11 KiB
Phase 14: Selective Analysis and Byte Cache - Discussion Log
Audit trail only. Do not use as input to planning, research, or execution agents. Decisions are captured in CONTEXT.md - this log preserves the alternatives considered.
Date: 2026-06-23 Phase: 14-selective-analysis-and-byte-cache Areas discussed: Analysis Scope Selection, Job Progress and Controls, Byte Cache Semantics, Idempotency and Reanalysis
Analysis Scope Selection
| Option | Description | Selected |
|---|---|---|
| Rows + toolbar | Single-file actions live on rows; multi-select/folder/connection analysis lives in the shared browser toolbar. | yes |
| Toolbar only | All analysis starts from selected items or current location, keeping rows quieter. | |
| Rows only | Simple for individual files, but awkward for folder or whole-connection analysis. |
User's choice: Rows + toolbar. Notes: Single-file analysis belongs on file rows. Multi-select, folder, and whole-connection analysis belong in the shared browser toolbar.
| Option | Description | Selected |
|---|---|---|
| Before folder/connection jobs only | Individual files and explicit multi-file selections start directly; folders and whole connections require estimate review. | |
| Before every job | Even one file shows a confirmation with byte count and estimated cost/time. | |
| Only for large scopes | Start small jobs directly, but estimate when item count or bytes cross a threshold. | yes |
User's choice: Only for large scopes. Notes: Threshold-based estimate/review protects users from unexpectedly large operations.
| Option | Description | Selected |
|---|---|---|
| Count + bytes + unsupported items | Show total files, provider-reported bytes, skipped/unsupported file types, and a clear start button. | yes |
| Detailed file list | Show every discovered item before starting, with per-file include/exclude. | |
| Cost-forward estimate | Emphasize AI cost/time estimates in addition to count and bytes. |
User's choice: Count + bytes + unsupported items. Notes: Phase 14 should not become a full file-by-file review UI.
| Option | Description | Selected |
|---|---|---|
| Recursive by default | Analyzing a folder includes supported files in all descendants, with threshold estimate protection. | |
| Current folder only | Only files directly inside the folder are analyzed unless the user opens subfolders. | |
| Ask each time | Every folder analysis asks current-only vs recursive. | yes |
User's choice: Ask each time. Notes: Whole-connection analysis remains inherently recursive.
Job Progress and Controls
| Option | Description | Selected |
|---|---|---|
| Aggregate + expandable item list | Show overall progress in toolbar/status area, with expandable per-item list for details. | yes |
| Per-item always visible | Every queued/running/failed item is visible in the main browser surface. | |
| Aggregate only | Show one progress indicator and surface only failures. |
User's choice: Aggregate + expandable item list. Notes: Keep the main browser clean but allow detailed inspection.
| Option | Description | Selected |
|---|---|---|
| Roadmap statuses exactly | Show queued, downloading, extracting, classifying, indexed, cancelled, failed. | |
| Simplified statuses | Show waiting, working, done, failed. | yes |
| Verbose technical statuses | Include cache hit/miss, provider fetch, extraction, topic assignment, semantic placeholders. |
User's choice: User-configurable detail with simplified default. Notes: Settings should let the user choose how much analysis progress detail to see. Simplified labels are the default; advanced detail can expose roadmap-level states.
| Option | Description | Selected |
|---|---|---|
| Cancel queued work only | Users can cancel jobs/items that have not started; running stages finish or fail naturally. | |
| Cancel queued + running downloads | Queued work cancels cleanly, and active provider downloads are interrupted where feasible. | |
| Cancel everything immediately | Attempt to interrupt every running stage, including extraction/classification. | |
| Queue controls with download + scan indicators | Similar queue/error model to upload; per-item cancel/skip/abort and global cancel everything. | yes |
User's choice: Queue controls with download and scanning indicators. Notes: The queue should cover both download/cache hydration and scanning/classification. It should offer per-file or queued-item cancel, skip, and abort controls plus a cancel-everything action.
| Option | Description | Selected |
|---|---|---|
| Pause queue on failure | Current item failure pauses the queue for retry/skip/abort. | yes |
| Continue queue automatically | Failed item is marked failed and later items continue; user retries failures later. | |
| Stop whole job | First failure aborts the remaining queue. |
User's choice: Pause queue on failure, with configurable default behavior. Notes: User added that default failure behavior should be configurable. Options include pause whole queue and wait for input, or pause failed item and continue the queue. The settings model should remain extensible for future behaviors.
Byte Cache Semantics
| Option | Description | Selected |
|---|---|---|
| Yes, while retained | Cached bytes count toward quota until evicted; metadata/extracted text does not. | yes |
| No, never | Cache is operational infrastructure and does not affect user quota. | |
| Only persistent cache | Active job bytes do not count, retained bytes after completion count. |
User's choice: Yes, while retained. Notes: User also requested preparation for separate scan quota, such as free-tier users scanning 1 or 5 files per day.
| Option | Description | Selected |
|---|---|---|
| Per-user cache limit with LRU eviction | Each user has a cache byte cap; least-recently-used inactive cached bytes evict first. | yes |
| Global cache limit only | One system-wide cap; eviction can affect any user but serving still respects ownership. | |
| Per-connection cache limit | Each cloud connection has its own cap. |
User's choice: Per-user cache limit with LRU eviction. Notes: Eviction applies only to inactive cached bytes.
| Option | Description | Selected |
|---|---|---|
| Tier-bounded user setting | Users choose a cache limit up to their tier maximum. | yes |
| Admin/system only | Users see cache usage but cannot change the limit. | |
| Fixed default for now | Implement one default limit and leave configuration for later. |
User's choice: Tier-bounded user setting. Notes: Admin/tier config defines the ceiling.
| Option | Description | Selected |
|---|---|---|
| Active jobs + open previews | Running analysis/download/preview bytes are pinned; everything else is LRU-evictable. | yes |
| Active jobs only | Previews can be evicted after response streaming begins. | |
| Recent successful analysis too | Keep newly analyzed files for a short grace window even if inactive. |
User's choice: Active jobs + open previews. Notes: Active use must not be interrupted by eviction.
| Option | Description | Selected |
|---|---|---|
| MinIO with cache metadata table | Store cached bytes in MinIO with UUID keys; track owner/connection/item/version/pin/size in PostgreSQL. | yes |
| Filesystem cache on worker/backend disk | Simpler, but weaker for horizontal scaling and cleanup. | |
| Database large objects | Keeps DB transactional, but poor fit for file bytes and quota accounting. |
User's choice: MinIO with cache metadata table. Notes: Preserve UUID/private key pattern and PostgreSQL metadata for quota/eviction.
Idempotency and Reanalysis
| Option | Description | Selected |
|---|---|---|
| Skip as already current | Do not re-download or reclassify; show already indexed/current. | yes |
| Allow manual force reanalysis | Default skip, but offer a force option. | |
| Always reanalyze | User request always downloads and reclassifies even when unchanged. |
User's choice: Skip as already current. Notes: Unchanged etag/version should prevent duplicate analysis.
| Option | Description | Selected |
|---|---|---|
| Use fallback fingerprint | Combine provider item id, size, modified time, and content type; reanalyze when it changes. | yes |
| Always reanalyze missing-etag items | Safest freshness-wise, but expensive. | |
| Never consider them current | Mark as needs manual refresh until provider supports versioning. |
User's choice: Use fallback fingerprint, plus content hash when bytes are already hydrated. Notes: User suggested storing a hash. Decision: do not download bytes solely for hashing; compute content hash during analysis because bytes are already present.
| Option | Description | Selected |
|---|---|---|
| Finish then mark stale if changed | Complete current job safely, compare version/fingerprint, and mark stale if changed. | yes |
| Abort current item immediately | Stop work when a newer provider version is detected. | |
| Ignore until next refresh | Current job writes results; Phase 16 later handles staleness. |
User's choice: Finish then mark stale if changed. Notes: Analysis remains non-destructive and provider-owned file is never mutated.
| Option | Description | Selected |
|---|---|---|
| Retry failed stage only when possible | Reuse cached bytes if valid; otherwise restart from download. | |
| Restart full item every time | Simpler and predictable, but repeats work. | yes |
| Retry whole batch | Retry all failed/skipped items together. |
User's choice: Restart full item every time. Notes: Predictability is preferred over partial-stage optimization.
| Option | Description | Selected |
|---|---|---|
| Process only actionable items | Skip current, queue stale and failed-prior, report unsupported. | yes |
| Ask before starting | Show categories and require user to choose which groups to include. | |
| Queue everything | Current items still get queued and skipped individually. |
User's choice: Process only actionable items. Notes: Unsupported/current items should be reported without blocking stale or failed-prior items.
Codex's Discretion
- Choose exact labels, icons, thresholds, endpoint shapes, schema names, retry mechanics, and default values consistent with the decisions captured in
14-CONTEXT.md.
Deferred Ideas
- Unified smart search remains Phase 15.
- Provider delta/change tracking reliability remains Phase 16.
- Permanent import/pinning and paid tier/billing implementation remain future scope.