kite/.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md

Phase 14: Selective Analysis and Byte Cache - Context

Gathered: 2026-06-23 Status: Ready for planning

## Phase Boundary

Phase 14 lets users analyze provider-owned cloud files without importing them into local DocuVault documents. Users can analyze individual files, selected items, folder scopes, or whole connections through observable, controllable background queues. DocuVault hydrates provider bytes only for opening, preview, or analysis; retained bytes live in a private bounded cache with owner-scoped metadata, quota accounting, and safe eviction. Phase 15 owns unified keyword/semantic search over persisted analysis output, and Phase 16 owns provider change tracking/delta reliability beyond the staleness checks needed for analysis idempotency.

## Implementation Decisions

Analysis scope selection

• D-01: Analysis entry points use both rows and the shared browser toolbar. Single-file analysis appears on file rows; multi-select, folder-scope, and whole-connection analysis appear in the shared StorageBrowser toolbar.
• D-02: Estimate/review is threshold-based. Small obvious jobs may start directly; item count or provider-reported byte thresholds trigger an estimate step. Whole-connection analysis always shows an estimate unless trivially small.
• D-03: The estimate step shows total supported file count, total provider-reported bytes, unsupported/skipped item count, and a clear start action. It is not a full per-file include/exclude review UI in Phase 14.
• D-04: Folder analysis asks each time whether to analyze the current folder only or recurse through descendants. Whole-connection analysis is inherently recursive.

Job progress and controls

• D-05: Progress presentation uses an aggregate indicator plus an expandable queue/item list. The browser stays quiet by default, but detailed per-item state is available.
• D-06: Analysis progress detail is user-configurable in Settings. The default is simplified labels; advanced users can opt into roadmap-level detail such as queued, downloading, extracting, classifying, indexed, cancelled, and failed.
• D-07: Analysis uses a queue model similar to the cloud upload queue, covering both download/cache hydration and scanning/classification stages.
• D-08: The queue displays separate indicators for download/cache hydration and scanning/analysis.
• D-09: Queue rows support per-item cancel, skip, and abort/stop controls where meaningful. Batch-level controls include Cancel everything.
• D-10: Error handling mirrors the existing upload queue pattern: a failed item can pause for user decision, then the user can retry, skip, abort/stop, or cancel the whole batch.
• D-11: Failure handling has a user setting for the default behavior. Initial options are pause whole queue and wait for input, or pause the failed item and continue the queue. The model must leave room for future default behaviors.

Byte cache, storage quota, and scan quota

• D-12: Temporary cached cloud bytes count against the user's storage quota while retained. Metadata, extracted text, topics, and semantic/index records do not count as stored bytes.
• D-13: Phase 14 must prepare a separate tier-ready scan quota seam distinct from storage quota. Free-tier users may eventually be limited to a small number of scanned files per day, such as 1 or 5.
• D-14: Cache limits are per-user with LRU eviction. Eviction removes only inactive cached bytes and must preserve owner isolation.
• D-15: Cache limit is a tier-bounded user setting. Users choose a preferred cache limit up to a tier/admin-defined maximum.
• D-16: Eviction protects active jobs and open previews. Bytes used by running download, analysis, or preview work are pinned until no longer active.
• D-17: Cached bytes are stored in MinIO with a cache metadata table. Object keys remain UUID-based/private; PostgreSQL tracks owner, connection, cloud item/provider reference, version or etag, size, active/pin state, last access, and eviction metadata.

Idempotency and reanalysis

• D-18: If a cloud item has already been analyzed and its provider version or etag is unchanged, DocuVault skips re-download/reclassification and reports it as already current/indexed.
• D-19: If the provider lacks reliable etag/version data, use a fallback metadata fingerprint from provider item id, size, modified time, and content type before download.
• D-20: When bytes are already hydrated for analysis, compute and store a content hash. Do not download bytes solely to compute a hash.
• D-21: If an item changes externally during analysis, finish the current job safely, then compare version/fingerprint and mark the resulting analysis stale if it no longer matches the provider state.
• D-22: Retry after failure restarts the full item from the beginning. This favors predictable behavior over partial-stage retry optimization.
• D-23: Mixed batches process only actionable items. Current items are skipped as already indexed/current, stale and failed-prior items are queued, and unsupported items are reported without blocking actionable work.

Codex's Discretion

• Choose exact status labels, icons, thresholds, queue wording, and accessible control copy while preserving user-configurable detail and the decisions above.
• Choose the storage schema names and service decomposition consistent with existing owner-scoped cloud services, MinIO storage patterns, and Celery retry conventions.
• Choose initial defaults for threshold values, cache limit values, and failure behavior, provided they are tier/config ready and covered by tests.

<canonical_refs>

Canonical References

Downstream agents MUST read these before planning or implementing.

Milestone and phase contracts

• .planning/ROADMAP.md - Phase 14 goal, requirements, success criteria, and Phase 15/16 boundaries.
• .planning/REQUIREMENTS.md - canonical ANALYZE-01 through ANALYZE-07 and CACHE-03 through CACHE-05 requirements; SEARCH and SYNC requirements remain later phases.
• .planning/PROJECT.md - virtual-local cloud storage model, privacy boundary, quota/tier direction, and shared-component architecture.
• .planning/phases/12-cloud-resource-foundation/12-CONTEXT.md - normalized cloud item metadata, byte availability, storage-quota distinction, and no-byte browse boundary inherited by Phase 14.
• .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md - truthful listing/freshness and provider-neutral correctness constraints.
• .planning/phases/13-virtual-local-cloud-operations/13-CONTEXT.md - authorized cloud open/preview, shared browser operations, queue semantics, provider mutation safeguards, and Phase 14 byte-cache boundary.
• AGENTS.md - non-negotiable shared-module, security, testing, documentation, environment, and Git rules.

Backend cloud, cache, and analysis surfaces

• backend/db/models.py - CloudItem, CloudItemTopic, and reserved analysis fields; new cache/scan quota schema should preserve owner and connection boundaries.
• backend/storage/cloud_base.py - provider-neutral resource/capability contract and normalized action vocabulary.
• backend/storage/cloud_backend_factory.py - adapter construction boundary where provider credentials are consumed.
• backend/services/cloud_items.py - existing owner-scoped cloud metadata reconciliation and lookup patterns.
• backend/services/cloud_operations.py - Phase 13 cloud operation orchestration and normalized provider error handling.
• backend/tasks/cloud_tasks.py - Celery bridge, retry sentinel pattern, owner revalidation, and credential decryption inside workers.
• backend/tasks/document_tasks.py - existing extraction/classification task pattern and retry harness; Phase 14 should not force provider-owned items into local Document rows.
• backend/services/extractor.py - byte-to-text extraction used by both local documents and future cloud analysis jobs.
• backend/services/classifier.py - AI topic classification path to reuse for cloud item analysis without leaking provider bytes or credentials.
• backend/api/cloud/operations.py - authorized cloud content routes and no-credential response boundary.
• backend/api/cloud/schemas.py - credential-free whitelisted response schemas.
• backend/storage/minio_backend.py - UUID/private object key and byte storage patterns to mirror for cached cloud bytes.

Frontend shared browser and state

• frontend/src/components/storage/StorageBrowser.vue - single shared browser; analysis row actions, toolbar actions, aggregate status, and expandable queue UI belong here rather than in a parallel cloud grid.
• frontend/src/views/CloudFolderView.vue - thin cloud data-provider integration for cloud browse, upload queue, open/preview, and future analysis handlers.
• frontend/src/views/FileManagerView.vue - local interaction reference for row actions and toolbar behavior.
• frontend/src/stores/cloudConnections.js - cloud browse, capability, freshness, health, byte-availability, and session navigation state.
• frontend/src/api/cloud.js and frontend/src/api/client.js - cloud API client surface and authenticated fetch behavior.
• frontend/src/views/SettingsView.vue - likely host for progress-detail, failure-behavior, and cache-limit settings.

Verification patterns

• backend/tests/test_cloud.py - owner-scoped cloud API integration patterns.
• backend/tests/test_cloud_items.py - cloud item persistence, analysis placeholder, reconciliation, and no-quota metadata tests.
• backend/tests/test_cloud_security.py - owner/admin/credential/SSRF/no-byte negative contract.
• backend/tests/test_cloud_backends.py - provider contract test patterns.
• frontend/src/components/storage/__tests__/StorageBrowser.cloud-queue.test.js - existing shared queue behavior to extend for analysis.
• frontend/src/components/storage/__tests__/StorageBrowser.capabilities.test.js - shared action rendering and disabled-capability coverage.
• frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js - rendered cloud browser flow coverage.

</canonical_refs>

<code_context>

Existing Code Insights

Reusable Assets

• CloudItem: already persists owner, connection, provider item id, parent, name, kind, content type, provider size, etag/version, extracted text, analysis status, semantic index status, and topic links without a local Document row.
• StorageBrowser.vue: already owns local/cloud rows, upload queue UI, toolbar/browser layout, and operation event emissions; Phase 14 should extend it rather than create another grid.
• CloudFolderView.vue: already maps opaque provider_item_id, server freshness, byte availability, upload queue behavior, and cloud open/preview handlers into StorageBrowser.
• cloud_tasks.py and document_tasks.py: provide established Celery sync-to-async bridge patterns, retry sentinels, owner revalidation, and worker-side credential/DB lookup boundaries.
• extractor.py and classifier.py: existing text extraction and AI classification services should be reused for cloud analysis output, adapted to CloudItem instead of local Document.
• MinIOBackend: existing private UUID object-key pattern is the right storage precedent for retained cloud byte cache objects.

Established Patterns

• Views own stores and route params; smart components own interaction/layout and emit upward.
• Cloud provider IDs remain opaque and navigation/mutations use connection UUID plus provider_item_id; never derive provider paths in Vue.
• Provider-owned bytes are authoritative. Cached bytes are temporary and distinct from permanent local imports.
• Credentials are decrypted only at provider/task boundaries and never travel through broker payloads, API responses, logs, browser storage, or planning artifacts.
• Service modules raise domain errors or ValueError; routers translate to HTTPException/typed responses.
• Metadata-only browse and refresh never download bytes or mutate quotas; Phase 14 byte cache is the first place retained cloud bytes affect quota.

Integration Points

• Add API routes under backend/api/cloud/ for estimate, enqueue analysis, list job state, cancel/skip/abort/retry controls, and cache settings.
• Add services for cloud analysis orchestration, scan quota checks, byte-cache metadata, LRU eviction, active pinning, and idempotency/fingerprint decisions.
• Add Celery tasks for folder/connection expansion, provider byte hydration, text extraction, classification, cache eviction, and status updates.
• Extend CloudItem or related tables with durable analysis job/cache/version/fingerprint data while preserving existing owner and connection indexes.
• Extend StorageBrowser, CloudFolderView, cloudConnections store, and Settings to expose analysis actions, progress detail preference, default failure behavior, and tier-bounded cache limit.
• Add backend/frontend tests for ownership, admin-negative access, credential secrecy, cache isolation, quota accounting, scan quota readiness, duplicate jobs, cancellation, failure defaults, and eviction pinning.

</code_context>

## Specific Ideas

• The analysis queue should feel similar to the existing upload queue, including pause-and-decide error handling and a global cancel-all action.
• Users can choose simplified versus detailed analysis progress in Settings; simplified is the default.
• Failure defaults are also user-configurable, with an extensible settings model so future behaviors can be added without reshaping the queue engine.
• The project should be ready for tiered scan limits, including free-tier daily scan quotas, even if billing/subscription tiers are not implemented in Phase 14.
• Content hashes are useful, but only when bytes are already being read for analysis; hashing must not become a hidden byte-download trigger.

## Deferred Ideas

• Unified keyword/semantic search over local and analyzed cloud documents remains Phase 15.
• Provider delta feeds, external-delete handling, and broader refresh/change reliability remain Phase 16.
• Permanent local import or pinning of provider files remains future IMPORT-01; Phase 14 cache is temporary and bounded.
• Specific paid tier definitions and billing workflows remain future scope, though Phase 14 should leave clean scan/cache quota seams.

---

Phase: 14-selective-analysis-and-byte-cache Context gathered: 2026-06-23