diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index ca4fc5f..2b00780 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -111,6 +111,20 @@ Plans: **Depends on:** Phase 13 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05 +**Plans:** 0/9 plans complete +**Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08. + +Plans: + +- [ ] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives. +- [ ] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers. +- [ ] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API. +- [ ] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation. +- [ ] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation. +- [ ] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle. +- [ ] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls. +- [ ] 14-08-PLAN.md — Add Settings controls for cache limit, progress detail, and failure behavior. +- [ ] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push. **Success Criteria:** diff --git a/.planning/STATE.md b/.planning/STATE.md index 10ef1ce..1fc755a 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,17 +2,17 @@ gsd_state_version: 1.0 milestone: v0.3 milestone_name: Reimagining Cloud Storage integration -current_phase: 13 -current_phase_name: virtual-local-cloud-operations -status: complete -stopped_at: Phase 14 context gathered -last_updated: "2026-06-23T12:04:31.598Z" +current_phase: 14 +current_phase_name: selective-analysis-and-byte-cache +status: planned +stopped_at: Phase 14 planned +last_updated: "2026-06-23T00:00:00.000Z" last_activity: 2026-06-23 last_activity_desc: Phase 13 complete, v0.3.0 shipped progress: total_phases: 6 completed_phases: 3 - total_plans: 21 + total_plans: 30 completed_plans: 21 percent: 50 --- @@ -20,15 +20,15 @@ progress: # Project State **Project:** DocuVault -**Status:** Phase 13 complete — ready to begin Phase 14 +**Status:** Phase 14 planned — ready to execute Plan 14-01 **Last Updated:** 2026-06-23 ## Current Position -Phase: 13 (virtual-local-cloud-operations) — COMPLETE -Plan: 11 of 11 -Status: Phase 13 complete. Next: Phase 14 (Selective Analysis and Byte Cache) -Last activity: 2026-06-23 — Phase 13 complete, v0.3.0 shipped +Phase: 14 (selective-analysis-and-byte-cache) — PLANNED +Plan: 0 of 9 +Status: Phase 14 planned. Next: execute 14-01 red contract tests +Last activity: 2026-06-23 — Phase 14 plan files created ## Phase Status @@ -37,7 +37,7 @@ Last activity: 2026-06-23 — Phase 13 complete, v0.3.0 shipped | 12. Cloud Resource Foundation | CONN-04, CLOUD-01, CLOUD-08, CACHE-01, CACHE-02, SYNC-01 | **Complete** | | 12.1 Fix Nextcloud Root Listing and Sync Visibility | CONN-04, CLOUD-01, CACHE-01, SYNC-01 | **Complete** | | 13. Virtual-Local Cloud Operations | CONN-01..03, CLOUD-02..07, CLOUD-09 | **Complete** | -| 14. Selective Analysis and Byte Cache | ANALYZE-01..07, CACHE-03..05 | **Not started** | +| 14. Selective Analysis and Byte Cache | ANALYZE-01..07, CACHE-03..05 | **Planned** | | 15. Unified Smart Search | SEARCH-01..07 | **Not started** | | 16. Change Tracking and Reliability | SYNC-02..04 | **Not started** | @@ -91,7 +91,7 @@ Last activity: 2026-06-23 — Phase 13 complete, v0.3.0 shipped - v0.2 completed: UI overhaul, admin panel rearchitecture, responsive layout, codebase quality (2026-06-17) - v0.2 archived to `.planning/milestones/v0.2-ROADMAP.md` - v0.3 in progress: Phase 12 (cloud resource foundation) complete 2026-06-21; Phase 12.1 (Nextcloud fix) complete; Phase 13 (virtual-local cloud operations) complete 2026-06-23 — v0.3.0 shipped -- Phase 14 (selective analysis and byte cache) is next +- Phase 14 (selective analysis and byte cache) planned 2026-06-23 ### Open Questions @@ -103,16 +103,16 @@ None. ## Session Continuity -**Stopped at:** Phase 14 context gathered +**Stopped at:** Phase 14 planned _Updated at each phase transition._ | Field | Value | |---|---| | Last session | 2026-06-23T12:04:31.592Z | -| Next action | Begin Phase 14 (Selective Analysis and Byte Cache) | +| Next action | Execute 14-01 red contract tests | | Pending decisions | None | -| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md | +| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md | ## Decisions diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md new file mode 100644 index 0000000..2e11a1d --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md @@ -0,0 +1,106 @@ +--- +phase: "14" +plan: "01" +type: execute +wave: 0 +depends_on: [] +files_modified: + - backend/tests/test_cloud_analysis_contract.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_analysis_api.py + - frontend/src/components/storage/__tests__/StorageBrowser.analysis-queue.test.js + - frontend/src/stores/__tests__/cloudConnections.analysis.test.js +autonomous: true +requirements: + - ANALYZE-01 + - ANALYZE-02 + - ANALYZE-03 + - ANALYZE-04 + - ANALYZE-05 + - ANALYZE-06 + - ANALYZE-07 + - CACHE-03 + - CACHE-04 + - CACHE-05 +--- + + +Create the Phase 14 executable test contract before implementation. Establish red backend and frontend tests for analysis scope selection, estimates, duplicate-version skips, cache ownership/quota semantics, cancellation/retry controls, and the shared-browser queue surface. + +Scope fence: add tests and fixtures only. Do not implement production code in this plan except minimal test helpers/fakes. + + + +@.planning/ROADMAP.md +@.planning/REQUIREMENTS.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@backend/tests/test_cloud_items.py +@backend/tests/test_cloud_security.py +@frontend/src/components/storage/__tests__/StorageBrowser.cloud-queue.test.js + + + + + + Task 1: Add backend red tests for analysis and cache contracts + backend/tests/test_cloud_analysis_contract.py, backend/tests/test_cloud_cache.py, backend/tests/test_cloud_analysis_api.py + + - backend/tests/test_cloud_items.py + - backend/tests/test_cloud_security.py + - backend/tests/test_cloud_mutations.py + - backend/db/models.py + + + Add focused tests using fake cloud items/connections and fake provider adapters. Cover individual-file, selection, folder, and connection estimate/enqueue requests; whole-connection estimate requirement; recursive folder choice; unsupported item reporting; duplicate unchanged version skip without provider byte fetch; owner/admin-negative access; cache entry ownership; cache object key secrecy; quota increment/decrement expectations; active pin eviction protection; retry and cancel status transitions. Mark these tests red against missing modules/endpoints rather than weakening assertions. + + + - Tests name the exact ANALYZE/CACHE requirements they protect. + - Tests assert provider bytes are not downloaded during estimate or duplicate-current checks. + - Tests assert provider mutation methods are never invoked by analysis paths. + - Running the new backend tests fails for missing Phase 14 implementation, not syntax/import errors in the tests themselves once modules exist. + + cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py tests/test_cloud_analysis_api.py + + + + Task 2: Add frontend red tests for shared-browser analysis UX + frontend/src/components/storage/__tests__/StorageBrowser.analysis-queue.test.js, frontend/src/stores/__tests__/cloudConnections.analysis.test.js + + - frontend/src/components/storage/StorageBrowser.vue + - frontend/src/views/CloudFolderView.vue + - frontend/src/stores/cloudConnections.js + - frontend/src/components/storage/__tests__/StorageBrowser.cloud-queue.test.js + + + Add tests proving row-level file analysis action, toolbar analysis for multi-select/folder/current connection, estimate modal thresholds, aggregate progress, expandable item list, simplified vs detailed labels, retry/skip/cancel controls, and Settings-backed default failure behavior. Tests must assert that `CloudFolderView` passes props and handles emits without owning layout, and that no parallel file grid is introduced. + + + - Tests fail because analysis UI/store code is absent, not because of brittle selectors. + - Shared browser remains the component under test for browser layout and controls. + - Store tests cover status translation and API client calls without using localStorage/sessionStorage for credentials. + + cd frontend && npm run test -- StorageBrowser.analysis-queue.test.js cloudConnections.analysis.test.js + + + + + +| Threat ID | Threat | Required test evidence | +|-----------|--------|------------------------| +| T-14-01 | Analysis IDOR | Foreign user cannot estimate/enqueue/status/cancel/retry another user's connection or item | +| T-14-02 | Credential/object-key disclosure | API/audit responses never include credentials, raw provider URLs, or MinIO object keys | +| T-14-03 | Hidden provider mutation | Fake adapter mutation method counters remain zero during analysis | +| T-14-04 | Hidden byte download | Estimate and already-current checks do not call provider byte fetch | +| T-14-05 | Cache quota corruption | Tests require atomic quota increment on retain and decrement on eviction | + + + +1. `cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py tests/test_cloud_analysis_api.py` +2. `cd frontend && npm run test -- StorageBrowser.analysis-queue.test.js cloudConnections.analysis.test.js` +3. Confirm failures are expected missing-implementation failures only. + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-01-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-02-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-02-PLAN.md new file mode 100644 index 0000000..8b0acdb --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-02-PLAN.md @@ -0,0 +1,87 @@ +--- +phase: "14" +plan: "02" +type: execute +wave: 1 +depends_on: + - "14-01" +files_modified: + - backend/migrations/versions/0007_cloud_analysis_cache.py + - backend/db/models.py + - backend/services/cloud_analysis_versioning.py + - backend/services/cloud_analysis_settings.py + - backend/tests/test_cloud_analysis_contract.py + - backend/tests/test_cloud_cache.py +autonomous: true +requirements: + - ANALYZE-04 + - ANALYZE-06 + - CACHE-04 + - CACHE-05 +--- + + +Add durable Phase 14 schema and shared helpers for cache entries, analysis jobs, per-item status, version-key idempotency, and user/tier-ready analysis settings. + +Scope fence: no provider byte download, Celery processing, or frontend UI in this plan. + + + +@.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@backend/db/models.py +@backend/migrations/versions/0006_cloud_resource_foundation.py +@backend/services/cloud_items.py + + + + + + Task 1: Add migration and ORM models + backend/migrations/versions/0007_cloud_analysis_cache.py, backend/db/models.py + + - backend/migrations/versions/0006_cloud_resource_foundation.py + - backend/db/models.py + + + Create the next Alembic revision for `cloud_byte_cache_entries`, `cloud_analysis_jobs`, `cloud_analysis_job_items`, and either owner-scoped analysis settings or extensions to the existing settings model. Include strict user/connection/item foreign keys, unique cache version key constraints, LRU eviction indexes, status fields, controlled error fields, aggregate counters, and reversible downgrade. Models must mirror the migration and preserve the existing `CloudItem` identity model. + + + - Cache entries are scoped by user, connection, cloud item, and version key. + - Job and job-item rows can represent file, selection, folder, and connection analysis. + - Settings support progress detail, failure behavior, and tier-capped cache limit preference. + - Downgrade drops only Phase 14 objects. + + cd backend && pytest -q tests/test_cloud_cache.py -k "schema or model" + + + + Task 2: Add version-key and settings helpers + backend/services/cloud_analysis_versioning.py, backend/services/cloud_analysis_settings.py, backend/tests/test_cloud_analysis_contract.py, backend/tests/test_cloud_cache.py + + - backend/services/cloud_items.py + - backend/db/models.py + - .planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md + + + Implement one shared version-key helper using version, etag, fallback metadata fingerprint, and optional post-hydration content hash. Implement settings helpers that return defaults, validate enum values, enforce cache limit bounds, and leave a clean tier/admin maximum seam. Services raise `ValueError` or domain exceptions only. + + + - Duplicate-current and cache lookup code can consume one canonical version key. + - Content hash is never required before bytes are hydrated. + - Invalid setting values are rejected by service validation. + - `rg "HTTPException" backend/services/cloud_analysis_versioning.py backend/services/cloud_analysis_settings.py` returns no matches. + + cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py -k "version_key or settings" + + + + + +1. `cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py` +2. Run migration upgrade/downgrade against the PostgreSQL integration environment. +3. `rg "HTTPException" backend/services/cloud_analysis_versioning.py backend/services/cloud_analysis_settings.py` + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-02-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md new file mode 100644 index 0000000..242a445 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md @@ -0,0 +1,99 @@ +--- +phase: "14" +plan: "03" +type: execute +wave: 2 +depends_on: + - "14-02" +files_modified: + - backend/services/cloud_byte_cache.py + - backend/api/cloud/cache.py + - backend/api/cloud/schemas.py + - backend/api/cloud/__init__.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_security.py +autonomous: true +requirements: + - CACHE-03 + - CACHE-04 + - CACHE-05 +--- + + +Implement the owner-scoped byte cache service and read/control API: cache lookup, retain, pin/release, per-user LRU eviction, quota accounting, settings update, and metadata-safe cache status responses. + +Scope fence: do not wire analysis workers or open/preview hydration yet. This plan builds the cache primitive those paths will call. + + + +@.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@backend/storage/minio_backend.py +@backend/services/cloud_items.py +@backend/db/models.py +@backend/api/cloud/schemas.py + + + + + + Task 1: Build cache service with quota and pinning semantics + backend/services/cloud_byte_cache.py, backend/tests/test_cloud_cache.py + + - backend/storage/minio_backend.py + - backend/services/quota.py + - backend/db/models.py + - backend/tests/test_quota.py + + + Add cache service functions to resolve owned cache entries, reserve quota atomically before retaining bytes, store bytes using UUID/private object keys, record metadata, reuse matching non-evicted version entries, pin/release active job or preview use, update last access, evict inactive LRU entries per user limit, decrement quota when retained bytes are evicted, and delete MinIO objects best-effort after DB state is consistent. Protect entries with `pin_count > 0` or `active_job_count > 0`. + + + - Cache retention increments quota and eviction decrements quota with atomic quota helpers. + - Eviction never selects another user's entry and never evicts pinned/active entries. + - Object keys are generated UUID/private keys and never include provider filenames. + - Service functions raise domain errors or `ValueError`, never `HTTPException`. + + cd backend && pytest -q tests/test_cloud_cache.py + + + + Task 2: Add cache settings/status API + backend/api/cloud/cache.py, backend/api/cloud/schemas.py, backend/api/cloud/__init__.py, backend/tests/test_cloud_cache.py, backend/tests/test_cloud_security.py + + - backend/api/cloud/connections.py + - backend/api/cloud/operations.py + - backend/api/cloud/schemas.py + - backend/tests/test_cloud_security.py + + + Add credential-free endpoints for current cache usage/settings and updating the user's preferred cache limit/progress/failure settings. Use explicit Pydantic fields only. Return aggregate bytes/counts and tier cap, not object keys. Enforce `get_regular_user`, owner scope, and rate limiting consistent with existing cloud routes. + + + - Admin users are blocked. + - Responses contain no `object_key`, credentials, raw provider URLs, or extracted text. + - Invalid cache limits above the tier/admin cap return a controlled validation response. + - Security tests cover foreign user and admin-negative cases. + + cd backend && pytest -q tests/test_cloud_cache.py tests/test_cloud_security.py -k "cache" + + + + + +| Threat ID | Threat | Mitigation | +|-----------|--------|------------| +| T-14-06 | Cross-user cache access | All cache entry queries include user_id and connection_id; negative tests | +| T-14-07 | Quota race | Existing atomic quota update helpers required for retain/evict deltas | +| T-14-08 | Object key leakage | API schemas omit object_key and tests assert absence | +| T-14-09 | Active work eviction | Pin/active counters in selection predicate and regression tests | + + + +1. `cd backend && pytest -q tests/test_cloud_cache.py` +2. `cd backend && pytest -q tests/test_cloud_security.py -k "cache"` +3. `rg "object_key|credentials_enc" backend/api/cloud/cache.py backend/api/cloud/schemas.py` + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-03-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-04-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-04-PLAN.md new file mode 100644 index 0000000..85a717e --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-04-PLAN.md @@ -0,0 +1,93 @@ +--- +phase: "14" +plan: "04" +type: execute +wave: 3 +depends_on: + - "14-02" + - "14-03" +files_modified: + - backend/services/cloud_analysis.py + - backend/api/cloud/analysis.py + - backend/api/cloud/schemas.py + - backend/api/cloud/__init__.py + - backend/tests/test_cloud_analysis_api.py + - backend/tests/test_cloud_analysis_contract.py + - backend/tests/test_cloud_security.py +autonomous: true +requirements: + - ANALYZE-01 + - ANALYZE-02 + - ANALYZE-03 + - ANALYZE-05 + - ANALYZE-06 + - ANALYZE-07 +--- + + +Implement Phase 14 analysis estimate, enqueue, status, cancellation, skip, and retry APIs backed by durable job rows and idempotency checks. + +Scope fence: enqueue and control jobs only. Actual provider byte hydration/extraction/classification is Plan 05. + + + +@.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@backend/services/cloud_items.py +@backend/services/cloud_analysis_versioning.py +@backend/services/cloud_analysis_settings.py +@backend/api/cloud/operations.py + + + + + + Task 1: Add analysis orchestration service + backend/services/cloud_analysis.py, backend/tests/test_cloud_analysis_contract.py + + - backend/services/cloud_items.py + - backend/services/cloud_analysis_versioning.py + - backend/db/models.py + + + Implement owner-scoped functions for estimating file/selection/folder/connection scopes, expanding known folder descendants from metadata where available, reporting partial estimates when provider expansion is incomplete, creating analysis jobs and job items, skipping unsupported/current items, and rejecting duplicate active jobs for the same user/item/version. Folder analysis must preserve current-only vs recursive choice; whole-connection analysis is recursive. Estimates never download bytes. + + + - File, selection, folder, and connection scope requests create correct job metadata. + - Whole-connection jobs require estimate acknowledgement unless trivially small by configured thresholds. + - Current unchanged items are marked `already_current` and do not enqueue byte work. + - Provider mutation methods are never called. + + cd backend && pytest -q tests/test_cloud_analysis_contract.py + + + + Task 2: Add analysis API routes and typed schemas + backend/api/cloud/analysis.py, backend/api/cloud/schemas.py, backend/api/cloud/__init__.py, backend/tests/test_cloud_analysis_api.py, backend/tests/test_cloud_security.py + + - backend/api/cloud/operations.py + - backend/api/cloud/schemas.py + - backend/tests/test_cloud_security.py + + + Add routes under `/api/cloud/connections/{connection_id}/analysis` for estimate, enqueue/start, job status/list, cancel item, cancel batch, skip failed/queued item, and retry failed item. Use explicit request schemas and credential-free response schemas. Convert service domain errors into typed JSON bodies with stable `kind`/`reason` fields. + + + - Routes use `get_regular_user` and owner-scoped connection/item resolution. + - Status responses expose aggregate and per-item state without credentials, object keys, provider URLs, or raw provider errors. + - Retry creates a fresh item attempt from the beginning. + - Cancel queued work immediately and marks running work cancellation-requested for worker cooperation. + + cd backend && pytest -q tests/test_cloud_analysis_api.py tests/test_cloud_security.py -k "analysis" + + + + + +1. `cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_analysis_api.py` +2. `cd backend && pytest -q tests/test_cloud_security.py -k "analysis"` +3. Confirm `rg "download|get_object|put_object" backend/services/cloud_analysis.py backend/api/cloud/analysis.py` has no estimate-time byte fetch. + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-04-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-05-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-05-PLAN.md new file mode 100644 index 0000000..e09f6cf --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-05-PLAN.md @@ -0,0 +1,104 @@ +--- +phase: "14" +plan: "05" +type: execute +wave: 4 +depends_on: + - "14-03" + - "14-04" +files_modified: + - backend/tasks/cloud_analysis_tasks.py + - backend/celery_app.py + - backend/services/cloud_analysis_processing.py + - backend/services/cloud_byte_cache.py + - backend/services/classifier.py + - backend/tests/test_cloud_analysis_contract.py + - backend/tests/test_cloud_cache.py +autonomous: true +requirements: + - ANALYZE-04 + - ANALYZE-05 + - ANALYZE-06 + - ANALYZE-07 + - CACHE-03 + - CACHE-04 +--- + + +Implement Celery-backed cloud analysis processing: hydrate bytes on demand, retain/reuse cache entries, extract text, classify topics onto `CloudItem`, update per-item and aggregate states, and honor cancellation/retry semantics. + +Scope fence: do not build frontend controls in this plan. + + + +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@backend/tasks/cloud_tasks.py +@backend/tasks/document_tasks.py +@backend/services/extractor.py +@backend/services/classifier.py +@backend/storage/cloud_backend_factory.py +@backend/services/cloud_byte_cache.py + + + + + + Task 1: Add processing service for one job item + backend/services/cloud_analysis_processing.py, backend/services/cloud_byte_cache.py, backend/tests/test_cloud_analysis_contract.py, backend/tests/test_cloud_cache.py + + - backend/services/extractor.py + - backend/services/classifier.py + - backend/services/cloud_byte_cache.py + - backend/services/cloud_analysis_versioning.py + + + Implement a service that processes a single job item by revalidating owner/connection/item, checking cancellation, checking version-key currentness, reusing or hydrating cache bytes, extracting text, persisting `CloudItem.extracted_text`, classifying topics through existing classifier logic adapted for cloud items, updating status to indexed/failed/cancelled/stale, and releasing cache pins. Compute content hash only while bytes are already available. Never create a local `Document` row and never call provider mutation methods. + + + - Status transitions include downloading, extracting, classifying, indexed, cancelled, failed, stale. + - Unchanged version skips processing before provider byte fetch. + - Provider byte fetch occurs only for actionable analysis/open/preview work. + - Cache pins are released on success, failure, and cancellation. + + cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py -k "processing or idempotent or cancellation" + + + + Task 2: Add Celery task and retry harness + backend/tasks/cloud_analysis_tasks.py, backend/celery_app.py, backend/tests/test_cloud_analysis_contract.py + + - backend/tasks/cloud_tasks.py + - backend/tasks/document_tasks.py + - backend/celery_app.py + + + Add sync Celery tasks that bridge to async processing with the existing sentinel retry pattern. Task payloads include job id/item id only. Decrypt credentials inside the worker after owner revalidation. Support bounded transient provider/classifier retry, terminal auth failures, max-retry final failure status, and cooperative cancellation checks between stages. + + + - Celery broker payload contains no credentials, provider URLs, filenames, or bytes. + - `self.retry()` is raised only in the outer sync task layer. + - Max retry writes controlled failure status. + - Aggregate job counters update after item transitions. + + cd backend && pytest -q tests/test_cloud_analysis_contract.py -k "celery or retry or broker" + + + + + +| Threat ID | Threat | Mitigation | +|-----------|--------|------------| +| T-14-10 | Credentials in broker payload | IDs only in Celery task args; decrypt in worker | +| T-14-11 | Provider file mutation | Processing service uses content-read adapter path only and tests fake mutation counters | +| T-14-12 | Cache pin leak | finally block releases pins; tests cover failure/cancel paths | +| T-14-13 | Stale result presented as current | Final version/fingerprint comparison marks item stale if provider metadata changed | + + + +1. `cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py` +2. `rg "credentials|credentials_enc|file_bytes" backend/tasks/cloud_analysis_tasks.py` and confirm no broker payload exposure. +3. `rg "Document\\(" backend/services/cloud_analysis_processing.py backend/tasks/cloud_analysis_tasks.py` returns no local import creation. + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-05-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-06-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-06-PLAN.md new file mode 100644 index 0000000..c93b36d --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-06-PLAN.md @@ -0,0 +1,87 @@ +--- +phase: "14" +plan: "06" +type: execute +wave: 5 +depends_on: + - "14-03" + - "14-05" +files_modified: + - backend/api/cloud/operations.py + - backend/services/cloud_byte_cache.py + - backend/api/cloud/schemas.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_security.py + - frontend/src/views/__tests__/CloudFolderOpenPreview.test.js +autonomous: true +requirements: + - CACHE-03 + - CACHE-04 + - CACHE-05 +--- + + +Route cloud open/preview byte hydration through the Phase 14 byte cache so opening and previewing use the same bounded, owner-scoped, quota-counted cache lifecycle as analysis. + +Scope fence: do not change the shared browser analysis queue; this plan only updates existing content routes and their tests. + + + +@.planning/phases/13-virtual-local-cloud-operations/13-CONTEXT.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@backend/api/cloud/operations.py +@backend/services/cloud_byte_cache.py +@frontend/src/views/CloudFolderView.vue + + + + + + Task 1: Update backend open/preview/download routes to use cache leases + backend/api/cloud/operations.py, backend/services/cloud_byte_cache.py, backend/tests/test_cloud_cache.py, backend/tests/test_cloud_security.py + + - backend/api/cloud/operations.py + - backend/services/cloud_byte_cache.py + - backend/storage/cloud_backend_factory.py + + + Modify authorized cloud content routes so provider bytes are hydrated into the cache on demand, pinned during active streaming/preview, last-accessed timestamp is updated, and eviction runs after release when needed. Preserve binary-only preview enforcement and no raw provider URL exposure. Existing download fallback remains authorized through DocuVault, not a provider URL. + + + - Preview/open/download never serve bytes without owner-scoped connection/item resolution. + - Cache hits avoid provider byte fetch. + - Active preview/download cache entries are not evicted mid-response. + - Responses still omit object keys, credentials, and raw provider URLs. + + cd backend && pytest -q tests/test_cloud_cache.py tests/test_cloud_security.py -k "preview or download or cache" + + + + Task 2: Preserve frontend open/preview behavior against cached backend + frontend/src/views/__tests__/CloudFolderOpenPreview.test.js + + - frontend/src/views/CloudFolderView.vue + - frontend/src/api/cloud.js + - frontend/src/views/__tests__/CloudFolderOpenPreview.test.js + + + Update frontend tests only if route response metadata changes. Preserve the invariant that `CloudFolderView` calls `api.openCloudFile`/`previewCloudFile` and never opens provider URLs directly. Add assertions that cache-related response fields, if any, do not leak object keys. + + + - Existing open/preview tests remain green. + - No browser-to-provider URL path is introduced. + - Cache semantics remain backend-owned and do not require frontend object-key knowledge. + + cd frontend && npm run test -- CloudFolderOpenPreview.test.js + + + + + +1. `cd backend && pytest -q tests/test_cloud_cache.py tests/test_cloud_security.py -k "cache or preview or download"` +2. `cd frontend && npm run test -- CloudFolderOpenPreview.test.js` +3. `rg "window.open|provider_url|object_key" frontend/src/views/CloudFolderView.vue frontend/src/api/cloud.js backend/api/cloud/operations.py` + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-06-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-07-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-07-PLAN.md new file mode 100644 index 0000000..d738848 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-07-PLAN.md @@ -0,0 +1,93 @@ +--- +phase: "14" +plan: "07" +type: execute +wave: 6 +depends_on: + - "14-04" + - "14-05" +files_modified: + - frontend/src/api/cloud.js + - frontend/src/stores/cloudConnections.js + - frontend/src/views/CloudFolderView.vue + - frontend/src/components/storage/StorageBrowser.vue + - frontend/src/components/storage/__tests__/StorageBrowser.analysis-queue.test.js + - frontend/src/stores/__tests__/cloudConnections.analysis.test.js + - frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js +autonomous: true +requirements: + - ANALYZE-01 + - ANALYZE-02 + - ANALYZE-03 + - ANALYZE-04 + - ANALYZE-05 + - ANALYZE-06 +--- + + +Implement the shared-browser Phase 14 analysis experience: row action, toolbar actions, estimate dialog, aggregate progress, expandable queue, and per-item/batch controls using the backend analysis API. + +Scope fence: do not build Settings controls in this plan beyond consuming already-returned settings values. + + + +@.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md +@frontend/src/components/storage/StorageBrowser.vue +@frontend/src/views/CloudFolderView.vue +@frontend/src/stores/cloudConnections.js +@frontend/src/api/cloud.js + + + + + + Task 1: Add API/store analysis queue surface + frontend/src/api/cloud.js, frontend/src/stores/cloudConnections.js, frontend/src/stores/__tests__/cloudConnections.analysis.test.js + + - frontend/src/api/cloud.js + - frontend/src/stores/cloudConnections.js + - frontend/src/stores/__tests__/cloudConnections.test.js + + + Add API helpers for estimate, start, list/status, cancel batch, cancel item, skip item, and retry item. Extend the store with analysis jobs, aggregate queue state, status label mapping, simplified/detailed progress mode, and failure behavior. Keep tokens in Pinia memory only and never persist queue credentials or provider refs outside route/session semantics already allowed. + + + - Store translates backend statuses in one place. + - API helpers use connection UUID and opaque provider item ids. + - Tests cover current/already-indexed, failed, cancelled, retry, and aggregate states. + + cd frontend && npm run test -- cloudConnections.analysis.test.js + + + + Task 2: Extend StorageBrowser and CloudFolderView + frontend/src/components/storage/StorageBrowser.vue, frontend/src/views/CloudFolderView.vue, frontend/src/components/storage/__tests__/StorageBrowser.analysis-queue.test.js, frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js + + - frontend/src/components/storage/StorageBrowser.vue + - frontend/src/views/CloudFolderView.vue + - frontend/src/components/storage/__tests__/StorageBrowser.cloud-queue.test.js + - frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js + + + Add file-row analyze action, toolbar actions for selected items/current folder/current connection, folder current-only vs recursive prompt, estimate review dialog, aggregate analysis progress, expandable item queue, retry/skip/cancel controls, and cancel-all. `StorageBrowser` emits actions; `CloudFolderView` handles API/store calls and refresh behavior. Use icons/tooltips consistent with existing AppIcon patterns and keep text compact enough for mobile. + + + - No parallel file grid or cloud-only browser component is introduced. + - Individual file, multi-select, folder, and connection analysis can be initiated. + - Whole connection and thresholded large scopes show estimate before start. + - Aggregate and per-item statuses render with accessible controls. + + cd frontend && npm run test -- StorageBrowser.analysis-queue.test.js CloudFolderRenderedFlow.test.js + + + + + +1. `cd frontend && npm run test -- StorageBrowser.analysis-queue.test.js cloudConnections.analysis.test.js CloudFolderRenderedFlow.test.js` +2. `rg "analysis" frontend/src/components/storage frontend/src/views/CloudFolderView.vue frontend/src/stores/cloudConnections.js` +3. Rendered smoke check in browser if the dev server is available. + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-07-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-08-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-08-PLAN.md new file mode 100644 index 0000000..0fdc059 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-08-PLAN.md @@ -0,0 +1,87 @@ +--- +phase: "14" +plan: "08" +type: execute +wave: 7 +depends_on: + - "14-03" + - "14-07" +files_modified: + - frontend/src/views/SettingsView.vue + - frontend/src/components/settings/SettingsCloudTab.vue + - frontend/src/components/settings/__tests__/SettingsCloudTab.test.js + - frontend/src/components/settings/__tests__/SettingsCloudTab.health.test.js + - frontend/src/stores/cloudConnections.js + - frontend/src/api/cloud.js + - backend/tests/test_cloud_cache.py +autonomous: true +requirements: + - ANALYZE-04 + - ANALYZE-05 + - CACHE-04 +--- + + +Expose Phase 14 user settings for cache limit, progress detail, and default failure behavior in Settings, backed by the cache/settings API and tier-bound validation. + +Scope fence: do not change backend cache internals except fixing issues found by Settings integration tests. + + + +@.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md +@frontend/src/views/SettingsView.vue +@frontend/src/components/settings/SettingsCloudTab.vue +@frontend/src/stores/cloudConnections.js +@backend/api/cloud/cache.py + + + + + + Task 1: Add Settings API/store wiring + frontend/src/api/cloud.js, frontend/src/stores/cloudConnections.js, frontend/src/stores/__tests__/cloudConnections.analysis.test.js + + - frontend/src/api/cloud.js + - frontend/src/stores/cloudConnections.js + - backend/api/cloud/cache.py + + + Add client/store methods to load and update cache/settings state. Store tier cap, current usage, preferred limit, progress detail mode, and default failure behavior. Handle validation errors without clearing existing settings. + + + - Store keeps settings in memory and never browser-stores credentials. + - Invalid cache limit responses surface controlled messages. + - Updated progress/failure settings affect analysis queue label/control behavior. + + cd frontend && npm run test -- cloudConnections.analysis.test.js + + + + Task 2: Add Settings Cloud tab controls + frontend/src/components/settings/SettingsCloudTab.vue, frontend/src/components/settings/__tests__/SettingsCloudTab.test.js, frontend/src/components/settings/__tests__/SettingsCloudTab.health.test.js + + - frontend/src/components/settings/SettingsCloudTab.vue + - frontend/src/components/settings/__tests__/SettingsCloudTab.test.js + + + Add compact controls for cache limit up to tier maximum, simplified vs detailed analysis progress, and pause-batch vs continue-item failure behavior. Keep the existing connection health/test/reconnect UX intact. Use form controls appropriate to settings: slider/number for bytes, segmented/select controls for modes, and clear validation text. + + + - Settings fit existing Cloud tab layout without nested cards. + - Existing health/reconnect tests still pass. + - Accessibility labels identify each control. + - No visible explanatory tutorial copy is added beyond concise labels and validation messages. + + cd frontend && npm run test -- SettingsCloudTab.test.js SettingsCloudTab.health.test.js + + + + + +1. `cd frontend && npm run test -- SettingsCloudTab.test.js SettingsCloudTab.health.test.js cloudConnections.analysis.test.js` +2. `cd backend && pytest -q tests/test_cloud_cache.py -k "settings"` +3. Render Settings at mobile and desktop widths if a dev server is available. + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-08-SUMMARY.md` when complete. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-09-PLAN.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-09-PLAN.md new file mode 100644 index 0000000..d6783f7 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-09-PLAN.md @@ -0,0 +1,109 @@ +--- +phase: "14" +plan: "09" +type: execute +wave: 8 +depends_on: + - "14-01" + - "14-02" + - "14-03" + - "14-04" + - "14-05" + - "14-06" + - "14-07" + - "14-08" +files_modified: + - AGENTS.md + - README.md + - SECURITY.md + - backend/main.py + - frontend/package.json + - .planning/ROADMAP.md + - .planning/STATE.md + - .planning/phases/14-selective-analysis-and-byte-cache/14-VALIDATION.md + - .planning/phases/14-selective-analysis-and-byte-cache/14-SECURITY.md +autonomous: true +requirements: + - ANALYZE-01 + - ANALYZE-02 + - ANALYZE-03 + - ANALYZE-04 + - ANALYZE-05 + - ANALYZE-06 + - ANALYZE-07 + - CACHE-03 + - CACHE-04 + - CACHE-05 +--- + + +Close Phase 14 with documentation, version bump, full verification gates, security evidence, and an atomic commit/push once all implementation plans pass. + +Scope fence: do not add new feature behavior here except small fixes required by gate failures. + + + +@AGENTS.md +@README.md +@SECURITY.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md +@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md + + + + + + Task 1: Update docs, versions, roadmap, and closeout artifacts + AGENTS.md, README.md, SECURITY.md, backend/main.py, frontend/package.json, .planning/ROADMAP.md, .planning/STATE.md, .planning/phases/14-selective-analysis-and-byte-cache/14-VALIDATION.md, .planning/phases/14-selective-analysis-and-byte-cache/14-SECURITY.md + + - AGENTS.md + - README.md + - SECURITY.md + - backend/main.py + - frontend/package.json + + + Update current-state docs for Phase 14 completion, shared module maps for new analysis/cache services and routes, README feature/API/settings documentation, SECURITY threat register and gate evidence, ROADMAP plan checkboxes, and STATE next action. Apply the project version bump rule after Phase 14 completion using `backend/main.py` and `frontend/package.json` as sources of truth. + + + - Docs describe selective analysis, byte cache, settings, API surface, and alpha/non-public status accurately. + - No secrets or provider item names from live data are included. + - ROADMAP marks Phase 14 complete and Phase 15 next. + - Version numbers are consistent across backend/frontend/docs. + + + + + Task 2: Run full gates and commit/push + all changed files + + - AGENTS.md testing, security, and git protocols + - .planning/phases/14-selective-analysis-and-byte-cache/14-SECURITY.md + + + Run focused Phase 14 tests, full backend suite, full frontend suite, `docker compose config --quiet`, Bandit, npm audit, Python dependency audit if local tooling supports Python 3.12, and an executable secret scan (`gitleaks` or `trufflehog`). Fix root-cause failures within scope. Commit and push the Phase 14 closeout only after gates pass or accepted pre-existing/tooling gaps are documented. + + + - Backend and frontend tests pass with zero Phase 14 failures. + - Security scans report zero new high/critical issues. + - Secret scan reports no new secrets. + - Atomic commit uses the project commit format and is pushed. + + docker compose config --quiet + + + + + +1. `cd backend && pytest -q` +2. `cd frontend && npm run test` +3. `python3 -m bandit -r backend/ --exclude backend/tests` +4. `cd frontend && npm audit --audit-level=high` +5. `docker compose config --quiet` +6. Run available secret scan and document pre-existing findings only. + + +Create `.planning/phases/14-selective-analysis-and-byte-cache/14-09-SUMMARY.md` and mark Phase 14 complete when all gates pass. + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md new file mode 100644 index 0000000..de66982 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md @@ -0,0 +1,93 @@ +# Phase 14 Patterns + +## Shared Boundaries + +- `StorageBrowser.vue` remains the only file browser. Analysis actions, aggregate progress, and expandable queue UI belong there. +- `CloudFolderView.vue` remains a thin data provider. It calls the store/API helpers and refreshes browse state; it does not own browser layout or queue rendering. +- Backend routers translate typed request/response schemas and HTTP status codes only. Service modules own analysis, cache, quota, and idempotency behavior and must not raise `HTTPException`. +- Cloud provider credentials are decrypted only inside route/task provider boundaries. Celery payloads contain IDs and control flags only. + +## Status Vocabulary + +Durable internal item statuses: + +- `queued` +- `downloading` +- `extracting` +- `classifying` +- `indexed` +- `already_current` +- `cancelled` +- `failed` +- `unsupported` +- `stale` + +UI label mapping: + +- simple default: waiting, working, done, skipped, failed +- detailed setting: queued, downloading, extracting, classifying, indexed, cancelled, failed + +Do not let frontend components invent a second status translation. Put API-to-UI translation in the Pinia store or a shared formatter/helper. + +## Version Keys + +Create one shared helper for cloud analysis version keys. All estimate, enqueue, duplicate-check, cache lookup, and worker paths must call it. + +Precedence: + +1. `version` +2. `etag` +3. metadata fingerprint from provider item id, size, modified time, content type +4. content hash as an additional post-hydration fact, never as a pre-download requirement + +## Cache Lifecycle + +1. Resolve owner and item. +2. Compute version key. +3. Reuse non-evicted matching cache entry if present. +4. Otherwise hydrate provider bytes through the mutable/content adapter boundary. +5. Store bytes under a UUID/private MinIO key. +6. Atomically increment user quota by retained size. +7. Pin for active job or preview. +8. Release pin when finished. +9. Evict inactive LRU entries when the user's configured limit or quota pressure requires it. + +Eviction failure must not hide successful analysis. It should surface controlled warning state and leave quota/cache metadata consistent. + +## Estimate Semantics + +Small file jobs may start directly. Folder and connection jobs estimate first when thresholds are crossed, and whole-connection analysis always estimates unless trivially small. + +Estimate responses include: + +- supported file count +- provider-reported byte total +- unsupported/skipped item count +- whether recursion applies +- whether the estimate is complete or partial + +The estimate endpoint must never download bytes. + +## Error and Control Semantics + +- Batch failure behavior defaults to `pause_batch`. +- `continue_item` pauses only the failed item and keeps later queued items running. +- Per-item retry restarts the full item. +- Per-item skip/cancel affects that item only. +- Batch cancel cancels queued items and requests cooperative cancellation for running work. +- Typed error bodies use controlled `kind` and `reason` fields, matching the Phase 13 mutation response style. + +## Test Pattern + +Each implementation plan starts with focused red tests for its slice, then implementation, then targeted verification. + +Required negative coverage by the end of the phase: + +- foreign user cannot estimate, enqueue, read status, cancel, retry, or read cache metadata +- admin cannot access cloud analysis/cache routes +- cache object keys and credentials never appear in responses or audit logs +- unchanged etag/version skips duplicate analysis without byte download +- cached bytes count against quota while retained and decrement after eviction +- active jobs/previews are not evicted +- folder/connection analysis does not mutate provider files + diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md new file mode 100644 index 0000000..0d3782a --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md @@ -0,0 +1,129 @@ +# Phase 14 Research: Selective Analysis and Byte Cache + +**Phase:** 14 - Selective Analysis and Byte Cache +**Prepared:** 2026-06-23 +**Status:** Ready for execution planning + +## Scope Boundary + +Phase 14 adds controlled analysis of provider-owned cloud files and a bounded private byte cache. It does not import cloud files into local `documents`, does not implement unified search, and does not implement provider delta feeds beyond version/fingerprint checks needed to avoid duplicate or stale analysis. + +The existing system already provides the correct foundation: + +- `CloudItem` stores owner, connection, provider item id, parent, kind, content type, size, etag/version, extracted text, analysis status, semantic status, and topic links without a local `Document`. +- `CloudFolderView` remains a route/store data provider; `StorageBrowser` owns the shared local/cloud browser surface. +- `cloud_tasks.py` and `document_tasks.py` show the accepted Celery sync-to-async bridge and retry sentinel pattern. +- `MinIOBackend` already generates UUID-based private object keys and is the right precedent for temporary cached cloud bytes. + +## Recommended Schema Additions + +Add a new migration after the current head with owner-scoped tables rather than overloading local `documents`. + +### `cloud_byte_cache_entries` + +Purpose: track retained cloud bytes in MinIO and make eviction/quota decisions without exposing provider credentials or paths. + +Suggested fields: + +- `id` UUID primary key +- `user_id` FK users, required +- `connection_id` FK cloud_connections, required +- `cloud_item_id` FK cloud_items, required +- `provider_item_id` text, required snapshot for audit/debugging without joins +- `version_key` text, required, derived from etag/version/fallback fingerprint/content hash +- `object_key` text, required, UUID/private key only +- `content_type` text nullable +- `size_bytes` bigint required +- `content_hash` text nullable, computed only when bytes are already hydrated +- `pin_count` integer required default 0 +- `active_job_count` integer required default 0 +- `last_accessed_at`, `created_at`, `updated_at`, `evicted_at` +- unique `(user_id, connection_id, cloud_item_id, version_key)` +- indexes on `(user_id, evicted_at, last_accessed_at)` and `(user_id, pin_count, active_job_count, last_accessed_at)` + +### `cloud_analysis_jobs` + +Purpose: represent user-visible batches for file, selection, folder, or connection analysis. + +Suggested fields: + +- `id` UUID primary key +- `user_id`, `connection_id` +- `scope_kind`: `file`, `selection`, `folder`, `connection` +- `scope_ref` nullable provider item id or root sentinel +- `recursive` boolean +- `status`: `queued`, `running`, `paused`, `completed`, `cancelled`, `failed` +- aggregate counts: total, queued, downloading, extracting, classifying, indexed, skipped, cancelled, failed, unsupported +- aggregate provider bytes estimate +- `failure_behavior`: `pause_batch` or `continue_item` +- timestamps + +### `cloud_analysis_job_items` + +Purpose: drive idempotent per-item processing and precise user controls. + +Suggested fields: + +- `id`, `job_id`, `user_id`, `connection_id`, `cloud_item_id` +- `provider_item_id`, `version_key`, `fingerprint` +- `status`: `queued`, `downloading`, `extracting`, `classifying`, `indexed`, `already_current`, `cancelled`, `failed`, `unsupported`, `stale` +- `error_code`, `error_message` controlled strings only +- `cache_entry_id` nullable +- timestamps and retry counter +- unique `(job_id, cloud_item_id)` +- index `(user_id, cloud_item_id, version_key)` for duplicate/current checks + +### User Settings + +Prefer explicit columns or a small owner-scoped settings table for: + +- `analysis_progress_detail`: `simple` default, `detailed` optional +- `analysis_failure_behavior`: `pause_batch` default, `continue_item` optional +- `cloud_cache_limit_bytes`: user preference capped by tier/admin maximum + +If the project already has a settings table at implementation time, extend it rather than creating a second settings authority. + +## Version and Fingerprint Rules + +Use this precedence for idempotency: + +1. Provider `version` when available. +2. Provider `etag` when available. +3. Fallback metadata fingerprint: provider item id, provider size, modified time, content type. +4. Content hash only after bytes are already hydrated for analysis/open/preview. + +Never download bytes solely to compute a hash. + +## Byte Cache Rules + +- Cache bytes only for active open, preview, or analysis work. +- Cached bytes count against `quotas.used_bytes` while retained. +- Quota changes must use the project's existing atomic update pattern. +- Eviction may delete only inactive entries with `pin_count == 0`, `active_job_count == 0`, and no active preview lease. +- Eviction must be per-user LRU and must never delete another user's entries. +- MinIO object keys must remain private UUID keys; provider filenames stay in metadata only. + +## Job Execution Rules + +- API requests enqueue jobs; Celery workers expand folders/connections and process items. +- Worker broker payloads include only IDs and control flags; credentials are decrypted inside the worker. +- Processing an unchanged cloud item reports `already_current` without provider byte download. +- Retry restarts the whole item from the beginning. +- Cancellation is cooperative: queued work is cancelled immediately, running download/extract/classify checks cancellation between stages and finishes safely if it cannot interrupt. +- If provider metadata changes during analysis, finish the job safely, compare final version/fingerprint, and mark the result stale rather than mutating the provider item. + +## Frontend Rules + +- Add analysis actions to `StorageBrowser` row actions and toolbar. +- `CloudFolderView` only maps route/store/API state and handles emitted events. +- Use an aggregate indicator plus expandable item list. +- Simplified progress labels are default; detailed labels are exposed from Settings. +- The analysis queue can reuse the existing upload queue interaction pattern, but it must be a distinct queue model because controls and statuses differ. + +## Security Notes + +- Admins remain blocked by `get_regular_user`. +- All analysis/cache routes must resolve `(current_user.id, connection_id, provider_item_id/cloud_item_id)` before taking action. +- Responses never include credentials, raw provider URLs, MinIO object keys, extracted text beyond explicitly authorized analysis/search surfaces, or provider error payloads. +- Audit logs contain metadata only: connection id, provider item id, cloud item id, action/status, count/byte totals where needed. +