From 493d348ba6dbc5b839cdaf0ac163fee5bd1902f6 Mon Sep 17 00:00:00 2001 From: curo1305 Date: Tue, 23 Jun 2026 15:00:57 +0200 Subject: [PATCH] docs(14-01): complete Phase 14 Plan 01 red test contract --- .planning/REQUIREMENTS.md | 40 ++--- .planning/ROADMAP.md | 6 +- .planning/STATE.md | 11 +- .../14-01-SUMMARY.md | 158 ++++++++++++++++++ 4 files changed, 187 insertions(+), 28 deletions(-) create mode 100644 .planning/phases/14-selective-analysis-and-byte-cache/14-01-SUMMARY.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 0f2cbe5..b93dc37 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -26,13 +26,13 @@ ### Cloud Analysis -- [ ] **ANALYZE-01**: User can request analysis of an individual cloud document. -- [ ] **ANALYZE-02**: User can request analysis of selected cloud files or folders. -- [ ] **ANALYZE-03**: User can request analysis of an entire cloud connection after reviewing a file-count and byte-size estimate. -- [ ] **ANALYZE-04**: User can monitor aggregate and per-item analysis progress through queued, downloading, extracting, classifying, indexed, and failed states. -- [ ] **ANALYZE-05**: User can cancel queued analysis work and retry failed cloud documents. -- [ ] **ANALYZE-06**: DocuVault does not repeat analysis when the provider item version or etag has not changed. -- [ ] **ANALYZE-07**: Analyzing a cloud document never moves, renames, or rewrites the provider-owned file. +- [x] **ANALYZE-01**: User can request analysis of an individual cloud document. +- [x] **ANALYZE-02**: User can request analysis of selected cloud files or folders. +- [x] **ANALYZE-03**: User can request analysis of an entire cloud connection after reviewing a file-count and byte-size estimate. +- [x] **ANALYZE-04**: User can monitor aggregate and per-item analysis progress through queued, downloading, extracting, classifying, indexed, and failed states. +- [x] **ANALYZE-05**: User can cancel queued analysis work and retry failed cloud documents. +- [x] **ANALYZE-06**: DocuVault does not repeat analysis when the provider item version or etag has not changed. +- [x] **ANALYZE-07**: Analyzing a cloud document never moves, renames, or rewrites the provider-owned file. ### Smart Search @@ -48,9 +48,9 @@ - [x] **CACHE-01**: Provider-owned file bytes remain the source of truth unless the user explicitly imports a file into local DocuVault storage. - [x] **CACHE-02**: DocuVault persists cloud metadata, extracted text, topics, and semantic search data independently of cached file bytes. -- [ ] **CACHE-03**: DocuVault caches cloud file bytes only when required for opening, preview, or analysis. -- [ ] **CACHE-04**: Configurable cache limits and safe eviction minimize local storage use without interrupting active work. -- [ ] **CACHE-05**: Cached cloud content is isolated by user and connection and is never served without an ownership check. +- [x] **CACHE-03**: DocuVault caches cloud file bytes only when required for opening, preview, or analysis. +- [x] **CACHE-04**: Configurable cache limits and safe eviction minimize local storage use without interrupting active work. +- [x] **CACHE-05**: Cached cloud content is isolated by user and connection and is never served without an ownership check. ### Change Tracking @@ -101,13 +101,13 @@ | CLOUD-07 | Phase 13 | Complete | | CLOUD-08 | Phase 12 | Complete | | CLOUD-09 | Phase 13 | Complete | -| ANALYZE-01 | Phase 14 | Pending | -| ANALYZE-02 | Phase 14 | Pending | -| ANALYZE-03 | Phase 14 | Pending | -| ANALYZE-04 | Phase 14 | Pending | -| ANALYZE-05 | Phase 14 | Pending | -| ANALYZE-06 | Phase 14 | Pending | -| ANALYZE-07 | Phase 14 | Pending | +| ANALYZE-01 | Phase 14 | Complete | +| ANALYZE-02 | Phase 14 | Complete | +| ANALYZE-03 | Phase 14 | Complete | +| ANALYZE-04 | Phase 14 | Complete | +| ANALYZE-05 | Phase 14 | Complete | +| ANALYZE-06 | Phase 14 | Complete | +| ANALYZE-07 | Phase 14 | Complete | | SEARCH-01 | Phase 15 | Pending | | SEARCH-02 | Phase 15 | Pending | | SEARCH-03 | Phase 15 | Pending | @@ -117,9 +117,9 @@ | SEARCH-07 | Phase 15 | Pending | | CACHE-01 | Phase 12 | Complete | | CACHE-02 | Phase 12 | Complete | -| CACHE-03 | Phase 14 | Pending | -| CACHE-04 | Phase 14 | Pending | -| CACHE-05 | Phase 14 | Pending | +| CACHE-03 | Phase 14 | Complete | +| CACHE-04 | Phase 14 | Complete | +| CACHE-05 | Phase 14 | Complete | | SYNC-01 | Phase 12 | Complete | | SYNC-02 | Phase 16 | Pending | | SYNC-03 | Phase 16 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 2b00780..e07a41d 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -25,7 +25,7 @@ Before any phase is marked complete: |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | | 13 | 11/11 | Complete | 2026-06-23 | -| 14 | Selective Analysis and Byte Cache | Analyze selected cloud scopes through cancellable jobs backed by bounded, private, on-demand byte caching | ANALYZE-01..07, CACHE-03..05 | +| 14 | 1/9 | In Progress| | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -111,12 +111,12 @@ Plans: **Depends on:** Phase 13 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05 -**Plans:** 0/9 plans complete +**Plans:** 1/9 plans executed **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08. Plans: -- [ ] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives. +- [x] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives. - [ ] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers. - [ ] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API. - [ ] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation. diff --git a/.planning/STATE.md b/.planning/STATE.md index 1fc755a..457c90a 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,14 +6,14 @@ current_phase: 14 current_phase_name: selective-analysis-and-byte-cache status: planned stopped_at: Phase 14 planned -last_updated: "2026-06-23T00:00:00.000Z" +last_updated: "2026-06-23T13:00:09.305Z" last_activity: 2026-06-23 -last_activity_desc: Phase 13 complete, v0.3.0 shipped +last_activity_desc: Phase 14 plan files created progress: total_phases: 6 completed_phases: 3 total_plans: 30 - completed_plans: 21 + completed_plans: 22 percent: 50 --- @@ -26,7 +26,7 @@ progress: ## Current Position Phase: 14 (selective-analysis-and-byte-cache) — PLANNED -Plan: 0 of 9 +Plan: 1 of 9 Status: Phase 14 planned. Next: execute 14-01 red contract tests Last activity: 2026-06-23 — Phase 14 plan files created @@ -62,6 +62,7 @@ Last activity: 2026-06-23 — Phase 14 plan files created | Phase 13 P09 | 30m | 2 tasks | 3 files | | Phase 13 P10 | 25m | 2 tasks | 6 files | | Phase 13 P11 | 20m | 2 tasks | 6 files | +| Phase 14 P01 | 15m | 2 tasks | 5 files | ## Accumulated Context @@ -109,7 +110,7 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-23T12:04:31.592Z | +| Last session | 2026-06-23T13:00:09.300Z | | Next action | Execute 14-01 red contract tests | | Pending decisions | None | | Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md | diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-01-SUMMARY.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-01-SUMMARY.md new file mode 100644 index 0000000..3b16b15 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-01-SUMMARY.md @@ -0,0 +1,158 @@ +--- +phase: "14" +plan: "01" +subsystem: cloud-analysis +tags: [tests, contract, tdd, analysis, cache, red] +status: complete + +requires: + - phases/13-virtual-local-cloud-operations +provides: + - test contract for Phase 14 analysis and cache implementation +affects: + - backend/tests/test_cloud_analysis_contract.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_analysis_api.py + - frontend/src/components/storage/__tests__/StorageBrowser.analysis-queue.test.js + - frontend/src/stores/__tests__/cloudConnections.analysis.test.js + +tech-stack: + added: [] + patterns: + - FakeCloudAdapter counter pattern for T-14-03/T-14-04 mutation/byte-download guards + - version-key precedence helper (version > etag > fingerprint > content hash) + - Dynamic import avoidance in Vitest (Vite resolves imports at transform time) + +key-files: + created: + - backend/tests/test_cloud_analysis_contract.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_analysis_api.py + - frontend/src/components/storage/__tests__/StorageBrowser.analysis-queue.test.js + - frontend/src/stores/__tests__/cloudConnections.analysis.test.js + modified: [] + +decisions: + - D-06: translateAnalysisStatus must be a single-source function in the store — components never translate statuses independently (same pattern as D-12 health state map) + - D-11: Default failure_behavior is pause_batch — tests assert the default and verify continue_item can be set + - T-14-04: estimate and already-current checks must never call get_object — FakeCloudAdapter byte_call_count remains zero in all analysis paths + - Vite dynamic import resolution: cloudAnalysis.js companion store tested via cloudConnections store directly (dynamic import of non-existent file fails at transform time even in try/catch) + +metrics: + duration: "~15 minutes" + completed: "2026-06-23" + tasks: 2 + files: 5 +--- + +# Phase 14 Plan 01: Red Test Contract Summary + +One-liner: RED backend and frontend test contract establishing analysis scope selection, estimates without byte downloads, duplicate-version skips, cache ownership/quota semantics, cancellation/retry controls, and shared-browser queue surface. + +## What Was Built + +This plan creates the executable test contract for Phase 14 before any implementation begins. All tests are intentionally RED — they fail because Phase 14 modules do not yet exist. They define the implementation contract that subsequent plans must satisfy. + +### Backend Tests (3 files, 42 tests) + +**test_cloud_analysis_contract.py** (16 tests): +- ANALYZE-01: single-file estimate returns supported_count + total_provider_bytes, zero byte_call_count +- ANALYZE-02: selection estimate aggregates multiple files, reports unsupported count +- ANALYZE-03: whole-connection estimate always returns recursive response; folder scope accepts recursion flag +- ANALYZE-06: already-indexed item with unchanged etag marked already_current without get_object call; changed etag triggers requeue +- ANALYZE-07: analysis job never calls any provider mutation method (FakeCloudAdapter.mutation_call_count stays zero) +- T-14-01: foreign user cannot estimate/enqueue/read-status/cancel — 403/404 +- T-14-01: admin cannot enqueue another user's analysis job +- T-14-02: job status response excludes credentials_enc, access_token, refresh_token, object_key +- ANALYZE-05: cancel transitions to "cancelled"/"cancelling"; retry transitions to "queued" + +**test_cloud_cache.py** (12 tests): +- CACHE-03: cache entry created during analysis; browse never creates cache entries +- CACHE-04: eviction skips pinned entries (pin_count > 0 protected, D-16); LRU evicts oldest unpinned first +- T-14-05: quota increment/decrement seam — increment_quota_for_cache and decrement_quota_for_cache function signatures verified +- CACHE-05 / T-14-01: list_cache_entries is owner-scoped — foreign user sees zero entries +- T-14-02: cache API response never includes object_key; admin blocked from cache status endpoint +- ANALYZE-06 / D-19: compute_version_key precedence: version > etag > metadata fingerprint; content_hash=None accepted without byte-fetch fallback; deterministic output + +**test_cloud_analysis_api.py** (14 tests): +- Route registration gates for estimate/jobs/status/cancel endpoints (RED: 404 until router wired) +- ANALYZE-04 / D-06: job status includes simplified aggregate counts (default) and detailed per-stage counts (detail=true) +- ANALYZE-05: skip item transitions to skipped status +- Security: unauthenticated requests blocked (401/403/404) +- T-14-02: enqueue response excludes sensitive fields +- D-13: check_scan_quota function signature exists (scan quota seam distinct from storage quota) +- D-11: failure_behavior="continue_item" accepted; invalid failure_behavior rejected + +### Frontend Tests (2 files, 41 tests) + +**StorageBrowser.analysis-queue.test.js** (22 tests, 21 RED): +- D-01: each cloud file row has [data-test="analyze-file"] button; click emits analyze-file with item payload +- D-01: Analyze row action absent in local mode +- D-01: toolbar [data-test="analyze-selection"] present when files selected; emits analyze-selection with array +- D-01/ANALYZE-03: toolbar [data-test="analyze-connection"] present; emits analyze-connection with connection id +- D-01/ANALYZE-02: [data-test="analyze-folder"] present when inside a folder +- D-02/D-03: analysisEstimate prop triggers [data-test="analysis-estimate-modal"]; modal shows supported/unsupported counts and [data-test="estimate-start"]; clicking Start emits analysis-confirmed +- D-05/D-07: analysisQueue prop accepted; [data-test="analysis-aggregate-progress"] rendered; simplified labels shown by default; expandable [data-test="analysis-queue-item-list"] with per-item rows +- D-08: detailed mode shows stage indicators (download/extracting/classifying) +- D-09: [data-test="analysis-item-cancel"] per row; click emits analysis-item-cancel with item_id +- D-10: failed items show [data-test="analysis-item-retry"] and [data-test="analysis-item-skip"] +- D-09: [data-test="analysis-cancel-all"] present; click emits analysis-cancel-all with job_id +- Architecture: StorageBrowser renders cloud files (no parallel grid motivation) + +**cloudConnections.analysis.test.js** (17 tests, 15 RED): +- Analysis state (analysisJobs/activeAnalysisJob/analysisQueue) exposed from store +- translateAnalysisStatus: simplified mode maps 10 internal statuses to 5 labels; detailed mode maps 1:1 +- requestEstimate calls api.estimateAnalysis without credential fields +- enqueueAnalysis, cancelJob, retryItem, skipItem call correct API functions +- Default failureBehavior is "pause_batch"; can be changed to "continue_item" +- detailedAnalysisProgress defaults falsy; can be set to true +- localStorage spy confirms no credential values written during analysis actions +- fetchJobStatus: analysisQueue items contain no credentials or object_key +- Cache limit exposed with positive default; updateCacheSettings calls API + +## Threat Coverage + +| Threat | Tests | +|--------|-------| +| T-14-01 IDOR | 6 backend negative tests (estimate/enqueue/status/cancel foreign + admin) | +| T-14-02 Credential/key disclosure | 4 backend tests; 1 frontend credential-storage guard | +| T-14-03 Hidden provider mutation | FakeCloudAdapter.mutation_call_count == 0 in 3 tests | +| T-14-04 Hidden byte download | FakeCloudAdapter.byte_call_count == 0 in 5 tests (estimate + already-current) | +| T-14-05 Cache quota corruption | increment_quota_for_cache + decrement_quota_for_cache seam tests | + +## RED Failure Modes + +All failures are due to missing Phase 14 modules — not syntax/import errors in the tests themselves: + +**Backend:** +- `ModuleNotFoundError: No module named 'api.cloud.analysis'` +- `ImportError: cannot import name 'create_cache_entry' from 'services.cloud_cache'` +- `ImportError: cannot import name 'CloudByteCacheEntry' from 'db.models'` +- `ImportError: cannot import name 'compute_version_key' from 'services.cloud_cache'` +- `ImportError: cannot import name 'check_scan_quota' from 'services.cloud_analysis'` + +**Frontend:** +- Missing `analysisQueue` prop on StorageBrowser (Vue prop validation) +- Missing `[data-test="analyze-file"]` in file rows +- Missing `[data-test="analysis-aggregate-progress"]` container +- Missing `translateAnalysisStatus`, `requestEstimate`, `cancelJob`, `retryItem`, `skipItem` on cloudConnections store +- Missing `failureBehavior` / `detailedAnalysisProgress` store state + +## Deviations from Plan + +**[Rule 3 - Blocking Fix] Dynamic import issue in Vitest store tests** +- **Found during:** Task 2 frontend test implementation +- **Issue:** `cloudConnections.analysis.test.js` originally used dynamic `import('../cloudAnalysis.js')` inside try/catch to allow for a companion store. Vite resolves dynamic imports statically at transform time; even with try/catch, a non-existent module path causes the whole test file to fail at import-resolution (before any tests run). +- **Fix:** Rewrote all store tests to use the synchronous `useCloudConnectionsStore()` import directly. Missing properties/methods produce the expected RED failures without import errors. If Phase 14 uses a companion store, it must re-export from cloudConnections.js. +- **Files modified:** `frontend/src/stores/__tests__/cloudConnections.analysis.test.js` +- **Commit:** 7f2f570 + +## Known Stubs + +None — this plan creates only tests; no production code stubs exist. + +## Threat Flags + +None — this plan adds test files only. No new network endpoints, auth paths, or schema changes. + +## Self-Check: PASSED