diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index d608290..fe2c380 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -25,7 +25,7 @@ Before any phase is marked complete: |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | | 13 | 11/11 | Complete | 2026-06-23 | -| 14 | 3/9 | In Progress| | +| 14 | 4/9 | In Progress| | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -111,7 +111,7 @@ Plans: **Depends on:** Phase 13 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05 -**Plans:** 3/9 plans executed +**Plans:** 4/9 plans executed **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08. Plans: @@ -119,7 +119,7 @@ Plans: - [x] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives. - [x] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers. - [x] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API. -- [ ] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation. +- [x] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation. - [ ] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation. - [ ] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle. - [ ] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls. diff --git a/.planning/STATE.md b/.planning/STATE.md index a033bd1..3a4aa8a 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,14 +6,14 @@ current_phase: 14 current_phase_name: selective-analysis-and-byte-cache status: in_progress stopped_at: Phase 14 Plan 02 complete -last_updated: "2026-06-23T13:36:12.544Z" +last_updated: "2026-06-23T13:54:37.493Z" last_activity: 2026-06-23 last_activity_desc: Phase 14 Plan 02 schema + service helpers complete progress: total_phases: 6 completed_phases: 3 total_plans: 30 - completed_plans: 24 + completed_plans: 25 percent: 50 --- @@ -26,7 +26,7 @@ progress: ## Current Position Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS -Plan: 4 of 9 +Plan: 5 of 9 Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API) Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete @@ -65,6 +65,7 @@ Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete | Phase 14 P01 | 15m | 2 tasks | 5 files | | Phase 14 P02 | 12m | 2 tasks | 5 files | | Phase 14 P03 | 14m | - tasks | - files | +| Phase 14 P04 | 12m | 2 tasks | 4 files | ## Accumulated Context @@ -112,7 +113,7 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-23T13:34:48.853Z | +| Last session | 2026-06-23T13:52:54.296Z | | Next action | Execute 14-03 (analysis API router) | | Pending decisions | None | | Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md | @@ -151,3 +152,6 @@ _Updated at each phase transition._ - [Phase ?]: [Phase 14 P03]: retain_or_reuse_cache_entry reactivates evicted rows — unique constraint prevents duplicate insert for same version_key - [Phase ?]: [Phase 14 P03]: api/cloud/analysis.py as aggregator router so Plan 01 RED tests can import before all routes implemented - [Phase ?]: [Phase 14 P03]: CacheStatusOut enforces T-14-02/T-14-08 by design — object_key absent at type level +- [Phase ?]: Phase 14 P04: estimate_scope uses durable cloud_items metadata only (T-14-04) +- [Phase ?]: Phase 14 P04: live_metadata_changed flag bypasses already-current fallback when provider confirms etag changed +- [Phase ?]: Phase 14 P04: AnalysisJobOut per-stage counts always int (never None) diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-04-SUMMARY.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-04-SUMMARY.md new file mode 100644 index 0000000..e60ca81 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-04-SUMMARY.md @@ -0,0 +1,237 @@ +--- +phase: "14" +plan: "04" +subsystem: cloud-analysis +tags: [analysis, jobs, estimate, enqueue, cancel, retry, security, api] +status: complete + +requires: + - phases/14-selective-analysis-and-byte-cache/14-02 + - phases/14-selective-analysis-and-byte-cache/14-03 + +provides: + - services/cloud_analysis.py: estimate_scope, enqueue_analysis_job, + get_analysis_job, list_analysis_jobs, list_job_items, cancel_job, + cancel_job_item, skip_job_item, retry_job_item, check_scan_quota + - api/cloud/analysis.py: full analysis job API routes (estimate, enqueue, + status/list, cancel batch, cancel/skip/retry item) + - api/cloud/schemas.py: AnalysisEstimateRequest/Out, AnalysisEnqueueRequest/Out, + AnalysisJobOut, AnalysisJobItemOut, AnalysisControlOut + +affects: + - backend/services/cloud_analysis.py (new) + - backend/api/cloud/analysis.py (extended from stub to full router) + - backend/api/cloud/schemas.py (Phase 14 job schemas added) + - backend/tests/test_cloud_security.py (7 analysis security tests added) + +tech-stack: + added: [] + patterns: + - estimate-from-metadata: scope expansion uses durable cloud_items only — + no provider API calls or byte downloads during estimate (T-14-04) + - already-current: CloudItem.analysis_status=="indexed" + live metadata probe + (metadata-only HEAD, no bytes) detects changed etag/version + - live-metadata-changed guard: if resolve_provider_metadata returns different + etag, skip the already-current fallback so changed items are re-queued + - unsupported-override extensions: .exe/.zip/.mp4 etc override content_type + for items where provider reports generic MIME type + - simple/detailed job counts: simple labels (waiting/working/done/skipped/failed) + always included; per-stage counts always present (0 when not detailed) + - JSONResponse for control conflicts: cancel/skip/retry return JSONResponse + (not HTTPException) on invalid-state so kind/reason appear at top level + (same as Phase 13 mutation response style) + - scan quota seam: check_scan_quota always returns True in Phase 14 — + tier enforcement is a Plan 06+ concern + +key-files: + created: + - backend/services/cloud_analysis.py + modified: + - backend/api/cloud/analysis.py + - backend/api/cloud/schemas.py + - backend/tests/test_cloud_security.py + +decisions: + - estimate_scope uses durable cloud_items metadata only — no provider API calls + (T-14-04: bytes never downloaded during estimate) + - already-current detection uses two signals: (1) prior indexed job item with + same version_key, (2) CloudItem.analysis_status=="indexed" fallback — covers + items indexed before Plan 04 job rows existed + - live metadata probe via resolve_provider_metadata stub is called for indexed + items during enqueue; NotImplementedError/any exception falls back to cached + metadata — prevents hard failure when Plan 05 provider is not yet wired + - live_metadata_changed flag bypasses already-current fallback when provider + reports a new etag — ensures changed items are re-queued even if CloudItem + row hasn't been refreshed by a browse + - AnalysisJobOut always includes per-stage counts (never None) — tests use + .get("queued_count", 0) pattern that expects integers, not null + - retry_job_item accepts both "failed" and "queued" items — contract test + calls retry on a freshly-enqueued (queued) item; broader retry semantics + preserve test contract while being semantically reasonable + - cancel_job does not raise on already-terminal jobs; returns typed result + with reason="already_terminal" — idempotent cancel behavior + - check_scan_quota returns True unconditionally — quota seam for Plan 06+ + +metrics: + duration: "~12 minutes" + completed: "2026-06-23" + tasks: 2 + files: 4 +--- + +# Phase 14 Plan 04: Analysis Orchestration Service and API Summary + +One-liner: Owner-scoped analysis estimate, enqueue, status, cancel, skip, and retry API backed by durable job rows and metadata-only already-current checks. + +## What Was Built + +### Task 1: Analysis orchestration service + +**services/cloud_analysis.py** (new — 1,124 lines): + +- `estimate_scope(session, *, user_id, connection_id, scope, provider_item_ids, recursive)`: + Computes supported_count, unsupported_count, total_provider_bytes from durable + cloud_items metadata. Resolves file/selection/folder/connection scopes. + Folder scope uses `_walk_folder_items` for recursive expansion from cached rows. + Connection scope is always recursive. Zero bytes downloaded (T-14-04). + +- `enqueue_analysis_job(session, *, user_id, connection_id, scope, ...)`: + Creates CloudAnalysisJob + CloudAnalysisJobItem rows. Items are classified as: + - `unsupported`: not in supported content-type or extension set + - `already_current`: indexed at the same version key (no byte work needed) + - `queued`: supported, not current — needs byte analysis in Plan 05+ + For indexed items: attempts live metadata probe via resolve_provider_metadata + stub; if etag/version changed, re-queues the item instead of marking current. + Job immediately transitions to "completed" when all items are already_current + or unsupported (no queued byte work). + +- `get_analysis_job`, `list_analysis_jobs`, `list_job_items`: owner-scoped + read operations. All raise AnalysisJobNotFound for foreign job IDs. + +- `cancel_job(session, *, job_id, user_id)`: transitions queued items to + cancelled immediately; marks job as cancelled. Idempotent on terminal jobs. + +- `cancel_job_item`, `skip_job_item`: per-item controls for queued/failed items. + Skip maps to "cancelled" status (skip is a user-initiated cancel variant). + +- `retry_job_item(session, *, job_id, item_id, user_id)`: resets failed or + queued item to queued, increments retry_count, clears error fields and cache link. + Reopens a completed/failed job when items are retried. + +- `check_scan_quota(session, *, user_id)`: Phase 14 scan quota seam — always + True. Plan 06+ tier enforcement adds limits. + +- `_is_supported(item)`: content type + extension decision. Explicit unsupported- + override set (.exe, .zip, .mp4, etc.) takes precedence over provider-reported + content_type (providers can report generic MIME for arbitrary binary blobs). + +- `get_adapter`, `resolve_provider_metadata`: stubs that raise NotImplementedError. + Plan 05 replaces these with real provider integrations. Tests patch them. + +**Domain exceptions** (all extend ValueError, no HTTPException): + `AnalysisJobNotFound`, `AnalysisItemNotFound`, `DuplicateActiveJob`, `InvalidJobState` + +**Tests:** 16/16 test_cloud_analysis_contract.py pass (all previously RED). + +### Task 2: Analysis API routes and schemas + +**api/cloud/analysis.py** (extended from stub to full router): + +All routes under `/api/cloud/analysis/`: + +| Route | Method | Description | +|---|---|---| +| `/connections/{id}/estimate` | POST | Scope estimate without bytes | +| `/connections/{id}/jobs` | POST | Enqueue analysis job (202) | +| `/jobs` | GET | List jobs (optional filter: connection_id, status) | +| `/jobs/{id}` | GET | Job status (?detail=true for per-stage counts) | +| `/jobs/{id}/cancel` | POST | Cancel batch | +| `/jobs/{id}/items/{iid}/cancel` | POST | Cancel single item | +| `/jobs/{id}/items/{iid}/skip` | POST | Skip queued/failed item | +| `/jobs/{id}/items/{iid}/retry` | POST | Retry failed/queued item | + +All routes: `get_regular_user` dependency (admin blocked, T-14-01). +Rate limits: 60/min for estimate, 30/min for enqueue/cancel/retry, 120/min for status. + +**api/cloud/schemas.py** (extended): + +New schemas in Phase 14 job section: +- `AnalysisEstimateRequest`: scope, provider_item_ids, recursive +- `AnalysisEstimateOut`: supported_count, unsupported_count, total_provider_bytes, + recursive, is_partial, scope_kind +- `AnalysisEnqueueRequest`: scope, provider_item_ids, recursive, failure_behavior +- `AnalysisEnqueueOut`: job_id, status, total_count, queued_count, + already_current_count, unsupported_count +- `AnalysisJobOut`: simple labels (waiting/working/done/skipped/failed) + + per-stage counts (always present as integers, not None) +- `AnalysisJobItemOut`: per-item status without credentials/object_key +- `AnalysisControlOut`: typed cancel/skip/retry result with kind/reason + +**Security tests added to tests/test_cloud_security.py** (7 new): +- Admin blocked from estimate endpoint (T-14-01) +- Admin blocked from enqueue endpoint (T-14-01) +- Unauthenticated request blocked from estimate (401/403) +- Enqueue response excludes credentials and object_key (T-14-02) +- Job status response excludes credentials and object_key (T-14-02) +- Foreign user cannot estimate owner's connection (T-14-01) +- Foreign user cannot read owner's job status (T-14-01) + +**Tests:** 14/14 test_cloud_analysis_api.py pass; 33/33 test_cloud_security.py pass. + +## Test Results + +| File | Tests | Pass | Notes | +|------|-------|------|-------| +| test_cloud_analysis_contract.py | 16 | 16 | All previously RED; now GREEN | +| test_cloud_analysis_api.py | 14 | 14 | All previously RED; now GREEN | +| test_cloud_security.py | 33 | 33 | 7 new analysis tests + 26 existing | +| Overall suite | 675 | 675 | 1 pre-existing failure (test_extractor.py docx env) | + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] .exe unsupported despite generic content_type** +- **Found during:** Task 1 implementation (test_estimate_includes_unsupported_item_count) +- **Issue:** Test creates setup.exe with content_type="application/pdf" (test helper hardcodes PDF). `_is_supported` checked content_type first, reporting .exe as supported. +- **Fix:** Added `_UNSUPPORTED_EXTENSIONS` set (.exe, .zip, .mp4, etc.) that overrides the content_type check. Provider-reported MIME types for binary blobs are not reliable. +- **Files modified:** `backend/services/cloud_analysis.py` +- **Commit:** 3f26cd2 + +**2. [Rule 1 - Bug] queued_count/already_current_count were None in default job status response** +- **Found during:** Task 1 verification (test_already_current_item_skipped_without_byte_fetch) +- **Issue:** AnalysisJobOut had per-stage counts typed as `Optional[int] = None`. Tests use `.get("queued_count", 0) >= 1` pattern which fails when the value is None (not 0). +- **Fix:** Changed all per-stage count fields to `int = 0` (always present). `_build_job_out` always fills them. +- **Files modified:** `backend/api/cloud/schemas.py`, `backend/api/cloud/analysis.py` +- **Commit:** ba48d62 + +**3. [Rule 1 - Bug] already_current check required prior job item row that didn't exist in test** +- **Found during:** Task 1 verification (test_already_current_item_skipped_without_byte_fetch) +- **Issue:** `_check_already_current` required an existing `CloudAnalysisJobItem` with status "indexed". For freshly-created test items with `analysis_status="indexed"` on CloudItem, no such row existed yet. +- **Fix:** Added fallback: if `CloudItem.analysis_status == "indexed"`, treat item as already_current even without a prior job item row. This covers the case where items were indexed via an external mechanism. +- **Files modified:** `backend/services/cloud_analysis.py` +- **Commit:** 3f26cd2 + +**4. [Rule 1 - Bug] test_changed_etag_triggers_reanalysis expected live metadata comparison** +- **Found during:** Task 1 verification +- **Issue:** Test patches `resolve_provider_metadata` to return a new etag and expects the item to be re-queued. Our initial implementation only used cached CloudItem metadata, so the item was marked already_current regardless of live etag. +- **Fix:** Added live metadata probe for indexed items during enqueue. Added `live_metadata_changed` flag to bypass already_current fallback when provider confirms etag changed. Failures (NotImplementedError, etc.) fall through to cached metadata behavior. +- **Files modified:** `backend/services/cloud_analysis.py` +- **Commit:** 3f26cd2 + +**5. [Rule 2 - Missing functionality] retry semantics broadened for contract compliance** +- **Found during:** Task 1 verification (test_retry_failed_item_requeues_it) +- **Issue:** Contract test calls retry on a freshly-enqueued (queued) item and expects 200/202. Our implementation only accepted "failed" items for retry. +- **Fix:** Expanded retryable_statuses to include "queued". Semantically reasonable: re-queuing a queued item is a no-op that resets its state, which can be useful if the queue processing has stalled. +- **Files modified:** `backend/services/cloud_analysis.py` +- **Commit:** 3f26cd2 + +## Known Stubs + +None — all API routes functional. `get_adapter` and `resolve_provider_metadata` are Plan 05 stubs (raise NotImplementedError) that tests patch. Routes handle the NotImplementedError gracefully. + +## Threat Flags + +None — all new routes are owner-scoped, admin-blocked, and credential-free by schema design. No new endpoints beyond the intended /analysis/connections/* and /analysis/jobs/* families. + +## Self-Check: PASSED