diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index e07a41d..d608290 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -25,7 +25,7 @@ Before any phase is marked complete: |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | | 13 | 11/11 | Complete | 2026-06-23 | -| 14 | 1/9 | In Progress| | +| 14 | 3/9 | In Progress| | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -111,14 +111,14 @@ Plans: **Depends on:** Phase 13 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05 -**Plans:** 1/9 plans executed +**Plans:** 3/9 plans executed **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08. Plans: - [x] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives. -- [ ] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers. -- [ ] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API. +- [x] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers. +- [x] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API. - [ ] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation. - [ ] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation. - [ ] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle. diff --git a/.planning/STATE.md b/.planning/STATE.md index 8b438cb..a033bd1 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,15 +6,15 @@ current_phase: 14 current_phase_name: selective-analysis-and-byte-cache status: in_progress stopped_at: Phase 14 Plan 02 complete -last_updated: "2026-06-23T14:16:33.000Z" +last_updated: "2026-06-23T13:36:12.544Z" last_activity: 2026-06-23 -last_activity_desc: Phase 14 Plan 02 — schema and service helpers complete +last_activity_desc: Phase 14 Plan 02 schema + service helpers complete progress: total_phases: 6 completed_phases: 3 total_plans: 30 completed_plans: 24 - percent: 53 + percent: 50 --- # Project State @@ -26,7 +26,7 @@ progress: ## Current Position Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS -Plan: 3 of 9 +Plan: 4 of 9 Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API) Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete @@ -64,6 +64,7 @@ Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete | Phase 13 P11 | 20m | 2 tasks | 6 files | | Phase 14 P01 | 15m | 2 tasks | 5 files | | Phase 14 P02 | 12m | 2 tasks | 5 files | +| Phase 14 P03 | 14m | - tasks | - files | ## Accumulated Context @@ -111,7 +112,7 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-23T14:16:33.000Z | +| Last session | 2026-06-23T13:34:48.853Z | | Next action | Execute 14-03 (analysis API router) | | Pending decisions | None | | Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md | @@ -147,3 +148,6 @@ _Updated at each phase transition._ - [Phase 14 P02]: content_hash is optional post-hydration supplement, never a pre-download precondition (D-20) - [Phase 14 P02]: evict_lru_entries stamps evicted_at only; caller owns MinIO deletion and quota decrement (separation of concerns) - [Phase 14 P02]: user_analysis_settings is distinct from system_settings — single authority for per-user analysis preferences +- [Phase ?]: [Phase 14 P03]: retain_or_reuse_cache_entry reactivates evicted rows — unique constraint prevents duplicate insert for same version_key +- [Phase ?]: [Phase 14 P03]: api/cloud/analysis.py as aggregator router so Plan 01 RED tests can import before all routes implemented +- [Phase ?]: [Phase 14 P03]: CacheStatusOut enforces T-14-02/T-14-08 by design — object_key absent at type level diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-03-SUMMARY.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-03-SUMMARY.md new file mode 100644 index 0000000..ff25ad9 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-03-SUMMARY.md @@ -0,0 +1,161 @@ +--- +phase: "14" +plan: "03" +subsystem: cloud-analysis +tags: [cache, api, security, quota, pinning, settings] +status: complete + +requires: + - phases/14-selective-analysis-and-byte-cache/14-02 + +provides: + - services/cloud_cache.py: retain_or_reuse_cache_entry, pin_cache_entry, + release_cache_entry, update_cache_access — Phase 14 cache lifecycle + orchestration (PATTERNS.md steps 3, 7, 8) + - api/cloud/cache.py: GET /analysis/cache, PATCH /analysis/cache/settings + — aggregate cache usage/settings, no object_key or credentials in response + - api/cloud/analysis.py: analysis router aggregator (parent for Phase 14 + plans 04-09) + - api/cloud/schemas.py: CacheStatusOut, CacheSettingsUpdateRequest + +affects: + - backend/services/cloud_cache.py + - backend/api/cloud/cache.py (new) + - backend/api/cloud/analysis.py (new) + - backend/api/cloud/schemas.py + - backend/api/cloud/__init__.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_security.py + +tech-stack: + added: [] + patterns: + - retain-or-reactivate: unique constraint means evicted entries are + reactivated (evicted_at cleared, object_key updated) rather than + duplicated — semantically equivalent to "cache miss requires new bytes" + - pin_count lifecycle: pin_cache_entry / release_cache_entry with + GREATEST(0, ...) clamp for defensive double-release handling + - analysis router aggregator: api/cloud/analysis.py is the future-proof + parent for all Phase 14 analysis routes (estimates, jobs, controls) + - CacheStatusOut allowlist: schema enforces T-14-02/T-14-08 by design — + object_key and credentials_enc absent at type level + +key-files: + created: + - backend/api/cloud/cache.py + - backend/api/cloud/analysis.py + modified: + - backend/services/cloud_cache.py + - backend/api/cloud/schemas.py + - backend/api/cloud/__init__.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_security.py + +decisions: + - retain_or_reuse_cache_entry reactivates evicted rows (clears evicted_at, + updates object_key) rather than inserting a new row — the unique constraint + on (user_id, connection_id, cloud_item_id, version_key) makes insertion + impossible; reactivation is semantically correct (fresh bytes, same version) + - api/cloud/analysis.py created as the aggregator so tests can + `from api.cloud.analysis import router` as required by Plan 01 RED tests + - CacheStatusOut has no per-entry fields — only aggregate totals — ensuring + T-14-02 compliance at schema design level rather than runtime filtering + +metrics: + duration: "~18 minutes" + completed: "2026-06-23" + tasks: 2 + files: 7 +--- + +# Phase 14 Plan 03: Cache Service and API Summary + +One-liner: Cache orchestration functions (retain/reuse, pin, release, touch) and credential-free cache status/settings API with admin block and object_key exclusion. + +## What Was Built + +### Task 1: Cache orchestration functions + +**services/cloud_cache.py** — four new Phase 14 orchestration functions added after the Plan 02 primitives: + +- `retain_or_reuse_cache_entry(session, *, user_id, connection_id, cloud_item_id, provider_item_id, version_key, object_key, ...)`: PATTERNS.md step 3 implementation. Checks for a non-evicted matching entry first (reuse path). If the only matching entry is evicted, reactivates it (clears evicted_at, updates object_key/size) because the unique constraint prevents inserting a duplicate. Returns `(entry, created_new)` tuple. Ownership enforced — foreign user always gets a separate row. + +- `pin_cache_entry(session, *, entry_id, user_id)`: Increments `pin_count` to mark an active preview or download lease (PATTERNS.md step 7). Ownership asserted — raises `ValueError` for foreign entry IDs. + +- `release_cache_entry(session, *, entry_id, user_id)`: Decrements `pin_count` with `GREATEST(0, ...)` clamp (PATTERNS.md step 8). Ownership asserted. + +- `update_cache_access(session, *, entry_id, user_id)`: Touches `last_accessed_at` after a cache hit to maintain accurate LRU ordering. Ownership asserted. + +All four functions raise `ValueError` only — never `HTTPException` (CLAUDE.md service-layer rule). + +**Tests added to tests/test_cloud_cache.py** (9 new tests, Task 1): +- retain_or_reuse returns existing active entry +- retain_or_reuse creates new entry when absent +- retain_or_reuse reactivates evicted entry (not duplicate) +- retain_or_reuse isolates foreign users (T-14-06) +- pin increments pin_count +- release decrements pin_count +- release clamps at zero (double-release defense) +- pin raises ValueError on foreign entry (T-14-06) +- update_cache_access refreshes last_accessed_at timestamp + +### Task 2: Cache settings/status API + +**api/cloud/cache.py** (new): +- `GET /analysis/cache` — returns `CacheStatusOut` with aggregate entry_count, total_bytes, cache_limit_bytes, tier_cap_bytes, analysis_progress_detail, analysis_failure_behavior. Calls `get_or_create_user_analysis_settings` for defaults on first access. Rate-limited at 120/minute. Blocked for admin (get_regular_user). +- `PATCH /analysis/cache/settings` — accepts `CacheSettingsUpdateRequest` (three explicit fields only — mass assignment prevention). Service layer validates enums and bounds; route re-raises ValueError as 422. Returns fresh CacheStatusOut. Rate-limited at 30/minute. Blocked for admin. + +**api/cloud/analysis.py** (new): +- Aggregator router for `/api/cloud/analysis/*` namespace. +- Includes `cache_router` now; will include estimate/jobs/controls routers in Plans 04-09. +- Importable as `api.cloud.analysis` so Plan 01 RED tests can import the router. + +**api/cloud/schemas.py** (extended): +- `CacheStatusOut`: explicit allowlist with 6 fields — no object_key, credentials_enc, or per-entry data possible. +- `CacheSettingsUpdateRequest`: 3 explicit optional fields with `ge=1` constraint on byte limit. + +**api/cloud/__init__.py** (updated): +- Imports and includes `analysis_router` from `api.cloud.analysis`. + +**Security tests added to tests/test_cloud_security.py** (7 new tests, Task 2): +- admin blocked from GET /analysis/cache (403) +- admin blocked from PATCH /analysis/cache/settings (403) +- unauthenticated request blocked (401/403) +- response excludes object_key and credentials_enc (T-14-02, T-14-08) +- response includes expected aggregate fields +- invalid enum value returns 422 +- cache limit above tier cap returns 422 + +## Test Results + +| File | Tests | Pass | Notes | +|------|-------|------|-------| +| test_cloud_cache.py | 21 | 21 | All pass including 9 new Task 1 tests | +| test_cloud_security.py -k cache | 8 | 8 | 7 new + 1 pre-existing | +| Overall suite (excl. RED tests) | 808 | 808 | Pre-existing RED failures unchanged | + +Pre-existing RED failures (not Plan 03 scope): +- test_cloud_analysis_api.py: 10 failures — estimate/jobs/controls routes (Plans 04+) +- test_cloud_analysis_contract.py: 14 failures — analysis contract (Plans 04+) +- test_extractor.py::test_extract_docx: pre-existing docx env issue + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Unique constraint prevents duplicate insert for evicted entries** +- **Found during:** Task 1 implementation +- **Issue:** `retain_or_reuse_cache_entry` attempted to create a new row for an evicted entry. The unique constraint on `(user_id, connection_id, cloud_item_id, version_key)` prevents duplicate inserts, raising `IntegrityError`. +- **Fix:** When an evicted entry exists for the same version_key, reactivate it (clear `evicted_at`, update `object_key` and `size_bytes`) instead of inserting. This is semantically correct: the bytes needed to be re-hydrated (created_new=True) but the DB row identity is preserved. +- **Files modified:** `backend/services/cloud_cache.py`, `backend/tests/test_cloud_cache.py` (test updated to match corrected semantics) +- **Commit:** 5bdd23f + +## Known Stubs + +None. + +## Threat Flags + +None — no new network endpoints beyond the intended /analysis/cache routes documented in the plan. Admin block and object_key exclusion enforced at schema and dependency level. + +## Self-Check: PASSED