docs(14-03): complete cache service and API plan

.planning/ROADMAP.md

@@ -25,7 +25,7 @@ Before any phase is marked complete:
 |------:|------|------|--------------|
 | 12 | 6/6 | Complete   | 2026-06-21 |
 | 13 | 11/11 | Complete   | 2026-06-23 |
-| 14 | 1/9 | In Progress|  |
+| 14 | 3/9 | In Progress|  |
 | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 |
 | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 |
 
@@ -111,14 +111,14 @@ Plans:
 
 **Depends on:** Phase 13
 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05
-**Plans:** 1/9 plans executed
+**Plans:** 3/9 plans executed
 **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08.
 
 Plans:
 
 - [x] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives.
-- [ ] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers.
+- [x] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers.
-- [ ] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API.
+- [x] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API.
 - [ ] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation.
 - [ ] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation.
 - [ ] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle.

.planning/STATE.md

@@ -6,15 +6,15 @@ current_phase: 14
 current_phase_name: selective-analysis-and-byte-cache
 status: in_progress
 stopped_at: Phase 14 Plan 02 complete
-last_updated: "2026-06-23T14:16:33.000Z"
+last_updated: "2026-06-23T13:36:12.544Z"
 last_activity: 2026-06-23
-last_activity_desc: Phase 14 Plan 02 — schema and service helpers complete
+last_activity_desc: Phase 14 Plan 02 schema + service helpers complete
 progress:
   total_phases: 6
   completed_phases: 3
   total_plans: 30
   completed_plans: 24
-  percent: 53
+  percent: 50
 ---
 
 # Project State
@@ -26,7 +26,7 @@ progress:
 ## Current Position
 
 Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS
-Plan: 3 of 9
+Plan: 4 of 9
 Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API)
 Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete
 
@@ -64,6 +64,7 @@ Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete
 | Phase 13 P11 | 20m | 2 tasks | 6 files |
 | Phase 14 P01 | 15m | 2 tasks | 5 files |
 | Phase 14 P02 | 12m | 2 tasks | 5 files |
+| Phase 14 P03 | 14m | - tasks | - files |
 
 ## Accumulated Context
 
@@ -111,7 +112,7 @@ _Updated at each phase transition._
 
 | Field | Value |
 |---|---|
-| Last session | 2026-06-23T14:16:33.000Z |
+| Last session | 2026-06-23T13:34:48.853Z |
 | Next action | Execute 14-03 (analysis API router) |
 | Pending decisions | None |
 | Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md |
@@ -147,3 +148,6 @@ _Updated at each phase transition._
 - [Phase 14 P02]: content_hash is optional post-hydration supplement, never a pre-download precondition (D-20)
 - [Phase 14 P02]: evict_lru_entries stamps evicted_at only; caller owns MinIO deletion and quota decrement (separation of concerns)
 - [Phase 14 P02]: user_analysis_settings is distinct from system_settings — single authority for per-user analysis preferences
+- [Phase ?]: [Phase 14 P03]: retain_or_reuse_cache_entry reactivates evicted rows — unique constraint prevents duplicate insert for same version_key
+- [Phase ?]: [Phase 14 P03]: api/cloud/analysis.py as aggregator router so Plan 01 RED tests can import before all routes implemented
+- [Phase ?]: [Phase 14 P03]: CacheStatusOut enforces T-14-02/T-14-08 by design — object_key absent at type level

.planning/phases/14-selective-analysis-and-byte-cache/14-03-SUMMARY.md

@@ -0,0 +1,161 @@
+---
+phase: "14"
+plan: "03"
+subsystem: cloud-analysis
+tags: [cache, api, security, quota, pinning, settings]
+status: complete
+
+requires:
+  - phases/14-selective-analysis-and-byte-cache/14-02
+
+provides:
+  - services/cloud_cache.py: retain_or_reuse_cache_entry, pin_cache_entry,
+    release_cache_entry, update_cache_access — Phase 14 cache lifecycle
+    orchestration (PATTERNS.md steps 3, 7, 8)
+  - api/cloud/cache.py: GET /analysis/cache, PATCH /analysis/cache/settings
+    — aggregate cache usage/settings, no object_key or credentials in response
+  - api/cloud/analysis.py: analysis router aggregator (parent for Phase 14
+    plans 04-09)
+  - api/cloud/schemas.py: CacheStatusOut, CacheSettingsUpdateRequest
+
+affects:
+  - backend/services/cloud_cache.py
+  - backend/api/cloud/cache.py (new)
+  - backend/api/cloud/analysis.py (new)
+  - backend/api/cloud/schemas.py
+  - backend/api/cloud/__init__.py
+  - backend/tests/test_cloud_cache.py
+  - backend/tests/test_cloud_security.py
+
+tech-stack:
+  added: []
+  patterns:
+    - retain-or-reactivate: unique constraint means evicted entries are
+      reactivated (evicted_at cleared, object_key updated) rather than
+      duplicated — semantically equivalent to "cache miss requires new bytes"
+    - pin_count lifecycle: pin_cache_entry / release_cache_entry with
+      GREATEST(0, ...) clamp for defensive double-release handling
+    - analysis router aggregator: api/cloud/analysis.py is the future-proof
+      parent for all Phase 14 analysis routes (estimates, jobs, controls)
+    - CacheStatusOut allowlist: schema enforces T-14-02/T-14-08 by design —
+      object_key and credentials_enc absent at type level
+
+key-files:
+  created:
+    - backend/api/cloud/cache.py
+    - backend/api/cloud/analysis.py
+  modified:
+    - backend/services/cloud_cache.py
+    - backend/api/cloud/schemas.py
+    - backend/api/cloud/__init__.py
+    - backend/tests/test_cloud_cache.py
+    - backend/tests/test_cloud_security.py
+
+decisions:
+  - retain_or_reuse_cache_entry reactivates evicted rows (clears evicted_at,
+    updates object_key) rather than inserting a new row — the unique constraint
+    on (user_id, connection_id, cloud_item_id, version_key) makes insertion
+    impossible; reactivation is semantically correct (fresh bytes, same version)
+  - api/cloud/analysis.py created as the aggregator so tests can
+    `from api.cloud.analysis import router` as required by Plan 01 RED tests
+  - CacheStatusOut has no per-entry fields — only aggregate totals — ensuring
+    T-14-02 compliance at schema design level rather than runtime filtering
+
+metrics:
+  duration: "~18 minutes"
+  completed: "2026-06-23"
+  tasks: 2
+  files: 7
+---
+
+# Phase 14 Plan 03: Cache Service and API Summary
+
+One-liner: Cache orchestration functions (retain/reuse, pin, release, touch) and credential-free cache status/settings API with admin block and object_key exclusion.
+
+## What Was Built
+
+### Task 1: Cache orchestration functions
+
+**services/cloud_cache.py** — four new Phase 14 orchestration functions added after the Plan 02 primitives:
+
+- `retain_or_reuse_cache_entry(session, *, user_id, connection_id, cloud_item_id, provider_item_id, version_key, object_key, ...)`: PATTERNS.md step 3 implementation. Checks for a non-evicted matching entry first (reuse path). If the only matching entry is evicted, reactivates it (clears evicted_at, updates object_key/size) because the unique constraint prevents inserting a duplicate. Returns `(entry, created_new)` tuple. Ownership enforced — foreign user always gets a separate row.
+
+- `pin_cache_entry(session, *, entry_id, user_id)`: Increments `pin_count` to mark an active preview or download lease (PATTERNS.md step 7). Ownership asserted — raises `ValueError` for foreign entry IDs.
+
+- `release_cache_entry(session, *, entry_id, user_id)`: Decrements `pin_count` with `GREATEST(0, ...)` clamp (PATTERNS.md step 8). Ownership asserted.
+
+- `update_cache_access(session, *, entry_id, user_id)`: Touches `last_accessed_at` after a cache hit to maintain accurate LRU ordering. Ownership asserted.
+
+All four functions raise `ValueError` only — never `HTTPException` (CLAUDE.md service-layer rule).
+
+**Tests added to tests/test_cloud_cache.py** (9 new tests, Task 1):
+- retain_or_reuse returns existing active entry
+- retain_or_reuse creates new entry when absent
+- retain_or_reuse reactivates evicted entry (not duplicate)
+- retain_or_reuse isolates foreign users (T-14-06)
+- pin increments pin_count
+- release decrements pin_count
+- release clamps at zero (double-release defense)
+- pin raises ValueError on foreign entry (T-14-06)
+- update_cache_access refreshes last_accessed_at timestamp
+
+### Task 2: Cache settings/status API
+
+**api/cloud/cache.py** (new):
+- `GET /analysis/cache` — returns `CacheStatusOut` with aggregate entry_count, total_bytes, cache_limit_bytes, tier_cap_bytes, analysis_progress_detail, analysis_failure_behavior. Calls `get_or_create_user_analysis_settings` for defaults on first access. Rate-limited at 120/minute. Blocked for admin (get_regular_user).
+- `PATCH /analysis/cache/settings` — accepts `CacheSettingsUpdateRequest` (three explicit fields only — mass assignment prevention). Service layer validates enums and bounds; route re-raises ValueError as 422. Returns fresh CacheStatusOut. Rate-limited at 30/minute. Blocked for admin.
+
+**api/cloud/analysis.py** (new):
+- Aggregator router for `/api/cloud/analysis/*` namespace.
+- Includes `cache_router` now; will include estimate/jobs/controls routers in Plans 04-09.
+- Importable as `api.cloud.analysis` so Plan 01 RED tests can import the router.
+
+**api/cloud/schemas.py** (extended):
+- `CacheStatusOut`: explicit allowlist with 6 fields — no object_key, credentials_enc, or per-entry data possible.
+- `CacheSettingsUpdateRequest`: 3 explicit optional fields with `ge=1` constraint on byte limit.
+
+**api/cloud/__init__.py** (updated):
+- Imports and includes `analysis_router` from `api.cloud.analysis`.
+
+**Security tests added to tests/test_cloud_security.py** (7 new tests, Task 2):
+- admin blocked from GET /analysis/cache (403)
+- admin blocked from PATCH /analysis/cache/settings (403)
+- unauthenticated request blocked (401/403)
+- response excludes object_key and credentials_enc (T-14-02, T-14-08)
+- response includes expected aggregate fields
+- invalid enum value returns 422
+- cache limit above tier cap returns 422
+
+## Test Results
+
+| File | Tests | Pass | Notes |
+|------|-------|------|-------|
+| test_cloud_cache.py | 21 | 21 | All pass including 9 new Task 1 tests |
+| test_cloud_security.py -k cache | 8 | 8 | 7 new + 1 pre-existing |
+| Overall suite (excl. RED tests) | 808 | 808 | Pre-existing RED failures unchanged |
+
+Pre-existing RED failures (not Plan 03 scope):
+- test_cloud_analysis_api.py: 10 failures — estimate/jobs/controls routes (Plans 04+)
+- test_cloud_analysis_contract.py: 14 failures — analysis contract (Plans 04+)
+- test_extractor.py::test_extract_docx: pre-existing docx env issue
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] Unique constraint prevents duplicate insert for evicted entries**
+- **Found during:** Task 1 implementation
+- **Issue:** `retain_or_reuse_cache_entry` attempted to create a new row for an evicted entry. The unique constraint on `(user_id, connection_id, cloud_item_id, version_key)` prevents duplicate inserts, raising `IntegrityError`.
+- **Fix:** When an evicted entry exists for the same version_key, reactivate it (clear `evicted_at`, update `object_key` and `size_bytes`) instead of inserting. This is semantically correct: the bytes needed to be re-hydrated (created_new=True) but the DB row identity is preserved.
+- **Files modified:** `backend/services/cloud_cache.py`, `backend/tests/test_cloud_cache.py` (test updated to match corrected semantics)
+- **Commit:** 5bdd23f
+
+## Known Stubs
+
+None.
+
+## Threat Flags
+
+None — no new network endpoints beyond the intended /analysis/cache routes documented in the plan. Admin block and object_key exclusion enforced at schema and dependency level.
+
+## Self-Check: PASSED