docs(14-06): complete byte cache for open/preview/download plan

.planning/ROADMAP.md

@@ -25,7 +25,7 @@ Before any phase is marked complete:
 |------:|------|------|--------------|
 | 12 | 6/6 | Complete   | 2026-06-21 |
 | 13 | 11/11 | Complete   | 2026-06-23 |
-| 14 | 5/9 | In Progress|  |
+| 14 | 6/9 | In Progress|  |
 | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 |
 | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 |
 
@@ -111,7 +111,7 @@ Plans:
 
 **Depends on:** Phase 13
 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05
-**Plans:** 5/9 plans executed
+**Plans:** 6/9 plans executed
 **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08.
 
 Plans:
@@ -121,7 +121,7 @@ Plans:
 - [x] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API.
 - [x] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation.
 - [x] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation.
-- [ ] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle.
+- [x] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle.
 - [ ] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls.
 - [ ] 14-08-PLAN.md — Add Settings controls for cache limit, progress detail, and failure behavior.
 - [ ] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push.

.planning/STATE.md

@@ -6,29 +6,29 @@ current_phase: 14
 current_phase_name: selective-analysis-and-byte-cache
 status: in_progress
 stopped_at: Phase 14 Plan 02 complete
-last_updated: "2026-06-23T14:10:28.736Z"
+last_updated: "2026-06-23T14:31:34.473Z"
 last_activity: 2026-06-23
 last_activity_desc: Phase 14 Plan 02 schema + service helpers complete
 progress:
   total_phases: 6
   completed_phases: 3
   total_plans: 30
-  completed_plans: 26
+  completed_plans: 27
   percent: 50
 ---
 
 # Project State
 
 **Project:** DocuVault
-**Status:** Phase 14 in progress — Plan 14-02 complete
+**Status:** Phase 14 in progress — Plan 14-06 complete
 **Last Updated:** 2026-06-23
 
 ## Current Position
 
 Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS
-Plan: 6 of 9
+Plan: 7 of 9
-Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API)
+Status: Plans 14-01 through 14-06 complete. Next: execute 14-07
-Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete
+Last activity: 2026-06-23 — Phase 14 Plan 06 byte cache for open/preview/download complete
 
 ## Phase Status
 
@@ -67,6 +67,7 @@ Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete
 | Phase 14 P03 | 14m | - tasks | - files |
 | Phase 14 P04 | 12m | 2 tasks | 4 files |
 | Phase 14 P05 | 10m | 2 tasks | 4 files |
+| Phase 14 P06 | 10m | 2 tasks | 5 files |
 
 ## Accumulated Context
 
@@ -108,16 +109,16 @@ None.
 
 ## Session Continuity
 
-**Stopped at:** Phase 14 Plan 02 complete
+**Stopped at:** Phase 14 Plan 06 complete
 
 _Updated at each phase transition._
 
 | Field | Value |
 |---|---|
-| Last session | 2026-06-23T14:10:23.947Z |
+| Last session | 2026-06-23T14:31:29.032Z |
-| Next action | Execute 14-03 (analysis API router) |
+| Next action | Execute 14-07 |
 | Pending decisions | None |
-| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md |
+| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-07-PLAN.md |
 
 ## Decisions
 
@@ -156,3 +157,5 @@ _Updated at each phase transition._
 - [Phase ?]: Phase 14 P04: estimate_scope uses durable cloud_items metadata only (T-14-04)
 - [Phase ?]: Phase 14 P04: live_metadata_changed flag bypasses already-current fallback when provider confirms etag changed
 - [Phase ?]: Phase 14 P04: AnalysisJobOut per-stage counts always int (never None)
+- [Phase 14 P06]: hydrate_and_cache_bytes is the single cache lifecycle entry point for preview/download — never inline adapter.get_object in open/preview/download routes
+- [Phase 14 P06]: preview and download cache integration is backend-transparent — response shapes unchanged from Phase 13 (T-14-02)

.planning/phases/14-selective-analysis-and-byte-cache/14-06-SUMMARY.md

@@ -0,0 +1,191 @@
+---
+phase: "14"
+plan: "06"
+subsystem: cloud-byte-cache
+tags: [cache, preview, download, security, pin, hydration, minio]
+status: complete
+
+requires:
+  - phases/14-selective-analysis-and-byte-cache/14-03
+  - phases/14-selective-analysis-and-byte-cache/14-05
+
+provides:
+  - services/cloud_cache.py: hydrate_and_cache_bytes — full cache lifecycle for
+    open/preview/download: cache-hit avoids provider fetch, cache-miss stores bytes
+    in MinIO and pins entry, pin always released in finally block (PATTERNS.md steps 3, 7, 8)
+  - api/cloud/operations.py: _make_minio_cache_wrapper — shared MinIO adapter for
+    cache operations (DRY, used by both preview and download endpoints)
+  - api/cloud/operations.py: preview_cloud_file and download_cloud_file now route
+    through hydrate_and_cache_bytes; version_key computed from CloudItem metadata,
+    no provider call required for cache key resolution
+
+affects:
+  - backend/services/cloud_cache.py
+  - backend/api/cloud/operations.py
+  - backend/tests/test_cloud_cache.py
+  - backend/tests/test_cloud_security.py
+  - frontend/src/views/__tests__/CloudFolderOpenPreview.test.js
+
+tech-stack:
+  added: []
+  patterns:
+    - cache-hit-pin-read-release: hydrate_and_cache_bytes pins before MinIO read,
+      releases in finally — eviction cannot occur mid-response (PATTERNS.md steps 7, 8)
+    - cache-miss-fetch-store-pin-release: provider fetch → MinIO put → retain_or_reuse →
+      pin → release; all steps non-fatal (bytes returned even on MinIO/quota failure)
+    - quota-bypass-on-failure: CacheQuotaExceeded causes MinIO object deletion and
+      immediate return of provider bytes without caching — quota is never left inconsistent
+    - version-key-from-metadata: compute_version_key called with CloudItem.version/etag/
+      provider_size/modified_at — no provider round-trip needed for key resolution
+    - shared-minio-wrapper: _make_minio_cache_wrapper in operations.py eliminates
+      duplicate adapter code at two call sites (DRY)
+    - cache-transparent-frontend: response shapes from preview and download routes
+      unchanged — frontend never sees object_key, MinIO paths, or cache metadata
+
+key-files:
+  created: []
+  modified:
+    - backend/services/cloud_cache.py
+    - backend/api/cloud/operations.py
+    - backend/tests/test_cloud_cache.py
+    - backend/tests/test_cloud_security.py
+    - frontend/src/views/__tests__/CloudFolderOpenPreview.test.js
+
+decisions:
+  - hydrate_and_cache_bytes lives in services/cloud_cache.py (service layer), not in
+    the router — CLAUDE.md service-layer rule; router only calls the helper
+  - MinIO write failure and quota failure are non-fatal — provider bytes returned
+    without caching; caller experience is identical (just no cache benefit on miss)
+  - Cache-hit MinIO read failure falls through to provider fetch — robust degradation
+    so a corrupted MinIO entry does not break user access to the file
+  - Pin is acquired before MinIO read on cache hit so eviction cannot occur mid-response
+  - evict_lru_entries called after pin release on cache-miss path — post-hydration
+    LRU maintenance stays within user's configured cache limit
+  - Frontend tests required no response-shape changes — all 8 pre-existing tests pass
+    unchanged; 1 new test added to document T-14-02 cache transparency invariant
+  - _make_minio_cache_wrapper helper avoids inline class duplication at preview/download
+
+metrics:
+  duration: "~10 minutes"
+  completed: "2026-06-23"
+  tasks: 2
+  files: 5
+---
+
+# Phase 14 Plan 06: Cache-Backed Open/Preview/Download Summary
+
+One-liner: Preview and download routes now hydrate provider bytes through the Phase 14 durable byte cache — cache hits avoid provider round-trips, active entries are pinned during response streaming, and pins are always released in finally blocks.
+
+## What Was Built
+
+### Task 1: Backend open/preview/download routes use cache leases
+
+**services/cloud_cache.py** — `hydrate_and_cache_bytes` (new, ~140 lines):
+
+The canonical cache lifecycle helper for content routes (PATTERNS.md steps 3, 7, 8).
+
+**Cache-hit path:**
+1. Query `CloudByteCacheEntry` for a non-evicted matching row by `(user_id, connection_id, cloud_item_id, version_key)`.
+2. Pin the entry before reading (`pin_count += 1`) so LRU eviction cannot occur mid-response (T-14-12 equivalent for preview/download).
+3. Touch `last_accessed_at` to update LRU position.
+4. Read bytes from MinIO via the private `object_key`.
+5. Release pin in a `finally` block on all exit paths.
+6. Return `(bytes, cache_entry_id)`.
+
+**Cache-miss path:**
+1. Call `fetch_fn()` to download bytes from the provider.
+2. Write bytes to MinIO under a private UUID key.
+3. Attempt atomic quota increment via `increment_quota_for_cache`.
+4. If quota exceeded: delete the MinIO object, return provider bytes without caching.
+5. Create or reactivate the cache entry via `retain_or_reuse_cache_entry`.
+6. Pin the entry for the response duration.
+7. Release pin in a `finally` block.
+8. Run `evict_lru_entries` to stay within the user's configured byte limit.
+9. Return `(bytes, cache_entry_id)`.
+
+**Non-fatal failure policy:** MinIO write failure, quota exhaustion, and any other non-provider error return the bytes from the provider fetch path without bubbling up an exception. The caller always gets bytes.
+
+---
+
+**api/cloud/operations.py** — three changes:
+
+1. **`_make_minio_cache_wrapper(backend)` helper** (new): Returns a `_MinIOCacheWrapper` instance adapting `MinIOBackend` to the `hydrate_and_cache_bytes` interface (`get_object`, `put_object`, `delete_object`). Shared by both `preview_cloud_file` and `download_cloud_file` (DRY).
+
+2. **`preview_cloud_file` updated** (Steps 4-5 refactored):
+   - Step 4 now resolves the adapter only (no `get_object` call inline).
+   - Step 5 computes `version_key` from `CloudItem.version/etag/provider_size/modified_at` — no provider call needed.
+   - Reads user's `cloud_cache_limit_bytes` from `UserAnalysisSettings` (default 512 MB).
+   - Calls `hydrate_and_cache_bytes`; falls through to direct `adapter.get_object()` if no `CloudItem` row found.
+   - Binary-only preview enforcement (D-18) unchanged — `PreviewSupport.is_supported()` still gates before the adapter is resolved.
+
+3. **`download_cloud_file` updated**:
+   - Same pattern: version_key from metadata, hydrate through cache, fallback to direct provider fetch for items without metadata rows.
+   - Error handling preserved — `_normalize_error` on adapter still surfaces `reauth_required` as 401.
+
+**New tests (Task 1):**
+
+`tests/test_cloud_cache.py` — 5 new tests (prefixed `test_hydrate_*`):
+- `test_hydrate_cache_miss_fetches_from_provider`: cache miss calls `fetch_fn` once, stores in MinIO, creates cache entry
+- `test_hydrate_cache_hit_skips_provider`: cache hit reads from MinIO, `fetch_fn` not called (T-14-03 equivalent)
+- `test_hydrate_pin_released_after_cache_hit`: `pin_count == 0` after successful hit (PATTERNS.md step 8)
+- `test_hydrate_cache_miss_pin_released`: no pin leak after miss path completes
+- `test_hydrate_foreign_user_gets_separate_cache_entry`: foreign user cannot reuse owner's cache entry (T-14-06)
+
+`tests/test_cloud_security.py` — 3 new tests:
+- `test_preview_cache_hit_response_excludes_object_key`: response body/headers contain no `object_key` or `credentials_enc` (T-14-02)
+- `test_download_response_excludes_object_key`: download headers contain no cache internals (T-14-02)
+- `test_foreign_user_cannot_download_via_cache`: IDOR — ownership check runs before cache lookup (T-13-01)
+
+### Task 2: Frontend tests preserve open/preview behavior
+
+No response schema changes were required — the byte cache is entirely backend-transparent. The existing 8 frontend tests in `CloudFolderOpenPreview.test.js` pass unchanged.
+
+**New test added (1):**
+- `test_hydrate_cache_hit_skips_provider` in `cache_backed_response_has_no_object_key` describe block: verifies that the `openCloudFile` response processed by `CloudFolderView` never exposes `object_key`, MinIO cache paths, or `credentials_enc` in rendered HTML or API call arguments (T-14-02 frontend invariant).
+
+## Test Results
+
+| File | Tests | Pass | Notes |
+|------|-------|------|-------|
+| test_cloud_cache.py | 30 → 35 | 35 | 5 new hydrate tests |
+| test_cloud_security.py -k cache/preview/download | 10 → 13 | 13 | 3 new Plan 06 security tests |
+| CloudFolderOpenPreview.test.js | 8 → 9 | 9 | 1 new cache transparency test |
+| Overall backend (excl. test_extractor.py docx env) | 693 | 693 | No regressions |
+
+Pre-existing failures (not Plan 06 scope):
+- `test_extractor.py::test_extract_docx`: pre-existing docx env issue (ModuleNotFoundError)
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] CacheQuotaExceeded with no Quota row in test SQLite DB**
+- **Found during:** Task 1 test authoring
+- **Issue:** `hydrate_and_cache_bytes` calls `increment_quota_for_cache` which uses a raw SQL `UPDATE quotas SET...` — but test fixtures use random UUIDs without creating User+Quota rows. SQLite returns 0 rows → `CacheQuotaExceeded` raised → cache entry never created → assertion `len(entries) == 1` fails.
+- **Fix:** Patched `services.cloud_cache.increment_quota_for_cache` with `AsyncMock()` in the three tests that need cache entry creation (`test_hydrate_cache_miss_fetches_from_provider`, `test_hydrate_cache_miss_pin_released`, `test_hydrate_foreign_user_gets_separate_cache_entry`). The quota contract is already tested by `test_cache_retain_increments_quota` and `test_cache_eviction_decrements_quota` in the existing RED suite.
+- **Files modified:** `backend/tests/test_cloud_cache.py`
+- **Commit:** dfc3350
+
+**2. [Rule 2 - Missing] Shared MinIO wrapper to avoid code duplication**
+- **Found during:** Task 1 implementation (both preview and download needed identical wrapper)
+- **Issue:** Both `preview_cloud_file` and `download_cloud_file` would require identical inline `_MinIOWrapper` class definitions.
+- **Fix:** Extracted `_make_minio_cache_wrapper(backend)` helper function at the top of `operations.py`, used at both call sites.
+- **Files modified:** `backend/api/cloud/operations.py`
+- **Commit:** dfc3350
+
+## Known Stubs
+
+None. The byte cache integration is complete for the preview and download routes. Caching is non-fatal on MinIO failure or quota exhaustion — the routes always return provider bytes as a fallback.
+
+## Threat Flags
+
+None new. The plan's mitigations are fully implemented:
+
+| Threat | Mitigation | Status |
+|--------|-----------|--------|
+| T-14-02 object_key secrecy | object_key never in any response body/header | Verified by test |
+| T-13-01 IDOR | ownership check (resolve_owned_connection) runs before cache lookup | Verified by test |
+| T-14-12 cache pin leak | finally block releases pin on all exit paths (success, failure, MinIO error) | Implemented + tested |
+| Provider URL exposure | No provider URL in open/preview/download response — DocuVault URLs only | Verified by rg check |
+
+## Self-Check: PASSED