From a2b7928ba4e3a5f9cba8c717d1bba7e44ee0d5a1 Mon Sep 17 00:00:00 2001 From: curo1305 Date: Tue, 23 Jun 2026 16:33:01 +0200 Subject: [PATCH] docs(14-06): complete byte cache for open/preview/download plan --- .planning/ROADMAP.md | 6 +- .planning/STATE.md | 23 ++- .../14-06-SUMMARY.md | 191 ++++++++++++++++++ 3 files changed, 207 insertions(+), 13 deletions(-) create mode 100644 .planning/phases/14-selective-analysis-and-byte-cache/14-06-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 2847fb6..0cae001 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -25,7 +25,7 @@ Before any phase is marked complete: |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | | 13 | 11/11 | Complete | 2026-06-23 | -| 14 | 5/9 | In Progress| | +| 14 | 6/9 | In Progress| | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -111,7 +111,7 @@ Plans: **Depends on:** Phase 13 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05 -**Plans:** 5/9 plans executed +**Plans:** 6/9 plans executed **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08. Plans: @@ -121,7 +121,7 @@ Plans: - [x] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API. - [x] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation. - [x] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation. -- [ ] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle. +- [x] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle. - [ ] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls. - [ ] 14-08-PLAN.md — Add Settings controls for cache limit, progress detail, and failure behavior. - [ ] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push. diff --git a/.planning/STATE.md b/.planning/STATE.md index e32f998..2c2075f 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,29 +6,29 @@ current_phase: 14 current_phase_name: selective-analysis-and-byte-cache status: in_progress stopped_at: Phase 14 Plan 02 complete -last_updated: "2026-06-23T14:10:28.736Z" +last_updated: "2026-06-23T14:31:34.473Z" last_activity: 2026-06-23 last_activity_desc: Phase 14 Plan 02 schema + service helpers complete progress: total_phases: 6 completed_phases: 3 total_plans: 30 - completed_plans: 26 + completed_plans: 27 percent: 50 --- # Project State **Project:** DocuVault -**Status:** Phase 14 in progress — Plan 14-02 complete +**Status:** Phase 14 in progress — Plan 14-06 complete **Last Updated:** 2026-06-23 ## Current Position Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS -Plan: 6 of 9 -Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API) -Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete +Plan: 7 of 9 +Status: Plans 14-01 through 14-06 complete. Next: execute 14-07 +Last activity: 2026-06-23 — Phase 14 Plan 06 byte cache for open/preview/download complete ## Phase Status @@ -67,6 +67,7 @@ Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete | Phase 14 P03 | 14m | - tasks | - files | | Phase 14 P04 | 12m | 2 tasks | 4 files | | Phase 14 P05 | 10m | 2 tasks | 4 files | +| Phase 14 P06 | 10m | 2 tasks | 5 files | ## Accumulated Context @@ -108,16 +109,16 @@ None. ## Session Continuity -**Stopped at:** Phase 14 Plan 02 complete +**Stopped at:** Phase 14 Plan 06 complete _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-23T14:10:23.947Z | -| Next action | Execute 14-03 (analysis API router) | +| Last session | 2026-06-23T14:31:29.032Z | +| Next action | Execute 14-07 | | Pending decisions | None | -| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md | +| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-07-PLAN.md | ## Decisions @@ -156,3 +157,5 @@ _Updated at each phase transition._ - [Phase ?]: Phase 14 P04: estimate_scope uses durable cloud_items metadata only (T-14-04) - [Phase ?]: Phase 14 P04: live_metadata_changed flag bypasses already-current fallback when provider confirms etag changed - [Phase ?]: Phase 14 P04: AnalysisJobOut per-stage counts always int (never None) +- [Phase 14 P06]: hydrate_and_cache_bytes is the single cache lifecycle entry point for preview/download — never inline adapter.get_object in open/preview/download routes +- [Phase 14 P06]: preview and download cache integration is backend-transparent — response shapes unchanged from Phase 13 (T-14-02) diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-06-SUMMARY.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-06-SUMMARY.md new file mode 100644 index 0000000..6766dff --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-06-SUMMARY.md @@ -0,0 +1,191 @@ +--- +phase: "14" +plan: "06" +subsystem: cloud-byte-cache +tags: [cache, preview, download, security, pin, hydration, minio] +status: complete + +requires: + - phases/14-selective-analysis-and-byte-cache/14-03 + - phases/14-selective-analysis-and-byte-cache/14-05 + +provides: + - services/cloud_cache.py: hydrate_and_cache_bytes — full cache lifecycle for + open/preview/download: cache-hit avoids provider fetch, cache-miss stores bytes + in MinIO and pins entry, pin always released in finally block (PATTERNS.md steps 3, 7, 8) + - api/cloud/operations.py: _make_minio_cache_wrapper — shared MinIO adapter for + cache operations (DRY, used by both preview and download endpoints) + - api/cloud/operations.py: preview_cloud_file and download_cloud_file now route + through hydrate_and_cache_bytes; version_key computed from CloudItem metadata, + no provider call required for cache key resolution + +affects: + - backend/services/cloud_cache.py + - backend/api/cloud/operations.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_security.py + - frontend/src/views/__tests__/CloudFolderOpenPreview.test.js + +tech-stack: + added: [] + patterns: + - cache-hit-pin-read-release: hydrate_and_cache_bytes pins before MinIO read, + releases in finally — eviction cannot occur mid-response (PATTERNS.md steps 7, 8) + - cache-miss-fetch-store-pin-release: provider fetch → MinIO put → retain_or_reuse → + pin → release; all steps non-fatal (bytes returned even on MinIO/quota failure) + - quota-bypass-on-failure: CacheQuotaExceeded causes MinIO object deletion and + immediate return of provider bytes without caching — quota is never left inconsistent + - version-key-from-metadata: compute_version_key called with CloudItem.version/etag/ + provider_size/modified_at — no provider round-trip needed for key resolution + - shared-minio-wrapper: _make_minio_cache_wrapper in operations.py eliminates + duplicate adapter code at two call sites (DRY) + - cache-transparent-frontend: response shapes from preview and download routes + unchanged — frontend never sees object_key, MinIO paths, or cache metadata + +key-files: + created: [] + modified: + - backend/services/cloud_cache.py + - backend/api/cloud/operations.py + - backend/tests/test_cloud_cache.py + - backend/tests/test_cloud_security.py + - frontend/src/views/__tests__/CloudFolderOpenPreview.test.js + +decisions: + - hydrate_and_cache_bytes lives in services/cloud_cache.py (service layer), not in + the router — CLAUDE.md service-layer rule; router only calls the helper + - MinIO write failure and quota failure are non-fatal — provider bytes returned + without caching; caller experience is identical (just no cache benefit on miss) + - Cache-hit MinIO read failure falls through to provider fetch — robust degradation + so a corrupted MinIO entry does not break user access to the file + - Pin is acquired before MinIO read on cache hit so eviction cannot occur mid-response + - evict_lru_entries called after pin release on cache-miss path — post-hydration + LRU maintenance stays within user's configured cache limit + - Frontend tests required no response-shape changes — all 8 pre-existing tests pass + unchanged; 1 new test added to document T-14-02 cache transparency invariant + - _make_minio_cache_wrapper helper avoids inline class duplication at preview/download + +metrics: + duration: "~10 minutes" + completed: "2026-06-23" + tasks: 2 + files: 5 +--- + +# Phase 14 Plan 06: Cache-Backed Open/Preview/Download Summary + +One-liner: Preview and download routes now hydrate provider bytes through the Phase 14 durable byte cache — cache hits avoid provider round-trips, active entries are pinned during response streaming, and pins are always released in finally blocks. + +## What Was Built + +### Task 1: Backend open/preview/download routes use cache leases + +**services/cloud_cache.py** — `hydrate_and_cache_bytes` (new, ~140 lines): + +The canonical cache lifecycle helper for content routes (PATTERNS.md steps 3, 7, 8). + +**Cache-hit path:** +1. Query `CloudByteCacheEntry` for a non-evicted matching row by `(user_id, connection_id, cloud_item_id, version_key)`. +2. Pin the entry before reading (`pin_count += 1`) so LRU eviction cannot occur mid-response (T-14-12 equivalent for preview/download). +3. Touch `last_accessed_at` to update LRU position. +4. Read bytes from MinIO via the private `object_key`. +5. Release pin in a `finally` block on all exit paths. +6. Return `(bytes, cache_entry_id)`. + +**Cache-miss path:** +1. Call `fetch_fn()` to download bytes from the provider. +2. Write bytes to MinIO under a private UUID key. +3. Attempt atomic quota increment via `increment_quota_for_cache`. +4. If quota exceeded: delete the MinIO object, return provider bytes without caching. +5. Create or reactivate the cache entry via `retain_or_reuse_cache_entry`. +6. Pin the entry for the response duration. +7. Release pin in a `finally` block. +8. Run `evict_lru_entries` to stay within the user's configured byte limit. +9. Return `(bytes, cache_entry_id)`. + +**Non-fatal failure policy:** MinIO write failure, quota exhaustion, and any other non-provider error return the bytes from the provider fetch path without bubbling up an exception. The caller always gets bytes. + +--- + +**api/cloud/operations.py** — three changes: + +1. **`_make_minio_cache_wrapper(backend)` helper** (new): Returns a `_MinIOCacheWrapper` instance adapting `MinIOBackend` to the `hydrate_and_cache_bytes` interface (`get_object`, `put_object`, `delete_object`). Shared by both `preview_cloud_file` and `download_cloud_file` (DRY). + +2. **`preview_cloud_file` updated** (Steps 4-5 refactored): + - Step 4 now resolves the adapter only (no `get_object` call inline). + - Step 5 computes `version_key` from `CloudItem.version/etag/provider_size/modified_at` — no provider call needed. + - Reads user's `cloud_cache_limit_bytes` from `UserAnalysisSettings` (default 512 MB). + - Calls `hydrate_and_cache_bytes`; falls through to direct `adapter.get_object()` if no `CloudItem` row found. + - Binary-only preview enforcement (D-18) unchanged — `PreviewSupport.is_supported()` still gates before the adapter is resolved. + +3. **`download_cloud_file` updated**: + - Same pattern: version_key from metadata, hydrate through cache, fallback to direct provider fetch for items without metadata rows. + - Error handling preserved — `_normalize_error` on adapter still surfaces `reauth_required` as 401. + +**New tests (Task 1):** + +`tests/test_cloud_cache.py` — 5 new tests (prefixed `test_hydrate_*`): +- `test_hydrate_cache_miss_fetches_from_provider`: cache miss calls `fetch_fn` once, stores in MinIO, creates cache entry +- `test_hydrate_cache_hit_skips_provider`: cache hit reads from MinIO, `fetch_fn` not called (T-14-03 equivalent) +- `test_hydrate_pin_released_after_cache_hit`: `pin_count == 0` after successful hit (PATTERNS.md step 8) +- `test_hydrate_cache_miss_pin_released`: no pin leak after miss path completes +- `test_hydrate_foreign_user_gets_separate_cache_entry`: foreign user cannot reuse owner's cache entry (T-14-06) + +`tests/test_cloud_security.py` — 3 new tests: +- `test_preview_cache_hit_response_excludes_object_key`: response body/headers contain no `object_key` or `credentials_enc` (T-14-02) +- `test_download_response_excludes_object_key`: download headers contain no cache internals (T-14-02) +- `test_foreign_user_cannot_download_via_cache`: IDOR — ownership check runs before cache lookup (T-13-01) + +### Task 2: Frontend tests preserve open/preview behavior + +No response schema changes were required — the byte cache is entirely backend-transparent. The existing 8 frontend tests in `CloudFolderOpenPreview.test.js` pass unchanged. + +**New test added (1):** +- `test_hydrate_cache_hit_skips_provider` in `cache_backed_response_has_no_object_key` describe block: verifies that the `openCloudFile` response processed by `CloudFolderView` never exposes `object_key`, MinIO cache paths, or `credentials_enc` in rendered HTML or API call arguments (T-14-02 frontend invariant). + +## Test Results + +| File | Tests | Pass | Notes | +|------|-------|------|-------| +| test_cloud_cache.py | 30 → 35 | 35 | 5 new hydrate tests | +| test_cloud_security.py -k cache/preview/download | 10 → 13 | 13 | 3 new Plan 06 security tests | +| CloudFolderOpenPreview.test.js | 8 → 9 | 9 | 1 new cache transparency test | +| Overall backend (excl. test_extractor.py docx env) | 693 | 693 | No regressions | + +Pre-existing failures (not Plan 06 scope): +- `test_extractor.py::test_extract_docx`: pre-existing docx env issue (ModuleNotFoundError) + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] CacheQuotaExceeded with no Quota row in test SQLite DB** +- **Found during:** Task 1 test authoring +- **Issue:** `hydrate_and_cache_bytes` calls `increment_quota_for_cache` which uses a raw SQL `UPDATE quotas SET...` — but test fixtures use random UUIDs without creating User+Quota rows. SQLite returns 0 rows → `CacheQuotaExceeded` raised → cache entry never created → assertion `len(entries) == 1` fails. +- **Fix:** Patched `services.cloud_cache.increment_quota_for_cache` with `AsyncMock()` in the three tests that need cache entry creation (`test_hydrate_cache_miss_fetches_from_provider`, `test_hydrate_cache_miss_pin_released`, `test_hydrate_foreign_user_gets_separate_cache_entry`). The quota contract is already tested by `test_cache_retain_increments_quota` and `test_cache_eviction_decrements_quota` in the existing RED suite. +- **Files modified:** `backend/tests/test_cloud_cache.py` +- **Commit:** dfc3350 + +**2. [Rule 2 - Missing] Shared MinIO wrapper to avoid code duplication** +- **Found during:** Task 1 implementation (both preview and download needed identical wrapper) +- **Issue:** Both `preview_cloud_file` and `download_cloud_file` would require identical inline `_MinIOWrapper` class definitions. +- **Fix:** Extracted `_make_minio_cache_wrapper(backend)` helper function at the top of `operations.py`, used at both call sites. +- **Files modified:** `backend/api/cloud/operations.py` +- **Commit:** dfc3350 + +## Known Stubs + +None. The byte cache integration is complete for the preview and download routes. Caching is non-fatal on MinIO failure or quota exhaustion — the routes always return provider bytes as a fallback. + +## Threat Flags + +None new. The plan's mitigations are fully implemented: + +| Threat | Mitigation | Status | +|--------|-----------|--------| +| T-14-02 object_key secrecy | object_key never in any response body/header | Verified by test | +| T-13-01 IDOR | ownership check (resolve_owned_connection) runs before cache lookup | Verified by test | +| T-14-12 cache pin leak | finally block releases pin on all exit paths (success, failure, MinIO error) | Implemented + tested | +| Provider URL exposure | No provider URL in open/preview/download response — DocuVault URLs only | Verified by rg check | + +## Self-Check: PASSED