192 lines
11 KiB
Markdown
192 lines
11 KiB
Markdown
---
|
|
phase: "14"
|
|
plan: "06"
|
|
subsystem: cloud-byte-cache
|
|
tags: [cache, preview, download, security, pin, hydration, minio]
|
|
status: complete
|
|
|
|
requires:
|
|
- phases/14-selective-analysis-and-byte-cache/14-03
|
|
- phases/14-selective-analysis-and-byte-cache/14-05
|
|
|
|
provides:
|
|
- services/cloud_cache.py: hydrate_and_cache_bytes — full cache lifecycle for
|
|
open/preview/download: cache-hit avoids provider fetch, cache-miss stores bytes
|
|
in MinIO and pins entry, pin always released in finally block (PATTERNS.md steps 3, 7, 8)
|
|
- api/cloud/operations.py: _make_minio_cache_wrapper — shared MinIO adapter for
|
|
cache operations (DRY, used by both preview and download endpoints)
|
|
- api/cloud/operations.py: preview_cloud_file and download_cloud_file now route
|
|
through hydrate_and_cache_bytes; version_key computed from CloudItem metadata,
|
|
no provider call required for cache key resolution
|
|
|
|
affects:
|
|
- backend/services/cloud_cache.py
|
|
- backend/api/cloud/operations.py
|
|
- backend/tests/test_cloud_cache.py
|
|
- backend/tests/test_cloud_security.py
|
|
- frontend/src/views/__tests__/CloudFolderOpenPreview.test.js
|
|
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- cache-hit-pin-read-release: hydrate_and_cache_bytes pins before MinIO read,
|
|
releases in finally — eviction cannot occur mid-response (PATTERNS.md steps 7, 8)
|
|
- cache-miss-fetch-store-pin-release: provider fetch → MinIO put → retain_or_reuse →
|
|
pin → release; all steps non-fatal (bytes returned even on MinIO/quota failure)
|
|
- quota-bypass-on-failure: CacheQuotaExceeded causes MinIO object deletion and
|
|
immediate return of provider bytes without caching — quota is never left inconsistent
|
|
- version-key-from-metadata: compute_version_key called with CloudItem.version/etag/
|
|
provider_size/modified_at — no provider round-trip needed for key resolution
|
|
- shared-minio-wrapper: _make_minio_cache_wrapper in operations.py eliminates
|
|
duplicate adapter code at two call sites (DRY)
|
|
- cache-transparent-frontend: response shapes from preview and download routes
|
|
unchanged — frontend never sees object_key, MinIO paths, or cache metadata
|
|
|
|
key-files:
|
|
created: []
|
|
modified:
|
|
- backend/services/cloud_cache.py
|
|
- backend/api/cloud/operations.py
|
|
- backend/tests/test_cloud_cache.py
|
|
- backend/tests/test_cloud_security.py
|
|
- frontend/src/views/__tests__/CloudFolderOpenPreview.test.js
|
|
|
|
decisions:
|
|
- hydrate_and_cache_bytes lives in services/cloud_cache.py (service layer), not in
|
|
the router — CLAUDE.md service-layer rule; router only calls the helper
|
|
- MinIO write failure and quota failure are non-fatal — provider bytes returned
|
|
without caching; caller experience is identical (just no cache benefit on miss)
|
|
- Cache-hit MinIO read failure falls through to provider fetch — robust degradation
|
|
so a corrupted MinIO entry does not break user access to the file
|
|
- Pin is acquired before MinIO read on cache hit so eviction cannot occur mid-response
|
|
- evict_lru_entries called after pin release on cache-miss path — post-hydration
|
|
LRU maintenance stays within user's configured cache limit
|
|
- Frontend tests required no response-shape changes — all 8 pre-existing tests pass
|
|
unchanged; 1 new test added to document T-14-02 cache transparency invariant
|
|
- _make_minio_cache_wrapper helper avoids inline class duplication at preview/download
|
|
|
|
metrics:
|
|
duration: "~10 minutes"
|
|
completed: "2026-06-23"
|
|
tasks: 2
|
|
files: 5
|
|
---
|
|
|
|
# Phase 14 Plan 06: Cache-Backed Open/Preview/Download Summary
|
|
|
|
One-liner: Preview and download routes now hydrate provider bytes through the Phase 14 durable byte cache — cache hits avoid provider round-trips, active entries are pinned during response streaming, and pins are always released in finally blocks.
|
|
|
|
## What Was Built
|
|
|
|
### Task 1: Backend open/preview/download routes use cache leases
|
|
|
|
**services/cloud_cache.py** — `hydrate_and_cache_bytes` (new, ~140 lines):
|
|
|
|
The canonical cache lifecycle helper for content routes (PATTERNS.md steps 3, 7, 8).
|
|
|
|
**Cache-hit path:**
|
|
1. Query `CloudByteCacheEntry` for a non-evicted matching row by `(user_id, connection_id, cloud_item_id, version_key)`.
|
|
2. Pin the entry before reading (`pin_count += 1`) so LRU eviction cannot occur mid-response (T-14-12 equivalent for preview/download).
|
|
3. Touch `last_accessed_at` to update LRU position.
|
|
4. Read bytes from MinIO via the private `object_key`.
|
|
5. Release pin in a `finally` block on all exit paths.
|
|
6. Return `(bytes, cache_entry_id)`.
|
|
|
|
**Cache-miss path:**
|
|
1. Call `fetch_fn()` to download bytes from the provider.
|
|
2. Write bytes to MinIO under a private UUID key.
|
|
3. Attempt atomic quota increment via `increment_quota_for_cache`.
|
|
4. If quota exceeded: delete the MinIO object, return provider bytes without caching.
|
|
5. Create or reactivate the cache entry via `retain_or_reuse_cache_entry`.
|
|
6. Pin the entry for the response duration.
|
|
7. Release pin in a `finally` block.
|
|
8. Run `evict_lru_entries` to stay within the user's configured byte limit.
|
|
9. Return `(bytes, cache_entry_id)`.
|
|
|
|
**Non-fatal failure policy:** MinIO write failure, quota exhaustion, and any other non-provider error return the bytes from the provider fetch path without bubbling up an exception. The caller always gets bytes.
|
|
|
|
---
|
|
|
|
**api/cloud/operations.py** — three changes:
|
|
|
|
1. **`_make_minio_cache_wrapper(backend)` helper** (new): Returns a `_MinIOCacheWrapper` instance adapting `MinIOBackend` to the `hydrate_and_cache_bytes` interface (`get_object`, `put_object`, `delete_object`). Shared by both `preview_cloud_file` and `download_cloud_file` (DRY).
|
|
|
|
2. **`preview_cloud_file` updated** (Steps 4-5 refactored):
|
|
- Step 4 now resolves the adapter only (no `get_object` call inline).
|
|
- Step 5 computes `version_key` from `CloudItem.version/etag/provider_size/modified_at` — no provider call needed.
|
|
- Reads user's `cloud_cache_limit_bytes` from `UserAnalysisSettings` (default 512 MB).
|
|
- Calls `hydrate_and_cache_bytes`; falls through to direct `adapter.get_object()` if no `CloudItem` row found.
|
|
- Binary-only preview enforcement (D-18) unchanged — `PreviewSupport.is_supported()` still gates before the adapter is resolved.
|
|
|
|
3. **`download_cloud_file` updated**:
|
|
- Same pattern: version_key from metadata, hydrate through cache, fallback to direct provider fetch for items without metadata rows.
|
|
- Error handling preserved — `_normalize_error` on adapter still surfaces `reauth_required` as 401.
|
|
|
|
**New tests (Task 1):**
|
|
|
|
`tests/test_cloud_cache.py` — 5 new tests (prefixed `test_hydrate_*`):
|
|
- `test_hydrate_cache_miss_fetches_from_provider`: cache miss calls `fetch_fn` once, stores in MinIO, creates cache entry
|
|
- `test_hydrate_cache_hit_skips_provider`: cache hit reads from MinIO, `fetch_fn` not called (T-14-03 equivalent)
|
|
- `test_hydrate_pin_released_after_cache_hit`: `pin_count == 0` after successful hit (PATTERNS.md step 8)
|
|
- `test_hydrate_cache_miss_pin_released`: no pin leak after miss path completes
|
|
- `test_hydrate_foreign_user_gets_separate_cache_entry`: foreign user cannot reuse owner's cache entry (T-14-06)
|
|
|
|
`tests/test_cloud_security.py` — 3 new tests:
|
|
- `test_preview_cache_hit_response_excludes_object_key`: response body/headers contain no `object_key` or `credentials_enc` (T-14-02)
|
|
- `test_download_response_excludes_object_key`: download headers contain no cache internals (T-14-02)
|
|
- `test_foreign_user_cannot_download_via_cache`: IDOR — ownership check runs before cache lookup (T-13-01)
|
|
|
|
### Task 2: Frontend tests preserve open/preview behavior
|
|
|
|
No response schema changes were required — the byte cache is entirely backend-transparent. The existing 8 frontend tests in `CloudFolderOpenPreview.test.js` pass unchanged.
|
|
|
|
**New test added (1):**
|
|
- `test_hydrate_cache_hit_skips_provider` in `cache_backed_response_has_no_object_key` describe block: verifies that the `openCloudFile` response processed by `CloudFolderView` never exposes `object_key`, MinIO cache paths, or `credentials_enc` in rendered HTML or API call arguments (T-14-02 frontend invariant).
|
|
|
|
## Test Results
|
|
|
|
| File | Tests | Pass | Notes |
|
|
|------|-------|------|-------|
|
|
| test_cloud_cache.py | 30 → 35 | 35 | 5 new hydrate tests |
|
|
| test_cloud_security.py -k cache/preview/download | 10 → 13 | 13 | 3 new Plan 06 security tests |
|
|
| CloudFolderOpenPreview.test.js | 8 → 9 | 9 | 1 new cache transparency test |
|
|
| Overall backend (excl. test_extractor.py docx env) | 693 | 693 | No regressions |
|
|
|
|
Pre-existing failures (not Plan 06 scope):
|
|
- `test_extractor.py::test_extract_docx`: pre-existing docx env issue (ModuleNotFoundError)
|
|
|
|
## Deviations from Plan
|
|
|
|
### Auto-fixed Issues
|
|
|
|
**1. [Rule 1 - Bug] CacheQuotaExceeded with no Quota row in test SQLite DB**
|
|
- **Found during:** Task 1 test authoring
|
|
- **Issue:** `hydrate_and_cache_bytes` calls `increment_quota_for_cache` which uses a raw SQL `UPDATE quotas SET...` — but test fixtures use random UUIDs without creating User+Quota rows. SQLite returns 0 rows → `CacheQuotaExceeded` raised → cache entry never created → assertion `len(entries) == 1` fails.
|
|
- **Fix:** Patched `services.cloud_cache.increment_quota_for_cache` with `AsyncMock()` in the three tests that need cache entry creation (`test_hydrate_cache_miss_fetches_from_provider`, `test_hydrate_cache_miss_pin_released`, `test_hydrate_foreign_user_gets_separate_cache_entry`). The quota contract is already tested by `test_cache_retain_increments_quota` and `test_cache_eviction_decrements_quota` in the existing RED suite.
|
|
- **Files modified:** `backend/tests/test_cloud_cache.py`
|
|
- **Commit:** dfc3350
|
|
|
|
**2. [Rule 2 - Missing] Shared MinIO wrapper to avoid code duplication**
|
|
- **Found during:** Task 1 implementation (both preview and download needed identical wrapper)
|
|
- **Issue:** Both `preview_cloud_file` and `download_cloud_file` would require identical inline `_MinIOWrapper` class definitions.
|
|
- **Fix:** Extracted `_make_minio_cache_wrapper(backend)` helper function at the top of `operations.py`, used at both call sites.
|
|
- **Files modified:** `backend/api/cloud/operations.py`
|
|
- **Commit:** dfc3350
|
|
|
|
## Known Stubs
|
|
|
|
None. The byte cache integration is complete for the preview and download routes. Caching is non-fatal on MinIO failure or quota exhaustion — the routes always return provider bytes as a fallback.
|
|
|
|
## Threat Flags
|
|
|
|
None new. The plan's mitigations are fully implemented:
|
|
|
|
| Threat | Mitigation | Status |
|
|
|--------|-----------|--------|
|
|
| T-14-02 object_key secrecy | object_key never in any response body/header | Verified by test |
|
|
| T-13-01 IDOR | ownership check (resolve_owned_connection) runs before cache lookup | Verified by test |
|
|
| T-14-12 cache pin leak | finally block releases pin on all exit paths (success, failure, MinIO error) | Implemented + tested |
|
|
| Provider URL exposure | No provider URL in open/preview/download response — DocuVault URLs only | Verified by rg check |
|
|
|
|
## Self-Check: PASSED
|