docs(14-09): Phase 14 closeout — docs, version bump v0.3.0→v0.4.0, security evidence

- CLAUDE.md/AGENTS.md: update current-state line to v0.4.0 Phase 14 complete;
  add Phase 14 shared module map entries (cloud_cache, cloud_analysis,
  cloud_analysis_processing, cloud_analysis_tasks, analysis router, cache API);
  add Phase 14 non-negotiable rules (hydrate_and_cache_bytes single entry point,
  IDs-only broker payload, object_key/credentials_enc excluded at schema level)
- README.md: bump to v0.4.0; add Selective cloud analysis and Cloud byte cache
  features; add Analysis API table (Phase 14 endpoints)
- SECURITY.md: add Phase 14 threat register (T-14-01..T-14-15) with evidence,
  security gate results, and accepted risks
- backend/main.py: bump FastAPI version to 0.4.0
- frontend/package.json: bump version to 0.4.0
- ROADMAP.md: mark Phase 14 complete (9/9 plans), plan 14-09 checked off
- 14-VALIDATION.md: gate evidence, requirement coverage, acceptance criteria check
- 14-SECURITY.md: security evidence summary and gate checklist

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
2026-06-23 20:05:49 +02:00
co-authored by Claude Sonnet 4.6
parent 2d697a6036
commit 90e1b525f1
9 changed files with 287 additions and 18 deletions
+3 -3
View File
@@ -25,7 +25,7 @@ Before any phase is marked complete:
|------:|------|------|--------------|
| 12 | 6/6 | Complete | 2026-06-21 |
| 13 | 11/11 | Complete | 2026-06-23 |
| 14 | 8/9 | In Progress| |
| 14 | 9/9 | Complete | 2026-06-23 |
| 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 |
| 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 |
@@ -111,7 +111,7 @@ Plans:
**Depends on:** Phase 13
**Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05
**Plans:** 8/9 plans executed
**Plans:** 9/9 plans complete
**Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08.
Plans:
@@ -124,7 +124,7 @@ Plans:
- [x] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle.
- [x] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls.
- [x] 14-08-PLAN.md — Add Settings controls for cache limit, progress detail, and failure behavior.
- [ ] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push.
- [x] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push.
**Success Criteria:**
@@ -0,0 +1,52 @@
# Phase 14 — Security Evidence
**Phase:** 14 — Selective Analysis and Byte Cache
**Audit date:** 2026-06-23
**Auditor:** gsd-execute-phase security gate (claude-sonnet-4-6)
**ASVS Level:** L2
**Status:** SECURED — 0 open threats, 3 accepted risks
## Threat Register Summary
| Threat ID | Threat | Status |
|-----------|--------|--------|
| T-14-01 | IDOR — analysis job or cache entry crosses user boundary | CLOSED |
| T-14-02 | Credential/object_key disclosure in analysis or cache response | CLOSED |
| T-14-03 | Hidden provider file mutation during analysis | CLOSED |
| T-14-04 | Provider bytes downloaded during estimate | CLOSED |
| T-14-05 | Cache quota corruption | CLOSED |
| T-14-06 | Cross-user cache entry access | CLOSED |
| T-14-07 | Analysis job remains visible to admin | CLOSED |
| T-14-08 | Cache status endpoint exposes object_key or credentials_enc | CLOSED |
| T-14-09 | Eviction deletes bytes for pinned active job | CLOSED |
| T-14-10 | Provider credentials in Celery broker payload | CLOSED |
| T-14-11 | Provider mutation during Celery processing | CLOSED |
| T-14-12 | Cache pin leak on processing failure/cancellation | CLOSED |
| T-14-13 | Stale analysis result presented when provider metadata changed | CLOSED |
| T-14-14 | Byte cache delivers stale/wrong bytes | CLOSED |
| T-14-15 | Cache status endpoint exposes per-entry metadata | CLOSED |
Full threat register evidence is documented in `SECURITY.md` at the repository root
under the "Phase 14 — Selective Analysis and Byte Cache" section.
## Security Gate Checklist
- [x] `bandit -r backend/ --exclude backend/tests --severity-level high` — 0 HIGH, 0 MEDIUM
- [x] `npm audit --audit-level=high` — 0 vulnerabilities
- [x] `docker compose config --quiet` — no output (all env vars resolve)
- [x] `gitleaks detect --redact` — 0 new findings (3 pre-existing test-fixture strings from May 2026)
- [x] Backend tests pass: 856 passed
- [x] Frontend tests pass: 471 passed
- [x] Admin blocked from all /analysis/ and /analysis/cache endpoints
- [x] `object_key` and `credentials_enc` absent from `CacheStatusOut`, `AnalysisJobOut`, `AnalysisJobItemOut`
- [x] `process_cloud_analysis_item` Celery task payload contains IDs only
- [x] `FakeCloudAdapter.mutation_call_count == 0` after estimate and full pipeline test
- [x] `FakeCloudAdapter.byte_call_count == 0` after estimate test
## Accepted Risks
| Risk ID | Component | Accepted Risk | Rationale |
|---------|-----------|---------------|-----------|
| T-14-SC-pip | pip-audit local tooling | pip-audit not runnable against Python 3.12 requirements in Python 3.9 local env | No new Python packages added in Phase 14; existing packages audited in Phase 8; same accepted gap as Phases 12 and 13 |
| T-14-docx | test_extract_docx failure | ModuleNotFoundError: python-docx/libmagic not in local test environment | Pre-existing; unrelated to Phase 14 |
| T-14-quota-nonfatal | Quota increment failure is non-fatal for analysis | When quota increment fails, processing continues; cache entry skipped | Only MinIO storage step is omitted; extracted_text and topics durably persisted |
@@ -0,0 +1,62 @@
# Phase 14 — Validation Record
**Phase:** 14 — Selective Analysis and Byte Cache
**Validated:** 2026-06-23
**Validator:** gsd-execute-phase (claude-sonnet-4-6)
**Status:** COMPLETE — all gates passed
## Summary
Phase 14 ships selective cloud analysis (ANALYZE-01 through ANALYZE-07) and a durable byte cache
(CACHE-03, CACHE-04, CACHE-05). All 9 plans were executed, all verification gates pass, and the
version was bumped to v0.4.0 (minor increment per CLAUDE.md protocol for full phase completion).
## Gate Results
| Gate | Check | Result | Evidence |
|------|-------|--------|---------|
| Backend test suite | `pytest -q` | PASS | 856 passed, 18 skipped, 7 xfailed; 1 pre-existing failure (test_extract_docx — docx env) |
| Frontend test suite | `npm run test` | PASS | 471 passed (50 test files) |
| Bandit HIGH severity | `bandit -r backend/ --exclude backend/tests --severity-level high` | PASS | 0 HIGH, 0 MEDIUM; 23 Low (all pre-existing) |
| npm audit | `npm audit --audit-level=high` | PASS | 0 vulnerabilities |
| docker compose config | `docker compose config --quiet` | PASS | No output — all env vars resolve |
| Secret scan | `gitleaks detect --redact` | PASS | 3 pre-existing findings (all before Phase 14) |
| pip-audit | Tooling gap (Python 3.9 local env) | ACCEPTED | No new Python packages in Phase 14; Phase 8 audit clean |
## Requirement Coverage
| Requirement | What Was Built | Plans |
|-------------|---------------|-------|
| ANALYZE-01 | Single-file estimate from metadata; Celery processing | 14-01, 14-04, 14-05 |
| ANALYZE-02 | Multi-selection estimate with unsupported count | 14-01, 14-04 |
| ANALYZE-03 | Folder-tree and whole-connection estimate (recursive) | 14-01, 14-04 |
| ANALYZE-04 | Job status with simple/detailed aggregate and per-stage counts | 14-04, 14-07 |
| ANALYZE-05 | Per-item cancel, skip, retry; Cancel all batch | 14-04, 14-07 |
| ANALYZE-06 | Already-current detection via version_key; changed etag triggers requeue | 14-02, 14-04 |
| ANALYZE-07 | Provider mutation methods never called during analysis | 14-03, 14-05 |
| CACHE-03 | Cache entry created during analysis; browse never creates cache entries | 14-02, 14-03 |
| CACHE-04 | Configurable cache limit; LRU eviction skips pinned entries | 14-02, 14-03, 14-08 |
| CACHE-05 | Cache quota increment/decrement; per-user isolation | 14-02, 14-03 |
## Phase Acceptance Criteria Check
1. **Users can analyze one file, multi-selection, folder tree, or whole connection after reviewing recursive estimates.**
Status: PASS — estimate API + frontend modal implemented (14-04, 14-07).
2. **UI exposes aggregate and per-item states with retry and cancellation.**
Status: PASS — analysis queue with expand/collapse, per-item cancel/retry/skip, Cancel all batch (14-07).
3. **Analysis jobs are idempotent by cloud item and provider version/etag; never mutate the provider.**
Status: PASS — already-current detection, stale guard, mutation_call_count==0 test (14-04, 14-05).
4. **Cloud bytes cached only for active work and evicted by configurable limits without interrupting pinned active jobs.**
Status: PASS — LRU eviction skips pin_count>0, cache limit configurable, Settings UI (14-03, 14-08).
5. **Cache ownership, cache-key versioning, cancellation, duplicate-job, failure-retry, and eviction have dedicated tests.**
Status: PASS — test_cloud_cache.py (30 tests), test_cloud_analysis_contract.py (47 tests), test_cloud_analysis_api.py (14 tests), test_cloud_security.py (analysis/cache subset).
## Pre-Existing Issues (Not Phase 14 Scope)
- `test_extract_docx``ModuleNotFoundError: No module named 'docx'` — python-docx/libmagic not installed in local test environment. Pre-existing; not a Phase 14 regression.
- pip-audit local tooling gap — Python 3.9 local environment cannot install pip-audit for Python 3.12 requirements. Accepted risk (same gap as Phases 12 and 13). No new Python packages were added in Phase 14.
- gitleaks 3 pre-existing findings — all in test files committed before Phase 14 (May 2026).
+23 -6
View File
@@ -4,7 +4,7 @@
DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV).
**Current state:** v0.2.5 — Phase 12.1 Plan 03 complete (2026-06-22). Cloud frontend aligned with normalized API: `kind` field for item classification, `provider_item_id` for opaque provider navigation, server freshness state mapped verbatim, explicit breadcrumb lineage. Phase 12.1 in progress. Not cleared for public internet deployment.
**Current state:** v0.4.0 — Phase 14 complete (2026-06-23). Selective cloud analysis with per-file/selection/folder/connection scope, byte cache with LRU eviction and ownership-scoped pinning, Celery-backed processing pipeline (download/extract/classify), authorized open/preview/download routed through the durable cache, frontend analysis queue with per-item controls, and cache/settings controls in Settings. 856 backend + 471 frontend tests pass. Not cleared for public internet deployment.
## Stack
@@ -42,9 +42,18 @@ Before adding a helper, check if it belongs in an existing shared module:
| `backend/storage/exceptions.py` | `CloudConnectionError` — single canonical definition; all files import from here |
| `backend/ai/utils.py` | `strip_code_fences`, `parse_classification`, `parse_suggestions` — AI response parsing shared by all providers |
| `backend/services/auth.py` | `validate_password_strength(password)` — raises `ValueError`; routers catch and re-raise as `HTTPException` |
| `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability` — Phase 12 browse contract |
| `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing` — owner-scoped metadata service |
| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut` — credential-free response schemas |
| `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability`, `CloudMutableAdapter` — Phase 12/13 browse+mutation contract; all 4 providers implement both |
| `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` |
| `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state`, `apply_listing_and_finalize`, `ListingResult` — owner-scoped cloud metadata service |
| `backend/services/cloud_operations.py` | `keep_both_name`, `run_health_check`, `run_reconnect`, `run_upload`, `run_create_folder`, `run_rename`, `run_move`, `run_delete` — all mutation orchestration with stale guard, reconciliation, and metadata-only audit |
| `backend/services/cloud_cache.py` | `compute_version_key`, `create_cache_entry`, `evict_lru_entries`, `retain_or_reuse_cache_entry`, `pin_cache_entry`, `release_cache_entry`, `hydrate_and_cache_bytes` — Phase 14 byte cache lifecycle (CACHE-03/CACHE-04/CACHE-05) |
| `backend/services/cloud_analysis.py` | `estimate_scope`, `enqueue_analysis_job`, `get_analysis_job`, `cancel_job`, `cancel_job_item`, `skip_job_item`, `retry_job_item`, `check_scan_quota` — Phase 14 analysis orchestration (ANALYZE-01..07) |
| `backend/services/cloud_analysis_processing.py` | `process_job_item`, `ProcessingResult` — single-item pipeline: stale guard, cache hydration, extraction, classification, cooperative cancellation |
| `backend/tasks/cloud_analysis_tasks.py` | `process_cloud_analysis_item` — Celery task, ID-only broker payload, bounded retry (30s/90s/270s) |
| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `CloudMutationResult`, `CloudUploadResult`, `AnalysisEstimateOut`, `AnalysisJobOut`, `CacheStatusOut` — credential-free response schemas |
| `backend/api/cloud/analysis.py` | Analysis router aggregator — `/api/cloud/analysis/*` namespace |
| `backend/api/cloud/operations.py` | Cloud mutation routes (Phase 13) and authorized open/preview/download (Phase 13/14) |
| `backend/api/cloud/cache.py` | `GET /analysis/cache`, `PATCH /analysis/cache/settings` — aggregate cache status/settings, no object_key |
**Rules:**
- No router may define `_ip()`, `_get_ip()`, or any other local variant of `get_client_ip`. Import from `deps.utils`.
@@ -54,6 +63,14 @@ Before adding a helper, check if it belongs in an existing shared module:
- Service layer raises `ValueError` (or domain exceptions), never `HTTPException`. Only the router layer raises `HTTPException`.
- Cloud browse responses must use `api/cloud/schemas.py` whitelisted types — never return raw ORM objects.
- `reconcile_cloud_listing` is the single reconciliation entry point — no provider may update `cloud_items` rows directly.
- `NextcloudBackend` must NOT override `list_folder` — it inherits the canonical `WebDAVBackend` implementation.
- No caller of `adapter.list_folder` may independently call `update_folder_state(refresh_state="fresh")`. Use `apply_listing_and_finalize` from `services.cloud_items`.
- Phase 13/14 mutation responses use `JSONResponse` (not `HTTPException`) so `kind`/`reason` appear at response top level.
- All cloud mutation orchestration routes through `services.cloud_operations` — never inline in the router.
- All byte cache lifecycle calls route through `hydrate_and_cache_bytes` in `services.cloud_cache` — never call `adapter.get_object` directly in open/preview/download routes.
- `process_cloud_analysis_item` Celery task accepts UUIDs only — credentials decrypted inside the worker after revalidation (no credentials in broker payload).
- `CacheStatusOut` and `AnalysisJobOut` never include `object_key` or `credentials_enc` — enforced at schema level.
- `testCloudConnection` is an explicit user-initiated action only — never called as a side effect of folder navigation (D-13).
### Frontend: shared module map
@@ -127,9 +144,9 @@ Before any GSD workflow or agent edits project files, create or switch into a de
/gsd:progress — check status and advance workflow
```
### Current state: v0.2.1 — Phase 12 gap-closure complete (2026-06-20)
### Current state: v0.4.0 — Phase 14 complete (2026-06-23)
Phase 12 gap-closure: migration-gated Compose startup (migrate service, service_completed_successfully dependency gate on backend/worker/beat), 0005→0006 upgrade regression tests (test_migration_0006.py), RUNBOOK migration ops section. Full Phase 12 cloud browse foundation — CloudResourceAdapter contract, connection-ID browse API (`GET /api/cloud/connections/{id}/items`), durable cloud item metadata (CloudItem ORM), connection-ID routing (`/cloud/:connectionId`), capability-aware action rendering (CapabilityButton / aria-disabled), breadcrumb freshness indicators (refreshing/fresh/stale), session-only folder memory, display-name rename for cloud connections, and a dedicated security-negative integration suite (`test_cloud_security.py`). Cloud and local files share one StorageBrowser row/action implementation. The browse foundation is complete; Phase 13 adds cloud mutations (upload/move/delete) and Phase 14 adds byte caching. The app is functional but alpha-quality — not cleared for public internet deployment.
Phase 14 complete (2026-06-23): Selective cloud analysis with per-file, selection, folder, and connection scopes; scope estimates from durable metadata only (zero provider bytes during estimate); idempotent analysis jobs with already-current/unsupported detection; Celery processing pipeline (download → extract → classify) with cooperative cancellation, stale guard, bounded retry, and cache pinning in finally block; durable byte cache (CloudByteCacheEntry) with LRU eviction, per-user quota accounting, and ownership-scoped pinning; open/preview/download routed through the byte cache lifecycle; frontend analysis queue with per-item cancel/retry/skip and aggregate progress indicator; Settings controls for cache limit, progress detail, and failure behavior. 856 backend + 471 frontend tests pass. Not cleared for public internet deployment.
## Development Setup
+15 -4
View File
@@ -4,7 +4,7 @@
DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV).
**Current state:** v0.3.0 — Phase 13 complete 2026-06-23. Full virtual-local cloud operations: connection health/reconnect/disconnect, authorized open/preview, sequential upload queue with typed conflict resolution, create-folder/rename with collision retry and stale guard, move with descendant safety, delete with disclosure confirmation, and metadata-only audit events. 766 backend + 429 frontend tests pass. Not cleared for public internet deployment.
**Current state:** v0.4.0 — Phase 14 complete 2026-06-23. Selective cloud analysis with per-file/selection/folder/connection scope estimates (metadata-only, no bytes), idempotent Celery analysis jobs with cooperative cancellation, stale guard, and bounded retry, durable byte cache (CloudByteCacheEntry) with LRU eviction and ownership-scoped pinning, open/preview/download routed through the cache, frontend analysis queue with per-item controls and Settings integration. 856 backend + 471 frontend tests pass. Not cleared for public internet deployment.
## Stack
@@ -44,9 +44,15 @@ Before adding a helper, check if it belongs in an existing shared module:
| `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` — Phase 12.1 adds `normalize_nextcloud_url` |
| `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state`, `apply_listing_and_finalize`, `ListingResult` — owner-scoped cloud metadata service; `apply_listing_and_finalize` is the single shared freshness gate |
| `backend/services/cloud_operations.py` | `keep_both_name(filename, counter)` — canonical D-03/D-05 keep-both collision naming (counter before first extension); `run_health_check`, `run_reconnect`, `run_upload`, `run_create_folder`, `run_rename`, `run_move`, `run_delete` — all mutation orchestration with stale guard, reconciliation, and metadata-only audit |
| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest`, `CloudMutationResult`, `CloudConflictResult`, `CloudUploadResult` — credential-free cloud response schemas |
| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest`, `CloudMutationResult`, `CloudConflictResult`, `CloudUploadResult`, `AnalysisEstimateOut`, `AnalysisJobOut`, `CacheStatusOut` — credential-free cloud response schemas |
| `backend/storage/cloud_base.py` | Phase 13 adds `CloudMutableAdapter` extending `CloudResourceAdapter` — normalized mutation contract (upload, create_folder, rename, move, delete) with `CloudMutationResult` typed returns; all 4 providers implement this |
| `backend/api/cloud/operations.py` | Owner-scoped cloud mutation routes: `POST /upload`, `POST /folders`, `PATCH /rename`, `POST /move`, `DELETE /items/{id}`, `POST /connections/{id}/test`, `POST /connections/{id}/reconnect`, `GET /connections/{id}/open/{item_id}`, `GET /connections/{id}/download/{item_id}` |
| `backend/services/cloud_cache.py` | Phase 14: `compute_version_key`, `create_cache_entry`, `evict_lru_entries`, `retain_or_reuse_cache_entry`, `pin_cache_entry`, `release_cache_entry`, `update_cache_access`, `hydrate_and_cache_bytes` — byte cache lifecycle; `hydrate_and_cache_bytes` is the single entry point for open/preview/download byte hydration |
| `backend/services/cloud_analysis.py` | Phase 14: `estimate_scope`, `enqueue_analysis_job`, `get_analysis_job`, `list_analysis_jobs`, `cancel_job`, `cancel_job_item`, `skip_job_item`, `retry_job_item`, `check_scan_quota` — analysis job orchestration |
| `backend/services/cloud_analysis_processing.py` | Phase 14: `process_job_item`, `ProcessingResult` — single-item Celery processing pipeline (stale guard, cache hydration, extraction, classification, cooperative cancellation) |
| `backend/tasks/cloud_analysis_tasks.py` | Phase 14: `process_cloud_analysis_item` — Celery task (IDs only in broker payload; credentials decrypted inside worker after revalidation) |
| `backend/api/cloud/analysis.py` | Phase 14: analysis router aggregator — `/api/cloud/analysis/*` namespace (estimate, jobs, controls, cache settings) |
| `backend/api/cloud/cache.py` | Phase 14: `GET /analysis/cache`, `PATCH /analysis/cache/settings` — aggregate cache status/settings; admin blocked; object_key excluded at schema level |
**Rules:**
- No router may define `_ip()`, `_get_ip()`, or any other local variant of `get_client_ip`. Import from `deps.utils`.
@@ -60,6 +66,11 @@ Before adding a helper, check if it belongs in an existing shared module:
- All cloud mutation orchestration (health, reconnect, upload, create-folder, rename, move, delete) routes through `services.cloud_operations` — never inline in the router.
- No mutation route may independently manage folder state after a mutation. Reconciliation (upsert_cloud_item + update_folder_state) is called only via `services.cloud_operations` mutation helpers.
- `testCloudConnection` is an explicit user-initiated action only — never called as a side effect of folder navigation (D-13).
- Phase 14: All byte hydration for open/preview/download/analysis routes through `hydrate_and_cache_bytes` in `services.cloud_cache` — never call `adapter.get_object` directly in router handlers.
- Phase 14: `process_cloud_analysis_item` Celery task accepts UUIDs only — credentials decrypted inside the worker after ownership revalidation (no credentials in broker payload, T-14-10).
- Phase 14: `CacheStatusOut` and `AnalysisJobOut` schemas never include `object_key` or `credentials_enc` — enforced at schema design level.
- Phase 14: Analysis estimate and already-current detection use durable `cloud_items` metadata only — zero provider bytes downloaded during estimate (T-14-04).
- Phase 14: `translateAnalysisStatus` in `cloudConnections` store is the single translation source — components never translate internal status strings independently (D-06).
### Frontend: shared module map
@@ -133,9 +144,9 @@ Before any GSD workflow or agent edits project files, create or switch into a de
/gsd:progress — check status and advance workflow
```
### Current state: v0.3.0 — Phase 13 complete (2026-06-23)
### Current state: v0.4.0 — Phase 14 complete (2026-06-23)
Phase 13 (virtual-local cloud operations) complete. Cloud browse, health/reconnect/disconnect, authorized open/preview, upload queue, create-folder, rename, move, and delete are all shipped. Phase 14 (selective analysis and byte cache) is next. Not cleared for public internet deployment.
Phase 14 (selective analysis and byte cache) complete. Selective cloud analysis with scope estimates (metadata-only), Celery processing pipeline, durable byte cache with LRU eviction and pinning, analysis queue UI with per-item controls, and Settings integration are all shipped. Phase 15 (unified smart search) is next. Not cleared for public internet deployment.
## Development Setup
+21 -2
View File
@@ -1,6 +1,6 @@
# DocuVault
**Version 0.3.0 — Alpha**
**Version 0.4.0 — Alpha**
> **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared.
@@ -22,6 +22,8 @@ A self-hosted, multi-user document management platform with AI-powered topic cla
- **Cloud file management** — upload files into cloud folders via a sequential queue with typed conflict resolution (keep-both/replace/skip/cancel-all); create folders with bounded collision-retry naming; rename items; move items within a connection; delete with trash-or-permanent disclosure
- **Connection health and reconnect** — per-connection health badge and error diagnostics in Settings; auto-test after connect/reconnect; Reconnect and requires-reauth banners in the shared browser; no health probe on folder navigation (D-13)
- **Authorized cloud preview** — open and preview supported binary formats (PDF/images) through DocuVault's authorized proxy; Office/Workspace formats fall back to an ownership-checked authorized download endpoint; provider credentials and raw provider URLs are never exposed
- **Selective cloud analysis** — analyze individual files, multi-selections, folder trees, or entire connections after reviewing scope estimates; background Celery jobs download, extract, and classify cloud files without mutating the provider; per-item cancel, skip, and retry controls with aggregate and expandable queue progress
- **Cloud byte cache** — downloaded file bytes are stored in MinIO with LRU eviction, per-user quota accounting, and ownership-scoped pinning; open/preview/download reuse cached bytes transparently; configurable cache limit, progress detail level, and failure behavior in Settings
- **Auth** — email/password registration with HIBP breach check, optional TOTP 2FA, 810 backup codes, password reset by email, sign-out-all-devices
- **Admin panel** — user CRUD, quota assignment, per-user AI provider override, AI provider system configuration, audit log viewer with CSV export
- **Observability** — structured JSON logging via structlog, per-request correlation IDs, Loki + Promtail + Grafana stack included in Docker Compose
@@ -356,7 +358,24 @@ cd backend && pytest -m live_nextcloud tests/test_nextcloud_live.py
The live suite is excluded from ordinary CI runs. It is read-only — PROPFIND metadata requests only. No bytes are downloaded and no provider mutations are made. Missing variables produce a safe skip.
**Phase 13 complete (v0.3.0):** Cloud file upload, rename, move, delete, and create-folder are all implemented. See Cloud file management feature above. Phase 14 (byte download/cache for offline preview and selective analysis) remains future work.
**Phase 13 complete (v0.3.0):** Cloud file upload, rename, move, delete, and create-folder are all implemented. See Cloud file management feature above.
**Phase 14 complete (v0.4.0):** Selective cloud analysis and byte cache are implemented. See Selective cloud analysis and Cloud byte cache features above. Phase 15 (unified smart search) is next.
### Analysis API (Phase 14)
| Endpoint | Purpose |
|----------|---------|
| `POST /api/cloud/analysis/connections/{id}/estimate` | Scope estimate — file/selection/folder/connection; no bytes downloaded |
| `POST /api/cloud/analysis/connections/{id}/jobs` | Enqueue analysis job; returns job_id |
| `GET /api/cloud/analysis/jobs` | List jobs (optional connection_id/status filter) |
| `GET /api/cloud/analysis/jobs/{id}` | Job status (?detail=true for per-stage counts) |
| `POST /api/cloud/analysis/jobs/{id}/cancel` | Cancel batch |
| `POST /api/cloud/analysis/jobs/{id}/items/{iid}/cancel` | Cancel single item |
| `POST /api/cloud/analysis/jobs/{id}/items/{iid}/skip` | Skip queued/failed item |
| `POST /api/cloud/analysis/jobs/{id}/items/{iid}/retry` | Retry failed item |
| `GET /api/cloud/analysis/cache` | Aggregate cache usage and settings |
| `PATCH /api/cloud/analysis/cache/settings` | Update cache limit, progress detail, failure behavior |
---
+109 -1
View File
@@ -512,7 +512,14 @@ Phase 12.1 scopes only read-only browse, freshness truthfulness, and cross-provi
| T-13-02 | Credential exposure in mutation response | mitigate | CLOSED | All mutation endpoints return `JSONResponse` with `CloudMutationResult`-shaped body; `credentials_enc`, `access_token`, `refresh_token` absent; `test_cloud_security.py::test_browse_response_excludes_credentials_and_raw_fields` still passes |
| T-13-03 | SSRF via reconnect or upload URL | mitigate | CLOSED | WebDAV/Nextcloud upload and reconnect paths reuse `validate_cloud_url` from `storage.cloud_utils`; `test_ssrf_url_validation_invariants` still passes |
| T-13-04 | Audit log contains provider bytes or credentials | mitigate | CLOSED | `write_audit_log` in `services/cloud_operations.py` writes only `metadata_`-level fields (`kind`, `provider_item_id`, `display_name`, `byte_size`); `test_cloud_audit.py::test_upload_success_writes_metadata_only_audit_row`, `test_move_audit_contains_no_credentials`, `test_delete_audit_contains_no_bytes` |
| T-13-11 | Credential plaintext at provider boundary | mitigate | CLOSED | Credentials decrypted inside `run_*` helpers only; decrypted value not returned in any `CloudMutationResult`; `test_cloud_mutations.py` asserts typed results contain no token fields |
| T-13-05 | Audit trail accuracy | mitigate | CLOSED | `write_audit_log` called only on authoritative success paths (upload, move, delete in `api/cloud/operations.py` lines 804819, 927940, 10961112); non-success branches (conflict, offline, reauth) never reach `write_audit_log`; `test_cloud_audit.py::test_upload_conflict_does_not_write_false_overwrite_audit`, `test_rename_stale_does_not_write_false_rename_audit`, `test_delete_failed_does_not_write_false_audit_event` |
| T-13-06 | Cloud health UI — no background probe on navigation | mitigate | CLOSED | `CloudFolderView.vue` comment at line 86 "D-13: navigateTo does NOT probe provider health"; `testConnection` store action called nowhere from `navigateTo`, `load`, `loadFolder`, `handleBreadcrumbNavigate`, or `onMounted`; `CloudFolderRenderedFlow.test.js::no_health_probe_on_folder_navigate_D13` |
| T-13-07 | Preview and download UI — no raw provider URLs | mitigate | CLOSED | `CloudFolderView.vue::onFileOpen` calls `api.openCloudFile` (DocuVault-authorized endpoint); comment at line 379 "D-02: Never calls window.open() with a raw provider URL"; unsupported preview falls back to `api.downloadCloudFile` (also DocuVault-proxied, not a direct provider URL); `test_cloud_mutations.py::test_open_file_returns_authorized_download_url` asserts URL is a DocuVault-scoped endpoint |
| T-13-08 | Queue conflict decisions — explicit pause/resume | mitigate | CLOSED | `StorageBrowser.vue` conflict dialog requires explicit `upload-queue-resolve` emit for all five actions: `keep_both`, `replace`, `skip`, `cancel_all`, `retry`; no implicit or silent path exists; `StorageBrowser.capabilities.test.js::test_queue_requires_explicit_resolution` |
| T-13-09 | Reconnect/consent UX — Google Drive broader scope | mitigate | CLOSED | `SettingsCloudTab.vue` lines 249-280: `data-test="gdrive-scope-notice"` with copy "DocuVault requests access to all files in your Google Drive so you can browse, open, …"; reconnect affordance present; `SettingsCloudTab.health.test.js::test_gdrive_scope_notice` |
| T-13-10 | Mutable contract layer — centralized kind/reason vocabulary | mitigate | CLOSED | `storage/cloud_base.py` lines 259292: `MUT_KIND_*` and `MUT_REASON_*` constants defined as module-level strings; `MUT_KINDS = frozenset({…})` seals the vocabulary; all provider adapters import these constants; `test_cloud_provider_contract.py` exercises the contract |
| T-13-11 | Credential plaintext at provider boundary | mitigate | CLOSED | Credentials decrypted inside `_resolve_and_get_adapter` only (operations.py lines 103132); decrypted value never returned in any mutation result dict; `test_cloud_reconnect.py::test_reconnect_response_excludes_credentials` asserts no credential fields in response |
| T-13-12 | Reconciliation path — all writes via cloud_items.py | mitigate | CLOSED | Provider backends (google_drive_backend.py, onedrive_backend.py, webdav_backend.py, nextcloud_backend.py) contain zero calls to `upsert_cloud_item`, `reconcile_cloud_listing`, or `update_folder_state`; those functions are called exclusively from `api/cloud/operations.py` and `services/cloud_items.py`; grep confirms no provider-layer ORM writes |
| T-13-13 | Health check exposes connection status to wrong user | mitigate | CLOSED | `POST /connections/{id}/test` asserts ownership via `resolve_owned_connection`; returns `{"state": ..., "error_code": ..., "error_message": ...}` only (no credentials); `test_cloud_reconnect.py::test_health_check_wrong_owner_403` |
| T-13-14 | Reconnect stores tokens in browser/response | mitigate | CLOSED | `run_reconnect` persists refreshed credentials via `encrypt_credentials`; response is `{"status": "active"}` only; `test_cloud_reconnect.py::test_reconnect_persists_token_not_exposes_it` |
| T-13-15 | Drive broader-scope consent missing from UI | mitigate | CLOSED | `SettingsCloudTab.vue` contains `data-test="gdrive-scope-notice"` with "all files in your Google Drive" copy (D-17); `SettingsCloudTab.health.test.js::test_gdrive_scope_notice` |
@@ -599,3 +606,104 @@ docker compose config --quiet
### Phase 13 Unregistered Flags
None. Phase 13 adds a write surface (mutations) but all mutation endpoints are owner-scoped, credential-free in responses, SSRF-protected on WebDAV/Nextcloud paths, and produce metadata-only audit events. No unregistered attack surfaces were discovered during plan execution.
---
## Phase 14 — Selective Analysis and Byte Cache (2026-06-23)
**Audit date:** 2026-06-23
**Phase:** 14 — Selective Analysis and Byte Cache (plans 14-01 through 14-09)
**ASVS Level:** L2
**Auditor:** gsd-security-auditor (claude-sonnet-4-6)
**Phase scope:** Cloud analysis scope estimate without byte download, idempotent Celery jobs with cooperative cancellation and bounded retry, durable byte cache with LRU eviction and per-user quota accounting, ownership-scoped cache pinning in finally block, stale guard before provider byte fetch, open/preview/download routed through the byte cache lifecycle, admin blocked from all analysis/cache routes, object_key never in API responses.
### Phase 14 Threat Register
| Threat ID | Threat | Disposition | Status | Evidence |
|-----------|--------|-------------|--------|----------|
| T-14-01 | IDOR — analysis job or cache entry crosses user boundary | mitigate | CLOSED | `resolve_owned_connection` in all analysis orchestration functions; `get_analysis_job`/`list_job_items` raise `AnalysisJobNotFound` for foreign job IDs; `pin_cache_entry`/`release_cache_entry` assert user_id ownership; `test_cloud_security.py` — `test_admin_cannot_estimate`, `test_foreign_user_cannot_read_job_status`, `test_cache_entries_are_owner_scoped` |
| T-14-02 | Credential/object_key disclosure in analysis or cache response | mitigate | CLOSED | `CacheStatusOut` schema excludes `object_key` at design level (no field declared); `AnalysisJobOut`/`AnalysisJobItemOut` exclude `credentials_enc`, `access_token`, `object_key`; `test_cloud_security.py::test_job_response_excludes_credentials`, `test_cache_status_excludes_object_key` |
| T-14-03 | Hidden provider file mutation during analysis | mitigate | CLOSED | `process_job_item` uses `adapter.get_object` only; `test_cloud_analysis_contract.py::test_processing_never_calls_provider_mutation_methods` — `FakeCloudAdapter.mutation_call_count == 0` after full pipeline execution |
| T-14-04 | Provider bytes downloaded during estimate | mitigate | CLOSED | `estimate_scope` reads durable `cloud_items` metadata only; no adapter call in estimate path; `test_cloud_analysis_contract.py::test_estimate_no_byte_download` — `FakeCloudAdapter.byte_call_count == 0` after estimate |
| T-14-05 | Cache quota corruption | mitigate | CLOSED | `increment_quota_for_cache` and `decrement_quota_for_cache` use atomic `UPDATE quotas SET used_bytes = used_bytes + :delta WHERE …` pattern; `test_cloud_cache.py` — increment/decrement seam tests; quota failure is non-fatal (processing proceeds without cache entry) |
| T-14-06 | Cross-user cache entry access | mitigate | CLOSED | `retain_or_reuse_cache_entry` enforces `user_id` equality before any pin/read operation; `test_cloud_cache.py::test_retain_or_reuse_isolates_foreign_users`, `test_pin_raises_for_foreign_entry` |
| T-14-07 | Analysis job remains visible to admin | mitigate | CLOSED | All analysis routes use `get_regular_user` dependency — admin tokens receive 403; `test_cloud_security.py::test_admin_cannot_estimate`, `test_admin_cannot_enqueue` |
| T-14-08 | Cache status endpoint exposes object_key or credentials_enc | mitigate | CLOSED | `CacheStatusOut` schema lists only 6 aggregate fields; `GET /analysis/cache` returns `CacheStatusOut` only; `test_cloud_cache.py::test_cache_status_excludes_object_key`, `test_cloud_security.py::test_cache_response_excludes_object_key` |
| T-14-09 | Eviction deletes bytes for pinned active job | mitigate | CLOSED | `evict_lru_entries` filters `pin_count == 0`; pinned entries are never candidates for eviction; `test_cloud_cache.py::test_eviction_skips_pinned_entries` |
| T-14-10 | Provider credentials in Celery broker payload | mitigate | CLOSED | `process_cloud_analysis_item` task parameters are `job_id, item_id, user_id, connection_id, cloud_item_id` (UUIDs only); credentials decrypted inside the worker after ownership revalidation via `decrypt_credentials`; `test_cloud_analysis_contract.py::test_celery_broker_payload_contains_ids_only` |
| T-14-11 | Provider mutation during Celery processing | mitigate | CLOSED | Worker uses `adapter.get_object` only; `test_cloud_analysis_contract.py::test_processing_never_calls_provider_mutation_methods` |
| T-14-12 | Cache pin leak on processing failure/cancellation | mitigate | CLOSED | `release_cache_entry` called in `finally` block after `pin_cache_entry`; applies on success, exception, and cooperative cancellation; `test_cloud_analysis_contract.py::test_processing_cache_pin_released_on_cancellation` |
| T-14-13 | Stale analysis result presented when provider metadata changed | mitigate | CLOSED | Version key re-computed from current `CloudItem` metadata before byte fetch; if different, item marked `"stale"` and processing stops without get_object; final stale check after classification for late concurrent changes; `test_cloud_analysis_contract.py::test_processing_unchanged_version_skips_before_provider_byte_fetch` |
| T-14-14 | Byte cache used for analysis delivers stale/wrong bytes | mitigate | CLOSED | Cache key is `(user_id, connection_id, cloud_item_id, version_key)` — version key includes etag/version_id/fingerprint; wrong-version cache entries cannot match the current version_key |
| T-14-15 | Cache status endpoint exposes per-entry metadata | mitigate | CLOSED | `CacheStatusOut` is aggregate-only (entry_count, total_bytes, cache_limit_bytes, tier_cap_bytes, analysis_progress_detail, analysis_failure_behavior) — no per-entry rows, file names, or provider IDs in the response |
### Phase 14 Security Gate Evidence
**Gate 1 — Full backend test suite:**
```
cd backend && python3 -m pytest -q --tb=no
856 passed, 18 skipped, 4 deselected, 7 xfailed, 73 warnings in 109.48s
(1 pre-existing failure: test_extract_docx — ModuleNotFoundError: libmagic/python-docx not installed locally; unrelated to Phase 14)
```
**Gate 2 — Frontend test suite:**
```
cd frontend && npm run test
471 passed (50 test files) — 0 failures
```
**Gate 3 — Bandit (backend static analysis):**
```
python3 -m bandit -r backend/ --exclude backend/tests --severity-level high
Total issues (by severity): High: 0, Medium: 0, Low: 23
(All 23 Low-severity findings are pre-existing; 0 HIGH or MEDIUM findings; 0 #nosec suppressions)
```
**Gate 4 — npm audit (frontend dependency scan):**
```
cd frontend && npm audit --audit-level=high
found 0 vulnerabilities
```
**Gate 5 — pip-audit (backend dependency scan):**
pip-audit not installable in local Python 3.9 environment (requirements.txt targets Python 3.12). No new Python packages added in Phase 14 plans 01 through 09. Existing packages audited in Phase 8 with 0 critical/high CVEs. Security-critical packages remain pinned: PyJWT 2.12+, pwdlib[argon2], cryptography, pyotp, slowapi. Same tooling gap accepted in Phase 12 and Phase 13.
**Gate 6 — Hardcoded-secret scan:**
```
gitleaks detect --redact
3 findings — all pre-existing in commits from before Phase 14:
- auth.test.js:92 (2026-05-31) — test fixture access_token string
- test_cloud_utils.py:116 (2026-05-28) — test fixture credentials dict
- test_auth_deps.py:94 (2026-05-22) — tampered JWT test token
No findings in any Phase 14 file.
git ls-files '.env' '.env.*': only .env.example tracked
```
**Gate 7 — Owner/admin/credential-secrecy invariants:**
- `test_cloud_security.py::test_foreign_user_cannot_browse_cloud_item` — PASS (IDOR block)
- `test_cloud_security.py::test_admin_cannot_browse_cloud_connection` — PASS (admin block)
- `test_cloud_security.py::test_admin_cannot_estimate` — PASS (analysis admin block)
- `test_cloud_security.py::test_admin_cannot_enqueue` — PASS (enqueue admin block)
- `test_cloud_security.py::test_job_response_excludes_credentials` — PASS (no credentials in job response)
- `test_cloud_security.py::test_cache_response_excludes_object_key` — PASS (no object_key in cache status)
- `test_cloud_analysis_contract.py::test_estimate_no_byte_download` — PASS (zero byte calls)
- `test_cloud_analysis_contract.py::test_processing_never_calls_provider_mutation_methods` — PASS
- `test_cloud_cache.py::test_eviction_skips_pinned_entries` — PASS
**Gate 8 — docker compose config:**
```
docker compose config --quiet
(no output — all required environment variables resolve without error)
```
### Phase 14 Accepted Risks
| Risk ID | Component | Accepted Risk | Rationale |
|---------|-----------|---------------|-----------|
| T-14-SC-pip | pip-audit local tooling | pip-audit not runnable against Python 3.12 requirements in Python 3.9 local env | No new Python packages added in Phase 14; existing packages audited in Phase 8; same accepted gap as Phases 12 and 13 |
| T-14-docx | test_extract_docx failure | ModuleNotFoundError: python-docx/libmagic not in local test environment | Pre-existing; unrelated to Phase 14; all Phase 14 test coverage passes |
| T-14-quota-nonfatal | Quota increment failure is non-fatal for analysis | When quota increment fails (limit exceeded), processing continues and cache entry is skipped | Quota refusal does not block text extraction or classification; only the MinIO cache storage step is omitted; extracted_text and topics are still durably persisted |
### Phase 14 Unregistered Flags
None. Phase 14 adds read-only analysis (no provider mutation, no provider URL exposure) and a byte cache keyed by version/etag (no cross-user data path). All new routes are owner-scoped, admin-blocked, and credential-free by schema design. No unregistered attack surfaces were discovered during plan execution.
+1 -1
View File
@@ -244,7 +244,7 @@ async def lifespan(app: FastAPI):
# ── Application factory ───────────────────────────────────────────────────────
app = FastAPI(title="Document Scanner API", version="0.3.0", lifespan=lifespan)
app = FastAPI(title="Document Scanner API", version="0.4.0", lifespan=lifespan)
# Rate limiter state (slowapi)
app.state.limiter = auth_limiter
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "document-scanner-frontend",
"version": "0.3.0",
"version": "0.4.0",
"type": "module",
"scripts": {
"dev": "vite",