diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index efde4c6..3d79797 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -25,7 +25,7 @@ Before any phase is marked complete: |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | | 13 | 11/11 | Complete | 2026-06-23 | -| 14 | 8/9 | In Progress| | +| 14 | 9/9 | Complete | 2026-06-23 | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -111,7 +111,7 @@ Plans: **Depends on:** Phase 13 **Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05 -**Plans:** 8/9 plans executed +**Plans:** 9/9 plans complete **Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08. Plans: @@ -124,7 +124,7 @@ Plans: - [x] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle. - [x] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls. - [x] 14-08-PLAN.md — Add Settings controls for cache limit, progress detail, and failure behavior. -- [ ] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push. +- [x] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push. **Success Criteria:** diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-SECURITY.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-SECURITY.md new file mode 100644 index 0000000..8a7b725 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-SECURITY.md @@ -0,0 +1,52 @@ +# Phase 14 — Security Evidence + +**Phase:** 14 — Selective Analysis and Byte Cache +**Audit date:** 2026-06-23 +**Auditor:** gsd-execute-phase security gate (claude-sonnet-4-6) +**ASVS Level:** L2 +**Status:** SECURED — 0 open threats, 3 accepted risks + +## Threat Register Summary + +| Threat ID | Threat | Status | +|-----------|--------|--------| +| T-14-01 | IDOR — analysis job or cache entry crosses user boundary | CLOSED | +| T-14-02 | Credential/object_key disclosure in analysis or cache response | CLOSED | +| T-14-03 | Hidden provider file mutation during analysis | CLOSED | +| T-14-04 | Provider bytes downloaded during estimate | CLOSED | +| T-14-05 | Cache quota corruption | CLOSED | +| T-14-06 | Cross-user cache entry access | CLOSED | +| T-14-07 | Analysis job remains visible to admin | CLOSED | +| T-14-08 | Cache status endpoint exposes object_key or credentials_enc | CLOSED | +| T-14-09 | Eviction deletes bytes for pinned active job | CLOSED | +| T-14-10 | Provider credentials in Celery broker payload | CLOSED | +| T-14-11 | Provider mutation during Celery processing | CLOSED | +| T-14-12 | Cache pin leak on processing failure/cancellation | CLOSED | +| T-14-13 | Stale analysis result presented when provider metadata changed | CLOSED | +| T-14-14 | Byte cache delivers stale/wrong bytes | CLOSED | +| T-14-15 | Cache status endpoint exposes per-entry metadata | CLOSED | + +Full threat register evidence is documented in `SECURITY.md` at the repository root +under the "Phase 14 — Selective Analysis and Byte Cache" section. + +## Security Gate Checklist + +- [x] `bandit -r backend/ --exclude backend/tests --severity-level high` — 0 HIGH, 0 MEDIUM +- [x] `npm audit --audit-level=high` — 0 vulnerabilities +- [x] `docker compose config --quiet` — no output (all env vars resolve) +- [x] `gitleaks detect --redact` — 0 new findings (3 pre-existing test-fixture strings from May 2026) +- [x] Backend tests pass: 856 passed +- [x] Frontend tests pass: 471 passed +- [x] Admin blocked from all /analysis/ and /analysis/cache endpoints +- [x] `object_key` and `credentials_enc` absent from `CacheStatusOut`, `AnalysisJobOut`, `AnalysisJobItemOut` +- [x] `process_cloud_analysis_item` Celery task payload contains IDs only +- [x] `FakeCloudAdapter.mutation_call_count == 0` after estimate and full pipeline test +- [x] `FakeCloudAdapter.byte_call_count == 0` after estimate test + +## Accepted Risks + +| Risk ID | Component | Accepted Risk | Rationale | +|---------|-----------|---------------|-----------| +| T-14-SC-pip | pip-audit local tooling | pip-audit not runnable against Python 3.12 requirements in Python 3.9 local env | No new Python packages added in Phase 14; existing packages audited in Phase 8; same accepted gap as Phases 12 and 13 | +| T-14-docx | test_extract_docx failure | ModuleNotFoundError: python-docx/libmagic not in local test environment | Pre-existing; unrelated to Phase 14 | +| T-14-quota-nonfatal | Quota increment failure is non-fatal for analysis | When quota increment fails, processing continues; cache entry skipped | Only MinIO storage step is omitted; extracted_text and topics durably persisted | diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-VALIDATION.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-VALIDATION.md new file mode 100644 index 0000000..f0568d0 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-VALIDATION.md @@ -0,0 +1,62 @@ +# Phase 14 — Validation Record + +**Phase:** 14 — Selective Analysis and Byte Cache +**Validated:** 2026-06-23 +**Validator:** gsd-execute-phase (claude-sonnet-4-6) +**Status:** COMPLETE — all gates passed + +## Summary + +Phase 14 ships selective cloud analysis (ANALYZE-01 through ANALYZE-07) and a durable byte cache +(CACHE-03, CACHE-04, CACHE-05). All 9 plans were executed, all verification gates pass, and the +version was bumped to v0.4.0 (minor increment per CLAUDE.md protocol for full phase completion). + +## Gate Results + +| Gate | Check | Result | Evidence | +|------|-------|--------|---------| +| Backend test suite | `pytest -q` | PASS | 856 passed, 18 skipped, 7 xfailed; 1 pre-existing failure (test_extract_docx — docx env) | +| Frontend test suite | `npm run test` | PASS | 471 passed (50 test files) | +| Bandit HIGH severity | `bandit -r backend/ --exclude backend/tests --severity-level high` | PASS | 0 HIGH, 0 MEDIUM; 23 Low (all pre-existing) | +| npm audit | `npm audit --audit-level=high` | PASS | 0 vulnerabilities | +| docker compose config | `docker compose config --quiet` | PASS | No output — all env vars resolve | +| Secret scan | `gitleaks detect --redact` | PASS | 3 pre-existing findings (all before Phase 14) | +| pip-audit | Tooling gap (Python 3.9 local env) | ACCEPTED | No new Python packages in Phase 14; Phase 8 audit clean | + +## Requirement Coverage + +| Requirement | What Was Built | Plans | +|-------------|---------------|-------| +| ANALYZE-01 | Single-file estimate from metadata; Celery processing | 14-01, 14-04, 14-05 | +| ANALYZE-02 | Multi-selection estimate with unsupported count | 14-01, 14-04 | +| ANALYZE-03 | Folder-tree and whole-connection estimate (recursive) | 14-01, 14-04 | +| ANALYZE-04 | Job status with simple/detailed aggregate and per-stage counts | 14-04, 14-07 | +| ANALYZE-05 | Per-item cancel, skip, retry; Cancel all batch | 14-04, 14-07 | +| ANALYZE-06 | Already-current detection via version_key; changed etag triggers requeue | 14-02, 14-04 | +| ANALYZE-07 | Provider mutation methods never called during analysis | 14-03, 14-05 | +| CACHE-03 | Cache entry created during analysis; browse never creates cache entries | 14-02, 14-03 | +| CACHE-04 | Configurable cache limit; LRU eviction skips pinned entries | 14-02, 14-03, 14-08 | +| CACHE-05 | Cache quota increment/decrement; per-user isolation | 14-02, 14-03 | + +## Phase Acceptance Criteria Check + +1. **Users can analyze one file, multi-selection, folder tree, or whole connection after reviewing recursive estimates.** + Status: PASS — estimate API + frontend modal implemented (14-04, 14-07). + +2. **UI exposes aggregate and per-item states with retry and cancellation.** + Status: PASS — analysis queue with expand/collapse, per-item cancel/retry/skip, Cancel all batch (14-07). + +3. **Analysis jobs are idempotent by cloud item and provider version/etag; never mutate the provider.** + Status: PASS — already-current detection, stale guard, mutation_call_count==0 test (14-04, 14-05). + +4. **Cloud bytes cached only for active work and evicted by configurable limits without interrupting pinned active jobs.** + Status: PASS — LRU eviction skips pin_count>0, cache limit configurable, Settings UI (14-03, 14-08). + +5. **Cache ownership, cache-key versioning, cancellation, duplicate-job, failure-retry, and eviction have dedicated tests.** + Status: PASS — test_cloud_cache.py (30 tests), test_cloud_analysis_contract.py (47 tests), test_cloud_analysis_api.py (14 tests), test_cloud_security.py (analysis/cache subset). + +## Pre-Existing Issues (Not Phase 14 Scope) + +- `test_extract_docx` — `ModuleNotFoundError: No module named 'docx'` — python-docx/libmagic not installed in local test environment. Pre-existing; not a Phase 14 regression. +- pip-audit local tooling gap — Python 3.9 local environment cannot install pip-audit for Python 3.12 requirements. Accepted risk (same gap as Phases 12 and 13). No new Python packages were added in Phase 14. +- gitleaks 3 pre-existing findings — all in test files committed before Phase 14 (May 2026). diff --git a/AGENTS.md b/AGENTS.md index a63c66a..a418eb4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV). -**Current state:** v0.2.5 — Phase 12.1 Plan 03 complete (2026-06-22). Cloud frontend aligned with normalized API: `kind` field for item classification, `provider_item_id` for opaque provider navigation, server freshness state mapped verbatim, explicit breadcrumb lineage. Phase 12.1 in progress. Not cleared for public internet deployment. +**Current state:** v0.4.0 — Phase 14 complete (2026-06-23). Selective cloud analysis with per-file/selection/folder/connection scope, byte cache with LRU eviction and ownership-scoped pinning, Celery-backed processing pipeline (download/extract/classify), authorized open/preview/download routed through the durable cache, frontend analysis queue with per-item controls, and cache/settings controls in Settings. 856 backend + 471 frontend tests pass. Not cleared for public internet deployment. ## Stack @@ -42,9 +42,18 @@ Before adding a helper, check if it belongs in an existing shared module: | `backend/storage/exceptions.py` | `CloudConnectionError` — single canonical definition; all files import from here | | `backend/ai/utils.py` | `strip_code_fences`, `parse_classification`, `parse_suggestions` — AI response parsing shared by all providers | | `backend/services/auth.py` | `validate_password_strength(password)` — raises `ValueError`; routers catch and re-raise as `HTTPException` | -| `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability` — Phase 12 browse contract | -| `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing` — owner-scoped metadata service | -| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut` — credential-free response schemas | +| `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability`, `CloudMutableAdapter` — Phase 12/13 browse+mutation contract; all 4 providers implement both | +| `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` | +| `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state`, `apply_listing_and_finalize`, `ListingResult` — owner-scoped cloud metadata service | +| `backend/services/cloud_operations.py` | `keep_both_name`, `run_health_check`, `run_reconnect`, `run_upload`, `run_create_folder`, `run_rename`, `run_move`, `run_delete` — all mutation orchestration with stale guard, reconciliation, and metadata-only audit | +| `backend/services/cloud_cache.py` | `compute_version_key`, `create_cache_entry`, `evict_lru_entries`, `retain_or_reuse_cache_entry`, `pin_cache_entry`, `release_cache_entry`, `hydrate_and_cache_bytes` — Phase 14 byte cache lifecycle (CACHE-03/CACHE-04/CACHE-05) | +| `backend/services/cloud_analysis.py` | `estimate_scope`, `enqueue_analysis_job`, `get_analysis_job`, `cancel_job`, `cancel_job_item`, `skip_job_item`, `retry_job_item`, `check_scan_quota` — Phase 14 analysis orchestration (ANALYZE-01..07) | +| `backend/services/cloud_analysis_processing.py` | `process_job_item`, `ProcessingResult` — single-item pipeline: stale guard, cache hydration, extraction, classification, cooperative cancellation | +| `backend/tasks/cloud_analysis_tasks.py` | `process_cloud_analysis_item` — Celery task, ID-only broker payload, bounded retry (30s/90s/270s) | +| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `CloudMutationResult`, `CloudUploadResult`, `AnalysisEstimateOut`, `AnalysisJobOut`, `CacheStatusOut` — credential-free response schemas | +| `backend/api/cloud/analysis.py` | Analysis router aggregator — `/api/cloud/analysis/*` namespace | +| `backend/api/cloud/operations.py` | Cloud mutation routes (Phase 13) and authorized open/preview/download (Phase 13/14) | +| `backend/api/cloud/cache.py` | `GET /analysis/cache`, `PATCH /analysis/cache/settings` — aggregate cache status/settings, no object_key | **Rules:** - No router may define `_ip()`, `_get_ip()`, or any other local variant of `get_client_ip`. Import from `deps.utils`. @@ -54,6 +63,14 @@ Before adding a helper, check if it belongs in an existing shared module: - Service layer raises `ValueError` (or domain exceptions), never `HTTPException`. Only the router layer raises `HTTPException`. - Cloud browse responses must use `api/cloud/schemas.py` whitelisted types — never return raw ORM objects. - `reconcile_cloud_listing` is the single reconciliation entry point — no provider may update `cloud_items` rows directly. +- `NextcloudBackend` must NOT override `list_folder` — it inherits the canonical `WebDAVBackend` implementation. +- No caller of `adapter.list_folder` may independently call `update_folder_state(refresh_state="fresh")`. Use `apply_listing_and_finalize` from `services.cloud_items`. +- Phase 13/14 mutation responses use `JSONResponse` (not `HTTPException`) so `kind`/`reason` appear at response top level. +- All cloud mutation orchestration routes through `services.cloud_operations` — never inline in the router. +- All byte cache lifecycle calls route through `hydrate_and_cache_bytes` in `services.cloud_cache` — never call `adapter.get_object` directly in open/preview/download routes. +- `process_cloud_analysis_item` Celery task accepts UUIDs only — credentials decrypted inside the worker after revalidation (no credentials in broker payload). +- `CacheStatusOut` and `AnalysisJobOut` never include `object_key` or `credentials_enc` — enforced at schema level. +- `testCloudConnection` is an explicit user-initiated action only — never called as a side effect of folder navigation (D-13). ### Frontend: shared module map @@ -127,9 +144,9 @@ Before any GSD workflow or agent edits project files, create or switch into a de /gsd:progress — check status and advance workflow ``` -### Current state: v0.2.1 — Phase 12 gap-closure complete (2026-06-20) +### Current state: v0.4.0 — Phase 14 complete (2026-06-23) -Phase 12 gap-closure: migration-gated Compose startup (migrate service, service_completed_successfully dependency gate on backend/worker/beat), 0005→0006 upgrade regression tests (test_migration_0006.py), RUNBOOK migration ops section. Full Phase 12 cloud browse foundation — CloudResourceAdapter contract, connection-ID browse API (`GET /api/cloud/connections/{id}/items`), durable cloud item metadata (CloudItem ORM), connection-ID routing (`/cloud/:connectionId`), capability-aware action rendering (CapabilityButton / aria-disabled), breadcrumb freshness indicators (refreshing/fresh/stale), session-only folder memory, display-name rename for cloud connections, and a dedicated security-negative integration suite (`test_cloud_security.py`). Cloud and local files share one StorageBrowser row/action implementation. The browse foundation is complete; Phase 13 adds cloud mutations (upload/move/delete) and Phase 14 adds byte caching. The app is functional but alpha-quality — not cleared for public internet deployment. +Phase 14 complete (2026-06-23): Selective cloud analysis with per-file, selection, folder, and connection scopes; scope estimates from durable metadata only (zero provider bytes during estimate); idempotent analysis jobs with already-current/unsupported detection; Celery processing pipeline (download → extract → classify) with cooperative cancellation, stale guard, bounded retry, and cache pinning in finally block; durable byte cache (CloudByteCacheEntry) with LRU eviction, per-user quota accounting, and ownership-scoped pinning; open/preview/download routed through the byte cache lifecycle; frontend analysis queue with per-item cancel/retry/skip and aggregate progress indicator; Settings controls for cache limit, progress detail, and failure behavior. 856 backend + 471 frontend tests pass. Not cleared for public internet deployment. ## Development Setup diff --git a/CLAUDE.md b/CLAUDE.md index 5c68995..22c5a8b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV). -**Current state:** v0.3.0 — Phase 13 complete 2026-06-23. Full virtual-local cloud operations: connection health/reconnect/disconnect, authorized open/preview, sequential upload queue with typed conflict resolution, create-folder/rename with collision retry and stale guard, move with descendant safety, delete with disclosure confirmation, and metadata-only audit events. 766 backend + 429 frontend tests pass. Not cleared for public internet deployment. +**Current state:** v0.4.0 — Phase 14 complete 2026-06-23. Selective cloud analysis with per-file/selection/folder/connection scope estimates (metadata-only, no bytes), idempotent Celery analysis jobs with cooperative cancellation, stale guard, and bounded retry, durable byte cache (CloudByteCacheEntry) with LRU eviction and ownership-scoped pinning, open/preview/download routed through the cache, frontend analysis queue with per-item controls and Settings integration. 856 backend + 471 frontend tests pass. Not cleared for public internet deployment. ## Stack @@ -44,9 +44,15 @@ Before adding a helper, check if it belongs in an existing shared module: | `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` — Phase 12.1 adds `normalize_nextcloud_url` | | `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state`, `apply_listing_and_finalize`, `ListingResult` — owner-scoped cloud metadata service; `apply_listing_and_finalize` is the single shared freshness gate | | `backend/services/cloud_operations.py` | `keep_both_name(filename, counter)` — canonical D-03/D-05 keep-both collision naming (counter before first extension); `run_health_check`, `run_reconnect`, `run_upload`, `run_create_folder`, `run_rename`, `run_move`, `run_delete` — all mutation orchestration with stale guard, reconciliation, and metadata-only audit | -| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest`, `CloudMutationResult`, `CloudConflictResult`, `CloudUploadResult` — credential-free cloud response schemas | +| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest`, `CloudMutationResult`, `CloudConflictResult`, `CloudUploadResult`, `AnalysisEstimateOut`, `AnalysisJobOut`, `CacheStatusOut` — credential-free cloud response schemas | | `backend/storage/cloud_base.py` | Phase 13 adds `CloudMutableAdapter` extending `CloudResourceAdapter` — normalized mutation contract (upload, create_folder, rename, move, delete) with `CloudMutationResult` typed returns; all 4 providers implement this | | `backend/api/cloud/operations.py` | Owner-scoped cloud mutation routes: `POST /upload`, `POST /folders`, `PATCH /rename`, `POST /move`, `DELETE /items/{id}`, `POST /connections/{id}/test`, `POST /connections/{id}/reconnect`, `GET /connections/{id}/open/{item_id}`, `GET /connections/{id}/download/{item_id}` | +| `backend/services/cloud_cache.py` | Phase 14: `compute_version_key`, `create_cache_entry`, `evict_lru_entries`, `retain_or_reuse_cache_entry`, `pin_cache_entry`, `release_cache_entry`, `update_cache_access`, `hydrate_and_cache_bytes` — byte cache lifecycle; `hydrate_and_cache_bytes` is the single entry point for open/preview/download byte hydration | +| `backend/services/cloud_analysis.py` | Phase 14: `estimate_scope`, `enqueue_analysis_job`, `get_analysis_job`, `list_analysis_jobs`, `cancel_job`, `cancel_job_item`, `skip_job_item`, `retry_job_item`, `check_scan_quota` — analysis job orchestration | +| `backend/services/cloud_analysis_processing.py` | Phase 14: `process_job_item`, `ProcessingResult` — single-item Celery processing pipeline (stale guard, cache hydration, extraction, classification, cooperative cancellation) | +| `backend/tasks/cloud_analysis_tasks.py` | Phase 14: `process_cloud_analysis_item` — Celery task (IDs only in broker payload; credentials decrypted inside worker after revalidation) | +| `backend/api/cloud/analysis.py` | Phase 14: analysis router aggregator — `/api/cloud/analysis/*` namespace (estimate, jobs, controls, cache settings) | +| `backend/api/cloud/cache.py` | Phase 14: `GET /analysis/cache`, `PATCH /analysis/cache/settings` — aggregate cache status/settings; admin blocked; object_key excluded at schema level | **Rules:** - No router may define `_ip()`, `_get_ip()`, or any other local variant of `get_client_ip`. Import from `deps.utils`. @@ -60,6 +66,11 @@ Before adding a helper, check if it belongs in an existing shared module: - All cloud mutation orchestration (health, reconnect, upload, create-folder, rename, move, delete) routes through `services.cloud_operations` — never inline in the router. - No mutation route may independently manage folder state after a mutation. Reconciliation (upsert_cloud_item + update_folder_state) is called only via `services.cloud_operations` mutation helpers. - `testCloudConnection` is an explicit user-initiated action only — never called as a side effect of folder navigation (D-13). +- Phase 14: All byte hydration for open/preview/download/analysis routes through `hydrate_and_cache_bytes` in `services.cloud_cache` — never call `adapter.get_object` directly in router handlers. +- Phase 14: `process_cloud_analysis_item` Celery task accepts UUIDs only — credentials decrypted inside the worker after ownership revalidation (no credentials in broker payload, T-14-10). +- Phase 14: `CacheStatusOut` and `AnalysisJobOut` schemas never include `object_key` or `credentials_enc` — enforced at schema design level. +- Phase 14: Analysis estimate and already-current detection use durable `cloud_items` metadata only — zero provider bytes downloaded during estimate (T-14-04). +- Phase 14: `translateAnalysisStatus` in `cloudConnections` store is the single translation source — components never translate internal status strings independently (D-06). ### Frontend: shared module map @@ -133,9 +144,9 @@ Before any GSD workflow or agent edits project files, create or switch into a de /gsd:progress — check status and advance workflow ``` -### Current state: v0.3.0 — Phase 13 complete (2026-06-23) +### Current state: v0.4.0 — Phase 14 complete (2026-06-23) -Phase 13 (virtual-local cloud operations) complete. Cloud browse, health/reconnect/disconnect, authorized open/preview, upload queue, create-folder, rename, move, and delete are all shipped. Phase 14 (selective analysis and byte cache) is next. Not cleared for public internet deployment. +Phase 14 (selective analysis and byte cache) complete. Selective cloud analysis with scope estimates (metadata-only), Celery processing pipeline, durable byte cache with LRU eviction and pinning, analysis queue UI with per-item controls, and Settings integration are all shipped. Phase 15 (unified smart search) is next. Not cleared for public internet deployment. ## Development Setup diff --git a/README.md b/README.md index 86df5ab..239ced3 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # DocuVault -**Version 0.3.0 — Alpha** +**Version 0.4.0 — Alpha** > **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared. @@ -22,6 +22,8 @@ A self-hosted, multi-user document management platform with AI-powered topic cla - **Cloud file management** — upload files into cloud folders via a sequential queue with typed conflict resolution (keep-both/replace/skip/cancel-all); create folders with bounded collision-retry naming; rename items; move items within a connection; delete with trash-or-permanent disclosure - **Connection health and reconnect** — per-connection health badge and error diagnostics in Settings; auto-test after connect/reconnect; Reconnect and requires-reauth banners in the shared browser; no health probe on folder navigation (D-13) - **Authorized cloud preview** — open and preview supported binary formats (PDF/images) through DocuVault's authorized proxy; Office/Workspace formats fall back to an ownership-checked authorized download endpoint; provider credentials and raw provider URLs are never exposed +- **Selective cloud analysis** — analyze individual files, multi-selections, folder trees, or entire connections after reviewing scope estimates; background Celery jobs download, extract, and classify cloud files without mutating the provider; per-item cancel, skip, and retry controls with aggregate and expandable queue progress +- **Cloud byte cache** — downloaded file bytes are stored in MinIO with LRU eviction, per-user quota accounting, and ownership-scoped pinning; open/preview/download reuse cached bytes transparently; configurable cache limit, progress detail level, and failure behavior in Settings - **Auth** — email/password registration with HIBP breach check, optional TOTP 2FA, 8–10 backup codes, password reset by email, sign-out-all-devices - **Admin panel** — user CRUD, quota assignment, per-user AI provider override, AI provider system configuration, audit log viewer with CSV export - **Observability** — structured JSON logging via structlog, per-request correlation IDs, Loki + Promtail + Grafana stack included in Docker Compose @@ -356,7 +358,24 @@ cd backend && pytest -m live_nextcloud tests/test_nextcloud_live.py The live suite is excluded from ordinary CI runs. It is read-only — PROPFIND metadata requests only. No bytes are downloaded and no provider mutations are made. Missing variables produce a safe skip. -**Phase 13 complete (v0.3.0):** Cloud file upload, rename, move, delete, and create-folder are all implemented. See Cloud file management feature above. Phase 14 (byte download/cache for offline preview and selective analysis) remains future work. +**Phase 13 complete (v0.3.0):** Cloud file upload, rename, move, delete, and create-folder are all implemented. See Cloud file management feature above. + +**Phase 14 complete (v0.4.0):** Selective cloud analysis and byte cache are implemented. See Selective cloud analysis and Cloud byte cache features above. Phase 15 (unified smart search) is next. + +### Analysis API (Phase 14) + +| Endpoint | Purpose | +|----------|---------| +| `POST /api/cloud/analysis/connections/{id}/estimate` | Scope estimate — file/selection/folder/connection; no bytes downloaded | +| `POST /api/cloud/analysis/connections/{id}/jobs` | Enqueue analysis job; returns job_id | +| `GET /api/cloud/analysis/jobs` | List jobs (optional connection_id/status filter) | +| `GET /api/cloud/analysis/jobs/{id}` | Job status (?detail=true for per-stage counts) | +| `POST /api/cloud/analysis/jobs/{id}/cancel` | Cancel batch | +| `POST /api/cloud/analysis/jobs/{id}/items/{iid}/cancel` | Cancel single item | +| `POST /api/cloud/analysis/jobs/{id}/items/{iid}/skip` | Skip queued/failed item | +| `POST /api/cloud/analysis/jobs/{id}/items/{iid}/retry` | Retry failed item | +| `GET /api/cloud/analysis/cache` | Aggregate cache usage and settings | +| `PATCH /api/cloud/analysis/cache/settings` | Update cache limit, progress detail, failure behavior | --- diff --git a/SECURITY.md b/SECURITY.md index e651da8..7938b73 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -512,7 +512,14 @@ Phase 12.1 scopes only read-only browse, freshness truthfulness, and cross-provi | T-13-02 | Credential exposure in mutation response | mitigate | CLOSED | All mutation endpoints return `JSONResponse` with `CloudMutationResult`-shaped body; `credentials_enc`, `access_token`, `refresh_token` absent; `test_cloud_security.py::test_browse_response_excludes_credentials_and_raw_fields` still passes | | T-13-03 | SSRF via reconnect or upload URL | mitigate | CLOSED | WebDAV/Nextcloud upload and reconnect paths reuse `validate_cloud_url` from `storage.cloud_utils`; `test_ssrf_url_validation_invariants` still passes | | T-13-04 | Audit log contains provider bytes or credentials | mitigate | CLOSED | `write_audit_log` in `services/cloud_operations.py` writes only `metadata_`-level fields (`kind`, `provider_item_id`, `display_name`, `byte_size`); `test_cloud_audit.py::test_upload_success_writes_metadata_only_audit_row`, `test_move_audit_contains_no_credentials`, `test_delete_audit_contains_no_bytes` | -| T-13-11 | Credential plaintext at provider boundary | mitigate | CLOSED | Credentials decrypted inside `run_*` helpers only; decrypted value not returned in any `CloudMutationResult`; `test_cloud_mutations.py` asserts typed results contain no token fields | +| T-13-05 | Audit trail accuracy | mitigate | CLOSED | `write_audit_log` called only on authoritative success paths (upload, move, delete in `api/cloud/operations.py` lines 804–819, 927–940, 1096–1112); non-success branches (conflict, offline, reauth) never reach `write_audit_log`; `test_cloud_audit.py::test_upload_conflict_does_not_write_false_overwrite_audit`, `test_rename_stale_does_not_write_false_rename_audit`, `test_delete_failed_does_not_write_false_audit_event` | +| T-13-06 | Cloud health UI — no background probe on navigation | mitigate | CLOSED | `CloudFolderView.vue` comment at line 86 "D-13: navigateTo does NOT probe provider health"; `testConnection` store action called nowhere from `navigateTo`, `load`, `loadFolder`, `handleBreadcrumbNavigate`, or `onMounted`; `CloudFolderRenderedFlow.test.js::no_health_probe_on_folder_navigate_D13` | +| T-13-07 | Preview and download UI — no raw provider URLs | mitigate | CLOSED | `CloudFolderView.vue::onFileOpen` calls `api.openCloudFile` (DocuVault-authorized endpoint); comment at line 379 "D-02: Never calls window.open() with a raw provider URL"; unsupported preview falls back to `api.downloadCloudFile` (also DocuVault-proxied, not a direct provider URL); `test_cloud_mutations.py::test_open_file_returns_authorized_download_url` asserts URL is a DocuVault-scoped endpoint | +| T-13-08 | Queue conflict decisions — explicit pause/resume | mitigate | CLOSED | `StorageBrowser.vue` conflict dialog requires explicit `upload-queue-resolve` emit for all five actions: `keep_both`, `replace`, `skip`, `cancel_all`, `retry`; no implicit or silent path exists; `StorageBrowser.capabilities.test.js::test_queue_requires_explicit_resolution` | +| T-13-09 | Reconnect/consent UX — Google Drive broader scope | mitigate | CLOSED | `SettingsCloudTab.vue` lines 249-280: `data-test="gdrive-scope-notice"` with copy "DocuVault requests access to all files in your Google Drive so you can browse, open, …"; reconnect affordance present; `SettingsCloudTab.health.test.js::test_gdrive_scope_notice` | +| T-13-10 | Mutable contract layer — centralized kind/reason vocabulary | mitigate | CLOSED | `storage/cloud_base.py` lines 259–292: `MUT_KIND_*` and `MUT_REASON_*` constants defined as module-level strings; `MUT_KINDS = frozenset({…})` seals the vocabulary; all provider adapters import these constants; `test_cloud_provider_contract.py` exercises the contract | +| T-13-11 | Credential plaintext at provider boundary | mitigate | CLOSED | Credentials decrypted inside `_resolve_and_get_adapter` only (operations.py lines 103–132); decrypted value never returned in any mutation result dict; `test_cloud_reconnect.py::test_reconnect_response_excludes_credentials` asserts no credential fields in response | +| T-13-12 | Reconciliation path — all writes via cloud_items.py | mitigate | CLOSED | Provider backends (google_drive_backend.py, onedrive_backend.py, webdav_backend.py, nextcloud_backend.py) contain zero calls to `upsert_cloud_item`, `reconcile_cloud_listing`, or `update_folder_state`; those functions are called exclusively from `api/cloud/operations.py` and `services/cloud_items.py`; grep confirms no provider-layer ORM writes | | T-13-13 | Health check exposes connection status to wrong user | mitigate | CLOSED | `POST /connections/{id}/test` asserts ownership via `resolve_owned_connection`; returns `{"state": ..., "error_code": ..., "error_message": ...}` only (no credentials); `test_cloud_reconnect.py::test_health_check_wrong_owner_403` | | T-13-14 | Reconnect stores tokens in browser/response | mitigate | CLOSED | `run_reconnect` persists refreshed credentials via `encrypt_credentials`; response is `{"status": "active"}` only; `test_cloud_reconnect.py::test_reconnect_persists_token_not_exposes_it` | | T-13-15 | Drive broader-scope consent missing from UI | mitigate | CLOSED | `SettingsCloudTab.vue` contains `data-test="gdrive-scope-notice"` with "all files in your Google Drive" copy (D-17); `SettingsCloudTab.health.test.js::test_gdrive_scope_notice` | @@ -599,3 +606,104 @@ docker compose config --quiet ### Phase 13 Unregistered Flags None. Phase 13 adds a write surface (mutations) but all mutation endpoints are owner-scoped, credential-free in responses, SSRF-protected on WebDAV/Nextcloud paths, and produce metadata-only audit events. No unregistered attack surfaces were discovered during plan execution. + +--- + +## Phase 14 — Selective Analysis and Byte Cache (2026-06-23) + +**Audit date:** 2026-06-23 +**Phase:** 14 — Selective Analysis and Byte Cache (plans 14-01 through 14-09) +**ASVS Level:** L2 +**Auditor:** gsd-security-auditor (claude-sonnet-4-6) +**Phase scope:** Cloud analysis scope estimate without byte download, idempotent Celery jobs with cooperative cancellation and bounded retry, durable byte cache with LRU eviction and per-user quota accounting, ownership-scoped cache pinning in finally block, stale guard before provider byte fetch, open/preview/download routed through the byte cache lifecycle, admin blocked from all analysis/cache routes, object_key never in API responses. + +### Phase 14 Threat Register + +| Threat ID | Threat | Disposition | Status | Evidence | +|-----------|--------|-------------|--------|----------| +| T-14-01 | IDOR — analysis job or cache entry crosses user boundary | mitigate | CLOSED | `resolve_owned_connection` in all analysis orchestration functions; `get_analysis_job`/`list_job_items` raise `AnalysisJobNotFound` for foreign job IDs; `pin_cache_entry`/`release_cache_entry` assert user_id ownership; `test_cloud_security.py` — `test_admin_cannot_estimate`, `test_foreign_user_cannot_read_job_status`, `test_cache_entries_are_owner_scoped` | +| T-14-02 | Credential/object_key disclosure in analysis or cache response | mitigate | CLOSED | `CacheStatusOut` schema excludes `object_key` at design level (no field declared); `AnalysisJobOut`/`AnalysisJobItemOut` exclude `credentials_enc`, `access_token`, `object_key`; `test_cloud_security.py::test_job_response_excludes_credentials`, `test_cache_status_excludes_object_key` | +| T-14-03 | Hidden provider file mutation during analysis | mitigate | CLOSED | `process_job_item` uses `adapter.get_object` only; `test_cloud_analysis_contract.py::test_processing_never_calls_provider_mutation_methods` — `FakeCloudAdapter.mutation_call_count == 0` after full pipeline execution | +| T-14-04 | Provider bytes downloaded during estimate | mitigate | CLOSED | `estimate_scope` reads durable `cloud_items` metadata only; no adapter call in estimate path; `test_cloud_analysis_contract.py::test_estimate_no_byte_download` — `FakeCloudAdapter.byte_call_count == 0` after estimate | +| T-14-05 | Cache quota corruption | mitigate | CLOSED | `increment_quota_for_cache` and `decrement_quota_for_cache` use atomic `UPDATE quotas SET used_bytes = used_bytes + :delta WHERE …` pattern; `test_cloud_cache.py` — increment/decrement seam tests; quota failure is non-fatal (processing proceeds without cache entry) | +| T-14-06 | Cross-user cache entry access | mitigate | CLOSED | `retain_or_reuse_cache_entry` enforces `user_id` equality before any pin/read operation; `test_cloud_cache.py::test_retain_or_reuse_isolates_foreign_users`, `test_pin_raises_for_foreign_entry` | +| T-14-07 | Analysis job remains visible to admin | mitigate | CLOSED | All analysis routes use `get_regular_user` dependency — admin tokens receive 403; `test_cloud_security.py::test_admin_cannot_estimate`, `test_admin_cannot_enqueue` | +| T-14-08 | Cache status endpoint exposes object_key or credentials_enc | mitigate | CLOSED | `CacheStatusOut` schema lists only 6 aggregate fields; `GET /analysis/cache` returns `CacheStatusOut` only; `test_cloud_cache.py::test_cache_status_excludes_object_key`, `test_cloud_security.py::test_cache_response_excludes_object_key` | +| T-14-09 | Eviction deletes bytes for pinned active job | mitigate | CLOSED | `evict_lru_entries` filters `pin_count == 0`; pinned entries are never candidates for eviction; `test_cloud_cache.py::test_eviction_skips_pinned_entries` | +| T-14-10 | Provider credentials in Celery broker payload | mitigate | CLOSED | `process_cloud_analysis_item` task parameters are `job_id, item_id, user_id, connection_id, cloud_item_id` (UUIDs only); credentials decrypted inside the worker after ownership revalidation via `decrypt_credentials`; `test_cloud_analysis_contract.py::test_celery_broker_payload_contains_ids_only` | +| T-14-11 | Provider mutation during Celery processing | mitigate | CLOSED | Worker uses `adapter.get_object` only; `test_cloud_analysis_contract.py::test_processing_never_calls_provider_mutation_methods` | +| T-14-12 | Cache pin leak on processing failure/cancellation | mitigate | CLOSED | `release_cache_entry` called in `finally` block after `pin_cache_entry`; applies on success, exception, and cooperative cancellation; `test_cloud_analysis_contract.py::test_processing_cache_pin_released_on_cancellation` | +| T-14-13 | Stale analysis result presented when provider metadata changed | mitigate | CLOSED | Version key re-computed from current `CloudItem` metadata before byte fetch; if different, item marked `"stale"` and processing stops without get_object; final stale check after classification for late concurrent changes; `test_cloud_analysis_contract.py::test_processing_unchanged_version_skips_before_provider_byte_fetch` | +| T-14-14 | Byte cache used for analysis delivers stale/wrong bytes | mitigate | CLOSED | Cache key is `(user_id, connection_id, cloud_item_id, version_key)` — version key includes etag/version_id/fingerprint; wrong-version cache entries cannot match the current version_key | +| T-14-15 | Cache status endpoint exposes per-entry metadata | mitigate | CLOSED | `CacheStatusOut` is aggregate-only (entry_count, total_bytes, cache_limit_bytes, tier_cap_bytes, analysis_progress_detail, analysis_failure_behavior) — no per-entry rows, file names, or provider IDs in the response | + +### Phase 14 Security Gate Evidence + +**Gate 1 — Full backend test suite:** +``` +cd backend && python3 -m pytest -q --tb=no +856 passed, 18 skipped, 4 deselected, 7 xfailed, 73 warnings in 109.48s +(1 pre-existing failure: test_extract_docx — ModuleNotFoundError: libmagic/python-docx not installed locally; unrelated to Phase 14) +``` + +**Gate 2 — Frontend test suite:** +``` +cd frontend && npm run test +471 passed (50 test files) — 0 failures +``` + +**Gate 3 — Bandit (backend static analysis):** +``` +python3 -m bandit -r backend/ --exclude backend/tests --severity-level high +Total issues (by severity): High: 0, Medium: 0, Low: 23 +(All 23 Low-severity findings are pre-existing; 0 HIGH or MEDIUM findings; 0 #nosec suppressions) +``` + +**Gate 4 — npm audit (frontend dependency scan):** +``` +cd frontend && npm audit --audit-level=high +found 0 vulnerabilities +``` + +**Gate 5 — pip-audit (backend dependency scan):** +pip-audit not installable in local Python 3.9 environment (requirements.txt targets Python 3.12). No new Python packages added in Phase 14 plans 01 through 09. Existing packages audited in Phase 8 with 0 critical/high CVEs. Security-critical packages remain pinned: PyJWT 2.12+, pwdlib[argon2], cryptography, pyotp, slowapi. Same tooling gap accepted in Phase 12 and Phase 13. + +**Gate 6 — Hardcoded-secret scan:** +``` +gitleaks detect --redact +3 findings — all pre-existing in commits from before Phase 14: + - auth.test.js:92 (2026-05-31) — test fixture access_token string + - test_cloud_utils.py:116 (2026-05-28) — test fixture credentials dict + - test_auth_deps.py:94 (2026-05-22) — tampered JWT test token +No findings in any Phase 14 file. +git ls-files '.env' '.env.*': only .env.example tracked +``` + +**Gate 7 — Owner/admin/credential-secrecy invariants:** +- `test_cloud_security.py::test_foreign_user_cannot_browse_cloud_item` — PASS (IDOR block) +- `test_cloud_security.py::test_admin_cannot_browse_cloud_connection` — PASS (admin block) +- `test_cloud_security.py::test_admin_cannot_estimate` — PASS (analysis admin block) +- `test_cloud_security.py::test_admin_cannot_enqueue` — PASS (enqueue admin block) +- `test_cloud_security.py::test_job_response_excludes_credentials` — PASS (no credentials in job response) +- `test_cloud_security.py::test_cache_response_excludes_object_key` — PASS (no object_key in cache status) +- `test_cloud_analysis_contract.py::test_estimate_no_byte_download` — PASS (zero byte calls) +- `test_cloud_analysis_contract.py::test_processing_never_calls_provider_mutation_methods` — PASS +- `test_cloud_cache.py::test_eviction_skips_pinned_entries` — PASS + +**Gate 8 — docker compose config:** +``` +docker compose config --quiet +(no output — all required environment variables resolve without error) +``` + +### Phase 14 Accepted Risks + +| Risk ID | Component | Accepted Risk | Rationale | +|---------|-----------|---------------|-----------| +| T-14-SC-pip | pip-audit local tooling | pip-audit not runnable against Python 3.12 requirements in Python 3.9 local env | No new Python packages added in Phase 14; existing packages audited in Phase 8; same accepted gap as Phases 12 and 13 | +| T-14-docx | test_extract_docx failure | ModuleNotFoundError: python-docx/libmagic not in local test environment | Pre-existing; unrelated to Phase 14; all Phase 14 test coverage passes | +| T-14-quota-nonfatal | Quota increment failure is non-fatal for analysis | When quota increment fails (limit exceeded), processing continues and cache entry is skipped | Quota refusal does not block text extraction or classification; only the MinIO cache storage step is omitted; extracted_text and topics are still durably persisted | + +### Phase 14 Unregistered Flags + +None. Phase 14 adds read-only analysis (no provider mutation, no provider URL exposure) and a byte cache keyed by version/etag (no cross-user data path). All new routes are owner-scoped, admin-blocked, and credential-free by schema design. No unregistered attack surfaces were discovered during plan execution. diff --git a/backend/main.py b/backend/main.py index d405bc4..97f50f0 100644 --- a/backend/main.py +++ b/backend/main.py @@ -244,7 +244,7 @@ async def lifespan(app: FastAPI): # ── Application factory ─────────────────────────────────────────────────────── -app = FastAPI(title="Document Scanner API", version="0.3.0", lifespan=lifespan) +app = FastAPI(title="Document Scanner API", version="0.4.0", lifespan=lifespan) # Rate limiter state (slowapi) app.state.limiter = auth_limiter diff --git a/frontend/package.json b/frontend/package.json index ce01a6e..d365632 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "document-scanner-frontend", - "version": "0.3.0", + "version": "0.4.0", "type": "module", "scripts": { "dev": "vite",