test(12.1): verify live cloud browse integration

Phase 12.1 close — docs, version, and security evidence: - Bump version 0.2.5 → 0.2.6 in backend/main.py, frontend/package.json - CLAUDE.md: update current state to Phase 12.1 complete v0.2.6 - README.md: add Phase 12.1 corrections, live smoke variable names, Phase 13/14 exclusions - RUNBOOK.md: add live smoke invocation, warning semantics table, fixture reconciliation procedure - SECURITY.md: add Phase 12.1 Plan 04 threat table (T-12.1-16..20, T-12.1-SC) and gate evidence - 12.1-VALIDATION.md: mark Plan 04 complete with final gate evidence
2026-06-22 09:46:25 +02:00
parent 6d294ea7e6
commit 46b7ea6c12
7 changed files with 105 additions and 5 deletions
@@ -1,6 +1,6 @@
 # Phase 12.1: Fix Nextcloud Root Listing and Sync Visibility — Validation Matrix

-**Status:** Plan 04 Task 3 complete. All regression, security, and rendered-flow gates pass.
+**Status:** Plan 04 complete. Phase 12.1 closed. All gates pass, docs updated, v0.2.6 committed.
 **Updated:** 2026-06-22

 ---
@@ -4,7 +4,7 @@

 DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV).

-**Current state:** v0.2.5 — Phase 12.1 Plan 03 complete 2026-06-22. Cloud frontend aligned with normalized API: `kind` field used for item classification (not `is_dir`); folder navigation uses `provider_item_id` as the opaque provider reference (DocuVault `id` retained for row identity/Vue keys only); server freshness (`refresh_state`, `last_refreshed_at`) mapped verbatim — no `new Date()` evidence; breadcrumb lineage built explicitly from visited nodes (never by splitting `provider_item_id`). Phase 12.1 in progress. Not cleared for public internet deployment.
+**Current state:** v0.2.6 — Phase 12.1 complete 2026-06-22. Nextcloud root listing functional: legacy `list_folder` signature override removed; incomplete listings blocked from `fresh`; frontend uses `kind` and `provider_item_id`; server freshness mapped verbatim. Live smoke test suite (opt-in, read-only) with owner-confirmed exact-name fixture. Full validation, security, documentation, versioning, and push complete. Not cleared for public internet deployment.

 ## Stack

@@ -1,6 +1,6 @@
 # DocuVault

-**Version 0.2.5 — Alpha**
+**Version 0.2.6 — Alpha**

 > **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared.

@@ -323,6 +323,24 @@ Each connected account is independently addressable by its connection UUID:

 Browsing returns durable cached rows immediately and schedules a background `refresh_cloud_folder` Celery task to reconcile provider changes. First-visit fetches are synchronous and bounded. All browse responses are credential-free — `credentials_enc` is never serialized in the response.

+**Freshness states:** Browse responses include `freshness.refresh_state` (`fresh` | `warning` | `refreshing` | `stale`). `warning` means the last reconciliation was incomplete — cached rows remain visible. The frontend maps server freshness verbatim; it never infers `fresh` from an HTTP 200 alone.
+
+**Phase 12.1 corrections (v0.2.6):**
+- Nextcloud root listing is now visible — the legacy signature override that broke the canonical four-argument `list_folder` contract has been removed
+- Incomplete provider listings (`complete=False`) no longer advance `last_refreshed_at` or set `refresh_state=fresh`
+- Frontend uses `item.kind` (`folder`|`file`) and `item.provider_item_id` for navigation — `is_dir` and DocuVault `id` are no longer used for routing
+
+**Opt-in live smoke tests (read-only):**
+
+```bash
+# Requires NEXTCLOUD_URL, NEXTCLOUD_USER, and NEXTCLOUD_APP_PASSWORD in .env (never committed)
+cd backend && pytest -m live_nextcloud tests/test_nextcloud_live.py
+```
+
+The live suite is excluded from ordinary CI runs. It is read-only — PROPFIND metadata requests only. No bytes are downloaded and no provider mutations are made. Missing variables produce a safe skip.
+
+**Phase 13/14 not yet implemented:** Cloud file upload, rename, move, delete, and create-folder (Phase 13) and byte download/cache (Phase 14) are future work. Current browse is read-only.
+
 ---

 ## Security Highlights
@@ -734,6 +734,54 @@ WHERE cc.id IS NULL;
 - **Credential fields:** `credentials_enc` is never included in browse API responses. Verify with: `grep -r "credentials_enc" backend/api/cloud/` — must only appear in connection create/update service code, never in response schemas.
 - **SSRF guard:** `storage/cloud_utils.validate_cloud_url` blocks RFC1918, link-local, file://, and cloud metadata IPs. All WebDAV/Nextcloud URLs pass through this guard before any connection is established.

+### Phase 12.1 Live Smoke Tests (Opt-In, Read-Only)
+
+The `live_nextcloud` pytest marker selects opt-in tests that use real Nextcloud credentials. These tests are **excluded from ordinary CI runs** (`addopts = -m "not live_nextcloud"` in `pytest.ini`).
+
+**Running the live smoke:**
+```bash
+# Set in .env (never committed):
+#   NEXTCLOUD_URL=<base URL of your Nextcloud instance>
+#   NEXTCLOUD_USER=<Nextcloud username>
+#   NEXTCLOUD_APP_PASSWORD=<Nextcloud app password (not account password)>
+
+cd backend
+pytest -m live_nextcloud tests/test_nextcloud_live.py
+```
+
+**What the live tests do:**
+- `test_nextcloud_adapter_read_only_root_metadata` — invokes production adapter directly via `PROPFIND`; asserts no byte/mutation method exists; verifies item ownership identity
+- `test_nextcloud_root_diagnostic` — sanitized count/kind diagnostic; prints total, expected-match count, missing expected names, unexpected count (names withheld); exits successfully even if candidate manifest has drift
+- `test_nextcloud_connection_id_browse_as_designated_user` — end-to-end browse via `GET /api/cloud/connections/{id}/items` in isolated test DB; asserts foreign-user 403/404 and admin 403/404
+- `test_nextcloud_expected_root_manifest` — exact set and kind equality against owner-confirmed fixture (requires `-k expected_root_manifest` or running all live tests)
+
+**What they do NOT do:**
+- Never download file bytes
+- Never issue PUT/POST/PATCH/DELETE/MKCOL/MOVE/COPY
+- Never print credential values, full URLs, or unexpected provider names
+- Never change MinIO objects, quota, or provider content
+- Missing credentials produce a descriptive skip, not a failure
+
+**Fixture reconciliation procedure:**
+If the expected root manifest (`backend/tests/fixtures/cloud/nextcloud_expected_root.json`) drifts:
+1. Run `pytest -m live_nextcloud -k root_diagnostic` and read the sanitized output (missing expected names only)
+2. Compare the missing names with the Nextcloud web UI (do not print the unexpected names)
+3. Update the fixture with the owner-confirmed correct names and set `owner_confirmed: true`
+4. Never record unexpected live names in any committed artifact
+
+### Phase 12 / 12.1 Warning Semantics
+
+A `warning` freshness state in the browse response means the last reconciliation was incomplete or failed. **The UI shows a warning indicator; cached rows remain visible.** This is not an error — it is honest state:
+
+| State | Meaning | UI behavior |
+|-------|---------|-------------|
+| `fresh` | Last `complete=True` listing succeeded | Green freshness indicator |
+| `refreshing` | Background refresh in progress | Spinner |
+| `warning` | Last listing was incomplete or failed | Warning banner; cached rows visible |
+| `stale` | Transport error during load | Error message; no freshness indicator |
+
+`complete=False` from any provider ALWAYS produces `warning`, never `fresh`. `last_refreshed_at` is not advanced for incomplete listings.
+
 ### Phase 12 Deferred Items

 | Item | Current state | Future path |
@@ -459,3 +459,37 @@ npm audit --audit-level=high: 0 vulnerabilities
 npm run build: built in ~500ms, 0 errors
 ```
 No bandit-equivalent issues: frontend-only plan; TypeScript/ESLint static analysis implicit via Vite build.
+
+---
+
+## Phase 12.1 Plan 04 — Live Smoke, Exact Acceptance, and Phase Close
+
+| Threat ID | Threat | Mitigation disposition | Status | Evidence |
+|-----------|--------|----------------------|--------|---------|
+| T-12.1-16 | Live credentials/full URL leak through pytest/logs | mitigate | CLOSED | `_load_live_credentials()` reads env vars in process memory only; `_assert_no_secrets_in_string` asserts captured output/logs on each test; no credential values in parametrize IDs, assertion diffs, or exception messages |
+| T-12.1-17 | Live smoke mutates/downloads user data | mitigate | CLOSED | `_NetworkMethodGuard` blocks GET/PUT/POST/PATCH/DELETE/MKCOL/MOVE/COPY before dispatch; mutation method absence assertion on adapter object; no MinIO/quota change confirmed |
+| T-12.1-18 | Live connection crosses user/admin boundary | mitigate | CLOSED | `test_nextcloud_connection_id_browse_as_designated_user` provisions isolated test DB user, foreign-user gets 403/404, admin gets 403/404 |
+| T-12.1-19 | Count-only test accepts wrong root | mitigate | CLOSED | `test_nextcloud_expected_root_manifest` asserts exact set equality AND exact kind equality; `owner_confirmed: true` guard prevents fixture activation without explicit reconciliation |
+| T-12.1-20 | Unexpected provider names become persisted sensitive metadata | mitigate | CLOSED | Diagnostic prints only `unexpected_count` (names withheld); fixture contains only confirmed names; no unexpected names in any committed artifact, log, or response |
+| T-12.1-SC | Supply-chain or committed-secret regression | mitigate | CLOSED | Bandit 0 HIGH; npm audit 0 high/critical; gitleaks 3 findings all pre-existing in commits prior to Phase 12.1 (May 2026), none from Phase 12.1 files; no `.env` in git index |
+
+### Phase 12.1 Plan 04 Security Gate Evidence
+
+```
+bandit -r backend/ -q: 0 HIGH, 0 MEDIUM (10 LOW — all pre-existing, none Phase 12.1)
+npm audit --audit-level=high: 0 vulnerabilities
+gitleaks detect --redact: 3 pre-existing findings (auth.test.js 2026-05-31,
+  test_cloud_utils.py 2026-05-28, test_auth_deps.py 2026-05-22); none in Phase 12.1 files
+git ls-files '.env' '.env.*': only .env.example tracked
+docker compose config --quiet: resolves all required variables without printing values
+Backend suite: 595 pass, 1 pre-existing failure (test_extract_docx, ModuleNotFoundError unrelated)
+Frontend suite: 376 pass (7 new rendered-flow tests)
+Frontend build: clean
+```
+
+### Phase 12.1 Exclusions
+
+Phase 12.1 scopes only read-only browse, freshness truthfulness, and cross-provider UI normalization:
+- **Phase 13 mutations excluded:** upload, rename, move, delete, create-folder are not implemented
+- **Phase 14 byte cache excluded:** no document content download, local copy, or cache layer
+- No threat for cloud mutations (T-13-xx) or byte cache (T-14-xx) is evaluated in Phase 12.1
@@ -244,7 +244,7 @@ async def lifespan(app: FastAPI):

 # ── Application factory ───────────────────────────────────────────────────────

-app = FastAPI(title="Document Scanner API", version="0.2.5", lifespan=lifespan)
+app = FastAPI(title="Document Scanner API", version="0.2.6", lifespan=lifespan)

 # Rate limiter state (slowapi)
 app.state.limiter = auth_limiter
@@ -1,6 +1,6 @@
 {
  "name": "document-scanner-frontend",
-  "version": "0.2.5",
+  "version": "0.2.6",
  "type": "module",
  "scripts": {
    "dev": "vite",