fix(12.1): make cloud freshness authoritative — plan 02 documentation and version

- Version bump 0.2.3 → 0.2.4 (user-visible sync status corrected)
- CLAUDE.md: update current state; add apply_listing_and_finalize to module map; add no-independent-fresh rule
- README.md: bump version
- RUNBOOK.md: update freshness state docs; explain complete=True gate and last_refreshed_at behavior
- SECURITY.md: add T-12.1-06..10 threat closure evidence for Plan 02
This commit is contained in:
curo1305
2026-06-22 08:34:51 +02:00
parent bfa4dc502a
commit 70a543a8c1
6 changed files with 28 additions and 6 deletions
+3 -2
View File
@@ -4,7 +4,7 @@
DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV). DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV).
**Current state:** v0.2.3 — Phase 12.1 Plan 01 complete 2026-06-22. Cloud provider adapter contract repaired: Nextcloud now inherits canonical `WebDAVBackend.list_folder(connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing`; legacy `list_folder(folder_path="") -> list[dict]` override removed. `normalize_nextcloud_url()` added to `cloud_utils.py`. OneDrive `@odata.nextLink` cross-origin guard added. Four-provider parametrized contract suite in `test_cloud_provider_contract.py`. Phase 12.1 in progress. Not cleared for public internet deployment. **Current state:** v0.2.4 — Phase 12.1 Plan 02 complete 2026-06-22. Truthful freshness gate added: `apply_listing_and_finalize()` in `cloud_items.py` is the single shared gate used by both `browse.py` and `cloud_tasks.py`. Incomplete listings (`CloudListing.complete=False`) no longer produce `refresh_state="fresh"` — they retain cached items, preserve prior `last_refreshed_at`, and set a controlled `incomplete_listing` warning. `ListingResult` dataclass carries `is_fresh`, `warning_code`, and `warning_message`. Phase 12.1 in progress. Not cleared for public internet deployment.
## Stack ## Stack
@@ -42,7 +42,7 @@ Before adding a helper, check if it belongs in an existing shared module:
| `backend/services/auth.py` | `validate_password_strength(password)` — raises `ValueError`; routers catch and re-raise as `HTTPException` | | `backend/services/auth.py` | `validate_password_strength(password)` — raises `ValueError`; routers catch and re-raise as `HTTPException` |
| `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability` — Phase 12 browse contract; all 4 providers implement this | | `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability` — Phase 12 browse contract; all 4 providers implement this |
| `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` — Phase 12.1 adds `normalize_nextcloud_url` | | `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` — Phase 12.1 adds `normalize_nextcloud_url` |
| `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state` — owner-scoped cloud metadata service | | `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state`, `apply_listing_and_finalize`, `ListingResult` — owner-scoped cloud metadata service; `apply_listing_and_finalize` is the single shared freshness gate |
| `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest` — credential-free cloud response schemas | | `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest` — credential-free cloud response schemas |
**Rules:** **Rules:**
@@ -52,6 +52,7 @@ Before adding a helper, check if it belongs in an existing shared module:
- No AI provider may define its own `_strip_code_fences` or `_parse_*`. Import from `ai.utils`. - No AI provider may define its own `_strip_code_fences` or `_parse_*`. Import from `ai.utils`.
- No API file may define `_validate_password_strength`. Import from `services.auth`. - No API file may define `_validate_password_strength`. Import from `services.auth`.
- Service layer raises `ValueError` (or domain exceptions), never `HTTPException`. Only the router layer raises `HTTPException`. - Service layer raises `ValueError` (or domain exceptions), never `HTTPException`. Only the router layer raises `HTTPException`.
- No caller of `adapter.list_folder` may independently call `update_folder_state(refresh_state="fresh")`. Callers must use `apply_listing_and_finalize` from `services.cloud_items`.
### Frontend: shared module map ### Frontend: shared module map
+1 -1
View File
@@ -1,6 +1,6 @@
# DocuVault # DocuVault
**Version 0.2.3 — Alpha** **Version 0.2.4 — Alpha**
> **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared. > **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared.
+2 -1
View File
@@ -682,7 +682,8 @@ UPDATE cloud_connections SET status = 'ACTIVE' WHERE id = '<conn-uuid>';
### Cloud browse background refresh ### Cloud browse background refresh
The browse endpoint (`GET /api/cloud/connections/{id}/items`) serves cached metadata and schedules a background refresh when stale: The browse endpoint (`GET /api/cloud/connections/{id}/items`) serves cached metadata and schedules a background refresh when stale:
- **Freshness states:** `fresh` (just refreshed), `refreshing` (refresh in progress), `stale` (last refresh > threshold), `warning` (refresh failed) - **Freshness states:** `fresh` (complete authoritative listing), `refreshing` (refresh in progress), `warning` (incomplete listing or provider error)
- **`fresh` requires `complete=True`** — a `CloudListing(complete=False)` result from any provider always produces `warning` state (`incomplete_listing` error code), never `fresh`. `last_refreshed_at` is only advanced on successful `complete=True` listings.
- **Cached data is always served** — the UI shows a warning banner for stale/error states without blocking navigation - **Cached data is always served** — the UI shows a warning banner for stale/error states without blocking navigation
- **No bytes are downloaded** during browse; `size_bytes` values come from provider-reported metadata only - **No bytes are downloaded** during browse; `size_bytes` values come from provider-reported metadata only
+20
View File
@@ -417,3 +417,23 @@ Integration tests activate with `INTEGRATION=1` against a disposable PostgreSQL
**Gate 3 — No new Python or frontend packages introduced.** **Gate 3 — No new Python or frontend packages introduced.**
**Gate 4 — No hardcoded credentials in Compose or test files.** **Gate 4 — No hardcoded credentials in Compose or test files.**
---
## Phase 12.1 Plan 02 — Truthful Freshness Gate
| Threat ID | Threat | Mitigation disposition | Status | Evidence |
|-----------|--------|----------------------|--------|---------|
| T-12.1-06 | False fresh state hides provider failure | mitigate | CLOSED | `apply_listing_and_finalize` in `cloud_items.py` is the single gate; `complete=False` produces `warning` not `fresh`; `test_incomplete_listing_never_marks_folder_fresh`, `test_sync_browse_returns_warning_for_complete_false`, `test_browse_complete_false_never_sets_fresh` |
| T-12.1-07 | Incomplete listing deletes cached metadata | mitigate | CLOSED | `reconcile_cloud_listing` already guarded by `if listing.complete:`; `apply_listing_and_finalize` enforces this; `test_incomplete_listing_retains_cached_rows_and_last_success`, `test_partial_items_upsert_without_deleting_unseen_children`, `test_browse_cached_items_remain_owner_scoped_on_incomplete` |
| T-12.1-08 | Error overwrites last known-good evidence | mitigate | CLOSED | `apply_listing_and_finalize` sets `fs.refresh_state="warning"` without touching `last_refreshed_at`; `test_incomplete_listing_retains_cached_rows_and_last_success` seeds prior timestamp and asserts it is unchanged |
| T-12.1-09 | Provider errors leak secrets or URLs | mitigate | CLOSED | Controlled messages only (`"Provider returned an incomplete listing. Showing cached results."`); `ListingResult.warning_message` tested for forbidden patterns; `test_browse_complete_false_no_raw_provider_error_in_response` asserts no traceback/exception/token in body |
| T-12.1-10 | Refresh changes bytes/quota | mitigate | CLOSED | Existing no-byte/no-MinIO/no-quota assertions still pass; `apply_listing_and_finalize` only calls `reconcile_cloud_listing` and `update_folder_state` — no byte IO |
### Phase 12.1 Plan 02 Security Gate Evidence
```
pytest -q backend/tests/test_cloud_items.py backend/tests/test_cloud.py backend/tests/test_cloud_security.py
89 passed
```
Bandit over modified files (`services/cloud_items.py`, `api/cloud/browse.py`, `tasks/cloud_tasks.py`): 0 HIGH, 0 MEDIUM findings.
+1 -1
View File
@@ -244,7 +244,7 @@ async def lifespan(app: FastAPI):
# ── Application factory ─────────────────────────────────────────────────────── # ── Application factory ───────────────────────────────────────────────────────
app = FastAPI(title="Document Scanner API", version="0.2.3", lifespan=lifespan) app = FastAPI(title="Document Scanner API", version="0.2.4", lifespan=lifespan)
# Rate limiter state (slowapi) # Rate limiter state (slowapi)
app.state.limiter = auth_limiter app.state.limiter = auth_limiter
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "document-scanner-frontend", "name": "document-scanner-frontend",
"version": "0.2.3", "version": "0.2.4",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "vite", "dev": "vite",