diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 6af4e23..6f8866e 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -51,13 +51,13 @@ Before any phase is marked complete: **Goal:** Restore visible, truthful, metadata-only cloud browsing by repairing Nextcloud's adapter contract, applying one completeness/freshness contract across all providers, and aligning the shared frontend browser with normalized provider identities. **Requirements:** CONN-04, CLOUD-01, CACHE-01, SYNC-01 **Depends on:** Phase 12 -**Plans:** 1/4 plans executed +**Plans:** 2/4 plans executed **Execution waves:** Wave 1: 12.1-01; Wave 2: 12.1-02 after 01; Wave 3: 12.1-03 after 02; Wave 4: 12.1-04 after 01-03. Plans: - [x] 12.1-01-PLAN.md -- [ ] 12.1-02-PLAN.md +- [x] 12.1-02-PLAN.md - [ ] 12.1-03-PLAN.md - [ ] 12.1-04-PLAN.md - [x] 12.1-01 (Wave 1) — Restore the shared four-provider adapter contract, secure Nextcloud URL/redirect handling, and metadata-only listing behavior. diff --git a/.planning/STATE.md b/.planning/STATE.md index cc8103c..9413932 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,14 +5,15 @@ milestone_name: Reimagining Cloud Storage integration current_phase: 12.1 current_phase_name: fix-nextcloud-root-listing-and-sync-visibility status: executing -last_updated: "2026-06-22T06:19:49.338Z" +stopped_at: Completed Phase 12.1 Plan 02 — truthful freshness gate +last_updated: "2026-06-22T06:37:54.133Z" last_activity: 2026-06-22 last_activity_desc: Phase 12.1 execution started progress: total_phases: 6 completed_phases: 1 total_plans: 10 - completed_plans: 7 + completed_plans: 8 percent: 17 --- @@ -25,7 +26,7 @@ progress: ## Current Position Phase: 12.1 (fix-nextcloud-root-listing-and-sync-visibility) — EXECUTING -Plan: 2 of 4 +Plan: 3 of 4 Status: Ready to execute Last activity: 2026-06-22 — Phase 12.1 execution started @@ -49,6 +50,7 @@ Last activity: 2026-06-22 — Phase 12.1 execution started | Plans complete | 6 / 10 | | Tests at milestone start | 277 | | Phase 12.1 P01 | 823 | 4 tasks | 14 files | +| Phase 12.1 P02 | 2100 | 3 tasks | 13 files | ## Accumulated Context @@ -86,11 +88,13 @@ None. ## Session Continuity +**Stopped at:** Completed Phase 12.1 Plan 02 — truthful freshness gate + _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-22T06:19:43.071Z | +| Last session | 2026-06-22T06:37:54.128Z | | Next action | Execute Phase 12.1 Plan 01 | | Pending decisions | None | | Resume file | None | @@ -99,3 +103,4 @@ _Updated at each phase transition._ - [Phase ?]: Remove NextcloudBackend.list_folder override — inherit canonical WebDAVBackend method - [Phase ?]: OneDrive nextLink restricted to graph.microsoft.com host +- [Phase ?]: apply_listing_and_finalize is the single freshness gate — callers must not set refresh_state=fresh independently after list_folder diff --git a/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-02-SUMMARY.md b/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-02-SUMMARY.md new file mode 100644 index 0000000..feae409 --- /dev/null +++ b/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-02-SUMMARY.md @@ -0,0 +1,171 @@ +--- +phase: "12.1" +plan: "02" +subsystem: cloud-storage-backends +status: complete +tags: [cloud, freshness, tdd, celery, reconciliation, security] +completed: "2026-06-22" +duration_seconds: 2100 +task_count: 3 +file_count: 13 + +requires: + - "12.1-01" +provides: + - "apply_listing_and_finalize — single shared freshness gate for API and worker" + - "ListingResult dataclass — structured result for complete vs incomplete listings" + - "Truthful refresh_state: complete=False can never produce refresh_state=fresh" + - "Preserved last_refreshed_at on incomplete/error refresh" +affects: [cloud-browse-api, cloud-tasks, cloud-items-service, security-gate] + +tech_stack: + added: [] + patterns: + - "TDD RED/GREEN — 9 failing tests written first; implementation makes them pass" + - "Single shared service gate (apply_listing_and_finalize) replaces two independent transitions" + - "ListingResult dataclass carries is_fresh, warning_code, warning_message — controlled API surface" + +key_files: + created: [] + modified: + - backend/services/cloud_items.py + - backend/api/cloud/browse.py + - backend/tasks/cloud_tasks.py + - backend/tests/test_cloud_items.py + - backend/tests/test_cloud.py + - backend/tests/test_cloud_security.py + - backend/main.py + - frontend/package.json + - CLAUDE.md + - README.md + - RUNBOOK.md + - SECURITY.md + +decisions: + - "apply_listing_and_finalize is the canonical gate — browse.py and cloud_tasks.py may not call update_folder_state('fresh') independently after list_folder" + - "ListingResult.warning_message must never contain traceback, URLs, or credential strings" + - "SQLite test timezone: strip tzinfo before comparing naive vs aware datetime (test-only issue, not production)" + +requirements: + - CLOUD-01 + - CACHE-01 + - SYNC-01 +--- + +# Phase 12.1 Plan 02: Truthful Cloud Freshness Gate — Summary + +**One-liner:** Added `apply_listing_and_finalize` as the single shared gate: `CloudListing.complete=False` now produces a controlled `warning` state instead of false `fresh`, preserving cached items and prior `last_refreshed_at`. + +## What Was Built + +### Task 1 — TDD RED: Failing tests for incomplete listing semantics + +Added 9 new failing tests across three test files before touching production code: + +**`backend/tests/test_cloud_items.py` (5 new tests):** +- `test_incomplete_listing_never_marks_folder_fresh` — asserts `apply_listing_and_finalize` (import error = RED) never sets `refresh_state=fresh` on `complete=False` +- `test_incomplete_listing_retains_cached_rows_and_last_success` — seeds prior timestamp, asserts both cached items and `last_refreshed_at` survive incomplete refresh +- `test_complete_empty_listing_is_authoritative_and_fresh` — asserts `complete=True, items=[]` soft-deletes unseen children AND marks fresh +- `test_partial_items_upsert_without_deleting_unseen_children` — asserts seen items upsert but unseen survive when `complete=False` +- `test_apply_listing_returns_warning_for_complete_false` — asserts `ListingResult(is_fresh=False, warning_code="incomplete_listing")` returned; tests warning_message for forbidden patterns (traceback, URL, credentials) + +**`backend/tests/test_cloud.py` (2 new tests):** +- `test_sync_browse_returns_warning_for_complete_false` — first-visit browse with `complete=False` adapter must not produce `refresh_state=fresh` +- `test_worker_returns_warning_for_complete_false` — Celery `_run()` must not call `update_folder_state('fresh')` when listing is incomplete + +**`backend/tests/test_cloud_security.py` (3 new tests):** +- `test_browse_complete_false_never_sets_fresh` — T-12.1-06 API-level assertion +- `test_browse_complete_false_no_raw_provider_error_in_response` — T-12.1-09 no traceback/exception/token in response body +- `test_browse_cached_items_remain_owner_scoped_on_incomplete` — T-12.1-07 IDOR holds even during incomplete refresh path + +**Result:** 6 tests immediately failed with `ImportError: cannot import name 'apply_listing_and_finalize'` (RED confirmed). + +### Task 2 — GREEN: Centralize listing application and freshness finalization + +**`backend/services/cloud_items.py`:** +- Added `ListingResult` dataclass (`is_fresh: bool`, `warning_code: Optional[str]`, `warning_message: Optional[str]`) +- Added `apply_listing_and_finalize(session, *, user_id, connection_id, parent_ref, listing) -> ListingResult`: + - Calls `reconcile_cloud_listing` (existing, correct guard: deletion only when `complete=True`) + - `listing.complete=True`: calls `update_folder_state("fresh")`, returns `ListingResult(is_fresh=True)` + - `listing.complete=False`: sets `refresh_state="warning"`, `error_code="incomplete_listing"`, controlled message; does NOT touch `last_refreshed_at`; returns `ListingResult(is_fresh=False, warning_code="incomplete_listing", ...)` + +**`backend/api/cloud/browse.py`:** +- Added `apply_listing_and_finalize` to imports +- First-visit synchronous refresh: replaced `reconcile_cloud_listing(...)` + `update_folder_state("fresh")` pair with a single `apply_listing_and_finalize(...)` call + +**`backend/tasks/cloud_tasks.py`:** +- Added `apply_listing_and_finalize` to `_run()` imports +- Replaced `reconcile_cloud_listing(...)` + `update_folder_state("fresh")` pair with `apply_listing_and_finalize(...)` +- Task return value: `{"status": "ok", ...}` when `is_fresh=True`; `{"status": "warning", "warning_code": ..., ...}` when `is_fresh=False` (no credentials, no provider item names in broker result) + +**Verification search:** +``` +rg 'refresh_state="fresh"' api/cloud/browse.py tasks/cloud_tasks.py +(no output — no unconditional fresh assignments remain) +``` + +### Task 3 — Documentation, security review, version bump + +- Version bumped `0.2.3 → 0.2.4` in `backend/main.py` and `frontend/package.json` +- `CLAUDE.md`: updated current state line; added `apply_listing_and_finalize` and `ListingResult` to module map; added "no independent fresh" rule +- `README.md`: version bump +- `RUNBOOK.md`: updated freshness state descriptions; documented `complete=True` gate requirement; `last_refreshed_at` advancement semantics +- `SECURITY.md`: added T-12.1-06 through T-12.1-10 threat closure table and security gate evidence +- Bandit over modified files: 0 HIGH, 0 MEDIUM findings +- Full backend suite: 595 passed, 1 pre-existing failure (`test_extract_docx` — missing `python-docx`, unrelated) +- Pushed to remote + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] SQLite naive vs. aware datetime comparison in test** +- **Found during:** Task 2 (test_incomplete_listing_retains_cached_rows_and_last_success) +- **Issue:** SQLite strips timezone info when storing datetime objects. The test seeded `prior_ts = datetime(2026, 6, 1, 12, 0, 0, tzinfo=timezone.utc)` and retrieved it back as naive `datetime(2026, 6, 1, 12, 0)`. Direct `==` comparison failed with `AssertionError`. +- **Fix:** Normalize `stored_ts` by attaching `tzinfo=timezone.utc` when the stored value is naive, before comparing with `prior_ts`. +- **Files modified:** `backend/tests/test_cloud_items.py` +- **Scope:** Test only — not a production defect (PostgreSQL preserves timezone; SQLite is used for unit tests only). + +## Test Evidence + +| Suite | Tests | Status | +|-------|-------|--------| +| `test_cloud_items.py` | 45 | All pass (40 pre-existing + 5 new) | +| `test_cloud.py` | 25 | All pass (23 pre-existing + 2 new) | +| `test_cloud_security.py` | 22 | All pass (19 pre-existing + 3 new) | +| Full backend suite | 595 | 1 pre-existing failure (`test_extract_docx`) | + +**RED/GREEN gate compliance:** +- RED commit `fe08afd` — 6 tests fail with ImportError before implementation +- GREEN commit `bfa4dc5` — all 12 new tests pass; existing 79 still pass + +## Security Verification + +Bandit over `services/cloud_items.py`, `api/cloud/browse.py`, `tasks/cloud_tasks.py`: 0 HIGH/MEDIUM findings. + +Threat model coverage: + +| Threat ID | Mitigation | Evidence | +|-----------|-----------|---------| +| T-12.1-06 | False fresh state on provider failure | `apply_listing_and_finalize` single gate; `test_incomplete_listing_never_marks_folder_fresh`, `test_browse_complete_false_never_sets_fresh` | +| T-12.1-07 | Incomplete listing deletes cached metadata | `reconcile_cloud_listing` still guarded by `complete=True`; `test_incomplete_listing_retains_cached_rows_and_last_success`, `test_partial_items_upsert_without_deleting_unseen_children` | +| T-12.1-08 | Error overwrites last known-good timestamp | `apply_listing_and_finalize` does not touch `last_refreshed_at` on `complete=False`; timestamp preservation test | +| T-12.1-09 | Provider errors leak secrets or URLs | Controlled `warning_message` only; `test_apply_listing_returns_warning_for_complete_false` checks forbidden patterns; `test_browse_complete_false_no_raw_provider_error_in_response` | +| T-12.1-10 | Refresh changes bytes/quota | No byte IO in `apply_listing_and_finalize`; existing no-byte/no-quota assertions still pass | + +## Known Stubs + +None — all warning messages, codes, and state transitions are fully wired. + +## Self-Check: PASSED + +| Check | Result | +|-------|--------| +| `backend/services/cloud_items.py` (apply_listing_and_finalize) | FOUND | +| `backend/api/cloud/browse.py` (apply_listing_and_finalize import) | FOUND | +| `backend/tasks/cloud_tasks.py` (apply_listing_and_finalize import) | FOUND | +| Commit fe08afd (RED tests) | FOUND | +| Commit bfa4dc5 (GREEN implementation) | FOUND | +| Commit 70a543a (docs + version) | FOUND | +| No unconditional refresh_state="fresh" in browse.py or cloud_tasks.py | CONFIRMED | +| Full test suite: 595 passed | CONFIRMED |