diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 60a4feb..aacd421 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -10,18 +10,18 @@ - [ ] **CONN-01**: User can connect, reconnect, test, and disconnect each supported cloud provider. - [ ] **CONN-02**: User can see connection health and actionable errors for expired, revoked, or invalid credentials. - [ ] **CONN-03**: Reconnecting or refreshing credentials invalidates stale provider caches without exposing credentials. -- [ ] **CONN-04**: User interface reflects the file operations supported by each connected provider. +- [x] **CONN-04**: User interface reflects the file operations supported by each connected provider. ### Cloud File Management -- [ ] **CLOUD-01**: User can browse connected cloud files and folders through the shared `StorageBrowser`. +- [x] **CLOUD-01**: User can browse connected cloud files and folders through the shared `StorageBrowser`. - [ ] **CLOUD-02**: User can open and preview supported cloud documents through DocuVault authorization. - [ ] **CLOUD-03**: User can upload files into the currently viewed cloud folder. - [ ] **CLOUD-04**: User can create folders in connected cloud storage where the provider supports it. - [ ] **CLOUD-05**: User can rename cloud files and folders where the provider supports it. - [ ] **CLOUD-06**: User can move files and folders within the same cloud connection where the provider supports it. - [ ] **CLOUD-07**: User can delete cloud files and folders after explicit confirmation. -- [ ] **CLOUD-08**: User sees unsupported cloud actions disabled with a provider-specific explanation. +- [x] **CLOUD-08**: User sees unsupported cloud actions disabled with a provider-specific explanation. - [ ] **CLOUD-09**: Successful cloud mutations update navigation promptly and produce metadata-only audit events. ### Cloud Analysis @@ -46,15 +46,15 @@ ### Cache and Storage Minimization -- [ ] **CACHE-01**: Provider-owned file bytes remain the source of truth unless the user explicitly imports a file into local DocuVault storage. -- [ ] **CACHE-02**: DocuVault persists cloud metadata, extracted text, topics, and semantic search data independently of cached file bytes. +- [x] **CACHE-01**: Provider-owned file bytes remain the source of truth unless the user explicitly imports a file into local DocuVault storage. +- [x] **CACHE-02**: DocuVault persists cloud metadata, extracted text, topics, and semantic search data independently of cached file bytes. - [ ] **CACHE-03**: DocuVault caches cloud file bytes only when required for opening, preview, or analysis. - [ ] **CACHE-04**: Configurable cache limits and safe eviction minimize local storage use without interrupting active work. - [ ] **CACHE-05**: Cached cloud content is isolated by user and connection and is never served without an ownership check. ### Change Tracking -- [ ] **SYNC-01**: DocuVault records provider item identifiers, parent/location, version or etag, size, and modification time for indexed cloud items. +- [x] **SYNC-01**: DocuVault records provider item identifiers, parent/location, version or etag, size, and modification time for indexed cloud items. - [ ] **SYNC-02**: DocuVault marks analyzed cloud items stale when their provider version changes. - [ ] **SYNC-03**: Files deleted outside DocuVault are removed from cloud navigation and excluded from search results after refresh. - [ ] **SYNC-04**: Provider refresh jobs are idempotent and retry transient failures with bounded exponential backoff. @@ -91,15 +91,15 @@ | CONN-01 | Phase 13 | Pending | | CONN-02 | Phase 13 | Pending | | CONN-03 | Phase 13 | Pending | -| CONN-04 | Phase 12 | Pending | -| CLOUD-01 | Phase 12 | Pending | +| CONN-04 | Phase 12 | Complete | +| CLOUD-01 | Phase 12 | Complete | | CLOUD-02 | Phase 13 | Pending | | CLOUD-03 | Phase 13 | Pending | | CLOUD-04 | Phase 13 | Pending | | CLOUD-05 | Phase 13 | Pending | | CLOUD-06 | Phase 13 | Pending | | CLOUD-07 | Phase 13 | Pending | -| CLOUD-08 | Phase 12 | Pending | +| CLOUD-08 | Phase 12 | Complete | | CLOUD-09 | Phase 13 | Pending | | ANALYZE-01 | Phase 14 | Pending | | ANALYZE-02 | Phase 14 | Pending | @@ -115,12 +115,12 @@ | SEARCH-05 | Phase 15 | Pending | | SEARCH-06 | Phase 15 | Pending | | SEARCH-07 | Phase 15 | Pending | -| CACHE-01 | Phase 12 | Pending | -| CACHE-02 | Phase 12 | Pending | +| CACHE-01 | Phase 12 | Complete | +| CACHE-02 | Phase 12 | Complete | | CACHE-03 | Phase 14 | Pending | | CACHE-04 | Phase 14 | Pending | | CACHE-05 | Phase 14 | Pending | -| SYNC-01 | Phase 12 | Pending | +| SYNC-01 | Phase 12 | Complete | | SYNC-02 | Phase 16 | Pending | | SYNC-03 | Phase 16 | Pending | | SYNC-04 | Phase 16 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 4d36e79..6af4e23 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -46,6 +46,34 @@ Before any phase is marked complete: 4. Provider-owned bytes remain outside DocuVault while metadata, extracted text placeholders, topic links, and semantic-index state can be persisted independently. 5. Ownership and admin-negative tests prove cloud connection and cloud item metadata cannot cross user boundaries. +### Phase 12.1: Fix Nextcloud root listing and sync visibility (INSERTED) + +**Goal:** Restore visible, truthful, metadata-only cloud browsing by repairing Nextcloud's adapter contract, applying one completeness/freshness contract across all providers, and aligning the shared frontend browser with normalized provider identities. +**Requirements:** CONN-04, CLOUD-01, CACHE-01, SYNC-01 +**Depends on:** Phase 12 +**Plans:** 1/4 plans executed +**Execution waves:** Wave 1: 12.1-01; Wave 2: 12.1-02 after 01; Wave 3: 12.1-03 after 02; Wave 4: 12.1-04 after 01-03. + +Plans: + +- [x] 12.1-01-PLAN.md +- [ ] 12.1-02-PLAN.md +- [ ] 12.1-03-PLAN.md +- [ ] 12.1-04-PLAN.md +- [x] 12.1-01 (Wave 1) — Restore the shared four-provider adapter contract, secure Nextcloud URL/redirect handling, and metadata-only listing behavior. +- [ ] 12.1-02 (Wave 2; after 12.1-01) — Centralize complete/incomplete reconciliation so incomplete provider results cannot become fresh or delete cached metadata. +- [ ] 12.1-03 (Wave 3; after 12.1-02) — Align the shared browser and trees with `kind`, opaque `provider_item_id` routing, lineage-based breadcrumbs, and server freshness. +- [ ] 12.1-04 (Wave 4; after 12.1-01 through 12.1-03) — Run sanitized two-mode live Nextcloud acceptance, rendered/security/secret gates, and phase closeout. + +**Success Criteria:** + +1. Nextcloud and generic WebDAV expose one canonical `CloudResourceAdapter.list_folder` implementation, and all four providers pass the same runtime identity/completeness/no-byte/no-mutation contract with provider-specific pagination fixtures. +2. Every outbound Nextcloud/WebDAV target, including redirects, is rejected unless each hop is HTTPS, same-origin, and passes URL plus resolved-address SSRF validation; automatic redirects are disabled. +3. `complete=False` never marks a folder fresh, advances `last_refreshed_at`, or authorizes unseen-item deletion in synchronous or worker paths; cached rows remain usable with a controlled warning. +4. The frontend classifies by `kind`, navigates through named routes/query serialization with opaque `provider_item_id` values, builds breadcrumbs from explicit navigation lineage rather than splitting provider IDs, and renders backend freshness verbatim. +5. Live Nextcloud validation first runs a nonblocking sanitized diagnostic; exact names/kinds become a blocking gate only after the owner-confirmed manifest is stored in a tracked, non-secret fixture. Unexpected live names are never printed or persisted. +6. Automated backend/frontend/rendered-flow tests, dedicated per-plan security reviews, dependency audits, an executable secret scan, documentation/version updates, and one atomic commit/push per plan pass without exposing credentials or provider bytes. + ### Phase 13: Virtual-Local Cloud Operations **Goal:** Users can maintain healthy cloud connections and perform the main file-management workflows in connected storage with the same shared browser interactions used for local files. diff --git a/.planning/STATE.md b/.planning/STATE.md index c072414..cc8103c 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,36 +2,39 @@ gsd_state_version: 1.0 milestone: v0.3 milestone_name: Reimagining Cloud Storage integration -status: ready_to_plan -last_updated: 2026-06-19T04:04:58.418Z -last_activity: 2026-06-18 -- Phase 12 execution started +current_phase: 12.1 +current_phase_name: fix-nextcloud-root-listing-and-sync-visibility +status: executing +last_updated: "2026-06-22T06:19:49.338Z" +last_activity: 2026-06-22 +last_activity_desc: Phase 12.1 execution started progress: - total_phases: 5 + total_phases: 6 completed_phases: 1 - total_plans: 4 - completed_plans: 69 - percent: 20 -stopped_at: Phase 12 complete (4/4) — ready to discuss Phase 13 + total_plans: 10 + completed_plans: 7 + percent: 17 --- # Project State **Project:** DocuVault -**Status:** Ready to plan -**Last Updated:** 2026-06-17 +**Status:** Ready to execute +**Last Updated:** 2026-06-22 ## Current Position -Phase: 13 -Plan: Not started -Status: Executing Phase 12 -Last activity: 2026-06-19 +Phase: 12.1 (fix-nextcloud-root-listing-and-sync-visibility) — EXECUTING +Plan: 2 of 4 +Status: Ready to execute +Last activity: 2026-06-22 — Phase 12.1 execution started ## Phase Status | Phase | Requirements | Status | |-------|-------------|--------| -| 12. Cloud Resource Foundation | CONN-04, CLOUD-01, CLOUD-08, CACHE-01, CACHE-02, SYNC-01 | **Not started** | +| 12. Cloud Resource Foundation | CONN-04, CLOUD-01, CLOUD-08, CACHE-01, CACHE-02, SYNC-01 | **Complete** | +| 12.1 Fix Nextcloud Root Listing and Sync Visibility | CONN-04, CLOUD-01, CACHE-01, SYNC-01 | **Planned** | | 13. Virtual-Local Cloud Operations | CONN-01..03, CLOUD-02..07, CLOUD-09 | **Not started** | | 14. Selective Analysis and Byte Cache | ANALYZE-01..07, CACHE-03..05 | **Not started** | | 15. Unified Smart Search | SEARCH-01..07 | **Not started** | @@ -41,10 +44,11 @@ Last activity: 2026-06-19 | Metric | Value | |---|---| -| Phases complete | 0 / 5 | -| Requirements satisfied | 0 / 36 | -| Plans complete | 0 / 0 | +| Phases complete | 1 / 6 | +| Requirements satisfied | 6 / 36 | +| Plans complete | 6 / 10 | | Tests at milestone start | 277 | +| Phase 12.1 P01 | 823 | 4 tasks | 14 files | ## Accumulated Context @@ -70,6 +74,7 @@ Last activity: 2026-06-19 - v0.2 completed: UI overhaul, admin panel rearchitecture, responsive layout, codebase quality (2026-06-17) - v0.2 archived to `.planning/milestones/v0.2-ROADMAP.md` - v0.3 proposed: virtual-local cloud operations, selective cloud analysis, bounded byte caching, unified smart search, and provider change tracking +- Phase 12.1 inserted after Phase 12: Fix Nextcloud root listing and sync visibility (URGENT) ### Open Questions @@ -85,7 +90,12 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-17 — v0.3 roadmap proposed | -| Next action | Approve roadmap, then discuss or plan Phase 12 | +| Last session | 2026-06-22T06:19:43.071Z | +| Next action | Execute Phase 12.1 Plan 01 | | Pending decisions | None | | Resume file | None | + +## Decisions + +- [Phase ?]: Remove NextcloudBackend.list_folder override — inherit canonical WebDAVBackend method +- [Phase ?]: OneDrive nextLink restricted to graph.microsoft.com host diff --git a/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-01-SUMMARY.md b/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-01-SUMMARY.md new file mode 100644 index 0000000..c54693a --- /dev/null +++ b/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-01-SUMMARY.md @@ -0,0 +1,156 @@ +--- +phase: "12.1" +plan: "01" +subsystem: cloud-storage-backends +status: complete +tags: [nextcloud, webdav, google-drive, onedrive, cloud-resource-adapter, contract-tests, security] +completed: "2026-06-22" +duration_seconds: 823 +task_count: 4 +file_count: 14 + +requires: [] +provides: + - "Canonical four-provider CloudResourceAdapter contract suite" + - "Nextcloud list_folder signature defect eliminated" + - "normalize_nextcloud_url shared helper" + - "OneDrive @odata.nextLink cross-origin guard" +affects: [cloud-browse-api, cloud-backend-factory, provider-tests] + +tech_stack: + added: [] + patterns: + - "TDD RED/GREEN — contract tests written first, production defect confirmed, then implementation fixed" + - "normalize_nextcloud_url idempotent URL derivation with SSRF passthrough" + - "OneDrive cross-origin nextLink validation before following" + +key_files: + created: + - backend/tests/test_cloud_provider_contract.py + - backend/tests/fixtures/cloud/nextcloud_root.xml + - backend/tests/fixtures/cloud/webdav_root.xml + - backend/tests/fixtures/cloud/google_drive_pages.json + - backend/tests/fixtures/cloud/onedrive_pages.json + modified: + - backend/storage/nextcloud_backend.py + - backend/storage/cloud_utils.py + - backend/storage/cloud_backend_factory.py + - backend/storage/onedrive_backend.py + - backend/tests/test_cloud_backends.py + - backend/tests/test_webdav_backend.py + - backend/main.py + - frontend/package.json + - CLAUDE.md + - README.md + +decisions: + - "Remove NextcloudBackend.list_folder override so Nextcloud inherits canonical WebDAVBackend method (direct fix for P0)" + - "normalize_nextcloud_url lives in cloud_utils.py — called at factory construction, not in every request" + - "OneDrive nextLink restricted to graph.microsoft.com origin — cross-origin returns complete=False with prior items retained" + - "asyncio import restored to nextcloud_backend.py (health_check uses asyncio.to_thread)" +--- + +# Phase 12.1 Plan 01: Restore Cloud Adapter Contract — Summary + +**One-liner:** Removed incompatible `NextcloudBackend.list_folder(folder_path="")` override, added `normalize_nextcloud_url()`, cross-origin OneDrive nextLink guard, and parametrized four-provider contract suite. + +## What Was Built + +### Task 1 — TDD RED: Four-provider contract tests and fixtures + +- `backend/tests/test_cloud_provider_contract.py` — 48 parametrized tests across Nextcloud, WebDAV, Google Drive, OneDrive covering: + - Canonical signature verification (`connection_id, user_id, parent_ref=None, page_token=None`) + - Return type is always `CloudListing` (never `list[dict]`) + - Trusted caller identity (`connection_id`, `user_id`) is never overrideable by provider response + - `provider_item_id` is set; `kind` is exactly "file" or "folder" + - `parent_ref` is propagated to each resource + - Absent optional metadata normalizes to `None` + - All pages consumed before `complete=True`; page failure → `complete=False` + - Forbidden-operation spies for `get_object`, `put_object`, `delete_object`, etc. +- Synthetic credential-free fixtures: `nextcloud_root.xml`, `webdav_root.xml`, `google_drive_pages.json`, `onedrive_pages.json` +- 9 Nextcloud-specific tests confirmed RED (TypeError on canonical invocation) as expected + +### Task 2 — GREEN: Repair Nextcloud/WebDAV listing + +- Removed `async def list_folder(self, folder_path: str = "") -> list[dict]` from `NextcloudBackend` — root cause of P0 defect +- `NextcloudBackend` now inherits `WebDAVBackend.list_folder` via standard Python inheritance +- Added `normalize_nextcloud_url(base_url, username)` to `cloud_utils.py`: + - Accepts bare origin, origin with subpath, already-canonical URL + - Username percent-encoded as single path segment + - HTTPS-only; rejects userinfo/query/fragment + - Idempotent — no double-append of WebDAV suffix + - SSRF-validated via existing `validate_cloud_url()` before return +- `cloud_backend_factory.py` calls `normalize_nextcloud_url` before `NextcloudBackend` construction; `ValueError` from DNS failures is caught gracefully +- Added `TestNextcloudBackendNoListFolderOverride` and `TestNextcloudUrlNormalization` to `test_webdav_backend.py` +- Restored `asyncio` import (was inadvertently removed with the legacy `list_folder`; `health_check` still uses `asyncio.to_thread`) + +### Task 3 — Drive/OneDrive contract completion + +- Added `_GRAPH_HOST` constant to `onedrive_backend.py`; `@odata.nextLink` hostname validated against `graph.microsoft.com` before following; cross-origin → `complete=False` with prior items retained +- Added `TestOneDriveCrossOriginNextLink`: cross-origin rejection, same-origin follows, page failure retains prior items +- Added `TestGoogleDriveAuthFailureControl`: 401 → `complete=False`, page1+page2 error retains page1 items + +### Task 4 — Documentation and version bump + +- Version bumped to `0.2.3` in `backend/main.py` and `frontend/package.json` +- `CLAUDE.md` updated: current state, shared module map, Nextcloud no-override rule +- `README.md` updated: version, Nextcloud URL normalization note, OneDrive nextLink origin guard + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Missing asyncio import in NextcloudBackend after legacy method removal** +- **Found during:** Task 4 (full test suite run) +- **Issue:** Removing the legacy `list_folder` method also removed the only active use of `asyncio` in the file, but the `health_check` method still calls `asyncio.to_thread`. This caused `test_nextcloud_connect_persists` and `test_same_provider_connections_are_independent` to fail. +- **Fix:** Re-added `import asyncio` to `nextcloud_backend.py` +- **Files modified:** `backend/storage/nextcloud_backend.py` +- **Commit:** 6212873 + +**2. [Rule 2 - Missing security] OneDrive @odata.nextLink cross-origin validation** +- **Found during:** Task 3 (plan specifies "reject cross-origin/untrusted continuation URLs") +- **Issue:** `onedrive_backend.py` followed `@odata.nextLink` without verifying the target stays on `graph.microsoft.com`; an SSRF-adjacent CSRF vector +- **Fix:** Added `_GRAPH_HOST` constant and `urllib.parse` hostname check before following any pagination URL +- **Files modified:** `backend/storage/onedrive_backend.py` +- **Commit:** 805fe44 + +## Test Evidence + +| Suite | Tests | Status | +|-------|-------|--------| +| `test_cloud_provider_contract.py` | 48 | All pass | +| `test_cloud_backends.py` | 67 | All pass | +| `test_webdav_backend.py` | 29 | All pass (9 warnings for sync tests with asyncio mark — module-level `pytestmark` issue, tests behave correctly) | +| `test_cloud_security.py` | 19 | All pass | +| Full backend suite | 585 | 1 pre-existing failure (`test_extract_docx` — missing `python-docx` module, unrelated) | + +## Security Verification + +Bandit scanned `nextcloud_backend.py`, `cloud_utils.py`, `cloud_backend_factory.py`, `onedrive_backend.py` — zero HIGH/MEDIUM/LOW findings. + +Threat model coverage: + +| Threat ID | Mitigation | Evidence | +|-----------|-----------|---------| +| T-12.1-01 | Provider response fields cannot override connection_id/user_id | `TestProviderHostileIdentityRejection` in contract suite | +| T-12.1-02 | SSRF through Nextcloud URL normalization | `normalize_nextcloud_url` always calls `validate_cloud_url`; HTTPS-only; rejects userinfo | +| T-12.1-03 | Partial page authorizes deletion | `complete=False` on error; `test_provider_page_failure_is_incomplete` | +| T-12.1-04 | Browse downloads or mutates provider content | `test_provider_listing_never_downloads_or_mutates` spies on forbidden methods | +| T-12.1-05 | Secrets in fixtures or error messages | Synthetic fixtures only; controlled exceptions | + +## Known Stubs + +None — all contract tests use real provider implementations (not mocks of the contract itself). + +## Self-Check: PASSED + +| Check | Result | +|-------|--------| +| `backend/tests/test_cloud_provider_contract.py` | FOUND | +| `backend/tests/fixtures/cloud/nextcloud_root.xml` | FOUND | +| `backend/storage/nextcloud_backend.py` | FOUND | +| `backend/storage/cloud_utils.py` | FOUND | +| Commit eb68fac (RED tests) | FOUND | +| Commit 2b46f74 (GREEN fix) | FOUND | +| Commit 805fe44 (Drive/OneDrive) | FOUND | +| Commit 6212873 (docs, asyncio fix) | FOUND |