From 52b110acef249466b8acd55125f56fd11ed605e7 Mon Sep 17 00:00:00 2001 From: curo1305 Date: Thu, 18 Jun 2026 22:13:23 +0200 Subject: [PATCH] docs(phase-12): add research and validation strategy --- .../12-RESEARCH.md | 237 ++++++++++++++++++ .../12-VALIDATION.md | 82 ++++++ 2 files changed, 319 insertions(+) create mode 100644 .planning/phases/12-cloud-resource-foundation/12-RESEARCH.md create mode 100644 .planning/phases/12-cloud-resource-foundation/12-VALIDATION.md diff --git a/.planning/phases/12-cloud-resource-foundation/12-RESEARCH.md b/.planning/phases/12-cloud-resource-foundation/12-RESEARCH.md new file mode 100644 index 0000000..e458803 --- /dev/null +++ b/.planning/phases/12-cloud-resource-foundation/12-RESEARCH.md @@ -0,0 +1,237 @@ +# Phase 12: Cloud Resource Foundation - Research + +**Researched:** 2026-06-18 +**Status:** Complete + +## Research Question + +What must be understood to plan a provider-neutral cloud capability contract, durable owner-scoped cloud item index, and capability-aware shared browser without copying provider-owned files into DocuVault storage? + +## Executive Summary + +Phase 12 should introduce a cloud-resource layer beside the existing object-storage `StorageBackend`, not expand that byte-oriented contract until it becomes a file manager. The durable identity is `CloudConnection.id`; provider type is only an adapter selector. This is required for multiple accounts from one provider and means current provider-keyed routes, cache keys, and `_get_active_connection()` queries must change. + +Browsing should use stale-while-revalidate semantics backed by PostgreSQL: return normalized metadata immediately, refresh from the provider, then upsert owner/connection-scoped rows. Provider-owned bytes remain absent. Capability responses need both operation support and a reason state so `StorageBrowser.vue` can distinguish unsupported actions from temporarily blocked actions without provider-specific branches. + +Two cross-phase constraints must be made explicit. First, Google currently requests the `drive.file` scope, which is limited to files created by or opened with the app and cannot satisfy whole-existing-drive discovery; Phase 13 reconnect/credential work must request and communicate appropriate broader access. Second, retained byte-cache quota accounting belongs to Phase 14, but Phase 12 schemas should not conflate metadata size with stored byte usage. + +## Current-State Findings + +### Connection Identity Is Provider-Keyed + +- `CloudConnection` already has a UUID primary key and `display_name`, but `_upsert_cloud_connection()` and `_get_active_connection()` select by `(user_id, provider)`. +- Browse routes are `/api/cloud/folders/{provider}/{folder_id}` and frontend routes are `/cloud/:provider/:folderId`; both make a second account of the same provider ambiguous. +- `cloud_cache.py` keys listings as `{user_id}:{provider}:{folder_id}`, so same-provider accounts would collide. +- `CloudStorageView.vue` correctly iterates connection IDs, but navigates with `conn.provider` and discards the connection identity. + +**Planning consequence:** migrate browse identity to connection UUID throughout API, route, cache, store, and view boundaries. Do not expose encrypted credentials or infer ownership from a client-supplied provider. + +### Existing Storage Contract Is Byte-Oriented + +`StorageBackend` defines put/get/delete, presigned URLs, health, and stat. It does not model folders, normalized resource metadata, rename/move, provider capabilities, or change cursors. Nextcloud adds `list_folder()` outside the abstract contract, while Google and OneDrive listing functions live directly in `api/cloud.py`. + +**Planning consequence:** add a separate cloud resource adapter contract. Keep `StorageBackend` for DocuVault-owned/local object storage and existing document byte operations. A cloud adapter can later support: + +- `list_folder(parent_ref)` returning normalized resources and a listing result +- `capabilities()` for connection-level support +- item-level capability overrides from provider metadata +- future `open`, `preview`, `upload`, `create_folder`, `rename`, `move`, `delete`, and change-cursor operations + +Phase 12 implements browse and capability reporting; Phase 13 implements mutations. + +### Listings Are Too Sparse and Ephemeral + +Current Google/OneDrive/WebDAV listing dictionaries expose only `id`, `name`, `is_dir`, and `size`. The in-process TTL cache lasts 60 seconds and disappears on restart or another application instance. There is no persisted parent, MIME type, modified time, version/etag, last-seen time, or analysis/index placeholder. + +**Planning consequence:** a `cloud_items` table should persist normalized metadata with a uniqueness boundary at `(connection_id, provider_item_id)` and explicit `user_id` for defense-in-depth ownership queries. Recommended fields: + +- UUID `id`, UUID `user_id`, UUID `connection_id` +- provider-native `provider_item_id`, optional canonical path snapshot +- optional `parent_item_id` plus provider parent reference/path +- `name`, `item_type` (`file`/`folder`), optional `content_type`, `size_bytes` +- optional `etag`, `version`, provider `modified_at` +- `last_seen_at`, optional `deleted_at`, timestamps +- analysis/index placeholders that do not imply byte ownership (`analysis_status`, optional extracted-text/index relation or future-compatible status fields) + +Use foreign keys with cascade from user/connection, indexes for owner+connection+parent browsing, and a unique connection+provider-item constraint. Never make path the stable primary identity: rename and move change paths, while provider IDs are generally more stable. + +### Capability Data Has Two Levels + +Provider APIs expose capability information differently: + +- Google Drive `files.capabilities` includes item-level booleans such as `canRename`, `canTrash`, and `canMoveItemWithinDrive`. +- Microsoft Graph `driveItem` supplies stable IDs, parent references, eTags/cTags, facets, and permissions; some effective capability decisions require granted scopes and item context rather than one universal provider matrix. +- WebDAV capability discovery can use standards-visible methods/properties and server responses, but arbitrary servers vary. Safe discovery must use `OPTIONS`, `PROPFIND`, configured permission information, and observed non-mutating responses, never test mutations. + +Model each action as structured data rather than booleans alone: + +- state: `supported`, `unsupported`, or `temporarily_unavailable` +- reason code: stable machine identifier such as `provider_unsupported`, `insufficient_scope`, `read_only`, `reauth_required`, `offline`, `item_restricted` +- message: short user-facing explanation +- optional remedy code for frontend action routing in later phases + +Connection-level defaults should be merged with item-level provider capabilities. Keep the action vocabulary fixed across providers: browse, open, preview, upload, create-folder, rename, move, delete, and change-tracking. + +### Google OAuth Scope Is a Functional Blocker for the Milestone Vision + +The current OAuth flow requests `https://www.googleapis.com/auth/drive.file`. Google documents this as access to files the app created or that users explicitly opened with the app. It is insufficient for discovering and analyzing all documents already organized in a user’s Drive. + +**Planning consequence:** Phase 12 should expose an `insufficient_scope` capability/health reason when credentials cannot browse the intended corpus. Phase 13 owns reconnect and credential lifecycle, including a deliberate broader-scope consent flow. Do not silently widen permissions in Phase 12. + +### Metadata and Byte Quota Must Stay Separate + +The context locks these semantics: + +- provider bytes are authoritative; +- metadata/search/index records do not consume document-storage quota; +- actual file bytes retained in DocuVault storage consume quota while present; +- transient streaming that is immediately discarded does not consume storage quota. + +**Planning consequence:** Phase 12 must not create MinIO objects or increment quota during browse. A future byte-cache table/service should reference cloud items and account for actual retained bytes atomically in Phase 14. `cloud_items.size_bytes` is provider metadata, not DocuVault quota usage. + +## Recommended Architecture + +### Backend Modules + +Keep routers thin and split the growing cloud monolith along current project conventions: + +- `backend/storage/cloud_base.py`: normalized enums/dataclasses or Pydantic-neutral domain types and the abstract cloud resource adapter. +- `backend/storage/cloud_backend_factory.py`: return resource-capable provider adapters by connection/provider. +- Provider adapters: normalize metadata and capabilities close to provider-specific SDK/API responses. +- `backend/services/cloud_items.py`: owner-scoped upsert/list/reconciliation service; raises domain errors, never `HTTPException`. +- `backend/api/cloud/connections.py` and `backend/api/cloud/browse.py` or an equivalent sub-router split using the project’s no-prefix sub-router rule. +- `backend/api/schemas.py` or `backend/api/cloud/schemas.py`: explicit whitelisted response models; never include `credentials_enc`. + +Avoid a generic repository abstraction unless it removes repeated owner-scoped queries. The important abstraction is the provider resource contract and normalized response. + +### Browse/Reconciliation Flow + +1. Authenticate regular user and resolve `connection_id` with `(connection.id, connection.user_id)`. +2. Read durable child metadata for `(user_id, connection_id, parent_ref)`. +3. Return cached rows promptly with freshness fields and normalized capabilities. +4. Refresh provider metadata in a bounded request/background path. +5. Upsert observed resources by `(connection_id, provider_item_id)`; update parent/name/type/size/mtime/etag/version and `last_seen_at`. +6. Reconcile disappeared children conservatively. Phase 12 may mark listing membership stale/deleted only after a successful complete listing; never delete rows after a failed or partial provider response. +7. Preserve stable DocuVault cloud item UUIDs so the frontend can reconcile silently without losing selection/scroll. + +The API should distinguish `fresh`, `refreshing`, and `warning` folder states and include `last_refreshed_at`/safe error metadata. Exact asynchronous transport (request-triggered task, Celery refresh, or immediate refresh after cached response) is planner discretion, but it must work across multiple backend instances and not rely solely on process memory. + +### Frontend Contract + +`StorageBrowser.vue` stays the only grid. Prefer data-driven props: + +- normalized rows with stable DocuVault IDs plus provider IDs hidden from presentation +- connection root/breadcrumb data +- action capability map with state/message/remedy metadata +- folder freshness state and last-successful timestamp +- byte-availability state (`cloud_only` now; cache states become active in Phase 14) + +Emit generic actions. `CloudStorageView` and `CloudFolderView` remain thin route/store data providers. Replace `mode === 'local'` action hiding with capability rendering; local mode can supply a local capability set so one rendering path serves both sources. + +Disabled native HTML buttons do not receive click events, so the touch requirement needs an accessible wrapper/trigger or `aria-disabled` control that blocks execution while still accepting focus/tap for explanation. The plan must test keyboard focus and touch/click explanation behavior. + +## Security Threat Model Inputs + +| Threat | Phase 12 mitigation to plan | +|--------|-----------------------------| +| IDOR across cloud connections | Resolve every connection by both UUID and current user; return 404 for foreign IDs | +| IDOR across cloud items | Query item by user and connection; never trust provider ID alone | +| Credential disclosure | Explicit response schemas; negative tests for `credentials_enc`, access tokens, refresh tokens, passwords | +| SSRF regression | Preserve `validate_cloud_url()` before all WebDAV/Nextcloud outbound requests and redirects | +| Cache/data collision | Include user and connection UUID in every durable/in-memory key | +| Destructive capability probing | Only provider declarations and non-mutating calls; tests assert no mutation methods invoked | +| Partial-list data loss | Reconcile removals only after a successful complete listing/page traversal | +| Admin content access | Continue `get_regular_user`; admin-negative tests for browse and metadata endpoints | +| Quota inflation/escape | Browse writes metadata only; assert quota and MinIO remain unchanged | +| Stored XSS through filenames/messages | Vue text rendering only; no `v-html`; provider errors mapped to controlled messages | + +## Provider-Specific Planning Notes + +### Google Drive + +- Request fields explicitly: IDs, parents, name, mime type, size, modified time, version/md5 where applicable, capabilities, and pagination token. +- Handle all pages and shared-drive flags where intended; a partial first page must not trigger removals. +- Native Google Docs may have no byte size and require export later; metadata normalization must allow nullable/zero size without treating the item as a folder. +- Scope remediation belongs to Phase 13. + +### OneDrive + +- Normalize `id`, `parentReference`, `file`/`folder` facets, `size`, `lastModifiedDateTime`, `eTag`, and `cTag` when present. +- Follow `@odata.nextLink`; incomplete pagination cannot be considered a complete reconciliation. +- Preserve drive identity where IDs are only meaningful with a drive/account context. + +### Nextcloud/WebDAV + +- Replace N+1 `list()` + `info()` where practical with a Depth-1 PROPFIND that requests needed properties in one response. +- Normalize `getetag`, `getlastmodified`, `getcontentlength`, `getcontenttype`, and `resourcetype` where available. +- Paths can change on rename/move and require careful percent encoding; retain provider path as a reference/snapshot, not global identity. +- Treat missing properties as unknown, not proof that an action is unsupported. + +## Validation Architecture + +### Test Layers + +| Layer | What to prove | Suggested location | +|-------|---------------|--------------------| +| Domain/unit | Capability merge/state/reason behavior; provider normalization; no destructive probes | `backend/tests/test_cloud_capabilities.py` | +| Service/unit | Owner-scoped upsert, stable identity, parent move/rename metadata update, complete-vs-partial reconciliation | `backend/tests/test_cloud_items.py` | +| DB integration | Constraints/indexes/cascades and same provider item IDs isolated across connections/users using PostgreSQL | integration-marked cloud item tests | +| API integration | Connection-ID routes, multiple same-provider accounts, foreign-owner/admin 404/403, credential exclusion, cached-first response schema | `backend/tests/test_cloud.py` or split cloud API tests | +| Provider contract | Google/OneDrive/WebDAV fixtures normalize to the same resource and capability schema; pagination is complete | `backend/tests/test_cloud_backends.py` plus focused adapter tests | +| Frontend unit | Connection-ID navigation, shared browser capabilities, disabled tooltip on focus/hover/tap, warning states, breadcrumb freshness | component/store/view Vitest files | +| Regression | Existing local browser actions and cloud connect/disconnect remain intact | current backend/frontend suites | + +### Required Commands + +- Fast backend loop: `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_backends.py tests/test_cloud_capabilities.py tests/test_cloud_items.py` +- Full backend gate: `cd backend && pytest -v` +- Frontend focused loop: `cd frontend && npm test -- --run StorageBrowser CloudFolderView CloudStorageView cloudConnections` +- Full frontend gate: `cd frontend && npm test` +- Production build: `cd frontend && npm run build` +- PostgreSQL integration gate for schema/ownership constraints using the project integration environment. + +### Requirement-to-Evidence Map + +| Requirement | Primary evidence | +|-------------|------------------| +| CONN-04 | Provider contract tests plus shared-browser capability state tests | +| CLOUD-01 | Connection-ID browse API and `StorageBrowser` integration tests | +| CLOUD-08 | Unsupported/warning explanation tests across pointer, keyboard, and touch behavior | +| CACHE-01 | Browse tests asserting no byte fetch, MinIO object, or quota mutation | +| CACHE-02 | Cloud item schema/service tests proving metadata/index placeholders persist independently | +| SYNC-01 | Provider normalization and durable metadata persistence tests for IDs, parent, size, mtime, etag/version | + +### Nyquist Rule + +Every implementation task that creates a contract, service, endpoint, or component behavior should create or update its focused tests in the same plan. Do not defer all tests to a final plan. The final integration plan may add cross-layer and negative-security tests, but it must not be the first coverage for earlier functionality. + +## Planning Risks and Mitigations + +1. **Schema overreach into Phase 14:** Persist metadata and analysis/index placeholders, not byte-cache implementation or analysis jobs. +2. **Provider-specific branches leaking into UI:** Normalize capabilities and items in backend/provider adapters. +3. **Same-provider account ambiguity:** Connection UUID is mandatory in route, cache, and uniqueness boundaries. +4. **Silent OAuth mismatch:** Surface insufficient scope; leave consent/reconnect implementation to Phase 13. +5. **False deletion from pagination/failure:** Track complete successful listing before marking missing children absent. +6. **SQLite-only confidence:** Run PostgreSQL integration tests for UUID/FK/index/uniqueness and ownership-sensitive queries. +7. **Tooltip inaccessible on touch:** Use focusable/tappable `aria-disabled` controls, not an inert `disabled` button. + +## Recommended Plan Shape + +1. **Cloud resource contract and schema:** normalized capability/item types, migrations, owner-scoped cloud item service, tests. +2. **Provider normalization and durable browsing API:** connection-ID routes, pagination/completeness, metadata reconciliation, security tests. +3. **Shared browser integration:** connection roots, breadcrumbs/freshness, capability-aware actions and accessible explanations, frontend tests. +4. **Cross-layer verification and documentation:** PostgreSQL integration, admin/owner negatives, no-byte/no-quota assertions, version/docs updates required by project protocol. + +Plans 1 and frontend scaffolding can begin in parallel only if their shared response schema is locked first. Provider/API work depends on the domain schema. Final integration depends on all prior plans. + +## Primary References + +- Microsoft Graph driveItem: https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0 +- Microsoft Graph OneDrive resource model: https://learn.microsoft.com/en-us/graph/api/resources/onedrive?view=graph-rest-1.0 +- Google Drive files resource and item capabilities: https://developers.google.com/workspace/drive/api/reference/rest/v3/files +- Google Drive OAuth scopes: https://developers.google.com/workspace/drive/api/guides/api-specific-auth +- Nextcloud WebDAV API: https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/basic.html +- RFC 4918 WebDAV: https://www.rfc-editor.org/rfc/rfc4918 +- Celery retry behavior: https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying + +## RESEARCH COMPLETE diff --git a/.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md b/.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md new file mode 100644 index 0000000..4a47fff --- /dev/null +++ b/.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md @@ -0,0 +1,82 @@ +--- +phase: 12 +slug: cloud-resource-foundation +status: draft +nyquist_compliant: true +wave_0_complete: false +created: 2026-06-18 +--- + +# Phase 12 — Validation Strategy + +> Per-phase validation contract for feedback sampling during execution. + +--- + +## Test Infrastructure + +| Property | Value | +|----------|-------| +| **Framework** | pytest 8.x, Vitest 4.x, Vue Test Utils 2.x | +| **Config file** | `backend/pytest.ini`, `frontend/vitest.config.js` | +| **Quick run command** | `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_backends.py tests/test_cloud_capabilities.py tests/test_cloud_items.py` and focused frontend Vitest files | +| **Full suite command** | `cd backend && pytest -v`; `cd frontend && npm test`; `cd frontend && npm run build` | +| **Estimated runtime** | Establish baseline during Wave 0; keep focused feedback under 60 seconds | + +--- + +## Sampling Rate + +- **After every task commit:** Run the focused test file(s) named by that task. +- **After every plan wave:** Run backend and frontend full suites for all touched layers. +- **Before `/gsd:verify-work`:** Full backend suite, frontend suite, production build, and required PostgreSQL integration tests must be green. +- **Max feedback latency:** 60 seconds for focused task verification. + +--- + +## Per-Task Verification Map + +| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status | +|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------| +| 12-01-01 | 01 | 1 | CONN-04, CLOUD-08 | T-12-06 | Capability discovery never mutates provider content | unit | `cd backend && pytest -q tests/test_cloud_capabilities.py` | ❌ W0 | ⬜ pending | +| 12-01-02 | 01 | 1 | CACHE-02, SYNC-01 | T-12-01, T-12-02 | Cloud item metadata is owner and connection scoped | unit + PostgreSQL integration | `cd backend && pytest -q tests/test_cloud_items.py` | ❌ W0 | ⬜ pending | +| 12-02-01 | 02 | 2 | CLOUD-01, SYNC-01 | T-12-01, T-12-07 | Connection-ID browse cannot cross owners and only reconciles complete listings | API integration | `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_items.py` | Existing + W0 additions | ⬜ pending | +| 12-02-02 | 02 | 2 | CACHE-01 | T-12-09 | Browse performs no byte download, MinIO write, or quota mutation | negative integration | `cd backend && pytest -q tests/test_cloud.py -k 'no_bytes or quota'` | ❌ W0 | ⬜ pending | +| 12-03-01 | 03 | 2 | CONN-04, CLOUD-08 | T-12-10 | Unsupported controls remain safe, focusable, and explanatory on pointer/keyboard/touch | component | `cd frontend && npm test -- --run StorageBrowser` | Existing file needs expansion | ⬜ pending | +| 12-03-02 | 03 | 2 | CLOUD-01 | T-12-01 | Frontend routes and requests use connection UUID, including duplicate providers | store/view component | `cd frontend && npm test -- --run CloudFolderView CloudStorageView cloudConnections` | Partial/W0 | ⬜ pending | +| 12-04-01 | 04 | 3 | All Phase 12 | T-12-01..10 | Full owner/admin, regression, build, and PostgreSQL gates pass | integration/regression | full suite commands above | Existing + additions | ⬜ pending | + +*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* + +--- + +## Wave 0 Requirements + +- [ ] `backend/tests/test_cloud_capabilities.py` — capability normalization, reason states, safe discovery, item overrides. +- [ ] `backend/tests/test_cloud_items.py` — owner-scoped persistence, stable identity, reconciliation, metadata-only invariants. +- [ ] PostgreSQL fixtures/marker for unique constraints, foreign keys, UUIDs, indexes, and cascade behavior. +- [ ] Frontend test files for `CloudFolderView.vue` and `CloudStorageView.vue` if focused files do not already exist. +- [ ] Shared provider response fixtures for Google Drive, OneDrive, Nextcloud, and generic WebDAV normalization. +- [ ] Baseline focused and full-suite runtimes recorded before implementation begins. + +--- + +## Manual-Only Verifications + +| Behavior | Requirement | Why Manual | Test Instructions | +|----------|-------------|------------|-------------------| +| Tooltip placement and clarity across desktop/mobile viewports | CLOUD-08 | Visual polish and viewport clipping need rendered-browser review | Use Playwright at mobile and desktop widths; hover, keyboard-focus, and tap unsupported controls; verify explanation is readable and does not overlap adjacent UI | +| Cached-first folder transition preserves perceived continuity | CLOUD-01 | Perceived flicker and scroll stability benefit from browser observation | Seed cached metadata, delay provider refresh, navigate folders, and verify rows remain usable with only the subtle folder-level refresh indicator | + +--- + +## Validation Sign-Off + +- [x] All proposed implementation tasks have an automated verification route or Wave 0 dependency. +- [x] Sampling continuity: no three consecutive tasks may occur without automated verification. +- [x] Wave 0 identifies every currently missing focused test/fixture. +- [x] Commands use one-shot test modes, not watch mode. +- [x] Focused feedback target is under 60 seconds. +- [x] `nyquist_compliant: true` is set in frontmatter. + +**Approval:** pending plan-checker validation