docs(phase-12): add research and validation strategy

This commit is contained in:
curo1305
2026-06-18 22:13:23 +02:00
parent 09814c28a9
commit 52b110acef
2 changed files with 319 additions and 0 deletions
@@ -0,0 +1,237 @@
# Phase 12: Cloud Resource Foundation - Research
**Researched:** 2026-06-18
**Status:** Complete
## Research Question
What must be understood to plan a provider-neutral cloud capability contract, durable owner-scoped cloud item index, and capability-aware shared browser without copying provider-owned files into DocuVault storage?
## Executive Summary
Phase 12 should introduce a cloud-resource layer beside the existing object-storage `StorageBackend`, not expand that byte-oriented contract until it becomes a file manager. The durable identity is `CloudConnection.id`; provider type is only an adapter selector. This is required for multiple accounts from one provider and means current provider-keyed routes, cache keys, and `_get_active_connection()` queries must change.
Browsing should use stale-while-revalidate semantics backed by PostgreSQL: return normalized metadata immediately, refresh from the provider, then upsert owner/connection-scoped rows. Provider-owned bytes remain absent. Capability responses need both operation support and a reason state so `StorageBrowser.vue` can distinguish unsupported actions from temporarily blocked actions without provider-specific branches.
Two cross-phase constraints must be made explicit. First, Google currently requests the `drive.file` scope, which is limited to files created by or opened with the app and cannot satisfy whole-existing-drive discovery; Phase 13 reconnect/credential work must request and communicate appropriate broader access. Second, retained byte-cache quota accounting belongs to Phase 14, but Phase 12 schemas should not conflate metadata size with stored byte usage.
## Current-State Findings
### Connection Identity Is Provider-Keyed
- `CloudConnection` already has a UUID primary key and `display_name`, but `_upsert_cloud_connection()` and `_get_active_connection()` select by `(user_id, provider)`.
- Browse routes are `/api/cloud/folders/{provider}/{folder_id}` and frontend routes are `/cloud/:provider/:folderId`; both make a second account of the same provider ambiguous.
- `cloud_cache.py` keys listings as `{user_id}:{provider}:{folder_id}`, so same-provider accounts would collide.
- `CloudStorageView.vue` correctly iterates connection IDs, but navigates with `conn.provider` and discards the connection identity.
**Planning consequence:** migrate browse identity to connection UUID throughout API, route, cache, store, and view boundaries. Do not expose encrypted credentials or infer ownership from a client-supplied provider.
### Existing Storage Contract Is Byte-Oriented
`StorageBackend` defines put/get/delete, presigned URLs, health, and stat. It does not model folders, normalized resource metadata, rename/move, provider capabilities, or change cursors. Nextcloud adds `list_folder()` outside the abstract contract, while Google and OneDrive listing functions live directly in `api/cloud.py`.
**Planning consequence:** add a separate cloud resource adapter contract. Keep `StorageBackend` for DocuVault-owned/local object storage and existing document byte operations. A cloud adapter can later support:
- `list_folder(parent_ref)` returning normalized resources and a listing result
- `capabilities()` for connection-level support
- item-level capability overrides from provider metadata
- future `open`, `preview`, `upload`, `create_folder`, `rename`, `move`, `delete`, and change-cursor operations
Phase 12 implements browse and capability reporting; Phase 13 implements mutations.
### Listings Are Too Sparse and Ephemeral
Current Google/OneDrive/WebDAV listing dictionaries expose only `id`, `name`, `is_dir`, and `size`. The in-process TTL cache lasts 60 seconds and disappears on restart or another application instance. There is no persisted parent, MIME type, modified time, version/etag, last-seen time, or analysis/index placeholder.
**Planning consequence:** a `cloud_items` table should persist normalized metadata with a uniqueness boundary at `(connection_id, provider_item_id)` and explicit `user_id` for defense-in-depth ownership queries. Recommended fields:
- UUID `id`, UUID `user_id`, UUID `connection_id`
- provider-native `provider_item_id`, optional canonical path snapshot
- optional `parent_item_id` plus provider parent reference/path
- `name`, `item_type` (`file`/`folder`), optional `content_type`, `size_bytes`
- optional `etag`, `version`, provider `modified_at`
- `last_seen_at`, optional `deleted_at`, timestamps
- analysis/index placeholders that do not imply byte ownership (`analysis_status`, optional extracted-text/index relation or future-compatible status fields)
Use foreign keys with cascade from user/connection, indexes for owner+connection+parent browsing, and a unique connection+provider-item constraint. Never make path the stable primary identity: rename and move change paths, while provider IDs are generally more stable.
### Capability Data Has Two Levels
Provider APIs expose capability information differently:
- Google Drive `files.capabilities` includes item-level booleans such as `canRename`, `canTrash`, and `canMoveItemWithinDrive`.
- Microsoft Graph `driveItem` supplies stable IDs, parent references, eTags/cTags, facets, and permissions; some effective capability decisions require granted scopes and item context rather than one universal provider matrix.
- WebDAV capability discovery can use standards-visible methods/properties and server responses, but arbitrary servers vary. Safe discovery must use `OPTIONS`, `PROPFIND`, configured permission information, and observed non-mutating responses, never test mutations.
Model each action as structured data rather than booleans alone:
- state: `supported`, `unsupported`, or `temporarily_unavailable`
- reason code: stable machine identifier such as `provider_unsupported`, `insufficient_scope`, `read_only`, `reauth_required`, `offline`, `item_restricted`
- message: short user-facing explanation
- optional remedy code for frontend action routing in later phases
Connection-level defaults should be merged with item-level provider capabilities. Keep the action vocabulary fixed across providers: browse, open, preview, upload, create-folder, rename, move, delete, and change-tracking.
### Google OAuth Scope Is a Functional Blocker for the Milestone Vision
The current OAuth flow requests `https://www.googleapis.com/auth/drive.file`. Google documents this as access to files the app created or that users explicitly opened with the app. It is insufficient for discovering and analyzing all documents already organized in a users Drive.
**Planning consequence:** Phase 12 should expose an `insufficient_scope` capability/health reason when credentials cannot browse the intended corpus. Phase 13 owns reconnect and credential lifecycle, including a deliberate broader-scope consent flow. Do not silently widen permissions in Phase 12.
### Metadata and Byte Quota Must Stay Separate
The context locks these semantics:
- provider bytes are authoritative;
- metadata/search/index records do not consume document-storage quota;
- actual file bytes retained in DocuVault storage consume quota while present;
- transient streaming that is immediately discarded does not consume storage quota.
**Planning consequence:** Phase 12 must not create MinIO objects or increment quota during browse. A future byte-cache table/service should reference cloud items and account for actual retained bytes atomically in Phase 14. `cloud_items.size_bytes` is provider metadata, not DocuVault quota usage.
## Recommended Architecture
### Backend Modules
Keep routers thin and split the growing cloud monolith along current project conventions:
- `backend/storage/cloud_base.py`: normalized enums/dataclasses or Pydantic-neutral domain types and the abstract cloud resource adapter.
- `backend/storage/cloud_backend_factory.py`: return resource-capable provider adapters by connection/provider.
- Provider adapters: normalize metadata and capabilities close to provider-specific SDK/API responses.
- `backend/services/cloud_items.py`: owner-scoped upsert/list/reconciliation service; raises domain errors, never `HTTPException`.
- `backend/api/cloud/connections.py` and `backend/api/cloud/browse.py` or an equivalent sub-router split using the projects no-prefix sub-router rule.
- `backend/api/schemas.py` or `backend/api/cloud/schemas.py`: explicit whitelisted response models; never include `credentials_enc`.
Avoid a generic repository abstraction unless it removes repeated owner-scoped queries. The important abstraction is the provider resource contract and normalized response.
### Browse/Reconciliation Flow
1. Authenticate regular user and resolve `connection_id` with `(connection.id, connection.user_id)`.
2. Read durable child metadata for `(user_id, connection_id, parent_ref)`.
3. Return cached rows promptly with freshness fields and normalized capabilities.
4. Refresh provider metadata in a bounded request/background path.
5. Upsert observed resources by `(connection_id, provider_item_id)`; update parent/name/type/size/mtime/etag/version and `last_seen_at`.
6. Reconcile disappeared children conservatively. Phase 12 may mark listing membership stale/deleted only after a successful complete listing; never delete rows after a failed or partial provider response.
7. Preserve stable DocuVault cloud item UUIDs so the frontend can reconcile silently without losing selection/scroll.
The API should distinguish `fresh`, `refreshing`, and `warning` folder states and include `last_refreshed_at`/safe error metadata. Exact asynchronous transport (request-triggered task, Celery refresh, or immediate refresh after cached response) is planner discretion, but it must work across multiple backend instances and not rely solely on process memory.
### Frontend Contract
`StorageBrowser.vue` stays the only grid. Prefer data-driven props:
- normalized rows with stable DocuVault IDs plus provider IDs hidden from presentation
- connection root/breadcrumb data
- action capability map with state/message/remedy metadata
- folder freshness state and last-successful timestamp
- byte-availability state (`cloud_only` now; cache states become active in Phase 14)
Emit generic actions. `CloudStorageView` and `CloudFolderView` remain thin route/store data providers. Replace `mode === 'local'` action hiding with capability rendering; local mode can supply a local capability set so one rendering path serves both sources.
Disabled native HTML buttons do not receive click events, so the touch requirement needs an accessible wrapper/trigger or `aria-disabled` control that blocks execution while still accepting focus/tap for explanation. The plan must test keyboard focus and touch/click explanation behavior.
## Security Threat Model Inputs
| Threat | Phase 12 mitigation to plan |
|--------|-----------------------------|
| IDOR across cloud connections | Resolve every connection by both UUID and current user; return 404 for foreign IDs |
| IDOR across cloud items | Query item by user and connection; never trust provider ID alone |
| Credential disclosure | Explicit response schemas; negative tests for `credentials_enc`, access tokens, refresh tokens, passwords |
| SSRF regression | Preserve `validate_cloud_url()` before all WebDAV/Nextcloud outbound requests and redirects |
| Cache/data collision | Include user and connection UUID in every durable/in-memory key |
| Destructive capability probing | Only provider declarations and non-mutating calls; tests assert no mutation methods invoked |
| Partial-list data loss | Reconcile removals only after a successful complete listing/page traversal |
| Admin content access | Continue `get_regular_user`; admin-negative tests for browse and metadata endpoints |
| Quota inflation/escape | Browse writes metadata only; assert quota and MinIO remain unchanged |
| Stored XSS through filenames/messages | Vue text rendering only; no `v-html`; provider errors mapped to controlled messages |
## Provider-Specific Planning Notes
### Google Drive
- Request fields explicitly: IDs, parents, name, mime type, size, modified time, version/md5 where applicable, capabilities, and pagination token.
- Handle all pages and shared-drive flags where intended; a partial first page must not trigger removals.
- Native Google Docs may have no byte size and require export later; metadata normalization must allow nullable/zero size without treating the item as a folder.
- Scope remediation belongs to Phase 13.
### OneDrive
- Normalize `id`, `parentReference`, `file`/`folder` facets, `size`, `lastModifiedDateTime`, `eTag`, and `cTag` when present.
- Follow `@odata.nextLink`; incomplete pagination cannot be considered a complete reconciliation.
- Preserve drive identity where IDs are only meaningful with a drive/account context.
### Nextcloud/WebDAV
- Replace N+1 `list()` + `info()` where practical with a Depth-1 PROPFIND that requests needed properties in one response.
- Normalize `getetag`, `getlastmodified`, `getcontentlength`, `getcontenttype`, and `resourcetype` where available.
- Paths can change on rename/move and require careful percent encoding; retain provider path as a reference/snapshot, not global identity.
- Treat missing properties as unknown, not proof that an action is unsupported.
## Validation Architecture
### Test Layers
| Layer | What to prove | Suggested location |
|-------|---------------|--------------------|
| Domain/unit | Capability merge/state/reason behavior; provider normalization; no destructive probes | `backend/tests/test_cloud_capabilities.py` |
| Service/unit | Owner-scoped upsert, stable identity, parent move/rename metadata update, complete-vs-partial reconciliation | `backend/tests/test_cloud_items.py` |
| DB integration | Constraints/indexes/cascades and same provider item IDs isolated across connections/users using PostgreSQL | integration-marked cloud item tests |
| API integration | Connection-ID routes, multiple same-provider accounts, foreign-owner/admin 404/403, credential exclusion, cached-first response schema | `backend/tests/test_cloud.py` or split cloud API tests |
| Provider contract | Google/OneDrive/WebDAV fixtures normalize to the same resource and capability schema; pagination is complete | `backend/tests/test_cloud_backends.py` plus focused adapter tests |
| Frontend unit | Connection-ID navigation, shared browser capabilities, disabled tooltip on focus/hover/tap, warning states, breadcrumb freshness | component/store/view Vitest files |
| Regression | Existing local browser actions and cloud connect/disconnect remain intact | current backend/frontend suites |
### Required Commands
- Fast backend loop: `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_backends.py tests/test_cloud_capabilities.py tests/test_cloud_items.py`
- Full backend gate: `cd backend && pytest -v`
- Frontend focused loop: `cd frontend && npm test -- --run StorageBrowser CloudFolderView CloudStorageView cloudConnections`
- Full frontend gate: `cd frontend && npm test`
- Production build: `cd frontend && npm run build`
- PostgreSQL integration gate for schema/ownership constraints using the project integration environment.
### Requirement-to-Evidence Map
| Requirement | Primary evidence |
|-------------|------------------|
| CONN-04 | Provider contract tests plus shared-browser capability state tests |
| CLOUD-01 | Connection-ID browse API and `StorageBrowser` integration tests |
| CLOUD-08 | Unsupported/warning explanation tests across pointer, keyboard, and touch behavior |
| CACHE-01 | Browse tests asserting no byte fetch, MinIO object, or quota mutation |
| CACHE-02 | Cloud item schema/service tests proving metadata/index placeholders persist independently |
| SYNC-01 | Provider normalization and durable metadata persistence tests for IDs, parent, size, mtime, etag/version |
### Nyquist Rule
Every implementation task that creates a contract, service, endpoint, or component behavior should create or update its focused tests in the same plan. Do not defer all tests to a final plan. The final integration plan may add cross-layer and negative-security tests, but it must not be the first coverage for earlier functionality.
## Planning Risks and Mitigations
1. **Schema overreach into Phase 14:** Persist metadata and analysis/index placeholders, not byte-cache implementation or analysis jobs.
2. **Provider-specific branches leaking into UI:** Normalize capabilities and items in backend/provider adapters.
3. **Same-provider account ambiguity:** Connection UUID is mandatory in route, cache, and uniqueness boundaries.
4. **Silent OAuth mismatch:** Surface insufficient scope; leave consent/reconnect implementation to Phase 13.
5. **False deletion from pagination/failure:** Track complete successful listing before marking missing children absent.
6. **SQLite-only confidence:** Run PostgreSQL integration tests for UUID/FK/index/uniqueness and ownership-sensitive queries.
7. **Tooltip inaccessible on touch:** Use focusable/tappable `aria-disabled` controls, not an inert `disabled` button.
## Recommended Plan Shape
1. **Cloud resource contract and schema:** normalized capability/item types, migrations, owner-scoped cloud item service, tests.
2. **Provider normalization and durable browsing API:** connection-ID routes, pagination/completeness, metadata reconciliation, security tests.
3. **Shared browser integration:** connection roots, breadcrumbs/freshness, capability-aware actions and accessible explanations, frontend tests.
4. **Cross-layer verification and documentation:** PostgreSQL integration, admin/owner negatives, no-byte/no-quota assertions, version/docs updates required by project protocol.
Plans 1 and frontend scaffolding can begin in parallel only if their shared response schema is locked first. Provider/API work depends on the domain schema. Final integration depends on all prior plans.
## Primary References
- Microsoft Graph driveItem: https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0
- Microsoft Graph OneDrive resource model: https://learn.microsoft.com/en-us/graph/api/resources/onedrive?view=graph-rest-1.0
- Google Drive files resource and item capabilities: https://developers.google.com/workspace/drive/api/reference/rest/v3/files
- Google Drive OAuth scopes: https://developers.google.com/workspace/drive/api/guides/api-specific-auth
- Nextcloud WebDAV API: https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/basic.html
- RFC 4918 WebDAV: https://www.rfc-editor.org/rfc/rfc4918
- Celery retry behavior: https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying
## RESEARCH COMPLETE
@@ -0,0 +1,82 @@
---
phase: 12
slug: cloud-resource-foundation
status: draft
nyquist_compliant: true
wave_0_complete: false
created: 2026-06-18
---
# Phase 12 — Validation Strategy
> Per-phase validation contract for feedback sampling during execution.
---
## Test Infrastructure
| Property | Value |
|----------|-------|
| **Framework** | pytest 8.x, Vitest 4.x, Vue Test Utils 2.x |
| **Config file** | `backend/pytest.ini`, `frontend/vitest.config.js` |
| **Quick run command** | `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_backends.py tests/test_cloud_capabilities.py tests/test_cloud_items.py` and focused frontend Vitest files |
| **Full suite command** | `cd backend && pytest -v`; `cd frontend && npm test`; `cd frontend && npm run build` |
| **Estimated runtime** | Establish baseline during Wave 0; keep focused feedback under 60 seconds |
---
## Sampling Rate
- **After every task commit:** Run the focused test file(s) named by that task.
- **After every plan wave:** Run backend and frontend full suites for all touched layers.
- **Before `/gsd:verify-work`:** Full backend suite, frontend suite, production build, and required PostgreSQL integration tests must be green.
- **Max feedback latency:** 60 seconds for focused task verification.
---
## Per-Task Verification Map
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
| 12-01-01 | 01 | 1 | CONN-04, CLOUD-08 | T-12-06 | Capability discovery never mutates provider content | unit | `cd backend && pytest -q tests/test_cloud_capabilities.py` | ❌ W0 | ⬜ pending |
| 12-01-02 | 01 | 1 | CACHE-02, SYNC-01 | T-12-01, T-12-02 | Cloud item metadata is owner and connection scoped | unit + PostgreSQL integration | `cd backend && pytest -q tests/test_cloud_items.py` | ❌ W0 | ⬜ pending |
| 12-02-01 | 02 | 2 | CLOUD-01, SYNC-01 | T-12-01, T-12-07 | Connection-ID browse cannot cross owners and only reconciles complete listings | API integration | `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_items.py` | Existing + W0 additions | ⬜ pending |
| 12-02-02 | 02 | 2 | CACHE-01 | T-12-09 | Browse performs no byte download, MinIO write, or quota mutation | negative integration | `cd backend && pytest -q tests/test_cloud.py -k 'no_bytes or quota'` | ❌ W0 | ⬜ pending |
| 12-03-01 | 03 | 2 | CONN-04, CLOUD-08 | T-12-10 | Unsupported controls remain safe, focusable, and explanatory on pointer/keyboard/touch | component | `cd frontend && npm test -- --run StorageBrowser` | Existing file needs expansion | ⬜ pending |
| 12-03-02 | 03 | 2 | CLOUD-01 | T-12-01 | Frontend routes and requests use connection UUID, including duplicate providers | store/view component | `cd frontend && npm test -- --run CloudFolderView CloudStorageView cloudConnections` | Partial/W0 | ⬜ pending |
| 12-04-01 | 04 | 3 | All Phase 12 | T-12-01..10 | Full owner/admin, regression, build, and PostgreSQL gates pass | integration/regression | full suite commands above | Existing + additions | ⬜ pending |
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
---
## Wave 0 Requirements
- [ ] `backend/tests/test_cloud_capabilities.py` — capability normalization, reason states, safe discovery, item overrides.
- [ ] `backend/tests/test_cloud_items.py` — owner-scoped persistence, stable identity, reconciliation, metadata-only invariants.
- [ ] PostgreSQL fixtures/marker for unique constraints, foreign keys, UUIDs, indexes, and cascade behavior.
- [ ] Frontend test files for `CloudFolderView.vue` and `CloudStorageView.vue` if focused files do not already exist.
- [ ] Shared provider response fixtures for Google Drive, OneDrive, Nextcloud, and generic WebDAV normalization.
- [ ] Baseline focused and full-suite runtimes recorded before implementation begins.
---
## Manual-Only Verifications
| Behavior | Requirement | Why Manual | Test Instructions |
|----------|-------------|------------|-------------------|
| Tooltip placement and clarity across desktop/mobile viewports | CLOUD-08 | Visual polish and viewport clipping need rendered-browser review | Use Playwright at mobile and desktop widths; hover, keyboard-focus, and tap unsupported controls; verify explanation is readable and does not overlap adjacent UI |
| Cached-first folder transition preserves perceived continuity | CLOUD-01 | Perceived flicker and scroll stability benefit from browser observation | Seed cached metadata, delay provider refresh, navigate folders, and verify rows remain usable with only the subtle folder-level refresh indicator |
---
## Validation Sign-Off
- [x] All proposed implementation tasks have an automated verification route or Wave 0 dependency.
- [x] Sampling continuity: no three consecutive tasks may occur without automated verification.
- [x] Wave 0 identifies every currently missing focused test/fixture.
- [x] Commands use one-shot test modes, not watch mode.
- [x] Focused feedback target is under 60 seconds.
- [x] `nyquist_compliant: true` is set in frontmatter.
**Approval:** pending plan-checker validation