kite/.planning/phases/12-cloud-resource-foundation/12-RESEARCH.md

# Phase 12: Cloud Resource Foundation - Research

**Researched:** 2026-06-18
**Status:** Complete

## Research Question

What must be understood to plan a provider-neutral cloud capability contract, durable owner-scoped cloud item index, and capability-aware shared browser without copying provider-owned files into DocuVault storage?

## Executive Summary

Phase 12 should introduce a cloud-resource layer beside the existing object-storage `StorageBackend`, not expand that byte-oriented contract until it becomes a file manager. The durable identity is `CloudConnection.id`; provider type is only an adapter selector. This is required for multiple accounts from one provider and means current provider-keyed routes, cache keys, and `_get_active_connection()` queries must change.

Browsing should use stale-while-revalidate semantics backed by PostgreSQL: return normalized metadata immediately, refresh from the provider, then upsert owner/connection-scoped rows. Provider-owned bytes remain absent. Capability responses need both operation support and a reason state so `StorageBrowser.vue` can distinguish unsupported actions from temporarily blocked actions without provider-specific branches.

Two cross-phase constraints must be made explicit. First, Google currently requests the `drive.file` scope, which is limited to files created by or opened with the app and cannot satisfy whole-existing-drive discovery; Phase 13 reconnect/credential work must request and communicate appropriate broader access. Second, retained byte-cache quota accounting belongs to Phase 14, but Phase 12 schemas should not conflate metadata size with stored byte usage.

## Current-State Findings

### Connection Identity Is Provider-Keyed

- `CloudConnection` already has a UUID primary key and `display_name`, but `_upsert_cloud_connection()` and `_get_active_connection()` select by `(user_id, provider)`.
- Browse routes are `/api/cloud/folders/{provider}/{folder_id}` and frontend routes are `/cloud/:provider/:folderId`; both make a second account of the same provider ambiguous.
- `cloud_cache.py` keys listings as `{user_id}:{provider}:{folder_id}`, so same-provider accounts would collide.
- `CloudStorageView.vue` correctly iterates connection IDs, but navigates with `conn.provider` and discards the connection identity.

**Planning consequence:** migrate browse identity to connection UUID throughout API, route, cache, store, and view boundaries. Do not expose encrypted credentials or infer ownership from a client-supplied provider.

### Existing Storage Contract Is Byte-Oriented

`StorageBackend` defines put/get/delete, presigned URLs, health, and stat. It does not model folders, normalized resource metadata, rename/move, provider capabilities, or change cursors. Nextcloud adds `list_folder()` outside the abstract contract, while Google and OneDrive listing functions live directly in `api/cloud.py`.

**Planning consequence:** add a separate cloud resource adapter contract. Keep `StorageBackend` for DocuVault-owned/local object storage and existing document byte operations. A cloud adapter can later support:

- `list_folder(parent_ref)` returning normalized resources and a listing result
- `capabilities()` for connection-level support
- item-level capability overrides from provider metadata
- future `open`, `preview`, `upload`, `create_folder`, `rename`, `move`, `delete`, and change-cursor operations

Phase 12 implements browse and capability reporting; Phase 13 implements mutations.

### Listings Are Too Sparse and Ephemeral

Current Google/OneDrive/WebDAV listing dictionaries expose only `id`, `name`, `is_dir`, and `size`. The in-process TTL cache lasts 60 seconds and disappears on restart or another application instance. There is no persisted parent, MIME type, modified time, version/etag, last-seen time, or analysis/index placeholder.

**Planning consequence:** a `cloud_items` table should persist normalized metadata with a uniqueness boundary at `(connection_id, provider_item_id)` and explicit `user_id` for defense-in-depth ownership queries. Recommended fields:

- UUID `id`, UUID `user_id`, UUID `connection_id`
- provider-native `provider_item_id`, optional canonical path snapshot
- optional `parent_item_id` plus provider parent reference/path
- `name`, `item_type` (`file`/`folder`), optional `content_type`, `size_bytes`
- optional `etag`, `version`, provider `modified_at`
- `last_seen_at`, optional `deleted_at`, timestamps
- analysis/index placeholders that do not imply byte ownership (`analysis_status`, optional extracted-text/index relation or future-compatible status fields)

Use foreign keys with cascade from user/connection, indexes for owner+connection+parent browsing, and a unique connection+provider-item constraint. Never make path the stable primary identity: rename and move change paths, while provider IDs are generally more stable.

### Capability Data Has Two Levels

Provider APIs expose capability information differently:

- Google Drive `files.capabilities` includes item-level booleans such as `canRename`, `canTrash`, and `canMoveItemWithinDrive`.
- Microsoft Graph `driveItem` supplies stable IDs, parent references, eTags/cTags, facets, and permissions; some effective capability decisions require granted scopes and item context rather than one universal provider matrix.
- WebDAV capability discovery can use standards-visible methods/properties and server responses, but arbitrary servers vary. Safe discovery must use `OPTIONS`, `PROPFIND`, configured permission information, and observed non-mutating responses, never test mutations.

Model each action as structured data rather than booleans alone:

- state: `supported`, `unsupported`, or `temporarily_unavailable`
- reason code: stable machine identifier such as `provider_unsupported`, `insufficient_scope`, `read_only`, `reauth_required`, `offline`, `item_restricted`
- message: short user-facing explanation
- optional remedy code for frontend action routing in later phases

Connection-level defaults should be merged with item-level provider capabilities. Keep the action vocabulary fixed across providers: browse, open, preview, upload, create-folder, rename, move, delete, and change-tracking.

### Google OAuth Scope Is a Functional Blocker for the Milestone Vision

The current OAuth flow requests `https://www.googleapis.com/auth/drive.file`. Google documents this as access to files the app created or that users explicitly opened with the app. It is insufficient for discovering and analyzing all documents already organized in a user’s Drive.

**Planning consequence:** Phase 12 should expose an `insufficient_scope` capability/health reason when credentials cannot browse the intended corpus. Phase 13 owns reconnect and credential lifecycle, including a deliberate broader-scope consent flow. Do not silently widen permissions in Phase 12.

### Metadata and Byte Quota Must Stay Separate

The context locks these semantics:

- provider bytes are authoritative;
- metadata/search/index records do not consume document-storage quota;
- actual file bytes retained in DocuVault storage consume quota while present;
- transient streaming that is immediately discarded does not consume storage quota.

**Planning consequence:** Phase 12 must not create MinIO objects or increment quota during browse. A future byte-cache table/service should reference cloud items and account for actual retained bytes atomically in Phase 14. `cloud_items.size_bytes` is provider metadata, not DocuVault quota usage.

## Recommended Architecture

### Backend Modules

Keep routers thin and split the growing cloud monolith along current project conventions:

- `backend/storage/cloud_base.py`: normalized enums/dataclasses or Pydantic-neutral domain types and the abstract cloud resource adapter.
- `backend/storage/cloud_backend_factory.py`: return resource-capable provider adapters by connection/provider.
- Provider adapters: normalize metadata and capabilities close to provider-specific SDK/API responses.
- `backend/services/cloud_items.py`: owner-scoped upsert/list/reconciliation service; raises domain errors, never `HTTPException`.
- `backend/api/cloud/connections.py` and `backend/api/cloud/browse.py` or an equivalent sub-router split using the project’s no-prefix sub-router rule.
- `backend/api/schemas.py` or `backend/api/cloud/schemas.py`: explicit whitelisted response models; never include `credentials_enc`.

Avoid a generic repository abstraction unless it removes repeated owner-scoped queries. The important abstraction is the provider resource contract and normalized response.

### Browse/Reconciliation Flow

1. Authenticate regular user and resolve `connection_id` with `(connection.id, connection.user_id)`.
2. Read durable child metadata for `(user_id, connection_id, parent_ref)`.
3. Return cached rows promptly with freshness fields and normalized capabilities.
4. Refresh provider metadata in a bounded request/background path.
5. Upsert observed resources by `(connection_id, provider_item_id)`; update parent/name/type/size/mtime/etag/version and `last_seen_at`.
6. Reconcile disappeared children conservatively. Phase 12 may mark listing membership stale/deleted only after a successful complete listing; never delete rows after a failed or partial provider response.
7. Preserve stable DocuVault cloud item UUIDs so the frontend can reconcile silently without losing selection/scroll.

The API should distinguish `fresh`, `refreshing`, and `warning` folder states and include `last_refreshed_at`/safe error metadata. Exact asynchronous transport (request-triggered task, Celery refresh, or immediate refresh after cached response) is planner discretion, but it must work across multiple backend instances and not rely solely on process memory.

### Frontend Contract

`StorageBrowser.vue` stays the only grid. Prefer data-driven props:

- normalized rows with stable DocuVault IDs plus provider IDs hidden from presentation
- connection root/breadcrumb data
- action capability map with state/message/remedy metadata
- folder freshness state and last-successful timestamp
- byte-availability state (`cloud_only` now; cache states become active in Phase 14)

Emit generic actions. `CloudStorageView` and `CloudFolderView` remain thin route/store data providers. Replace `mode === 'local'` action hiding with capability rendering; local mode can supply a local capability set so one rendering path serves both sources.

Disabled native HTML buttons do not receive click events, so the touch requirement needs an accessible wrapper/trigger or `aria-disabled` control that blocks execution while still accepting focus/tap for explanation. The plan must test keyboard focus and touch/click explanation behavior.

## Security Threat Model Inputs

| Threat | Phase 12 mitigation to plan |
|--------|-----------------------------|
| IDOR across cloud connections | Resolve every connection by both UUID and current user; return 404 for foreign IDs |
| IDOR across cloud items | Query item by user and connection; never trust provider ID alone |
| Credential disclosure | Explicit response schemas; negative tests for `credentials_enc`, access tokens, refresh tokens, passwords |
| SSRF regression | Preserve `validate_cloud_url()` before all WebDAV/Nextcloud outbound requests and redirects |
| Cache/data collision | Include user and connection UUID in every durable/in-memory key |
| Destructive capability probing | Only provider declarations and non-mutating calls; tests assert no mutation methods invoked |
| Partial-list data loss | Reconcile removals only after a successful complete listing/page traversal |
| Admin content access | Continue `get_regular_user`; admin-negative tests for browse and metadata endpoints |
| Quota inflation/escape | Browse writes metadata only; assert quota and MinIO remain unchanged |
| Stored XSS through filenames/messages | Vue text rendering only; no `v-html`; provider errors mapped to controlled messages |

## Provider-Specific Planning Notes

### Google Drive

- Request fields explicitly: IDs, parents, name, mime type, size, modified time, version/md5 where applicable, capabilities, and pagination token.
- Handle all pages and shared-drive flags where intended; a partial first page must not trigger removals.
- Native Google Docs may have no byte size and require export later; metadata normalization must allow nullable/zero size without treating the item as a folder.
- Scope remediation belongs to Phase 13.

### OneDrive

- Normalize `id`, `parentReference`, `file`/`folder` facets, `size`, `lastModifiedDateTime`, `eTag`, and `cTag` when present.
- Follow `@odata.nextLink`; incomplete pagination cannot be considered a complete reconciliation.
- Preserve drive identity where IDs are only meaningful with a drive/account context.

### Nextcloud/WebDAV

- Replace N+1 `list()` + `info()` where practical with a Depth-1 PROPFIND that requests needed properties in one response.
- Normalize `getetag`, `getlastmodified`, `getcontentlength`, `getcontenttype`, and `resourcetype` where available.
- Paths can change on rename/move and require careful percent encoding; retain provider path as a reference/snapshot, not global identity.
- Treat missing properties as unknown, not proof that an action is unsupported.

## Validation Architecture

### Test Layers

| Layer | What to prove | Suggested location |
|-------|---------------|--------------------|
| Domain/unit | Capability merge/state/reason behavior; provider normalization; no destructive probes | `backend/tests/test_cloud_capabilities.py` |
| Service/unit | Owner-scoped upsert, stable identity, parent move/rename metadata update, complete-vs-partial reconciliation | `backend/tests/test_cloud_items.py` |
| DB integration | Constraints/indexes/cascades and same provider item IDs isolated across connections/users using PostgreSQL | integration-marked cloud item tests |
| API integration | Connection-ID routes, multiple same-provider accounts, foreign-owner/admin 404/403, credential exclusion, cached-first response schema | `backend/tests/test_cloud.py` or split cloud API tests |
| Provider contract | Google/OneDrive/WebDAV fixtures normalize to the same resource and capability schema; pagination is complete | `backend/tests/test_cloud_backends.py` plus focused adapter tests |
| Frontend unit | Connection-ID navigation, shared browser capabilities, disabled tooltip on focus/hover/tap, warning states, breadcrumb freshness | component/store/view Vitest files |
| Regression | Existing local browser actions and cloud connect/disconnect remain intact | current backend/frontend suites |

### Required Commands

- Fast backend loop: `cd backend && pytest -q tests/test_cloud.py tests/test_cloud_backends.py tests/test_cloud_capabilities.py tests/test_cloud_items.py`
- Full backend gate: `cd backend && pytest -v`
- Frontend focused loop: `cd frontend && npm test -- --run StorageBrowser CloudFolderView CloudStorageView cloudConnections`
- Full frontend gate: `cd frontend && npm test`
- Production build: `cd frontend && npm run build`
- PostgreSQL integration gate for schema/ownership constraints using the project integration environment.

### Requirement-to-Evidence Map

| Requirement | Primary evidence |
|-------------|------------------|
| CONN-04 | Provider contract tests plus shared-browser capability state tests |
| CLOUD-01 | Connection-ID browse API and `StorageBrowser` integration tests |
| CLOUD-08 | Unsupported/warning explanation tests across pointer, keyboard, and touch behavior |
| CACHE-01 | Browse tests asserting no byte fetch, MinIO object, or quota mutation |
| CACHE-02 | Cloud item schema/service tests proving metadata/index placeholders persist independently |
| SYNC-01 | Provider normalization and durable metadata persistence tests for IDs, parent, size, mtime, etag/version |

### Nyquist Rule

Every implementation task that creates a contract, service, endpoint, or component behavior should create or update its focused tests in the same plan. Do not defer all tests to a final plan. The final integration plan may add cross-layer and negative-security tests, but it must not be the first coverage for earlier functionality.

## Planning Risks and Mitigations

1. **Schema overreach into Phase 14:** Persist metadata and analysis/index placeholders, not byte-cache implementation or analysis jobs.
2. **Provider-specific branches leaking into UI:** Normalize capabilities and items in backend/provider adapters.
3. **Same-provider account ambiguity:** Connection UUID is mandatory in route, cache, and uniqueness boundaries.
4. **Silent OAuth mismatch:** Surface insufficient scope; leave consent/reconnect implementation to Phase 13.
5. **False deletion from pagination/failure:** Track complete successful listing before marking missing children absent.
6. **SQLite-only confidence:** Run PostgreSQL integration tests for UUID/FK/index/uniqueness and ownership-sensitive queries.
7. **Tooltip inaccessible on touch:** Use focusable/tappable `aria-disabled` controls, not an inert `disabled` button.

## Recommended Plan Shape

1. **Cloud resource contract and schema:** normalized capability/item types, migrations, owner-scoped cloud item service, tests.
2. **Provider normalization and durable browsing API:** connection-ID routes, pagination/completeness, metadata reconciliation, security tests.
3. **Shared browser integration:** connection roots, breadcrumbs/freshness, capability-aware actions and accessible explanations, frontend tests.
4. **Cross-layer verification and documentation:** PostgreSQL integration, admin/owner negatives, no-byte/no-quota assertions, version/docs updates required by project protocol.

Plans 1 and frontend scaffolding can begin in parallel only if their shared response schema is locked first. Provider/API work depends on the domain schema. Final integration depends on all prior plans.

## Primary References

- Microsoft Graph driveItem: https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0
- Microsoft Graph OneDrive resource model: https://learn.microsoft.com/en-us/graph/api/resources/onedrive?view=graph-rest-1.0
- Google Drive files resource and item capabilities: https://developers.google.com/workspace/drive/api/reference/rest/v3/files
- Google Drive OAuth scopes: https://developers.google.com/workspace/drive/api/guides/api-specific-auth
- Nextcloud WebDAV API: https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/basic.html
- RFC 4918 WebDAV: https://www.rfc-editor.org/rfc/rfc4918
- Celery retry behavior: https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying

## RESEARCH COMPLETE