curo/kite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
kite/.planning/phases/12-cloud-resource-foundation/12-RESEARCH.md
T

18 KiB
Raw Permalink Blame History

Phase 12: Cloud Resource Foundation - Research

Researched: 2026-06-18 Status: Complete

Research Question

What must be understood to plan a provider-neutral cloud capability contract, durable owner-scoped cloud item index, and capability-aware shared browser without copying provider-owned files into DocuVault storage?

Executive Summary

Phase 12 should introduce a cloud-resource layer beside the existing object-storage StorageBackend, not expand that byte-oriented contract until it becomes a file manager. The durable identity is CloudConnection.id; provider type is only an adapter selector. This is required for multiple accounts from one provider and means current provider-keyed routes, cache keys, and _get_active_connection() queries must change.

Browsing should use stale-while-revalidate semantics backed by PostgreSQL: return normalized metadata immediately, refresh from the provider, then upsert owner/connection-scoped rows. Provider-owned bytes remain absent. Capability responses need both operation support and a reason state so StorageBrowser.vue can distinguish unsupported actions from temporarily blocked actions without provider-specific branches.

Two cross-phase constraints must be made explicit. First, Google currently requests the drive.file scope, which is limited to files created by or opened with the app and cannot satisfy whole-existing-drive discovery; Phase 13 reconnect/credential work must request and communicate appropriate broader access. Second, retained byte-cache quota accounting belongs to Phase 14, but Phase 12 schemas should not conflate metadata size with stored byte usage.

Current-State Findings

Connection Identity Is Provider-Keyed

  • CloudConnection already has a UUID primary key and display_name, but _upsert_cloud_connection() and _get_active_connection() select by (user_id, provider).
  • Browse routes are /api/cloud/folders/{provider}/{folder_id} and frontend routes are /cloud/:provider/:folderId; both make a second account of the same provider ambiguous.
  • cloud_cache.py keys listings as {user_id}:{provider}:{folder_id}, so same-provider accounts would collide.
  • CloudStorageView.vue correctly iterates connection IDs, but navigates with conn.provider and discards the connection identity.

Planning consequence: migrate browse identity to connection UUID throughout API, route, cache, store, and view boundaries. Do not expose encrypted credentials or infer ownership from a client-supplied provider.

Existing Storage Contract Is Byte-Oriented

StorageBackend defines put/get/delete, presigned URLs, health, and stat. It does not model folders, normalized resource metadata, rename/move, provider capabilities, or change cursors. Nextcloud adds list_folder() outside the abstract contract, while Google and OneDrive listing functions live directly in api/cloud.py.

Planning consequence: add a separate cloud resource adapter contract. Keep StorageBackend for DocuVault-owned/local object storage and existing document byte operations. A cloud adapter can later support:

  • list_folder(parent_ref) returning normalized resources and a listing result
  • capabilities() for connection-level support
  • item-level capability overrides from provider metadata
  • future open, preview, upload, create_folder, rename, move, delete, and change-cursor operations

Phase 12 implements browse and capability reporting; Phase 13 implements mutations.

Listings Are Too Sparse and Ephemeral

Current Google/OneDrive/WebDAV listing dictionaries expose only id, name, is_dir, and size. The in-process TTL cache lasts 60 seconds and disappears on restart or another application instance. There is no persisted parent, MIME type, modified time, version/etag, last-seen time, or analysis/index placeholder.

Planning consequence: a cloud_items table should persist normalized metadata with a uniqueness boundary at (connection_id, provider_item_id) and explicit user_id for defense-in-depth ownership queries. Recommended fields:

  • UUID id, UUID user_id, UUID connection_id
  • provider-native provider_item_id, optional canonical path snapshot
  • optional parent_item_id plus provider parent reference/path
  • name, item_type (file/folder), optional content_type, size_bytes
  • optional etag, version, provider modified_at
  • last_seen_at, optional deleted_at, timestamps
  • analysis/index placeholders that do not imply byte ownership (analysis_status, optional extracted-text/index relation or future-compatible status fields)

Use foreign keys with cascade from user/connection, indexes for owner+connection+parent browsing, and a unique connection+provider-item constraint. Never make path the stable primary identity: rename and move change paths, while provider IDs are generally more stable.

Capability Data Has Two Levels

Provider APIs expose capability information differently:

  • Google Drive files.capabilities includes item-level booleans such as canRename, canTrash, and canMoveItemWithinDrive.
  • Microsoft Graph driveItem supplies stable IDs, parent references, eTags/cTags, facets, and permissions; some effective capability decisions require granted scopes and item context rather than one universal provider matrix.
  • WebDAV capability discovery can use standards-visible methods/properties and server responses, but arbitrary servers vary. Safe discovery must use OPTIONS, PROPFIND, configured permission information, and observed non-mutating responses, never test mutations.

Model each action as structured data rather than booleans alone:

  • state: supported, unsupported, or temporarily_unavailable
  • reason code: stable machine identifier such as provider_unsupported, insufficient_scope, read_only, reauth_required, offline, item_restricted
  • message: short user-facing explanation
  • optional remedy code for frontend action routing in later phases

Connection-level defaults should be merged with item-level provider capabilities. Keep the action vocabulary fixed across providers: browse, open, preview, upload, create-folder, rename, move, delete, and change-tracking.

Google OAuth Scope Is a Functional Blocker for the Milestone Vision

The current OAuth flow requests https://www.googleapis.com/auth/drive.file. Google documents this as access to files the app created or that users explicitly opened with the app. It is insufficient for discovering and analyzing all documents already organized in a user’s Drive.

Planning consequence: Phase 12 should expose an insufficient_scope capability/health reason when credentials cannot browse the intended corpus. Phase 13 owns reconnect and credential lifecycle, including a deliberate broader-scope consent flow. Do not silently widen permissions in Phase 12.

Metadata and Byte Quota Must Stay Separate

The context locks these semantics:

  • provider bytes are authoritative;
  • metadata/search/index records do not consume document-storage quota;
  • actual file bytes retained in DocuVault storage consume quota while present;
  • transient streaming that is immediately discarded does not consume storage quota.

Planning consequence: Phase 12 must not create MinIO objects or increment quota during browse. A future byte-cache table/service should reference cloud items and account for actual retained bytes atomically in Phase 14. cloud_items.size_bytes is provider metadata, not DocuVault quota usage.

Recommended Architecture

Backend Modules

Keep routers thin and split the growing cloud monolith along current project conventions:

  • backend/storage/cloud_base.py: normalized enums/dataclasses or Pydantic-neutral domain types and the abstract cloud resource adapter.
  • backend/storage/cloud_backend_factory.py: return resource-capable provider adapters by connection/provider.
  • Provider adapters: normalize metadata and capabilities close to provider-specific SDK/API responses.
  • backend/services/cloud_items.py: owner-scoped upsert/list/reconciliation service; raises domain errors, never HTTPException.
  • backend/api/cloud/connections.py and backend/api/cloud/browse.py or an equivalent sub-router split using the project’s no-prefix sub-router rule.
  • backend/api/schemas.py or backend/api/cloud/schemas.py: explicit whitelisted response models; never include credentials_enc.

Avoid a generic repository abstraction unless it removes repeated owner-scoped queries. The important abstraction is the provider resource contract and normalized response.

Browse/Reconciliation Flow

  1. Authenticate regular user and resolve connection_id with (connection.id, connection.user_id).
  2. Read durable child metadata for (user_id, connection_id, parent_ref).
  3. Return cached rows promptly with freshness fields and normalized capabilities.
  4. Refresh provider metadata in a bounded request/background path.
  5. Upsert observed resources by (connection_id, provider_item_id); update parent/name/type/size/mtime/etag/version and last_seen_at.
  6. Reconcile disappeared children conservatively. Phase 12 may mark listing membership stale/deleted only after a successful complete listing; never delete rows after a failed or partial provider response.
  7. Preserve stable DocuVault cloud item UUIDs so the frontend can reconcile silently without losing selection/scroll.

The API should distinguish fresh, refreshing, and warning folder states and include last_refreshed_at/safe error metadata. Exact asynchronous transport (request-triggered task, Celery refresh, or immediate refresh after cached response) is planner discretion, but it must work across multiple backend instances and not rely solely on process memory.

Frontend Contract

StorageBrowser.vue stays the only grid. Prefer data-driven props:

  • normalized rows with stable DocuVault IDs plus provider IDs hidden from presentation
  • connection root/breadcrumb data
  • action capability map with state/message/remedy metadata
  • folder freshness state and last-successful timestamp
  • byte-availability state (cloud_only now; cache states become active in Phase 14)

Emit generic actions. CloudStorageView and CloudFolderView remain thin route/store data providers. Replace mode === 'local' action hiding with capability rendering; local mode can supply a local capability set so one rendering path serves both sources.

Disabled native HTML buttons do not receive click events, so the touch requirement needs an accessible wrapper/trigger or aria-disabled control that blocks execution while still accepting focus/tap for explanation. The plan must test keyboard focus and touch/click explanation behavior.

Security Threat Model Inputs

Threat Phase 12 mitigation to plan
IDOR across cloud connections Resolve every connection by both UUID and current user; return 404 for foreign IDs
IDOR across cloud items Query item by user and connection; never trust provider ID alone
Credential disclosure Explicit response schemas; negative tests for credentials_enc, access tokens, refresh tokens, passwords
SSRF regression Preserve validate_cloud_url() before all WebDAV/Nextcloud outbound requests and redirects
Cache/data collision Include user and connection UUID in every durable/in-memory key
Destructive capability probing Only provider declarations and non-mutating calls; tests assert no mutation methods invoked
Partial-list data loss Reconcile removals only after a successful complete listing/page traversal
Admin content access Continue get_regular_user; admin-negative tests for browse and metadata endpoints
Quota inflation/escape Browse writes metadata only; assert quota and MinIO remain unchanged
Stored XSS through filenames/messages Vue text rendering only; no v-html; provider errors mapped to controlled messages

Provider-Specific Planning Notes

Google Drive

  • Request fields explicitly: IDs, parents, name, mime type, size, modified time, version/md5 where applicable, capabilities, and pagination token.
  • Handle all pages and shared-drive flags where intended; a partial first page must not trigger removals.
  • Native Google Docs may have no byte size and require export later; metadata normalization must allow nullable/zero size without treating the item as a folder.
  • Scope remediation belongs to Phase 13.

OneDrive

  • Normalize id, parentReference, file/folder facets, size, lastModifiedDateTime, eTag, and cTag when present.
  • Follow @odata.nextLink; incomplete pagination cannot be considered a complete reconciliation.
  • Preserve drive identity where IDs are only meaningful with a drive/account context.

Nextcloud/WebDAV

  • Replace N+1 list() + info() where practical with a Depth-1 PROPFIND that requests needed properties in one response.
  • Normalize getetag, getlastmodified, getcontentlength, getcontenttype, and resourcetype where available.
  • Paths can change on rename/move and require careful percent encoding; retain provider path as a reference/snapshot, not global identity.
  • Treat missing properties as unknown, not proof that an action is unsupported.

Validation Architecture

Test Layers

Layer What to prove Suggested location
Domain/unit Capability merge/state/reason behavior; provider normalization; no destructive probes backend/tests/test_cloud_capabilities.py
Service/unit Owner-scoped upsert, stable identity, parent move/rename metadata update, complete-vs-partial reconciliation backend/tests/test_cloud_items.py
DB integration Constraints/indexes/cascades and same provider item IDs isolated across connections/users using PostgreSQL integration-marked cloud item tests
API integration Connection-ID routes, multiple same-provider accounts, foreign-owner/admin 404/403, credential exclusion, cached-first response schema backend/tests/test_cloud.py or split cloud API tests
Provider contract Google/OneDrive/WebDAV fixtures normalize to the same resource and capability schema; pagination is complete backend/tests/test_cloud_backends.py plus focused adapter tests
Frontend unit Connection-ID navigation, shared browser capabilities, disabled tooltip on focus/hover/tap, warning states, breadcrumb freshness component/store/view Vitest files
Regression Existing local browser actions and cloud connect/disconnect remain intact current backend/frontend suites

Required Commands

  • Fast backend loop: cd backend && pytest -q tests/test_cloud.py tests/test_cloud_backends.py tests/test_cloud_capabilities.py tests/test_cloud_items.py
  • Full backend gate: cd backend && pytest -v
  • Frontend focused loop: cd frontend && npm test -- --run StorageBrowser CloudFolderView CloudStorageView cloudConnections
  • Full frontend gate: cd frontend && npm test
  • Production build: cd frontend && npm run build
  • PostgreSQL integration gate for schema/ownership constraints using the project integration environment.

Requirement-to-Evidence Map

Requirement Primary evidence
CONN-04 Provider contract tests plus shared-browser capability state tests
CLOUD-01 Connection-ID browse API and StorageBrowser integration tests
CLOUD-08 Unsupported/warning explanation tests across pointer, keyboard, and touch behavior
CACHE-01 Browse tests asserting no byte fetch, MinIO object, or quota mutation
CACHE-02 Cloud item schema/service tests proving metadata/index placeholders persist independently
SYNC-01 Provider normalization and durable metadata persistence tests for IDs, parent, size, mtime, etag/version

Nyquist Rule

Every implementation task that creates a contract, service, endpoint, or component behavior should create or update its focused tests in the same plan. Do not defer all tests to a final plan. The final integration plan may add cross-layer and negative-security tests, but it must not be the first coverage for earlier functionality.

Planning Risks and Mitigations

  1. Schema overreach into Phase 14: Persist metadata and analysis/index placeholders, not byte-cache implementation or analysis jobs.
  2. Provider-specific branches leaking into UI: Normalize capabilities and items in backend/provider adapters.
  3. Same-provider account ambiguity: Connection UUID is mandatory in route, cache, and uniqueness boundaries.
  4. Silent OAuth mismatch: Surface insufficient scope; leave consent/reconnect implementation to Phase 13.
  5. False deletion from pagination/failure: Track complete successful listing before marking missing children absent.
  6. SQLite-only confidence: Run PostgreSQL integration tests for UUID/FK/index/uniqueness and ownership-sensitive queries.
  7. Tooltip inaccessible on touch: Use focusable/tappable aria-disabled controls, not an inert disabled button.

Recommended Plan Shape

  1. Cloud resource contract and schema: normalized capability/item types, migrations, owner-scoped cloud item service, tests.
  2. Provider normalization and durable browsing API: connection-ID routes, pagination/completeness, metadata reconciliation, security tests.
  3. Shared browser integration: connection roots, breadcrumbs/freshness, capability-aware actions and accessible explanations, frontend tests.
  4. Cross-layer verification and documentation: PostgreSQL integration, admin/owner negatives, no-byte/no-quota assertions, version/docs updates required by project protocol.

Plans 1 and frontend scaffolding can begin in parallel only if their shared response schema is locked first. Provider/API work depends on the domain schema. Final integration depends on all prior plans.

Primary References

RESEARCH COMPLETE

Reference in New Issue View Git Blame Copy Permalink