kite/.planning/research/ARCHITECTURE.md

v0.3 Research: Architecture

Milestone: v0.3 Reimagining Cloud Storage integration Date: 2026-06-17

Target Model

Cloud files become virtual documents: provider-owned bytes remain in the connected storage, while DocuVault stores metadata, extracted text, topics, embeddings/search data, status, and a bounded cache of temporary bytes.

Proposed Components

Provider Capability Layer

Add a cloud file operation interface separate from the current object-storage StorageBackend contract:

• list_folder
• get_metadata
• download_bytes / streaming equivalent
• upload_file
• rename_item
• move_item
• delete_item
• create_folder
• get_change_cursor / list_changes where available
• capabilities() returning action support and limitations

This avoids forcing local/MinIO object semantics onto provider file-manager behavior.

Cloud Item Index

Add durable tables for cloud-visible items and analysis/index state:

• cloud_items: user, connection, provider, provider item id/path, parent id/path, name, kind, MIME/content type, size, etag/version, mtime, path snapshot, last_seen_at, deleted_at.
• cloud_item_analysis: item, status, extracted_text, topic links, semantic index status, last_analyzed_etag, error, job id, timestamps.
• Optional cloud_file_cache: item, etag/version, byte location, size, last_accessed_at, expires_at, pin reason, eviction state.

Keep documents for DocuVault-owned local/imported files unless a later design intentionally merges local and cloud into one polymorphic table.

API Shape

Split backend/api/cloud.py when implementation starts:

• api/cloud/connections.py
• api/cloud/browse.py
• api/cloud/actions.py
• api/cloud/analysis.py
• api/cloud/search.py or reuse a shared search API
• shared schemas in api/schemas.py or api/cloud/schemas.py

Every endpoint must assert connection.user_id == current_user.id and cloud_item.user_id == current_user.id.

Frontend Shape

FileManagerView and CloudFolderView remain thin data providers. StorageBrowser.vue becomes capability-aware:

• accepts capabilities prop
• shows action buttons based on capabilities
• emits generic events for open/upload/rename/move/delete/analyze
• displays status badges for cached/indexed/analyzing/stale/error

Cloud-specific stores own provider state and API calls; StorageBrowser stays store-agnostic.

Byte Cache and Offload

Cache flow:

1. User opens/previews/analyzes cloud item.
2. Backend verifies ownership and provider metadata.
3. Backend downloads bytes into a bounded cache or streams through memory if small.
4. Extraction/preview reads from cache.
5. Cache entry records item id, etag/version, byte size, last access, expiry.
6. Eviction removes bytes once unpinned and stale or over budget.

The cache is not the source of truth. Provider metadata and extracted/indexed data are.

Analysis Flow

1. User selects file/folder/provider and clicks Analyze.
2. Backend resolves selection into cloud items, stores/updates metadata rows, and enqueues Celery jobs.
3. Jobs hydrate bytes as needed, extract text, classify topics, optionally create embeddings, and persist analysis state.
4. Search reads only persisted text/vector/topic metadata; it should not download cloud bytes during query.
5. Opening a result hydrates bytes on demand if preview/download needs them.

Build Order

1. Capability model + provider action interface + tests.
2. Cloud item metadata/index schema + browse persistence.
3. Unified browser actions for cloud files/folders.
4. Cloud analysis jobs and cache/offload service.
5. Unified smart search over local and analyzed cloud documents.
6. Provider change detection, stale markers, and polish.