kite/.planning/research/STACK.md

v0.3 Research: Stack

Milestone: v0.3 Reimagining Cloud Storage integration Date: 2026-06-17

Recommendation

Keep the existing FastAPI + SQLAlchemy + PostgreSQL + MinIO + Celery + Vue stack. Add a first-class cloud metadata/index layer and a bounded byte cache instead of adding a separate sync server, filesystem mount, or full local mirror.

Existing Pieces to Extend

• backend/storage/base.py: current StorageBackend is object-storage shaped. v0.3 needs a second interface or capability extension for file-manager operations: list, open/download stream, upload to path, rename, move, delete, mkdir, stat, and provider change cursors where available.
• backend/api/cloud.py: currently owns connect/list folders. It should split into focused modules as v0.2 did for admin/documents/auth once behavior expands.
• backend/services/cloud_cache.py: current TTL folder-listing cache is metadata-only. Keep it for short folder listings, but add a durable DB-backed cloud item index and a separate byte-cache service for extraction/preview.
• backend/tasks/document_tasks.py: extraction/classification already retrieves bytes from cloud backends for cloud-backed Document rows. v0.3 should add dedicated cloud analysis jobs that create/update index records from existing provider files without requiring upload into MinIO.
• frontend/src/components/storage/StorageBrowser.vue: already unifies local/cloud browsing. v0.3 should make actions capability-aware instead of mode === "local" gated.

Provider APIs

• Microsoft Graph driveItem supports list children, update item, upload, download, delete, move, copy, search, preview, and delta tracking. It exposes eTags/cTags and short-lived download URLs that must not be cached. Source: https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0
• Google Drive API supports resumable uploads and file metadata operations. v0.3 should normalize Drive file IDs, parent IDs, MIME types, and Workspace-document export limitations. Source: https://developers.google.com/workspace/drive/api/guides/manage-uploads
• Nextcloud/WebDAV supports standard PROPFIND, PUT, GET, DELETE, MOVE, COPY, and metadata response headers such as OC-Etag and OC-FileId. Source: https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/basic.html

Cache and Index Stack

• Store cloud item metadata in PostgreSQL, not Redis: provider, connection, provider item id/path, parent id/path, name, MIME/content type, size, etag/content hash, mtime, folder/file flag, provider capabilities, analysis/index status, last seen at, and error state.
• Store extracted text and classification/search metadata in PostgreSQL, linked to either local documents or cloud item records. Avoid copying full bytes to MinIO unless the user explicitly imports or provider capability requires it.
• Use MinIO or a local temp directory only as a bounded byte cache for preview/extraction. Cache entries need owner, provider item version, byte size, last access, and eviction status.
• Redis can remain a coordination/cache primitive. If used for byte-cache metadata or job locks, configure memory limits and eviction intentionally; Redis recommends choosing policies such as allkeys-lru for cache-like workloads. Source: https://redis.io/docs/latest/develop/reference/eviction/

Background Jobs

• Use Celery for analysis, indexing, cache hydration, cache eviction, and provider refresh jobs. Celery supports autoretry with exponential backoff and jitter for external services. Source: https://docs.celeryq.dev/en/stable/userguide/tasks.html
• Jobs should be idempotent by (connection_id, provider_item_id, version/etag) to survive retries, duplicate selections, and provider race conditions.

Search Stack

• Keep PostgreSQL full-text search for keyword/sentence search over extracted text.
• Add semantic search as a scoped phase using pgvector if the project wants idea search in v0.3. pgvector keeps vectors in Postgres, supports exact/approximate nearest-neighbor search, and avoids introducing a separate vector database. Source: https://github.com/pgvector/pgvector
• Semantic indexing should be optional per analyzed item and provider-independent.

What Not to Add

• Do not mount user cloud drives into the container filesystem.
• Do not build a full Dropbox/Nextcloud sync daemon in v0.3.
• Do not mirror entire cloud accounts into MinIO by default.
• Do not create a second cloud file grid. Extend StorageBrowser.vue.
• Do not expose provider download URLs directly to the browser unless proxied and authorized.