49 lines
4.5 KiB
Markdown
49 lines
4.5 KiB
Markdown
# v0.3 Research: Stack
|
|
|
|
**Milestone:** v0.3 Reimagining Cloud Storage integration
|
|
**Date:** 2026-06-17
|
|
|
|
## Recommendation
|
|
|
|
Keep the existing FastAPI + SQLAlchemy + PostgreSQL + MinIO + Celery + Vue stack. Add a first-class cloud metadata/index layer and a bounded byte cache instead of adding a separate sync server, filesystem mount, or full local mirror.
|
|
|
|
## Existing Pieces to Extend
|
|
|
|
- `backend/storage/base.py`: current `StorageBackend` is object-storage shaped. v0.3 needs a second interface or capability extension for file-manager operations: list, open/download stream, upload to path, rename, move, delete, mkdir, stat, and provider change cursors where available.
|
|
- `backend/api/cloud.py`: currently owns connect/list folders. It should split into focused modules as v0.2 did for admin/documents/auth once behavior expands.
|
|
- `backend/services/cloud_cache.py`: current TTL folder-listing cache is metadata-only. Keep it for short folder listings, but add a durable DB-backed cloud item index and a separate byte-cache service for extraction/preview.
|
|
- `backend/tasks/document_tasks.py`: extraction/classification already retrieves bytes from cloud backends for cloud-backed `Document` rows. v0.3 should add dedicated cloud analysis jobs that create/update index records from existing provider files without requiring upload into MinIO.
|
|
- `frontend/src/components/storage/StorageBrowser.vue`: already unifies local/cloud browsing. v0.3 should make actions capability-aware instead of `mode === "local"` gated.
|
|
|
|
## Provider APIs
|
|
|
|
- Microsoft Graph `driveItem` supports list children, update item, upload, download, delete, move, copy, search, preview, and delta tracking. It exposes eTags/cTags and short-lived download URLs that must not be cached. Source: https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0
|
|
- Google Drive API supports resumable uploads and file metadata operations. v0.3 should normalize Drive file IDs, parent IDs, MIME types, and Workspace-document export limitations. Source: https://developers.google.com/workspace/drive/api/guides/manage-uploads
|
|
- Nextcloud/WebDAV supports standard `PROPFIND`, `PUT`, `GET`, `DELETE`, `MOVE`, `COPY`, and metadata response headers such as `OC-Etag` and `OC-FileId`. Source: https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/basic.html
|
|
|
|
## Cache and Index Stack
|
|
|
|
- Store cloud item metadata in PostgreSQL, not Redis: provider, connection, provider item id/path, parent id/path, name, MIME/content type, size, etag/content hash, mtime, folder/file flag, provider capabilities, analysis/index status, last seen at, and error state.
|
|
- Store extracted text and classification/search metadata in PostgreSQL, linked to either local `documents` or cloud item records. Avoid copying full bytes to MinIO unless the user explicitly imports or provider capability requires it.
|
|
- Use MinIO or a local temp directory only as a bounded byte cache for preview/extraction. Cache entries need owner, provider item version, byte size, last access, and eviction status.
|
|
- Redis can remain a coordination/cache primitive. If used for byte-cache metadata or job locks, configure memory limits and eviction intentionally; Redis recommends choosing policies such as `allkeys-lru` for cache-like workloads. Source: https://redis.io/docs/latest/develop/reference/eviction/
|
|
|
|
## Background Jobs
|
|
|
|
- Use Celery for analysis, indexing, cache hydration, cache eviction, and provider refresh jobs. Celery supports autoretry with exponential backoff and jitter for external services. Source: https://docs.celeryq.dev/en/stable/userguide/tasks.html
|
|
- Jobs should be idempotent by `(connection_id, provider_item_id, version/etag)` to survive retries, duplicate selections, and provider race conditions.
|
|
|
|
## Search Stack
|
|
|
|
- Keep PostgreSQL full-text search for keyword/sentence search over extracted text.
|
|
- Add semantic search as a scoped phase using `pgvector` if the project wants idea search in v0.3. pgvector keeps vectors in Postgres, supports exact/approximate nearest-neighbor search, and avoids introducing a separate vector database. Source: https://github.com/pgvector/pgvector
|
|
- Semantic indexing should be optional per analyzed item and provider-independent.
|
|
|
|
## What Not to Add
|
|
|
|
- Do not mount user cloud drives into the container filesystem.
|
|
- Do not build a full Dropbox/Nextcloud sync daemon in v0.3.
|
|
- Do not mirror entire cloud accounts into MinIO by default.
|
|
- Do not create a second cloud file grid. Extend `StorageBrowser.vue`.
|
|
- Do not expose provider download URLs directly to the browser unless proxied and authorized.
|