92 lines
3.7 KiB
Markdown
92 lines
3.7 KiB
Markdown
# v0.3 Research: Architecture
|
|
|
|
**Milestone:** v0.3 Reimagining Cloud Storage integration
|
|
**Date:** 2026-06-17
|
|
|
|
## Target Model
|
|
|
|
Cloud files become virtual documents: provider-owned bytes remain in the connected storage, while DocuVault stores metadata, extracted text, topics, embeddings/search data, status, and a bounded cache of temporary bytes.
|
|
|
|
## Proposed Components
|
|
|
|
### Provider Capability Layer
|
|
|
|
Add a cloud file operation interface separate from the current object-storage `StorageBackend` contract:
|
|
|
|
- `list_folder`
|
|
- `get_metadata`
|
|
- `download_bytes` / streaming equivalent
|
|
- `upload_file`
|
|
- `rename_item`
|
|
- `move_item`
|
|
- `delete_item`
|
|
- `create_folder`
|
|
- `get_change_cursor` / `list_changes` where available
|
|
- `capabilities()` returning action support and limitations
|
|
|
|
This avoids forcing local/MinIO object semantics onto provider file-manager behavior.
|
|
|
|
### Cloud Item Index
|
|
|
|
Add durable tables for cloud-visible items and analysis/index state:
|
|
|
|
- `cloud_items`: user, connection, provider, provider item id/path, parent id/path, name, kind, MIME/content type, size, etag/version, mtime, path snapshot, last_seen_at, deleted_at.
|
|
- `cloud_item_analysis`: item, status, extracted_text, topic links, semantic index status, last_analyzed_etag, error, job id, timestamps.
|
|
- Optional `cloud_file_cache`: item, etag/version, byte location, size, last_accessed_at, expires_at, pin reason, eviction state.
|
|
|
|
Keep `documents` for DocuVault-owned local/imported files unless a later design intentionally merges local and cloud into one polymorphic table.
|
|
|
|
### API Shape
|
|
|
|
Split `backend/api/cloud.py` when implementation starts:
|
|
|
|
- `api/cloud/connections.py`
|
|
- `api/cloud/browse.py`
|
|
- `api/cloud/actions.py`
|
|
- `api/cloud/analysis.py`
|
|
- `api/cloud/search.py` or reuse a shared search API
|
|
- shared schemas in `api/schemas.py` or `api/cloud/schemas.py`
|
|
|
|
Every endpoint must assert `connection.user_id == current_user.id` and `cloud_item.user_id == current_user.id`.
|
|
|
|
### Frontend Shape
|
|
|
|
`FileManagerView` and `CloudFolderView` remain thin data providers. `StorageBrowser.vue` becomes capability-aware:
|
|
|
|
- accepts `capabilities` prop
|
|
- shows action buttons based on capabilities
|
|
- emits generic events for open/upload/rename/move/delete/analyze
|
|
- displays status badges for cached/indexed/analyzing/stale/error
|
|
|
|
Cloud-specific stores own provider state and API calls; `StorageBrowser` stays store-agnostic.
|
|
|
|
### Byte Cache and Offload
|
|
|
|
Cache flow:
|
|
|
|
1. User opens/previews/analyzes cloud item.
|
|
2. Backend verifies ownership and provider metadata.
|
|
3. Backend downloads bytes into a bounded cache or streams through memory if small.
|
|
4. Extraction/preview reads from cache.
|
|
5. Cache entry records item id, etag/version, byte size, last access, expiry.
|
|
6. Eviction removes bytes once unpinned and stale or over budget.
|
|
|
|
The cache is not the source of truth. Provider metadata and extracted/indexed data are.
|
|
|
|
### Analysis Flow
|
|
|
|
1. User selects file/folder/provider and clicks Analyze.
|
|
2. Backend resolves selection into cloud items, stores/updates metadata rows, and enqueues Celery jobs.
|
|
3. Jobs hydrate bytes as needed, extract text, classify topics, optionally create embeddings, and persist analysis state.
|
|
4. Search reads only persisted text/vector/topic metadata; it should not download cloud bytes during query.
|
|
5. Opening a result hydrates bytes on demand if preview/download needs them.
|
|
|
|
## Build Order
|
|
|
|
1. Capability model + provider action interface + tests.
|
|
2. Cloud item metadata/index schema + browse persistence.
|
|
3. Unified browser actions for cloud files/folders.
|
|
4. Cloud analysis jobs and cache/offload service.
|
|
5. Unified smart search over local and analyzed cloud documents.
|
|
6. Provider change detection, stale markers, and polish.
|