docs(phase-14): plan selective analysis and byte cache
This commit is contained in:
curo1305
@@ -0,0 +1,129 @@
|
||||
# Phase 14 Research: Selective Analysis and Byte Cache
|
||||
|
||||
**Phase:** 14 - Selective Analysis and Byte Cache
|
||||
**Prepared:** 2026-06-23
|
||||
**Status:** Ready for execution planning
|
||||
|
||||
## Scope Boundary
|
||||
|
||||
Phase 14 adds controlled analysis of provider-owned cloud files and a bounded private byte cache. It does not import cloud files into local `documents`, does not implement unified search, and does not implement provider delta feeds beyond version/fingerprint checks needed to avoid duplicate or stale analysis.
|
||||
|
||||
The existing system already provides the correct foundation:
|
||||
|
||||
- `CloudItem` stores owner, connection, provider item id, parent, kind, content type, size, etag/version, extracted text, analysis status, semantic status, and topic links without a local `Document`.
|
||||
- `CloudFolderView` remains a route/store data provider; `StorageBrowser` owns the shared local/cloud browser surface.
|
||||
- `cloud_tasks.py` and `document_tasks.py` show the accepted Celery sync-to-async bridge and retry sentinel pattern.
|
||||
- `MinIOBackend` already generates UUID-based private object keys and is the right precedent for temporary cached cloud bytes.
|
||||
|
||||
## Recommended Schema Additions
|
||||
|
||||
Add a new migration after the current head with owner-scoped tables rather than overloading local `documents`.
|
||||
|
||||
### `cloud_byte_cache_entries`
|
||||
|
||||
Purpose: track retained cloud bytes in MinIO and make eviction/quota decisions without exposing provider credentials or paths.
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id` UUID primary key
|
||||
- `user_id` FK users, required
|
||||
- `connection_id` FK cloud_connections, required
|
||||
- `cloud_item_id` FK cloud_items, required
|
||||
- `provider_item_id` text, required snapshot for audit/debugging without joins
|
||||
- `version_key` text, required, derived from etag/version/fallback fingerprint/content hash
|
||||
- `object_key` text, required, UUID/private key only
|
||||
- `content_type` text nullable
|
||||
- `size_bytes` bigint required
|
||||
- `content_hash` text nullable, computed only when bytes are already hydrated
|
||||
- `pin_count` integer required default 0
|
||||
- `active_job_count` integer required default 0
|
||||
- `last_accessed_at`, `created_at`, `updated_at`, `evicted_at`
|
||||
- unique `(user_id, connection_id, cloud_item_id, version_key)`
|
||||
- indexes on `(user_id, evicted_at, last_accessed_at)` and `(user_id, pin_count, active_job_count, last_accessed_at)`
|
||||
|
||||
### `cloud_analysis_jobs`
|
||||
|
||||
Purpose: represent user-visible batches for file, selection, folder, or connection analysis.
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id` UUID primary key
|
||||
- `user_id`, `connection_id`
|
||||
- `scope_kind`: `file`, `selection`, `folder`, `connection`
|
||||
- `scope_ref` nullable provider item id or root sentinel
|
||||
- `recursive` boolean
|
||||
- `status`: `queued`, `running`, `paused`, `completed`, `cancelled`, `failed`
|
||||
- aggregate counts: total, queued, downloading, extracting, classifying, indexed, skipped, cancelled, failed, unsupported
|
||||
- aggregate provider bytes estimate
|
||||
- `failure_behavior`: `pause_batch` or `continue_item`
|
||||
- timestamps
|
||||
|
||||
### `cloud_analysis_job_items`
|
||||
|
||||
Purpose: drive idempotent per-item processing and precise user controls.
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`, `job_id`, `user_id`, `connection_id`, `cloud_item_id`
|
||||
- `provider_item_id`, `version_key`, `fingerprint`
|
||||
- `status`: `queued`, `downloading`, `extracting`, `classifying`, `indexed`, `already_current`, `cancelled`, `failed`, `unsupported`, `stale`
|
||||
- `error_code`, `error_message` controlled strings only
|
||||
- `cache_entry_id` nullable
|
||||
- timestamps and retry counter
|
||||
- unique `(job_id, cloud_item_id)`
|
||||
- index `(user_id, cloud_item_id, version_key)` for duplicate/current checks
|
||||
|
||||
### User Settings
|
||||
|
||||
Prefer explicit columns or a small owner-scoped settings table for:
|
||||
|
||||
- `analysis_progress_detail`: `simple` default, `detailed` optional
|
||||
- `analysis_failure_behavior`: `pause_batch` default, `continue_item` optional
|
||||
- `cloud_cache_limit_bytes`: user preference capped by tier/admin maximum
|
||||
|
||||
If the project already has a settings table at implementation time, extend it rather than creating a second settings authority.
|
||||
|
||||
## Version and Fingerprint Rules
|
||||
|
||||
Use this precedence for idempotency:
|
||||
|
||||
1. Provider `version` when available.
|
||||
2. Provider `etag` when available.
|
||||
3. Fallback metadata fingerprint: provider item id, provider size, modified time, content type.
|
||||
4. Content hash only after bytes are already hydrated for analysis/open/preview.
|
||||
|
||||
Never download bytes solely to compute a hash.
|
||||
|
||||
## Byte Cache Rules
|
||||
|
||||
- Cache bytes only for active open, preview, or analysis work.
|
||||
- Cached bytes count against `quotas.used_bytes` while retained.
|
||||
- Quota changes must use the project's existing atomic update pattern.
|
||||
- Eviction may delete only inactive entries with `pin_count == 0`, `active_job_count == 0`, and no active preview lease.
|
||||
- Eviction must be per-user LRU and must never delete another user's entries.
|
||||
- MinIO object keys must remain private UUID keys; provider filenames stay in metadata only.
|
||||
|
||||
## Job Execution Rules
|
||||
|
||||
- API requests enqueue jobs; Celery workers expand folders/connections and process items.
|
||||
- Worker broker payloads include only IDs and control flags; credentials are decrypted inside the worker.
|
||||
- Processing an unchanged cloud item reports `already_current` without provider byte download.
|
||||
- Retry restarts the whole item from the beginning.
|
||||
- Cancellation is cooperative: queued work is cancelled immediately, running download/extract/classify checks cancellation between stages and finishes safely if it cannot interrupt.
|
||||
- If provider metadata changes during analysis, finish the job safely, compare final version/fingerprint, and mark the result stale rather than mutating the provider item.
|
||||
|
||||
## Frontend Rules
|
||||
|
||||
- Add analysis actions to `StorageBrowser` row actions and toolbar.
|
||||
- `CloudFolderView` only maps route/store/API state and handles emitted events.
|
||||
- Use an aggregate indicator plus expandable item list.
|
||||
- Simplified progress labels are default; detailed labels are exposed from Settings.
|
||||
- The analysis queue can reuse the existing upload queue interaction pattern, but it must be a distinct queue model because controls and statuses differ.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- Admins remain blocked by `get_regular_user`.
|
||||
- All analysis/cache routes must resolve `(current_user.id, connection_id, provider_item_id/cloud_item_id)` before taking action.
|
||||
- Responses never include credentials, raw provider URLs, MinIO object keys, extracted text beyond explicitly authorized analysis/search surfaces, or provider error payloads.
|
||||
- Audit logs contain metadata only: connection id, provider item id, cloud item id, action/status, count/byte totals where needed.
|
||||
|
||||
Reference in New Issue
Block a user