kite/.planning/phases/14-selective-analysis-and-byte-cache/14-RESEARCH.md

Phase 14 Research: Selective Analysis and Byte Cache

Phase: 14 - Selective Analysis and Byte Cache Prepared: 2026-06-23 Status: Ready for execution planning

Scope Boundary

Phase 14 adds controlled analysis of provider-owned cloud files and a bounded private byte cache. It does not import cloud files into local documents, does not implement unified search, and does not implement provider delta feeds beyond version/fingerprint checks needed to avoid duplicate or stale analysis.

The existing system already provides the correct foundation:

• CloudItem stores owner, connection, provider item id, parent, kind, content type, size, etag/version, extracted text, analysis status, semantic status, and topic links without a local Document.
• CloudFolderView remains a route/store data provider; StorageBrowser owns the shared local/cloud browser surface.
• cloud_tasks.py and document_tasks.py show the accepted Celery sync-to-async bridge and retry sentinel pattern.
• MinIOBackend already generates UUID-based private object keys and is the right precedent for temporary cached cloud bytes.

Recommended Schema Additions

Add a new migration after the current head with owner-scoped tables rather than overloading local documents.

cloud_byte_cache_entries

Purpose: track retained cloud bytes in MinIO and make eviction/quota decisions without exposing provider credentials or paths.

Suggested fields:

• id UUID primary key
• user_id FK users, required
• connection_id FK cloud_connections, required
• cloud_item_id FK cloud_items, required
• provider_item_id text, required snapshot for audit/debugging without joins
• version_key text, required, derived from etag/version/fallback fingerprint/content hash
• object_key text, required, UUID/private key only
• content_type text nullable
• size_bytes bigint required
• content_hash text nullable, computed only when bytes are already hydrated
• pin_count integer required default 0
• active_job_count integer required default 0
• last_accessed_at, created_at, updated_at, evicted_at
• unique (user_id, connection_id, cloud_item_id, version_key)
• indexes on (user_id, evicted_at, last_accessed_at) and (user_id, pin_count, active_job_count, last_accessed_at)

cloud_analysis_jobs

Purpose: represent user-visible batches for file, selection, folder, or connection analysis.

Suggested fields:

• id UUID primary key
• user_id, connection_id
• scope_kind: file, selection, folder, connection
• scope_ref nullable provider item id or root sentinel
• recursive boolean
• status: queued, running, paused, completed, cancelled, failed
• aggregate counts: total, queued, downloading, extracting, classifying, indexed, skipped, cancelled, failed, unsupported
• aggregate provider bytes estimate
• failure_behavior: pause_batch or continue_item
• timestamps

cloud_analysis_job_items

Purpose: drive idempotent per-item processing and precise user controls.

Suggested fields:

• id, job_id, user_id, connection_id, cloud_item_id
• provider_item_id, version_key, fingerprint
• status: queued, downloading, extracting, classifying, indexed, already_current, cancelled, failed, unsupported, stale
• error_code, error_message controlled strings only
• cache_entry_id nullable
• timestamps and retry counter
• unique (job_id, cloud_item_id)
• index (user_id, cloud_item_id, version_key) for duplicate/current checks

User Settings

Prefer explicit columns or a small owner-scoped settings table for:

• analysis_progress_detail: simple default, detailed optional
• analysis_failure_behavior: pause_batch default, continue_item optional
• cloud_cache_limit_bytes: user preference capped by tier/admin maximum

If the project already has a settings table at implementation time, extend it rather than creating a second settings authority.

Version and Fingerprint Rules

Use this precedence for idempotency:

1. Provider version when available.
2. Provider etag when available.
3. Fallback metadata fingerprint: provider item id, provider size, modified time, content type.
4. Content hash only after bytes are already hydrated for analysis/open/preview.

Never download bytes solely to compute a hash.

Byte Cache Rules

• Cache bytes only for active open, preview, or analysis work.
• Cached bytes count against quotas.used_bytes while retained.
• Quota changes must use the project's existing atomic update pattern.
• Eviction may delete only inactive entries with pin_count == 0, active_job_count == 0, and no active preview lease.
• Eviction must be per-user LRU and must never delete another user's entries.
• MinIO object keys must remain private UUID keys; provider filenames stay in metadata only.

Job Execution Rules

• API requests enqueue jobs; Celery workers expand folders/connections and process items.
• Worker broker payloads include only IDs and control flags; credentials are decrypted inside the worker.
• Processing an unchanged cloud item reports already_current without provider byte download.
• Retry restarts the whole item from the beginning.
• Cancellation is cooperative: queued work is cancelled immediately, running download/extract/classify checks cancellation between stages and finishes safely if it cannot interrupt.
• If provider metadata changes during analysis, finish the job safely, compare final version/fingerprint, and mark the result stale rather than mutating the provider item.

Frontend Rules

• Add analysis actions to StorageBrowser row actions and toolbar.
• CloudFolderView only maps route/store/API state and handles emitted events.
• Use an aggregate indicator plus expandable item list.
• Simplified progress labels are default; detailed labels are exposed from Settings.
• The analysis queue can reuse the existing upload queue interaction pattern, but it must be a distinct queue model because controls and statuses differ.

Security Notes

• Admins remain blocked by get_regular_user.
• All analysis/cache routes must resolve (current_user.id, connection_id, provider_item_id/cloud_item_id) before taking action.
• Responses never include credentials, raw provider URLs, MinIO object keys, extracted text beyond explicitly authorized analysis/search surfaces, or provider error payloads.
• Audit logs contain metadata only: connection id, provider item id, cloud item id, action/status, count/byte totals where needed.