From d3700e20c45bb2d1f7f35c2b14681d9e570dc8f0 Mon Sep 17 00:00:00 2001 From: curo1305 Date: Tue, 23 Jun 2026 13:59:53 +0200 Subject: [PATCH] docs(14): capture phase context --- .../14-CONTEXT.md | 154 ++++++++++++++ .../14-DISCUSSION-LOG.md | 199 ++++++++++++++++++ 2 files changed, 353 insertions(+) create mode 100644 .planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md create mode 100644 .planning/phases/14-selective-analysis-and-byte-cache/14-DISCUSSION-LOG.md diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md new file mode 100644 index 0000000..a851bdc --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-CONTEXT.md @@ -0,0 +1,154 @@ +# Phase 14: Selective Analysis and Byte Cache - Context + +**Gathered:** 2026-06-23 +**Status:** Ready for planning + + +## Phase Boundary + +Phase 14 lets users analyze provider-owned cloud files without importing them into local DocuVault documents. Users can analyze individual files, selected items, folder scopes, or whole connections through observable, controllable background queues. DocuVault hydrates provider bytes only for opening, preview, or analysis; retained bytes live in a private bounded cache with owner-scoped metadata, quota accounting, and safe eviction. Phase 15 owns unified keyword/semantic search over persisted analysis output, and Phase 16 owns provider change tracking/delta reliability beyond the staleness checks needed for analysis idempotency. + + + + +## Implementation Decisions + +### Analysis scope selection +- **D-01:** Analysis entry points use both rows and the shared browser toolbar. Single-file analysis appears on file rows; multi-select, folder-scope, and whole-connection analysis appear in the shared `StorageBrowser` toolbar. +- **D-02:** Estimate/review is threshold-based. Small obvious jobs may start directly; item count or provider-reported byte thresholds trigger an estimate step. Whole-connection analysis always shows an estimate unless trivially small. +- **D-03:** The estimate step shows total supported file count, total provider-reported bytes, unsupported/skipped item count, and a clear start action. It is not a full per-file include/exclude review UI in Phase 14. +- **D-04:** Folder analysis asks each time whether to analyze the current folder only or recurse through descendants. Whole-connection analysis is inherently recursive. + +### Job progress and controls +- **D-05:** Progress presentation uses an aggregate indicator plus an expandable queue/item list. The browser stays quiet by default, but detailed per-item state is available. +- **D-06:** Analysis progress detail is user-configurable in Settings. The default is simplified labels; advanced users can opt into roadmap-level detail such as queued, downloading, extracting, classifying, indexed, cancelled, and failed. +- **D-07:** Analysis uses a queue model similar to the cloud upload queue, covering both download/cache hydration and scanning/classification stages. +- **D-08:** The queue displays separate indicators for download/cache hydration and scanning/analysis. +- **D-09:** Queue rows support per-item cancel, skip, and abort/stop controls where meaningful. Batch-level controls include Cancel everything. +- **D-10:** Error handling mirrors the existing upload queue pattern: a failed item can pause for user decision, then the user can retry, skip, abort/stop, or cancel the whole batch. +- **D-11:** Failure handling has a user setting for the default behavior. Initial options are pause whole queue and wait for input, or pause the failed item and continue the queue. The model must leave room for future default behaviors. + +### Byte cache, storage quota, and scan quota +- **D-12:** Temporary cached cloud bytes count against the user's storage quota while retained. Metadata, extracted text, topics, and semantic/index records do not count as stored bytes. +- **D-13:** Phase 14 must prepare a separate tier-ready scan quota seam distinct from storage quota. Free-tier users may eventually be limited to a small number of scanned files per day, such as 1 or 5. +- **D-14:** Cache limits are per-user with LRU eviction. Eviction removes only inactive cached bytes and must preserve owner isolation. +- **D-15:** Cache limit is a tier-bounded user setting. Users choose a preferred cache limit up to a tier/admin-defined maximum. +- **D-16:** Eviction protects active jobs and open previews. Bytes used by running download, analysis, or preview work are pinned until no longer active. +- **D-17:** Cached bytes are stored in MinIO with a cache metadata table. Object keys remain UUID-based/private; PostgreSQL tracks owner, connection, cloud item/provider reference, version or etag, size, active/pin state, last access, and eviction metadata. + +### Idempotency and reanalysis +- **D-18:** If a cloud item has already been analyzed and its provider version or etag is unchanged, DocuVault skips re-download/reclassification and reports it as already current/indexed. +- **D-19:** If the provider lacks reliable etag/version data, use a fallback metadata fingerprint from provider item id, size, modified time, and content type before download. +- **D-20:** When bytes are already hydrated for analysis, compute and store a content hash. Do not download bytes solely to compute a hash. +- **D-21:** If an item changes externally during analysis, finish the current job safely, then compare version/fingerprint and mark the resulting analysis stale if it no longer matches the provider state. +- **D-22:** Retry after failure restarts the full item from the beginning. This favors predictable behavior over partial-stage retry optimization. +- **D-23:** Mixed batches process only actionable items. Current items are skipped as already indexed/current, stale and failed-prior items are queued, and unsupported items are reported without blocking actionable work. + +### Codex's Discretion +- Choose exact status labels, icons, thresholds, queue wording, and accessible control copy while preserving user-configurable detail and the decisions above. +- Choose the storage schema names and service decomposition consistent with existing owner-scoped cloud services, MinIO storage patterns, and Celery retry conventions. +- Choose initial defaults for threshold values, cache limit values, and failure behavior, provided they are tier/config ready and covered by tests. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Milestone and phase contracts +- `.planning/ROADMAP.md` - Phase 14 goal, requirements, success criteria, and Phase 15/16 boundaries. +- `.planning/REQUIREMENTS.md` - canonical ANALYZE-01 through ANALYZE-07 and CACHE-03 through CACHE-05 requirements; SEARCH and SYNC requirements remain later phases. +- `.planning/PROJECT.md` - virtual-local cloud storage model, privacy boundary, quota/tier direction, and shared-component architecture. +- `.planning/phases/12-cloud-resource-foundation/12-CONTEXT.md` - normalized cloud item metadata, byte availability, storage-quota distinction, and no-byte browse boundary inherited by Phase 14. +- `.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md` - truthful listing/freshness and provider-neutral correctness constraints. +- `.planning/phases/13-virtual-local-cloud-operations/13-CONTEXT.md` - authorized cloud open/preview, shared browser operations, queue semantics, provider mutation safeguards, and Phase 14 byte-cache boundary. +- `AGENTS.md` - non-negotiable shared-module, security, testing, documentation, environment, and Git rules. + +### Backend cloud, cache, and analysis surfaces +- `backend/db/models.py` - `CloudItem`, `CloudItemTopic`, and reserved analysis fields; new cache/scan quota schema should preserve owner and connection boundaries. +- `backend/storage/cloud_base.py` - provider-neutral resource/capability contract and normalized action vocabulary. +- `backend/storage/cloud_backend_factory.py` - adapter construction boundary where provider credentials are consumed. +- `backend/services/cloud_items.py` - existing owner-scoped cloud metadata reconciliation and lookup patterns. +- `backend/services/cloud_operations.py` - Phase 13 cloud operation orchestration and normalized provider error handling. +- `backend/tasks/cloud_tasks.py` - Celery bridge, retry sentinel pattern, owner revalidation, and credential decryption inside workers. +- `backend/tasks/document_tasks.py` - existing extraction/classification task pattern and retry harness; Phase 14 should not force provider-owned items into local `Document` rows. +- `backend/services/extractor.py` - byte-to-text extraction used by both local documents and future cloud analysis jobs. +- `backend/services/classifier.py` - AI topic classification path to reuse for cloud item analysis without leaking provider bytes or credentials. +- `backend/api/cloud/operations.py` - authorized cloud content routes and no-credential response boundary. +- `backend/api/cloud/schemas.py` - credential-free whitelisted response schemas. +- `backend/storage/minio_backend.py` - UUID/private object key and byte storage patterns to mirror for cached cloud bytes. + +### Frontend shared browser and state +- `frontend/src/components/storage/StorageBrowser.vue` - single shared browser; analysis row actions, toolbar actions, aggregate status, and expandable queue UI belong here rather than in a parallel cloud grid. +- `frontend/src/views/CloudFolderView.vue` - thin cloud data-provider integration for cloud browse, upload queue, open/preview, and future analysis handlers. +- `frontend/src/views/FileManagerView.vue` - local interaction reference for row actions and toolbar behavior. +- `frontend/src/stores/cloudConnections.js` - cloud browse, capability, freshness, health, byte-availability, and session navigation state. +- `frontend/src/api/cloud.js` and `frontend/src/api/client.js` - cloud API client surface and authenticated fetch behavior. +- `frontend/src/views/SettingsView.vue` - likely host for progress-detail, failure-behavior, and cache-limit settings. + +### Verification patterns +- `backend/tests/test_cloud.py` - owner-scoped cloud API integration patterns. +- `backend/tests/test_cloud_items.py` - cloud item persistence, analysis placeholder, reconciliation, and no-quota metadata tests. +- `backend/tests/test_cloud_security.py` - owner/admin/credential/SSRF/no-byte negative contract. +- `backend/tests/test_cloud_backends.py` - provider contract test patterns. +- `frontend/src/components/storage/__tests__/StorageBrowser.cloud-queue.test.js` - existing shared queue behavior to extend for analysis. +- `frontend/src/components/storage/__tests__/StorageBrowser.capabilities.test.js` - shared action rendering and disabled-capability coverage. +- `frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js` - rendered cloud browser flow coverage. + + + + +## Existing Code Insights + +### Reusable Assets +- `CloudItem`: already persists owner, connection, provider item id, parent, name, kind, content type, provider size, etag/version, extracted text, analysis status, semantic index status, and topic links without a local `Document` row. +- `StorageBrowser.vue`: already owns local/cloud rows, upload queue UI, toolbar/browser layout, and operation event emissions; Phase 14 should extend it rather than create another grid. +- `CloudFolderView.vue`: already maps opaque `provider_item_id`, server freshness, byte availability, upload queue behavior, and cloud open/preview handlers into `StorageBrowser`. +- `cloud_tasks.py` and `document_tasks.py`: provide established Celery sync-to-async bridge patterns, retry sentinels, owner revalidation, and worker-side credential/DB lookup boundaries. +- `extractor.py` and `classifier.py`: existing text extraction and AI classification services should be reused for cloud analysis output, adapted to `CloudItem` instead of local `Document`. +- `MinIOBackend`: existing private UUID object-key pattern is the right storage precedent for retained cloud byte cache objects. + +### Established Patterns +- Views own stores and route params; smart components own interaction/layout and emit upward. +- Cloud provider IDs remain opaque and navigation/mutations use connection UUID plus `provider_item_id`; never derive provider paths in Vue. +- Provider-owned bytes are authoritative. Cached bytes are temporary and distinct from permanent local imports. +- Credentials are decrypted only at provider/task boundaries and never travel through broker payloads, API responses, logs, browser storage, or planning artifacts. +- Service modules raise domain errors or `ValueError`; routers translate to `HTTPException`/typed responses. +- Metadata-only browse and refresh never download bytes or mutate quotas; Phase 14 byte cache is the first place retained cloud bytes affect quota. + +### Integration Points +- Add API routes under `backend/api/cloud/` for estimate, enqueue analysis, list job state, cancel/skip/abort/retry controls, and cache settings. +- Add services for cloud analysis orchestration, scan quota checks, byte-cache metadata, LRU eviction, active pinning, and idempotency/fingerprint decisions. +- Add Celery tasks for folder/connection expansion, provider byte hydration, text extraction, classification, cache eviction, and status updates. +- Extend `CloudItem` or related tables with durable analysis job/cache/version/fingerprint data while preserving existing owner and connection indexes. +- Extend `StorageBrowser`, `CloudFolderView`, `cloudConnections` store, and Settings to expose analysis actions, progress detail preference, default failure behavior, and tier-bounded cache limit. +- Add backend/frontend tests for ownership, admin-negative access, credential secrecy, cache isolation, quota accounting, scan quota readiness, duplicate jobs, cancellation, failure defaults, and eviction pinning. + + + + +## Specific Ideas + +- The analysis queue should feel similar to the existing upload queue, including pause-and-decide error handling and a global cancel-all action. +- Users can choose simplified versus detailed analysis progress in Settings; simplified is the default. +- Failure defaults are also user-configurable, with an extensible settings model so future behaviors can be added without reshaping the queue engine. +- The project should be ready for tiered scan limits, including free-tier daily scan quotas, even if billing/subscription tiers are not implemented in Phase 14. +- Content hashes are useful, but only when bytes are already being read for analysis; hashing must not become a hidden byte-download trigger. + + + + +## Deferred Ideas + +- Unified keyword/semantic search over local and analyzed cloud documents remains Phase 15. +- Provider delta feeds, external-delete handling, and broader refresh/change reliability remain Phase 16. +- Permanent local import or pinning of provider files remains future `IMPORT-01`; Phase 14 cache is temporary and bounded. +- Specific paid tier definitions and billing workflows remain future scope, though Phase 14 should leave clean scan/cache quota seams. + + + +--- + +*Phase: 14-selective-analysis-and-byte-cache* +*Context gathered: 2026-06-23* diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-DISCUSSION-LOG.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-DISCUSSION-LOG.md new file mode 100644 index 0000000..00f4d7e --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-DISCUSSION-LOG.md @@ -0,0 +1,199 @@ +# Phase 14: Selective Analysis and Byte Cache - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md - this log preserves the alternatives considered. + +**Date:** 2026-06-23 +**Phase:** 14-selective-analysis-and-byte-cache +**Areas discussed:** Analysis Scope Selection, Job Progress and Controls, Byte Cache Semantics, Idempotency and Reanalysis + +--- + +## Analysis Scope Selection + +| Option | Description | Selected | +|--------|-------------|----------| +| Rows + toolbar | Single-file actions live on rows; multi-select/folder/connection analysis lives in the shared browser toolbar. | yes | +| Toolbar only | All analysis starts from selected items or current location, keeping rows quieter. | | +| Rows only | Simple for individual files, but awkward for folder or whole-connection analysis. | | + +**User's choice:** Rows + toolbar. +**Notes:** Single-file analysis belongs on file rows. Multi-select, folder, and whole-connection analysis belong in the shared browser toolbar. + +| Option | Description | Selected | +|--------|-------------|----------| +| Before folder/connection jobs only | Individual files and explicit multi-file selections start directly; folders and whole connections require estimate review. | | +| Before every job | Even one file shows a confirmation with byte count and estimated cost/time. | | +| Only for large scopes | Start small jobs directly, but estimate when item count or bytes cross a threshold. | yes | + +**User's choice:** Only for large scopes. +**Notes:** Threshold-based estimate/review protects users from unexpectedly large operations. + +| Option | Description | Selected | +|--------|-------------|----------| +| Count + bytes + unsupported items | Show total files, provider-reported bytes, skipped/unsupported file types, and a clear start button. | yes | +| Detailed file list | Show every discovered item before starting, with per-file include/exclude. | | +| Cost-forward estimate | Emphasize AI cost/time estimates in addition to count and bytes. | | + +**User's choice:** Count + bytes + unsupported items. +**Notes:** Phase 14 should not become a full file-by-file review UI. + +| Option | Description | Selected | +|--------|-------------|----------| +| Recursive by default | Analyzing a folder includes supported files in all descendants, with threshold estimate protection. | | +| Current folder only | Only files directly inside the folder are analyzed unless the user opens subfolders. | | +| Ask each time | Every folder analysis asks current-only vs recursive. | yes | + +**User's choice:** Ask each time. +**Notes:** Whole-connection analysis remains inherently recursive. + +--- + +## Job Progress and Controls + +| Option | Description | Selected | +|--------|-------------|----------| +| Aggregate + expandable item list | Show overall progress in toolbar/status area, with expandable per-item list for details. | yes | +| Per-item always visible | Every queued/running/failed item is visible in the main browser surface. | | +| Aggregate only | Show one progress indicator and surface only failures. | | + +**User's choice:** Aggregate + expandable item list. +**Notes:** Keep the main browser clean but allow detailed inspection. + +| Option | Description | Selected | +|--------|-------------|----------| +| Roadmap statuses exactly | Show queued, downloading, extracting, classifying, indexed, cancelled, failed. | | +| Simplified statuses | Show waiting, working, done, failed. | yes | +| Verbose technical statuses | Include cache hit/miss, provider fetch, extraction, topic assignment, semantic placeholders. | | + +**User's choice:** User-configurable detail with simplified default. +**Notes:** Settings should let the user choose how much analysis progress detail to see. Simplified labels are the default; advanced detail can expose roadmap-level states. + +| Option | Description | Selected | +|--------|-------------|----------| +| Cancel queued work only | Users can cancel jobs/items that have not started; running stages finish or fail naturally. | | +| Cancel queued + running downloads | Queued work cancels cleanly, and active provider downloads are interrupted where feasible. | | +| Cancel everything immediately | Attempt to interrupt every running stage, including extraction/classification. | | +| Queue controls with download + scan indicators | Similar queue/error model to upload; per-item cancel/skip/abort and global cancel everything. | yes | + +**User's choice:** Queue controls with download and scanning indicators. +**Notes:** The queue should cover both download/cache hydration and scanning/classification. It should offer per-file or queued-item cancel, skip, and abort controls plus a cancel-everything action. + +| Option | Description | Selected | +|--------|-------------|----------| +| Pause queue on failure | Current item failure pauses the queue for retry/skip/abort. | yes | +| Continue queue automatically | Failed item is marked failed and later items continue; user retries failures later. | | +| Stop whole job | First failure aborts the remaining queue. | | + +**User's choice:** Pause queue on failure, with configurable default behavior. +**Notes:** User added that default failure behavior should be configurable. Options include pause whole queue and wait for input, or pause failed item and continue the queue. The settings model should remain extensible for future behaviors. + +--- + +## Byte Cache Semantics + +| Option | Description | Selected | +|--------|-------------|----------| +| Yes, while retained | Cached bytes count toward quota until evicted; metadata/extracted text does not. | yes | +| No, never | Cache is operational infrastructure and does not affect user quota. | | +| Only persistent cache | Active job bytes do not count, retained bytes after completion count. | | + +**User's choice:** Yes, while retained. +**Notes:** User also requested preparation for separate scan quota, such as free-tier users scanning 1 or 5 files per day. + +| Option | Description | Selected | +|--------|-------------|----------| +| Per-user cache limit with LRU eviction | Each user has a cache byte cap; least-recently-used inactive cached bytes evict first. | yes | +| Global cache limit only | One system-wide cap; eviction can affect any user but serving still respects ownership. | | +| Per-connection cache limit | Each cloud connection has its own cap. | | + +**User's choice:** Per-user cache limit with LRU eviction. +**Notes:** Eviction applies only to inactive cached bytes. + +| Option | Description | Selected | +|--------|-------------|----------| +| Tier-bounded user setting | Users choose a cache limit up to their tier maximum. | yes | +| Admin/system only | Users see cache usage but cannot change the limit. | | +| Fixed default for now | Implement one default limit and leave configuration for later. | | + +**User's choice:** Tier-bounded user setting. +**Notes:** Admin/tier config defines the ceiling. + +| Option | Description | Selected | +|--------|-------------|----------| +| Active jobs + open previews | Running analysis/download/preview bytes are pinned; everything else is LRU-evictable. | yes | +| Active jobs only | Previews can be evicted after response streaming begins. | | +| Recent successful analysis too | Keep newly analyzed files for a short grace window even if inactive. | | + +**User's choice:** Active jobs + open previews. +**Notes:** Active use must not be interrupted by eviction. + +| Option | Description | Selected | +|--------|-------------|----------| +| MinIO with cache metadata table | Store cached bytes in MinIO with UUID keys; track owner/connection/item/version/pin/size in PostgreSQL. | yes | +| Filesystem cache on worker/backend disk | Simpler, but weaker for horizontal scaling and cleanup. | | +| Database large objects | Keeps DB transactional, but poor fit for file bytes and quota accounting. | | + +**User's choice:** MinIO with cache metadata table. +**Notes:** Preserve UUID/private key pattern and PostgreSQL metadata for quota/eviction. + +--- + +## Idempotency and Reanalysis + +| Option | Description | Selected | +|--------|-------------|----------| +| Skip as already current | Do not re-download or reclassify; show already indexed/current. | yes | +| Allow manual force reanalysis | Default skip, but offer a force option. | | +| Always reanalyze | User request always downloads and reclassifies even when unchanged. | | + +**User's choice:** Skip as already current. +**Notes:** Unchanged etag/version should prevent duplicate analysis. + +| Option | Description | Selected | +|--------|-------------|----------| +| Use fallback fingerprint | Combine provider item id, size, modified time, and content type; reanalyze when it changes. | yes | +| Always reanalyze missing-etag items | Safest freshness-wise, but expensive. | | +| Never consider them current | Mark as needs manual refresh until provider supports versioning. | | + +**User's choice:** Use fallback fingerprint, plus content hash when bytes are already hydrated. +**Notes:** User suggested storing a hash. Decision: do not download bytes solely for hashing; compute content hash during analysis because bytes are already present. + +| Option | Description | Selected | +|--------|-------------|----------| +| Finish then mark stale if changed | Complete current job safely, compare version/fingerprint, and mark stale if changed. | yes | +| Abort current item immediately | Stop work when a newer provider version is detected. | | +| Ignore until next refresh | Current job writes results; Phase 16 later handles staleness. | | + +**User's choice:** Finish then mark stale if changed. +**Notes:** Analysis remains non-destructive and provider-owned file is never mutated. + +| Option | Description | Selected | +|--------|-------------|----------| +| Retry failed stage only when possible | Reuse cached bytes if valid; otherwise restart from download. | | +| Restart full item every time | Simpler and predictable, but repeats work. | yes | +| Retry whole batch | Retry all failed/skipped items together. | | + +**User's choice:** Restart full item every time. +**Notes:** Predictability is preferred over partial-stage optimization. + +| Option | Description | Selected | +|--------|-------------|----------| +| Process only actionable items | Skip current, queue stale and failed-prior, report unsupported. | yes | +| Ask before starting | Show categories and require user to choose which groups to include. | | +| Queue everything | Current items still get queued and skipped individually. | | + +**User's choice:** Process only actionable items. +**Notes:** Unsupported/current items should be reported without blocking stale or failed-prior items. + +--- + +## Codex's Discretion + +- Choose exact labels, icons, thresholds, endpoint shapes, schema names, retry mechanics, and default values consistent with the decisions captured in `14-CONTEXT.md`. + +## Deferred Ideas + +- Unified smart search remains Phase 15. +- Provider delta/change tracking reliability remains Phase 16. +- Permanent import/pinning and paid tier/billing implementation remain future scope.