diff --git a/.planning/STATE.md b/.planning/STATE.md index 457c90a..8b438cb 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,31 +4,31 @@ milestone: v0.3 milestone_name: Reimagining Cloud Storage integration current_phase: 14 current_phase_name: selective-analysis-and-byte-cache -status: planned -stopped_at: Phase 14 planned -last_updated: "2026-06-23T13:00:09.305Z" +status: in_progress +stopped_at: Phase 14 Plan 02 complete +last_updated: "2026-06-23T14:16:33.000Z" last_activity: 2026-06-23 -last_activity_desc: Phase 14 plan files created +last_activity_desc: Phase 14 Plan 02 — schema and service helpers complete progress: total_phases: 6 completed_phases: 3 total_plans: 30 - completed_plans: 22 - percent: 50 + completed_plans: 24 + percent: 53 --- # Project State **Project:** DocuVault -**Status:** Phase 14 planned — ready to execute Plan 14-01 +**Status:** Phase 14 in progress — Plan 14-02 complete **Last Updated:** 2026-06-23 ## Current Position -Phase: 14 (selective-analysis-and-byte-cache) — PLANNED -Plan: 1 of 9 -Status: Phase 14 planned. Next: execute 14-01 red contract tests -Last activity: 2026-06-23 — Phase 14 plan files created +Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS +Plan: 3 of 9 +Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API) +Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete ## Phase Status @@ -63,6 +63,7 @@ Last activity: 2026-06-23 — Phase 14 plan files created | Phase 13 P10 | 25m | 2 tasks | 6 files | | Phase 13 P11 | 20m | 2 tasks | 6 files | | Phase 14 P01 | 15m | 2 tasks | 5 files | +| Phase 14 P02 | 12m | 2 tasks | 5 files | ## Accumulated Context @@ -104,16 +105,16 @@ None. ## Session Continuity -**Stopped at:** Phase 14 planned +**Stopped at:** Phase 14 Plan 02 complete _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-23T13:00:09.300Z | -| Next action | Execute 14-01 red contract tests | +| Last session | 2026-06-23T14:16:33.000Z | +| Next action | Execute 14-03 (analysis API router) | | Pending decisions | None | -| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md | +| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md | ## Decisions @@ -142,3 +143,7 @@ _Updated at each phase transition._ - [Phase 13 P09]: Stale move guard stops mutation, refreshes source folder state, returns typed stale result - [Phase 13 P09]: Delete audit metadata: only kind/provider_item_id/display_name/is_folder — no bytes or credentials - [Phase 13 P11]: Version bump to v0.3.0 on Phase 13 completion (minor bump for full phase per CLAUDE.md protocol) +- [Phase 14 P02]: compute_version_key defined in cloud_analysis_versioning.py and re-exported from cloud_cache.py — single import site for tests and callers +- [Phase 14 P02]: content_hash is optional post-hydration supplement, never a pre-download precondition (D-20) +- [Phase 14 P02]: evict_lru_entries stamps evicted_at only; caller owns MinIO deletion and quota decrement (separation of concerns) +- [Phase 14 P02]: user_analysis_settings is distinct from system_settings — single authority for per-user analysis preferences diff --git a/.planning/phases/14-selective-analysis-and-byte-cache/14-02-SUMMARY.md b/.planning/phases/14-selective-analysis-and-byte-cache/14-02-SUMMARY.md new file mode 100644 index 0000000..727c1c7 --- /dev/null +++ b/.planning/phases/14-selective-analysis-and-byte-cache/14-02-SUMMARY.md @@ -0,0 +1,145 @@ +--- +phase: "14" +plan: "02" +subsystem: cloud-analysis +tags: [schema, migration, services, cache, versioning, settings] +status: complete + +requires: + - phases/14-selective-analysis-and-byte-cache/14-01 + +provides: + - Alembic migration 0007: cloud_byte_cache_entries, cloud_analysis_jobs, + cloud_analysis_job_items, user_analysis_settings + - ORM models: CloudByteCacheEntry, CloudAnalysisJob, CloudAnalysisJobItem, + UserAnalysisSettings + - services/cloud_analysis_versioning.py: compute_version_key, compute_fingerprint + - services/cloud_analysis_settings.py: get_or_create_user_analysis_settings, + update_user_analysis_settings, build_default_settings + - services/cloud_cache.py: Phase 14 durable cache functions (create_cache_entry, + list_cache_entries, evict_lru_entries, increment_quota_for_cache, + decrement_quota_for_cache) + re-export of compute_version_key + +affects: + - backend/migrations/versions/0007_cloud_analysis_cache.py + - backend/db/models.py + - backend/services/cloud_analysis_versioning.py + - backend/services/cloud_analysis_settings.py + - backend/services/cloud_cache.py + +tech-stack: + added: [] + patterns: + - version-key precedence (version > etag > metadata fingerprint > content hash) + - LRU eviction with pin_count/active_job_count guards + - atomic quota UPDATE pattern for cache retain/release + - upsert-on-first-access for user settings row + - re-export pattern to unify import sites for compute_version_key + +key-files: + created: + - backend/migrations/versions/0007_cloud_analysis_cache.py + - backend/services/cloud_analysis_versioning.py + - backend/services/cloud_analysis_settings.py + modified: + - backend/db/models.py + - backend/services/cloud_cache.py + +decisions: + - compute_version_key defined in cloud_analysis_versioning.py and re-exported + from cloud_cache.py so tests have one canonical import site + - content_hash accepted as optional post-hydration supplement — never required + as a pre-download precondition (D-20) + - user_analysis_settings is distinct from system_settings — single settings + authority for per-user analysis preferences + - evict_lru_entries stamps evicted_at only — caller is responsible for MinIO + object deletion and quota decrement (separation of concerns) + - increment_quota_for_cache uses atomic UPDATE…RETURNING pattern (same as + upload quota enforcement) — raises CacheQuotaExceeded on overflow + +metrics: + duration: "~12 minutes" + completed: "2026-06-23" + tasks: 2 + files: 5 +--- + +# Phase 14 Plan 02: Schema and Service Helpers Summary + +One-liner: Alembic migration 0007 adds four Phase 14 tables (byte cache, analysis jobs, job items, user settings) with ORM models, version-key precedence helper, tier-seamed settings service, and LRU eviction cache functions. + +## What Was Built + +### Task 1: Migration and ORM Models + +**migration 0007_cloud_analysis_cache.py:** +- `cloud_byte_cache_entries`: owner-scoped durable byte cache backed by MinIO. + Unique on (user_id, connection_id, cloud_item_id, version_key). Fields include + pin_count, active_job_count, last_accessed_at, evicted_at. Two LRU eviction + indexes (ix_cache_entries_user_evicted_accessed, ix_cache_entries_user_active). +- `cloud_analysis_jobs`: user-visible batch rows with scope_kind (file/selection/ + folder/connection), 11 aggregate item counters, provider_bytes_estimate, status, + and failure_behavior. +- `cloud_analysis_job_items`: per-item idempotency rows with the full status + vocabulary (queued/downloading/extracting/classifying/indexed/already_current/ + cancelled/failed/unsupported/stale), controlled error fields, cache_entry_id FK, + fingerprint snapshot, and retry_count. Unique on (job_id, cloud_item_id). + Fast duplicate-current index: (user_id, cloud_item_id, version_key). +- `user_analysis_settings`: per-user preferences (analysis_progress_detail, + analysis_failure_behavior, cloud_cache_limit_bytes). Separate from system_settings. +- Reversible downgrade drops Phase 14 tables only in reverse-creation order. + +**db/models.py additions:** +- CloudByteCacheEntry, CloudAnalysisJob, CloudAnalysisJobItem, UserAnalysisSettings + ORM models mirror the migration exactly. + +### Task 2: Service Helpers + +**services/cloud_analysis_versioning.py** (new): +- `compute_version_key(provider_item_id, version, etag, size, modified_at, content_type, content_hash=None)`: canonical version-key helper with v:/e:/fp:/ch: prefix tags. + Precedence: provider version → etag → SHA-256 metadata fingerprint → optional + post-hydration content hash suffix. Deterministic and raises ValueError only. +- `compute_fingerprint(...)`: public entry point for metadata-only fingerprint. +- No HTTPException import or raise (CLAUDE.md service-layer rule). + +**services/cloud_analysis_settings.py** (new): +- `get_or_create_user_analysis_settings(session, user_id)`: upsert on first access. +- `update_user_analysis_settings(session, user_id, ...)`: validates enum fields + (VALID_PROGRESS_DETAIL, VALID_FAILURE_BEHAVIOR) and bounds-checks cache limit + against MIN_CACHE_LIMIT_BYTES and caller-supplied max_cache_limit_bytes. +- `build_default_settings()`: returns defaults dict without touching the DB. +- Tier/admin maximum is an explicit parameter seam (not hardcoded in the service). +- No HTTPException import or raise. + +**services/cloud_cache.py** (extended): +- Phase 14 functions added after the existing Phase 12 in-memory TTL cache code: + `create_cache_entry`, `list_cache_entries` (owner-scoped, excludes evicted), + `evict_lru_entries` (LRU, skips pin_count>0 or active_job_count>0), + `increment_quota_for_cache` (atomic UPDATE…RETURNING, raises CacheQuotaExceeded), + `decrement_quota_for_cache` (GREATEST(0, used_bytes - delta)). +- Re-exports `compute_version_key` from cloud_analysis_versioning.py so test/caller + import sites are unified. +- Phase 12 get_cloud_folders_cached / invalidate_provider_cache unchanged. + +## Test Results + +| File | Pass | Fail | Notes | +|------|------|------|-------| +| test_cloud_cache.py | 10 | 2 | 2 failures require api.cloud.analysis (future plan) | +| test_cloud_analysis_contract.py | 0 | 16 | All require api.cloud.analysis (future plan) | +| Overall suite (excl. Phase 14 RED) | 610+ | 1 pre-existing | test_extractor.py docx env issue — pre-existing | + +## Deviations from Plan + +None — plan executed exactly as written. + +## Known Stubs + +None. + +## Threat Flags + +None — this plan adds schema, migration, and service layer only. No new network +endpoints, auth paths, or file access patterns. + +## Self-Check: PASSED