docs(14-02): complete Phase 14 Plan 02 — schema and service helpers
- 14-02-SUMMARY.md: documents migration 0007, ORM models, version-key precedence helper, settings service, and Phase 14 cache functions - STATE.md: advance to Plan 3 of 9, record P02 metrics and decisions
This commit is contained in:
curo1305
+20
-15
@@ -4,31 +4,31 @@ milestone: v0.3
|
||||
milestone_name: Reimagining Cloud Storage integration
|
||||
current_phase: 14
|
||||
current_phase_name: selective-analysis-and-byte-cache
|
||||
status: planned
|
||||
stopped_at: Phase 14 planned
|
||||
last_updated: "2026-06-23T13:00:09.305Z"
|
||||
status: in_progress
|
||||
stopped_at: Phase 14 Plan 02 complete
|
||||
last_updated: "2026-06-23T14:16:33.000Z"
|
||||
last_activity: 2026-06-23
|
||||
last_activity_desc: Phase 14 plan files created
|
||||
last_activity_desc: Phase 14 Plan 02 — schema and service helpers complete
|
||||
progress:
|
||||
total_phases: 6
|
||||
completed_phases: 3
|
||||
total_plans: 30
|
||||
completed_plans: 22
|
||||
percent: 50
|
||||
completed_plans: 24
|
||||
percent: 53
|
||||
---
|
||||
|
||||
# Project State
|
||||
|
||||
**Project:** DocuVault
|
||||
**Status:** Phase 14 planned — ready to execute Plan 14-01
|
||||
**Status:** Phase 14 in progress — Plan 14-02 complete
|
||||
**Last Updated:** 2026-06-23
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 14 (selective-analysis-and-byte-cache) — PLANNED
|
||||
Plan: 1 of 9
|
||||
Status: Phase 14 planned. Next: execute 14-01 red contract tests
|
||||
Last activity: 2026-06-23 — Phase 14 plan files created
|
||||
Phase: 14 (selective-analysis-and-byte-cache) — IN PROGRESS
|
||||
Plan: 3 of 9
|
||||
Status: Plans 14-01 and 14-02 complete. Next: execute 14-03 (analysis API)
|
||||
Last activity: 2026-06-23 — Phase 14 Plan 02 schema + service helpers complete
|
||||
|
||||
## Phase Status
|
||||
|
||||
@@ -63,6 +63,7 @@ Last activity: 2026-06-23 — Phase 14 plan files created
|
||||
| Phase 13 P10 | 25m | 2 tasks | 6 files |
|
||||
| Phase 13 P11 | 20m | 2 tasks | 6 files |
|
||||
| Phase 14 P01 | 15m | 2 tasks | 5 files |
|
||||
| Phase 14 P02 | 12m | 2 tasks | 5 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -104,16 +105,16 @@ None.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
**Stopped at:** Phase 14 planned
|
||||
**Stopped at:** Phase 14 Plan 02 complete
|
||||
|
||||
_Updated at each phase transition._
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Last session | 2026-06-23T13:00:09.300Z |
|
||||
| Next action | Execute 14-01 red contract tests |
|
||||
| Last session | 2026-06-23T14:16:33.000Z |
|
||||
| Next action | Execute 14-03 (analysis API router) |
|
||||
| Pending decisions | None |
|
||||
| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-01-PLAN.md |
|
||||
| Resume file | .planning/phases/14-selective-analysis-and-byte-cache/14-03-PLAN.md |
|
||||
|
||||
## Decisions
|
||||
|
||||
@@ -142,3 +143,7 @@ _Updated at each phase transition._
|
||||
- [Phase 13 P09]: Stale move guard stops mutation, refreshes source folder state, returns typed stale result
|
||||
- [Phase 13 P09]: Delete audit metadata: only kind/provider_item_id/display_name/is_folder — no bytes or credentials
|
||||
- [Phase 13 P11]: Version bump to v0.3.0 on Phase 13 completion (minor bump for full phase per CLAUDE.md protocol)
|
||||
- [Phase 14 P02]: compute_version_key defined in cloud_analysis_versioning.py and re-exported from cloud_cache.py — single import site for tests and callers
|
||||
- [Phase 14 P02]: content_hash is optional post-hydration supplement, never a pre-download precondition (D-20)
|
||||
- [Phase 14 P02]: evict_lru_entries stamps evicted_at only; caller owns MinIO deletion and quota decrement (separation of concerns)
|
||||
- [Phase 14 P02]: user_analysis_settings is distinct from system_settings — single authority for per-user analysis preferences
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
phase: "14"
|
||||
plan: "02"
|
||||
subsystem: cloud-analysis
|
||||
tags: [schema, migration, services, cache, versioning, settings]
|
||||
status: complete
|
||||
|
||||
requires:
|
||||
- phases/14-selective-analysis-and-byte-cache/14-01
|
||||
|
||||
provides:
|
||||
- Alembic migration 0007: cloud_byte_cache_entries, cloud_analysis_jobs,
|
||||
cloud_analysis_job_items, user_analysis_settings
|
||||
- ORM models: CloudByteCacheEntry, CloudAnalysisJob, CloudAnalysisJobItem,
|
||||
UserAnalysisSettings
|
||||
- services/cloud_analysis_versioning.py: compute_version_key, compute_fingerprint
|
||||
- services/cloud_analysis_settings.py: get_or_create_user_analysis_settings,
|
||||
update_user_analysis_settings, build_default_settings
|
||||
- services/cloud_cache.py: Phase 14 durable cache functions (create_cache_entry,
|
||||
list_cache_entries, evict_lru_entries, increment_quota_for_cache,
|
||||
decrement_quota_for_cache) + re-export of compute_version_key
|
||||
|
||||
affects:
|
||||
- backend/migrations/versions/0007_cloud_analysis_cache.py
|
||||
- backend/db/models.py
|
||||
- backend/services/cloud_analysis_versioning.py
|
||||
- backend/services/cloud_analysis_settings.py
|
||||
- backend/services/cloud_cache.py
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- version-key precedence (version > etag > metadata fingerprint > content hash)
|
||||
- LRU eviction with pin_count/active_job_count guards
|
||||
- atomic quota UPDATE pattern for cache retain/release
|
||||
- upsert-on-first-access for user settings row
|
||||
- re-export pattern to unify import sites for compute_version_key
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- backend/migrations/versions/0007_cloud_analysis_cache.py
|
||||
- backend/services/cloud_analysis_versioning.py
|
||||
- backend/services/cloud_analysis_settings.py
|
||||
modified:
|
||||
- backend/db/models.py
|
||||
- backend/services/cloud_cache.py
|
||||
|
||||
decisions:
|
||||
- compute_version_key defined in cloud_analysis_versioning.py and re-exported
|
||||
from cloud_cache.py so tests have one canonical import site
|
||||
- content_hash accepted as optional post-hydration supplement — never required
|
||||
as a pre-download precondition (D-20)
|
||||
- user_analysis_settings is distinct from system_settings — single settings
|
||||
authority for per-user analysis preferences
|
||||
- evict_lru_entries stamps evicted_at only — caller is responsible for MinIO
|
||||
object deletion and quota decrement (separation of concerns)
|
||||
- increment_quota_for_cache uses atomic UPDATE…RETURNING pattern (same as
|
||||
upload quota enforcement) — raises CacheQuotaExceeded on overflow
|
||||
|
||||
metrics:
|
||||
duration: "~12 minutes"
|
||||
completed: "2026-06-23"
|
||||
tasks: 2
|
||||
files: 5
|
||||
---
|
||||
|
||||
# Phase 14 Plan 02: Schema and Service Helpers Summary
|
||||
|
||||
One-liner: Alembic migration 0007 adds four Phase 14 tables (byte cache, analysis jobs, job items, user settings) with ORM models, version-key precedence helper, tier-seamed settings service, and LRU eviction cache functions.
|
||||
|
||||
## What Was Built
|
||||
|
||||
### Task 1: Migration and ORM Models
|
||||
|
||||
**migration 0007_cloud_analysis_cache.py:**
|
||||
- `cloud_byte_cache_entries`: owner-scoped durable byte cache backed by MinIO.
|
||||
Unique on (user_id, connection_id, cloud_item_id, version_key). Fields include
|
||||
pin_count, active_job_count, last_accessed_at, evicted_at. Two LRU eviction
|
||||
indexes (ix_cache_entries_user_evicted_accessed, ix_cache_entries_user_active).
|
||||
- `cloud_analysis_jobs`: user-visible batch rows with scope_kind (file/selection/
|
||||
folder/connection), 11 aggregate item counters, provider_bytes_estimate, status,
|
||||
and failure_behavior.
|
||||
- `cloud_analysis_job_items`: per-item idempotency rows with the full status
|
||||
vocabulary (queued/downloading/extracting/classifying/indexed/already_current/
|
||||
cancelled/failed/unsupported/stale), controlled error fields, cache_entry_id FK,
|
||||
fingerprint snapshot, and retry_count. Unique on (job_id, cloud_item_id).
|
||||
Fast duplicate-current index: (user_id, cloud_item_id, version_key).
|
||||
- `user_analysis_settings`: per-user preferences (analysis_progress_detail,
|
||||
analysis_failure_behavior, cloud_cache_limit_bytes). Separate from system_settings.
|
||||
- Reversible downgrade drops Phase 14 tables only in reverse-creation order.
|
||||
|
||||
**db/models.py additions:**
|
||||
- CloudByteCacheEntry, CloudAnalysisJob, CloudAnalysisJobItem, UserAnalysisSettings
|
||||
ORM models mirror the migration exactly.
|
||||
|
||||
### Task 2: Service Helpers
|
||||
|
||||
**services/cloud_analysis_versioning.py** (new):
|
||||
- `compute_version_key(provider_item_id, version, etag, size, modified_at, content_type, content_hash=None)`: canonical version-key helper with v:/e:/fp:/ch: prefix tags.
|
||||
Precedence: provider version → etag → SHA-256 metadata fingerprint → optional
|
||||
post-hydration content hash suffix. Deterministic and raises ValueError only.
|
||||
- `compute_fingerprint(...)`: public entry point for metadata-only fingerprint.
|
||||
- No HTTPException import or raise (CLAUDE.md service-layer rule).
|
||||
|
||||
**services/cloud_analysis_settings.py** (new):
|
||||
- `get_or_create_user_analysis_settings(session, user_id)`: upsert on first access.
|
||||
- `update_user_analysis_settings(session, user_id, ...)`: validates enum fields
|
||||
(VALID_PROGRESS_DETAIL, VALID_FAILURE_BEHAVIOR) and bounds-checks cache limit
|
||||
against MIN_CACHE_LIMIT_BYTES and caller-supplied max_cache_limit_bytes.
|
||||
- `build_default_settings()`: returns defaults dict without touching the DB.
|
||||
- Tier/admin maximum is an explicit parameter seam (not hardcoded in the service).
|
||||
- No HTTPException import or raise.
|
||||
|
||||
**services/cloud_cache.py** (extended):
|
||||
- Phase 14 functions added after the existing Phase 12 in-memory TTL cache code:
|
||||
`create_cache_entry`, `list_cache_entries` (owner-scoped, excludes evicted),
|
||||
`evict_lru_entries` (LRU, skips pin_count>0 or active_job_count>0),
|
||||
`increment_quota_for_cache` (atomic UPDATE…RETURNING, raises CacheQuotaExceeded),
|
||||
`decrement_quota_for_cache` (GREATEST(0, used_bytes - delta)).
|
||||
- Re-exports `compute_version_key` from cloud_analysis_versioning.py so test/caller
|
||||
import sites are unified.
|
||||
- Phase 12 get_cloud_folders_cached / invalidate_provider_cache unchanged.
|
||||
|
||||
## Test Results
|
||||
|
||||
| File | Pass | Fail | Notes |
|
||||
|------|------|------|-------|
|
||||
| test_cloud_cache.py | 10 | 2 | 2 failures require api.cloud.analysis (future plan) |
|
||||
| test_cloud_analysis_contract.py | 0 | 16 | All require api.cloud.analysis (future plan) |
|
||||
| Overall suite (excl. Phase 14 RED) | 610+ | 1 pre-existing | test_extractor.py docx env issue — pre-existing |
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None — plan executed exactly as written.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None.
|
||||
|
||||
## Threat Flags
|
||||
|
||||
None — this plan adds schema, migration, and service layer only. No new network
|
||||
endpoints, auth paths, or file access patterns.
|
||||
|
||||
## Self-Check: PASSED
|
||||
Reference in New Issue
Block a user