diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index b2f68f9..3b9ae66 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -4,7 +4,7 @@ _Last updated: 2026-05-21_ ## Phases -- [ ] **Phase 1: Infrastructure Foundation** — PostgreSQL + MinIO wired into Docker Compose; Alembic migrations running; existing app still works +- [x] **Phase 1: Infrastructure Foundation** — PostgreSQL + MinIO wired into Docker Compose; Alembic migrations running; existing app still works - [ ] **Phase 2: Users & Authentication** — Full auth flow end-to-end (register, login, TOTP, backup codes, password reset, sign-out-all) with admin panel for user management - [ ] **Phase 3: Document Migration & Multi-User Isolation** — All documents in PostgreSQL + MinIO; per-user isolation enforced; existing UI still works - [ ] **Phase 4: Folders, Sharing, Quotas & Document UX** — Full document management UX (folders, sharing, quota bar, PDF preview, search, audit log) @@ -31,7 +31,7 @@ _Last updated: 2026-05-21_ - [x] 01-02-PLAN.md — Wave 0 test scaffolds (xfail/skip stubs) + async pytest fixtures - [x] 01-03-PLAN.md — SQLAlchemy ORM models + async engine + Alembic async migration (incl. alembic upgrade head) - [x] 01-04-PLAN.md — StorageBackend ABC + MinIO backend + rewritten async services/storage.py -- [ ] 01-05-PLAN.md — Lifespan + /health + API cutover + Celery worker + walking-skeleton e2e verify +- [x] 01-05-PLAN.md — Lifespan + /health + API cutover + Celery worker + walking-skeleton e2e verify --- diff --git a/.planning/STATE.md b/.planning/STATE.md index 2e503c6..abf9fc4 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,29 +2,29 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone -current_phase: 1 -status: executing -last_updated: "2026-05-22T07:40:00Z" +current_phase: 2 +status: ready +last_updated: "2026-05-22T12:00:00Z" progress: total_phases: 5 - completed_phases: 0 + completed_phases: 1 total_plans: 5 - completed_plans: 4 - percent: 80 + completed_plans: 5 + percent: 20 --- # Project State **Project:** DocuVault -**Status:** Executing Phase 1 -**Current Phase:** 1 +**Status:** Phase 1 Complete — Ready for Phase 2 +**Current Phase:** 2 **Last Updated:** 2026-05-22 ## Phase Status | Phase | Name | Status | |---|---|---| -| 1 | Infrastructure Foundation | In Progress (4/5 plans) | +| 1 | Infrastructure Foundation | ✓ Complete | | 2 | Users & Authentication | Not Started | | 3 | Document Migration & Multi-User Isolation | Not Started | | 4 | Folders, Sharing, Quotas & Document UX | Not Started | @@ -32,20 +32,18 @@ progress: ## Current Position -Phase: 1 (Infrastructure Foundation) — EXECUTING -Plan: 5 of 5 -**Phase:** 01-infrastructure-foundation -**Plan:** 01-04 COMPLETE → advancing to 01-05 -**Progress:** ████████░░ 80% +**Phase:** 01-infrastructure-foundation — COMPLETE ✓ +**Plan:** 5/5 complete +**Progress:** ██░░░░░░░░ 20% (1/5 phases) ## Performance Metrics | Metric | Value | |---|---| -| Phases complete | 0 / 5 | +| Phases complete | 1 / 5 | | Requirements mapped | 54 / 54 | | Plans written | 5 (Phase 1) | -| Plans complete | 4 | +| Plans complete | 5 | ## Accumulated Context @@ -92,6 +90,6 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-05-22 — Executed 01-04-PLAN.md (StorageBackend ABC + MinIOBackend + async services/storage.py; all 6 test_storage.py xfail tests PASSED) | -| Next action | Execute 01-05-PLAN.md (Lifespan + /health + API cutover + Celery worker + walking-skeleton e2e verify) | +| Last session | 2026-05-22 — Executed Phase 1 (all 5 plans complete); walking-skeleton e2e verified live against Docker stack | +| Next action | Run `/gsd:discuss-phase 2` to begin Phase 2 (Users & Authentication) | | Pending decisions | See Open Questions above | diff --git a/.planning/phases/01-infrastructure-foundation/01-05-SUMMARY.md b/.planning/phases/01-infrastructure-foundation/01-05-SUMMARY.md new file mode 100644 index 0000000..1de69be --- /dev/null +++ b/.planning/phases/01-infrastructure-foundation/01-05-SUMMARY.md @@ -0,0 +1,100 @@ +--- +plan: 01-05 +phase: 01-infrastructure-foundation +status: complete +completed: "2026-05-22" +tasks_total: 4 +tasks_complete: 4 +requirements_satisfied: + - STORE-01 + - STORE-07 +self_check: PASSED +--- + +# Plan 01-05 Summary — Lifespan + /health + API Cutover + Celery + Walking Skeleton + +## What Was Built + +### Task 1 — Celery app + task + session-aware classifier (commit 32d67de) + +**`backend/celery_app.py`** — Minimal Celery instance (`celery_app = Celery("docuvault")`). Reads `REDIS_URL` directly from `os.environ` (no config import — Pitfall 7). JSON serialization, `documents` queue route for `tasks.document_tasks.*`, `autodiscover_tasks(["tasks"])`. + +**`backend/tasks/__init__.py`** — Empty package file. + +**`backend/tasks/document_tasks.py`** — Sync `def extract_and_classify(document_id: str)` Celery task (NOT async). Uses `asyncio.run(_run(document_id))` to drive the async body. Opens a fresh `AsyncSessionLocal` session, fetches the Document ORM row, pulls bytes from MinIO via `MinIOBackend.get_object`, calls `extractor.extract_text_from_bytes`, persists extracted text, then calls `classifier.classify_document(session, doc_id)`. Non-fatal classification failures set `status = "classification_failed"`. + +**`backend/services/extractor.py`** — Added `extract_text_from_bytes(file_bytes, content_type)` helper that writes to a `NamedTemporaryFile` and delegates to the existing `extract_text(path, mime)` function. + +**`backend/services/classifier.py`** — Added `session: AsyncSession` as first parameter to `classify_document` and `suggest_topics_for_document`. All internal `storage.*` calls updated to pass session. + +### Task 2 — Lifespan + /health + async API wiring (commit c1931fd) + +**`backend/main.py`** — Lifespan creates `Minio` client, auto-creates `docuvault` bucket if absent (via `asyncio.to_thread`), attaches to `app.state.minio`, disposes `engine` on shutdown. No longer calls `ensure_data_dirs()`. `/health` endpoint probes PostgreSQL (`SELECT 1` via `AsyncSessionLocal`) and MinIO (`bucket_exists` via `asyncio.to_thread`); returns `{"status": "ok"|"degraded", "checks": {"postgres": ..., "minio": ...}}`. + +**`backend/api/documents.py`** — All 5 route handlers inject `session: AsyncSession = Depends(get_db)`. Upload handler: calls `await storage.save_upload(session, ...)`, uses in-memory `content` bytes for extraction (no filesystem path needed), enqueues `extract_and_classify.delay(saved["id"])` for async classification, returns `topics: []` immediately. `/classify` endpoint retains synchronous `await classifier.classify_document(session, doc_id)` for backward compatibility. + +**`backend/api/topics.py`** — All 5 route handlers inject session dependency; all `storage.*` calls are async with session. + +### Task 3 — Final cutover (commit 970c8e4) + +- `backend/data/` — All tracked files removed via `git rm -rf`; `backend/data/` added to `.gitignore` +- `backend/config.py` — Removed `DATA_DIR`, `UPLOADS_DIR`, `METADATA_DIR`, `TOPICS_FILE`, `ensure_data_dirs()`, `import os`. Retained `DEFAULT_SETTINGS`, `DEFAULT_SYSTEM_PROMPT`, `class Settings(BaseSettings)`, `settings = Settings()`. `SETTINGS_FILE` rebased as `Path(settings.data_dir) / "settings.json"` after `settings = Settings()`. +- `backend/tests/conftest.py` — Removed `isolated_data_dir` fixture and sync `TestClient` `client` fixture. Promoted `db_session` and `async_client` fixtures (removed `try/except ImportError` wrappers — deps now exist). Added `live_services_available` session fixture that probes localhost:5432/9000/6379 via socket. +- `backend/tests/test_documents.py` — Deleted 9 legacy sync tests. Removed all `@pytest.mark.xfail` markers from async ports. +- `backend/tests/test_health.py` — Removed `@pytest.mark.xfail` from `test_health_checks_postgres_and_minio`. Deleted legacy `test_health(client)` sync test. +- `backend/tests/test_settings.py`, `backend/tests/test_topics.py` — Updated to remove any remaining sync client references. + +### Task 4 — Walking-skeleton e2e verification (human-approved ✓) + +All 12 verification steps passed: + +1. `.env` created from `.env.example` +2. `docker compose down -v` — clean state +3. `docker compose up --build -d` — all 5 services booted +4. `docker compose ps` — `postgres`, `minio`, `redis`, `backend`, `celery-worker` all `Up (healthy)` +5. `alembic upgrade head` — exit 0, `Running upgrade -> 0001` +6. `/health` response: + ```json + {"status": "ok", "checks": {"postgres": "ok", "minio": "ok"}} + ``` +7. Upload `test.txt` — returned `{"id": "", "original_name": "test.txt", "topics": [], ...}` +8. PostgreSQL confirmed: one row, `object_key` starts with `null-user/` +9. MinIO confirmed: object present in `docuvault` bucket +10. Celery confirmed: `Task tasks.document_tasks.extract_and_classify[...] succeeded` +11. Delete confirmed: `{"success": true}`, MinIO object removed +12. Integration tests: zero FAILED, zero XFAIL + +## ROADMAP.md Phase 1 Success Criteria — All Met + +| # | Criterion | Status | +|---|-----------|--------| +| 1 | `docker compose up` starts all services healthy | ✓ Verified (Task 4, step 4) | +| 2 | `alembic upgrade head` applies cleanly | ✓ Verified (Plan 03 Task 3 + Task 4 step 5) | +| 3 | Full upload/extract/classify workflow works — no regression | ✓ Verified (Task 4, steps 7-10) | +| 4 | MinIO object key schema `{user_id}/{document_id}/{uuid4()}{ext}` enforced | ✓ Verified (Plan 04 + Task 4 step 8) | + +## Deviations + +| Rule | Deviation | Resolution | +|------|-----------|------------| +| 1 (Bug) | `extract_text_from_bytes` helper did not exist in extractor.py | Added to `services/extractor.py` as specified in Plan 05 Task 1 action block | +| 2 (Enhancement) | `live_services_available` uses env var `INTEGRATION=1` as fallback | Socket probe primary, env var secondary — matches plan intent | + +## Key Files Created/Modified + +| File | Status | Notes | +|------|--------|-------| +| backend/celery_app.py | Created | Minimal Celery — no config import | +| backend/tasks/document_tasks.py | Created | Sync task wrapping asyncio.run | +| backend/tasks/__init__.py | Created | Package marker | +| backend/main.py | Rewritten | Lifespan + /health | +| backend/api/documents.py | Rewritten | Async session injection | +| backend/api/topics.py | Rewritten | Async session injection | +| backend/services/classifier.py | Updated | Session-aware | +| backend/services/extractor.py | Updated | Added bytes helper | +| backend/config.py | Pruned | Flat-file constants removed | +| backend/tests/conftest.py | Pruned | Async-only fixtures | +| backend/tests/test_documents.py | Pruned | Async-only tests | +| backend/data/ | Deleted | D-04 complete | + +## Self-Check: PASSED