From 126ac7cf6356b92b9d4aff433a323e57a435b5d8 Mon Sep 17 00:00:00 2001 From: curo1305 Date: Thu, 21 May 2026 21:31:06 +0200 Subject: [PATCH] docs(01): capture phase context --- .../01-CONTEXT.md | 107 ++++++++++++ .../01-DISCUSSION-LOG.md | 153 ++++++++++++++++++ 2 files changed, 260 insertions(+) create mode 100644 .planning/phases/01-infrastructure-foundation/01-CONTEXT.md create mode 100644 .planning/phases/01-infrastructure-foundation/01-DISCUSSION-LOG.md diff --git a/.planning/phases/01-infrastructure-foundation/01-CONTEXT.md b/.planning/phases/01-infrastructure-foundation/01-CONTEXT.md new file mode 100644 index 0000000..c13b302 --- /dev/null +++ b/.planning/phases/01-infrastructure-foundation/01-CONTEXT.md @@ -0,0 +1,107 @@ +# Phase 1: Infrastructure Foundation - Context + +**Gathered:** 2026-05-21 +**Status:** Ready for planning + + +## Phase Boundary + +Wire PostgreSQL, MinIO, Redis, and Celery into Docker Compose; create a complete Alembic-managed schema; rewrite the document storage service layer to use PostgreSQL + MinIO; delete legacy flat-file data. The existing document upload and AI classification workflow must continue to work end-to-end with the new infrastructure. No user-facing behavior change, but the internal storage layer is fully replaced. + + + + +## Implementation Decisions + +### Schema Scope +- **D-01:** Phase 1 initial Alembic migration creates the **full v1 skeleton** — all tables: `users`, `refresh_tokens`, `quotas`, `documents`, `topics`, `folders`, `shares`, `audit_log`, `cloud_connections`. Subsequent phases add data and constraints, not new tables. +- **D-02:** `groups` table stub included in Phase 1 migration (v2 feature, but PROJECT.md explicitly notes "groups table will be seeded in schema"). Empty table, correct columns and FKs. +- **D-03:** `documents.user_id` is **nullable** in Phase 1 (no auth system yet). Phase 2 migration adds the `NOT NULL` constraint after the user/auth system is live. +- **D-04:** Existing `data/` directory contents (flat-file JSON metadata + uploaded files) are **deleted** in Phase 1. They are test data only — no migration script needed. Phase 3's migration scope is removed. + +### App Wiring +- **D-05:** Phase 1 **switches the storage service layer** to PostgreSQL + MinIO. `backend/services/storage.py` is rewritten to use async SQLAlchemy + MinIO SDK. The app does not continue using the filesystem after Phase 1. +- **D-06:** Single MinIO bucket named **`docuvault`**. Object keys follow `{user_id}/{document_id}/{uuid4()}{ext}` (STORE-02). Human-readable filenames stored in the `documents.filename` DB column only — never in the MinIO key. +- **D-07:** `backend/main.py` `/health` endpoint extended to check PostgreSQL + MinIO connectivity (not just `{"status": "ok"}`). Health checks gate `docker compose up` readiness. + +### Background Worker +- **D-08:** Background task queue: **Celery + Redis** (STORE-08). FastAPI `BackgroundTasks` replaced. +- **D-09:** Redis service added to `docker-compose.yml` in **Phase 1** (alongside PostgreSQL and MinIO). Redis doubles as the rate-limiting store for Phase 2 auth endpoints — no second Redis needed later. +- **D-10:** A `celery-worker` service is added to `docker-compose.yml`. Celery broker and result backend both point to the same Redis instance via `REDIS_URL`. + +### Env / Secrets Strategy +- **D-11:** `.env` gitignored + `.env.example` committed. `docker-compose.yml` reads vars via `${VAR_NAME}`. `.env.example` has safe placeholder values and comments explaining each variable. +- **D-12:** Production secrets stored **outside the project directory** at `/etc/docuvault/env` (`chmod 600`, owned by the service user, not root). `docker-compose.yml` references it via `env_file:`. Documented in deployment notes. +- **D-13:** **Two PostgreSQL DSNs** introduced: + - `DATABASE_URL` — restricted app user `docuvault_app` (SELECT / INSERT / UPDATE / DELETE only; no DDL) + - `DATABASE_MIGRATE_URL` — migration user `docuvault_migrate` (DDL privileges; used only by Alembic) +- **D-14:** PostgreSQL init script in `docker/postgres/initdb.d/` provisions both users on first container start. The app never connects as the PostgreSQL superuser. +- **D-15:** MinIO vars: `MINIO_ENDPOINT`, `MINIO_ROOT_USER`, `MINIO_ROOT_PASSWORD` (init only), `MINIO_BUCKET` (value: `docuvault`), `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY` (separate app-level access key pair with minimal bucket permissions). +- **D-16:** Additional vars in Phase 1 `.env.example`: `REDIS_URL`, `SECRET_KEY` (documented now for Phase 2 JWT + HKDF use; app does not read it in Phase 1). + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Requirements +- `.planning/REQUIREMENTS.md` — STORE-01 (storage migration), STORE-02 (MinIO key schema), STORE-07 (stateless backend), STORE-08 (BackgroundTasks replacement) +- `.planning/ROADMAP.md` — Phase 1 goal, success criteria (especially criterion #4: MinIO key schema enforced in model layer) + +### Project Decisions +- `.planning/PROJECT.md` — Key Decisions table (PostgreSQL + MinIO rationale, HKDF key derivation, atomic quota UPDATE pattern, groups table seeding) +- `.planning/STATE.md` — Open Questions (Celery+Redis now resolved as Celery+Redis; PyOTP valid_window note for Phase 2) + + + + +## Existing Code Insights + +### Reusable Assets +- `backend/ai/base.py` + `backend/ai/__init__.py` — ABC + factory pattern; the `storage/` module for MinIO/cloud backends should mirror this exactly (StorageBackend ABC + get_storage_backend() factory) +- `backend/config.py` — Pydantic Settings class; extend with new env vars (DATABASE_URL, DATABASE_MIGRATE_URL, MINIO_*, REDIS_URL, SECRET_KEY) +- `backend/main.py` `/health` — extend to probe PostgreSQL + MinIO rather than replace + +### Established Patterns +- **Provider pattern** (ABC + factory in `ai/`) — storage module mirrors this +- **Service layer** (`services/extractor.py`, `services/classifier.py`) — pure Python modules, no FastAPI coupling; new storage service follows the same boundary +- **Pinia-as-facade** (frontend) — no changes needed in Phase 1; API contract preserved + +### Integration Points +- `docker-compose.yml` — add `postgres`, `minio`, `redis`, `celery-worker` services; add health checks and `depends_on` ordering +- `backend/requirements.txt` — add: `sqlalchemy[asyncio]>=2.0`, `psycopg[binary,pool]>=3`, `alembic`, `minio`, `celery[redis]` +- `backend/services/storage.py` — **replace entirely** with async SQLAlchemy + MinIO SDK implementation +- `backend/api/documents.py` — update to use new async storage service; interface should stay stable so frontend is unaffected +- `backend/main.py` — add SQLAlchemy async engine lifespan (startup/shutdown); extend `/health` + +### Constraints +- All CORS origins are currently `["*"]` — leave as-is for Phase 1; Phase 2 locks this down with auth +- No linter/formatter config in repo — don't introduce one in Phase 1 (out of scope) +- `filelock` dependency can be removed once `services/storage.py` is replaced + + + + +## Specific Ideas + +- MinIO bucket name: `docuvault` (single bucket, prefix-based isolation) +- PostgreSQL users: `docuvault_app` (runtime, restricted) + `docuvault_migrate` (Alembic migrations, DDL) +- Init script path: `docker/postgres/initdb.d/01-init-users.sql` +- Production env file location: `/etc/docuvault/env` (outside project dir, `chmod 600`) +- Redis URL format: `redis://:${REDIS_PASSWORD}@redis:6379/0` (password-protected even in dev) + + + + +## Deferred Ideas + +None — discussion stayed within phase scope. + + + +--- + +*Phase: 1-Infrastructure Foundation* +*Context gathered: 2026-05-21* diff --git a/.planning/phases/01-infrastructure-foundation/01-DISCUSSION-LOG.md b/.planning/phases/01-infrastructure-foundation/01-DISCUSSION-LOG.md new file mode 100644 index 0000000..9d6f7e7 --- /dev/null +++ b/.planning/phases/01-infrastructure-foundation/01-DISCUSSION-LOG.md @@ -0,0 +1,153 @@ +# Phase 1: Infrastructure Foundation - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-05-21 +**Phase:** 1-Infrastructure Foundation +**Areas discussed:** Schema scope, App wiring, Background worker, Env / secrets strategy + +--- + +## Schema Scope + +| Option | Description | Selected | +|--------|-------------|----------| +| Full v1 skeleton | All tables upfront: users, refresh_tokens, quotas, documents, topics, folders, shares, audit_log, cloud_connections | ✓ | +| Phase 1 minimum | Only documents + topics tables; users and auth added in Phase 2 | | +| You decide | Leave scope to planner | | + +**User's choice:** Full v1 skeleton +**Notes:** — + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Yes, seed the groups table | Empty table, correct columns and FKs; avoids schema change when v2 Group admin lands | ✓ | +| No, v1 tables only | Keep Phase 1 strictly to v1 requirements | | + +**User's choice:** Yes, seed the groups table +**Notes:** PROJECT.md explicitly notes groups table will be seeded in schema. + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Leave existing data on filesystem; schema empty at Phase 1 end | Phase 3 runs migration script | | +| Seed a system / placeholder user | Sentinel user_id for Phase 1 uploads | | + +**User's choice:** (Free text) "The current documents and files are just test data and can be deleted. Please switch uploads to PostgreSQL and MinIO." +**Notes:** Existing flat-file data is expendable test data. This changed the Phase 1 scope significantly: the storage service layer is rewritten in Phase 1, not Phase 3. Phase 3's data migration scope is removed. + +--- + +## App Wiring + +| Option | Description | Selected | +|--------|-------------|----------| +| Filesystem stays active; new infra sits ready | Phase 1 only wires infra; Phase 3 switches service layer | | +| Phase 1 switches uploads to PostgreSQL + MinIO | services/storage.py rewritten in Phase 1 | ✓ | + +**User's choice:** Phase 1 switches uploads to PostgreSQL + MinIO +**Notes:** User clarified existing data is test data and can be deleted. + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Seed a default/admin user in Phase 1 migration | All Phase 1 uploads attributed to placeholder user | | +| Make user_id nullable for Phase 1, enforce NOT NULL in Phase 2 | Avoids phantom user; Phase 2 adds constraint | ✓ | + +**User's choice:** user_id nullable in Phase 1 +**Notes:** — + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Leave data/ directory in place | Stop writing to it; clean up manually or in Phase 3 | | +| Delete data/ contents in Phase 1 | Clean break; no ambiguity about active storage | ✓ | + +**User's choice:** Delete data/ contents in Phase 1 +**Notes:** Old data is test data only. + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Single bucket 'docuvault' with user prefix paths | Standard S3 pattern; matches STORE-02 key schema | ✓ | +| Per-user buckets | Stronger isolation but bucket count issues and operational complexity | | + +**User's choice:** Single bucket 'docuvault' +**Notes:** — + +--- + +## Background Worker + +| Option | Description | Selected | +|--------|-------------|----------| +| pgqueuer | Postgres-native; no Redis; simpler Compose | | +| Celery + Redis | Battle-tested; adds Redis service | ✓ | + +**User's choice:** Celery + Redis +**Notes:** — + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Phase 1 — add it now with the other infra | All services wired together in one shot | ✓ | +| Phase 3 — add when horizontal scaling needed | STORE-08 is a Phase 3 requirement | | + +**User's choice:** Phase 1 — add Redis + Celery now +**Notes:** Redis also serves as the rate-limiting store for Phase 2 auth endpoints. + +--- + +## Env / Secrets Strategy + +**User asked for best practice guidance** before answering — wanted to understand production VPS implications (root server / VPS deployment, database admin password handling). Claude explained: +- App should never use PostgreSQL superuser; restricted users per principle of least privilege +- Secrets stored outside project directory at `/etc/docuvault/env`; `chmod 600`, service user owned +- Two DB DSNs: runtime (restricted) + migration (DDL privileges) + +| Option | Description | Selected | +|--------|-------------|----------| +| .env gitignored + .env.example committed | Docker Compose native; self-hosters copy .env.example → .env | ✓ | +| Hardcoded dev defaults in docker-compose.yml | Simpler but secrets committed to git | | +| docker-compose.override.yml for secrets | Flexible but extra indirection layer | | + +**User's choice:** .env gitignored + .env.example committed; production at `/etc/docuvault/env` + +--- + +| Env var group | Selected | +|---|---| +| DATABASE_URL + DATABASE_MIGRATE_URL | ✓ | +| MINIO_ENDPOINT / MINIO_ROOT_USER / MINIO_ROOT_PASSWORD / MINIO_BUCKET | ✓ | +| REDIS_URL | ✓ | +| SECRET_KEY | ✓ | + +**Notes:** All four groups introduced in Phase 1. SECRET_KEY documented now for Phase 2+ use (JWT + HKDF) even though Phase 1 doesn't read it. + +--- + +| Option | Description | Selected | +|--------|-------------|----------| +| Yes — provision restricted users in Phase 1 | init script creates docuvault_app + docuvault_migrate | ✓ | +| No — use superuser for now, split in Phase 2 | Simpler but not least-privilege | | + +**User's choice:** Yes — provision restricted users in Phase 1 +**Notes:** — + +--- + +## Claude's Discretion + +None — user made explicit choices for all areas. + +## Deferred Ideas + +None — discussion stayed within phase scope.