docs(01): create phase 1 plan — 5 plans in 4 waves
Research, pattern mapping, and verification complete. Walking Skeleton mode active (MVP Phase 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
@@ -0,0 +1,357 @@
|
||||
---
|
||||
phase: 01-infrastructure-foundation
|
||||
plan: 03
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on:
|
||||
- 01-01
|
||||
- 01-02
|
||||
files_modified:
|
||||
- backend/db/__init__.py
|
||||
- backend/db/models.py
|
||||
- backend/db/session.py
|
||||
- backend/deps/__init__.py
|
||||
- backend/deps/db.py
|
||||
- backend/alembic.ini
|
||||
- backend/migrations/env.py
|
||||
- backend/migrations/script.py.mako
|
||||
- backend/migrations/versions/0001_initial_schema.py
|
||||
autonomous: false
|
||||
requirements:
|
||||
- STORE-01
|
||||
- STORE-07
|
||||
user_setup: []
|
||||
tags:
|
||||
- schema
|
||||
- sqlalchemy
|
||||
- alembic
|
||||
- postgresql
|
||||
- migration
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "`backend/db/models.py` declares all 11 SQLAlchemy ORM model classes for the full v1 schema (D-01, D-02)"
|
||||
- "`documents.user_id` is declared as nullable in the ORM and emitted as nullable in the generated migration (D-03)"
|
||||
- "`backend/db/session.py` exposes `engine` and `AsyncSessionLocal` reading from `config.settings.database_url` with `expire_on_commit=False`"
|
||||
- "`backend/deps/db.py` exposes an async `get_db()` dependency yielding an `AsyncSession`"
|
||||
- "`backend/alembic.ini` references `DATABASE_MIGRATE_URL` via env interpolation (`%(DATABASE_MIGRATE_URL)s`)"
|
||||
- "`backend/migrations/env.py` uses `async_engine_from_config` + `asyncio.run`, imports `db.models.Base` so `target_metadata = Base.metadata` populates"
|
||||
- "`backend/migrations/versions/0001_initial_schema.py` creates all 11 tables and ends with `op.execute(\"ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO docuvault_app;\")` plus an immediate `GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO docuvault_app;`"
|
||||
- "`alembic upgrade head` runs cleanly against a live PostgreSQL instance"
|
||||
- "Test scaffolds `tests/test_alembic.py::test_migration_creates_all_tables` and `tests/test_alembic.py::test_documents_user_id_nullable` flip from XFAIL to PASSED"
|
||||
artifacts:
|
||||
- path: "backend/db/models.py"
|
||||
provides: "Full v1 ORM schema — User, Quota, RefreshToken, Folder, Document, Topic, DocumentTopic, Share, AuditLog, CloudConnection, Group"
|
||||
contains: "class Document(Base)"
|
||||
min_lines: 130
|
||||
- path: "backend/db/session.py"
|
||||
provides: "Async engine + async_sessionmaker"
|
||||
contains: "create_async_engine"
|
||||
- path: "backend/deps/db.py"
|
||||
provides: "FastAPI dependency yielding AsyncSession"
|
||||
contains: "async def get_db"
|
||||
- path: "backend/alembic.ini"
|
||||
provides: "Alembic config pointing at DATABASE_MIGRATE_URL"
|
||||
contains: "script_location = migrations"
|
||||
- path: "backend/migrations/env.py"
|
||||
provides: "Async migration env wiring Base.metadata"
|
||||
contains: "async_engine_from_config"
|
||||
- path: "backend/migrations/versions/0001_initial_schema.py"
|
||||
provides: "Initial v1 schema migration"
|
||||
contains: "op.create_table"
|
||||
key_links:
|
||||
- from: "backend/migrations/env.py"
|
||||
to: "backend/db/models.py"
|
||||
via: "from db.models import Base"
|
||||
pattern: "from db\\.models import Base"
|
||||
- from: "backend/db/session.py"
|
||||
to: "config.settings.database_url"
|
||||
via: "create_async_engine(settings.database_url, ...)"
|
||||
pattern: "create_async_engine\\(settings\\.database_url"
|
||||
- from: "backend/alembic.ini"
|
||||
to: "env var DATABASE_MIGRATE_URL"
|
||||
via: "%(DATABASE_MIGRATE_URL)s interpolation"
|
||||
pattern: "%\\(DATABASE_MIGRATE_URL\\)s"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Establish the database layer: declare the full v1 SQLAlchemy ORM schema in `backend/db/models.py` (D-01, D-02, D-03), wire async engine + session factory in `backend/db/session.py`, expose a FastAPI dependency in `backend/deps/db.py`, scaffold Alembic with the async template, and author the initial migration `0001_initial_schema.py`. The plan ends with a human-verify checkpoint where we boot PostgreSQL via Docker Compose and run `alembic upgrade head` to confirm the migration applies cleanly against the live database — the Phase 1 success criterion (ROADMAP.md criterion #2).
|
||||
|
||||
Purpose: Without a working schema, no document can be stored, no health check can probe PostgreSQL, and the walking skeleton cannot complete its end-to-end loop. This plan delivers the data foundation that Plans 04 and 05 will write to.
|
||||
|
||||
Output: Five new Python files under `backend/db/`, `backend/deps/`, and `backend/migrations/`; an `alembic.ini`; the initial migration file; and verified `alembic upgrade head` success against the live Docker PostgreSQL.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@CLAUDE.md
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/01-infrastructure-foundation/01-CONTEXT.md
|
||||
@.planning/phases/01-infrastructure-foundation/01-RESEARCH.md
|
||||
@.planning/phases/01-infrastructure-foundation/01-PATTERNS.md
|
||||
@.planning/phases/01-infrastructure-foundation/01-VALIDATION.md
|
||||
@.planning/phases/01-infrastructure-foundation/SKELETON.md
|
||||
@.planning/phases/01-infrastructure-foundation/01-01-SUMMARY.md
|
||||
@.planning/phases/01-infrastructure-foundation/01-02-SUMMARY.md
|
||||
|
||||
<interfaces>
|
||||
This plan creates the contracts other plans depend on. Subsequent plans MUST import from these locations only:
|
||||
|
||||
```python
|
||||
# backend/db/models.py — exports
|
||||
class Base(DeclarativeBase): pass
|
||||
class User(Base): __tablename__ = "users"
|
||||
class Quota(Base): __tablename__ = "quotas"
|
||||
class RefreshToken(Base): __tablename__ = "refresh_tokens"
|
||||
class Folder(Base): __tablename__ = "folders"
|
||||
class Document(Base): __tablename__ = "documents" # user_id NULLABLE per D-03
|
||||
class Topic(Base): __tablename__ = "topics"
|
||||
class DocumentTopic(Base): __tablename__ = "document_topics"
|
||||
class Share(Base): __tablename__ = "shares"
|
||||
class AuditLog(Base): __tablename__ = "audit_log"
|
||||
class CloudConnection(Base): __tablename__ = "cloud_connections"
|
||||
class Group(Base): __tablename__ = "groups" # D-02 stub
|
||||
|
||||
# backend/db/session.py — exports
|
||||
engine: AsyncEngine # bound to settings.database_url
|
||||
AsyncSessionLocal: async_sessionmaker[AsyncSession] # expire_on_commit=False
|
||||
|
||||
# backend/deps/db.py — exports
|
||||
async def get_db() -> AsyncGenerator[AsyncSession, None]
|
||||
```
|
||||
|
||||
Existing code that consumes the new layer:
|
||||
- Plan 04 imports `Base`, all model classes, and `AsyncSessionLocal`
|
||||
- Plan 05 imports `AsyncSessionLocal` (lifespan) and `get_db` (route handlers)
|
||||
- Plan 02's `tests/conftest.py` already references `from db.models import Base` and `from deps.db import get_db` inside try/except blocks — those imports must succeed after this plan
|
||||
|
||||
URL prefix is `postgresql+psycopg://` (RESEARCH.md Pattern 1 — same driver for sync Alembic and async FastAPI).
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: Author backend/db/models.py — full v1 SQLAlchemy 2.0 ORM schema</name>
|
||||
<files>backend/db/__init__.py, backend/db/models.py, backend/db/session.py, backend/deps/__init__.py, backend/deps/db.py</files>
|
||||
<behavior>
|
||||
- Importing `from db.models import Base` succeeds and `Base.metadata.tables` contains all 11 expected table names
|
||||
- `Document.__table__.c.user_id.nullable is True` (D-03)
|
||||
- `Group` class exists with `__tablename__ = "groups"` even though it has no rows (D-02 stub)
|
||||
- `from db.session import engine, AsyncSessionLocal` returns objects whose URL matches `settings.database_url` and whose sessionmaker has `expire_on_commit=False`
|
||||
- `from deps.db import get_db` returns an async generator function
|
||||
- `Quota.limit_bytes` default equals `104857600` (100 MB)
|
||||
- All UUID-primary-keyed tables use `UUID(as_uuid=True)` with `default=uuid.uuid4`
|
||||
- All timestamp columns use `TIMESTAMP(timezone=True)` with `server_default=func.now()`
|
||||
- The `Document` model has indexes `ix_documents_user_folder` and `ix_documents_user_created`
|
||||
- The `AuditLog` model uses `INET` for `ip_address` and `JSONB` for `metadata` (PostgreSQL-specific dialect types)
|
||||
</behavior>
|
||||
<read_first>
|
||||
- .planning/phases/01-infrastructure-foundation/01-RESEARCH.md (Code Examples lines 769-908 — full schema; Pattern 1 lines 240-289 — engine + session factory; Config Extension lines 914-937)
|
||||
- .planning/phases/01-infrastructure-foundation/01-PATTERNS.md (backend/db/models.py, backend/db/session.py, backend/deps/db.py sections)
|
||||
- .planning/phases/01-infrastructure-foundation/01-CONTEXT.md (D-01 full schema; D-02 groups stub; D-03 documents.user_id nullable)
|
||||
- backend/config.py (after Plan 01 — must contain `settings.database_url`)
|
||||
- backend/tests/test_alembic.py (Plan 02 output — defines expected table set and nullable assertion)
|
||||
</read_first>
|
||||
<action>
|
||||
Create directory `backend/db/` containing an empty `__init__.py` and a `models.py` that declares the full v1 schema using SQLAlchemy 2.0 typed `Mapped[]` syntax. Imports: `import uuid`, `from datetime import datetime`, `from sqlalchemy import Boolean, BigInteger, ForeignKey, Index, String, Text, TIMESTAMP, UniqueConstraint, Integer`, `from sqlalchemy.dialects.postgresql import UUID, INET, JSONB`, `from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship`, `from sqlalchemy.sql import func`. Declare `class Base(DeclarativeBase): pass`. Declare the eleven model classes exactly per RESEARCH.md lines 788-908 (verbatim — that block was designed to be implementation-ready):
|
||||
`User`, `Quota`, `RefreshToken`, `Folder`, `Document` (with `user_id: Mapped[uuid.UUID | None]` and `nullable=True` per D-03), `Topic`, `DocumentTopic` (composite PK), `Share`, `AuditLog` (with `INET` for `ip_address`, `JSONB` for the `metadata_` column — rename to `metadata_` and pass `name="metadata"` as a `mapped_column` kwarg because `metadata` shadows the SQLAlchemy reserved attribute on `DeclarativeBase`; reflect this column name choice in the migration too), `CloudConnection`, `Group` (D-02 stub).
|
||||
|
||||
Create `backend/db/session.py` per RESEARCH.md Pattern 1: import `from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession`, import `from config import settings`. Declare module-level `engine = create_async_engine(settings.database_url, pool_pre_ping=True, echo=False)` and `AsyncSessionLocal = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)` — `expire_on_commit=False` is mandatory (Pitfall 1).
|
||||
|
||||
Create directory `backend/deps/` containing an empty `__init__.py` and a `db.py` with: `from db.session import AsyncSessionLocal`; `async def get_db():` that wraps `async with AsyncSessionLocal() as session: try: yield session finally: await session.close()`.
|
||||
|
||||
For the SQLAlchemy `metadata` reserved-name issue on `AuditLog`: declare the column as `metadata_: Mapped[dict | None] = mapped_column("metadata", JSONB, nullable=True)` so the ORM attribute is `metadata_` but the DB column is named `metadata`. Document this in a code comment.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /Users/nik/Documents/Progamming/document_scanner/backend && python3 -c "
|
||||
from db.models import Base, User, Quota, RefreshToken, Folder, Document, Topic, DocumentTopic, Share, AuditLog, CloudConnection, Group
|
||||
expected = {'users','quotas','refresh_tokens','folders','documents','topics','document_topics','shares','audit_log','cloud_connections','groups'}
|
||||
actual = set(Base.metadata.tables.keys())
|
||||
assert expected == actual, f'mismatch: missing={expected-actual} extra={actual-expected}'
|
||||
assert Document.__table__.c.user_id.nullable is True, 'documents.user_id must be nullable (D-03)'
|
||||
assert Quota.__table__.c.limit_bytes.default.arg == 104857600, 'quota limit_bytes default must be 100MB'
|
||||
from db.session import engine, AsyncSessionLocal
|
||||
from deps.db import get_db
|
||||
print('models-ok')
|
||||
"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `backend/db/models.py` exists and the verify command prints `models-ok` exiting 0
|
||||
- File contains exactly 11 `__tablename__ = "<name>"` lines covering all of `users`, `quotas`, `refresh_tokens`, `folders`, `documents`, `topics`, `document_topics`, `shares`, `audit_log`, `cloud_connections`, `groups` (verifiable via `grep -E "__tablename__ = \"(users|quotas|refresh_tokens|folders|documents|topics|document_topics|shares|audit_log|cloud_connections|groups)\"" backend/db/models.py | sort -u | wc -l` equals 11)
|
||||
- File contains `nullable=True` on the `Document.user_id` mapped_column declaration (D-03)
|
||||
- File contains `class Base(DeclarativeBase)`
|
||||
- File imports `UUID`, `INET`, `JSONB` from `sqlalchemy.dialects.postgresql`
|
||||
- File contains `mapped_column("metadata", JSONB` for `AuditLog.metadata_` (avoids the reserved-attribute conflict)
|
||||
- `backend/db/session.py` contains `create_async_engine(settings.database_url`
|
||||
- `backend/db/session.py` contains `expire_on_commit=False`
|
||||
- `backend/db/session.py` contains `pool_pre_ping=True`
|
||||
- `backend/deps/db.py` contains `async def get_db()`
|
||||
- `backend/deps/db.py` contains `async with AsyncSessionLocal() as session:`
|
||||
- Files `backend/db/__init__.py` and `backend/deps/__init__.py` exist (empty is acceptable)
|
||||
- From the backend dir, `python3 -c "from db.models import Base; from db.session import engine, AsyncSessionLocal; from deps.db import get_db"` exits 0
|
||||
- `Base.metadata.tables['documents'].c.user_id.nullable` is `True` (verifiable through the verify command's inline assertion)
|
||||
</acceptance_criteria>
|
||||
<done>The full v1 ORM schema is declared, the async engine + session factory is wired, and the FastAPI dependency is exposed. Plans 04 and 05 can now import these symbols.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Scaffold Alembic async + author 0001_initial_schema.py migration</name>
|
||||
<files>backend/alembic.ini, backend/migrations/env.py, backend/migrations/script.py.mako, backend/migrations/versions/0001_initial_schema.py</files>
|
||||
<behavior>
|
||||
- `alembic.ini` declares `script_location = migrations` and `sqlalchemy.url = %(DATABASE_MIGRATE_URL)s` so Alembic reads the migration DSN from the environment
|
||||
- `migrations/env.py` uses `async_engine_from_config` + `asyncio.run(run_async_migrations())` and imports `from db.models import Base` so `target_metadata = Base.metadata` is populated (Pitfall 2)
|
||||
- `migrations/versions/0001_initial_schema.py` has `revision = "0001"`, `down_revision = None`, and an `upgrade()` that creates all 11 tables in dependency order (users → quotas, folders, refresh_tokens, topics → documents → document_topics, shares, audit_log → cloud_connections; groups standalone)
|
||||
- `upgrade()` ends with two `op.execute()` calls: (a) `GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO docuvault_app;` and (b) `ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO docuvault_app;` (Pitfall 4)
|
||||
- `downgrade()` is a no-op or drops tables in reverse dependency order (Phase 1 will never call downgrade in practice but a defensive drop is acceptable)
|
||||
- All UUID primary keys use `sa.dialects.postgresql.UUID(as_uuid=True)` and a `sa.text("gen_random_uuid()")` server_default OR are populated by application-level `default=uuid.uuid4` (matching the ORM declaration)
|
||||
</behavior>
|
||||
<read_first>
|
||||
- backend/db/models.py (Task 1 output — schema definitions the migration must materialize)
|
||||
- .planning/phases/01-infrastructure-foundation/01-RESEARCH.md (Pattern 2 — Alembic async env.py; Pitfall 2 — must import models; Pitfall 4 — ALTER DEFAULT PRIVILEGES inside migration)
|
||||
- .planning/phases/01-infrastructure-foundation/01-PATTERNS.md (backend/alembic.ini section + backend/migrations/env.py section)
|
||||
- .planning/phases/01-infrastructure-foundation/01-CONTEXT.md (D-13 two DSNs; D-14 init script handles role creation, not table grants)
|
||||
- .env.example (Plan 01 output — confirm `DATABASE_MIGRATE_URL` variable name)
|
||||
</read_first>
|
||||
<action>
|
||||
From the `backend/` directory, run `alembic init -t async migrations` once to generate the standard async template (`migrations/env.py`, `migrations/script.py.mako`, `migrations/versions/`, and a root `alembic.ini`). Then make these specific edits:
|
||||
|
||||
**`alembic.ini`**: set `script_location = migrations`. Set `sqlalchemy.url = %(DATABASE_MIGRATE_URL)s`. Remove any leftover example URL. Leave other defaults from the template.
|
||||
|
||||
**`migrations/env.py`**: replace the placeholder `target_metadata = None` with the block per RESEARCH.md Pattern 2:
|
||||
`from db.models import Base # noqa: F401 — must import to register all models (Pitfall 2)`
|
||||
`target_metadata = Base.metadata`
|
||||
Also add `import os` at the top, and BEFORE `config = context.config`, inject the runtime DSN by calling `config.set_main_option("sqlalchemy.url", os.environ.get("DATABASE_MIGRATE_URL", config.get_main_option("sqlalchemy.url") or ""))` — this is necessary because `%(DATABASE_MIGRATE_URL)s` interpolation in `alembic.ini` only reads from `[alembic]` section variables, not OS env. Keep the rest of the generated `run_migrations_offline` / `run_async_migrations` / `run_migrations_online` structure verbatim from the template.
|
||||
|
||||
**`migrations/versions/0001_initial_schema.py`**: hand-author this file (do NOT use `alembic revision --autogenerate` here — Phase 1 has no prior schema to diff). File header sets `revision = "0001"`, `down_revision = None`, `branch_labels = None`, `depends_on = None`. The `upgrade()` function uses `op.create_table()` calls for all 11 tables matching the ORM declarations in `backend/db/models.py`, with `op.create_index()` calls for every `Index(...)` declared on the ORM side (`ix_refresh_tokens_user_revoked`, `ix_documents_user_folder`, `ix_documents_user_created`, `ix_shares_recipient`, `ix_audit_user_created`, `ix_audit_event_created`, `ix_cloud_connections_user`). Use `sa.dialects.postgresql.UUID(as_uuid=True)` for UUID columns, `sa.dialects.postgresql.INET()` for `audit_log.ip_address`, `sa.dialects.postgresql.JSONB()` for the `metadata` column (use the literal column name `metadata` here since SQL doesn't care). The `documents.user_id` column MUST be `nullable=True` (D-03). Add `sa.UniqueConstraint("user_id", "parent_id", "name", name="uq_folders_user_parent_name")` on folders, and similar named constraints for topics and shares. End the `upgrade()` function with these two literal statements:
|
||||
|
||||
```
|
||||
op.execute("GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO docuvault_app;")
|
||||
op.execute("ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO docuvault_app;")
|
||||
op.execute("GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO docuvault_app;")
|
||||
op.execute("ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO docuvault_app;")
|
||||
```
|
||||
|
||||
The last two SEQUENCES grants are needed because the `audit_log.id` autoincrement uses a sequence and the `docuvault_app` user must be able to call `nextval()`. Add a comment block above these statements explaining "Pitfall 4: ALTER DEFAULT PRIVILEGES required so future migrations inherit grants automatically; the `docuvault_app` user is created in `docker/postgres/initdb.d/01-init-users.sql` with CONNECT but no table privileges."
|
||||
|
||||
Implement `downgrade()` to drop all tables in reverse dependency order: `op.drop_table("cloud_connections")`, then `audit_log`, `shares`, `document_topics`, `topics` (after document_topics), `documents`, `folders`, `refresh_tokens`, `quotas`, `groups`, `users`. Drop indexes before dropping tables where applicable.
|
||||
|
||||
Keep `migrations/script.py.mako` exactly as generated by `alembic init`.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /Users/nik/Documents/Progamming/document_scanner/backend && python3 -c "
|
||||
import os, configparser
|
||||
cp = configparser.ConfigParser()
|
||||
cp.read('alembic.ini')
|
||||
assert cp.get('alembic', 'script_location') == 'migrations'
|
||||
assert '%(DATABASE_MIGRATE_URL)s' in cp.get('alembic', 'sqlalchemy.url')
|
||||
print('alembic-ini-ok')
|
||||
" && grep -q "from db.models import Base" backend/migrations/env.py && grep -q "target_metadata = Base.metadata" backend/migrations/env.py && grep -q "ALTER DEFAULT PRIVILEGES" backend/migrations/versions/0001_initial_schema.py && grep -q "GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES" backend/migrations/versions/0001_initial_schema.py && echo "migration-files-ok"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `backend/alembic.ini` exists and the verify script prints `alembic-ini-ok`
|
||||
- `backend/migrations/env.py` exists and contains the line `from db.models import Base`
|
||||
- `backend/migrations/env.py` contains `target_metadata = Base.metadata`
|
||||
- `backend/migrations/env.py` contains `async_engine_from_config` (preserved from template)
|
||||
- `backend/migrations/env.py` contains `config.set_main_option("sqlalchemy.url", os.environ.get("DATABASE_MIGRATE_URL"`
|
||||
- `backend/migrations/script.py.mako` exists (from `alembic init`)
|
||||
- `backend/migrations/versions/0001_initial_schema.py` exists and contains `revision = "0001"`
|
||||
- `backend/migrations/versions/0001_initial_schema.py` contains `op.create_table` exactly 11 times (one per table — verifiable via `grep -c "^ op.create_table" backend/migrations/versions/0001_initial_schema.py | grep -q "^11$"`)
|
||||
- The migration contains all 11 table-name literals as the first positional arg to `op.create_table` (verifiable: `for t in users quotas refresh_tokens folders documents topics document_topics shares audit_log cloud_connections groups; do grep -q "op.create_table(\"$t\"" backend/migrations/versions/0001_initial_schema.py || echo MISSING:$t; done` produces no output)
|
||||
- The migration contains both `op.execute("GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES` and `op.execute("ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT`
|
||||
- The migration contains `op.execute("GRANT USAGE, SELECT ON ALL SEQUENCES`
|
||||
- The `documents` table creation block contains `nullable=True` on the `user_id` column line (D-03)
|
||||
- The migration contains `sa.dialects.postgresql.INET` (for `audit_log.ip_address`)
|
||||
- The migration contains `sa.dialects.postgresql.JSONB` (for `audit_log.metadata`)
|
||||
- `cd backend && python3 -c "from alembic.config import Config; from alembic.script import ScriptDirectory; cfg = Config('alembic.ini'); sd = ScriptDirectory.from_config(cfg); revs = list(sd.walk_revisions()); assert any(r.revision == '0001' for r in revs), 'revision 0001 not found'; print('script-discovery-ok')"` exits 0 with `script-discovery-ok`
|
||||
</acceptance_criteria>
|
||||
<done>Alembic is scaffolded with the async template; `env.py` imports the Base metadata so autogenerate would work; the initial migration creates all 11 tables and issues both immediate and default-privilege GRANTs for `docuvault_app`.</done>
|
||||
</task>
|
||||
|
||||
<task type="checkpoint:human-verify" gate="blocking">
|
||||
<name>Task 3: Boot PostgreSQL via Docker Compose and run `alembic upgrade head` against the live DB</name>
|
||||
<files>(verification only)</files>
|
||||
<read_first>
|
||||
- docker-compose.yml (Plan 01 output)
|
||||
- .env.example (Plan 01 output — values to copy into .env)
|
||||
- backend/migrations/versions/0001_initial_schema.py (Task 2 output)
|
||||
</read_first>
|
||||
<what-built>
|
||||
Plans 01-03 together: a `docker-compose.yml` with a healthy `postgres` service, a `docuvault_app` + `docuvault_migrate` PostgreSQL user pair, an Alembic async config, and an initial migration that creates the full v1 schema. We now boot PostgreSQL and apply the migration as the final acceptance step for ROADMAP.md Phase 1 success criterion #2: "Running `alembic upgrade head` applies the initial migration cleanly against the fresh PostgreSQL instance with no errors."
|
||||
</what-built>
|
||||
<how-to-verify>
|
||||
Run these commands from the project root and confirm each one exits 0 and produces the expected output:
|
||||
|
||||
1. `cp .env.example .env` (only if `.env` does not already exist — never overwrite a real `.env`)
|
||||
2. `docker compose up -d postgres` — wait ~15 seconds, then `docker compose ps postgres` and confirm the `STATUS` column shows `Up (healthy)`. If `unhealthy`, run `docker compose logs postgres` and report the first error.
|
||||
3. From inside the backend container: `docker compose run --rm backend bash -lc "cd /app && alembic upgrade head"` — confirm the output ends with `Running upgrade -> 0001, <message>` and exits 0. No `error`, `permission denied`, or `relation already exists` messages may appear.
|
||||
4. Confirm all 11 tables exist:
|
||||
`docker compose exec postgres psql -U postgres -d docuvault -c "\dt"`
|
||||
— output must list all 11 tables: `users`, `quotas`, `refresh_tokens`, `folders`, `documents`, `topics`, `document_topics`, `shares`, `audit_log`, `cloud_connections`, `groups`.
|
||||
5. Confirm `documents.user_id` is nullable:
|
||||
`docker compose exec postgres psql -U postgres -d docuvault -c "SELECT column_name, is_nullable FROM information_schema.columns WHERE table_name='documents' AND column_name='user_id';"`
|
||||
— output must show `user_id | YES`.
|
||||
6. Confirm `docuvault_app` can read tables (default-privileges grant):
|
||||
`docker compose exec postgres psql -U docuvault_app -d docuvault -c "SELECT count(*) FROM users;"`
|
||||
— output must show `0` (zero rows, no permission error).
|
||||
7. Re-run `cd backend && python3 -m pytest tests/test_alembic.py -v` from the host (or `docker compose run --rm backend bash -lc "cd /app && pytest tests/test_alembic.py -v"`) — both tests previously marked XFAIL should now report XPASSED (or PASSED if the xfail marker was removed) — at minimum, neither should report FAILED.
|
||||
</how-to-verify>
|
||||
<expected-outcome>
|
||||
All 7 verification commands succeed; the migration applies cleanly; `docuvault_app` has working SELECT access; the two test_alembic.py scaffolds pass.
|
||||
</expected-outcome>
|
||||
<if-broken>
|
||||
Common failures and fixes:
|
||||
- `permission denied for schema public`: the `ALTER DEFAULT PRIVILEGES` grant must run as the `docuvault_migrate` user. Either re-run the migration as that user (set `DATABASE_MIGRATE_URL` to use `docuvault_migrate`), or temporarily run the migration as `postgres` (superuser) for the very first run. Fix the migration approach and retry.
|
||||
- `relation "alembic_version" does not exist`: this is the first migration on an empty DB — normal; not an error.
|
||||
- `alembic.util.exc.CommandError: Can't locate revision`: the `env.py` likely failed to import `db.models`. Add `import sys; sys.path.insert(0, '/app')` to the top of `env.py` if needed.
|
||||
- Healthcheck stuck `starting`: increase `start_period` on the postgres service to `30s`.
|
||||
</if-broken>
|
||||
<resume-signal>Type "approved" once all 7 verification steps pass, or describe the specific failure to resume planning with a fix.</resume-signal>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Alembic process → PostgreSQL | Connects with `docuvault_migrate` (DDL privileges); never used by the runtime app |
|
||||
| Runtime app → PostgreSQL | Connects with `docuvault_app` (DML only, no DDL); granted via the migration's `ALTER DEFAULT PRIVILEGES` |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|-------------|-----------------|
|
||||
| T-01-03-01 | Elevation of Privilege | App connecting with DDL-capable user | mitigate | Migration uses `DATABASE_MIGRATE_URL` (docuvault_migrate); `db/session.py` uses `settings.database_url` (docuvault_app). The two DSNs are read from disjoint env vars; cannot be cross-wired without a config change (D-13). |
|
||||
| T-01-03-02 | Tampering | SQL injection via ORM string concatenation | mitigate | All schema access is through SQLAlchemy ORM with `mapped_column` declarations; no raw string interpolation in models or migration (SEC-03 / CLAUDE.md). The only raw SQL strings in the migration are constant DDL/GRANT statements with no user input. |
|
||||
| T-01-03-03 | Information Disclosure | `audit_log.metadata` JSONB column leaking PII | accept | Phase 1 has no audit-log writes; `audit_log` table is created empty (D-01 schema seeding). Phase 2/4 must enforce no-document-content rule on writes (SEC-07, ADMIN-06). Documented in SKELETON.md "Out of Scope". |
|
||||
| T-01-03-04 | Tampering | Migration applied as superuser leaves orphan privileges | mitigate | Migration ends with explicit `GRANT ... TO docuvault_app` and `ALTER DEFAULT PRIVILEGES ... TO docuvault_app` (Pitfall 4); only `docuvault_app` gets DML; superuser is not used by the runtime app. |
|
||||
| T-01-03-SC | Tampering | npm/pip installs | N/A | No new package installs in this plan; uses sqlalchemy/psycopg/alembic from Plan 01. RESEARCH.md audit covers all three (slopcheck OK). |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- Tasks 1 + 2 are autonomous; Task 3 is a blocking human-verify checkpoint.
|
||||
- After Task 3 approval: `cd backend && python3 -m pytest tests/test_alembic.py -v` must show both tests as XPASSED or PASSED (no FAILED).
|
||||
- `docker compose exec postgres psql -U postgres -d docuvault -c "\dt"` lists all 11 tables.
|
||||
- `docker compose exec postgres psql -U docuvault_app -d docuvault -c "SELECT count(*) FROM users;"` returns `0` without permission error.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- `backend/db/models.py` declares the full v1 schema (11 tables) with `Document.user_id` nullable (D-03) and `Group` table stub (D-02).
|
||||
- `backend/db/session.py` and `backend/deps/db.py` provide the async engine, session factory, and FastAPI dependency.
|
||||
- `backend/alembic.ini`, `backend/migrations/env.py`, and `backend/migrations/versions/0001_initial_schema.py` exist and work end-to-end.
|
||||
- `alembic upgrade head` against the live Docker PostgreSQL applies cleanly — ROADMAP.md Phase 1 success criterion #2 met.
|
||||
- `docuvault_app` user has working SELECT/INSERT/UPDATE/DELETE on all tables (D-14 + Pitfall 4 fix).
|
||||
- `tests/test_alembic.py` tests stop xfailing.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/01-infrastructure-foundation/01-03-SUMMARY.md` when done. Include: full table list materialized, the exact Alembic command output from Task 3, any deviations from the schema in RESEARCH.md (e.g., the `metadata_` rename), and the PostgreSQL version that actually booted (image tag from `docker compose images postgres`).
|
||||
</output>
|
||||
Reference in New Issue
Block a user