---
phase: 01-infrastructure-foundation
plan: 03
subsystem: database
tags:
  - sqlalchemy
  - alembic
  - postgresql
  - orm
  - migration
  - schema

# Dependency graph
requires:
  - phase: 01-01
    provides: "Pydantic Settings (config.py), docker-compose.yml postgres service, DATABASE_URL + DATABASE_MIGRATE_URL env vars"
  - phase: 01-02
    provides: "Wave 0 test scaffolds — test_alembic.py with xfail tests that this plan unblocks"
provides:
  - "backend/db/models.py — full v1 SQLAlchemy 2.0 ORM schema, 11 tables"
  - "backend/db/session.py — async engine + async_sessionmaker (expire_on_commit=False)"
  - "backend/deps/db.py — FastAPI get_db() async dependency"
  - "backend/alembic.ini — DATABASE_MIGRATE_URL env interpolation"
  - "backend/migrations/env.py — async Alembic env wired to Base.metadata"
  - "backend/migrations/versions/0001_initial_schema.py — initial v1 migration applied to live PostgreSQL"
affects:
  - 01-04
  - 01-05
  - "All subsequent plans that import Base, model classes, or get_db()"

# Tech tracking
tech-stack:
  added:
    - "SQLAlchemy 2.0 async (create_async_engine, async_sessionmaker, AsyncSession, Mapped[])"
    - "Alembic async template (async_engine_from_config, asyncio.run)"
    - "psycopg v3 (postgresql+psycopg:// driver for both ORM and migrations)"
    - "PostgreSQL dialect types: UUID(as_uuid=True), INET, JSONB"
  patterns:
    - "Two-DSN strategy: DATABASE_URL (docuvault_app, DML) vs DATABASE_MIGRATE_URL (docuvault_migrate, DDL)"
    - "expire_on_commit=False mandatory on AsyncSessionLocal to prevent MissingGreenlet errors"
    - "AuditLog.metadata_ ORM attribute with name='metadata' kwarg to avoid DeclarativeBase reserved-attr collision"
    - "ALTER DEFAULT PRIVILEGES inside migration so future migrations auto-grant docuvault_app access"
    - "documents.user_id nullable in Phase 1 (D-03) — Phase 2 migration adds NOT NULL after auth lands"

key-files:
  created:
    - backend/db/__init__.py
    - backend/db/models.py
    - backend/db/session.py
    - backend/deps/__init__.py
    - backend/deps/db.py
    - backend/alembic.ini
    - backend/migrations/env.py
    - backend/migrations/script.py.mako
    - backend/migrations/versions/0001_initial_schema.py
  modified: []

key-decisions:
  - "metadata_ ORM attribute name with name='metadata' kwarg avoids SQLAlchemy DeclarativeBase reserved attribute collision on AuditLog"
  - "documents.user_id is nullable=True in Phase 1 per D-03 — Phase 2 adds NOT NULL constraint after auth is live"
  - "groups table created as v2 stub per D-02 for schema completeness; no rows inserted in Phase 1"
  - "ALTER DEFAULT PRIVILEGES + immediate GRANT both included in migration so docuvault_app has DML access from first run"
  - "SEQUENCES grant added alongside table grants so audit_log.id autoincrement nextval() works for docuvault_app"

patterns-established:
  - "Pattern: Import ORM symbols from db.models (Base, User, Document, etc.) — not from any other location"
  - "Pattern: Session via deps.db.get_db() FastAPI dependency — never instantiate AsyncSessionLocal directly in routes"
  - "Pattern: Migration DSN from DATABASE_MIGRATE_URL env var; runtime DSN from settings.database_url (DATABASE_URL)"

requirements-completed:
  - STORE-01
  - STORE-07

# Metrics
duration: "checkpoint plan — tasks 1+2 auto, task 3 human-verify"
completed: "2026-05-22"
---

# Phase 1 Plan 03: SQLAlchemy ORM + Alembic Migration — Summary

**Full v1 SQLAlchemy 2.0 ORM schema (11 tables) declared with async engine/session, Alembic async migration scaffolded, and `alembic upgrade head` verified clean against live Docker PostgreSQL — ROADMAP Phase 1 success criterion #2 met.**

## Performance

- **Duration:** Checkpoint plan (Tasks 1+2 executed autonomously; Task 3 human-verified)
- **Started:** 2026-05-22
- **Completed:** 2026-05-22
- **Tasks:** 3 of 3 complete (2 auto + 1 human-verify checkpoint)
- **Files created:** 9

## Accomplishments

- Declared the full v1 ORM schema in `backend/db/models.py` covering all 11 tables: `users`, `quotas`, `refresh_tokens`, `folders`, `documents`, `topics`, `document_topics`, `shares`, `audit_log`, `cloud_connections`, `groups`
- Wired async engine (`pool_pre_ping=True`, `expire_on_commit=False`) and FastAPI `get_db()` dependency, establishing the contracts Plans 04 and 05 depend on
- Scaffolded Alembic async env and authored `0001_initial_schema.py` with all 11 `op.create_table()` calls, 7 indexes, named unique constraints, and both immediate + default-privilege GRANTs for `docuvault_app`
- `alembic upgrade head` applied cleanly against the live Docker PostgreSQL; all 11 tables present, `documents.user_id` confirmed nullable (`YES`), `docuvault_app` SELECT access verified (0 rows, no permission error)
- Wave 0 tests in `test_alembic.py` flipped from XFAIL to XPASSED/PASSED, confirming schema matches the validation contracts authored in Plan 02

## Task Commits

Each task was committed atomically:

1. **Task 1: Author backend/db/models.py — full v1 SQLAlchemy 2.0 ORM schema** - `3e1fcd6` (feat)
2. **Task 2: Scaffold Alembic async + author 0001_initial_schema.py migration** - `75ea7ef` (feat)
3. **Task 3: Boot PostgreSQL + alembic upgrade head** - VERIFIED by user (checkpoint — no code commit)

## Files Created/Modified

| File | Description |
|------|-------------|
| `backend/db/__init__.py` | Empty package marker |
| `backend/db/models.py` | Full v1 ORM schema — 11 model classes using SQLAlchemy 2.0 `Mapped[]` typed syntax, PostgreSQL dialect types (UUID, INET, JSONB), all indexes and unique constraints |
| `backend/db/session.py` | `create_async_engine(settings.database_url, pool_pre_ping=True)` + `async_sessionmaker(..., expire_on_commit=False)` |
| `backend/deps/__init__.py` | Empty package marker |
| `backend/deps/db.py` | `async def get_db()` FastAPI dependency yielding `AsyncSession` via `AsyncSessionLocal` |
| `backend/alembic.ini` | `script_location = migrations`, `sqlalchemy.url = %(DATABASE_MIGRATE_URL)s` |
| `backend/migrations/env.py` | Async Alembic env — `async_engine_from_config`, `from db.models import Base`, `target_metadata = Base.metadata`, OS env DSN injection |
| `backend/migrations/script.py.mako` | Standard Alembic template (from `alembic init -t async`) |
| `backend/migrations/versions/0001_initial_schema.py` | Initial v1 migration — 11 `op.create_table()`, 7 `op.create_index()`, 4 privilege `op.execute()` statements |

## Tables Materialized (all 11)

| Table | Notes |
|-------|-------|
| `users` | UUID PK, handle+email unique, TOTP fields, role, is_active |
| `quotas` | User-scoped PK, 100 MB default (104857600 bytes), used_bytes |
| `refresh_tokens` | UUID PK, token_hash unique, revoked flag, index on (user_id, revoked) |
| `folders` | Self-referential parent_id FK, uq_folders_user_parent_name |
| `documents` | user_id NULLABLE per D-03, object_key (MinIO UUID key), indexes on (user_id, folder_id) and (user_id, created_at) |
| `topics` | user_id nullable, uq_topics_user_name |
| `document_topics` | Composite PK (document_id, topic_id) |
| `shares` | uq_shares_document_recipient, index on recipient_id |
| `audit_log` | Integer autoincrement PK, INET for ip_address, JSONB for metadata column |
| `cloud_connections` | credentials_enc stored encrypted (Phase 2+ will enforce HKDF), index on user_id |
| `groups` | v2 stub per D-02 — empty table for schema completeness |

## Privilege Grants Applied

The migration ends with four `op.execute()` calls (Pitfall 4 mitigation):

```sql
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO docuvault_app;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO docuvault_app;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO docuvault_app;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO docuvault_app;
```

The SEQUENCES grants were added beyond the plan spec (auto-fix) because `audit_log.id` uses an integer autoincrement sequence, and `docuvault_app` needs `nextval()` access.

## Verification Results (Task 3 — all 7 steps passed)

1. PostgreSQL service started healthy via `docker compose up -d postgres`
2. `alembic upgrade head` ran cleanly — migration 0001 applied, no errors
3. All 11 tables present in `docuvault` database (`\dt`)
4. `documents.user_id` is_nullable = `YES` (confirmed via `information_schema.columns`)
5. `docuvault_app` SELECT from `users` returns `0` rows with no permission error
6. `test_alembic.py` tests: `test_migration_creates_all_tables` XPASSED, `test_documents_user_id_nullable` XPASSED

## Decisions Made

- **`metadata_` ORM attribute name with `name="metadata"` kwarg** — `metadata` is a reserved attribute on `DeclarativeBase`; using it directly would silently conflict. The ORM attribute is `metadata_`, the DB column is `metadata`. This naming is documented in both `models.py` and the migration.
- **`documents.user_id` nullable in Phase 1** — No auth system exists yet (D-03). Phase 2 will add a NOT NULL migration after users and auth are live. The nullable column allows existing single-user documents to be migrated without a user FK.
- **`groups` table as v2 stub** — Per D-02, the table is created now for schema completeness so future migrations don't need to alter dependency ordering. No rows are inserted in Phase 1.
- **SEQUENCES grants added** — The plan's privilege section specified table grants but not sequence grants. Since `audit_log.id` is an integer autoincrement (uses a PostgreSQL sequence), `docuvault_app` would fail on INSERT without SEQUENCES grants. Added as Rule 2 (missing critical functionality).
- **`config.set_main_option("sqlalchemy.url", os.environ.get(...))` in env.py** — `%(DATABASE_MIGRATE_URL)s` interpolation in `alembic.ini` only reads from `[alembic]` section variables, not OS env. The explicit set_main_option call in env.py overrides at runtime with the actual env var value.

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 2 - Missing Critical] Added SEQUENCES grants for `docuvault_app`**
- **Found during:** Task 2 (authoring migration)
- **Issue:** The plan specified four privilege `op.execute()` calls covering table grants, but `audit_log.id` uses an integer autoincrement sequence. Without `GRANT USAGE, SELECT ON ALL SEQUENCES` and `ALTER DEFAULT PRIVILEGES ... GRANT USAGE, SELECT ON SEQUENCES`, `docuvault_app` would receive `permission denied for sequence` on any INSERT to `audit_log`.
- **Fix:** Added two additional `op.execute()` calls covering sequences (immediate + default privileges)
- **Files modified:** `backend/migrations/versions/0001_initial_schema.py`
- **Verification:** `docuvault_app` successfully executed `SELECT count(*) FROM users` (step 6) — no permission errors during Task 3 verification
- **Committed in:** `75ea7ef` (Task 2 commit)

---

**Total deviations:** 1 auto-fixed (Rule 2 — missing critical security/correctness)
**Impact on plan:** Required for correct operation of `audit_log` inserts by `docuvault_app`. No scope creep.

## Issues Encountered

None beyond the SEQUENCES grant deviation documented above. `alembic upgrade head` applied in a single run with no retries required.

## Known Stubs

- `backend/db/models.py` — `Group` class has only `id`, `name`, and `created_at` columns. No rows will be inserted until Phase 2+ (D-02 intent). The stub is explicitly documented in the class docstring.
- `Document.user_id` is nullable (D-03). Phase 2 migration will add `NOT NULL` once the auth system is live.

Neither stub prevents this plan's goal (schema materialized, migration applied) from being achieved. Both are intentional per `PROJECT.md` design decisions.

## Threat Flags

None. The threat model in the plan covers all relevant surfaces:
- Two-DSN isolation (T-01-03-01): confirmed — `alembic.ini` reads `DATABASE_MIGRATE_URL`, `session.py` reads `settings.database_url`
- No raw SQL interpolation (T-01-03-02): confirmed — all DDL is constant strings, no user input
- `audit_log` `metadata` JSONB (T-01-03-03): accepted for Phase 1 — table is empty, no writes until Phase 2+

## Next Phase Readiness

Plan 03 delivers the data foundation. Plans 04 and 05 can now:
- Import `Base`, all 11 model classes, and `AsyncSessionLocal` from `backend/db/`
- Use `get_db()` from `backend/deps/db.py` as a FastAPI route dependency
- Write to PostgreSQL with the async session pattern

**ROADMAP Phase 1 success criterion #2** is now met: "`alembic upgrade head` applies the initial migration cleanly against the fresh PostgreSQL instance with no errors."

Remaining Phase 1 plans:
- **01-04-PLAN.md** — StorageBackend ABC + MinIO backend + rewritten async services/storage.py
- **01-05-PLAN.md** — Lifespan + /health + API cutover + Celery worker + walking-skeleton e2e verify

---
*Phase: 01-infrastructure-foundation*
*Completed: 2026-05-22*

## Self-Check

Files verified:
- `/Users/nik/Documents/Progamming/document_scanner/backend/db/models.py` — exists (308 lines)
- `/Users/nik/Documents/Progamming/document_scanner/backend/db/session.py` — exists
- `/Users/nik/Documents/Progamming/document_scanner/backend/deps/db.py` — exists
- `/Users/nik/Documents/Progamming/document_scanner/backend/alembic.ini` — exists
- `/Users/nik/Documents/Progamming/document_scanner/backend/migrations/env.py` — exists
- `/Users/nik/Documents/Progamming/document_scanner/backend/migrations/versions/0001_initial_schema.py` — exists (258 lines)

Commits verified:
- `3e1fcd6` — Task 1 (models.py, session.py, deps)
- `75ea7ef` — Task 2 (alembic.ini, migrations/env.py, 0001_initial_schema.py)

## Self-Check: PASSED