diff --git a/.planning/phases/12-cloud-resource-foundation/12-05-PLAN.md b/.planning/phases/12-cloud-resource-foundation/12-05-PLAN.md new file mode 100644 index 0000000..6b522b8 --- /dev/null +++ b/.planning/phases/12-cloud-resource-foundation/12-05-PLAN.md @@ -0,0 +1,174 @@ +--- +phase: "12" +plan: "05" +type: gap_closure +wave: 1 +depends_on: + - "12-04" +files_modified: + - docker-compose.yml + - backend/tests/test_compose_migrations.py + - backend/tests/test_migration_0006.py + - backend/main.py + - frontend/package.json + - frontend/package-lock.json + - AGENTS.md + - README.md + - RUNBOOK.md + - SECURITY.md +autonomous: true +requirements: + - CONN-04 + - CLOUD-01 + +must_haves: + truths: + - "A clean `docker compose up` applies Alembic through head before backend, Celery worker, or Celery beat starts" + - "An existing database at revision 0005 upgrades to 0006 and contains display_name_override plus all Phase 12 cloud metadata tables" + - "GET /api/cloud/connections no longer fails with UndefinedColumn after a normal Compose restart" + - "POST /api/cloud/connections/webdav can persist a valid Nextcloud connection after migration" + - "A failed migration blocks application process startup instead of serving with schema drift" + artifacts: + - path: "docker-compose.yml" + provides: "One-shot Alembic migration service and service_completed_successfully dependency gate" + contains: "alembic upgrade head" + - path: "backend/tests/test_compose_migrations.py" + provides: "Static/Compose configuration regression for migration ordering" + contains: "service_completed_successfully" + - path: "backend/tests/test_migration_0006.py" + provides: "PostgreSQL upgrade regression from revision 0005 to head" + contains: "display_name_override" + - path: "RUNBOOK.md" + provides: "Normal migration lifecycle, revision verification, and schema-drift recovery" + contains: "alembic current" + key_links: + - from: "docker-compose.yml migrate service" + to: "backend/celery-worker/celery-beat services" + via: "depends_on migrate condition service_completed_successfully" + pattern: "service_completed_successfully" + - from: "backend/migrations/versions/0006_cloud_resource_foundation.py" + to: "backend/tests/test_migration_0006.py" + via: "real PostgreSQL revision and information_schema assertions" + pattern: "0005.*0006|display_name_override" +--- + + +Close the Phase 12 UAT blockers caused by application/schema drift. Make database migration a mandatory Compose startup gate, add a real upgrade regression, repair the running environment, and verify both cloud connection listing and Nextcloud connection creation before UAT resumes. + +Root cause: `.planning/debug/phase-12-cloud-schema-cold-start.md` proves the live database remained at Alembic `0005` because Compose started application processes without running `alembic upgrade head`. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/phases/12-cloud-resource-foundation/12-UAT.md +@.planning/debug/phase-12-cloud-schema-cold-start.md +@.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md +@backend/migrations/versions/0006_cloud_resource_foundation.py +@docker-compose.yml + + + + + + Task 1: Add a migration-gated Compose startup path + docker-compose.yml, backend/tests/test_compose_migrations.py + + - docker-compose.yml — current backend/worker/beat dependencies, hardening, volumes, and environment + - backend/migrations/env.py — runtime `DATABASE_MIGRATE_URL` selection + - backend/Dockerfile — image user/workdir and Alembic availability + - .planning/debug/phase-12-cloud-schema-cold-start.md — confirmed deployment root cause + + + Add a one-shot `migrate` service built from `./backend` that runs `alembic upgrade head` from `/app`, receives `DATABASE_MIGRATE_URL` and only migration-required environment, mounts backend source consistently for development, waits for PostgreSQL health (and any migration-time MinIO prerequisite required by revision 0004), and uses the same non-root/read-only/tmpfs/cap-drop/no-new-privileges hardening as backend. Change backend, celery-worker, and celery-beat so each has `depends_on.migrate.condition: service_completed_successfully` in addition to its runtime dependencies. Do not run Alembic independently in every replica and do not grant DDL privileges to `docuvault_app`. Add focused tests that load/render the Compose configuration and assert the migrate command, migrate-role URL, one-shot dependency, and absence of `alembic upgrade` from ordinary backend/worker commands. Include a failure-path assertion showing a nonzero migration exit prevents dependent services from starting. + + + - `docker compose config` succeeds and contains exactly one migration service command `alembic upgrade head` + - backend, celery-worker, and celery-beat each wait for successful migration completion + - migration service uses `DATABASE_MIGRATE_URL`; application services continue using `DATABASE_URL` + - migration service preserves container hardening and stores no credentials in source + - `pytest -q tests/test_compose_migrations.py` passes + + docker compose config >/tmp/docuvault-compose.yml && docker compose run --rm backend pytest -q tests/test_compose_migrations.py + Compose has one hardened migration gate, every application process depends on it, and configuration tests pass. + + + + Task 2: Prove PostgreSQL upgrade from 0005 to Phase 12 head + backend/tests/test_migration_0006.py, RUNBOOK.md + + - backend/migrations/versions/0005_system_settings.py — starting revision + - backend/migrations/versions/0006_cloud_resource_foundation.py — required upgrade objects + - backend/tests/test_alembic.py — existing migration tests and their limitations + - docker/postgres/initdb.d/01-init-users.sql — migrate/app role privileges + - RUNBOOK.md — existing database operations guidance + + + Add an integration test that uses a disposable PostgreSQL database or isolated test schema, applies Alembic through `0005`, then upgrades to head. Assert `alembic_version` is `0006`; `cloud_connections.display_name_override` exists; `cloud_items`, `cloud_item_topics`, and `cloud_folder_states` exist; named unique constraints/indexes exist; and `docuvault_app` can perform intended DML but not DDL. The test must never downgrade or mutate the developer's normal database and must skip with an explicit prerequisite message when a disposable integration database cannot be provisioned. Add a cold-start Compose test path suitable for CI that starts from an isolated project/volume, waits for migrate success, checks revision/head, exercises `/health`, and tears down only that isolated project. Document normal upgrade, revision inspection, failed-migration recovery, and the immediate `0005` to head remediation commands in RUNBOOK. + + + - integration test proves a real 0005-to-0006/head transition without touching the normal development database + - all four Phase 12 schema additions are asserted against PostgreSQL catalog/information_schema + - migration failure is observable and prevents backend startup + - RUNBOOK gives exact safe commands for current revision, upgrade, logs, restart, and recovery + - focused migration tests pass in the Compose environment + + docker compose run --rm -e INTEGRATION=1 backend pytest -q tests/test_migration_0006.py + A disposable PostgreSQL test upgrades from 0005 to head and proves the column/tables/privilege boundaries; operational recovery is documented. + + + + Task 3: Repair the environment, rerun blocker paths, and close protocol + backend/main.py, frontend/package.json, frontend/package-lock.json, AGENTS.md, README.md, RUNBOOK.md, SECURITY.md + + - .planning/phases/12-cloud-resource-foundation/12-UAT.md — two confirmed blockers and four blocked tests + - backend/api/cloud/connections.py — failing list and WebDAV/Nextcloud paths + - AGENTS.md — bug regression, documentation, version, security, commit, and push protocols + - SECURITY.md — Phase 12 threat register and gate evidence + + + Apply `alembic upgrade head` through the new migration service in the current Compose environment, verify `alembic current` reports `0006`, and restart/recreate backend plus Celery processes through the gated path. Add API regressions against PostgreSQL for `GET /api/cloud/connections` and valid mocked/controlled `POST /api/cloud/connections/webdav`, asserting neither raises UndefinedColumn and the connection response excludes credentials. Run focused cloud/migration tests, full backend/frontend suites, frontend build, Bandit, npm audit, and available pip-audit/secret checks; resolve introduced high-severity findings. Bump the patch version from 0.2.0 to 0.2.1 in tracked version sources. Update AGENTS current state to Phase 12 UAT schema-drift gap closure, README startup behavior, RUNBOOK final commands, and SECURITY evidence. Commit and push atomically under `fix(12-uat): gate startup on database migrations`. + + + - live `alembic current` reports 0006/head before backend starts + - connection list returns successfully for an authenticated user + - Nextcloud/WebDAV connection creation reaches normal validation/persistence without schema exceptions + - focused regression, full suites, build, and mandatory security gates pass or document only pre-existing external blockers + - backend and frontend tracked versions equal 0.2.1; docs accurately describe migration-gated startup + + docker compose run --rm migrate && docker compose run --rm backend alembic current && docker compose run --rm backend pytest -q tests/test_compose_migrations.py tests/test_migration_0006.py tests/test_cloud.py tests/test_cloud_security.py && cd frontend && npm test && npm run build + The running schema is at head, both UAT blocker paths work, regression/security gates pass, version/docs are 0.2.1, and Phase 12 UAT can resume at Test 1. + + + + + +| Threat ID | Threat | Mitigation and evidence | +|-----------|--------|-------------------------| +| T-12-05-01 | App starts against stale schema | One-shot migrate service and successful-completion dependency gate | +| T-12-05-02 | DDL privilege escalation | Migration uses migrate URL; app retains DML-only URL and privilege regression test | +| T-12-05-03 | Concurrent migrations | Exactly one Compose migration service; replicas only wait on completion | +| T-12-05-04 | Secrets exposed in Compose/tests | Environment references only; no rendered config or logs committed with values | +| T-12-05-05 | Test destroys developer DB | Disposable database/schema/project requirement; explicit skip rather than fallback to normal DB | +| T-12-05-SC | Dependency risk | No new runtime dependencies; standard security/audit gates rerun | + + + +1. GSD plan structure validator reports zero errors/warnings. +2. Compose config proves migration dependency ordering and hardening. +3. Disposable PostgreSQL upgrade proves 0005 -> 0006/head and schema objects. +4. Live revision and both failed API paths pass after gated restart. +5. Full tests, build, and security gates complete before UAT resumes. + + + +- Normal startup cannot serve application traffic with unapplied migrations. +- The exact `display_name_override` regression is caught automatically. +- Current Nextcloud and connection-list blockers are resolved at their deployment root cause. +- Documentation and versioning reflect the gap closure. + + +Create `.planning/phases/12-cloud-resource-foundation/12-05-SUMMARY.md` when complete.