kite/.planning/phases/12-cloud-resource-foundation/12-05-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
12	05	gap_closure	1	12-04	docker-compose.yml backend/tests/test_compose_migrations.py backend/tests/test_migration_0006.py backend/main.py frontend/package.json frontend/package-lock.json AGENTS.md README.md RUNBOOK.md SECURITY.md	true	CONN-04 CLOUD-01	truths artifacts key_links A clean `docker compose up` applies Alembic through head before backend, Celery worker, or Celery beat starts An existing database at revision 0005 upgrades to 0006 and contains display_name_override plus all Phase 12 cloud metadata tables GET /api/cloud/connections no longer fails with UndefinedColumn after a normal Compose restart POST /api/cloud/connections/webdav can persist a valid Nextcloud connection after migration A failed migration blocks application process startup instead of serving with schema drift path provides contains docker-compose.yml One-shot Alembic migration service and service_completed_successfully dependency gate alembic upgrade head path provides contains backend/tests/test_compose_migrations.py Static/Compose configuration regression for migration ordering service_completed_successfully path provides contains backend/tests/test_migration_0006.py PostgreSQL upgrade regression from revision 0005 to head display_name_override path provides contains RUNBOOK.md Normal migration lifecycle, revision verification, and schema-drift recovery alembic current from to via pattern docker-compose.yml migrate service backend/celery-worker/celery-beat services depends_on migrate condition service_completed_successfully service_completed_successfully from to via pattern backend/migrations/versions/0006_cloud_resource_foundation.py backend/tests/test_migration_0006.py real PostgreSQL revision and information_schema assertions 0005.*0006|display_name_override

Close the Phase 12 UAT blockers caused by application/schema drift. Make database migration a mandatory Compose startup gate, add a real upgrade regression, repair the running environment, and verify both cloud connection listing and Nextcloud connection creation before UAT resumes.

Root cause: .planning/debug/phase-12-cloud-schema-cold-start.md proves the live database remained at Alembic 0005 because Compose started application processes without running alembic upgrade head.

<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/phases/12-cloud-resource-foundation/12-UAT.md @.planning/debug/phase-12-cloud-schema-cold-start.md @.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md @backend/migrations/versions/0006_cloud_resource_foundation.py @docker-compose.yml Task 1: Add a migration-gated Compose startup path docker-compose.yml, backend/tests/test_compose_migrations.py - docker-compose.yml — current backend/worker/beat dependencies, hardening, volumes, and environment - backend/migrations/env.py — runtime `DATABASE_MIGRATE_URL` selection - backend/Dockerfile — image user/workdir and Alembic availability - .planning/debug/phase-12-cloud-schema-cold-start.md — confirmed deployment root cause Add a one-shot `migrate` service built from `./backend` that runs `alembic upgrade head` from `/app`, receives `DATABASE_MIGRATE_URL` and only migration-required environment, mounts backend source consistently for development, waits for PostgreSQL health (and any migration-time MinIO prerequisite required by revision 0004), and uses the same non-root/read-only/tmpfs/cap-drop/no-new-privileges hardening as backend. Change backend, celery-worker, and celery-beat so each has `depends_on.migrate.condition: service_completed_successfully` in addition to its runtime dependencies. Do not run Alembic independently in every replica and do not grant DDL privileges to `docuvault_app`. Add focused tests that load/render the Compose configuration and assert the migrate command, migrate-role URL, one-shot dependency, and absence of `alembic upgrade` from ordinary backend/worker commands. Include a failure-path assertion showing a nonzero migration exit prevents dependent services from starting. - `docker compose config` succeeds and contains exactly one migration service command `alembic upgrade head` - backend, celery-worker, and celery-beat each wait for successful migration completion - migration service uses `DATABASE_MIGRATE_URL`; application services continue using `DATABASE_URL` - migration service preserves container hardening and stores no credentials in source - `pytest -q tests/test_compose_migrations.py` passes docker compose config >/tmp/docuvault-compose.yml && docker compose run --rm backend pytest -q tests/test_compose_migrations.py Compose has one hardened migration gate, every application process depends on it, and configuration tests pass. Task 2: Prove PostgreSQL upgrade from 0005 to Phase 12 head backend/tests/test_migration_0006.py, RUNBOOK.md - backend/migrations/versions/0005_system_settings.py — starting revision - backend/migrations/versions/0006_cloud_resource_foundation.py — required upgrade objects - backend/tests/test_alembic.py — existing migration tests and their limitations - docker/postgres/initdb.d/01-init-users.sql — migrate/app role privileges - RUNBOOK.md — existing database operations guidance Add an integration test that uses a disposable PostgreSQL database or isolated test schema, applies Alembic through `0005`, then upgrades to head. Assert `alembic_version` is `0006`; `cloud_connections.display_name_override` exists; `cloud_items`, `cloud_item_topics`, and `cloud_folder_states` exist; named unique constraints/indexes exist; and `docuvault_app` can perform intended DML but not DDL. The test must never downgrade or mutate the developer's normal database and must skip with an explicit prerequisite message when a disposable integration database cannot be provisioned. Add a cold-start Compose test path suitable for CI that starts from an isolated project/volume, waits for migrate success, checks revision/head, exercises `/health`, and tears down only that isolated project. Document normal upgrade, revision inspection, failed-migration recovery, and the immediate `0005` to head remediation commands in RUNBOOK. - integration test proves a real 0005-to-0006/head transition without touching the normal development database - all four Phase 12 schema additions are asserted against PostgreSQL catalog/information_schema - migration failure is observable and prevents backend startup - RUNBOOK gives exact safe commands for current revision, upgrade, logs, restart, and recovery - focused migration tests pass in the Compose environment docker compose run --rm -e INTEGRATION=1 backend pytest -q tests/test_migration_0006.py A disposable PostgreSQL test upgrades from 0005 to head and proves the column/tables/privilege boundaries; operational recovery is documented. Task 3: Repair the environment, rerun blocker paths, and close protocol backend/main.py, frontend/package.json, frontend/package-lock.json, AGENTS.md, README.md, RUNBOOK.md, SECURITY.md - .planning/phases/12-cloud-resource-foundation/12-UAT.md — two confirmed blockers and four blocked tests - backend/api/cloud/connections.py — failing list and WebDAV/Nextcloud paths - AGENTS.md — bug regression, documentation, version, security, commit, and push protocols - SECURITY.md — Phase 12 threat register and gate evidence Apply `alembic upgrade head` through the new migration service in the current Compose environment, verify `alembic current` reports `0006`, and restart/recreate backend plus Celery processes through the gated path. Add API regressions against PostgreSQL for `GET /api/cloud/connections` and valid mocked/controlled `POST /api/cloud/connections/webdav`, asserting neither raises UndefinedColumn and the connection response excludes credentials. Run focused cloud/migration tests, full backend/frontend suites, frontend build, Bandit, npm audit, and available pip-audit/secret checks; resolve introduced high-severity findings. Bump the patch version from 0.2.0 to 0.2.1 in tracked version sources. Update AGENTS current state to Phase 12 UAT schema-drift gap closure, README startup behavior, RUNBOOK final commands, and SECURITY evidence. Commit and push atomically under `fix(12-uat): gate startup on database migrations`. - live `alembic current` reports 0006/head before backend starts - connection list returns successfully for an authenticated user - Nextcloud/WebDAV connection creation reaches normal validation/persistence without schema exceptions - focused regression, full suites, build, and mandatory security gates pass or document only pre-existing external blockers - backend and frontend tracked versions equal 0.2.1; docs accurately describe migration-gated startup docker compose run --rm migrate && docker compose run --rm backend alembic current && docker compose run --rm backend pytest -q tests/test_compose_migrations.py tests/test_migration_0006.py tests/test_cloud.py tests/test_cloud_security.py && cd frontend && npm test && npm run build The running schema is at head, both UAT blocker paths work, regression/security gates pass, version/docs are 0.2.1, and Phase 12 UAT can resume at Test 1.

<threat_model>

Threat ID	Threat	Mitigation and evidence
T-12-05-01	App starts against stale schema	One-shot migrate service and successful-completion dependency gate
T-12-05-02	DDL privilege escalation	Migration uses migrate URL; app retains DML-only URL and privilege regression test
T-12-05-03	Concurrent migrations	Exactly one Compose migration service; replicas only wait on completion
T-12-05-04	Secrets exposed in Compose/tests	Environment references only; no rendered config or logs committed with values
T-12-05-05	Test destroys developer DB	Disposable database/schema/project requirement; explicit skip rather than fallback to normal DB
T-12-05-SC	Dependency risk	No new runtime dependencies; standard security/audit gates rerun
</threat_model>

1. GSD plan structure validator reports zero errors/warnings. 2. Compose config proves migration dependency ordering and hardening. 3. Disposable PostgreSQL upgrade proves 0005 -> 0006/head and schema objects. 4. Live revision and both failed API paths pass after gated restart. 5. Full tests, build, and security gates complete before UAT resumes.

<success_criteria>

• Normal startup cannot serve application traffic with unapplied migrations.
• The exact display_name_override regression is caught automatically.
• Current Nextcloud and connection-list blockers are resolved at their deployment root cause.
• Documentation and versioning reflect the gap closure. </success_criteria>

Create .planning/phases/12-cloud-resource-foundation/12-05-SUMMARY.md when complete.