docs(12): plan UAT migration gap closure

.planning/debug/phase-12-cloud-schema-cold-start.md

@@ -0,0 +1,49 @@
+# Debug: Phase 12 Cloud Schema Cold Start
+
+**Status:** root cause found
+**Date:** 2026-06-19
+**UAT tests:** 1, 2
+
+## Symptoms
+
+- `GET /api/cloud/connections` raises `psycopg.errors.UndefinedColumn` for `cloud_connections.display_name_override`.
+- `POST /api/cloud/connections/webdav` reaches `_upsert_cloud_connection` and raises the same error, preventing Nextcloud account connection.
+
+## Root Cause
+
+The running application code and ORM are at Phase 12, but the live PostgreSQL schema remains at Alembic revision `0005`.
+
+Live evidence:
+
+```text
+$ docker compose run --rm backend alembic current
+0005
+```
+
+Migration `backend/migrations/versions/0006_cloud_resource_foundation.py` correctly adds `cloud_connections.display_name_override` and the Phase 12 cloud metadata tables. The ORM correctly maps that column. The defect is that `docker-compose.yml` has no migration service and the backend command starts Uvicorn directly. Backend, Celery worker, and Celery beat depend on PostgreSQL health, but none depends on `alembic upgrade head` completing. Therefore `docker compose up` can run new application code against an old persistent database.
+
+## Why Automated Tests Missed It
+
+- Most backend tests build schema directly from `Base.metadata.create_all`, which validates the final ORM shape but bypasses Alembic history and deployment ordering.
+- Existing Alembic tests target early migrations and do not exercise an existing PostgreSQL database upgraded from `0005` to `head`.
+- Phase verification checked migration file structure and model parity, not a real cold-start Compose upgrade path.
+
+## Files Involved
+
+- `docker-compose.yml` — no one-shot migration service; backend/workers start after DB health only.
+- `backend/migrations/versions/0006_cloud_resource_foundation.py` — contains the required column and tables but was not applied.
+- `backend/db/models.py` — queries `display_name_override`, exposing schema drift immediately.
+- `backend/tests/test_alembic.py` — lacks a 0005-to-head PostgreSQL upgrade regression.
+- `.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md` — cold-start test expected migration completion, but execution evidence did not actually verify Compose migration orchestration.
+
+## Required Fix Direction
+
+1. Add a one-shot Compose migration service that runs `alembic upgrade head` with `DATABASE_MIGRATE_URL` after PostgreSQL becomes healthy.
+2. Make backend, Celery worker, and Celery beat wait for that migration service to complete successfully.
+3. Add a PostgreSQL/Compose regression proving an existing revision `0005` advances to `0006/head` before the API starts, and assert `display_name_override`, `cloud_items`, `cloud_item_topics`, and `cloud_folder_states` exist.
+4. Document the migration lifecycle and recovery command in README/RUNBOOK.
+5. For the currently running environment, apply `alembic upgrade head` and restart application processes before resuming UAT.
+
+## Scope
+
+Both UAT blockers share this root cause. No evidence currently indicates a separate Nextcloud credential or WebDAV defect.

.planning/phases/12-cloud-resource-foundation/12-05-PLAN.md

@@ -0,0 +1,174 @@
+---
+phase: "12"
+plan: "05"
+type: gap_closure
+wave: 1
+depends_on:
+  - "12-04"
+files_modified:
+  - docker-compose.yml
+  - backend/tests/test_compose_migrations.py
+  - backend/tests/test_migration_0006.py
+  - backend/main.py
+  - frontend/package.json
+  - frontend/package-lock.json
+  - AGENTS.md
+  - README.md
+  - RUNBOOK.md
+  - SECURITY.md
+autonomous: true
+requirements:
+  - CONN-04
+  - CLOUD-01
+
+must_haves:
+  truths:
+    - "A clean `docker compose up` applies Alembic through head before backend, Celery worker, or Celery beat starts"
+    - "An existing database at revision 0005 upgrades to 0006 and contains display_name_override plus all Phase 12 cloud metadata tables"
+    - "GET /api/cloud/connections no longer fails with UndefinedColumn after a normal Compose restart"
+    - "POST /api/cloud/connections/webdav can persist a valid Nextcloud connection after migration"
+    - "A failed migration blocks application process startup instead of serving with schema drift"
+  artifacts:
+    - path: "docker-compose.yml"
+      provides: "One-shot Alembic migration service and service_completed_successfully dependency gate"
+      contains: "alembic upgrade head"
+    - path: "backend/tests/test_compose_migrations.py"
+      provides: "Static/Compose configuration regression for migration ordering"
+      contains: "service_completed_successfully"
+    - path: "backend/tests/test_migration_0006.py"
+      provides: "PostgreSQL upgrade regression from revision 0005 to head"
+      contains: "display_name_override"
+    - path: "RUNBOOK.md"
+      provides: "Normal migration lifecycle, revision verification, and schema-drift recovery"
+      contains: "alembic current"
+  key_links:
+    - from: "docker-compose.yml migrate service"
+      to: "backend/celery-worker/celery-beat services"
+      via: "depends_on migrate condition service_completed_successfully"
+      pattern: "service_completed_successfully"
+    - from: "backend/migrations/versions/0006_cloud_resource_foundation.py"
+      to: "backend/tests/test_migration_0006.py"
+      via: "real PostgreSQL revision and information_schema assertions"
+      pattern: "0005.*0006|display_name_override"
+---
+
+<objective>
+Close the Phase 12 UAT blockers caused by application/schema drift. Make database migration a mandatory Compose startup gate, add a real upgrade regression, repair the running environment, and verify both cloud connection listing and Nextcloud connection creation before UAT resumes.
+
+Root cause: `.planning/debug/phase-12-cloud-schema-cold-start.md` proves the live database remained at Alembic `0005` because Compose started application processes without running `alembic upgrade head`.
+</objective>
+
+<execution_context>
+@$HOME/.claude/get-shit-done/workflows/execute-plan.md
+@$HOME/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/phases/12-cloud-resource-foundation/12-UAT.md
+@.planning/debug/phase-12-cloud-schema-cold-start.md
+@.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md
+@backend/migrations/versions/0006_cloud_resource_foundation.py
+@docker-compose.yml
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Add a migration-gated Compose startup path</name>
+  <files>docker-compose.yml, backend/tests/test_compose_migrations.py</files>
+  <read_first>
+    - docker-compose.yml — current backend/worker/beat dependencies, hardening, volumes, and environment
+    - backend/migrations/env.py — runtime `DATABASE_MIGRATE_URL` selection
+    - backend/Dockerfile — image user/workdir and Alembic availability
+    - .planning/debug/phase-12-cloud-schema-cold-start.md — confirmed deployment root cause
+  </read_first>
+  <action>
+    Add a one-shot `migrate` service built from `./backend` that runs `alembic upgrade head` from `/app`, receives `DATABASE_MIGRATE_URL` and only migration-required environment, mounts backend source consistently for development, waits for PostgreSQL health (and any migration-time MinIO prerequisite required by revision 0004), and uses the same non-root/read-only/tmpfs/cap-drop/no-new-privileges hardening as backend. Change backend, celery-worker, and celery-beat so each has `depends_on.migrate.condition: service_completed_successfully` in addition to its runtime dependencies. Do not run Alembic independently in every replica and do not grant DDL privileges to `docuvault_app`. Add focused tests that load/render the Compose configuration and assert the migrate command, migrate-role URL, one-shot dependency, and absence of `alembic upgrade` from ordinary backend/worker commands. Include a failure-path assertion showing a nonzero migration exit prevents dependent services from starting.
+  </action>
+  <acceptance_criteria>
+    - `docker compose config` succeeds and contains exactly one migration service command `alembic upgrade head`
+    - backend, celery-worker, and celery-beat each wait for successful migration completion
+    - migration service uses `DATABASE_MIGRATE_URL`; application services continue using `DATABASE_URL`
+    - migration service preserves container hardening and stores no credentials in source
+    - `pytest -q tests/test_compose_migrations.py` passes
+  </acceptance_criteria>
+  <verify><automated>docker compose config >/tmp/docuvault-compose.yml && docker compose run --rm backend pytest -q tests/test_compose_migrations.py</automated></verify>
+  <done>Compose has one hardened migration gate, every application process depends on it, and configuration tests pass.</done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Prove PostgreSQL upgrade from 0005 to Phase 12 head</name>
+  <files>backend/tests/test_migration_0006.py, RUNBOOK.md</files>
+  <read_first>
+    - backend/migrations/versions/0005_system_settings.py — starting revision
+    - backend/migrations/versions/0006_cloud_resource_foundation.py — required upgrade objects
+    - backend/tests/test_alembic.py — existing migration tests and their limitations
+    - docker/postgres/initdb.d/01-init-users.sql — migrate/app role privileges
+    - RUNBOOK.md — existing database operations guidance
+  </read_first>
+  <action>
+    Add an integration test that uses a disposable PostgreSQL database or isolated test schema, applies Alembic through `0005`, then upgrades to head. Assert `alembic_version` is `0006`; `cloud_connections.display_name_override` exists; `cloud_items`, `cloud_item_topics`, and `cloud_folder_states` exist; named unique constraints/indexes exist; and `docuvault_app` can perform intended DML but not DDL. The test must never downgrade or mutate the developer's normal database and must skip with an explicit prerequisite message when a disposable integration database cannot be provisioned. Add a cold-start Compose test path suitable for CI that starts from an isolated project/volume, waits for migrate success, checks revision/head, exercises `/health`, and tears down only that isolated project. Document normal upgrade, revision inspection, failed-migration recovery, and the immediate `0005` to head remediation commands in RUNBOOK.
+  </action>
+  <acceptance_criteria>
+    - integration test proves a real 0005-to-0006/head transition without touching the normal development database
+    - all four Phase 12 schema additions are asserted against PostgreSQL catalog/information_schema
+    - migration failure is observable and prevents backend startup
+    - RUNBOOK gives exact safe commands for current revision, upgrade, logs, restart, and recovery
+    - focused migration tests pass in the Compose environment
+  </acceptance_criteria>
+  <verify><automated>docker compose run --rm -e INTEGRATION=1 backend pytest -q tests/test_migration_0006.py</automated></verify>
+  <done>A disposable PostgreSQL test upgrades from 0005 to head and proves the column/tables/privilege boundaries; operational recovery is documented.</done>
+</task>
+
+<task type="auto">
+  <name>Task 3: Repair the environment, rerun blocker paths, and close protocol</name>
+  <files>backend/main.py, frontend/package.json, frontend/package-lock.json, AGENTS.md, README.md, RUNBOOK.md, SECURITY.md</files>
+  <read_first>
+    - .planning/phases/12-cloud-resource-foundation/12-UAT.md — two confirmed blockers and four blocked tests
+    - backend/api/cloud/connections.py — failing list and WebDAV/Nextcloud paths
+    - AGENTS.md — bug regression, documentation, version, security, commit, and push protocols
+    - SECURITY.md — Phase 12 threat register and gate evidence
+  </read_first>
+  <action>
+    Apply `alembic upgrade head` through the new migration service in the current Compose environment, verify `alembic current` reports `0006`, and restart/recreate backend plus Celery processes through the gated path. Add API regressions against PostgreSQL for `GET /api/cloud/connections` and valid mocked/controlled `POST /api/cloud/connections/webdav`, asserting neither raises UndefinedColumn and the connection response excludes credentials. Run focused cloud/migration tests, full backend/frontend suites, frontend build, Bandit, npm audit, and available pip-audit/secret checks; resolve introduced high-severity findings. Bump the patch version from 0.2.0 to 0.2.1 in tracked version sources. Update AGENTS current state to Phase 12 UAT schema-drift gap closure, README startup behavior, RUNBOOK final commands, and SECURITY evidence. Commit and push atomically under `fix(12-uat): gate startup on database migrations`.
+  </action>
+  <acceptance_criteria>
+    - live `alembic current` reports 0006/head before backend starts
+    - connection list returns successfully for an authenticated user
+    - Nextcloud/WebDAV connection creation reaches normal validation/persistence without schema exceptions
+    - focused regression, full suites, build, and mandatory security gates pass or document only pre-existing external blockers
+    - backend and frontend tracked versions equal 0.2.1; docs accurately describe migration-gated startup
+  </acceptance_criteria>
+  <verify><automated>docker compose run --rm migrate && docker compose run --rm backend alembic current && docker compose run --rm backend pytest -q tests/test_compose_migrations.py tests/test_migration_0006.py tests/test_cloud.py tests/test_cloud_security.py && cd frontend && npm test && npm run build</automated></verify>
+  <done>The running schema is at head, both UAT blocker paths work, regression/security gates pass, version/docs are 0.2.1, and Phase 12 UAT can resume at Test 1.</done>
+</task>
+
+</tasks>
+
+<threat_model>
+| Threat ID | Threat | Mitigation and evidence |
+|-----------|--------|-------------------------|
+| T-12-05-01 | App starts against stale schema | One-shot migrate service and successful-completion dependency gate |
+| T-12-05-02 | DDL privilege escalation | Migration uses migrate URL; app retains DML-only URL and privilege regression test |
+| T-12-05-03 | Concurrent migrations | Exactly one Compose migration service; replicas only wait on completion |
+| T-12-05-04 | Secrets exposed in Compose/tests | Environment references only; no rendered config or logs committed with values |
+| T-12-05-05 | Test destroys developer DB | Disposable database/schema/project requirement; explicit skip rather than fallback to normal DB |
+| T-12-05-SC | Dependency risk | No new runtime dependencies; standard security/audit gates rerun |
+</threat_model>
+
+<verification>
+1. GSD plan structure validator reports zero errors/warnings.
+2. Compose config proves migration dependency ordering and hardening.
+3. Disposable PostgreSQL upgrade proves 0005 -> 0006/head and schema objects.
+4. Live revision and both failed API paths pass after gated restart.
+5. Full tests, build, and security gates complete before UAT resumes.
+</verification>
+
+<success_criteria>
+- Normal startup cannot serve application traffic with unapplied migrations.
+- The exact `display_name_override` regression is caught automatically.
+- Current Nextcloud and connection-list blockers are resolved at their deployment root cause.
+- Documentation and versioning reflect the gap closure.
+</success_criteria>
+
+<output>Create `.planning/phases/12-cloud-resource-foundation/12-05-SUMMARY.md` when complete.</output>

.planning/phases/12-cloud-resource-foundation/12-UAT.md

@@ -0,0 +1,96 @@
+---
+status: diagnosed
+phase: 12-cloud-resource-foundation
+source:
+  - 12-01-SUMMARY.md
+  - 12-02-SUMMARY.md
+  - 12-03-SUMMARY.md
+  - 12-04-SUMMARY.md
+started: 2026-06-19T20:21:31+02:00
+updated: 2026-06-19T20:28:00+02:00
+---
+
+## Current Test
+
+[testing paused — 4 items blocked by cloud provider connection failure]
+
+## Tests
+
+### 1. Cold Start Smoke Test
+expected: Start DocuVault from a clean stopped state. PostgreSQL migration 0006 completes, the backend and frontend become healthy without startup errors, and the authenticated Cloud Storage page loads normally.
+result: issue
+reported: "GET /api/cloud/connections raises an ASGI exception: psycopg.errors.UndefinedColumn because cloud_connections.display_name_override does not exist in the live PostgreSQL schema."
+severity: blocker
+
+### 2. Connection Roots and Custom Names
+expected: Cloud Storage shows each connected account as its own top-level row. Two accounts from the same provider remain distinct; default duplicate names are disambiguated, and renaming one connection in Settings updates only that connection.
+result: issue
+reported: "Cannot connect Nextcloud test users; POST /api/cloud/connections/webdav raises psycopg.errors.UndefinedColumn because cloud_connections.display_name_override does not exist."
+severity: blocker
+
+### 3. Shared Cloud Folder Browser
+expected: Opening a connection shows its files and folders in the same StorageBrowser layout used for local documents. The breadcrumb begins with the connection name, folder navigation works, and provider IDs or technical metadata do not clutter normal rows.
+result: blocked
+blocked_by: server
+reason: "Cannot test because cloud provider connection fails with the live database schema error."
+
+### 4. Unsupported Action Explanations
+expected: Unsupported cloud actions remain visible in their normal positions but appear gray; temporarily unavailable actions appear amber. Hover or keyboard focus shows the specific explanation, and tapping/clicking an unavailable action explains it without executing it.
+result: blocked
+blocked_by: server
+reason: "Cannot reach a connected cloud folder while provider connection creation is failing."
+
+### 5. Cached-First Refresh and Warning State
+expected: Revisiting a previously browsed folder shows saved metadata immediately while a subtle refresh indicator runs. If provider refresh fails, cached rows remain usable and a warning reports the last successful refresh and automatic retry instead of replacing the folder with an error.
+result: blocked
+blocked_by: server
+reason: "Cannot establish the cloud connection needed to seed and revisit provider metadata."
+
+### 6. Session-Only Folder Memory
+expected: Leaving and reopening a connection in the same browser session resumes its last folder. Starting a fresh tab/session begins at the connection root, and no credentials or document content are stored in browser session storage.
+result: blocked
+blocked_by: server
+reason: "Cannot enter a connected cloud folder while provider connection creation is failing."
+
+## Summary
+
+total: 6
+passed: 0
+issues: 2
+pending: 0
+skipped: 0
+blocked: 4
+
+## Gaps
+
+- truth: "A clean start applies migration 0006 and the authenticated Cloud Storage page loads without backend schema errors."
+  status: failed
+  reason: "User reported: GET /api/cloud/connections raises psycopg.errors.UndefinedColumn because cloud_connections.display_name_override does not exist in the live PostgreSQL schema."
+  severity: blocker
+  test: 1
+  root_cause: "The live database is at Alembic revision 0005 because docker-compose.yml starts backend/workers directly and has no migration service or dependency that runs alembic upgrade head. ORM metadata expects revision 0006."
+  artifacts:
+    - path: "docker-compose.yml"
+      issue: "No migration service; backend and Celery services depend only on PostgreSQL health."
+    - path: "backend/migrations/versions/0006_cloud_resource_foundation.py"
+      issue: "Required migration exists but is not applied during Compose startup."
+    - path: "backend/tests/test_alembic.py"
+      issue: "No PostgreSQL regression covering upgrade from revision 0005 to head."
+  missing:
+    - "Run alembic upgrade head before backend and worker startup."
+    - "Add cold-start migration regression and deployment documentation."
+  debug_session: ".planning/debug/phase-12-cloud-schema-cold-start.md"
+- truth: "A user can connect Nextcloud accounts and each connected account appears as an independently renameable cloud root."
+  status: failed
+  reason: "User reported: Cannot connect Nextcloud test users; POST /api/cloud/connections/webdav fails because cloud_connections.display_name_override does not exist."
+  severity: blocker
+  test: 2
+  root_cause: "Same schema-drift root cause as Test 1: Nextcloud connection creation queries the Phase 12 ORM while PostgreSQL remains at revision 0005."
+  artifacts:
+    - path: "docker-compose.yml"
+      issue: "No migration gate before the cloud connection API starts."
+    - path: "backend/api/cloud/connections.py"
+      issue: "Correctly queries CloudConnection, which exposes the unapplied schema immediately."
+  missing:
+    - "Apply revision 0006/head before accepting cloud connection requests."
+  debug_session: ".planning/debug/phase-12-cloud-schema-cold-start.md"