--- phase: 06-performance-production-hardening plan: 06 type: execute wave: 3 depends_on: - 06-04 - 06-05 files_modified: - RUNBOOK.md autonomous: false requirements: - D-10 - D-14 user_setup: - service: docker-hub why: "docker scout cves requires an authenticated Docker Hub session to submit the image manifest for CVE analysis (Pitfall 5)" env_vars: [] dashboard_config: - task: "Run `docker login` from the terminal where the security gate will be invoked" location: "Local shell / CI environment that will run docker scout cves" must_haves: truths: - "RUNBOOK.md exists at the repo root and is the single operational reference for Phase 6 deployment + on-call response" - "RUNBOOK.md documents the docker scout cves invocation with zero-critical-CVEs gate (D-10)" - "RUNBOOK.md enumerates every required env var with description, example value, and source (.env vs Docker secret vs cloud provider dashboard)" - "RUNBOOK.md documents startup/shutdown procedures for the full docker-compose stack including the new Loki/Promtail/Grafana services from 06-02" - "RUNBOOK.md documents backup strategy: pg_dump for Postgres + mc mirror for MinIO, with restore commands" - "RUNBOOK.md documents health-check verification: /health endpoint, Loki :3100/ready, Grafana :3000/api/health, MinIO mc ready, redis-cli ping" - "RUNBOOK.md documents on-call escalation: who to contact, in what order, for which alert types" - "RUNBOOK.md documents common failure modes from RESEARCH.md Pitfalls 1, 5, 6, 7 with recovery commands" - "A human verifies the docker scout cves run returns zero critical CVEs on the built image AFTER 06-04 ships the hardened Dockerfile — this is the final phase gate" artifacts: - path: "RUNBOOK.md" provides: "Operational runbook — env vars, startup/shutdown, backup, health checks, escalation, failure modes" contains: "docker scout cves" min_lines: 200 key_links: - from: "RUNBOOK.md docker scout section" to: "06-04 hardened Dockerfile" via: "image tag built by 06-04 is the scan target" pattern: "docker scout cves" - from: "RUNBOOK.md backup section" to: "docker-compose.yml postgres + minio services" via: "pg_dump and mc mirror against the volumes named in compose" pattern: "pg_dump|mc mirror" - from: "RUNBOOK.md health-check section" to: "06-02 Loki/Grafana endpoints + existing /health" via: "documented HTTP probes per service" pattern: "loki.*3100|grafana.*3000" --- Close Phase 6 with the two remaining decisions: run the `docker scout cves` zero-critical-CVE gate against the hardened image (D-10), and ship `RUNBOOK.md` at the repo root as the single operational reference (D-14). This plan is the final gate before the phase is marked complete in STATE.md. Purpose: The hardened Dockerfile (06-04) and rate-limiting changes (06-05) are inert without (a) proof the resulting image has no critical CVEs and (b) a written runbook so a fresh operator can stand up the stack, find the right env var, and respond to a page at 03:00 without reading PLAN.md. RUNBOOK.md is the artifact that turns "we hardened it" into "anyone on-call can run it." Output: One RUNBOOK.md file at the repo root covering env vars, startup/shutdown, backups, health checks, escalation, and failure modes — plus a human-executed docker scout cves gate whose pass/fail decides whether the phase ships. @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md @.planning/phases/06-performance-production-hardening/06-CONTEXT.md @.planning/phases/06-performance-production-hardening/06-RESEARCH.md @.planning/phases/06-performance-production-hardening/06-PATTERNS.md @.planning/phases/06-performance-production-hardening/06-04-PLAN.md @.planning/phases/06-performance-production-hardening/06-05-PLAN.md @CLAUDE.md @docker-compose.yml @backend/Dockerfile @backend/config.py D-14 required sections (verbatim from CONTEXT.md): 1. All required env vars with descriptions and examples 2. Docker Compose startup/shutdown procedures 3. Backup strategy: PostgreSQL (pg_dump cron) + MinIO (mc mirror) 4. Health check verification steps 5. On-call escalation path (who, in what order, for which alert types) 6. Common failure modes and recovery steps Env var source (read backend/config.py — Settings class fields): - DATABASE_URL, DATABASE_MIGRATE_URL - MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MINIO_BUCKET, MINIO_PUBLIC_ENDPOINT - REDIS_URL, REDIS_PASSWORD - SECRET_KEY, REFRESH_TOKEN_SECRET (per Phase 2) - ADMIN_EMAIL, ADMIN_PASSWORD - CORS_ORIGINS, FRONTEND_URL - CLOUD_CREDS_KEY (Phase 5) - google_client_id / google_client_secret, onedrive_client_id / onedrive_client_secret (Phase 5) - LOG_LEVEL, LOG_JSON (Phase 6 — added by 06-02) - TRUSTED_PROXY_CIDRS (Phase 6 — referenced by 06-05; default list lives in deps/utils.py) - LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD (Phase 6 — added by 06-03) Health-check endpoints to document: - Backend: GET http://localhost:8000/health (existing endpoint) - Loki: GET http://localhost:3100/ready - Grafana: GET http://localhost:3000/api/health - MinIO: `docker compose exec minio mc ready local` - Redis: `docker compose exec redis redis-cli -a $REDIS_PASSWORD ping` (expect "PONG") - Postgres: `docker compose exec postgres pg_isready -U postgres -d docuvault` docker scout cves gate command (from RESEARCH.md Code Examples + Pitfall 5): Build: `cd backend && docker build -t docuvault-backend:phase6 .` (already produced by 06-04) Scan: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code` Exit code 0 = clean; exit code 2 = critical CVEs found Prerequisite: `docker login` (Pitfall 5 — scout uploads manifest to Docker's service) Fallback if docker scout is unavailable: `trivy image docuvault-backend:phase6` (note: trivy not installed by default — RUNBOOK documents brew install trivy) Failure-mode entries to include (from RESEARCH.md Pitfalls): - Pitfall 1: tempfile.NamedTemporaryFile fails → check /tmp tmpfs mode=1777 - Pitfall 5: docker scout returns auth error → run docker login - Pitfall 6: PyMuPDF read-only error → may need additional tmpfs mount for /var/cache/fontconfig - Pitfall 7: celery-beat won't start under read_only → confirm celery-beat block has no read_only key On-call escalation template (since this is a solo dev project, document the realistic pattern): - Primary: project owner (curo1305@curonet.de) — first responder for all alerts - Secondary: empty (single-operator deployment) — RUNBOOK acknowledges escalation tier is the operator themselves; documents alert types and the recovery commands directly so the owner can self-serve - Alert classes: backend-down (page immediately), celery-stuck (warn within 1h), loki-full-disk (warn within 4h), docker scout critical CVE (page within 24h) RUNBOOK location: repo root, sibling to CLAUDE.md and README.md (D-14 verbatim). Task 1: Write RUNBOOK.md at repo root RUNBOOK.md - .planning/phases/06-performance-production-hardening/06-CONTEXT.md (D-14 — full content list) - .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pitfalls 1, 5, 6, 7; Standard Stack; Environment Availability) - .planning/phases/06-performance-production-hardening/06-PATTERNS.md (interfaces section in this plan summarises env vars) - backend/config.py (authoritative list of env vars — every field in Settings class must appear in RUNBOOK with its env var name and default) - docker-compose.yml (services, ports, volumes, healthchecks — RUNBOOK references all of them) - backend/Dockerfile (hardened image tag from 06-04 — RUNBOOK uses it as scan target) - CLAUDE.md ("Stack" + "Development Setup" + "Security Protocol" — RUNBOOK aligns with these conventions) Create RUNBOOK.md at the repository root (sibling to CLAUDE.md, README.md, docker-compose.yml). Top of file: H1 `# DocuVault Operational Runbook`. One-paragraph purpose statement: this is the single reference for running DocuVault in production-like environments — env vars, startup, backups, health checks, escalation, recovery. Cross-reference: "For architecture and rationale see CLAUDE.md; for phase history see .planning/ROADMAP.md." Section 1 — `## Environment Variables`. Markdown table with columns: Name | Required | Description | Example | Source. Enumerate every Settings field from backend/config.py: DATABASE_URL, DATABASE_MIGRATE_URL, MINIO_*, REDIS_*, SECRET_KEY, REFRESH_TOKEN_SECRET (if defined), ADMIN_EMAIL, ADMIN_PASSWORD, CORS_ORIGINS, FRONTEND_URL, CLOUD_CREDS_KEY, google_client_id/secret, onedrive_client_id/secret, LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS, LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD. For each row provide a realistic example (e.g. `postgresql+psycopg://docuvault_app:****@postgres:5432/docuvault`) and where it's set (`.env` file at repo root). Note: SECRET_KEY and CLOUD_CREDS_KEY must NEVER be committed; document the rotation procedure. Section 2 — `## Startup / Shutdown`. Document the docker-compose startup sequence (`docker compose up -d --build`) and orderly shutdown (`docker compose down`, `docker compose down -v` for nuke-from-orbit including volumes). Reference the new services from 06-02 (loki, promtail, grafana) and their access URLs (Grafana :3000, Loki :3100). Include the dev-mode reload note (bind mount + --reload override). Document the celery-beat exception (Pitfall 7) — it does NOT have read_only hardening. Section 3 — `## Backup Strategy`. Postgres: command `docker compose exec postgres pg_dump -U postgres docuvault > backups/docuvault_$(date +%F).sql`; restore command `cat backups/docuvault_YYYY-MM-DD.sql | docker compose exec -T postgres psql -U postgres docuvault`. Document cron pattern for daily backups: a crontab line like `0 3 * * * cd /path/to/docuvault && docker compose exec -T postgres pg_dump -U postgres docuvault | gzip > backups/$(date +\%F).sql.gz`. Note: D-14 says backup strategy must be documented; automation is deferred per Deferred Idea "Backup automation". MinIO: command `docker compose exec minio mc mirror /data /backup/minio-mirror` (mc client is installed inside the minio container). Document offsite-backup pattern using `mc mirror local-minio/ s3://offsite-bucket/` against an S3-compatible target. Restore command: `mc mirror /backup/minio-mirror /data`. Document retention policy: keep daily for 7 days, weekly for 4 weeks, monthly for 12 months — operator implements via cron + rotate. Section 4 — `## Health Checks`. Markdown table with Service | Probe Command | Healthy Output: Backend `curl -sf http://localhost:8000/health` returns `{"status":"ok",...}` Loki `curl -sf http://localhost:3100/ready` returns `ready` Grafana `curl -sf http://localhost:3000/api/health` returns JSON with `database: ok` Postgres `docker compose exec postgres pg_isready -U postgres -d docuvault` exit 0 MinIO `docker compose exec minio mc ready local` exit 0 Redis `docker compose exec redis redis-cli -a $REDIS_PASSWORD ping` returns `PONG` Celery worker `docker compose exec celery-worker celery -A celery_app inspect ping` returns `pong` Section 5 — `## Security Gate — docker scout CVE scan (D-10)`. Prerequisite: `docker login` (Pitfall 5 — scout uploads image manifest to Docker's analysis service). Build step (already done by 06-04): `cd backend && docker build -t docuvault-backend:phase6 .` Scan command: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code` Pass criterion: exit code 0. Fail criterion: exit code 2 (critical CVEs found). Phase advancement requires pass. Fallback if `docker scout` is unavailable (e.g. offline environment): `brew install trivy` (or `apt install trivy`), then `trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6`. Cadence: run on every image rebuild; gate any prod deploy on pass. On failure recovery: re-pull base image (`docker pull python:3.12-slim`), rebuild, rescan. If still failing, pin to a newer Python 3.12 patch release or rebuild against the latest python:3.13-slim after compatibility testing. Section 6 — `## On-Call Escalation`. State the realistic single-operator model: primary on-call is the project owner (curo1305@curonet.de); secondary escalation is the operator themselves; this is a solo deployment. Document the alert classes and the immediate action for each: - `backend-down` (HTTP 5xx > 5% for 5min OR backend container restarting) → page immediately. Action: `docker compose logs --tail 100 backend`; check for `Read-only file system` (Pitfall 1/6) or `Permission denied` (Pitfall 5). If extractor errors, see Section 7 → "tempfile permission" entry. - `celery-stuck` (queue depth > 1000 OR no task completion for 30min) → warn within 1h. Action: `docker compose logs --tail 200 celery-worker`; restart `docker compose restart celery-worker`. - `loki-full-disk` (Loki container reports > 80% disk usage on /loki) → warn within 4h. Action: prune old chunks (Loki retention tuning — link to upstream docs); or rotate the loki_data volume. - `docker scout critical-CVE` (scheduled re-scan finds new critical CVE in base image) → page within 24h. Action: rebuild image with latest python:3.12-slim and re-scan; if persistent, see Section 5 fallback. - `quota-violation-spike` (audit log shows > 10 quota_exceeded events per hour from same user) → warn. Action: check user; possible abuse — disable account via admin endpoint. - `failed-login-spike` (audit log shows > 100 failed_login events per hour from same IP) → warn. Action: verify per-IP limiter is firing (10/min); if not, check that 06-05 deployed and TRUSTED_PROXY_CIDRS is correct for the deployment topology. Section 7 — `## Common Failure Modes`. Markdown table or H3 entries — each with Symptom | Cause | Recovery: - Symptom: "PermissionError: [Errno 13] /tmp/..." in backend logs → Cause: tmpfs missing mode=1777 (Pitfall 1) → Recovery: verify docker-compose.yml backend.tmpfs value contains "/tmp:mode=1777"; recreate the container with `docker compose up -d --force-recreate backend`. - Symptom: docker scout returns "authentication required" → Cause: Pitfall 5 → Recovery: run `docker login` first, then rerun the scan. - Symptom: PyMuPDF "Read-only file system" when extracting PDF → Cause: Pitfall 6 — PyMuPDF writing to /var/cache/fontconfig under read_only → Recovery: add `tmpfs: ["/var/cache/fontconfig"]` to backend service in docker-compose.yml. - Symptom: celery-beat container exits "Permission denied: 'celerybeat-schedule'" → Cause: Pitfall 7 — read_only was accidentally added to celery-beat → Recovery: remove `read_only:` from celery-beat block in docker-compose.yml; restart. - Symptom: After 06-05 ships, requests from external clients suddenly hit rate-limit despite low traffic → Cause: a real reverse proxy is in front but its IP is not in TRUSTED_PROXY_CIDRS, so its peer IP is used as the rate-limit key for all traffic → Recovery: add the proxy's CIDR to TRUSTED_PROXY_CIDRS env var and restart backend. - Symptom: per-account 429s when expected per-IP 429s → Cause: route handler missing the `request.state.current_user = current_user` first-line assignment → Recovery: locate the handler missing the line, add it as the first executable statement, redeploy. - Symptom: Grafana UI loads but no Loki datasource → Cause: Loki container not ready when Grafana started OR loki-config.yaml malformed → Recovery: check `curl http://localhost:3100/ready`; if not 200, inspect `docker compose logs loki` for YAML errors. Section 8 — `## Phase 6 Deferred Items`. List the four deferred ideas from CONTEXT.md so the next operator knows the runbook acknowledges them: HTTPS/TLS termination (add reverse proxy — pattern documented in Section 5), horizontal scaling (Redis-backed limiter — Phase 7), CI/CD pipeline (GitHub Actions for scout + locust — Phase 7), backup automation (cron service — manual procedure above is the current state). Format requirements: GitHub-flavored Markdown, line wrap at ~100 chars where possible, consistent code-fence language tags (`bash` for shell, `yaml` for compose snippets, `text` for output samples). Add a one-line "Last updated: {today}" footer. test -f RUNBOOK.md && wc -l RUNBOOK.md | awk '{print ($1 >= 200) ? "OK" : "TOO_SHORT"}' && grep -cE "docker scout cves|pg_dump|mc mirror|TRUSTED_PROXY_CIDRS|LOG_JSON|celery-beat" RUNBOOK.md - `test -f RUNBOOK.md` exits 0 (file present at repo root). - `wc -l RUNBOOK.md` returns ≥ 200 lines. - All eight section headings present: `grep -cE "^##\\s+(Environment Variables|Startup|Backup Strategy|Health Checks|Security Gate|On-Call Escalation|Common Failure Modes|Phase 6 Deferred Items)" RUNBOOK.md` returns 8. - Docker scout command appears: `grep -c "docker scout cves" RUNBOOK.md` returns ≥ 1. - All required env vars appear by name (test 12 representative): `grep -cE "DATABASE_URL|MINIO_ENDPOINT|REDIS_URL|SECRET_KEY|CLOUD_CREDS_KEY|LOG_LEVEL|LOG_JSON|TRUSTED_PROXY_CIDRS|CORS_ORIGINS|FRONTEND_URL|ADMIN_EMAIL|LOAD_TEST_EMAIL" RUNBOOK.md` returns ≥ 12. - Backup commands present: `grep -c "pg_dump" RUNBOOK.md` returns ≥ 1 AND `grep -c "mc mirror" RUNBOOK.md` returns ≥ 1. - Health-check commands present for at least 6 services: `grep -cE "/health|/ready|/api/health|pg_isready|mc ready|redis-cli.*ping" RUNBOOK.md` returns ≥ 6. - Pitfall recoveries documented: `grep -cE "mode=1777|docker login|fontconfig|celerybeat-schedule|TRUSTED_PROXY_CIDRS|request\\.state\\.current_user" RUNBOOK.md` returns ≥ 6. - Escalation section present with at least 4 alert classes: `grep -cE "backend-down|celery-stuck|loki-full-disk|critical-CVE" RUNBOOK.md` returns ≥ 4. - Cross-reference to CLAUDE.md present: `grep -c "CLAUDE.md" RUNBOOK.md` returns ≥ 1. - Markdown is parseable (basic check — no unbalanced code fences): `awk '/^```/ {n++} END {exit (n % 2 == 0) ? 0 : 1}' RUNBOOK.md` exits 0. RUNBOOK.md present at repo root, ≥200 lines, all 8 D-14 content areas covered, every Phase 6 env var documented with example, all major Pitfall recoveries entered into Failure Modes section. Task 2: Run docker scout cves zero-critical gate (D-10) - .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Code Examples — docker scout cves command; Pitfall 5) - .planning/phases/06-performance-production-hardening/06-04-PLAN.md (Task 1 — produces the image tag docuvault-backend:phase6) - RUNBOOK.md (Section 5 — Security Gate — the runbook is the source of truth for the command and prerequisite) Plan 06-04 produces a hardened multi-stage backend image tagged `docuvault-backend:phase6` (uid=1000, no build tools, read-only-friendly). This checkpoint runs the D-10 zero-critical-CVE gate against it. Pitfall 5 requires `docker login` before scout can submit the image manifest for analysis. 1. Confirm 06-04 has shipped: `docker image inspect docuvault-backend:phase6 >/dev/null 2>&1 && echo IMAGE_PRESENT` should print `IMAGE_PRESENT`. If absent, run `cd backend && docker build -t docuvault-backend:phase6 .` first. 2. Authenticate Docker Hub (Pitfall 5 prerequisite): `docker login` — enter Docker Hub credentials. Skip if already logged in. 3. Run the scan: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code` 4. Read the output. Expected pass: exit code 0 and the summary line shows `0 critical` (and ideally low numbers in High/Medium/Low). Expected fail: exit code 2 with a list of CRITICAL CVEs and their package origins. 5. If FAIL: identify the failing package(s) from the report, attempt remediation: a. Rebuild against latest base image: `docker pull python:3.12-slim` then `cd backend && docker build --no-cache -t docuvault-backend:phase6 .` and rerun step 3. b. If CVE comes from a Python package, bump the pin in backend/requirements.txt to a patched version and rebuild. c. If unresolvable in this phase, escalate to the operator with the CVE list — the phase cannot advance until critical CVEs are zero. 6. If docker scout is unavailable (offline / interpreter error), use the documented fallback: `trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6` (install trivy first per RUNBOOK Section 5). - User runs `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code` (or documented trivy fallback) and reports the exit code. - Exit code 0 → gate passes; respond "approved" with the summary line pasted (e.g. "0 critical, 2 high, 15 medium, 40 low"). - Exit code non-zero → gate fails; user pastes the CVE list. Planner triages: either bump a package pin (within this plan as a follow-up task) or escalate as a blocking issue that prevents phase completion. - User confirms they ran `docker login` first (or that trivy was used as the offline fallback) so the Pitfall 5 prerequisite is documented in the SUMMARY. Type "approved" with the pasted "X critical" summary line, OR paste the CVE list for triage. ## Trust Boundaries | Boundary | Description | |----------|-------------| | Operator → production stack | Operator runs the documented commands; RUNBOOK is the contract for what's safe to run | | Image base layers → runtime | Upstream python:3.12-slim and apt packages may ship CVEs over time; scout gate catches them | | Repository file → secrets exposure | RUNBOOK must reference env vars without hardcoding actual secrets | ## STRIDE Threat Register | Threat ID | Category | Component | Disposition | Mitigation Plan | |-----------|----------|-----------|-------------|-----------------| | T-06-06-01 | Tampering | Critical CVE in base image / dependency ships to production | mitigate | docker scout cves zero-critical gate (D-10) — human checkpoint blocks phase advancement on non-zero exit (Task 2) | | T-06-06-02 | Information Disclosure | RUNBOOK accidentally hardcodes a real secret value | mitigate | All example values use obvious placeholder patterns (****, CHANGEME, your-token-here); explicit warning that SECRET_KEY and CLOUD_CREDS_KEY must never be committed | | T-06-06-03 | Repudiation | Operator does not know which alert maps to which recovery procedure | mitigate | RUNBOOK Section 6 enumerates alert classes; Section 7 maps each Pitfall symptom to its recovery command — operator can self-serve at 03:00 | | T-06-06-04 | Tampering | docker scout requires Docker Hub auth (Pitfall 5) → gate silently fails open if operator skips login | mitigate | RUNBOOK Section 5 documents the prerequisite explicitly; Task 2 checkpoint instructions force the operator to confirm login before running scan; gate uses --exit-code so scan failures cannot be ignored | | T-06-06-05 | Denial of Service | Backup procedure not exercised → restore on production failure does not work | accept | D-14 says document the strategy; restore commands are present (Section 3); rehearsal cadence is operator responsibility documented as TODO in the runbook | | T-06-06-SC | Tampering | docker scout itself may be tampered with (binary integrity) | accept | docker scout ships with Docker Engine 29.5.2 (verified at research time); supply-chain risk delegated to Docker Inc.; trivy fallback exists if scout is compromised | - RUNBOOK.md exists at repo root with ≥200 lines and all 8 D-14 content sections. - `grep -c "docker scout cves" RUNBOOK.md` returns ≥ 1. - All Phase 6 env vars (LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS, LOAD_TEST_*) documented. - Pitfall recoveries (1, 5, 6, 7) entered into Failure Modes section. - Human-executed docker scout cves gate returns exit code 0 (or trivy equivalent with zero critical CVEs) — confirmed in Task 2 checkpoint. - Pitfall 5 prerequisite (docker login) executed before the scan; confirmation recorded in SUMMARY. - D-10 satisfied: docker scout cves run against docuvault-backend:phase6 returns zero critical CVEs (or fallback trivy scan does); result documented in 06-06-SUMMARY.md. - D-14 satisfied: RUNBOOK.md exists at repo root and covers env vars, startup/shutdown, backup strategy, health checks, on-call escalation, and common failure modes. - Phase 6 gate is green: all six decisions covered by plans 06-04/05/06 are implemented and verified, and the security scan blocks the phase from advancing if critical CVEs are present. Create `.planning/phases/06-performance-production-hardening/06-06-SUMMARY.md` when done. Include: RUNBOOK.md final line count and section headings list, the exact docker scout command run and its full output (or trivy equivalent), the pass/fail decision, any package pin bumps made to resolve CVEs, and a one-line confirmation that `docker login` was executed before the scan (Pitfall 5).