6 plans across 4 waves covering structlog/Loki observability, Locust load testing, multi-stage Dockerfile hardening, trusted-proxy rate limiting, and RUNBOOK.md. Verification passed (0 blockers). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
26 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, user_setup, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | user_setup | must_haves | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 06-performance-production-hardening | 06 | execute | 3 |
|
|
false |
|
|
|
Purpose: The hardened Dockerfile (06-04) and rate-limiting changes (06-05) are inert without (a) proof the resulting image has no critical CVEs and (b) a written runbook so a fresh operator can stand up the stack, find the right env var, and respond to a page at 03:00 without reading PLAN.md. RUNBOOK.md is the artifact that turns "we hardened it" into "anyone on-call can run it."
Output: One RUNBOOK.md file at the repo root covering env vars, startup/shutdown, backups, health checks, escalation, and failure modes — plus a human-executed docker scout cves gate whose pass/fail decides whether the phase ships.
<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md @.planning/phases/06-performance-production-hardening/06-RESEARCH.md @.planning/phases/06-performance-production-hardening/06-PATTERNS.md @.planning/phases/06-performance-production-hardening/06-04-PLAN.md @.planning/phases/06-performance-production-hardening/06-05-PLAN.md @CLAUDE.md @docker-compose.yml @backend/Dockerfile @backend/config.pyD-14 required sections (verbatim from CONTEXT.md):
- All required env vars with descriptions and examples
- Docker Compose startup/shutdown procedures
- Backup strategy: PostgreSQL (pg_dump cron) + MinIO (mc mirror)
- Health check verification steps
- On-call escalation path (who, in what order, for which alert types)
- Common failure modes and recovery steps
Env var source (read backend/config.py — Settings class fields):
- DATABASE_URL, DATABASE_MIGRATE_URL
- MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MINIO_BUCKET, MINIO_PUBLIC_ENDPOINT
- REDIS_URL, REDIS_PASSWORD
- SECRET_KEY, REFRESH_TOKEN_SECRET (per Phase 2)
- ADMIN_EMAIL, ADMIN_PASSWORD
- CORS_ORIGINS, FRONTEND_URL
- CLOUD_CREDS_KEY (Phase 5)
- google_client_id / google_client_secret, onedrive_client_id / onedrive_client_secret (Phase 5)
- LOG_LEVEL, LOG_JSON (Phase 6 — added by 06-02)
- TRUSTED_PROXY_CIDRS (Phase 6 — referenced by 06-05; default list lives in deps/utils.py)
- LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD (Phase 6 — added by 06-03)
Health-check endpoints to document:
- Backend: GET http://localhost:8000/health (existing endpoint)
- Loki: GET http://localhost:3100/ready
- Grafana: GET http://localhost:3000/api/health
- MinIO:
docker compose exec minio mc ready local - Redis:
docker compose exec redis redis-cli -a $REDIS_PASSWORD ping(expect "PONG") - Postgres:
docker compose exec postgres pg_isready -U postgres -d docuvault
docker scout cves gate command (from RESEARCH.md Code Examples + Pitfall 5):
Build: cd backend && docker build -t docuvault-backend:phase6 . (already produced by 06-04)
Scan: docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code
Exit code 0 = clean; exit code 2 = critical CVEs found
Prerequisite: docker login (Pitfall 5 — scout uploads manifest to Docker's service)
Fallback if docker scout is unavailable: trivy image docuvault-backend:phase6 (note: trivy not installed by default — RUNBOOK documents brew install trivy)
Failure-mode entries to include (from RESEARCH.md Pitfalls):
- Pitfall 1: tempfile.NamedTemporaryFile fails → check /tmp tmpfs mode=1777
- Pitfall 5: docker scout returns auth error → run docker login
- Pitfall 6: PyMuPDF read-only error → may need additional tmpfs mount for /var/cache/fontconfig
- Pitfall 7: celery-beat won't start under read_only → confirm celery-beat block has no read_only key
On-call escalation template (since this is a solo dev project, document the realistic pattern):
- Primary: project owner (curo1305@curonet.de) — first responder for all alerts
- Secondary: empty (single-operator deployment) — RUNBOOK acknowledges escalation tier is the operator themselves; documents alert types and the recovery commands directly so the owner can self-serve
- Alert classes: backend-down (page immediately), celery-stuck (warn within 1h), loki-full-disk (warn within 4h), docker scout critical CVE (page within 24h)
RUNBOOK location: repo root, sibling to CLAUDE.md and README.md (D-14 verbatim).
Task 1: Write RUNBOOK.md at repo root RUNBOOK.md - .planning/phases/06-performance-production-hardening/06-CONTEXT.md (D-14 — full content list) - .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pitfalls 1, 5, 6, 7; Standard Stack; Environment Availability) - .planning/phases/06-performance-production-hardening/06-PATTERNS.md (interfaces section in this plan summarises env vars) - backend/config.py (authoritative list of env vars — every field in Settings class must appear in RUNBOOK with its env var name and default) - docker-compose.yml (services, ports, volumes, healthchecks — RUNBOOK references all of them) - backend/Dockerfile (hardened image tag from 06-04 — RUNBOOK uses it as scan target) - CLAUDE.md ("Stack" + "Development Setup" + "Security Protocol" — RUNBOOK aligns with these conventions) Create RUNBOOK.md at the repository root (sibling to CLAUDE.md, README.md, docker-compose.yml). Top of file: H1 `# DocuVault Operational Runbook`. One-paragraph purpose statement: this is the single reference for running DocuVault in production-like environments — env vars, startup, backups, health checks, escalation, recovery. Cross-reference: "For architecture and rationale see CLAUDE.md; for phase history see .planning/ROADMAP.md."Section 1 — `## Environment Variables`. Markdown table with columns: Name | Required | Description | Example | Source. Enumerate every Settings field from backend/config.py: DATABASE_URL, DATABASE_MIGRATE_URL, MINIO_*, REDIS_*, SECRET_KEY, REFRESH_TOKEN_SECRET (if defined), ADMIN_EMAIL, ADMIN_PASSWORD, CORS_ORIGINS, FRONTEND_URL, CLOUD_CREDS_KEY, google_client_id/secret, onedrive_client_id/secret, LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS, LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD. For each row provide a realistic example (e.g. `postgresql+psycopg://docuvault_app:****@postgres:5432/docuvault`) and where it's set (`.env` file at repo root). Note: SECRET_KEY and CLOUD_CREDS_KEY must NEVER be committed; document the rotation procedure.
Section 2 — `## Startup / Shutdown`. Document the docker-compose startup sequence (`docker compose up -d --build`) and orderly shutdown (`docker compose down`, `docker compose down -v` for nuke-from-orbit including volumes). Reference the new services from 06-02 (loki, promtail, grafana) and their access URLs (Grafana :3000, Loki :3100). Include the dev-mode reload note (bind mount + --reload override). Document the celery-beat exception (Pitfall 7) — it does NOT have read_only hardening.
Section 3 — `## Backup Strategy`.
Postgres: command `docker compose exec postgres pg_dump -U postgres docuvault > backups/docuvault_$(date +%F).sql`; restore command `cat backups/docuvault_YYYY-MM-DD.sql | docker compose exec -T postgres psql -U postgres docuvault`. Document cron pattern for daily backups: a crontab line like `0 3 * * * cd /path/to/docuvault && docker compose exec -T postgres pg_dump -U postgres docuvault | gzip > backups/$(date +\%F).sql.gz`. Note: D-14 says backup strategy must be documented; automation is deferred per Deferred Idea "Backup automation".
MinIO: command `docker compose exec minio mc mirror /data /backup/minio-mirror` (mc client is installed inside the minio container). Document offsite-backup pattern using `mc mirror local-minio/<bucket> s3://offsite-bucket/<prefix>` against an S3-compatible target. Restore command: `mc mirror /backup/minio-mirror /data`.
Document retention policy: keep daily for 7 days, weekly for 4 weeks, monthly for 12 months — operator implements via cron + rotate.
Section 4 — `## Health Checks`. Markdown table with Service | Probe Command | Healthy Output:
Backend `curl -sf http://localhost:8000/health` returns `{"status":"ok",...}`
Loki `curl -sf http://localhost:3100/ready` returns `ready`
Grafana `curl -sf http://localhost:3000/api/health` returns JSON with `database: ok`
Postgres `docker compose exec postgres pg_isready -U postgres -d docuvault` exit 0
MinIO `docker compose exec minio mc ready local` exit 0
Redis `docker compose exec redis redis-cli -a $REDIS_PASSWORD ping` returns `PONG`
Celery worker `docker compose exec celery-worker celery -A celery_app inspect ping` returns `pong`
Section 5 — `## Security Gate — docker scout CVE scan (D-10)`.
Prerequisite: `docker login` (Pitfall 5 — scout uploads image manifest to Docker's analysis service).
Build step (already done by 06-04): `cd backend && docker build -t docuvault-backend:phase6 .`
Scan command: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code`
Pass criterion: exit code 0. Fail criterion: exit code 2 (critical CVEs found). Phase advancement requires pass.
Fallback if `docker scout` is unavailable (e.g. offline environment): `brew install trivy` (or `apt install trivy`), then `trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6`.
Cadence: run on every image rebuild; gate any prod deploy on pass.
On failure recovery: re-pull base image (`docker pull python:3.12-slim`), rebuild, rescan. If still failing, pin to a newer Python 3.12 patch release or rebuild against the latest python:3.13-slim after compatibility testing.
Section 6 — `## On-Call Escalation`.
State the realistic single-operator model: primary on-call is the project owner (curo1305@curonet.de); secondary escalation is the operator themselves; this is a solo deployment. Document the alert classes and the immediate action for each:
- `backend-down` (HTTP 5xx > 5% for 5min OR backend container restarting) → page immediately. Action: `docker compose logs --tail 100 backend`; check for `Read-only file system` (Pitfall 1/6) or `Permission denied` (Pitfall 5). If extractor errors, see Section 7 → "tempfile permission" entry.
- `celery-stuck` (queue depth > 1000 OR no task completion for 30min) → warn within 1h. Action: `docker compose logs --tail 200 celery-worker`; restart `docker compose restart celery-worker`.
- `loki-full-disk` (Loki container reports > 80% disk usage on /loki) → warn within 4h. Action: prune old chunks (Loki retention tuning — link to upstream docs); or rotate the loki_data volume.
- `docker scout critical-CVE` (scheduled re-scan finds new critical CVE in base image) → page within 24h. Action: rebuild image with latest python:3.12-slim and re-scan; if persistent, see Section 5 fallback.
- `quota-violation-spike` (audit log shows > 10 quota_exceeded events per hour from same user) → warn. Action: check user; possible abuse — disable account via admin endpoint.
- `failed-login-spike` (audit log shows > 100 failed_login events per hour from same IP) → warn. Action: verify per-IP limiter is firing (10/min); if not, check that 06-05 deployed and TRUSTED_PROXY_CIDRS is correct for the deployment topology.
Section 7 — `## Common Failure Modes`. Markdown table or H3 entries — each with Symptom | Cause | Recovery:
- Symptom: "PermissionError: [Errno 13] /tmp/..." in backend logs → Cause: tmpfs missing mode=1777 (Pitfall 1) → Recovery: verify docker-compose.yml backend.tmpfs value contains "/tmp:mode=1777"; recreate the container with `docker compose up -d --force-recreate backend`.
- Symptom: docker scout returns "authentication required" → Cause: Pitfall 5 → Recovery: run `docker login` first, then rerun the scan.
- Symptom: PyMuPDF "Read-only file system" when extracting PDF → Cause: Pitfall 6 — PyMuPDF writing to /var/cache/fontconfig under read_only → Recovery: add `tmpfs: ["/var/cache/fontconfig"]` to backend service in docker-compose.yml.
- Symptom: celery-beat container exits "Permission denied: 'celerybeat-schedule'" → Cause: Pitfall 7 — read_only was accidentally added to celery-beat → Recovery: remove `read_only:` from celery-beat block in docker-compose.yml; restart.
- Symptom: After 06-05 ships, requests from external clients suddenly hit rate-limit despite low traffic → Cause: a real reverse proxy is in front but its IP is not in TRUSTED_PROXY_CIDRS, so its peer IP is used as the rate-limit key for all traffic → Recovery: add the proxy's CIDR to TRUSTED_PROXY_CIDRS env var and restart backend.
- Symptom: per-account 429s when expected per-IP 429s → Cause: route handler missing the `request.state.current_user = current_user` first-line assignment → Recovery: locate the handler missing the line, add it as the first executable statement, redeploy.
- Symptom: Grafana UI loads but no Loki datasource → Cause: Loki container not ready when Grafana started OR loki-config.yaml malformed → Recovery: check `curl http://localhost:3100/ready`; if not 200, inspect `docker compose logs loki` for YAML errors.
Section 8 — `## Phase 6 Deferred Items`. List the four deferred ideas from CONTEXT.md so the next operator knows the runbook acknowledges them: HTTPS/TLS termination (add reverse proxy — pattern documented in Section 5), horizontal scaling (Redis-backed limiter — Phase 7), CI/CD pipeline (GitHub Actions for scout + locust — Phase 7), backup automation (cron service — manual procedure above is the current state).
Format requirements: GitHub-flavored Markdown, line wrap at ~100 chars where possible, consistent code-fence language tags (`bash` for shell, `yaml` for compose snippets, `text` for output samples). Add a one-line "Last updated: {today}" footer.
<threat_model>
Trust Boundaries
| Boundary | Description |
|---|---|
| Operator → production stack | Operator runs the documented commands; RUNBOOK is the contract for what's safe to run |
| Image base layers → runtime | Upstream python:3.12-slim and apt packages may ship CVEs over time; scout gate catches them |
| Repository file → secrets exposure | RUNBOOK must reference env vars without hardcoding actual secrets |
STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|---|---|---|---|---|
| T-06-06-01 | Tampering | Critical CVE in base image / dependency ships to production | mitigate | docker scout cves zero-critical gate (D-10) — human checkpoint blocks phase advancement on non-zero exit (Task 2) |
| T-06-06-02 | Information Disclosure | RUNBOOK accidentally hardcodes a real secret value | mitigate | All example values use obvious placeholder patterns (****, CHANGEME, your-token-here); explicit warning that SECRET_KEY and CLOUD_CREDS_KEY must never be committed |
| T-06-06-03 | Repudiation | Operator does not know which alert maps to which recovery procedure | mitigate | RUNBOOK Section 6 enumerates alert classes; Section 7 maps each Pitfall symptom to its recovery command — operator can self-serve at 03:00 |
| T-06-06-04 | Tampering | docker scout requires Docker Hub auth (Pitfall 5) → gate silently fails open if operator skips login | mitigate | RUNBOOK Section 5 documents the prerequisite explicitly; Task 2 checkpoint instructions force the operator to confirm login before running scan; gate uses --exit-code so scan failures cannot be ignored |
| T-06-06-05 | Denial of Service | Backup procedure not exercised → restore on production failure does not work | accept | D-14 says document the strategy; restore commands are present (Section 3); rehearsal cadence is operator responsibility documented as TODO in the runbook |
| T-06-06-SC | Tampering | docker scout itself may be tampered with (binary integrity) | accept | docker scout ships with Docker Engine 29.5.2 (verified at research time); supply-chain risk delegated to Docker Inc.; trivy fallback exists if scout is compromised |
| </threat_model> |
<success_criteria>
- D-10 satisfied: docker scout cves run against docuvault-backend:phase6 returns zero critical CVEs (or fallback trivy scan does); result documented in 06-06-SUMMARY.md.
- D-14 satisfied: RUNBOOK.md exists at repo root and covers env vars, startup/shutdown, backup strategy, health checks, on-call escalation, and common failure modes.
- Phase 6 gate is green: all six decisions covered by plans 06-04/05/06 are implemented and verified, and the security scan blocks the phase from advancing if critical CVEs are present. </success_criteria>