docs(06): create phase 6 plan — performance & production hardening

6 plans across 4 waves covering structlog/Loki observability, Locust
load testing, multi-stage Dockerfile hardening, trusted-proxy rate
limiting, and RUNBOOK.md. Verification passed (0 blockers).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
2026-06-02 20:35:03 +02:00
co-authored by Claude Sonnet 4.6
parent b7503cdff4
commit 70c09f6cd4
10 changed files with 1778 additions and 14 deletions
+36 -1
View File
@@ -286,7 +286,32 @@ Before any phase is marked complete, all three gates must pass:
4. Container hardening is complete: non-root user, read-only root filesystem, dropped Linux capabilities; `docker scout` or equivalent reports zero critical CVEs
5. A runbook documents all environment variables, startup/shutdown procedures, backup strategy, and on-call escalation path; the app can be stood up from scratch using only the runbook
**Plans**: TBD
**Plans**: 6 plans (4 waves)
**Wave 0** — Test scaffolds + package verification
- [ ] 06-01-PLAN.md — Nyquist Wave 0: xfail stubs (test_logging.py, test_rate_limiting.py), Locust skeleton, package legitimacy checkpoint (D-01, D-04, D-11, D-12)
**Wave 1** *(blocked on Wave 0 completion)*
- [ ] 06-02-PLAN.md — structlog JSON logging + CorrelationIDMiddleware + Loki/Promtail/Grafana Docker Compose stack (D-01, D-02, D-03)
- [ ] 06-03-PLAN.md — Locust locustfile.py with JWT auth + SLA gate listener (D-04, D-05, D-06)
**Wave 2** *(blocked on Wave 1 completion)*
- [ ] 06-04-PLAN.md — Multi-stage Dockerfile + read_only/tmpfs/cap_drop on backend + celery-worker (D-07, D-08, D-09)
- [ ] 06-05-PLAN.md — Trusted-proxy get_client_ip body + per-account rate limiter on document/cloud endpoints (D-11, D-12, D-13)
**Wave 3** *(blocked on Wave 2 completion)*
- [ ] 06-06-PLAN.md — docker scout CVE gate + RUNBOOK.md (D-10, D-14)
**Cross-cutting constraints:**
- get_client_ip lives ONLY in backend/deps/utils.py — replace body in-place, no new function (Plans 05)
- celery-beat intentionally excluded from read_only: true (Plan 04)
- locust must NOT be in requirements.txt — use requirements-dev.txt (Plan 03)
- CorrelationIDMiddleware registered LAST in main.py — Starlette reverse order (Plan 02)
---
@@ -358,6 +383,16 @@ Before any phase is marked complete, all three gates must pass:
**Status: ✓ Complete (2026-06-01)**
### Phase 7: Redo and optimize LLM integration
**Goal:** [To be planned]
**Requirements**: TBD
**Depends on:** Phase 6
**Plans:** 0 plans
Plans:
- [ ] TBD (run /gsd-plan-phase 7 to break down)
---
## Progress Table
+9 -9
View File
@@ -1,16 +1,16 @@
---
gsd_state_version: 1.0
milestone: v1.0
milestone_name: "audit gaps: SHARE-02/STORE-06/ADMIN-06"
current_phase: 06.2
status: executing
last_updated: "2026-06-02T18:24:12.300Z"
milestone_name: "Performance & Production Hardening"
current_phase: "06"
status: planned
last_updated: "2026-06-02T00:00:00.000Z"
progress:
total_phases: 3
completed_phases: 2
total_plans: 7
completed_plans: 7
percent: 67
total_phases: 1
completed_phases: 0
total_plans: 6
completed_plans: 0
percent: 0
---
# Project State
@@ -0,0 +1,255 @@
---
phase: 06-performance-production-hardening
plan: 01
type: execute
wave: 0
depends_on: []
files_modified:
- backend/tests/test_logging.py
- backend/tests/test_rate_limiting.py
- backend/load_tests/__init__.py
- backend/load_tests/locustfile.py
autonomous: false
requirements:
- D-01
- D-02
- D-04
- D-05
- D-06
- D-11
- D-12
user_setup: []
must_haves:
truths:
- "pytest -v collects test_logging.py and test_rate_limiting.py without import errors"
- "All Wave 0 stubs run as xfail(strict=False) so no false-positive xpass breaks CI"
- "backend/load_tests/ is excluded from pytest discovery via empty __init__.py"
- "Package legitimacy for structlog and locust is verified by a human before any requirements file is modified"
- "An automated unit test asserts the slowapi key_func ordering assumption (A1) before per-account limiters ship"
artifacts:
- path: "backend/tests/test_logging.py"
provides: "xfail stubs for D-01/D-02 (correlation_id field, context cleared between requests, JSON renderer wired)"
contains: "pytest.mark.xfail"
- path: "backend/tests/test_rate_limiting.py"
provides: "xfail stubs for D-11 (get_client_ip trusted/untrusted) and D-12 (account_limiter key, 429 after 100 req/min, key_func ordering assumption A1)"
contains: "pytest.mark.xfail"
- path: "backend/load_tests/__init__.py"
provides: "Empty marker file that makes load_tests an importable package and stops pytest from descending into it"
- path: "backend/load_tests/locustfile.py"
provides: "Locust HttpUser skeleton with TODO body — full implementation in 06-03"
contains: "class DocuVaultUser"
key_links:
- from: "backend/tests/test_rate_limiting.py::test_account_limiter_key_ordering"
to: "Pattern 4 / Pitfall 3 in 06-RESEARCH.md (A1 verification)"
via: "ASGI request fixture with request.state.current_user pre-set"
pattern: "request\\.state\\.current_user"
- from: "backend/load_tests/__init__.py"
to: "pytest collection"
via: "package import gate"
pattern: "__init__\\.py"
---
<objective>
Establish the Nyquist Wave 0 scaffold for Phase 6: failing test stubs for structured logging (D-01/D-02), trusted-proxy rate limiting (D-11), and per-account rate limiting (D-12); a Locust load-test skeleton (D-04/D-05) excluded from pytest discovery; and a blocking package-legitimacy checkpoint for the two [ASSUMED] dependencies (structlog 25.5.0, locust 2.34.0) before any requirements file is touched.
Purpose: Lock down behavior expectations before implementation; surface the slowapi key_func ordering assumption (RESEARCH.md A1) as an explicit test that downstream plans must turn green; gate package installs behind a blocking human verification per package-legitimacy protocol.
Output: Four scaffolded files plus a verified human-approved package list ready for 06-02 / 06-03 to install.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
@.planning/phases/06-performance-production-hardening/06-VALIDATION.md
@CLAUDE.md
@backend/tests/conftest.py
@backend/deps/utils.py
@backend/pytest.ini
</context>
<tasks>
<task type="checkpoint:human-verify" gate="blocking-human">
<name>Task 1: Package legitimacy verification — structlog and locust</name>
<read_first>
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (sections: Package Legitimacy Audit, Standard Stack)
</read_first>
<what-built>
Two PyPI packages are flagged `[ASSUMED]` by the researcher because slopcheck was unavailable at research time. No code change has been made yet — this checkpoint exists so the user can confirm legitimacy before any requirements file is modified.
</what-built>
<how-to-verify>
1. Open https://pypi.org/project/structlog/ — confirm version 25.5.0 is published; confirm the homepage points to https://github.com/hynek/structlog; confirm author is Hynek Schlawack (a CPython core contributor).
2. Open https://pypi.org/project/locust/ — confirm version 2.34.0 is published; confirm the homepage points to https://github.com/locustio/locust; confirm the project has >10 years of release history and multi-million monthly downloads.
3. Confirm neither package name is a typosquat of a similar published name (e.g. `struct-log`, `locusts`).
</how-to-verify>
<acceptance_criteria>
- User responds with "approved" or "approved with notes" — execution may proceed.
- User responds with anything else — execution halts; planner reconsiders alternatives or aborts the phase.
</acceptance_criteria>
<resume-signal>Type "approved" to allow 06-02 to add structlog and 06-03 to add locust; otherwise describe the concern.</resume-signal>
</task>
<task type="auto">
<name>Task 2: Create test_logging.py with xfail stubs (D-01/D-02)</name>
<files>backend/tests/test_logging.py</files>
<read_first>
- backend/tests/conftest.py (fixture conventions: async_client, auth_user, pytest_asyncio.fixture, asyncio_mode=auto)
- backend/pytest.ini (asyncio_mode setting; verify xfail is a recognised marker)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 1, Pattern 2, Pitfall 2)
</read_first>
<action>
Create backend/tests/test_logging.py with single-line-body xfail stubs (strict=False) for these behaviours, one test function each — body is exactly `pytest.xfail("not implemented yet")`:
(1) test_setup_logging_emits_json_when_LOG_JSON_true — D-01 contract that setup_logging(json_logs=True) installs JSONRenderer through ProcessorFormatter and root logger emits JSON to stdout.
(2) test_correlation_id_middleware_binds_contextvar — CorrelationIDMiddleware binds correlation_id, path, method into structlog.contextvars per request.
(3) test_correlation_id_response_header_present — every HTTP response includes X-Correlation-ID header set to the bound correlation_id.
(4) test_contextvars_cleared_between_requests — Pitfall 2 — clear_contextvars() runs first in the middleware so user_id from a prior request never bleeds into the next.
(5) test_uvicorn_access_log_suppressed — uvicorn.access propagate=False after setup_logging() so the middleware owns request logging.
Use the same async_client fixture pattern from conftest.py for tests that need ASGI requests. Mark all five with `@pytest.mark.xfail(strict=False, reason="implementation in 06-02")`. Single-line body only — no assertion code that could xpass accidentally.
</action>
<verify>
<automated>cd backend &amp;&amp; pytest tests/test_logging.py -v --no-header 2>&amp;1 | grep -v '^#' | grep -cE 'XFAIL|xfailed'</automated>
</verify>
<acceptance_criteria>
- File backend/tests/test_logging.py exists and is importable: `cd backend &amp;&amp; python -c "import tests.test_logging"` exits 0.
- `cd backend &amp;&amp; pytest tests/test_logging.py -v --no-header 2>&amp;1` shows exactly 5 collected tests, all XFAIL, zero PASSED, zero FAILED, zero ERROR.
- Every test body is the single line `pytest.xfail("not implemented yet")``grep -c "pytest.xfail" backend/tests/test_logging.py` returns 5.
- All five tests are decorated `@pytest.mark.xfail(strict=False, reason=...)``grep -c "xfail(strict=False" backend/tests/test_logging.py` returns 5.
</acceptance_criteria>
<done>Five xfail stubs collected, all show XFAIL status, full suite still passes the same count as before this plan.</done>
</task>
<task type="auto">
<name>Task 3: Create test_rate_limiting.py with xfail stubs (D-11/D-12 + A1)</name>
<files>backend/tests/test_rate_limiting.py</files>
<read_first>
- backend/deps/utils.py (current get_client_ip body; the function that 06-04 will replace)
- backend/tests/conftest.py (async_client, auth_user, second_auth_user fixtures)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 3, Pattern 4, Pitfall 3, Assumption A1)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (deps/utils.py section, account_limiter section)
</read_first>
<action>
Create backend/tests/test_rate_limiting.py with single-line-body xfail stubs (strict=False), one test function each — body is exactly `pytest.xfail("not implemented yet")`:
(1) test_get_client_ip_untrusted_returns_direct_peer — D-11 — when request.client.host is 8.8.8.8 (not in trusted CIDRs), ignore X-Forwarded-For and return "8.8.8.8".
(2) test_get_client_ip_trusted_proxy_reads_xff_leftmost — D-11 — when request.client.host is 127.0.0.1 (trusted), return the leftmost IP of X-Forwarded-For: "1.2.3.4, 5.6.7.8" → "1.2.3.4".
(3) test_get_client_ip_trusted_proxy_no_xff_falls_back — D-11 — trusted peer with no XFF header returns the direct peer IP.
(4) test_get_client_ip_invalid_peer_returns_none_or_string — D-11 — request.client is None returns None without raising.
(5) test_account_limiter_key_uses_user_id — D-12 — _account_key(request) where request.state.current_user has id=UUID(...) returns str(user.id), not request.client.host.
(6) test_account_limiter_key_falls_back_to_ip_when_no_user — D-12 — when request.state.current_user is missing, key function returns the direct peer IP (Pitfall 3 — must not crash).
(7) test_account_limiter_key_ordering_assumption — A1 verification — construct a FastAPI app with one endpoint that sets `request.state.current_user = current_user` as its first line, decorated with `@account_limiter.limit("100/minute")`; call it 101 times with the same user; assert the 101st response is 429 AND the limiter recorded the key as str(user.id), not as the IP. This test is the gate that turns A1 from assumption to fact before 06-04 wires the decorator on all endpoints.
(8) test_authenticated_endpoint_429_after_100_per_minute — D-12 — full integration: GET /api/documents/ 101 times with the same auth_user; 101st returns 429.
Mark all eight with `@pytest.mark.xfail(strict=False, reason="implementation in 06-04")`. Single-line body only.
</action>
<verify>
<automated>cd backend &amp;&amp; pytest tests/test_rate_limiting.py -v --no-header 2>&amp;1 | grep -v '^#' | grep -cE 'XFAIL|xfailed'</automated>
</verify>
<acceptance_criteria>
- File backend/tests/test_rate_limiting.py exists and is importable: `cd backend &amp;&amp; python -c "import tests.test_rate_limiting"` exits 0.
- `cd backend &amp;&amp; pytest tests/test_rate_limiting.py -v --no-header` shows exactly 8 collected tests, all XFAIL, zero PASSED, zero FAILED.
- `grep -c "pytest.xfail" backend/tests/test_rate_limiting.py` returns 8.
- `grep -c "xfail(strict=False" backend/tests/test_rate_limiting.py` returns 8.
- At least one test name contains the substring "ordering" — `grep -c "def test.*ordering" backend/tests/test_rate_limiting.py` returns 1.
</acceptance_criteria>
<done>Eight xfail stubs collected, A1 verification test present, no regressions in existing suite.</done>
</task>
<task type="auto">
<name>Task 4: Create backend/load_tests/ package skeleton with Locust stub</name>
<files>backend/load_tests/__init__.py, backend/load_tests/locustfile.py</files>
<read_first>
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 7)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (locustfile.py section)
- backend/pytest.ini (testpaths setting — confirm pytest does not include load_tests)
</read_first>
<action>
Create two files:
(1) backend/load_tests/__init__.py — empty file (zero bytes). This serves a single purpose: turn load_tests/ into a Python package so locust can import it cleanly, while pytest's default test discovery (which looks for test_*.py files) will not collect locustfile.py because its filename does not match the test_*.py pattern. Add a one-line comment `# Locust load-test package — not a pytest test target` if any content is needed.
(2) backend/load_tests/locustfile.py — skeleton with the import block (`from locust import HttpUser, task, between, events`), TEST_EMAIL / TEST_PASSWORD env-var pickup with safe defaults, a class `DocuVaultUser(HttpUser)` containing wait_time = between(0.5, 2.0), an empty access_token attribute, and method stubs `on_start`, `_auth_headers`, `list_documents`, `upload_document`, `refresh_token`, plus an `@events.quitting.add_listener def check_sla(...)` registered listener. Every method body is exactly `raise NotImplementedError("implementation in 06-03")`. Top of file: docstring referencing D-04/D-05/D-06 and the run command from RESEARCH.md Pattern 7. Do NOT import anything from the application code (no `from backend...` or `from api...` lines).
</action>
<verify>
<automated>test -f backend/load_tests/__init__.py &amp;&amp; test -f backend/load_tests/locustfile.py &amp;&amp; cd backend &amp;&amp; python -c "import ast; ast.parse(open('load_tests/locustfile.py').read())" &amp;&amp; cd backend &amp;&amp; pytest tests/ -v --collect-only --no-header 2>&amp;1 | grep -c "load_tests"</automated>
</verify>
<acceptance_criteria>
- File backend/load_tests/__init__.py exists; `wc -c backend/load_tests/__init__.py` returns ≤ 70 bytes (empty or one comment line).
- File backend/load_tests/locustfile.py exists and is syntactically valid Python: `python -c "import ast; ast.parse(open('backend/load_tests/locustfile.py').read())"` exits 0.
- locustfile.py defines exactly one class containing the string `class DocuVaultUser``grep -c "class DocuVaultUser" backend/load_tests/locustfile.py` returns 1.
- locustfile.py contains the SLA listener marker — `grep -c "events.quitting.add_listener" backend/load_tests/locustfile.py` returns 1.
- locustfile.py contains no application imports — `grep -cE "^from (backend|api|services|db|deps)" backend/load_tests/locustfile.py` returns 0.
- `cd backend &amp;&amp; pytest tests/ --collect-only --no-header 2>&amp;1 | grep -c "load_tests"` returns 0 (pytest must NOT discover anything in load_tests).
- Each method body raises NotImplementedError — `grep -c "raise NotImplementedError" backend/load_tests/locustfile.py` returns ≥ 5.
</acceptance_criteria>
<done>Package present, locustfile importable by `python -c "import sys; sys.path.insert(0,'backend/load_tests'); import locustfile"`, pytest does not discover the directory, no application imports.</done>
</task>
<task type="auto">
<name>Task 5: Full-suite regression check</name>
<files></files>
<read_first>
- backend/pytest.ini
- backend/tests/test_logging.py (just created)
- backend/tests/test_rate_limiting.py (just created)
</read_first>
<action>
Run the full backend pytest suite to confirm: (a) the 13 new xfail stubs (5 + 8) all collect and report XFAIL, (b) zero pre-existing tests changed status, (c) no collection errors from the new files or the new load_tests/ package. Capture the XFAIL count and the PASSED count for the SUMMARY.md.
</action>
<verify>
<automated>cd backend &amp;&amp; pytest tests/ -v --no-header --tb=short 2>&amp;1 | tail -5 | grep -vE '^#' | grep -E 'passed|xfailed'</automated>
</verify>
<acceptance_criteria>
- `cd backend &amp;&amp; pytest tests/ --no-header --tb=line 2>&amp;1 | tail -1` shows a green summary line containing "passed" with no "failed" or "error".
- Total xfailed count increased by exactly 13 versus the baseline recorded in STATE.md (Phase 6.2: 344 passed, 1 pre-existing failure).
- Zero NEW failures: the pre-existing test_extract_docx failure remains the only failure, no other tests turn red.
- Plan SUMMARY records the exact passed/xfailed numbers from the run.
</acceptance_criteria>
<done>Full suite green-with-known-pre-existing-failure, 13 new XFAIL items present, ready for implementation plans 06-02/06-03/06-04 to promote them to PASS.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Untrusted client → reverse proxy | External HTTP clients can spoof headers; proxy must overwrite or strip them |
| Reverse proxy → FastAPI | Internal-network requests; XFF is trusted only when peer matches CIDR |
| Application code → PyPI registry | Third-party packages may be malicious typosquats; legitimacy gate required |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-06-01-01 | Tampering | structlog/locust packages from PyPI | mitigate | Blocking human checkpoint (Task 1) — verify PyPI URL, author, age, downloads before any requirements file modification |
| T-06-01-02 | Tampering | A1 assumption (slowapi key_func ordering) ships untested | mitigate | Dedicated xfail test (Task 3, test 7) — 06-04 must turn it green before applying the decorator to all routes; un-promoted test blocks phase gate |
| T-06-01-03 | Information Disclosure | Locust credentials hardcoded in version-controlled file | mitigate | Skeleton reads from env vars; .env file already in .gitignore; documented in RUNBOOK.md (06-06) |
| T-06-01-SC | Tampering | npm/pip/cargo installs | mitigate | Package legitimacy gate (Task 1) is `gate="blocking-human"`; cannot auto-advance |
</threat_model>
<verification>
After all five tasks:
- `cd backend && pytest tests/ -v --no-header` shows the existing pass count plus exactly 13 new XFAIL items (5 in test_logging.py + 8 in test_rate_limiting.py).
- `cd backend && pytest tests/ --collect-only 2>&1 | grep -c "load_tests"` returns 0.
- `python -c "import ast; ast.parse(open('backend/load_tests/locustfile.py').read())"` exits 0.
- `grep -cE "^from (backend|api|services|db|deps)" backend/load_tests/locustfile.py` returns 0.
- User approved the package legitimacy checkpoint (recorded in SUMMARY).
</verification>
<success_criteria>
- All 13 xfail stubs collected and shown as XFAIL.
- backend/load_tests/ package exists, pytest skips it, locustfile.py imports cleanly under Python's stdlib only.
- Package legitimacy human-verify checkpoint passed and recorded.
- Zero regressions: the only red test in the suite is the pre-existing test_extract_docx (carried over from 06.2).
</success_criteria>
<output>
Create `.planning/phases/06-performance-production-hardening/06-01-SUMMARY.md` when done. Include: passed/xfailed/failed counts from the final pytest run, the package legitimacy decision verbatim, and any open notes for downstream plans.
</output>
@@ -0,0 +1,269 @@
---
phase: 06-performance-production-hardening
plan: 02
type: execute
wave: 1
depends_on:
- 06-01
files_modified:
- backend/requirements.txt
- backend/services/logging.py
- backend/config.py
- backend/main.py
- docker-compose.yml
- docker/loki/loki-config.yaml
- docker/loki/promtail-config.yaml
- backend/tests/test_logging.py
autonomous: true
requirements:
- D-01
- D-02
- D-03
user_setup:
- service: grafana
why: "Local log query UI for Loki"
env_vars: []
dashboard_config:
- task: "Open http://localhost:3000 after compose up; Loki datasource is preconfigured anonymous; Explore → Loki → query {service=\"backend\"}"
location: "Grafana UI → Explore → Loki"
must_haves:
truths:
- "Every HTTP request emits at least one structured JSON log line containing correlation_id, path, method, duration_ms (user_id added by the per-account work in 06-04)"
- "Every HTTP response carries an X-Correlation-ID header matching the bound contextvar"
- "structlog contextvars are cleared at the start of every request — no bleed between requests on the same worker"
- "docker compose up brings up loki, promtail, grafana services that read backend container stdout via the docker_sd label logging=promtail"
- "Setting LOG_JSON=true switches the renderer from ConsoleRenderer to JSONRenderer without code changes"
artifacts:
- path: "backend/services/logging.py"
provides: "setup_logging(json_logs, log_level) — single entry-point for structlog + stdlib bridge"
exports: ["setup_logging"]
min_lines: 40
- path: "backend/main.py"
provides: "CorrelationIDMiddleware raw-ASGI class + setup_logging() call in lifespan + middleware registered LAST"
contains: "class CorrelationIDMiddleware"
- path: "backend/config.py"
provides: "log_level + log_json Settings fields"
contains: "log_level"
- path: "docker-compose.yml"
provides: "loki, promtail, grafana services + loki_data and grafana_data named volumes + logging:promtail label on backend"
contains: "grafana/loki"
- path: "docker/loki/loki-config.yaml"
provides: "single-binary filesystem-mode Loki config (schema v13, tsdb)"
contains: "auth_enabled: false"
- path: "docker/loki/promtail-config.yaml"
provides: "Promtail docker_sd_configs scrape with label filter logging=promtail; ships to http://loki:3100/loki/api/v1/push"
contains: "docker_sd_configs"
key_links:
- from: "backend/main.py CorrelationIDMiddleware"
to: "structlog.contextvars"
via: "clear_contextvars() then bind_contextvars(correlation_id, path, method)"
pattern: "clear_contextvars"
- from: "docker-compose.yml backend.labels.logging"
to: "promtail-config.yaml docker_sd_configs.filters"
via: "label match logging=promtail"
pattern: "logging.*promtail"
- from: "main.py app.add_middleware(CorrelationIDMiddleware)"
to: "Starlette reverse-insertion order"
via: "registered LAST so it runs FIRST"
pattern: "add_middleware\\(CorrelationIDMiddleware"
---
<objective>
Wire structured JSON logging (D-01) with correlation IDs across every FastAPI request, and stand up a local Loki+Promtail+Grafana log aggregation stack via docker-compose (D-02). Skip OpenTelemetry per D-03. Promote 5 xfail stubs from 06-01 to PASS.
Purpose: Make every request observable end-to-end with a single grep on correlation_id. The Loki stack closes Phase 6 success criterion 2 ("Structured JSON logging is emitted to stdout; a local log aggregation stack captures and queries them").
Output: setup_logging() service module, CorrelationIDMiddleware in main.py, config keys, docker/loki/{loki,promtail}-config.yaml, docker-compose additions, and all 5 logging xfail stubs flipped to PASS.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
@.planning/phases/06-performance-production-hardening/06-VALIDATION.md
@CLAUDE.md
@backend/main.py
@backend/config.py
@backend/services/auth.py
@backend/tests/test_logging.py
@docker-compose.yml
<interfaces>
<!-- Key signatures the executor must respect — extracted from existing code. -->
From backend/main.py (current lifespan + middleware shape):
- `@asynccontextmanager async def lifespan(app: FastAPI)` — add `setup_logging(json_logs=settings.log_json, log_level=settings.log_level)` as the FIRST statement inside lifespan(), before MinIO/Redis init.
- Existing `app.add_middleware(...)` calls (in order of insertion): SecurityHeadersMiddleware → CORSMiddleware → OriginValidationMiddleware. Starlette runs middleware in REVERSE insertion order. CorrelationIDMiddleware must be registered LAST so it runs FIRST.
From backend/config.py (current Settings class):
- Uses pydantic-settings `Settings(BaseSettings)` with `model_config = SettingsConfigDict(env_file=".env", env_list_separator=",")`. Add new fields with type annotations and defaults; env vars are upper-snake-case of the field name.
From RESEARCH.md Pattern 1 / 2 (target shape):
- `setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None`
- `class CorrelationIDMiddleware:` constructor `__init__(self, app: ASGIApp)` + `async def __call__(self, scope, receive, send)`. NOT BaseHTTPMiddleware.
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Add structlog dependency + create services/logging.py</name>
<files>backend/requirements.txt, backend/services/logging.py</files>
<read_first>
- backend/requirements.txt
- backend/services/auth.py (module structure analog)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 1)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (services/logging.py section)
</read_first>
<behavior>
- setup_logging(json_logs=True) installs a JSONRenderer in the root logger handler chain.
- setup_logging(json_logs=False) installs ConsoleRenderer.
- shared_processors list places structlog.contextvars.merge_contextvars FIRST.
- Stdlib loggers uvicorn and uvicorn.error propagate through the structlog formatter; uvicorn.access propagate is set to False so the middleware owns request logging.
- Calling setup_logging twice does not duplicate root handlers (idempotent — clear existing handlers before adding).
</behavior>
<action>
Append `structlog>=25.5.0` to backend/requirements.txt under a new comment `# Observability (Phase 6 — D-01)`.
Create backend/services/logging.py implementing `setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None`. Module docstring per the services/auth.py analog: `from __future__ import annotations`, import logging + structlog + `from config import settings`, no FastAPI coupling. Build the shared_processors list in this exact order: merge_contextvars, add_log_level, add_logger_name, PositionalArgumentsFormatter, ExtraAdder, TimeStamper(fmt="iso"), StackInfoRenderer. If json_logs True, append format_exc_info to that list. Call structlog.configure(processors=shared+wrap_for_formatter, logger_factory=LoggerFactory, cache_logger_on_first_use=True). Pick JSONRenderer when json_logs else ConsoleRenderer. Build a ProcessorFormatter(foreign_pre_chain=shared, processors=[remove_processors_meta, log_renderer]). Clear existing root logger handlers, add a single StreamHandler with the formatter, set root level to log_level.upper(). For uvicorn and uvicorn.error loggers, clear handlers and set propagate=True. For uvicorn.access, clear handlers and set propagate=False.
</action>
<verify>
<automated>cd backend &amp;&amp; pip install structlog &amp;&amp; python -c "from services.logging import setup_logging; setup_logging(json_logs=True, log_level='INFO'); import structlog; structlog.get_logger().info('hello', user_id='abc')" 2>&amp;1 | grep -E '"event":\s*"hello"'</automated>
</verify>
<acceptance_criteria>
- `grep -c '^structlog' backend/requirements.txt` returns 1.
- `grep -c 'def setup_logging' backend/services/logging.py` returns 1.
- Module imports cleanly: `cd backend &amp;&amp; python -c "from services.logging import setup_logging"` exits 0.
- merge_contextvars appears BEFORE add_log_level in shared_processors: `grep -nE "merge_contextvars|add_log_level" backend/services/logging.py | head -2` shows merge_contextvars first.
- JSON branch emits a JSON-shaped line: `cd backend &amp;&amp; python -c "from services.logging import setup_logging; setup_logging(json_logs=True); import structlog; structlog.get_logger().info('e', k=1)" 2>&amp;1 | grep -cE '^\\{.*"event":\\s*"e"'` returns ≥ 1.
- `grep -cE "uvicorn.access.*propagate.*False|propagate.*False.*uvicorn.access" backend/services/logging.py` returns ≥ 1.
</acceptance_criteria>
<done>structlog installed, setup_logging defined and idempotent, JSON/console branches selectable via boolean parameter.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Wire CorrelationIDMiddleware in main.py + add config fields + promote 5 test stubs</name>
<files>backend/main.py, backend/config.py, backend/tests/test_logging.py</files>
<read_first>
- backend/main.py (lines 65131 — lifespan, middleware registration order)
- backend/config.py (current Settings fields and pattern)
- backend/tests/test_logging.py (the 5 xfail stubs from 06-01 — flip to real assertions)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 2, Pitfall 2, Anti-Patterns section)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (main.py section — "Register LAST")
</read_first>
<behavior>
- GET /health response includes an X-Correlation-ID header that is a UUID4-shaped string.
- Two consecutive requests on the same worker produce two distinct correlation_id values (Pitfall 2 — no contextvars bleed).
- Log lines emitted during a request include correlation_id, path, method, and duration_ms keys.
- clear_contextvars() runs as the first operation inside CorrelationIDMiddleware.__call__ — a contextvar bound by a faked earlier request does not appear in the current request's log.
- CorrelationIDMiddleware is the LAST middleware registered in main.py (Starlette reverse-insertion order makes it run FIRST in the request chain).
</behavior>
<action>
Edit backend/config.py to append two fields inside the Settings class after the existing Cloud Storage block, before `settings = Settings()`: `log_level: str = "INFO"` and `log_json: bool = False`. Use the existing comment pattern `# Observability (Phase 6 — D-01)`. Pydantic-settings picks them up automatically as LOG_LEVEL / LOG_JSON env vars.
Edit backend/main.py:
(a) Add imports near the top: `import uuid`, `import time`, `import structlog`, `from starlette.types import ASGIApp, Receive, Scope, Send`, `from services.logging import setup_logging`.
(b) Inside the existing lifespan() function, insert as the FIRST statement (before MinIO client init): `setup_logging(json_logs=settings.log_json, log_level=settings.log_level)`.
(c) Define a new top-level class CorrelationIDMiddleware as raw ASGI — NOT BaseHTTPMiddleware. Constructor `__init__(self, app: ASGIApp) -> None` stores app on self. Async `__call__(self, scope, receive, send) -> None`: if scope type is not "http" delegate to self.app and return; generate correlation_id = str(uuid.uuid4()); capture start_ns = time.perf_counter_ns(); call structlog.contextvars.clear_contextvars() FIRST; then structlog.contextvars.bind_contextvars(correlation_id=correlation_id, path=scope.get("path",""), method=scope.get("method","")); define `async def send_with_header(message)` that on http.response.start appends (b"x-correlation-id", correlation_id.encode()) to message["headers"] (preserve existing headers); await self.app(scope, receive, send_with_header); after await, compute duration_ms = (time.perf_counter_ns() - start_ns) / 1_000_000 and bind_contextvars(duration_ms=round(duration_ms, 2)).
(d) Register the new middleware as the LAST `app.add_middleware()` call (after OriginValidationMiddleware), with a header comment `# 4. CorrelationID — added last so it runs FIRST (Starlette reverse-insertion order)`.
Edit backend/tests/test_logging.py: remove the `@pytest.mark.xfail` decorator from all 5 stubs and replace the single-line `pytest.xfail(...)` body with real assertions matching the behaviour titles set in 06-01. For tests that need a working app, use the existing async_client fixture from conftest.py.
</action>
<verify>
<automated>cd backend &amp;&amp; pytest tests/test_logging.py -v --no-header 2>&amp;1 | tail -3 | grep -E '5 passed'</automated>
</verify>
<acceptance_criteria>
- `grep -c "log_level: str" backend/config.py` returns 1 and `grep -c "log_json: bool" backend/config.py` returns 1.
- `grep -c "class CorrelationIDMiddleware" backend/main.py` returns 1.
- `grep -cE "class CorrelationIDMiddleware\\(BaseHTTPMiddleware" backend/main.py` returns 0 (must NOT inherit BaseHTTPMiddleware).
- `grep -nE "app.add_middleware\\(CorrelationIDMiddleware" backend/main.py` returns a line number greater than the line `app.add_middleware(OriginValidationMiddleware)`.
- `grep -c "clear_contextvars" backend/main.py` returns ≥ 1.
- `grep -c "setup_logging" backend/main.py` returns ≥ 2 (one import, one call).
- All 5 tests in backend/tests/test_logging.py PASS: `cd backend &amp;&amp; pytest tests/test_logging.py -v --no-header 2>&amp;1 | tail -3` shows "5 passed".
- `grep -c "pytest.mark.xfail" backend/tests/test_logging.py` returns 0 (all xfail markers removed).
- Full backend suite shows no NEW failures versus the 06-01 baseline.
</acceptance_criteria>
<done>5 test_logging.py tests pass, no regressions, correlation_id flows from middleware to logs to response header, contextvars are cleared per request.</done>
</task>
<task type="auto">
<name>Task 3: Add Loki + Promtail + Grafana stack to docker-compose</name>
<files>docker-compose.yml, docker/loki/loki-config.yaml, docker/loki/promtail-config.yaml</files>
<read_first>
- docker-compose.yml (current backend, celery-worker, celery-beat blocks)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 6 — Loki/Promtail/Grafana yaml blocks)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (docker-compose.yml section)
</read_first>
<action>
Create docker/loki/loki-config.yaml per RESEARCH.md Pattern 6 (single-binary filesystem-mode block): auth_enabled false; server http_listen_port 3100, grpc_listen_port 9096; common instance_addr 127.0.0.1, path_prefix /loki, storage.filesystem chunks_directory /loki/chunks rules_directory /loki/rules, replication_factor 1, ring kvstore store inmemory; schema_config configs from 2020-10-24 store tsdb object_store filesystem schema v13 index prefix index_ period 24h; query_range.results_cache.cache.embedded_cache enabled true max_size_mb 100.
Create docker/loki/promtail-config.yaml per RESEARCH.md Pattern 6: server http_listen_port 9080 grpc_listen_port 0; positions filename /tmp/positions.yaml; single clients entry url http://loki:3100/loki/api/v1/push; scrape_configs[0] job_name docker with docker_sd_configs (host unix:///var/run/docker.sock, refresh_interval 5s, filters one entry name=label values=["logging=promtail"]); relabel_configs that map __meta_docker_container_name (regex "/(.*)") to label `container`, and map __meta_docker_container_label_com_docker_compose_service to label `service`.
Edit docker-compose.yml:
(a) Add three new services AFTER the existing celery-beat block, BEFORE the frontend service. Service `loki` (image grafana/loki:latest, ports "3100:3100", volumes ./docker/loki/loki-config.yaml:/etc/loki/local-config.yaml + loki_data:/loki, command "-config.file=/etc/loki/local-config.yaml"). Service `promtail` (image grafana/promtail:latest, volumes ./docker/loki/promtail-config.yaml:/etc/promtail/config.yaml + /var/lib/docker/containers:/var/lib/docker/containers:ro + /var/run/docker.sock:/var/run/docker.sock, command "-config.file=/etc/promtail/config.yaml", depends_on loki). Service `grafana` (image grafana/grafana:latest, ports "3000:3000", environment GF_AUTH_ANONYMOUS_ENABLED=true GF_AUTH_ANONYMOUS_ORG_ROLE=Admin, volumes grafana_data:/var/lib/grafana, depends_on loki).
(b) Append `loki_data:` and `grafana_data:` entries under the existing top-level `volumes:` block.
(c) Add a `labels:` map to the backend service block containing `logging: "promtail"` so promtail's docker_sd label filter matches.
(d) Add the same `logging: "promtail"` label to the celery-worker service block.
(e) Add `LOG_LEVEL=${LOG_LEVEL:-INFO}` and `LOG_JSON=${LOG_JSON:-false}` to the backend service environment block.
Do NOT modify celery-beat (it remains unhardened per Pitfall 7; it will not ship logs via promtail in this phase because that requires read-only safety considerations deferred to a later phase).
</action>
<verify>
<automated>test -f docker/loki/loki-config.yaml &amp;&amp; test -f docker/loki/promtail-config.yaml &amp;&amp; python -c "import yaml; [yaml.safe_load(open(f)) for f in ['docker-compose.yml','docker/loki/loki-config.yaml','docker/loki/promtail-config.yaml']]" &amp;&amp; docker compose config --quiet</automated>
</verify>
<acceptance_criteria>
- All three YAML files parse without error: `python -c "import yaml; [yaml.safe_load(open(f)) for f in ['docker-compose.yml','docker/loki/loki-config.yaml','docker/loki/promtail-config.yaml']]"` exits 0.
- `grep -c 'grafana/loki' docker-compose.yml` returns 1.
- `grep -c 'grafana/promtail' docker-compose.yml` returns 1.
- `grep -c 'grafana/grafana' docker-compose.yml` returns 1.
- `grep -cE 'logging:\\s*"?promtail"?' docker-compose.yml` returns ≥ 2 (backend + celery-worker).
- `grep -c '^ loki_data:' docker-compose.yml` returns 1 and `grep -c '^ grafana_data:' docker-compose.yml` returns 1.
- `grep -c 'LOG_JSON' docker-compose.yml` returns ≥ 1.
- `grep -c 'schema:\\s*v13' docker/loki/loki-config.yaml` returns 1.
- `grep -c 'docker_sd_configs' docker/loki/promtail-config.yaml` returns 1.
- `docker compose config --quiet` exits 0 (compose file is valid).
- celery-beat block contains NO `logging:` label and NO `read_only:` key — `awk '/^ celery-beat:/,/^ [a-z]/' docker-compose.yml | grep -cE 'logging:|read_only:'` returns 0.
</acceptance_criteria>
<done>Loki stack defined in compose, two backend services labelled for promtail scraping, all yaml validates, celery-beat untouched.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Untrusted user input → log fields | User-controlled strings (path, query params, body) may attempt log injection |
| Backend stdout → Promtail → Loki | Internal-only network; no external exposure |
| Grafana UI :3000 → local network | Anonymous admin enabled for local dev only — production hardening deferred (see RUNBOOK.md in 06-06) |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-06-02-01 | Tampering | Log injection via user-controlled strings in log fields | mitigate | structlog JSONRenderer serialises values as JSON strings — newlines and quotes are escaped automatically; no `%s` format strings concatenate user input |
| T-06-02-02 | Information Disclosure | structlog contextvars leaking user_id across requests | mitigate | clear_contextvars() is the FIRST call inside CorrelationIDMiddleware.__call__; verified by test_contextvars_cleared_between_requests in 06-01/06-02 |
| T-06-02-03 | Information Disclosure | Grafana anonymous admin exposes Loki query UI to anyone on the docker network | accept | Local dev convenience; RUNBOOK.md (06-06) documents production-time hardening (disable anonymous, add Grafana auth) |
| T-06-02-04 | Denial of Service | Loki disk fill via unbounded log retention | accept | Single-binary filesystem mode; RUNBOOK.md documents log rotation and retention tuning for production |
</threat_model>
<verification>
- 5 tests in backend/tests/test_logging.py PASS (no XFAIL).
- `grep -c "class CorrelationIDMiddleware" backend/main.py` returns 1.
- CorrelationIDMiddleware does NOT inherit BaseHTTPMiddleware.
- `docker compose config --quiet` exits 0.
- celery-beat block has neither `logging:` label nor `read_only:` key.
- All 3 YAML files (compose, loki-config, promtail-config) parse without error.
</verification>
<success_criteria>
- D-01 satisfied: setup_logging emits JSON when LOG_JSON=true; correlation_id appears in every log line; X-Correlation-ID header on every response.
- D-02 satisfied: docker compose up brings loki on :3100, grafana on :3000, promtail scrapes containers labelled logging=promtail.
- D-03 satisfied: no opentelemetry dependency added, no tracing middleware introduced.
- Zero new failing tests; the 5 logging xfails from 06-01 now PASS.
</success_criteria>
<output>
Create `.planning/phases/06-performance-production-hardening/06-02-SUMMARY.md` when done. Include: full-suite pytest summary, exact lines added to docker-compose.yml (count of new keys), and a one-line note confirming celery-beat was deliberately left untouched.
</output>
@@ -0,0 +1,235 @@
---
phase: 06-performance-production-hardening
plan: 03
type: execute
wave: 1
depends_on:
- 06-01
files_modified:
- backend/requirements-dev.txt
- backend/load_tests/locustfile.py
- backend/load_tests/README.md
autonomous: false
requirements:
- D-04
- D-05
- D-06
user_setup:
- service: locust
why: "Load testing runs OUTSIDE the production Docker image; install on the host or in a dedicated venv"
env_vars:
- name: LOAD_TEST_EMAIL
source: "Local .env or shell export — defaults to loadtest@example.com"
- name: LOAD_TEST_PASSWORD
source: "Local .env or shell export — defaults to a fixed value matching AUTH-01 strength rules"
must_haves:
truths:
- "locust is installable from requirements-dev.txt and is NOT installed by the production Dockerfile"
- "locustfile.py exposes a DocuVaultUser HttpUser class with login → list → upload tasks weighted realistically"
- "Running `locust --headless --users 50 --spawn-rate 10 --run-time 5m` against a running stack exits 0 only when p95 < 200ms AND p99 < 500ms AND fail_ratio ≤ 1%"
- "Locust on_start is self-bootstrapping — calls /api/auth/register first (409 conflict ignored), then /api/auth/login"
- "The upload task path matches the actual documents.py shape (presigned upload-url → PUT → confirm OR direct /upload — confirmed against documents.py before commit)"
artifacts:
- path: "backend/requirements-dev.txt"
provides: "locust pin (separate from production requirements.txt)"
contains: "locust>=2.34.0"
- path: "backend/load_tests/locustfile.py"
provides: "Full Locust user class + SLA gating quitting listener"
contains: "class DocuVaultUser"
min_lines: 80
- path: "backend/load_tests/README.md"
provides: "Run instructions, prerequisites, env var documentation, expected exit codes"
key_links:
- from: "backend/load_tests/locustfile.py on_start()"
to: "/api/auth/register and /api/auth/login"
via: "POST JSON {email, password, handle}"
pattern: "auth/(register|login)"
- from: "backend/load_tests/locustfile.py upload_document()"
to: "documents.py upload endpoint shape (presigned vs direct)"
via: "verified by reading backend/api/documents.py first"
pattern: "api/documents/(upload|upload-url)"
- from: "backend/load_tests/locustfile.py check_sla()"
to: "environment.process_exit_code"
via: "@events.quitting.add_listener"
pattern: "events.quitting.add_listener"
---
<objective>
Implement the Locust load test (D-04, D-05, D-06) that satisfies the SLA targets in Phase 6 success criterion 1: 50 concurrent users for a 5-minute soak with p50 < 100ms, p95 < 200ms, p99 < 500ms, and zero endpoint failures. Promote the locustfile.py skeleton from 06-01 to a full implementation. Resolve RESEARCH.md Open Question 1 by reading documents.py and using the correct upload flow shape.
Purpose: Make Phase 6 SC-01 measurable and gate-able. Provide a single command (`locust --headless ...`) whose exit code tells the on-call engineer whether the latest deployment meets SLA.
Output: requirements-dev.txt with locust pinned, a complete locustfile.py with SLA-gating quitting listener, and a load_tests/README.md so a fresh operator can run the test without reading PLAN.md.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
@.planning/phases/06-performance-production-hardening/06-VALIDATION.md
@CLAUDE.md
@backend/api/documents.py
@backend/api/auth.py
@backend/tests/conftest.py
@backend/load_tests/locustfile.py
@backend/load_tests/__init__.py
<interfaces>
<!-- The Locust task structure mirrors a realistic user session per D-05. -->
From backend/api/auth.py:
- POST /api/auth/register — body {handle, email, password}, returns 201 on success or 409 if email exists.
- POST /api/auth/login — body {email, password, totp_code?, backup_code?}, returns 200 with {access_token: str, ...} on success.
- POST /api/auth/refresh — relies on the httpOnly refresh_token cookie which the Locust HttpUser session maintains automatically via the requests Session it wraps.
From backend/api/documents.py (verified at planning time):
- GET /api/documents/ — list endpoint, returns array.
- POST /api/documents/upload-url — body {filename, ...}, returns {upload_url, document_id}. Two-step presigned flow.
- POST /api/documents/{id}/confirm — finalises after MinIO PUT, atomic quota UPDATE.
- POST /api/documents/upload — alternative direct multipart flow used for cloud backends (also accepts target_backend=minio). Use the DIRECT /upload endpoint for the load test because it is a single HTTP call from the client's perspective and avoids needing a MinIO presigned PUT from the Locust process.
From RESEARCH.md Pattern 7 + Open Question 1 resolution:
- Locust HttpUser maintains an HttpSession (cookies preserved across calls within one virtual user) — refresh_token cookie persists between login and refresh calls.
</interfaces>
</context>
<tasks>
<task type="checkpoint:human-verify" gate="blocking-human">
<name>Task 1: Confirm load-test credential strategy</name>
<read_first>
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pitfall 4 — load test user not pre-created)
- .planning/phases/06-performance-production-hardening/06-CONTEXT.md (D-04 prerequisites)
</read_first>
<what-built>
Two strategies for the load-test user. The planner needs the operator to pick one before implementation.
Option A — self-bootstrapping: locustfile on_start calls POST /api/auth/register (silently swallows 409 if user already exists), then POST /api/auth/login. Pros: zero out-of-band setup; works against a clean DB. Cons: writes a real user into the production-shape DB; password defaults to a fixed string unless LOAD_TEST_PASSWORD env var is set.
Option B — pre-seeded user via Alembic seed: planner writes a one-off migration or makefile target to create the load-test user, locustfile only logs in. Pros: load test does not pollute the user table during runs. Cons: extra setup step; operator must run the seed before every fresh DB.
</what-built>
<how-to-verify>
1. Decide: Option A (self-bootstrap, default) or Option B (pre-seeded).
2. If Option A: confirm the load-test handle/email default — the planner uses `loadtestuser` / `loadtest@example.com` / `Loadtest123!@#` unless overridden by LOAD_TEST_EMAIL / LOAD_TEST_PASSWORD env vars. Confirm this password meets AUTH-01 strength rules: ≥12 chars, upper, lower, digit, special — "Loadtest123!@#" is 14 chars and has all classes.
3. If Option B: provide the seeding command you want the planner to document in README.md.
</how-to-verify>
<acceptance_criteria>
- User selects "A" or "B"; planner proceeds with that strategy in Task 3.
- If A: README.md documents that the loadtest user persists in the DB and can be cleaned up with `DELETE FROM users WHERE email='loadtest@example.com'`.
- If B: README.md documents the seeding command before-each-run.
</acceptance_criteria>
<resume-signal>Type "A" for self-bootstrap (default) or "B" for pre-seeded with the seeding command.</resume-signal>
</task>
<task type="auto">
<name>Task 2: Add locust to requirements-dev.txt</name>
<files>backend/requirements-dev.txt</files>
<read_first>
- backend/requirements.txt (current pins — do NOT modify this file)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Standard Stack section — "Locust is a dev/test dependency only")
</read_first>
<action>
Create backend/requirements-dev.txt (new file). Top of file: comment block stating "Development and load-test dependencies. NOT installed by the production Dockerfile. Install separately on the host or in a dedicated venv via: `pip install -r backend/requirements-dev.txt`." First include line: `-r requirements.txt` so the dev environment also installs everything from production. Then a `# Load testing (Phase 6 — D-04)` comment followed by `locust>=2.34.0`. Do NOT modify backend/requirements.txt — locust must NOT enter the production image (per RESEARCH.md and the gates checklist).
</action>
<verify>
<automated>test -f backend/requirements-dev.txt &amp;&amp; grep -c '^locust' backend/requirements-dev.txt &amp;&amp; ! grep -q '^locust' backend/requirements.txt</automated>
</verify>
<acceptance_criteria>
- File backend/requirements-dev.txt exists: `test -f backend/requirements-dev.txt` returns 0.
- File contains `-r requirements.txt`: `grep -c '^-r requirements.txt' backend/requirements-dev.txt` returns 1.
- File contains `locust>=2.34.0`: `grep -c '^locust>=2.34.0' backend/requirements-dev.txt` returns 1.
- backend/requirements.txt does NOT contain locust: `grep -c '^locust' backend/requirements.txt` returns 0.
- File parses as valid pip requirements: `pip install --dry-run -r backend/requirements-dev.txt 2>&amp;1 | grep -cE 'ERROR|error'` returns 0.
</acceptance_criteria>
<done>locust installable from requirements-dev.txt, production requirements.txt unchanged.</done>
</task>
<task type="auto">
<name>Task 3: Implement locustfile.py + README.md</name>
<files>backend/load_tests/locustfile.py, backend/load_tests/README.md</files>
<read_first>
- backend/api/documents.py (lines around `upload_document` and `confirm_upload` — confirm the request body shape for /upload and /upload-url + /{id}/confirm before finalising the upload task)
- backend/api/auth.py (register/login request body shapes)
- backend/load_tests/locustfile.py (the skeleton from 06-01 — replace bodies)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 7, Pitfall 4, Open Question 1 + 2)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (locustfile.py section)
</read_first>
<action>
Replace the body of backend/load_tests/locustfile.py with the full Pattern 7 implementation, parameterised by the credential strategy chosen in Task 1.
Top of file docstring: name, purpose, run command (`locust --headless --users 50 --spawn-rate 10 --run-time 5m --host http://localhost:8000 --csv backend/load_tests/results -f backend/load_tests/locustfile.py`), prerequisites, SLA exit-code semantics.
Imports: `import os`, `from io import BytesIO`, `from locust import HttpUser, task, between, events`. NO imports from backend application code.
Constants: TEST_EMAIL = os.environ.get("LOAD_TEST_EMAIL", "loadtest@example.com"); TEST_PASSWORD = os.environ.get("LOAD_TEST_PASSWORD", "Loadtest123!@#"); TEST_HANDLE = "loadtestuser".
Class DocuVaultUser(HttpUser): wait_time = between(0.5, 2.0); access_token: str = "".
Method on_start(self): if strategy A — POST /api/auth/register with {handle: TEST_HANDLE, email: TEST_EMAIL, password: TEST_PASSWORD} catching 409 silently; THEN POST /api/auth/login with {email, password}; on 200 set self.access_token from response JSON access_token; on any other status call self.environment.runner.quit().
Method _auth_headers(self): return {"Authorization": f"Bearer {self.access_token}"}.
Tasks (weighted per D-05 realistic session):
@task(5) list_documents(self): self.client.get("/api/documents/", headers=self._auth_headers(), name="GET /api/documents/").
@task(2) upload_document(self): use the DIRECT /api/documents/upload endpoint with form-data (target_backend="minio", file fake PDF bytes b"%PDF-1.4 ..." in a BytesIO), headers from _auth_headers, name="POST /api/documents/upload". Confirm before committing that documents.py /upload accepts the target_backend form field.
@task(3) get_document(self): list first to pick an id; if list is non-empty pick the first doc and GET /api/documents/{id}, name="GET /api/documents/{id}"; if empty, skip silently (do not record as failure).
@task(1) refresh_token(self): POST /api/auth/refresh, name="POST /api/auth/refresh" — relies on HttpSession cookie jar to carry refresh_token.
SLA listener (module-level): `@events.quitting.add_listener def check_sla(environment, **kwargs):` reads environment.runner.stats.total; if fail_ratio > 0.01 → environment.process_exit_code = 1; elif p95 > 200 → 1; elif p99 > 500 → 1.
Create backend/load_tests/README.md documenting: purpose, install command (`pip install -r backend/requirements-dev.txt`), run command (full headless example), env vars (LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD with safe defaults), SLA semantics, exit-code mapping, cleanup tip for the loadtest user, and a note that load tests run OUTSIDE the production container.
</action>
<verify>
<automated>cd backend &amp;&amp; pip install -q locust &amp;&amp; python -c "import ast; ast.parse(open('load_tests/locustfile.py').read())" &amp;&amp; cd backend &amp;&amp; locust -f load_tests/locustfile.py --headless --users 1 --spawn-rate 1 --run-time 5s --host http://localhost:8000 --only-summary 2>&amp;1 | tail -5</automated>
</verify>
<acceptance_criteria>
- locustfile.py is syntactically valid: `python -c "import ast; ast.parse(open('backend/load_tests/locustfile.py').read())"` exits 0.
- `grep -c "class DocuVaultUser" backend/load_tests/locustfile.py` returns 1.
- `grep -c "raise NotImplementedError" backend/load_tests/locustfile.py` returns 0 (all skeleton stubs replaced).
- `grep -c "events.quitting.add_listener" backend/load_tests/locustfile.py` returns 1.
- `grep -cE "p95|get_response_time_percentile.0\\.95" backend/load_tests/locustfile.py` returns ≥ 1.
- `grep -cE "p99|get_response_time_percentile.0\\.99" backend/load_tests/locustfile.py` returns ≥ 1.
- `grep -cE "^from (backend|api|services|db|deps)" backend/load_tests/locustfile.py` returns 0 (no application imports).
- `grep -c "@task" backend/load_tests/locustfile.py` returns 4 (list, upload, get, refresh).
- locust accepts the file without parse errors: `locust -f backend/load_tests/locustfile.py --list-commands 2>&amp;1 | grep -cE 'error|Error|invalid'` returns 0.
- README.md exists: `test -f backend/load_tests/README.md` returns 0.
- README.md mentions LOAD_TEST_EMAIL: `grep -c LOAD_TEST_EMAIL backend/load_tests/README.md` returns ≥ 1.
- The actual /api/documents/upload endpoint shape was confirmed against documents.py before commit (planner records this in SUMMARY).
</acceptance_criteria>
<done>locustfile.py runs without parse errors, defines four weighted tasks, SLA listener gates exit code on p95/p99/fail_ratio, README.md explains how to operate it.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Locust host process → backend API | Same as any HTTP client; uses real auth tokens, exercises real rate limits |
| LOAD_TEST_PASSWORD env var → process memory | Credential lives only in env vars and the HTTP session — never written to disk by locust |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-06-03-01 | Information Disclosure | Load-test credentials hardcoded in version control | mitigate | TEST_EMAIL/TEST_PASSWORD read from env vars with non-sensitive defaults; README documents using a real password via LOAD_TEST_PASSWORD; .env already in .gitignore |
| T-06-03-02 | Tampering | Locust added to production Docker image | mitigate | locust placed in requirements-dev.txt only; production Dockerfile installs only requirements.txt; grep gate verifies locust is absent from requirements.txt |
| T-06-03-03 | Denial of Service | Load test against production by mistake | accept | Operator responsibility; --host flag is required for locust to run; README warns to use http://localhost:8000 or a dedicated staging URL |
| T-06-03-04 | Tampering | locustfile import of application code couples test harness to deployment | mitigate | grep gate enforces zero imports from backend/api/services/db/deps in locustfile.py |
| T-06-03-SC | Tampering | locust install from PyPI | mitigate | Package legitimacy verified in 06-01 Task 1 (blocking human checkpoint); install gated behind that approval |
</threat_model>
<verification>
- backend/requirements.txt has no `^locust` line.
- backend/requirements-dev.txt has `^locust>=2.34.0`.
- locustfile.py parses, defines DocuVaultUser, has four @task decorators, has the SLA quitting listener, and zero application imports.
- README.md exists and documents the run command, env vars, SLA semantics.
- Operator-confirmed credential strategy (A or B) recorded in SUMMARY.
</verification>
<success_criteria>
- D-04 satisfied: Locust is the load tester, file lives at backend/load_tests/locustfile.py, runnable headless.
- D-05 satisfied: Tasks cover login (on_start) → list → get → upload, weighted realistically; cloud endpoints excluded.
- D-06 satisfied: SLA listener exits non-zero when p95 > 200ms OR p99 > 500ms OR fail_ratio > 1%.
- locust kept out of production image (gates checklist item).
</success_criteria>
<output>
Create `.planning/phases/06-performance-production-hardening/06-03-SUMMARY.md` when done. Include: credential strategy chosen (A/B), the exact upload endpoint shape used (presigned three-step vs direct one-step) and why, full locustfile line count, sample dry-run output (5s run against a running backend).
</output>
@@ -0,0 +1,269 @@
---
phase: 06-performance-production-hardening
plan: 04
type: execute
wave: 2
depends_on:
- 06-02
files_modified:
- backend/Dockerfile
- docker-compose.yml
autonomous: false
requirements:
- D-07
- D-08
- D-09
user_setup: []
must_haves:
truths:
- "backend image runs as uid=1000 (appuser), not root — docker run --rm <image> id returns uid=1000(appuser)"
- "backend container's root filesystem is read-only (read_only: true) and writes to /tmp succeed because tmpfs is mounted mode=1777"
- "celery-worker container has the same read_only + tmpfs + cap_drop + no-new-privileges hardening as backend"
- "celery-beat is INTENTIONALLY left without read_only/cap_drop because it writes celerybeat-schedule to its working directory (Pitfall 7)"
- "Both backend and celery-worker drop ALL Linux capabilities (cap_drop: [ALL]) and forbid privilege escalation (security_opt: no-new-privileges)"
- "tempfile.NamedTemporaryFile (services/extractor.py) still succeeds inside the hardened container — the mode=1777 sticky-bit tmpfs is writable by appuser"
artifacts:
- path: "backend/Dockerfile"
provides: "Multi-stage builder/runtime image; runtime stage creates appuser uid=1000 and runs USER appuser"
contains: "FROM python:3.12-slim AS builder"
min_lines: 15
- path: "docker-compose.yml"
provides: "Hardened backend + celery-worker service definitions (read_only/tmpfs/cap_drop/security_opt); celery-beat deliberately unmodified"
contains: "read_only: true"
key_links:
- from: "backend/Dockerfile runtime stage"
to: "docker-compose.yml backend.read_only"
via: "appuser uid=1000 + tmpfs mode=1777 cooperate so non-root writes to /tmp succeed"
pattern: "USER appuser"
- from: "docker-compose.yml backend.tmpfs"
to: "services/extractor.py tempfile.NamedTemporaryFile"
via: "/tmp:mode=1777 is the destination tempfile picks via TMPDIR/default"
pattern: "/tmp:mode=1777"
- from: "docker-compose.yml celery-beat block"
to: "Pitfall 7 in 06-RESEARCH.md"
via: "EXPLICITLY NOT hardened — preserves writable filesystem for celerybeat-schedule"
pattern: "celery-beat:"
---
<objective>
Harden the production container image (D-07) and the docker-compose runtime for backend + celery-worker (D-08, D-09). Replace the current single-stage root-running Dockerfile with a multi-stage build whose runtime stage creates appuser (uid=1000) and copies installed packages from a discardable builder stage. Add `read_only: true`, `tmpfs: /tmp:mode=1777`, `cap_drop: [ALL]`, and `security_opt: no-new-privileges:true` to the backend and celery-worker services in docker-compose.yml. Leave celery-beat untouched (Pitfall 7 — it writes celerybeat-schedule to its working directory).
Purpose: Satisfy Phase 6 success criteria 3 + 4 (non-root container, read-only rootfs, dropped capabilities). Reduce blast radius of a backend RCE: an attacker cannot write to the image filesystem, escalate privileges, or use a kernel capability to break out.
Output: A rewritten Dockerfile that produces a runtime image with appuser; docker-compose.yml service blocks that apply the four hardening keys to backend and celery-worker only; verification commands that prove uid=1000, read-only rootfs writes fail, and tmpfs writes succeed.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
@CLAUDE.md
@backend/Dockerfile
@docker-compose.yml
@backend/services/extractor.py
@backend/requirements.txt
<interfaces>
<!-- Key constraints the executor must respect. -->
Current Dockerfile shape (single stage, root user):
- FROM python:3.12-slim
- Runs apt-get install tesseract-ocr libgl1 libglib2.0-0 then `pip install -r requirements.txt`
- COPY . . and EXPOSE 8000 — no USER directive (defaults to root).
Multi-stage target shape (from RESEARCH.md Pattern 5):
- Stage 1 `builder` (python:3.12-slim) — installs build tooling (gcc) + `pip install --no-cache-dir --prefix=/install -r requirements.txt`.
- Stage 2 `runtime` (python:3.12-slim) — installs RUNTIME apt deps (tesseract-ocr, libgl1, libglib2.0-0), `COPY --from=builder /install /usr/local`, creates appuser via groupadd/useradd uid=1000, COPY --chown=appuser:appgroup . /app, `USER appuser`, EXPOSE 8000, CMD uvicorn.
docker-compose service shape (per RESEARCH.md Pattern 6 + Critical Note 4/5):
- Add to backend AND celery-worker:
- `read_only: true`
- `tmpfs: ["/tmp:mode=1777"]` (long-form list with mode option; mode=1777 = world-writable + sticky bit so appuser uid=1000 can write tempfile.NamedTemporaryFile output)
- `cap_drop: [ALL]`
- `security_opt: ["no-new-privileges:true"]`
- DO NOT add to celery-beat.
Existing `volumes:` mount `./backend:/app` on backend and celery-worker is a bind mount; bind mounts are writable even when read_only=true on the container rootfs. The /app path remains writable for dev reload — verified by the docker spec (read_only applies to the image layer, not declared mounts).
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Rewrite backend/Dockerfile to multi-stage with appuser</name>
<files>backend/Dockerfile</files>
<read_first>
- backend/Dockerfile (current single-stage form — line 117)
- backend/requirements.txt (verify list of packages compatible with --prefix install)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 5; Assumption A3 — pip --prefix copy)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (backend/Dockerfile section)
</read_first>
<behavior>
- The built image's default user is appuser (uid=1000), not root: `docker run --rm <image> id` returns `uid=1000(appuser) gid=1000(appgroup)`.
- All Python imports from requirements.txt resolve in the runtime stage: `docker run --rm <image> python -c "import fastapi, structlog, sqlalchemy, minio, celery"` exits 0.
- Runtime system deps are installed in the runtime stage (tesseract-ocr, libgl1, libglib2.0-0): `docker run --rm <image> tesseract --version` exits 0.
- The builder stage is discarded — final image does NOT contain gcc: `docker run --rm <image> sh -c "command -v gcc" || true` returns empty.
- The runtime stage uses `--no-install-recommends` to keep apt footprint minimal.
- Image still exposes port 8000 and CMD launches uvicorn with the same host/port as before.
</behavior>
<action>
Replace backend/Dockerfile content with a two-stage build per RESEARCH.md Pattern 5.
Stage 1 header: `FROM python:3.12-slim AS builder`. WORKDIR /build. Install build tooling: `apt-get update && apt-get install -y --no-install-recommends gcc && rm -rf /var/lib/apt/lists/*`. Copy requirements.txt. Run `pip install --no-cache-dir --prefix=/install -r requirements.txt`.
Stage 2 header: `FROM python:3.12-slim AS runtime`. Install RUNTIME apt deps: `apt-get update && apt-get install -y --no-install-recommends tesseract-ocr libgl1 libglib2.0-0 && rm -rf /var/lib/apt/lists/*`. `COPY --from=builder /install /usr/local`. Create non-root user: `RUN groupadd --gid 1000 appgroup && useradd --uid 1000 --gid appgroup --shell /bin/sh --no-create-home appuser`. WORKDIR /app. `COPY --chown=appuser:appgroup . .`. `USER appuser`. `EXPOSE 8000`. `CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]`.
Per the docker-compose `command:` override (`uvicorn main:app --host 0.0.0.0 --port 8000 --reload`), the CMD is replaced at runtime in dev — but the Dockerfile CMD remains the production default. Do NOT bake `--reload` into the CMD.
Use `--no-install-recommends` on BOTH apt-get invocations.
Do NOT add the locust dependency anywhere in this Dockerfile (per 06-03 boundary: locust lives in requirements-dev.txt only).
</action>
<verify>
<automated>cd backend &amp;&amp; docker build -t docuvault-backend:phase6 . &amp;&amp; docker run --rm docuvault-backend:phase6 id | grep -c "uid=1000(appuser)"</automated>
</verify>
<acceptance_criteria>
- `grep -c "FROM python:3.12-slim AS builder" backend/Dockerfile` returns 1.
- `grep -c "FROM python:3.12-slim AS runtime" backend/Dockerfile` returns 1.
- `grep -c "^USER appuser" backend/Dockerfile` returns 1.
- `grep -c "useradd --uid 1000" backend/Dockerfile` returns 1.
- `grep -c "COPY --from=builder" backend/Dockerfile` returns 1.
- `grep -c "no-install-recommends" backend/Dockerfile` returns 2 (both apt-get blocks).
- `grep -c "tesseract-ocr" backend/Dockerfile` returns 1 (in runtime stage only — NOT in builder stage; the builder needs only gcc).
- `grep -c "EXPOSE 8000" backend/Dockerfile` returns 1.
- `grep -c "CMD .uvicorn" backend/Dockerfile` returns 1.
- `grep -c "\\--reload" backend/Dockerfile` returns 0 (reload is a compose-level override only).
- Build succeeds: `cd backend &amp;&amp; docker build -t docuvault-backend:phase6 .` exits 0.
- `docker run --rm docuvault-backend:phase6 id` output contains `uid=1000(appuser)`.
- `docker run --rm docuvault-backend:phase6 python -c "import fastapi, structlog, sqlalchemy, minio, celery, structlog"` exits 0 (verifies A3 — the --prefix/COPY pattern populated site-packages correctly).
- `docker run --rm docuvault-backend:phase6 tesseract --version` exits 0 (runtime apt deps present).
- `docker run --rm docuvault-backend:phase6 sh -c "command -v gcc"` exits non-zero or returns empty (gcc was in builder stage only).
</acceptance_criteria>
<done>Multi-stage image builds, runs as uid=1000, all production Python imports resolve, runtime apt deps present, build tooling absent.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Apply read_only + tmpfs + cap_drop + no-new-privileges to backend and celery-worker in docker-compose.yml</name>
<files>docker-compose.yml</files>
<read_first>
- docker-compose.yml (lines 49104 — backend, celery-worker, celery-beat blocks; volumes block)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 6 hardening additions; Pitfall 1, 3, 6, 7; Critical Note 5)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (docker-compose.yml section; Critical Note 4)
- backend/services/extractor.py (line 18 area — tempfile.NamedTemporaryFile usage that drives the mode=1777 requirement)
</read_first>
<behavior>
- `docker compose config --quiet` exits 0 (compose file is still valid YAML/schema).
- After `docker compose up backend`, `docker compose exec backend sh -c 'touch /tmp/probe && echo ok'` writes successfully (tmpfs writable by appuser).
- After `docker compose up backend`, `docker compose exec backend sh -c 'touch /probe 2>&1 || echo readonly'` reports "readonly" (rootfs is read_only).
- `docker compose exec backend cat /proc/self/status | grep CapEff` shows an empty/zero effective capability set (cap_drop: ALL took effect).
- The celery-beat block contains NEITHER `read_only:` NOR `cap_drop:` NOR `tmpfs:` keys (Pitfall 7 — celerybeat-schedule must remain writable).
- The celery-worker block has the SAME four hardening keys as backend.
- The backend bind mount `./backend:/app` remains and continues to provide live source for `--reload` dev mode (bind mounts are unaffected by `read_only: true` on the container rootfs).
</behavior>
<action>
Edit docker-compose.yml in place. For the backend service block (currently lines 4979), add the following four keys as siblings of existing keys (build, ports, volumes, environment, command, depends_on). Insert after `depends_on:` (or at a stable, readable location):
- `read_only: true`
- `tmpfs:` with a single list entry `"/tmp:mode=1777"` (use the long-form string with mode option — mode=1777 makes the tmpfs world-writable with the sticky bit so appuser uid=1000 can write the tempfile.NamedTemporaryFile output that services/extractor.py creates).
- `cap_drop:` with a single list entry `ALL`.
- `security_opt:` with a single list entry `no-new-privileges:true`.
For the celery-worker block (currently lines 81103), add the SAME four keys with the SAME values. The celery worker runs the same image and the same temp-file-producing extractor; it needs identical hardening.
For the celery-beat block (currently lines 105124), DO NOT add any of those four keys. Leave the block exactly as-is. Add a single comment line directly above the celery-beat key declaration: ` # celery-beat: NOT hardened — writes celerybeat-schedule to working directory (Phase 6 Pitfall 7).`
Do NOT remove or modify the existing `volumes: - ./backend:/app` bind mounts — bind mounts remain writable under read_only because read_only applies only to the image layer.
Do NOT remove or modify environment variables, healthchecks, or extra_hosts.
Do NOT add the Loki/Promtail/Grafana labels here — those landed in 06-02 already (06-02 added `labels: logging: "promtail"` on backend and celery-worker). If those labels are missing because 06-02 has not yet shipped, ADD them now alongside the four hardening keys to keep the two plans composable — but if they are already present, leave them untouched.
</action>
<verify>
<automated>docker compose config --quiet &amp;&amp; awk '/^ backend:/,/^ [a-z]/' docker-compose.yml | grep -c "read_only: true"</automated>
</verify>
<acceptance_criteria>
- `docker compose config --quiet` exits 0.
- `awk '/^ backend:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -c "read_only: true"` returns 1.
- `awk '/^ backend:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -cE "/tmp:mode=1777"` returns 1.
- `awk '/^ backend:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -cE "^\\s*-\\s*ALL\\s*$"` returns 1.
- `awk '/^ backend:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -cE "no-new-privileges:true"` returns 1.
- `awk '/^ celery-worker:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -c "read_only: true"` returns 1.
- `awk '/^ celery-worker:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -cE "/tmp:mode=1777"` returns 1.
- `awk '/^ celery-worker:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -cE "no-new-privileges:true"` returns 1.
- celery-beat must have NONE of these keys: `awk '/^ celery-beat:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -cE "read_only:|cap_drop:|tmpfs:|security_opt:|no-new-privileges"` returns 0.
- Bind mount preserved: `awk '/^ backend:/,/^ [a-z][a-z-]*:/' docker-compose.yml | grep -c "./backend:/app"` returns 1.
- Smoke test (runtime): `docker compose up -d backend &amp;&amp; sleep 5 &amp;&amp; docker compose exec -T backend sh -c 'touch /tmp/probe &amp;&amp; rm /tmp/probe &amp;&amp; echo TMP_OK'` prints "TMP_OK" (tmpfs writable).
- Smoke test (rootfs read-only): `docker compose exec -T backend sh -c 'touch /probe 2>&amp;1; ls /probe 2>/dev/null || echo ROOTFS_RO'` prints "ROOTFS_RO".
- Smoke test (extractor): upload a small PDF via the API and confirm classify works (proves tempfile.NamedTemporaryFile succeeded under the new constraints). If a full upload is heavy, instead `docker compose exec -T backend python -c "import tempfile; t = tempfile.NamedTemporaryFile(delete=False); t.write(b'x'); t.close(); print('TEMPFILE_OK')"` exits 0 and prints "TEMPFILE_OK".
</acceptance_criteria>
<done>Backend and celery-worker run read_only with mode=1777 tmpfs /tmp, all capabilities dropped, no privilege escalation; celery-beat untouched; live-reload bind mount still works; extractor's tempfile pattern still succeeds.</done>
</task>
<task type="checkpoint:human-verify" gate="blocking">
<name>Task 3: Human smoke test — full stack up, upload + extract under hardened runtime</name>
<read_first>
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pitfall 6 — PyMuPDF cache paths; Pitfall 1 — tmpfs mode)
- backend/services/extractor.py
- backend/tasks/document_tasks.py
</read_first>
<what-built>
The Dockerfile produces a uid=1000 runtime image; docker-compose.yml runs backend and celery-worker with read_only rootfs, tmpfs /tmp mode=1777, ALL capabilities dropped, no-new-privileges enabled. Celery-beat is intentionally left writable.
</what-built>
<how-to-verify>
1. From repo root: `docker compose down -v` then `docker compose up -d --build` and wait ~30s.
2. Confirm appuser: `docker compose exec -T backend id` should print `uid=1000(appuser) gid=1000(appgroup) groups=1000(appgroup)`.
3. Confirm read_only rootfs: `docker compose exec -T backend sh -c 'touch /readonly_probe 2>&1 || echo "rootfs is read-only"'` should print "rootfs is read-only".
4. Confirm tmpfs writable: `docker compose exec -T backend sh -c 'touch /tmp/ok &amp;&amp; ls -la /tmp/ok'` should succeed.
5. Confirm celery-worker has the same hardening: `docker compose exec -T celery-worker id` returns uid=1000; `docker compose exec -T celery-worker sh -c 'touch /readonly_probe 2>&1 || echo RO'` prints RO; `docker compose exec -T celery-worker sh -c 'touch /tmp/ok'` succeeds.
6. Confirm celery-beat is UNHARDENED (intentionally): `docker compose exec -T celery-beat sh -c 'touch /tmp/probe &amp;&amp; touch /app/probe; ls /app/probe'` should succeed (writable rootfs — required for celerybeat-schedule).
7. End-to-end upload test: register a fresh user via the UI (or curl /api/auth/register), log in, upload a small PDF, wait ~10s for the Celery task. Confirm in the UI/API that the document appears with extracted text and classified topics. This validates Pitfall 6 (PyMuPDF cache under read_only) is not blocking extraction.
8. Inspect docker stats: `docker stats --no-stream` — backend and celery-worker should be running healthy (not restarting in a loop).
9. Inspect logs for any "Read-only file system" or "Permission denied" errors: `docker compose logs backend celery-worker | grep -iE "read-only|permission denied"` — should be empty (or only contain expected ignored writes, not failures).
</how-to-verify>
<acceptance_criteria>
- User confirms steps 28 all pass; step 9 produces zero hard errors that crash request handling.
- If step 7 (extraction) fails with a Read-only/Permission error, user reports it — planner adds an additional tmpfs mount (e.g. `/var/cache/fontconfig`) in a follow-up before approving.
- User responds "approved" to advance Wave 3 (06-06 security gate + RUNBOOK).
</acceptance_criteria>
<resume-signal>Type "approved" if all checks pass; otherwise paste the failing command output for triage.</resume-signal>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Attacker exploits backend RCE → container filesystem | Hardening (read_only + cap_drop + no-new-privileges) limits post-RCE actions |
| Container process → host kernel | Dropped capabilities + no privilege escalation make kernel-API abuse harder |
| Container process → other containers on same network | Out of scope for hardening (compose default bridge network) — network policy is Phase 7+ |
| Build-time gcc → runtime image | Multi-stage build keeps build tools out of the runtime image, reducing CVE attack surface |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-06-04-01 | Elevation of Privilege | Container escape via root-owned process exploit | mitigate | Multi-stage Dockerfile creates appuser uid=1000; USER appuser drops root before CMD runs (D-07) |
| T-06-04-02 | Tampering | Attacker writes a webshell or modifies application code on container filesystem | mitigate | read_only: true on backend + celery-worker prevents all writes outside declared tmpfs/volumes (D-08) |
| T-06-04-03 | Elevation of Privilege | Attacker uses Linux capability (CAP_NET_RAW, CAP_SYS_PTRACE, etc) to expand reach | mitigate | cap_drop: [ALL]; no cap_add — port 8000 is unprivileged (D-09) |
| T-06-04-04 | Elevation of Privilege | Attacker invokes setuid binary to regain root inside container | mitigate | security_opt: no-new-privileges:true blocks setuid bit from elevating (D-09 belt-and-braces) |
| T-06-04-05 | Denial of Service | tempfile.NamedTemporaryFile fails under read_only because /tmp is not writable by appuser | mitigate | tmpfs: /tmp:mode=1777 — sticky world-writable bit per Pitfall 1 (RESEARCH.md) — verified by Task 3 step 4 |
| T-06-04-06 | Denial of Service | celery-beat cannot write celerybeat-schedule under read_only | accept | celery-beat INTENTIONALLY left unhardened per Pitfall 7; security risk lower (no inbound network port); RUNBOOK.md (06-06) documents the deferral |
| T-06-04-07 | Information Disclosure | Build tooling (gcc) shipped in runtime image expands CVE surface | mitigate | Multi-stage build discards builder; runtime image contains only runtime apt deps + Python site-packages |
</threat_model>
<verification>
- `docker compose config --quiet` exits 0.
- `cd backend && docker build -t docuvault-backend:phase6 .` exits 0; resulting image `id` returns uid=1000(appuser).
- backend and celery-worker blocks each contain `read_only: true`, `/tmp:mode=1777` tmpfs, `cap_drop: [ALL]`, `security_opt: no-new-privileges:true`.
- celery-beat block contains NONE of those four keys.
- Live smoke (Task 3) confirms /tmp writable, rootfs writes fail, full upload+extract round-trip works.
- Existing backend test suite still passes (no test regressions): `cd backend && pytest tests/ --no-header -q` baseline unchanged.
</verification>
<success_criteria>
- D-07 satisfied: Dockerfile is multi-stage with appuser uid=1000; runtime image contains only runtime deps + Python packages.
- D-08 satisfied: backend + celery-worker run with `read_only: true` and `tmpfs: /tmp:mode=1777`; celery-beat preserved as writable.
- D-09 satisfied: backend + celery-worker drop ALL capabilities and forbid privilege escalation.
- No regression: existing pytest suite green; full Docker stack starts; upload + extract works end-to-end under the hardened runtime.
</success_criteria>
<output>
Create `.planning/phases/06-performance-production-hardening/06-04-SUMMARY.md` when done. Include: built image ID + size before/after multi-stage, `docker run --rm <image> id` output, `docker compose config --quiet` exit code, smoke test results from Task 3 (each numbered step + pass/fail), explicit confirmation that celery-beat was left untouched, and any tmpfs additions made beyond /tmp:mode=1777 (e.g. /var/cache/fontconfig if Pitfall 6 surfaced).
</output>
@@ -0,0 +1,323 @@
---
phase: 06-performance-production-hardening
plan: 05
type: execute
wave: 2
depends_on:
- 06-02
files_modified:
- backend/deps/utils.py
- backend/api/auth.py
- backend/services/rate_limiting.py
- backend/main.py
- backend/api/documents.py
- backend/api/cloud.py
- backend/tests/test_rate_limiting.py
autonomous: true
requirements:
- D-11
- D-12
- D-13
user_setup: []
must_haves:
truths:
- "get_client_ip in deps/utils.py uses a trusted-proxy CIDR check — requests from untrusted peers ignore X-Forwarded-For; requests from 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, ::1 read the leftmost XFF entry"
- "backend/api/auth.py uses key_func=get_client_ip on its Limiter — get_remote_address from slowapi.util is no longer imported"
- "A second Limiter instance (account_limiter) keyed by request.state.current_user.id exists in backend/services/rate_limiting.py and is reused by documents.py and cloud.py"
- "Every authenticated endpoint in documents.py and cloud.py that previously had no per-account limit now has @account_limiter.limit('100/minute') AND sets request.state.current_user = current_user as the first statement of its handler body"
- "Existing per-IP limits on auth endpoints (10/minute, 5/hour) are PRESERVED — only the key_func changed"
- "All 8 xfail stubs in backend/tests/test_rate_limiting.py from 06-01 now PASS (xfail markers removed)"
- "Assumption A1 (slowapi key_func evaluation order with request.state) is converted to a passing test before per-account decorators are applied to all endpoints"
artifacts:
- path: "backend/deps/utils.py"
provides: "Replaced get_client_ip body with trusted-proxy CIDR logic; module-level _TRUSTED_PROXY_NETS list; private _is_trusted_proxy helper"
contains: "_TRUSTED_PROXY_NETS"
- path: "backend/services/rate_limiting.py"
provides: "Single canonical account_limiter Limiter instance + _account_key function for per-user rate limiting (D-12)"
exports: ["account_limiter"]
min_lines: 25
- path: "backend/api/auth.py"
provides: "Limiter now uses key_func=get_client_ip (D-11/D-13); existing @limiter.limit decorators on register/login/refresh/password endpoints unchanged"
contains: "key_func=get_client_ip"
- path: "backend/api/documents.py"
provides: "Per-account 100/minute limit on every authenticated endpoint; request.state.current_user set as first handler line"
contains: "@account_limiter.limit"
- path: "backend/api/cloud.py"
provides: "Same per-account 100/minute limit pattern on every endpoint that uses get_regular_user"
contains: "@account_limiter.limit"
- path: "backend/main.py"
provides: "account_limiter imported alongside the existing auth limiter; both limiters live as module-level singletons"
contains: "from services.rate_limiting import account_limiter"
key_links:
- from: "backend/deps/utils.py get_client_ip"
to: "backend/api/auth.py Limiter(key_func=...)"
via: "import + key_func wiring"
pattern: "key_func=get_client_ip"
- from: "backend/services/rate_limiting.py _account_key"
to: "request.state.current_user (set by route handler)"
via: "getattr fallback to IP for safety (Pitfall 3)"
pattern: "request\\.state\\.current_user"
- from: "Route handlers in documents.py / cloud.py"
to: "_account_key"
via: "first line of handler body: request.state.current_user = current_user"
pattern: "request\\.state\\.current_user\\s*=\\s*current_user"
---
<objective>
Close the rate-limit bypass and per-account fairness gaps for Phase 6. Replace the body of `get_client_ip` in backend/deps/utils.py with trusted-proxy CIDR logic (D-11). Switch the existing auth Limiter from slowapi's `get_remote_address` to `get_client_ip` (D-13 — preserves the existing 10/min and 5/hour limits but stops header spoofing from external clients). Add a second `account_limiter` Limiter instance keyed by `request.state.current_user.id` in a new shared module `backend/services/rate_limiting.py`, and decorate every authenticated endpoint in `documents.py` and `cloud.py` with `@account_limiter.limit("100/minute")` (D-12). Promote all 8 xfail stubs in `tests/test_rate_limiting.py` (from 06-01) to passing.
Purpose: Header-spoofing currently lets any external client claim an arbitrary IP and bypass the IP-based rate limiter. D-11 closes that. D-12 adds a second axis (per-account) so a single compromised account cannot exhaust the application's request capacity by rotating IPs. Together they form the rate-limit half of Phase 6's hardening.
Output: One body-replacement in deps/utils.py (NO new function, NO new public API), two lines changed in auth.py, a new ~30-line rate_limiting.py service module, and decorator/first-line additions across every authenticated endpoint in documents.py and cloud.py.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
@CLAUDE.md
@backend/deps/utils.py
@backend/api/auth.py
@backend/api/documents.py
@backend/api/cloud.py
@backend/main.py
@backend/tests/test_rate_limiting.py
@backend/tests/conftest.py
<interfaces>
<!-- Key signatures and contracts the executor must respect. -->
Existing get_client_ip (backend/deps/utils.py:10):
- Signature: `def get_client_ip(request: Request) -> Optional[str]:`
- Currently returns `request.headers.get("X-Forwarded-For") or (request.client.host if request.client else None)`
- TASK: replace BODY only. Signature is unchanged. Add no new public functions. `ipaddress` is stdlib — no new package dependency.
- Existing callers: backend/api/auth.py, audit log helpers, etc. They import by name from deps.utils; the import chain stays identical.
Existing slowapi limiter (backend/api/auth.py:3744):
- `from slowapi.util import get_remote_address`
- `limiter = Limiter(key_func=get_remote_address)`
- TASK: drop the slowapi.util import; add `from deps.utils import get_client_ip`; change `key_func=get_remote_address` to `key_func=get_client_ip`. Two-line patch. Existing @limiter.limit decorators stay exactly as written.
New shared module (backend/services/rate_limiting.py):
- Exports: `account_limiter: Limiter`, `_account_key(request: Request) -> str` (module-private; reused only inside this file).
- `_account_key` reads `getattr(request.state, "current_user", None)` and returns `str(user.id)` when set; falls back to `request.client.host or "anonymous"` to avoid raising in untested code paths (Pitfall 3).
- Why a separate module: CLAUDE.md mandates one canonical definition for shared utilities; both documents.py and cloud.py will import the same `account_limiter` instance so rate-limit counters are shared across routers.
Per-account decorator contract on each authenticated route:
- `request: Request` MUST appear in the signature (slowapi looks it up by name on the function's first parameters).
- `@account_limiter.limit("100/minute")` decorates the handler.
- First line of handler body: `request.state.current_user = current_user` (so _account_key has the user object available when slowapi evaluates the key — A1 assumption).
- Apply to every route in documents.py and cloud.py that uses `Depends(get_regular_user)`.
Routes in documents.py to decorate (verified at planning time, lines 88, 129, 290, 399, 521, 565, 616, 693, 746):
POST /api/documents/upload-url
POST /api/documents/upload
POST /api/documents/{doc_id}/confirm
GET /api/documents
GET /api/documents/{doc_id}
PATCH /api/documents/{doc_id}
DELETE /api/documents/{doc_id}
POST /api/documents/{doc_id}/classify
GET /api/documents/{doc_id}/content
That's 9 endpoints. All currently take `current_user: User = Depends(get_regular_user)`.
Routes in cloud.py to decorate (verified at planning time, those using get_regular_user — at lines 315, 407, 553, 644, 677, 728, 773, 928 and others):
GET /api/cloud/oauth/initiate/{provider}
GET /api/cloud/oauth/callback/{provider}
POST /api/cloud/connections/webdav
GET /api/cloud/connections
GET /api/cloud/connections/{connection_id}/config
DELETE /api/cloud/connections/{connection_id}
GET /api/cloud/folders/{provider}/{folder_id:path}
(… and any other route in cloud.py that injects get_regular_user — apply uniformly)
Note: oauth_callback returns a RedirectResponse — still safe to apply the limit; per-account is rate-limit on internal users, not OAuth providers.
Wave 0 stubs to promote (backend/tests/test_rate_limiting.py — 8 xfails):
1. test_get_client_ip_untrusted_returns_direct_peer
2. test_get_client_ip_trusted_proxy_reads_xff_leftmost
3. test_get_client_ip_trusted_proxy_no_xff_falls_back
4. test_get_client_ip_invalid_peer_returns_none_or_string
5. test_account_limiter_key_uses_user_id
6. test_account_limiter_key_falls_back_to_ip_when_no_user
7. test_account_limiter_key_ordering_assumption (A1 verification — gates D-12 rollout)
8. test_authenticated_endpoint_429_after_100_per_minute
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Replace get_client_ip body in deps/utils.py + create services/rate_limiting.py + promote unit-level xfails (tests 17)</name>
<files>backend/deps/utils.py, backend/services/rate_limiting.py, backend/tests/test_rate_limiting.py</files>
<read_first>
- backend/deps/utils.py (current body, lines 135 — replace the body of get_client_ip ONLY; do NOT add a second function, do NOT rename, do NOT add a module-level import that breaks existing callers)
- backend/tests/test_rate_limiting.py (the 8 xfail stubs from 06-01 — flip the first 7 to real assertions)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 3, Pattern 4, Pitfall 3, Assumption A1)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (deps/utils.py section; account_limiter section; Critical Note 1)
- CLAUDE.md ("Backend: shared module map" — `deps/utils.py` and the rule that no router may define a local variant)
</read_first>
<behavior>
- get_client_ip(request) with request.client.host == "8.8.8.8" and X-Forwarded-For: "1.2.3.4" returns "8.8.8.8" (untrusted peer → ignore XFF; D-11).
- get_client_ip(request) with request.client.host == "127.0.0.1" and X-Forwarded-For: "1.2.3.4, 5.6.7.8" returns "1.2.3.4" (trusted peer → leftmost XFF; D-11).
- get_client_ip(request) with request.client.host == "172.16.5.5" and no XFF returns "172.16.5.5" (trusted peer fallback).
- get_client_ip(request) with request.client is None returns None and does not raise.
- _account_key(request) with request.state.current_user.id == UUID(...) returns the str(UUID) — never the IP.
- _account_key(request) with no request.state.current_user returns the IP (or "anonymous") and does NOT raise (Pitfall 3 — failure mode is "more limiting", not crash).
- A FastAPI ASGI app with one endpoint decorated `@account_limiter.limit("100/minute")` that sets `request.state.current_user = current_user` as its FIRST line, called 101 times within a minute with the same authenticated user, returns 429 on the 101st call AND the limiter's internal counter records the key as `str(user.id)`, not the IP. This proves A1.
</behavior>
<action>
Edit backend/deps/utils.py:
(a) Add module-level imports at the top of the existing import block: `import ipaddress`. The `Optional` and `Request` imports already exist; do NOT re-import.
(b) Add a module-level constant directly after the existing imports: `_TRUSTED_PROXY_NETS` as a list of `ipaddress.ip_network(...)` objects for "127.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "::1/128".
(c) Add a private helper `_is_trusted_proxy(host: str) -> bool` that wraps `ipaddress.ip_address(host)` in a try/except ValueError and returns True if the address is in any of the trusted nets.
(d) Replace the BODY of the existing `get_client_ip(request: Request) -> Optional[str]` function with the trusted-proxy logic (RESEARCH.md Pattern 3): compute `direct_peer = request.client.host if request.client else None`; if `direct_peer and _is_trusted_proxy(direct_peer)`, read `request.headers.get("X-Forwarded-For")` and if present return its leftmost comma-split entry stripped of whitespace; otherwise return `direct_peer`. Update the docstring to note "D-11 — trusted-proxy CIDR check" and remove the old "use for audit logging only" warning (the function is now safe for rate-limiting too).
(e) DO NOT add a second function. DO NOT rename. DO NOT touch parse_uuid.
Create backend/services/rate_limiting.py (new file). Module docstring: "Per-account rate limiter shared across document and cloud routers (D-12)." Pattern: `from __future__ import annotations`, `from fastapi import Request`, `from slowapi import Limiter`. Define private function `_account_key(request: Request) -> str` per the behaviour spec — read `getattr(request.state, "current_user", None)`; when set return `str(user.id)`; when None fall back to `request.client.host if request.client else "anonymous"`. Define module-level singleton `account_limiter = Limiter(key_func=_account_key)`. Export both symbols (no __all__ required — they are top-level names).
Edit backend/tests/test_rate_limiting.py:
For tests 17 (the unit-level tests — get_client_ip variants, _account_key key selection, A1 ordering), REMOVE the `@pytest.mark.xfail(...)` decorators and REPLACE the single-line `pytest.xfail(...)` bodies with real assertions per the behaviour spec above. Use a lightweight mock Request (or starlette.requests.Request constructed from a minimal scope dict) — do NOT spin up the full app for unit tests 16.
For test 7 (A1 ordering): build a minimal FastAPI app inline (within the test), register one GET endpoint that sets `request.state.current_user = SimpleNamespace(id=uuid.uuid4())` as its first line, decorate it with `@account_limiter.limit("100/minute")`, mount via TestClient. Send 101 requests in a tight loop with the SAME peer IP but VARYING X-Forwarded-For; assert the 101st response is 429 (proves the key is the constant user.id, NOT the per-request IP). Skip integration test #8 in this task — Task 2 handles it.
</action>
<verify>
<automated>cd backend &amp;&amp; pytest tests/test_rate_limiting.py -v --no-header -k "not test_authenticated_endpoint_429_after_100_per_minute" 2>&amp;1 | tail -5 | grep -E "7 passed"</automated>
</verify>
<acceptance_criteria>
- `grep -c "^def get_client_ip" backend/deps/utils.py` returns 1 (single canonical definition — no new function added).
- `grep -c "_TRUSTED_PROXY_NETS" backend/deps/utils.py` returns ≥ 2 (definition + use).
- `grep -c "^import ipaddress" backend/deps/utils.py` returns 1.
- `grep -c "_is_trusted_proxy" backend/deps/utils.py` returns ≥ 2 (definition + use).
- The replaced get_client_ip body no longer reads X-Forwarded-For unconditionally: `awk '/^def get_client_ip/,/^def [a-zA-Z]/' backend/deps/utils.py | grep -cE 'request\\.headers\\.get\\("X-Forwarded-For"\\)\\s*or'` returns 0.
- `test -f backend/services/rate_limiting.py` exits 0.
- `grep -c "^account_limiter = Limiter" backend/services/rate_limiting.py` returns 1.
- `grep -c "def _account_key" backend/services/rate_limiting.py` returns 1.
- Module imports cleanly: `cd backend &amp;&amp; python -c "from services.rate_limiting import account_limiter, _account_key"` exits 0.
- Tests 17 in test_rate_limiting.py are no longer xfail and now PASS: `cd backend &amp;&amp; pytest tests/test_rate_limiting.py -v --no-header -k "not test_authenticated_endpoint_429_after_100_per_minute" | tail -3 | grep -c "7 passed"` returns 1.
- `grep -c "pytest.mark.xfail" backend/tests/test_rate_limiting.py` returns 1 (only the integration test #8 remains xfail until Task 2).
- All other existing pytest tests continue to pass: no NEW failures introduced.
</acceptance_criteria>
<done>get_client_ip body replaced in-place per CLAUDE.md single-canonical-definition rule; account_limiter singleton lives in services/rate_limiting.py; 7 unit tests pass including the A1 ordering test that gates Task 2.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Switch auth.py Limiter to get_client_ip + decorate documents.py and cloud.py with @account_limiter + promote integration xfail (test 8)</name>
<files>backend/api/auth.py, backend/api/documents.py, backend/api/cloud.py, backend/main.py, backend/tests/test_rate_limiting.py</files>
<read_first>
- backend/api/auth.py (lines 3045 — Limiter declaration; lines 3744 — current key_func)
- backend/api/documents.py (the 9 authenticated endpoints listed in the <interfaces> block; signatures around lines 88, 129, 290, 399, 521, 565, 616, 693, 746)
- backend/api/cloud.py (the endpoints using get_regular_user listed in the <interfaces> block)
- backend/main.py (existing app.state.limiter assignment and SlowAPIMiddleware registration — confirm both stay as-is for the IP limiter; account_limiter does not need app.state wiring)
- backend/tests/test_rate_limiting.py (the remaining xfail integration test #8)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 4 + Pitfall 3 — slowapi key_func evaluation order and Request-first-param requirement)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (auth.py section; documents.py per-account pattern; cloud.py section)
</read_first>
<behavior>
- backend/api/auth.py imports get_client_ip from deps.utils and passes it as key_func to its Limiter; get_remote_address is no longer imported.
- Existing @limiter.limit("10/minute") and @limiter.limit("5/hour") decorators on register/login/refresh/password endpoints remain unchanged (D-13 — limits preserved).
- Every authenticated endpoint in documents.py (9 endpoints) has `@account_limiter.limit("100/minute")` directly above the route function definition and `request.state.current_user = current_user` as the FIRST line of the handler body.
- Every authenticated endpoint in cloud.py (the routes using Depends(get_regular_user)) has the same decorator and same first-line assignment.
- Every decorated endpoint's signature has `request: Request` as a parameter (some already have it; for those that don't, add it as a parameter so slowapi can find it by name).
- main.py imports `account_limiter` from `services.rate_limiting` (alongside the existing auth limiter wiring); no additional app.state wiring is required for account_limiter because its decorators are applied directly to handlers.
- Integration test #8 (test_authenticated_endpoint_429_after_100_per_minute) now PASSES — 101 GET /api/documents/ requests with the SAME auth token returns 429 on the 101st.
- Per-account limit shape: 100 requests per minute per authenticated user.id, shared across all decorated endpoints (because they share the same account_limiter instance).
- Existing IP limits on auth endpoints still trigger correctly — register/login 429s after 10 attempts/min from the same client IP (verified by re-running the existing auth rate-limit tests).
</behavior>
<action>
Edit backend/api/auth.py:
(a) Remove the `from slowapi.util import get_remote_address` import line.
(b) The existing `from deps.utils import get_client_ip` import is already present (line 34) — do NOT re-add.
(c) Change line 44: `limiter = Limiter(key_func=get_remote_address)` to `limiter = Limiter(key_func=get_client_ip)`.
(d) Do NOT modify any of the existing @limiter.limit(...) decorators — they preserve the existing 10/min and 5/hour limits per D-13.
Edit backend/main.py:
(a) Add an import line near the other service imports: `from services.rate_limiting import account_limiter`.
(b) Leave `app.state.limiter = auth_limiter` (or whatever name is currently used) UNCHANGED — SlowAPIMiddleware drives only the limiter assigned to app.state. The account_limiter is invoked directly via its decorators.
Edit backend/api/documents.py:
For EACH of the 9 endpoints listed in the <interfaces> block (request_upload_url, upload_document, confirm_upload, list_documents, get_document, patch_document, delete_document, classify_document, stream_document_content):
(a) Add at the top of the file (alongside the existing imports): `from services.rate_limiting import account_limiter`.
(b) Above each route function definition, immediately under the `@router.<verb>(...)` decorator, add `@account_limiter.limit("100/minute")`.
(c) Ensure the function signature has `request: Request` as a parameter (some handlers may already have it — keep). If a handler doesn't, ADD `request: Request` as the first parameter after any path/body params (FastAPI parameter order rules allow Request anywhere; slowapi finds it by type/name).
(d) Make `request.state.current_user = current_user` the FIRST executable statement of the handler body — before any other logic. This is the line that satisfies the A1 contract proven by Task 1's ordering test.
Edit backend/api/cloud.py:
Apply the same three changes (import account_limiter; decorate with @account_limiter.limit("100/minute"); first-line `request.state.current_user = current_user`) to EVERY endpoint in cloud.py whose signature contains `Depends(get_regular_user)`. Use `grep -n "Depends(get_regular_user)" backend/api/cloud.py` to enumerate the targets and ensure 100% coverage. The endpoint at /api/cloud/oauth/callback/{provider} returns a RedirectResponse — that's fine; rate-limit applies to the authenticated user regardless of response type.
Edit backend/tests/test_rate_limiting.py:
Remove the `@pytest.mark.xfail(...)` decorator from test_authenticated_endpoint_429_after_100_per_minute. Replace its single-line body with real assertions: use the async_client fixture from conftest.py and the auth_user fixture, hit GET /api/documents/ 101 times in a tight loop with the same Bearer token, assert the 101st response is 429. Implementation note: because slowapi's default in-memory storage is per-process, the test must run within a single test client lifetime; reset isn't required because each test gets a fresh app from conftest.
Do NOT modify any endpoint signature beyond adding `request: Request` where missing. Do NOT change @router.<verb>(...) URL paths, body models, response models, or dependency lists.
</action>
<verify>
<automated>cd backend &amp;&amp; pytest tests/test_rate_limiting.py tests/test_auth*.py -v --no-header 2>&amp;1 | tail -5 | grep -E "passed"</automated>
</verify>
<acceptance_criteria>
- `grep -c "from slowapi.util import get_remote_address" backend/api/auth.py` returns 0 (import removed).
- `grep -c "key_func=get_client_ip" backend/api/auth.py` returns 1 (Limiter switched to trusted-proxy key).
- `grep -c "key_func=get_remote_address" backend/api/auth.py` returns 0.
- All existing `@limiter.limit(` decorators in auth.py remain — `grep -c "@limiter.limit(" backend/api/auth.py` returns the SAME number as before this plan (use git diff to confirm).
- `grep -c "from services.rate_limiting import account_limiter" backend/main.py` returns 1.
- `grep -c "from services.rate_limiting import account_limiter" backend/api/documents.py` returns 1.
- `grep -c "from services.rate_limiting import account_limiter" backend/api/cloud.py` returns 1.
- Every authenticated endpoint in documents.py has the per-account decorator: `grep -c "@account_limiter.limit" backend/api/documents.py` returns 9 (matches the 9 endpoints listed in <interfaces>).
- Every documents.py decorated handler sets request.state.current_user as its first line: `grep -c "request\\.state\\.current_user\\s*=\\s*current_user" backend/api/documents.py` returns 9.
- Every cloud.py endpoint using get_regular_user has the decorator: `[ "$(grep -c 'Depends(get_regular_user)' backend/api/cloud.py)" = "$(grep -c '@account_limiter.limit' backend/api/cloud.py)" ]` exits 0 (equal counts).
- Every cloud.py decorated handler sets request.state.current_user as first line: `[ "$(grep -c '@account_limiter.limit' backend/api/cloud.py)" = "$(grep -c 'request\\.state\\.current_user\\s*=\\s*current_user' backend/api/cloud.py)" ]` exits 0.
- All 8 tests in test_rate_limiting.py PASS: `cd backend &amp;&amp; pytest tests/test_rate_limiting.py -v --no-header | tail -3 | grep -c "8 passed"` returns 1.
- `grep -c "pytest.mark.xfail" backend/tests/test_rate_limiting.py` returns 0 (no remaining xfails).
- Existing auth rate-limit tests still pass (no regression on the per-IP limits): `cd backend &amp;&amp; pytest tests/ -v --no-header -k "rate_limit or limiter or test_auth" | tail -3 | grep -c "failed"` returns 0.
- Full backend pytest run shows no NEW failures vs. the 06-04 baseline.
</acceptance_criteria>
<done>auth.py keyed by get_client_ip with old limits preserved; documents.py and cloud.py uniformly decorated with the per-account limiter; all 8 rate-limiting tests pass; no signature/URL/body-model changes leaked into the patch.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| External client → FastAPI | Client controls all HTTP headers, including X-Forwarded-For; FastAPI must distinguish trusted-proxy origin from arbitrary internet origin |
| Reverse proxy → FastAPI (intra-cluster) | Trusted CIDR; X-Forwarded-For is authoritative for the originating client |
| Authenticated user → application capacity | A compromised account rotating IPs bypasses the IP limiter; per-account limit closes that hole |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-06-05-01 | Tampering | Rate-limit bypass via X-Forwarded-For spoofing from external clients | mitigate | get_client_ip body replaced with trusted-proxy CIDR check (D-11); external clients can no longer claim arbitrary IPs |
| T-06-05-02 | Denial of Service | Compromised account exhausts request capacity by rotating IPs (defeats per-IP limit) | mitigate | account_limiter keyed by str(current_user.id) caps each user at 100 req/min across all decorated endpoints (D-12) |
| T-06-05-03 | Tampering | account_limiter key_func runs BEFORE request.state.current_user is set → silently falls back to IP, defeats D-12 | mitigate | A1 verification test (test_account_limiter_key_ordering_assumption) in Task 1 — the first-line `request.state.current_user = current_user` contract is verified before applying decorators to all endpoints in Task 2 |
| T-06-05-04 | Denial of Service | _account_key raises when request.state.current_user is unset → crashes request handling | mitigate | _account_key returns IP fallback or "anonymous" string instead of raising (Pitfall 3) |
| T-06-05-05 | Information Disclosure | Per-account 429 reveals authenticated user enumeration to attacker | accept | The 429 only fires for already-authenticated users; the attacker already proved knowledge of credentials. Information leak is bounded by the existing per-IP limit on login endpoints (preserved by D-13). |
| T-06-05-06 | Tampering | Duplicate get_client_ip definitions drift apart over time (CLAUDE.md anti-pattern) | mitigate | Replace BODY in-place; no new function created; grep gate verifies single canonical definition |
</threat_model>
<verification>
- `grep -c "^def get_client_ip" backend/deps/utils.py` returns 1.
- `grep -c "key_func=get_remote_address" backend/api/auth.py` returns 0.
- `grep -c "key_func=get_client_ip" backend/api/auth.py` returns 1.
- `grep -c "from services.rate_limiting import account_limiter" backend/main.py backend/api/documents.py backend/api/cloud.py` returns 3 (one per file).
- All 8 tests in backend/tests/test_rate_limiting.py PASS.
- Full pytest suite green; no regressions in existing auth rate-limit tests.
- Per-account decorator count matches authenticated endpoint count in documents.py (9) and equals the get_regular_user count in cloud.py.
</verification>
<success_criteria>
- D-11 satisfied: get_client_ip uses trusted-proxy CIDR validation; external clients can no longer spoof XFF.
- D-12 satisfied: account_limiter Limiter keyed by user.id; all documents.py + cloud.py authenticated endpoints decorated; per-account 100/minute enforced.
- D-13 satisfied: existing per-IP limits on auth endpoints (10/min, 5/hour) preserved verbatim; only the key_func changed.
- All 8 Wave 0 xfails promoted to PASS — including A1 ordering verification.
- Zero new failures in the full backend pytest suite.
</success_criteria>
<output>
Create `.planning/phases/06-performance-production-hardening/06-05-SUMMARY.md` when done. Include: line-by-line diff summary of backend/deps/utils.py (before/after body), count of @account_limiter.limit decorators added to documents.py and cloud.py, final pytest summary (passed/failed/xfailed), explicit confirmation that no new public function was added (CLAUDE.md compliance), and any auth/documents/cloud endpoints that were left undecorated with rationale.
</output>
@@ -0,0 +1,288 @@
---
phase: 06-performance-production-hardening
plan: 06
type: execute
wave: 3
depends_on:
- 06-04
- 06-05
files_modified:
- RUNBOOK.md
autonomous: false
requirements:
- D-10
- D-14
user_setup:
- service: docker-hub
why: "docker scout cves requires an authenticated Docker Hub session to submit the image manifest for CVE analysis (Pitfall 5)"
env_vars: []
dashboard_config:
- task: "Run `docker login` from the terminal where the security gate will be invoked"
location: "Local shell / CI environment that will run docker scout cves"
must_haves:
truths:
- "RUNBOOK.md exists at the repo root and is the single operational reference for Phase 6 deployment + on-call response"
- "RUNBOOK.md documents the docker scout cves invocation with zero-critical-CVEs gate (D-10)"
- "RUNBOOK.md enumerates every required env var with description, example value, and source (.env vs Docker secret vs cloud provider dashboard)"
- "RUNBOOK.md documents startup/shutdown procedures for the full docker-compose stack including the new Loki/Promtail/Grafana services from 06-02"
- "RUNBOOK.md documents backup strategy: pg_dump for Postgres + mc mirror for MinIO, with restore commands"
- "RUNBOOK.md documents health-check verification: /health endpoint, Loki :3100/ready, Grafana :3000/api/health, MinIO mc ready, redis-cli ping"
- "RUNBOOK.md documents on-call escalation: who to contact, in what order, for which alert types"
- "RUNBOOK.md documents common failure modes from RESEARCH.md Pitfalls 1, 5, 6, 7 with recovery commands"
- "A human verifies the docker scout cves run returns zero critical CVEs on the built image AFTER 06-04 ships the hardened Dockerfile — this is the final phase gate"
artifacts:
- path: "RUNBOOK.md"
provides: "Operational runbook — env vars, startup/shutdown, backup, health checks, escalation, failure modes"
contains: "docker scout cves"
min_lines: 200
key_links:
- from: "RUNBOOK.md docker scout section"
to: "06-04 hardened Dockerfile"
via: "image tag built by 06-04 is the scan target"
pattern: "docker scout cves"
- from: "RUNBOOK.md backup section"
to: "docker-compose.yml postgres + minio services"
via: "pg_dump and mc mirror against the volumes named in compose"
pattern: "pg_dump|mc mirror"
- from: "RUNBOOK.md health-check section"
to: "06-02 Loki/Grafana endpoints + existing /health"
via: "documented HTTP probes per service"
pattern: "loki.*3100|grafana.*3000"
---
<objective>
Close Phase 6 with the two remaining decisions: run the `docker scout cves` zero-critical-CVE gate against the hardened image (D-10), and ship `RUNBOOK.md` at the repo root as the single operational reference (D-14). This plan is the final gate before the phase is marked complete in STATE.md.
Purpose: The hardened Dockerfile (06-04) and rate-limiting changes (06-05) are inert without (a) proof the resulting image has no critical CVEs and (b) a written runbook so a fresh operator can stand up the stack, find the right env var, and respond to a page at 03:00 without reading PLAN.md. RUNBOOK.md is the artifact that turns "we hardened it" into "anyone on-call can run it."
Output: One RUNBOOK.md file at the repo root covering env vars, startup/shutdown, backups, health checks, escalation, and failure modes — plus a human-executed docker scout cves gate whose pass/fail decides whether the phase ships.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
@.planning/phases/06-performance-production-hardening/06-04-PLAN.md
@.planning/phases/06-performance-production-hardening/06-05-PLAN.md
@CLAUDE.md
@docker-compose.yml
@backend/Dockerfile
@backend/config.py
<interfaces>
<!-- The runbook content matrix — what MUST appear, organized by D-14 sub-clause. -->
D-14 required sections (verbatim from CONTEXT.md):
1. All required env vars with descriptions and examples
2. Docker Compose startup/shutdown procedures
3. Backup strategy: PostgreSQL (pg_dump cron) + MinIO (mc mirror)
4. Health check verification steps
5. On-call escalation path (who, in what order, for which alert types)
6. Common failure modes and recovery steps
Env var source (read backend/config.py — Settings class fields):
- DATABASE_URL, DATABASE_MIGRATE_URL
- MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MINIO_BUCKET, MINIO_PUBLIC_ENDPOINT
- REDIS_URL, REDIS_PASSWORD
- SECRET_KEY, REFRESH_TOKEN_SECRET (per Phase 2)
- ADMIN_EMAIL, ADMIN_PASSWORD
- CORS_ORIGINS, FRONTEND_URL
- CLOUD_CREDS_KEY (Phase 5)
- google_client_id / google_client_secret, onedrive_client_id / onedrive_client_secret (Phase 5)
- LOG_LEVEL, LOG_JSON (Phase 6 — added by 06-02)
- TRUSTED_PROXY_CIDRS (Phase 6 — referenced by 06-05; default list lives in deps/utils.py)
- LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD (Phase 6 — added by 06-03)
Health-check endpoints to document:
- Backend: GET http://localhost:8000/health (existing endpoint)
- Loki: GET http://localhost:3100/ready
- Grafana: GET http://localhost:3000/api/health
- MinIO: `docker compose exec minio mc ready local`
- Redis: `docker compose exec redis redis-cli -a $REDIS_PASSWORD ping` (expect "PONG")
- Postgres: `docker compose exec postgres pg_isready -U postgres -d docuvault`
docker scout cves gate command (from RESEARCH.md Code Examples + Pitfall 5):
Build: `cd backend && docker build -t docuvault-backend:phase6 .` (already produced by 06-04)
Scan: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code`
Exit code 0 = clean; exit code 2 = critical CVEs found
Prerequisite: `docker login` (Pitfall 5 — scout uploads manifest to Docker's service)
Fallback if docker scout is unavailable: `trivy image docuvault-backend:phase6` (note: trivy not installed by default — RUNBOOK documents brew install trivy)
Failure-mode entries to include (from RESEARCH.md Pitfalls):
- Pitfall 1: tempfile.NamedTemporaryFile fails → check /tmp tmpfs mode=1777
- Pitfall 5: docker scout returns auth error → run docker login
- Pitfall 6: PyMuPDF read-only error → may need additional tmpfs mount for /var/cache/fontconfig
- Pitfall 7: celery-beat won't start under read_only → confirm celery-beat block has no read_only key
On-call escalation template (since this is a solo dev project, document the realistic pattern):
- Primary: project owner (curo1305@curonet.de) — first responder for all alerts
- Secondary: empty (single-operator deployment) — RUNBOOK acknowledges escalation tier is the operator themselves; documents alert types and the recovery commands directly so the owner can self-serve
- Alert classes: backend-down (page immediately), celery-stuck (warn within 1h), loki-full-disk (warn within 4h), docker scout critical CVE (page within 24h)
RUNBOOK location: repo root, sibling to CLAUDE.md and README.md (D-14 verbatim).
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Write RUNBOOK.md at repo root</name>
<files>RUNBOOK.md</files>
<read_first>
- .planning/phases/06-performance-production-hardening/06-CONTEXT.md (D-14 — full content list)
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pitfalls 1, 5, 6, 7; Standard Stack; Environment Availability)
- .planning/phases/06-performance-production-hardening/06-PATTERNS.md (interfaces section in this plan summarises env vars)
- backend/config.py (authoritative list of env vars — every field in Settings class must appear in RUNBOOK with its env var name and default)
- docker-compose.yml (services, ports, volumes, healthchecks — RUNBOOK references all of them)
- backend/Dockerfile (hardened image tag from 06-04 — RUNBOOK uses it as scan target)
- CLAUDE.md ("Stack" + "Development Setup" + "Security Protocol" — RUNBOOK aligns with these conventions)
</read_first>
<action>
Create RUNBOOK.md at the repository root (sibling to CLAUDE.md, README.md, docker-compose.yml).
Top of file: H1 `# DocuVault Operational Runbook`. One-paragraph purpose statement: this is the single reference for running DocuVault in production-like environments — env vars, startup, backups, health checks, escalation, recovery. Cross-reference: "For architecture and rationale see CLAUDE.md; for phase history see .planning/ROADMAP.md."
Section 1 — `## Environment Variables`. Markdown table with columns: Name | Required | Description | Example | Source. Enumerate every Settings field from backend/config.py: DATABASE_URL, DATABASE_MIGRATE_URL, MINIO_*, REDIS_*, SECRET_KEY, REFRESH_TOKEN_SECRET (if defined), ADMIN_EMAIL, ADMIN_PASSWORD, CORS_ORIGINS, FRONTEND_URL, CLOUD_CREDS_KEY, google_client_id/secret, onedrive_client_id/secret, LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS, LOAD_TEST_EMAIL, LOAD_TEST_PASSWORD. For each row provide a realistic example (e.g. `postgresql+psycopg://docuvault_app:****@postgres:5432/docuvault`) and where it's set (`.env` file at repo root). Note: SECRET_KEY and CLOUD_CREDS_KEY must NEVER be committed; document the rotation procedure.
Section 2 — `## Startup / Shutdown`. Document the docker-compose startup sequence (`docker compose up -d --build`) and orderly shutdown (`docker compose down`, `docker compose down -v` for nuke-from-orbit including volumes). Reference the new services from 06-02 (loki, promtail, grafana) and their access URLs (Grafana :3000, Loki :3100). Include the dev-mode reload note (bind mount + --reload override). Document the celery-beat exception (Pitfall 7) — it does NOT have read_only hardening.
Section 3 — `## Backup Strategy`.
Postgres: command `docker compose exec postgres pg_dump -U postgres docuvault > backups/docuvault_$(date +%F).sql`; restore command `cat backups/docuvault_YYYY-MM-DD.sql | docker compose exec -T postgres psql -U postgres docuvault`. Document cron pattern for daily backups: a crontab line like `0 3 * * * cd /path/to/docuvault && docker compose exec -T postgres pg_dump -U postgres docuvault | gzip > backups/$(date +\%F).sql.gz`. Note: D-14 says backup strategy must be documented; automation is deferred per Deferred Idea "Backup automation".
MinIO: command `docker compose exec minio mc mirror /data /backup/minio-mirror` (mc client is installed inside the minio container). Document offsite-backup pattern using `mc mirror local-minio/<bucket> s3://offsite-bucket/<prefix>` against an S3-compatible target. Restore command: `mc mirror /backup/minio-mirror /data`.
Document retention policy: keep daily for 7 days, weekly for 4 weeks, monthly for 12 months — operator implements via cron + rotate.
Section 4 — `## Health Checks`. Markdown table with Service | Probe Command | Healthy Output:
Backend `curl -sf http://localhost:8000/health` returns `{"status":"ok",...}`
Loki `curl -sf http://localhost:3100/ready` returns `ready`
Grafana `curl -sf http://localhost:3000/api/health` returns JSON with `database: ok`
Postgres `docker compose exec postgres pg_isready -U postgres -d docuvault` exit 0
MinIO `docker compose exec minio mc ready local` exit 0
Redis `docker compose exec redis redis-cli -a $REDIS_PASSWORD ping` returns `PONG`
Celery worker `docker compose exec celery-worker celery -A celery_app inspect ping` returns `pong`
Section 5 — `## Security Gate — docker scout CVE scan (D-10)`.
Prerequisite: `docker login` (Pitfall 5 — scout uploads image manifest to Docker's analysis service).
Build step (already done by 06-04): `cd backend && docker build -t docuvault-backend:phase6 .`
Scan command: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code`
Pass criterion: exit code 0. Fail criterion: exit code 2 (critical CVEs found). Phase advancement requires pass.
Fallback if `docker scout` is unavailable (e.g. offline environment): `brew install trivy` (or `apt install trivy`), then `trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6`.
Cadence: run on every image rebuild; gate any prod deploy on pass.
On failure recovery: re-pull base image (`docker pull python:3.12-slim`), rebuild, rescan. If still failing, pin to a newer Python 3.12 patch release or rebuild against the latest python:3.13-slim after compatibility testing.
Section 6 — `## On-Call Escalation`.
State the realistic single-operator model: primary on-call is the project owner (curo1305@curonet.de); secondary escalation is the operator themselves; this is a solo deployment. Document the alert classes and the immediate action for each:
- `backend-down` (HTTP 5xx > 5% for 5min OR backend container restarting) → page immediately. Action: `docker compose logs --tail 100 backend`; check for `Read-only file system` (Pitfall 1/6) or `Permission denied` (Pitfall 5). If extractor errors, see Section 7 → "tempfile permission" entry.
- `celery-stuck` (queue depth > 1000 OR no task completion for 30min) → warn within 1h. Action: `docker compose logs --tail 200 celery-worker`; restart `docker compose restart celery-worker`.
- `loki-full-disk` (Loki container reports > 80% disk usage on /loki) → warn within 4h. Action: prune old chunks (Loki retention tuning — link to upstream docs); or rotate the loki_data volume.
- `docker scout critical-CVE` (scheduled re-scan finds new critical CVE in base image) → page within 24h. Action: rebuild image with latest python:3.12-slim and re-scan; if persistent, see Section 5 fallback.
- `quota-violation-spike` (audit log shows > 10 quota_exceeded events per hour from same user) → warn. Action: check user; possible abuse — disable account via admin endpoint.
- `failed-login-spike` (audit log shows > 100 failed_login events per hour from same IP) → warn. Action: verify per-IP limiter is firing (10/min); if not, check that 06-05 deployed and TRUSTED_PROXY_CIDRS is correct for the deployment topology.
Section 7 — `## Common Failure Modes`. Markdown table or H3 entries — each with Symptom | Cause | Recovery:
- Symptom: "PermissionError: [Errno 13] /tmp/..." in backend logs → Cause: tmpfs missing mode=1777 (Pitfall 1) → Recovery: verify docker-compose.yml backend.tmpfs value contains "/tmp:mode=1777"; recreate the container with `docker compose up -d --force-recreate backend`.
- Symptom: docker scout returns "authentication required" → Cause: Pitfall 5 → Recovery: run `docker login` first, then rerun the scan.
- Symptom: PyMuPDF "Read-only file system" when extracting PDF → Cause: Pitfall 6 — PyMuPDF writing to /var/cache/fontconfig under read_only → Recovery: add `tmpfs: ["/var/cache/fontconfig"]` to backend service in docker-compose.yml.
- Symptom: celery-beat container exits "Permission denied: 'celerybeat-schedule'" → Cause: Pitfall 7 — read_only was accidentally added to celery-beat → Recovery: remove `read_only:` from celery-beat block in docker-compose.yml; restart.
- Symptom: After 06-05 ships, requests from external clients suddenly hit rate-limit despite low traffic → Cause: a real reverse proxy is in front but its IP is not in TRUSTED_PROXY_CIDRS, so its peer IP is used as the rate-limit key for all traffic → Recovery: add the proxy's CIDR to TRUSTED_PROXY_CIDRS env var and restart backend.
- Symptom: per-account 429s when expected per-IP 429s → Cause: route handler missing the `request.state.current_user = current_user` first-line assignment → Recovery: locate the handler missing the line, add it as the first executable statement, redeploy.
- Symptom: Grafana UI loads but no Loki datasource → Cause: Loki container not ready when Grafana started OR loki-config.yaml malformed → Recovery: check `curl http://localhost:3100/ready`; if not 200, inspect `docker compose logs loki` for YAML errors.
Section 8 — `## Phase 6 Deferred Items`. List the four deferred ideas from CONTEXT.md so the next operator knows the runbook acknowledges them: HTTPS/TLS termination (add reverse proxy — pattern documented in Section 5), horizontal scaling (Redis-backed limiter — Phase 7), CI/CD pipeline (GitHub Actions for scout + locust — Phase 7), backup automation (cron service — manual procedure above is the current state).
Format requirements: GitHub-flavored Markdown, line wrap at ~100 chars where possible, consistent code-fence language tags (`bash` for shell, `yaml` for compose snippets, `text` for output samples). Add a one-line "Last updated: {today}" footer.
</action>
<verify>
<automated>test -f RUNBOOK.md &amp;&amp; wc -l RUNBOOK.md | awk '{print ($1 >= 200) ? "OK" : "TOO_SHORT"}' &amp;&amp; grep -cE "docker scout cves|pg_dump|mc mirror|TRUSTED_PROXY_CIDRS|LOG_JSON|celery-beat" RUNBOOK.md</automated>
</verify>
<acceptance_criteria>
- `test -f RUNBOOK.md` exits 0 (file present at repo root).
- `wc -l RUNBOOK.md` returns ≥ 200 lines.
- All eight section headings present: `grep -cE "^##\\s+(Environment Variables|Startup|Backup Strategy|Health Checks|Security Gate|On-Call Escalation|Common Failure Modes|Phase 6 Deferred Items)" RUNBOOK.md` returns 8.
- Docker scout command appears: `grep -c "docker scout cves" RUNBOOK.md` returns ≥ 1.
- All required env vars appear by name (test 12 representative): `grep -cE "DATABASE_URL|MINIO_ENDPOINT|REDIS_URL|SECRET_KEY|CLOUD_CREDS_KEY|LOG_LEVEL|LOG_JSON|TRUSTED_PROXY_CIDRS|CORS_ORIGINS|FRONTEND_URL|ADMIN_EMAIL|LOAD_TEST_EMAIL" RUNBOOK.md` returns ≥ 12.
- Backup commands present: `grep -c "pg_dump" RUNBOOK.md` returns ≥ 1 AND `grep -c "mc mirror" RUNBOOK.md` returns ≥ 1.
- Health-check commands present for at least 6 services: `grep -cE "/health|/ready|/api/health|pg_isready|mc ready|redis-cli.*ping" RUNBOOK.md` returns ≥ 6.
- Pitfall recoveries documented: `grep -cE "mode=1777|docker login|fontconfig|celerybeat-schedule|TRUSTED_PROXY_CIDRS|request\\.state\\.current_user" RUNBOOK.md` returns ≥ 6.
- Escalation section present with at least 4 alert classes: `grep -cE "backend-down|celery-stuck|loki-full-disk|critical-CVE" RUNBOOK.md` returns ≥ 4.
- Cross-reference to CLAUDE.md present: `grep -c "CLAUDE.md" RUNBOOK.md` returns ≥ 1.
- Markdown is parseable (basic check — no unbalanced code fences): `awk '/^```/ {n++} END {exit (n % 2 == 0) ? 0 : 1}' RUNBOOK.md` exits 0.
</acceptance_criteria>
<done>RUNBOOK.md present at repo root, ≥200 lines, all 8 D-14 content areas covered, every Phase 6 env var documented with example, all major Pitfall recoveries entered into Failure Modes section.</done>
</task>
<task type="checkpoint:human-verify" gate="blocking-human">
<name>Task 2: Run docker scout cves zero-critical gate (D-10)</name>
<read_first>
- .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Code Examples — docker scout cves command; Pitfall 5)
- .planning/phases/06-performance-production-hardening/06-04-PLAN.md (Task 1 — produces the image tag docuvault-backend:phase6)
- RUNBOOK.md (Section 5 — Security Gate — the runbook is the source of truth for the command and prerequisite)
</read_first>
<what-built>
Plan 06-04 produces a hardened multi-stage backend image tagged `docuvault-backend:phase6` (uid=1000, no build tools, read-only-friendly). This checkpoint runs the D-10 zero-critical-CVE gate against it. Pitfall 5 requires `docker login` before scout can submit the image manifest for analysis.
</what-built>
<how-to-verify>
1. Confirm 06-04 has shipped: `docker image inspect docuvault-backend:phase6 >/dev/null 2>&amp;1 &amp;&amp; echo IMAGE_PRESENT` should print `IMAGE_PRESENT`. If absent, run `cd backend &amp;&amp; docker build -t docuvault-backend:phase6 .` first.
2. Authenticate Docker Hub (Pitfall 5 prerequisite): `docker login` — enter Docker Hub credentials. Skip if already logged in.
3. Run the scan: `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code`
4. Read the output. Expected pass: exit code 0 and the summary line shows `0 critical` (and ideally low numbers in High/Medium/Low). Expected fail: exit code 2 with a list of CRITICAL CVEs and their package origins.
5. If FAIL: identify the failing package(s) from the report, attempt remediation:
a. Rebuild against latest base image: `docker pull python:3.12-slim` then `cd backend &amp;&amp; docker build --no-cache -t docuvault-backend:phase6 .` and rerun step 3.
b. If CVE comes from a Python package, bump the pin in backend/requirements.txt to a patched version and rebuild.
c. If unresolvable in this phase, escalate to the operator with the CVE list — the phase cannot advance until critical CVEs are zero.
6. If docker scout is unavailable (offline / interpreter error), use the documented fallback: `trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6` (install trivy first per RUNBOOK Section 5).
</how-to-verify>
<acceptance_criteria>
- User runs `docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code` (or documented trivy fallback) and reports the exit code.
- Exit code 0 → gate passes; respond "approved" with the summary line pasted (e.g. "0 critical, 2 high, 15 medium, 40 low").
- Exit code non-zero → gate fails; user pastes the CVE list. Planner triages: either bump a package pin (within this plan as a follow-up task) or escalate as a blocking issue that prevents phase completion.
- User confirms they ran `docker login` first (or that trivy was used as the offline fallback) so the Pitfall 5 prerequisite is documented in the SUMMARY.
</acceptance_criteria>
<resume-signal>Type "approved" with the pasted "X critical" summary line, OR paste the CVE list for triage.</resume-signal>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Operator → production stack | Operator runs the documented commands; RUNBOOK is the contract for what's safe to run |
| Image base layers → runtime | Upstream python:3.12-slim and apt packages may ship CVEs over time; scout gate catches them |
| Repository file → secrets exposure | RUNBOOK must reference env vars without hardcoding actual secrets |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-06-06-01 | Tampering | Critical CVE in base image / dependency ships to production | mitigate | docker scout cves zero-critical gate (D-10) — human checkpoint blocks phase advancement on non-zero exit (Task 2) |
| T-06-06-02 | Information Disclosure | RUNBOOK accidentally hardcodes a real secret value | mitigate | All example values use obvious placeholder patterns (****, CHANGEME, your-token-here); explicit warning that SECRET_KEY and CLOUD_CREDS_KEY must never be committed |
| T-06-06-03 | Repudiation | Operator does not know which alert maps to which recovery procedure | mitigate | RUNBOOK Section 6 enumerates alert classes; Section 7 maps each Pitfall symptom to its recovery command — operator can self-serve at 03:00 |
| T-06-06-04 | Tampering | docker scout requires Docker Hub auth (Pitfall 5) → gate silently fails open if operator skips login | mitigate | RUNBOOK Section 5 documents the prerequisite explicitly; Task 2 checkpoint instructions force the operator to confirm login before running scan; gate uses --exit-code so scan failures cannot be ignored |
| T-06-06-05 | Denial of Service | Backup procedure not exercised → restore on production failure does not work | accept | D-14 says document the strategy; restore commands are present (Section 3); rehearsal cadence is operator responsibility documented as TODO in the runbook |
| T-06-06-SC | Tampering | docker scout itself may be tampered with (binary integrity) | accept | docker scout ships with Docker Engine 29.5.2 (verified at research time); supply-chain risk delegated to Docker Inc.; trivy fallback exists if scout is compromised |
</threat_model>
<verification>
- RUNBOOK.md exists at repo root with ≥200 lines and all 8 D-14 content sections.
- `grep -c "docker scout cves" RUNBOOK.md` returns ≥ 1.
- All Phase 6 env vars (LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS, LOAD_TEST_*) documented.
- Pitfall recoveries (1, 5, 6, 7) entered into Failure Modes section.
- Human-executed docker scout cves gate returns exit code 0 (or trivy equivalent with zero critical CVEs) — confirmed in Task 2 checkpoint.
- Pitfall 5 prerequisite (docker login) executed before the scan; confirmation recorded in SUMMARY.
</verification>
<success_criteria>
- D-10 satisfied: docker scout cves run against docuvault-backend:phase6 returns zero critical CVEs (or fallback trivy scan does); result documented in 06-06-SUMMARY.md.
- D-14 satisfied: RUNBOOK.md exists at repo root and covers env vars, startup/shutdown, backup strategy, health checks, on-call escalation, and common failure modes.
- Phase 6 gate is green: all six decisions covered by plans 06-04/05/06 are implemented and verified, and the security scan blocks the phase from advancing if critical CVEs are present.
</success_criteria>
<output>
Create `.planning/phases/06-performance-production-hardening/06-06-SUMMARY.md` when done. Include: RUNBOOK.md final line count and section headings list, the exact docker scout command run and its full output (or trivy equivalent), the pass/fail decision, any package pin bumps made to resolve CVEs, and a one-line confirmation that `docker login` was executed before the scan (Pitfall 5).
</output>
@@ -845,19 +845,19 @@ locust \
---
## Open Questions
## Open Questions (RESOLVED)
1. **Upload endpoint for load testing**
1. **Upload endpoint for load testing** (RESOLVED: 06-03 `<interfaces>` block — use POST /api/documents/upload direct multipart endpoint for load test; presigned flow excluded per D-05)
- What we know: `services/extractor.py` uses `tempfile.NamedTemporaryFile`; the upload flow may be two-step (presigned URL) or one-step depending on the current documents.py implementation
- What's unclear: Whether Locust needs to implement the three-step presigned flow (upload-url → PUT to MinIO → confirm) or if a simpler direct POST exists
- Recommendation: Read `backend/api/documents.py` lines for the upload endpoint before finalizing the locustfile; if two-step, implement the MinIO PUT step using `self.client.put` to MinIO's host:9000
2. **Load test user bootstrap**
2. **Load test user bootstrap** (RESOLVED: 06-03 Task 1 human checkpoint — use `on_start` register-then-login pattern; catch 409 conflict)
- What we know: Locust needs a valid user to authenticate; no seed user for load testing exists yet
- What's unclear: Whether to create the user via the registration API (on_start) or pre-seed via Alembic
- Recommendation: Use `on_start` to register if not exists (catch 409), then login — self-contained, no DB dependency
3. **Loki tmpfs mount persistence**
3. **Loki tmpfs mount persistence** (RESOLVED: 06-02 Task 3 — named volume `loki_data:/loki` used; loki service not subject to read_only)
- What we know: Loki service needs a writable `/loki` directory for chunk storage
- What's unclear: Whether the named volume `loki_data:/loki` is sufficient or if Loki's own container permissions require a matching UID
- Recommendation: Use named volume (not read_only on loki service — it's not a user-facing service)
@@ -0,0 +1,90 @@
---
phase: 6
slug: performance-production-hardening
status: draft
nyquist_compliant: false
wave_0_complete: false
created: 2026-06-02
---
# Phase 6 — Validation Strategy
> Per-phase validation contract for feedback sampling during execution.
---
## Test Infrastructure
| Property | Value |
|----------|-------|
| **Framework** | pytest 8.2 + pytest-asyncio (asyncio_mode=auto) |
| **Config file** | `backend/pytest.ini` |
| **Quick run command** | `cd backend && pytest tests/ -v -x --tb=short` |
| **Full suite command** | `cd backend && pytest tests/ -v` |
| **Estimated runtime** | ~45 seconds |
---
## Sampling Rate
- **After every task commit:** Run `cd backend && pytest tests/ -v -x --tb=short`
- **After every plan wave:** Run `cd backend && pytest tests/ -v`
- **Before `/gsd:verify-work`:** Full suite must be green
- **Max feedback latency:** 60 seconds
---
## Per-Task Verification Map
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
| 06-W0-01 | Wave 0 | 0 | D-01 | — | structlog emits JSON with correlation_id | unit | `pytest tests/test_logging.py -x` | ❌ Wave 0 | ⬜ pending |
| 06-W0-02 | Wave 0 | 0 | D-11 | T-06-01 | get_client_ip returns direct IP when peer is untrusted | unit | `pytest tests/test_rate_limiting.py::test_get_client_ip_untrusted -x` | ❌ Wave 0 | ⬜ pending |
| 06-W0-03 | Wave 0 | 0 | D-11 | T-06-01 | get_client_ip reads XFF when peer is trusted proxy | unit | `pytest tests/test_rate_limiting.py::test_get_client_ip_trusted_proxy -x` | ❌ Wave 0 | ⬜ pending |
| 06-W0-04 | Wave 0 | 0 | D-12 | T-06-01 | per-account limiter key is user.id not IP | unit | `pytest tests/test_rate_limiting.py::test_account_limiter_key -x` | ❌ Wave 0 | ⬜ pending |
| 06-W0-05 | Wave 0 | 0 | D-12 | T-06-01 | authenticated endpoint returns 429 after 100 req/min | integration | `pytest tests/test_rate_limiting.py::test_account_rate_limit -x` | ❌ Wave 0 | ⬜ pending |
| 06-W0-06 | Wave 0 | 0 | D-04..D-06 | — | Locust locustfile.py exists and is discoverable | smoke | `python -c "import locust; import sys; sys.path.insert(0,'backend/load_tests'); import locustfile"` | ❌ Wave 0 | ⬜ pending |
| 06-LOG-01 | structlog | 1 | D-01/D-02 | — | JSON log line contains correlation_id and method | unit | `pytest tests/test_logging.py -x` | ❌ Wave 0 | ⬜ pending |
| 06-LOG-02 | structlog | 1 | D-01 | — | structlog contextvars cleared between requests | unit | `pytest tests/test_logging.py::test_context_cleared -x` | ❌ Wave 0 | ⬜ pending |
| 06-RL-01 | rate limiting | 5 | D-11 | T-06-01 | IP rate limiter uses get_client_ip not get_remote_address | unit | `pytest tests/test_rate_limiting.py -x` | ❌ Wave 0 | ⬜ pending |
| 06-RL-02 | rate limiting | 5 | D-12 | T-06-01 | per-account 429 after 100 req/min on documents endpoint | integration | `pytest tests/test_rate_limiting.py::test_account_rate_limit -x` | ❌ Wave 0 | ⬜ pending |
| 06-SCOUT-01 | docker scout | manual | D-10 | CVE | Zero critical CVEs in built image | manual | `docker scout cves local://docuvault-backend:latest --only-severity critical --exit-code` | N/A manual | ⬜ pending |
| 06-DOCKER-01 | Dockerfile | manual | D-07 | EoP | Container runs as uid=1000 not root | manual | `docker run --rm docuvault-backend:latest id` outputs `uid=1000` | N/A manual | ⬜ pending |
| 06-DOCKER-02 | docker-compose | manual | D-08 | EoP | read_only container can write to /tmp | manual | `docker compose up backend` + upload a document — no PermissionError | N/A manual | ⬜ pending |
| 06-LOCUST-01 | Locust | manual | D-06 | — | SLA: p95 < 200ms, p99 < 500ms at 50 users | load test | `locust --headless --users 50 --spawn-rate 10 --run-time 5m -f backend/load_tests/locustfile.py --host http://localhost:8000` | ❌ Wave 0 | ⬜ pending |
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
---
## Wave 0 Requirements
- [ ] `backend/tests/test_logging.py` — structlog config tests (D-01); xfail stubs for: JSON output contains correlation_id, context cleared between requests
- [ ] `backend/tests/test_rate_limiting.py` — get_client_ip unit tests + per-account limiter tests (D-11, D-12); xfail stubs for: untrusted peer, trusted proxy XFF, account_limiter key, 429 on account limit
- [ ] `backend/load_tests/__init__.py` — empty marker file (prevents pytest from discovering locustfile.py as a test file)
- [ ] `backend/load_tests/locustfile.py` — Locust HttpUser skeleton (can be a stub with TODO body; full implementation in load test wave)
---
## Manual-Only Verifications
| Behavior | Requirement | Why Manual | Test Instructions |
|----------|-------------|------------|-------------------|
| Container runs as uid=1000 | D-07 | Requires built Docker image, cannot be run in unit test | `docker build -t docuvault-backend:latest backend/ && docker run --rm docuvault-backend:latest id` — must show `uid=1000(appuser)` |
| read_only fs allows /tmp writes | D-08 | Requires docker compose stack | `docker compose up backend` → upload a document → confirm no PermissionError in logs |
| SLA targets met at 50 concurrent users | D-06 | Load test requires running stack | `locust --headless --users 50 --spawn-rate 10 --run-time 5m -f backend/load_tests/locustfile.py --host http://localhost:8000` — must exit 0 |
| docker scout reports zero critical CVEs | D-10 | Requires built image + Docker Hub login | `docker login && docker scout cves local://docuvault-backend:latest --only-severity critical --exit-code` — must exit 0 |
| Grafana dashboard shows Loki logs | D-02 | UI verification | Open http://localhost:3000, Explore → Loki datasource → query `{service="backend"}` — should show JSON log lines with correlation_id |
---
## Validation Sign-Off
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
- [ ] Wave 0 covers all MISSING references
- [ ] No watch-mode flags
- [ ] Feedback latency < 60s
- [ ] `nyquist_compliant: true` set in frontmatter
**Approval:** pending