docs(06): create phase 6 plan — performance & production hardening

6 plans across 4 waves covering structlog/Loki observability, Locust
load testing, multi-stage Dockerfile hardening, trusted-proxy rate
limiting, and RUNBOOK.md. Verification passed (0 blockers).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.planning/phases/06-performance-production-hardening/06-02-PLAN.md

@@ -0,0 +1,269 @@
+---
+phase: 06-performance-production-hardening
+plan: 02
+type: execute
+wave: 1
+depends_on:
+  - 06-01
+files_modified:
+  - backend/requirements.txt
+  - backend/services/logging.py
+  - backend/config.py
+  - backend/main.py
+  - docker-compose.yml
+  - docker/loki/loki-config.yaml
+  - docker/loki/promtail-config.yaml
+  - backend/tests/test_logging.py
+autonomous: true
+requirements:
+  - D-01
+  - D-02
+  - D-03
+user_setup:
+  - service: grafana
+    why: "Local log query UI for Loki"
+    env_vars: []
+    dashboard_config:
+      - task: "Open http://localhost:3000 after compose up; Loki datasource is preconfigured anonymous; Explore → Loki → query {service=\"backend\"}"
+        location: "Grafana UI → Explore → Loki"
+
+must_haves:
+  truths:
+    - "Every HTTP request emits at least one structured JSON log line containing correlation_id, path, method, duration_ms (user_id added by the per-account work in 06-04)"
+    - "Every HTTP response carries an X-Correlation-ID header matching the bound contextvar"
+    - "structlog contextvars are cleared at the start of every request — no bleed between requests on the same worker"
+    - "docker compose up brings up loki, promtail, grafana services that read backend container stdout via the docker_sd label logging=promtail"
+    - "Setting LOG_JSON=true switches the renderer from ConsoleRenderer to JSONRenderer without code changes"
+  artifacts:
+    - path: "backend/services/logging.py"
+      provides: "setup_logging(json_logs, log_level) — single entry-point for structlog + stdlib bridge"
+      exports: ["setup_logging"]
+      min_lines: 40
+    - path: "backend/main.py"
+      provides: "CorrelationIDMiddleware raw-ASGI class + setup_logging() call in lifespan + middleware registered LAST"
+      contains: "class CorrelationIDMiddleware"
+    - path: "backend/config.py"
+      provides: "log_level + log_json Settings fields"
+      contains: "log_level"
+    - path: "docker-compose.yml"
+      provides: "loki, promtail, grafana services + loki_data and grafana_data named volumes + logging:promtail label on backend"
+      contains: "grafana/loki"
+    - path: "docker/loki/loki-config.yaml"
+      provides: "single-binary filesystem-mode Loki config (schema v13, tsdb)"
+      contains: "auth_enabled: false"
+    - path: "docker/loki/promtail-config.yaml"
+      provides: "Promtail docker_sd_configs scrape with label filter logging=promtail; ships to http://loki:3100/loki/api/v1/push"
+      contains: "docker_sd_configs"
+  key_links:
+    - from: "backend/main.py CorrelationIDMiddleware"
+      to: "structlog.contextvars"
+      via: "clear_contextvars() then bind_contextvars(correlation_id, path, method)"
+      pattern: "clear_contextvars"
+    - from: "docker-compose.yml backend.labels.logging"
+      to: "promtail-config.yaml docker_sd_configs.filters"
+      via: "label match logging=promtail"
+      pattern: "logging.*promtail"
+    - from: "main.py app.add_middleware(CorrelationIDMiddleware)"
+      to: "Starlette reverse-insertion order"
+      via: "registered LAST so it runs FIRST"
+      pattern: "add_middleware\\(CorrelationIDMiddleware"
+---
+
+<objective>
+Wire structured JSON logging (D-01) with correlation IDs across every FastAPI request, and stand up a local Loki+Promtail+Grafana log aggregation stack via docker-compose (D-02). Skip OpenTelemetry per D-03. Promote 5 xfail stubs from 06-01 to PASS.
+
+Purpose: Make every request observable end-to-end with a single grep on correlation_id. The Loki stack closes Phase 6 success criterion 2 ("Structured JSON logging is emitted to stdout; a local log aggregation stack captures and queries them").
+
+Output: setup_logging() service module, CorrelationIDMiddleware in main.py, config keys, docker/loki/{loki,promtail}-config.yaml, docker-compose additions, and all 5 logging xfail stubs flipped to PASS.
+</objective>
+
+<execution_context>
+@$HOME/.claude/get-shit-done/workflows/execute-plan.md
+@$HOME/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/phases/06-performance-production-hardening/06-CONTEXT.md
+@.planning/phases/06-performance-production-hardening/06-RESEARCH.md
+@.planning/phases/06-performance-production-hardening/06-PATTERNS.md
+@.planning/phases/06-performance-production-hardening/06-VALIDATION.md
+@CLAUDE.md
+@backend/main.py
+@backend/config.py
+@backend/services/auth.py
+@backend/tests/test_logging.py
+@docker-compose.yml
+
+<interfaces>
+<!-- Key signatures the executor must respect — extracted from existing code. -->
+
+From backend/main.py (current lifespan + middleware shape):
+- `@asynccontextmanager async def lifespan(app: FastAPI)` — add `setup_logging(json_logs=settings.log_json, log_level=settings.log_level)` as the FIRST statement inside lifespan(), before MinIO/Redis init.
+- Existing `app.add_middleware(...)` calls (in order of insertion): SecurityHeadersMiddleware → CORSMiddleware → OriginValidationMiddleware. Starlette runs middleware in REVERSE insertion order. CorrelationIDMiddleware must be registered LAST so it runs FIRST.
+
+From backend/config.py (current Settings class):
+- Uses pydantic-settings `Settings(BaseSettings)` with `model_config = SettingsConfigDict(env_file=".env", env_list_separator=",")`. Add new fields with type annotations and defaults; env vars are upper-snake-case of the field name.
+
+From RESEARCH.md Pattern 1 / 2 (target shape):
+- `setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None`
+- `class CorrelationIDMiddleware:` constructor `__init__(self, app: ASGIApp)` + `async def __call__(self, scope, receive, send)`. NOT BaseHTTPMiddleware.
+</interfaces>
+</context>
+
+<tasks>
+
+<task type="auto" tdd="true">
+  <name>Task 1: Add structlog dependency + create services/logging.py</name>
+  <files>backend/requirements.txt, backend/services/logging.py</files>
+  <read_first>
+    - backend/requirements.txt
+    - backend/services/auth.py (module structure analog)
+    - .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 1)
+    - .planning/phases/06-performance-production-hardening/06-PATTERNS.md (services/logging.py section)
+  </read_first>
+  <behavior>
+    - setup_logging(json_logs=True) installs a JSONRenderer in the root logger handler chain.
+    - setup_logging(json_logs=False) installs ConsoleRenderer.
+    - shared_processors list places structlog.contextvars.merge_contextvars FIRST.
+    - Stdlib loggers uvicorn and uvicorn.error propagate through the structlog formatter; uvicorn.access propagate is set to False so the middleware owns request logging.
+    - Calling setup_logging twice does not duplicate root handlers (idempotent — clear existing handlers before adding).
+  </behavior>
+  <action>
+    Append `structlog>=25.5.0` to backend/requirements.txt under a new comment `# Observability (Phase 6 — D-01)`.
+    Create backend/services/logging.py implementing `setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None`. Module docstring per the services/auth.py analog: `from __future__ import annotations`, import logging + structlog + `from config import settings`, no FastAPI coupling. Build the shared_processors list in this exact order: merge_contextvars, add_log_level, add_logger_name, PositionalArgumentsFormatter, ExtraAdder, TimeStamper(fmt="iso"), StackInfoRenderer. If json_logs True, append format_exc_info to that list. Call structlog.configure(processors=shared+wrap_for_formatter, logger_factory=LoggerFactory, cache_logger_on_first_use=True). Pick JSONRenderer when json_logs else ConsoleRenderer. Build a ProcessorFormatter(foreign_pre_chain=shared, processors=[remove_processors_meta, log_renderer]). Clear existing root logger handlers, add a single StreamHandler with the formatter, set root level to log_level.upper(). For uvicorn and uvicorn.error loggers, clear handlers and set propagate=True. For uvicorn.access, clear handlers and set propagate=False.
+  </action>
+  <verify>
+    <automated>cd backend &amp;&amp; pip install structlog &amp;&amp; python -c "from services.logging import setup_logging; setup_logging(json_logs=True, log_level='INFO'); import structlog; structlog.get_logger().info('hello', user_id='abc')" 2>&amp;1 | grep -E '"event":\s*"hello"'</automated>
+  </verify>
+  <acceptance_criteria>
+    - `grep -c '^structlog' backend/requirements.txt` returns 1.
+    - `grep -c 'def setup_logging' backend/services/logging.py` returns 1.
+    - Module imports cleanly: `cd backend &amp;&amp; python -c "from services.logging import setup_logging"` exits 0.
+    - merge_contextvars appears BEFORE add_log_level in shared_processors: `grep -nE "merge_contextvars|add_log_level" backend/services/logging.py | head -2` shows merge_contextvars first.
+    - JSON branch emits a JSON-shaped line: `cd backend &amp;&amp; python -c "from services.logging import setup_logging; setup_logging(json_logs=True); import structlog; structlog.get_logger().info('e', k=1)" 2>&amp;1 | grep -cE '^\\{.*"event":\\s*"e"'` returns ≥ 1.
+    - `grep -cE "uvicorn.access.*propagate.*False|propagate.*False.*uvicorn.access" backend/services/logging.py` returns ≥ 1.
+  </acceptance_criteria>
+  <done>structlog installed, setup_logging defined and idempotent, JSON/console branches selectable via boolean parameter.</done>
+</task>
+
+<task type="auto" tdd="true">
+  <name>Task 2: Wire CorrelationIDMiddleware in main.py + add config fields + promote 5 test stubs</name>
+  <files>backend/main.py, backend/config.py, backend/tests/test_logging.py</files>
+  <read_first>
+    - backend/main.py (lines 65–131 — lifespan, middleware registration order)
+    - backend/config.py (current Settings fields and pattern)
+    - backend/tests/test_logging.py (the 5 xfail stubs from 06-01 — flip to real assertions)
+    - .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 2, Pitfall 2, Anti-Patterns section)
+    - .planning/phases/06-performance-production-hardening/06-PATTERNS.md (main.py section — "Register LAST")
+  </read_first>
+  <behavior>
+    - GET /health response includes an X-Correlation-ID header that is a UUID4-shaped string.
+    - Two consecutive requests on the same worker produce two distinct correlation_id values (Pitfall 2 — no contextvars bleed).
+    - Log lines emitted during a request include correlation_id, path, method, and duration_ms keys.
+    - clear_contextvars() runs as the first operation inside CorrelationIDMiddleware.__call__ — a contextvar bound by a faked earlier request does not appear in the current request's log.
+    - CorrelationIDMiddleware is the LAST middleware registered in main.py (Starlette reverse-insertion order makes it run FIRST in the request chain).
+  </behavior>
+  <action>
+    Edit backend/config.py to append two fields inside the Settings class after the existing Cloud Storage block, before `settings = Settings()`: `log_level: str = "INFO"` and `log_json: bool = False`. Use the existing comment pattern `# Observability (Phase 6 — D-01)`. Pydantic-settings picks them up automatically as LOG_LEVEL / LOG_JSON env vars.
+    Edit backend/main.py:
+    (a) Add imports near the top: `import uuid`, `import time`, `import structlog`, `from starlette.types import ASGIApp, Receive, Scope, Send`, `from services.logging import setup_logging`.
+    (b) Inside the existing lifespan() function, insert as the FIRST statement (before MinIO client init): `setup_logging(json_logs=settings.log_json, log_level=settings.log_level)`.
+    (c) Define a new top-level class CorrelationIDMiddleware as raw ASGI — NOT BaseHTTPMiddleware. Constructor `__init__(self, app: ASGIApp) -> None` stores app on self. Async `__call__(self, scope, receive, send) -> None`: if scope type is not "http" delegate to self.app and return; generate correlation_id = str(uuid.uuid4()); capture start_ns = time.perf_counter_ns(); call structlog.contextvars.clear_contextvars() FIRST; then structlog.contextvars.bind_contextvars(correlation_id=correlation_id, path=scope.get("path",""), method=scope.get("method","")); define `async def send_with_header(message)` that on http.response.start appends (b"x-correlation-id", correlation_id.encode()) to message["headers"] (preserve existing headers); await self.app(scope, receive, send_with_header); after await, compute duration_ms = (time.perf_counter_ns() - start_ns) / 1_000_000 and bind_contextvars(duration_ms=round(duration_ms, 2)).
+    (d) Register the new middleware as the LAST `app.add_middleware()` call (after OriginValidationMiddleware), with a header comment `# 4. CorrelationID — added last so it runs FIRST (Starlette reverse-insertion order)`.
+    Edit backend/tests/test_logging.py: remove the `@pytest.mark.xfail` decorator from all 5 stubs and replace the single-line `pytest.xfail(...)` body with real assertions matching the behaviour titles set in 06-01. For tests that need a working app, use the existing async_client fixture from conftest.py.
+  </action>
+  <verify>
+    <automated>cd backend &amp;&amp; pytest tests/test_logging.py -v --no-header 2>&amp;1 | tail -3 | grep -E '5 passed'</automated>
+  </verify>
+  <acceptance_criteria>
+    - `grep -c "log_level: str" backend/config.py` returns 1 and `grep -c "log_json: bool" backend/config.py` returns 1.
+    - `grep -c "class CorrelationIDMiddleware" backend/main.py` returns 1.
+    - `grep -cE "class CorrelationIDMiddleware\\(BaseHTTPMiddleware" backend/main.py` returns 0 (must NOT inherit BaseHTTPMiddleware).
+    - `grep -nE "app.add_middleware\\(CorrelationIDMiddleware" backend/main.py` returns a line number greater than the line `app.add_middleware(OriginValidationMiddleware)`.
+    - `grep -c "clear_contextvars" backend/main.py` returns ≥ 1.
+    - `grep -c "setup_logging" backend/main.py` returns ≥ 2 (one import, one call).
+    - All 5 tests in backend/tests/test_logging.py PASS: `cd backend &amp;&amp; pytest tests/test_logging.py -v --no-header 2>&amp;1 | tail -3` shows "5 passed".
+    - `grep -c "pytest.mark.xfail" backend/tests/test_logging.py` returns 0 (all xfail markers removed).
+    - Full backend suite shows no NEW failures versus the 06-01 baseline.
+  </acceptance_criteria>
+  <done>5 test_logging.py tests pass, no regressions, correlation_id flows from middleware to logs to response header, contextvars are cleared per request.</done>
+</task>
+
+<task type="auto">
+  <name>Task 3: Add Loki + Promtail + Grafana stack to docker-compose</name>
+  <files>docker-compose.yml, docker/loki/loki-config.yaml, docker/loki/promtail-config.yaml</files>
+  <read_first>
+    - docker-compose.yml (current backend, celery-worker, celery-beat blocks)
+    - .planning/phases/06-performance-production-hardening/06-RESEARCH.md (Pattern 6 — Loki/Promtail/Grafana yaml blocks)
+    - .planning/phases/06-performance-production-hardening/06-PATTERNS.md (docker-compose.yml section)
+  </read_first>
+  <action>
+    Create docker/loki/loki-config.yaml per RESEARCH.md Pattern 6 (single-binary filesystem-mode block): auth_enabled false; server http_listen_port 3100, grpc_listen_port 9096; common instance_addr 127.0.0.1, path_prefix /loki, storage.filesystem chunks_directory /loki/chunks rules_directory /loki/rules, replication_factor 1, ring kvstore store inmemory; schema_config configs from 2020-10-24 store tsdb object_store filesystem schema v13 index prefix index_ period 24h; query_range.results_cache.cache.embedded_cache enabled true max_size_mb 100.
+    Create docker/loki/promtail-config.yaml per RESEARCH.md Pattern 6: server http_listen_port 9080 grpc_listen_port 0; positions filename /tmp/positions.yaml; single clients entry url http://loki:3100/loki/api/v1/push; scrape_configs[0] job_name docker with docker_sd_configs (host unix:///var/run/docker.sock, refresh_interval 5s, filters one entry name=label values=["logging=promtail"]); relabel_configs that map __meta_docker_container_name (regex "/(.*)") to label `container`, and map __meta_docker_container_label_com_docker_compose_service to label `service`.
+    Edit docker-compose.yml:
+    (a) Add three new services AFTER the existing celery-beat block, BEFORE the frontend service. Service `loki` (image grafana/loki:latest, ports "3100:3100", volumes ./docker/loki/loki-config.yaml:/etc/loki/local-config.yaml + loki_data:/loki, command "-config.file=/etc/loki/local-config.yaml"). Service `promtail` (image grafana/promtail:latest, volumes ./docker/loki/promtail-config.yaml:/etc/promtail/config.yaml + /var/lib/docker/containers:/var/lib/docker/containers:ro + /var/run/docker.sock:/var/run/docker.sock, command "-config.file=/etc/promtail/config.yaml", depends_on loki). Service `grafana` (image grafana/grafana:latest, ports "3000:3000", environment GF_AUTH_ANONYMOUS_ENABLED=true GF_AUTH_ANONYMOUS_ORG_ROLE=Admin, volumes grafana_data:/var/lib/grafana, depends_on loki).
+    (b) Append `loki_data:` and `grafana_data:` entries under the existing top-level `volumes:` block.
+    (c) Add a `labels:` map to the backend service block containing `logging: "promtail"` so promtail's docker_sd label filter matches.
+    (d) Add the same `logging: "promtail"` label to the celery-worker service block.
+    (e) Add `LOG_LEVEL=${LOG_LEVEL:-INFO}` and `LOG_JSON=${LOG_JSON:-false}` to the backend service environment block.
+    Do NOT modify celery-beat (it remains unhardened per Pitfall 7; it will not ship logs via promtail in this phase because that requires read-only safety considerations deferred to a later phase).
+  </action>
+  <verify>
+    <automated>test -f docker/loki/loki-config.yaml &amp;&amp; test -f docker/loki/promtail-config.yaml &amp;&amp; python -c "import yaml; [yaml.safe_load(open(f)) for f in ['docker-compose.yml','docker/loki/loki-config.yaml','docker/loki/promtail-config.yaml']]" &amp;&amp; docker compose config --quiet</automated>
+  </verify>
+  <acceptance_criteria>
+    - All three YAML files parse without error: `python -c "import yaml; [yaml.safe_load(open(f)) for f in ['docker-compose.yml','docker/loki/loki-config.yaml','docker/loki/promtail-config.yaml']]"` exits 0.
+    - `grep -c 'grafana/loki' docker-compose.yml` returns 1.
+    - `grep -c 'grafana/promtail' docker-compose.yml` returns 1.
+    - `grep -c 'grafana/grafana' docker-compose.yml` returns 1.
+    - `grep -cE 'logging:\\s*"?promtail"?' docker-compose.yml` returns ≥ 2 (backend + celery-worker).
+    - `grep -c '^  loki_data:' docker-compose.yml` returns 1 and `grep -c '^  grafana_data:' docker-compose.yml` returns 1.
+    - `grep -c 'LOG_JSON' docker-compose.yml` returns ≥ 1.
+    - `grep -c 'schema:\\s*v13' docker/loki/loki-config.yaml` returns 1.
+    - `grep -c 'docker_sd_configs' docker/loki/promtail-config.yaml` returns 1.
+    - `docker compose config --quiet` exits 0 (compose file is valid).
+    - celery-beat block contains NO `logging:` label and NO `read_only:` key — `awk '/^  celery-beat:/,/^  [a-z]/' docker-compose.yml | grep -cE 'logging:|read_only:'` returns 0.
+  </acceptance_criteria>
+  <done>Loki stack defined in compose, two backend services labelled for promtail scraping, all yaml validates, celery-beat untouched.</done>
+</task>
+
+</tasks>
+
+<threat_model>
+## Trust Boundaries
+
+| Boundary | Description |
+|----------|-------------|
+| Untrusted user input → log fields | User-controlled strings (path, query params, body) may attempt log injection |
+| Backend stdout → Promtail → Loki | Internal-only network; no external exposure |
+| Grafana UI :3000 → local network | Anonymous admin enabled for local dev only — production hardening deferred (see RUNBOOK.md in 06-06) |
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-06-02-01 | Tampering | Log injection via user-controlled strings in log fields | mitigate | structlog JSONRenderer serialises values as JSON strings — newlines and quotes are escaped automatically; no `%s` format strings concatenate user input |
+| T-06-02-02 | Information Disclosure | structlog contextvars leaking user_id across requests | mitigate | clear_contextvars() is the FIRST call inside CorrelationIDMiddleware.__call__; verified by test_contextvars_cleared_between_requests in 06-01/06-02 |
+| T-06-02-03 | Information Disclosure | Grafana anonymous admin exposes Loki query UI to anyone on the docker network | accept | Local dev convenience; RUNBOOK.md (06-06) documents production-time hardening (disable anonymous, add Grafana auth) |
+| T-06-02-04 | Denial of Service | Loki disk fill via unbounded log retention | accept | Single-binary filesystem mode; RUNBOOK.md documents log rotation and retention tuning for production |
+</threat_model>
+
+<verification>
+- 5 tests in backend/tests/test_logging.py PASS (no XFAIL).
+- `grep -c "class CorrelationIDMiddleware" backend/main.py` returns 1.
+- CorrelationIDMiddleware does NOT inherit BaseHTTPMiddleware.
+- `docker compose config --quiet` exits 0.
+- celery-beat block has neither `logging:` label nor `read_only:` key.
+- All 3 YAML files (compose, loki-config, promtail-config) parse without error.
+</verification>
+
+<success_criteria>
+- D-01 satisfied: setup_logging emits JSON when LOG_JSON=true; correlation_id appears in every log line; X-Correlation-ID header on every response.
+- D-02 satisfied: docker compose up brings loki on :3100, grafana on :3000, promtail scrapes containers labelled logging=promtail.
+- D-03 satisfied: no opentelemetry dependency added, no tracing middleware introduced.
+- Zero new failing tests; the 5 logging xfails from 06-01 now PASS.
+</success_criteria>
+
+<output>
+Create `.planning/phases/06-performance-production-hardening/06-02-SUMMARY.md` when done. Include: full-suite pytest summary, exact lines added to docker-compose.yml (count of new keys), and a one-line note confirming celery-beat was deliberately left untouched.
+</output>