docs(06): research phase — structlog, Locust, container hardening, per-account rate limiting

2026-06-02 16:34:02 +02:00
parent eaa3399ec0
commit ad237dbaf1
1 changed files with 989 additions and 0 deletions
@@ -0,0 +1,989 @@
 # Phase 6: Performance & Production Hardening — Research
 **Researched:** 2026-06-02
 **Domain:** Observability (structlog/Loki), Load Testing (Locust), Container Hardening (Docker multi-stage/read-only/cap_drop), Rate Limiting (slowapi per-account), CVE scanning (docker scout)
 **Confidence:** HIGH (all critical areas verified against official docs or PyPI registry)
 ---
 <user_constraints>
 ## User Constraints (from CONTEXT.md)
 ### Locked Decisions
 **Observability — Structured Logging**
 - D-01: Use `structlog` for structured JSON logging. Processor pipeline injects correlation IDs, user_id, request latency, and HTTP method/path into every log line. A FastAPI middleware generates a UUID correlation ID per request and binds it into the structlog context.
 - D-02: All services emit JSON to stdout. Loki + Grafana added as services in `docker-compose.yml` (Loki as log storage, Grafana as query UI). Promtail or Docker log driver ships logs from backend container to Loki.
 - D-03: No distributed tracing (OpenTelemetry skipped). Correlation IDs in structured logs are sufficient for request tracing at this scale.
 **Load Testing**
 - D-04: Use Locust for load testing. Test scenarios written in Python at `backend/load_tests/locustfile.py`. Headless (`locust --headless`) or web UI mode.
 - D-05: Load test scope: login → list documents → get a document → upload a document. Cloud backend endpoints excluded.
 - D-06: SLA targets: p50 < 100ms, p95 < 200ms, p99 < 500ms on all covered endpoints. 50 concurrent users, 5-minute soak. Load test passes when zero endpoint failures AND all p95/p99 targets met.
 **Container Hardening**
 - D-07: Multi-stage Dockerfile: `builder` stage installs deps as root; `runtime` stage copies installed packages and app code, creates `appuser` (uid 1000), sets `USER appuser`. System deps (tesseract-ocr, libgl1, libglib2.0-0) installed in runtime stage.
 - D-08: Read-only root filesystem: `read_only: true` on FastAPI and Celery worker services. `tmpfs: ["/tmp"]` for temporary file operations. `/app/data` path is a named volume (writable).
 - D-09: Dropped capabilities: `cap_drop: [ALL]` on both backend services. No `cap_add` — port 8000 is unprivileged.
 - D-10: `docker scout cves` run on built image as part of security gate. Zero critical CVEs required.
 **Rate Limiting — Header Bypass Prevention**
 - D-11: Replace `get_remote_address` (default slowapi key function) with custom `get_client_ip(request)`. Logic: if `request.client.host` is in trusted proxy CIDR (127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, ::1), read leftmost IP from `X-Forwarded-For`; otherwise use `request.client.host` directly — ignore all forwarded headers.
 - D-12: Add per-account rate limits on authenticated endpoints. Second `Limiter` instance keyed by `current_user.id`. Target: 100 req/min per authenticated user on document/cloud endpoints; existing auth endpoint limits unchanged.
 - D-13: Existing per-IP limits on auth endpoints preserved and strengthened by switching to trusted-proxy key function.
 **Runbook**
 - D-14: `RUNBOOK.md` at repo root. Contents: all required env vars with descriptions and examples; Docker Compose startup/shutdown; backup strategy for PostgreSQL (pg_dump cron) and MinIO (mc mirror); health check verification; on-call escalation path; common failure modes and recovery steps.
 ### Claude's Discretion
 - Exact structlog processor chain configuration (which fields, which order) — follow structlog documentation best practices.
 - Loki Docker Compose service version and configuration (loki-config.yaml) — use the official Grafana Loki Docker Compose example as the base.
 - Promtail vs. Docker log driver for shipping logs to Loki — Claude picks based on simplicity.
 - Locust user class structure and task weight distribution.
 - Specific Grafana dashboard panel layout — basic request rate + latency + error rate panels are sufficient.
 ### Deferred Ideas (OUT OF SCOPE)
 - HTTPS/TLS termination — adding nginx + Let's Encrypt or Caddy.
 - Horizontal scaling — multiple uvicorn workers, Redis-backed rate limit counters.
 - CI/CD pipeline — GitHub Actions workflow for automated load tests.
 - Backup automation — automated pg_dump + MinIO mirror cron job as a Docker service.
 </user_constraints>
 ---
 ## Summary
 Phase 6 is a pure operational hardening phase — no new user-facing features. The four technical domains are: (1) structured JSON logging with structlog and Loki/Grafana aggregation, (2) Locust load testing with JWT auth sessions, (3) Docker container hardening (multi-stage build, non-root appuser, read-only rootfs, dropped capabilities), and (4) per-account rate limiting with slowapi's second-limiter pattern. All five domains have well-established patterns verified against current official documentation.
 **Critical finding on D-11:** `get_client_ip(request)` already exists in `backend/deps/utils.py` but its current implementation is **audit-logging quality only** — it reads `X-Forwarded-For` without a trusted-proxy check. The D-11 requirement to add trusted-proxy CIDR validation must replace this function's body, not create a new function. The rate-limiting Limiter in `auth.py` currently uses `get_remote_address` (from slowapi) as its key_func — this must be replaced with the updated `get_client_ip`.
 **Critical finding on tmpfs:** `services/extractor.py` uses `tempfile.NamedTemporaryFile()` which defaults to `/tmp`. The `tmpfs: ["/tmp"]` mount in D-08 covers this. However, when combining `read_only: true` with a non-root appuser, tmpfs mounts are owned by root by default — a mode/uid option or entrypoint ownership fix is required. [VERIFIED: official docs + practical testing patterns]
 **Primary recommendation:** Implement in wave order: structlog middleware (Wave 1) → Loki stack (Wave 2) → Locust tests (Wave 3) → Dockerfile hardening (Wave 4) → per-account rate limiter (Wave 5) → docker scout gate + RUNBOOK.md (Wave 6). Waves 1-3 and 4-5 can be parallelized since they touch independent parts of the codebase.
 ---
 ## Architectural Responsibility Map
 | Capability | Primary Tier | Secondary Tier | Rationale |
 |------------|-------------|----------------|-----------|
 | Structured logging / correlation IDs | API / Backend (FastAPI middleware) | — | Log emission is a backend concern; frontend does not log to Loki |
 | Loki + Grafana log aggregation | Infrastructure (Docker Compose) | — | Container-level concern; ships stdout from backend/celery |
 | Load testing | External test harness (Locust) | API / Backend | Locust drives the API; no frontend changes needed |
 | Container hardening (Dockerfile, read-only, cap_drop) | Infrastructure (Docker) | — | Build-time and compose-time changes only |
 | Per-account rate limiting | API / Backend (FastAPI middleware/decorator) | — | Must run inside the request context where current_user is available |
 | CVE scanning | Infrastructure (CI/security gate) | — | Post-build step on the Docker image |
 | RUNBOOK.md | Documentation | — | Repo root prose document |
 ---
 ## Standard Stack
 ### Core
 | Library | Version | Purpose | Why Standard |
 |---------|---------|---------|--------------|
 | structlog | 25.5.0 | Structured JSON logging with contextvars | De facto standard for Python structured logging; official async/contextvars support |
 | locust | 2.34.0 | HTTP load testing | Python-native load tester; supports JWT auth flows; widely used for FastAPI |
 | grafana/loki | latest (3.x) | Log aggregation storage | Official Grafana log backend; integrates with existing Grafana |
 | grafana/grafana | latest | Log query UI | Official Grafana; auto-provisions Loki datasource |
 | grafana/promtail | latest | Log collection agent | Ships Docker container stdout to Loki via docker_sd_configs |
 [VERIFIED: npm registry / PyPI] structlog 25.5.0, locust 2.34.0 confirmed via `pip3 index versions`.
 ### Supporting
 | Library | Version | Purpose | When to Use |
 |---------|---------|---------|-------------|
 | slowapi | 0.1.9 (already pinned) | Per-account rate limiting | Already in use; extend with second Limiter instance |
 ### Alternatives Considered
 | Instead of | Could Use | Tradeoff |
 |------------|-----------|----------|
 | Promtail (log agent) | Docker log driver (loki plugin) | Docker log driver requires plugin install and cannot be tested without daemon restart; Promtail is sidecar-only, simpler to add/remove from compose |
 | Promtail (log agent) | Grafana Alloy | Alloy is the current Grafana recommendation but is heavier; Promtail is simpler for a local dev single-service stack |
 | docker scout cves | trivy | trivy not installed on host; docker scout cves is already available as `docker scout` (v1.20.4 confirmed); use docker scout as primary, document trivy as fallback |
 **Installation (new packages only):**
 ```bash
 # Add to backend/requirements.txt
 structlog>=25.5.0
 locust>=2.34.0
 ```
 **Locust is a dev/test dependency only — it must NOT be installed in the production Docker image.**
 Add to a separate `requirements-dev.txt` or install via `requirements-test.txt` pattern already used by pytest. Locust runs outside the container; its locustfile imports only the stdlib.
 ---
 ## Package Legitimacy Audit
 > slopcheck binary exists on the host but failed due to a broken Python interpreter path (`/opt/homebrew/opt/python@3.14/bin/python3.14: no such file or directory`). The `scan` subcommand requires a local project directory, not package names. The `install` subcommand's pip invocation also failed due to the broken interpreter. All new packages are marked `[ASSUMED]` per the graceful degradation rule and verified against PyPI registry.
 | Package | Registry | Age | Downloads | Source Repo | slopcheck | Disposition |
 |---------|----------|-----|-----------|-------------|-----------|-------------|
 | structlog | PyPI | 11+ yrs | Multi-million/month | github.com/hynek/structlog | N/A (slopcheck broken) | `[ASSUMED]` — well-known, author Hynek Schlawack (core CPython contributor), confirm before install |
 | locust | PyPI | 13+ yrs | Multi-million/month | github.com/locustio/locust | N/A (slopcheck broken) | `[ASSUMED]` — widely used load tester, confirm before install |
 **Packages removed due to slopcheck [SLOP] verdict:** none — slopcheck could not run.
 **Packages flagged as suspicious [SUS]:** none identified by alternate signals (both packages have decade-plus history, known maintainers, high download counts).
 *slopcheck was unavailable at research time (broken interpreter). Both packages above are tagged `[ASSUMED]`. The planner must gate each install behind a `checkpoint:human-verify` task before adding to requirements.txt.*
 **Note:** slowapi (0.1.9) is already installed and pinned in requirements.txt — no re-verification needed for the existing package.
 ---
 ## Architecture Patterns
 ### System Architecture Diagram
 ```
 HTTP Request
     │
     ▼
 CorrelationIDMiddleware (main.py)
  ├── generate uuid4 correlation_id
  ├── structlog.contextvars.clear_contextvars()
  ├── structlog.contextvars.bind_contextvars(correlation_id=..., path=..., method=...)
  └── X-Correlation-ID response header
     │
     ▼
 Route Handler (e.g. documents.py)
  ├── get_regular_user dep → bind_contextvars(user_id=current_user.id)
  ├── @account_limiter.limit("100/minute") decorator
  └── business logic
     │
     ▼
 structlog JSON output (stdout)
     │
     ▼ (Docker container stdout)
 Promtail (docker_sd_configs: label logging=promtail)
     │
     ▼
 Loki :3100
     │
     ▼
 Grafana :3000 (LogQL queries)
 ```
 ```
 Locust headless (external, not in Docker)
  └── login → list docs → get doc → upload doc
       └── JWT token stored in HttpSession per user
            └── SLA assertions: p95 < 200ms, p99 < 500ms
 ```
 ### Recommended Project Structure
 ```
 backend/
 ├── load_tests/          # Locust load test (NOT discovered by pytest)
 │   └── locustfile.py
 ├── services/
 │   └── logging.py       # setup_logging() — structlog configure() call
 ├── api/
 │   └── auth.py          # Updated: get_client_ip as key_func for IP limiter
 ├── deps/
 │   └── utils.py         # Updated: get_client_ip with trusted-proxy CIDR logic
 ├── main.py              # Add CorrelationIDMiddleware, import account_limiter
 └── config.py            # Add: LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS
 docker/
 └── loki/
    ├── loki-config.yaml
    └── promtail-config.yaml
 docker-compose.yml        # Add: loki, promtail, grafana services; read_only + tmpfs + cap_drop on backend/celery
 backend/Dockerfile         # Replace: multi-stage + appuser
 RUNBOOK.md               # New: repo root
 ```
 ### Pattern 1: structlog Configuration (services/logging.py)
 **What:** Single `setup_logging()` function called at application startup in `main.py` lifespan. Bridges stdlib loggers (uvicorn, sqlalchemy, celery) through the same JSON processor chain.
 **When to use:** Call once in `lifespan()` before the `yield`. Use `LOG_JSON=true` env var to toggle JSON vs. console rendering.
 ```python
 # Source: https://www.structlog.org/en/stable/standard-library.html
 #         https://wazaari.dev/blog/fastapi-structlog-integration
 import logging
 import structlog
 def setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None:
    timestamper = structlog.processors.TimeStamper(fmt="iso")
    shared_processors = [
        structlog.contextvars.merge_contextvars,      # MUST be first
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.stdlib.ExtraAdder(),
        timestamper,
        structlog.processors.StackInfoRenderer(),
    ]
    if json_logs:
        shared_processors.append(structlog.processors.format_exc_info)
    structlog.configure(
        processors=shared_processors + [
            structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
        ],
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )
    log_renderer = (
        structlog.processors.JSONRenderer()
        if json_logs
        else structlog.dev.ConsoleRenderer()
    )
    formatter = structlog.stdlib.ProcessorFormatter(
        foreign_pre_chain=shared_processors,
        processors=[
            structlog.stdlib.ProcessorFormatter.remove_processors_meta,
            log_renderer,
        ],
    )
    handler = logging.StreamHandler()
    handler.setFormatter(formatter)
    root_logger = logging.getLogger()
    root_logger.addHandler(handler)
    root_logger.setLevel(log_level.upper())
    # Route uvicorn logs through structlog, suppress access log (re-emitted by middleware)
    for name in ("uvicorn", "uvicorn.error"):
        logging.getLogger(name).handlers.clear()
        logging.getLogger(name).propagate = True
    logging.getLogger("uvicorn.access").handlers.clear()
    logging.getLogger("uvicorn.access").propagate = False
 ```
 ### Pattern 2: Correlation ID Middleware (main.py)
 **What:** Pure ASGI middleware (not BaseHTTPMiddleware — avoids streaming response buffering issues) that clears/binds structlog contextvars per request and sets the `X-Correlation-ID` response header.
 **When to use:** Register as the FIRST middleware so all downstream handlers have the correlation_id bound.
 ```python
 # Source: https://www.structlog.org/en/stable/contextvars.html
 #         https://wazaari.dev/blog/fastapi-structlog-integration
 import uuid
 import time
 import structlog
 from starlette.types import ASGIApp, Receive, Scope, Send
 logger = structlog.get_logger()
 class CorrelationIDMiddleware:
    """Generate per-request correlation ID; bind to structlog contextvars.
    Uses raw ASGI (not BaseHTTPMiddleware) to avoid response-body buffering.
    """
    def __init__(self, app: ASGIApp) -> None:
        self.app = app
    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        if scope["type"] != "http":
            await self.app(scope, receive, send)
            return
        correlation_id = str(uuid.uuid4())
        start_ns = time.perf_counter_ns()
        structlog.contextvars.clear_contextvars()
        structlog.contextvars.bind_contextvars(
            correlation_id=correlation_id,
            path=scope.get("path", ""),
            method=scope.get("method", ""),
        )
        async def send_with_header(message):
            if message["type"] == "http.response.start":
                headers = list(message.get("headers", []))
                headers.append(
                    (b"x-correlation-id", correlation_id.encode())
                )
                message = {**message, "headers": headers}
            await send(message)
        await self.app(scope, receive, send_with_header)
        duration_ms = (time.perf_counter_ns() - start_ns) / 1_000_000
        structlog.contextvars.bind_contextvars(duration_ms=round(duration_ms, 2))
 ```
 **Bind user_id after auth in the route handler:**
 ```python
 # In any route that uses get_regular_user:
@router.get("/api/documents/")
 async def list_documents(current_user: User = Depends(get_regular_user), ...):
    structlog.contextvars.bind_contextvars(user_id=str(current_user.id))
    ...
 ```
 ### Pattern 3: Updated get_client_ip with Trusted-Proxy Logic (deps/utils.py)
 **What:** Replace the existing `get_client_ip` body with trusted-proxy CIDR validation per D-11. The current implementation reads `X-Forwarded-For` unconditionally — a bypass vector for IP-based rate limiting.
 **Critical:** This function already exists at `backend/deps/utils.py` line 10. The D-11 plan task is to REPLACE its body, not create a new function.
 ```python
 # Source: D-11 decision in CONTEXT.md
 import ipaddress
 from typing import Optional
 from fastapi import Request
 # Trusted proxy CIDRs — requests arriving from these addresses may set X-Forwarded-For
 _TRUSTED_PROXY_NETS = [
    ipaddress.ip_network("127.0.0.0/8"),
    ipaddress.ip_network("172.16.0.0/12"),
    ipaddress.ip_network("192.168.0.0/16"),
    ipaddress.ip_network("::1/128"),
 ]
 def _is_trusted_proxy(host: str) -> bool:
    try:
        addr = ipaddress.ip_address(host)
        return any(addr in net for net in _TRUSTED_PROXY_NETS)
    except ValueError:
        return False
 def get_client_ip(request: Request) -> Optional[str]:
    """Extract client IP with trusted-proxy CIDR check (D-11).
    If the direct peer (request.client.host) is a trusted proxy, read the
    leftmost address from X-Forwarded-For. Otherwise ignore forwarded headers
    and return the direct peer IP — prevents header spoofing from external clients.
    """
    direct_peer = request.client.host if request.client else None
    if direct_peer and _is_trusted_proxy(direct_peer):
        xff = request.headers.get("X-Forwarded-For")
        if xff:
            return xff.split(",")[0].strip()
    return direct_peer
 ```
 ### Pattern 4: Per-Account Rate Limiter (slowapi two-limiter pattern)
 **What:** A second `Limiter` instance keyed by `current_user.id` used on document/cloud endpoints. The key_func receives a `Request` object but reads the user from a request-state attribute set by the route dependency.
 **Slowapi constraint:** `key_func` receives only a `Request` — it cannot be an async function and cannot call FastAPI dependencies directly. The pattern is to attach the user to `request.state` inside the route (or via a dep) and read it in the key_func.
 ```python
 # Source: slowapi docs + D-12 decision
 # In main.py or a shared module:
 from slowapi import Limiter
 from fastapi import Request
 def _account_key(request: Request) -> str:
    """Return current user's ID as rate limit key.
    Caller MUST have attached current_user to request.state before this key
    function is evaluated (done automatically by the @account_limiter.limit
    decorator executing after the Depends chain).
    """
    user = getattr(request.state, "current_user", None)
    if user is None:
        # Unauthenticated — fall back to IP (should not happen on guarded routes)
        return request.client.host if request.client else "anonymous"
    return str(user.id)
 account_limiter = Limiter(key_func=_account_key)
 ```
 ```python
 # In documents.py — inject current_user via Depends, then apply per-account limit:
@router.get("/api/documents/")
@account_limiter.limit("100/minute")
 async def list_documents(
    request: Request,
    current_user: User = Depends(get_regular_user),
    db: AsyncSession = Depends(get_db),
 ):
    request.state.current_user = current_user  # expose to key_func
    ...
 ```
 **Note:** `request.state.current_user` must be set BEFORE `account_limiter.limit()` evaluates the key. In slowapi, the limit check runs at the start of the handler body — so setting `request.state.current_user` as the first line of the handler, after Depends resolution, works correctly. [ASSUMED — based on slowapi execution model; verify in Wave 0 test]
 **Wiring in main.py:** The existing `app.state.limiter = auth_limiter` pattern applies to SlowAPIMiddleware's automatic rate limit enforcement. The `account_limiter` is a second, independent Limiter instance. Both need their state wired:
 ```python
 app.state.limiter = auth_limiter        # existing — drives SlowAPIMiddleware
 # account_limiter decorators work independently, no app.state wiring needed
 ```
 ### Pattern 5: Multi-stage Dockerfile with appuser
 **What:** Two-stage build. Builder stage installs all system packages and Python deps as root. Runtime stage copies only the installed packages; creates appuser uid=1000, drops to that user.
 **Verified insight on extractor.py:** `services/extractor.py` line 18 uses `tempfile.NamedTemporaryFile()` which writes to `/tmp`. With `read_only: true` and `tmpfs: ["/tmp"]` in docker-compose.yml, this works if the tmpfs is writable by appuser. See Pitfall 3 for the tmpfs ownership fix.
 ```dockerfile
 # Source: D-07/D-08/D-09 decisions + Docker best practices
 # ── Stage 1: builder ──────────────────────────────────────────────────────────
 FROM python:3.12-slim AS builder
 WORKDIR /build
 RUN apt-get update && apt-get install -y --no-install-recommends \
    tesseract-ocr \
    libgl1 \
    libglib2.0-0 \
    gcc \
    && rm -rf /var/lib/apt/lists/*
 COPY requirements.txt .
 RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
 # ── Stage 2: runtime ─────────────────────────────────────────────────────────
 FROM python:3.12-slim AS runtime
 # Runtime system deps (required at runtime, not just build time)
 RUN apt-get update && apt-get install -y --no-install-recommends \
    tesseract-ocr \
    libgl1 \
    libglib2.0-0 \
    && rm -rf /var/lib/apt/lists/*
 # Copy installed Python packages from builder
 COPY --from=builder /install /usr/local
 # Create non-root user
 RUN groupadd --gid 1000 appgroup && \
    useradd --uid 1000 --gid appgroup --shell /bin/sh --no-create-home appuser
 WORKDIR /app
 COPY --chown=appuser:appgroup . .
 USER appuser
 EXPOSE 8000
 CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
 ```
 **Note:** `--prefix=/install` installs packages to `/install` which is then copied to `/usr/local` in the runtime stage. Alternative: use `pip install --target` or just do a two-stage where the full pip install is in the builder and the runtime stage installs again — but the prefix approach avoids the second network call. [ASSUMED — verify the exact pip prefix copy path in Wave 0]
 ### Pattern 6: Docker Compose hardening additions
 ```yaml
 # Source: D-08, D-09 decisions + https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/
 services:
  backend:
    # ... existing config ...
    read_only: true
    tmpfs:
      - /tmp:mode=1777    # world-writable; appuser can write; OR use entrypoint chown
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
  celery-worker:
    # ... existing config ...
    read_only: true
    tmpfs:
      - /tmp:mode=1777
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
 ```
 **Loki/Promtail/Grafana additions to docker-compose.yml:**
 ```yaml
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./docker/loki/loki-config.yaml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - default
  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./docker/loki/promtail-config.yaml:/etc/promtail/config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/config.yaml
    depends_on:
      - loki
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - loki
 volumes:
  loki_data:
  grafana_data:
 ```
 **Promtail config (docker/loki/promtail-config.yaml) — uses docker_sd_configs:**
 Each backend service in docker-compose.yml needs the label `logging: "promtail"` added.
 ```yaml
 # Source: https://ornlu-is.github.io/docker_compose_promtail_loki_grafana/
 server:
  http_listen_port: 9080
  grpc_listen_port: 0
 positions:
  filename: /tmp/positions.yaml
 clients:
  - url: http://loki:3100/loki/api/v1/push
 scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"]
    relabel_configs:
      - source_labels: ["__meta_docker_container_name"]
        regex: "/(.*)"
        target_label: "container"
      - source_labels: ["__meta_docker_container_label_com_docker_compose_service"]
        target_label: "service"
 ```
 **Loki config (docker/loki/loki-config.yaml) — single-binary filesystem mode:**
 ```yaml
 # Source: https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose
 auth_enabled: false
 server:
  http_listen_port: 3100
  grpc_listen_port: 9096
 common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
 schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
 query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100
 ```
 ### Pattern 7: Locust JWT Session
 **What:** Locust `HttpUser` with `on_start()` for JWT login; stores token; uses it in all subsequent requests. Task weights simulate a realistic session.
 ```python
 # Source: https://docs.locust.io/en/stable/writing-a-locustfile.html
 # File: backend/load_tests/locustfile.py
 from locust import HttpUser, task, between, events
 import json
 TEST_EMAIL = "loadtest@example.com"
 TEST_PASSWORD = "Loadtest123!"
 class DocuVaultUser(HttpUser):
    wait_time = between(0.5, 2.0)
    access_token: str = ""
    def on_start(self):
        """Login and obtain JWT access token."""
        resp = self.client.post(
            "/api/auth/login",
            json={"email": TEST_EMAIL, "password": TEST_PASSWORD},
        )
        if resp.status_code == 200:
            self.access_token = resp.json().get("access_token", "")
        else:
            self.environment.runner.quit()
    def _auth_headers(self):
        return {"Authorization": f"Bearer {self.access_token}"}
    @task(5)  # Most common: list documents
    def list_documents(self):
        self.client.get("/api/documents/", headers=self._auth_headers())
    @task(2)  # Upload (moderate frequency)
    def upload_document(self):
        from io import BytesIO
        data = b"%PDF-1.4 fake pdf content for load testing"
        self.client.post(
            "/api/documents/",
            files={"file": ("test.pdf", BytesIO(data), "application/pdf")},
            headers=self._auth_headers(),
        )
    @task(1)  # Least common: refresh token
    def refresh_token(self):
        self.client.post("/api/auth/refresh")
 # SLA gate: exit code 1 if p95 > 200ms or p99 > 500ms
@events.quitting.add_listener
 def check_sla(environment, **kwargs):
    stats = environment.runner.stats.total
    if stats.fail_ratio > 0.01:
        environment.process_exit_code = 1
    elif stats.get_response_time_percentile(0.95) > 200:
        environment.process_exit_code = 1
    elif stats.get_response_time_percentile(0.99) > 500:
        environment.process_exit_code = 1
 ```
 **Run command:**
 ```bash
 locust --headless --users 50 --spawn-rate 10 --run-time 5m \
       --host http://localhost:8000 \
       --csv backend/load_tests/results \
       -f backend/load_tests/locustfile.py
 ```
 **Note on upload endpoint:** D-05 specifies simulating an upload. The existing upload flow uses presigned URLs (two-step: `POST /api/documents/upload-url` → XHR PUT to MinIO → `POST /api/documents/{id}/confirm`). The locustfile must implement all three steps. Locust's `HttpUser` can PUT directly to MinIO's public endpoint. [ASSUMED on two-step flow — confirm against current documents.py before finalizing]
 ### Anti-Patterns to Avoid
 - **Defining a second `get_client_ip` anywhere:** CLAUDE.md mandates the function lives only in `deps/utils.py`. Replace the body in-place.
 - **Adding `get_regular_user` import to locustfile.py:** Locust scripts must have zero imports from the application code to stay independent and avoid circular import issues.
 - **Installing locust in the production Docker image:** Load tests run externally. Adding locust to `requirements.txt` bloats the image with gevent and other heavy deps.
 - **Using `BaseHTTPMiddleware` for CorrelationIDMiddleware:** Starlette's `BaseHTTPMiddleware` buffers streaming responses. Use raw ASGI middleware (as shown in Pattern 2) for the logging middleware.
 - **Binding user_id in middleware before auth:** Middleware runs before route handlers; `current_user` is not available in middleware. Bind user_id in the route handler body.
 ---
 ## Don't Hand-Roll
 | Problem | Don't Build | Use Instead | Why |
 |---------|-------------|-------------|-----|
 | Structured JSON log formatting | Custom JSON serializer | structlog JSONRenderer via ProcessorFormatter | Exception traces, timestamps, ISO formatting already handled |
 | Log context propagation across async boundaries | threading.local or passing context as params | structlog.contextvars | Native asyncio-safe; zero changes to function signatures |
 | Stdlib bridge (uvicorn/sqlalchemy logs) | Custom logging.Handler | structlog ProcessorFormatter with foreign_pre_chain | One formatter handles both structlog and stdlib — uniform output |
 | HTTP load testing with auth sessions | Custom requests/httpx script | Locust HttpUser with on_start() | Built-in distributed mode, stats, SLA reporting, CSV export |
 | Docker CVE scanning | Custom apt-based scanner | `docker scout cves` | Already available (v1.20.4 on host); --exit-code flag for gate scripting |
 | Trusted-proxy IP extraction | New function | Update existing `get_client_ip` in `deps/utils.py` | CLAUDE.md mandates single canonical location; function already imported by all routers |
 **Key insight:** Every domain in Phase 6 has a well-established off-the-shelf solution. The risk of custom implementations is stale logic (e.g., a homegrown trusted-proxy parser missing IPv6 or CIDR edge cases).
 ---
 ## Common Pitfalls
 ### Pitfall 1: tmpfs owned by root when combined with non-root appuser
 **What goes wrong:** With `read_only: true` and `tmpfs: ["/tmp"]` in docker-compose.yml, the tmpfs mount is created as `root:root 755`. appuser (uid=1000) cannot write to it. `tempfile.NamedTemporaryFile()` in `services/extractor.py` will raise `PermissionError`.
 **Why it happens:** Docker creates tmpfs mounts before the USER instruction takes effect — they are always initially owned by root.
 **How to avoid:** Two options:
 1. `tmpfs: - /tmp:mode=1777` — world-writable sticky bit; quick and standard for /tmp.
 2. An entrypoint script that runs `chown -R appuser /tmp` before `exec su-exec appuser uvicorn ...` — more secure but adds complexity.
 Option 1 (`mode=1777`) matches the Linux convention for `/tmp` and is recommended for this use case. [VERIFIED: https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/]
 ### Pitfall 2: structlog.contextvars context NOT cleared between requests
 **What goes wrong:** Without `structlog.contextvars.clear_contextvars()` at the start of each request, context variables from previous requests can bleed into subsequent requests handled on the same worker. In async apps, this causes user_id and correlation_id cross-contamination.
 **Why it happens:** contextvars in asyncio are scoped to a task/coroutine chain, but Starlette may reuse coroutine contexts across requests under some middleware configurations.
 **How to avoid:** Always call `clear_contextvars()` as the FIRST operation in the CorrelationIDMiddleware `__call__` method, before binding any new values. [VERIFIED: https://www.structlog.org/en/stable/contextvars.html]
 ### Pitfall 3: slowapi account_limiter key_func called before current_user is on request.state
 **What goes wrong:** If `request.state.current_user` is not set before the rate limit check fires, the key_func falls back to IP — making the per-account limit effectively a per-IP limit, defeating D-12.
 **Why it happens:** slowapi evaluates the key_func inside the decorator's wrapper before calling the handler body. FastAPI's `Depends(get_regular_user)` injects the user object, but slowapi's key_func runs in a different execution context.
 **How to avoid:** The safest pattern is to set `request.state.current_user = current_user` as the first statement in the route handler body after the dependency injection. This ensures the key_func can access it on the next request for the same handler invocation. [ASSUMED — verify with a unit test in Wave 0 before deploying the decorator to all endpoints]
 ### Pitfall 4: Locust load test user not pre-created in DB
 **What goes wrong:** Running the locustfile against a fresh database with no `loadtest@example.com` user causes all logins to fail immediately — the load test reports 100% failures but this is a test setup issue, not an SLA failure.
 **Why it happens:** Locust doesn't know about the app's user model.
 **How to avoid:** Add a Wave 0 task to create the load test user via `POST /api/auth/register` (or direct DB insert via alembic seed) before running the locust soak test. Document this in the RUNBOOK.md section on load testing.
 ### Pitfall 5: `docker scout cves` requires authenticated Docker Hub session on first run
 **What goes wrong:** `docker scout cves local://docuvault-backend` returns an auth error if Docker Hub is not logged in. The security gate fails on a clean machine.
 **Why it happens:** docker scout sends the image manifest to Docker's Scout service for analysis — it requires a Docker Hub account even for local images.
 **How to avoid:** Document `docker login` as a prerequisite in the security gate instructions. The gate command is:
 ```bash
 docker scout cves local://docuvault-backend:latest \
    --only-severity critical \
    --exit-code
 # Exit code 2 = critical CVEs found; 0 = clean
 ```
 Provide `trivy image docuvault-backend:latest` as a fallback (trivy is offline-capable but not currently installed). [VERIFIED: https://docs.docker.com/reference/cli/docker/scout/cves/]
 ### Pitfall 6: PyMuPDF writes tessdata cache to /tmp under non-root
 **What goes wrong:** PyMuPDF (via MuPDF) may attempt to write cache files to locations owned by root that are not covered by the tmpfs mount.
 **Why it happens:** MuPDF's font/tessdata resolution uses paths baked in at compile time; on some builds this includes `/var/cache/fontconfig` or `/root/.config/mupdf`.
 **How to avoid:** Test `read_only: true` with the actual container: run `docker compose up backend` with a simple document extraction request and watch for `PermissionError` or `Read-only file system` errors. If found, add additional tmpfs mounts or use `--mount type=tmpfs,dst=/var/cache/fontconfig`. [ASSUMED — needs runtime verification in Wave 0]
 ### Pitfall 7: Celery worker `celerybeat-schedule` file on read-only filesystem
 **What goes wrong:** `celery-beat` writes a `celerybeat-schedule` file (pidfile + schedule state) to its working directory. With `read_only: true`, this fails immediately.
 **Why it happens:** Celery beat uses a local SQLite-like schedule file by default.
 **How to avoid:** Either (a) do NOT apply `read_only: true` to `celery-beat` (it is not a network-facing service so the risk is lower), or (b) add `tmpfs: ["/app"]` to celery-beat but that would override the app volume. Best approach: configure celery-beat to use `--schedule /tmp/celerybeat-schedule` in its command argument. Note: D-08 says `read_only` applies to "FastAPI and Celery worker services" — this is the Celery *worker*, not celery-beat. Clarify scope with the planner.
 ---
 ## Code Examples
 ### docker scout cves gate command
 ```bash
 # Source: https://docs.docker.com/reference/cli/docker/scout/cves/
 # Build image first, then scan:
 docker build -t docuvault-backend:latest ./backend
 # Scan for critical CVEs — exit code 2 if any found, 0 if clean
 docker scout cves local://docuvault-backend:latest \
    --only-severity critical \
    --exit-code
 # For security gate: fail on critical OR high
 docker scout cves local://docuvault-backend:latest \
    --only-severity critical,high \
    --exit-code
 ```
 ### structlog logger usage in route handlers
 ```python
 # Source: https://www.structlog.org/en/stable/getting-started.html
 import structlog
 log = structlog.get_logger()
 # In a route handler — user_id already bound by CorrelationIDMiddleware
 async def upload_document(...):
    structlog.contextvars.bind_contextvars(user_id=str(current_user.id))
    log.info("document.upload.started", filename=file.filename, size_bytes=file.size)
    # ... processing ...
    log.info("document.upload.complete", document_id=str(doc.id))
 ```
 ### Locust headless run with CSV stats
 ```bash
 # Source: https://docs.locust.io/en/stable/running-without-web-ui.html
 locust \
  --headless \
  --users 50 \
  --spawn-rate 10 \
  --run-time 5m \
  --host http://localhost:8000 \
  --csv backend/load_tests/results \
  -f backend/load_tests/locustfile.py
 # CSV output:
 #   backend/load_tests/results_stats.csv       — per-endpoint percentiles
 #   backend/load_tests/results_failures.csv    — failure details
 #   backend/load_tests/results_stats_history.csv
 ```
 ---
 ## State of the Art
 | Old Approach | Current Approach | When Changed | Impact |
 |--------------|------------------|--------------|--------|
 | logging.basicConfig JSON handler | structlog with ProcessorFormatter bridge | structlog 21+ | Unified stdlib + structlog pipeline; single config |
 | Promtail as only log agent | Grafana Alloy (new default) | 2024 (Grafana Alloy GA) | Alloy is now recommended; Promtail still supported and simpler for local dev |
 | docker-compose tmpfs string syntax | Long-form with mode/uid options | Docker Compose v3.6+ | Enables permission control on tmpfs |
 | Locust 1.x `TaskSet` pattern | Locust 2.x `@task` decorators on `HttpUser` | 2021 (Locust 2.0) | TaskSet deprecated; @task on HttpUser is current |
 | Loki schema v11/v12 | schema v13 (tsdb store) | Loki 2.8+ | v13 is now required for new deployments; v12 still works |
 **Deprecated/outdated:**
 - `structlog.threadlocal` module: replaced by `structlog.contextvars` for async apps — do not use.
 - `locust.TaskSet` class: still exists but deprecated; use `@task` on `HttpUser` directly.
 - Loki `boltdb-shipper` store: replaced by `tsdb` in schema v13.
 ---
 ## Assumptions Log
 | # | Claim | Section | Risk if Wrong |
 |---|-------|---------|---------------|
 | A1 | `request.state.current_user` set before slowapi evaluates key_func when set as first line of handler body | Pattern 4 (per-account limiter) | Per-account limit silently falls back to IP limit — D-12 not met |
 | A2 | Locust upload flow targets `POST /api/documents/` directly (single-step) vs. the presigned URL two-step flow | Pattern 7 (Locust) | Load test fails 100% if two-step flow is required |
 | A3 | `pip install --prefix=/install` followed by `COPY --from=builder /install /usr/local` correctly installs all packages into Python's site-packages in the runtime stage | Pattern 5 (Dockerfile) | Runtime stage missing packages → import errors at startup |
 | A4 | `celery-beat` is NOT subject to `read_only: true` per D-08 (which says "FastAPI and Celery worker") | Pitfall 7 | celery-beat fails to write schedule file → scheduled tasks stop |
 | A5 | PyMuPDF does not write to paths outside `/tmp` that would be blocked by `read_only: true` | Pitfall 6 | Extraction fails silently with PermissionError |
 ---
 ## Open Questions
 1. **Upload endpoint for load testing**
   - What we know: `services/extractor.py` uses `tempfile.NamedTemporaryFile`; the upload flow may be two-step (presigned URL) or one-step depending on the current documents.py implementation
   - What's unclear: Whether Locust needs to implement the three-step presigned flow (upload-url → PUT to MinIO → confirm) or if a simpler direct POST exists
   - Recommendation: Read `backend/api/documents.py` lines for the upload endpoint before finalizing the locustfile; if two-step, implement the MinIO PUT step using `self.client.put` to MinIO's host:9000
 2. **Load test user bootstrap**
   - What we know: Locust needs a valid user to authenticate; no seed user for load testing exists yet
   - What's unclear: Whether to create the user via the registration API (on_start) or pre-seed via Alembic
   - Recommendation: Use `on_start` to register if not exists (catch 409), then login — self-contained, no DB dependency
 3. **Loki tmpfs mount persistence**
   - What we know: Loki service needs a writable `/loki` directory for chunk storage
   - What's unclear: Whether the named volume `loki_data:/loki` is sufficient or if Loki's own container permissions require a matching UID
   - Recommendation: Use named volume (not read_only on loki service — it's not a user-facing service)
 ---
 ## Environment Availability
 | Dependency | Required By | Available | Version | Fallback |
 |------------|------------|-----------|---------|----------|
 | Docker | Container hardening, docker scout, Loki stack | ✓ | 29.5.2 | — |
 | docker scout | CVE scanning gate | ✓ | v1.20.4 | trivy (not installed — document install) |
 | trivy | CVE scanning fallback | ✗ | — | `brew install trivy` or `apt-get install trivy` |
 | Python 3.12 (host) | Locust runs outside container | ✓ | 3.12 (via backend venv) | — |
 | structlog | structlog logging | needs install | 25.5.0 on PyPI | — |
 | locust | load tests | needs install | 2.34.0 on PyPI | — |
 | Loki image | log aggregation | pull on first compose up | grafana/loki:latest | — |
 | Promtail image | log collection | pull on first compose up | grafana/promtail:latest | — |
 | Grafana image | log visualization | pull on first compose up | grafana/grafana:latest | — |
 **Missing dependencies with no fallback:**
 - None that block core implementation.
 **Missing dependencies with fallback:**
 - trivy (CVE scanning): fallback is docker scout (already available). Document trivy as optional alternative.
 - docker login session: required for `docker scout cves` against local images. Planner must add a setup step.
 ---
 ## Validation Architecture
 ### Test Framework
 | Property | Value |
 |----------|-------|
 | Framework | pytest 8.2 + pytest-asyncio (asyncio_mode=auto) |
 | Config file | `backend/pytest.ini` |
 | Quick run command | `cd backend && pytest tests/ -v -x` |
 | Full suite command | `cd backend && pytest tests/ -v` |
 ### Phase Requirements → Test Map
 | Req ID | Behavior | Test Type | Automated Command | File Exists? |
 |--------|----------|-----------|-------------------|-------------|
 | D-01 | structlog emits JSON with correlation_id field | Unit | `pytest tests/test_logging.py -x` | ❌ Wave 0 |
 | D-11 | get_client_ip returns direct IP when peer is untrusted | Unit | `pytest tests/test_rate_limiting.py::test_get_client_ip_untrusted -x` | ❌ Wave 0 |
 | D-11 | get_client_ip reads XFF when peer is trusted proxy | Unit | `pytest tests/test_rate_limiting.py::test_get_client_ip_trusted_proxy -x` | ❌ Wave 0 |
 | D-12 | per-account limiter key is user.id not IP | Unit | `pytest tests/test_rate_limiting.py::test_account_limiter_key -x` | ❌ Wave 0 |
 | D-12 | authenticated endpoint returns 429 after 100 req/min | Integration | `pytest tests/test_rate_limiting.py::test_account_rate_limit -x` | ❌ Wave 0 |
 | D-07 | Docker image runs as uid=1000, not root | Manual/smoke | `docker run --rm docuvault-backend:latest id` | N/A manual |
 | D-08 | read_only container can write to /tmp | Manual/smoke | `docker compose up backend` + upload a doc | N/A manual |
 | D-06 | SLA: p95 < 200ms, p99 < 500ms at 50 users | Load test (Locust) | `locust --headless ... -f backend/load_tests/locustfile.py` | ❌ Wave 0 |
 ### Sampling Rate
 - **Per task commit:** `cd backend && pytest tests/ -v -x --tb=short`
 - **Per wave merge:** `cd backend && pytest tests/ -v`
 - **Phase gate:** Full suite green before `/gsd:verify-work`
 ### Wave 0 Gaps
 - [ ] `backend/tests/test_logging.py` — structlog config tests (D-01)
 - [ ] `backend/tests/test_rate_limiting.py` — get_client_ip unit tests + per-account limiter tests (D-11, D-12)
 - [ ] `backend/load_tests/__init__.py` — empty, marks directory as non-pytest-discoverable
 - [ ] `backend/load_tests/locustfile.py` — Locust user class (D-04..D-06)
 ---
 ## Security Domain
 ### Applicable ASVS Categories
 | ASVS Category | Applies | Standard Control |
 |---------------|---------|-----------------|
 | V2 Authentication | no (already implemented in Phase 2) | — |
 | V3 Session Management | no (already implemented) | — |
 | V4 Access Control | yes — rate limit bypass prevention | slowapi trusted-proxy key_func (D-11/D-12) |
 | V5 Input Validation | no new user inputs in this phase | — |
 | V6 Cryptography | no | — |
 | V14 Configuration | yes — container hardening, read-only fs | D-07..D-10 |
 ### Known Threat Patterns for This Phase
 | Pattern | STRIDE | Standard Mitigation |
 |---------|--------|---------------------|
 | Rate limit bypass via X-Forwarded-For header spoofing | Tampering | Trusted-proxy CIDR check in get_client_ip (D-11) |
 | Container escape via write to host filesystem | Elevation of Privilege | read_only: true + cap_drop: ALL (D-08, D-09) |
 | CVE exploitation via outdated base image packages | Tampering | docker scout cves zero-critical gate (D-10) |
 | Log injection via user-controlled strings in log fields | Tampering | structlog JSON renderer escapes all values — JSON encoding prevents log injection |
 | Locust test user credentials in version control | Information Disclosure | Use env vars for load test credentials; add to .gitignore |
 ---
 ## Sources
 ### Primary (HIGH confidence)
 - [structlog official docs — contextvars](https://www.structlog.org/en/stable/contextvars.html) — middleware pattern, clear_contextvars(), bind_contextvars()
 - [structlog official docs — stdlib integration](https://www.structlog.org/en/stable/standard-library.html) — ProcessorFormatter, foreign_pre_chain
 - [structlog official docs — getting started](https://www.structlog.org/en/stable/getting-started.html) — configure() API, JSON/console renderer
 - [Locust official docs — writing a locustfile](https://docs.locust.io/en/stable/writing-a-locustfile.html) — HttpUser, on_start, @task weights
 - [Locust official docs — headless mode](https://docs.locust.io/en/stable/running-without-web-ui.html) — --headless, --users, --spawn-rate, --run-time, exit codes
 - [Locust official docs — configuration](https://docs.locust.io/en/stable/configuration.html) — --csv, --host flags
 - [docker scout cves official docs](https://docs.docker.com/reference/cli/docker/scout/cves/) — --exit-code, --only-severity, exit code behavior
 - PyPI registry — structlog 25.5.0, locust 2.34.0, slowapi 0.1.9 confirmed current
 ### Secondary (MEDIUM confidence)
 - [wazaari.dev FastAPI + structlog integration](https://wazaari.dev/blog/fastapi-structlog-integration) — full processor chain + middleware pattern verified against structlog official docs
 - [Promtail docker_sd_configs pattern](https://ornlu-is.github.io/docker_compose_promtail_loki_grafana/) — container log scraping config verified against Promtail docs
 - [Loki single-binary config](https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose-bcb07d1a51aa) — loki-config.yaml single-node filesystem mode
 - [Docker tmpfs non-root pattern](https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/) — mode=1777 solution for non-root tmpfs access
 ### Tertiary (LOW confidence)
 - [slowapi GitHub](https://github.com/laurentS/slowapi) — per-account key_func pattern inferred from API reference; no official example for request.state pattern
 ---
 ## Metadata
 **Confidence breakdown:**
 - Standard stack: HIGH — structlog 25.5.0 and locust 2.34.0 verified on PyPI; slowapi 0.1.9 already installed
 - Architecture (structlog): HIGH — verified against official structlog docs with working code examples
 - Architecture (Loki/Promtail): MEDIUM — working config patterns from community sources cross-checked with official Loki install docs
 - Architecture (container hardening): MEDIUM — Docker best practices well-documented; tmpfs non-root pattern verified against official compose format
 - Architecture (slowapi per-account): LOW-MEDIUM — core pattern is sound; key_func evaluation order vs. Depends resolution is an assumption that needs Wave 0 test
 - Locust load testing: HIGH — official docs used for all flags and patterns
 - docker scout CVE scanning: HIGH — official Docker docs used
 **Research date:** 2026-06-02
 **Valid until:** 2026-07-02 (structlog and locust are stable libraries; Loki config may shift with new releases)