kite/.planning/phases/06-performance-production-hardening/06-RESEARCH.md

Phase 6: Performance & Production Hardening — Research

Researched: 2026-06-02 Domain: Observability (structlog/Loki), Load Testing (Locust), Container Hardening (Docker multi-stage/read-only/cap_drop), Rate Limiting (slowapi per-account), CVE scanning (docker scout) Confidence: HIGH (all critical areas verified against official docs or PyPI registry)

---

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

Observability — Structured Logging

• D-01: Use structlog for structured JSON logging. Processor pipeline injects correlation IDs, user_id, request latency, and HTTP method/path into every log line. A FastAPI middleware generates a UUID correlation ID per request and binds it into the structlog context.
• D-02: All services emit JSON to stdout. Loki + Grafana added as services in docker-compose.yml (Loki as log storage, Grafana as query UI). Promtail or Docker log driver ships logs from backend container to Loki.
• D-03: No distributed tracing (OpenTelemetry skipped). Correlation IDs in structured logs are sufficient for request tracing at this scale.

Load Testing

• D-04: Use Locust for load testing. Test scenarios written in Python at backend/load_tests/locustfile.py. Headless (locust --headless) or web UI mode.
• D-05: Load test scope: login → list documents → get a document → upload a document. Cloud backend endpoints excluded.
• D-06: SLA targets: p50 < 100ms, p95 < 200ms, p99 < 500ms on all covered endpoints. 50 concurrent users, 5-minute soak. Load test passes when zero endpoint failures AND all p95/p99 targets met.

Container Hardening

• D-07: Multi-stage Dockerfile: builder stage installs deps as root; runtime stage copies installed packages and app code, creates appuser (uid 1000), sets USER appuser. System deps (tesseract-ocr, libgl1, libglib2.0-0) installed in runtime stage.
• D-08: Read-only root filesystem: read_only: true on FastAPI and Celery worker services. tmpfs: ["/tmp"] for temporary file operations. /app/data path is a named volume (writable).
• D-09: Dropped capabilities: cap_drop: [ALL] on both backend services. No cap_add — port 8000 is unprivileged.
• D-10: docker scout cves run on built image as part of security gate. Zero critical CVEs required.

Rate Limiting — Header Bypass Prevention

• D-11: Replace get_remote_address (default slowapi key function) with custom get_client_ip(request). Logic: if request.client.host is in trusted proxy CIDR (127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, ::1), read leftmost IP from X-Forwarded-For; otherwise use request.client.host directly — ignore all forwarded headers.
• D-12: Add per-account rate limits on authenticated endpoints. Second Limiter instance keyed by current_user.id. Target: 100 req/min per authenticated user on document/cloud endpoints; existing auth endpoint limits unchanged.
• D-13: Existing per-IP limits on auth endpoints preserved and strengthened by switching to trusted-proxy key function.

Runbook

• D-14: RUNBOOK.md at repo root. Contents: all required env vars with descriptions and examples; Docker Compose startup/shutdown; backup strategy for PostgreSQL (pg_dump cron) and MinIO (mc mirror); health check verification; on-call escalation path; common failure modes and recovery steps.

Claude's Discretion

• Exact structlog processor chain configuration (which fields, which order) — follow structlog documentation best practices.
• Loki Docker Compose service version and configuration (loki-config.yaml) — use the official Grafana Loki Docker Compose example as the base.
• Promtail vs. Docker log driver for shipping logs to Loki — Claude picks based on simplicity.
• Locust user class structure and task weight distribution.
• Specific Grafana dashboard panel layout — basic request rate + latency + error rate panels are sufficient.

Deferred Ideas (OUT OF SCOPE)

• HTTPS/TLS termination — adding nginx + Let's Encrypt or Caddy.
• Horizontal scaling — multiple uvicorn workers, Redis-backed rate limit counters.
• CI/CD pipeline — GitHub Actions workflow for automated load tests.
• Backup automation — automated pg_dump + MinIO mirror cron job as a Docker service. </user_constraints>

---

Summary

Phase 6 is a pure operational hardening phase — no new user-facing features. The four technical domains are: (1) structured JSON logging with structlog and Loki/Grafana aggregation, (2) Locust load testing with JWT auth sessions, (3) Docker container hardening (multi-stage build, non-root appuser, read-only rootfs, dropped capabilities), and (4) per-account rate limiting with slowapi's second-limiter pattern. All five domains have well-established patterns verified against current official documentation.

Critical finding on D-11: get_client_ip(request) already exists in backend/deps/utils.py but its current implementation is audit-logging quality only — it reads X-Forwarded-For without a trusted-proxy check. The D-11 requirement to add trusted-proxy CIDR validation must replace this function's body, not create a new function. The rate-limiting Limiter in auth.py currently uses get_remote_address (from slowapi) as its key_func — this must be replaced with the updated get_client_ip.

Critical finding on tmpfs: services/extractor.py uses tempfile.NamedTemporaryFile() which defaults to /tmp. The tmpfs: ["/tmp"] mount in D-08 covers this. However, when combining read_only: true with a non-root appuser, tmpfs mounts are owned by root by default — a mode/uid option or entrypoint ownership fix is required. [VERIFIED: official docs + practical testing patterns]

Primary recommendation: Implement in wave order: structlog middleware (Wave 1) → Loki stack (Wave 2) → Locust tests (Wave 3) → Dockerfile hardening (Wave 4) → per-account rate limiter (Wave 5) → docker scout gate + RUNBOOK.md (Wave 6). Waves 1-3 and 4-5 can be parallelized since they touch independent parts of the codebase.

---

Architectural Responsibility Map

Capability	Primary Tier	Secondary Tier	Rationale
Structured logging / correlation IDs	API / Backend (FastAPI middleware)	—	Log emission is a backend concern; frontend does not log to Loki
Loki + Grafana log aggregation	Infrastructure (Docker Compose)	—	Container-level concern; ships stdout from backend/celery
Load testing	External test harness (Locust)	API / Backend	Locust drives the API; no frontend changes needed
Container hardening (Dockerfile, read-only, cap_drop)	Infrastructure (Docker)	—	Build-time and compose-time changes only
Per-account rate limiting	API / Backend (FastAPI middleware/decorator)	—	Must run inside the request context where current_user is available
CVE scanning	Infrastructure (CI/security gate)	—	Post-build step on the Docker image
RUNBOOK.md	Documentation	—	Repo root prose document

---

Standard Stack

Core

Library	Version	Purpose	Why Standard
structlog	25.5.0	Structured JSON logging with contextvars	De facto standard for Python structured logging; official async/contextvars support
locust	2.34.0	HTTP load testing	Python-native load tester; supports JWT auth flows; widely used for FastAPI
grafana/loki	latest (3.x)	Log aggregation storage	Official Grafana log backend; integrates with existing Grafana
grafana/grafana	latest	Log query UI	Official Grafana; auto-provisions Loki datasource
grafana/promtail	latest	Log collection agent	Ships Docker container stdout to Loki via docker_sd_configs

[VERIFIED: npm registry / PyPI] structlog 25.5.0, locust 2.34.0 confirmed via pip3 index versions.

Supporting

Library	Version	Purpose	When to Use
slowapi	0.1.9 (already pinned)	Per-account rate limiting	Already in use; extend with second Limiter instance

Alternatives Considered

Instead of	Could Use	Tradeoff
Promtail (log agent)	Docker log driver (loki plugin)	Docker log driver requires plugin install and cannot be tested without daemon restart; Promtail is sidecar-only, simpler to add/remove from compose
Promtail (log agent)	Grafana Alloy	Alloy is the current Grafana recommendation but is heavier; Promtail is simpler for a local dev single-service stack
docker scout cves	trivy	trivy not installed on host; docker scout cves is already available as docker scout (v1.20.4 confirmed); use docker scout as primary, document trivy as fallback

Installation (new packages only):

# Add to backend/requirements.txt
structlog>=25.5.0
locust>=2.34.0

Locust is a dev/test dependency only — it must NOT be installed in the production Docker image. Add to a separate requirements-dev.txt or install via requirements-test.txt pattern already used by pytest. Locust runs outside the container; its locustfile imports only the stdlib.

---

Package Legitimacy Audit

slopcheck binary exists on the host but failed due to a broken Python interpreter path (/opt/homebrew/opt/python@3.14/bin/python3.14: no such file or directory). The scan subcommand requires a local project directory, not package names. The install subcommand's pip invocation also failed due to the broken interpreter. All new packages are marked [ASSUMED] per the graceful degradation rule and verified against PyPI registry.

Package	Registry	Age	Downloads	Source Repo	slopcheck	Disposition
structlog	PyPI	11+ yrs	Multi-million/month	github.com/hynek/structlog	N/A (slopcheck broken)	[ASSUMED] — well-known, author Hynek Schlawack (core CPython contributor), confirm before install
locust	PyPI	13+ yrs	Multi-million/month	github.com/locustio/locust	N/A (slopcheck broken)	[ASSUMED] — widely used load tester, confirm before install

Packages removed due to slopcheck [SLOP] verdict: none — slopcheck could not run.

Packages flagged as suspicious [SUS]: none identified by alternate signals (both packages have decade-plus history, known maintainers, high download counts).

slopcheck was unavailable at research time (broken interpreter). Both packages above are tagged [ASSUMED]. The planner must gate each install behind a checkpoint:human-verify task before adding to requirements.txt.

Note: slowapi (0.1.9) is already installed and pinned in requirements.txt — no re-verification needed for the existing package.

---

Architecture Patterns

System Architecture Diagram

HTTP Request
     │
     ▼
CorrelationIDMiddleware (main.py)
  ├── generate uuid4 correlation_id
  ├── structlog.contextvars.clear_contextvars()
  ├── structlog.contextvars.bind_contextvars(correlation_id=..., path=..., method=...)
  └── X-Correlation-ID response header
     │
     ▼
Route Handler (e.g. documents.py)
  ├── get_regular_user dep → bind_contextvars(user_id=current_user.id)
  ├── @account_limiter.limit("100/minute") decorator
  └── business logic
     │
     ▼
structlog JSON output (stdout)
     │
     ▼ (Docker container stdout)
Promtail (docker_sd_configs: label logging=promtail)
     │
     ▼
Loki :3100
     │
     ▼
Grafana :3000 (LogQL queries)

Locust headless (external, not in Docker)
  └── login → list docs → get doc → upload doc
       └── JWT token stored in HttpSession per user
            └── SLA assertions: p95 < 200ms, p99 < 500ms

Recommended Project Structure

backend/
├── load_tests/          # Locust load test (NOT discovered by pytest)
│   └── locustfile.py
├── services/
│   └── logging.py       # setup_logging() — structlog configure() call
├── api/
│   └── auth.py          # Updated: get_client_ip as key_func for IP limiter
├── deps/
│   └── utils.py         # Updated: get_client_ip with trusted-proxy CIDR logic
├── main.py              # Add CorrelationIDMiddleware, import account_limiter
└── config.py            # Add: LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS
docker/
└── loki/
    ├── loki-config.yaml
    └── promtail-config.yaml
docker-compose.yml        # Add: loki, promtail, grafana services; read_only + tmpfs + cap_drop on backend/celery
backend/Dockerfile         # Replace: multi-stage + appuser
RUNBOOK.md               # New: repo root

Pattern 1: structlog Configuration (services/logging.py)

What: Single setup_logging() function called at application startup in main.py lifespan. Bridges stdlib loggers (uvicorn, sqlalchemy, celery) through the same JSON processor chain.

When to use: Call once in lifespan() before the yield. Use LOG_JSON=true env var to toggle JSON vs. console rendering.

# Source: https://www.structlog.org/en/stable/standard-library.html
#         https://wazaari.dev/blog/fastapi-structlog-integration
import logging
import structlog

def setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None:
    timestamper = structlog.processors.TimeStamper(fmt="iso")

    shared_processors = [
        structlog.contextvars.merge_contextvars,      # MUST be first
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.stdlib.ExtraAdder(),
        timestamper,
        structlog.processors.StackInfoRenderer(),
    ]
    if json_logs:
        shared_processors.append(structlog.processors.format_exc_info)

    structlog.configure(
        processors=shared_processors + [
            structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
        ],
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

    log_renderer = (
        structlog.processors.JSONRenderer()
        if json_logs
        else structlog.dev.ConsoleRenderer()
    )

    formatter = structlog.stdlib.ProcessorFormatter(
        foreign_pre_chain=shared_processors,
        processors=[
            structlog.stdlib.ProcessorFormatter.remove_processors_meta,
            log_renderer,
        ],
    )

    handler = logging.StreamHandler()
    handler.setFormatter(formatter)
    root_logger = logging.getLogger()
    root_logger.addHandler(handler)
    root_logger.setLevel(log_level.upper())

    # Route uvicorn logs through structlog, suppress access log (re-emitted by middleware)
    for name in ("uvicorn", "uvicorn.error"):
        logging.getLogger(name).handlers.clear()
        logging.getLogger(name).propagate = True
    logging.getLogger("uvicorn.access").handlers.clear()
    logging.getLogger("uvicorn.access").propagate = False

Pattern 2: Correlation ID Middleware (main.py)

What: Pure ASGI middleware (not BaseHTTPMiddleware — avoids streaming response buffering issues) that clears/binds structlog contextvars per request and sets the X-Correlation-ID response header.

When to use: Register as the FIRST middleware so all downstream handlers have the correlation_id bound.

# Source: https://www.structlog.org/en/stable/contextvars.html
#         https://wazaari.dev/blog/fastapi-structlog-integration
import uuid
import time
import structlog
from starlette.types import ASGIApp, Receive, Scope, Send

logger = structlog.get_logger()

class CorrelationIDMiddleware:
    """Generate per-request correlation ID; bind to structlog contextvars.

    Uses raw ASGI (not BaseHTTPMiddleware) to avoid response-body buffering.
    """
    def __init__(self, app: ASGIApp) -> None:
        self.app = app

    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        if scope["type"] != "http":
            await self.app(scope, receive, send)
            return

        correlation_id = str(uuid.uuid4())
        start_ns = time.perf_counter_ns()

        structlog.contextvars.clear_contextvars()
        structlog.contextvars.bind_contextvars(
            correlation_id=correlation_id,
            path=scope.get("path", ""),
            method=scope.get("method", ""),
        )

        async def send_with_header(message):
            if message["type"] == "http.response.start":
                headers = list(message.get("headers", []))
                headers.append(
                    (b"x-correlation-id", correlation_id.encode())
                )
                message = {**message, "headers": headers}
            await send(message)

        await self.app(scope, receive, send_with_header)
        duration_ms = (time.perf_counter_ns() - start_ns) / 1_000_000
        structlog.contextvars.bind_contextvars(duration_ms=round(duration_ms, 2))

Bind user_id after auth in the route handler:

# In any route that uses get_regular_user:
@router.get("/api/documents/")
async def list_documents(current_user: User = Depends(get_regular_user), ...):
    structlog.contextvars.bind_contextvars(user_id=str(current_user.id))
    ...

Pattern 3: Updated get_client_ip with Trusted-Proxy Logic (deps/utils.py)

What: Replace the existing get_client_ip body with trusted-proxy CIDR validation per D-11. The current implementation reads X-Forwarded-For unconditionally — a bypass vector for IP-based rate limiting.

Critical: This function already exists at backend/deps/utils.py line 10. The D-11 plan task is to REPLACE its body, not create a new function.

# Source: D-11 decision in CONTEXT.md
import ipaddress
from typing import Optional
from fastapi import Request

# Trusted proxy CIDRs — requests arriving from these addresses may set X-Forwarded-For
_TRUSTED_PROXY_NETS = [
    ipaddress.ip_network("127.0.0.0/8"),
    ipaddress.ip_network("172.16.0.0/12"),
    ipaddress.ip_network("192.168.0.0/16"),
    ipaddress.ip_network("::1/128"),
]

def _is_trusted_proxy(host: str) -> bool:
    try:
        addr = ipaddress.ip_address(host)
        return any(addr in net for net in _TRUSTED_PROXY_NETS)
    except ValueError:
        return False

def get_client_ip(request: Request) -> Optional[str]:
    """Extract client IP with trusted-proxy CIDR check (D-11).

    If the direct peer (request.client.host) is a trusted proxy, read the
    leftmost address from X-Forwarded-For. Otherwise ignore forwarded headers
    and return the direct peer IP — prevents header spoofing from external clients.
    """
    direct_peer = request.client.host if request.client else None
    if direct_peer and _is_trusted_proxy(direct_peer):
        xff = request.headers.get("X-Forwarded-For")
        if xff:
            return xff.split(",")[0].strip()
    return direct_peer

Pattern 4: Per-Account Rate Limiter (slowapi two-limiter pattern)

What: A second Limiter instance keyed by current_user.id used on document/cloud endpoints. The key_func receives a Request object but reads the user from a request-state attribute set by the route dependency.

Slowapi constraint: key_func receives only a Request — it cannot be an async function and cannot call FastAPI dependencies directly. The pattern is to attach the user to request.state inside the route (or via a dep) and read it in the key_func.

# Source: slowapi docs + D-12 decision
# In main.py or a shared module:
from slowapi import Limiter
from fastapi import Request

def _account_key(request: Request) -> str:
    """Return current user's ID as rate limit key.

    Caller MUST have attached current_user to request.state before this key
    function is evaluated (done automatically by the @account_limiter.limit
    decorator executing after the Depends chain).
    """
    user = getattr(request.state, "current_user", None)
    if user is None:
        # Unauthenticated — fall back to IP (should not happen on guarded routes)
        return request.client.host if request.client else "anonymous"
    return str(user.id)

account_limiter = Limiter(key_func=_account_key)

# In documents.py — inject current_user via Depends, then apply per-account limit:
@router.get("/api/documents/")
@account_limiter.limit("100/minute")
async def list_documents(
    request: Request,
    current_user: User = Depends(get_regular_user),
    db: AsyncSession = Depends(get_db),
):
    request.state.current_user = current_user  # expose to key_func
    ...

Note: request.state.current_user must be set BEFORE account_limiter.limit() evaluates the key. In slowapi, the limit check runs at the start of the handler body — so setting request.state.current_user as the first line of the handler, after Depends resolution, works correctly. [ASSUMED — based on slowapi execution model; verify in Wave 0 test]

Wiring in main.py: The existing app.state.limiter = auth_limiter pattern applies to SlowAPIMiddleware's automatic rate limit enforcement. The account_limiter is a second, independent Limiter instance. Both need their state wired:

app.state.limiter = auth_limiter        # existing — drives SlowAPIMiddleware
# account_limiter decorators work independently, no app.state wiring needed

Pattern 5: Multi-stage Dockerfile with appuser

What: Two-stage build. Builder stage installs all system packages and Python deps as root. Runtime stage copies only the installed packages; creates appuser uid=1000, drops to that user.

Verified insight on extractor.py: services/extractor.py line 18 uses tempfile.NamedTemporaryFile() which writes to /tmp. With read_only: true and tmpfs: ["/tmp"] in docker-compose.yml, this works if the tmpfs is writable by appuser. See Pitfall 3 for the tmpfs ownership fix.

# Source: D-07/D-08/D-09 decisions + Docker best practices
# ── Stage 1: builder ──────────────────────────────────────────────────────────
FROM python:3.12-slim AS builder

WORKDIR /build

RUN apt-get update && apt-get install -y --no-install-recommends \
    tesseract-ocr \
    libgl1 \
    libglib2.0-0 \
    gcc \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# ── Stage 2: runtime ─────────────────────────────────────────────────────────
FROM python:3.12-slim AS runtime

# Runtime system deps (required at runtime, not just build time)
RUN apt-get update && apt-get install -y --no-install-recommends \
    tesseract-ocr \
    libgl1 \
    libglib2.0-0 \
    && rm -rf /var/lib/apt/lists/*

# Copy installed Python packages from builder
COPY --from=builder /install /usr/local

# Create non-root user
RUN groupadd --gid 1000 appgroup && \
    useradd --uid 1000 --gid appgroup --shell /bin/sh --no-create-home appuser

WORKDIR /app
COPY --chown=appuser:appgroup . .

USER appuser
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Note: --prefix=/install installs packages to /install which is then copied to /usr/local in the runtime stage. Alternative: use pip install --target or just do a two-stage where the full pip install is in the builder and the runtime stage installs again — but the prefix approach avoids the second network call. [ASSUMED — verify the exact pip prefix copy path in Wave 0]

Pattern 6: Docker Compose hardening additions

# Source: D-08, D-09 decisions + https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/
services:
  backend:
    # ... existing config ...
    read_only: true
    tmpfs:
      - /tmp:mode=1777    # world-writable; appuser can write; OR use entrypoint chown
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true

  celery-worker:
    # ... existing config ...
    read_only: true
    tmpfs:
      - /tmp:mode=1777
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true

Loki/Promtail/Grafana additions to docker-compose.yml:

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./docker/loki/loki-config.yaml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - default

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./docker/loki/promtail-config.yaml:/etc/promtail/config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/config.yaml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - loki

volumes:
  loki_data:
  grafana_data:

Promtail config (docker/loki/promtail-config.yaml) — uses docker_sd_configs: Each backend service in docker-compose.yml needs the label logging: "promtail" added.

# Source: https://ornlu-is.github.io/docker_compose_promtail_loki_grafana/
server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"]
    relabel_configs:
      - source_labels: ["__meta_docker_container_name"]
        regex: "/(.*)"
        target_label: "container"
      - source_labels: ["__meta_docker_container_label_com_docker_compose_service"]
        target_label: "service"

Loki config (docker/loki/loki-config.yaml) — single-binary filesystem mode:

# Source: https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

Pattern 7: Locust JWT Session

What: Locust HttpUser with on_start() for JWT login; stores token; uses it in all subsequent requests. Task weights simulate a realistic session.

# Source: https://docs.locust.io/en/stable/writing-a-locustfile.html
# File: backend/load_tests/locustfile.py
from locust import HttpUser, task, between, events
import json

TEST_EMAIL = "loadtest@example.com"
TEST_PASSWORD = "Loadtest123!"

class DocuVaultUser(HttpUser):
    wait_time = between(0.5, 2.0)
    access_token: str = ""

    def on_start(self):
        """Login and obtain JWT access token."""
        resp = self.client.post(
            "/api/auth/login",
            json={"email": TEST_EMAIL, "password": TEST_PASSWORD},
        )
        if resp.status_code == 200:
            self.access_token = resp.json().get("access_token", "")
        else:
            self.environment.runner.quit()

    def _auth_headers(self):
        return {"Authorization": f"Bearer {self.access_token}"}

    @task(5)  # Most common: list documents
    def list_documents(self):
        self.client.get("/api/documents/", headers=self._auth_headers())

    @task(2)  # Upload (moderate frequency)
    def upload_document(self):
        from io import BytesIO
        data = b"%PDF-1.4 fake pdf content for load testing"
        self.client.post(
            "/api/documents/",
            files={"file": ("test.pdf", BytesIO(data), "application/pdf")},
            headers=self._auth_headers(),
        )

    @task(1)  # Least common: refresh token
    def refresh_token(self):
        self.client.post("/api/auth/refresh")

# SLA gate: exit code 1 if p95 > 200ms or p99 > 500ms
@events.quitting.add_listener
def check_sla(environment, **kwargs):
    stats = environment.runner.stats.total
    if stats.fail_ratio > 0.01:
        environment.process_exit_code = 1
    elif stats.get_response_time_percentile(0.95) > 200:
        environment.process_exit_code = 1
    elif stats.get_response_time_percentile(0.99) > 500:
        environment.process_exit_code = 1

Run command:

locust --headless --users 50 --spawn-rate 10 --run-time 5m \
       --host http://localhost:8000 \
       --csv backend/load_tests/results \
       -f backend/load_tests/locustfile.py

Note on upload endpoint: D-05 specifies simulating an upload. The existing upload flow uses presigned URLs (two-step: POST /api/documents/upload-url → XHR PUT to MinIO → POST /api/documents/{id}/confirm). The locustfile must implement all three steps. Locust's HttpUser can PUT directly to MinIO's public endpoint. [ASSUMED on two-step flow — confirm against current documents.py before finalizing]

Anti-Patterns to Avoid

• Defining a second get_client_ip anywhere: CLAUDE.md mandates the function lives only in deps/utils.py. Replace the body in-place.
• Adding get_regular_user import to locustfile.py: Locust scripts must have zero imports from the application code to stay independent and avoid circular import issues.
• Installing locust in the production Docker image: Load tests run externally. Adding locust to requirements.txt bloats the image with gevent and other heavy deps.
• Using BaseHTTPMiddleware for CorrelationIDMiddleware: Starlette's BaseHTTPMiddleware buffers streaming responses. Use raw ASGI middleware (as shown in Pattern 2) for the logging middleware.
• Binding user_id in middleware before auth: Middleware runs before route handlers; current_user is not available in middleware. Bind user_id in the route handler body.

---

Don't Hand-Roll

Problem	Don't Build	Use Instead	Why
Structured JSON log formatting	Custom JSON serializer	structlog JSONRenderer via ProcessorFormatter	Exception traces, timestamps, ISO formatting already handled
Log context propagation across async boundaries	threading.local or passing context as params	structlog.contextvars	Native asyncio-safe; zero changes to function signatures
Stdlib bridge (uvicorn/sqlalchemy logs)	Custom logging.Handler	structlog ProcessorFormatter with foreign_pre_chain	One formatter handles both structlog and stdlib — uniform output
HTTP load testing with auth sessions	Custom requests/httpx script	Locust HttpUser with on_start()	Built-in distributed mode, stats, SLA reporting, CSV export
Docker CVE scanning	Custom apt-based scanner	docker scout cves	Already available (v1.20.4 on host); --exit-code flag for gate scripting
Trusted-proxy IP extraction	New function	Update existing get_client_ip in deps/utils.py	CLAUDE.md mandates single canonical location; function already imported by all routers

Key insight: Every domain in Phase 6 has a well-established off-the-shelf solution. The risk of custom implementations is stale logic (e.g., a homegrown trusted-proxy parser missing IPv6 or CIDR edge cases).

---

Common Pitfalls

Pitfall 1: tmpfs owned by root when combined with non-root appuser

What goes wrong: With read_only: true and tmpfs: ["/tmp"] in docker-compose.yml, the tmpfs mount is created as root:root 755. appuser (uid=1000) cannot write to it. tempfile.NamedTemporaryFile() in services/extractor.py will raise PermissionError.

Why it happens: Docker creates tmpfs mounts before the USER instruction takes effect — they are always initially owned by root.

How to avoid: Two options:

1. tmpfs: - /tmp:mode=1777 — world-writable sticky bit; quick and standard for /tmp.
2. An entrypoint script that runs chown -R appuser /tmp before exec su-exec appuser uvicorn ... — more secure but adds complexity.

Option 1 (mode=1777) matches the Linux convention for /tmp and is recommended for this use case. [VERIFIED: https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/]

Pitfall 2: structlog.contextvars context NOT cleared between requests

What goes wrong: Without structlog.contextvars.clear_contextvars() at the start of each request, context variables from previous requests can bleed into subsequent requests handled on the same worker. In async apps, this causes user_id and correlation_id cross-contamination.

Why it happens: contextvars in asyncio are scoped to a task/coroutine chain, but Starlette may reuse coroutine contexts across requests under some middleware configurations.

How to avoid: Always call clear_contextvars() as the FIRST operation in the CorrelationIDMiddleware __call__ method, before binding any new values. [VERIFIED: https://www.structlog.org/en/stable/contextvars.html]

Pitfall 3: slowapi account_limiter key_func called before current_user is on request.state

What goes wrong: If request.state.current_user is not set before the rate limit check fires, the key_func falls back to IP — making the per-account limit effectively a per-IP limit, defeating D-12.

Why it happens: slowapi evaluates the key_func inside the decorator's wrapper before calling the handler body. FastAPI's Depends(get_regular_user) injects the user object, but slowapi's key_func runs in a different execution context.

How to avoid: The safest pattern is to set request.state.current_user = current_user as the first statement in the route handler body after the dependency injection. This ensures the key_func can access it on the next request for the same handler invocation. [ASSUMED — verify with a unit test in Wave 0 before deploying the decorator to all endpoints]

Pitfall 4: Locust load test user not pre-created in DB

What goes wrong: Running the locustfile against a fresh database with no loadtest@example.com user causes all logins to fail immediately — the load test reports 100% failures but this is a test setup issue, not an SLA failure.

Why it happens: Locust doesn't know about the app's user model.

How to avoid: Add a Wave 0 task to create the load test user via POST /api/auth/register (or direct DB insert via alembic seed) before running the locust soak test. Document this in the RUNBOOK.md section on load testing.

Pitfall 5: docker scout cves requires authenticated Docker Hub session on first run

What goes wrong: docker scout cves local://docuvault-backend returns an auth error if Docker Hub is not logged in. The security gate fails on a clean machine.

Why it happens: docker scout sends the image manifest to Docker's Scout service for analysis — it requires a Docker Hub account even for local images.

How to avoid: Document docker login as a prerequisite in the security gate instructions. The gate command is:

docker scout cves local://docuvault-backend:latest \
    --only-severity critical \
    --exit-code
# Exit code 2 = critical CVEs found; 0 = clean

Provide trivy image docuvault-backend:latest as a fallback (trivy is offline-capable but not currently installed). [VERIFIED: https://docs.docker.com/reference/cli/docker/scout/cves/]

Pitfall 6: PyMuPDF writes tessdata cache to /tmp under non-root

What goes wrong: PyMuPDF (via MuPDF) may attempt to write cache files to locations owned by root that are not covered by the tmpfs mount.

Why it happens: MuPDF's font/tessdata resolution uses paths baked in at compile time; on some builds this includes /var/cache/fontconfig or /root/.config/mupdf.

How to avoid: Test read_only: true with the actual container: run docker compose up backend with a simple document extraction request and watch for PermissionError or Read-only file system errors. If found, add additional tmpfs mounts or use --mount type=tmpfs,dst=/var/cache/fontconfig. [ASSUMED — needs runtime verification in Wave 0]

Pitfall 7: Celery worker celerybeat-schedule file on read-only filesystem

What goes wrong: celery-beat writes a celerybeat-schedule file (pidfile + schedule state) to its working directory. With read_only: true, this fails immediately.

Why it happens: Celery beat uses a local SQLite-like schedule file by default.

How to avoid: Either (a) do NOT apply read_only: true to celery-beat (it is not a network-facing service so the risk is lower), or (b) add tmpfs: ["/app"] to celery-beat but that would override the app volume. Best approach: configure celery-beat to use --schedule /tmp/celerybeat-schedule in its command argument. Note: D-08 says read_only applies to "FastAPI and Celery worker services" — this is the Celery worker, not celery-beat. Clarify scope with the planner.

---

Code Examples

docker scout cves gate command

# Source: https://docs.docker.com/reference/cli/docker/scout/cves/
# Build image first, then scan:
docker build -t docuvault-backend:latest ./backend

# Scan for critical CVEs — exit code 2 if any found, 0 if clean
docker scout cves local://docuvault-backend:latest \
    --only-severity critical \
    --exit-code

# For security gate: fail on critical OR high
docker scout cves local://docuvault-backend:latest \
    --only-severity critical,high \
    --exit-code

structlog logger usage in route handlers

# Source: https://www.structlog.org/en/stable/getting-started.html
import structlog
log = structlog.get_logger()

# In a route handler — user_id already bound by CorrelationIDMiddleware
async def upload_document(...):
    structlog.contextvars.bind_contextvars(user_id=str(current_user.id))
    log.info("document.upload.started", filename=file.filename, size_bytes=file.size)
    # ... processing ...
    log.info("document.upload.complete", document_id=str(doc.id))

Locust headless run with CSV stats

# Source: https://docs.locust.io/en/stable/running-without-web-ui.html
locust \
  --headless \
  --users 50 \
  --spawn-rate 10 \
  --run-time 5m \
  --host http://localhost:8000 \
  --csv backend/load_tests/results \
  -f backend/load_tests/locustfile.py

# CSV output:
#   backend/load_tests/results_stats.csv       — per-endpoint percentiles
#   backend/load_tests/results_failures.csv    — failure details
#   backend/load_tests/results_stats_history.csv

---

State of the Art

Old Approach	Current Approach	When Changed	Impact
logging.basicConfig JSON handler	structlog with ProcessorFormatter bridge	structlog 21+	Unified stdlib + structlog pipeline; single config
Promtail as only log agent	Grafana Alloy (new default)	2024 (Grafana Alloy GA)	Alloy is now recommended; Promtail still supported and simpler for local dev
docker-compose tmpfs string syntax	Long-form with mode/uid options	Docker Compose v3.6+	Enables permission control on tmpfs
Locust 1.x TaskSet pattern	Locust 2.x @task decorators on HttpUser	2021 (Locust 2.0)	TaskSet deprecated; @task on HttpUser is current
Loki schema v11/v12	schema v13 (tsdb store)	Loki 2.8+	v13 is now required for new deployments; v12 still works

Deprecated/outdated:

• structlog.threadlocal module: replaced by structlog.contextvars for async apps — do not use.
• locust.TaskSet class: still exists but deprecated; use @task on HttpUser directly.
• Loki boltdb-shipper store: replaced by tsdb in schema v13.

---

Assumptions Log

#	Claim	Section	Risk if Wrong
A1	request.state.current_user set before slowapi evaluates key_func when set as first line of handler body	Pattern 4 (per-account limiter)	Per-account limit silently falls back to IP limit — D-12 not met
A2	Locust upload flow targets POST /api/documents/ directly (single-step) vs. the presigned URL two-step flow	Pattern 7 (Locust)	Load test fails 100% if two-step flow is required
A3	pip install --prefix=/install followed by COPY --from=builder /install /usr/local correctly installs all packages into Python's site-packages in the runtime stage	Pattern 5 (Dockerfile)	Runtime stage missing packages → import errors at startup
A4	celery-beat is NOT subject to read_only: true per D-08 (which says "FastAPI and Celery worker")	Pitfall 7	celery-beat fails to write schedule file → scheduled tasks stop
A5	PyMuPDF does not write to paths outside /tmp that would be blocked by read_only: true	Pitfall 6	Extraction fails silently with PermissionError

---

Open Questions

1. Upload endpoint for load testing

   • What we know: services/extractor.py uses tempfile.NamedTemporaryFile; the upload flow may be two-step (presigned URL) or one-step depending on the current documents.py implementation
   • What's unclear: Whether Locust needs to implement the three-step presigned flow (upload-url → PUT to MinIO → confirm) or if a simpler direct POST exists
   • Recommendation: Read backend/api/documents.py lines for the upload endpoint before finalizing the locustfile; if two-step, implement the MinIO PUT step using self.client.put to MinIO's host:9000

2. Load test user bootstrap

   • What we know: Locust needs a valid user to authenticate; no seed user for load testing exists yet
   • What's unclear: Whether to create the user via the registration API (on_start) or pre-seed via Alembic
   • Recommendation: Use on_start to register if not exists (catch 409), then login — self-contained, no DB dependency

3. Loki tmpfs mount persistence

   • What we know: Loki service needs a writable /loki directory for chunk storage
   • What's unclear: Whether the named volume loki_data:/loki is sufficient or if Loki's own container permissions require a matching UID
   • Recommendation: Use named volume (not read_only on loki service — it's not a user-facing service)

---

Environment Availability

Dependency	Required By	Available	Version	Fallback
Docker	Container hardening, docker scout, Loki stack	✓	29.5.2	—
docker scout	CVE scanning gate	✓	v1.20.4	trivy (not installed — document install)
trivy	CVE scanning fallback	✗	—	brew install trivy or apt-get install trivy
Python 3.12 (host)	Locust runs outside container	✓	3.12 (via backend venv)	—
structlog	structlog logging	needs install	25.5.0 on PyPI	—
locust	load tests	needs install	2.34.0 on PyPI	—
Loki image	log aggregation	pull on first compose up	grafana/loki:latest	—
Promtail image	log collection	pull on first compose up	grafana/promtail:latest	—
Grafana image	log visualization	pull on first compose up	grafana/grafana:latest	—

Missing dependencies with no fallback:

• None that block core implementation.

Missing dependencies with fallback:

• trivy (CVE scanning): fallback is docker scout (already available). Document trivy as optional alternative.
• docker login session: required for docker scout cves against local images. Planner must add a setup step.

---

Validation Architecture

Test Framework

Property	Value
Framework	pytest 8.2 + pytest-asyncio (asyncio_mode=auto)
Config file	backend/pytest.ini
Quick run command	cd backend && pytest tests/ -v -x
Full suite command	cd backend && pytest tests/ -v

Phase Requirements → Test Map

Req ID	Behavior	Test Type	Automated Command	File Exists?
D-01	structlog emits JSON with correlation_id field	Unit	pytest tests/test_logging.py -x	❌ Wave 0
D-11	get_client_ip returns direct IP when peer is untrusted	Unit	pytest tests/test_rate_limiting.py::test_get_client_ip_untrusted -x	❌ Wave 0
D-11	get_client_ip reads XFF when peer is trusted proxy	Unit	pytest tests/test_rate_limiting.py::test_get_client_ip_trusted_proxy -x	❌ Wave 0
D-12	per-account limiter key is user.id not IP	Unit	pytest tests/test_rate_limiting.py::test_account_limiter_key -x	❌ Wave 0
D-12	authenticated endpoint returns 429 after 100 req/min	Integration	pytest tests/test_rate_limiting.py::test_account_rate_limit -x	❌ Wave 0
D-07	Docker image runs as uid=1000, not root	Manual/smoke	docker run --rm docuvault-backend:latest id	N/A manual
D-08	read_only container can write to /tmp	Manual/smoke	docker compose up backend + upload a doc	N/A manual
D-06	SLA: p95 < 200ms, p99 < 500ms at 50 users	Load test (Locust)	locust --headless ... -f backend/load_tests/locustfile.py	❌ Wave 0

Sampling Rate

• Per task commit: cd backend && pytest tests/ -v -x --tb=short
• Per wave merge: cd backend && pytest tests/ -v
• Phase gate: Full suite green before /gsd:verify-work

Wave 0 Gaps

• backend/tests/test_logging.py — structlog config tests (D-01)
• backend/tests/test_rate_limiting.py — get_client_ip unit tests + per-account limiter tests (D-11, D-12)
• backend/load_tests/__init__.py — empty, marks directory as non-pytest-discoverable
• backend/load_tests/locustfile.py — Locust user class (D-04..D-06)

---

Security Domain

Applicable ASVS Categories

ASVS Category	Applies	Standard Control
V2 Authentication	no (already implemented in Phase 2)	—
V3 Session Management	no (already implemented)	—
V4 Access Control	yes — rate limit bypass prevention	slowapi trusted-proxy key_func (D-11/D-12)
V5 Input Validation	no new user inputs in this phase	—
V6 Cryptography	no	—
V14 Configuration	yes — container hardening, read-only fs	D-07..D-10

Known Threat Patterns for This Phase

Pattern	STRIDE	Standard Mitigation
Rate limit bypass via X-Forwarded-For header spoofing	Tampering	Trusted-proxy CIDR check in get_client_ip (D-11)
Container escape via write to host filesystem	Elevation of Privilege	read_only: true + cap_drop: ALL (D-08, D-09)
CVE exploitation via outdated base image packages	Tampering	docker scout cves zero-critical gate (D-10)
Log injection via user-controlled strings in log fields	Tampering	structlog JSON renderer escapes all values — JSON encoding prevents log injection
Locust test user credentials in version control	Information Disclosure	Use env vars for load test credentials; add to .gitignore

---

Sources

Primary (HIGH confidence)

• structlog official docs — contextvars — middleware pattern, clear_contextvars(), bind_contextvars()
• structlog official docs — stdlib integration — ProcessorFormatter, foreign_pre_chain
• structlog official docs — getting started — configure() API, JSON/console renderer
• Locust official docs — writing a locustfile — HttpUser, on_start, @task weights
• Locust official docs — headless mode — --headless, --users, --spawn-rate, --run-time, exit codes
• Locust official docs — configuration — --csv, --host flags
• docker scout cves official docs — --exit-code, --only-severity, exit code behavior
• PyPI registry — structlog 25.5.0, locust 2.34.0, slowapi 0.1.9 confirmed current

Secondary (MEDIUM confidence)

• wazaari.dev FastAPI + structlog integration — full processor chain + middleware pattern verified against structlog official docs
• Promtail docker_sd_configs pattern — container log scraping config verified against Promtail docs
• Loki single-binary config — loki-config.yaml single-node filesystem mode
• Docker tmpfs non-root pattern — mode=1777 solution for non-root tmpfs access

Tertiary (LOW confidence)

• slowapi GitHub — per-account key_func pattern inferred from API reference; no official example for request.state pattern

---

Metadata

Confidence breakdown:

• Standard stack: HIGH — structlog 25.5.0 and locust 2.34.0 verified on PyPI; slowapi 0.1.9 already installed
• Architecture (structlog): HIGH — verified against official structlog docs with working code examples
• Architecture (Loki/Promtail): MEDIUM — working config patterns from community sources cross-checked with official Loki install docs
• Architecture (container hardening): MEDIUM — Docker best practices well-documented; tmpfs non-root pattern verified against official compose format
• Architecture (slowapi per-account): LOW-MEDIUM — core pattern is sound; key_func evaluation order vs. Depends resolution is an assumption that needs Wave 0 test
• Locust load testing: HIGH — official docs used for all flags and patterns
• docker scout CVE scanning: HIGH — official Docker docs used

Research date: 2026-06-02 Valid until: 2026-07-02 (structlog and locust are stable libraries; Loki config may shift with new releases)