diff --git a/.planning/phases/06-performance-production-hardening/06-RESEARCH.md b/.planning/phases/06-performance-production-hardening/06-RESEARCH.md new file mode 100644 index 0000000..3fab2d8 --- /dev/null +++ b/.planning/phases/06-performance-production-hardening/06-RESEARCH.md @@ -0,0 +1,989 @@ +# Phase 6: Performance & Production Hardening — Research + +**Researched:** 2026-06-02 +**Domain:** Observability (structlog/Loki), Load Testing (Locust), Container Hardening (Docker multi-stage/read-only/cap_drop), Rate Limiting (slowapi per-account), CVE scanning (docker scout) +**Confidence:** HIGH (all critical areas verified against official docs or PyPI registry) + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +**Observability — Structured Logging** +- D-01: Use `structlog` for structured JSON logging. Processor pipeline injects correlation IDs, user_id, request latency, and HTTP method/path into every log line. A FastAPI middleware generates a UUID correlation ID per request and binds it into the structlog context. +- D-02: All services emit JSON to stdout. Loki + Grafana added as services in `docker-compose.yml` (Loki as log storage, Grafana as query UI). Promtail or Docker log driver ships logs from backend container to Loki. +- D-03: No distributed tracing (OpenTelemetry skipped). Correlation IDs in structured logs are sufficient for request tracing at this scale. + +**Load Testing** +- D-04: Use Locust for load testing. Test scenarios written in Python at `backend/load_tests/locustfile.py`. Headless (`locust --headless`) or web UI mode. +- D-05: Load test scope: login → list documents → get a document → upload a document. Cloud backend endpoints excluded. +- D-06: SLA targets: p50 < 100ms, p95 < 200ms, p99 < 500ms on all covered endpoints. 50 concurrent users, 5-minute soak. Load test passes when zero endpoint failures AND all p95/p99 targets met. + +**Container Hardening** +- D-07: Multi-stage Dockerfile: `builder` stage installs deps as root; `runtime` stage copies installed packages and app code, creates `appuser` (uid 1000), sets `USER appuser`. System deps (tesseract-ocr, libgl1, libglib2.0-0) installed in runtime stage. +- D-08: Read-only root filesystem: `read_only: true` on FastAPI and Celery worker services. `tmpfs: ["/tmp"]` for temporary file operations. `/app/data` path is a named volume (writable). +- D-09: Dropped capabilities: `cap_drop: [ALL]` on both backend services. No `cap_add` — port 8000 is unprivileged. +- D-10: `docker scout cves` run on built image as part of security gate. Zero critical CVEs required. + +**Rate Limiting — Header Bypass Prevention** +- D-11: Replace `get_remote_address` (default slowapi key function) with custom `get_client_ip(request)`. Logic: if `request.client.host` is in trusted proxy CIDR (127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, ::1), read leftmost IP from `X-Forwarded-For`; otherwise use `request.client.host` directly — ignore all forwarded headers. +- D-12: Add per-account rate limits on authenticated endpoints. Second `Limiter` instance keyed by `current_user.id`. Target: 100 req/min per authenticated user on document/cloud endpoints; existing auth endpoint limits unchanged. +- D-13: Existing per-IP limits on auth endpoints preserved and strengthened by switching to trusted-proxy key function. + +**Runbook** +- D-14: `RUNBOOK.md` at repo root. Contents: all required env vars with descriptions and examples; Docker Compose startup/shutdown; backup strategy for PostgreSQL (pg_dump cron) and MinIO (mc mirror); health check verification; on-call escalation path; common failure modes and recovery steps. + +### Claude's Discretion +- Exact structlog processor chain configuration (which fields, which order) — follow structlog documentation best practices. +- Loki Docker Compose service version and configuration (loki-config.yaml) — use the official Grafana Loki Docker Compose example as the base. +- Promtail vs. Docker log driver for shipping logs to Loki — Claude picks based on simplicity. +- Locust user class structure and task weight distribution. +- Specific Grafana dashboard panel layout — basic request rate + latency + error rate panels are sufficient. + +### Deferred Ideas (OUT OF SCOPE) +- HTTPS/TLS termination — adding nginx + Let's Encrypt or Caddy. +- Horizontal scaling — multiple uvicorn workers, Redis-backed rate limit counters. +- CI/CD pipeline — GitHub Actions workflow for automated load tests. +- Backup automation — automated pg_dump + MinIO mirror cron job as a Docker service. + + +--- + +## Summary + +Phase 6 is a pure operational hardening phase — no new user-facing features. The four technical domains are: (1) structured JSON logging with structlog and Loki/Grafana aggregation, (2) Locust load testing with JWT auth sessions, (3) Docker container hardening (multi-stage build, non-root appuser, read-only rootfs, dropped capabilities), and (4) per-account rate limiting with slowapi's second-limiter pattern. All five domains have well-established patterns verified against current official documentation. + +**Critical finding on D-11:** `get_client_ip(request)` already exists in `backend/deps/utils.py` but its current implementation is **audit-logging quality only** — it reads `X-Forwarded-For` without a trusted-proxy check. The D-11 requirement to add trusted-proxy CIDR validation must replace this function's body, not create a new function. The rate-limiting Limiter in `auth.py` currently uses `get_remote_address` (from slowapi) as its key_func — this must be replaced with the updated `get_client_ip`. + +**Critical finding on tmpfs:** `services/extractor.py` uses `tempfile.NamedTemporaryFile()` which defaults to `/tmp`. The `tmpfs: ["/tmp"]` mount in D-08 covers this. However, when combining `read_only: true` with a non-root appuser, tmpfs mounts are owned by root by default — a mode/uid option or entrypoint ownership fix is required. [VERIFIED: official docs + practical testing patterns] + +**Primary recommendation:** Implement in wave order: structlog middleware (Wave 1) → Loki stack (Wave 2) → Locust tests (Wave 3) → Dockerfile hardening (Wave 4) → per-account rate limiter (Wave 5) → docker scout gate + RUNBOOK.md (Wave 6). Waves 1-3 and 4-5 can be parallelized since they touch independent parts of the codebase. + +--- + +## Architectural Responsibility Map + +| Capability | Primary Tier | Secondary Tier | Rationale | +|------------|-------------|----------------|-----------| +| Structured logging / correlation IDs | API / Backend (FastAPI middleware) | — | Log emission is a backend concern; frontend does not log to Loki | +| Loki + Grafana log aggregation | Infrastructure (Docker Compose) | — | Container-level concern; ships stdout from backend/celery | +| Load testing | External test harness (Locust) | API / Backend | Locust drives the API; no frontend changes needed | +| Container hardening (Dockerfile, read-only, cap_drop) | Infrastructure (Docker) | — | Build-time and compose-time changes only | +| Per-account rate limiting | API / Backend (FastAPI middleware/decorator) | — | Must run inside the request context where current_user is available | +| CVE scanning | Infrastructure (CI/security gate) | — | Post-build step on the Docker image | +| RUNBOOK.md | Documentation | — | Repo root prose document | + +--- + +## Standard Stack + +### Core + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| structlog | 25.5.0 | Structured JSON logging with contextvars | De facto standard for Python structured logging; official async/contextvars support | +| locust | 2.34.0 | HTTP load testing | Python-native load tester; supports JWT auth flows; widely used for FastAPI | +| grafana/loki | latest (3.x) | Log aggregation storage | Official Grafana log backend; integrates with existing Grafana | +| grafana/grafana | latest | Log query UI | Official Grafana; auto-provisions Loki datasource | +| grafana/promtail | latest | Log collection agent | Ships Docker container stdout to Loki via docker_sd_configs | + +[VERIFIED: npm registry / PyPI] structlog 25.5.0, locust 2.34.0 confirmed via `pip3 index versions`. + +### Supporting + +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| slowapi | 0.1.9 (already pinned) | Per-account rate limiting | Already in use; extend with second Limiter instance | + +### Alternatives Considered + +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| Promtail (log agent) | Docker log driver (loki plugin) | Docker log driver requires plugin install and cannot be tested without daemon restart; Promtail is sidecar-only, simpler to add/remove from compose | +| Promtail (log agent) | Grafana Alloy | Alloy is the current Grafana recommendation but is heavier; Promtail is simpler for a local dev single-service stack | +| docker scout cves | trivy | trivy not installed on host; docker scout cves is already available as `docker scout` (v1.20.4 confirmed); use docker scout as primary, document trivy as fallback | + +**Installation (new packages only):** +```bash +# Add to backend/requirements.txt +structlog>=25.5.0 +locust>=2.34.0 +``` + +**Locust is a dev/test dependency only — it must NOT be installed in the production Docker image.** +Add to a separate `requirements-dev.txt` or install via `requirements-test.txt` pattern already used by pytest. Locust runs outside the container; its locustfile imports only the stdlib. + +--- + +## Package Legitimacy Audit + +> slopcheck binary exists on the host but failed due to a broken Python interpreter path (`/opt/homebrew/opt/python@3.14/bin/python3.14: no such file or directory`). The `scan` subcommand requires a local project directory, not package names. The `install` subcommand's pip invocation also failed due to the broken interpreter. All new packages are marked `[ASSUMED]` per the graceful degradation rule and verified against PyPI registry. + +| Package | Registry | Age | Downloads | Source Repo | slopcheck | Disposition | +|---------|----------|-----|-----------|-------------|-----------|-------------| +| structlog | PyPI | 11+ yrs | Multi-million/month | github.com/hynek/structlog | N/A (slopcheck broken) | `[ASSUMED]` — well-known, author Hynek Schlawack (core CPython contributor), confirm before install | +| locust | PyPI | 13+ yrs | Multi-million/month | github.com/locustio/locust | N/A (slopcheck broken) | `[ASSUMED]` — widely used load tester, confirm before install | + +**Packages removed due to slopcheck [SLOP] verdict:** none — slopcheck could not run. + +**Packages flagged as suspicious [SUS]:** none identified by alternate signals (both packages have decade-plus history, known maintainers, high download counts). + +*slopcheck was unavailable at research time (broken interpreter). Both packages above are tagged `[ASSUMED]`. The planner must gate each install behind a `checkpoint:human-verify` task before adding to requirements.txt.* + +**Note:** slowapi (0.1.9) is already installed and pinned in requirements.txt — no re-verification needed for the existing package. + +--- + +## Architecture Patterns + +### System Architecture Diagram + +``` +HTTP Request + │ + ▼ +CorrelationIDMiddleware (main.py) + ├── generate uuid4 correlation_id + ├── structlog.contextvars.clear_contextvars() + ├── structlog.contextvars.bind_contextvars(correlation_id=..., path=..., method=...) + └── X-Correlation-ID response header + │ + ▼ +Route Handler (e.g. documents.py) + ├── get_regular_user dep → bind_contextvars(user_id=current_user.id) + ├── @account_limiter.limit("100/minute") decorator + └── business logic + │ + ▼ +structlog JSON output (stdout) + │ + ▼ (Docker container stdout) +Promtail (docker_sd_configs: label logging=promtail) + │ + ▼ +Loki :3100 + │ + ▼ +Grafana :3000 (LogQL queries) +``` + +``` +Locust headless (external, not in Docker) + └── login → list docs → get doc → upload doc + └── JWT token stored in HttpSession per user + └── SLA assertions: p95 < 200ms, p99 < 500ms +``` + +### Recommended Project Structure + +``` +backend/ +├── load_tests/ # Locust load test (NOT discovered by pytest) +│ └── locustfile.py +├── services/ +│ └── logging.py # setup_logging() — structlog configure() call +├── api/ +│ └── auth.py # Updated: get_client_ip as key_func for IP limiter +├── deps/ +│ └── utils.py # Updated: get_client_ip with trusted-proxy CIDR logic +├── main.py # Add CorrelationIDMiddleware, import account_limiter +└── config.py # Add: LOG_LEVEL, LOG_JSON, TRUSTED_PROXY_CIDRS +docker/ +└── loki/ + ├── loki-config.yaml + └── promtail-config.yaml +docker-compose.yml # Add: loki, promtail, grafana services; read_only + tmpfs + cap_drop on backend/celery +backend/Dockerfile # Replace: multi-stage + appuser +RUNBOOK.md # New: repo root +``` + +### Pattern 1: structlog Configuration (services/logging.py) + +**What:** Single `setup_logging()` function called at application startup in `main.py` lifespan. Bridges stdlib loggers (uvicorn, sqlalchemy, celery) through the same JSON processor chain. + +**When to use:** Call once in `lifespan()` before the `yield`. Use `LOG_JSON=true` env var to toggle JSON vs. console rendering. + +```python +# Source: https://www.structlog.org/en/stable/standard-library.html +# https://wazaari.dev/blog/fastapi-structlog-integration +import logging +import structlog + +def setup_logging(json_logs: bool = False, log_level: str = "INFO") -> None: + timestamper = structlog.processors.TimeStamper(fmt="iso") + + shared_processors = [ + structlog.contextvars.merge_contextvars, # MUST be first + structlog.stdlib.add_log_level, + structlog.stdlib.add_logger_name, + structlog.stdlib.PositionalArgumentsFormatter(), + structlog.stdlib.ExtraAdder(), + timestamper, + structlog.processors.StackInfoRenderer(), + ] + if json_logs: + shared_processors.append(structlog.processors.format_exc_info) + + structlog.configure( + processors=shared_processors + [ + structlog.stdlib.ProcessorFormatter.wrap_for_formatter, + ], + logger_factory=structlog.stdlib.LoggerFactory(), + cache_logger_on_first_use=True, + ) + + log_renderer = ( + structlog.processors.JSONRenderer() + if json_logs + else structlog.dev.ConsoleRenderer() + ) + + formatter = structlog.stdlib.ProcessorFormatter( + foreign_pre_chain=shared_processors, + processors=[ + structlog.stdlib.ProcessorFormatter.remove_processors_meta, + log_renderer, + ], + ) + + handler = logging.StreamHandler() + handler.setFormatter(formatter) + root_logger = logging.getLogger() + root_logger.addHandler(handler) + root_logger.setLevel(log_level.upper()) + + # Route uvicorn logs through structlog, suppress access log (re-emitted by middleware) + for name in ("uvicorn", "uvicorn.error"): + logging.getLogger(name).handlers.clear() + logging.getLogger(name).propagate = True + logging.getLogger("uvicorn.access").handlers.clear() + logging.getLogger("uvicorn.access").propagate = False +``` + +### Pattern 2: Correlation ID Middleware (main.py) + +**What:** Pure ASGI middleware (not BaseHTTPMiddleware — avoids streaming response buffering issues) that clears/binds structlog contextvars per request and sets the `X-Correlation-ID` response header. + +**When to use:** Register as the FIRST middleware so all downstream handlers have the correlation_id bound. + +```python +# Source: https://www.structlog.org/en/stable/contextvars.html +# https://wazaari.dev/blog/fastapi-structlog-integration +import uuid +import time +import structlog +from starlette.types import ASGIApp, Receive, Scope, Send + +logger = structlog.get_logger() + +class CorrelationIDMiddleware: + """Generate per-request correlation ID; bind to structlog contextvars. + + Uses raw ASGI (not BaseHTTPMiddleware) to avoid response-body buffering. + """ + def __init__(self, app: ASGIApp) -> None: + self.app = app + + async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: + if scope["type"] != "http": + await self.app(scope, receive, send) + return + + correlation_id = str(uuid.uuid4()) + start_ns = time.perf_counter_ns() + + structlog.contextvars.clear_contextvars() + structlog.contextvars.bind_contextvars( + correlation_id=correlation_id, + path=scope.get("path", ""), + method=scope.get("method", ""), + ) + + async def send_with_header(message): + if message["type"] == "http.response.start": + headers = list(message.get("headers", [])) + headers.append( + (b"x-correlation-id", correlation_id.encode()) + ) + message = {**message, "headers": headers} + await send(message) + + await self.app(scope, receive, send_with_header) + duration_ms = (time.perf_counter_ns() - start_ns) / 1_000_000 + structlog.contextvars.bind_contextvars(duration_ms=round(duration_ms, 2)) +``` + +**Bind user_id after auth in the route handler:** +```python +# In any route that uses get_regular_user: +@router.get("/api/documents/") +async def list_documents(current_user: User = Depends(get_regular_user), ...): + structlog.contextvars.bind_contextvars(user_id=str(current_user.id)) + ... +``` + +### Pattern 3: Updated get_client_ip with Trusted-Proxy Logic (deps/utils.py) + +**What:** Replace the existing `get_client_ip` body with trusted-proxy CIDR validation per D-11. The current implementation reads `X-Forwarded-For` unconditionally — a bypass vector for IP-based rate limiting. + +**Critical:** This function already exists at `backend/deps/utils.py` line 10. The D-11 plan task is to REPLACE its body, not create a new function. + +```python +# Source: D-11 decision in CONTEXT.md +import ipaddress +from typing import Optional +from fastapi import Request + +# Trusted proxy CIDRs — requests arriving from these addresses may set X-Forwarded-For +_TRUSTED_PROXY_NETS = [ + ipaddress.ip_network("127.0.0.0/8"), + ipaddress.ip_network("172.16.0.0/12"), + ipaddress.ip_network("192.168.0.0/16"), + ipaddress.ip_network("::1/128"), +] + +def _is_trusted_proxy(host: str) -> bool: + try: + addr = ipaddress.ip_address(host) + return any(addr in net for net in _TRUSTED_PROXY_NETS) + except ValueError: + return False + +def get_client_ip(request: Request) -> Optional[str]: + """Extract client IP with trusted-proxy CIDR check (D-11). + + If the direct peer (request.client.host) is a trusted proxy, read the + leftmost address from X-Forwarded-For. Otherwise ignore forwarded headers + and return the direct peer IP — prevents header spoofing from external clients. + """ + direct_peer = request.client.host if request.client else None + if direct_peer and _is_trusted_proxy(direct_peer): + xff = request.headers.get("X-Forwarded-For") + if xff: + return xff.split(",")[0].strip() + return direct_peer +``` + +### Pattern 4: Per-Account Rate Limiter (slowapi two-limiter pattern) + +**What:** A second `Limiter` instance keyed by `current_user.id` used on document/cloud endpoints. The key_func receives a `Request` object but reads the user from a request-state attribute set by the route dependency. + +**Slowapi constraint:** `key_func` receives only a `Request` — it cannot be an async function and cannot call FastAPI dependencies directly. The pattern is to attach the user to `request.state` inside the route (or via a dep) and read it in the key_func. + +```python +# Source: slowapi docs + D-12 decision +# In main.py or a shared module: +from slowapi import Limiter +from fastapi import Request + +def _account_key(request: Request) -> str: + """Return current user's ID as rate limit key. + + Caller MUST have attached current_user to request.state before this key + function is evaluated (done automatically by the @account_limiter.limit + decorator executing after the Depends chain). + """ + user = getattr(request.state, "current_user", None) + if user is None: + # Unauthenticated — fall back to IP (should not happen on guarded routes) + return request.client.host if request.client else "anonymous" + return str(user.id) + +account_limiter = Limiter(key_func=_account_key) +``` + +```python +# In documents.py — inject current_user via Depends, then apply per-account limit: +@router.get("/api/documents/") +@account_limiter.limit("100/minute") +async def list_documents( + request: Request, + current_user: User = Depends(get_regular_user), + db: AsyncSession = Depends(get_db), +): + request.state.current_user = current_user # expose to key_func + ... +``` + +**Note:** `request.state.current_user` must be set BEFORE `account_limiter.limit()` evaluates the key. In slowapi, the limit check runs at the start of the handler body — so setting `request.state.current_user` as the first line of the handler, after Depends resolution, works correctly. [ASSUMED — based on slowapi execution model; verify in Wave 0 test] + +**Wiring in main.py:** The existing `app.state.limiter = auth_limiter` pattern applies to SlowAPIMiddleware's automatic rate limit enforcement. The `account_limiter` is a second, independent Limiter instance. Both need their state wired: +```python +app.state.limiter = auth_limiter # existing — drives SlowAPIMiddleware +# account_limiter decorators work independently, no app.state wiring needed +``` + +### Pattern 5: Multi-stage Dockerfile with appuser + +**What:** Two-stage build. Builder stage installs all system packages and Python deps as root. Runtime stage copies only the installed packages; creates appuser uid=1000, drops to that user. + +**Verified insight on extractor.py:** `services/extractor.py` line 18 uses `tempfile.NamedTemporaryFile()` which writes to `/tmp`. With `read_only: true` and `tmpfs: ["/tmp"]` in docker-compose.yml, this works if the tmpfs is writable by appuser. See Pitfall 3 for the tmpfs ownership fix. + +```dockerfile +# Source: D-07/D-08/D-09 decisions + Docker best practices +# ── Stage 1: builder ────────────────────────────────────────────────────────── +FROM python:3.12-slim AS builder + +WORKDIR /build + +RUN apt-get update && apt-get install -y --no-install-recommends \ + tesseract-ocr \ + libgl1 \ + libglib2.0-0 \ + gcc \ + && rm -rf /var/lib/apt/lists/* + +COPY requirements.txt . +RUN pip install --no-cache-dir --prefix=/install -r requirements.txt + +# ── Stage 2: runtime ───────────────────────────────────────────────────────── +FROM python:3.12-slim AS runtime + +# Runtime system deps (required at runtime, not just build time) +RUN apt-get update && apt-get install -y --no-install-recommends \ + tesseract-ocr \ + libgl1 \ + libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* + +# Copy installed Python packages from builder +COPY --from=builder /install /usr/local + +# Create non-root user +RUN groupadd --gid 1000 appgroup && \ + useradd --uid 1000 --gid appgroup --shell /bin/sh --no-create-home appuser + +WORKDIR /app +COPY --chown=appuser:appgroup . . + +USER appuser +EXPOSE 8000 +CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +**Note:** `--prefix=/install` installs packages to `/install` which is then copied to `/usr/local` in the runtime stage. Alternative: use `pip install --target` or just do a two-stage where the full pip install is in the builder and the runtime stage installs again — but the prefix approach avoids the second network call. [ASSUMED — verify the exact pip prefix copy path in Wave 0] + +### Pattern 6: Docker Compose hardening additions + +```yaml +# Source: D-08, D-09 decisions + https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/ +services: + backend: + # ... existing config ... + read_only: true + tmpfs: + - /tmp:mode=1777 # world-writable; appuser can write; OR use entrypoint chown + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + + celery-worker: + # ... existing config ... + read_only: true + tmpfs: + - /tmp:mode=1777 + cap_drop: + - ALL + security_opt: + - no-new-privileges:true +``` + +**Loki/Promtail/Grafana additions to docker-compose.yml:** +```yaml + loki: + image: grafana/loki:latest + ports: + - "3100:3100" + volumes: + - ./docker/loki/loki-config.yaml:/etc/loki/local-config.yaml + - loki_data:/loki + command: -config.file=/etc/loki/local-config.yaml + networks: + - default + + promtail: + image: grafana/promtail:latest + volumes: + - ./docker/loki/promtail-config.yaml:/etc/promtail/config.yaml + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - /var/run/docker.sock:/var/run/docker.sock + command: -config.file=/etc/promtail/config.yaml + depends_on: + - loki + + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + environment: + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin + volumes: + - grafana_data:/var/lib/grafana + depends_on: + - loki + +volumes: + loki_data: + grafana_data: +``` + +**Promtail config (docker/loki/promtail-config.yaml) — uses docker_sd_configs:** +Each backend service in docker-compose.yml needs the label `logging: "promtail"` added. +```yaml +# Source: https://ornlu-is.github.io/docker_compose_promtail_loki_grafana/ +server: + http_listen_port: 9080 + grpc_listen_port: 0 +positions: + filename: /tmp/positions.yaml +clients: + - url: http://loki:3100/loki/api/v1/push +scrape_configs: + - job_name: docker + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 5s + filters: + - name: label + values: ["logging=promtail"] + relabel_configs: + - source_labels: ["__meta_docker_container_name"] + regex: "/(.*)" + target_label: "container" + - source_labels: ["__meta_docker_container_label_com_docker_compose_service"] + target_label: "service" +``` + +**Loki config (docker/loki/loki-config.yaml) — single-binary filesystem mode:** +```yaml +# Source: https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose +auth_enabled: false +server: + http_listen_port: 3100 + grpc_listen_port: 9096 +common: + instance_addr: 127.0.0.1 + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory +schema_config: + configs: + - from: 2020-10-24 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h +query_range: + results_cache: + cache: + embedded_cache: + enabled: true + max_size_mb: 100 +``` + +### Pattern 7: Locust JWT Session + +**What:** Locust `HttpUser` with `on_start()` for JWT login; stores token; uses it in all subsequent requests. Task weights simulate a realistic session. + +```python +# Source: https://docs.locust.io/en/stable/writing-a-locustfile.html +# File: backend/load_tests/locustfile.py +from locust import HttpUser, task, between, events +import json + +TEST_EMAIL = "loadtest@example.com" +TEST_PASSWORD = "Loadtest123!" + +class DocuVaultUser(HttpUser): + wait_time = between(0.5, 2.0) + access_token: str = "" + + def on_start(self): + """Login and obtain JWT access token.""" + resp = self.client.post( + "/api/auth/login", + json={"email": TEST_EMAIL, "password": TEST_PASSWORD}, + ) + if resp.status_code == 200: + self.access_token = resp.json().get("access_token", "") + else: + self.environment.runner.quit() + + def _auth_headers(self): + return {"Authorization": f"Bearer {self.access_token}"} + + @task(5) # Most common: list documents + def list_documents(self): + self.client.get("/api/documents/", headers=self._auth_headers()) + + @task(2) # Upload (moderate frequency) + def upload_document(self): + from io import BytesIO + data = b"%PDF-1.4 fake pdf content for load testing" + self.client.post( + "/api/documents/", + files={"file": ("test.pdf", BytesIO(data), "application/pdf")}, + headers=self._auth_headers(), + ) + + @task(1) # Least common: refresh token + def refresh_token(self): + self.client.post("/api/auth/refresh") + +# SLA gate: exit code 1 if p95 > 200ms or p99 > 500ms +@events.quitting.add_listener +def check_sla(environment, **kwargs): + stats = environment.runner.stats.total + if stats.fail_ratio > 0.01: + environment.process_exit_code = 1 + elif stats.get_response_time_percentile(0.95) > 200: + environment.process_exit_code = 1 + elif stats.get_response_time_percentile(0.99) > 500: + environment.process_exit_code = 1 +``` + +**Run command:** +```bash +locust --headless --users 50 --spawn-rate 10 --run-time 5m \ + --host http://localhost:8000 \ + --csv backend/load_tests/results \ + -f backend/load_tests/locustfile.py +``` + +**Note on upload endpoint:** D-05 specifies simulating an upload. The existing upload flow uses presigned URLs (two-step: `POST /api/documents/upload-url` → XHR PUT to MinIO → `POST /api/documents/{id}/confirm`). The locustfile must implement all three steps. Locust's `HttpUser` can PUT directly to MinIO's public endpoint. [ASSUMED on two-step flow — confirm against current documents.py before finalizing] + +### Anti-Patterns to Avoid + +- **Defining a second `get_client_ip` anywhere:** CLAUDE.md mandates the function lives only in `deps/utils.py`. Replace the body in-place. +- **Adding `get_regular_user` import to locustfile.py:** Locust scripts must have zero imports from the application code to stay independent and avoid circular import issues. +- **Installing locust in the production Docker image:** Load tests run externally. Adding locust to `requirements.txt` bloats the image with gevent and other heavy deps. +- **Using `BaseHTTPMiddleware` for CorrelationIDMiddleware:** Starlette's `BaseHTTPMiddleware` buffers streaming responses. Use raw ASGI middleware (as shown in Pattern 2) for the logging middleware. +- **Binding user_id in middleware before auth:** Middleware runs before route handlers; `current_user` is not available in middleware. Bind user_id in the route handler body. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Structured JSON log formatting | Custom JSON serializer | structlog JSONRenderer via ProcessorFormatter | Exception traces, timestamps, ISO formatting already handled | +| Log context propagation across async boundaries | threading.local or passing context as params | structlog.contextvars | Native asyncio-safe; zero changes to function signatures | +| Stdlib bridge (uvicorn/sqlalchemy logs) | Custom logging.Handler | structlog ProcessorFormatter with foreign_pre_chain | One formatter handles both structlog and stdlib — uniform output | +| HTTP load testing with auth sessions | Custom requests/httpx script | Locust HttpUser with on_start() | Built-in distributed mode, stats, SLA reporting, CSV export | +| Docker CVE scanning | Custom apt-based scanner | `docker scout cves` | Already available (v1.20.4 on host); --exit-code flag for gate scripting | +| Trusted-proxy IP extraction | New function | Update existing `get_client_ip` in `deps/utils.py` | CLAUDE.md mandates single canonical location; function already imported by all routers | + +**Key insight:** Every domain in Phase 6 has a well-established off-the-shelf solution. The risk of custom implementations is stale logic (e.g., a homegrown trusted-proxy parser missing IPv6 or CIDR edge cases). + +--- + +## Common Pitfalls + +### Pitfall 1: tmpfs owned by root when combined with non-root appuser + +**What goes wrong:** With `read_only: true` and `tmpfs: ["/tmp"]` in docker-compose.yml, the tmpfs mount is created as `root:root 755`. appuser (uid=1000) cannot write to it. `tempfile.NamedTemporaryFile()` in `services/extractor.py` will raise `PermissionError`. + +**Why it happens:** Docker creates tmpfs mounts before the USER instruction takes effect — they are always initially owned by root. + +**How to avoid:** Two options: +1. `tmpfs: - /tmp:mode=1777` — world-writable sticky bit; quick and standard for /tmp. +2. An entrypoint script that runs `chown -R appuser /tmp` before `exec su-exec appuser uvicorn ...` — more secure but adds complexity. + +Option 1 (`mode=1777`) matches the Linux convention for `/tmp` and is recommended for this use case. [VERIFIED: https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/] + +### Pitfall 2: structlog.contextvars context NOT cleared between requests + +**What goes wrong:** Without `structlog.contextvars.clear_contextvars()` at the start of each request, context variables from previous requests can bleed into subsequent requests handled on the same worker. In async apps, this causes user_id and correlation_id cross-contamination. + +**Why it happens:** contextvars in asyncio are scoped to a task/coroutine chain, but Starlette may reuse coroutine contexts across requests under some middleware configurations. + +**How to avoid:** Always call `clear_contextvars()` as the FIRST operation in the CorrelationIDMiddleware `__call__` method, before binding any new values. [VERIFIED: https://www.structlog.org/en/stable/contextvars.html] + +### Pitfall 3: slowapi account_limiter key_func called before current_user is on request.state + +**What goes wrong:** If `request.state.current_user` is not set before the rate limit check fires, the key_func falls back to IP — making the per-account limit effectively a per-IP limit, defeating D-12. + +**Why it happens:** slowapi evaluates the key_func inside the decorator's wrapper before calling the handler body. FastAPI's `Depends(get_regular_user)` injects the user object, but slowapi's key_func runs in a different execution context. + +**How to avoid:** The safest pattern is to set `request.state.current_user = current_user` as the first statement in the route handler body after the dependency injection. This ensures the key_func can access it on the next request for the same handler invocation. [ASSUMED — verify with a unit test in Wave 0 before deploying the decorator to all endpoints] + +### Pitfall 4: Locust load test user not pre-created in DB + +**What goes wrong:** Running the locustfile against a fresh database with no `loadtest@example.com` user causes all logins to fail immediately — the load test reports 100% failures but this is a test setup issue, not an SLA failure. + +**Why it happens:** Locust doesn't know about the app's user model. + +**How to avoid:** Add a Wave 0 task to create the load test user via `POST /api/auth/register` (or direct DB insert via alembic seed) before running the locust soak test. Document this in the RUNBOOK.md section on load testing. + +### Pitfall 5: `docker scout cves` requires authenticated Docker Hub session on first run + +**What goes wrong:** `docker scout cves local://docuvault-backend` returns an auth error if Docker Hub is not logged in. The security gate fails on a clean machine. + +**Why it happens:** docker scout sends the image manifest to Docker's Scout service for analysis — it requires a Docker Hub account even for local images. + +**How to avoid:** Document `docker login` as a prerequisite in the security gate instructions. The gate command is: +```bash +docker scout cves local://docuvault-backend:latest \ + --only-severity critical \ + --exit-code +# Exit code 2 = critical CVEs found; 0 = clean +``` +Provide `trivy image docuvault-backend:latest` as a fallback (trivy is offline-capable but not currently installed). [VERIFIED: https://docs.docker.com/reference/cli/docker/scout/cves/] + +### Pitfall 6: PyMuPDF writes tessdata cache to /tmp under non-root + +**What goes wrong:** PyMuPDF (via MuPDF) may attempt to write cache files to locations owned by root that are not covered by the tmpfs mount. + +**Why it happens:** MuPDF's font/tessdata resolution uses paths baked in at compile time; on some builds this includes `/var/cache/fontconfig` or `/root/.config/mupdf`. + +**How to avoid:** Test `read_only: true` with the actual container: run `docker compose up backend` with a simple document extraction request and watch for `PermissionError` or `Read-only file system` errors. If found, add additional tmpfs mounts or use `--mount type=tmpfs,dst=/var/cache/fontconfig`. [ASSUMED — needs runtime verification in Wave 0] + +### Pitfall 7: Celery worker `celerybeat-schedule` file on read-only filesystem + +**What goes wrong:** `celery-beat` writes a `celerybeat-schedule` file (pidfile + schedule state) to its working directory. With `read_only: true`, this fails immediately. + +**Why it happens:** Celery beat uses a local SQLite-like schedule file by default. + +**How to avoid:** Either (a) do NOT apply `read_only: true` to `celery-beat` (it is not a network-facing service so the risk is lower), or (b) add `tmpfs: ["/app"]` to celery-beat but that would override the app volume. Best approach: configure celery-beat to use `--schedule /tmp/celerybeat-schedule` in its command argument. Note: D-08 says `read_only` applies to "FastAPI and Celery worker services" — this is the Celery *worker*, not celery-beat. Clarify scope with the planner. + +--- + +## Code Examples + +### docker scout cves gate command + +```bash +# Source: https://docs.docker.com/reference/cli/docker/scout/cves/ +# Build image first, then scan: +docker build -t docuvault-backend:latest ./backend + +# Scan for critical CVEs — exit code 2 if any found, 0 if clean +docker scout cves local://docuvault-backend:latest \ + --only-severity critical \ + --exit-code + +# For security gate: fail on critical OR high +docker scout cves local://docuvault-backend:latest \ + --only-severity critical,high \ + --exit-code +``` + +### structlog logger usage in route handlers + +```python +# Source: https://www.structlog.org/en/stable/getting-started.html +import structlog +log = structlog.get_logger() + +# In a route handler — user_id already bound by CorrelationIDMiddleware +async def upload_document(...): + structlog.contextvars.bind_contextvars(user_id=str(current_user.id)) + log.info("document.upload.started", filename=file.filename, size_bytes=file.size) + # ... processing ... + log.info("document.upload.complete", document_id=str(doc.id)) +``` + +### Locust headless run with CSV stats + +```bash +# Source: https://docs.locust.io/en/stable/running-without-web-ui.html +locust \ + --headless \ + --users 50 \ + --spawn-rate 10 \ + --run-time 5m \ + --host http://localhost:8000 \ + --csv backend/load_tests/results \ + -f backend/load_tests/locustfile.py + +# CSV output: +# backend/load_tests/results_stats.csv — per-endpoint percentiles +# backend/load_tests/results_failures.csv — failure details +# backend/load_tests/results_stats_history.csv +``` + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| logging.basicConfig JSON handler | structlog with ProcessorFormatter bridge | structlog 21+ | Unified stdlib + structlog pipeline; single config | +| Promtail as only log agent | Grafana Alloy (new default) | 2024 (Grafana Alloy GA) | Alloy is now recommended; Promtail still supported and simpler for local dev | +| docker-compose tmpfs string syntax | Long-form with mode/uid options | Docker Compose v3.6+ | Enables permission control on tmpfs | +| Locust 1.x `TaskSet` pattern | Locust 2.x `@task` decorators on `HttpUser` | 2021 (Locust 2.0) | TaskSet deprecated; @task on HttpUser is current | +| Loki schema v11/v12 | schema v13 (tsdb store) | Loki 2.8+ | v13 is now required for new deployments; v12 still works | + +**Deprecated/outdated:** +- `structlog.threadlocal` module: replaced by `structlog.contextvars` for async apps — do not use. +- `locust.TaskSet` class: still exists but deprecated; use `@task` on `HttpUser` directly. +- Loki `boltdb-shipper` store: replaced by `tsdb` in schema v13. + +--- + +## Assumptions Log + +| # | Claim | Section | Risk if Wrong | +|---|-------|---------|---------------| +| A1 | `request.state.current_user` set before slowapi evaluates key_func when set as first line of handler body | Pattern 4 (per-account limiter) | Per-account limit silently falls back to IP limit — D-12 not met | +| A2 | Locust upload flow targets `POST /api/documents/` directly (single-step) vs. the presigned URL two-step flow | Pattern 7 (Locust) | Load test fails 100% if two-step flow is required | +| A3 | `pip install --prefix=/install` followed by `COPY --from=builder /install /usr/local` correctly installs all packages into Python's site-packages in the runtime stage | Pattern 5 (Dockerfile) | Runtime stage missing packages → import errors at startup | +| A4 | `celery-beat` is NOT subject to `read_only: true` per D-08 (which says "FastAPI and Celery worker") | Pitfall 7 | celery-beat fails to write schedule file → scheduled tasks stop | +| A5 | PyMuPDF does not write to paths outside `/tmp` that would be blocked by `read_only: true` | Pitfall 6 | Extraction fails silently with PermissionError | + +--- + +## Open Questions + +1. **Upload endpoint for load testing** + - What we know: `services/extractor.py` uses `tempfile.NamedTemporaryFile`; the upload flow may be two-step (presigned URL) or one-step depending on the current documents.py implementation + - What's unclear: Whether Locust needs to implement the three-step presigned flow (upload-url → PUT to MinIO → confirm) or if a simpler direct POST exists + - Recommendation: Read `backend/api/documents.py` lines for the upload endpoint before finalizing the locustfile; if two-step, implement the MinIO PUT step using `self.client.put` to MinIO's host:9000 + +2. **Load test user bootstrap** + - What we know: Locust needs a valid user to authenticate; no seed user for load testing exists yet + - What's unclear: Whether to create the user via the registration API (on_start) or pre-seed via Alembic + - Recommendation: Use `on_start` to register if not exists (catch 409), then login — self-contained, no DB dependency + +3. **Loki tmpfs mount persistence** + - What we know: Loki service needs a writable `/loki` directory for chunk storage + - What's unclear: Whether the named volume `loki_data:/loki` is sufficient or if Loki's own container permissions require a matching UID + - Recommendation: Use named volume (not read_only on loki service — it's not a user-facing service) + +--- + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| Docker | Container hardening, docker scout, Loki stack | ✓ | 29.5.2 | — | +| docker scout | CVE scanning gate | ✓ | v1.20.4 | trivy (not installed — document install) | +| trivy | CVE scanning fallback | ✗ | — | `brew install trivy` or `apt-get install trivy` | +| Python 3.12 (host) | Locust runs outside container | ✓ | 3.12 (via backend venv) | — | +| structlog | structlog logging | needs install | 25.5.0 on PyPI | — | +| locust | load tests | needs install | 2.34.0 on PyPI | — | +| Loki image | log aggregation | pull on first compose up | grafana/loki:latest | — | +| Promtail image | log collection | pull on first compose up | grafana/promtail:latest | — | +| Grafana image | log visualization | pull on first compose up | grafana/grafana:latest | — | + +**Missing dependencies with no fallback:** +- None that block core implementation. + +**Missing dependencies with fallback:** +- trivy (CVE scanning): fallback is docker scout (already available). Document trivy as optional alternative. +- docker login session: required for `docker scout cves` against local images. Planner must add a setup step. + +--- + +## Validation Architecture + +### Test Framework + +| Property | Value | +|----------|-------| +| Framework | pytest 8.2 + pytest-asyncio (asyncio_mode=auto) | +| Config file | `backend/pytest.ini` | +| Quick run command | `cd backend && pytest tests/ -v -x` | +| Full suite command | `cd backend && pytest tests/ -v` | + +### Phase Requirements → Test Map + +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| D-01 | structlog emits JSON with correlation_id field | Unit | `pytest tests/test_logging.py -x` | ❌ Wave 0 | +| D-11 | get_client_ip returns direct IP when peer is untrusted | Unit | `pytest tests/test_rate_limiting.py::test_get_client_ip_untrusted -x` | ❌ Wave 0 | +| D-11 | get_client_ip reads XFF when peer is trusted proxy | Unit | `pytest tests/test_rate_limiting.py::test_get_client_ip_trusted_proxy -x` | ❌ Wave 0 | +| D-12 | per-account limiter key is user.id not IP | Unit | `pytest tests/test_rate_limiting.py::test_account_limiter_key -x` | ❌ Wave 0 | +| D-12 | authenticated endpoint returns 429 after 100 req/min | Integration | `pytest tests/test_rate_limiting.py::test_account_rate_limit -x` | ❌ Wave 0 | +| D-07 | Docker image runs as uid=1000, not root | Manual/smoke | `docker run --rm docuvault-backend:latest id` | N/A manual | +| D-08 | read_only container can write to /tmp | Manual/smoke | `docker compose up backend` + upload a doc | N/A manual | +| D-06 | SLA: p95 < 200ms, p99 < 500ms at 50 users | Load test (Locust) | `locust --headless ... -f backend/load_tests/locustfile.py` | ❌ Wave 0 | + +### Sampling Rate + +- **Per task commit:** `cd backend && pytest tests/ -v -x --tb=short` +- **Per wave merge:** `cd backend && pytest tests/ -v` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps + +- [ ] `backend/tests/test_logging.py` — structlog config tests (D-01) +- [ ] `backend/tests/test_rate_limiting.py` — get_client_ip unit tests + per-account limiter tests (D-11, D-12) +- [ ] `backend/load_tests/__init__.py` — empty, marks directory as non-pytest-discoverable +- [ ] `backend/load_tests/locustfile.py` — Locust user class (D-04..D-06) + +--- + +## Security Domain + +### Applicable ASVS Categories + +| ASVS Category | Applies | Standard Control | +|---------------|---------|-----------------| +| V2 Authentication | no (already implemented in Phase 2) | — | +| V3 Session Management | no (already implemented) | — | +| V4 Access Control | yes — rate limit bypass prevention | slowapi trusted-proxy key_func (D-11/D-12) | +| V5 Input Validation | no new user inputs in this phase | — | +| V6 Cryptography | no | — | +| V14 Configuration | yes — container hardening, read-only fs | D-07..D-10 | + +### Known Threat Patterns for This Phase + +| Pattern | STRIDE | Standard Mitigation | +|---------|--------|---------------------| +| Rate limit bypass via X-Forwarded-For header spoofing | Tampering | Trusted-proxy CIDR check in get_client_ip (D-11) | +| Container escape via write to host filesystem | Elevation of Privilege | read_only: true + cap_drop: ALL (D-08, D-09) | +| CVE exploitation via outdated base image packages | Tampering | docker scout cves zero-critical gate (D-10) | +| Log injection via user-controlled strings in log fields | Tampering | structlog JSON renderer escapes all values — JSON encoding prevents log injection | +| Locust test user credentials in version control | Information Disclosure | Use env vars for load test credentials; add to .gitignore | + +--- + +## Sources + +### Primary (HIGH confidence) +- [structlog official docs — contextvars](https://www.structlog.org/en/stable/contextvars.html) — middleware pattern, clear_contextvars(), bind_contextvars() +- [structlog official docs — stdlib integration](https://www.structlog.org/en/stable/standard-library.html) — ProcessorFormatter, foreign_pre_chain +- [structlog official docs — getting started](https://www.structlog.org/en/stable/getting-started.html) — configure() API, JSON/console renderer +- [Locust official docs — writing a locustfile](https://docs.locust.io/en/stable/writing-a-locustfile.html) — HttpUser, on_start, @task weights +- [Locust official docs — headless mode](https://docs.locust.io/en/stable/running-without-web-ui.html) — --headless, --users, --spawn-rate, --run-time, exit codes +- [Locust official docs — configuration](https://docs.locust.io/en/stable/configuration.html) — --csv, --host flags +- [docker scout cves official docs](https://docs.docker.com/reference/cli/docker/scout/cves/) — --exit-code, --only-severity, exit code behavior +- PyPI registry — structlog 25.5.0, locust 2.34.0, slowapi 0.1.9 confirmed current + +### Secondary (MEDIUM confidence) +- [wazaari.dev FastAPI + structlog integration](https://wazaari.dev/blog/fastapi-structlog-integration) — full processor chain + middleware pattern verified against structlog official docs +- [Promtail docker_sd_configs pattern](https://ornlu-is.github.io/docker_compose_promtail_loki_grafana/) — container log scraping config verified against Promtail docs +- [Loki single-binary config](https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose-bcb07d1a51aa) — loki-config.yaml single-node filesystem mode +- [Docker tmpfs non-root pattern](https://www.tutorialpedia.org/blog/docker-compose-mounting-a-tmpfs-usable-by-non-root-user/) — mode=1777 solution for non-root tmpfs access + +### Tertiary (LOW confidence) +- [slowapi GitHub](https://github.com/laurentS/slowapi) — per-account key_func pattern inferred from API reference; no official example for request.state pattern + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — structlog 25.5.0 and locust 2.34.0 verified on PyPI; slowapi 0.1.9 already installed +- Architecture (structlog): HIGH — verified against official structlog docs with working code examples +- Architecture (Loki/Promtail): MEDIUM — working config patterns from community sources cross-checked with official Loki install docs +- Architecture (container hardening): MEDIUM — Docker best practices well-documented; tmpfs non-root pattern verified against official compose format +- Architecture (slowapi per-account): LOW-MEDIUM — core pattern is sound; key_func evaluation order vs. Depends resolution is an assumption that needs Wave 0 test +- Locust load testing: HIGH — official docs used for all flags and patterns +- docker scout CVE scanning: HIGH — official Docker docs used + +**Research date:** 2026-06-02 +**Valid until:** 2026-07-02 (structlog and locust are stable libraries; Loki config may shift with new releases)