kite/RUNBOOK.md

DocuVault Operational Runbook

This is the single reference for running DocuVault in production-like environments. It covers environment variables, startup/shutdown procedures, backup strategy, health checks, on-call escalation, and common failure-mode recovery. For architecture and rationale see CLAUDE.md; for phase history and roadmap see .planning/ROADMAP.md.

Last updated: 2026-06-04

---

Table of Contents

1. Environment Variables
2. Startup / Shutdown
3. Backup Strategy
4. Health Checks
5. Security Gate — docker scout CVE Scan (D-10)
6. On-Call Escalation
7. Common Failure Modes
8. Phase 6 Deferred Items

---

1. Environment Variables

All variables are read from a .env file at the repo root (or injected by the container orchestrator). Copy .env.example to .env and populate every field before starting services.

Security note: SECRET_KEY, CLOUD_CREDS_KEY, and all *_PASSWORD/*_SECRET fields must never be committed to version control. Rotate them if exposure is suspected.

PostgreSQL

Name	Required	Description	Example	Source
DATABASE_URL	Yes	Application DSN (docuvault_app role — DML only).	postgresql+psycopg://docuvault_app:****@postgres:5432/docuvault	.env
DATABASE_MIGRATE_URL	Yes	Migration DSN (docuvault_migrate role — DDL). Used by Alembic only.	postgresql+psycopg://docuvault_migrate:****@postgres:5432/docuvault	.env
POSTGRES_PASSWORD	Yes	Root postgres superuser password (used by the docker-compose postgres service).	change-me-postgres	.env

MinIO

Name	Required	Description	Example	Source
MINIO_ENDPOINT	Yes	Internal Docker hostname:port for MinIO SDK calls.	minio:9000	.env
MINIO_ACCESS_KEY	Yes	MinIO application access key.	docuvault_app	.env
MINIO_SECRET_KEY	Yes	MinIO application secret key.	****	.env
MINIO_BUCKET	Yes	Primary document storage bucket name.	docuvault	.env
MINIO_PUBLIC_ENDPOINT	No	Browser-visible hostname for presigned upload/download URLs. Empty → falls back to MINIO_ENDPOINT.	localhost:9000	.env
MINIO_ROOT_USER	Yes	MinIO root/admin username (used by the minio compose service).	minioadmin	.env
MINIO_ROOT_PASSWORD	Yes	MinIO root/admin password.	****	.env

Redis / Celery

Name	Required	Description	Example	Source
REDIS_URL	Yes	Redis connection string including password.	redis://:****@redis:6379/0	.env
REDIS_PASSWORD	Yes	Redis requirepass password (used by the redis compose service).	****	.env

Security / Auth

Name	Required	Description	Example	Source
SECRET_KEY	Yes	HMAC/JWT signing secret. Must be a high-entropy random string ≥32 bytes. Rotate to invalidate all active sessions.	openssl rand -hex 32 output	.env
CLOUD_CREDS_KEY	Yes	Master key for HKDF per-user cloud credential encryption. Must be exactly 32 bytes.	openssl rand -hex 16 (pad to 32)	.env
ADMIN_EMAIL	Yes	Bootstrap admin account email. Created on first startup if absent.	admin@example.com	.env
ADMIN_PASSWORD	Yes	Bootstrap admin account password. Must meet AUTH-01 strength requirements.	Ch@ngeMe1!	.env
ACCESS_TOKEN_EXPIRE_MINUTES	No	JWT access token lifetime in minutes. Default: 15. Do not increase beyond 30.	15	.env
REFRESH_TOKEN_EXPIRE_DAYS	No	Refresh token cookie lifetime in days. Default: 30.	30	.env

Rotation procedure for SECRET_KEY:

1. Generate a new value: openssl rand -hex 32
2. Update .env and restart the backend service.
3. All active JWT sessions are immediately invalidated — users must log in again.

Rotation procedure for CLOUD_CREDS_KEY:

Requires re-encrypting all stored cloud credentials. Until a migration tool exists, this rotation is a breaking change — treat the key as long-lived and protect it accordingly.

CORS / Frontend

Name	Required	Description	Example	Source
CORS_ORIGINS	No	Comma-separated list of allowed CORS origins. Default: http://localhost:5173.	https://app.example.com	.env
FRONTEND_URL	No	Base URL of the Vue frontend — used in password-reset emails. Default: http://localhost:5173.	https://app.example.com	.env
BACKEND_URL	No	Externally reachable backend URL — used to construct OAuth callback URLs. Default: http://localhost:8000.	https://api.example.com	.env

SMTP (password reset emails)

Name	Required	Description	Example	Source
SMTP_HOST	No	SMTP relay hostname. Leave empty to disable email.	smtp.sendgrid.net	.env
SMTP_PORT	No	SMTP port. Default: 587 (STARTTLS).	587	.env
SMTP_USER	No	SMTP login username.	apikey	.env
SMTP_PASSWORD	No	SMTP login password / API key.	****	.env
SMTP_FROM	No	Sender address for outgoing emails. Default: noreply@docuvault.local.	noreply@example.com	.env

AI Classification

Name	Required	Description	Example	Source
DEFAULT_AI_PROVIDER	No	Default LLM provider. Default: ollama. Options: ollama, openai, anthropic.	openai	.env
DEFAULT_AI_MODEL	No	Default model name for the chosen provider. Default: llama3.2.	gpt-4o-mini	.env
SYSTEM_PROMPT	No	Custom system prompt override for the classifier. Hardcoded fallback lives in backend/ai/classifier.py.	Classify documents concisely.	.env

Cloud Storage Providers (Phase 5)

Name	Required	Description	Example	Source
GOOGLE_CLIENT_ID	No	Google OAuth 2.0 client ID for Drive integration.	1234567890-abc.apps.googleusercontent.com	Google Cloud Console
GOOGLE_CLIENT_SECRET	No	Google OAuth 2.0 client secret.	GOCSPX-****	Google Cloud Console
ONEDRIVE_CLIENT_ID	No	Azure AD application (client) ID for OneDrive.	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx	Azure Portal
ONEDRIVE_CLIENT_SECRET	No	Azure AD client secret value.	****	Azure Portal
ONEDRIVE_TENANT_ID	No	Azure AD tenant. Default: common (personal + org accounts).	common	.env

Observability (Phase 6)

Name	Required	Description	Example	Source
LOG_LEVEL	No	structlog minimum level. Default: INFO. Options: DEBUG, INFO, WARNING, ERROR.	INFO	.env
LOG_JSON	No	Emit structured JSON log lines instead of plain text. Default: false. Set true in production.	true	.env

Load Testing (Phase 6 — dev only)

Name	Required	Description	Example	Source
LOAD_TEST_EMAIL	No	Email of the pre-created load-test user. Default: loadtest@example.com.	loadtest@staging.example.com	.env (dev only)
LOAD_TEST_PASSWORD	No	Password for the load-test user. Default: Loadtest123!@#.	MyS3cure!Pass	.env (dev only)

---

2. Startup / Shutdown

Full stack startup

# Build images and start all services in the background
docker compose up -d --build

# Tail logs for the first 60 s to confirm everything came up cleanly
docker compose logs -f --tail 50

Services start in dependency order:

postgres / minio / redis  →  backend / celery-worker / celery-beat
                          →  loki  →  promtail / grafana
                          →  frontend

Access points after a healthy startup:

Service	URL
Vue frontend	http://localhost:5173
FastAPI backend (health)	http://localhost:8000/health
MinIO console	http://localhost:9001
Grafana (log UI)	http://localhost:3000
Loki (API/ready)	http://localhost:3100/ready

Development mode (live reload)

The docker-compose.yml already mounts ./backend:/app with --reload, so Python file changes trigger automatic reload. To force a rebuild after changing requirements.txt or the Dockerfile:

docker compose build backend celery-worker celery-beat
docker compose up -d backend celery-worker celery-beat

Orderly shutdown

# Stop all containers, preserve volumes (normal dev shutdown)
docker compose down

# Stop and remove named volumes (WARNING: deletes all data)
docker compose down -v

Warning: docker compose down -v deletes postgres_data, minio_data, loki_data, and grafana_data. Only use this for a full environment reset.

celery-beat note (Pitfall 7)

The celery-beat service does not have read_only: true in docker-compose.yml because it writes the celerybeat-schedule file to its working directory at startup. Do not add read_only: to the celery-beat block — it will exit immediately with Permission denied: 'celerybeat-schedule'.

---

3. Backup Strategy

PostgreSQL

Manual backup:

docker compose exec -T postgres pg_dump -U postgres docuvault \
  > backups/docuvault_$(date +%F).sql

Restore:

cat backups/docuvault_YYYY-MM-DD.sql \
  | docker compose exec -T postgres psql -U postgres docuvault

Cron pattern (daily at 03:00, keep 7 days):

# Add to crontab (crontab -e)
0 3 * * * cd /path/to/docuvault && \
  docker compose exec -T postgres pg_dump -U postgres docuvault \
  | gzip > backups/$(date +\%F).sql.gz && \
  find backups/ -name "*.sql.gz" -mtime +7 -delete

MinIO

Mirror to local path:

docker compose exec minio mc mirror /data /backup/minio-mirror

Mirror to an S3-compatible offsite target:

docker compose exec minio mc mirror \
  local-minio/docuvault s3://offsite-bucket/docuvault-mirror

Restore from mirror:

docker compose exec minio mc mirror /backup/minio-mirror /data

Retention policy (recommendation)

Frequency	Keep
Daily	7 days
Weekly	4 weeks
Monthly	12 months

Implement via cron + find … -mtime +N -delete. Backup automation as a Docker service is a deferred item (see Section 8).

---

4. Health Checks

Run these commands after startup or after an incident to verify each service is healthy.

Service	Probe Command	Expected Output
Backend API	curl -sf http://localhost:8000/health	{"status":"ok",...} (HTTP 200)
Loki	curl -sf http://localhost:3100/ready	ready
Grafana	curl -sf http://localhost:3000/api/health	JSON with "database":"ok"
Postgres	docker compose exec postgres pg_isready -U postgres -d docuvault	accepting connections (exit 0)
MinIO	docker compose exec minio mc ready local	exit 0
Redis	docker compose exec redis redis-cli -a $REDIS_PASSWORD ping	PONG
Celery worker	docker compose exec celery-worker celery -A celery_app inspect ping	pong response

All-in-one check script:

#!/usr/bin/env bash
set -e
echo "Backend:  $(curl -sf http://localhost:8000/health | jq -r .status)"
echo "Loki:     $(curl -sf http://localhost:3100/ready)"
echo "Grafana:  $(curl -sf http://localhost:3000/api/health | jq -r .database)"
docker compose exec -T postgres pg_isready -U postgres -d docuvault
docker compose exec -T minio mc ready local
docker compose exec -T redis redis-cli -a "$REDIS_PASSWORD" ping
echo "All services healthy."

---

5. Security Gate — docker scout CVE Scan (D-10)

Run this gate before every production deploy and after every base-image rebuild. Zero critical CVEs is the hard pass requirement.

Prerequisites

1. Docker Desktop (or Docker Engine with the scout plugin) must be installed.

2. Authenticate with Docker Hub — docker scout uploads the image manifest to Docker's CVE analysis service (Pitfall 5):

   docker login
   

   Skip if already logged in (check with docker system info | grep Username).

Build the Phase 6 hardened image

The multi-stage backend/Dockerfile (Phase 6 D-07) produces a lean runtime image with appuser (uid 1000), no build toolchain, and a read-only-friendly layout:

cd backend
docker build -t docuvault-backend:phase6 .

Run the CVE scan

docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code

Exit code	Meaning
0	PASS — zero critical CVEs. Safe to deploy.
2	FAIL — one or more critical CVEs found. Do not deploy until resolved.

A passing scan looks like:

CRITICAL: 0 critical
HIGH:     2 high
MEDIUM:   15 medium
LOW:      40 low

Remediation if the scan fails

1. Pull the latest base image and rebuild:

   docker pull python:3.12-slim
   cd backend && docker build --no-cache -t docuvault-backend:phase6 .
   docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code
   

2. If a Python package is the source of the CVE, bump its pin in backend/requirements.txt to a patched version and rebuild.

3. If the CVE is in the base OS packages (Debian), try pinning to a newer Python 3.12 patch release (python:3.12.X-slim) or, after compatibility testing, migrate to python:3.13-slim.

4. If the CVE cannot be resolved in this release cycle, escalate to the operator with the full CVE list. The phase must not be marked complete until the critical count reaches zero.

Fallback: Trivy (offline / docker scout unavailable)

If docker scout is unavailable (offline environment or Docker Hub outage):

# macOS
brew install trivy

# Debian/Ubuntu
apt install -y trivy

# Run scan
trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6

Exit code 0 = no critical CVEs. Exit code 1 = critical CVEs found.

Scan cadence

• Before every production deploy — mandatory gate.
• Weekly scheduled re-scan — new CVEs are published daily; old images can become vulnerable after release. CI/CD automation of this cadence is a deferred item (see Section 8).

---

6. On-Call Escalation

DocuVault runs as a single-operator deployment. There is no secondary on-call tier. The operator is the first and only responder. This section documents alert classes and the self-service recovery action for each so the operator can act at 03:00 without reading PLAN.md.

Alert classes and actions

Alert	Urgency	Immediate action
backend-down	Page immediately	docker compose logs --tail 100 backend — check for Read-only file system (Pitfall 1/6) or permission errors. See Section 7 for recovery.
celery-stuck	Warn within 1 h	docker compose logs --tail 200 celery-worker — check for queue depth or task errors. Restart: docker compose restart celery-worker.
loki-full-disk	Warn within 4 h	Loki container >80% disk on /loki. Prune old chunks via Loki retention tuning or rotate the loki_data volume.
docker-scout-critical-CVE	Page within 24 h	Rebuild image with docker pull python:3.12-slim && docker compose build backend. Re-run Section 5 scan.
quota-violation-spike	Warn	>10 quota_exceeded events/hour from same user → possible abuse. Disable account via admin endpoint.
failed-login-spike	Warn	>100 failed_login events/hour from same IP → verify per-IP rate limiter is firing (10/min). If not, confirm 06-05 deployed and TRUSTED_PROXY_CIDRS / proxy topology is correct.

Contact

Role	Contact
Primary on-call	curo1305@curonet.de (project owner)
Secondary	(none — single-operator deployment)

---

7. Common Failure Modes

7.1 PermissionError: [Errno 13] /tmp/… in backend logs

Symptom: PermissionError: [Errno 13] Permission denied: '/tmp/tmpXXXXXX'

Cause: Pitfall 1 — tmpfs mount missing mode=1777. By default, tmpfs mounts are owned by root and non-root containers (appuser, uid 1000) cannot write to them.

Recovery:

1. Verify docker-compose.yml backend service has:

   tmpfs:
     - "/tmp:mode=1777"
   

2. Recreate the container:

   docker compose up -d --force-recreate backend
   

---

7.2 docker scout returns "authentication required"

Symptom: docker scout cves exits with an authentication error.

Cause: Pitfall 5 — docker scout requires a Docker Hub session to submit the image manifest to Docker's CVE analysis service.

Recovery:

docker login
# Re-run the scan
docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code

---

7.3 PyMuPDF Read-only file system when extracting PDF

Symptom: RuntimeError: Read-only file system originating from PyMuPDF (Celery worker or backend during extraction).

Cause: Pitfall 6 — PyMuPDF writes font cache data to /var/cache/fontconfig at runtime. Under read_only: true, this directory is not writable.

Recovery: Add a tmpfs mount for fontconfig in docker-compose.yml:

services:
  backend:
    tmpfs:
      - "/tmp:mode=1777"
      - "/var/cache/fontconfig"   # ← add this line
  celery-worker:
    tmpfs:
      - "/tmp:mode=1777"
      - "/var/cache/fontconfig"   # ← add this line

Then recreate:

docker compose up -d --force-recreate backend celery-worker

---

7.4 celery-beat exits with Permission denied: 'celerybeat-schedule'

Symptom: celery-beat container exits immediately with permission denied on celerybeat-schedule.

Cause: Pitfall 7 — read_only: true was accidentally added to the celery-beat service block. celery-beat writes its schedule file to its working directory at startup.

Recovery: Remove read_only: from the celery-beat block in docker-compose.yml:

# celery-beat: NOT hardened — writes celerybeat-schedule (Pitfall 7)
celery-beat:
  build: ./backend
  # read_only: true   ← remove this line if present
  ...

Then recreate:

docker compose up -d --force-recreate celery-beat

---

7.5 External clients hit rate limit despite low traffic

Symptom: After deploying 06-05, external clients receive 429 Too Many Requests at far below the expected threshold.

Cause: A reverse proxy (nginx, Caddy, load balancer) sits in front of the backend but its IP is not in the trusted proxy CIDR list. The rate limiter sees the proxy's IP as the client IP for all traffic and exhausts the per-IP limit immediately.

Recovery: The trusted proxy CIDRs are hardcoded in backend/deps/utils.py as _TRUSTED_PROXY_NETS. Add the proxy's CIDR to that list and redeploy:

_TRUSTED_PROXY_NETS = [
    ipaddress.ip_network("127.0.0.0/8"),
    ipaddress.ip_network("172.16.0.0/12"),
    ipaddress.ip_network("192.168.0.0/16"),
    ipaddress.ip_network("::1/128"),
    ipaddress.ip_network("10.0.0.0/8"),   # ← add your proxy CIDR here
]

Then rebuild and redeploy the backend.

---

7.6 Per-account 429s when expecting per-IP 429s

Symptom: A single user receives 429 errors on authenticated endpoints far earlier than the 100 req/min per-account limit, or all users receive 429s at the same time.

Cause: A route handler is missing the first-line assignment request.state.current_user = current_user. Without it, _account_key in services/rate_limiting.py falls back to request.client.host, making the limiter key per-IP instead of per-account.

Recovery: Find the handler missing the assignment and add it as the first executable statement inside the function body:

@router.get("/some-endpoint")
@account_limiter.limit("100/minute")
async def some_endpoint(request: Request, current_user: User = Depends(get_regular_user)):
    request.state.current_user = current_user   # ← first line, always
    ...

---

7.7 Grafana loads but no Loki datasource

Symptom: Grafana UI loads but the Loki datasource shows "Data source connected and labels found" or cannot connect.

Cause: Loki container not ready when Grafana started, or loki-config.yaml is malformed.

Recovery:

1. Check Loki readiness:

   curl -sf http://localhost:3100/ready
   docker compose logs --tail 50 loki
   

2. If Loki is not ready, look for YAML parse errors in its logs.
3. Restart the stack after Loki is confirmed healthy:

   docker compose restart grafana
   

---

8. Phase 6 Deferred Items

The following improvements are out of scope for Phase 6. They are documented here so the next operator knows the current runbook acknowledges them:

Item	Current state	Future path
HTTPS/TLS termination	Backend serves plain HTTP on port 8000.	Add nginx or Caddy in front; see nginx reverse proxy pattern. The runbook will gain a TLS section when this ships.
Horizontal scaling	Rate limit counters are in-memory (single-instance only).	Switch to Redis-backed counters (slowapi supports this); add uvicorn worker count config. Phase 7+ concern.
CI/CD pipeline	docker scout cves and Locust load tests are run manually.	GitHub Actions workflow for automated scan + load test on every PR. Deferred — no CI setup exists yet.
Backup automation	pg_dump and mc mirror run manually or via user-managed cron.	Add a backup service to docker-compose.yml that runs the cron commands inside the stack. Documented procedure above is the current state.