# DocuVault Operational Runbook

This is the single reference for running DocuVault in production-like environments. It covers
environment variables, startup/shutdown procedures, backup strategy, health checks, on-call
escalation, and common failure-mode recovery. For architecture and rationale see `CLAUDE.md`;
for phase history and roadmap see `.planning/ROADMAP.md`.

> **Last updated:** 2026-06-04

---

## Table of Contents

1. [Environment Variables](#1-environment-variables)
2. [Startup / Shutdown](#2-startup--shutdown)
3. [Database Migrations](#3-database-migrations)
4. [Backup Strategy](#4-backup-strategy)
5. [Health Checks](#5-health-checks)
6. [Security Gate — docker scout CVE Scan (D-10)](#6-security-gate--docker-scout-cve-scan-d-10)
7. [On-Call Escalation](#7-on-call-escalation)
8. [Common Failure Modes](#8-common-failure-modes)
9. [Phase 6 Deferred Items](#9-phase-6-deferred-items)

---

## 1. Environment Variables

All variables are read from a `.env` file at the repo root (or injected by the container
orchestrator). Copy `.env.example` to `.env` and populate every field before starting services.

> **Security note:** `SECRET_KEY`, `CLOUD_CREDS_KEY`, and all `*_PASSWORD`/`*_SECRET` fields
> must **never** be committed to version control. Rotate them if exposure is suspected.

### PostgreSQL

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `DATABASE_URL` | Yes | Application DSN (`docuvault_app` role — DML only). | `postgresql+psycopg://docuvault_app:****@postgres:5432/docuvault` | `.env` |
| `DATABASE_MIGRATE_URL` | Yes | Migration DSN (`docuvault_migrate` role — DDL). Used by Alembic only. | `postgresql+psycopg://docuvault_migrate:****@postgres:5432/docuvault` | `.env` |
| `POSTGRES_PASSWORD` | Yes | Root postgres superuser password (used by the docker-compose `postgres` service). | `change-me-postgres` | `.env` |

### MinIO

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `MINIO_ENDPOINT` | Yes | Internal Docker hostname:port for MinIO SDK calls. | `minio:9000` | `.env` |
| `MINIO_ACCESS_KEY` | Yes | MinIO application access key. | `docuvault_app` | `.env` |
| `MINIO_SECRET_KEY` | Yes | MinIO application secret key. | `****` | `.env` |
| `MINIO_BUCKET` | Yes | Primary document storage bucket name. | `docuvault` | `.env` |
| `MINIO_PUBLIC_ENDPOINT` | No | Browser-visible hostname for presigned upload/download URLs. Empty → falls back to `MINIO_ENDPOINT`. | `localhost:9000` | `.env` |
| `MINIO_ROOT_USER` | Yes | MinIO root/admin username (used by the `minio` compose service). | `minioadmin` | `.env` |
| `MINIO_ROOT_PASSWORD` | Yes | MinIO root/admin password. | `****` | `.env` |

### Redis / Celery

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `REDIS_URL` | Yes | Redis connection string including password. | `redis://:****@redis:6379/0` | `.env` |
| `REDIS_PASSWORD` | Yes | Redis `requirepass` password (used by the `redis` compose service). | `****` | `.env` |

### Security / Auth

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `SECRET_KEY` | Yes | HMAC/JWT signing secret. Must be a high-entropy random string ≥32 bytes. Rotate to invalidate all active sessions. | `openssl rand -hex 32` output | `.env` |
| `CLOUD_CREDS_KEY` | Yes | Master key for HKDF per-user cloud credential encryption. Must be exactly 32 bytes. | `openssl rand -hex 16` (pad to 32) | `.env` |
| `ADMIN_EMAIL` | Yes | Bootstrap admin account email. Created on first startup if absent. | `admin@example.com` | `.env` |
| `ADMIN_PASSWORD` | Yes | Bootstrap admin account password. Must meet AUTH-01 strength requirements. | `Ch@ngeMe1!` | `.env` |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | No | JWT access token lifetime in minutes. Default: `15`. Do not increase beyond 30. | `15` | `.env` |
| `REFRESH_TOKEN_EXPIRE_DAYS` | No | Refresh token cookie lifetime in days. Default: `30`. | `30` | `.env` |

**Rotation procedure for `SECRET_KEY`:**

1. Generate a new value: `openssl rand -hex 32`
2. Update `.env` and restart the `backend` service.
3. All active JWT sessions are immediately invalidated — users must log in again.

**Rotation procedure for `CLOUD_CREDS_KEY`:**

Requires re-encrypting all stored cloud credentials. Until a migration tool exists, this
rotation is a breaking change — treat the key as long-lived and protect it accordingly.

### CORS / Frontend

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `CORS_ORIGINS` | No | Comma-separated list of allowed CORS origins. Default: `http://localhost:5173`. | `https://app.example.com` | `.env` |
| `FRONTEND_URL` | No | Base URL of the Vue frontend — used in password-reset emails. Default: `http://localhost:5173`. | `https://app.example.com` | `.env` |
| `BACKEND_URL` | No | Externally reachable backend URL — used to construct OAuth callback URLs. Default: `http://localhost:8000`. | `https://api.example.com` | `.env` |

### SMTP (password reset emails)

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `SMTP_HOST` | No | SMTP relay hostname. Leave empty to disable email. | `smtp.sendgrid.net` | `.env` |
| `SMTP_PORT` | No | SMTP port. Default: `587` (STARTTLS). | `587` | `.env` |
| `SMTP_USER` | No | SMTP login username. | `apikey` | `.env` |
| `SMTP_PASSWORD` | No | SMTP login password / API key. | `****` | `.env` |
| `SMTP_FROM` | No | Sender address for outgoing emails. Default: `noreply@docuvault.local`. | `noreply@example.com` | `.env` |

### AI Classification

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `DEFAULT_AI_PROVIDER` | No | Default LLM provider. Default: `ollama`. Options: `ollama`, `openai`, `anthropic`. | `openai` | `.env` |
| `DEFAULT_AI_MODEL` | No | Default model name for the chosen provider. Default: `llama3.2`. | `gpt-4o-mini` | `.env` |
| `SYSTEM_PROMPT` | No | Custom system prompt override for the classifier. Hardcoded fallback lives in `backend/ai/classifier.py`. | `Classify documents concisely.` | `.env` |

### Cloud Storage Providers (Phase 5)

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `GOOGLE_CLIENT_ID` | No | Google OAuth 2.0 client ID for Drive integration. | `1234567890-abc.apps.googleusercontent.com` | Google Cloud Console |
| `GOOGLE_CLIENT_SECRET` | No | Google OAuth 2.0 client secret. | `GOCSPX-****` | Google Cloud Console |
| `ONEDRIVE_CLIENT_ID` | No | Azure AD application (client) ID for OneDrive. | `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` | Azure Portal |
| `ONEDRIVE_CLIENT_SECRET` | No | Azure AD client secret value. | `****` | Azure Portal |
| `ONEDRIVE_TENANT_ID` | No | Azure AD tenant. Default: `common` (personal + org accounts). | `common` | `.env` |

### Observability (Phase 6)

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `LOG_LEVEL` | No | structlog minimum level. Default: `INFO`. Options: `DEBUG`, `INFO`, `WARNING`, `ERROR`. | `INFO` | `.env` |
| `LOG_JSON` | No | Emit structured JSON log lines instead of plain text. Default: `false`. Set `true` in production. | `true` | `.env` |

### Load Testing (Phase 6 — dev only)

| Name | Required | Description | Example | Source |
|------|----------|-------------|---------|--------|
| `LOAD_TEST_EMAIL` | No | Email of the pre-created load-test user. Default: `loadtest@example.com`. | `loadtest@staging.example.com` | `.env` (dev only) |
| `LOAD_TEST_PASSWORD` | No | Password for the load-test user. Default: `Loadtest123!@#`. | `MyS3cure!Pass` | `.env` (dev only) |

---

## 2. Startup / Shutdown

### Full stack startup

```bash
# Build images and start all services in the background
docker compose up -d --build

# Tail logs for the first 60 s to confirm everything came up cleanly
docker compose logs -f --tail 50
```

Services start in dependency order:

```
postgres / minio / redis  →  migrate (alembic upgrade head)
                          →  backend / celery-worker / celery-beat
                          →  loki  →  promtail / grafana
                          →  frontend
```

The `migrate` service is a one-shot container that runs `alembic upgrade head` using
`DATABASE_MIGRATE_URL` (DDL-capable `docuvault_migrate` role) and exits 0 on success.
All application services (`backend`, `celery-worker`, `celery-beat`) depend on it with
`condition: service_completed_successfully`. If `migrate` exits non-zero, no application
process starts and the deployment is blocked until the migration issue is resolved.

Access points after a healthy startup:

| Service | URL |
|---------|-----|
| Vue frontend | http://localhost:5173 |
| FastAPI backend (health) | http://localhost:8000/health |
| MinIO console | http://localhost:9001 |
| Grafana (log UI) | http://localhost:3000 |
| Loki (API/ready) | http://localhost:3100/ready |

### Development mode (live reload)

The `docker-compose.yml` already mounts `./backend:/app` with `--reload`, so
Python file changes trigger automatic reload. To force a rebuild after changing
`requirements.txt` or the `Dockerfile`:

```bash
docker compose build backend celery-worker celery-beat
docker compose up -d backend celery-worker celery-beat
```

### Orderly shutdown

```bash
# Stop all containers, preserve volumes (normal dev shutdown)
docker compose down

# Stop and remove named volumes (WARNING: deletes all data)
docker compose down -v
```

> **Warning:** `docker compose down -v` deletes `postgres_data`, `minio_data`,
> `loki_data`, and `grafana_data`. Only use this for a full environment reset.

### celery-beat note (Pitfall 7)

The `celery-beat` service does **not** have `read_only: true` in `docker-compose.yml`
because it writes the `celerybeat-schedule` file to its working directory at startup.
Do not add `read_only:` to the celery-beat block — it will exit immediately with
`Permission denied: 'celerybeat-schedule'`.

---

## 3. Database Migrations

### Checking the current revision

```bash
# Inside the running backend container
docker compose exec backend alembic current

# Via a one-off migrate container (preferred — uses DDL role)
docker compose run --rm migrate alembic current
```

A healthy deployment at Phase 12 head reports:

```
0006 (head)
```

### Upgrading to the latest migration

```bash
# Compose will run migrate automatically on docker compose up.
# To run it manually (e.g., after a git pull):
docker compose run --rm migrate
```

`migrate` runs `alembic upgrade head` using `DATABASE_MIGRATE_URL`. On success it exits 0.
Application services will not start until this exits 0 — this is the deployment gate.

### Viewing migration history

```bash
docker compose run --rm migrate alembic history --verbose
```

### Recovering from a failed migration

1. **Read the logs** — the migrate container logs contain the SQL and the error:
   ```bash
   docker compose logs migrate
   ```

2. **Fix the migration** — edit the failing migration revision file in `backend/migrations/versions/`.

3. **Re-run** — the migrate container is idempotent; Alembic skips already-applied revisions:
   ```bash
   docker compose run --rm migrate
   ```

4. **Check the revision** — confirm head is applied before restarting backend:
   ```bash
   docker compose run --rm migrate alembic current
   ```

5. **Restart application services** after a successful migration:
   ```bash
   docker compose up -d backend celery-worker celery-beat
   ```

### Emergency: schema drift on live database (Phase 12 gap-closure scenario)

If the backend is already running against a database that is behind head (e.g., it started
before the migrate service was introduced and reports `UndefinedColumn: display_name_override`):

```bash
# 1. Stop application processes (not postgres)
docker compose stop backend celery-worker celery-beat

# 2. Apply missing migrations via the dedicated migrate service
docker compose run --rm migrate

# 3. Verify the revision
docker compose run --rm migrate alembic current
# Expected: 0006 (head)

# 4. Restart application processes through the gated path
docker compose up -d backend celery-worker celery-beat
```

---

## 4. Backup Strategy

### PostgreSQL

**Manual backup:**
```bash
docker compose exec -T postgres pg_dump -U postgres docuvault \
  > backups/docuvault_$(date +%F).sql
```

**Restore:**
```bash
cat backups/docuvault_YYYY-MM-DD.sql \
  | docker compose exec -T postgres psql -U postgres docuvault
```

**Cron pattern (daily at 03:00, keep 7 days):**
```bash
# Add to crontab (crontab -e)
0 3 * * * cd /path/to/docuvault && \
  docker compose exec -T postgres pg_dump -U postgres docuvault \
  | gzip > backups/$(date +\%F).sql.gz && \
  find backups/ -name "*.sql.gz" -mtime +7 -delete
```

### MinIO

**Mirror to local path:**
```bash
docker compose exec minio mc mirror /data /backup/minio-mirror
```

**Mirror to an S3-compatible offsite target:**
```bash
docker compose exec minio mc mirror \
  local-minio/docuvault s3://offsite-bucket/docuvault-mirror
```

**Restore from mirror:**
```bash
docker compose exec minio mc mirror /backup/minio-mirror /data
```

### Retention policy (recommendation)

| Frequency | Keep |
|-----------|------|
| Daily | 7 days |
| Weekly | 4 weeks |
| Monthly | 12 months |

Implement via cron + `find … -mtime +N -delete`. Backup automation as a Docker
service is a deferred item (see Section 8).

---

## 4. Health Checks

Run these commands after startup or after an incident to verify each service is
healthy.

| Service | Probe Command | Expected Output |
|---------|--------------|-----------------|
| Backend API | `curl -sf http://localhost:8000/health` | `{"status":"ok",...}` (HTTP 200) |
| Loki | `curl -sf http://localhost:3100/ready` | `ready` |
| Grafana | `curl -sf http://localhost:3000/api/health` | JSON with `"database":"ok"` |
| Postgres | `docker compose exec postgres pg_isready -U postgres -d docuvault` | `accepting connections` (exit 0) |
| MinIO | `docker compose exec minio mc ready local` | exit 0 |
| Redis | `docker compose exec redis redis-cli -a $REDIS_PASSWORD ping` | `PONG` |
| Celery worker | `docker compose exec celery-worker celery -A celery_app inspect ping` | `pong` response |

**All-in-one check script:**

```bash
#!/usr/bin/env bash
set -e
echo "Backend:  $(curl -sf http://localhost:8000/health | jq -r .status)"
echo "Loki:     $(curl -sf http://localhost:3100/ready)"
echo "Grafana:  $(curl -sf http://localhost:3000/api/health | jq -r .database)"
docker compose exec -T postgres pg_isready -U postgres -d docuvault
docker compose exec -T minio mc ready local
docker compose exec -T redis redis-cli -a "$REDIS_PASSWORD" ping
echo "All services healthy."
```

---

## 5. Security Gate — docker scout CVE Scan (D-10)

Run this gate **before every production deploy** and after every base-image rebuild.
Zero critical CVEs is the hard pass requirement.

### Prerequisites

1. Docker Desktop (or Docker Engine with the `scout` plugin) must be installed.
2. Authenticate with Docker Hub — `docker scout` uploads the image manifest to
   Docker's CVE analysis service (Pitfall 5):

   ```bash
   docker login
   ```

   Skip if already logged in (check with `docker system info | grep Username`).

### Build the Phase 6 hardened image

The multi-stage `backend/Dockerfile` (Phase 6 D-07) produces a lean runtime image
with `appuser` (uid 1000), no build toolchain, and a read-only-friendly layout:

```bash
cd backend
docker build -t docuvault-backend:phase6 .
```

### Run the CVE scan

```bash
docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code
```

| Exit code | Meaning |
|-----------|---------|
| `0` | **PASS** — zero critical CVEs. Safe to deploy. |
| `2` | **FAIL** — one or more critical CVEs found. Do not deploy until resolved. |

A passing scan looks like:

```text
CRITICAL: 0 critical
HIGH:     2 high
MEDIUM:   15 medium
LOW:      40 low
```

### Remediation if the scan fails

1. Pull the latest base image and rebuild:
   ```bash
   docker pull python:3.12-slim
   cd backend && docker build --no-cache -t docuvault-backend:phase6 .
   docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code
   ```

2. If a Python package is the source of the CVE, bump its pin in
   `backend/requirements.txt` to a patched version and rebuild.

3. If the CVE is in the base OS packages (Debian), try pinning to a newer
   Python 3.12 patch release (`python:3.12.X-slim`) or, after compatibility
   testing, migrate to `python:3.13-slim`.

4. If the CVE cannot be resolved in this release cycle, escalate to the operator
   with the full CVE list. The phase must not be marked complete until the
   critical count reaches zero.

### Fallback: Trivy (offline / docker scout unavailable)

If `docker scout` is unavailable (offline environment or Docker Hub outage):

```bash
# macOS
brew install trivy

# Debian/Ubuntu
apt install -y trivy

# Run scan
trivy image --severity CRITICAL --exit-code 1 docuvault-backend:phase6
```

Exit code `0` = no critical CVEs. Exit code `1` = critical CVEs found.

### Scan cadence

- **Before every production deploy** — mandatory gate.
- **Weekly scheduled re-scan** — new CVEs are published daily; old images can
  become vulnerable after release. CI/CD automation of this cadence is a
  deferred item (see Section 8).

---

## 6. On-Call Escalation

DocuVault runs as a **single-operator deployment**. There is no secondary on-call tier.
The operator is the first and only responder. This section documents alert classes and
the self-service recovery action for each so the operator can act at 03:00 without
reading PLAN.md.

### Alert classes and actions

| Alert | Urgency | Immediate action |
|-------|---------|-----------------|
| `backend-down` | **Page immediately** | `docker compose logs --tail 100 backend` — check for `Read-only file system` (Pitfall 1/6) or permission errors. See Section 7 for recovery. |
| `celery-stuck` | Warn within 1 h | `docker compose logs --tail 200 celery-worker` — check for queue depth or task errors. Restart: `docker compose restart celery-worker`. |
| `loki-full-disk` | Warn within 4 h | Loki container >80% disk on `/loki`. Prune old chunks via Loki retention tuning or rotate the `loki_data` volume. |
| `docker-scout-critical-CVE` | Page within 24 h | Rebuild image with `docker pull python:3.12-slim && docker compose build backend`. Re-run Section 5 scan. |
| `quota-violation-spike` | Warn | >10 `quota_exceeded` events/hour from same user → possible abuse. Disable account via admin endpoint. |
| `failed-login-spike` | Warn | >100 `failed_login` events/hour from same IP → verify per-IP rate limiter is firing (10/min). If not, confirm 06-05 deployed and `TRUSTED_PROXY_CIDRS` / proxy topology is correct. |

### Contact

| Role | Contact |
|------|---------|
| Primary on-call | curo1305@curonet.de (project owner) |
| Secondary | *(none — single-operator deployment)* |

---

## 7. Common Failure Modes

### 7.1 `PermissionError: [Errno 13] /tmp/…` in backend logs

**Symptom:** `PermissionError: [Errno 13] Permission denied: '/tmp/tmpXXXXXX'`

**Cause:** Pitfall 1 — `tmpfs` mount missing `mode=1777`. By default, tmpfs mounts
are owned by root and non-root containers (`appuser`, uid 1000) cannot write to them.

**Recovery:**

1. Verify `docker-compose.yml` backend service has:
   ```yaml
   tmpfs:
     - "/tmp:mode=1777"
   ```
2. Recreate the container:
   ```bash
   docker compose up -d --force-recreate backend
   ```

---

### 7.2 `docker scout` returns "authentication required"

**Symptom:** `docker scout cves` exits with an authentication error.

**Cause:** Pitfall 5 — `docker scout` requires a Docker Hub session to submit the
image manifest to Docker's CVE analysis service.

**Recovery:**

```bash
docker login
# Re-run the scan
docker scout cves local://docuvault-backend:phase6 --only-severity critical --exit-code
```

---

### 7.3 PyMuPDF `Read-only file system` when extracting PDF

**Symptom:** `RuntimeError: Read-only file system` originating from PyMuPDF
(Celery worker or backend during extraction).

**Cause:** Pitfall 6 — PyMuPDF writes font cache data to `/var/cache/fontconfig`
at runtime. Under `read_only: true`, this directory is not writable.

**Recovery:** Add a tmpfs mount for fontconfig in `docker-compose.yml`:

```yaml
services:
  backend:
    tmpfs:
      - "/tmp:mode=1777"
      - "/var/cache/fontconfig"   # ← add this line
  celery-worker:
    tmpfs:
      - "/tmp:mode=1777"
      - "/var/cache/fontconfig"   # ← add this line
```

Then recreate:
```bash
docker compose up -d --force-recreate backend celery-worker
```

---

### 7.4 `celery-beat` exits with `Permission denied: 'celerybeat-schedule'`

**Symptom:** `celery-beat` container exits immediately with permission denied on
`celerybeat-schedule`.

**Cause:** Pitfall 7 — `read_only: true` was accidentally added to the `celery-beat`
service block. celery-beat writes its schedule file to its working directory at startup.

**Recovery:** Remove `read_only:` from the `celery-beat` block in `docker-compose.yml`:

```yaml
# celery-beat: NOT hardened — writes celerybeat-schedule (Pitfall 7)
celery-beat:
  build: ./backend
  # read_only: true   ← remove this line if present
  ...
```

Then recreate:
```bash
docker compose up -d --force-recreate celery-beat
```

---

### 7.5 External clients hit rate limit despite low traffic

**Symptom:** After deploying 06-05, external clients receive `429 Too Many Requests`
at far below the expected threshold.

**Cause:** A reverse proxy (nginx, Caddy, load balancer) sits in front of the backend
but its IP is not in the trusted proxy CIDR list. The rate limiter sees the proxy's IP
as the client IP for all traffic and exhausts the per-IP limit immediately.

**Recovery:** The trusted proxy CIDRs are hardcoded in `backend/deps/utils.py`
as `_TRUSTED_PROXY_NETS`. Add the proxy's CIDR to that list and redeploy:

```python
_TRUSTED_PROXY_NETS = [
    ipaddress.ip_network("127.0.0.0/8"),
    ipaddress.ip_network("172.16.0.0/12"),
    ipaddress.ip_network("192.168.0.0/16"),
    ipaddress.ip_network("::1/128"),
    ipaddress.ip_network("10.0.0.0/8"),   # ← add your proxy CIDR here
]
```

Then rebuild and redeploy the backend.

---

### 7.6 Per-account 429s when expecting per-IP 429s

**Symptom:** A single user receives 429 errors on authenticated endpoints far earlier
than the 100 req/min per-account limit, or all users receive 429s at the same time.

**Cause:** A route handler is missing the first-line assignment
`request.state.current_user = current_user`. Without it, `_account_key` in
`services/rate_limiting.py` falls back to `request.client.host`, making the limiter
key per-IP instead of per-account.

**Recovery:** Find the handler missing the assignment and add it as the first
executable statement inside the function body:

```python
@router.get("/some-endpoint")
@account_limiter.limit("100/minute")
async def some_endpoint(request: Request, current_user: User = Depends(get_regular_user)):
    request.state.current_user = current_user   # ← first line, always
    ...
```

---

### 7.7 Grafana loads but no Loki datasource

**Symptom:** Grafana UI loads but the Loki datasource shows "Data source connected and
labels found" or cannot connect.

**Cause:** Loki container not ready when Grafana started, or `loki-config.yaml` is
malformed.

**Recovery:**

1. Check Loki readiness:
   ```bash
   curl -sf http://localhost:3100/ready
   docker compose logs --tail 50 loki
   ```
2. If Loki is not ready, look for YAML parse errors in its logs.
3. Restart the stack after Loki is confirmed healthy:
   ```bash
   docker compose restart grafana
   ```

---

## 9. Phase 12 — Cloud Resource Foundation Operations

### Cloud connection management

**List a user's cloud connections (admin):**
```bash
# Via API (admin token required for admin panel only — not cloud browse endpoints)
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8000/api/admin/users
# Cloud connections are user-scoped. Admins cannot browse connection items.
```

**Check connection status from DB:**
```sql
SELECT id, user_id, provider, display_name, status, created_at
FROM cloud_connections
WHERE user_id = '<user-uuid>';
```

**Re-enable a DISABLED connection:**
```sql
UPDATE cloud_connections SET status = 'ACTIVE' WHERE id = '<conn-uuid>';
```

### Cloud browse background refresh

The browse endpoint (`GET /api/cloud/connections/{id}/items`) serves cached metadata and schedules a background refresh when stale:
- **Freshness states:** `fresh` (complete authoritative listing), `refreshing` (refresh in progress), `warning` (incomplete listing or provider error)
- **`fresh` requires `complete=True`** — a `CloudListing(complete=False)` result from any provider always produces `warning` state (`incomplete_listing` error code), never `fresh`. `last_refreshed_at` is only advanced on successful `complete=True` listings.
- **Cached data is always served** — the UI shows a warning banner for stale/error states without blocking navigation
- **No bytes are downloaded** during browse; `size_bytes` values come from provider-reported metadata only

**Trigger a manual refresh (no direct endpoint — use the UI Refresh button):**
The UI CloudFolderView sends a refresh request that updates `cloud_folder_states.refresh_state` to `refreshing` and schedules a background task.

**Inspect refresh state:**
```sql
SELECT connection_id, folder_ref, refresh_state, refreshed_at, error_code
FROM cloud_folder_states
ORDER BY refreshed_at DESC NULLS LAST
LIMIT 20;
```

**Clear stuck refreshing state:**
```sql
UPDATE cloud_folder_states
SET refresh_state = 'stale', error_code = NULL
WHERE refresh_state = 'refreshing'
  AND refreshed_at < NOW() - INTERVAL '15 minutes';
```

### Cloud item metadata

Cloud items are stored in `cloud_items`. They are metadata-only — no file bytes are in the database.

**Count items per connection:**
```sql
SELECT connection_id, count(*) AS item_count
FROM cloud_items
GROUP BY connection_id;
```

**Purge orphaned items (after connection deleted):**
```sql
-- cloud_items.connection_id has ON DELETE CASCADE — orphaned items are removed automatically
-- Verify with:
SELECT count(*) FROM cloud_items ci
LEFT JOIN cloud_connections cc ON ci.connection_id = cc.id
WHERE cc.id IS NULL;
-- Should return 0
```

### Security operations — cloud browse

- **IDOR protection:** All browse endpoints use `get_regular_user` and assert `connection.user_id == current_user.id`. Violations return 404, not 403, to avoid enumeration.
- **Admin tokens:** Blocked by `get_regular_user`. Admin users cannot browse other users' connections.
- **Credential fields:** `credentials_enc` is never included in browse API responses. Verify with: `grep -r "credentials_enc" backend/api/cloud/` — must only appear in connection create/update service code, never in response schemas.
- **SSRF guard:** `storage/cloud_utils.validate_cloud_url` blocks RFC1918, link-local, file://, and cloud metadata IPs. All WebDAV/Nextcloud URLs pass through this guard before any connection is established.

### Phase 12.1 Live Smoke Tests (Opt-In, Read-Only)

The `live_nextcloud` pytest marker selects opt-in tests that use real Nextcloud credentials. These tests are **excluded from ordinary CI runs** (`addopts = -m "not live_nextcloud"` in `pytest.ini`).

**Running the live smoke:**
```bash
# Set in .env (never committed):
#   NEXTCLOUD_URL=<base URL of your Nextcloud instance>
#   NEXTCLOUD_USER=<Nextcloud username>
#   NEXTCLOUD_APP_PASSWORD=<Nextcloud app password (not account password)>

cd backend
pytest -m live_nextcloud tests/test_nextcloud_live.py
```

**What the live tests do:**
- `test_nextcloud_adapter_read_only_root_metadata` — invokes production adapter directly via `PROPFIND`; asserts no byte/mutation method exists; verifies item ownership identity
- `test_nextcloud_root_diagnostic` — sanitized count/kind diagnostic; prints total, expected-match count, missing expected names, unexpected count (names withheld); exits successfully even if candidate manifest has drift
- `test_nextcloud_connection_id_browse_as_designated_user` — end-to-end browse via `GET /api/cloud/connections/{id}/items` in isolated test DB; asserts foreign-user 403/404 and admin 403/404
- `test_nextcloud_expected_root_manifest` — exact set and kind equality against owner-confirmed fixture (requires `-k expected_root_manifest` or running all live tests)

**What they do NOT do:**
- Never download file bytes
- Never issue PUT/POST/PATCH/DELETE/MKCOL/MOVE/COPY
- Never print credential values, full URLs, or unexpected provider names
- Never change MinIO objects, quota, or provider content
- Missing credentials produce a descriptive skip, not a failure

**Fixture reconciliation procedure:**
If the expected root manifest (`backend/tests/fixtures/cloud/nextcloud_expected_root.json`) drifts:
1. Run `pytest -m live_nextcloud -k root_diagnostic` and read the sanitized output (missing expected names only)
2. Compare the missing names with the Nextcloud web UI (do not print the unexpected names)
3. Update the fixture with the owner-confirmed correct names and set `owner_confirmed: true`
4. Never record unexpected live names in any committed artifact

### Phase 12 / 12.1 Warning Semantics

A `warning` freshness state in the browse response means the last reconciliation was incomplete or failed. **The UI shows a warning indicator; cached rows remain visible.** This is not an error — it is honest state:

| State | Meaning | UI behavior |
|-------|---------|-------------|
| `fresh` | Last `complete=True` listing succeeded | Green freshness indicator |
| `refreshing` | Background refresh in progress | Spinner |
| `warning` | Last listing was incomplete or failed | Warning banner; cached rows visible |
| `stale` | Transport error during load | Error message; no freshness indicator |

`complete=False` from any provider ALWAYS produces `warning`, never `fresh`. `last_refreshed_at` is not advanced for incomplete listings.

### Phase 12 Deferred Items

| Item | Current state | Future path |
|------|---------------|-------------|
| **Cloud file upload/move/delete** | Browse is metadata-only. Write operations return `temporarily_unavailable`. | Phase 13 will implement cloud mutations (upload, move, delete) with full quota integration. |
| **Byte-level caching** | Size metadata is shown; file bytes are never fetched during browse. | Phase 14 will implement a byte cache for offline/preview access. |
| **Real-time push refresh** | Background refresh is poll-based via the UI Refresh button. | Future: WebSocket or SSE push from background task completion. |
| **pip-audit in CI** | pip-audit not in local Python 3.9 environment. | Add `pip-audit` to requirements-dev.txt and CI pipeline when moving to Python 3.12. |

---

## 8. Phase 6 Deferred Items

The following improvements are out of scope for Phase 6. They are documented here so the
next operator knows the current runbook acknowledges them:

| Item | Current state | Future path |
|------|---------------|-------------|
| **HTTPS/TLS termination** | Backend serves plain HTTP on port 8000. | Add nginx or Caddy in front; see [nginx reverse proxy pattern](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/). The runbook will gain a TLS section when this ships. |
| **Horizontal scaling** | Rate limit counters are in-memory (single-instance only). | Switch to Redis-backed counters (slowapi supports this); add uvicorn worker count config. Phase 7+ concern. |
| **CI/CD pipeline** | `docker scout cves` and Locust load tests are run manually. | GitHub Actions workflow for automated scan + load test on every PR. Deferred — no CI setup exists yet. |
| **Backup automation** | `pg_dump` and `mc mirror` run manually or via user-managed cron. | Add a `backup` service to `docker-compose.yml` that runs the cron commands inside the stack. Documented procedure above is the current state. |