kite/README.md

# DocuVault

**Version 0.3.0 — Alpha**

> **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared.

A self-hosted, multi-user document management platform with AI-powered topic classification, pluggable cloud storage backends, and a security-first architecture.

---

## Features

- **Document management** — upload PDF, DOCX, images, and plain text; full-text extraction stored in PostgreSQL
- **AI classification** — automatic topic tagging via LM Studio (local), Ollama, OpenAI, Anthropic, or any OpenAI-compatible provider (Groq, xAI, DeepSeek, OpenRouter, Gemini, Mistral)
- **Folder organisation** — hierarchical folders with move, rename, and delete; breadcrumb navigation
- **Responsive file browser** — mobile sidebar drawer, compact search/sort/new-folder controls, visible touch-safe row actions, and responsive document columns
- **Document sharing** — share by user handle with view or edit permission; "Shared with me" virtual folder; per-recipient revocation
- **Storage quota** — per-user limit enforced atomically; amber/red quota bar at 80 % / 95 %; quota decremented on delete
- **Cloud storage backends** — connect OneDrive, Google Drive, Nextcloud, or any WebDAV server as a personal storage backend; credentials encrypted with HKDF per-user keys
- **Unified connection-root browsing** — each connected account appears as a distinct top-level root navigable by connection UUID; duplicate same-provider accounts are disambiguated automatically; local and cloud files share one row and action implementation
- **Capability-aware actions** — unsupported cloud actions render gray with accessible explanations; temporarily unavailable actions render amber; all remain keyboard-focusable
- **Cloud file management** — upload files into cloud folders via a sequential queue with typed conflict resolution (keep-both/replace/skip/cancel-all); create folders with bounded collision-retry naming; rename items; move items within a connection; delete with trash-or-permanent disclosure
- **Connection health and reconnect** — per-connection health badge and error diagnostics in Settings; auto-test after connect/reconnect; Reconnect and requires-reauth banners in the shared browser; no health probe on folder navigation (D-13)
- **Authorized cloud preview** — open and preview supported binary formats (PDF/images) through DocuVault's authorized proxy; Office/Workspace formats fall back to an ownership-checked authorized download endpoint; provider credentials and raw provider URLs are never exposed
- **Auth** — email/password registration with HIBP breach check, optional TOTP 2FA, 8–10 backup codes, password reset by email, sign-out-all-devices
- **Admin panel** — user CRUD, quota assignment, per-user AI provider override, AI provider system configuration, audit log viewer with CSV export
- **Observability** — structured JSON logging via structlog, per-request correlation IDs, Loki + Promtail + Grafana stack included in Docker Compose
- **Production hardening** — non-root containers, read-only filesystems, CSP/CSRF/Origin-validation middleware, per-IP and per-account rate limiting, CVE scanning gate

---

## Architecture

```
┌─────────────────────────────────────────┐
│  Vue 3 frontend (Vite + Pinia + Tailwind) │
│  Port 5173                               │
└────────────────┬────────────────────────┘
                 │ REST / httpOnly cookie
┌────────────────▼────────────────────────┐
│  FastAPI backend (Python 3.12)          │
│  Port 8000                              │
│  ├── api/auth.py      (JWT + TOTP)      │
│  ├── api/documents.py                   │
│  ├── api/folders.py                     │
│  ├── api/shares.py                      │
│  ├── api/cloud/         (cloud backends package)  │
│  ├── api/admin.py                       │
│  └── api/audit.py                       │
└──┬──────────┬──────────┬───────────────┘
   │          │          │
   ▼          ▼          ▼
PostgreSQL  MinIO     Redis
(data)    (objects) (rate limit / TOTP replay)
              │
              └── Celery worker
                  (extract + classify tasks)
                  Celery beat
                  (daily audit export)
```

**Storage object key schema:** `{user_id}/{document_id}/{uuid4()}{ext}` — human filenames live in PostgreSQL only.

**Token flow:** Access token (15 min, ES256) in Pinia memory only. Refresh token (30 day) in an httpOnly `SameSite=Strict` cookie. Refresh rotation on every use; reuse revokes the entire token family.

---

## Stack

| Layer | Technology |
|-------|------------|
| Backend | Python 3.12, FastAPI 0.136+, SQLAlchemy 2.0 async, psycopg v3, Alembic |
| Task queue | Celery 5 + Redis, celery-beat for scheduled tasks |
| Object store | MinIO (S3-compatible) |
| Database | PostgreSQL 17 |
| Frontend | Vue 3 (Options API), Pinia, Vue Router 4, Vite, Tailwind CSS |
| Auth | PyJWT (ES256), pwdlib[argon2], pyotp (TOTP), cryptography (HKDF/AES-GCM) |
| AI providers | Anthropic, OpenAI, Ollama, LM Studio, GenericOpenAI (Groq/xAI/DeepSeek/Gemini/…) |
| Cloud backends | Google Drive, OneDrive (MSAL), Nextcloud, WebDAV |
| Observability | structlog, Loki, Promtail, Grafana |
| Rate limiting | slowapi (per-IP + per-account) |
| Load testing | Locust (`backend/load_tests/`) |

---

## Prerequisites

- Docker and Docker Compose v2
- `openssl` (for generating secrets)
- *(Optional)* LM Studio or Ollama running on the host for local AI classification

---

## Quick Start

```bash
# 1. Clone
git clone <repo-url>
cd document_scanner

# 2. Create .env from the example
cp .env.example .env

# 3. Generate required secrets
python3 -c "import secrets; print(secrets.token_hex(64))"   # → SECRET_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))" # → CLOUD_CREDS_KEY

# Generate ES256 JWT keypair (required — see JWT Key Generation section below)
python3 -c "
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization
import base64
k = ec.generate_private_key(ec.SECP256R1())
print('JWT_PRIVATE_KEY=' + base64.b64encode(k.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())).decode())
print('JWT_PUBLIC_KEY='  + base64.b64encode(k.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)).decode())
"   # → paste both lines into .env

# 4. Edit .env — at minimum set:
#   SECRET_KEY, CLOUD_CREDS_KEY, JWT_PRIVATE_KEY, JWT_PUBLIC_KEY, ADMIN_EMAIL, ADMIN_PASSWORD
#   (everything else has usable defaults for local dev)
nano .env

# 5. Start all services (migrations run automatically before backend starts)
docker compose up -d --build

# 6. Open the app
open http://localhost:5173
```

The `migrate` service runs `alembic upgrade head` automatically before the backend, Celery worker, and Celery beat start. No manual migration step is needed on a normal `docker compose up`.

The bootstrap admin account (email + password from `.env`) is created automatically on first startup if no users exist.

---

## Service URLs

| Service | URL | Notes |
|---------|-----|-------|
| Vue frontend | http://localhost:5173 | Main app |
| FastAPI backend | http://localhost:8000 | REST API |
| API health | http://localhost:8000/health | Postgres + MinIO probe |
| OpenAPI docs | http://localhost:8000/docs | Swagger UI (dev) |
| MinIO console | http://localhost:9001 | Object browser |
| Grafana | http://localhost:3000 | Log dashboard (admin/changeme) |
| Loki | http://localhost:3100/ready | Log aggregator |

---

## Environment Variables

Copy `.env.example` to `.env`. Only the fields marked **Required** must be set before first boot. See `RUNBOOK.md` for the full reference.

### Required

| Variable | Description |
|----------|-------------|
| `SECRET_KEY` | JWT signing secret — generate with `openssl rand -hex 32` |
| `JWT_PRIVATE_KEY` | Base64-encoded PEM private key for ES256 JWT signing (required) — see [JWT Key Generation](#jwt-key-generation) |
| `JWT_PUBLIC_KEY` | Base64-encoded PEM public key for ES256 JWT verification (required) — see [JWT Key Generation](#jwt-key-generation) |
| `CLOUD_CREDS_KEY` | Master key for cloud credential encryption — generate with `openssl rand -hex 16` (pad to 32 chars) |
| `ADMIN_EMAIL` | Bootstrap admin email |
| `ADMIN_PASSWORD` | Bootstrap admin password (must pass strength check) |
| `POSTGRES_PASSWORD` | PostgreSQL superuser password |
| `MINIO_ROOT_PASSWORD` | MinIO root password |
| `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` | App-level MinIO credentials; Docker Compose provisions this user and bucket policy |
| `REDIS_PASSWORD` | Redis `requirepass` password |

### Optional (sensible defaults for local dev)

| Variable | Default | Description |
|----------|---------|-------------|
| `CORS_ORIGINS` | `["http://localhost:5173"]` | JSON list of allowed origins |
| `SMTP_HOST` | *(unset)* | Leave empty to log password-reset links to stdout |
| `LOG_JSON` | `false` | Set `true` in production for structured JSON logs |
| `DEFAULT_AI_PROVIDER` | `ollama` | Used on first boot seed; overridable per-user in admin panel |
| `GOOGLE_CLIENT_ID/SECRET` | *(unset)* | Required only if using Google Drive backend |
| `ONEDRIVE_CLIENT_ID/SECRET` | *(unset)* | Required only if using OneDrive backend |

### JWT Key Generation

Phase 7.3 uses ES256 (ECDSA P-256) for JWT signing. Unlike HS256, ES256 is asymmetric — the private key signs tokens and the public key verifies them. A leaked public key cannot forge tokens.

Generate the key pair with this Python one-liner (requires `cryptography`, already in `requirements.txt`):

```bash
python3 -c "
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization
import base64
k = ec.generate_private_key(ec.SECP256R1())
priv = base64.b64encode(k.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())).decode()
pub = base64.b64encode(k.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)).decode()
print(f'JWT_PRIVATE_KEY={priv}')
print(f'JWT_PUBLIC_KEY={pub}')
"
```

Paste the two output lines into your `.env` file at the project root.

> **Warning:** Rotating these keys invalidates every active session — the startup rotation hook will bulk-revoke all refresh tokens on the next boot.

---

## Development

### Backend (local, no Docker)

```bash
cd backend
pip install -r requirements.txt -r requirements-dev.txt
# Requires a running PostgreSQL and Redis — use docker compose up postgres redis minio
uvicorn main:app --reload --port 8000
```

### Frontend (local, no Docker)

```bash
cd frontend
npm install
npm run dev          # http://localhost:5173
```

### Database migrations

```bash
# Migrations run automatically on docker compose up via the migrate service.
# To run them manually (e.g., after a git pull without restarting):
docker compose run --rm migrate

# Check current revision
docker compose run --rm migrate alembic current

# Create a new migration after changing db/models.py
docker compose exec backend alembic revision --autogenerate -m "describe change"
```

### Live reload in Docker

`docker-compose.yml` mounts `./backend:/app` so Python changes reload automatically. After changing `requirements.txt` or `Dockerfile`:

```bash
docker compose build backend celery-worker celery-beat
docker compose up -d backend celery-worker celery-beat
```

---

## Testing

```bash
# Backend — all tests
cd backend && pytest -v

# Backend — integration tests (requires live PostgreSQL + MinIO)
INTEGRATION=1 pytest -v

# Frontend
cd frontend && npm run test

# Load testing (requires a running stack + load-test user)
cd backend
pip install locust
locust -f load_tests/locustfile.py --host http://localhost:8000
```

### Test coverage by layer

| Layer | Approach |
|-------|----------|
| Service / business logic | Unit tests with mocked dependencies |
| DB queries | Integration tests against real PostgreSQL |
| API endpoints | `httpx.AsyncClient` against a real DB |
| Auth flows | Full round-trip (register → login → TOTP → refresh → revoke) |
| Security invariants | Negative tests (wrong owner → 403/404, admin block, token replay) |
| Frontend | Vitest unit tests in `frontend/tests/` |

---

## AI Providers

AI provider settings live in the `system_settings` database table and are managed through the Admin → AI Providers panel. API keys are encrypted at rest with HKDF/AES-GCM.

| Provider | Type | `provider_id` |
|----------|------|---------------|
| LM Studio | Local (OpenAI-compat) | `lmstudio` |
| Ollama | Local (OpenAI-compat) | `ollama` |
| OpenAI | Cloud | `openai` |
| Anthropic | Cloud (native) | `anthropic` |
| Groq | Cloud (OpenAI-compat) | `groq` |
| xAI Grok | Cloud (OpenAI-compat) | `xai` |
| DeepSeek | Cloud (OpenAI-compat) | `deepseek` |
| OpenRouter | Cloud (OpenAI-compat) | `openrouter` |
| Gemini (compat) | Cloud | `gemini` |
| Mistral (compat) | Cloud | `mistral` |

Documents whose classification fails are automatically retried by Celery at 30 s, 90 s, and 270 s. After three failures the document shows a red "Classification failed" badge with a "Re-analyze" button.

---

## Cloud Storage Backends

Users connect cloud storage through **Settings → Cloud Storage**. Credentials are encrypted per-user with an HKDF-derived key and never returned in API responses.

| Provider | Auth method | Notes |
|----------|-------------|-------|
| Google Drive | OAuth 2.0 | Requires `GOOGLE_CLIENT_ID/SECRET` |
| Microsoft OneDrive | OAuth 2.0 (MSAL) | Requires `ONEDRIVE_CLIENT_ID/SECRET` |
| Nextcloud | Username + password | Custom server URL; SSRF allowlist enforced; URL normalized to canonical DAV root |
| WebDAV | Username + password | Any RFC 4918 server; SSRF allowlist enforced |

Connection statuses: `ACTIVE`, `REQUIRES_REAUTH`, `ERROR`. An externally revoked OAuth token transitions to `REQUIRES_REAUTH` without a 500 error.

All four providers implement the canonical `CloudResourceAdapter.list_folder(connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing` contract. `@odata.nextLink` pagination for OneDrive is validated to stay on `graph.microsoft.com` before following.

### Connection-ID Browse API (Phase 12)

Each connected account is independently addressable by its connection UUID:

| Endpoint | Purpose |
|----------|---------|
| `GET /api/cloud/connections` | List all connections (all providers, no credentials) |
| `GET /api/cloud/connections/{id}/items` | Browse folder by connection UUID (stale-while-revalidate) |
| `PATCH /api/cloud/connections/{id}` | Rename connection display name |
| `DELETE /api/cloud/connections/{id}` | Remove connection and credentials |

### Cloud Mutation API (Phase 13)

| Endpoint | Purpose |
|----------|---------|
| `POST /api/cloud/connections/{id}/test` | Explicit connection health test (never called on navigate) |
| `POST /api/cloud/connections/{id}/reconnect` | Reconnect with credentials refresh; preserves cached metadata |
| `GET /api/cloud/connections/{id}/open/{item_id}` | Authorized open/preview proxy — never exposes raw provider URL |
| `GET /api/cloud/connections/{id}/download/{item_id}` | Authorized download fallback for unsupported preview formats |
| `POST /api/cloud/connections/{id}/upload` | Upload file; returns typed result (success/conflict/offline/reauth_required) |
| `POST /api/cloud/connections/{id}/folders` | Create folder with bounded collision-retry naming |
| `PATCH /api/cloud/connections/{id}/items/{item_id}/rename` | Rename; stale guard stops on external change |
| `POST /api/cloud/connections/{id}/items/{item_id}/move` | Move within same connection; descendant check rejects self-moves |
| `DELETE /api/cloud/connections/{id}/items/{item_id}` | Delete (trash or permanent depending on provider capability) |

Browsing returns durable cached rows immediately and schedules a background `refresh_cloud_folder` Celery task to reconcile provider changes. First-visit fetches are synchronous and bounded. All browse responses are credential-free — `credentials_enc` is never serialized in the response.

**Freshness states:** Browse responses include `freshness.refresh_state` (`fresh` | `warning` | `refreshing` | `stale`). `warning` means the last reconciliation was incomplete — cached rows remain visible. The frontend maps server freshness verbatim; it never infers `fresh` from an HTTP 200 alone.

**Phase 12.1 corrections (v0.2.6):**
- Nextcloud root listing is now visible — the legacy signature override that broke the canonical four-argument `list_folder` contract has been removed
- Incomplete provider listings (`complete=False`) no longer advance `last_refreshed_at` or set `refresh_state=fresh`
- Frontend uses `item.kind` (`folder`|`file`) and `item.provider_item_id` for navigation — `is_dir` and DocuVault `id` are no longer used for routing

**Opt-in live smoke tests (read-only):**

```bash
# Requires NEXTCLOUD_URL, NEXTCLOUD_USER, and NEXTCLOUD_APP_PASSWORD in .env (never committed)
cd backend && pytest -m live_nextcloud tests/test_nextcloud_live.py
```

The live suite is excluded from ordinary CI runs. It is read-only — PROPFIND metadata requests only. No bytes are downloaded and no provider mutations are made. Missing variables produce a safe skip.

**Phase 13 complete (v0.3.0):** Cloud file upload, rename, move, delete, and create-folder are all implemented. See Cloud file management feature above. Phase 14 (byte download/cache for offline preview and selective analysis) remains future work.

---

## Security Highlights

- **Passwords** hashed with Argon2id; HIBP breach check on registration and password change
- **JWT** signed with ES256 (ECDSA P-256); access token 15 min; refresh token 30 days in httpOnly `SameSite=Strict` cookie
- **Token fingerprinting** — access token embeds HMAC of `User-Agent + Accept-Language`; validated on every request
- **TOTP replay prevention** — used codes marked in Redis within the 90 s validity window
- **Quota** enforced by a single atomic `UPDATE … RETURNING` — no read-then-write race
- **IDOR protection** — every resource endpoint asserts `resource.user_id == current_user.id`
- **SSRF prevention** — user-supplied WebDAV/Nextcloud URLs pass a hostname allowlist
- **CSP / security headers** on every response
- **Container hardening** — non-root user (`appuser` uid 1000), read-only root filesystem, all Linux capabilities dropped
- **Audit log** — all auth events, quota violations, and admin actions logged without document content

Security gate (run before every production deploy):

```bash
bandit -r backend/                                   # zero HIGH
pip audit                                            # zero critical/high CVEs
npm audit --audit-level=high                         # zero high/critical
docker scout cves local://docuvault-backend:latest --only-severity critical --exit-code
```

See `SECURITY.md` for the full threat model and `RUNBOOK.md` for operational procedures.

---

## Observability

Structured JSON logs are emitted to stdout and collected by Promtail → Loki → Grafana.

Every request gets an `X-Correlation-ID` header bound to all log lines for that request. Log fields include `correlation_id`, `path`, `method`, `status_code`, `duration_ms`, and `user_id` (when authenticated).

Access Grafana at http://localhost:3000 (default credentials: `admin` / `changeme` — change via `GRAFANA_ADMIN_PASSWORD` in `.env`). Add a Loki datasource pointing to `http://loki:3100` to query logs.

---

## Project Structure

```
.
├── backend/
│   ├── ai/              # Provider implementations + classifier
│   ├── api/             # FastAPI routers (auth, documents, folders, shares, cloud, admin, audit)
│   ├── db/              # SQLAlchemy models + async session
│   ├── deps/            # FastAPI dependency injection (auth, db, utils)
│   ├── migrations/      # Alembic migration versions
│   ├── services/        # Business logic (auth, classifier, storage, email, audit, rate limiting)
│   ├── storage/         # StorageBackend ABC + MinIO/Google/OneDrive/Nextcloud/WebDAV backends
│   ├── tasks/           # Celery tasks (document processing, email, audit export)
│   ├── load_tests/      # Locust load test scripts
│   ├── tests/           # pytest test suite
│   ├── main.py          # FastAPI application factory + middleware
│   └── config.py        # Pydantic settings
├── frontend/
│   └── src/
│       ├── api/         # Axios API client modules
│       ├── components/  # UI components (documents, folders, cloud, admin, sharing, layout)
│       ├── stores/      # Pinia stores (auth, documents, folders, cloud connections)
│       ├── views/       # Page-level view components
│       └── utils/       # formatters.js (formatDate, formatSize, providerColor, …)
├── docker/
│   ├── postgres/        # initdb.d — user + role bootstrap SQL
│   └── loki/            # Loki + Promtail config
├── docker-compose.yml
├── .env.example
├── RUNBOOK.md           # Operational runbook (env vars, backup, on-call, failure modes)
└── SECURITY.md          # Threat model and security gate results
```

---

## Contributing

1. All new endpoints require at least one integration test
2. Run `pytest -v` and `npm run test` — zero failures required
3. Run `bandit -r backend/` — zero HIGH findings
4. No raw SQL string interpolation; no `pickle`; no `eval`; no `innerHTML` with user data
5. Service layer raises `ValueError` (never `HTTPException`); only the router layer raises `HTTPException`
6. Before writing a new helper, check the shared module map in `CLAUDE.md`