DocuVault

Version 0.4.0 — Alpha

Not production-ready. DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared.

A self-hosted, multi-user document management platform with AI-powered topic classification, pluggable cloud storage backends, and a security-first architecture.

---

Features

• Document management — upload PDF, DOCX, images, and plain text; full-text extraction stored in PostgreSQL
• AI classification — automatic topic tagging via LM Studio (local), Ollama, OpenAI, Anthropic, or any OpenAI-compatible provider (Groq, xAI, DeepSeek, OpenRouter, Gemini, Mistral)
• Folder organisation — hierarchical folders with move, rename, and delete; breadcrumb navigation
• Responsive file browser — mobile sidebar drawer, compact search/sort/new-folder controls, visible touch-safe row actions, and responsive document columns
• Document sharing — share by user handle with view or edit permission; "Shared with me" virtual folder; per-recipient revocation
• Storage quota — per-user limit enforced atomically; amber/red quota bar at 80 % / 95 %; quota decremented on delete
• Cloud storage backends — connect OneDrive, Google Drive, Nextcloud, or any WebDAV server as a personal storage backend; credentials encrypted with HKDF per-user keys
• Unified connection-root browsing — each connected account appears as a distinct top-level root navigable by connection UUID; duplicate same-provider accounts are disambiguated automatically; local and cloud files share one row and action implementation
• Capability-aware actions — unsupported cloud actions render gray with accessible explanations; temporarily unavailable actions render amber; all remain keyboard-focusable
• Cloud file management — upload files into cloud folders via a sequential queue with typed conflict resolution (keep-both/replace/skip/cancel-all); create folders with bounded collision-retry naming; rename items; move items within a connection; delete with trash-or-permanent disclosure
• Connection health and reconnect — per-connection health badge and error diagnostics in Settings; auto-test after connect/reconnect; Reconnect and requires-reauth banners in the shared browser; no health probe on folder navigation (D-13)
• Authorized cloud preview — open and preview supported binary formats (PDF/images) through DocuVault's authorized proxy; Office/Workspace formats fall back to an ownership-checked authorized download endpoint; provider credentials and raw provider URLs are never exposed
• Selective cloud analysis — analyze individual files, multi-selections, folder trees, or entire connections after reviewing scope estimates; background Celery jobs download, extract, and classify cloud files without mutating the provider; per-item cancel, skip, and retry controls with aggregate and expandable queue progress
• Cloud byte cache — downloaded file bytes are stored in MinIO with LRU eviction, per-user quota accounting, and ownership-scoped pinning; open/preview/download reuse cached bytes transparently; configurable cache limit, progress detail level, and failure behavior in Settings
• Auth — email/password registration with HIBP breach check, optional TOTP 2FA, 8–10 backup codes, password reset by email, sign-out-all-devices
• Admin panel — user CRUD, quota assignment, per-user AI provider override, AI provider system configuration, audit log viewer with CSV export
• Observability — structured JSON logging via structlog, per-request correlation IDs, Loki + Promtail + Grafana stack included in Docker Compose
• Production hardening — non-root containers, read-only filesystems, CSP/CSRF/Origin-validation middleware, per-IP and per-account rate limiting, CVE scanning gate

---

Architecture

┌─────────────────────────────────────────┐
│  Vue 3 frontend (Vite + Pinia + Tailwind) │
│  Port 5173                               │
└────────────────┬────────────────────────┘
                 │ REST / httpOnly cookie
┌────────────────▼────────────────────────┐
│  FastAPI backend (Python 3.12)          │
│  Port 8000                              │
│  ├── api/auth.py      (JWT + TOTP)      │
│  ├── api/documents.py                   │
│  ├── api/folders.py                     │
│  ├── api/shares.py                      │
│  ├── api/cloud/         (cloud backends package)  │
│  ├── api/admin.py                       │
│  └── api/audit.py                       │
└──┬──────────┬──────────┬───────────────┘
   │          │          │
   ▼          ▼          ▼
PostgreSQL  MinIO     Redis
(data)    (objects) (rate limit / TOTP replay)
              │
              └── Celery worker
                  (extract + classify tasks)
                  Celery beat
                  (daily audit export)

Storage object key schema: {user_id}/{document_id}/{uuid4()}{ext} — human filenames live in PostgreSQL only.

Token flow: Access token (15 min, ES256) in Pinia memory only. Refresh token (30 day) in an httpOnly SameSite=Strict cookie. Refresh rotation on every use; reuse revokes the entire token family.

---

Stack

Layer	Technology
Backend	Python 3.12, FastAPI 0.136+, SQLAlchemy 2.0 async, psycopg v3, Alembic
Task queue	Celery 5 + Redis, celery-beat for scheduled tasks
Object store	MinIO (S3-compatible)
Database	PostgreSQL 17
Frontend	Vue 3 (Options API), Pinia, Vue Router 4, Vite, Tailwind CSS
Auth	PyJWT (ES256), pwdlib[argon2], pyotp (TOTP), cryptography (HKDF/AES-GCM)
AI providers	Anthropic, OpenAI, Ollama, LM Studio, GenericOpenAI (Groq/xAI/DeepSeek/Gemini/…)
Cloud backends	Google Drive, OneDrive (MSAL), Nextcloud, WebDAV
Observability	structlog, Loki, Promtail, Grafana
Rate limiting	slowapi (per-IP + per-account)
Load testing	Locust (backend/load_tests/)

---

Prerequisites

• Docker and Docker Compose v2
• openssl (for generating secrets)
• (Optional) LM Studio or Ollama running on the host for local AI classification

---

Quick Start

# 1. Clone
git clone <repo-url>
cd document_scanner

# 2. Create .env from the example
cp .env.example .env

# 3. Generate required secrets
python3 -c "import secrets; print(secrets.token_hex(64))"   # → SECRET_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))" # → CLOUD_CREDS_KEY

# Generate ES256 JWT keypair (required — see JWT Key Generation section below)
python3 -c "
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization
import base64
k = ec.generate_private_key(ec.SECP256R1())
print('JWT_PRIVATE_KEY=' + base64.b64encode(k.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())).decode())
print('JWT_PUBLIC_KEY='  + base64.b64encode(k.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)).decode())
"   # → paste both lines into .env

# 4. Edit .env — at minimum set:
#   SECRET_KEY, CLOUD_CREDS_KEY, JWT_PRIVATE_KEY, JWT_PUBLIC_KEY, ADMIN_EMAIL, ADMIN_PASSWORD
#   (everything else has usable defaults for local dev)
nano .env

# 5. Start all services (migrations run automatically before backend starts)
docker compose up -d --build

# 6. Open the app
open http://localhost:5173

The migrate service runs alembic upgrade head automatically before the backend, Celery worker, and Celery beat start. No manual migration step is needed on a normal docker compose up.

The bootstrap admin account (email + password from .env) is created automatically on first startup if no users exist.

---

Service URLs

Service	URL	Notes
Vue frontend	http://localhost:5173	Main app
FastAPI backend	http://localhost:8000	REST API
API health	http://localhost:8000/health	Postgres + MinIO probe
OpenAPI docs	http://localhost:8000/docs	Swagger UI (dev)
MinIO console	http://localhost:9001	Object browser
Grafana	http://localhost:3000	Log dashboard (admin/changeme)
Loki	http://localhost:3100/ready	Log aggregator

---

Environment Variables

Copy .env.example to .env. Only the fields marked Required must be set before first boot. See RUNBOOK.md for the full reference.

Required

Variable	Description
SECRET_KEY	JWT signing secret — generate with openssl rand -hex 32
JWT_PRIVATE_KEY	Base64-encoded PEM private key for ES256 JWT signing (required) — see JWT Key Generation
JWT_PUBLIC_KEY	Base64-encoded PEM public key for ES256 JWT verification (required) — see JWT Key Generation
CLOUD_CREDS_KEY	Master key for cloud credential encryption — generate with openssl rand -hex 16 (pad to 32 chars)
ADMIN_EMAIL	Bootstrap admin email
ADMIN_PASSWORD	Bootstrap admin password (must pass strength check)
POSTGRES_PASSWORD	PostgreSQL superuser password
MINIO_ROOT_PASSWORD	MinIO root password
MINIO_ACCESS_KEY / MINIO_SECRET_KEY	App-level MinIO credentials; Docker Compose provisions this user and bucket policy
REDIS_PASSWORD	Redis requirepass password

Optional (sensible defaults for local dev)

Variable	Default	Description
CORS_ORIGINS	["http://localhost:5173"]	JSON list of allowed origins
SMTP_HOST	(unset)	Leave empty to log password-reset links to stdout
LOG_JSON	false	Set true in production for structured JSON logs
DEFAULT_AI_PROVIDER	ollama	Used on first boot seed; overridable per-user in admin panel
GOOGLE_CLIENT_ID/SECRET	(unset)	Required only if using Google Drive backend
ONEDRIVE_CLIENT_ID/SECRET	(unset)	Required only if using OneDrive backend

JWT Key Generation

Phase 7.3 uses ES256 (ECDSA P-256) for JWT signing. Unlike HS256, ES256 is asymmetric — the private key signs tokens and the public key verifies them. A leaked public key cannot forge tokens.

Generate the key pair with this Python one-liner (requires cryptography, already in requirements.txt):

python3 -c "
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization
import base64
k = ec.generate_private_key(ec.SECP256R1())
priv = base64.b64encode(k.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())).decode()
pub = base64.b64encode(k.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo)).decode()
print(f'JWT_PRIVATE_KEY={priv}')
print(f'JWT_PUBLIC_KEY={pub}')
"

Paste the two output lines into your .env file at the project root.

Warning: Rotating these keys invalidates every active session — the startup rotation hook will bulk-revoke all refresh tokens on the next boot.

---

Development

Backend (local, no Docker)

cd backend
pip install -r requirements.txt -r requirements-dev.txt
# Requires a running PostgreSQL and Redis — use docker compose up postgres redis minio
uvicorn main:app --reload --port 8000

Frontend (local, no Docker)

cd frontend
npm install
npm run dev          # http://localhost:5173

Database migrations

# Migrations run automatically on docker compose up via the migrate service.
# To run them manually (e.g., after a git pull without restarting):
docker compose run --rm migrate

# Check current revision
docker compose run --rm migrate alembic current

# Create a new migration after changing db/models.py
docker compose exec backend alembic revision --autogenerate -m "describe change"

Live reload in Docker

docker-compose.yml mounts ./backend:/app so Python changes reload automatically. After changing requirements.txt or Dockerfile:

docker compose build backend celery-worker celery-beat
docker compose up -d backend celery-worker celery-beat

---

Testing

# Backend — all tests
cd backend && pytest -v

# Backend — integration tests (requires live PostgreSQL + MinIO)
INTEGRATION=1 pytest -v

# Frontend
cd frontend && npm run test

# Load testing (requires a running stack + load-test user)
cd backend
pip install locust
locust -f load_tests/locustfile.py --host http://localhost:8000

Test coverage by layer

Layer	Approach
Service / business logic	Unit tests with mocked dependencies
DB queries	Integration tests against real PostgreSQL
API endpoints	httpx.AsyncClient against a real DB
Auth flows	Full round-trip (register → login → TOTP → refresh → revoke)
Security invariants	Negative tests (wrong owner → 403/404, admin block, token replay)
Frontend	Vitest unit tests in frontend/tests/

---

AI Providers

AI provider settings live in the system_settings database table and are managed through the Admin → AI Providers panel. API keys are encrypted at rest with HKDF/AES-GCM.

Provider	Type	provider_id
LM Studio	Local (OpenAI-compat)	lmstudio
Ollama	Local (OpenAI-compat)	ollama
OpenAI	Cloud	openai
Anthropic	Cloud (native)	anthropic
Groq	Cloud (OpenAI-compat)	groq
xAI Grok	Cloud (OpenAI-compat)	xai
DeepSeek	Cloud (OpenAI-compat)	deepseek
OpenRouter	Cloud (OpenAI-compat)	openrouter
Gemini (compat)	Cloud	gemini
Mistral (compat)	Cloud	mistral

Documents whose classification fails are automatically retried by Celery at 30 s, 90 s, and 270 s. After three failures the document shows a red "Classification failed" badge with a "Re-analyze" button.

---

Cloud Storage Backends

Users connect cloud storage through Settings → Cloud Storage. Credentials are encrypted per-user with an HKDF-derived key and never returned in API responses.

Provider	Auth method	Notes
Google Drive	OAuth 2.0	Requires GOOGLE_CLIENT_ID/SECRET
Microsoft OneDrive	OAuth 2.0 (MSAL)	Requires ONEDRIVE_CLIENT_ID/SECRET
Nextcloud	Username + password	Custom server URL; SSRF allowlist enforced; URL normalized to canonical DAV root
WebDAV	Username + password	Any RFC 4918 server; SSRF allowlist enforced

Connection statuses: ACTIVE, REQUIRES_REAUTH, ERROR. An externally revoked OAuth token transitions to REQUIRES_REAUTH without a 500 error.

All four providers implement the canonical CloudResourceAdapter.list_folder(connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing contract. @odata.nextLink pagination for OneDrive is validated to stay on graph.microsoft.com before following.

Connection-ID Browse API (Phase 12)

Each connected account is independently addressable by its connection UUID:

Endpoint	Purpose
GET /api/cloud/connections	List all connections (all providers, no credentials)
GET /api/cloud/connections/{id}/items	Browse folder by connection UUID (stale-while-revalidate)
PATCH /api/cloud/connections/{id}	Rename connection display name
DELETE /api/cloud/connections/{id}	Remove connection and credentials

Cloud Mutation API (Phase 13)

Endpoint	Purpose
POST /api/cloud/connections/{id}/test	Explicit connection health test (never called on navigate)
POST /api/cloud/connections/{id}/reconnect	Reconnect with credentials refresh; preserves cached metadata
GET /api/cloud/connections/{id}/open/{item_id}	Authorized open/preview proxy — never exposes raw provider URL
GET /api/cloud/connections/{id}/download/{item_id}	Authorized download fallback for unsupported preview formats
POST /api/cloud/connections/{id}/upload	Upload file; returns typed result (success/conflict/offline/reauth_required)
POST /api/cloud/connections/{id}/folders	Create folder with bounded collision-retry naming
PATCH /api/cloud/connections/{id}/items/{item_id}/rename	Rename; stale guard stops on external change
POST /api/cloud/connections/{id}/items/{item_id}/move	Move within same connection; descendant check rejects self-moves
DELETE /api/cloud/connections/{id}/items/{item_id}	Delete (trash or permanent depending on provider capability)

Browsing returns durable cached rows immediately and schedules a background refresh_cloud_folder Celery task to reconcile provider changes. First-visit fetches are synchronous and bounded. All browse responses are credential-free — credentials_enc is never serialized in the response.

Freshness states: Browse responses include freshness.refresh_state (fresh | warning | refreshing | stale). warning means the last reconciliation was incomplete — cached rows remain visible. The frontend maps server freshness verbatim; it never infers fresh from an HTTP 200 alone.

Phase 12.1 corrections (v0.2.6):

• Nextcloud root listing is now visible — the legacy signature override that broke the canonical four-argument list_folder contract has been removed
• Incomplete provider listings (complete=False) no longer advance last_refreshed_at or set refresh_state=fresh
• Frontend uses item.kind (folder|file) and item.provider_item_id for navigation — is_dir and DocuVault id are no longer used for routing

Opt-in live smoke tests (read-only):

# Requires NEXTCLOUD_URL, NEXTCLOUD_USER, and NEXTCLOUD_APP_PASSWORD in .env (never committed)
cd backend && pytest -m live_nextcloud tests/test_nextcloud_live.py

The live suite is excluded from ordinary CI runs. It is read-only — PROPFIND metadata requests only. No bytes are downloaded and no provider mutations are made. Missing variables produce a safe skip.

Phase 13 complete (v0.3.0): Cloud file upload, rename, move, delete, and create-folder are all implemented. See Cloud file management feature above.

Phase 14 complete (v0.4.0): Selective cloud analysis and byte cache are implemented. See Selective cloud analysis and Cloud byte cache features above. Phase 15 (unified smart search) is next.

Analysis API (Phase 14)

Endpoint	Purpose
POST /api/cloud/analysis/connections/{id}/estimate	Scope estimate — file/selection/folder/connection; no bytes downloaded
POST /api/cloud/analysis/connections/{id}/jobs	Enqueue analysis job; returns job_id
GET /api/cloud/analysis/jobs	List jobs (optional connection_id/status filter)
GET /api/cloud/analysis/jobs/{id}	Job status (?detail=true for per-stage counts)
POST /api/cloud/analysis/jobs/{id}/cancel	Cancel batch
POST /api/cloud/analysis/jobs/{id}/items/{iid}/cancel	Cancel single item
POST /api/cloud/analysis/jobs/{id}/items/{iid}/skip	Skip queued/failed item
POST /api/cloud/analysis/jobs/{id}/items/{iid}/retry	Retry failed item
GET /api/cloud/analysis/cache	Aggregate cache usage and settings
PATCH /api/cloud/analysis/cache/settings	Update cache limit, progress detail, failure behavior

---

Security Highlights

• Passwords hashed with Argon2id; HIBP breach check on registration and password change
• JWT signed with ES256 (ECDSA P-256); access token 15 min; refresh token 30 days in httpOnly SameSite=Strict cookie
• Token fingerprinting — access token embeds HMAC of User-Agent + Accept-Language; validated on every request
• TOTP replay prevention — used codes marked in Redis within the 90 s validity window
• Quota enforced by a single atomic UPDATE … RETURNING — no read-then-write race
• IDOR protection — every resource endpoint asserts resource.user_id == current_user.id
• SSRF prevention — user-supplied WebDAV/Nextcloud URLs pass a hostname allowlist
• CSP / security headers on every response
• Container hardening — non-root user (appuser uid 1000), read-only root filesystem, all Linux capabilities dropped
• Audit log — all auth events, quota violations, and admin actions logged without document content

Security gate (run before every production deploy):

bandit -r backend/                                   # zero HIGH
pip audit                                            # zero critical/high CVEs
npm audit --audit-level=high                         # zero high/critical
docker scout cves local://docuvault-backend:latest --only-severity critical --exit-code

See SECURITY.md for the full threat model and RUNBOOK.md for operational procedures.

---

Observability

Structured JSON logs are emitted to stdout and collected by Promtail → Loki → Grafana.

Every request gets an X-Correlation-ID header bound to all log lines for that request. Log fields include correlation_id, path, method, status_code, duration_ms, and user_id (when authenticated).

Access Grafana at http://localhost:3000 (default credentials: admin / changeme — change via GRAFANA_ADMIN_PASSWORD in .env). Add a Loki datasource pointing to http://loki:3100 to query logs.

---

Project Structure

.
├── backend/
│   ├── ai/              # Provider implementations + classifier
│   ├── api/             # FastAPI routers (auth, documents, folders, shares, cloud, admin, audit)
│   ├── db/              # SQLAlchemy models + async session
│   ├── deps/            # FastAPI dependency injection (auth, db, utils)
│   ├── migrations/      # Alembic migration versions
│   ├── services/        # Business logic (auth, classifier, storage, email, audit, rate limiting)
│   ├── storage/         # StorageBackend ABC + MinIO/Google/OneDrive/Nextcloud/WebDAV backends
│   ├── tasks/           # Celery tasks (document processing, email, audit export)
│   ├── load_tests/      # Locust load test scripts
│   ├── tests/           # pytest test suite
│   ├── main.py          # FastAPI application factory + middleware
│   └── config.py        # Pydantic settings
├── frontend/
│   └── src/
│       ├── api/         # Axios API client modules
│       ├── components/  # UI components (documents, folders, cloud, admin, sharing, layout)
│       ├── stores/      # Pinia stores (auth, documents, folders, cloud connections)
│       ├── views/       # Page-level view components
│       └── utils/       # formatters.js (formatDate, formatSize, providerColor, …)
├── docker/
│   ├── postgres/        # initdb.d — user + role bootstrap SQL
│   └── loki/            # Loki + Promtail config
├── docker-compose.yml
├── .env.example
├── RUNBOOK.md           # Operational runbook (env vars, backup, on-call, failure modes)
└── SECURITY.md          # Threat model and security gate results

---

Contributing

1. All new endpoints require at least one integration test
2. Run pytest -v and npm run test — zero failures required
3. Run bandit -r backend/ — zero HIGH findings
4. No raw SQL string interpolation; no pickle; no eval; no innerHTML with user data
5. Service layer raises ValueError (never HTTPException); only the router layer raises HTTPException
6. Before writing a new helper, check the shared module map in CLAUDE.md