docs(readme): add project README for v0.1 alpha

Covers features, architecture diagram, token flow, stack, environment variables, quick-start, running tests, cloud backend setup, AI provider configuration, and alpha status disclaimer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 14:44:54 +02:00
parent 14c1bf437b
commit 6ec2748b84
1 changed files with 349 additions and 0 deletions
@@ -0,0 +1,349 @@
+# DocuVault
+
+**Version 0.1.0 — Alpha**
+
+> **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared.
+
+A self-hosted, multi-user document management platform with AI-powered topic classification, pluggable cloud storage backends, and a security-first architecture.
+
+---
+
+## Features
+
+- **Document management** — upload PDF, DOCX, images, and plain text; full-text extraction stored in PostgreSQL
+- **AI classification** — automatic topic tagging via LM Studio (local), Ollama, OpenAI, Anthropic, or any OpenAI-compatible provider (Groq, xAI, DeepSeek, OpenRouter, Gemini, Mistral)
+- **Folder organisation** — hierarchical folders with move, rename, and delete; breadcrumb navigation
+- **Document sharing** — share by user handle with view or edit permission; "Shared with me" virtual folder; per-recipient revocation
+- **Storage quota** — per-user limit enforced atomically; amber/red quota bar at 80 % / 95 %; quota decremented on delete
+- **Cloud storage backends** — connect OneDrive, Google Drive, Nextcloud, or any WebDAV server as a personal storage backend; credentials encrypted with HKDF per-user keys
+- **Auth** — email/password registration with HIBP breach check, optional TOTP 2FA, 8–10 backup codes, password reset by email, sign-out-all-devices
+- **Admin panel** — user CRUD, quota assignment, per-user AI provider override, AI provider system configuration, audit log viewer with CSV export
+- **Observability** — structured JSON logging via structlog, per-request correlation IDs, Loki + Promtail + Grafana stack included in Docker Compose
+- **Production hardening** — non-root containers, read-only filesystems, CSP/CSRF/Origin-validation middleware, per-IP and per-account rate limiting, CVE scanning gate
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────┐
+│  Vue 3 frontend (Vite + Pinia + Tailwind) │
+│  Port 5173                               │
+└────────────────┬────────────────────────┘
+                 │ REST / httpOnly cookie
+┌────────────────▼────────────────────────┐
+│  FastAPI backend (Python 3.12)          │
+│  Port 8000                              │
+│  ├── api/auth.py      (JWT + TOTP)      │
+│  ├── api/documents.py                   │
+│  ├── api/folders.py                     │
+│  ├── api/shares.py                      │
+│  ├── api/cloud.py     (cloud backends)  │
+│  ├── api/admin.py                       │
+│  └── api/audit.py                       │
+└──┬──────────┬──────────┬───────────────┘
+   │          │          │
+   ▼          ▼          ▼
+PostgreSQL  MinIO     Redis
+(data)    (objects) (rate limit / TOTP replay)
+              │
+              └── Celery worker
+                  (extract + classify tasks)
+                  Celery beat
+                  (daily audit export)
+```
+
+**Storage object key schema:** `{user_id}/{document_id}/{uuid4()}{ext}` — human filenames live in PostgreSQL only.
+
+**Token flow:** Access token (15 min, ES256) in Pinia memory only. Refresh token (30 day) in an httpOnly `SameSite=Strict` cookie. Refresh rotation on every use; reuse revokes the entire token family.
+
+---
+
+## Stack
+
+| Layer | Technology |
+|-------|------------|
+| Backend | Python 3.12, FastAPI 0.136+, SQLAlchemy 2.0 async, psycopg v3, Alembic |
+| Task queue | Celery 5 + Redis, celery-beat for scheduled tasks |
+| Object store | MinIO (S3-compatible) |
+| Database | PostgreSQL 17 |
+| Frontend | Vue 3 (Options API), Pinia, Vue Router 4, Vite, Tailwind CSS |
+| Auth | PyJWT (ES256), pwdlib[argon2], pyotp (TOTP), cryptography (HKDF/AES-GCM) |
+| AI providers | Anthropic, OpenAI, Ollama, LM Studio, GenericOpenAI (Groq/xAI/DeepSeek/Gemini/…) |
+| Cloud backends | Google Drive, OneDrive (MSAL), Nextcloud, WebDAV |
+| Observability | structlog, Loki, Promtail, Grafana |
+| Rate limiting | slowapi (per-IP + per-account) |
+| Load testing | Locust (`backend/load_tests/`) |
+
+---
+
+## Prerequisites
+
+- Docker and Docker Compose v2
+- `openssl` (for generating secrets)
+- *(Optional)* LM Studio or Ollama running on the host for local AI classification
+
+---
+
+## Quick Start
+
+```bash
+# 1. Clone
+git clone <repo-url>
+cd document_scanner
+
+# 2. Create .env from the example
+cp .env.example .env
+
+# 3. Generate required secrets
+python3 -c "import secrets; print(secrets.token_hex(64))"   # → SECRET_KEY
+python3 -c "import secrets; print(secrets.token_urlsafe(32))" # → CLOUD_CREDS_KEY
+
+# 4. Edit .env — at minimum set:
+#   SECRET_KEY, CLOUD_CREDS_KEY, ADMIN_EMAIL, ADMIN_PASSWORD
+#   (everything else has usable defaults for local dev)
+nano .env
+
+# 5. Start all services
+docker compose up -d --build
+
+# 6. Run database migrations
+docker compose exec backend alembic upgrade head
+
+# 7. Open the app
+open http://localhost:5173
+```
+
+The bootstrap admin account (email + password from `.env`) is created automatically on first startup if no users exist.
+
+---
+
+## Service URLs
+
+| Service | URL | Notes |
+|---------|-----|-------|
+| Vue frontend | http://localhost:5173 | Main app |
+| FastAPI backend | http://localhost:8000 | REST API |
+| API health | http://localhost:8000/health | Postgres + MinIO probe |
+| OpenAPI docs | http://localhost:8000/docs | Swagger UI (dev) |
+| MinIO console | http://localhost:9001 | Object browser |
+| Grafana | http://localhost:3000 | Log dashboard (admin/changeme) |
+| Loki | http://localhost:3100/ready | Log aggregator |
+
+---
+
+## Environment Variables
+
+Copy `.env.example` to `.env`. Only the fields marked **Required** must be set before first boot. See `RUNBOOK.md` for the full reference.
+
+### Required
+
+| Variable | Description |
+|----------|-------------|
+| `SECRET_KEY` | JWT signing secret — generate with `openssl rand -hex 32` |
+| `CLOUD_CREDS_KEY` | Master key for cloud credential encryption — generate with `openssl rand -hex 16` (pad to 32 chars) |
+| `ADMIN_EMAIL` | Bootstrap admin email |
+| `ADMIN_PASSWORD` | Bootstrap admin password (must pass strength check) |
+| `POSTGRES_PASSWORD` | PostgreSQL superuser password |
+| `MINIO_ROOT_PASSWORD` | MinIO root password |
+| `REDIS_PASSWORD` | Redis `requirepass` password |
+
+### Optional (sensible defaults for local dev)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CORS_ORIGINS` | `http://localhost:5173` | Comma-separated allowed origins |
+| `SMTP_HOST` | *(unset)* | Leave empty to log password-reset links to stdout |
+| `LOG_JSON` | `false` | Set `true` in production for structured JSON logs |
+| `DEFAULT_AI_PROVIDER` | `ollama` | Used on first boot seed; overridable per-user in admin panel |
+| `GOOGLE_CLIENT_ID/SECRET` | *(unset)* | Required only if using Google Drive backend |
+| `ONEDRIVE_CLIENT_ID/SECRET` | *(unset)* | Required only if using OneDrive backend |
+
+---
+
+## Development
+
+### Backend (local, no Docker)
+
+```bash
+cd backend
+pip install -r requirements.txt -r requirements-dev.txt
+# Requires a running PostgreSQL and Redis — use docker compose up postgres redis minio
+uvicorn main:app --reload --port 8000
+```
+
+### Frontend (local, no Docker)
+
+```bash
+cd frontend
+npm install
+npm run dev          # http://localhost:5173
+```
+
+### Database migrations
+
+```bash
+# Apply all pending migrations
+docker compose exec backend alembic upgrade head
+
+# Create a new migration after changing db/models.py
+docker compose exec backend alembic revision --autogenerate -m "describe change"
+```
+
+### Live reload in Docker
+
+`docker-compose.yml` mounts `./backend:/app` so Python changes reload automatically. After changing `requirements.txt` or `Dockerfile`:
+
+```bash
+docker compose build backend celery-worker celery-beat
+docker compose up -d backend celery-worker celery-beat
+```
+
+---
+
+## Testing
+
+```bash
+# Backend — all tests
+cd backend && pytest -v
+
+# Backend — integration tests (requires live PostgreSQL + MinIO)
+INTEGRATION=1 pytest -v
+
+# Frontend
+cd frontend && npm run test
+
+# Load testing (requires a running stack + load-test user)
+cd backend
+pip install locust
+locust -f load_tests/locustfile.py --host http://localhost:8000
+```
+
+### Test coverage by layer
+
+| Layer | Approach |
+|-------|----------|
+| Service / business logic | Unit tests with mocked dependencies |
+| DB queries | Integration tests against real PostgreSQL |
+| API endpoints | `httpx.AsyncClient` against a real DB |
+| Auth flows | Full round-trip (register → login → TOTP → refresh → revoke) |
+| Security invariants | Negative tests (wrong owner → 403/404, admin block, token replay) |
+| Frontend | Vitest unit tests in `frontend/tests/` |
+
+---
+
+## AI Providers
+
+AI provider settings live in the `system_settings` database table and are managed through the Admin → AI Providers panel. API keys are encrypted at rest with HKDF/AES-GCM.
+
+| Provider | Type | `provider_id` |
+|----------|------|---------------|
+| LM Studio | Local (OpenAI-compat) | `lmstudio` |
+| Ollama | Local (OpenAI-compat) | `ollama` |
+| OpenAI | Cloud | `openai` |
+| Anthropic | Cloud (native) | `anthropic` |
+| Groq | Cloud (OpenAI-compat) | `groq` |
+| xAI Grok | Cloud (OpenAI-compat) | `xai` |
+| DeepSeek | Cloud (OpenAI-compat) | `deepseek` |
+| OpenRouter | Cloud (OpenAI-compat) | `openrouter` |
+| Gemini (compat) | Cloud | `gemini` |
+| Mistral (compat) | Cloud | `mistral` |
+
+Documents whose classification fails are automatically retried by Celery at 30 s, 90 s, and 270 s. After three failures the document shows a red "Classification failed" badge with a "Re-analyze" button.
+
+---
+
+## Cloud Storage Backends
+
+Users connect cloud storage through **Settings → Cloud Storage**. Credentials are encrypted per-user with an HKDF-derived key and never returned in API responses.
+
+| Provider | Auth method | Notes |
+|----------|-------------|-------|
+| Google Drive | OAuth 2.0 | Requires `GOOGLE_CLIENT_ID/SECRET` |
+| Microsoft OneDrive | OAuth 2.0 (MSAL) | Requires `ONEDRIVE_CLIENT_ID/SECRET` |
+| Nextcloud | Username + password | Custom server URL; SSRF allowlist enforced |
+| WebDAV | Username + password | Any RFC 4918 server; SSRF allowlist enforced |
+
+Connection statuses: `ACTIVE`, `REQUIRES_REAUTH`, `ERROR`. An externally revoked OAuth token transitions to `REQUIRES_REAUTH` without a 500 error.
+
+---
+
+## Security Highlights
+
+- **Passwords** hashed with Argon2id; HIBP breach check on registration and password change
+- **JWT** signed with ES256 (ECDSA P-256); access token 15 min; refresh token 30 days in httpOnly `SameSite=Strict` cookie
+- **Token fingerprinting** — access token embeds HMAC of `User-Agent + Accept-Language`; validated on every request
+- **TOTP replay prevention** — used codes marked in Redis within the 90 s validity window
+- **Quota** enforced by a single atomic `UPDATE … RETURNING` — no read-then-write race
+- **IDOR protection** — every resource endpoint asserts `resource.user_id == current_user.id`
+- **SSRF prevention** — user-supplied WebDAV/Nextcloud URLs pass a hostname allowlist
+- **CSP / security headers** on every response
+- **Container hardening** — non-root user (`appuser` uid 1000), read-only root filesystem, all Linux capabilities dropped
+- **Audit log** — all auth events, quota violations, and admin actions logged without document content
+
+Security gate (run before every production deploy):
+
+```bash
+bandit -r backend/                                   # zero HIGH
+pip audit                                            # zero critical/high CVEs
+npm audit --audit-level=high                         # zero high/critical
+docker scout cves local://docuvault-backend:latest --only-severity critical --exit-code
+```
+
+See `SECURITY.md` for the full threat model and `RUNBOOK.md` for operational procedures.
+
+---
+
+## Observability
+
+Structured JSON logs are emitted to stdout and collected by Promtail → Loki → Grafana.
+
+Every request gets an `X-Correlation-ID` header bound to all log lines for that request. Log fields include `correlation_id`, `path`, `method`, `status_code`, `duration_ms`, and `user_id` (when authenticated).
+
+Access Grafana at http://localhost:3000 (default credentials: `admin` / `changeme` — change via `GRAFANA_ADMIN_PASSWORD` in `.env`). Add a Loki datasource pointing to `http://loki:3100` to query logs.
+
+---
+
+## Project Structure
+
+```
+.
+├── backend/
+│   ├── ai/              # Provider implementations + classifier
+│   ├── api/             # FastAPI routers (auth, documents, folders, shares, cloud, admin, audit)
+│   ├── db/              # SQLAlchemy models + async session
+│   ├── deps/            # FastAPI dependency injection (auth, db, utils)
+│   ├── migrations/      # Alembic migration versions
+│   ├── services/        # Business logic (auth, classifier, storage, email, audit, rate limiting)
+│   ├── storage/         # StorageBackend ABC + MinIO/Google/OneDrive/Nextcloud/WebDAV backends
+│   ├── tasks/           # Celery tasks (document processing, email, audit export)
+│   ├── load_tests/      # Locust load test scripts
+│   ├── tests/           # pytest test suite
+│   ├── main.py          # FastAPI application factory + middleware
+│   └── config.py        # Pydantic settings
+├── frontend/
+│   └── src/
+│       ├── api/         # Axios API client modules
+│       ├── components/  # UI components (documents, folders, cloud, admin, sharing, layout)
+│       ├── stores/      # Pinia stores (auth, documents, folders, cloud connections)
+│       ├── views/       # Page-level view components
+│       └── utils/       # formatters.js (formatDate, formatSize, providerColor, …)
+├── docker/
+│   ├── postgres/        # initdb.d — user + role bootstrap SQL
+│   └── loki/            # Loki + Promtail config
+├── docker-compose.yml
+├── .env.example
+├── RUNBOOK.md           # Operational runbook (env vars, backup, on-call, failure modes)
+└── SECURITY.md          # Threat model and security gate results
+```
+
+---
+
+## Contributing
+
+1. All new endpoints require at least one integration test
+2. Run `pytest -v` and `npm run test` — zero failures required
+3. Run `bandit -r backend/` — zero HIGH findings
+4. No raw SQL string interpolation; no `pickle`; no `eval`; no `innerHTML` with user data
+5. Service layer raises `ValueError` (never `HTTPException`); only the router layer raises `HTTPException`
+6. Before writing a new helper, check the shared module map in `CLAUDE.md`