From 6ec2748b840dd97ef1dc4798cba722fc9a5eda78 Mon Sep 17 00:00:00 2001 From: curo1305 Date: Fri, 5 Jun 2026 14:44:54 +0200 Subject: [PATCH] docs(readme): add project README for v0.1 alpha Covers features, architecture diagram, token flow, stack, environment variables, quick-start, running tests, cloud backend setup, AI provider configuration, and alpha status disclaimer. Co-Authored-By: Claude Sonnet 4.6 --- README.md | 349 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 349 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..4a0b5e0 --- /dev/null +++ b/README.md @@ -0,0 +1,349 @@ +# DocuVault + +**Version 0.1.0 — Alpha** + +> **Not production-ready.** DocuVault is functional for local and self-hosted use but has not been audited or hardened for public internet exposure. APIs, environment variables, and the database schema may change without notice until a stable 1.0 release is declared. + +A self-hosted, multi-user document management platform with AI-powered topic classification, pluggable cloud storage backends, and a security-first architecture. + +--- + +## Features + +- **Document management** — upload PDF, DOCX, images, and plain text; full-text extraction stored in PostgreSQL +- **AI classification** — automatic topic tagging via LM Studio (local), Ollama, OpenAI, Anthropic, or any OpenAI-compatible provider (Groq, xAI, DeepSeek, OpenRouter, Gemini, Mistral) +- **Folder organisation** — hierarchical folders with move, rename, and delete; breadcrumb navigation +- **Document sharing** — share by user handle with view or edit permission; "Shared with me" virtual folder; per-recipient revocation +- **Storage quota** — per-user limit enforced atomically; amber/red quota bar at 80 % / 95 %; quota decremented on delete +- **Cloud storage backends** — connect OneDrive, Google Drive, Nextcloud, or any WebDAV server as a personal storage backend; credentials encrypted with HKDF per-user keys +- **Auth** — email/password registration with HIBP breach check, optional TOTP 2FA, 8–10 backup codes, password reset by email, sign-out-all-devices +- **Admin panel** — user CRUD, quota assignment, per-user AI provider override, AI provider system configuration, audit log viewer with CSV export +- **Observability** — structured JSON logging via structlog, per-request correlation IDs, Loki + Promtail + Grafana stack included in Docker Compose +- **Production hardening** — non-root containers, read-only filesystems, CSP/CSRF/Origin-validation middleware, per-IP and per-account rate limiting, CVE scanning gate + +--- + +## Architecture + +``` +┌─────────────────────────────────────────┐ +│ Vue 3 frontend (Vite + Pinia + Tailwind) │ +│ Port 5173 │ +└────────────────┬────────────────────────┘ + │ REST / httpOnly cookie +┌────────────────▼────────────────────────┐ +│ FastAPI backend (Python 3.12) │ +│ Port 8000 │ +│ ├── api/auth.py (JWT + TOTP) │ +│ ├── api/documents.py │ +│ ├── api/folders.py │ +│ ├── api/shares.py │ +│ ├── api/cloud.py (cloud backends) │ +│ ├── api/admin.py │ +│ └── api/audit.py │ +└──┬──────────┬──────────┬───────────────┘ + │ │ │ + ▼ ▼ ▼ +PostgreSQL MinIO Redis +(data) (objects) (rate limit / TOTP replay) + │ + └── Celery worker + (extract + classify tasks) + Celery beat + (daily audit export) +``` + +**Storage object key schema:** `{user_id}/{document_id}/{uuid4()}{ext}` — human filenames live in PostgreSQL only. + +**Token flow:** Access token (15 min, ES256) in Pinia memory only. Refresh token (30 day) in an httpOnly `SameSite=Strict` cookie. Refresh rotation on every use; reuse revokes the entire token family. + +--- + +## Stack + +| Layer | Technology | +|-------|------------| +| Backend | Python 3.12, FastAPI 0.136+, SQLAlchemy 2.0 async, psycopg v3, Alembic | +| Task queue | Celery 5 + Redis, celery-beat for scheduled tasks | +| Object store | MinIO (S3-compatible) | +| Database | PostgreSQL 17 | +| Frontend | Vue 3 (Options API), Pinia, Vue Router 4, Vite, Tailwind CSS | +| Auth | PyJWT (ES256), pwdlib[argon2], pyotp (TOTP), cryptography (HKDF/AES-GCM) | +| AI providers | Anthropic, OpenAI, Ollama, LM Studio, GenericOpenAI (Groq/xAI/DeepSeek/Gemini/…) | +| Cloud backends | Google Drive, OneDrive (MSAL), Nextcloud, WebDAV | +| Observability | structlog, Loki, Promtail, Grafana | +| Rate limiting | slowapi (per-IP + per-account) | +| Load testing | Locust (`backend/load_tests/`) | + +--- + +## Prerequisites + +- Docker and Docker Compose v2 +- `openssl` (for generating secrets) +- *(Optional)* LM Studio or Ollama running on the host for local AI classification + +--- + +## Quick Start + +```bash +# 1. Clone +git clone +cd document_scanner + +# 2. Create .env from the example +cp .env.example .env + +# 3. Generate required secrets +python3 -c "import secrets; print(secrets.token_hex(64))" # → SECRET_KEY +python3 -c "import secrets; print(secrets.token_urlsafe(32))" # → CLOUD_CREDS_KEY + +# 4. Edit .env — at minimum set: +# SECRET_KEY, CLOUD_CREDS_KEY, ADMIN_EMAIL, ADMIN_PASSWORD +# (everything else has usable defaults for local dev) +nano .env + +# 5. Start all services +docker compose up -d --build + +# 6. Run database migrations +docker compose exec backend alembic upgrade head + +# 7. Open the app +open http://localhost:5173 +``` + +The bootstrap admin account (email + password from `.env`) is created automatically on first startup if no users exist. + +--- + +## Service URLs + +| Service | URL | Notes | +|---------|-----|-------| +| Vue frontend | http://localhost:5173 | Main app | +| FastAPI backend | http://localhost:8000 | REST API | +| API health | http://localhost:8000/health | Postgres + MinIO probe | +| OpenAPI docs | http://localhost:8000/docs | Swagger UI (dev) | +| MinIO console | http://localhost:9001 | Object browser | +| Grafana | http://localhost:3000 | Log dashboard (admin/changeme) | +| Loki | http://localhost:3100/ready | Log aggregator | + +--- + +## Environment Variables + +Copy `.env.example` to `.env`. Only the fields marked **Required** must be set before first boot. See `RUNBOOK.md` for the full reference. + +### Required + +| Variable | Description | +|----------|-------------| +| `SECRET_KEY` | JWT signing secret — generate with `openssl rand -hex 32` | +| `CLOUD_CREDS_KEY` | Master key for cloud credential encryption — generate with `openssl rand -hex 16` (pad to 32 chars) | +| `ADMIN_EMAIL` | Bootstrap admin email | +| `ADMIN_PASSWORD` | Bootstrap admin password (must pass strength check) | +| `POSTGRES_PASSWORD` | PostgreSQL superuser password | +| `MINIO_ROOT_PASSWORD` | MinIO root password | +| `REDIS_PASSWORD` | Redis `requirepass` password | + +### Optional (sensible defaults for local dev) + +| Variable | Default | Description | +|----------|---------|-------------| +| `CORS_ORIGINS` | `http://localhost:5173` | Comma-separated allowed origins | +| `SMTP_HOST` | *(unset)* | Leave empty to log password-reset links to stdout | +| `LOG_JSON` | `false` | Set `true` in production for structured JSON logs | +| `DEFAULT_AI_PROVIDER` | `ollama` | Used on first boot seed; overridable per-user in admin panel | +| `GOOGLE_CLIENT_ID/SECRET` | *(unset)* | Required only if using Google Drive backend | +| `ONEDRIVE_CLIENT_ID/SECRET` | *(unset)* | Required only if using OneDrive backend | + +--- + +## Development + +### Backend (local, no Docker) + +```bash +cd backend +pip install -r requirements.txt -r requirements-dev.txt +# Requires a running PostgreSQL and Redis — use docker compose up postgres redis minio +uvicorn main:app --reload --port 8000 +``` + +### Frontend (local, no Docker) + +```bash +cd frontend +npm install +npm run dev # http://localhost:5173 +``` + +### Database migrations + +```bash +# Apply all pending migrations +docker compose exec backend alembic upgrade head + +# Create a new migration after changing db/models.py +docker compose exec backend alembic revision --autogenerate -m "describe change" +``` + +### Live reload in Docker + +`docker-compose.yml` mounts `./backend:/app` so Python changes reload automatically. After changing `requirements.txt` or `Dockerfile`: + +```bash +docker compose build backend celery-worker celery-beat +docker compose up -d backend celery-worker celery-beat +``` + +--- + +## Testing + +```bash +# Backend — all tests +cd backend && pytest -v + +# Backend — integration tests (requires live PostgreSQL + MinIO) +INTEGRATION=1 pytest -v + +# Frontend +cd frontend && npm run test + +# Load testing (requires a running stack + load-test user) +cd backend +pip install locust +locust -f load_tests/locustfile.py --host http://localhost:8000 +``` + +### Test coverage by layer + +| Layer | Approach | +|-------|----------| +| Service / business logic | Unit tests with mocked dependencies | +| DB queries | Integration tests against real PostgreSQL | +| API endpoints | `httpx.AsyncClient` against a real DB | +| Auth flows | Full round-trip (register → login → TOTP → refresh → revoke) | +| Security invariants | Negative tests (wrong owner → 403/404, admin block, token replay) | +| Frontend | Vitest unit tests in `frontend/tests/` | + +--- + +## AI Providers + +AI provider settings live in the `system_settings` database table and are managed through the Admin → AI Providers panel. API keys are encrypted at rest with HKDF/AES-GCM. + +| Provider | Type | `provider_id` | +|----------|------|---------------| +| LM Studio | Local (OpenAI-compat) | `lmstudio` | +| Ollama | Local (OpenAI-compat) | `ollama` | +| OpenAI | Cloud | `openai` | +| Anthropic | Cloud (native) | `anthropic` | +| Groq | Cloud (OpenAI-compat) | `groq` | +| xAI Grok | Cloud (OpenAI-compat) | `xai` | +| DeepSeek | Cloud (OpenAI-compat) | `deepseek` | +| OpenRouter | Cloud (OpenAI-compat) | `openrouter` | +| Gemini (compat) | Cloud | `gemini` | +| Mistral (compat) | Cloud | `mistral` | + +Documents whose classification fails are automatically retried by Celery at 30 s, 90 s, and 270 s. After three failures the document shows a red "Classification failed" badge with a "Re-analyze" button. + +--- + +## Cloud Storage Backends + +Users connect cloud storage through **Settings → Cloud Storage**. Credentials are encrypted per-user with an HKDF-derived key and never returned in API responses. + +| Provider | Auth method | Notes | +|----------|-------------|-------| +| Google Drive | OAuth 2.0 | Requires `GOOGLE_CLIENT_ID/SECRET` | +| Microsoft OneDrive | OAuth 2.0 (MSAL) | Requires `ONEDRIVE_CLIENT_ID/SECRET` | +| Nextcloud | Username + password | Custom server URL; SSRF allowlist enforced | +| WebDAV | Username + password | Any RFC 4918 server; SSRF allowlist enforced | + +Connection statuses: `ACTIVE`, `REQUIRES_REAUTH`, `ERROR`. An externally revoked OAuth token transitions to `REQUIRES_REAUTH` without a 500 error. + +--- + +## Security Highlights + +- **Passwords** hashed with Argon2id; HIBP breach check on registration and password change +- **JWT** signed with ES256 (ECDSA P-256); access token 15 min; refresh token 30 days in httpOnly `SameSite=Strict` cookie +- **Token fingerprinting** — access token embeds HMAC of `User-Agent + Accept-Language`; validated on every request +- **TOTP replay prevention** — used codes marked in Redis within the 90 s validity window +- **Quota** enforced by a single atomic `UPDATE … RETURNING` — no read-then-write race +- **IDOR protection** — every resource endpoint asserts `resource.user_id == current_user.id` +- **SSRF prevention** — user-supplied WebDAV/Nextcloud URLs pass a hostname allowlist +- **CSP / security headers** on every response +- **Container hardening** — non-root user (`appuser` uid 1000), read-only root filesystem, all Linux capabilities dropped +- **Audit log** — all auth events, quota violations, and admin actions logged without document content + +Security gate (run before every production deploy): + +```bash +bandit -r backend/ # zero HIGH +pip audit # zero critical/high CVEs +npm audit --audit-level=high # zero high/critical +docker scout cves local://docuvault-backend:latest --only-severity critical --exit-code +``` + +See `SECURITY.md` for the full threat model and `RUNBOOK.md` for operational procedures. + +--- + +## Observability + +Structured JSON logs are emitted to stdout and collected by Promtail → Loki → Grafana. + +Every request gets an `X-Correlation-ID` header bound to all log lines for that request. Log fields include `correlation_id`, `path`, `method`, `status_code`, `duration_ms`, and `user_id` (when authenticated). + +Access Grafana at http://localhost:3000 (default credentials: `admin` / `changeme` — change via `GRAFANA_ADMIN_PASSWORD` in `.env`). Add a Loki datasource pointing to `http://loki:3100` to query logs. + +--- + +## Project Structure + +``` +. +├── backend/ +│ ├── ai/ # Provider implementations + classifier +│ ├── api/ # FastAPI routers (auth, documents, folders, shares, cloud, admin, audit) +│ ├── db/ # SQLAlchemy models + async session +│ ├── deps/ # FastAPI dependency injection (auth, db, utils) +│ ├── migrations/ # Alembic migration versions +│ ├── services/ # Business logic (auth, classifier, storage, email, audit, rate limiting) +│ ├── storage/ # StorageBackend ABC + MinIO/Google/OneDrive/Nextcloud/WebDAV backends +│ ├── tasks/ # Celery tasks (document processing, email, audit export) +│ ├── load_tests/ # Locust load test scripts +│ ├── tests/ # pytest test suite +│ ├── main.py # FastAPI application factory + middleware +│ └── config.py # Pydantic settings +├── frontend/ +│ └── src/ +│ ├── api/ # Axios API client modules +│ ├── components/ # UI components (documents, folders, cloud, admin, sharing, layout) +│ ├── stores/ # Pinia stores (auth, documents, folders, cloud connections) +│ ├── views/ # Page-level view components +│ └── utils/ # formatters.js (formatDate, formatSize, providerColor, …) +├── docker/ +│ ├── postgres/ # initdb.d — user + role bootstrap SQL +│ └── loki/ # Loki + Promtail config +├── docker-compose.yml +├── .env.example +├── RUNBOOK.md # Operational runbook (env vars, backup, on-call, failure modes) +└── SECURITY.md # Threat model and security gate results +``` + +--- + +## Contributing + +1. All new endpoints require at least one integration test +2. Run `pytest -v` and `npm run test` — zero failures required +3. Run `bandit -r backend/` — zero HIGH findings +4. No raw SQL string interpolation; no `pickle`; no `eval`; no `innerHTML` with user data +5. Service layer raises `ValueError` (never `HTTPException`); only the router layer raises `HTTPException` +6. Before writing a new helper, check the shared module map in `CLAUDE.md`