# DocuVault — v1 Roadmap

_Last updated: 2026-06-02_

## Mandatory Cross-Cutting Gates (every phase)

Before any phase is marked complete, all three gates must pass:

1. **Test gate** — `pytest -v` passes with zero failures; every new function/endpoint has at least one test; all security invariant tests pass (wrong owner, admin block, token replay)
2. **Security gate** — Security agent runs `bandit -r backend/` (zero HIGH), `pip audit` (zero critical/high), `npm audit --audit-level=high` (zero high/critical); admin endpoints verified to never return `password_hash`, `credentials_enc`, or document content; no hardcoded secrets
3. **Bug fix rule** — Any bug fix during execution must: (a) target the root cause, (b) change ≤50 lines, (c) include a regression test — no workarounds permitted

---

## Phases

- [x] **Phase 1: Infrastructure Foundation** — PostgreSQL + MinIO wired into Docker Compose; Alembic migrations running; existing app still works
- [x] **Phase 2: Users & Authentication** — Full auth flow end-to-end (register, login, TOTP, backup codes, password reset, sign-out-all) with admin panel for user management
- [x] **Phase 3: Document Migration & Multi-User Isolation** — All documents in PostgreSQL + MinIO; per-user isolation enforced; existing UI still works
- [x] **Phase 4: Folders, Sharing, Quotas & Document UX** — Full document management UX (folders, sharing, quota bar, PDF preview, search, audit log)
- [x] **Phase 5: Cloud Storage Backends** — Users can connect OneDrive, Google Drive, Nextcloud, or WebDAV as a personal storage backend

---

## Phase Details

### Phase 1: Infrastructure Foundation

**Goal**: PostgreSQL + MinIO are wired into Docker Compose with a complete Alembic-managed schema; all services boot cleanly and the existing single-user document scanner continues to work exactly as before — no user-facing behavior change.
**Mode:** mvp
**Depends on**: Nothing (first phase)
**Requirements**: STORE-01, STORE-02, STORE-07

**Success Criteria** (what must be TRUE):

1. `docker compose up` starts PostgreSQL, MinIO, and the FastAPI backend with no errors; health checks pass for all three services
2. Running `alembic upgrade head` applies the initial migration cleanly against the fresh PostgreSQL instance with no errors
3. The full existing document upload, text extraction, and AI classification workflow completes successfully — no regression in single-user behavior
4. MinIO object key schema `{user_id}/{document_id}/{uuid4()}{ext}` is enforced in the model layer; human-readable filenames are stored in the DB column, not in the MinIO key

**Plans**: 5 plans

- [x] 01-01-PLAN.md — Docker Compose service topology + Postgres init + Pydantic Settings + requirements
- [x] 01-02-PLAN.md — Wave 0 test scaffolds (xfail/skip stubs) + async pytest fixtures
- [x] 01-03-PLAN.md — SQLAlchemy ORM models + async engine + Alembic async migration (incl. alembic upgrade head)
- [x] 01-04-PLAN.md — StorageBackend ABC + MinIO backend + rewritten async services/storage.py
- [x] 01-05-PLAN.md — Lifespan + /health + API cutover + Celery worker + walking-skeleton e2e verify

---

### Phase 2: Users & Authentication

**Goal**: Users can register, log in (with optional TOTP 2FA), reset their password, and sign out all active sessions; admins can manage user accounts and assign AI providers — all enforced by a complete FastAPI dependency chain.
**Mode:** mvp
**Depends on**: Phase 1
**Requirements**: AUTH-01, AUTH-02, AUTH-03, AUTH-04, AUTH-05, AUTH-06, AUTH-07, AUTH-08, SEC-01, SEC-02, SEC-03, SEC-05, SEC-06, SEC-07, ADMIN-01, ADMIN-02, ADMIN-03, ADMIN-04, ADMIN-05, ADMIN-07

**Success Criteria** (what must be TRUE):

1. A new user can register with an email and password that passes strength validation; a password from the HaveIBeenPwned list is rejected with a clear error
2. A logged-in user can enroll a TOTP authenticator app, receive 8–10 backup codes, explicitly acknowledge them, and thereafter be required to supply a TOTP code (or backup code) on every login — a backup code is invalidated on first use
3. A user who forgets their password can receive a reset email, follow the link within 1 hour, set a new password, and is then returned to the TOTP login gate (not auto-logged in)
4. A user can trigger "sign out all devices" from account settings; all other active sessions are immediately invalidated and any reuse of a rotated refresh token revokes the entire token family
5. An admin user can create, deactivate, and reset a user account, and assign an AI provider and model to that user; admin API endpoints never return document content or credentials_enc (per-user document auth enforcement deferred to Phase 3 per D-07)

**Plans**: 5 plans

**Wave 1** — Foundation

- [x] 02-01-PLAN.md — Auth service layer (Argon2, JWT, refresh tokens, TOTP, backup codes, HIBP, security alert), FastAPI deps, BackupCode model + password_must_change migration

**Wave 2** *(blocked on Wave 1 completion)*

- [x] 02-02-PLAN.md — Register/login (TOTP + backup code paths) + refresh/logout/change-password endpoints + CSP/Origin validation/rate-limit (IP + per-account) + Vue auth store + router guard + Login/Register views

**Wave 3** *(blocked on Wave 2 completion)*

- [x] 02-03-PLAN.md — TOTP enrollment + backup codes + password reset + sign-out-all endpoints + AccountView + TotpEnrollment + BackupCodesDisplay + PasswordReset views

**Wave 4** *(blocked on Wave 3 completion)*

- [x] 02-04-PLAN.md — Admin backend: user CRUD, quota, AI config endpoints with get_current_admin enforced + tests

**Wave 5** *(blocked on Wave 4 completion)*

- [x] 02-05-PLAN.md — Admin panel frontend: AdminView + three tab components + AppSidebar admin link and user identity footer

**Cross-cutting constraints:**

- JWT access token in Pinia memory only — never localStorage (Plans 02, 03, 05)
- Refresh token httpOnly SameSite=Strict cookie on all token issuance (Plans 02, 03)
- Admin endpoints never return document content or credentials_enc (Plans 04, 05)
- All auth endpoints rate-limited per-IP and per-account (Plans 02, 03)

**UI hint**: yes

---

### Phase 3: Document Migration & Multi-User Isolation

**Goal**: All existing documents have been migrated from flat-file JSON + filesystem into PostgreSQL + MinIO; all new uploads use the presigned URL flow; per-user isolation is enforced at the DB level; the existing document UI works without regression; the backend is stateless and ready for horizontal scaling.
**Mode:** mvp
**Depends on**: Phase 2
**Requirements**: STORE-03, STORE-04, STORE-05, STORE-06, STORE-08, SEC-04, DOC-03, DOC-04, DOC-05

**Success Criteria** (what must be TRUE):

1. Every document present before migration is accessible after migration with the same metadata and extracted text; a count reconciliation check confirms zero document loss
2. Two concurrent uploads that would together exceed a user's 100 MB quota result in exactly one success and one 413 rejection — the quota never goes over limit
3. A document delete atomically decrements the user's recorded quota usage; after deletion the quota reflects the freed bytes
4. Requesting a document object key or presigned URL for a document owned by a different user returns 403 — no cross-user object access is possible through any request parameter manipulation; all /api/documents/* endpoints enforce get_current_user and return 403 when the requesting user's role is admin (completing SC5 from Phase 2)
5. AI classification for each document uses the provider and model assigned to that user by the admin, not any user-supplied or default value

**Plans**: 5 plans

**Wave 1** — Migration + test scaffolds

- [x] 03-01-PLAN.md — Wave 0 test scaffolds (auth_user/admin_user/MinIO mock fixtures + 19 xfail stubs) + Alembic migration 0003 (null-user cleanup, NOT NULL constraint, topic cleanup, quota reconciliation, ix_topics_user_id) — Complete 2026-05-23

**Wave 2** *(blocked on Wave 1)*

- [ ] 03-02-PLAN.md — Presigned upload backend: StorageBackend ABC + MinIOBackend dual client + generate_presigned_put_url/stat_object + /api/documents/upload-url + /api/documents/{id}/confirm with atomic quota UPDATE + GET /api/auth/me/quota + delete-with-quota + abandoned-upload Celery beat + docker-compose CORS/celery-beat

**Wave 3** *(blocked on Wave 2)*

- [x] 03-03-PLAN.md — Auth guards: get_regular_user dep + ownership assertions on every /api/documents/* handler (404 not 403) + admin 403 + real user_id in object_key + namespace-scoped /api/topics/* + POST /api/admin/topics + classifier topic-namespace plumbing

**Wave 4** *(blocked on Wave 3)*

- [x] 03-04-PLAN.md — Settings retirement + per-user AI: delete /api/settings + remove load_settings/save_settings + classifier accepts ai_provider/ai_model kwargs + Celery task resolves user.ai_provider via DB + frontend SettingsView placeholder + remove settings store/API — Complete 2026-05-23

**Wave 5** *(blocked on Wave 4)*

- [ ] 03-05-PLAN.md — Frontend upload flow + quota bar: 3-step upload action with XHR progress + UploadProgress.vue progress bar and quota rejection error block + QuotaBar.vue + AppSidebar embed + quota state in auth store + human checkpoint

**Cross-cutting constraints:**

- Atomic quota UPDATE pattern only lives in Plan 02; never duplicate (CLAUDE.md)
- Every /api/documents/* handler injects get_regular_user (Plan 03)
- AI provider/model resolved only via Celery task DB lookup (Plan 04)
- Browser XHR PUT to MinIO sends NO Authorization header (Plan 05)

**Phase gates (must pass before Phase 3 is complete):**

- [ ] `pytest -v` — zero failures; presigned URL, quota enforcement, ownership isolation, and admin-403 all covered
- [ ] Security agent: path traversal check on object key construction; cross-user IDOR tests; quota race condition test
- [ ] Bandit + pip audit + npm audit all clean

**UI hint**: yes

---

### Phase 4: Folders, Sharing, Quotas & Document UX

**Goal**: Users have a complete document management experience — organized with folders, shared by handle, warned before they hit quota, able to preview PDFs in-browser, and served by a searchable document list; admins can view the append-only audit log.
**Mode:** mvp
**Depends on**: Phase 3
**Requirements**: FOLD-01, FOLD-02, FOLD-03, FOLD-04, FOLD-05, SHARE-01, SHARE-02, SHARE-03, SHARE-04, SHARE-05, SEC-08, SEC-09, ADMIN-06, DOC-01, DOC-02

**Success Criteria** (what must be TRUE):

1. A user can create, rename, and delete folders; moving a document between folders preserves its metadata and AI classification; deleting a non-empty folder prompts with the content count before proceeding
2. A user can share a document with another user by handle; the recipient sees it appear in a "Shared with me" virtual folder with no storage quota charged against them; the owner can revoke access and the shared entry disappears immediately for the recipient
3. The sidebar quota bar displays current usage in MB; it turns amber at 80% and red at 95%; an upload that would exceed the limit is rejected with an error showing current usage, the rejected file size, and a link to storage settings
4. Any document in the user's library can be previewed in-browser as a PDF; document bytes are proxied through the app and no presigned URLs are exposed to the browser (native browser PDF rendering via Content-Type header)
5. An admin can view the audit log filtered by date range, user, and action type; the log contains no document content, filenames, or extracted text; account deletion triggers cleanup of all user files before DB records are removed

**Plans**: 9 plans

**Wave 1** — Test scaffolds + DB migration (parallel)

- [x] 04-01-PLAN.md — Wave 0 test stubs: test_folders.py + test_shares.py + test_audit.py + proxy stubs in test_documents.py + SEC-08/SEC-09 stubs in test_security.py
- [x] 04-02-PLAN.md — Alembic migration 0004 (users.pdf_open_mode, GIN FTS index, audit-logs bucket) + MinIOBackend.put_object_raw()

**Wave 2** *(blocked on Wave 1)*

- [x] 04-03-PLAN.md — Audit service (write_audit_log) + Folders API (FOLD-01..05): POST/GET/PATCH/DELETE /api/folders + PATCH /api/documents/{id}/folder + document list sort/search/is_shared extension
- [ ] 04-04-PLAN.md — Shares API (SHARE-01..05): POST/GET /api/shares + GET /api/shares/received + DELETE /api/shares/{id} with IDOR protection

**Wave 3** *(blocked on Wave 2)*

- [ ] 04-05-PLAN.md — PDF streaming proxy GET /api/documents/{id}/content with Range header support + PATCH /api/auth/me/preferences (pdf_open_mode)
- [ ] 04-06-PLAN.md — Admin audit log API (GET /api/admin/audit-log, CSV export) + Celery beat daily audit export task + celery_app.py beat schedule

**Wave 4** *(blocked on Wave 3)*

- [ ] 04-07-PLAN.md — SEC-08/SEC-09 hardening + audit log backfill into auth.py/admin.py/documents.py + CloudConnectionOut Pydantic model + delete-user file cleanup

**Wave 5** *(blocked on Wave 4)*

- [ ] 04-08-PLAN.md — Frontend data layer: API client functions + useFoldersStore + documents store extension + Vue Router routes (/folders/:folderId, /shared)

**Wave 6** *(blocked on Wave 5)*

- [ ] 04-09-PLAN.md — Frontend UI: all new components (FolderRow, FolderBreadcrumb, FolderDeleteModal, ShareModal, DocumentPreviewModal, SearchBar, SortControls, AuditLogTab) + view wiring (AppSidebar, DocumentCard, HomeView, FolderView, SharedView, SettingsView, AdminView) + human checkpoint

**Phase gates (must pass before Phase 4 is complete):**

- [ ] `pytest -v` — zero failures; folder ownership, share revocation, quota bar, PDF proxy (no presigned URL exposure) all covered
- [ ] Security agent: audit log verified to contain zero document content; sharing IDOR tests; PDF proxy verified to not leak presigned URLs or object keys
- [ ] Bandit + pip audit + npm audit all clean

**UI hint**: yes

---

### Phase 5: Cloud Storage Backends

**Goal**: Users can connect OneDrive, Google Drive, Nextcloud, or a generic WebDAV server as a personal storage backend; credentials are encrypted with a per-user HKDF-derived key; connection status is visible; local and cloud storage coexist; the `StorageBackend` ABC makes adding further backends straightforward.
**Mode:** mvp
**Depends on**: Phase 4
**Requirements**: CLOUD-01, CLOUD-02, CLOUD-03, CLOUD-04, CLOUD-05, CLOUD-06, CLOUD-07

**Success Criteria** (what must be TRUE):

1. A user can connect OneDrive, Google Drive, Nextcloud, or a WebDAV endpoint through an OAuth or credential flow; the connection status is displayed as `ACTIVE`, `REQUIRES_REAUTH`, or `ERROR` — never shows raw credentials
2. When an OAuth token is revoked externally (simulated `invalid_grant` response), the connection status transitions to `REQUIRES_REAUTH` without a 500 error; the user is shown a re-authentication prompt
3. A user can select their connected cloud backend as the default storage destination for new uploads; local MinIO storage remains available as an alternative; existing local documents are unaffected
4. A user can disconnect a cloud backend; credentials are permanently deleted from the DB and a subsequent attempt to use that backend returns an appropriate error — no orphaned data remains
5. An admin API response for a user's cloud connections returns only `provider, display_name, connected_at, status` — the `credentials_enc` column is never present in any serialized response

**Plans**: 12 plans (8 original + 3 UAT gap closure + 1 gap closure wave)

**Wave 1** — Test scaffold + dependencies

- [x] 05-01-PLAN.md — Wave 0 xfail stubs, conftest cloud fixtures, requirements.txt packages, config.py settings

**Wave 2** — Shared utilities

- [x] 05-02-PLAN.md — cloud_utils.py (SSRF + HKDF), cloud_cache.py (TTLCache), storage factory extension

**Wave 3** — Cloud backends (parallel, both blocked on Wave 2 / Plan 05-02)

- [x] 05-03-PLAN.md — GoogleDriveBackend + OneDriveBackend (all 7 StorageBackend methods)
- [x] 05-04-PLAN.md — NextcloudBackend + WebDAVBackend (all 7 StorageBackend methods)

**Wave 4** — Cloud API

- [x] 05-05-PLAN.md — All /api/cloud/* endpoints + /api/users/me/default-storage + main.py router registration

**Wave 5** — Document routing + full test suite

- [x] 05-06-PLAN.md — Upload/content proxy cloud routing + all 15 tests promoted to passing

**Wave 6** — Frontend settings UI

- [x] 05-07-PLAN.md — cloudConnections store + API client + SettingsView 3-tab + SettingsCloudTab + CloudCredentialModal

**Wave 7** — Frontend sidebar (human checkpoint)

- [x] 05-08-PLAN.md — AppSidebar cloud section + CloudProviderTreeItem + CloudFolderTreeItem + human checkpoint

**Wave 8** — UAT gap closure (parallel, all independent)

- [x] 05-09-PLAN.md — Cloud document open/re-analyze/edit: authenticated fetch+Blob URL, cloud-aware Celery task, PATCH /api/documents/{id}
- [x] 05-10-PLAN.md — OAuth initiate fix (JSON response), Nextcloud custom endpoint edit round-trip, Edit button on ERROR rows, confirmation text overflow
- [x] 05-11-PLAN.md — Admin hard-delete with password confirmation: UserDeleteConfirm backend model + inline frontend panel

**Wave 9** — Post-UAT gap closure

- [x] 05-12-PLAN.md — OAuth 400 preflight (unconfigured creds), 502 cloud fallback, upload hint in CloudStorageView, celery-worker volume mount

**Phase gates (must pass before Phase 5 is complete):**

- [x] `pytest -v` — zero failures; SSRF prevention on WebDAV/Nextcloud user-supplied URLs; credential encryption/decryption round-trip; admin response never exposes `credentials_enc`; OAuth invalid_grant handling
- [x] Security agent: SSRF allowlist verification; credential key derivation correctness; connection status never leaks raw credential values
- [x] Bandit + pip audit + npm audit all clean
- [x] UAT gaps resolved and re-tested (05-09, 05-10, 05-11, 05-12)

**UI hint**: yes

---

### Phase 6: Performance & Production Hardening

**Goal**: The application is ready for production deployment — observable, load-tested, and hardened; response times meet SLA targets under concurrent load; all auth and document endpoints are rate-limited; structured logging and distributed tracing are in place; the Docker image runs as a non-root user with a read-only filesystem.
**Mode:** mvp
**Depends on**: Phase 5
**Requirements**: TBD

**Success Criteria** (what must be TRUE):

1. All API endpoints respond within defined latency targets (p50/p95/p99) under a realistic load test (e.g., 50 concurrent users, 5-minute soak)
2. Structured JSON logging (correlation IDs, user ID, request latency) is emitted to stdout; a local log aggregation stack (Loki or similar) captures and queries them
3. All auth endpoints (login, register, password reset, TOTP) enforce per-IP and per-account rate limits that cannot be bypassed by header manipulation
4. Container hardening is complete: non-root user, read-only root filesystem, dropped Linux capabilities; `docker scout` or equivalent reports zero critical CVEs
5. A runbook documents all environment variables, startup/shutdown procedures, backup strategy, and on-call escalation path; the app can be stood up from scratch using only the runbook

**Plans**: 6 plans (4 waves)

**Wave 0** — Test scaffolds + package verification

- [ ] 06-01-PLAN.md — Nyquist Wave 0: xfail stubs (test_logging.py, test_rate_limiting.py), Locust skeleton, package legitimacy checkpoint (D-01, D-04, D-11, D-12)

**Wave 1** *(blocked on Wave 0 completion)*

- [ ] 06-02-PLAN.md — structlog JSON logging + CorrelationIDMiddleware + Loki/Promtail/Grafana Docker Compose stack (D-01, D-02, D-03)
- [ ] 06-03-PLAN.md — Locust locustfile.py with JWT auth + SLA gate listener (D-04, D-05, D-06)

**Wave 2** *(blocked on Wave 1 completion)*

- [ ] 06-04-PLAN.md — Multi-stage Dockerfile + read_only/tmpfs/cap_drop on backend + celery-worker (D-07, D-08, D-09)
- [ ] 06-05-PLAN.md — Trusted-proxy get_client_ip body + per-account rate limiter on document/cloud endpoints (D-11, D-12, D-13)

**Wave 3** *(blocked on Wave 2 completion)*

- [ ] 06-06-PLAN.md — docker scout CVE gate + RUNBOOK.md (D-10, D-14)

**Cross-cutting constraints:**

- get_client_ip lives ONLY in backend/deps/utils.py — replace body in-place, no new function (Plans 05)
- celery-beat intentionally excluded from read_only: true (Plan 04)
- locust must NOT be in requirements.txt — use requirements-dev.txt (Plan 03)
- CorrelationIDMiddleware registered LAST in main.py — Starlette reverse order (Plan 02)

---

### Phase 6.1: Close v1.0 audit gaps: SHARE-02/STORE-06/ADMIN-06

**Goal**: Close three v1.0 requirements that remain unimplemented — atomic quota decrement on document delete (STORE-06), "Shared with me" virtual folder without recipient quota charge (SHARE-02), and admin audit log viewer with date/user/action type filters (ADMIN-06).
**Mode:** mvp
**Depends on**: Phase 6
**Requirements**: STORE-06, SHARE-02, ADMIN-06

**Success Criteria** (what must be TRUE):

1. Deleting a document atomically decrements the owning user's quota; after deletion the quota reflects the freed bytes with no race condition under concurrent deletes
2. A user who receives a shared document sees it appear in a "Shared with me" virtual folder; the recipient's quota usage is not charged for the shared document's storage
3. An admin can view the audit log filtered independently by date range, user, and action type; filtered results contain no document content, filenames, or extracted text

**Plans**: 2 plans

**Wave 1** — Test promotion (parallel)

- [x] 06.1-01-PLAN.md — Promote test_shares.py stubs to real tests + second_auth_user fixture (SHARE-01..05)
- [x] 06.1-02-PLAN.md — Promote test_audit.py stubs to real tests (ADMIN-06)

**Phase gates (must pass before Phase 6.1 is complete):**

- [ ] `pytest -v` — zero failures; all 7 share tests + 4 audit log tests passing
- [ ] Security agent: bandit + pip audit + npm audit all clean
- [ ] STORE-06 confirmed: `test_delete_decrements_quota` passes under `INTEGRATION=1`

---

### Phase 6.2: Close v1 sharing + cloud-delete + CSV export gaps

**Goal**: Close remaining v1 gaps — sharing edge cases (SHARE-03/SHARE-05), cloud document deletion propagation to the remote backend, and CSV export + daily export UI for the admin audit log (ADMIN-06).
**Mode:** mvp
**Depends on**: Phase 6.1
**Requirements**: SHARE-03, SHARE-05, ADMIN-06

**Success Criteria** (what must be TRUE):

1. Documents shared with others display a "Shared" badge in the owner's list view (reads doc.is_shared, not doc.share_count)
2. Owner can set permission to "view" or "edit" when creating a share and toggle it per-recipient afterward; PATCH /api/shares/{id} enforces IDOR protection (404 on wrong owner)
3. Deleting a cloud document propagates the delete to the cloud provider; failure shows a warning modal with "Remove from app" fallback; ?remove_only=true removes only the DB record; cloud docs never affect quota on delete
4. Admin can download filtered audit log CSV via fetch+Blob (not window.location.href); audit log entries show user handles instead of raw UUIDs; user filter accepts handles (not UUIDs)
5. Admin can list and download Celery-generated daily audit export files from a new section in the Audit Log tab

**Plans**: 4 plans

**Wave 0** — Test stubs

- [x] 06.2-01-PLAN.md — 11 xfail stubs across test_shares.py, test_documents.py, test_audit.py

**Wave 1** — Feature slices (parallel)

- [x] 06.2-02-PLAN.md — SHARE-05 badge fix + SHARE-03 permission control (backend PATCH + frontend dropdown + toggle)
- [x] 06.2-03-PLAN.md — Cloud-delete propagation + structured error response + remove_only path + DocumentView warning modal

**Wave 2** — Audit log enrichment

- [x] 06.2-04-PLAN.md — Audit handle JOIN + user_handle filter + CSV fetch+Blob fix + daily-export list + download endpoints + AuditLogTab UI

**Phase gates (must pass before Phase 6.2 is complete):**

- [x] `pytest -v` — 344 passed, 1 pre-existing unrelated failure (test_extract_docx missing module)
- [x] Security agent: bandit + pip audit + npm audit all clean (SECURITY.md threats_open: 0)
- [x] IDOR on PATCH /api/shares/{id}: test_share_patch_idor passes
- [x] Date regex validation confirmed: GET /api/admin/audit-log/daily-exports/invalid-date returns 404
- [x] window.location.href removed from AuditLogTab.vue confirmed by grep

**Status: ✓ Complete (2026-06-01)**

### Phase 7: Redo and optimize LLM integration

**Goal:** A fully refactored, production-reliable AI provider layer: AI provider settings live in a new `system_settings` DB table (replacing env-only config), API keys encrypted at rest via HKDF/AES-GCM (same pattern as cloud credentials), all OpenAI-compatible providers (Groq, xAI, DeepSeek, OpenRouter, Gemini-compat, Mistral-compat, Ollama, LMStudio) handled by a single `GenericOpenAIProvider` class, Anthropic uses native `output_config.format.type="json_schema"` structured output, JSON-mode enforced everywhere with `parse_classification()` as fallback, Celery exponential-backoff retry (30s/90s/270s) replaces silent failure on classification errors, per-provider context window size with smart 60/40 truncation replaces the global `MAX_AI_CHARS`, the singleton `_client` pattern restores httpx connection-pool reuse, an admin AI Providers panel allows interactive configuration plus test-connection, and a "Re-analyze" button on the document card re-queues failed classifications.
**Mode:** standard
**Depends on:** Phase 6.2
**Requirements**: D-01..D-18 (CONTEXT.md decisions; no REQ-IDs yet mapped — phase introduces new infrastructure)

**Success Criteria** (what must be TRUE):

1. AI provider settings live in `system_settings`; admin can view, edit, and atomically activate any provider via `PUT /api/admin/ai-config`; `GET /api/admin/ai-config` never returns `api_key_enc` or any decrypted key value
2. A document whose classification raises an exception is automatically retried by Celery at 30 s, 90 s, and 270 s; after the third failure the document's status remains `classification_failed` and no further retries occur
3. All OpenAI-compatible providers route through `GenericOpenAIProvider`; classify/suggest pass `response_format={"type":"json_object"}` when `supports_json_mode` is True (Gemini preset is False and falls back to `parse_classification()`); Anthropic uses `output_config.format.type="json_schema"` with a constrained schema
4. `MAX_AI_CHARS` no longer exists in `openai_provider.py`, `anthropic_provider.py`, or `classifier.py`; each provider truncates input text using its own `context_chars` value via a 60% head + 40% tail strategy
5. A failed document shows a red "Classification failed" badge and a "Re-analyze" button on the document card; clicking the button calls `POST /api/documents/{id}/classify`, which sets the document to `processing` and re-queues the Celery task

**Plans**: 5 plans (5 waves)

**Wave 1** — Foundation: migration, ORM model, encryption helpers, Wave 0 test stubs

- [ ] 07-01-PLAN.md — Alembic migration 0005 (system_settings table) + SystemSettings ORM model + services/ai_config.py (HKDF helpers + load_provider_config + env seed on startup) + Wave 0 xfail stubs for D-01..D-16 (test_ai_providers.py, test_ai_config.py, test_admin_ai_config.py, test_document_tasks.py)

**Wave 2** *(blocked on Wave 1)* — Provider layer: ProviderConfig, GenericOpenAIProvider, singleton client fix, registry

- [ ] 07-02-PLAN.md — ProviderConfig Pydantic model + PROVIDER_DEFAULTS + SUPPORTS_JSON_MODE + GenericOpenAIProvider(OpenAIProvider) + OpenAIProvider singleton refactor + ollama/lmstudio context_chars + MAX_AI_CHARS removal from openai_provider.py + classifier.py + registry-based ai/__init__.py get_provider(config: ProviderConfig) + anthropic>=0.95.0 pin

**Wave 3** *(blocked on Wave 2)* — Anthropic + Classifier wiring

- [ ] 07-03-PLAN.md — AnthropicProvider singleton + output_config json_schema + smart truncation + MAX_AI_CHARS removal + classifier.classify_document driven by load_provider_config (no inline _settings dict) + ai_config stub removed in favor of real ProviderConfig + load_provider_config_by_id helper

**Wave 4** *(blocked on Wave 3)* — Celery retry harness + Re-queue endpoint

- [ ] 07-04-PLAN.md — Celery extract_and_classify decorator gains bind=True + max_retries=3 + _ClassificationError sentinel + 30/90/270 backoff + _mark_classification_failed on exhaustion + POST /api/documents/{id}/classify changes to re-queue Celery (sets status=processing, calls .delay()) — promotes test_retry_backoff, test_exhaustion_sets_failed_status, test_reclassify_requeues_celery

**Wave 5** *(blocked on Wave 4)* — Admin UI + Frontend badge

- [ ] 07-05-PLAN.md — Admin AI-config backend (GET/PUT/test-connection with whitelist + audit logging + atomic is_active flip) + frontend API client helpers (getAiConfig/saveAiConfig/testAiConnection) + AdminAiConfigTab.vue global System AI Providers section ABOVE existing per-user table + DocumentCard.vue classification_failed badge + Re-analyze button + human checkpoint UAT

**Cross-cutting constraints:**

- `api_key_enc` is never returned by any admin endpoint — enforced by `_ai_config_to_dict()` whitelist (Plan 05)
- HKDF domain separation: AI settings use `info=b"ai-provider-settings"`, cloud credentials use `info=b"cloud-credentials"` (Plan 01)
- `is_active` flip is atomic: single `UPDATE SET is_active = (provider_id = $target)` — never read-then-write (Plan 05)
- Provider clients are stored as `self._client` in `__init__` — never recreated per call (Plans 02, 03)
- `MAX_AI_CHARS` removal must cover all three locations: openai_provider.py L5, anthropic_provider.py L5, classifier.py L28 (Plans 02 + 03)
- Celery `self.retry()` must be raised from the outer sync task body, never inside `asyncio.run()` (Plan 04 — Pitfall 3)
- `extra_hosts: ["host.docker.internal:host-gateway"]` already present in docker-compose.yml — D-14 is already done; no docker-compose changes required (RESEARCH.md)
- AdminAiConfigTab.vue per-user assignment table is preserved untouched; the new global system section is added ABOVE it (Plan 05 — Pitfall 6)

**Phase gates (must pass before Phase 7 is complete):**

- [ ] `pytest -v` — zero failures; D-01/D-03/D-05/D-06/D-07/D-09/D-10/D-11/D-12/D-13/D-16 covered by promoted tests
- [ ] Security agent: bandit -r backend/ (zero HIGH), pip audit (zero critical/high), npm audit --audit-level=high (zero high/critical)
- [ ] `_ai_config_to_dict()` confirmed to never include `api_key_enc` (integration test asserts response.text does not contain the key)
- [ ] HKDF domain separation verified: same master key + same plaintext produces different ciphertexts for `provider_id` vs `user_id` derivations
- [ ] Atomic `is_active` flip verified: after two PUTs with `is_active=true` on different providers, exactly one row has `is_active=true`
- [ ] Human checkpoint UAT approves admin panel + DocumentCard badge end-to-end

**UI hint**: yes

---

## Progress Table

| Phase | Plans Complete | Status | Completed |
|-------|----------------|--------|-----------|
| 1. Infrastructure Foundation | 5/5 | Complete | 2026-05-22 |
| 2. Users & Authentication | 6/6 | Complete   | 2026-06-01 |
| 3. Document Migration & Multi-User Isolation | 5/5 | Complete | 2026-05-25 |
| 4. Folders, Sharing, Quotas & Document UX | 9/9 | Complete | 2026-05-28 |
| 5. Cloud Storage Backends | 12/12 | Complete | 2026-05-30 |
| 6. Performance & Production Hardening | 0/TBD | Not started | — |
| 6.1. Close v1.0 audit gaps | 2/2 | Complete | 2026-05-30 |
| 6.2. Close v1 sharing + cloud-delete + CSV export gaps | 5/5 | Complete   | 2026-05-31 |
| 7. Redo and optimize LLM integration | 0/5 | Planned | — |
