docs(07): create phase plan — 5 plans, verification passed

5 waves: system_settings DB + HKDF encryption (01), ProviderConfig +
GenericOpenAIProvider + singleton client fix (02), Anthropic output_config +
classifier wiring (03), Celery retry 30/90/270s + re-queue endpoint (04),
admin AI panel + DocumentCard badge + human checkpoint (05).

All 18 decisions D-01..D-18 covered. Plan checker passed after 1 revision
round (4 blockers fixed: D-02/D-14 coverage, D-11 Vitest tests, Plan 05
files_modified).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
2026-06-03 18:25:00 +02:00
co-authored by Claude Sonnet 4.6
parent 6c8e0d8bde
commit 3df62506c9
11 changed files with 2328 additions and 26 deletions
+57 -7
View File
@@ -1,6 +1,6 @@
# DocuVault — v1 Roadmap
_Last updated: 2026-05-31_
_Last updated: 2026-06-02_
## Mandatory Cross-Cutting Gates (every phase)
@@ -385,13 +385,62 @@ Before any phase is marked complete, all three gates must pass:
### Phase 7: Redo and optimize LLM integration
**Goal:** [To be planned]
**Requirements**: TBD
**Depends on:** Phase 6
**Plans:** 0 plans
**Goal:** A fully refactored, production-reliable AI provider layer: AI provider settings live in a new `system_settings` DB table (replacing env-only config), API keys encrypted at rest via HKDF/AES-GCM (same pattern as cloud credentials), all OpenAI-compatible providers (Groq, xAI, DeepSeek, OpenRouter, Gemini-compat, Mistral-compat, Ollama, LMStudio) handled by a single `GenericOpenAIProvider` class, Anthropic uses native `output_config.format.type="json_schema"` structured output, JSON-mode enforced everywhere with `parse_classification()` as fallback, Celery exponential-backoff retry (30s/90s/270s) replaces silent failure on classification errors, per-provider context window size with smart 60/40 truncation replaces the global `MAX_AI_CHARS`, the singleton `_client` pattern restores httpx connection-pool reuse, an admin AI Providers panel allows interactive configuration plus test-connection, and a "Re-analyze" button on the document card re-queues failed classifications.
**Mode:** standard
**Depends on:** Phase 6.2
**Requirements**: D-01..D-18 (CONTEXT.md decisions; no REQ-IDs yet mapped — phase introduces new infrastructure)
Plans:
- [ ] TBD (run /gsd-plan-phase 7 to break down)
**Success Criteria** (what must be TRUE):
1. AI provider settings live in `system_settings`; admin can view, edit, and atomically activate any provider via `PUT /api/admin/ai-config`; `GET /api/admin/ai-config` never returns `api_key_enc` or any decrypted key value
2. A document whose classification raises an exception is automatically retried by Celery at 30 s, 90 s, and 270 s; after the third failure the document's status remains `classification_failed` and no further retries occur
3. All OpenAI-compatible providers route through `GenericOpenAIProvider`; classify/suggest pass `response_format={"type":"json_object"}` when `supports_json_mode` is True (Gemini preset is False and falls back to `parse_classification()`); Anthropic uses `output_config.format.type="json_schema"` with a constrained schema
4. `MAX_AI_CHARS` no longer exists in `openai_provider.py`, `anthropic_provider.py`, or `classifier.py`; each provider truncates input text using its own `context_chars` value via a 60% head + 40% tail strategy
5. A failed document shows a red "Classification failed" badge and a "Re-analyze" button on the document card; clicking the button calls `POST /api/documents/{id}/classify`, which sets the document to `processing` and re-queues the Celery task
**Plans**: 5 plans (5 waves)
**Wave 1** — Foundation: migration, ORM model, encryption helpers, Wave 0 test stubs
- [ ] 07-01-PLAN.md — Alembic migration 0005 (system_settings table) + SystemSettings ORM model + services/ai_config.py (HKDF helpers + load_provider_config + env seed on startup) + Wave 0 xfail stubs for D-01..D-16 (test_ai_providers.py, test_ai_config.py, test_admin_ai_config.py, test_document_tasks.py)
**Wave 2** *(blocked on Wave 1)* — Provider layer: ProviderConfig, GenericOpenAIProvider, singleton client fix, registry
- [ ] 07-02-PLAN.md — ProviderConfig Pydantic model + PROVIDER_DEFAULTS + SUPPORTS_JSON_MODE + GenericOpenAIProvider(OpenAIProvider) + OpenAIProvider singleton refactor + ollama/lmstudio context_chars + MAX_AI_CHARS removal from openai_provider.py + classifier.py + registry-based ai/__init__.py get_provider(config: ProviderConfig) + anthropic>=0.95.0 pin
**Wave 3** *(blocked on Wave 2)* — Anthropic + Classifier wiring
- [ ] 07-03-PLAN.md — AnthropicProvider singleton + output_config json_schema + smart truncation + MAX_AI_CHARS removal + classifier.classify_document driven by load_provider_config (no inline _settings dict) + ai_config stub removed in favor of real ProviderConfig + load_provider_config_by_id helper
**Wave 4** *(blocked on Wave 3)* — Celery retry harness + Re-queue endpoint
- [ ] 07-04-PLAN.md — Celery extract_and_classify decorator gains bind=True + max_retries=3 + _ClassificationError sentinel + 30/90/270 backoff + _mark_classification_failed on exhaustion + POST /api/documents/{id}/classify changes to re-queue Celery (sets status=processing, calls .delay()) — promotes test_retry_backoff, test_exhaustion_sets_failed_status, test_reclassify_requeues_celery
**Wave 5** *(blocked on Wave 4)* — Admin UI + Frontend badge
- [ ] 07-05-PLAN.md — Admin AI-config backend (GET/PUT/test-connection with whitelist + audit logging + atomic is_active flip) + frontend API client helpers (getAiConfig/saveAiConfig/testAiConnection) + AdminAiConfigTab.vue global System AI Providers section ABOVE existing per-user table + DocumentCard.vue classification_failed badge + Re-analyze button + human checkpoint UAT
**Cross-cutting constraints:**
- `api_key_enc` is never returned by any admin endpoint — enforced by `_ai_config_to_dict()` whitelist (Plan 05)
- HKDF domain separation: AI settings use `info=b"ai-provider-settings"`, cloud credentials use `info=b"cloud-credentials"` (Plan 01)
- `is_active` flip is atomic: single `UPDATE SET is_active = (provider_id = $target)` — never read-then-write (Plan 05)
- Provider clients are stored as `self._client` in `__init__` — never recreated per call (Plans 02, 03)
- `MAX_AI_CHARS` removal must cover all three locations: openai_provider.py L5, anthropic_provider.py L5, classifier.py L28 (Plans 02 + 03)
- Celery `self.retry()` must be raised from the outer sync task body, never inside `asyncio.run()` (Plan 04 — Pitfall 3)
- `extra_hosts: ["host.docker.internal:host-gateway"]` already present in docker-compose.yml — D-14 is already done; no docker-compose changes required (RESEARCH.md)
- AdminAiConfigTab.vue per-user assignment table is preserved untouched; the new global system section is added ABOVE it (Plan 05 — Pitfall 6)
**Phase gates (must pass before Phase 7 is complete):**
- [ ] `pytest -v` — zero failures; D-01/D-03/D-05/D-06/D-07/D-09/D-10/D-11/D-12/D-13/D-16 covered by promoted tests
- [ ] Security agent: bandit -r backend/ (zero HIGH), pip audit (zero critical/high), npm audit --audit-level=high (zero high/critical)
- [ ] `_ai_config_to_dict()` confirmed to never include `api_key_enc` (integration test asserts response.text does not contain the key)
- [ ] HKDF domain separation verified: same master key + same plaintext produces different ciphertexts for `provider_id` vs `user_id` derivations
- [ ] Atomic `is_active` flip verified: after two PUTs with `is_active=true` on different providers, exactly one row has `is_active=true`
- [ ] Human checkpoint UAT approves admin panel + DocumentCard badge end-to-end
**UI hint**: yes
---
@@ -407,3 +456,4 @@ Plans:
| 6. Performance & Production Hardening | 0/TBD | Not started | — |
| 6.1. Close v1.0 audit gaps | 2/2 | Complete | 2026-05-30 |
| 6.2. Close v1 sharing + cloud-delete + CSV export gaps | 5/5 | Complete | 2026-05-31 |
| 7. Redo and optimize LLM integration | 0/5 | Planned | — |