docs(07-01): complete plan summary — system_settings DB + HKDF + Wave 0 stubs

Tasks: 3/3 complete. Commits: 4febe2f, 4eb3177, 0fd6930. Created system_settings table (Alembic 0005), SystemSettings ORM, HKDF AI key encryption (info=b"ai-provider-settings"), load_provider_config(), seed startup hook, and 13 Wave 0 xfail stubs across 4 new test files.
2026-06-04 18:51:12 +02:00
parent 0fd6930b41
commit 7986661333
1 changed files with 155 additions and 0 deletions
@@ -0,0 +1,155 @@
+---
+phase: 07-redo-and-optimize-llm-integration
+plan: "01"
+subsystem: backend/ai-config
+tags:
+  - database
+  - encryption
+  - hkdf
+  - testing
+  - wave-0-scaffold
+dependency_graph:
+  requires:
+    - "06-05 (trusted-proxy, per-account rate limiting)"
+  provides:
+    - "system_settings table (Alembic 0005)"
+    - "SystemSettings ORM model"
+    - "HKDF encryption helpers for AI API keys"
+    - "load_provider_config() DB reader"
+    - "seed_system_settings_from_env() startup hook"
+    - "Wave 0 xfail test stubs (13 new stubs)"
+  affects:
+    - "07-02 (ProviderConfig + GenericOpenAIProvider — depends on system_settings table)"
+    - "07-03 (Anthropic output_config — depends on load_provider_config)"
+    - "07-05 (Admin AI panel — depends on system_settings table)"
+tech_stack:
+  added: []
+  patterns:
+    - "HKDF-SHA256 with info=b\"ai-provider-settings\" (domain-separated from b\"cloud-credentials\")"
+    - "Fresh HKDF instance per call (AlreadyFinalized guard)"
+    - "Fernet symmetric encryption for API keys (no JSON wrapping)"
+    - "Lazy import of ProviderConfig inside load_provider_config() to avoid Plan 01/02 circular dep"
+    - "Try/except lifespan seed to survive pre-migration fresh container startup"
+key_files:
+  created:
+    - backend/migrations/versions/0005_system_settings.py
+    - backend/services/ai_config.py
+    - backend/tests/test_ai_providers.py
+    - backend/tests/test_ai_config.py
+    - backend/tests/test_admin_ai_config.py
+    - backend/tests/test_document_tasks.py
+  modified:
+    - backend/db/models.py
+    - backend/main.py
+    - backend/tests/test_documents.py
+decisions:
+  - "HKDF info=b\"ai-provider-settings\" enforces domain separation from cloud-credentials — same master key produces different derived Fernet keys"
+  - "Fresh HKDF instance per _derive_ai_settings_key() call (cryptography AlreadyFinalized invariant)"
+  - "_ProviderConfigStub introduced as stub until Plan 02 creates ai/provider_config.py; removed in Plan 03"
+  - "seed_system_settings_from_env wrapped in try/except in lifespan — pre-migration container startup must not crash"
+  - "UniqueConstraint uq_system_settings_provider_id on provider_id enforces one row per provider"
+  - "D-14 regression guard: docker-compose.yml retains 3 host-gateway entries (backend, celery-worker, celery-worker-two); all >= required 2"
+metrics:
+  duration: "~25 minutes"
+  completed: "2026-06-04"
+  tasks_completed: 3
+  tasks_total: 3
+  files_created: 6
+  files_modified: 3
+---
+
+# Phase 7 Plan 01: Database, Encryption, and Test Scaffolding Foundation Summary
+
+HKDF/Fernet encryption layer for AI provider API keys stored in a new `system_settings` DB table; Wave 0 xfail test stubs providing the full Phase 7 testing skeleton; startup seed from env vars.
+
+## Tasks Completed
+
+| Task | Description | Commit | Files |
+|------|-------------|--------|-------|
+| 1 | Wave 0 xfail stubs — 13 new stubs across 4 new files + 1 append | 4febe2f | test_ai_providers.py, test_ai_config.py, test_admin_ai_config.py, test_document_tasks.py, test_documents.py |
+| 2 | Alembic migration 0005 + SystemSettings ORM model | 4eb3177 | migrations/versions/0005_system_settings.py, db/models.py |
+| 3 | services/ai_config.py — HKDF helpers, loader, env seed + main.py wiring | 0fd6930 | services/ai_config.py, main.py |
+
+## What Was Built
+
+### Task 1: Wave 0 Test Scaffold
+
+Created four new test files and appended one stub to an existing file:
+
+- `test_ai_providers.py` — 7 stubs: test_generic_openai_json_mode, test_anthropic_structured_output, test_get_provider_typed, test_client_singleton, test_context_chars_truncation, test_smart_truncation, test_gemini_fallback_to_parse_classification
+- `test_ai_config.py` — 2 stubs: test_load_provider_config, test_api_key_encrypt_decrypt
+- `test_admin_ai_config.py` — 2 stubs: test_get_never_returns_key, test_put_writes_active_provider
+- `test_document_tasks.py` — 2 stubs: test_retry_backoff, test_exhaustion_sets_failed_status
+- `test_documents.py` — 1 appended stub: test_reclassify_requeues_celery
+
+All stubs use `@pytest.mark.xfail(strict=False, reason="Wave 0 stub — promoted in Plan 07-XX")` per the xfail(strict=False) STATE.md decision. Bodies are single `pytest.xfail()` calls only.
+
+D-14 regression guard verified: `docker-compose.yml` contains 3 `host.docker.internal:host-gateway` entries (backend + celery-worker + celery-worker-two), satisfying the minimum-2 requirement.
+
+### Task 2: Alembic Migration 0005 + SystemSettings ORM
+
+`backend/migrations/versions/0005_system_settings.py` creates the `system_settings` table with:
+- `id`: UUID PK with `gen_random_uuid()` server default
+- `provider_id`: String NOT NULL UNIQUE (uq_system_settings_provider_id constraint)
+- `api_key_enc`: Text NULL (Fernet-encrypted; NULL for local providers like Ollama)
+- `base_url`: Text NULL
+- `model_name`: Text NOT NULL, default ''
+- `context_chars`: Integer NOT NULL, default 8000
+- `is_active`: Boolean NOT NULL, default false
+- `created_at` / `updated_at`: TIMESTAMPTZ NOT NULL, default now()
+
+`backend/db/models.py` gains the `SystemSettings(Base)` ORM model with identical column declarations using `Mapped[]` type syntax, mirroring the CloudConnection pattern.
+
+### Task 3: services/ai_config.py
+
+Complete encryption + loader + seed module:
+
+- `_derive_ai_settings_key(master_key, provider_id)`: Creates FRESH HKDF on every call (AlreadyFinalized guard), `salt=provider_id.encode("utf-8")`, `info=b"ai-provider-settings"` (domain-separated from `b"cloud-credentials"`)
+- `encrypt_api_key(master_key, provider_id, api_key)`: `Fernet.encrypt(api_key.encode()).decode()` — no JSON wrapping
+- `decrypt_api_key(master_key, provider_id, api_key_enc)`: `Fernet.decrypt(...).decode()` — no JSON unwrapping
+- `load_provider_config(session)`: Reads `is_active=True` row, decrypts API key, returns `_ProviderConfigStub` (upgraded to real `ProviderConfig` lazily once Plan 02 lands)
+- `seed_system_settings_from_env(session)`: Inserts default provider row from `settings.default_ai_provider / default_ai_model` when no row exists for that provider_id; idempotent
+- `_ProviderConfigStub`: Minimal Pydantic model stub; removed in Plan 03
+
+`backend/main.py` updated to import and call `seed_system_settings_from_env` inside the async lifespan with `try/except` so that a missing table (pre-migration fresh container) logs a warning and skips rather than crashing.
+
+## Verification Results
+
+- Round-trip smoke test: `encrypt_api_key(mk, 'openai', 'sk-test')` → `decrypt_api_key(mk, 'openai', ct) == 'sk-test'` — PASS
+- Domain salt isolation: ciphertext for `provider_id="openai"` vs `provider_id="anthropic"` differs — PASS
+- AlreadyFinalized guard: `_derive_ai_settings_key()` called twice without error — PASS
+- Domain separation: `info=b"ai-provider-settings"` present; `info=b"cloud-credentials"` absent from ai_config.py — PASS
+- `SystemSettings.__table_args__` contains `UniqueConstraint(name="uq_system_settings_provider_id")` — PASS
+- `from db.models import SystemSettings` exits 0 — PASS
+- Full suite: 1 failed (pre-existing test_extract_docx), 357 passed, 21 xfailed — PASS (no new failures)
+
+## Deviations from Plan
+
+None — plan executed exactly as written.
+
+## Known Stubs
+
+- `_ProviderConfigStub` in `services/ai_config.py` lines 44-50: intentional temporary stub; Plan 02 creates `ai/provider_config.py` and `load_provider_config()` will import `ProviderConfig` from there via lazy import.
+
+## Threat Flags
+
+No new threat surface introduced beyond what is described in the plan's `<threat_model>`. The `system_settings` table is accessed only by the `seed_system_settings_from_env` startup hook and the `load_provider_config` service function. No new API endpoints or network-accessible paths are added in this plan.
+
+## Self-Check: PASSED
+
+Files created/modified:
+
+- [x] backend/migrations/versions/0005_system_settings.py — FOUND
+- [x] backend/db/models.py — FOUND (SystemSettings class present)
+- [x] backend/services/ai_config.py — FOUND
+- [x] backend/main.py — FOUND (seed call present)
+- [x] backend/tests/test_ai_providers.py — FOUND (7 xfail stubs)
+- [x] backend/tests/test_ai_config.py — FOUND (2 xfail stubs)
+- [x] backend/tests/test_admin_ai_config.py — FOUND (2 xfail stubs)
+- [x] backend/tests/test_document_tasks.py — FOUND (2 xfail stubs)
+- [x] backend/tests/test_documents.py — FOUND (test_reclassify_requeues_celery appended)
+
+Commits:
+- [x] 4febe2f — test(07-01): Wave 0 xfail stubs
+- [x] 4eb3177 — feat(07-01): Alembic migration + SystemSettings ORM
+- [x] 0fd6930 — feat(07-01): services/ai_config.py + main.py wiring