---
phase: 07-redo-and-optimize-llm-integration
plan: "01"
subsystem: backend/ai-config
tags:
  - database
  - encryption
  - hkdf
  - testing
  - wave-0-scaffold
dependency_graph:
  requires:
    - "06-05 (trusted-proxy, per-account rate limiting)"
  provides:
    - "system_settings table (Alembic 0005)"
    - "SystemSettings ORM model"
    - "HKDF encryption helpers for AI API keys"
    - "load_provider_config() DB reader"
    - "seed_system_settings_from_env() startup hook"
    - "Wave 0 xfail test stubs (13 new stubs)"
  affects:
    - "07-02 (ProviderConfig + GenericOpenAIProvider — depends on system_settings table)"
    - "07-03 (Anthropic output_config — depends on load_provider_config)"
    - "07-05 (Admin AI panel — depends on system_settings table)"
tech_stack:
  added: []
  patterns:
    - "HKDF-SHA256 with info=b\"ai-provider-settings\" (domain-separated from b\"cloud-credentials\")"
    - "Fresh HKDF instance per call (AlreadyFinalized guard)"
    - "Fernet symmetric encryption for API keys (no JSON wrapping)"
    - "Lazy import of ProviderConfig inside load_provider_config() to avoid Plan 01/02 circular dep"
    - "Try/except lifespan seed to survive pre-migration fresh container startup"
key_files:
  created:
    - backend/migrations/versions/0005_system_settings.py
    - backend/services/ai_config.py
    - backend/tests/test_ai_providers.py
    - backend/tests/test_ai_config.py
    - backend/tests/test_admin_ai_config.py
    - backend/tests/test_document_tasks.py
  modified:
    - backend/db/models.py
    - backend/main.py
    - backend/tests/test_documents.py
decisions:
  - "HKDF info=b\"ai-provider-settings\" enforces domain separation from cloud-credentials — same master key produces different derived Fernet keys"
  - "Fresh HKDF instance per _derive_ai_settings_key() call (cryptography AlreadyFinalized invariant)"
  - "_ProviderConfigStub introduced as stub until Plan 02 creates ai/provider_config.py; removed in Plan 03"
  - "seed_system_settings_from_env wrapped in try/except in lifespan — pre-migration container startup must not crash"
  - "UniqueConstraint uq_system_settings_provider_id on provider_id enforces one row per provider"
  - "D-14 regression guard: docker-compose.yml retains 3 host-gateway entries (backend, celery-worker, celery-worker-two); all >= required 2"
metrics:
  duration: "~25 minutes"
  completed: "2026-06-04"
  tasks_completed: 3
  tasks_total: 3
  files_created: 6
  files_modified: 3
---

# Phase 7 Plan 01: Database, Encryption, and Test Scaffolding Foundation Summary

HKDF/Fernet encryption layer for AI provider API keys stored in a new `system_settings` DB table; Wave 0 xfail test stubs providing the full Phase 7 testing skeleton; startup seed from env vars.

## Tasks Completed

| Task | Description | Commit | Files |
|------|-------------|--------|-------|
| 1 | Wave 0 xfail stubs — 13 new stubs across 4 new files + 1 append | 4febe2f | test_ai_providers.py, test_ai_config.py, test_admin_ai_config.py, test_document_tasks.py, test_documents.py |
| 2 | Alembic migration 0005 + SystemSettings ORM model | 4eb3177 | migrations/versions/0005_system_settings.py, db/models.py |
| 3 | services/ai_config.py — HKDF helpers, loader, env seed + main.py wiring | 0fd6930 | services/ai_config.py, main.py |

## What Was Built

### Task 1: Wave 0 Test Scaffold

Created four new test files and appended one stub to an existing file:

- `test_ai_providers.py` — 7 stubs: test_generic_openai_json_mode, test_anthropic_structured_output, test_get_provider_typed, test_client_singleton, test_context_chars_truncation, test_smart_truncation, test_gemini_fallback_to_parse_classification
- `test_ai_config.py` — 2 stubs: test_load_provider_config, test_api_key_encrypt_decrypt
- `test_admin_ai_config.py` — 2 stubs: test_get_never_returns_key, test_put_writes_active_provider
- `test_document_tasks.py` — 2 stubs: test_retry_backoff, test_exhaustion_sets_failed_status
- `test_documents.py` — 1 appended stub: test_reclassify_requeues_celery

All stubs use `@pytest.mark.xfail(strict=False, reason="Wave 0 stub — promoted in Plan 07-XX")` per the xfail(strict=False) STATE.md decision. Bodies are single `pytest.xfail()` calls only.

D-14 regression guard verified: `docker-compose.yml` contains 3 `host.docker.internal:host-gateway` entries (backend + celery-worker + celery-worker-two), satisfying the minimum-2 requirement.

### Task 2: Alembic Migration 0005 + SystemSettings ORM

`backend/migrations/versions/0005_system_settings.py` creates the `system_settings` table with:
- `id`: UUID PK with `gen_random_uuid()` server default
- `provider_id`: String NOT NULL UNIQUE (uq_system_settings_provider_id constraint)
- `api_key_enc`: Text NULL (Fernet-encrypted; NULL for local providers like Ollama)
- `base_url`: Text NULL
- `model_name`: Text NOT NULL, default ''
- `context_chars`: Integer NOT NULL, default 8000
- `is_active`: Boolean NOT NULL, default false
- `created_at` / `updated_at`: TIMESTAMPTZ NOT NULL, default now()

`backend/db/models.py` gains the `SystemSettings(Base)` ORM model with identical column declarations using `Mapped[]` type syntax, mirroring the CloudConnection pattern.

### Task 3: services/ai_config.py

Complete encryption + loader + seed module:

- `_derive_ai_settings_key(master_key, provider_id)`: Creates FRESH HKDF on every call (AlreadyFinalized guard), `salt=provider_id.encode("utf-8")`, `info=b"ai-provider-settings"` (domain-separated from `b"cloud-credentials"`)
- `encrypt_api_key(master_key, provider_id, api_key)`: `Fernet.encrypt(api_key.encode()).decode()` — no JSON wrapping
- `decrypt_api_key(master_key, provider_id, api_key_enc)`: `Fernet.decrypt(...).decode()` — no JSON unwrapping
- `load_provider_config(session)`: Reads `is_active=True` row, decrypts API key, returns `_ProviderConfigStub` (upgraded to real `ProviderConfig` lazily once Plan 02 lands)
- `seed_system_settings_from_env(session)`: Inserts default provider row from `settings.default_ai_provider / default_ai_model` when no row exists for that provider_id; idempotent
- `_ProviderConfigStub`: Minimal Pydantic model stub; removed in Plan 03

`backend/main.py` updated to import and call `seed_system_settings_from_env` inside the async lifespan with `try/except` so that a missing table (pre-migration fresh container) logs a warning and skips rather than crashing.

## Verification Results

- Round-trip smoke test: `encrypt_api_key(mk, 'openai', 'sk-test')` → `decrypt_api_key(mk, 'openai', ct) == 'sk-test'` — PASS
- Domain salt isolation: ciphertext for `provider_id="openai"` vs `provider_id="anthropic"` differs — PASS
- AlreadyFinalized guard: `_derive_ai_settings_key()` called twice without error — PASS
- Domain separation: `info=b"ai-provider-settings"` present; `info=b"cloud-credentials"` absent from ai_config.py — PASS
- `SystemSettings.__table_args__` contains `UniqueConstraint(name="uq_system_settings_provider_id")` — PASS
- `from db.models import SystemSettings` exits 0 — PASS
- Full suite: 1 failed (pre-existing test_extract_docx), 357 passed, 21 xfailed — PASS (no new failures)

## Deviations from Plan

None — plan executed exactly as written.

## Known Stubs

- `_ProviderConfigStub` in `services/ai_config.py` lines 44-50: intentional temporary stub; Plan 02 creates `ai/provider_config.py` and `load_provider_config()` will import `ProviderConfig` from there via lazy import.

## Threat Flags

No new threat surface introduced beyond what is described in the plan's `<threat_model>`. The `system_settings` table is accessed only by the `seed_system_settings_from_env` startup hook and the `load_provider_config` service function. No new API endpoints or network-accessible paths are added in this plan.

## Self-Check: PASSED

Files created/modified:

- [x] backend/migrations/versions/0005_system_settings.py — FOUND
- [x] backend/db/models.py — FOUND (SystemSettings class present)
- [x] backend/services/ai_config.py — FOUND
- [x] backend/main.py — FOUND (seed call present)
- [x] backend/tests/test_ai_providers.py — FOUND (7 xfail stubs)
- [x] backend/tests/test_ai_config.py — FOUND (2 xfail stubs)
- [x] backend/tests/test_admin_ai_config.py — FOUND (2 xfail stubs)
- [x] backend/tests/test_document_tasks.py — FOUND (2 xfail stubs)
- [x] backend/tests/test_documents.py — FOUND (test_reclassify_requeues_celery appended)

Commits:
- [x] 4febe2f — test(07-01): Wave 0 xfail stubs
- [x] 4eb3177 — feat(07-01): Alembic migration + SystemSettings ORM
- [x] 0fd6930 — feat(07-01): services/ai_config.py + main.py wiring