kite/.planning/phases/07-redo-and-optimize-llm-integration/07-VALIDATION.md

phase, slug, status, nyquist_compliant, wave_0_complete, created

phase	slug	status	nyquist_compliant	wave_0_complete	created
7	redo-and-optimize-llm-integration	draft	false	false	2026-06-02

Phase 7 — Validation Strategy

Per-phase validation contract for feedback sampling during execution.

---

Test Infrastructure

Property	Value
Framework	pytest + pytest-asyncio
Config file	backend/pyproject.toml or pytest.ini (existing)
Quick run command	pytest backend/tests/test_classifier.py backend/tests/test_ai_providers.py -x
Full suite command	pytest backend/tests/ -v
Estimated runtime	~60 seconds

---

Sampling Rate

• After every task commit: Run pytest backend/tests/test_classifier.py backend/tests/test_ai_providers.py -x
• After every plan wave: Run pytest backend/tests/ -v
• Before /gsd:verify-work: Full suite must be green
• Max feedback latency: 60 seconds

---

Per-Task Verification Map

Task ID	Plan	Wave	Requirement	Threat Ref	Secure Behavior	Test Type	Automated Command	File Exists	Status
07-01-01	01	1	D-04	T-07-01	system_settings API key never returned in GET response	unit	pytest backend/tests/test_ai_config.py::test_load_provider_config -x	❌ W0	⬜ pending
07-01-02	01	1	D-05	T-07-02	HKDF key derivation with provider_id salt	unit	pytest backend/tests/test_ai_config.py::test_api_key_encrypt_decrypt -x	❌ W0	⬜ pending
07-02-01	02	2	D-06	—	get_provider() accepts ProviderConfig, not raw dict	unit	pytest backend/tests/test_ai_providers.py::test_get_provider_typed -x	❌ W0	⬜ pending
07-02-02	02	2	D-07	—	OpenAIProvider._client is singleton (not recreated per call)	unit	pytest backend/tests/test_ai_providers.py::test_client_singleton -x	❌ W0	⬜ pending
07-02-03	02	2	D-16	—	GenericOpenAIProvider passes response_format json_object	unit	pytest backend/tests/test_ai_providers.py::test_generic_openai_json_mode -x	❌ W0	⬜ pending
07-03-01	03	3	D-03	—	AnthropicProvider passes output_config json_schema	unit	pytest backend/tests/test_ai_providers.py::test_anthropic_structured_output -x	❌ W0	⬜ pending
07-03-02	03	3	D-12/D-13	—	Smart truncation: 60% head + 40% tail per provider context_chars	unit	pytest backend/tests/test_ai_providers.py::test_smart_truncation -x	❌ W0	⬜ pending
07-04-01	04	4	D-09	—	Celery retry countdown: 30s, 90s, 270s	unit (mock)	pytest backend/tests/test_document_tasks.py::test_retry_backoff -x	❌ W0	⬜ pending
07-04-02	04	4	D-10	—	After 3 retries doc.status = classification_failed	unit (mock)	pytest backend/tests/test_document_tasks.py::test_exhaustion_sets_failed_status -x	❌ W0	⬜ pending
07-04-03	04	4	D-11	—	POST /api/documents/{id}/classify re-queues Celery	integration	pytest backend/tests/test_documents.py::test_reclassify_requeues_celery -x	❌ W0	⬜ pending
07-05-01	05	5	D-05/D-08	T-07-03	GET /api/admin/ai-config never returns api_key_enc	integration	pytest backend/tests/test_admin_ai_config.py::test_get_never_returns_key -x	❌ W0	⬜ pending

Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky

---

Wave 0 Requirements

• backend/tests/test_ai_providers.py — stubs for D-01, D-03, D-06, D-07, D-12, D-13, D-16
• backend/tests/test_ai_config.py — stubs for D-04, D-05 (encryption round-trip + DB read)
• backend/tests/test_admin_ai_config.py — stubs for admin endpoint security (never returns key)
• backend/tests/test_document_tasks.py — additional stubs for D-09, D-10 (existing file, add to it)

Note: backend/tests/test_classifier.py and backend/tests/test_documents.py already exist — add stubs for D-11 to the documents test file only.

---

Manual-Only Verifications

Behavior	Requirement	Why Manual	Test Instructions
Admin AI Providers panel shows masked API key field (never pre-filled from server)	D-08/D-05	Frontend write-only input cannot be automated without browser	Open admin panel, navigate to AI Providers tab, confirm API key field is empty/masked on load
Re-analyze button appears on DocumentCard for classification_failed documents	D-11	Visual state test	Upload a document, force classification_failed status in DB, reload page, confirm badge + button

---

Validation Sign-Off

• All tasks have <automated> verify or Wave 0 dependencies
• Sampling continuity: no 3 consecutive tasks without automated verify
• Wave 0 covers all MISSING references
• No watch-mode flags
• Feedback latency < 60s
• nyquist_compliant: true set in frontmatter

Approval: pending