kite/.planning/phases/07-redo-and-optimize-llm-integration/07-VALIDATION.md

phase, slug, status, nyquist_compliant, wave_0_complete, created, audited

phase	slug	status	nyquist_compliant	wave_0_complete	created	audited
7	redo-and-optimize-llm-integration	complete	true	true	2026-06-02	2026-06-05

Phase 7 — Validation Strategy

Per-phase validation contract for feedback sampling during execution.

---

Test Infrastructure

Property	Value
Framework	pytest + pytest-asyncio
Config file	backend/pyproject.toml or pytest.ini (existing)
Quick run command	pytest backend/tests/test_classifier.py backend/tests/test_ai_providers.py -x
Full suite command	pytest backend/tests/ -v
Estimated runtime	~60 seconds

---

Sampling Rate

• After every task commit: Run pytest backend/tests/test_classifier.py backend/tests/test_ai_providers.py -x
• After every plan wave: Run pytest backend/tests/ -v
• Before /gsd:verify-work: Full suite must be green
• Max feedback latency: 60 seconds

---

Per-Task Verification Map

Task ID	Plan	Wave	Requirement	Threat Ref	Secure Behavior	Test Type	Automated Command	File Exists	Status
07-01-01	01	1	D-04	T-07-01	system_settings API key never returned in GET response	integration	INTEGRATION=1 pytest backend/tests/test_ai_config.py::test_load_provider_config -x	✅	✅ green (skipped w/o INTEGRATION=1)
07-01-02	01	1	D-05	T-07-02	HKDF key derivation with provider_id salt	unit	pytest backend/tests/test_ai_config.py::test_api_key_encrypt_decrypt -x	✅	✅ green
07-02-01	02	2	D-06	—	get_provider() accepts ProviderConfig, not raw dict	unit	pytest backend/tests/test_ai_providers.py::test_get_provider_typed -x	✅	✅ green
07-02-02	02	2	D-07	—	OpenAIProvider._client is singleton (not recreated per call)	unit	pytest backend/tests/test_ai_providers.py::test_client_singleton -x	✅	✅ green
07-02-03	02	2	D-16	—	GenericOpenAIProvider passes response_format json_object	unit	pytest backend/tests/test_ai_providers.py::test_generic_openai_json_mode -x	✅	✅ green
07-03-01	03	3	D-03	—	AnthropicProvider passes output_config json_schema	unit	pytest backend/tests/test_ai_providers.py::test_anthropic_structured_output -x	✅	✅ green
07-03-02	03	3	D-12/D-13	—	Smart truncation: 60% head + 40% tail per provider context_chars	unit	pytest backend/tests/test_ai_providers.py::test_smart_truncation -x	✅	✅ green
07-04-01	04	4	D-09	—	Celery retry countdown: 30s, 90s, 270s	unit (mock)	pytest backend/tests/test_document_tasks.py::test_retry_backoff -x	✅	✅ green
07-04-02	04	4	D-10	—	After 3 retries doc.status = classification_failed	unit (mock)	pytest backend/tests/test_document_tasks.py::test_exhaustion_sets_failed_status -x	✅	✅ green
07-04-03	04	4	D-11	—	POST /api/documents/{id}/classify re-queues Celery	integration	pytest backend/tests/test_documents.py::test_reclassify_requeues_celery -x	✅	✅ green
07-05-01	05	5	D-05/D-08	T-07-03	GET /api/admin/ai-config never returns api_key_enc	integration	pytest backend/tests/test_admin_ai_config.py::test_get_never_returns_key -x	✅	✅ green

Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky

---

Wave 0 Requirements

• backend/tests/test_ai_providers.py — covers D-03, D-06, D-07, D-12, D-13, D-16
• backend/tests/test_ai_config.py — covers D-04, D-05 (encryption round-trip + DB read)
• backend/tests/test_admin_ai_config.py — covers admin endpoint security (never returns key)
• backend/tests/test_document_tasks.py — covers D-09, D-10

All Wave 0 files exist and tests are implemented.

---

Manual-Only Verifications

Behavior	Requirement	Why Manual	Test Instructions
Admin AI Providers panel shows masked API key field (never pre-filled from server)	D-08/D-05	Frontend write-only input cannot be automated without browser	Open admin panel, navigate to AI Providers tab, confirm API key field is empty/masked on load
Re-analyze button appears on DocumentCard for classification_failed documents	D-11	Visual state test	Upload a document, force classification_failed status in DB, reload page, confirm badge + button

---

Validation Sign-Off

• All tasks have automated verify command
• Sampling continuity: no 3 consecutive tasks without automated verify
• Wave 0 covers all MISSING references
• No watch-mode flags
• Feedback latency < 60s (actual: 1.11s for 11 tests)
• nyquist_compliant: true set in frontmatter

Approval: 2026-06-05

---

Validation Audit 2026-06-05

Metric	Count
Gaps found	0
Resolved	0
Escalated	0
Tests green	10
Tests skipped (integration, by design)	1
Total	11

All 11 test functions exist and are implemented. 10 pass in unit mode; 1 (test_load_provider_config) is correctly gated behind INTEGRATION=1 because it requires a live PostgreSQL session. The test implementation is complete and correct — it is not a gap.