kite/.planning/phases/07-redo-and-optimize-llm-integration/07-CONTEXT.md

Phase 7: Redo and Optimize LLM Integration - Context

Gathered: 2026-06-02 Status: Ready for planning

## Phase Boundary

Phase 7 delivers a fully refactored, production-reliable AI provider layer: a system_settings DB table so admins can configure all provider settings interactively, a GenericOpenAIProvider class that covers all OpenAI-compatible providers, native Gemini and Mistral integrations, JSON-mode structured output enforced across all providers, Celery-based retry with exponential backoff, per-provider context window configuration with smart truncation, a fix for the LMStudio Linux connectivity issue, and a re-classify UI button for failed documents. The admin panel gains an AI Providers configuration section.

## Implementation Decisions

Structured Output

• D-01: All OpenAI-compatible providers (OpenAI, Ollama, LMStudio, Groq, Gemini-compat, Mistral-compat, xAI/Grok, DeepSeek, OpenRouter) must pass response_format={"type": "json_object"} in every classify() and suggest_topics() call.
• D-02: Keep parse_classification() and parse_suggestions() in ai/utils.py as a last-resort fallback only — they are called when a provider does not support response_format (e.g., some local Ollama models that ignore the parameter).
• D-03: Anthropic structured output strategy is left to Claude's discretion (see below).

Provider Factory & Configuration

• D-04: All AI provider settings (API keys, base URLs, model names, active provider) are stored in a new system_settings DB table. Env vars (existing DEFAULT_AI_PROVIDER, DEFAULT_AI_MODEL, API key vars) serve as startup defaults that populate the table on first boot, but the DB record is authoritative at runtime.
• D-05: API keys stored in system_settings must be encrypted at rest — use the same HKDF/AES-GCM pattern already established for cloud credentials in backend/storage/cloud_utils.py.
• D-06: get_provider() accepts typed Pydantic ProviderConfig models (not raw dicts). The classifier assembles a ProviderConfig from the DB record — no more inline dict construction.
• D-07: Client lifecycle (singleton, pooled, per-request) is left to Claude's discretion per provider — the researcher must document the optimal pattern for each provider so new providers can follow the same established pattern.
• D-08: Admin panel gains an "AI Providers" configuration section where admins can set the active provider, configure API keys/base URLs/models, and test connectivity via the existing health_check() method.

Retry & Resilience

• D-09: The extract_and_classify Celery task uses Celery's built-in retry mechanism (self.retry(exc=…, countdown=…, max_retries=3)) with exponential backoff: first retry at 30 s, second at 90 s, third at 270 s.
• D-10: After all retries are exhausted the document status remains classification_failed. No silent failure.
• D-11: The document card in the frontend shows a "Classification failed" badge when status == "classification_failed". A "Re-analyze" button is shown that calls POST /api/documents/{id}/classify to re-queue the Celery task.

Context Window Strategy

• D-12: MAX_AI_CHARS is replaced by a per-provider context_chars value declared in each provider's config (or ProviderConfig default). The global constant is removed.
• D-13: Smart truncation strategy: take the first 60% of the per-provider limit + the last 40%. This captures both document introduction and conclusion, which carry the most topic signal for long documents.

LMStudio Linux Fix

• D-14: Add extra_hosts: ["host.docker.internal:host-gateway"] to the backend and celery-worker services in docker-compose.yml. This is the standard Linux Docker fix for accessing host services.
• D-15: LMStudio base URL and model are configurable via the admin AI Providers panel (D-08), so users are not locked to the hardcoded host.docker.internal:1234 default.

New Providers

• D-16: Introduce a GenericOpenAIProvider(base_url, api_key, model, context_chars) class in backend/ai/generic_openai_provider.py. It subclasses OpenAIProvider and adds JSON-mode enforcement. Named presets (Groq, xAI/Grok, DeepSeek, OpenRouter, Gemini-compat, Mistral-compat) are factory shortcuts that pass the known base_url default.
• D-17: Providers to add in this phase: Google Gemini (via OpenAI-compat endpoint at generativelanguage.googleapis.com/v1beta/openai/), Mistral (via OpenAI-compat), Groq, xAI/Grok, DeepSeek, OpenRouter — all via GenericOpenAIProvider.
• D-18: Mistral uses the OpenAI-compat endpoint (not the native mistralai SDK) to avoid adding a new dependency. Can be revisited in a later phase if Mistral-specific features (embeddings, FIM) are needed.

Claude's Discretion

• D-03 (Anthropic output): Researcher should determine whether Anthropic uses native tool_use JSON schema, Anthropic's response_format (if supported), or prompt-enforced JSON. Choose the most reliable option that is consistent with the JSON-mode-everywhere principle (D-01).
• D-07 (Client lifecycle): Researcher must investigate and document optimal client instantiation per provider (singleton, per-request, pooled). The documentation must be clear enough that adding a new provider in the future does not require re-researching this.

<canonical_refs>

Canonical References

Downstream agents MUST read these before planning or implementing.

Existing AI provider code

• backend/ai/base.py — AIProvider ABC and ClassificationResult dataclass; the interface all providers must implement
• backend/ai/utils.py — strip_code_fences, parse_classification, parse_suggestions; fallback parsers that remain but are demoted
• backend/ai/__init__.py — get_provider(settings: dict) factory; this is the function being replaced/refactored
• backend/ai/anthropic_provider.py — current Anthropic implementation; reference for what the refactored version must preserve
• backend/ai/openai_provider.py — current OpenAI/base-compat implementation; GenericOpenAIProvider builds on this
• backend/ai/ollama_provider.py — one-line subclass pattern for Ollama; shows the current named-preset approach
• backend/ai/lmstudio_provider.py — same pattern as Ollama; root cause of Linux connection issue is here (hardcoded host.docker.internal)

Classification orchestrator

• backend/services/classifier.py — classify_document() and suggest_topics_for_document(); inline _settings dict construction that must be replaced with ProviderConfig (D-06)
• backend/tasks/document_tasks.py — extract_and_classify Celery task; retry logic (D-09) goes here

Configuration and encryption

• backend/config.py — current default_ai_provider, default_ai_model, system_prompt settings; these become DB defaults (D-04)
• backend/storage/cloud_utils.py — HKDF/AES-GCM encryption pattern that D-05 requires for API key storage in system_settings

Infrastructure

• docker-compose.yml — extra_hosts must be added to backend and celery-worker services (D-14)

Architecture context

• .planning/codebase/ARCHITECTURE.md — system overview; AI providers shown at bottom-right of the diagram
• CLAUDE.md §"Backend: shared module map" — backend/ai/utils.py must remain the single location for AI parsing helpers

ROADMAP

• .planning/ROADMAP.md Phase 7 entry — goal and dependencies (depends on Phase 6)

</canonical_refs>

<code_context>

Existing Code Insights

Reusable Assets

• OpenAIProvider (backend/ai/openai_provider.py): GenericOpenAIProvider subclasses this directly — JSON mode is added in one override.
• HKDF key derivation in backend/storage/cloud_utils.py:_derive_fernet_key(): the same pattern (fresh HKDF instance per call, never cache) must be replicated for system_settings API key encryption (D-05).
• write_audit_log() in backend/services/storage.py: admin AI config changes should be logged as audit events.
• Existing health_check() on all providers: reuse this for the "Test connection" button in the admin AI Providers panel (D-08).

Established Patterns

• Celery retry: document_tasks.py currently catches exceptions and returns {"status": "classification_failed"}. Refactor to raise self.retry(exc=e, countdown=30, max_retries=3) in the task decorator.
• StorageBackend ABC pattern: the AIProvider ABC (base.py) already mirrors this. New providers follow the same subclassing pattern.
• Admin endpoint whitelist: _user_to_dict() pattern in backend/api/admin.py — new system_settings endpoints must never return encrypted API keys in responses (same principle as credentials_enc).
• get_provider() factory: currently a flat if/elif chain; refactor to a registry dict {"anthropic": AnthropicProvider, "groq": GenericOpenAIProvider, …} to make adding new providers O(1).

Integration Points

• backend/api/admin.py — new AI Providers endpoints (GET /api/admin/ai-config, PUT /api/admin/ai-config) added here; admin-only via get_current_admin dep.
• backend/services/classifier.py — replace inline _settings dict with a load_provider_config(session) helper that reads from system_settings table.
• frontend/src/views/AdminView.vue / frontend/src/components/admin/ — new AdminAITab.vue component following the existing tab pattern (AdminUsersTab, AdminQuotasTab, AdminAITab).
• frontend/src/components/documents/DocumentCard.vue — add "Classification failed" badge and "Re-analyze" button (D-11).

</code_context>

## Specific Ideas

• LMStudio Linux fix is purely a Docker Compose extra_hosts addition + JSON mode enforcement — no changes to lmstudio_provider.py logic itself needed beyond what the provider refactor provides.
• GenericOpenAIProvider named presets: the factory registry entry for "groq" passes base_url="https://api.groq.com/openai/v1", for "xai" passes base_url="https://api.x.ai/v1", for "deepseek" passes base_url="https://api.deepseek.com", for "openrouter" passes base_url="https://openrouter.ai/api/v1", for "gemini" passes base_url="https://generativelanguage.googleapis.com/v1beta/openai/".
• Admin panel "AI Providers" section should show: active provider selector, per-provider accordion (API key field masked, base URL, model, context window size), "Test connection" button that calls health_check().

## Deferred Ideas

• Chunk + merge multi-call classification — splitting long documents into N chunks and merging results was considered but deferred. More accurate for very long docs but 2-5x API cost. Worth a future phase once the provider layer is stable.
• Native mistralai SDK — Mistral-specific features (embeddings, FIM/code completion) are not needed for document classification. OpenAI-compat is sufficient now; revisit when embedding-based retrieval is added.
• Token counting — true token counting per provider (using tiktoken or provider-specific tokenizers) was considered for context window management. Per-provider context_chars approximation is sufficient for Phase 7; precise token counting can be added in a future optimization phase.
• Streaming classification — streaming LLM responses for faster UX was identified but deferred; classification is a background Celery task so streaming to the user is not straightforward without a websocket layer.

---

Phase: 07-redo-and-optimize-llm-integration Context gathered: 2026-06-02