curo/kite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(07): capture phase context

Browse Source
This commit is contained in:
curo1305
2026-06-02 20:24:08 +02:00
parent ad237dbaf1
commit b8d128ee5e
2 changed files with 310 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/phases/07-redo-and-optimize-llm-integration/07-CONTEXT.md
+131
View File
@@ -0,0 +1,131 @@
# Phase 7: Redo and Optimize LLM Integration - Context
**Gathered:** 2026-06-02
**Status:** Ready for planning
<domain>
## Phase Boundary
Phase 7 delivers a fully refactored, production-reliable AI provider layer: a `system_settings` DB table so admins can configure all provider settings interactively, a `GenericOpenAIProvider` class that covers all OpenAI-compatible providers, native Gemini and Mistral integrations, JSON-mode structured output enforced across all providers, Celery-based retry with exponential backoff, per-provider context window configuration with smart truncation, a fix for the LMStudio Linux connectivity issue, and a re-classify UI button for failed documents. The admin panel gains an AI Providers configuration section.
</domain>
<decisions>
## Implementation Decisions
### Structured Output
- **D-01:** All OpenAI-compatible providers (OpenAI, Ollama, LMStudio, Groq, Gemini-compat, Mistral-compat, xAI/Grok, DeepSeek, OpenRouter) must pass `response_format={"type": "json_object"}` in every `classify()` and `suggest_topics()` call.
- **D-02:** Keep `parse_classification()` and `parse_suggestions()` in `ai/utils.py` as a last-resort fallback only — they are called when a provider does not support `response_format` (e.g., some local Ollama models that ignore the parameter).
- **D-03:** Anthropic structured output strategy is left to Claude's discretion (see below).
### Provider Factory & Configuration
- **D-04:** All AI provider settings (API keys, base URLs, model names, active provider) are stored in a new `system_settings` DB table. Env vars (existing `DEFAULT_AI_PROVIDER`, `DEFAULT_AI_MODEL`, API key vars) serve as startup defaults that populate the table on first boot, but the DB record is authoritative at runtime.
- **D-05:** API keys stored in `system_settings` must be encrypted at rest — use the same HKDF/AES-GCM pattern already established for cloud credentials in `backend/storage/cloud_utils.py`.
- **D-06:** `get_provider()` accepts typed Pydantic `ProviderConfig` models (not raw dicts). The classifier assembles a `ProviderConfig` from the DB record — no more inline dict construction.
- **D-07:** Client lifecycle (singleton, pooled, per-request) is left to Claude's discretion per provider — the researcher must document the optimal pattern for each provider so new providers can follow the same established pattern.
- **D-08:** Admin panel gains an "AI Providers" configuration section where admins can set the active provider, configure API keys/base URLs/models, and test connectivity via the existing `health_check()` method.
### Retry & Resilience
- **D-09:** The `extract_and_classify` Celery task uses Celery's built-in retry mechanism (`self.retry(exc=…, countdown=…, max_retries=3)`) with exponential backoff: first retry at 30 s, second at 90 s, third at 270 s.
- **D-10:** After all retries are exhausted the document status remains `classification_failed`. No silent failure.
- **D-11:** The document card in the frontend shows a "Classification failed" badge when `status == "classification_failed"`. A "Re-analyze" button is shown that calls `POST /api/documents/{id}/classify` to re-queue the Celery task.
### Context Window Strategy
- **D-12:** `MAX_AI_CHARS` is replaced by a per-provider `context_chars` value declared in each provider's config (or `ProviderConfig` default). The global constant is removed.
- **D-13:** Smart truncation strategy: take the first 60% of the per-provider limit + the last 40%. This captures both document introduction and conclusion, which carry the most topic signal for long documents.
### LMStudio Linux Fix
- **D-14:** Add `extra_hosts: ["host.docker.internal:host-gateway"]` to the `backend` and `celery-worker` services in `docker-compose.yml`. This is the standard Linux Docker fix for accessing host services.
- **D-15:** LMStudio base URL and model are configurable via the admin AI Providers panel (D-08), so users are not locked to the hardcoded `host.docker.internal:1234` default.
### New Providers
- **D-16:** Introduce a `GenericOpenAIProvider(base_url, api_key, model, context_chars)` class in `backend/ai/generic_openai_provider.py`. It subclasses `OpenAIProvider` and adds JSON-mode enforcement. Named presets (Groq, xAI/Grok, DeepSeek, OpenRouter, Gemini-compat, Mistral-compat) are factory shortcuts that pass the known `base_url` default.
- **D-17:** Providers to add in this phase: **Google Gemini** (via OpenAI-compat endpoint at `generativelanguage.googleapis.com/v1beta/openai/`), **Mistral** (via OpenAI-compat), **Groq**, **xAI/Grok**, **DeepSeek**, **OpenRouter** — all via `GenericOpenAIProvider`.
- **D-18:** Mistral uses the OpenAI-compat endpoint (not the native `mistralai` SDK) to avoid adding a new dependency. Can be revisited in a later phase if Mistral-specific features (embeddings, FIM) are needed.
### Claude's Discretion
- **D-03 (Anthropic output):** Researcher should determine whether Anthropic uses native `tool_use` JSON schema, Anthropic's `response_format` (if supported), or prompt-enforced JSON. Choose the most reliable option that is consistent with the JSON-mode-everywhere principle (D-01).
- **D-07 (Client lifecycle):** Researcher must investigate and document optimal client instantiation per provider (singleton, per-request, pooled). The documentation must be clear enough that adding a new provider in the future does not require re-researching this.
</decisions>
<canonical_refs>
## Canonical References
**Downstream agents MUST read these before planning or implementing.**
### Existing AI provider code
- `backend/ai/base.py` — `AIProvider` ABC and `ClassificationResult` dataclass; the interface all providers must implement
- `backend/ai/utils.py` — `strip_code_fences`, `parse_classification`, `parse_suggestions`; fallback parsers that remain but are demoted
- `backend/ai/__init__.py` — `get_provider(settings: dict)` factory; this is the function being replaced/refactored
- `backend/ai/anthropic_provider.py` — current Anthropic implementation; reference for what the refactored version must preserve
- `backend/ai/openai_provider.py` — current OpenAI/base-compat implementation; `GenericOpenAIProvider` builds on this
- `backend/ai/ollama_provider.py` — one-line subclass pattern for Ollama; shows the current named-preset approach
- `backend/ai/lmstudio_provider.py` — same pattern as Ollama; root cause of Linux connection issue is here (hardcoded `host.docker.internal`)
### Classification orchestrator
- `backend/services/classifier.py` — `classify_document()` and `suggest_topics_for_document()`; inline `_settings` dict construction that must be replaced with `ProviderConfig` (D-06)
- `backend/tasks/document_tasks.py` — `extract_and_classify` Celery task; retry logic (D-09) goes here
### Configuration and encryption
- `backend/config.py` — current `default_ai_provider`, `default_ai_model`, `system_prompt` settings; these become DB defaults (D-04)
- `backend/storage/cloud_utils.py` — HKDF/AES-GCM encryption pattern that D-05 requires for API key storage in `system_settings`
### Infrastructure
- `docker-compose.yml` — `extra_hosts` must be added to `backend` and `celery-worker` services (D-14)
### Architecture context
- `.planning/codebase/ARCHITECTURE.md` — system overview; AI providers shown at bottom-right of the diagram
- `CLAUDE.md` §"Backend: shared module map" — `backend/ai/utils.py` must remain the single location for AI parsing helpers
### ROADMAP
- `.planning/ROADMAP.md` Phase 7 entry — goal and dependencies (depends on Phase 6)
</canonical_refs>
<code_context>
## Existing Code Insights
### Reusable Assets
- `OpenAIProvider` (`backend/ai/openai_provider.py`): `GenericOpenAIProvider` subclasses this directly — JSON mode is added in one override.
- HKDF key derivation in `backend/storage/cloud_utils.py:_derive_fernet_key()`: the same pattern (fresh HKDF instance per call, never cache) must be replicated for `system_settings` API key encryption (D-05).
- `write_audit_log()` in `backend/services/storage.py`: admin AI config changes should be logged as audit events.
- Existing `health_check()` on all providers: reuse this for the "Test connection" button in the admin AI Providers panel (D-08).
### Established Patterns
- **Celery retry**: `document_tasks.py` currently catches exceptions and returns `{"status": "classification_failed"}`. Refactor to `raise self.retry(exc=e, countdown=30, max_retries=3)` in the task decorator.
- **StorageBackend ABC pattern**: the `AIProvider` ABC (`base.py`) already mirrors this. New providers follow the same subclassing pattern.
- **Admin endpoint whitelist**: `_user_to_dict()` pattern in `backend/api/admin.py` — new `system_settings` endpoints must never return encrypted API keys in responses (same principle as `credentials_enc`).
- **`get_provider()` factory**: currently a flat `if/elif` chain; refactor to a registry dict `{"anthropic": AnthropicProvider, "groq": GenericOpenAIProvider, …}` to make adding new providers O(1).
### Integration Points
- `backend/api/admin.py` — new AI Providers endpoints (`GET /api/admin/ai-config`, `PUT /api/admin/ai-config`) added here; admin-only via `get_current_admin` dep.
- `backend/services/classifier.py` — replace inline `_settings` dict with a `load_provider_config(session)` helper that reads from `system_settings` table.
- `frontend/src/views/AdminView.vue` / `frontend/src/components/admin/` — new `AdminAITab.vue` component following the existing tab pattern (AdminUsersTab, AdminQuotasTab, AdminAITab).
- `frontend/src/components/documents/DocumentCard.vue` — add "Classification failed" badge and "Re-analyze" button (D-11).
</code_context>
<specifics>
## Specific Ideas
- LMStudio Linux fix is purely a Docker Compose `extra_hosts` addition + JSON mode enforcement — no changes to `lmstudio_provider.py` logic itself needed beyond what the provider refactor provides.
- `GenericOpenAIProvider` named presets: the factory registry entry for `"groq"` passes `base_url="https://api.groq.com/openai/v1"`, for `"xai"` passes `base_url="https://api.x.ai/v1"`, for `"deepseek"` passes `base_url="https://api.deepseek.com"`, for `"openrouter"` passes `base_url="https://openrouter.ai/api/v1"`, for `"gemini"` passes `base_url="https://generativelanguage.googleapis.com/v1beta/openai/"`.
- Admin panel "AI Providers" section should show: active provider selector, per-provider accordion (API key field masked, base URL, model, context window size), "Test connection" button that calls `health_check()`.
</specifics>
<deferred>
## Deferred Ideas
- **Chunk + merge multi-call classification** — splitting long documents into N chunks and merging results was considered but deferred. More accurate for very long docs but 2-5x API cost. Worth a future phase once the provider layer is stable.
- **Native `mistralai` SDK** — Mistral-specific features (embeddings, FIM/code completion) are not needed for document classification. OpenAI-compat is sufficient now; revisit when embedding-based retrieval is added.
- **Token counting** — true token counting per provider (using `tiktoken` or provider-specific tokenizers) was considered for context window management. Per-provider `context_chars` approximation is sufficient for Phase 7; precise token counting can be added in a future optimization phase.
- **Streaming classification** — streaming LLM responses for faster UX was identified but deferred; classification is a background Celery task so streaming to the user is not straightforward without a websocket layer.
</deferred>
---
*Phase: 07-redo-and-optimize-llm-integration*
*Context gathered: 2026-06-02*