docs(07): capture phase context

.planning/phases/07-redo-and-optimize-llm-integration/07-CONTEXT.md

@@ -0,0 +1,131 @@
+# Phase 7: Redo and Optimize LLM Integration - Context
+
+**Gathered:** 2026-06-02
+**Status:** Ready for planning
+
+<domain>
+## Phase Boundary
+
+Phase 7 delivers a fully refactored, production-reliable AI provider layer: a `system_settings` DB table so admins can configure all provider settings interactively, a `GenericOpenAIProvider` class that covers all OpenAI-compatible providers, native Gemini and Mistral integrations, JSON-mode structured output enforced across all providers, Celery-based retry with exponential backoff, per-provider context window configuration with smart truncation, a fix for the LMStudio Linux connectivity issue, and a re-classify UI button for failed documents. The admin panel gains an AI Providers configuration section.
+
+</domain>
+
+<decisions>
+## Implementation Decisions
+
+### Structured Output
+- **D-01:** All OpenAI-compatible providers (OpenAI, Ollama, LMStudio, Groq, Gemini-compat, Mistral-compat, xAI/Grok, DeepSeek, OpenRouter) must pass `response_format={"type": "json_object"}` in every `classify()` and `suggest_topics()` call.
+- **D-02:** Keep `parse_classification()` and `parse_suggestions()` in `ai/utils.py` as a last-resort fallback only — they are called when a provider does not support `response_format` (e.g., some local Ollama models that ignore the parameter).
+- **D-03:** Anthropic structured output strategy is left to Claude's discretion (see below).
+
+### Provider Factory & Configuration
+- **D-04:** All AI provider settings (API keys, base URLs, model names, active provider) are stored in a new `system_settings` DB table. Env vars (existing `DEFAULT_AI_PROVIDER`, `DEFAULT_AI_MODEL`, API key vars) serve as startup defaults that populate the table on first boot, but the DB record is authoritative at runtime.
+- **D-05:** API keys stored in `system_settings` must be encrypted at rest — use the same HKDF/AES-GCM pattern already established for cloud credentials in `backend/storage/cloud_utils.py`.
+- **D-06:** `get_provider()` accepts typed Pydantic `ProviderConfig` models (not raw dicts). The classifier assembles a `ProviderConfig` from the DB record — no more inline dict construction.
+- **D-07:** Client lifecycle (singleton, pooled, per-request) is left to Claude's discretion per provider — the researcher must document the optimal pattern for each provider so new providers can follow the same established pattern.
+- **D-08:** Admin panel gains an "AI Providers" configuration section where admins can set the active provider, configure API keys/base URLs/models, and test connectivity via the existing `health_check()` method.
+
+### Retry & Resilience
+- **D-09:** The `extract_and_classify` Celery task uses Celery's built-in retry mechanism (`self.retry(exc=…, countdown=…, max_retries=3)`) with exponential backoff: first retry at 30 s, second at 90 s, third at 270 s.
+- **D-10:** After all retries are exhausted the document status remains `classification_failed`. No silent failure.
+- **D-11:** The document card in the frontend shows a "Classification failed" badge when `status == "classification_failed"`. A "Re-analyze" button is shown that calls `POST /api/documents/{id}/classify` to re-queue the Celery task.
+
+### Context Window Strategy
+- **D-12:** `MAX_AI_CHARS` is replaced by a per-provider `context_chars` value declared in each provider's config (or `ProviderConfig` default). The global constant is removed.
+- **D-13:** Smart truncation strategy: take the first 60% of the per-provider limit + the last 40%. This captures both document introduction and conclusion, which carry the most topic signal for long documents.
+
+### LMStudio Linux Fix
+- **D-14:** Add `extra_hosts: ["host.docker.internal:host-gateway"]` to the `backend` and `celery-worker` services in `docker-compose.yml`. This is the standard Linux Docker fix for accessing host services.
+- **D-15:** LMStudio base URL and model are configurable via the admin AI Providers panel (D-08), so users are not locked to the hardcoded `host.docker.internal:1234` default.
+
+### New Providers
+- **D-16:** Introduce a `GenericOpenAIProvider(base_url, api_key, model, context_chars)` class in `backend/ai/generic_openai_provider.py`. It subclasses `OpenAIProvider` and adds JSON-mode enforcement. Named presets (Groq, xAI/Grok, DeepSeek, OpenRouter, Gemini-compat, Mistral-compat) are factory shortcuts that pass the known `base_url` default.
+- **D-17:** Providers to add in this phase: **Google Gemini** (via OpenAI-compat endpoint at `generativelanguage.googleapis.com/v1beta/openai/`), **Mistral** (via OpenAI-compat), **Groq**, **xAI/Grok**, **DeepSeek**, **OpenRouter** — all via `GenericOpenAIProvider`.
+- **D-18:** Mistral uses the OpenAI-compat endpoint (not the native `mistralai` SDK) to avoid adding a new dependency. Can be revisited in a later phase if Mistral-specific features (embeddings, FIM) are needed.
+
+### Claude's Discretion
+- **D-03 (Anthropic output):** Researcher should determine whether Anthropic uses native `tool_use` JSON schema, Anthropic's `response_format` (if supported), or prompt-enforced JSON. Choose the most reliable option that is consistent with the JSON-mode-everywhere principle (D-01).
+- **D-07 (Client lifecycle):** Researcher must investigate and document optimal client instantiation per provider (singleton, per-request, pooled). The documentation must be clear enough that adding a new provider in the future does not require re-researching this.
+
+</decisions>
+
+<canonical_refs>
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Existing AI provider code
+- `backend/ai/base.py` — `AIProvider` ABC and `ClassificationResult` dataclass; the interface all providers must implement
+- `backend/ai/utils.py` — `strip_code_fences`, `parse_classification`, `parse_suggestions`; fallback parsers that remain but are demoted
+- `backend/ai/__init__.py` — `get_provider(settings: dict)` factory; this is the function being replaced/refactored
+- `backend/ai/anthropic_provider.py` — current Anthropic implementation; reference for what the refactored version must preserve
+- `backend/ai/openai_provider.py` — current OpenAI/base-compat implementation; `GenericOpenAIProvider` builds on this
+- `backend/ai/ollama_provider.py` — one-line subclass pattern for Ollama; shows the current named-preset approach
+- `backend/ai/lmstudio_provider.py` — same pattern as Ollama; root cause of Linux connection issue is here (hardcoded `host.docker.internal`)
+
+### Classification orchestrator
+- `backend/services/classifier.py` — `classify_document()` and `suggest_topics_for_document()`; inline `_settings` dict construction that must be replaced with `ProviderConfig` (D-06)
+- `backend/tasks/document_tasks.py` — `extract_and_classify` Celery task; retry logic (D-09) goes here
+
+### Configuration and encryption
+- `backend/config.py` — current `default_ai_provider`, `default_ai_model`, `system_prompt` settings; these become DB defaults (D-04)
+- `backend/storage/cloud_utils.py` — HKDF/AES-GCM encryption pattern that D-05 requires for API key storage in `system_settings`
+
+### Infrastructure
+- `docker-compose.yml` — `extra_hosts` must be added to `backend` and `celery-worker` services (D-14)
+
+### Architecture context
+- `.planning/codebase/ARCHITECTURE.md` — system overview; AI providers shown at bottom-right of the diagram
+- `CLAUDE.md` §"Backend: shared module map" — `backend/ai/utils.py` must remain the single location for AI parsing helpers
+
+### ROADMAP
+- `.planning/ROADMAP.md` Phase 7 entry — goal and dependencies (depends on Phase 6)
+
+</canonical_refs>
+
+<code_context>
+## Existing Code Insights
+
+### Reusable Assets
+- `OpenAIProvider` (`backend/ai/openai_provider.py`): `GenericOpenAIProvider` subclasses this directly — JSON mode is added in one override.
+- HKDF key derivation in `backend/storage/cloud_utils.py:_derive_fernet_key()`: the same pattern (fresh HKDF instance per call, never cache) must be replicated for `system_settings` API key encryption (D-05).
+- `write_audit_log()` in `backend/services/storage.py`: admin AI config changes should be logged as audit events.
+- Existing `health_check()` on all providers: reuse this for the "Test connection" button in the admin AI Providers panel (D-08).
+
+### Established Patterns
+- **Celery retry**: `document_tasks.py` currently catches exceptions and returns `{"status": "classification_failed"}`. Refactor to `raise self.retry(exc=e, countdown=30, max_retries=3)` in the task decorator.
+- **StorageBackend ABC pattern**: the `AIProvider` ABC (`base.py`) already mirrors this. New providers follow the same subclassing pattern.
+- **Admin endpoint whitelist**: `_user_to_dict()` pattern in `backend/api/admin.py` — new `system_settings` endpoints must never return encrypted API keys in responses (same principle as `credentials_enc`).
+- **`get_provider()` factory**: currently a flat `if/elif` chain; refactor to a registry dict `{"anthropic": AnthropicProvider, "groq": GenericOpenAIProvider, …}` to make adding new providers O(1).
+
+### Integration Points
+- `backend/api/admin.py` — new AI Providers endpoints (`GET /api/admin/ai-config`, `PUT /api/admin/ai-config`) added here; admin-only via `get_current_admin` dep.
+- `backend/services/classifier.py` — replace inline `_settings` dict with a `load_provider_config(session)` helper that reads from `system_settings` table.
+- `frontend/src/views/AdminView.vue` / `frontend/src/components/admin/` — new `AdminAITab.vue` component following the existing tab pattern (AdminUsersTab, AdminQuotasTab, AdminAITab).
+- `frontend/src/components/documents/DocumentCard.vue` — add "Classification failed" badge and "Re-analyze" button (D-11).
+
+</code_context>
+
+<specifics>
+## Specific Ideas
+
+- LMStudio Linux fix is purely a Docker Compose `extra_hosts` addition + JSON mode enforcement — no changes to `lmstudio_provider.py` logic itself needed beyond what the provider refactor provides.
+- `GenericOpenAIProvider` named presets: the factory registry entry for `"groq"` passes `base_url="https://api.groq.com/openai/v1"`, for `"xai"` passes `base_url="https://api.x.ai/v1"`, for `"deepseek"` passes `base_url="https://api.deepseek.com"`, for `"openrouter"` passes `base_url="https://openrouter.ai/api/v1"`, for `"gemini"` passes `base_url="https://generativelanguage.googleapis.com/v1beta/openai/"`.
+- Admin panel "AI Providers" section should show: active provider selector, per-provider accordion (API key field masked, base URL, model, context window size), "Test connection" button that calls `health_check()`.
+
+</specifics>
+
+<deferred>
+## Deferred Ideas
+
+- **Chunk + merge multi-call classification** — splitting long documents into N chunks and merging results was considered but deferred. More accurate for very long docs but 2-5x API cost. Worth a future phase once the provider layer is stable.
+- **Native `mistralai` SDK** — Mistral-specific features (embeddings, FIM/code completion) are not needed for document classification. OpenAI-compat is sufficient now; revisit when embedding-based retrieval is added.
+- **Token counting** — true token counting per provider (using `tiktoken` or provider-specific tokenizers) was considered for context window management. Per-provider `context_chars` approximation is sufficient for Phase 7; precise token counting can be added in a future optimization phase.
+- **Streaming classification** — streaming LLM responses for faster UX was identified but deferred; classification is a background Celery task so streaming to the user is not straightforward without a websocket layer.
+
+</deferred>
+
+---
+
+*Phase: 07-redo-and-optimize-llm-integration*
+*Context gathered: 2026-06-02*

.planning/phases/07-redo-and-optimize-llm-integration/07-DISCUSSION-LOG.md

@@ -0,0 +1,179 @@
+# Phase 7: Redo and Optimize LLM Integration - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-06-02
+**Phase:** 07-redo-and-optimize-llm-integration
+**Areas discussed:** Structured Output, Provider Factory Refactor, Retry & Resilience, Context Window Strategy, LMStudio Fix, New Providers
+
+---
+
+## Structured Output
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| JSON mode everywhere | All OpenAI-compat providers use `response_format={"type":"json_object"}`. Anthropic uses native JSON output. Keep regex parser as fallback only. | ✓ |
+| JSON schema / tool_use | Define strict Pydantic schema, use OpenAI/Anthropic tool_use. Guaranteed conformance but local models hit-or-miss. | |
+| Keep current regex parsing | Improve error handling but keep current fragile approach. | |
+
+**User's choice:** JSON mode everywhere
+**Notes:** None.
+
+### Anthropic strategy sub-question
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Native tool_use schema | Define ClassificationResult as Anthropic tool. Schema-guaranteed, no parsing needed. | |
+| Prompt-enforced JSON mode | Explicit system prompt + strip fences. Simpler but not schema-enforced. | |
+| You decide | Leave Anthropic strategy to researcher/planner. | ✓ |
+
+**User's choice:** You decide
+**Notes:** Researcher should pick the most reliable option consistent with the JSON-mode-everywhere principle.
+
+---
+
+## Provider Factory Refactor
+
+### Client lifecycle sub-question
+
+**User's choice:** Free text — "I want you to pick the best method for each provider. Especially when adding new providers which can happen after you research it but even later down the road, new providers can be added and the best method for each provider needs to be elaborated."
+**Notes:** Researcher must investigate and document optimal client lifecycle per provider type so that the pattern is clear for future additions. This is captured as a Claude's Discretion item (D-07).
+
+### Configuration storage sub-question
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Pydantic ProviderConfig models | Typed config models; `get_provider()` accepts ProviderConfig not raw dict. | ✓ (implied) |
+| Keep raw dict with validation | Add Pydantic step before passing dict. Less refactoring. | |
+| Move config into Settings | All provider config in `config.py` Settings env-var driven. | |
+
+**User's choice:** Free text — "I want all settings inside the admin settings page and only default values inside any .env or other files. I want the settings to be as interactive as possible."
+**Notes:** This means a new `system_settings` DB table storing all AI provider configuration. Admin panel gets an AI Providers section. Env vars serve as startup defaults only.
+
+### DB storage sub-question
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| New system_settings table in DB | Key-value/JSON config table, admin writes via API. API keys encrypted. Multi-instance safe. | ✓ |
+| Extend existing users table / per-user only | Keep settings purely per-user. No global defaults. | |
+| JSON config file on disk | Admin writes JSON file mounted into container. Breaks multi-instance. | |
+
+**User's choice:** New system_settings table in DB
+**Notes:** None.
+
+---
+
+## Retry & Resilience
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Celery task auto-retry with backoff | Raise retryable exception; 3x retry at 30s/90s/270s. Uses existing Celery infra. | ✓ |
+| tenacity in-process retry | Wrap provider calls in tenacity. Blocks worker thread during wait. | |
+| No retry — better error logging | Fail-fast but structured logs. User manually re-triggers. | |
+
+**User's choice:** Celery task auto-retry with backoff
+**Notes:** None.
+
+### User visibility sub-question
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Show status badge + manual re-classify button | Document card shows 'Classification failed' badge + Re-analyze button calls POST /api/documents/{id}/classify | ✓ |
+| Silent auto-retry only | Only Celery retries. No UI feedback if all fail. | |
+| Already exists | Re-analyze button already in DocumentView. | |
+
+**User's choice:** Show status badge + manual re-classify button
+**Notes:** None.
+
+---
+
+## Context Window Strategy
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Per-provider token limit + smart truncation | Each provider declares context size. Truncation: first 60% + last 40% of limit. MAX_AI_CHARS becomes per-provider. | ✓ |
+| Chunk + merge (multi-call) | Split doc into N chunks, classify each, merge. More accurate but 2-5x API calls. | |
+| Keep 8,000 char limit globally | Keep current behavior. Simple but inaccurate for long docs. | |
+
+**User's choice:** Per-provider token limit + smart truncation
+**Notes:** None.
+
+---
+
+## LMStudio Fix
+
+### Failure mode sub-question
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Classification returns empty/wrong topics | API call succeeds but model returns non-JSON. Fixed by JSON mode. | |
+| Connection refused / not reachable | Docker can't reach host.docker.internal:1234. | |
+| Model name mismatch | Hardcoded 'gemma-4-e4b-it' doesn't match loaded model. | |
+| Other / not sure | Leave to researcher. | |
+
+**User's choice:** "Classification failed: Connection error."
+**Notes:** Connection error is the reported symptom.
+
+### OS sub-question
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| macOS or Windows | host.docker.internal works automatically. | |
+| Linux | Requires extra_hosts: ["host.docker.internal:host-gateway"] — almost certainly root cause. | ✓ |
+| Both / uncertain | Fix for all OSes. | |
+
+**User's choice:** Linux
+**Notes:** Root cause confirmed. Fix: add `extra_hosts: ["host.docker.internal:host-gateway"]` to backend and celery-worker in docker-compose.yml.
+
+---
+
+## New Providers
+
+### Which providers to add
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Google Gemini | OpenAI-compat endpoint, zero new deps. | ✓ |
+| Mistral | Native SDK or OpenAI-compat. | ✓ |
+| Groq | OpenAI-compat, fastest inference. | ✓ |
+| xAI/Grok + DeepSeek + OpenRouter | All OpenAI-compat, just base_url + api_key. | ✓ |
+
+**User's choice:** All four selected.
+**Notes:** None.
+
+### Implementation approach
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| One GenericOpenAIProvider class | Single `GenericOpenAIProvider(base_url, api_key, model, context_chars)`. Named presets are factory shortcuts. | ✓ |
+| Separate class per provider | Individual files for each. Explicit but repetitive. | |
+| You decide | Leave to planner. | |
+
+**User's choice:** One GenericOpenAIProvider class
+**Notes:** All OpenAI-compat providers (Groq, Grok, DeepSeek, OpenRouter, Gemini-compat, Mistral-compat) use this one class.
+
+### Mistral SDK vs compat
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| OpenAI-compat (simpler) | Use GenericOpenAIProvider with Mistral base_url. Zero new dependencies. | ✓ |
+| Native mistralai SDK | Full Mistral features (embeddings, FIM) but adds dependency. | |
+| You decide | Leave to researcher. | |
+
+**User's choice:** OpenAI-compat (simpler)
+**Notes:** Native SDK can be added later if Mistral-specific features are needed.
+
+---
+
+## Claude's Discretion
+
+- **Anthropic structured output strategy** — researcher to determine tool_use schema vs response_format vs prompt-enforcement. Must be consistent with JSON-mode-everywhere principle.
+- **Client lifecycle per provider** — researcher to investigate and document optimal instantiation pattern (singleton, pooled, per-request) per provider type, such that future providers can follow the documented pattern without re-researching.
+
+## Deferred Ideas
+
+- Chunk + merge multi-call classification — 2-5x API cost; revisit when provider layer is stable.
+- Native `mistralai` SDK — not needed for classification; revisit when embedding-based retrieval is added.
+- Token counting (tiktoken or provider tokenizers) — per-provider `context_chars` approximation sufficient for now.
+- Streaming classification — Celery background task makes streaming non-trivial without websockets.