docs(07): create phase plan — 5 plans, verification passed
5 waves: system_settings DB + HKDF encryption (01), ProviderConfig + GenericOpenAIProvider + singleton client fix (02), Anthropic output_config + classifier wiring (03), Celery retry 30/90/270s + re-queue endpoint (04), admin AI panel + DocumentCard badge + human checkpoint (05). All 18 decisions D-01..D-18 covered. Plan checker passed after 1 revision round (4 blockers fixed: D-02/D-14 coverage, D-11 Vitest tests, Plan 05 files_modified). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
6c8e0d8bde
commit
3df62506c9
@@ -0,0 +1,677 @@
|
||||
# Phase 7: Redo and Optimize LLM Integration - Pattern Map
|
||||
|
||||
**Mapped:** 2026-06-02
|
||||
**Files analyzed:** 14 new/modified files
|
||||
**Analogs found:** 13 / 14
|
||||
|
||||
---
|
||||
|
||||
## File Classification
|
||||
|
||||
| New/Modified File | Role | Data Flow | Closest Analog | Match Quality |
|
||||
|---|---|---|---|---|
|
||||
| `backend/ai/provider_config.py` | config/model | transform | `backend/api/admin.py` Pydantic models (lines 74–119) | role-match |
|
||||
| `backend/ai/generic_openai_provider.py` | service | request-response | `backend/ai/openai_provider.py` | exact |
|
||||
| `backend/ai/openai_provider.py` (modify) | service | request-response | self (current file) | exact |
|
||||
| `backend/ai/anthropic_provider.py` (modify) | service | request-response | `backend/ai/openai_provider.py` | exact |
|
||||
| `backend/ai/ollama_provider.py` (modify) | service | request-response | self + `backend/ai/ollama_provider.py` | exact |
|
||||
| `backend/ai/lmstudio_provider.py` (modify) | service | request-response | `backend/ai/ollama_provider.py` | exact |
|
||||
| `backend/ai/__init__.py` (refactor) | factory | request-response | self (current file) | exact |
|
||||
| `backend/db/models.py` (add SystemSettings) | model | CRUD | existing models in `backend/db/models.py` (CloudConnection, lines 299–318) | exact |
|
||||
| `backend/migrations/versions/0005_system_settings.py` | migration | batch | `backend/migrations/versions/0004_phase4_pdf_open_mode_tsvector.py` | exact |
|
||||
| `backend/services/ai_config.py` | service | CRUD + encrypt | `backend/storage/cloud_utils.py` | exact |
|
||||
| `backend/services/classifier.py` (refactor) | service | request-response | self (current file) | exact |
|
||||
| `backend/tasks/document_tasks.py` (refactor) | task | event-driven | self (current file) | exact |
|
||||
| `frontend/src/components/admin/AdminAiConfigTab.vue` (modify) | component | request-response | self + `frontend/src/components/admin/AdminUsersTab.vue` | exact |
|
||||
| `frontend/src/components/documents/DocumentCard.vue` (modify) | component | request-response | self (current file) | exact |
|
||||
|
||||
---
|
||||
|
||||
## Pattern Assignments
|
||||
|
||||
### `backend/ai/provider_config.py` (NEW — config model)
|
||||
|
||||
**Analog:** `backend/api/admin.py` Pydantic request models (lines 74–119)
|
||||
|
||||
**Imports pattern** (from analog lines 27–32):
|
||||
```python
|
||||
from __future__ import annotations
|
||||
from pydantic import BaseModel, field_validator
|
||||
from typing import Optional
|
||||
```
|
||||
|
||||
**Core model pattern** (analog lines 74–119 — `AiConfigUpdate`, `QuotaUpdate`):
|
||||
```python
|
||||
# From backend/api/admin.py lines 102–105
|
||||
class AiConfigUpdate(BaseModel):
|
||||
ai_provider: Optional[str] = None
|
||||
ai_model: Optional[str] = None
|
||||
```
|
||||
New file follows same `BaseModel` subclass pattern with typed fields and defaults. No `from_attributes` needed (not an ORM response model).
|
||||
|
||||
**No analog for `PROVIDER_DEFAULTS` dict** — use RESEARCH.md Pattern 2 directly (the dict of known base_urls and context_chars per provider_id).
|
||||
|
||||
---
|
||||
|
||||
### `backend/ai/generic_openai_provider.py` (NEW — service, request-response)
|
||||
|
||||
**Analog:** `backend/ai/openai_provider.py` (entire file — 72 lines)
|
||||
|
||||
**Imports pattern** (analog lines 1–4):
|
||||
```python
|
||||
from openai import AsyncOpenAI
|
||||
from ai.base import AIProvider, ClassificationResult
|
||||
from ai.utils import parse_classification, parse_suggestions
|
||||
```
|
||||
|
||||
**Subclass pattern** (analog lines 8–16):
|
||||
```python
|
||||
# From backend/ai/openai_provider.py lines 8–16
|
||||
class OpenAIProvider(AIProvider):
|
||||
def __init__(self, api_key: str, model: str = "gpt-4o", base_url=None):
|
||||
self._api_key = api_key
|
||||
self._model = model
|
||||
self._base_url = base_url
|
||||
|
||||
def _client(self) -> AsyncOpenAI:
|
||||
return AsyncOpenAI(api_key=self._api_key or "placeholder", base_url=self._base_url)
|
||||
```
|
||||
`GenericOpenAIProvider` subclasses `OpenAIProvider`. The constructor adds `context_chars: int` and `supports_json_mode: bool = True`. The `_client()` method is replaced by `self._client = AsyncOpenAI(...)` stored in `__init__` (D-07 singleton fix). The `api_key` placeholder changes from `"placeholder"` to `"not-needed"` (openai SDK 2.34+ pitfall).
|
||||
|
||||
**Core classify pattern** (analog lines 17–37 — override with json_mode):
|
||||
```python
|
||||
# From backend/ai/openai_provider.py lines 17–37
|
||||
async def classify(self, document_text, existing_topics, system_prompt):
|
||||
topics_str = ", ".join(existing_topics) if existing_topics else "(none yet)"
|
||||
user_msg = (
|
||||
f"Existing topics: [{topics_str}]\n\n"
|
||||
f"Document text:\n{document_text[:MAX_AI_CHARS]}"
|
||||
)
|
||||
response = await self._client().chat.completions.create(
|
||||
model=self._model,
|
||||
max_tokens=1024,
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_msg},
|
||||
],
|
||||
)
|
||||
raw = response.choices[0].message.content or ""
|
||||
return parse_classification(raw)
|
||||
```
|
||||
`GenericOpenAIProvider` overrides this to: (1) call `self._truncate(document_text)` instead of `[:MAX_AI_CHARS]`, (2) add `response_format={"type": "json_object"}` when `self.supports_json_mode` is `True`, (3) call `self._client.chat.completions.create(...)` on the stored singleton (no `()`).
|
||||
|
||||
**`_truncate` helper pattern** — no existing analog; copy verbatim from RESEARCH.md Pattern (smart truncation):
|
||||
```python
|
||||
def _truncate(self, text: str) -> str:
|
||||
"""First 60% + last 40% of context window (D-13)."""
|
||||
limit = self._context_chars
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
head_len = int(limit * 0.6)
|
||||
tail_len = limit - head_len
|
||||
return text[:head_len] + "\n[...truncated...]\n" + text[-tail_len:]
|
||||
```
|
||||
|
||||
**health_check pattern** (analog lines 60–69):
|
||||
```python
|
||||
# From backend/ai/openai_provider.py lines 60–69
|
||||
async def health_check(self) -> bool:
|
||||
try:
|
||||
await self._client().chat.completions.create(
|
||||
model=self._model, max_tokens=5,
|
||||
messages=[{"role": "user", "content": "ping"}],
|
||||
)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
```
|
||||
In `GenericOpenAIProvider` call `self._client.chat.completions.create(...)` (no `()`).
|
||||
|
||||
---
|
||||
|
||||
### `backend/ai/openai_provider.py` (MODIFY — singleton + context_chars)
|
||||
|
||||
**Analog:** self (current file)
|
||||
|
||||
**Changes from current pattern:**
|
||||
|
||||
Current `__init__` (lines 9–15):
|
||||
```python
|
||||
# CURRENT — creates client per-call (anti-pattern to replace)
|
||||
def __init__(self, api_key: str, model: str = "gpt-4o", base_url=None):
|
||||
self._api_key = api_key
|
||||
self._model = model
|
||||
self._base_url = base_url
|
||||
|
||||
def _client(self) -> AsyncOpenAI:
|
||||
return AsyncOpenAI(api_key=self._api_key or "placeholder", base_url=self._base_url)
|
||||
```
|
||||
|
||||
Replace with (D-07 singleton pattern):
|
||||
```python
|
||||
# TARGET — singleton stored in __init__
|
||||
def __init__(self, api_key: str, model: str, base_url: str | None, context_chars: int):
|
||||
self._api_key = api_key or "not-needed" # SDK 2.34+ rejects empty string
|
||||
self._model = model
|
||||
self._base_url = base_url
|
||||
self._context_chars = context_chars
|
||||
self._client = AsyncOpenAI(api_key=self._api_key, base_url=self._base_url)
|
||||
```
|
||||
|
||||
Remove `MAX_AI_CHARS = 8_000` constant (line 5). Replace `document_text[:MAX_AI_CHARS]` with `self._truncate(document_text)`. Add `_truncate()` method (same as GenericOpenAIProvider above). Remove `_client()` method entirely. Update all `self._client()` call-sites to `self._client` (no `()`).
|
||||
|
||||
---
|
||||
|
||||
### `backend/ai/anthropic_provider.py` (MODIFY — singleton + output_config)
|
||||
|
||||
**Analog:** self (current file) + `backend/ai/openai_provider.py`
|
||||
|
||||
**Current anti-pattern** (lines 9–14):
|
||||
```python
|
||||
# CURRENT — creates client per-call (anti-pattern to replace)
|
||||
def __init__(self, api_key: str, model: str = "claude-sonnet-4-6"):
|
||||
self._api_key = api_key
|
||||
self._model = model
|
||||
|
||||
def _client(self):
|
||||
return anthropic.AsyncAnthropic(api_key=self._api_key)
|
||||
```
|
||||
|
||||
**Target pattern** (D-07 + D-03 output_config):
|
||||
```python
|
||||
# TARGET — singleton + context_chars + output_config
|
||||
def __init__(self, api_key: str, model: str, context_chars: int):
|
||||
self._api_key = api_key
|
||||
self._model = model
|
||||
self._context_chars = context_chars
|
||||
self._client = anthropic.AsyncAnthropic(api_key=self._api_key)
|
||||
```
|
||||
|
||||
Remove `MAX_AI_CHARS = 8_000` (line 5). Replace `document_text[:MAX_AI_CHARS]` with `self._truncate(document_text)`. Add `_truncate()` method. Remove `_client()` method. Replace `client = self._client()` call-sites with direct `self._client`.
|
||||
|
||||
**output_config pattern for classify** (from RESEARCH.md D-03):
|
||||
```python
|
||||
# Add to messages.create() call in classify():
|
||||
output_config={
|
||||
"format": {"type": "json_schema", "schema": _CLASSIFICATION_SCHEMA}
|
||||
},
|
||||
```
|
||||
`_CLASSIFICATION_SCHEMA` defined as module-level constant. `response.content[0].text` read only when `response.stop_reason == "end_turn"`; otherwise fall back to `parse_classification("")`.
|
||||
|
||||
---
|
||||
|
||||
### `backend/ai/ollama_provider.py` / `lmstudio_provider.py` (MODIFY — context_chars)
|
||||
|
||||
**Analog:** `backend/ai/ollama_provider.py` (entire file — 10 lines)
|
||||
|
||||
**Current pattern** (lines 1–11):
|
||||
```python
|
||||
from ai.openai_provider import OpenAIProvider
|
||||
|
||||
class OllamaProvider(OpenAIProvider):
|
||||
def __init__(self, base_url: str = "http://host.docker.internal:11434", model: str = "llama3.2"):
|
||||
super().__init__(
|
||||
api_key="ollama",
|
||||
model=model,
|
||||
base_url=base_url.rstrip("/") + "/v1",
|
||||
)
|
||||
```
|
||||
These one-liner subclasses become aliases in the provider registry (both route to `GenericOpenAIProvider`). The files themselves may be reduced to empty stubs or removed if `get_provider()` registry handles instantiation directly. If kept, update `super().__init__()` to pass `context_chars=8_000` as a default.
|
||||
|
||||
---
|
||||
|
||||
### `backend/ai/__init__.py` (REFACTOR — registry pattern)
|
||||
|
||||
**Analog:** self (current file — lines 1–35)
|
||||
|
||||
**Current anti-pattern** (lines 8–35):
|
||||
```python
|
||||
# CURRENT — flat if/elif chain
|
||||
def get_provider(settings: dict) -> AIProvider:
|
||||
active = settings.get("active_provider", "lmstudio")
|
||||
providers = settings.get("providers", {})
|
||||
cfg = providers.get(active, {})
|
||||
if active == "anthropic":
|
||||
return AnthropicProvider(...)
|
||||
elif active == "openai":
|
||||
...
|
||||
else:
|
||||
raise ValueError(f"Unknown AI provider: {active}")
|
||||
```
|
||||
|
||||
**Target registry pattern** (from RESEARCH.md get_provider() Registry Pattern):
|
||||
```python
|
||||
# TARGET — O(1) dict lookup, typed ProviderConfig input
|
||||
from ai.provider_config import ProviderConfig
|
||||
|
||||
_REGISTRY: dict[str, type] = {
|
||||
"openai": OpenAIProvider,
|
||||
"anthropic": AnthropicProvider,
|
||||
"gemini": GenericOpenAIProvider,
|
||||
# ... all providers
|
||||
}
|
||||
|
||||
def get_provider(config: ProviderConfig) -> AIProvider:
|
||||
cls = _REGISTRY.get(config.provider_id)
|
||||
if cls is None:
|
||||
raise ValueError(f"Unknown AI provider: {config.provider_id!r}")
|
||||
# AnthropicProvider does not take base_url — handle in factory
|
||||
if config.provider_id == "anthropic":
|
||||
return cls(api_key=config.api_key or "not-needed", model=config.model, context_chars=config.context_chars)
|
||||
return cls(api_key=config.api_key or "not-needed", model=config.model,
|
||||
base_url=config.base_url, context_chars=config.context_chars)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `backend/db/models.py` — ADD `SystemSettings` model
|
||||
|
||||
**Analog:** `backend/db/models.py` — `CloudConnection` model (lines 299–318)
|
||||
|
||||
**Closest existing model pattern** (lines 299–318):
|
||||
```python
|
||||
# From backend/db/models.py lines 299–318
|
||||
class CloudConnection(Base):
|
||||
__tablename__ = "cloud_connections"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
|
||||
)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False,
|
||||
)
|
||||
provider: Mapped[str] = mapped_column(String, nullable=False)
|
||||
credentials_enc: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
status: Mapped[str] = mapped_column(String, nullable=False, default="ACTIVE")
|
||||
connected_at: Mapped[datetime] = mapped_column(
|
||||
TIMESTAMP(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
__table_args__ = (Index("ix_cloud_connections_user", "user_id"),)
|
||||
```
|
||||
|
||||
**Target SystemSettings model** — same `Mapped[]` typed column style, same `server_default=func.now()` for timestamps, same `UniqueConstraint` in `__table_args__`. No FK (global table, not per-user). Columns: `id` (UUID PK), `provider_id` (String UNIQUE NOT NULL), `api_key_enc` (Text nullable), `base_url` (Text nullable), `model_name` (Text NOT NULL default ""), `context_chars` (Integer NOT NULL default 8000), `is_active` (Boolean NOT NULL default False), `created_at` (TIMESTAMP), `updated_at` (TIMESTAMP).
|
||||
|
||||
**Import additions needed** (from top of models.py, lines 25–38):
|
||||
```python
|
||||
# Already imported — no new imports needed:
|
||||
# Boolean, String, Text, TIMESTAMP, UniqueConstraint, Integer
|
||||
# UUID, Mapped, mapped_column, func
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `backend/migrations/versions/0005_system_settings.py` (NEW — migration)
|
||||
|
||||
**Analog:** `backend/migrations/versions/0004_phase4_pdf_open_mode_tsvector.py` (entire file — 86 lines)
|
||||
|
||||
**Header pattern** (analog lines 1–36):
|
||||
```python
|
||||
"""Phase 4 schema additions: ..."""
|
||||
from __future__ import annotations
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0004"
|
||||
down_revision = "0003"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
```
|
||||
New migration uses: `revision = "0005"`, `down_revision = "0004"`.
|
||||
|
||||
**`op.create_table()` pattern** (from analog + `backend/migrations/versions/0001_initial_schema.py` lines 38–55):
|
||||
```python
|
||||
# From 0001 lines 38–55 — op.create_table with postgresql.UUID
|
||||
op.create_table("users",
|
||||
sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
|
||||
sa.Column("handle", sa.String(), nullable=False),
|
||||
...
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("email", name="uq_users_email"),
|
||||
)
|
||||
```
|
||||
New migration uses `sa.dialects.postgresql.UUID(as_uuid=True)` for the `id` column, `sa.text("gen_random_uuid()")` as `server_default` for `id`, and `sa.UniqueConstraint("provider_id", name="uq_system_settings_provider_id")` in the column list.
|
||||
|
||||
**downgrade pattern** (analog lines 77–86):
|
||||
```python
|
||||
def downgrade() -> None:
|
||||
op.drop_table("system_settings")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `backend/services/ai_config.py` (NEW — service, CRUD + encrypt)
|
||||
|
||||
**Analog:** `backend/storage/cloud_utils.py` (entire file — 181 lines)
|
||||
|
||||
**Module docstring pattern** (analog lines 1–19):
|
||||
```python
|
||||
"""
|
||||
Cloud storage shared utilities for DocuVault.
|
||||
|
||||
Security design:
|
||||
HKDF credential encryption (D-18, CLOUD-02): _derive_fernet_key() creates a FRESH
|
||||
HKDF instance on every call. The cryptography library raises AlreadyFinalized if
|
||||
.derive() is called twice on the same instance.
|
||||
...
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import base64
|
||||
from cryptography.fernet import Fernet
|
||||
from cryptography.hazmat.primitives import hashes
|
||||
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
|
||||
```
|
||||
|
||||
**HKDF key derivation pattern** (analog lines 114–141 — COPY VERBATIM, change salt + info):
|
||||
```python
|
||||
# From backend/storage/cloud_utils.py lines 114–141
|
||||
def _derive_fernet_key(master_key: bytes, user_id: str) -> Fernet:
|
||||
# FRESH HKDF instance on every call — AlreadyFinalized pitfall
|
||||
hkdf = HKDF(
|
||||
algorithm=hashes.SHA256(),
|
||||
length=32,
|
||||
salt=user_id.encode("utf-8"),
|
||||
info=b"cloud-credentials", # <-- change to b"ai-provider-settings" in new file
|
||||
)
|
||||
raw_key: bytes = hkdf.derive(master_key)
|
||||
fernet_key = base64.urlsafe_b64encode(raw_key)
|
||||
return Fernet(fernet_key)
|
||||
```
|
||||
In `ai_config.py`, function is renamed `_derive_ai_settings_key(master_key, provider_id)`. The salt is `provider_id.encode("utf-8")`. The info is `b"ai-provider-settings"` — different from `b"cloud-credentials"` (domain separation). Same AlreadyFinalized comment must be retained.
|
||||
|
||||
**encrypt/decrypt pattern** (analog lines 144–181):
|
||||
```python
|
||||
# From backend/storage/cloud_utils.py lines 144–161
|
||||
def encrypt_credentials(master_key: bytes, user_id: str, credentials: dict) -> str:
|
||||
f = _derive_fernet_key(master_key, user_id)
|
||||
plaintext = json.dumps(credentials).encode("utf-8")
|
||||
return f.encrypt(plaintext).decode("utf-8")
|
||||
|
||||
def decrypt_credentials(master_key: bytes, user_id: str, credentials_enc: str) -> dict:
|
||||
f = _derive_fernet_key(master_key, user_id)
|
||||
plaintext = f.decrypt(credentials_enc.encode("utf-8"))
|
||||
return json.loads(plaintext)
|
||||
```
|
||||
In `ai_config.py`: `encrypt_api_key(master_key, provider_id, api_key: str) -> str` (encrypt a plain string, not a dict — use `.encode()` / `.decode()` directly without `json.dumps`). `decrypt_api_key(master_key, provider_id, api_key_enc: str) -> str`.
|
||||
|
||||
**Additional function `load_provider_config(session)`** — no analog; use SQLAlchemy `select()` + `where(SystemSettings.is_active == True)` to load the active row, decrypt the key, construct `ProviderConfig`. Pattern for `session.execute(select(...).where(...))` from `backend/api/admin.py` lines 163–167:
|
||||
```python
|
||||
# From backend/api/admin.py lines 163–167
|
||||
result = await session.execute(
|
||||
select(User).order_by(User.created_at.desc())
|
||||
)
|
||||
users = result.scalars().all()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `backend/services/classifier.py` (REFACTOR — ProviderConfig injection)
|
||||
|
||||
**Analog:** self (current file — 128 lines)
|
||||
|
||||
**Current inline dict construction** (lines 57–64 — the anti-pattern to replace):
|
||||
```python
|
||||
# CURRENT — lines 57–64 in backend/services/classifier.py
|
||||
_ai_provider = ai_provider or app_settings.default_ai_provider
|
||||
_ai_model = ai_model or app_settings.default_ai_model
|
||||
system_prompt = app_settings.system_prompt or _DEFAULT_SYSTEM_PROMPT
|
||||
_settings = {
|
||||
"active_provider": _ai_provider,
|
||||
"providers": {_ai_provider: {"model": _ai_model}},
|
||||
}
|
||||
provider = get_provider(_settings)
|
||||
```
|
||||
|
||||
**Target pattern** (replace with `load_provider_config()` call):
|
||||
```python
|
||||
# TARGET — ProviderConfig loaded from DB (D-06)
|
||||
from services.ai_config import load_provider_config
|
||||
config = await load_provider_config(session)
|
||||
# Per-user override: if user has ai_provider set, build a ProviderConfig override
|
||||
provider = get_provider(config)
|
||||
```
|
||||
|
||||
Remove `MAX_AI_CHARS = 8_000` (line 28). Remove `text[:MAX_AI_CHARS]` (line 85) — truncation now lives in the provider's `_truncate()`. Remove the `ai_provider: str | None` and `ai_model: str | None` kwargs from both `classify_document()` and `suggest_topics_for_document()` signatures (or retain them as optional overrides for per-user assignment — see RESEARCH.md Runtime State Inventory note about per-user `ai_provider` column).
|
||||
|
||||
---
|
||||
|
||||
### `backend/tasks/document_tasks.py` (REFACTOR — Celery retry)
|
||||
|
||||
**Analog:** self (current file — 173 lines)
|
||||
|
||||
**Current task decorator** (lines 22–25):
|
||||
```python
|
||||
# CURRENT — no bind, no retry
|
||||
@celery_app.task(name="tasks.document_tasks.extract_and_classify")
|
||||
def extract_and_classify(document_id: str) -> dict:
|
||||
return asyncio.run(_run(document_id))
|
||||
```
|
||||
|
||||
**Target decorator** (D-09 — bind=True + max_retries):
|
||||
```python
|
||||
# TARGET
|
||||
@celery_app.task(
|
||||
name="tasks.document_tasks.extract_and_classify",
|
||||
bind=True,
|
||||
max_retries=3,
|
||||
)
|
||||
def extract_and_classify(self, document_id: str) -> dict:
|
||||
try:
|
||||
return asyncio.run(_run(document_id))
|
||||
except _ClassificationError as exc:
|
||||
countdowns = [30, 90, 270]
|
||||
countdown = countdowns[min(self.request.retries, 2)]
|
||||
raise self.retry(exc=exc, countdown=countdown)
|
||||
```
|
||||
|
||||
`_ClassificationError` is a new module-level sentinel exception class (defined above the task). The existing `except Exception as e: doc.status = "classification_failed"` block in `_run()` (lines 109–124) is replaced: instead of catching and returning a dict, `_run()` raises `_ClassificationError(str(e))` for classification failures specifically (not for extract/storage failures which still `return {...}`). On `MaxRetriesExceededError`, write `classification_failed` to DB via a new `_mark_classification_failed(document_id)` async helper.
|
||||
|
||||
**Existing non-retryable return pattern to preserve** (lines 47–106 — storage/extract failures):
|
||||
```python
|
||||
# From backend/tasks/document_tasks.py lines 47–50 — keep unchanged
|
||||
try:
|
||||
doc_uuid = _uuid.UUID(document_id)
|
||||
except ValueError:
|
||||
return {"document_id": document_id, "status": "invalid_id"}
|
||||
```
|
||||
All `return {... "status": "extract_failed" ...}` patterns in the retrieval and extraction steps are preserved — they are not retried.
|
||||
|
||||
---
|
||||
|
||||
### `frontend/src/components/admin/AdminAiConfigTab.vue` (MODIFY — add global config section)
|
||||
|
||||
**Analog:** self (current file — 136 lines) + `frontend/src/components/admin/AdminUsersTab.vue`
|
||||
|
||||
**Current imports/setup pattern** (lines 76–94):
|
||||
```javascript
|
||||
// From frontend/src/components/admin/AdminAiConfigTab.vue lines 76–94
|
||||
import { ref, reactive, onMounted } from 'vue'
|
||||
import * as api from '../../api/client.js'
|
||||
|
||||
const users = ref([])
|
||||
const loading = ref(false)
|
||||
const loadError = ref(null)
|
||||
const savingId = ref(null)
|
||||
const savedId = ref(null)
|
||||
const configs = reactive({})
|
||||
```
|
||||
|
||||
**Save function pattern** (lines 96–114):
|
||||
```javascript
|
||||
// From frontend/src/components/admin/AdminAiConfigTab.vue lines 96–114
|
||||
async function saveConfig(userId) {
|
||||
savingId.value = userId
|
||||
savedId.value = null
|
||||
try {
|
||||
await api.adminUpdateAiConfig(userId, configs[userId].provider || null, configs[userId].model || null)
|
||||
savedId.value = userId
|
||||
setTimeout(() => { if (savedId.value === userId) savedId.value = null }, 1500)
|
||||
} catch (e) {
|
||||
loadError.value = e.message
|
||||
} finally {
|
||||
savingId.value = null
|
||||
}
|
||||
}
|
||||
```
|
||||
New global config section adds a second `ref()` block (`systemConfig`, `savingSystem`, `savedSystem`) and a `saveSystemConfig()` function following the same try/catch/finally shape. New API functions `api.getAiConfig()` and `api.saveAiConfig(body)` are called instead of `adminUpdateAiConfig`.
|
||||
|
||||
**Loading state template pattern** (lines 2–9):
|
||||
```html
|
||||
<!-- From AdminAiConfigTab.vue lines 2–9 — reuse for global config section -->
|
||||
<div v-if="loading" class="bg-white rounded-xl border border-gray-200 p-8 text-center">
|
||||
<div class="flex items-center justify-center gap-2 text-gray-400 text-sm">
|
||||
<span class="animate-spin rounded-full border-2 border-current border-t-transparent w-4 h-4"></span>
|
||||
Loading AI config…
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
**Pitfall 6 guard:** The new "System AI Providers" section is added as a separate block *above* the existing per-user table, not replacing it. The existing `users` / `configs` reactive state and `saveConfig(userId)` function remain untouched.
|
||||
|
||||
---
|
||||
|
||||
### `frontend/src/components/documents/DocumentCard.vue` (MODIFY — classification failed badge + Re-analyze button)
|
||||
|
||||
**Analog:** self (current file — 142 lines)
|
||||
|
||||
**Existing status badge pattern** (lines 31–34 — shared indicator):
|
||||
```html
|
||||
<!-- From DocumentCard.vue lines 31–34 — copy badge style -->
|
||||
<div v-if="doc.is_shared" class="mt-2">
|
||||
<span class="bg-indigo-50 text-indigo-600 text-xs font-medium px-2 py-1 rounded-full">Shared</span>
|
||||
</div>
|
||||
```
|
||||
|
||||
**Target badge** — add alongside the shared badge block:
|
||||
```html
|
||||
<!-- Add after line 34 -->
|
||||
<div v-if="doc.status === 'classification_failed'" class="mt-2 flex items-center gap-2">
|
||||
<span class="bg-red-50 text-red-600 text-xs font-medium px-2 py-1 rounded-full">Classification failed</span>
|
||||
<button
|
||||
@click.stop="reanalyze"
|
||||
:disabled="reanalyzing"
|
||||
class="text-xs text-indigo-600 hover:text-indigo-700 font-semibold disabled:opacity-50"
|
||||
>
|
||||
<span v-if="reanalyzing">Re-analyzing…</span>
|
||||
<span v-else>Re-analyze</span>
|
||||
</button>
|
||||
</div>
|
||||
```
|
||||
|
||||
**Action button import pattern** (existing lines 97–98):
|
||||
```javascript
|
||||
// From DocumentCard.vue lines 97–98
|
||||
import { moveDocument } from '../../api/client.js'
|
||||
```
|
||||
Add `classifyDocument` to the same import. Add `const reanalyzing = ref(false)` to the `ref()` block. Add `reanalyze()` async function following the same try/catch shape as `moveToFolder()` (lines 128–135):
|
||||
```javascript
|
||||
// From DocumentCard.vue lines 128–135 — copy shape
|
||||
async function moveToFolder(folderId) {
|
||||
showFolderPicker.value = false
|
||||
try {
|
||||
await moveDocument(props.doc.id, folderId)
|
||||
} catch (e) {
|
||||
console.error('Move failed:', e.message)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Shared Patterns
|
||||
|
||||
### HKDF/Fernet Encryption
|
||||
**Source:** `backend/storage/cloud_utils.py` lines 114–181
|
||||
**Apply to:** `backend/services/ai_config.py`
|
||||
```python
|
||||
# Pattern: fresh HKDF instance per call — never reuse (AlreadyFinalized pitfall)
|
||||
hkdf = HKDF(
|
||||
algorithm=hashes.SHA256(),
|
||||
length=32,
|
||||
salt=<scope_id>.encode("utf-8"), # user_id for cloud, provider_id for AI settings
|
||||
info=b"<domain-label>", # b"cloud-credentials" vs b"ai-provider-settings"
|
||||
)
|
||||
raw_key: bytes = hkdf.derive(master_key)
|
||||
fernet_key = base64.urlsafe_b64encode(raw_key)
|
||||
return Fernet(fernet_key)
|
||||
```
|
||||
|
||||
### Admin Endpoint Whitelist Helper
|
||||
**Source:** `backend/api/admin.py` lines 52–68 (`_user_to_dict`)
|
||||
**Apply to:** New admin AI config endpoints in `backend/api/admin.py`
|
||||
```python
|
||||
# From backend/api/admin.py lines 52–68 — safe whitelist pattern
|
||||
def _user_to_dict(user: User) -> dict:
|
||||
"""Return a safe subset — never includes password_hash, credentials_enc."""
|
||||
return {
|
||||
"id": str(user.id),
|
||||
"handle": user.handle,
|
||||
# ... whitelisted fields only
|
||||
}
|
||||
```
|
||||
New `_ai_config_to_dict(row: SystemSettings) -> dict` follows same pattern: includes `provider_id, base_url, model_name, context_chars, is_active, updated_at`, **never** `api_key_enc`.
|
||||
|
||||
### Admin Endpoint Auth + Audit Log
|
||||
**Source:** `backend/api/admin.py` lines 153–167, 388–410
|
||||
**Apply to:** New `GET /api/admin/ai-config` and `PUT /api/admin/ai-config` endpoints
|
||||
```python
|
||||
# From backend/api/admin.py lines 153–158
|
||||
@router.get("/users")
|
||||
async def list_users(
|
||||
session: AsyncSession = Depends(get_db),
|
||||
_admin: User = Depends(get_current_admin), # <-- mandatory on every admin endpoint
|
||||
) -> dict:
|
||||
```
|
||||
```python
|
||||
# Audit log write pattern — from lines 390–399
|
||||
await write_audit_log(
|
||||
session,
|
||||
event_type="admin.ai_config_changed",
|
||||
user_id=None, # global config — no target user
|
||||
actor_id=_admin.id,
|
||||
resource_id=None,
|
||||
ip_address=get_client_ip(request),
|
||||
metadata_={"provider_id": body.provider_id},
|
||||
)
|
||||
```
|
||||
|
||||
### Frontend API Call Pattern
|
||||
**Source:** `frontend/src/api/client.js` lines 277–284
|
||||
**Apply to:** New `getAiConfig()`, `saveAiConfig()` functions in `client.js`
|
||||
```javascript
|
||||
// From frontend/src/api/client.js lines 277–284
|
||||
export function adminUpdateAiConfig(id, provider, model) {
|
||||
return request(`/api/admin/users/${id}/ai-config`, {
|
||||
method: 'PATCH',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ ai_provider: provider, ai_model: model }),
|
||||
})
|
||||
}
|
||||
```
|
||||
New functions follow same `request()` wrapper pattern with `method: 'GET'` or `method: 'PUT'` and `Content-Type: application/json`.
|
||||
|
||||
### Celery Task Structure (asyncio.run bridge)
|
||||
**Source:** `backend/tasks/document_tasks.py` lines 22–25, 127–136
|
||||
**Apply to:** Refactored `extract_and_classify` (same file)
|
||||
```python
|
||||
# From document_tasks.py lines 127–136 — second task shows the same pattern
|
||||
@celery_app.task(name="tasks.document_tasks.cleanup_abandoned_uploads")
|
||||
def cleanup_abandoned_uploads() -> dict:
|
||||
return asyncio.run(_cleanup_abandoned())
|
||||
```
|
||||
The `asyncio.run()` bridge pattern is the standard — sync task def wraps an async `_run()`. The retry exception must be raised from the sync layer, not inside `asyncio.run()`.
|
||||
|
||||
---
|
||||
|
||||
## No Analog Found
|
||||
|
||||
| File | Role | Data Flow | Reason |
|
||||
|---|---|---|---|
|
||||
| `_truncate()` method pattern | utility | transform | No existing truncation helper in the codebase; defined in RESEARCH.md Pattern (smart truncation 60/40 split) |
|
||||
|
||||
---
|
||||
|
||||
## Metadata
|
||||
|
||||
**Analog search scope:** `backend/ai/`, `backend/storage/`, `backend/api/`, `backend/services/`, `backend/tasks/`, `backend/db/`, `backend/migrations/versions/`, `frontend/src/components/admin/`, `frontend/src/components/documents/`, `frontend/src/api/`
|
||||
**Files scanned:** 14 source files read directly
|
||||
**Pattern extraction date:** 2026-06-02
|
||||
Reference in New Issue
Block a user