Commit Graph
530 Commits
Author SHA1 Message Date
curo1305andClaude Sonnet 4.6 888856aa8b docs(06): add code review report — 7 critical, 9 warning, 4 info findings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 23:01:53 +02:00
curo1305andClaude Sonnet 4.6 b8b0840729 docs(phase-7): update tracking after wave 3 — plan 07-03 complete
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:19:03 +02:00
curo1305 e0cd183a63 chore: merge executor worktree (wave3-recovery/07-03) 2026-06-04 19:17:50 +02:00
curo1305 5f72814f07 docs(07-03): complete plan summary — AnthropicProvider output_config + classifier ProviderConfig refactor 2026-06-04 19:16:43 +02:00
curo1305 95c386f764 feat(07-03): classifier wired to load_provider_config + ai_config stub removed — D-04/D-06
- Remove _ProviderConfigStub from services/ai_config.py (replaced by real ProviderConfig)
- Add module-level import: from ai.provider_config import ProviderConfig, PROVIDER_DEFAULTS
- load_provider_config() now returns ProviderConfig (no lazy import inside function body)
- classifier.py: replace inline _settings dict with load_provider_config(session) call (D-06)
- Per-user override path builds ProviderConfig from PROVIDER_DEFAULTS (no api_key — T-07-06)
- Fallback to app_settings defaults when load_provider_config returns None (D-15)
- Truncation delegated to provider._truncate() — no more text slices in classifier (D-12/D-13)
- Promote test_api_key_encrypt_decrypt to passing (round-trip + domain salt isolation)
- Update test_classifier.py: test_per_user_provider + test_default_provider_fallback use ProviderConfig assertions
2026-06-04 19:14:48 +02:00
curo1305andClaude Sonnet 4.6 420fecdacd docs(phase-6): update tracking after wave 3 — plan 06-06 complete, Phase 6 done
All 6 Phase 6 plans complete. Phase 6 status updated to Complete in STATE.md
and ROADMAP.md (plans 06-04, 06-05, 06-06 checked). UAT passed (06-UAT.md),
D-10 CVE gate passed (trivy exit 0), D-14 RUNBOOK.md in place.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:14:22 +02:00
curo1305andClaude Sonnet 4.6 54521e2b99 docs(06-06): create plan SUMMARY — RUNBOOK.md + D-10 CVE gate passed
Task 1: RUNBOOK.md (580 lines, 8 D-14 sections). Task 2: trivy scan exits 0
after patching OpenSSL CVE-2026-31789; Mesa/perl unfixable CVEs documented in
.trivyignore. Phase 6 plan 06-06 complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:13:45 +02:00
curo1305andClaude Sonnet 4.6 382f9bec6b fix(06-06): patch OpenSSL CVE-2026-31789 — add apt-get upgrade to runtime Dockerfile stage
Apply all available OS security updates in the runtime image stage to pull in
openssl 3.5.5-1~deb13u2 (fixes CVE-2026-31789 heap buffer overflow). Three
CVEs with no upstream fix (Mesa will_not_fix, perl-base affected) documented
and suppressed in .trivyignore with rationale. Trivy rescan exits 0 (D-10 gate
passes).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:13:05 +02:00
curo1305 efc177a155 feat(07-03): AnthropicProvider singleton + output_config + truncation — D-03/D-07/D-12/D-13
- Delete MAX_AI_CHARS constant and _client() method from anthropic_provider.py
- Add self._client = AsyncAnthropic(...) singleton in __init__ (D-07)
- Add _truncate() with 60/40 split using self._context_chars (D-13)
- Add _CLASSIFICATION_SCHEMA and _SUGGESTIONS_SCHEMA module-level constants
- classify() and suggest_topics() pass output_config with json_schema format (D-03)
- stop_reason != "end_turn" degrades to parse_classification("") (T-07-08)
- Widen __init__ signature to (api_key, model, context_chars, base_url) (uniform ctor)
- Update ai/__init__.py to pass context_chars + base_url to AnthropicProvider
- Promote test_anthropic_structured_output + add test_anthropic_stop_reason_fallback
2026-06-04 19:10:05 +02:00
curo1305andClaude Sonnet 4.6 ea8df02491 docs(phase-7): update tracking after wave 2 — plan 07-02 complete
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:06:07 +02:00
curo1305andClaude Sonnet 4.6 670df192d6 docs(06-06): add RUNBOOK.md — env vars, startup, backup, health checks, escalation, failure modes
Covers D-14: all required env vars with examples, Docker Compose startup/shutdown
procedures, PostgreSQL + MinIO backup strategy with restore commands, health check
verification for all 7 services, on-call escalation for 6 alert classes, and
recovery steps for all 7 Phase 6 failure modes (Pitfalls 1, 5, 6, 7 + rate limit
edge cases + Grafana/Loki). docker scout CVE scan gate (D-10) documented in
Section 5 with prerequisites, pass/fail criteria, remediation, and Trivy fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:05:05 +02:00
curo1305 60b621f424 chore: merge executor worktree (worktree-agent-afe278fe2f809d30b) 2026-06-04 19:04:52 +02:00
curo1305 00c154d61b docs(07-02): complete provider config + GenericOpenAIProvider + registry plan
- ProviderConfig + PROVIDER_DEFAULTS + SUPPORTS_JSON_MODE in provider_config.py
- GenericOpenAIProvider with JSON-mode conditional + D-02 fallback
- OpenAIProvider singleton _client + smart truncation (D-07/D-13)
- Registry-based get_provider(config: ProviderConfig) — no if/elif chain
- MAX_AI_CHARS removed from openai_provider.py + classifier.py
- anthropic floor bumped to >=0.95.0 for Plan 03 output_config
- 6 Wave-2 xfail tests promoted; suite: 363 passed, 15 xfailed, 1 pre-existing failure
2026-06-04 19:04:18 +02:00
curo1305 209b156af1 test(07-02): promote 6 Wave-2 xfail tests to passing — D-01/D-02/D-07/D-12/D-13
- test_get_provider_typed: registry get_provider() returns correct class + ValueError on bogus id
- test_client_singleton: AsyncOpenAI instantiated once per provider instance (D-07)
- test_generic_openai_json_mode: response_format present/absent based on supports_json_mode (D-01)
- test_context_chars_truncation: _truncate() shortens long text (D-12)
- test_smart_truncation: 60% head + 40% tail truncation strategy (D-13)
- test_gemini_fallback_to_parse_classification: supports_json_mode=False skips
  response_format AND routes through parse_classification() from ai.utils (D-02 contract)
- test_anthropic_structured_output remains xfail (Plan 07-03)
2026-06-04 19:01:38 +02:00
curo1305 13eef371bc feat(07-02): registry-based get_provider(config: ProviderConfig) — D-06
- Rewrites ai/__init__.py with _REGISTRY dict (10 entries) replacing the if/elif chain
- get_provider() accepts ProviderConfig (not raw dict), raises ValueError on unknown
- Resolves effective values: api_key fallback to "not-needed", model/base_url/context_chars
  from PROVIDER_DEFAULTS when config fields are empty/zero
- SUPPORTS_JSON_MODE lookup sets supports_json_mode on GenericOpenAIProvider instances
- AnthropicProvider instantiated without base_url (Plan 03 will widen the ctor)
- provider_config.py: context_chars default changed to 0 (sentinel for PROVIDER_DEFAULTS
  resolution); 8000 default was inconsistent with O(1) factory pattern (Rule 1 fix)
2026-06-04 19:01:26 +02:00
curo1305andClaude Sonnet 4.6 582136c120 test(06): complete UAT - 8 passed, 0 issues
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 19:00:19 +02:00
curo1305 02bcbb9143 feat(07-02): singleton OpenAIProvider + GenericOpenAIProvider + MAX_AI_CHARS removal
- openai_provider.py: singleton self._client=AsyncOpenAI(...) in __init__ (D-07),
  _truncate() 60/40 smart truncation (D-13), removed MAX_AI_CHARS constant,
  changed __init__ signature to include context_chars, removed _client() method
- generic_openai_provider.py: new class, subclasses OpenAIProvider, conditional
  response_format={"type":"json_object"} on supports_json_mode flag (D-01/D-02),
  imports parse_classification + parse_suggestions from ai.utils (D-02 contract)
- ollama_provider.py: added context_chars kwarg with default 8000, passes through
- lmstudio_provider.py: added context_chars kwarg with default 8000, passes through
- classifier.py: removed MAX_AI_CHARS constant and text[:MAX_AI_CHARS] slices;
  truncation now handled inside each provider via _truncate()
- requirements.txt: bumped anthropic floor to >=0.95.0 (D-03 output_config support)
2026-06-04 18:58:19 +02:00
curo1305 beb5b5e49d feat(07-02): ProviderConfig Pydantic model + PROVIDER_DEFAULTS + SUPPORTS_JSON_MODE
- Creates backend/ai/provider_config.py as a pure data module (no provider imports)
- ProviderConfig(BaseModel) with extra=forbid: provider_id, api_key, base_url, model, context_chars
- PROVIDER_DEFAULTS dict with all 10 providers and verbatim base_url/model/context_chars from RESEARCH.md
- SUPPORTS_JSON_MODE dict with gemini=False per JSON Compatibility Matrix (D-02)
2026-06-04 18:56:31 +02:00
curo1305andClaude Sonnet 4.6 651713fa7a docs(phase-7): update tracking after wave 1 — plan 07-01 complete
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:53:25 +02:00
curo1305 88f9175f28 chore: merge executor worktree (worktree-agent-a034ce22177165198) 2026-06-04 18:51:55 +02:00
curo1305 7986661333 docs(07-01): complete plan summary — system_settings DB + HKDF + Wave 0 stubs
Tasks: 3/3 complete. Commits: 4febe2f, 4eb3177, 0fd6930.
Created system_settings table (Alembic 0005), SystemSettings ORM, HKDF AI key
encryption (info=b"ai-provider-settings"), load_provider_config(), seed startup
hook, and 13 Wave 0 xfail stubs across 4 new test files.
2026-06-04 18:51:12 +02:00
curo1305 0fd6930b41 feat(07-01): services/ai_config.py HKDF helpers, ProviderConfig loader, env seed
- Create backend/services/ai_config.py with:
  - _derive_ai_settings_key(master_key, provider_id): fresh HKDF per call, salt=provider_id.encode("utf-8"), info=b"ai-provider-settings" (domain-separated from b"cloud-credentials")
  - encrypt_api_key / decrypt_api_key: Fernet round-trip without JSON wrapping
  - load_provider_config(session): reads is_active=True row from system_settings, decrypts api_key_enc; returns stub ProviderConfig (real class lands in Plan 02)
  - seed_system_settings_from_env(session): idempotent insert of default provider on first boot
  - _ProviderConfigStub: minimal Pydantic model stub until ai/provider_config.py exists (Plan 02)
- Update backend/main.py: import seed_system_settings_from_env, call in lifespan with try/except so missing table (pre-migration fresh boot) doesn't crash startup
- Round-trip smoke test: encrypt_api_key → decrypt_api_key == original plaintext
2026-06-04 18:50:01 +02:00
curo1305 4eb317749f feat(07-01): Alembic migration 0005 + SystemSettings ORM model
- Add backend/migrations/versions/0005_system_settings.py (revision 0005, down_revision 0004)
- Creates system_settings table with 9 columns: id (UUID PK gen_random_uuid()), provider_id (String NOT NULL UNIQUE), api_key_enc (Text NULL), base_url (Text NULL), model_name (Text NOT NULL default ''), context_chars (Integer NOT NULL default 8000), is_active (Boolean NOT NULL default false), created_at/updated_at (TIMESTAMPTZ default now())
- UniqueConstraint uq_system_settings_provider_id on provider_id
- Add SystemSettings ORM model to backend/db/models.py following CloudConnection style
- downgrade() drops system_settings table cleanly
2026-06-04 18:46:11 +02:00
curo1305 4febe2f704 test(07-01): Wave 0 xfail stubs for D-01..D-16 (13 new stubs)
- Create backend/tests/test_ai_providers.py with 7 stubs: test_generic_openai_json_mode, test_anthropic_structured_output, test_get_provider_typed, test_client_singleton, test_context_chars_truncation, test_smart_truncation, test_gemini_fallback_to_parse_classification
- Create backend/tests/test_ai_config.py with 2 stubs: test_load_provider_config, test_api_key_encrypt_decrypt
- Create backend/tests/test_admin_ai_config.py with 2 stubs: test_get_never_returns_key, test_put_writes_active_provider
- Create backend/tests/test_document_tasks.py with 2 stubs: test_retry_backoff, test_exhaustion_sets_failed_status
- Append test_reclassify_requeues_celery xfail stub to test_documents.py
- D-14 regression guard: grep confirms 3 host-gateway entries (>=2 required)
2026-06-04 18:44:53 +02:00
curo1305andClaude Sonnet 4.6 f9dfa3dd9b docs(06-04): add partial plan summary — Tasks 1 & 2 complete, Task 3 human checkpoint pending
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:32:10 +02:00
curo1305andClaude Sonnet 4.6 f98cab30e9 chore: merge executor worktree (06-05) — trusted-proxy rate limiting + per-account limiter
Resolved merge conflicts:
- backend/main.py: keep both setup_logging (06-02) and account_limiter (06-05) imports
- backend/tests/test_rate_limiting.py: take 06-05 version (all 8 xfails promoted)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:31:46 +02:00
curo1305andClaude Sonnet 4.6 5c82a9840a docs(06-05): add plan summary — D-11/D-12 trusted-proxy IP + per-account limiter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:28:30 +02:00
curo1305andClaude Sonnet 4.6 a826738e18 feat(06-05): trusted-proxy get_client_ip, per-account rate limiter, promote 8 xfail tests (D-11/D-12/D-13)
- D-11: replace get_client_ip body with trusted-proxy CIDR check (127/8, 172.16/12,
  192.168/16, ::1/128); untrusted peers always return their own IP, preventing XFF
  spoofing by external callers
- D-12: create services/rate_limiting.py with _account_key() keyed by user.id
  (falls back to peer IP when no authenticated user in request.state); exports
  account_limiter = Limiter(key_func=_account_key)
- Wire: auth.py switches from get_remote_address to get_client_ip; main.py imports
  account_limiter; all 9 documents.py endpoints and 7 cloud.py endpoints decorated
  @account_limiter.limit("100/minute") with request.state.current_user = current_user
  as the first handler statement (A1 ordering invariant)
- Promote all 8 xfail stubs to real assertions; add autouse fixture in conftest.py
  to reset MemoryStorage between tests preventing cross-test 429 contamination

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:27:49 +02:00
curo1305andClaude Sonnet 4.6 378822682c feat(06-04): multi-stage Dockerfile with appuser + docker-compose runtime hardening (D-07/D-08/D-09)
- backend/Dockerfile: two-stage build (builder/runtime); runtime stage creates
  appuser uid=1000, copies installed packages from discardable builder stage,
  runs as non-root USER appuser; no --reload baked in CMD
- docker-compose.yml: backend + celery-worker get read_only:true,
  tmpfs:/tmp:mode=1777, cap_drop:ALL, security_opt:no-new-privileges:true;
  celery-beat intentionally left unhardened (Pitfall 7 — writes celerybeat-schedule)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 18:20:38 +02:00
curo1305 6d1ecbd9e2 docs(phase-06): update tracking after wave 1 — 06-02 and 06-03 complete 2026-06-03 18:54:06 +02:00
curo1305 339cb9b991 chore: merge executor worktree (06-02) — structured logging + Loki stack 2026-06-03 18:52:13 +02:00
curo1305 7c624b0b6b docs(06-02): complete structured logging + Loki stack plan summary
- 3 tasks complete: structlog dependency, CorrelationIDMiddleware, Loki stack
- D-01: JSON logging with correlation IDs; X-Correlation-ID header on responses
- D-02: Loki+Promtail+Grafana in docker-compose; backend+celery-worker labelled
- D-03: no OpenTelemetry added
- 5 xfail stubs from 06-01 promoted to real assertions in test_logging.py
2026-06-03 18:51:29 +02:00
curo1305 203c225a3e feat(06-02): add Loki + Promtail + Grafana stack to docker-compose
- Create docker/loki/loki-config.yaml: single-binary filesystem-mode Loki
  (schema v13, tsdb store, embedded_cache 100MB, auth_enabled: false)
- Create docker/loki/promtail-config.yaml: docker_sd_configs scrape with
  label filter logging=promtail; relabels container name and service labels
- docker-compose.yml: add loki (:3100), promtail, grafana (:3000) services
  after celery-beat; loki_data and grafana_data named volumes; GF_AUTH
  anonymous enabled for local dev; backend + celery-worker get logging:
  "promtail" label; backend gets LOG_LEVEL/LOG_JSON env vars
- celery-beat deliberately left without logging: label (not a network-facing
  service; schedule file write concerns deferred per D-08 Pitfall 7 notes)
2026-06-03 18:49:41 +02:00
curo1305 e498d884bb docs(06-03): complete load test plan summary — strategy A, direct upload, SLA listener 2026-06-03 18:49:09 +02:00
curo1305 5a93257ac0 feat(06-03): implement Locust load test — self-bootstrap strategy, 4 tasks, SLA listener (D-04/D-05/D-06) 2026-06-03 18:48:47 +02:00
curo1305 abe8f8ee90 feat(06-02): wire CorrelationIDMiddleware + config fields + promote 5 test stubs
- backend/config.py: add log_level: str = "INFO" and log_json: bool = False
  fields under Observability (Phase 6 — D-01) comment; pydantic-settings reads
  LOG_LEVEL / LOG_JSON env vars automatically
- backend/main.py: add imports (uuid, time, structlog, ASGIApp/Receive/Scope/Send,
  setup_logging); add CorrelationIDMiddleware raw-ASGI class (NOT BaseHTTPMiddleware
  — avoids streaming buffering); call setup_logging() as first lifespan statement;
  register CorrelationIDMiddleware LAST so it runs FIRST (Starlette reverse order)
- backend/tests/test_logging.py: remove all 5 xfail decorators; replace single-line
  bodies with real assertions for JSON renderer, contextvar binding, X-Correlation-ID
  header, no-bleed between requests, uvicorn.access propagate=False
2026-06-03 18:47:49 +02:00
curo1305 fb35f0e988 feat(06-03): add backend/requirements-dev.txt with locust pin (D-04) 2026-06-03 18:47:43 +02:00
curo1305 9fa74a91f5 feat(06-02): add structlog dependency + create services/logging.py
- Append structlog>=25.5.0 to requirements.txt under Phase 6 D-01 comment
- Create backend/services/logging.py with setup_logging(json_logs, log_level)
- shared_processors: merge_contextvars first, then add_log_level, add_logger_name,
  PositionalArgumentsFormatter, ExtraAdder, TimeStamper(iso), StackInfoRenderer
- JSON branch appends format_exc_info to shared_processors
- JSONRenderer when json_logs=True, ConsoleRenderer when False
- ProcessorFormatter bridges uvicorn/stdlib loggers through same chain
- Idempotent: clears root handlers before installing new one
- uvicorn.access propagate=False — middleware owns request logging
2026-06-03 18:44:37 +02:00
curo1305 c11984c66c docs(phase-06): update tracking after wave 0 — 06-01 complete 2026-06-03 18:41:46 +02:00
curo1305 3e3f980466 docs(06-01): complete Wave 0 Nyquist scaffold plan summary
- 13 xfail stubs: 5 for D-01/D-02 (logging), 8 for D-11/D-12+A1 (rate limiting)
- load_tests package skeleton (D-04/D-05/D-06) excluded from pytest
- Package legitimacy decisions recorded (structlog + locust both approved)
- Full suite: 344 passed, 20 xfailed, 1 pre-existing failure, 0 regressions
2026-06-03 18:39:21 +02:00
curo1305 594eb46efe feat(06-01): add backend/load_tests/ package skeleton with Locust stub (D-04/D-05/D-06)
- backend/load_tests/__init__.py — empty package marker; stops pytest discovery
- backend/load_tests/locustfile.py — DocuVaultUser skeleton (on_start,
  _auth_headers, list_documents, upload_document, refresh_token methods)
  and check_sla SLA listener; all bodies raise NotImplementedError
- No application imports; TEST_EMAIL/TEST_PASSWORD from env vars only
- Full implementation in plan 06-03
2026-06-03 18:37:23 +02:00
curo1305 56d9da7be1 test(06-01): add 8 xfail stubs for rate limiting (D-11/D-12 + A1)
- D-11: 4 stubs for trusted-proxy CIDR logic in get_client_ip
  (untrusted peer, trusted+XFF, trusted+no-XFF, None client)
- D-12: 2 stubs for account_limiter key_func (user_id, fallback to IP)
- A1: test_account_limiter_key_ordering_assumption — explicit gate that
  request.state.current_user is readable by key_func before counting
- D-12 integration: 429 after 100 req/min with same auth_user
All strict=False xfail stubs; implementation target: plan 06-04
2026-06-03 18:36:35 +02:00
curo1305 e1f8874b9d test(06-01): add 5 xfail stubs for structured logging (D-01/D-02)
- test_setup_logging_emits_json_when_LOG_JSON_true
- test_correlation_id_middleware_binds_contextvar
- test_correlation_id_response_header_present
- test_contextvars_cleared_between_requests (Pitfall 2 guard)
- test_uvicorn_access_log_suppressed
All strict=False xfail stubs; implementation target: plan 06-02
2026-06-03 18:35:55 +02:00
curo1305andClaude Sonnet 4.6 bdd784341f wip: phase 7 planning paused — ready to execute
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-03 18:34:49 +02:00
curo1305andClaude Sonnet 4.6 3df62506c9 docs(07): create phase plan — 5 plans, verification passed
5 waves: system_settings DB + HKDF encryption (01), ProviderConfig +
GenericOpenAIProvider + singleton client fix (02), Anthropic output_config +
classifier wiring (03), Celery retry 30/90/270s + re-queue endpoint (04),
admin AI panel + DocumentCard badge + human checkpoint (05).

All 18 decisions D-01..D-18 covered. Plan checker passed after 1 revision
round (4 blockers fixed: D-02/D-14 coverage, D-11 Vitest tests, Plan 05
files_modified).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-03 18:25:00 +02:00
curo1305andClaude Sonnet 4.6 6c8e0d8bde docs(07): research phase — Anthropic output_config, client singleton pattern, Celery retry, system_settings design
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-02 20:41:03 +02:00
curo1305andClaude Sonnet 4.6 70c09f6cd4 docs(06): create phase 6 plan — performance & production hardening
6 plans across 4 waves covering structlog/Loki observability, Locust
load testing, multi-stage Dockerfile hardening, trusted-proxy rate
limiting, and RUNBOOK.md. Verification passed (0 blockers).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-02 20:35:03 +02:00
curo1305 b7503cdff4 docs(state): record phase 7 context session 2026-06-02 20:24:12 +02:00
curo1305 b8d128ee5e docs(07): capture phase context 2026-06-02 20:24:08 +02:00
curo1305 ad237dbaf1 docs(06): research phase — structlog, Locust, container hardening, per-account rate limiting 2026-06-02 16:34:02 +02:00