Compare commits

..

25 Commits

Author SHA1 Message Date
curo1305 0f760c379d fix: remove obsolete /data/documents and /config dirs from Dockerfiles
doc-service and ai-service no longer use local filesystem directories —
all file and config I/O goes through storage-service. Update README and
CLAUDE.md to reflect 6-service architecture, new volumes, and add
storage-service step to the "Adding a new resource" checklist.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 13:45:12 +02:00
curo1305 f13ef88711 Merge feat/storage-service: dedicated storage service with pluggable backends
All file/blob persistence now routes through storage-service (port 8020).
Replaces doc_data and app_config volumes. Supports local (default), S3-compatible,
and WebDAV backends with zero-data-loss migration flow.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 13:41:26 +02:00
curo1305 0d8e0366c6 docs: always use port 5173 for feature stacks (no per-branch ports)
Update feature branch workflow: stop main stack before starting feature
stack, always use :5173. Simplify feat override template — no port
remapping needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 11:48:22 +02:00
curo1305 3a66aeeec5 fix: rename download_file import to storage_download to avoid shadow
The route handler async def download_file() shadowed the storage import
of the same name, causing the endpoint to call itself recursively.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 11:48:04 +02:00
curo1305 248b2bb9d7 fix: remove unused imports in StorageAdminPage
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 11:25:59 +02:00
curo1305 cfec3bb906 feat: Phase 4+5 — admin storage UI, backend proxy, CLAUDE.md enforcement
- backend/app/routers/storage_config.py: 5 admin-only endpoints proxying
  storage-service config + migration API (GET/PATCH/POST/DELETE)
- backend/app/main.py: register storage_config router
- frontend/src/api/client.ts: StorageStatus, MigrationStatus,
  StorageBackendConfig interfaces + 5 API functions
- frontend/src/pages/StorageAdminPage.tsx: full admin UI — backend health
  dot, driver selector (local/S3/WebDAV), conditional credential fields,
  Test & Migrate button, live 2s-poll migration progress bar, Cancel
- frontend/src/App.tsx: /admin/storage route (AdminRoute guard)
- CLAUDE.md: storage enforcement rule, updated Docker tables (6 services,
  3 volumes), §20 in merge checklist
- backend/CLAUDE.md, frontend/CLAUDE.md, doc-service/CLAUDE.md,
  ai-service/CLAUDE.md: updated to reflect storage-service integration
- tests/ALL_TESTS.md + tests/storage-service_tests.md: §20 (20 tests)
- backend/STATUS.md, frontend/STATUS.md: updated with new endpoints/routes
- changelog/2026-04-20_storage-service.md: full change log

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 16:13:05 +02:00
curo1305 4c35d7a2a4 feat: migrate app_config volume to storage-service config bucket (Phase 3)
All JSON config files (AI settings, doc settings, appearance, themes) now live
in the 'config' bucket of storage-service instead of a shared Docker volume.

- backend/core/config_storage.py: new async HTTP helpers for config bucket r/w
- backend/core/app_config.py: fully async rewrite; all load_*/save_*/seed_*
  functions use config_storage instead of filesystem
- backend/routers/settings.py: all asyncio.to_thread() wrappers removed; direct
  await calls throughout; update_theme reads via load_theme_by_id()
- backend/main.py: await seed_builtin_themes() directly (no to_thread)
- ai-service: remove CONFIG_PATH, add STORAGE_SERVICE_URL; config_reader now
  fetches from storage-service via httpx
- doc-service: config_reader rewritten to fetch/write via storage-service
- docker-compose: remove app_config volume; add storage-service depends_on for
  ai-service; remove DATA_DIR and CONFIG_PATH from doc-service

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 16:02:57 +02:00
curo1305 2f3efb9bf9 feat: migrate doc-service to use storage-service for file I/O (Phase 2)
- storage.py: replace aiofiles filesystem ops with httpx calls to
  storage-service PUT/GET/DELETE /objects/documents/{key}
- Document model: rename file_path → storage_key (plain object key, no path prefix)
- Migration 0008: ALTER COLUMN + data migration strips /data/documents/ prefix
- documents.py: update upload, delete, download endpoints; _extract_pdf_text
  now takes bytes (pdfplumber.open(BytesIO)) instead of a filesystem path
- file_watcher.py: store storage_key instead of file_path on ingestion
- doc-service config: add STORAGE_SERVICE_URL env var

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 15:57:29 +02:00
curo1305 5349f21752 feat: add storage-service container with pluggable backends (Phase 1)
New FastAPI microservice (port 8020) providing unified blob storage via
PUT/GET/DELETE/LIST HTTP API. Local filesystem backend is the default (zero
extra deps). S3-compatible and WebDAV backends are built in. Backend is
switchable at runtime via POST /migrate, which copies all objects to the new
backend, verifies each one, atomically switches, then cleans up the old backend.

WebDAV XML parsing uses defusedxml to prevent XXE attacks.

Wired into docker-compose (storage_data volume) and registered in the backend
service-health poller as 'storage-service'.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 15:50:31 +02:00
curo1305 50d2348b36 refactor: rename MERGE_CHECKLIST to ALL_TESTS + add per-service test files
- tests/MERGE_CHECKLIST.md → tests/ALL_TESTS.md (git rename, updated header + index of sub-files)
- tests/backend_tests.md — §1–9, §18 (auth, users, admin, groups, appearance, service health, plugins, AI/doc settings, infra/security)
- tests/frontend_tests.md — §19 (UI & routing)
- tests/doc-service_tests.md — §10–16 (upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory)
- tests/ai-service_tests.md — §17 (AI queue & providers)
- CLAUDE.md: updated merge checklist section, file tree, and self-update checkpoint with mandatory test-file update rule
- settings.local.json: added docker inspect/ps, curl, lsof, git merge/branch/log/diff/status/config/mv permissions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 02:19:51 +02:00
curo1305 d345ace86d fix: admin delete bypass + update merge checklist for new features
- Fix doc-service delete endpoint: admins could not delete non-owned,
  non-shared documents — they hit 404 because the initial query filtered
  by owner/watch/group even before the is_admin bypass was checked.
  Admins now get an unconditional fetch, consistent with intent.
- Add 18 new checklist tests covering: group admin role (4.9–4.10),
  delete permission variants (12.16b–12.16e), can_delete sharing
  (13.11–13.14), category scopes / PascalCase naming (14.7–14.17),
  and three-dots portal fix (19.11).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 02:12:40 +02:00
curo1305 c59718171c Merge: resolve conflicts between feat/document-delete-permissions and feat/category-scopes-group-admin
- Keep HEAD's get_user_admin_groups dep and richer delete permission logic (can_delete via share OR group admin path)
- Use sa.text("false") for migration server_default (correct SQLAlchemy form)
- Preserve 0006/0007 migration entries in doc-service CLAUDE.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 01:06:04 +02:00
curo1305 99d22660f9 Merge branch 'feat/category-scopes-group-admin' 2026-04-18 22:36:55 +02:00
curo1305 fcfc06cda9 fix: rename existing system categories to PascalCase-with-dashes via migration
Migration 0007 converts all scope='system' category names in-place
(e.g. "invoices" → "Invoices", "vendor-invoices" → "Vendor-Invoices").

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 22:33:16 +02:00
curo f5bc28cda2 Merge pull request 'feat: document delete permissions + three-dots menu portal fix' (#2) from feat/document-delete-permissions into main
Reviewed-on: #2
2026-04-18 22:27:17 +02:00
curo1305 1c8b35399c fix: capitalize watch-folder names to PascalCase-with-dashes on ingest
Folder names like "invoices" and "vendor-invoices" are now converted to
"Invoices" and "Vendor-Invoices" when the watcher auto-creates categories,
matching the naming convention enforced on user-created categories.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 22:26:24 +02:00
curo1305 ebf97b6f4a fix: show manage controls for system categories when user is superuser
canManage() returned false for system-scope categories unconditionally.
Superusers can manage all categories (backend already permits it), so
check is_admin from getMe() and short-circuit to true.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 22:23:30 +02:00
curo1305 fec3953009 feat: category scopes, group-admin role, and permission model
- Three category scopes: personal / group / system (watch)
- PascalCase-with-dashes naming convention enforced at backend + frontend
- is_group_admin flag on GroupMembership; PATCH endpoint for admins to toggle it
- Categories router: scope-based list/create/rename/delete with _check_can_manage_cat
- Documents router: delete uses is_admin + can_delete share flag + group-admin check; remove_category requires doc ownership; assign_category accepts group/system categories
- Proxy layers inject x-user-is-admin and x-user-admin-groups headers
- Frontend: ManageCategoriesDialog grouped by scope with lock icons; SourcePanel scope picker + client-side name validation; AdminGroupsPage group-admin checkbox

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 22:16:49 +02:00
curo1305 6e5e5c08bf feat: document delete permissions + three-dots menu portal fix
- Add can_delete column to document_shares (migration 0005)
- Inject x-user-is-admin header from backend proxy to doc-service
- Add get_user_is_admin() dep in doc-service
- Delete endpoint now allows: owner, admin, or group member with can_delete=true
- Watch documents (user_id='watch') deletable by admins only
- DocumentOut gains viewer_can_delete (computed per-request)
- Share UI: 'Allow group members to delete' checkbox + trash badge on shares
- RowActionsMenu dropdown portaled to document.body — fixes overflow-hidden clipping
- Delete mutation onError handler — no more silent failures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 21:39:01 +02:00
curo1305 05d79d3d21 Fix 401 redirect loop on login page
The 401 handler was redirecting to /login unconditionally, causing an
infinite reload loop when useTheme fired unauthenticated API calls on
the login page itself. Now only redirects if not already on /login.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 21:16:45 +02:00
curo1305 75b7ae6062 Merge feat/replace-axios-with-fetch: replace Axios with native fetch + 401 redirect 2026-04-18 21:05:59 +02:00
curo1305 479108779f Replace Axios with native fetch; add global 401 session-expiry redirect
All API calls now go through a thin request() wrapper around native fetch.
Removes the axios dependency entirely. The wrapper injects the JWT on every
request and — the key fix — clears localStorage and redirects to /login on
any 401 response, so expired sessions no longer leave users on broken pages.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 21:04:18 +02:00
curo1305 c5976882be Split monolithic CLAUDE.md into per-service sub-files
Root CLAUDE.md now contains only project-wide concerns (stack, architecture,
Docker, workflows, security hook). Service-specific details moved to:
- backend/CLAUDE.md — DB models, API endpoints, JWT/bcrypt, naming conventions
- frontend/CLAUDE.md — routes, TanStack Query patterns, XSS prevention
- features/ai-service/CLAUDE.md — queue endpoints, provider notes
- features/doc-service/CLAUDE.md — document models, PDF limits, proxy endpoints

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 13:10:10 +02:00
curo1305 64808e0928 Edit the Workflow to include a plan phase and branching. 2026-04-18 12:53:50 +02:00
curo1305 94901fc30f Redesign doc service UX for scale + add group-based document sharing
- Three-column layout: Sidebar + SourcePanel (views + searchable category tree) + main
- DocumentSlideOver (480px right panel): inline editing, type picker, AI suggestion confirm/reject,
  categories combobox, tags editor, sharing section, raw text, re-analyse/delete actions
- ManageCategoriesDialog: inline rename, delete with confirm, search filter
- DocumentsPage rewrite: filter chip system, multi-file upload queue, drag-and-drop overlay,
  bulk actions bar (share/delete), smart TanStack Query polling, URL-driven view state
- Sidebar simplified: per-category NavLinks removed; Documents = single NavLink under Apps
- Backend: document_shares table (migration 0004), share CRUD endpoints, shared-with-me view,
  N+1-safe share_count via GROUP BY, recipient download access, X-User-Groups header enforcement
- Gateway proxy: injects X-User-Groups header into all document + category proxy requests
- Backend users: GET /api/users/me/groups endpoint for share picker combobox
- CLAUDE.md, STATUS.md files, and changelog updated

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 12:46:43 +02:00
95 changed files with 7302 additions and 1888 deletions
+14 -1
View File
@@ -5,8 +5,21 @@
"Bash(git add:*)",
"Bash(git commit -m ':*)",
"Bash(git push:*)",
"Bash(git pull:*)",
"Bash(git checkout:*)",
"Bash(git merge:*)",
"Bash(git branch:*)",
"Bash(git log:*)",
"Bash(git diff:*)",
"Bash(git status:*)",
"Bash(git config:*)",
"Bash(git mv:*)",
"Bash(docker compose:*)",
"Bash(docker run:*)"
"Bash(docker run:*)",
"Bash(docker inspect:*)",
"Bash(docker ps:*)",
"Bash(curl:*)",
"Bash(lsof:*)"
]
}
}
+98 -634
View File
@@ -1,19 +1,43 @@
# CLAUDE.md
This file provides permanent, authoritative guidance to Claude Code for every session. All sections below reflect the actual codebase state and must be kept up-to-date as the project evolves.
This file provides permanent, authoritative guidance to Claude Code for every session. It covers project-wide concerns only. Service-specific details live in sub-files — read them only when working in that service:
- `backend/CLAUDE.md` — auth/users/admin/settings/plugins endpoints; DB models; JWT/bcrypt/sanitization security; naming conventions
- `frontend/CLAUDE.md` — routes, components, API client patterns, XSS prevention
- `features/ai-service/CLAUDE.md` — /chat, /health, /queue endpoints; queue service
- `features/doc-service/CLAUDE.md` — document/category/share endpoints; DB models; PDF limits; file watcher
- `features/storage-service/CLAUDE.md` — storage API, pluggable backend drivers (local/S3/WebDAV), migration
---
## Merge checklist
Before merging any feature branch into `main`, every test relevant to the changed area in `tests/ALL_TESTS.md` (and the relevant service-specific file) must be marked passing. The test suite covers all 20 feature areas across five service files:
- `tests/backend_tests.md` — §19, §18
- `tests/frontend_tests.md` — §19
- `tests/doc-service_tests.md` — §1016
- `tests/ai-service_tests.md` — §17
- `tests/storage-service_tests.md` — §20
Do not merge without it.
---
## CLAUDE.md self-update checkpoint
**After every change to the codebase**, before committing, check whether CLAUDE.md needs updating:
**After every change to the codebase**, before committing, check which CLAUDE.md files need updating:
- New route added → update **All API Endpoints** and **Frontend Routes** tables
- New DB model or column → update **Database Models**
- New migration → update **Migration chains**
- New file or directory → update **File & Folder Tree**
- New limit or default value changed → update **Default Values & Limits**
- New dependency, auth mechanism, or security pattern → update **Security Standards**
- New Docker service, volume, network, or env var → update **Docker Infrastructure**
- Stack version changed → update **Stack**
- New route added → update **API Endpoints** in `backend/CLAUDE.md`, `features/doc-service/CLAUDE.md`, or `features/ai-service/CLAUDE.md`; update **Frontend Routes** in `frontend/CLAUDE.md`
- New DB model or column → update **Database Models** in `backend/CLAUDE.md` or `features/doc-service/CLAUDE.md`
- New migration → update **Migration chain** table in `backend/CLAUDE.md` or `features/doc-service/CLAUDE.md`
- New file or directory → update **File & Folder Tree** in the relevant sub-file; update the high-level tree in this root file only if a top-level directory changes
- New limit or default value changed → update **Default Values & Limits** in the relevant sub-file
- New dependency, auth mechanism, or security pattern → update **Security Standards** in the relevant sub-file
- New Docker service, volume, network, or env var → update **Docker Infrastructure** in this file
- Stack version changed → update **Stack** in this file
- New feature or endpoint added → add test rows to **both** `tests/ALL_TESTS.md` (in the relevant section) **and** the matching service-specific file (`tests/backend_tests.md`, `tests/frontend_tests.md`, `tests/doc-service_tests.md`, `tests/ai-service_tests.md`, or `tests/storage-service_tests.md`). Use the same test number and format as existing rows.
This check is mandatory — treat it the same as updating STATUS.md.
@@ -36,22 +60,6 @@ This check is mandatory — treat it the same as updating STATUS.md.
All test, build, and package-manager commands run **inside Docker** — never on the host. See the memory note: "Testing inside Docker only".
### Migrations (run in Docker)
```bash
docker compose exec backend alembic revision --autogenerate -m "describe change"
docker compose exec backend alembic upgrade head
docker compose exec backend alembic downgrade -1
```
### Lint (run in Docker)
```bash
docker compose exec backend ruff check . && ruff format .
docker compose exec frontend npm run typecheck
docker compose exec frontend npm run lint
```
### Full stack
```bash
@@ -63,13 +71,15 @@ docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
docker compose up --build -d
```
For service-specific commands (migrations, lint), see `backend/CLAUDE.md` and `frontend/CLAUDE.md`.
---
## File & Folder Tree
```
/
├── CLAUDE.md ← This file — authoritative session context
├── CLAUDE.md ← This file — project-wide context
├── README.md ← Project overview, containers table, Current State
├── TODO.md ← Task list
├── .env.example ← Template for backend/.env
@@ -78,113 +88,18 @@ docker compose up --build -d
├── .githooks/pre-commit ← Runs scripts/security_check.py before every commit
├── scripts/security_check.py ← Static analysis: secrets, weak crypto, SQLi, JWT
├── changelog/YYYY-MM-DD_<slug>.md ← Per-date change logs
├── tests/ALL_TESTS.md ← Full test suite (all 19 areas); must pass before merging to main
├── tests/backend_tests.md ← Backend-only tests (§19, §18)
├── tests/frontend_tests.md ← Frontend-only tests (§19)
├── tests/doc-service_tests.md ← Doc-service tests (§1016)
├── tests/ai-service_tests.md ← AI-service tests (§17)
├── dev-watch/ ← Dev bind-mount for file watcher testing (.gitkeep only)
├── backend/ ← FastAPI gateway (port 8000, internal)
│ ├── app/
│ │ ├── main.py ← App factory, router registration, lifespan (health loop)
│ │ ├── database.py ← AsyncEngine, AsyncSessionLocal, Base
│ │ ├── deps.py ← get_current_user, get_current_admin, get_service_admin(id), check_plugin_access
│ │ ├── core/
│ │ │ ├── config.py ← All settings via pydantic-settings (reads .env)
│ │ │ ├── security.py ← JWT sign/verify (RS256), bcrypt hash/verify
│ │ │ ├── sanitize.py ← Input sanitization helpers (see Security Standards)
│ │ │ └── app_config.py ← Per-service config load/save to /config volume; theme files in /config/themes/
│ │ ├── models/
│ │ │ ├── __init__.py ← Imports all models (required for Alembic autogenerate)
│ │ │ ├── user.py ← User model (see Database Models)
│ │ │ ├── profile.py ← Profile model
│ │ │ └── group.py ← Group, GroupMembership models
│ │ ├── schemas/
│ │ │ ├── user.py ← UserCreate/Out, Token, DashboardPrefsOut/Update
│ │ │ ├── profile.py ← ProfileRead, ProfileUpdate
│ │ │ └── group.py ← GroupCreate/Update/Out/DetailOut, GroupMemberOut
│ │ ├── routers/
│ │ │ ├── auth.py ← POST /register, POST /login
│ │ │ ├── users.py ← GET /me, GET+PATCH /me/preferences, PATCH /me/color-mode
│ │ │ ├── profile.py ← GET+PUT /me (profile)
│ │ │ ├── admin.py ← User admin CRUD (admin-only)
│ │ │ ├── groups.py ← Group CRUD + member management (admin-only)
│ │ │ ├── settings.py ← AI, doc limits, system prompts, appearance, themes (admin-only)
│ │ │ ├── services.py ← GET /services (health status)
│ │ │ ├── plugins.py ← Generic plugin proxy (GET/PATCH /api/plugins/*)
│ │ │ ├── categories_proxy.py ← Transparent proxy → doc-service /categories/*
│ │ │ └── documents_proxy.py ← Transparent proxy → doc-service /documents/*
│ │ └── services/
│ │ ├── service_health.py ← Background 30s health-check loop; caches /plugin/manifest per service
│ │ └── group_bootstrap.py ← Ensures {service-id}-admin group exists for every registered service at startup
│ ├── alembic/
│ │ ├── env.py ← Async migration runner
│ │ └── versions/ ← Migration chain (see Migrations section)
│ ├── scripts/seed.py ← Seed test user
│ ├── Dockerfile ← python:3.12-slim, non-root user 1001
│ └── STATUS.md
├── backend/ ← FastAPI gateway (port 8000, internal); see backend/CLAUDE.md
├── features/
│ ├── ai-service/ ← AI provider intermediary (port 8010, internal)
│ ├── app/
├── main.pyFastAPI, queue worker lifespan
│ │ │ ├── routers/chat.py ← POST /chat (sync, NORMAL priority queue)
│ │ │ ├── routers/health.py ← GET /health
│ │ │ ├── routers/queue.py ← GET /queue/status, /pause, /resume, /cancel/{id}
│ │ │ └── routers/plugin.py ← GET /plugin/manifest (access rules for ai-service-admin group)
│ │ │ ├── providers/base.py ← AIProvider abstract class
│ │ │ ├── providers/anthropic_provider.py
│ │ │ ├── providers/openai_compat.py ← Ollama / LM Studio
│ │ │ └── services/queue.py ← Priority queue (CRITICAL > HIGH > NORMAL)
│ │ ├── Dockerfile
│ │ └── STATUS.md
│ │
│ └── doc-service/ ← PDF extraction microservice (port 8001, internal)
│ ├── app/
│ │ ├── main.py ← FastAPI, lifespan (file watcher start/stop)
│ │ ├── database.py ← Same PostgreSQL instance as backend
│ │ ├── deps.py ← get_user_id (reads x-user-id header)
│ │ ├── models/
│ │ │ ├── document.py ← Document model (see Database Models)
│ │ │ ├── category.py ← DocumentCategory model
│ │ │ └── category_assignment.py ← CategoryAssignment (composite PK)
│ │ ├── schemas/
│ │ │ ├── document.py ← DocumentOut, DocumentPage, DocumentStatusOut, etc.
│ │ │ └── category.py ← CategoryOut, CategoryCreate, CategoryUpdate
│ │ ├── routers/
│ │ │ ├── documents.py ← Full document CRUD + file serving + reprocess + suggestion endpoints
│ │ │ ├── categories.py ← Category CRUD (includes watch-owned categories)
│ │ │ └── plugin.py ← GET /plugin/manifest, GET+PATCH /plugin/settings
│ │ └── services/
│ │ ├── storage.py ← File I/O
│ │ ├── ai_client.py ← classify_document() → ai-service:8010/chat
│ │ ├── config_reader.py ← Config load/save including storage/watch settings
│ │ └── file_watcher.py ← watchdog-based PDF watcher + startup scan + ingestion
│ ├── alembic/versions/ ← Doc-service migration chain
│ │ └── 0003_add_watch_columns.py ← source, watch_path, suggested_folder, suggested_filename
│ ├── Dockerfile
│ └── STATUS.md
└── frontend/ ← React SPA (port 5173 dev / 80 prod)
├── src/
│ ├── main.tsx ← React root, QueryClientProvider, BrowserRouter
│ ├── App.tsx ← Route tree, PrivateRoute, AdminRoute
│ ├── api/client.ts ← Axios instance + ALL API functions (single source of truth)
│ ├── hooks/
│ │ ├── useAuth.ts ← Token state (localStorage), login/logout
│ │ └── useTheme.ts ← Theme toggle
│ ├── components/
│ │ ├── AppShell.tsx ← Layout: Sidebar + scrollable main
│ │ ├── Sidebar.tsx ← Collapsible nav (icons ↔ icons+labels)
│ │ ├── ThemeToggle.tsx ← Light/dark mode toggle
│ │ ├── PluginSchemaForm.tsx ← JSON Schema → React form (boolean/string/number/readOnly)
│ │ └── ui/ ← shadcn/ui components (Button, Input, …)
│ ├── pages/ ← One file per route (see Routes section)
│ │ ├── DocServiceSettingsPage.tsx ← Combined doc-service settings: upload limits + watch directory
│ │ └── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest
│ ├── lib/utils.ts ← cn() = clsx + tailwind-merge
│ └── styles/theme.css ← CSS custom properties, Tailwind setup
├── vite.config.ts ← /api/* proxied to backend:8000
├── tailwind.config.ts
├── components.json ← shadcn/ui config
├── Dockerfile ← Multi-stage: Node build → nginx-unprivileged
└── STATUS.md
│ ├── ai-service/ ← AI provider intermediary (port 8010, internal); see features/ai-service/CLAUDE.md
└── doc-service/ ← PDF extraction microservice (port 8001, internal); see features/doc-service/CLAUDE.md
└── frontend/ React SPA (port 5173 dev / 80 prod); see frontend/CLAUDE.md
```
---
@@ -205,8 +120,8 @@ Browser (:5173 dev / :80 prod)
/users /groups /documents/categories/*
/profile /settings
/services │ │
JSON volume proxy (injects x-user-id)
(/config) │
JSON volume proxy (injects x-user-id,
(/config) x-user-groups)
doc-service:8001
ai-service:8010
@@ -222,334 +137,25 @@ Browser (:5173 dev / :80 prod)
---
## Database Models
### Backend (`users`, `profiles`, `groups`, `group_memberships`)
**`users`**
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | auto-generated |
| `email` | String | UNIQUE, indexed, NOT NULL | lowercased before storing |
| `hashed_password` | String | NOT NULL | bcrypt 13 rounds |
| `full_name` | String | nullable | sanitized max 128 chars |
| `is_active` | Boolean | default=True | soft-delete flag |
| `is_superuser` | Boolean | default=False | admin role; never exposed as-is (serialised as `is_admin`) |
| `dashboard_app_ids` | JSON | NOT NULL, default=[] | list of pinned service IDs |
| `color_mode` | String | nullable, default=NULL | user's preferred mode: "light" / "dark" / "system" / NULL (use admin default) |
Relationship: `profile` (one-to-one, cascade all+delete-orphan)
**`profiles`**
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | auto-generated |
| `user_id` | String | FK→users.id UNIQUE, cascade delete | one-to-one |
| `phone` | String(20) | nullable | validated format |
| `date_of_birth` | Date | nullable | 1900+ and not future |
| `position` | String(128) | nullable | job title |
| `address` | String(255) | nullable | |
| `updated_at` | DateTime(tz) | server_default=now(), onupdate=now() | |
**`groups`**
| Column | Type | Constraints |
|--------|------|-------------|
| `id` | String | PK, UUID |
| `name` | String(128) | UNIQUE indexed, NOT NULL |
| `description` | String(512) | nullable |
| `created_at` | DateTime(tz) | server_default=now() |
**`group_memberships`**
| Column | Type | Constraints |
|--------|------|-------------|
| `id` | String | PK, UUID |
| `group_id` | String | FK→groups.id, indexed, CASCADE |
| `user_id` | String | FK→users.id, indexed, CASCADE |
| `joined_at` | DateTime(tz) | server_default=now() |
Unique constraint: `(group_id, user_id)`
### Doc-service (`documents`, `document_categories`, `document_category_assignments`)
**`documents`**
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | |
| `user_id` | String | indexed | not FK — trusts x-user-id header |
| `filename` | String | NOT NULL | |
| `file_path` | String | NOT NULL | absolute path under /data/documents |
| `file_size` | Integer | NOT NULL | bytes |
| `status` | String | default="pending" | pending / processing / done / failed |
| `title` | String(500) | nullable | AI-extracted |
| `document_type` | String | nullable | invoice / bill / receipt / order / expense / revenue / unknown |
| `raw_text` | Text | nullable | first 500 k chars |
| `extracted_data` | Text | nullable | JSON string |
| `tags` | Text | nullable | JSON array string |
| `error_message` | String(500) | nullable | |
| `created_at` | DateTime(tz) | server_default=now() | |
| `processed_at` | DateTime(tz) | nullable | |
| `source` | String(16) | default="upload" | "upload" or "watch" |
| `watch_path` | String | nullable | original absolute path in watch directory |
| `suggested_folder` | String(128) | nullable | AI-suggested category (pending user confirm) |
| `suggested_filename` | String(500) | nullable | AI-suggested title/rename (pending user confirm) |
**`document_categories`**
| Column | Type | Constraints |
|--------|------|-------------|
| `id` | String | PK, UUID |
| `user_id` | String | indexed |
| `name` | String(128) | NOT NULL |
| `created_at` | DateTime(tz) | server_default=now() |
**`document_category_assignments`** (composite PK)
| Column | Type | Constraints |
|--------|------|-------------|
| `document_id` | String | PK + FK→documents.id CASCADE |
| `category_id` | String | PK + FK→document_categories.id CASCADE |
### Migration chains
**Backend** (must be applied in order):
| Rev ID | Slug |
|--------|------|
| `38efeff7c45a` | `create_users_table` |
| `676084df61d1` | `add_profiles_table` |
| `a3f9c2d14e87` | `add_groups_and_group_memberships` |
| `c7e8f9a0b1d2` | `add_dashboard_app_ids_to_users` |
| `dd6ad2f2c211` | `add_color_mode_to_users` |
**Doc-service**:
| Rev ID | Slug |
|--------|------|
| `0001` | `create_doc_tables` |
| `0002` | `add_document_title` |
| `0003` | `add_watch_columns` |
---
## All API Endpoints
### Auth (`/api/auth`) — public
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/api/auth/register` | — | Create account; returns `UserOut`; enforces password policy |
| POST | `/api/auth/login` | — | OAuth2 password flow; returns `{access_token, token_type}` |
### Users (`/api/users`) — authenticated
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/users/me` | user | Current user info → `UserOut` |
| GET | `/api/users/me/preferences` | user | Dashboard pinned app IDs → `{app_ids}` |
| PATCH | `/api/users/me/preferences` | user | Save pinned app IDs (max 50, slug-safe) |
| PATCH | `/api/users/me/color-mode` | user | Save colour mode preference ("light"/"dark"/"system") |
### Profile (`/api/profile`) — authenticated
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/profile/me` | user | Fetch profile; auto-creates if missing |
| PUT | `/api/profile/me` | user | Update profile fields |
### Admin — Users (`/api/admin`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/users` | List all users → `list[UserAdminOut]` |
| POST | `/api/admin/users` | Create user (with optional is_admin) |
| DELETE | `/api/admin/users/{user_id}` | Delete user (204) |
| PATCH | `/api/admin/users/{user_id}/active` | Toggle active status |
### Admin — Groups (`/api/admin/groups`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/groups` | List groups with member count |
| POST | `/api/admin/groups` | Create group |
| GET | `/api/admin/groups/{id}` | Group detail + members |
| PATCH | `/api/admin/groups/{id}` | Update name / description |
| DELETE | `/api/admin/groups/{id}` | Delete (cascades memberships) |
| POST | `/api/admin/groups/{id}/members/{user_id}` | Add member |
| DELETE | `/api/admin/groups/{id}/members/{user_id}` | Remove member |
### Settings (`/api/settings`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/settings/ai` | AI config (keys masked) |
| PATCH | `/api/settings/ai` | Update AI provider / credentials |
| POST | `/api/settings/ai/test` | Test AI connection |
| GET | `/api/settings/documents/limits` | PDF upload limits |
| PATCH | `/api/settings/documents/limits` | Update max PDF size |
| GET | `/api/settings/system-prompts` | All editable system prompts |
| PATCH | `/api/settings/system-prompts/{service_id}` | Update system prompt |
| GET | `/api/settings/appearance` | Active theme + default mode (auth) |
| PATCH | `/api/settings/appearance` | Update active theme + default mode (admin) |
| GET | `/api/settings/themes` | List all themes — built-in + custom (auth) |
| POST | `/api/settings/themes` | Create custom theme (admin) |
| PATCH | `/api/settings/themes/{id}` | Update custom theme label/colours (admin) |
| DELETE | `/api/settings/themes/{id}` | Delete custom theme (admin, 204) |
### Services (`/api/services`) — authenticated
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/services` | Health status of all registered services → `list[ServiceStatus]` |
### Documents (`/api/documents/*`) — authenticated, proxied to doc-service
| Method | Path | Description |
|--------|------|-------------|
| POST | `/api/documents/upload` | Upload PDF (202, background processing) |
| GET | `/api/documents` | Paginated list (filterable: search, status, type, category, sort) |
| GET | `/api/documents/{id}` | Document detail |
| GET | `/api/documents/{id}/status` | Processing status only |
| PATCH | `/api/documents/{id}/type` | Update document type |
| PATCH | `/api/documents/{id}/tags` | Update tags |
| PATCH | `/api/documents/{id}/title` | Update title |
| POST | `/api/documents/{id}/reprocess` | Re-run AI extraction |
| DELETE | `/api/documents/{id}` | Delete document (204) |
| GET | `/api/documents/{id}/file` | Download PDF (streaming) |
| POST | `/api/documents/{id}/categories/{cat_id}` | Assign category |
| DELETE | `/api/documents/{id}/categories/{cat_id}` | Remove category |
| POST | `/api/documents/{id}/suggestions/folder/confirm` | Confirm AI folder suggestion |
| POST | `/api/documents/{id}/suggestions/folder/reject` | Reject AI folder suggestion |
| POST | `/api/documents/{id}/suggestions/filename/confirm` | Confirm AI filename suggestion |
| POST | `/api/documents/{id}/suggestions/filename/reject` | Reject AI filename suggestion |
### Categories (`/api/documents/categories/*`) — authenticated, proxied to doc-service
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/documents/categories` | List user's categories |
| POST | `/api/documents/categories` | Create category (triggers background AI reanalysis) |
| PATCH | `/api/documents/categories/{id}` | Rename |
| DELETE | `/api/documents/categories/{id}` | Delete (204) |
### Plugins (`/api/plugins`) — authenticated, auth-per-plugin
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/plugins` | List plugins accessible to current user |
| GET | `/api/plugins/{id}/manifest` | Plugin manifest with settings JSON Schema (auth-gated) |
| GET | `/api/plugins/{id}/settings` | Proxy to feature `/plugin/settings` (auth-gated) |
| PATCH | `/api/plugins/{id}/settings` | Proxy to feature `/plugin/settings` (auth-gated) |
Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence.
### AI-service (internal only — not exposed to browser)
| Method | Path | Description |
|--------|------|-------------|
| POST | `/chat` | Chat request (queued at NORMAL priority) |
| GET | `/health` | Health check |
| GET | `/queue/status` | Queue state |
| POST | `/queue/pause` | Pause queue |
| POST | `/queue/resume` | Resume queue |
| POST | `/queue/cancel/{job_id}` | Cancel job |
---
## Frontend Routes
| Path | Component | Guard |
|------|-----------|-------|
| `/login` | `LoginPage` | Public |
| `/` | `DashboardPage` | PrivateRoute |
| `/apps` | `AppsPage` | PrivateRoute |
| `/apps/documents` | `DocumentsPage` | PrivateRoute |
| `/apps/documents/settings` | `DocServiceSettingsPage` | ServiceAdminRoute (is_admin OR doc-service-admin member) |
| `/apps/ai/settings` | `AIAdminSettingsPage` | ServiceAdminRoute (is_admin OR ai-service-admin member) |
| `/profile` | `ProfilePage` | PrivateRoute |
| `/settings` | `SettingsPage` | PrivateRoute |
| `/settings/plugins/:id` | `PluginSettingsPage` | PrivateRoute (auth enforced per-plugin by backend) |
| `/admin` | `AdminPage` (→ `/admin/users`) | AdminRoute |
| `/admin/users` | `AdminUsersPage` | AdminRoute |
| `/admin/groups` | `AdminGroupsPage` | AdminRoute |
| `/admin/appearance` | `AdminAppearancePage` | AdminRoute |
| `*` | redirect to `/` | — |
`PrivateRoute` — checks `token` from `useAuth`, redirects to `/login` if absent.
`AdminRoute` — checks token AND queries `GET /api/users/me` for `is_admin`; waits for query to avoid flash; redirects to `/login` (not `/`) if not admin.
---
## Security Standards
These standards are **non-negotiable**. Every change must comply.
### JWT
- **Algorithm**: RS256 (4096-bit RSA key pair, generated by `scripts/generate_jwt_keys.py`)
- **Keys**: PEM-encoded in `backend/.env` as `JWT_PRIVATE_KEY` / `JWT_PUBLIC_KEY` (gitignored)
- **Expiry**: 8 hours (`EXPIRE_MINUTES=480`) — never set longer; no refresh tokens
- **Claims**: `{sub: user_id, exp, iat}` — user_id is a UUID string
- **Validation**: `decode_access_token()` in `core/security.py`; called by `get_current_user`
- **Never**: set algorithm to `"none"`, disable `verify_exp`, or hardcode secrets in code
### Password hashing
- **Algorithm**: bcrypt, **13 rounds** (`bcrypt.gensalt(rounds=13)`)
- **Timing**: ~300 ms per hash (intentional brute-force resistance)
- **Never** use MD5, SHA1, or plain SHA256 for password storage
### Password policy (enforced in `UserCreate` schema)
All of the following must pass:
- ≥ 8 characters
- ≥ 1 uppercase (AZ)
- ≥ 1 lowercase (az)
- ≥ 1 digit (09)
- ≥ 1 special character: `!@#$%^&*()\-_=+[]{}|;:'"<>?/\`~`
- No common words (password, secret, login, admin, test, qwerty, welcome, …)
### Input sanitization
Every user-supplied string stored in the database **must** pass through `core/sanitize.py`:
```python
sanitize_str(value, max_len=255)
# → strips whitespace; rejects null bytes (\x00); rejects control chars
# (0x010x1F, 0x7F except \t \n \r); enforces max_len; returns None for ""
normalize_email(value) # lowercase + strip
validate_phone(value) # sanitize_str(max=20) + regex ^\+?[\d\s\-()\[\]]{7,20}$
validate_date_of_birth(v) # must be ≥ 1900, not future
```
Apply via Pydantic `@field_validator` on all request schemas.
### XSS prevention
- React JSX text interpolation (`{value}`) is HTML-escaped by the DOM renderer — **never** use `dangerouslySetInnerHTML` with user-supplied content.
- Server-side `sanitize_str` provides defense-in-depth (control char stripping, max length).
### SQL injection prevention
- Use SQLAlchemy ORM (bound parameters) — **never** raw SQL strings.
- If `text()` is needed, use `bindparam()` for all user-supplied values.
- **Never** use f-strings, `.format()`, or `%`-formatting for SQL.
### Admin route security
- Use `get_current_admin` dependency (checks `is_superuser`).
- Return **404** (not 403) for unauthorized access — hides both endpoint existence and permission model.
These standards are **non-negotiable**. Every change must comply. Implementation-specific security rules (JWT, bcrypt, input sanitization, XSS, SQLi, admin routes) are in the relevant sub-CLAUDE.md files.
### Network isolation
- `backend-net`: all containers except frontend; not reachable from host in prod.
- `frontend-net`: only frontend; single host port (80 prod / 5173 dev).
- DB, backend, doc-service, ai-service have **no** host port bindings in prod.
- DB, backend, doc-service, ai-service, storage-service have **no** host port bindings in prod.
### Storage rule (non-negotiable)
**No service may write to a filesystem path for persistent data.** All file/blob storage must go through the storage-service HTTP API (`PUT/GET/DELETE /objects/{bucket}/{key}`). Config JSON files must be stored in the `config` bucket. Uploaded files must be stored in the `documents` bucket. Violation is a security and architecture defect.
The only two persistent storage mechanisms in the project are:
1. **PostgreSQL** — structured/relational data
2. **storage-service** — all file/blob/config data (local filesystem by default; switchable to S3-compatible or WebDAV)
New services and features must follow this pattern. See `features/storage-service/CLAUDE.md` for the API reference.
### Pre-commit security hook
@@ -567,169 +173,14 @@ Apply via Pydantic `@field_validator` on all request schemas.
---
## Frontend Patterns & Conventions
### API client (`src/api/client.ts`)
Single Axios instance — **all** API calls live here, nowhere else:
```typescript
const api = axios.create({ baseURL: "/api" });
api.interceptors.request.use((config) => {
const token = localStorage.getItem("token");
if (token) config.headers.Authorization = `Bearer ${token}`;
return config;
});
```
Adding a new API call:
1. Define a TypeScript interface for the response if it's new.
2. Add a named export function (`getX`, `createX`, `updateX`, `deleteX`).
3. Use `api.get<T>(...)`, `api.post<T>(...)`, etc.; always `.then((r) => r.data)`.
### TanStack Query conventions
**Query keys** (flat arrays, lowercase):
```typescript
["me"] // current user
["services"] // service health list
["dashboard-prefs"] // user dashboard preferences
["categories"] // document categories
["documents", params] // document list (params object for cache isolation)
["document", id] // single document
["plugins"] // accessible plugin list (filtered by user access)
["plugin-manifest", id] // plugin manifest (cached)
["plugin-settings", id] // plugin current settings
```
**Mutation pattern**:
```typescript
const mutation = useMutation({
mutationFn: apiFunction,
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["affected-key"] });
// additional side effects (close dialog, reset form, etc.)
},
});
// Usage:
mutation.mutate(data);
mutation.isPending // show spinner / disable button
mutation.isError // show error message
```
**Polling**:
```typescript
useQuery({ queryKey: ["services"], queryFn: getServices,
refetchInterval: 30_000, refetchIntervalInBackground: true });
```
### Route guards
```typescript
// PrivateRoute — redirect to /login if no token
// AdminRoute — redirect to /login if no token OR not admin
// (waits for getMe() query to avoid flash; uses 404 semantics)
```
### Component patterns
- Functional components only.
- Local `useState` for UI-only state (edit mode, pending values, open/closed).
- Server state via `useQuery` / `useMutation` — no duplicated local copies.
- `cn()` from `lib/utils.ts` for conditional Tailwind classes.
- `lucide-react` for all icons.
- Never use `dangerouslySetInnerHTML` with user-supplied content.
---
## Naming & Code Conventions
### Database
- **Tables**: lowercase, plural, snake_case (`users`, `group_memberships`, `document_category_assignments`)
- **Columns**: lowercase, snake_case
- **ORM models**: PascalCase, singular (`User`, `Group`, `GroupMembership`, `Document`)
- Primary keys: `id` (String UUID, auto-generated)
- Timestamps: `created_at` / `updated_at` / `joined_at` / `processed_at` — always timezone-aware
### Pydantic schemas
| Suffix | Purpose |
|--------|---------|
| `Create` | POST request body (user-supplied input) |
| `Update` | PATCH request body (partial update) |
| `Out` | API response (safe subset of model) |
| `AdminOut` | Extended response for admin endpoints |
| `Read` | GET response (same as `Out`, used for profiles) |
Always set `model_config = {"from_attributes": True}` on response schemas.
Use `validation_alias` when the ORM field name differs from the JSON key (e.g., `is_superuser``is_admin`).
### HTTP status codes
| Code | Use |
|------|-----|
| 200 | Successful GET / PATCH / PUT |
| 201 | Successful POST that creates a resource |
| 202 | Accepted (async processing started, e.g., document upload) |
| 204 | Successful DELETE or action with no response body |
| 400 | Bad request (duplicates, invalid data beyond Pydantic) |
| 401 | Missing / invalid JWT |
| 404 | Not found **and** admin routes when not admin |
| 413 | Payload too large (file exceeds limit) |
| 415 | Unsupported media type (not a PDF) |
| 422 | Pydantic validation failure (FastAPI default) |
| 502 | Downstream service unreachable |
| 503 | Service unavailable (queue stopped, AI error) |
| 504 | Gateway timeout |
### Backend code style
- Async/await for **all** I/O (DB, HTTP, file).
- `raise HTTPException(status_code=..., detail="...")` for all errors.
- Response models always declared in route decorator: `@router.get("/path", response_model=XOut)`.
- Background tasks via `BackgroundTasks` param; tasks open their own `AsyncSessionLocal` session.
- Commit + refresh pattern after mutations:
```python
await db.commit()
await db.refresh(obj)
```
### Frontend code style
- TypeScript strict mode — no `any`.
- API response types inferred from interfaces in `client.ts` only.
- Error messages displayed inline (no alert); loading shown as disabled state or "…" text.
- All user-facing text: safe via React JSX rendering (not innerHTML).
---
## Default Values & Limits
## Default Values & Limits (cross-cutting)
| Parameter | Value | Location |
|-----------|-------|----------|
| JWT expiry | 480 min (8 h) | `core/security.py` |
| Bcrypt rounds | 13 | `core/security.py` |
| Token localStorage key | `"token"` | `useAuth.ts` |
| Health check interval | 30 s | `service_health.py` |
| Service poll (frontend) | 30 s | `AppsPage.tsx`, `DashboardPage.tsx` |
| User `color_mode` default | NULL (falls back to admin default_mode, then system) | `models/user.py` |
| Max dashboard pinned apps | 50 | `schemas/user.py` |
| App ID max length | 64 chars | `schemas/user.py` |
| App ID allowed chars | `[a-zA-Z0-9_\-]` | `schemas/user.py` |
| full_name max length | 128 chars | `schemas/user.py` |
| Group name max length | 128 chars | `schemas/group.py` |
| Group description max | 512 chars | `schemas/group.py` |
| Phone max length | 20 chars | `sanitize.py` |
| Position max length | 128 chars | `schemas/profile.py` |
| Address max length | 255 chars | `schemas/profile.py` |
| Document title max | 500 chars | `models/document.py` |
| Category name max | 128 chars | `models/category.py` |
| PDF max size (default) | 20 MB | admin settings (configurable) |
| Raw text cap | 500 k chars | `doc-service` AI client |
| Documents per_page | 1100, default 20 | `routers/documents.py` |
| AI service timeout | 60 s | `ai_client.py` |
| AI service max retries | 2 | `ai_client.py` |
All other per-service defaults are in the relevant sub-CLAUDE.md file.
---
@@ -740,9 +191,10 @@ Use `validation_alias` when the ORM field name differs from the JSON key (e.g.,
| Service | Image base | Internal port | User | Volumes | Network |
|---------|-----------|---------------|------|---------|---------|
| `db` | postgres:16-alpine | 5432 | 70:70 | `postgres_data` | backend-net |
| `backend` | python:3.12-slim | 8000 | 1001:1001 | `app_config` | backend-net |
| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | `app_config` | backend-net |
| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `doc_data`, `watch_data`, `app_config` | backend-net |
| `backend` | python:3.12-slim | 8000 | 1001:1001 | | backend-net |
| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | | backend-net |
| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `watch_data` | backend-net |
| `storage-service` | python:3.12-slim | 8020 | 1001:1001 | `storage_data` | backend-net |
| `frontend` | nginx-unprivileged:alpine | 8080 | 1001:1001 | — | backend-net, frontend-net |
### Volumes
@@ -750,15 +202,14 @@ Use `validation_alias` when the ORM field name differs from the JSON key (e.g.,
| Volume | Mount path | Contains |
|--------|-----------|---------|
| `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL data |
| `doc_data` | `/data/documents` | Uploaded PDF files |
| `storage_data` | `/data/storage` | All file/blob storage: PDFs (`documents/`) and config JSONs (`config/`) |
| `watch_data` | `/data/watch` | Watch directory (bind-mount NAS/Nextcloud via docker-compose.override.yml) |
| `app_config` | `/config` | Per-service runtime config JSON files |
### Networks
| Network | Host-accessible | Members |
|---------|----------------|---------|
| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, frontend |
| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, storage-service, frontend |
| `frontend-net` | Yes (port 80 → frontend:8080) | frontend |
### Environment variables (required in `backend/.env`)
@@ -774,6 +225,7 @@ Injected by docker-compose (not in `.env`):
```
DOC_SERVICE_URL=http://doc-service:8001
AI_SERVICE_URL=http://ai-service:8010
STORAGE_SERVICE_URL=http://storage-service:8020
```
---
@@ -840,8 +292,9 @@ Each entry must include:
4. Add router in `backend/app/routers/`, mount it in `main.py`
5. Add API function(s) to `frontend/src/api/client.ts`
6. Add page component in `frontend/src/pages/`, register route in `App.tsx`
7. Update `STATUS.md` for affected services
8. Add changelog entry
7. If the resource involves file or blob data: store it via `PUT /objects/{bucket}/{key}` on `storage-service:8020`. Never write to the local filesystem. See `features/storage-service/CLAUDE.md` for the API.
8. Update `STATUS.md` for affected services
9. Add changelog entry
---
@@ -855,31 +308,40 @@ Always run `git push` immediately after every `git commit`.
Every non-trivial implementation (anything beyond a one-line fix or doc change) **must** follow this workflow:
#### 0 — Mandatory planning phase (REQUIRED before any code changes)
Before touching any code, present a written plan and **wait for explicit user approval**. Do not open files to edit, do not create branches, do not write code until the user says the plan is approved.
The plan must include:
- **What** is changing and **why**
- **Which files** will be created or modified (with paths)
- **Database / migration impact** (if any)
- **API contract changes** (new endpoints, changed schemas)
- **Frontend route / component changes**
- **Risks or non-obvious decisions**
Only proceed to step 1 after the user responds with explicit approval (e.g. "looks good", "go ahead", "approved").
#### 1 — Create a feature branch
After the planning phase is approved, branch off `main`:
After the planning phase is approved, branch off `main`. Name the branch after the title of the change — use lowercase words separated by hyphens, descriptive enough to understand at a glance what the branch does:
```bash
git checkout main && git pull
git checkout -b feat/<slug> # e.g. feat/color-mode, feat/admin-appearance
git checkout -b feat/<descriptive-title> # e.g. feat/user-profile-avatar-upload, feat/document-bulk-delete
```
#### 2 — Spin up an isolated Docker stack for the feature
A dedicated compose stack runs alongside the main dev stack so both can be tested independently.
The feature stack always uses port `5173` (same as the main dev stack). Stop the main stack before starting a feature stack, and restart it when done.
**Find the next free port** (main dev stack owns 5173):
**Stop the main dev stack first:**
```bash
for port in $(seq 5174 5200); do
lsof -iTCP:$port -sTCP:LISTEN -t &>/dev/null || { echo "$port"; break; }
done
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
```
Use the first free port returned (call it `$PORT`).
**Create a per-feature override file** at `docker-compose.feat-<slug>.yml` (gitignored):
```yaml
# docker-compose.feat-<slug>.yml — feature test stack, never committed to main
services:
frontend:
ports:
- "$PORT:8080" # e.g. 5174:8080
container_name: frontend-<slug>
backend:
container_name: backend-<slug>
@@ -905,8 +367,7 @@ docker compose -f docker-compose.yml \
--project-name <slug> up --build
```
The feature frontend is now reachable at `http://localhost:$PORT`.
The main dev stack continues running unaffected on `:5173`.
The feature frontend is now reachable at `http://localhost:5173`.
#### 3 — Develop on the feature branch
All code changes happen on `feat/<slug>`. Commit and push normally:
@@ -917,7 +378,7 @@ git push -u origin feat/<slug>
```
#### 4 — Confirm functionality
Before merging, verify all of the following on `http://localhost:$PORT`:
Before merging, verify all of the following on `http://localhost:5173`:
- [ ] Login and registration work end-to-end
- [ ] The specific feature works as intended
- [ ] No regressions visible in the UI
@@ -934,13 +395,16 @@ git branch -d feat/<slug>
git push origin --delete feat/<slug>
```
#### 6 — Tear down the feature stack
#### 6 — Tear down the feature stack and restart main dev stack
```bash
docker compose -f docker-compose.yml \
-f docker-compose.dev.yml \
-f docker-compose.feat-<slug>.yml \
--project-name <slug> down --volumes --remove-orphans
rm docker-compose.feat-<slug>.yml
# Restart the main dev stack on :5173
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build -d
```
---
+9 -9
View File
@@ -20,10 +20,11 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
- All input sanitized before reaching the DB (null-byte rejection, length caps, format validation)
- **PDF Documents app** (`/apps/documents`): upload PDFs, async text extraction (pdfplumber), AI classification via ai-service, per-user categories, file download
- **AI Service** (`ai-service:8010`): shared AI intermediary container; routes prompts to Anthropic / Ollama / LM Studio; stateless; all feature containers talk to it via `POST /chat`
- Admin settings: `/apps/ai/settings/admin` (provider, credentials, test connection); `/apps/documents/settings/admin` (upload limits only)
- Config stored in shared Docker volume: `/config/ai_service_config.json` and `/config/doc_service_config.json`
- **Storage Service** (`storage-service:8020`): unified file/blob storage with pluggable backends (local filesystem default; S3-compatible and WebDAV built in); backend switchable via admin UI with zero-data-loss migration
- Admin settings: AI provider, doc upload limits, storage backend switching with live migration progress
- Config stored in storage-service (`config` bucket); PDFs stored in storage-service (`documents` bucket) — no shared filesystem volumes
- `/apps` launcher hub — one card per installed app with Open + Settings links
- 5 separate Docker containers: `db`, `backend`, `ai-service`, `doc-service`, `frontend`
- 6 separate Docker containers: `db`, `backend`, `ai-service`, `doc-service`, `storage-service`, `frontend`
- All containers run as non-root users (UID 1001 for app containers, UID 70 for db)
- Network-isolated: only the frontend exposes a host port (80/5173); all backend services are unreachable from outside Docker
- Dev environment seeds a test user automatically on startup (`test@example.com` / `Test123!`)
@@ -38,6 +39,7 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
| `backend` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | FastAPI management API + proxy to doc-service |
| `ai-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | Shared AI intermediary (routes to LM Studio / Ollama / Anthropic) |
| `doc-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | PDF extraction microservice (calls ai-service) |
| `storage-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | Unified file/blob storage (local / S3-compatible / WebDAV) |
| `frontend` | custom (nginxinc/nginx-unprivileged:alpine) | 80 (prod) / 5173 (dev) | backend-net + frontend-net | 1001:1001 | React UI + nginx reverse proxy |
**Networks:**
@@ -46,8 +48,8 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
**Volumes:**
- `postgres_data` — PostgreSQL data files
- `doc_data`uploaded PDF files (mounted into doc-service at `/data/documents`)
- `app_config` — per-service runtime config JSON files (mounted into backend, ai-service, and doc-service at `/config`)
- `storage_data`all file/blob storage: uploaded PDFs (`documents/` bucket) and service config JSON files (`config/` bucket); mounted into storage-service at `/data/storage`
- `watch_data` — file watcher input directory; mounted into doc-service at `/data/watch`
The frontend nginx proxies `/api/*` to `backend:8000` via `backend-net`. The backend proxies `/api/documents/*` and `/api/documents/categories/*` to `doc-service:8001`. The backend test-connection endpoint proxies to `ai-service:8010`. No backend service or database port is ever exposed to the host.
@@ -126,10 +128,8 @@ Copy `.env.example` to `backend/.env` and adjust:
| `JWT_PRIVATE_KEY` | — | RS256 private key PEM (generate with `scripts/generate_jwt_keys.py`) |
| `JWT_PUBLIC_KEY` | — | RS256 public key PEM (generate with `scripts/generate_jwt_keys.py`) |
| `CORS_ORIGINS` | `["http://localhost:5173"]` | Allowed frontend origins |
| `APP_CONFIG_DIR` | `/config` | Directory for per-service runtime config JSON files |
| `DOC_SERVICE_URL` | `http://doc-service:8001` | Internal URL of the doc-service (set by docker-compose) |
`doc-service` reads `DATABASE_URL`, `DATA_DIR`, and `CONFIG_PATH` from its own environment (set in `docker-compose.yml`).
| `DOC_SERVICE_URL` | `http://doc-service:8001` | Internal URL of doc-service (set by docker-compose) |
| `STORAGE_SERVICE_URL` | `http://storage-service:8020` | Internal URL of storage-service (set by docker-compose) |
## Development
+365
View File
@@ -0,0 +1,365 @@
# backend — Claude context
FastAPI async gateway, port 8000 (internal). Handles auth, user/group management, settings, and proxies document/category requests to `doc-service:8001`. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
---
## Commands
All commands run inside Docker — never on the host.
### Migrations
```bash
docker compose exec backend alembic revision --autogenerate -m "describe change"
docker compose exec backend alembic upgrade head
docker compose exec backend alembic downgrade -1
```
### Lint
```bash
docker compose exec backend ruff check . && ruff format .
```
---
## File & Folder Tree
```
backend/
├── app/
│ ├── main.py ← App factory, router registration, lifespan (health loop)
│ ├── database.py ← AsyncEngine, AsyncSessionLocal, Base
│ ├── deps.py ← get_current_user, get_current_admin, get_service_admin(id), check_plugin_access (also get_user_groups in doc-service)
│ ├── core/
│ │ ├── config.py ← All settings via pydantic-settings (reads .env)
│ │ ├── security.py ← JWT sign/verify (RS256), bcrypt hash/verify
│ │ ├── sanitize.py ← Input sanitization helpers (see Security Standards)
│ │ ├── app_config.py ← Per-service config load/save via storage-service; theme files in config/themes/{id}.json
│ │ └── config_storage.py ← Thin async HTTP helpers: read_json/write_json/delete_key/list_keys → storage-service config bucket
│ ├── models/
│ │ ├── __init__.py ← Imports all models (required for Alembic autogenerate)
│ │ ├── user.py ← User model
│ │ ├── profile.py ← Profile model
│ │ └── group.py ← Group, GroupMembership models
│ ├── schemas/
│ │ ├── user.py ← UserCreate/Out, Token, DashboardPrefsOut/Update
│ │ ├── profile.py ← ProfileRead, ProfileUpdate
│ │ └── group.py ← GroupCreate/Update/Out/DetailOut, GroupMemberOut
│ ├── routers/
│ │ ├── auth.py ← POST /register, POST /login
│ │ ├── users.py ← GET /me, GET+PATCH /me/preferences, PATCH /me/color-mode, GET /me/groups
│ │ ├── profile.py ← GET+PUT /me (profile)
│ │ ├── admin.py ← User admin CRUD (admin-only)
│ │ ├── groups.py ← Group CRUD + member management (admin-only)
│ │ ├── settings.py ← AI, doc limits, system prompts, appearance, themes (admin-only)
│ │ ├── services.py ← GET /services (health status)
│ │ ├── plugins.py ← Generic plugin proxy (GET/PATCH /api/plugins/*)
│ │ ├── categories_proxy.py ← Transparent proxy → doc-service /categories/*
│ │ ├── documents_proxy.py ← Transparent proxy → doc-service /documents/*
│ │ └── storage_config.py ← Admin proxy → storage-service config + migration endpoints
│ └── services/
│ ├── service_health.py ← Background 30s health-check loop; caches /plugin/manifest per service
│ └── group_bootstrap.py ← Ensures {service-id}-admin group exists for every registered service at startup
├── alembic/
│ ├── env.py ← Async migration runner
│ └── versions/ ← Migration chain (see Database Models)
├── scripts/seed.py ← Seed test user
├── Dockerfile ← python:3.12-slim, non-root user 1001
└── STATUS.md
```
---
## Database Models
### `users`
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | auto-generated |
| `email` | String | UNIQUE, indexed, NOT NULL | lowercased before storing |
| `hashed_password` | String | NOT NULL | bcrypt 13 rounds |
| `full_name` | String | nullable | sanitized max 128 chars |
| `is_active` | Boolean | default=True | soft-delete flag |
| `is_superuser` | Boolean | default=False | admin role; never exposed as-is (serialised as `is_admin`) |
| `dashboard_app_ids` | JSON | NOT NULL, default=[] | list of pinned service IDs |
| `color_mode` | String | nullable, default=NULL | user's preferred mode: "light" / "dark" / "system" / NULL (use admin default) |
Relationship: `profile` (one-to-one, cascade all+delete-orphan)
### `profiles`
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | auto-generated |
| `user_id` | String | FK→users.id UNIQUE, cascade delete | one-to-one |
| `phone` | String(20) | nullable | validated format |
| `date_of_birth` | Date | nullable | 1900+ and not future |
| `position` | String(128) | nullable | job title |
| `address` | String(255) | nullable | |
| `updated_at` | DateTime(tz) | server_default=now(), onupdate=now() | |
### `groups`
| Column | Type | Constraints |
|--------|------|-------------|
| `id` | String | PK, UUID |
| `name` | String(128) | UNIQUE indexed, NOT NULL |
| `description` | String(512) | nullable |
| `created_at` | DateTime(tz) | server_default=now() |
### `group_memberships`
| Column | Type | Constraints |
|--------|------|-------------|
| `id` | String | PK, UUID |
| `group_id` | String | FK→groups.id, indexed, CASCADE |
| `user_id` | String | FK→users.id, indexed, CASCADE |
| `is_group_admin` | Boolean | NOT NULL, default=false | grants group-admin rights (manage group categories, delete shared docs) |
| `joined_at` | DateTime(tz) | server_default=now() |
Unique constraint: `(group_id, user_id)`
### Migration chain (must be applied in order)
| Rev ID | Slug |
|--------|------|
| `38efeff7c45a` | `create_users_table` |
| `676084df61d1` | `add_profiles_table` |
| `a3f9c2d14e87` | `add_groups_and_group_memberships` |
| `c7e8f9a0b1d2` | `add_dashboard_app_ids_to_users` |
| `dd6ad2f2c211` | `add_color_mode_to_users` |
| `e1f2a3b4c5d6` | `add_group_member_is_admin` |
---
## API Endpoints
### Auth (`/api/auth`) — public
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/api/auth/register` | — | Create account; returns `UserOut`; enforces password policy |
| POST | `/api/auth/login` | — | OAuth2 password flow; returns `{access_token, token_type}` |
### Users (`/api/users`) — authenticated
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/users/me` | user | Current user info → `UserOut` |
| GET | `/api/users/me/preferences` | user | Dashboard pinned app IDs → `{app_ids}` |
| PATCH | `/api/users/me/preferences` | user | Save pinned app IDs (max 50, slug-safe) |
| PATCH | `/api/users/me/color-mode` | user | Save colour mode preference ("light"/"dark"/"system") |
| GET | `/api/users/me/groups` | user | Groups current user belongs to → `list[UserGroupOut]` |
### Profile (`/api/profile`) — authenticated
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/profile/me` | user | Fetch profile; auto-creates if missing |
| PUT | `/api/profile/me` | user | Update profile fields |
### Admin — Users (`/api/admin`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/users` | List all users → `list[UserAdminOut]` |
| POST | `/api/admin/users` | Create user (with optional is_admin) |
| DELETE | `/api/admin/users/{user_id}` | Delete user (204) |
| PATCH | `/api/admin/users/{user_id}/active` | Toggle active status |
### Admin — Groups (`/api/admin/groups`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/groups` | List groups with member count |
| POST | `/api/admin/groups` | Create group |
| GET | `/api/admin/groups/{id}` | Group detail + members |
| PATCH | `/api/admin/groups/{id}` | Update name / description |
| DELETE | `/api/admin/groups/{id}` | Delete (cascades memberships) |
| POST | `/api/admin/groups/{id}/members/{user_id}` | Add member |
| DELETE | `/api/admin/groups/{id}/members/{user_id}` | Remove member |
| PATCH | `/api/admin/groups/{id}/members/{user_id}/admin` | Set/unset group admin role (body: `{ is_group_admin: bool }`) |
### Settings (`/api/settings`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/settings/ai` | AI config (keys masked) |
| PATCH | `/api/settings/ai` | Update AI provider / credentials |
| POST | `/api/settings/ai/test` | Test AI connection |
| GET | `/api/settings/documents/limits` | PDF upload limits |
| PATCH | `/api/settings/documents/limits` | Update max PDF size |
| GET | `/api/settings/system-prompts` | All editable system prompts |
| PATCH | `/api/settings/system-prompts/{service_id}` | Update system prompt |
| GET | `/api/settings/appearance` | Active theme + default mode (auth) |
| PATCH | `/api/settings/appearance` | Update active theme + default mode (admin) |
| GET | `/api/settings/themes` | List all themes — built-in + custom (auth) |
| POST | `/api/settings/themes` | Create custom theme (admin) |
| PATCH | `/api/settings/themes/{id}` | Update custom theme label/colours (admin) |
| DELETE | `/api/settings/themes/{id}` | Delete custom theme (admin, 204) |
### Services (`/api/services`) — authenticated
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/services` | Health status of all registered services → `list[ServiceStatus]` |
### Plugins (`/api/plugins`) — authenticated, auth-per-plugin
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/plugins` | List plugins accessible to current user |
| GET | `/api/plugins/{id}/manifest` | Plugin manifest with settings JSON Schema (auth-gated) |
| GET | `/api/plugins/{id}/settings` | Proxy to feature `/plugin/settings` (auth-gated) |
| PATCH | `/api/plugins/{id}/settings` | Proxy to feature `/plugin/settings` (auth-gated) |
Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence.
### Admin — Storage (`/api/admin`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/storage-config` | Current backend driver + health → proxied from storage-service `/health` |
| PATCH | `/api/admin/storage-config` | Reconfigure backend without data migration (same-backend credential update) |
| POST | `/api/admin/storage-config/migrate` | Start async migration to a new backend (copy → verify → switch → cleanup) |
| GET | `/api/admin/storage-config/migrate/status` | Poll migration progress: `{state, total, done, failed, errors[]}` |
| DELETE | `/api/admin/storage-config/migrate` | Cancel a running migration; old backend remains active |
### Documents and Categories — proxied
`/api/documents/*` and `/api/documents/categories/*` are transparently proxied to `doc-service:8001`. The backend injects `x-user-id`, `x-user-groups`, and `x-user-is-admin` headers. See `features/doc-service/CLAUDE.md` for the internal endpoint list.
---
## Security Standards
These standards are **non-negotiable**. Every change must comply.
### JWT
- **Algorithm**: RS256 (4096-bit RSA key pair, generated by `scripts/generate_jwt_keys.py`)
- **Keys**: PEM-encoded in `backend/.env` as `JWT_PRIVATE_KEY` / `JWT_PUBLIC_KEY` (gitignored)
- **Expiry**: 8 hours (`EXPIRE_MINUTES=480`) — never set longer; no refresh tokens
- **Claims**: `{sub: user_id, exp, iat}` — user_id is a UUID string
- **Validation**: `decode_access_token()` in `core/security.py`; called by `get_current_user`
- **Never**: set algorithm to `"none"`, disable `verify_exp`, or hardcode secrets in code
### Password hashing
- **Algorithm**: bcrypt, **13 rounds** (`bcrypt.gensalt(rounds=13)`)
- **Timing**: ~300 ms per hash (intentional brute-force resistance)
- **Never** use MD5, SHA1, or plain SHA256 for password storage
### Password policy (enforced in `UserCreate` schema)
All of the following must pass:
- ≥ 8 characters
- ≥ 1 uppercase (AZ)
- ≥ 1 lowercase (az)
- ≥ 1 digit (09)
- ≥ 1 special character: `!@#$%^&*()\-_=+[]{}|;:'"<>?/\`~`
- No common words (password, secret, login, admin, test, qwerty, welcome, …)
### Input sanitization
Every user-supplied string stored in the database **must** pass through `core/sanitize.py`:
```python
sanitize_str(value, max_len=255)
# → strips whitespace; rejects null bytes (\x00); rejects control chars
# (0x010x1F, 0x7F except \t \n \r); enforces max_len; returns None for ""
normalize_email(value) # lowercase + strip
validate_phone(value) # sanitize_str(max=20) + regex ^\+?[\d\s\-()\[\]]{7,20}$
validate_date_of_birth(v) # must be ≥ 1900, not future
```
Apply via Pydantic `@field_validator` on all request schemas.
### SQL injection prevention
- Use SQLAlchemy ORM (bound parameters) — **never** raw SQL strings.
- If `text()` is needed, use `bindparam()` for all user-supplied values.
- **Never** use f-strings, `.format()`, or `%`-formatting for SQL.
### Admin route security
- Use `get_current_admin` dependency (checks `is_superuser`).
- Return **404** (not 403) for unauthorized access — hides both endpoint existence and permission model.
---
## Naming & Code Conventions
### Database
- **Tables**: lowercase, plural, snake_case (`users`, `group_memberships`, `document_category_assignments`)
- **Columns**: lowercase, snake_case
- **ORM models**: PascalCase, singular (`User`, `Group`, `GroupMembership`, `Document`)
- Primary keys: `id` (String UUID, auto-generated)
- Timestamps: `created_at` / `updated_at` / `joined_at` / `processed_at` — always timezone-aware
### Pydantic schemas
| Suffix | Purpose |
|--------|---------|
| `Create` | POST request body (user-supplied input) |
| `Update` | PATCH request body (partial update) |
| `Out` | API response (safe subset of model) |
| `AdminOut` | Extended response for admin endpoints |
| `Read` | GET response (same as `Out`, used for profiles) |
Always set `model_config = {"from_attributes": True}` on response schemas.
Use `validation_alias` when the ORM field name differs from the JSON key (e.g., `is_superuser``is_admin`).
### HTTP status codes
| Code | Use |
|------|-----|
| 200 | Successful GET / PATCH / PUT |
| 201 | Successful POST that creates a resource |
| 202 | Accepted (async processing started, e.g., document upload) |
| 204 | Successful DELETE or action with no response body |
| 400 | Bad request (duplicates, invalid data beyond Pydantic) |
| 401 | Missing / invalid JWT |
| 404 | Not found **and** admin routes when not admin |
| 413 | Payload too large (file exceeds limit) |
| 415 | Unsupported media type (not a PDF) |
| 422 | Pydantic validation failure (FastAPI default) |
| 502 | Downstream service unreachable |
| 503 | Service unavailable (queue stopped, AI error) |
| 504 | Gateway timeout |
### Backend code style
- Async/await for **all** I/O (DB, HTTP, file).
- `raise HTTPException(status_code=..., detail="...")` for all errors.
- Response models always declared in route decorator: `@router.get("/path", response_model=XOut)`.
- Background tasks via `BackgroundTasks` param; tasks open their own `AsyncSessionLocal` session.
- Commit + refresh pattern after mutations:
```python
await db.commit()
await db.refresh(obj)
```
---
## Default Values & Limits
| Parameter | Value | Location |
|-----------|-------|----------|
| JWT expiry | 480 min (8 h) | `core/security.py` |
| Bcrypt rounds | 13 | `core/security.py` |
| User `color_mode` default | NULL (falls back to admin default_mode, then system) | `models/user.py` |
| Max dashboard pinned apps | 50 | `schemas/user.py` |
| App ID max length | 64 chars | `schemas/user.py` |
| App ID allowed chars | `[a-zA-Z0-9_\-]` | `schemas/user.py` |
| full_name max length | 128 chars | `schemas/user.py` |
| Group name max length | 128 chars | `schemas/group.py` |
| Group description max | 512 chars | `schemas/group.py` |
| Phone max length | 20 chars | `sanitize.py` |
| Position max length | 128 chars | `schemas/profile.py` |
| Address max length | 255 chars | `schemas/profile.py` |
+17 -3
View File
@@ -27,6 +27,7 @@ JWT signing uses a 4096-bit RSA key pair (`RS256`). Keys are generated by `scrip
| `GET` | `/api/users/me` | Current user info |
| `GET` | `/api/users/me/preferences` | User's dashboard preferences (`app_ids` list) |
| `PATCH` | `/api/users/me/preferences` | Update pinned app IDs (max 50; validated as safe slugs) |
| `GET` | `/api/users/me/groups` | List groups the current user belongs to (for share picker) |
### Profile (`/api/profile`)
@@ -74,10 +75,20 @@ A background task (`service_health.py`) polls each service's `/health` endpoint
| `GET` | `/api/settings/system-prompts` | All editable system prompts — superuser OR `ai-service-admin` member |
| `PATCH` | `/api/settings/system-prompts/{id}` | Update system prompt — same access |
Settings are persisted to JSON files on the `app_config` Docker named volume and read by the respective feature services.
Settings are persisted to the `config` bucket of `storage-service:8020` via `core/config_storage.py`. All config I/O is async HTTP; no filesystem volumes are used.
Access to service-specific settings endpoints is enforced by `get_service_admin(service_id)` in `deps.py` — grants access to superusers OR members of the `{service_id}-admin` group.
### Storage config (`/api/admin`)
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/admin/storage-config` | Current backend driver + health (proxied from storage-service) |
| `PATCH` | `/api/admin/storage-config` | Reconfigure backend without migration |
| `POST` | `/api/admin/storage-config/migrate` | Start async migration to a new backend |
| `GET` | `/api/admin/storage-config/migrate/status` | Poll migration progress |
| `DELETE` | `/api/admin/storage-config/migrate` | Cancel running migration |
### Feature proxies
All `/api/documents/*` and `/api/documents/categories/*` requests are transparently proxied to `doc-service:8001` via `httpx.AsyncClient`. The proxy:
@@ -128,8 +139,11 @@ Browser (port 5173 dev / 80 prod)
┌───────────┼────────────┬──────────────┐
/auth /settings /documents/* /services
/users (JSON │ │
/admin volume) └── proxy → health-check loop
/profile doc-service:8001 (30s poll)
/admin /storage- └── proxy → health-check loop
/profile config doc-service:8001 (30s poll)
(proxy)
storage-service:8020
```
---
@@ -0,0 +1,32 @@
"""add is_group_admin to group_memberships
Revision ID: e1f2a3b4c5d6
Revises: dd6ad2f2c211
Create Date: 2026-04-18
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "e1f2a3b4c5d6"
down_revision: Union[str, None] = "dd6ad2f2c211"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"group_memberships",
sa.Column(
"is_group_admin",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
)
def downgrade() -> None:
op.drop_column("group_memberships", "is_group_admin")
+79 -93
View File
@@ -1,21 +1,25 @@
"""
Per-service runtime config helpers.
Config files live on the shared `app_config` Docker volume at /config/.
Each service has its own JSON file.
All config files are stored in the 'config' bucket of the storage-service.
Every function is async — callers must await them.
Atomic write pattern: write to .tmp in same dir, then os.replace() so
services never read a partial file.
Key layout in the config bucket:
ai_service_config.json
doc_service_config.json
appearance_config.json
themes/{id}.json
"""
import copy
import json
import os
import logging
import re
from pathlib import Path
from copy import deepcopy
from pydantic import BaseModel
_CONFIG_DIR = Path(os.environ.get("APP_CONFIG_DIR", "/config"))
from app.core import config_storage
logger = logging.getLogger(__name__)
# ── AI service config schemas ──────────────────────────────────────────────────
@@ -108,59 +112,50 @@ def _mask_ai_config(data: dict) -> dict:
# ── Load / Save ────────────────────────────────────────────────────────────────
def _config_path(service: str) -> Path:
return _CONFIG_DIR / f"{service}_config.json"
def load_service_config(service: str) -> dict:
path = _config_path(service)
if not path.exists():
async def load_service_config(service: str) -> dict:
data = await config_storage.read_json(f"{service}_config.json")
if data is None:
if service == "ai_service":
return AIServiceConfig().model_dump()
if service == "doc_service":
return DocServiceConfig().model_dump()
return {}
with path.open() as f:
return json.load(f)
return data
def save_service_config(service: str, data: dict) -> None:
path = _config_path(service)
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(data, indent=2))
os.replace(tmp, path)
async def save_service_config(service: str, data: dict) -> None:
await config_storage.write_json(f"{service}_config.json", data)
# AI service helpers
def load_ai_service_config() -> AIServiceConfig:
raw = load_service_config("ai_service")
async def load_ai_service_config() -> AIServiceConfig:
raw = await load_service_config("ai_service")
return AIServiceConfig.model_validate(raw)
def save_ai_service_config(config: AIServiceConfig) -> None:
save_service_config("ai_service", config.model_dump())
async def save_ai_service_config(config: AIServiceConfig) -> None:
await save_service_config("ai_service", config.model_dump())
def load_ai_service_config_masked() -> dict:
raw = load_service_config("ai_service")
async def load_ai_service_config_masked() -> dict:
raw = await load_service_config("ai_service")
return _mask_ai_config(raw)
# Doc service helpers
def load_doc_service_config() -> DocServiceConfig:
raw = load_service_config("doc_service")
async def load_doc_service_config() -> DocServiceConfig:
raw = await load_service_config("doc_service")
return DocServiceConfig.model_validate(raw)
def save_doc_service_config(config: DocServiceConfig) -> None:
save_service_config("doc_service", config.model_dump())
async def save_doc_service_config(config: DocServiceConfig) -> None:
await save_service_config("doc_service", config.model_dump())
def load_doc_service_config_masked() -> dict:
return load_service_config("doc_service")
async def load_doc_service_config_masked() -> dict:
return await load_service_config("doc_service")
def _merge_api_key(new_key: str, existing_key: str) -> str:
@@ -172,18 +167,16 @@ def _merge_api_key(new_key: str, existing_key: str) -> str:
# ── System prompts helpers ─────────────────────────────────────────────────────
# Registry of all services that have editable system prompts.
# key = service identifier, value = human-readable label
SYSTEM_PROMPT_SERVICES: dict[str, str] = {
"doc_service": "Document Service",
}
def load_all_system_prompts() -> dict:
async def load_all_system_prompts() -> dict:
"""Return {service_id: {label, system, user_template, default_system, default_user_template}}."""
result: dict = {}
for service_id, label in SYSTEM_PROMPT_SERVICES.items():
config = load_service_config(service_id)
config = await load_service_config(service_id)
prompts = config.get("system_prompts", {})
defaults = _get_service_prompt_defaults(service_id)
result[service_id] = {
@@ -196,15 +189,14 @@ def load_all_system_prompts() -> dict:
return result
def save_service_system_prompts(service_id: str, system: str, user_template: str) -> None:
"""Persist updated system prompts into the service's config file."""
async def save_service_system_prompts(service_id: str, system: str, user_template: str) -> None:
if service_id not in SYSTEM_PROMPT_SERVICES:
raise ValueError(f"Unknown service: {service_id!r}")
config = load_service_config(service_id)
config = await load_service_config(service_id)
config.setdefault("system_prompts", {})
config["system_prompts"]["system"] = system
config["system_prompts"]["user_template"] = user_template
save_service_config(service_id, config)
await save_service_config(service_id, config)
def _get_service_prompt_defaults(service_id: str) -> dict:
@@ -221,26 +213,19 @@ class AppearanceConfig(BaseModel):
default_mode: str = "system"
def load_appearance_config() -> AppearanceConfig:
path = _CONFIG_DIR / "appearance_config.json"
if not path.exists():
async def load_appearance_config() -> AppearanceConfig:
data = await config_storage.read_json("appearance_config.json")
if data is None:
return AppearanceConfig()
with path.open() as f:
return AppearanceConfig.model_validate(json.load(f))
return AppearanceConfig.model_validate(data)
def save_appearance_config(config: AppearanceConfig) -> None:
path = _CONFIG_DIR / "appearance_config.json"
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(config.model_dump(), indent=2))
os.replace(tmp, path)
async def save_appearance_config(config: AppearanceConfig) -> None:
await config_storage.write_json("appearance_config.json", config.model_dump())
# ── Theme file management ──────────────────────────────────────────────────────
_THEMES_DIR = _CONFIG_DIR / "themes"
# 9 required colour tokens per mode
_REQUIRED_TOKENS = frozenset({
"primary", "primary_hover", "accent", "accent_hover",
@@ -361,36 +346,57 @@ _BUILTIN_THEMES: list[dict] = [
]
def seed_builtin_themes() -> None:
"""Create /config/themes/ and write built-in theme files if missing."""
_THEMES_DIR.mkdir(parents=True, exist_ok=True)
async def seed_builtin_themes() -> None:
"""Write built-in theme files to storage-service if they are not already there."""
existing_keys = await config_storage.list_keys(prefix="themes/")
existing_ids = {k.removeprefix("themes/").removesuffix(".json") for k in existing_keys}
for theme in _BUILTIN_THEMES:
path = _THEMES_DIR / f"{theme['id']}.json"
if not path.exists():
path.write_text(json.dumps(theme, indent=2))
if theme["id"] not in existing_ids:
await config_storage.write_json(f"themes/{theme['id']}.json", theme)
logger.info("Built-in themes seeded (%d themes)", len(_BUILTIN_THEMES))
def load_all_themes() -> list[dict]:
"""Return all themes from /config/themes/*.json, built-ins first."""
if not _THEMES_DIR.exists():
seed_builtin_themes()
themes = []
for f in sorted(_THEMES_DIR.glob("*.json")):
try:
themes.append(json.loads(f.read_text()))
except (json.JSONDecodeError, OSError):
pass
# Sort: built-ins first (preserving their original order), then custom by label
async def load_all_themes() -> list[dict]:
"""Return all themes from storage-service, built-ins first then custom by label."""
keys = await config_storage.list_keys(prefix="themes/")
themes: list[dict] = []
for key in keys:
data = await config_storage.read_json(key)
if data:
themes.append(data)
builtin_ids = [t["id"] for t in _BUILTIN_THEMES]
def sort_key(t: dict) -> tuple:
tid = t.get("id", "")
try:
return (0, builtin_ids.index(tid))
except ValueError:
return (1, t.get("label", tid).lower())
return sorted(themes, key=sort_key)
async def load_theme_by_id(theme_id: str) -> dict | None:
"""Return a single theme dict, or None if not found."""
return await config_storage.read_json(f"themes/{theme_id}.json")
async def save_theme(theme: dict) -> None:
"""Write a theme to storage-service."""
await config_storage.write_json(f"themes/{theme['id']}.json", theme)
async def delete_theme(theme_id: str) -> None:
"""Delete a custom theme. Raises ValueError for built-ins, KeyError if not found."""
data = await config_storage.read_json(f"themes/{theme_id}.json")
if data is None:
raise FileNotFoundError(theme_id)
if data.get("builtin"):
raise ValueError("Cannot delete a built-in theme")
await config_storage.delete_key(f"themes/{theme_id}.json")
def validate_theme_tokens(colors: dict) -> list[str]:
"""Return a list of validation error messages, empty if valid."""
errors = []
@@ -401,23 +407,3 @@ def validate_theme_tokens(colors: dict) -> list[str]:
if key in _REQUIRED_TOKENS and not _RGB_RE.match(str(val)):
errors.append(f"Token '{key}' must be an RGB triplet like '37 99 235', got: {val!r}")
return errors
def save_theme(theme: dict) -> None:
"""Write a theme file atomically."""
_THEMES_DIR.mkdir(parents=True, exist_ok=True)
path = _THEMES_DIR / f"{theme['id']}.json"
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(theme, indent=2))
os.replace(tmp, path)
def delete_theme(theme_id: str) -> None:
"""Delete a custom theme file. Raises ValueError for built-ins, FileNotFoundError if missing."""
path = _THEMES_DIR / f"{theme_id}.json"
if not path.exists():
raise FileNotFoundError(theme_id)
data = json.loads(path.read_text())
if data.get("builtin"):
raise ValueError("Cannot delete a built-in theme")
path.unlink()
+1
View File
@@ -17,6 +17,7 @@ class Settings(BaseSettings):
DOC_SERVICE_URL: str = "http://doc-service:8001"
AI_SERVICE_URL: str = "http://ai-service:8010"
STORAGE_SERVICE_URL: str = "http://storage-service:8020"
@field_validator("JWT_PRIVATE_KEY", "JWT_PUBLIC_KEY", mode="before")
@classmethod
+63
View File
@@ -0,0 +1,63 @@
"""
Async HTTP client for the 'config' bucket in storage-service.
All JSON config files (AI settings, doc settings, appearance, themes, …) are stored
in the 'config' bucket under the storage-service. This module provides thin
async helpers so app_config.py does not depend on the filesystem at all.
"""
import json
import logging
import httpx
from app.core.config import settings
logger = logging.getLogger(__name__)
_BUCKET = "config"
_TIMEOUT = 10.0
def _url(key: str) -> str:
return f"{settings.STORAGE_SERVICE_URL}/objects/{_BUCKET}/{key}"
async def read_json(key: str) -> dict | None:
"""Return parsed JSON from the config bucket, or None if the key does not exist."""
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.get(_url(key))
if resp.status_code == 404:
return None
resp.raise_for_status()
return resp.json()
async def write_json(key: str, data: dict) -> None:
"""Serialise *data* to JSON and PUT it into the config bucket."""
payload = json.dumps(data, indent=2).encode()
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.put(
_url(key),
content=payload,
headers={"Content-Type": "application/octet-stream"},
)
resp.raise_for_status()
async def delete_key(key: str) -> None:
"""Delete a key from the config bucket. No-op if it does not exist."""
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.delete(_url(key))
if resp.status_code not in (204, 404):
resp.raise_for_status()
async def list_keys(prefix: str = "") -> list[str]:
"""List all keys in the config bucket, optionally filtered by *prefix*."""
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.get(f"{settings.STORAGE_SERVICE_URL}/objects/{_BUCKET}")
resp.raise_for_status()
keys: list[str] = resp.json().get("keys", [])
if prefix:
keys = [k for k in keys if k.startswith(prefix)]
return keys
+4 -1
View File
@@ -9,16 +9,18 @@ from app.core.config import settings
from app.database import AsyncSessionLocal
from app.routers import admin, auth, categories_proxy, documents_proxy, groups, plugins, profile, services, users
from app.routers import settings as settings_router
from app.routers import storage_config
from app.services.group_bootstrap import ensure_service_admin_groups
from app.services.service_health import check_all, health_check_loop, register_services
@asynccontextmanager
async def lifespan(app: FastAPI):
await asyncio.to_thread(seed_builtin_themes)
await seed_builtin_themes()
register_services(
doc_service_url=settings.DOC_SERVICE_URL,
ai_service_url=settings.AI_SERVICE_URL,
storage_service_url=settings.STORAGE_SERVICE_URL,
)
# Create <service-id>-admin groups for every registered service (idempotent)
async with AsyncSessionLocal() as db:
@@ -51,6 +53,7 @@ app.include_router(admin.router, prefix="/api/admin", tags=["admin"])
app.include_router(groups.router, prefix="/api/admin/groups", tags=["admin"])
app.include_router(settings_router.router, prefix="/api/settings", tags=["settings"])
app.include_router(services.router, prefix="/api/services", tags=["services"])
app.include_router(storage_config.router, prefix="/api/admin", tags=["admin"])
app.include_router(plugins.router, prefix="/api/plugins", tags=["plugins"])
# categories_proxy MUST be registered before documents_proxy —
# otherwise /api/documents/{path:path} swallows /api/documents/categories/*
+4 -1
View File
@@ -1,7 +1,7 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import DateTime, ForeignKey, String, UniqueConstraint
from sqlalchemy import Boolean, DateTime, ForeignKey, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.database import Base
@@ -35,6 +35,9 @@ class GroupMembership(Base):
user_id: Mapped[str] = mapped_column(
String, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
)
is_group_admin: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False, server_default="false"
)
joined_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=lambda: datetime.now(timezone.utc),
+20 -2
View File
@@ -9,8 +9,12 @@ import os
import httpx
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.responses import Response
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.deps import get_current_user
from app.models.group import GroupMembership
from app.models.user import User
DOC_SERVICE_URL = os.environ.get("DOC_SERVICE_URL", "http://doc-service:8001")
@@ -35,13 +39,26 @@ _HOP_BY_HOP = frozenset([
_STRIP_RESPONSE = frozenset([*_HOP_BY_HOP, "content-length", "content-type"])
def _forward_headers(request: Request, user_id: str) -> dict:
async def _forward_headers(
request: Request, user_id: str, is_admin: bool, db: AsyncSession
) -> dict:
headers = {
k: v
for k, v in request.headers.items()
if k.lower() not in _HOP_BY_HOP
}
headers["x-user-id"] = user_id
headers["x-user-is-admin"] = "true" if is_admin else "false"
mem_result = await db.execute(
select(GroupMembership.group_id, GroupMembership.is_group_admin)
.where(GroupMembership.user_id == user_id)
)
rows = mem_result.all()
group_ids = [row[0] for row in rows]
admin_group_ids = [row[0] for row in rows if row[1]]
headers["x-user-groups"] = ",".join(group_ids)
headers["x-user-admin-groups"] = ",".join(admin_group_ids)
return headers
@@ -50,10 +67,11 @@ def _forward_headers(request: Request, user_id: str) -> dict:
async def proxy_categories(
request: Request,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
path: str = "",
) -> Response:
url = f"/categories/{path}" if path else "/categories"
headers = _forward_headers(request, str(current_user.id))
headers = await _forward_headers(request, str(current_user.id), current_user.is_superuser, db)
body = await request.body()
try:
+26 -2
View File
@@ -3,14 +3,21 @@ Proxy all /api/documents/* requests to doc-service:8001/documents/*.
Uses a module-level AsyncClient for connection pooling.
Strips hop-by-hop headers that must not be forwarded.
Injects X-User-Id and X-User-Groups headers so the doc-service
can enforce ownership and group-sharing access without querying the
backend database directly.
"""
import os
import httpx
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.responses import Response
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.deps import get_current_user
from app.models.group import GroupMembership
from app.models.user import User
DOC_SERVICE_URL = os.environ.get("DOC_SERVICE_URL", "http://doc-service:8001")
@@ -43,13 +50,29 @@ _HOP_BY_HOP = frozenset([
_STRIP_RESPONSE = frozenset([*_HOP_BY_HOP, "content-length", "content-type"])
def _forward_headers(request: Request, user_id: str) -> dict:
async def _forward_headers(
request: Request, user_id: str, is_admin: bool, db: AsyncSession
) -> dict:
headers = {
k: v
for k, v in request.headers.items()
if k.lower() not in _HOP_BY_HOP
}
headers["x-user-id"] = user_id
headers["x-user-is-admin"] = "true" if is_admin else "false"
# Inject group memberships and group-admin status so the doc-service can
# evaluate ownership, sharing access, and group-admin permissions.
mem_result = await db.execute(
select(GroupMembership.group_id, GroupMembership.is_group_admin)
.where(GroupMembership.user_id == user_id)
)
rows = mem_result.all()
group_ids = [row[0] for row in rows]
admin_group_ids = [row[0] for row in rows if row[1]]
headers["x-user-groups"] = ",".join(group_ids)
headers["x-user-admin-groups"] = ",".join(admin_group_ids)
return headers
@@ -58,10 +81,11 @@ def _forward_headers(request: Request, user_id: str) -> dict:
async def proxy_documents(
request: Request,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
path: str = "",
) -> Response:
url = f"/documents/{path}" if path else "/documents"
headers = _forward_headers(request, str(current_user.id))
headers = await _forward_headers(request, str(current_user.id), current_user.is_superuser, db)
body = await request.body()
try:
+22 -1
View File
@@ -7,7 +7,7 @@ from app.database import get_db
from app.deps import get_current_admin
from app.models.group import Group, GroupMembership
from app.models.user import User
from app.schemas.group import GroupCreate, GroupDetailOut, GroupOut, GroupUpdate, GroupMemberOut
from app.schemas.group import GroupCreate, GroupDetailOut, GroupMemberAdminUpdate, GroupMemberOut, GroupOut, GroupUpdate
router = APIRouter()
@@ -111,6 +111,7 @@ async def get_group(
email=user.email,
full_name=user.full_name,
is_active=user.is_active,
is_group_admin=membership.is_group_admin,
joined_at=membership.joined_at,
)
for membership, user in rows
@@ -197,6 +198,26 @@ async def add_member(
await db.commit()
@router.patch("/{group_id}/members/{user_id}/admin", status_code=status.HTTP_204_NO_CONTENT)
async def set_member_admin(
group_id: str,
user_id: str,
body: GroupMemberAdminUpdate,
_admin: User = Depends(get_current_admin),
db: AsyncSession = Depends(get_db),
) -> None:
result = await db.execute(
select(GroupMembership).where(
GroupMembership.group_id == group_id, GroupMembership.user_id == user_id
)
)
membership = result.scalar_one_or_none()
if not membership:
raise HTTPException(status_code=404, detail="User is not a member of this group")
membership.is_group_admin = body.is_group_admin
await db.commit()
@router.delete("/{group_id}/members/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
async def remove_member(
group_id: str,
+27 -34
View File
@@ -2,10 +2,9 @@
Admin-only settings API for per-service runtime configuration.
All endpoints require the caller to be an admin (Depends(get_current_admin)).
Config files live on the shared app_config volume (/config/).
Config files are stored in the 'config' bucket of storage-service.
"""
import asyncio
import json
import re as _re
import httpx
from fastapi import APIRouter, Depends, HTTPException
@@ -21,10 +20,11 @@ from app.core.app_config import (
load_all_system_prompts,
load_all_themes,
load_appearance_config,
save_appearance_config,
load_doc_service_config,
load_doc_service_config_masked,
load_theme_by_id,
save_ai_service_config,
save_appearance_config,
save_doc_service_config,
save_service_system_prompts,
save_theme,
@@ -36,6 +36,8 @@ from app.models.user import User
router = APIRouter()
_THEME_ID_RE = _re.compile(r"^[a-z0-9_-]{1,64}$")
# ── Pydantic request bodies ────────────────────────────────────────────────────
@@ -98,7 +100,7 @@ class ThemeUpdate(BaseModel):
async def get_ai_settings(
_: User = Depends(get_service_admin("ai-service")),
) -> dict:
return load_ai_service_config_masked()
return await load_ai_service_config_masked()
@router.patch("/ai")
@@ -110,7 +112,7 @@ async def update_ai_settings(
if body.provider not in valid_providers:
raise HTTPException(status_code=422, detail=f"provider must be one of {valid_providers}")
config = load_ai_service_config()
config = await load_ai_service_config()
config.provider = body.provider
# Anthropic
@@ -139,8 +141,8 @@ async def update_ai_settings(
body.lmstudio_api_key, config.lmstudio.api_key
)
await asyncio.to_thread(save_ai_service_config, config)
return load_ai_service_config_masked()
await save_ai_service_config(config)
return await load_ai_service_config_masked()
@router.post("/ai/test")
@@ -173,7 +175,7 @@ async def test_ai_connection(
async def get_documents_limits(
_: User = Depends(get_service_admin("doc-service")),
) -> dict:
return load_doc_service_config_masked()
return await load_doc_service_config_masked()
@router.patch("/documents/limits")
@@ -184,10 +186,10 @@ async def update_documents_limits(
if body.max_pdf_mb < 1 or body.max_pdf_mb > 200:
raise HTTPException(status_code=422, detail="max_pdf_mb must be between 1 and 200")
config = load_doc_service_config()
config = await load_doc_service_config()
config.documents.max_pdf_bytes = body.max_pdf_mb * 1024 * 1024
await asyncio.to_thread(save_doc_service_config, config)
return load_doc_service_config_masked()
await save_doc_service_config(config)
return await load_doc_service_config_masked()
# ── System prompts ─────────────────────────────────────────────────────────────
@@ -197,8 +199,7 @@ async def update_documents_limits(
async def get_system_prompts(
_: User = Depends(get_service_admin("ai-service")),
) -> dict:
"""Return all editable system prompts, keyed by service id."""
return await asyncio.to_thread(load_all_system_prompts)
return await load_all_system_prompts()
@router.patch("/system-prompts/{service_id}")
@@ -207,26 +208,20 @@ async def update_system_prompt(
body: SystemPromptUpdate,
_: User = Depends(get_service_admin("ai-service")),
) -> dict:
"""Update the system prompts for a single service."""
if service_id not in SYSTEM_PROMPT_SERVICES:
raise HTTPException(status_code=404, detail=f"No system prompts registered for {service_id!r}")
await asyncio.to_thread(
save_service_system_prompts, service_id, body.system, body.user_template
)
return await asyncio.to_thread(load_all_system_prompts)
await save_service_system_prompts(service_id, body.system, body.user_template)
return await load_all_system_prompts()
# ── Appearance (global default — auth read, admin write) ───────────────────────
import re as _re
_THEME_ID_RE = _re.compile(r"^[a-z0-9_-]{1,64}$")
@router.get("/appearance")
async def get_appearance(
_: User = Depends(get_current_user),
) -> dict:
config = await asyncio.to_thread(load_appearance_config)
config = await load_appearance_config()
return config.model_dump()
@@ -237,12 +232,12 @@ async def update_appearance(
) -> dict:
if body.default_mode not in ("light", "dark", "system"):
raise HTTPException(status_code=422, detail="default_mode must be 'light', 'dark', or 'system'")
themes = await asyncio.to_thread(load_all_themes)
themes = await load_all_themes()
theme_ids = {t["id"] for t in themes}
if body.theme not in theme_ids:
raise HTTPException(status_code=422, detail=f"Unknown theme: {body.theme!r}")
config = AppearanceConfig(theme=body.theme, default_mode=body.default_mode)
await asyncio.to_thread(save_appearance_config, config)
await save_appearance_config(config)
return config.model_dump()
@@ -253,7 +248,7 @@ async def update_appearance(
async def list_themes(
_: User = Depends(get_current_user),
) -> list:
return await asyncio.to_thread(load_all_themes)
return await load_all_themes()
@router.post("/themes", status_code=201)
@@ -263,7 +258,7 @@ async def create_theme(
) -> dict:
if not _THEME_ID_RE.match(body.id):
raise HTTPException(status_code=422, detail="Theme ID must match [a-z0-9_-]{1,64}")
existing = {t["id"] for t in await asyncio.to_thread(load_all_themes)}
existing = {t["id"] for t in await load_all_themes()}
if body.id in existing:
raise HTTPException(status_code=400, detail=f"Theme ID already in use: {body.id!r}")
light = body.light.model_dump()
@@ -273,7 +268,7 @@ async def create_theme(
if errors:
raise HTTPException(status_code=422, detail=f"{mode}: {'; '.join(errors)}")
theme = {"id": body.id, "label": body.label, "builtin": False, "light": light, "dark": dark}
await asyncio.to_thread(save_theme, theme)
await save_theme(theme)
return theme
@@ -283,11 +278,9 @@ async def update_theme(
body: ThemeUpdate,
_: User = Depends(get_current_admin),
) -> dict:
from app.core.app_config import _THEMES_DIR
path = _THEMES_DIR / f"{theme_id}.json"
if not path.exists():
theme = await load_theme_by_id(theme_id)
if theme is None:
raise HTTPException(status_code=404, detail="Theme not found")
theme = json.loads(path.read_text())
if theme.get("builtin"):
raise HTTPException(status_code=400, detail="Cannot edit a built-in theme")
if body.label is not None:
@@ -304,7 +297,7 @@ async def update_theme(
if errors:
raise HTTPException(status_code=422, detail=f"dark: {'; '.join(errors)}")
theme["dark"] = dark
await asyncio.to_thread(save_theme, theme)
await save_theme(theme)
return theme
@@ -314,7 +307,7 @@ async def remove_theme(
_: User = Depends(get_current_admin),
) -> None:
try:
await asyncio.to_thread(delete_theme, theme_id)
await delete_theme(theme_id)
except FileNotFoundError:
raise HTTPException(status_code=404, detail="Theme not found")
except ValueError as exc:
+126
View File
@@ -0,0 +1,126 @@
"""
Admin-only endpoints for storage-service backend configuration.
GET /admin/storage-config — current backend driver + health
PATCH /admin/storage-config — update backend config (no data migration)
POST /admin/storage-config/migrate — start migration to a new backend
GET /admin/storage-config/migrate/status — poll migration progress
DELETE /admin/storage-config/migrate — cancel in-progress migration
All endpoints proxy to storage-service:8020.
"""
import logging
import httpx
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from app.core.config import settings
from app.deps import get_current_admin
from app.models.user import User
router = APIRouter()
logger = logging.getLogger(__name__)
_STORAGE_BASE = settings.STORAGE_SERVICE_URL
class BackendConfigUpdate(BaseModel):
driver: str
config: dict = {}
class MigrateRequest(BaseModel):
driver: str
config: dict = {}
def _storage_url(path: str) -> str:
return f"{_STORAGE_BASE}{path}"
async def _proxy_get(path: str) -> dict:
async with httpx.AsyncClient(timeout=15.0) as client:
resp = await client.get(_storage_url(path))
if resp.status_code == 404:
raise HTTPException(status_code=404, detail="Not found")
resp.raise_for_status()
return resp.json()
@router.get("/storage-config")
async def get_storage_config(
_: User = Depends(get_current_admin),
) -> dict:
"""Return current backend driver and health status."""
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.get(_storage_url("/health"))
resp.raise_for_status()
return resp.json()
@router.patch("/storage-config", status_code=204)
async def update_storage_config(
body: BackendConfigUpdate,
_: User = Depends(get_current_admin),
) -> None:
"""
Reconfigure the active backend without migrating data.
Use when changing credentials for the same backend type, or reverting to local.
To move data to a new backend, use POST /admin/storage-config/migrate instead.
"""
async with httpx.AsyncClient(timeout=15.0) as client:
resp = await client.patch(
_storage_url("/backend-config"),
json={"driver": body.driver, "config": body.config},
)
if resp.status_code == 400:
raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed"))
if resp.status_code == 409:
raise HTTPException(status_code=409, detail="Migration in progress — cannot reconfigure now")
resp.raise_for_status()
@router.post("/storage-config/migrate", status_code=202)
async def start_migration(
body: MigrateRequest,
_: User = Depends(get_current_admin),
) -> dict:
"""
Start an async migration to a new backend.
Flow: validate new backend → copy all objects → verify → switch → delete old objects.
The old backend stays active until 100% of objects are verified on the new one.
Poll GET /admin/storage-config/migrate/status to track progress.
"""
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.post(
_storage_url("/migrate"),
json={"driver": body.driver, "config": body.config},
)
if resp.status_code == 400:
raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed"))
if resp.status_code == 409:
raise HTTPException(status_code=409, detail="A migration is already in progress")
resp.raise_for_status()
return resp.json()
@router.get("/storage-config/migrate/status")
async def migration_status(
_: User = Depends(get_current_admin),
) -> dict:
"""Poll migration progress. State: idle → validating → migrating → switching → cleaning → done."""
return await _proxy_get("/migrate/status")
@router.delete("/storage-config/migrate", status_code=204)
async def cancel_migration(
_: User = Depends(get_current_admin),
) -> None:
"""Cancel a running migration. The old backend remains active."""
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.delete(_storage_url("/migrate"))
if resp.status_code == 409:
raise HTTPException(status_code=409, detail="No cancellable migration in progress")
resp.raise_for_status()
+22 -1
View File
@@ -1,10 +1,13 @@
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.database import get_db
from app.deps import get_current_user
from app.models.group import Group, GroupMembership
from app.models.user import User
from app.schemas.user import ColorModeUpdate, DashboardPrefsOut, DashboardPrefsUpdate, UserOut
from app.schemas.user import ColorModeUpdate, DashboardPrefsOut, DashboardPrefsUpdate, UserGroupOut, UserOut
router = APIRouter()
@@ -31,6 +34,24 @@ async def update_preferences(
return DashboardPrefsOut(app_ids=current_user.dashboard_app_ids or [])
@router.get("/me/groups", response_model=list[UserGroupOut])
async def get_my_groups(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Return all groups the current user belongs to, including their admin status."""
result = await db.execute(
select(Group, GroupMembership.is_group_admin)
.join(GroupMembership, GroupMembership.group_id == Group.id)
.where(GroupMembership.user_id == current_user.id)
.order_by(Group.name)
)
return [
UserGroupOut(id=g.id, name=g.name, description=g.description, is_group_admin=is_admin)
for g, is_admin in result.all()
]
@router.patch("/me/color-mode", response_model=UserOut)
async def update_color_mode(
body: ColorModeUpdate,
+5
View File
@@ -18,11 +18,16 @@ class GroupMemberOut(BaseModel):
email: str
full_name: str | None
is_active: bool
is_group_admin: bool = False
joined_at: datetime
model_config = {"from_attributes": True}
class GroupMemberAdminUpdate(BaseModel):
is_group_admin: bool
class GroupOut(BaseModel):
id: str
name: str
+11
View File
@@ -1,4 +1,5 @@
import re
from datetime import datetime
from pydantic import BaseModel, EmailStr, Field, field_validator
@@ -116,6 +117,16 @@ class ColorModeUpdate(BaseModel):
return v
class UserGroupOut(BaseModel):
"""A group the current user belongs to — used for the share picker."""
id: str
name: str
description: str | None
is_group_admin: bool = False
model_config = {"from_attributes": True}
class DashboardPrefsUpdate(BaseModel):
app_ids: list[str] = Field(default_factory=list)
+10 -1
View File
@@ -40,7 +40,7 @@ _health: dict[str, bool | None] = {}
_manifests: dict[str, dict | None] = {}
def register_services(doc_service_url: str, ai_service_url: str) -> None:
def register_services(doc_service_url: str, ai_service_url: str, storage_service_url: str) -> None:
"""Called once during app startup to populate the registry from config."""
global _REGISTRY, _health, _manifests
@@ -63,6 +63,15 @@ def register_services(doc_service_url: str, ai_service_url: str) -> None:
app_path="",
settings_path="/apps/ai/settings",
),
ServiceDefinition(
id="storage-service",
name="Storage",
description="Unified file storage. Manages all uploaded files with pluggable backends (local, S3, WebDAV).",
internal_url=storage_service_url,
health_path="/health",
app_path="",
settings_path="/admin/storage",
),
]
_health = {svc.id: None for svc in _REGISTRY}
@@ -0,0 +1,34 @@
# 2026-04-18 — Category scopes, group admin role, and permission model
**Timestamp:** 2026-04-18T00:00:00Z
## Summary
Introduces three category scopes (personal / group / system), a PascalCase-with-dashes naming convention, a group-admin role on group memberships, and a full permission model for who can create, rename, and delete categories and documents.
## Files Added
- `backend/alembic/versions/e1f2a3b4c5d6_add_group_member_is_admin.py` — adds `is_group_admin BOOLEAN` to `group_memberships`
- `features/doc-service/alembic/versions/0005_add_share_can_delete.py` — adds `can_delete BOOLEAN` to `document_shares` (backfill from feat/document-delete-permissions)
- `features/doc-service/alembic/versions/0006_add_category_scope.py` — adds `scope VARCHAR(16)` and `group_id VARCHAR` to `document_categories`; data-migrates watch categories to scope='system'
## Files Modified
- `backend/app/models/group.py` — added `is_group_admin` to `GroupMembership`
- `backend/app/schemas/group.py` — added `is_group_admin` to `GroupMemberOut`; new `GroupMemberAdminUpdate`
- `backend/app/schemas/user.py` — added `is_group_admin` to `UserGroupOut`
- `backend/app/routers/users.py``get_my_groups` now joins `GroupMembership` to include `is_group_admin`
- `backend/app/routers/groups.py``get_group` includes `is_group_admin`; new `PATCH /{id}/members/{user_id}/admin` endpoint
- `backend/app/routers/categories_proxy.py` — injects `x-user-is-admin` and `x-user-admin-groups` headers
- `backend/app/routers/documents_proxy.py` — injects `x-user-admin-groups` header (was already injecting `x-user-is-admin`)
- `features/doc-service/app/models/category.py` — added `scope`, `group_id` columns
- `features/doc-service/app/schemas/category.py``CategoryOut` includes `scope`/`group_id`; `CategoryCreate` accepts `group_id`
- `features/doc-service/app/deps.py` — added `get_user_is_admin`, `get_user_admin_groups`
- `features/doc-service/app/routers/categories.py` — full rewrite: name validation regex, scope-based list/create, `_check_can_manage_cat` permission helper, scope-aware rename/delete
- `features/doc-service/app/routers/documents.py``delete_document` enforces is_admin/can_delete/group-admin hierarchy; `remove_category` requires doc ownership; `assign_category` accepts group/system categories
- `frontend/src/api/client.ts``CategoryOut` gains `scope`/`group_id`; `createCategory` accepts optional `groupId`; `UserGroupOut`/`GroupMemberOut` gain `is_group_admin`; new `adminSetGroupMemberAdmin()`; `ApiError` exported
- `frontend/src/components/ManageCategoriesDialog.tsx` — categories grouped by scope; lock icons for unmanageable categories; rename/delete gated by scope permissions; inline rename error display
- `frontend/src/components/SourcePanel.tsx` — categories shown in sections (Mine / Group name / System); scope picker on new category form; client-side name validation
- `frontend/src/pages/AdminGroupsPage.tsx` — group admin checkbox column in members table
- `backend/CLAUDE.md` — updated `group_memberships` model, migration chain, endpoints
- `features/doc-service/CLAUDE.md` — updated `document_categories` model, `document_shares` model, migration chain, deps note
@@ -0,0 +1,43 @@
# 2026-04-18 — Doc Service Redesign: Scalable UX + Group-Based Sharing
**Timestamp:** 2026-04-18T00:00:00+00:00
## Summary
Complete redesign of the document management UX for scale (10 → 100 000 documents, 2 → 1 000 categories) and group-based document sharing. Replaced the monolithic DocumentsPage with a three-column layout (Sidebar + SourcePanel + main), a slide-over detail panel, a filter chip system, multi-file upload queue, and bulk actions. Added the full backend sharing stack: `document_shares` table, share CRUD endpoints, a shared-with-me view, and X-User-Groups header injection in the gateway proxy.
---
## Files Added
| File | Description |
|------|-------------|
| `features/doc-service/app/models/document_share.py` | DocumentShare ORM model (document_id, group_id, shared_by_user_id) |
| `features/doc-service/app/schemas/share.py` | DocumentShareOut, DocumentShareCreate, SharedDocumentOut schemas |
| `features/doc-service/alembic/versions/0004_add_document_shares.py` | Migration creating document_shares table with indexes |
| `frontend/src/components/SourcePanel.tsx` | Left panel (240px): views (All/Mine/Shared) + searchable category tree + new category form |
| `frontend/src/components/ManageCategoriesDialog.tsx` | Category CRUD modal (inline rename, delete with confirm, search) |
| `frontend/src/components/DocumentSlideOver.tsx` | Right slide-over (480px): metadata, inline title edit, type picker, AI suggestions, categories combobox, tags, sharing section, raw text, actions |
---
## Files Modified
| File | Change |
|------|--------|
| `features/doc-service/app/models/__init__.py` | Import DocumentShare |
| `features/doc-service/app/deps.py` | Added `get_user_groups` dependency (reads X-User-Groups header) |
| `features/doc-service/app/schemas/document.py` | Added `share_count: int = 0` to DocumentOut |
| `features/doc-service/app/routers/documents.py` | Complete rewrite: added share CRUD endpoints, shared-with-me endpoint, N+1-safe share_count, recipient download access, X-User-Groups enforcement |
| `features/doc-service/STATUS.md` | Added sharing section, migration 0004, updated future work |
| `backend/app/routers/documents_proxy.py` | Injects X-User-Groups header (queries GroupMembership per request) |
| `backend/app/routers/categories_proxy.py` | Same X-User-Groups injection pattern |
| `backend/app/routers/users.py` | Added GET /me/groups endpoint |
| `backend/app/schemas/user.py` | Added UserGroupOut schema |
| `backend/STATUS.md` | Added /me/groups endpoint |
| `frontend/src/api/client.ts` | Added share_count to DocumentOut, SharedDocumentOut, DocumentShareOut, listSharedWithMe, getDocumentShares, addDocumentShare, removeDocumentShare, getMyGroups |
| `frontend/src/components/Sidebar.tsx` | Removed per-category NavLinks; Documents is now a single NavLink under Apps |
| `frontend/src/components/AppShell.tsx` | Renders SourcePanel between Sidebar and main on /apps/documents route |
| `frontend/src/pages/DocumentsPage.tsx` | Complete rewrite: three-panel layout, view param (all/mine/shared), smart polling, drag-and-drop, multi-file upload queue, filter chip system, bulk actions bar |
| `frontend/STATUS.md` | Complete rewrite reflecting all new components and patterns |
| `CLAUDE.md` | Updated file tree, Database Models (DocumentShare), Migration chains (0004), API endpoints (shares, shared-with-me, /me/groups), TanStack Query keys, request flow diagram |
@@ -0,0 +1,25 @@
# 2026-04-18 — Document delete permissions + three-dots menu fix
**Timestamp:** 2026-04-18T00:00:00Z
## Summary
Added a proper permission model for document deletion: owners and admins can always delete; group members can delete only when the share was explicitly granted `can_delete=true`. Fixed silent delete failures (watch docs returning 404 with no user feedback) and fixed the three-dots context menu being clipped by `overflow-hidden` on the table container.
## Files Added / Modified / Deleted
### Added
- `features/doc-service/alembic/versions/0005_add_share_can_delete.py` — migration: adds `can_delete BOOLEAN NOT NULL DEFAULT false` to `document_shares`
### Modified
- `features/doc-service/app/models/document_share.py` — added `can_delete: Mapped[bool]` column
- `features/doc-service/app/schemas/share.py` — added `can_delete` to `DocumentShareOut` and `DocumentShareCreate`; added `viewer_can_delete` to `SharedDocumentOut`
- `features/doc-service/app/schemas/document.py` — added `viewer_can_delete: bool = False` to `DocumentOut`
- `features/doc-service/app/deps.py` — added `get_user_is_admin()` dep reading `x-user-is-admin` header
- `features/doc-service/app/routers/documents.py` — added `_get_deletable_doc_ids()` helper; updated list/get/delete endpoints with permission logic; updated `add_share` to store `can_delete`; updated shared-with-me to include `viewer_can_delete`
- `backend/app/routers/documents_proxy.py``_forward_headers()` now injects `x-user-is-admin` header
- `frontend/src/api/client.ts``DocumentOut`: added `viewer_can_delete`; `DocumentShareOut`: added `can_delete`; `addDocumentShare`: accepts `canDelete` param
- `frontend/src/pages/DocumentsPage.tsx``RowActionsMenu`: replaced absolute dropdown with `createPortal` to fix clipping; delete button now uses `doc.viewer_can_delete`; added `onError` handler for silent failures
- `frontend/src/components/DocumentSlideOver.tsx` — sharing section: shows trash icon badge on shares with `can_delete=true`; added "Allow group members to delete" checkbox before group picker; delete button uses `doc.viewer_can_delete`
- `features/doc-service/CLAUDE.md` — updated `document_shares` table docs + migration chain
- `backend/CLAUDE.md` — noted `x-user-is-admin` header injection
@@ -0,0 +1,12 @@
# 2026-04-19 — Merge checklist update + admin delete bug fix
**Timestamp:** 2026-04-19T00:15:00Z
## Summary
Updated `tests/MERGE_CHECKLIST.md` with all new tests for the two recently merged features (document delete permissions and category scopes / group-admin role). While running the new test 12.16b, discovered and fixed a bug where the doc-service delete endpoint returned 404 for admins deleting non-owned documents.
## Files Added / Modified / Deleted
- **Modified** `tests/MERGE_CHECKLIST.md` — added 18 new tests: 4.94.10 (group admin role), 12.16b12.16e (delete permissions), 13.1113.14 (can_delete sharing), 14.714.17 (category scopes, PascalCase naming), 19.11 (three-dots portal fix); updated 12.16 and 14.5 descriptions
- **Modified** `features/doc-service/app/routers/documents.py` — fixed `delete_document` to bypass group-membership filter for admins; previously admins got 404 on any document they didn't own or that wasn't a watch doc
+60
View File
@@ -0,0 +1,60 @@
# 2026-04-20 — Dedicated storage-service with pluggable backends
**Timestamp:** 2026-04-20T00:00:00Z
## Summary
Introduced a dedicated `storage-service` container (port 8020) as the single file/blob persistence layer for the entire stack. All services now route file and config I/O through this service's HTTP API. The service supports pluggable storage backends (local filesystem by default; S3-compatible and WebDAV built in) with a zero-data-loss migration flow. The `doc_data` and `app_config` Docker volumes were removed.
## Files Added
- `features/storage-service/app/main.py` — FastAPI app, lifespan (backend init)
- `features/storage-service/app/core/config.py` — Settings (DATA_DIR, STORAGE_BACKEND, S3_*, WEBDAV_*)
- `features/storage-service/app/routers/health.py` — GET /health
- `features/storage-service/app/routers/objects.py` — PUT/GET/DELETE /objects/{bucket}/{key:path}, GET /objects/{bucket}
- `features/storage-service/app/routers/migrate.py` — POST/GET/DELETE /migrate, PATCH /backend-config
- `features/storage-service/app/services/backend_manager.py` — Driver factory, singleton, atomic switch
- `features/storage-service/app/services/migration.py` — Async migration: copy → verify → switch → cleanup
- `features/storage-service/app/services/backends/base.py` — AbstractStorageBackend ABC
- `features/storage-service/app/services/backends/local.py` — LocalFSBackend (path traversal guard)
- `features/storage-service/app/services/backends/s3.py` — S3Backend (aiobotocore, endpoint_url configurable)
- `features/storage-service/app/services/backends/webdav.py` — WebDAVBackend (aiohttp + defusedxml)
- `features/storage-service/scripts/start.sh` — prod uvicorn start
- `features/storage-service/scripts/start_dev.sh` — dev uvicorn --reload start
- `features/storage-service/pyproject.toml` — Dependencies
- `features/storage-service/Dockerfile` — python:3.12-slim, non-root user 1001, port 8020
- `features/storage-service/CLAUDE.md` — API reference, bucket docs, driver docs
- `features/storage-service/STATUS.md` — Service status
- `backend/app/core/config_storage.py` — Thin async helpers: read_json/write_json/delete_key/list_keys
- `backend/app/routers/storage_config.py` — Admin proxy endpoints for storage config + migration
- `features/doc-service/alembic/versions/0008_rename_file_path_to_storage_key.py` — DB migration
- `frontend/src/pages/StorageAdminPage.tsx` — Admin UI: backend status, driver form, migration progress
- `tests/storage-service_tests.md` — §20 storage-service test suite
## Files Modified
- `docker-compose.yml` — Added storage-service, storage_data volume; removed doc_data, app_config; added depends_on service_healthy
- `docker-compose.dev.yml` — Added storage-service dev override
- `backend/app/core/config.py` — Added STORAGE_SERVICE_URL
- `backend/app/core/app_config.py` — Full async rewrite using config_storage HTTP helpers (no filesystem)
- `backend/app/routers/settings.py` — Removed all asyncio.to_thread wrappers; direct await calls
- `backend/app/main.py` — Register storage_config router; update register_services call
- `backend/app/services/service_health.py` — Register storage-service
- `features/doc-service/app/core/config.py` — Added STORAGE_SERVICE_URL
- `features/doc-service/app/models/document.py` — file_path → storage_key
- `features/doc-service/app/services/storage.py` — Complete rewrite: HTTP client calls to storage-service
- `features/doc-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket
- `features/doc-service/app/services/file_watcher.py` — Uses save_upload() → storage-service
- `features/doc-service/app/routers/documents.py` — storage_key refs, pdfplumber(io.BytesIO), streaming from storage-service
- `features/ai-service/app/core/config.py` — Added STORAGE_SERVICE_URL; removed CONFIG_PATH
- `features/ai-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket
- `frontend/src/api/client.ts` — Added StorageStatus, MigrationStatus, StorageBackendConfig interfaces + 5 API functions
- `frontend/src/App.tsx` — Added /admin/storage route (AdminRoute → StorageAdminPage)
- `tests/ALL_TESTS.md` — Updated to 20 feature areas; added §20 storage-service tests
- `CLAUDE.md` — Added storage-service to Services/Volumes/Networks tables; storage enforcement rule; §20 test file
- `backend/CLAUDE.md` — Added config_storage.py, storage_config.py to tree; added admin storage endpoints
- `frontend/CLAUDE.md` — Added StorageAdminPage to tree; added /admin/storage route
- `features/doc-service/CLAUDE.md` — Updated storage.py description; file_path → storage_key; added migration 0008
- `features/ai-service/CLAUDE.md` — Added config_reader.py description
- `backend/STATUS.md` — Added storage-config endpoints; updated settings persistence note
- `frontend/STATUS.md` — Added /admin/storage route; added StorageAdminPage description
+5
View File
@@ -29,6 +29,11 @@ services:
volumes:
- ./features/ai-service:/app
storage-service:
command: sh scripts/start_dev.sh
volumes:
- ./features/storage-service:/app
doc-service:
command: sh scripts/start_dev.sh
env_file: ./features/doc-service/.env
+32 -11
View File
@@ -19,6 +19,27 @@ services:
networks:
- backend-net
# ── Storage service (unified blob storage) ──────────────────────────────────
storage-service:
build:
context: ./features/storage-service
dockerfile: Dockerfile
network: host
user: "1001:1001"
restart: unless-stopped
environment:
STORAGE_BACKEND: local
DATA_DIR: /data/storage
volumes:
- storage_data:/data/storage
healthcheck:
test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8020/health')\""]
interval: 10s
timeout: 5s
retries: 5
networks:
- backend-net
# ── Backend (management) ────────────────────────────────────────────────────
backend:
build:
@@ -32,11 +53,12 @@ services:
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-password}@db:5432/${POSTGRES_DB:-destroying_sap}
DOC_SERVICE_URL: http://doc-service:8001
AI_SERVICE_URL: http://ai-service:8010
volumes:
- app_config:/config
STORAGE_SERVICE_URL: http://storage-service:8020
depends_on:
db:
condition: service_healthy
storage-service:
condition: service_healthy
networks:
- backend-net
@@ -49,9 +71,10 @@ services:
user: "1001:1001"
restart: unless-stopped
environment:
CONFIG_PATH: /config/ai_service_config.json
volumes:
- app_config:/config
STORAGE_SERVICE_URL: http://storage-service:8020
depends_on:
storage-service:
condition: service_healthy
networks:
- backend-net
@@ -65,18 +88,17 @@ services:
restart: unless-stopped
environment:
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-password}@db:5432/${POSTGRES_DB:-destroying_sap}
DATA_DIR: /data/documents
CONFIG_PATH: /config/doc_service_config.json
AI_SERVICE_URL: http://ai-service:8010
STORAGE_SERVICE_URL: http://storage-service:8020
volumes:
- doc_data:/data/documents
- watch_data:/data/watch
- app_config:/config
depends_on:
db:
condition: service_healthy
ai-service:
condition: service_started
storage-service:
condition: service_healthy
networks:
- backend-net
@@ -98,9 +120,8 @@ services:
volumes:
postgres_data:
doc_data: # PDF files persisted across restarts
storage_data: # All file/blob storage — managed by storage-service (documents + config)
watch_data: # Watch directory — bind-mount your NAS/Nextcloud here via docker-compose.override.yml
app_config: # Per-service runtime config JSON files
networks:
# backend-net: db ↔ backend ↔ doc-service. No host ports bound.
+51
View File
@@ -0,0 +1,51 @@
# ai-service — Claude context
AI provider intermediary, port 8010 (internal only — never proxied to the browser). Accepts chat requests from `doc-service` (and potentially other callers). Manages a priority queue and abstracts over multiple AI providers (Anthropic, Ollama/LM Studio). See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
---
## File & Folder Tree
```
features/ai-service/
├── app/
│ ├── main.py ← FastAPI, queue worker lifespan
│ ├── core/
│ │ └── config.py ← Settings via pydantic-settings
│ ├── providers/
│ │ ├── base.py ← AIProvider abstract class
│ │ ├── anthropic_provider.py ← Anthropic API integration
│ │ └── openai_compat.py ← Ollama / LM Studio compatibility
│ ├── routers/
│ │ ├── chat.py ← POST /chat (sync, NORMAL priority queue)
│ │ ├── health.py ← GET /health
│ │ ├── queue.py ← GET /queue/status, /pause, /resume, /cancel/{id}
│ │ └── plugin.py ← GET /plugin/manifest (access rules for ai-service-admin group)
│ └── services/
│ ├── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL)
│ └── config_reader.py ← Reads ai_service_config.json from storage-service config bucket (30 s TTL cache)
├── Dockerfile ← python:3.12-slim, non-root user 1001
└── STATUS.md
```
---
## API Endpoints (internal only)
| Method | Path | Description |
|--------|------|-------------|
| POST | `/chat` | Chat request (queued at NORMAL priority) |
| GET | `/health` | Health check |
| GET | `/queue/status` | Queue state |
| POST | `/queue/pause` | Pause queue |
| POST | `/queue/resume` | Resume queue |
| POST | `/queue/cancel/{job_id}` | Cancel job |
| GET | `/plugin/manifest` | Plugin manifest (access rules for ai-service-admin group) |
These endpoints are only reachable on `backend-net`. The backend does not expose them to the browser.
---
## Note on timeout and retry configuration
Caller-side timeout and retry settings live in `features/doc-service/app/services/ai_client.py` — see `features/doc-service/CLAUDE.md` for the values.
+1 -2
View File
@@ -15,8 +15,7 @@ FROM python:3.12-slim
RUN groupadd --gid 1001 appuser && \
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
# Pre-create the config directory with correct ownership
RUN mkdir -p /config && chown -R appuser:appuser /config
# No filesystem directories needed — all config goes through storage-service.
WORKDIR /app
+1 -1
View File
@@ -3,7 +3,7 @@ from pydantic_settings import BaseSettings
class Settings(BaseSettings):
PROJECT_NAME: str = "ai-service"
CONFIG_PATH: str = "/config/ai_service_config.json"
STORAGE_SERVICE_URL: str = "http://storage-service:8020"
model_config = {"env_file": ".env", "extra": "ignore"}
@@ -1,5 +1,5 @@
"""
Reads ai_service_config.json from the shared config volume.
Reads ai_service_config.json from the storage-service config bucket.
30-second TTL cache + env var overrides (dev credentials stay out of git).
Env var overrides (all optional):
@@ -8,15 +8,17 @@ Env var overrides (all optional):
OLLAMA_BASE_URL, OLLAMA_MODEL, OLLAMA_API_KEY
ANTHROPIC_API_KEY, ANTHROPIC_MODEL
"""
import asyncio
import json
import os
import time
from copy import deepcopy
from pathlib import Path
import httpx
from app.core.config import settings
_CONFIG_KEY = "ai_service_config.json"
_DEFAULT_CONFIG: dict = {
"provider": "lmstudio",
"timeout_seconds": 60,
@@ -31,12 +33,18 @@ _cache_at: float = 0.0
_CACHE_TTL = 30.0
def _read_config_sync() -> dict:
path = Path(settings.CONFIG_PATH)
if not path.exists():
return _apply_env_overrides(deepcopy(_DEFAULT_CONFIG))
with open(path) as f:
return _apply_env_overrides(json.load(f))
def _storage_url() -> str:
return f"{settings.STORAGE_SERVICE_URL}/objects/config/{_CONFIG_KEY}"
async def _fetch_config() -> dict:
"""Fetch config from storage-service. Returns defaults if not found."""
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.get(_storage_url())
if resp.status_code == 404:
return deepcopy(_DEFAULT_CONFIG)
resp.raise_for_status()
return resp.json()
def _apply_env_overrides(config: dict) -> dict:
@@ -75,7 +83,8 @@ async def load_ai_config() -> dict:
now = time.monotonic()
if _cache is not None and (now - _cache_at) < _CACHE_TTL:
return _cache
data = await asyncio.to_thread(_read_config_sync)
raw = await _fetch_config()
data = _apply_env_overrides(raw)
_cache = data
_cache_at = now
return data
+184
View File
@@ -0,0 +1,184 @@
# doc-service — Claude context
PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. All file/blob storage goes through `storage-service:8020` — no files are written directly to the filesystem. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
---
## Commands
All commands run inside Docker — never on the host.
```bash
docker compose exec doc-service alembic revision --autogenerate -m "describe change"
docker compose exec doc-service alembic upgrade head
docker compose exec doc-service alembic downgrade -1
```
---
## File & Folder Tree
```
features/doc-service/
├── app/
│ ├── main.py ← FastAPI, lifespan (file watcher start/stop)
│ ├── database.py ← Same PostgreSQL instance as backend
│ ├── deps.py ← get_user_id, get_user_groups, get_user_is_admin, get_user_admin_groups (injected headers)
│ ├── models/
│ │ ├── document.py ← Document model
│ │ ├── category.py ← DocumentCategory model
│ │ ├── category_assignment.py ← CategoryAssignment (composite PK)
│ │ └── document_share.py ← DocumentShare model (group-based sharing)
│ ├── schemas/
│ │ ├── document.py ← DocumentOut, DocumentPage, DocumentStatusOut, etc.
│ │ ├── category.py ← CategoryOut, CategoryCreate, CategoryUpdate
│ │ └── share.py ← DocumentShareOut, DocumentShareCreate, SharedDocumentOut
│ ├── routers/
│ │ ├── documents.py ← Full CRUD + file serving + reprocess + suggestions + sharing
│ │ ├── categories.py ← Category CRUD (includes watch-owned categories)
│ │ └── plugin.py ← GET /plugin/manifest, GET+PATCH /plugin/settings
│ └── services/
│ ├── storage.py ← Storage client: save_upload/download_file/delete_file → storage-service:8020 documents bucket
│ ├── ai_client.py ← classify_document() → ai-service:8010/chat
│ ├── config_reader.py ← Config load/save via storage-service config bucket (doc_service_config.json)
│ └── file_watcher.py ← watchdog-based PDF watcher + startup scan + ingestion
├── alembic/versions/ ← Migration chain
│ ├── 0003_add_watch_columns.py ← source, watch_path, suggested_folder, suggested_filename
│ ├── 0004_add_document_shares.py ← document_shares table (group-based sharing)
│ └── 0008_rename_file_path_to_storage_key.py ← file_path → storage_key; strips /data/documents/ prefix from existing rows
├── Dockerfile ← python:3.12-slim, non-root user 1001
└── STATUS.md
```
---
## Database Models
### `documents`
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | |
| `user_id` | String | indexed | not FK — trusts x-user-id header |
| `filename` | String | NOT NULL | |
| `storage_key` | String | NOT NULL | storage-service key: `{user_id}/{doc_id}.pdf` (documents bucket) |
| `file_size` | Integer | NOT NULL | bytes |
| `status` | String | default="pending" | pending / processing / done / failed |
| `title` | String(500) | nullable | AI-extracted |
| `document_type` | String | nullable | invoice / bill / receipt / order / expense / revenue / unknown |
| `raw_text` | Text | nullable | first 500 k chars |
| `extracted_data` | Text | nullable | JSON string |
| `tags` | Text | nullable | JSON array string |
| `error_message` | String(500) | nullable | |
| `created_at` | DateTime(tz) | server_default=now() | |
| `processed_at` | DateTime(tz) | nullable | |
| `source` | String(16) | default="upload" | "upload" or "watch" |
| `watch_path` | String | nullable | original absolute path in watch directory |
| `suggested_folder` | String(128) | nullable | AI-suggested category (pending user confirm) |
| `suggested_filename` | String(500) | nullable | AI-suggested title/rename (pending user confirm) |
### `document_categories`
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | |
| `user_id` | String | indexed | owner; "watch" for system categories |
| `name` | String(128) | NOT NULL | PascalCase-with-dashes convention enforced on create/rename |
| `scope` | String(16) | NOT NULL, default="personal" | "personal" / "group" / "system" |
| `group_id` | String | nullable, indexed | set when scope="group" |
| `created_at` | DateTime(tz) | server_default=now() | |
### `document_category_assignments` (composite PK)
| Column | Type | Constraints |
|--------|------|-------------|
| `document_id` | String | PK + FK→documents.id CASCADE |
| `category_id` | String | PK + FK→document_categories.id CASCADE |
### `document_shares`
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| `id` | String | PK, UUID | |
| `document_id` | String | indexed, NOT NULL | not FK — trusts proxy |
| `group_id` | String | indexed, NOT NULL | group from backend |
| `shared_by_user_id` | String | NOT NULL | owner who shared |
| `can_delete` | Boolean | NOT NULL, default=false | allows group members to delete the doc |
| `created_at` | DateTime(tz) | server_default=now() | |
Unique constraint: `(document_id, group_id)`
### Migration chain
| Rev ID | Slug |
|--------|------|
| `0001` | `create_doc_tables` |
| `0002` | `add_document_title` |
| `0003` | `add_watch_columns` |
| `0004` | `add_document_shares` |
| `0005` | `add_share_can_delete` |
| `0006` | `add_category_scope` |
| `0007` | `capitalize_system_category_names` |
| `0008` | `rename_file_path_to_storage_key` |
---
## API Endpoints (internal — reached via backend proxy)
All these endpoints are proxied from `backend:8000`. The backend injects `x-user-id` and `x-user-groups` before forwarding.
### Documents
| Method | Path | Description |
|--------|------|-------------|
| POST | `/documents/upload` | Upload PDF (202, background processing) |
| GET | `/documents` | Paginated list (filterable: search, status, type, category, sort) |
| GET | `/documents/{id}` | Document detail |
| GET | `/documents/{id}/status` | Processing status only |
| PATCH | `/documents/{id}/type` | Update document type |
| PATCH | `/documents/{id}/tags` | Update tags |
| PATCH | `/documents/{id}/title` | Update title |
| POST | `/documents/{id}/reprocess` | Re-run AI extraction |
| DELETE | `/documents/{id}` | Delete document (204) |
| GET | `/documents/{id}/file` | Download PDF (streaming) |
| POST | `/documents/{id}/categories/{cat_id}` | Assign category |
| DELETE | `/documents/{id}/categories/{cat_id}` | Remove category |
| POST | `/documents/{id}/suggestions/folder/confirm` | Confirm AI folder suggestion |
| POST | `/documents/{id}/suggestions/folder/reject` | Reject AI folder suggestion |
| POST | `/documents/{id}/suggestions/filename/confirm` | Confirm AI filename suggestion |
| POST | `/documents/{id}/suggestions/filename/reject` | Reject AI filename suggestion |
| GET | `/documents/shared-with-me` | Documents shared with current user via their groups |
| GET | `/documents/{id}/shares` | List groups the document is shared with (owner only) |
| POST | `/documents/{id}/shares` | Share with a group (owner only; group must be in user's groups) |
| DELETE | `/documents/{id}/shares/{group_id}` | Stop sharing with a group (owner only) |
### Categories
| Method | Path | Description |
|--------|------|-------------|
| GET | `/categories` | List user's categories |
| POST | `/categories` | Create category (triggers background AI reanalysis) |
| PATCH | `/categories/{id}` | Rename |
| DELETE | `/categories/{id}` | Delete (204) |
### Plugin
| Method | Path | Description |
|--------|------|-------------|
| GET | `/plugin/manifest` | Plugin manifest with settings JSON Schema |
| GET | `/plugin/settings` | Current plugin settings |
| PATCH | `/plugin/settings` | Update plugin settings |
---
## Default Values & Limits
| Parameter | Value | Location |
|-----------|-------|----------|
| Document title max | 500 chars | `models/document.py` |
| Category name max | 128 chars | `models/category.py` |
| PDF max size (default) | 20 MB | admin settings (configurable) |
| Raw text cap | 500 k chars | `services/ai_client.py` |
| Documents per_page | 1100, default 20 | `routers/documents.py` |
| AI service timeout | 60 s | `services/ai_client.py` |
| AI service max retries | 2 | `services/ai_client.py` |
+3 -3
View File
@@ -15,9 +15,9 @@ FROM python:3.12-slim
RUN groupadd --gid 1001 appuser && \
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
# Pre-create data and config dirs with correct ownership.
# Named volumes mounted over these paths will inherit ownership on first creation.
RUN mkdir -p /data/documents /data/watch /config && chown -R appuser:appuser /data /config
# Pre-create watch dir with correct ownership.
# /data/documents and /config are no longer used — all file/config storage goes through storage-service.
RUN mkdir -p /data/watch && chown -R appuser:appuser /data
WORKDIR /app
+24 -5
View File
@@ -123,6 +123,23 @@ Controlled via plugin settings (UI accessible to superusers and `doc-service-adm
On startup scan, the watcher walks the watch directory and ingests any PDFs not already in the database (idempotency check by `watch_path`). Subfolders are automatically mapped to categories (e.g. `watch/invoices/bill.pdf` → category "invoices"). No-remove policy: deleting a file from the watch directory does not delete the document record.
### Document sharing (`document_shares`)
Group-based sharing allows a document owner to share a document with all members of any group they belong to. Recipients can view and download the shared document; they cannot edit, re-analyse, delete, or re-share it.
The gateway injects `X-User-Groups: <group_id1>,<group_id2>,...` alongside the existing `X-User-Id` header, so doc-service can evaluate group access without querying the backend DB.
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/documents/shared-with-me` | Any user | Documents shared with the user via their groups; excludes own docs |
| `GET` | `/documents/{id}/shares` | Owner only | List all groups the document is shared with |
| `POST` | `/documents/{id}/shares` | Owner only | Share with a group (`{group_id}` in body); group must be in X-User-Groups |
| `DELETE` | `/documents/{id}/shares/{group_id}` | Owner only | Stop sharing with that group |
`DocumentOut` now includes `share_count: int` — the number of groups the document is shared with.
`GET /documents/{id}/file` also allows access to shared documents (recipients can download).
### Database migrations
| Revision | Description |
@@ -130,6 +147,7 @@ On startup scan, the watcher walks the watch directory and ingests any PDFs not
| 0001 | Initial schema (documents, categories, category_assignments) |
| 0002 | Add `title` column to documents |
| 0003 | Add `source`, `watch_path`, `suggested_folder`, `suggested_filename` columns |
| 0004 | Add `document_shares` table (document_id, group_id, shared_by_user_id, created_at) |
Run automatically on container start via `alembic upgrade head`.
@@ -168,8 +186,8 @@ file_watcher.py (watchdog Observer, daemon thread)
- **Re-process** — no endpoint to re-trigger AI extraction on an existing document (e.g. after changing the AI model or prompt)
- **Advanced field-level search** — `search` param matches text fields via ILIKE but does not query into `extracted_data` JSON (e.g. filter by `vendor` or `due_date`)
- **Bulk operations** — no bulk category assign/remove, no bulk delete
- **Document sharing** — documents are strictly per-user; no group sharing yet
- **Bulk operations** — no bulk category assign/remove endpoint (frontend handles bulk delete/share individually)
- **Advanced field-level search** — `search` matches text fields via ILIKE but does not query into `extracted_data` JSON
- **Pagination in categories** — categories are returned as a full list (no pagination)
- **File type** — only PDF supported
- **Concurrent uploads** — no rate limiting per user
@@ -183,9 +201,10 @@ file_watcher.py (watchdog Observer, daemon thread)
- [x] Plugin manifest endpoint (`/plugin/manifest`, `/plugin/settings`) for generic settings UI
- [ ] Advanced filter: query `extracted_data` JSON fields (vendor, due_date, amount) — requires PostgreSQL `jsonb` column or indexed virtual columns
- [ ] Bulk operations endpoint
- [ ] Document sharing via groups (blocked on groups/permissions system in backend)
- [x] Document sharing via groups `document_shares` table + share endpoints + shared-with-me view
- [x] Frontend UI for suggestion badges (suggested_folder / suggested_filename confirm/reject buttons in slide-over)
- [ ] Advanced filter: query `extracted_data` JSON fields (vendor, due_date, amount)
- [ ] Support additional file types (images via OCR, DOCX)
- [ ] Rate limiting on upload endpoint
- [ ] Soft delete with restore
- [ ] Category rename / delete with cascade handling
- [ ] Frontend UI for suggestion badges (suggested_folder / suggested_filename confirm/reject buttons)
- [ ] Edit rights for shared recipients (V2)
@@ -0,0 +1,42 @@
"""add document_shares table
Revision ID: 0004
Revises: 0003
Create Date: 2026-04-18
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0004"
down_revision: Union[str, None] = "0003"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"document_shares",
sa.Column("id", sa.String(), nullable=False),
sa.Column("document_id", sa.String(), nullable=False),
sa.Column("group_id", sa.String(), nullable=False),
sa.Column("shared_by_user_id", sa.String(), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("now()"),
nullable=False,
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("document_id", "group_id", name="uq_document_group_share"),
)
op.create_index("ix_document_shares_document_id", "document_shares", ["document_id"])
op.create_index("ix_document_shares_group_id", "document_shares", ["group_id"])
def downgrade() -> None:
op.drop_index("ix_document_shares_group_id", table_name="document_shares")
op.drop_index("ix_document_shares_document_id", table_name="document_shares")
op.drop_table("document_shares")
@@ -0,0 +1,32 @@
"""add can_delete to document_shares
Revision ID: 0005
Revises: 0004
Create Date: 2026-04-18
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0005"
down_revision: Union[str, None] = "0004"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"document_shares",
sa.Column(
"can_delete",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
)
def downgrade() -> None:
op.drop_column("document_shares", "can_delete")
@@ -0,0 +1,42 @@
"""add scope and group_id to document_categories
Revision ID: 0006
Revises: 0005
Create Date: 2026-04-18
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0006"
down_revision: Union[str, None] = "0005"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"document_categories",
sa.Column(
"scope",
sa.String(16),
nullable=False,
server_default="personal",
),
)
op.add_column(
"document_categories",
sa.Column("group_id", sa.String(), nullable=True),
)
op.create_index("ix_document_categories_group_id", "document_categories", ["group_id"])
# Migrate existing watch-owned categories to system scope
op.execute("UPDATE document_categories SET scope = 'system' WHERE user_id = 'watch'")
def downgrade() -> None:
op.drop_index("ix_document_categories_group_id", table_name="document_categories")
op.drop_column("document_categories", "group_id")
op.drop_column("document_categories", "scope")
@@ -0,0 +1,44 @@
"""capitalize existing system category names to PascalCase-with-dashes
Revision ID: 0007
Revises: 0006
Create Date: 2026-04-18
Converts names like "invoices""Invoices", "vendor-invoices""Vendor-Invoices"
for all categories with scope='system' (watch-ingested).
"""
from typing import Sequence, Union
import re
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text
revision: str = "0007"
down_revision: Union[str, None] = "0006"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _to_pascal(name: str) -> str:
return "-".join(p.capitalize() for p in re.split(r"[-_\s]+", name) if p)
def upgrade() -> None:
conn = op.get_bind()
rows = conn.execute(
text("SELECT id, name FROM document_categories WHERE scope = 'system'")
).fetchall()
for row in rows:
new_name = _to_pascal(row.name)
if new_name != row.name:
conn.execute(
text("UPDATE document_categories SET name = :name WHERE id = :id"),
{"name": new_name[:128], "id": row.id},
)
def downgrade() -> None:
pass # names before migration are unknown; downgrade is a no-op
@@ -0,0 +1,56 @@
"""rename file_path to storage_key and strip filesystem prefix from existing rows
Revision ID: 0008
Revises: 0007
Create Date: 2026-04-20
Renames the documents.file_path column to storage_key.
Existing rows have paths like '/data/documents/{user_id}/{doc_id}.pdf' or
'/data/documents/watch/{doc_id}.pdf'. The migration strips the leading
'/data/documents/' prefix so the value becomes a plain storage key
(e.g. '{user_id}/{doc_id}.pdf') that the storage-service uses as the object key.
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0008"
down_revision: Union[str, None] = "0007"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
with op.batch_alter_table("documents") as batch_op:
batch_op.alter_column(
"file_path",
new_column_name="storage_key",
existing_type=sa.String(),
existing_nullable=False,
)
# Strip the '/data/documents/' filesystem prefix from pre-migration rows.
op.execute(
sa.text(
"UPDATE documents SET storage_key = REPLACE(storage_key, '/data/documents/', '')"
" WHERE storage_key LIKE '/data/documents/%'"
)
)
def downgrade() -> None:
# Restore the filesystem prefix so old code can still find the files.
op.execute(
sa.text(
"UPDATE documents SET storage_key = '/data/documents/' || storage_key"
" WHERE storage_key NOT LIKE '/data/documents/%'"
)
)
with op.batch_alter_table("documents") as batch_op:
batch_op.alter_column(
"storage_key",
new_column_name="file_path",
existing_type=sa.String(),
existing_nullable=False,
)
+1
View File
@@ -7,6 +7,7 @@ class Settings(BaseSettings):
DATA_DIR: str = "/data/documents"
CONFIG_PATH: str = "/config/doc_service_config.json"
AI_SERVICE_URL: str = "http://ai-service:8010"
STORAGE_SERVICE_URL: str = "http://storage-service:8020"
class Config:
env_file = ".env"
+30
View File
@@ -10,3 +10,33 @@ async def get_user_id(x_user_id: str = Header(...)) -> str:
if not x_user_id:
raise HTTPException(status_code=400, detail="Missing X-User-Id header")
return x_user_id
async def get_user_groups(x_user_groups: str = Header(default="")) -> list[str]:
"""
Extract the group IDs injected by the main backend proxy.
Comma-separated list of group UUIDs the current user belongs to.
Returns an empty list if the header is absent or empty.
"""
if not x_user_groups:
return []
return [g.strip() for g in x_user_groups.split(",") if g.strip()]
async def get_user_is_admin(x_user_is_admin: str = Header(default="false")) -> bool:
"""
Extract the superuser flag injected by the main backend proxy.
Returns True only when the header value is exactly "true".
"""
return x_user_is_admin.lower() == "true"
async def get_user_admin_groups(x_user_admin_groups: str = Header(default="")) -> list[str]:
"""
Extract the group IDs for which the current user is a group admin.
Injected by the main backend proxy from GroupMembership.is_group_admin.
Returns an empty list if absent or empty.
"""
if not x_user_admin_groups:
return []
return [g.strip() for g in x_user_admin_groups.split(",") if g.strip()]
+2 -1
View File
@@ -1,5 +1,6 @@
from app.models.document import Document
from app.models.category import DocumentCategory
from app.models.category_assignment import CategoryAssignment
from app.models.document_share import DocumentShare
__all__ = ["Document", "DocumentCategory", "CategoryAssignment"]
__all__ = ["Document", "DocumentCategory", "CategoryAssignment", "DocumentShare"]
@@ -13,6 +13,8 @@ class DocumentCategory(Base):
id: Mapped[str] = mapped_column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
user_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
name: Mapped[str] = mapped_column(String(128), nullable=False)
scope: Mapped[str] = mapped_column(String(16), nullable=False, default="personal", server_default="personal")
group_id: Mapped[str | None] = mapped_column(String, nullable=True, index=True)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
+1 -1
View File
@@ -13,7 +13,7 @@ class Document(Base):
id: Mapped[str] = mapped_column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
user_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
filename: Mapped[str] = mapped_column(String, nullable=False)
file_path: Mapped[str] = mapped_column(String, nullable=False)
storage_key: Mapped[str] = mapped_column(String, nullable=False)
file_size: Mapped[int] = mapped_column(Integer, nullable=False)
status: Mapped[str] = mapped_column(String, nullable=False, default="pending")
title: Mapped[str | None] = mapped_column(String(500), nullable=True)
@@ -0,0 +1,24 @@
import uuid
from datetime import datetime
from sqlalchemy import Boolean, DateTime, String, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from app.database import Base
class DocumentShare(Base):
__tablename__ = "document_shares"
id: Mapped[str] = mapped_column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
document_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
group_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
shared_by_user_id: Mapped[str] = mapped_column(String, nullable=False)
can_delete: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
__table_args__ = (
UniqueConstraint("document_id", "group_id", name="uq_document_group_share"),
)
+82 -14
View File
@@ -1,5 +1,6 @@
import difflib
import json
import re
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
from sqlalchemy import select
@@ -8,7 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import or_
from app.database import AsyncSessionLocal, get_db
from app.deps import get_user_id
from app.deps import get_user_admin_groups, get_user_groups, get_user_id, get_user_is_admin
from app.models.category import DocumentCategory
from app.models.category_assignment import CategoryAssignment
from app.models.document import Document
@@ -22,14 +23,31 @@ _WATCH_USER_ID = "watch"
_SIMILARITY_THRESHOLD = 0.4
# PascalCase-with-dashes: each word starts with a capital, words joined by '-'
# Valid: Invoices, Vendor-Invoices, Q1-Reports
# Invalid: invoices, Invoice Reports, Invoice-reports
_NAME_RE = re.compile(r'^[A-Z][a-zA-Z0-9]*(-[A-Z][a-zA-Z0-9]*)*$')
def _validate_name(name: str) -> None:
if not _NAME_RE.match(name):
raise HTTPException(
status_code=422,
detail=(
"Category name must start with a capital letter. "
"Multiple words are joined with dashes, each starting with a capital "
"(e.g. Invoices, Vendor-Invoices, Q1-Reports)."
),
)
def _name_similarity(a: str, b: str) -> float:
"""Return similarity score (01) between two category names."""
a_low = a.lower()
b_low = b.lower()
# Word overlap is a strong signal
a_words = set(a_low.split())
b_words = set(b_low.split())
a_words = set(a_low.split("-"))
b_words = set(b_low.split("-"))
if a_words & b_words:
return 0.9
# Fallback: character sequence ratio
@@ -81,15 +99,44 @@ async def _reanalyze_documents_for_new_category(
pass
async def _check_can_manage_cat(
cat: DocumentCategory,
user_id: str,
is_admin: bool,
user_admin_groups: list[str],
) -> None:
"""Raise 403/404 if the current user may not rename or delete this category."""
if is_admin:
return # superuser can manage anything
if cat.scope == "system":
raise HTTPException(status_code=403, detail="Only admins can modify system categories")
if cat.scope == "personal" and cat.user_id != user_id:
raise HTTPException(status_code=404, detail="Category not found")
if cat.scope == "group" and cat.group_id not in user_admin_groups:
raise HTTPException(status_code=403, detail="Only group admins can modify group categories")
@router.get("", response_model=list[CategoryOut])
async def list_categories(
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
db: AsyncSession = Depends(get_db),
) -> list[DocumentCategory]:
# Include watch-ingested categories so they appear in the sidebar/filter
"""
Return all categories visible to the current user:
- personal (owned by the user)
- group (any group the user belongs to)
- system (watch-ingested, scope='system')
"""
result = await db.execute(
select(DocumentCategory)
.where(or_(DocumentCategory.user_id == user_id, DocumentCategory.user_id == _WATCH_USER_ID))
.where(
or_(
DocumentCategory.user_id == user_id, # personal
DocumentCategory.group_id.in_(user_groups), # group
DocumentCategory.scope == "system", # system
)
)
.order_by(DocumentCategory.name)
)
return result.scalars().all()
@@ -100,18 +147,32 @@ async def create_category(
body: CategoryCreate,
background_tasks: BackgroundTasks,
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
db: AsyncSession = Depends(get_db),
) -> DocumentCategory:
name = body.name.strip()
if not name:
raise HTTPException(status_code=422, detail="Category name cannot be empty")
_validate_name(name)
if body.group_id:
# User must be a member of the target group
if body.group_id not in user_groups:
raise HTTPException(status_code=403, detail="You are not a member of that group")
cat = DocumentCategory(
user_id=user_id,
name=name[:128],
scope="group",
group_id=body.group_id,
)
else:
cat = DocumentCategory(user_id=user_id, name=name[:128], scope="personal")
cat = DocumentCategory(user_id=user_id, name=name[:128])
db.add(cat)
await db.commit()
await db.refresh(cat)
# Find existing categories with similar names
# Find existing personal categories with similar names for background AI reanalysis
result = await db.execute(
select(DocumentCategory)
.where(DocumentCategory.user_id == user_id)
@@ -140,12 +201,18 @@ async def rename_category(
cat_id: str,
body: CategoryUpdate,
user_id: str = Depends(get_user_id),
is_admin: bool = Depends(get_user_is_admin),
user_admin_groups: list[str] = Depends(get_user_admin_groups),
db: AsyncSession = Depends(get_db),
) -> DocumentCategory:
cat = await _get_user_cat(cat_id, user_id, db)
cat = await _fetch_visible_cat(cat_id, user_id, db)
await _check_can_manage_cat(cat, user_id, is_admin, user_admin_groups)
name = body.name.strip()
if not name:
raise HTTPException(status_code=422, detail="Category name cannot be empty")
_validate_name(name)
cat.name = name[:128]
await db.commit()
await db.refresh(cat)
@@ -156,19 +223,20 @@ async def rename_category(
async def delete_category(
cat_id: str,
user_id: str = Depends(get_user_id),
is_admin: bool = Depends(get_user_is_admin),
user_admin_groups: list[str] = Depends(get_user_admin_groups),
db: AsyncSession = Depends(get_db),
) -> None:
cat = await _get_user_cat(cat_id, user_id, db)
cat = await _fetch_visible_cat(cat_id, user_id, db)
await _check_can_manage_cat(cat, user_id, is_admin, user_admin_groups)
await db.delete(cat)
await db.commit()
async def _get_user_cat(cat_id: str, user_id: str, db: AsyncSession) -> DocumentCategory:
async def _fetch_visible_cat(cat_id: str, user_id: str, db: AsyncSession) -> DocumentCategory:
"""Fetch a category that the user can see (personal/group/system). 404 if absent."""
result = await db.execute(
select(DocumentCategory).where(
DocumentCategory.id == cat_id,
DocumentCategory.user_id == user_id,
)
select(DocumentCategory).where(DocumentCategory.id == cat_id)
)
cat = result.scalar_one_or_none()
if cat is None:
+368 -30
View File
@@ -1,10 +1,10 @@
import asyncio
import io
import json
import math
import uuid
from datetime import datetime, timezone
import aiofiles
import pdfplumber
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, UploadFile
from fastapi.responses import StreamingResponse
@@ -13,14 +13,23 @@ from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.database import AsyncSessionLocal, get_db
from app.deps import get_user_id
from app.deps import get_user_admin_groups, get_user_groups, get_user_id, get_user_is_admin
from app.models.category import DocumentCategory
from app.models.category_assignment import CategoryAssignment
from app.models.document import Document
from app.schemas.document import DocumentOut, DocumentPage, DocumentStatusOut, DocumentTypeUpdate, TagsUpdate, TitleUpdate
from app.models.document_share import DocumentShare
from app.schemas.document import (
DocumentOut,
DocumentPage,
DocumentStatusOut,
DocumentTypeUpdate,
TagsUpdate,
TitleUpdate,
)
from app.schemas.share import DocumentShareCreate, DocumentShareOut, SharedDocumentOut
from app.services.ai_client import AIServiceError, classify_document
from app.services.config_reader import load_doc_config
from app.services.storage import delete_file, get_upload_path, save_upload
from app.services.storage import delete_file, download_file as storage_download, save_upload
router = APIRouter()
@@ -52,7 +61,38 @@ async def _get_user_doc(doc_id: str, user_id: str, db: AsyncSession) -> Document
return doc
def _doc_with_categories(doc: Document) -> DocumentOut:
async def _get_share_counts(doc_ids: list[str], db: AsyncSession) -> dict[str, int]:
"""Return a mapping of doc_id → share count for the given document IDs."""
if not doc_ids:
return {}
rows = await db.execute(
select(DocumentShare.document_id, func.count(DocumentShare.id))
.where(DocumentShare.document_id.in_(doc_ids))
.group_by(DocumentShare.document_id)
)
return {row[0]: row[1] for row in rows.all()}
async def _get_deletable_doc_ids(
doc_ids: list[str], user_groups: list[str], db: AsyncSession
) -> set[str]:
"""Return doc IDs for which the user has delete permission via a group share."""
if not doc_ids or not user_groups:
return set()
rows = await db.execute(
select(DocumentShare.document_id)
.where(
DocumentShare.document_id.in_(doc_ids),
DocumentShare.group_id.in_(user_groups),
DocumentShare.can_delete.is_(True),
)
)
return {row[0] for row in rows.all()}
def _doc_with_categories(
doc: Document, share_count: int = 0, viewer_can_delete: bool = False
) -> DocumentOut:
from app.schemas.document import CategoryOut
cats = [CategoryOut(id=a.category.id, name=a.category.name) for a in doc.category_assignments]
return DocumentOut(
@@ -73,13 +113,15 @@ def _doc_with_categories(doc: Document) -> DocumentOut:
watch_path=doc.watch_path,
suggested_folder=doc.suggested_folder,
suggested_filename=doc.suggested_filename,
share_count=share_count,
viewer_can_delete=viewer_can_delete,
)
def _extract_pdf_text(file_path: str) -> str:
def _extract_pdf_text(pdf_bytes: bytes) -> str:
"""Synchronous — must be called via asyncio.to_thread."""
text_parts = []
with pdfplumber.open(file_path) as pdf:
with pdfplumber.open(io.BytesIO(pdf_bytes)) as pdf:
for page in pdf.pages:
page_text = page.extract_text()
if page_text:
@@ -104,7 +146,8 @@ async def process_document(doc_id: str) -> None:
await db.commit()
try:
text = await asyncio.to_thread(_extract_pdf_text, doc.file_path)
pdf_bytes = await storage_download(doc.storage_key)
text = await asyncio.to_thread(_extract_pdf_text, pdf_bytes)
result = await classify_document(text)
doc.raw_text = text[:500_000] # cap stored text at 500k chars
@@ -145,13 +188,13 @@ async def upload_document(
)
doc_id = str(uuid.uuid4())
dest = await save_upload(file_data, user_id, doc_id)
storage_key = await save_upload(file_data, user_id, doc_id)
doc = Document(
id=doc_id,
user_id=user_id,
filename=file.filename or "upload.pdf",
file_path=str(dest),
storage_key=storage_key,
file_size=len(file_data),
status="pending",
)
@@ -161,8 +204,6 @@ async def upload_document(
background_tasks.add_task(process_document, doc_id)
# Re-query with selectinload so category_assignments is eagerly loaded.
# A new doc has no categories yet, but we need the relationship populated
# to avoid MissingGreenlet in the async session.
doc = await _get_user_doc(doc_id, user_id, db)
return _doc_with_categories(doc)
@@ -189,12 +230,13 @@ async def list_documents(
search: str | None = Query(default=None),
category_id: str | None = Query(default=None),
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
is_admin: bool = Depends(get_user_is_admin),
db: AsyncSession = Depends(get_db),
) -> DocumentPage:
sort_col = _SORT_COLUMNS.get(sort, Document.created_at)
sort_expr = sort_col.desc() if order == "desc" else sort_col.asc()
# Build filter conditions once and reuse for both count + items queries.
# Watch-ingested documents (user_id = "watch") are visible to all users.
conditions = [or_(Document.user_id == user_id, Document.user_id == _WATCH_USER_ID)]
if status:
@@ -233,7 +275,21 @@ async def list_documents(
.offset((page - 1) * per_page)
.limit(per_page)
)
items = [_doc_with_categories(d) for d in items_result.scalars().all()]
docs = items_result.scalars().all()
doc_ids = [d.id for d in docs]
share_counts = await _get_share_counts(doc_ids, db)
if is_admin:
deletable_ids = set(doc_ids)
else:
deletable_ids = {d.id for d in docs if d.user_id == user_id}
deletable_ids |= await _get_deletable_doc_ids(doc_ids, user_groups, db)
items = [
_doc_with_categories(d, share_counts.get(d.id, 0), viewer_can_delete=d.id in deletable_ids)
for d in docs
]
return DocumentPage(
items=items,
@@ -243,14 +299,137 @@ async def list_documents(
)
# NOTE: This route must be registered BEFORE /{doc_id} to avoid path collision.
@router.get("/shared-with-me", response_model=DocumentPage)
async def list_shared_with_me(
page: int = Query(default=1, ge=1),
per_page: int = Query(default=20, ge=1, le=100),
sort: str = Query(default="created_at"),
order: str = Query(default="desc", pattern="^(asc|desc)$"),
search: str | None = Query(default=None),
document_type: str | None = Query(default=None),
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
db: AsyncSession = Depends(get_db),
) -> DocumentPage:
"""Return documents shared with the current user via any of their groups.
Excludes documents the user owns (those appear in their regular list).
"""
if not user_groups:
return DocumentPage(items=[], total=0, page=page, pages=1)
sort_col = _SORT_COLUMNS.get(sort, Document.created_at)
sort_expr = sort_col.desc() if order == "desc" else sort_col.asc()
shared_doc_ids_subq = (
select(DocumentShare.document_id)
.where(DocumentShare.group_id.in_(user_groups))
.scalar_subquery()
)
conditions = [
Document.id.in_(shared_doc_ids_subq),
Document.user_id != user_id, # exclude own docs
]
if document_type:
conditions.append(Document.document_type == document_type)
if search:
like = f"%{search}%"
conditions.append(
or_(
Document.title.ilike(like),
Document.filename.ilike(like),
Document.tags.ilike(like),
Document.document_type.ilike(like),
)
)
count_result = await db.execute(
select(func.count(Document.id)).where(*conditions)
)
total = count_result.scalar_one()
items_result = await db.execute(
select(Document)
.where(*conditions)
.options(
selectinload(Document.category_assignments)
.selectinload(CategoryAssignment.category)
)
.order_by(sort_expr)
.offset((page - 1) * per_page)
.limit(per_page)
)
docs = items_result.scalars().all()
# For each doc, find which share (group) brought it in (pick first match)
share_rows_result = await db.execute(
select(DocumentShare)
.where(
DocumentShare.document_id.in_([d.id for d in docs]),
DocumentShare.group_id.in_(user_groups),
)
)
share_rows = share_rows_result.scalars().all()
# Map doc_id → first share row found
share_map: dict[str, DocumentShare] = {}
for share in share_rows:
if share.document_id not in share_map:
share_map[share.document_id] = share
from app.schemas.document import CategoryOut
items: list[SharedDocumentOut] = []
for doc in docs:
cats = [CategoryOut(id=a.category.id, name=a.category.name) for a in doc.category_assignments]
share = share_map.get(doc.id)
items.append(
SharedDocumentOut(
id=doc.id,
user_id=doc.user_id,
filename=doc.filename,
title=doc.title,
file_size=doc.file_size,
status=doc.status,
document_type=doc.document_type,
extracted_data=doc.extracted_data,
tags=doc.tags,
error_message=doc.error_message,
created_at=doc.created_at,
processed_at=doc.processed_at,
categories=cats,
source=doc.source,
shared_by_user_id=share.shared_by_user_id if share else "",
shared_via_group_id=share.group_id if share else "",
viewer_can_delete=bool(share and share.can_delete),
)
)
return DocumentPage(
items=items, # type: ignore[arg-type]
total=total,
page=page,
pages=max(1, math.ceil(total / per_page)),
)
@router.get("/{doc_id}", response_model=DocumentOut)
async def get_document(
doc_id: str,
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
is_admin: bool = Depends(get_user_is_admin),
db: AsyncSession = Depends(get_db),
) -> DocumentOut:
doc = await _get_user_doc(doc_id, user_id, db)
return _doc_with_categories(doc)
counts = await _get_share_counts([doc.id], db)
if is_admin:
viewer_can_delete = True
elif doc.user_id == user_id:
viewer_can_delete = True
else:
viewer_can_delete = bool(await _get_deletable_doc_ids([doc.id], user_groups, db))
return _doc_with_categories(doc, counts.get(doc.id, 0), viewer_can_delete=viewer_can_delete)
@router.get("/{doc_id}/status", response_model=DocumentStatusOut)
@@ -343,15 +522,64 @@ async def reprocess_document(
async def delete_document(
doc_id: str,
user_id: str = Depends(get_user_id),
is_admin: bool = Depends(get_user_is_admin),
user_groups: list[str] = Depends(get_user_groups),
user_admin_groups: list[str] = Depends(get_user_admin_groups),
db: AsyncSession = Depends(get_db),
) -> None:
if is_admin:
# Admins can delete any document — fetch unconditionally.
result = await db.execute(select(Document).where(Document.id == doc_id))
else:
# Fetch the document (owner, watch, or shared with user's groups)
result = await db.execute(
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
select(Document).where(
Document.id == doc_id,
or_(
Document.user_id == user_id,
Document.user_id == _WATCH_USER_ID,
Document.id.in_(
select(DocumentShare.document_id).where(
DocumentShare.group_id.in_(user_groups)
)
) if user_groups else False,
),
)
)
doc = result.scalar_one_or_none()
if doc is None:
raise HTTPException(status_code=404, detail="Document not found")
delete_file(doc.file_path)
is_owner = doc.user_id == user_id
if not is_owner and not is_admin:
# Check: shared with a group where the user has can_delete=True
can_delete_via_share = False
if user_groups:
share_result = await db.execute(
select(DocumentShare).where(
DocumentShare.document_id == doc_id,
DocumentShare.group_id.in_(user_groups),
DocumentShare.can_delete.is_(True),
)
)
can_delete_via_share = share_result.scalar_one_or_none() is not None
# Check: user is a group admin for a group the doc is shared with
can_delete_as_group_admin = False
if user_admin_groups:
admin_share_result = await db.execute(
select(DocumentShare).where(
DocumentShare.document_id == doc_id,
DocumentShare.group_id.in_(user_admin_groups),
)
)
can_delete_as_group_admin = admin_share_result.scalar_one_or_none() is not None
if not can_delete_via_share and not can_delete_as_group_admin:
raise HTTPException(status_code=403, detail="Not allowed to delete this document")
await delete_file(doc.storage_key)
await db.delete(doc)
await db.commit()
@@ -360,25 +588,35 @@ async def delete_document(
async def download_file(
doc_id: str,
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
db: AsyncSession = Depends(get_db),
) -> StreamingResponse:
# Allow access if: owner, watch doc, or shared with any of user's groups
result = await db.execute(
select(Document).where(
Document.id == doc_id,
or_(Document.user_id == user_id, Document.user_id == _WATCH_USER_ID),
or_(
Document.user_id == user_id,
Document.user_id == _WATCH_USER_ID,
Document.id.in_(
select(DocumentShare.document_id).where(
DocumentShare.group_id.in_(user_groups)
)
) if user_groups else False,
),
)
)
doc = result.scalar_one_or_none()
if doc is None:
raise HTTPException(status_code=404, detail="Document not found")
async def file_generator():
async with aiofiles.open(doc.file_path, "rb") as f:
while chunk := await f.read(64 * 1024):
yield chunk
try:
pdf_bytes = await storage_download(doc.storage_key)
except FileNotFoundError:
raise HTTPException(status_code=404, detail="File not found in storage")
return StreamingResponse(
file_generator(),
iter([pdf_bytes]),
media_type="application/pdf",
headers={"Content-Disposition": f'inline; filename="{doc.filename}"'},
)
@@ -391,9 +629,9 @@ async def assign_category(
doc_id: str,
cat_id: str,
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
db: AsyncSession = Depends(get_db),
) -> None:
# Verify the document is accessible (own or watch-ingested)
doc_result = await db.execute(
select(Document).where(
Document.id == doc_id,
@@ -403,15 +641,20 @@ async def assign_category(
if doc_result.scalar_one_or_none() is None:
raise HTTPException(status_code=404, detail="Document not found")
# Accept personal, group (user is a member), or system categories
cat_result = await db.execute(
select(DocumentCategory).where(
DocumentCategory.id == cat_id, DocumentCategory.user_id == user_id
DocumentCategory.id == cat_id,
or_(
DocumentCategory.user_id == user_id,
DocumentCategory.group_id.in_(user_groups) if user_groups else False,
DocumentCategory.scope == "system",
),
)
)
if cat_result.scalar_one_or_none() is None:
raise HTTPException(status_code=404, detail="Category not found")
# Upsert — ignore if already assigned
existing = await db.execute(
select(CategoryAssignment).where(
CategoryAssignment.document_id == doc_id,
@@ -430,6 +673,13 @@ async def remove_category(
user_id: str = Depends(get_user_id),
db: AsyncSession = Depends(get_db),
) -> None:
# Only the document owner may remove a category assignment
doc_result = await db.execute(
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
)
if doc_result.scalar_one_or_none() is None:
raise HTTPException(status_code=403, detail="Only the document owner can remove category assignments")
result = await db.execute(
select(CategoryAssignment).where(
CategoryAssignment.document_id == doc_id,
@@ -443,8 +693,6 @@ async def remove_category(
# ── AI suggestion confirmation ────────────────────────────────────────────────
# These endpoints allow users to confirm or reject AI suggestions on
# watch-ingested documents. No disk mutations — suggestions only update the DB.
@router.post("/{doc_id}/suggestions/folder/confirm", status_code=204)
async def confirm_folder_suggestion(
@@ -456,7 +704,6 @@ async def confirm_folder_suggestion(
if not doc.suggested_folder:
raise HTTPException(status_code=400, detail="No folder suggestion pending")
# Find or create the suggested category under the watch sentinel user
cat_result = await db.execute(
select(DocumentCategory).where(
DocumentCategory.user_id == _WATCH_USER_ID,
@@ -470,7 +717,6 @@ async def confirm_folder_suggestion(
await db.commit()
await db.refresh(cat)
# Assign if not already assigned
exists = await db.execute(
select(CategoryAssignment).where(
CategoryAssignment.document_id == doc_id,
@@ -518,3 +764,95 @@ async def reject_filename_suggestion(
doc = await _get_user_doc(doc_id, user_id, db)
doc.suggested_filename = None
await db.commit()
# ── Document sharing ──────────────────────────────────────────────────────────
@router.get("/{doc_id}/shares", response_model=list[DocumentShareOut])
async def list_document_shares(
doc_id: str,
user_id: str = Depends(get_user_id),
db: AsyncSession = Depends(get_db),
) -> list[DocumentShare]:
"""List all group shares for a document. Owner only."""
result = await db.execute(
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
)
if result.scalar_one_or_none() is None:
raise HTTPException(status_code=404, detail="Document not found")
shares_result = await db.execute(
select(DocumentShare).where(DocumentShare.document_id == doc_id)
)
return shares_result.scalars().all()
@router.post("/{doc_id}/shares", response_model=DocumentShareOut, status_code=201)
async def add_document_share(
doc_id: str,
body: DocumentShareCreate,
user_id: str = Depends(get_user_id),
user_groups: list[str] = Depends(get_user_groups),
db: AsyncSession = Depends(get_db),
) -> DocumentShare:
"""Share a document with a group. The sharing user must own the document
and must be a member of the target group."""
result = await db.execute(
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
)
if result.scalar_one_or_none() is None:
raise HTTPException(status_code=404, detail="Document not found")
if body.group_id not in user_groups:
raise HTTPException(
status_code=403,
detail="You can only share with groups you belong to",
)
# Idempotent — return existing share if already shared with this group
existing = await db.execute(
select(DocumentShare).where(
DocumentShare.document_id == doc_id,
DocumentShare.group_id == body.group_id,
)
)
share = existing.scalar_one_or_none()
if share is not None:
return share
share = DocumentShare(
document_id=doc_id,
group_id=body.group_id,
shared_by_user_id=user_id,
can_delete=body.can_delete,
)
db.add(share)
await db.commit()
await db.refresh(share)
return share
@router.delete("/{doc_id}/shares/{group_id}", status_code=204)
async def remove_document_share(
doc_id: str,
group_id: str,
user_id: str = Depends(get_user_id),
db: AsyncSession = Depends(get_db),
) -> None:
"""Remove a group share. Owner only."""
result = await db.execute(
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
)
if result.scalar_one_or_none() is None:
raise HTTPException(status_code=404, detail="Document not found")
share_result = await db.execute(
select(DocumentShare).where(
DocumentShare.document_id == doc_id,
DocumentShare.group_id == group_id,
)
)
share = share_result.scalar_one_or_none()
if share:
await db.delete(share)
await db.commit()
@@ -7,6 +7,8 @@ class CategoryOut(BaseModel):
id: str
user_id: str
name: str
scope: str = "personal"
group_id: str | None = None
created_at: datetime
model_config = {"from_attributes": True}
@@ -14,6 +16,7 @@ class CategoryOut(BaseModel):
class CategoryCreate(BaseModel):
name: str
group_id: str | None = None
class CategoryUpdate(BaseModel):
@@ -27,6 +27,8 @@ class DocumentOut(BaseModel):
watch_path: str | None = None
suggested_folder: str | None = None
suggested_filename: str | None = None
share_count: int = 0
viewer_can_delete: bool = False
model_config = {"from_attributes": True}
+44
View File
@@ -0,0 +1,44 @@
from datetime import datetime
from pydantic import BaseModel
class DocumentShareOut(BaseModel):
id: str
document_id: str
group_id: str
shared_by_user_id: str
can_delete: bool
created_at: datetime
model_config = {"from_attributes": True}
class DocumentShareCreate(BaseModel):
group_id: str
can_delete: bool = False
class SharedDocumentOut(BaseModel):
"""DocumentOut fields plus sharing context for the 'Shared with me' view."""
id: str
user_id: str
filename: str
title: str | None
file_size: int
status: str
document_type: str | None
extracted_data: str | None
tags: str | None
error_message: str | None
created_at: datetime
processed_at: datetime | None
categories: list = []
source: str = "upload"
share_count: int = 0
viewer_can_delete: bool = False
# Sharing context
shared_by_user_id: str
shared_via_group_id: str
model_config = {"from_attributes": True}
@@ -1,19 +1,20 @@
"""
Reads doc_service_config.json from the shared config volume.
Reads doc_service_config.json from the storage-service config bucket.
30-second TTL cache + env var overrides.
Env var overrides (all optional):
DOC_MAX_PDF_MB — max upload size in megabytes (e.g. "50")
"""
import asyncio
import json
import os
import time
from copy import deepcopy
from pathlib import Path
import httpx
from app.core.config import settings
_CONFIG_KEY = "doc_service_config.json"
_DEFAULT_STORAGE_CONFIG: dict = {
"watch_enabled": False,
"watch_path": "/data/watch",
@@ -63,33 +64,30 @@ _cache_at: float = 0.0
_CACHE_TTL = 30.0
def _read_config_sync() -> dict:
path = Path(settings.CONFIG_PATH)
if not path.exists():
base = deepcopy(_DEFAULT_CONFIG)
else:
with open(path) as f:
base = json.load(f)
return _apply_env_overrides(base)
def _storage_url() -> str:
return f"{settings.STORAGE_SERVICE_URL}/objects/config/{_CONFIG_KEY}"
def _read_config_sync_raw() -> dict:
"""Read without env overrides — used when we need to write back to disk."""
path = Path(settings.CONFIG_PATH)
if not path.exists():
async def _fetch_config() -> dict:
"""Fetch config from storage-service. Returns defaults if not found."""
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.get(_storage_url())
if resp.status_code == 404:
return deepcopy(_DEFAULT_CONFIG)
with open(path) as f:
return json.load(f)
resp.raise_for_status()
return resp.json()
def _write_config_sync(config: dict) -> None:
"""Atomically write config JSON to disk."""
path = Path(settings.CONFIG_PATH)
tmp = path.with_suffix(".tmp")
tmp.parent.mkdir(parents=True, exist_ok=True)
with open(tmp, "w") as f:
json.dump(config, f, indent=2)
os.replace(tmp, path)
async def _write_config(data: dict) -> None:
import json
payload = json.dumps(data, indent=2).encode()
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.put(
_storage_url(),
content=payload,
headers={"Content-Type": "application/octet-stream"},
)
resp.raise_for_status()
def _apply_env_overrides(config: dict) -> dict:
@@ -108,7 +106,8 @@ async def load_doc_config() -> dict:
now = time.monotonic()
if _cache is not None and (now - _cache_at) < _CACHE_TTL:
return _cache
data = await asyncio.to_thread(_read_config_sync)
raw = await _fetch_config()
data = _apply_env_overrides(raw)
_cache = data
_cache_at = now
return data
@@ -123,11 +122,10 @@ async def get_storage_config() -> dict:
async def save_storage_config(data: dict) -> None:
"""Merge data into the storage config block and persist to disk."""
"""Merge data into the storage config block and persist to storage-service."""
global _cache, _cache_at
raw = await asyncio.to_thread(_read_config_sync_raw)
raw = await _fetch_config()
raw.setdefault("storage", {}).update(data)
await asyncio.to_thread(_write_config_sync, raw)
# Invalidate cache so next read picks up the new values
await _write_config(raw)
_cache = None
_cache_at = 0.0
@@ -3,7 +3,7 @@ File-system watcher for the watch directory.
Uses the watchdog library to monitor a configured directory for new PDF files.
When a PDF is detected, it is automatically ingested into the document service
(copied to /data/documents, a DB record is created, and the AI pipeline runs).
(uploaded to storage-service, a DB record is created, and the AI pipeline runs).
Key design decisions:
- No-remove policy: on_deleted and on_moved events are intentionally ignored.
@@ -11,7 +11,8 @@ Key design decisions:
- Watch documents use user_id="watch" as a sentinel so they are visible to
all authenticated users in the document list.
- Subfolder names map to categories: a file at invoices/bill.pdf is assigned
to a "invoices" category (auto-created if needed).
to an "Invoices" category (auto-created if needed; folder name is converted
to PascalCase-with-dashes: "vendor-invoices""Vendor-Invoices").
- Suggestions: if ai_folder_suggestion or ai_rename_suggestion are enabled,
the relevant fields are set on the document after AI processing so users
can confirm/reject from the UI.
@@ -21,6 +22,7 @@ Key design decisions:
import asyncio
import json
import logging
import re
import uuid
from pathlib import Path
@@ -66,7 +68,10 @@ async def ingest_file(path_str: str, watch_root: Path, config: dict) -> None:
# Determine category from the first subfolder component
try:
rel = path.relative_to(watch_root)
folder_name = rel.parts[0] if len(rel.parts) > 1 else None
folder_name = (
"-".join(p.capitalize() for p in re.split(r"[-_\s]+", rel.parts[0]) if p)
if len(rel.parts) > 1 else None
)
except ValueError:
folder_name = None
@@ -77,13 +82,13 @@ async def ingest_file(path_str: str, watch_root: Path, config: dict) -> None:
logger.warning("[watcher] Cannot read %s: %s", path_str, exc)
return
# Save a copy to /data/documents/watch/{doc_id}.pdf
# Upload to storage-service under documents/watch/{doc_id}.pdf
doc_id = existing.id if existing is not None else str(uuid.uuid4())
dest = await save_upload(file_data, WATCH_USER_ID, doc_id)
storage_key = await save_upload(file_data, WATCH_USER_ID, doc_id)
if existing is not None:
# Re-ingest a previously failed document
existing.file_path = str(dest)
existing.storage_key = storage_key
existing.file_size = len(file_data)
existing.status = "pending"
existing.error_message = None
@@ -95,7 +100,7 @@ async def ingest_file(path_str: str, watch_root: Path, config: dict) -> None:
source="watch",
watch_path=path_str,
filename=path.name,
file_path=str(dest),
storage_key=storage_key,
file_size=len(file_data),
status="pending",
)
+51 -17
View File
@@ -1,27 +1,61 @@
import asyncio
from pathlib import Path
"""
Storage client for the storage-service HTTP API.
import aiofiles
All persistent file I/O goes through storage-service:8020.
The bucket for all document PDFs is 'documents'.
Keys follow the pattern:
uploaded: {user_id}/{doc_id}.pdf
watch-ingested: watch/{doc_id}.pdf
"""
import logging
import httpx
from app.core.config import settings
logger = logging.getLogger(__name__)
def get_upload_path(user_id: str, doc_id: str) -> Path:
"""Return /data/documents/{user_id}/{doc_id}.pdf, creating the directory if needed."""
user_dir = Path(settings.DATA_DIR) / user_id
user_dir.mkdir(parents=True, exist_ok=True)
return user_dir / f"{doc_id}.pdf"
_BUCKET = "documents"
async def save_upload(file_data: bytes, user_id: str, doc_id: str) -> Path:
dest = get_upload_path(user_id, doc_id)
async with aiofiles.open(dest, "wb") as f:
await f.write(file_data)
return dest
def _storage_url(key: str) -> str:
return f"{settings.STORAGE_SERVICE_URL}/objects/{_BUCKET}/{key}"
def delete_file(file_path: str) -> None:
def build_storage_key(user_id: str, doc_id: str) -> str:
"""Return the canonical storage key for a document."""
return f"{user_id}/{doc_id}.pdf"
async def save_upload(file_data: bytes, user_id: str, doc_id: str) -> str:
"""Upload bytes to storage-service. Returns the storage key."""
key = build_storage_key(user_id, doc_id)
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.put(
_storage_url(key),
content=file_data,
headers={"Content-Type": "application/octet-stream"},
)
resp.raise_for_status()
return key
async def download_file(storage_key: str) -> bytes:
"""Download bytes from storage-service by storage key."""
async with httpx.AsyncClient(timeout=60.0) as client:
resp = await client.get(_storage_url(storage_key))
if resp.status_code == 404:
raise FileNotFoundError(f"Object not found: {storage_key}")
resp.raise_for_status()
return resp.content
async def delete_file(storage_key: str) -> None:
"""Delete an object from storage-service. Swallows errors — deletion failure must not 500."""
try:
Path(file_path).unlink(missing_ok=True)
except OSError:
pass # log but do not raise — deletion failure must not 500
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.delete(_storage_url(storage_key))
if resp.status_code not in (204, 404):
logger.warning("storage-service DELETE returned %s for key %s", resp.status_code, storage_key)
except Exception as exc:
logger.warning("Could not delete %s from storage-service: %s", storage_key, exc)
+115
View File
@@ -0,0 +1,115 @@
# storage-service — Claude context
Unified file/blob storage microservice, port 8020 (internal). All services must use this service's
HTTP API for any file persistence — no service may write to a Docker volume directly. See root
`CLAUDE.md` for architecture, Docker, and project-wide workflows.
---
## Architecture rule (enforced)
**No service may write to a filesystem path for persistent data.**
All file/blob storage must go through the storage-service HTTP API.
Violation is a security/architecture defect.
---
## File & Folder Tree
```
features/storage-service/
├── app/
│ ├── main.py ← FastAPI, lifespan (backend init)
│ ├── core/config.py ← Settings (DATA_DIR, STORAGE_BACKEND, S3_*, WEBDAV_*)
│ ├── routers/
│ │ ├── health.py ← GET /health
│ │ ├── objects.py ← PUT/GET/DELETE /objects/{bucket}/{key:path}, GET /objects/{bucket}
│ │ └── migrate.py ← POST /migrate, GET /migrate/status, DELETE /migrate, PATCH /backend-config
│ └── services/
│ ├── backend_manager.py ← build_backend(), initialize_backend(), get_backend(), switch_backend()
│ ├── migration.py ← run_migration(), get_status(), cancel(); KNOWN_BUCKETS
│ └── backends/
│ ├── base.py ← AbstractStorageBackend (ABC)
│ ├── local.py ← LocalFSBackend — /data/storage/{bucket}/{key}
│ ├── s3.py ← S3Backend — aiobotocore, endpoint_url configurable
│ └── webdav.py ← WebDAVBackend — aiohttp + WebDAV PROPFIND/PUT/GET/DELETE
├── scripts/
│ ├── start.sh ← prod start (uvicorn port 8020)
│ └── start_dev.sh ← dev start (uvicorn --reload)
├── Dockerfile ← python:3.12-slim, non-root user 1001
└── STATUS.md
```
---
## HTTP API
### Objects
| Method | Path | Body | Response |
|--------|------|------|----------|
| PUT | `/objects/{bucket}/{key:path}` | Raw bytes | 204 |
| GET | `/objects/{bucket}/{key:path}` | — | 200 Raw bytes / 404 |
| DELETE | `/objects/{bucket}/{key:path}` | — | 204 |
| GET | `/objects/{bucket}` | — | `{"bucket": "...", "keys": [...]}` |
Keys may contain `/` (e.g. `user123/abc.pdf`). Path traversal (`..`) returns 400.
### Migration
| Method | Path | Body | Response |
|--------|------|------|----------|
| POST | `/migrate` | `{"driver": "s3", "config": {...}}` | 202 / 400 / 409 |
| GET | `/migrate/status` | — | `{state, total, done, failed, errors[]}` |
| DELETE | `/migrate` | — | 204 / 409 |
| PATCH | `/backend-config` | `{"driver": "...", "config": {...}}` | 204 / 400 / 409 |
Migration states: `idle → validating → migrating → switching → cleaning → done` (or `failed`/`cancelled`)
### Health
| Method | Path | Response |
|--------|------|----------|
| GET | `/health` | `{"status": "ok", "backend": "local"}` |
---
## Buckets
| Bucket | Contents | Key format |
|--------|----------|------------|
| `documents` | Uploaded PDFs | `{user_id}/{doc_id}.pdf` or `watch/{doc_id}.pdf` |
| `config` | JSON config files | `{service_name}_config.json` |
To add a new bucket: add it to `KNOWN_BUCKETS` in `services/migration.py` so it is included in migrations.
---
## Backend drivers
| Driver | Config fields | Notes |
|--------|---------------|-------|
| `local` | `data_dir` (optional) | Default. Files under `/data/storage/`. Zero external deps. |
| `s3` | `endpoint_url`, `access_key`, `secret_key`, `region` | Works with MinIO, AWS S3, Backblaze B2, Cloudflare R2. Set `endpoint_url=""` for real AWS. |
| `webdav` | `url`, `username`, `password`, `root_path` | Nextcloud: set root_path to `/remote.php/dav/files/{username}` |
---
## Adding a new backend driver
1. Create `app/services/backends/your_driver.py` implementing `AbstractStorageBackend`
2. Add a branch in `build_backend()` in `backend_manager.py`
3. Add config fields to `app/core/config.py` if env-based config is needed
4. Document driver name + config fields in this file
---
## Default Values & Limits
| Parameter | Value | Location |
|-----------|-------|----------|
| Default backend | `local` | `STORAGE_BACKEND` env var |
| Local data dir | `/data/storage` | `DATA_DIR` env var |
| S3 region default | `us-east-1` | `S3_REGION` env var |
| Migration error cap in response | 50 | `migration.py` |
| Port | 8020 | `scripts/start.sh` |
+32
View File
@@ -0,0 +1,32 @@
# ── Stage 1: dependency installation ─────────────────────────────────────────
FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --upgrade pip
COPY pyproject.toml .
RUN pip install --prefix=/install .
# ── Stage 2: runtime ──────────────────────────────────────────────────────────
FROM python:3.12-slim
# Create non-root user (UID/GID 1001)
RUN groupadd --gid 1001 appuser && \
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
# Pre-create data dir with correct ownership.
# Named volume mounted over this path will inherit ownership on first creation.
RUN mkdir -p /data/storage && chown -R appuser:appuser /data
WORKDIR /app
COPY --from=builder /install /usr/local
COPY --chown=appuser:appuser app ./app
COPY --chown=appuser:appuser scripts ./scripts
USER appuser
EXPOSE 8020
CMD ["sh", "scripts/start.sh"]
+83
View File
@@ -0,0 +1,83 @@
# Storage Service — Status
## What it is
Unified file/blob storage microservice, port 8020 (internal). All services store and retrieve files
through its HTTP API — no service writes to a Docker volume directly. Uses a pluggable backend
driver (local FS by default; S3-compatible and WebDAV available). Backend is switchable at runtime
via admin settings with automatic data migration.
## Current functionality
### Object API (`/objects`)
| Method | Path | Description |
|--------|------|-------------|
| PUT | `/objects/{bucket}/{key}` | Upload raw bytes |
| GET | `/objects/{bucket}/{key}` | Download raw bytes |
| DELETE | `/objects/{bucket}/{key}` | Delete object |
| GET | `/objects/{bucket}` | List all keys in bucket |
### Migration API (`/migrate`)
| Method | Path | Description |
|--------|------|-------------|
| POST | `/migrate` | Start migration to a new backend (validates, copies, switches, cleans) |
| GET | `/migrate/status` | Poll migration progress |
| DELETE | `/migrate` | Cancel in-progress migration |
| PATCH | `/backend-config` | Reconfigure backend without migrating data |
### Health
| Method | Path | Description |
|--------|------|-------------|
| GET | `/health` | `{"status": "ok", "backend": "<driver>"}` |
### Buckets
| Bucket | Contents |
|--------|----------|
| `documents` | Uploaded PDFs (keyed as `{user_id}/{doc_id}.pdf` or `watch/{doc_id}.pdf`) |
| `config` | JSON config files (replaces `app_config` volume — Phase 3) |
## Architecture
```
backend / doc-service / future-svc
│ HTTP
storage-service:8020
backend_manager
┌──────┴──────────────────┐
│ │
LocalFSBackend S3Backend / WebDAVBackend
/data/storage/ (configured via admin UI)
{bucket}/{key}
```
Migration flow:
```
POST /migrate { driver, config }
→ test_connection() (validate)
→ list all objects in KNOWN_BUCKETS (enumerate)
→ GET old / PUT new / exists verify (copy + verify, per object)
→ if 0 failures: switch_backend() (atomic switch)
→ DELETE old objects (cleanup)
```
## Known limitations / not implemented
- Migration state is in-memory — a container restart during migration loses progress (restart restarts from scratch)
- No presigned URL support (direct client downloads go through the API)
- rclone backends (Google Drive, OneDrive, Dropbox) not yet implemented
- No per-object metadata or content-type headers
- No multipart upload for very large files (> available RAM)
## Future work
- [ ] rclone-based backend adapter (GDrive, OneDrive, Dropbox)
- [ ] Presigned URL generation for direct browser downloads
- [ ] Persist migration state to DB so restarts can resume
- [ ] Streaming upload/download to avoid buffering entire file in memory
- [ ] Per-bucket access policies
@@ -0,0 +1,23 @@
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
DATA_DIR: str = "/data/storage"
STORAGE_BACKEND: str = "local" # local | s3 | webdav
# S3-compatible (MinIO, AWS S3, Backblaze B2, Cloudflare R2, …)
S3_ENDPOINT_URL: str = "" # leave empty for real AWS S3
S3_ACCESS_KEY: str = ""
S3_SECRET_KEY: str = ""
S3_REGION: str = "us-east-1"
# WebDAV (Nextcloud, …)
WEBDAV_URL: str = ""
WEBDAV_USERNAME: str = ""
WEBDAV_PASSWORD: str = ""
WEBDAV_ROOT_PATH: str = "/"
settings = Settings()
+29
View File
@@ -0,0 +1,29 @@
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from app.core.config import settings
from app.routers import health, objects, migrate
from app.services.backend_manager import initialize_backend
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
initialize_backend()
logger.info("storage-service started (backend=%s)", settings.STORAGE_BACKEND)
yield
logger.info("storage-service shutting down")
app = FastAPI(title="Storage Service", lifespan=lifespan)
app.include_router(health.router)
app.include_router(objects.router)
app.include_router(migrate.router)
@@ -0,0 +1,10 @@
from fastapi import APIRouter
from app.services.backend_manager import get_backend
router = APIRouter()
@router.get("/health")
async def health():
return {"status": "ok", "backend": get_backend().driver_name}
@@ -0,0 +1,88 @@
import logging
from fastapi import APIRouter, BackgroundTasks, HTTPException
from pydantic import BaseModel
from app.services import migration
from app.services.backend_manager import build_backend, switch_backend
router = APIRouter()
logger = logging.getLogger(__name__)
class MigrateRequest(BaseModel):
driver: str
config: dict = {}
class BackendConfigRequest(BaseModel):
driver: str
config: dict = {}
@router.post("/migrate", status_code=202)
async def start_migration(body: MigrateRequest, background_tasks: BackgroundTasks):
"""
Validate the new backend, then start an async migration job that:
1. Copies all objects from the current backend to the new one
2. Verifies every object
3. Atomically switches the active backend
4. Deletes all objects from the old backend
Returns 409 if a migration is already in progress.
Returns 400 if the new backend config fails validation.
"""
if migration.is_in_progress():
raise HTTPException(status_code=409, detail="A migration is already in progress")
# Reset status and enter validating state before any async work
migration._status.state = "validating"
migration._status.total = 0
migration._status.done = 0
migration._status.failed = 0
migration._status.errors.clear()
try:
new_backend = build_backend(body.driver, body.config)
await new_backend.test_connection()
except Exception as exc:
migration._status.state = "idle"
raise HTTPException(status_code=400, detail=f"Backend validation failed: {exc}")
background_tasks.add_task(migration.run_migration, new_backend)
return {"status": "started", "driver": body.driver}
@router.get("/migrate/status")
async def migration_status():
"""Poll this to track migration progress."""
return migration.get_status()
@router.delete("/migrate", status_code=204)
async def cancel_migration():
"""
Request cancellation of a running migration.
The old backend remains active. Returns 409 if no migration is running.
"""
cancelled = await migration.cancel()
if not cancelled:
raise HTTPException(status_code=409, detail="No cancellable migration in progress")
@router.patch("/backend-config", status_code=204)
async def update_backend_config(body: BackendConfigRequest):
"""
Reconfigure the active backend without migrating data (e.g. update S3 credentials
for the same endpoint, or switch back to local after a failed migration).
Use POST /migrate when you need data to be moved to the new backend.
"""
if migration.is_in_progress():
raise HTTPException(status_code=409, detail="Cannot reconfigure while migration is in progress")
try:
new_backend = build_backend(body.driver, body.config)
await new_backend.test_connection()
except Exception as exc:
raise HTTPException(status_code=400, detail=f"Backend validation failed: {exc}")
switch_backend(new_backend)
@@ -0,0 +1,60 @@
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import Response
from app.services.backend_manager import get_backend
router = APIRouter()
def _validate_key(key: str) -> str:
"""Reject path traversal. Key may contain '/' for nested objects (e.g. user/doc.pdf)."""
parts = key.split("/")
if ".." in parts:
raise HTTPException(status_code=400, detail="Invalid key: path traversal not allowed")
return key
@router.put("/objects/{bucket}/{key:path}", status_code=204)
async def put_object(bucket: str, key: str, request: Request):
"""Upload raw bytes. Body is read as-is (application/octet-stream)."""
_validate_key(key)
data = await request.body()
try:
await get_backend().put(bucket, key, data)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc))
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
@router.get("/objects/{bucket}/{key:path}")
async def get_object(bucket: str, key: str):
"""Download raw bytes."""
_validate_key(key)
try:
data = await get_backend().get(bucket, key)
except KeyError:
raise HTTPException(status_code=404, detail="Object not found")
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
return Response(content=data, media_type="application/octet-stream")
@router.delete("/objects/{bucket}/{key:path}", status_code=204)
async def delete_object(bucket: str, key: str):
"""Delete an object. No-op if it does not exist."""
_validate_key(key)
try:
await get_backend().delete(bucket, key)
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
@router.get("/objects/{bucket}")
async def list_objects(bucket: str):
"""List all keys in a bucket."""
try:
keys = await get_backend().list_keys(bucket)
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
return {"bucket": bucket, "keys": keys}
@@ -0,0 +1,70 @@
import logging
from app.core.config import settings
from app.services.backends.base import AbstractStorageBackend
from app.services.backends.local import LocalFSBackend
from app.services.backends.s3 import S3Backend
from app.services.backends.webdav import WebDAVBackend
logger = logging.getLogger(__name__)
_active_backend: AbstractStorageBackend | None = None
def build_backend(driver: str, config: dict) -> AbstractStorageBackend:
"""Construct a backend instance from a driver name + config dict."""
if driver == "local":
return LocalFSBackend(data_dir=config.get("data_dir", settings.DATA_DIR))
if driver == "s3":
return S3Backend(
endpoint_url=config.get("endpoint_url", ""),
access_key=config.get("access_key", ""),
secret_key=config.get("secret_key", ""),
region=config.get("region", "us-east-1"),
)
if driver == "webdav":
return WebDAVBackend(
url=config.get("url", ""),
username=config.get("username", ""),
password=config.get("password", ""),
root_path=config.get("root_path", "/"),
)
raise ValueError(f"Unknown driver: {driver!r}. Valid options: local, s3, webdav")
def initialize_backend() -> None:
"""Build the initial backend from environment variables at startup."""
global _active_backend
driver = settings.STORAGE_BACKEND
config: dict = {}
if driver == "s3":
config = {
"endpoint_url": settings.S3_ENDPOINT_URL,
"access_key": settings.S3_ACCESS_KEY,
"secret_key": settings.S3_SECRET_KEY,
"region": settings.S3_REGION,
}
elif driver == "webdav":
config = {
"url": settings.WEBDAV_URL,
"username": settings.WEBDAV_USERNAME,
"password": settings.WEBDAV_PASSWORD,
"root_path": settings.WEBDAV_ROOT_PATH,
}
# local needs no extra config — DATA_DIR is read from settings inside build_backend
_active_backend = build_backend(driver, config)
logger.info("Storage backend initialized: %s", driver)
def get_backend() -> AbstractStorageBackend:
if _active_backend is None:
raise RuntimeError("Backend not initialized — call initialize_backend() at startup")
return _active_backend
def switch_backend(new_backend: AbstractStorageBackend) -> None:
"""Replace the active backend. Called by the migration job after all data is verified."""
global _active_backend
old_name = _active_backend.driver_name if _active_backend else "none"
_active_backend = new_backend
logger.info("Storage backend switched: %s%s", old_name, new_backend.driver_name)
@@ -0,0 +1,34 @@
from abc import ABC, abstractmethod
class AbstractStorageBackend(ABC):
"""Common interface every storage backend must implement."""
@property
@abstractmethod
def driver_name(self) -> str:
"""Short identifier returned in /health: 'local', 's3', or 'webdav'."""
@abstractmethod
async def put(self, bucket: str, key: str, data: bytes) -> None:
"""Store *data* under bucket/key. Creates bucket/intermediate dirs as needed."""
@abstractmethod
async def get(self, bucket: str, key: str) -> bytes:
"""Return the stored bytes. Raises KeyError if the object does not exist."""
@abstractmethod
async def delete(self, bucket: str, key: str) -> None:
"""Delete the object. No-op if it does not exist."""
@abstractmethod
async def list_keys(self, bucket: str) -> list[str]:
"""Return all keys stored in *bucket*. Returns [] if bucket is empty/absent."""
@abstractmethod
async def exists(self, bucket: str, key: str) -> bool:
"""Return True if the object exists."""
@abstractmethod
async def test_connection(self) -> None:
"""Verify the backend is reachable and writable. Raise on failure."""
@@ -0,0 +1,67 @@
import asyncio
from pathlib import Path
import aiofiles
from .base import AbstractStorageBackend
class LocalFSBackend(AbstractStorageBackend):
"""Stores objects as files under <data_dir>/<bucket>/<key>."""
def __init__(self, data_dir: str) -> None:
self._root = Path(data_dir)
self._root.mkdir(parents=True, exist_ok=True)
@property
def driver_name(self) -> str:
return "local"
def _resolve(self, bucket: str, key: str) -> Path:
safe_key = key.lstrip("/")
if ".." in safe_key.split("/"):
raise ValueError(f"Invalid key: {key!r}")
return self._root / bucket / safe_key
async def put(self, bucket: str, key: str, data: bytes) -> None:
dest = self._resolve(bucket, key)
await asyncio.to_thread(dest.parent.mkdir, parents=True, exist_ok=True)
async with aiofiles.open(dest, "wb") as f:
await f.write(data)
async def get(self, bucket: str, key: str) -> bytes:
path = self._resolve(bucket, key)
if not path.exists():
raise KeyError(f"{bucket}/{key}")
async with aiofiles.open(path, "rb") as f:
return await f.read()
async def delete(self, bucket: str, key: str) -> None:
path = self._resolve(bucket, key)
try:
await asyncio.to_thread(path.unlink, missing_ok=True)
except OSError:
pass
async def list_keys(self, bucket: str) -> list[str]:
bucket_dir = self._root / bucket
if not bucket_dir.exists():
return []
def _scan() -> list[str]:
return [
str(p.relative_to(bucket_dir))
for p in bucket_dir.rglob("*")
if p.is_file()
]
return await asyncio.to_thread(_scan)
async def exists(self, bucket: str, key: str) -> bool:
return self._resolve(bucket, key).exists()
async def test_connection(self) -> None:
self._root.mkdir(parents=True, exist_ok=True)
probe = self._root / ".health_probe"
probe.write_bytes(b"ok")
probe.unlink()
@@ -0,0 +1,99 @@
import logging
from contextlib import asynccontextmanager
from aiobotocore.session import get_session
from .base import AbstractStorageBackend
logger = logging.getLogger(__name__)
class S3Backend(AbstractStorageBackend):
"""
S3-compatible backend. Works with AWS S3, MinIO, Backblaze B2, Cloudflare R2, etc.
Set endpoint_url to the service URL for non-AWS providers; leave empty for real AWS.
"""
def __init__(
self,
endpoint_url: str,
access_key: str,
secret_key: str,
region: str = "us-east-1",
) -> None:
self._endpoint_url = endpoint_url or None
self._access_key = access_key
self._secret_key = secret_key
self._region = region
self._session = get_session()
@property
def driver_name(self) -> str:
return "s3"
@asynccontextmanager
async def _client(self):
async with self._session.create_client(
"s3",
endpoint_url=self._endpoint_url,
aws_access_key_id=self._access_key,
aws_secret_access_key=self._secret_key,
region_name=self._region,
) as client:
yield client
async def _ensure_bucket(self, client, bucket: str) -> None:
try:
await client.head_bucket(Bucket=bucket)
except Exception:
try:
if self._region == "us-east-1":
await client.create_bucket(Bucket=bucket)
else:
await client.create_bucket(
Bucket=bucket,
CreateBucketConfiguration={"LocationConstraint": self._region},
)
except Exception as exc:
logger.debug("Bucket create skipped (may already exist): %s", exc)
async def put(self, bucket: str, key: str, data: bytes) -> None:
async with self._client() as client:
await self._ensure_bucket(client, bucket)
await client.put_object(Bucket=bucket, Key=key, Body=data)
async def get(self, bucket: str, key: str) -> bytes:
async with self._client() as client:
try:
response = await client.get_object(Bucket=bucket, Key=key)
return await response["Body"].read()
except Exception as exc:
raise KeyError(f"{bucket}/{key}") from exc
async def delete(self, bucket: str, key: str) -> None:
async with self._client() as client:
await client.delete_object(Bucket=bucket, Key=key)
async def list_keys(self, bucket: str) -> list[str]:
async with self._client() as client:
try:
paginator = client.get_paginator("list_objects_v2")
keys: list[str] = []
async for page in paginator.paginate(Bucket=bucket):
for obj in page.get("Contents", []):
keys.append(obj["Key"])
return keys
except Exception:
return []
async def exists(self, bucket: str, key: str) -> bool:
async with self._client() as client:
try:
await client.head_object(Bucket=bucket, Key=key)
return True
except Exception:
return False
async def test_connection(self) -> None:
async with self._client() as client:
await client.list_buckets()
@@ -0,0 +1,121 @@
import base64
from urllib.parse import quote
import defusedxml.ElementTree as ET
import aiohttp
from .base import AbstractStorageBackend
class WebDAVBackend(AbstractStorageBackend):
"""
WebDAV backend. Compatible with Nextcloud and any standard WebDAV server.
root_path should be the WebDAV root on the server, e.g. '/remote.php/dav/files/username'.
"""
def __init__(
self,
url: str,
username: str,
password: str,
root_path: str = "/",
) -> None:
self._base = url.rstrip("/")
self._root = root_path.rstrip("/")
creds = base64.b64encode(f"{username}:{password}".encode()).decode()
self._auth = f"Basic {creds}"
@property
def driver_name(self) -> str:
return "webdav"
def _url(self, *parts: str) -> str:
encoded = "/".join(quote(p, safe="") for p in parts)
return f"{self._base}{self._root}/{encoded}"
def _headers(self, extra: dict | None = None) -> dict[str, str]:
h = {"Authorization": self._auth}
if extra:
h.update(extra)
return h
async def _ensure_collection(self, session: aiohttp.ClientSession, *parts: str) -> None:
"""MKCOL is idempotent — ignore 405 (already exists)."""
url = self._url(*parts)
async with session.request("MKCOL", url, headers=self._headers()) as resp:
if resp.status not in (200, 201, 405):
pass # best-effort; PUT will fail if directory is truly missing
async def put(self, bucket: str, key: str, data: bytes) -> None:
async with aiohttp.ClientSession() as session:
await self._ensure_collection(session, bucket)
parts = key.split("/")
for i in range(1, len(parts)):
await self._ensure_collection(session, bucket, *parts[:i])
url = self._url(bucket, key)
async with session.put(url, data=data, headers=self._headers()) as resp:
if resp.status not in (200, 201, 204):
raise OSError(f"WebDAV PUT {url}{resp.status}")
async def get(self, bucket: str, key: str) -> bytes:
async with aiohttp.ClientSession() as session:
url = self._url(bucket, key)
async with session.get(url, headers=self._headers()) as resp:
if resp.status == 404:
raise KeyError(f"{bucket}/{key}")
if resp.status != 200:
raise OSError(f"WebDAV GET {url}{resp.status}")
return await resp.read()
async def delete(self, bucket: str, key: str) -> None:
async with aiohttp.ClientSession() as session:
url = self._url(bucket, key)
async with session.delete(url, headers=self._headers()) as resp:
if resp.status not in (200, 204, 404):
raise OSError(f"WebDAV DELETE {url}{resp.status}")
async def list_keys(self, bucket: str) -> list[str]:
async with aiohttp.ClientSession() as session:
url = self._url(bucket)
headers = self._headers({"Depth": "infinity", "Content-Type": "application/xml"})
body = '<?xml version="1.0"?><d:propfind xmlns:d="DAV:"><d:prop><d:resourcetype/></d:prop></d:propfind>'
async with session.request("PROPFIND", url, headers=headers, data=body) as resp:
if resp.status == 404:
return []
if resp.status != 207:
return []
xml_body = await resp.text()
ns = {"d": "DAV:"}
try:
root = ET.fromstring(xml_body)
except ET.ParseError:
return []
prefix = f"{self._base}{self._root}/{quote(bucket, safe='')}/"
keys: list[str] = []
for response in root.findall("d:response", ns):
href = response.findtext("d:href", namespaces=ns) or ""
prop = response.find(".//d:prop", ns)
if prop is not None:
rt = prop.find("d:resourcetype", ns)
if rt is not None and rt.find("d:collection", ns) is not None:
continue # skip directories
if href.startswith(prefix):
keys.append(href[len(prefix):])
return keys
async def exists(self, bucket: str, key: str) -> bool:
async with aiohttp.ClientSession() as session:
url = self._url(bucket, key)
async with session.request("HEAD", url, headers=self._headers()) as resp:
return resp.status == 200
async def test_connection(self) -> None:
async with aiohttp.ClientSession() as session:
root_url = f"{self._base}{self._root}/"
headers = self._headers({"Depth": "0"})
async with session.request("PROPFIND", root_url, headers=headers) as resp:
if resp.status not in (200, 207):
raise ConnectionError(f"WebDAV root PROPFIND → {resp.status}")
@@ -0,0 +1,139 @@
"""
Backend migration service.
Flow:
1. POST /migrate → validate new backend (test_connection)
2. Background task enumerates all objects in all known buckets
3. Each object is copied old → new, then verified
4. Only after 100 % success: atomically switch active backend
5. Delete all objects from old backend
6. If any copy fails: old backend stays active; state = "failed"
7. DELETE /migrate cancels a running migration (old backend stays active)
"""
import logging
from dataclasses import dataclass, field
from typing import Literal
from app.services.backends.base import AbstractStorageBackend
from app.services.backend_manager import get_backend, switch_backend
logger = logging.getLogger(__name__)
# All logical buckets the service knows about — enumerated during migration.
KNOWN_BUCKETS = ["documents", "config"]
MigrationState = Literal[
"idle", "validating", "migrating", "switching", "cleaning", "done", "failed", "cancelled"
]
@dataclass
class _MigrationStatus:
state: MigrationState = "idle"
total: int = 0
done: int = 0
failed: int = 0
errors: list[str] = field(default_factory=list)
_status = _MigrationStatus()
_cancel_requested: bool = False
def get_status() -> dict:
return {
"state": _status.state,
"total": _status.total,
"done": _status.done,
"failed": _status.failed,
"errors": _status.errors[:50], # cap to avoid huge responses
}
def is_in_progress() -> bool:
return _status.state in ("validating", "migrating", "switching", "cleaning")
async def cancel() -> bool:
global _cancel_requested
if _status.state == "migrating":
_cancel_requested = True
return True
return False
async def run_migration(new_backend: AbstractStorageBackend) -> None:
"""
Background task: copy all objects to new_backend, verify, switch, clean old.
Called after the caller has already validated new_backend.test_connection().
"""
global _cancel_requested
_cancel_requested = False
old_backend = get_backend()
_status.state = "migrating"
_status.done = 0
_status.failed = 0
_status.errors.clear()
try:
# Collect all objects across every known bucket
all_objects: list[tuple[str, str]] = []
for bucket in KNOWN_BUCKETS:
try:
keys = await old_backend.list_keys(bucket)
for key in keys:
all_objects.append((bucket, key))
except Exception as exc:
logger.warning("Could not list bucket %r: %s", bucket, exc)
_status.total = len(all_objects)
logger.info("Migration: %d objects to migrate across %d buckets", len(all_objects), len(KNOWN_BUCKETS))
for bucket, key in all_objects:
if _cancel_requested:
_status.state = "cancelled"
logger.info("Migration cancelled (%d/%d done)", _status.done, _status.total)
return
try:
data = await old_backend.get(bucket, key)
await new_backend.put(bucket, key, data)
if not await new_backend.exists(bucket, key):
raise OSError("Verification failed: object absent after PUT")
_status.done += 1
except Exception as exc:
_status.failed += 1
entry = f"{bucket}/{key}: {exc}"
_status.errors.append(entry)
logger.warning("Migration copy failed — %s", entry)
if _status.failed > 0:
_status.state = "failed"
logger.error(
"Migration failed: %d/%d objects could not be copied; old backend remains active",
_status.failed,
_status.total,
)
return
# All objects verified — atomically switch
_status.state = "switching"
switch_backend(new_backend)
# Remove all objects from old backend (best-effort)
_status.state = "cleaning"
for bucket, key in all_objects:
try:
await old_backend.delete(bucket, key)
except Exception as exc:
logger.warning("Cleanup failed for %s/%s: %s", bucket, key, exc)
_status.state = "done"
logger.info("Migration complete: %d objects moved to %s", _status.total, new_backend.driver_name)
except Exception as exc:
_status.state = "failed"
_status.errors.append(f"Unexpected error: {exc}")
logger.exception("Migration aborted with unexpected error")
+25
View File
@@ -0,0 +1,25 @@
[build-system]
requires = ["setuptools>=45"]
build-backend = "setuptools.build_meta"
[project]
name = "storage-service"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
"fastapi>=0.111",
"uvicorn[standard]>=0.29",
"pydantic-settings>=2.2",
"aiofiles>=23.0",
"aiobotocore>=2.13",
"aiohttp>=3.9",
"defusedxml>=0.7",
]
[project.optional-dependencies]
dev = [
"ruff>=0.4",
]
[tool.ruff]
line-length = 100
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
set -e
echo "[storage-service] starting uvicorn..."
exec uvicorn app.main:app --host 0.0.0.0 --port 8020
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
set -e
echo "[storage-service] starting uvicorn (dev)..."
exec uvicorn app.main:app --host 0.0.0.0 --port 8020 --reload
+190
View File
@@ -0,0 +1,190 @@
# frontend — Claude context
React 18 SPA built with Vite, port 5173 dev / 80 prod, served by nginx-unprivileged in production. All `/api/*` requests are proxied to `backend:8000`. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
---
## Commands
All commands run inside Docker — never on the host.
```bash
docker compose exec frontend npm run typecheck
docker compose exec frontend npm run lint
```
---
## File & Folder Tree
```
frontend/
├── src/
│ ├── main.tsx ← React root, QueryClientProvider, BrowserRouter
│ ├── App.tsx ← Route tree, PrivateRoute, AdminRoute
│ ├── api/client.ts ← Native fetch wrapper (`request()`) + ALL API functions (single source of truth); no Axios
│ ├── hooks/
│ │ ├── useAuth.ts ← Token state (localStorage), login/logout
│ │ └── useTheme.ts ← Theme toggle
│ ├── components/
│ │ ├── AppShell.tsx ← Layout: Sidebar + SourcePanel (on /apps/documents) + main
│ │ ├── Sidebar.tsx ← Collapsible nav (icons ↔ icons+labels)
│ │ ├── SourcePanel.tsx ← Views + searchable category tree (docs route only)
│ │ ├── ManageCategoriesDialog.tsx ← Category CRUD modal (rename, delete)
│ │ ├── DocumentSlideOver.tsx ← Right slide-over: detail, edit, share, AI suggestions
│ │ ├── ThemeToggle.tsx ← Light/dark mode toggle
│ │ ├── PluginSchemaForm.tsx ← JSON Schema → React form (boolean/string/number/readOnly)
│ │ └── ui/ ← shadcn/ui components (Button, Input, …)
│ ├── pages/ ← One file per route
│ │ ├── DocServiceSettingsPage.tsx ← Combined doc-service settings: upload limits + watch directory
│ │ ├── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest
│ │ └── StorageAdminPage.tsx ← Admin storage backend config + live migration progress
│ ├── lib/utils.ts ← cn() = clsx + tailwind-merge
│ └── styles/theme.css ← CSS custom properties, Tailwind setup
├── vite.config.ts ← /api/* proxied to backend:8000
├── tailwind.config.ts
├── components.json ← shadcn/ui config
├── Dockerfile ← Multi-stage: Node build → nginx-unprivileged
└── STATUS.md
```
---
## Frontend Routes
| Path | Component | Guard |
|------|-----------|-------|
| `/login` | `LoginPage` | Public |
| `/` | `DashboardPage` | PrivateRoute |
| `/apps` | `AppsPage` | PrivateRoute |
| `/apps/documents` | `DocumentsPage` | PrivateRoute |
| `/apps/documents/settings` | `DocServiceSettingsPage` | ServiceAdminRoute (is_admin OR doc-service-admin member) |
| `/apps/ai/settings` | `AIAdminSettingsPage` | ServiceAdminRoute (is_admin OR ai-service-admin member) |
| `/profile` | `ProfilePage` | PrivateRoute |
| `/settings` | `SettingsPage` | PrivateRoute |
| `/settings/plugins/:id` | `PluginSettingsPage` | PrivateRoute (auth enforced per-plugin by backend) |
| `/admin` | `AdminPage` (→ `/admin/users`) | AdminRoute |
| `/admin/users` | `AdminUsersPage` | AdminRoute |
| `/admin/groups` | `AdminGroupsPage` | AdminRoute |
| `/admin/appearance` | `AdminAppearancePage` | AdminRoute |
| `/admin/storage` | `StorageAdminPage` | AdminRoute |
| `*` | redirect to `/` | — |
`PrivateRoute` — checks `token` from `useAuth`, redirects to `/login` if absent.
`AdminRoute` — checks token AND queries `GET /api/users/me` for `is_admin`; waits for query to avoid flash; redirects to `/login` (not `/`) if not admin.
---
## Security Standards
### XSS prevention
- React JSX text interpolation (`{value}`) is HTML-escaped by the DOM renderer — **never** use `dangerouslySetInnerHTML` with user-supplied content.
- Server-side `sanitize_str` provides defense-in-depth (control char stripping, max length).
---
## Frontend Patterns & Conventions
### API client (`src/api/client.ts`)
**No Axios** — uses a thin native `fetch` wrapper. All API calls live here, nowhere else.
The core `request()` function handles:
- Prepends `/api` base URL
- Injects `Authorization: Bearer {token}` from `localStorage` on every request
- **Global 401 handler**: clears `localStorage` token and redirects to `/login` via `window.location.href` — this is the expired-session redirect
- Throws `ApiError(status, detail)` on non-2xx responses (detail parsed from JSON body)
- Returns `undefined` on 204 No Content
- Supports `blob: true` for file download/preview responses
```typescript
// The internal api object — use these methods in exported functions:
api.get<T>(path, params?) // GET with optional query params object
api.post<T>(path, json?) // POST with JSON body
api.postForm<T>(path, URLSearchParams) // POST with form-encoded body (login)
api.postFile<T>(path, FormData) // POST with multipart body (file upload)
api.patch<T>(path, json?) // PATCH with JSON body
api.delete<T>(path) // DELETE
api.getBlob(path) // GET → Blob (download / view)
```
Adding a new API call:
1. Define a TypeScript interface for the response if it's new.
2. Add a named export function (`getX`, `createX`, `updateX`, `deleteX`).
3. Use the appropriate `api.*` method — return the promise directly (no `.then((r) => r.data)`).
Error handling in components: catch blocks receive an `ApiError` instance with `.status` and `.message` (the detail string).
### TanStack Query conventions
**Query keys** (flat arrays, lowercase):
```typescript
["me"] // current user
["services"] // service health list
["dashboard-prefs"] // user dashboard preferences
["categories"] // document categories
["documents", params] // document list (params object for cache isolation)
["documents-shared", params] // shared-with-me list
["document", id] // single document
["document-shares", id] // share list for a specific document
["my-groups"] // current user's group memberships (for share picker)
["plugins"] // accessible plugin list (filtered by user access)
["plugin-manifest", id] // plugin manifest (cached)
["plugin-settings", id] // plugin current settings
```
**Mutation pattern**:
```typescript
const mutation = useMutation({
mutationFn: apiFunction,
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["affected-key"] });
// additional side effects (close dialog, reset form, etc.)
},
});
// Usage:
mutation.mutate(data);
mutation.isPending // show spinner / disable button
mutation.isError // show error message
```
**Polling**:
```typescript
useQuery({ queryKey: ["services"], queryFn: getServices,
refetchInterval: 30_000, refetchIntervalInBackground: true });
```
### Route guards
```typescript
// PrivateRoute — redirect to /login if no token
// AdminRoute — redirect to /login if no token OR not admin
// (waits for getMe() query to avoid flash; uses 404 semantics)
```
### Component patterns
- Functional components only.
- Local `useState` for UI-only state (edit mode, pending values, open/closed).
- Server state via `useQuery` / `useMutation` — no duplicated local copies.
- `cn()` from `lib/utils.ts` for conditional Tailwind classes.
- `lucide-react` for all icons.
- Never use `dangerouslySetInnerHTML` with user-supplied content.
---
## Naming & Code Conventions
- TypeScript strict mode — no `any`.
- API response types inferred from interfaces in `client.ts` only.
- Error messages displayed inline (no alert); loading shown as disabled state or "…" text.
- All user-facing text: safe via React JSX rendering (not innerHTML).
---
## Default Values & Limits
| Parameter | Value | Location |
|-----------|-------|----------|
| Token localStorage key | `"token"` | `useAuth.ts` |
+111 -110
View File
@@ -21,6 +21,7 @@ All API calls go through `src/api/client.ts` (single Axios instance, JWT injecte
| `/admin` | `AdminPage` (redirects to `/admin/users`) | Admin only |
| `/admin/users` | `AdminUsersPage` | Admin only |
| `/admin/groups` | `AdminGroupsPage` | Admin only |
| `/admin/storage` | `StorageAdminPage` | Admin only |
| `/profile` | `ProfilePage` | Required |
| `/settings` | `SettingsPage` (placeholder) | Required |
| `/settings/plugins/:id` | `PluginSettingsPage` | Required (per-plugin access control) |
@@ -40,138 +41,133 @@ All API calls go through `src/api/client.ts` (single Axios instance, JWT injecte
### Home dashboard (`/`)
Personalised landing page per user:
- Time-aware greeting with the user's display name (`full_name` or email). React JSX text rendering HTML-escapes all values — no `dangerouslySetInnerHTML` is used anywhere on this page.
- Grid of **pinned app cards** drawn from `GET /api/services`, filtered to the user's saved list.
- **Customize mode** (pencil button): shows all services; `+` / `` toggle buttons on each card; changes committed with **Save** via `PATCH /api/users/me/preferences`.
- Empty-state prompt when no apps are pinned.
- Time-aware greeting with the user's display name
- Grid of **pinned app cards** from `GET /api/services`, filtered to user's saved list
- **Customize mode** (pencil button): shows all services; `+` / `` toggle; commits via `PATCH /api/users/me/preferences`
### Apps page (`/apps`)
Cards are rendered dynamically from `GET /api/services` (polled every 30 s via TanStack Query):
- **healthy=true + app_path set** — clickable card with "Available" badge
- **healthy=true + no app_path** — non-clickable card (e.g. AI Service — no user UI)
- **healthy=false** — non-clickable, dimmed card with "Unavailable" badge and explanation text
- Single **Settings** button per card — visible to global admins OR members of the service's admin group (checked via `GET /api/plugins` which backend filters by access). Links to `svc.settings_path`.
Cards from `GET /api/services` (polled every 30 s):
- healthy + app_path clickable card with "Available" badge
- healthy + no app_path non-clickable card
- unhealthy → dimmed, non-clickable, "Unavailable"
- Settings button visible to admins and service-admin group members
### Sidebar navigation
`Apps` is an expandable accordion in the sidebar:
- **Documents** sub-item (expandable) — lists all user categories beneath it; clicking a category navigates to `/apps/documents?category_id=<id>`
- AI Service is not listed (no openable UI)
- Sections auto-open when navigating to their route
- In collapsed (icons-only) mode, clicking the Apps icon navigates to `/apps`
`Apps` expandable accordion: **Documents** single NavLink to `/apps/documents`. Category navigation moved to SourcePanel (only visible on `/apps/documents` route). Admin section (Users, Groups, Appearance) for admins. Collapsible to icon-only mode.
### Documents page (`/apps/documents`)
### Documents page (`/apps/documents`) — three-column layout
**Upload:** PDF file input, 202 response, error display.
**SourcePanel** (240px, left): Appears only on `/apps/documents`.
- Views: All Documents / Mine / Shared with me (URL param `?view=`)
- Category tree with client-side search (searchable when > 4 categories)
- Inline new category form
- "Manage categories" button opens `ManageCategoriesDialog`
**Filter bar:**
- Search input (400ms debounce) — matches title, filename, tags, document_type
- Status dropdown (all / pending / processing / done / failed)
- Type dropdown (all / invoice / bill / receipt / order / expense / revenue / unknown)
- Sort selector (upload date / processed date / title / filename / file size / type / status)
- Asc/Desc toggle
- "Clear filters" button (appears when any filter is active)
**Toolbar:** Debounced search input (400ms) + filter chips system.
- Filter chips: Status, Document type, Category (each adds a removable chip)
- "Add filter" button opens a two-step picker (dimension → value)
- Sort via clickable column headers (↑/↓ chevron)
**Pagination:** Prev/Next with "XY of Z" count. Only shown when total > per_page.
**Compact table rows:**
- Columns: checkbox | title/filename | type | status dot | categories (2 + overflow) | sharing icon | date | size | 3-dot actions
- Row click opens `DocumentSlideOver`
- Shared-with-me rows show a primary border accent
**Document row (collapsed):**
- Inline title editor (pencil icon, Enter to save, Esc to cancel; shows filename in italic when no title)
- Status badge (colour-coded)
- Document type label
- File size
- View button (opens PDF in new tab via blob URL — auth-gated)
- Download button
- Delete button (confirm dialog)
**DocumentSlideOver** (480px, right slide-over):
- Metadata (status dot, size, dates, source)
- Inline title edit (pencil icon)
- Type picker (chips for each doc type)
- **AI Suggestions** — folder and filename confirm/reject buttons (was missing before, now implemented)
- Extracted data key-value table
- Categories multi-select combobox (search-to-filter)
- AI-suggested categories with Assign / Create & Assign actions
- Tags chip editor (add/remove inline)
- **Sharing section** (owner only): lists groups the doc is shared with; group picker combobox (filtered to user's own groups); remove share button
- Raw text section (collapsed by default)
- Re-analyse / Delete actions (owner only)
**Document row (expanded):**
- **Extracted data table** — all AI-extracted JSON fields (excludes `tags`, `suggested_categories`)
- **Error message** — shown if status=failed
- **Categories** — assigned chips with remove; dropdown to assign existing; AI-suggested chips with Accept / Create & Assign / Dismiss
- **Status polling** — auto-refetches every 3s while status is pending/processing; invalidates document list on done/failed
**Bulk actions bar** (floating, bottom center, owner view only):
- Appears when any rows are checked
- Share with group (opens group picker → shares all selected)
- Delete (confirm dialog)
- Clear selection
**Upload experience:**
- Full-page drag-and-drop overlay (activates on `dragenter`)
- Multi-file upload (iterates all selected/dropped files)
- Bottom-right upload queue panel (collapsible toast) with per-file status + "Review →" link after upload
**Document sharing:**
- Owner shares doc with any of their own groups from the slide-over
- Recipient sees shared docs in "Shared with me" view
- Recipient can View + Download only (no edit/delete/share)
- `share_count` indicator (Users icon) in table rows
**Polling:** List query refetches every 3s automatically when any visible doc is pending/processing (single query, not per-document). Uses TanStack Query `refetchInterval` function.
### AI Service Settings (`/apps/ai/settings`)
Accessible to global admins and `ai-service-admin` group members (`ServiceAdminRoute`).
- Provider selector (lmstudio / ollama / anthropic)
- Per-provider fields (base URL, model, API key)
- Test Connection button (`POST /api/settings/ai/test`)
- Save button
Provider selector, per-provider fields, Test Connection, Save.
### Document Service Settings (`/apps/documents/settings`)
Accessible to global admins and `doc-service-admin` group members (`ServiceAdminRoute`).
Combined settings on one page, accessed via the single "Settings" button on the app card:
- **Upload Limits** — max PDF size in MB (`GET/PATCH /api/settings/documents/limits`)
- **Watch Directory** — file watcher config rendered via `PluginSchemaForm` from manifest (`GET/PATCH /api/plugins/doc-service/settings`)
Upload limits + watch directory config.
### Admin — Storage page (`/admin/storage`)
Current backend status (green/red health dot). Driver selector (local/S3/WebDAV) with conditional credential fields. "Test & Migrate" button triggers an async migration that copies all objects to the new backend, verifies, then switches atomically. Live progress bar with 2s polling (states: validating → migrating → switching → cleaning → done). Cancel button during in-progress migrations.
### Admin — Users page (`/admin/users`)
- User list with role and active status
- Inline active status toggle
- Create user form (email, name, password, admin flag)
- Delete user
User list, toggle active, create user, delete user.
### Admin — Groups page (`/admin/groups`)
- Group list with name, description, member count
- Create group (name, optional description)
- Edit group name / description inline panel
- Delete group (with confirmation)
- Expand group row to manage members: view members, remove members, add non-members from dropdown
Group list, create, edit name/description, delete, add/remove members.
### Profile page (`/profile`)
- Display and edit personal information
Display and edit personal information.
---
## API client (`src/api/client.ts`)
Key functions:
**No Axios** — replaced with a native `fetch` wrapper (`request()`). Global 401 handler clears the token and redirects to `/login`, fixing the expired-session blank-page bug. All exported function signatures are unchanged.
Key document-related functions:
| Function | Description |
|----------|-------------|
| `listDocuments(params)` | `GET /documents`returns `DocumentPage`; supports `category_id` filter |
| `listDocuments(params)` | `GET /documents`paginated with filters |
| `listSharedWithMe(params)` | `GET /documents/shared-with-me` |
| `uploadDocument(file)` | `POST /documents/upload` |
| `deleteDocument(id)` | `DELETE /documents/{id}` |
| `downloadDocument(id, filename)` | Blob URL download |
| `viewDocument(id)` | Blob URL → `window.open`, auto-revoke after 60s |
| `getDocumentStatus(id)` | Poll endpoint |
| `listCategories()` | All categories for user |
| `createCategory(name)` | Create category |
| `assignCategory(docId, catId)` | Assign |
| `removeCategory(docId, catId)` | Remove |
| `updateDocumentTags(id, tags)` | `PATCH /documents/{id}/tags` |
| `updateDocumentTitle(id, title)` | `PATCH /documents/{id}/title` |
| `confirmFolderSuggestion(docId)` | `POST /documents/{id}/suggestions/folder/confirm` |
| `rejectFolderSuggestion(docId)` | `POST /documents/{id}/suggestions/folder/reject` |
| `confirmFilenameSuggestion(docId)` | `POST /documents/{id}/suggestions/filename/confirm` |
| `rejectFilenameSuggestion(docId)` | `POST /documents/{id}/suggestions/filename/reject` |
| `getAISettings()` | `GET /settings/ai` (masked) |
| `updateAISettings(data)` | `PATCH /settings/ai` |
| `testAIConnection()` | `POST /settings/ai/test` |
| `getDocumentLimits()` | `GET /settings/documents/limits` |
| `adminListGroups()` | `GET /admin/groups` |
| `adminCreateGroup(data)` | `POST /admin/groups` |
| `adminGetGroup(id)` | `GET /admin/groups/{id}` with members |
| `adminUpdateGroup(id, data)` | `PATCH /admin/groups/{id}` |
| `adminDeleteGroup(id)` | `DELETE /admin/groups/{id}` |
| `adminAddGroupMember(gId, uId)` | `POST /admin/groups/{gId}/members/{uId}` |
| `adminRemoveGroupMember(gId, uId)` | `DELETE /admin/groups/{gId}/members/{uId}` |
| `updateDocumentLimits(data)` | `PATCH /settings/documents/limits` |
| `getPlugins()` | `GET /plugins` — list accessible plugins |
| `getPluginManifest(id)` | `GET /plugins/{id}/manifest` |
| `getPluginSettings(id)` | `GET /plugins/{id}/settings` |
| `updatePluginSettings(id, data)` | `PATCH /plugins/{id}/settings` |
| `viewDocument(id)` | Blob URL → new tab, 60s revoke |
| `getDocumentShares(docId)` | `GET /documents/{id}/shares` |
| `addDocumentShare(docId, groupId)` | `POST /documents/{id}/shares` |
| `removeDocumentShare(docId, groupId)` | `DELETE /documents/{id}/shares/{group_id}` |
| `getMyGroups()` | `GET /users/me/groups` (for share picker) |
| `confirmFolderSuggestion(docId)` | Apply AI folder suggestion |
| `rejectFolderSuggestion(docId)` | Dismiss AI folder suggestion |
| `confirmFilenameSuggestion(docId)` | Apply AI filename suggestion |
| `rejectFilenameSuggestion(docId)` | Dismiss AI filename suggestion |
---
## State management
- **TanStack Query** — all server state; `queryKey: ["documents", params]` for cache isolation per filter/page combination
- **No global store** — local `useState` for UI-only state (editing mode, filter params, etc.)
- **Token** — `localStorage`, read by `useAuth` hook, injected by Axios interceptor
- **TanStack Query** — all server state
- `["documents", params]` — owned doc list (refetchInterval when pending/processing)
- `["documents-shared", params]` — shared-with-me list
- `["categories"]` — all user categories (shared across SourcePanel + DocumentSlideOver)
- `["document-shares", docId]` — shares for a specific document
- `["my-groups"]` — current user's group memberships
- **URL search params** — `view`, `page`, `sort`, `order`, `search`, `status`, `document_type`, `category_id`
- **Local `useState`** — UI-only state (drag, upload queue, active doc ID, selected IDs, slide-over open)
---
@@ -179,37 +175,42 @@ Key functions:
| Component | Path | Description |
|-----------|------|-------------|
| `AppShell` | `src/components/AppShell.tsx` | Layout wrapper: Sidebar + scrollable main content |
| `Sidebar` | `src/components/Sidebar.tsx` | Collapsible left nav; includes dynamic "Extensions" section |
| `ThemeToggle` | `src/components/ThemeToggle.tsx` | Sun/moon ghost icon button; persists to localStorage |
| `PluginSchemaForm` | `src/components/PluginSchemaForm.tsx` | JSON Schema → React form (boolean/string/number/readOnly fields) |
| `PluginSettingsPage` | `src/pages/PluginSettingsPage.tsx` | Generic plugin settings page (manifest-driven) |
| `Button` | `src/components/ui/button.tsx` | shadcn/ui Button (default, ghost, outline, destructive) |
| `AppShell` | `src/components/AppShell.tsx` | Layout: Sidebar + SourcePanel (on /apps/documents) + main |
| `Sidebar` | `src/components/Sidebar.tsx` | Collapsible left nav (categories removed, replaced by SourcePanel) |
| `SourcePanel` | `src/components/SourcePanel.tsx` | Views + searchable category tree (docs route only) |
| `ManageCategoriesDialog` | `src/components/ManageCategoriesDialog.tsx` | Category CRUD modal |
| `DocumentSlideOver` | `src/components/DocumentSlideOver.tsx` | Right slide-over: detail, edit, share, AI suggestions |
| `ThemeToggle` | `src/components/ThemeToggle.tsx` | Sun/moon toggle |
| `PluginSchemaForm` | `src/components/PluginSchemaForm.tsx` | JSON Schema → React form |
| `Button` | `src/components/ui/button.tsx` | shadcn/ui Button |
| `Input` | `src/components/ui/input.tsx` | shadcn/ui Input |
---
## Known limitations / not implemented
- **JWT in `localStorage`** — XSS risk; migrate to `httpOnly` cookie when backend supports it
- **No toast / notification system** — errors shown inline; success is silent
- **No loading skeletons** — "Loading…" text only
- **No app permission UI** per group — groups exist but permission grants are not yet implemented
- **No app permission UI** — all apps visible to all authenticated users
- **No loading skeletons** — spinner only
- **Raw text not in DocumentOut** — slide-over shows a placeholder; full text requires direct backend API call
---
## Future work
- [x] UI component library: shadcn/ui + Tailwind CSS — installed and wired up
- [x] AppShell + Sidebar replacing inline Nav component
- [x] Light/dark theme context with OS preference detection
- [x] Generic plugin infrastructure: Extensions sidebar section, PluginSchemaForm, PluginSettingsPage
- [ ] Suggestion badges in DocumentsPage for `suggested_folder` / `suggested_filename` (confirm/reject buttons)
- [ ] Toast notification system (upload success, save feedback, errors)
- [x] SourcePanel with views + searchable category navigation
- [x] DocumentSlideOver replacing expand-in-row
- [x] Filter chip system
- [x] Multi-file upload with queue panel + drag-and-drop
- [x] Bulk actions bar (share, delete)
- [x] Document sharing UI (Sharing section + Shared with me view)
- [x] AI suggestion confirm/reject UI (folder + filename)
- [x] Groups admin UI
- [x] Replace Axios with native fetch; add global 401 → `/login` redirect for expired sessions
- [x] Admin storage page with live migration progress bar
- [ ] Toast notification system
- [ ] Loading skeletons
- [ ] `POST /queue/jobs` integration — show AI processing queue status / progress per document
- [ ] Advanced filter: extracted data fields (vendor, due date, amount) — needs backend support
- [x] Groups admin UI — list, create, edit, delete, add/remove members
- [ ] App permissions UI per group (blocked on backend group_app_permissions)
- [ ] Document sharing UI (blocked on backend)
- [ ] Cmd+K global search (`CommandDialog`)
- [ ] Advanced filter: extracted data fields (needs backend support)
- [ ] `httpOnly` cookie auth (requires backend change)
- [ ] Bulk document operations (select multiple, bulk delete / bulk categorise)
- [ ] TanStack Virtual for category list > 200 items
-1
View File
@@ -13,7 +13,6 @@
"dependencies": {
"@radix-ui/react-slot": "^1.0.2",
"@tanstack/react-query": "^5.40.0",
"axios": "^1.7.2",
"class-variance-authority": "^0.7.0",
"clsx": "^2.1.0",
"lucide-react": "^0.400.0",
+2
View File
@@ -16,6 +16,7 @@ import DocServiceSettingsPage from "./pages/DocServiceSettingsPage";
import AIAdminSettingsPage from "./pages/AIAdminSettingsPage";
import SettingsPage from "./pages/SettingsPage";
import PluginSettingsPage from "./pages/PluginSettingsPage";
import StorageAdminPage from "./pages/StorageAdminPage";
function PrivateRoute({ children }: { children: React.ReactNode }) {
const { token } = useAuth();
@@ -102,6 +103,7 @@ export default function App() {
<Route path="/admin/users" element={<AdminRoute><AdminUsersPage /></AdminRoute>} />
<Route path="/admin/groups" element={<AdminRoute><AdminGroupsPage /></AdminRoute>} />
<Route path="/admin/appearance" element={<AdminRoute><AdminAppearancePage /></AdminRoute>} />
<Route path="/admin/storage" element={<AdminRoute><StorageAdminPage /></AdminRoute>} />
{/* Catch-all */}
<Route path="*" element={<Navigate to="/" replace />} />
+303 -96
View File
@@ -1,25 +1,127 @@
import axios from "axios";
const BASE = "/api";
const api = axios.create({ baseURL: "/api" });
// ---------------------------------------------------------------------------
// Core fetch wrapper
// ---------------------------------------------------------------------------
api.interceptors.request.use((config) => {
export class ApiError extends Error {
status: number;
detail: string;
constructor(status: number, detail: string) {
super(detail);
this.status = status;
this.detail = detail;
}
}
async function request<T>(
method: string,
path: string,
options: {
json?: unknown;
form?: URLSearchParams;
body?: FormData;
params?: Record<string, string | number | undefined>;
blob?: boolean;
} = {}
): Promise<T> {
const { json, form, body, params, blob: asBlob } = options;
// Build URL with optional query params
let url = `${BASE}${path}`;
if (params) {
const qs = new URLSearchParams();
for (const [k, v] of Object.entries(params)) {
if (v !== undefined && v !== "") qs.set(k, String(v));
}
const str = qs.toString();
if (str) url += `?${str}`;
}
// Build headers
const headers: Record<string, string> = {};
const token = localStorage.getItem("token");
if (token) config.headers.Authorization = `Bearer ${token}`;
return config;
});
if (token) headers["Authorization"] = `Bearer ${token}`;
if (json !== undefined) headers["Content-Type"] = "application/json";
if (form !== undefined) headers["Content-Type"] = "application/x-www-form-urlencoded";
// FormData: intentionally omit Content-Type so browser sets multipart boundary
// Build body
let requestBody: BodyInit | undefined;
if (json !== undefined) requestBody = JSON.stringify(json);
else if (form !== undefined) requestBody = form;
else if (body !== undefined) requestBody = body;
const response = await fetch(url, { method, headers, body: requestBody });
// Global 401 handler — expired or invalid token
if (response.status === 401) {
localStorage.removeItem("token");
if (window.location.pathname !== "/login") {
window.location.href = "/login";
}
throw new ApiError(401, "Session expired");
}
// Parse error responses
if (!response.ok) {
let detail = `HTTP ${response.status}`;
try {
const err = await response.json();
if (typeof err.detail === "string") detail = err.detail;
else if (Array.isArray(err.detail)) detail = err.detail.map((e: { msg: string }) => e.msg).join(", ");
} catch {
// non-JSON error body — keep default message
}
throw new ApiError(response.status, detail);
}
// 204 No Content
if (response.status === 204) return undefined as T;
if (asBlob) return response.blob() as Promise<T>;
const text = await response.text();
return text ? (JSON.parse(text) as T) : (undefined as T);
}
const api = {
get: <T>(path: string, params?: Record<string, string | number | undefined>) =>
request<T>("GET", path, { params }),
post: <T>(path: string, json?: unknown) =>
request<T>("POST", path, { json }),
postForm: <T>(path: string, form: URLSearchParams) =>
request<T>("POST", path, { form }),
postFile: <T>(path: string, body: FormData) =>
request<T>("POST", path, { body }),
patch: <T>(path: string, json?: unknown) =>
request<T>("PATCH", path, { json }),
delete: <T>(path: string) =>
request<T>("DELETE", path),
getBlob: (path: string) =>
request<Blob>("GET", path, { blob: true }),
};
export default api;
// --- Auth ---
// ---------------------------------------------------------------------------
// Auth
// ---------------------------------------------------------------------------
export const login = (email: string, password: string) =>
api
.post<{ access_token: string }>("/auth/login", new URLSearchParams({ username: email, password }))
.then((r) => r.data.access_token);
api.postForm<{ access_token: string }>(
"/auth/login",
new URLSearchParams({ username: email, password })
).then((r) => r.access_token);
export const register = (email: string, password: string, full_name?: string) =>
api.post("/auth/register", { email, password, full_name }).then((r) => r.data);
api.post("/auth/register", { email, password, full_name });
// ---------------------------------------------------------------------------
// Users
// ---------------------------------------------------------------------------
// --- Users ---
export interface UserData {
id: string;
email: string;
@@ -29,19 +131,21 @@ export interface UserData {
color_mode: string | null;
}
export const getMe = () => api.get<UserData>("/users/me").then((r) => r.data);
export const getMe = () => api.get<UserData>("/users/me");
export interface DashboardPrefs {
app_ids: string[];
}
export const getDashboardPrefs = () =>
api.get<DashboardPrefs>("/users/me/preferences").then((r) => r.data);
export const getDashboardPrefs = () => api.get<DashboardPrefs>("/users/me/preferences");
export const updateDashboardPrefs = (app_ids: string[]) =>
api.patch<DashboardPrefs>("/users/me/preferences", { app_ids }).then((r) => r.data);
api.patch<DashboardPrefs>("/users/me/preferences", { app_ids });
// ---------------------------------------------------------------------------
// Admin — Users
// ---------------------------------------------------------------------------
// --- Admin ---
export interface AdminUserCreate {
email: string;
password: string;
@@ -49,19 +153,21 @@ export interface AdminUserCreate {
is_admin?: boolean;
}
export const adminGetUsers = () =>
api.get<UserData[]>("/admin/users").then((r) => r.data);
export const adminGetUsers = () => api.get<UserData[]>("/admin/users");
export const adminCreateUser = (data: AdminUserCreate) =>
api.post<UserData>("/admin/users", data).then((r) => r.data);
api.post<UserData>("/admin/users", data);
export const adminDeleteUser = (userId: string) =>
api.delete(`/admin/users/${userId}`);
export const adminToggleActive = (userId: string) =>
api.patch<UserData>(`/admin/users/${userId}/active`).then((r) => r.data);
api.patch<UserData>(`/admin/users/${userId}/active`);
// ---------------------------------------------------------------------------
// Profile
// ---------------------------------------------------------------------------
// --- Profile ---
export interface ProfileData {
id: string;
user_id: string;
@@ -79,18 +185,22 @@ export interface ProfileUpdate {
address?: string | null;
}
export const getProfile = () =>
api.get<ProfileData>("/profile/me").then((r) => r.data);
export const getProfile = () => api.get<ProfileData>("/profile/me");
export const updateProfile = (data: ProfileUpdate) =>
api.put<ProfileData>("/profile/me", data).then((r) => r.data);
api.patch<ProfileData>("/profile/me", data);
// ---------------------------------------------------------------------------
// Documents
// ---------------------------------------------------------------------------
// --- Documents ---
export type DocumentStatus = "pending" | "processing" | "done" | "failed";
export interface CategoryOut {
id: string;
name: string;
scope: "personal" | "group" | "system";
group_id: string | null;
}
export interface DocumentOut {
@@ -111,6 +221,22 @@ export interface DocumentOut {
watch_path: string | null;
suggested_folder: string | null;
suggested_filename: string | null;
share_count: number;
viewer_can_delete: boolean;
}
export interface SharedDocumentOut extends DocumentOut {
shared_by_user_id: string;
shared_via_group_id: string;
}
export interface DocumentShareOut {
id: string;
document_id: string;
group_id: string;
shared_by_user_id: string;
can_delete: boolean;
created_at: string;
}
export interface DocumentPage {
@@ -140,29 +266,40 @@ export interface DocumentStatusOut {
}
export const listDocuments = (params: DocumentListParams = {}) =>
api.get<DocumentPage>("/documents", { params }).then((r) => r.data);
api.get<DocumentPage>("/documents", params as Record<string, string | number | undefined>);
export const listSharedWithMe = (params: DocumentListParams = {}) =>
api.get<DocumentPage>("/documents/shared-with-me", params as Record<string, string | number | undefined>);
export const getDocumentShares = (docId: string) =>
api.get<DocumentShareOut[]>(`/documents/${docId}/shares`);
export const addDocumentShare = (docId: string, groupId: string, canDelete = false) =>
api.post<DocumentShareOut>(`/documents/${docId}/shares`, { group_id: groupId, can_delete: canDelete });
export const removeDocumentShare = (docId: string, groupId: string) =>
api.delete(`/documents/${docId}/shares/${groupId}`);
export const getDocument = (id: string) =>
api.get<DocumentOut>(`/documents/${id}`).then((r) => r.data);
api.get<DocumentOut>(`/documents/${id}`);
export const getDocumentStatus = (id: string) =>
api.get<DocumentStatusOut>(`/documents/${id}/status`).then((r) => r.data);
api.get<DocumentStatusOut>(`/documents/${id}/status`);
export const uploadDocument = (file: File) => {
const form = new FormData();
form.append("file", file);
return api.post<DocumentOut>("/documents/upload", form).then((r) => r.data);
return api.postFile<DocumentOut>("/documents/upload", form);
};
export const updateDocumentType = (id: string, document_type: string) =>
api.patch<DocumentOut>(`/documents/${id}/type`, { document_type }).then((r) => r.data);
api.patch<DocumentOut>(`/documents/${id}/type`, { document_type });
export const deleteDocument = (id: string) =>
api.delete(`/documents/${id}`);
export const deleteDocument = (id: string) => api.delete(`/documents/${id}`);
export const downloadDocument = async (id: string, filename: string) => {
const response = await api.get(`/documents/${id}/file`, { responseType: "blob" });
const url = URL.createObjectURL(response.data);
const blob = await api.getBlob(`/documents/${id}/file`);
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.href = url;
a.download = filename;
@@ -171,22 +308,21 @@ export const downloadDocument = async (id: string, filename: string) => {
};
export const viewDocument = async (id: string): Promise<void> => {
const response = await api.get(`/documents/${id}/file`, { responseType: "blob" });
const url = URL.createObjectURL(response.data);
const blob = await api.getBlob(`/documents/${id}/file`);
const url = URL.createObjectURL(blob);
const win = window.open(url, "_blank");
// Revoke after a generous delay — the new tab needs time to load the blob
setTimeout(() => URL.revokeObjectURL(url), 60_000);
if (!win) alert("Pop-up blocked. Please allow pop-ups for this site to preview PDFs.");
};
export const updateDocumentTags = (id: string, tags: string[]) =>
api.patch<DocumentOut>(`/documents/${id}/tags`, { tags }).then((r) => r.data);
api.patch<DocumentOut>(`/documents/${id}/tags`, { tags });
export const updateDocumentTitle = (id: string, title: string) =>
api.patch<DocumentOut>(`/documents/${id}/title`, { title }).then((r) => r.data);
api.patch<DocumentOut>(`/documents/${id}/title`, { title });
export const reprocessDocument = (id: string) =>
api.post<DocumentOut>(`/documents/${id}/reprocess`).then((r) => r.data);
api.post<DocumentOut>(`/documents/${id}/reprocess`);
export const assignCategory = (docId: string, catId: string) =>
api.post(`/documents/${docId}/categories/${catId}`);
@@ -194,20 +330,24 @@ export const assignCategory = (docId: string, catId: string) =>
export const removeCategory = (docId: string, catId: string) =>
api.delete(`/documents/${docId}/categories/${catId}`);
// --- Categories ---
export const listCategories = () =>
api.get<CategoryOut[]>("/documents/categories").then((r) => r.data);
// ---------------------------------------------------------------------------
// Categories
// ---------------------------------------------------------------------------
export const createCategory = (name: string) =>
api.post<CategoryOut>("/documents/categories", { name }).then((r) => r.data);
export const listCategories = () => api.get<CategoryOut[]>("/documents/categories");
export const createCategory = (name: string, groupId?: string) =>
api.post<CategoryOut>("/documents/categories", { name, group_id: groupId ?? null });
export const renameCategory = (id: string, name: string) =>
api.patch<CategoryOut>(`/documents/categories/${id}`, { name }).then((r) => r.data);
api.patch<CategoryOut>(`/documents/categories/${id}`, { name });
export const deleteCategory = (id: string) =>
api.delete(`/documents/categories/${id}`);
export const deleteCategory = (id: string) => api.delete(`/documents/categories/${id}`);
// ---------------------------------------------------------------------------
// Appearance & Themes
// ---------------------------------------------------------------------------
// --- Appearance & Themes ---
export interface ThemeColors {
primary: string;
primary_hover: string;
@@ -233,31 +373,31 @@ export interface AppearanceSettings {
default_mode: string;
}
export const getAppearanceSettings = (): Promise<AppearanceSettings> =>
api.get<AppearanceSettings>("/settings/appearance").then((r) => r.data);
export const getAppearanceSettings = () =>
api.get<AppearanceSettings>("/settings/appearance");
export const updateAppearanceSettings = (data: AppearanceSettings): Promise<AppearanceSettings> =>
api.patch<AppearanceSettings>("/settings/appearance", data).then((r) => r.data);
export const updateAppearanceSettings = (data: AppearanceSettings) =>
api.patch<AppearanceSettings>("/settings/appearance", data);
export const getThemes = (): Promise<ThemeDefinition[]> =>
api.get<ThemeDefinition[]>("/settings/themes").then((r) => r.data);
export const getThemes = () => api.get<ThemeDefinition[]>("/settings/themes");
export const createTheme = (data: Omit<ThemeDefinition, "builtin">): Promise<ThemeDefinition> =>
api.post<ThemeDefinition>("/settings/themes", data).then((r) => r.data);
export const createTheme = (data: Omit<ThemeDefinition, "builtin">) =>
api.post<ThemeDefinition>("/settings/themes", data);
export const updateTheme = (
id: string,
data: { label?: string; light?: ThemeColors; dark?: ThemeColors }
): Promise<ThemeDefinition> =>
api.patch<ThemeDefinition>(`/settings/themes/${id}`, data).then((r) => r.data);
) => api.patch<ThemeDefinition>(`/settings/themes/${id}`, data);
export const deleteTheme = (id: string): Promise<void> =>
api.delete(`/settings/themes/${id}`).then((r) => r.data);
export const deleteTheme = (id: string) => api.delete(`/settings/themes/${id}`);
export const updateColorMode = (color_mode: string): Promise<UserData> =>
api.patch<UserData>("/users/me/color-mode", { color_mode }).then((r) => r.data);
export const updateColorMode = (color_mode: string) =>
api.patch<UserData>("/users/me/color-mode", { color_mode });
// ---------------------------------------------------------------------------
// Settings (admin only)
// ---------------------------------------------------------------------------
// --- Settings (admin only) ---
export interface AIProviderUpdate {
provider: string;
anthropic_api_key?: string;
@@ -271,25 +411,39 @@ export interface AIProviderUpdate {
}
export const getAISettings = () =>
api.get<Record<string, unknown>>("/settings/ai").then((r) => r.data);
api.get<Record<string, unknown>>("/settings/ai");
export const updateAISettings = (data: AIProviderUpdate) =>
api.patch<Record<string, unknown>>("/settings/ai", data).then((r) => r.data);
api.patch<Record<string, unknown>>("/settings/ai", data);
export const testAIConnection = () =>
api.post<{ ok: boolean; provider: string; response?: string; error?: string }>(
"/settings/ai/test"
).then((r) => r.data);
export const updateDocumentLimits = (max_pdf_mb: number) =>
api.patch<Record<string, unknown>>("/settings/documents/limits", { max_pdf_mb }).then(
(r) => r.data
);
export const getDocumentLimits = () =>
api.get<Record<string, unknown>>("/settings/documents/limits").then((r) => r.data);
export const updateDocumentLimits = (max_pdf_mb: number) =>
api.patch<Record<string, unknown>>("/settings/documents/limits", { max_pdf_mb });
export const getDocumentLimits = () =>
api.get<Record<string, unknown>>("/settings/documents/limits");
// ---------------------------------------------------------------------------
// User groups (current user's own memberships)
// ---------------------------------------------------------------------------
export interface UserGroupOut {
id: string;
name: string;
description: string | null;
is_group_admin: boolean;
}
export const getMyGroups = () => api.get<UserGroupOut[]>("/users/me/groups");
// ---------------------------------------------------------------------------
// Groups (admin only)
// ---------------------------------------------------------------------------
// --- Groups (admin only) ---
export interface GroupOut {
id: string;
name: string;
@@ -304,6 +458,7 @@ export interface GroupMemberOut {
full_name: string | null;
is_active: boolean;
joined_at: string;
is_group_admin: boolean;
}
export interface GroupDetailOut extends GroupOut {
@@ -320,17 +475,16 @@ export interface GroupUpdate {
description?: string | null;
}
export const adminListGroups = () =>
api.get<GroupOut[]>("/admin/groups").then((r) => r.data);
export const adminListGroups = () => api.get<GroupOut[]>("/admin/groups");
export const adminCreateGroup = (data: GroupCreate) =>
api.post<GroupOut>("/admin/groups", data).then((r) => r.data);
api.post<GroupOut>("/admin/groups", data);
export const adminGetGroup = (groupId: string) =>
api.get<GroupDetailOut>(`/admin/groups/${groupId}`).then((r) => r.data);
api.get<GroupDetailOut>(`/admin/groups/${groupId}`);
export const adminUpdateGroup = (groupId: string, data: GroupUpdate) =>
api.patch<GroupOut>(`/admin/groups/${groupId}`, data).then((r) => r.data);
api.patch<GroupOut>(`/admin/groups/${groupId}`, data);
export const adminDeleteGroup = (groupId: string) =>
api.delete(`/admin/groups/${groupId}`);
@@ -341,7 +495,13 @@ export const adminAddGroupMember = (groupId: string, userId: string) =>
export const adminRemoveGroupMember = (groupId: string, userId: string) =>
api.delete(`/admin/groups/${groupId}/members/${userId}`);
// --- Services ---
export const adminSetGroupMemberAdmin = (groupId: string, userId: string, isAdmin: boolean) =>
api.patch(`/admin/groups/${groupId}/members/${userId}/admin`, { is_group_admin: isAdmin });
// ---------------------------------------------------------------------------
// Services
// ---------------------------------------------------------------------------
export interface ServiceStatus {
id: string;
name: string;
@@ -351,10 +511,12 @@ export interface ServiceStatus {
settings_path: string;
}
export const getServices = () =>
api.get<ServiceStatus[]>("/services").then((r) => r.data);
export const getServices = () => api.get<ServiceStatus[]>("/services");
// ---------------------------------------------------------------------------
// System Prompts (admin only)
// ---------------------------------------------------------------------------
// --- System Prompts (admin only) ---
export interface ServiceSystemPrompt {
label: string;
system: string;
@@ -366,17 +528,17 @@ export interface ServiceSystemPrompt {
export type SystemPromptsData = Record<string, ServiceSystemPrompt>;
export const getSystemPrompts = () =>
api.get<SystemPromptsData>("/settings/system-prompts").then((r) => r.data);
api.get<SystemPromptsData>("/settings/system-prompts");
export const updateSystemPrompt = (
serviceId: string,
data: { system: string; user_template: string }
) =>
api
.patch<SystemPromptsData>(`/settings/system-prompts/${serviceId}`, data)
.then((r) => r.data);
) => api.patch<SystemPromptsData>(`/settings/system-prompts/${serviceId}`, data);
// ---------------------------------------------------------------------------
// Document suggestions
// ---------------------------------------------------------------------------
// --- Document suggestions (watch-ingested documents) ---
export const confirmFolderSuggestion = (docId: string) =>
api.post(`/documents/${docId}/suggestions/folder/confirm`);
@@ -389,7 +551,10 @@ export const confirmFilenameSuggestion = (docId: string) =>
export const rejectFilenameSuggestion = (docId: string) =>
api.post(`/documents/${docId}/suggestions/filename/reject`);
// --- Plugins ---
// ---------------------------------------------------------------------------
// Plugins
// ---------------------------------------------------------------------------
export interface PluginOut {
id: string;
name: string;
@@ -420,14 +585,56 @@ export interface PluginManifest {
};
}
export const getPlugins = () =>
api.get<PluginOut[]>("/plugins").then((r) => r.data);
// ── Storage admin ──────────────────────────────────────────────────────────────
export interface StorageStatus {
status: string;
backend: string;
}
export interface MigrationStatus {
state:
| "idle"
| "validating"
| "migrating"
| "switching"
| "cleaning"
| "done"
| "failed"
| "cancelled";
total: number;
done: number;
failed: number;
errors: string[];
}
export interface StorageBackendConfig {
driver: string;
config: Record<string, string>;
}
export const getStorageConfig = () => api.get<StorageStatus>("/admin/storage-config");
export const updateStorageConfig = (body: StorageBackendConfig) =>
api.patch<void>("/admin/storage-config", body);
export const startStorageMigration = (body: StorageBackendConfig) =>
api.post<{ status: string; driver: string }>("/admin/storage-config/migrate", body);
export const getMigrationStatus = () =>
api.get<MigrationStatus>("/admin/storage-config/migrate/status");
export const cancelMigration = () => api.delete<void>("/admin/storage-config/migrate");
// ── Plugins ────────────────────────────────────────────────────────────────────
export const getPlugins = () => api.get<PluginOut[]>("/plugins");
export const getPluginManifest = (id: string) =>
api.get<PluginManifest>(`/plugins/${id}/manifest`).then((r) => r.data);
api.get<PluginManifest>(`/plugins/${id}/manifest`);
export const getPluginSettings = (id: string) =>
api.get<Record<string, unknown>>(`/plugins/${id}/settings`).then((r) => r.data);
api.get<Record<string, unknown>>(`/plugins/${id}/settings`);
export const updatePluginSettings = (id: string, data: Record<string, unknown>) =>
api.patch<Record<string, unknown>>(`/plugins/${id}/settings`, data).then((r) => r.data);
api.patch<Record<string, unknown>>(`/plugins/${id}/settings`, data);
+6
View File
@@ -1,13 +1,19 @@
import { useLocation } from "react-router-dom";
import Sidebar from "@/components/Sidebar";
import SourcePanel from "@/components/SourcePanel";
interface AppShellProps {
children: React.ReactNode;
}
export default function AppShell({ children }: AppShellProps) {
const location = useLocation();
const showSourcePanel = location.pathname === "/apps/documents";
return (
<div className="flex h-screen bg-background">
<Sidebar />
{showSourcePanel && <SourcePanel />}
<main className="flex-1 overflow-y-auto p-6">{children}</main>
</div>
);
@@ -0,0 +1,792 @@
import { useState, useRef, useEffect, useCallback } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import {
X, Download, Eye, RefreshCw, Trash2, Check, Pencil, Plus,
ChevronDown, ChevronRight, Users, UserMinus,
} from "lucide-react";
import {
DocumentOut, CategoryOut,
updateDocumentTitle, updateDocumentTags, updateDocumentType,
assignCategory, removeCategory, deleteDocument,
downloadDocument, viewDocument, reprocessDocument,
confirmFolderSuggestion, rejectFolderSuggestion,
confirmFilenameSuggestion, rejectFilenameSuggestion,
listCategories,
getDocumentShares, addDocumentShare, removeDocumentShare,
getMyGroups,
} from "@/api/client";
import { Input } from "@/components/ui/input";
import { Button } from "@/components/ui/button";
import { cn } from "@/lib/utils";
interface Props {
doc: DocumentOut | null;
isOwner: boolean; // false for "shared with me" view
onClose: () => void;
onDeleted: (id: string) => void;
}
const STATUS_COLORS: Record<string, string> = {
pending: "bg-orange-400",
processing: "bg-blue-400",
done: "bg-emerald-400",
failed: "bg-red-400",
};
const DOC_TYPES = ["invoice", "bill", "receipt", "order", "expense", "revenue", "unknown"];
function formatBytes(bytes: number) {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
}
function formatDate(iso: string | null) {
if (!iso) return "—";
return new Date(iso).toLocaleDateString(undefined, { year: "numeric", month: "short", day: "numeric" });
}
// ── Category combobox ─────────────────────────────────────────────────────────
function CategoryCombobox({
categories, assigned, onAssign,
}: { categories: CategoryOut[]; assigned: CategoryOut[]; onAssign: (id: string) => void }) {
const [open, setOpen] = useState(false);
const [search, setSearch] = useState("");
const ref = useRef<HTMLDivElement>(null);
const assignedIds = new Set(assigned.map((c) => c.id));
const unassigned = categories.filter((c) => !assignedIds.has(c.id));
const filtered = search
? unassigned.filter((c) => c.name.toLowerCase().includes(search.toLowerCase()))
: unassigned;
useEffect(() => {
const handler = (e: MouseEvent) => {
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
};
document.addEventListener("mousedown", handler);
return () => document.removeEventListener("mousedown", handler);
}, []);
if (unassigned.length === 0) return null;
return (
<div className="relative" ref={ref}>
<button
onClick={() => setOpen((o) => !o)}
className="flex items-center gap-1 text-xs text-muted hover:text-foreground transition-colors border border-dashed border-border rounded px-2 py-0.5"
>
<Plus className="h-3 w-3" /> Add category
</button>
{open && (
<div className="absolute left-0 top-full mt-1 z-20 bg-surface border border-border rounded-lg shadow-lg w-52">
{categories.length > 5 && (
<div className="p-2 border-b border-border">
<Input
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder="Search…"
className="h-7 text-xs"
autoFocus
/>
</div>
)}
<div className="max-h-48 overflow-y-auto py-1">
{filtered.map((cat) => (
<button
key={cat.id}
className="w-full text-left px-3 py-1.5 text-sm hover:bg-muted/20 transition-colors"
onClick={() => { onAssign(cat.id); setOpen(false); setSearch(""); }}
>
{cat.name}
</button>
))}
{filtered.length === 0 && (
<p className="text-xs text-muted px-3 py-2">No categories</p>
)}
</div>
</div>
)}
</div>
);
}
// ── Group picker combobox ─────────────────────────────────────────────────────
function GroupCombobox({
groups, sharedGroupIds, onShare,
}: { groups: { id: string; name: string }[]; sharedGroupIds: Set<string>; onShare: (id: string) => void }) {
const [open, setOpen] = useState(false);
const [search, setSearch] = useState("");
const ref = useRef<HTMLDivElement>(null);
const available = groups.filter((g) => !sharedGroupIds.has(g.id));
const filtered = search
? available.filter((g) => g.name.toLowerCase().includes(search.toLowerCase()))
: available;
useEffect(() => {
const handler = (e: MouseEvent) => {
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
};
document.addEventListener("mousedown", handler);
return () => document.removeEventListener("mousedown", handler);
}, []);
if (available.length === 0) return null;
return (
<div className="relative" ref={ref}>
<button
onClick={() => setOpen((o) => !o)}
className="flex items-center gap-1 text-xs text-muted hover:text-foreground transition-colors border border-dashed border-border rounded px-2 py-0.5 mt-1"
>
<Users className="h-3 w-3" /> Share with a group
</button>
{open && (
<div className="absolute left-0 top-full mt-1 z-20 bg-surface border border-border rounded-lg shadow-lg w-52">
{groups.length > 5 && (
<div className="p-2 border-b border-border">
<Input
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder="Search groups…"
className="h-7 text-xs"
autoFocus
/>
</div>
)}
<div className="max-h-48 overflow-y-auto py-1">
{filtered.map((g) => (
<button
key={g.id}
className="w-full text-left px-3 py-1.5 text-sm hover:bg-muted/20 transition-colors"
onClick={() => { onShare(g.id); setOpen(false); setSearch(""); }}
>
{g.name}
</button>
))}
{filtered.length === 0 && (
<p className="text-xs text-muted px-3 py-2">No groups available</p>
)}
</div>
</div>
)}
</div>
);
}
// ── Main component ────────────────────────────────────────────────────────────
export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }: Props) {
const queryClient = useQueryClient();
const [rawOpen, setRawOpen] = useState(false);
const [editingTitle, setEditingTitle] = useState(false);
const [titleValue, setTitleValue] = useState("");
const [editingType, setEditingType] = useState(false);
const [tagInput, setTagInput] = useState("");
const [canDeleteNew, setCanDeleteNew] = useState(false);
const titleInputRef = useRef<HTMLInputElement>(null);
useEffect(() => {
if (doc) {
setTitleValue(doc.title ?? "");
setEditingTitle(false);
setEditingType(false);
setRawOpen(false);
setTagInput("");
}
}, [doc?.id]);
useEffect(() => {
if (editingTitle) titleInputRef.current?.focus();
}, [editingTitle]);
const { data: allCategories = [] } = useQuery({
queryKey: ["categories"],
queryFn: listCategories,
});
const { data: myGroups = [] } = useQuery({
queryKey: ["my-groups"],
queryFn: getMyGroups,
enabled: isOwner,
});
const { data: shares = [] } = useQuery({
queryKey: ["document-shares", doc?.id],
queryFn: () => getDocumentShares(doc!.id),
enabled: isOwner && !!doc,
});
const invalidateDoc = useCallback(() => {
queryClient.invalidateQueries({ queryKey: ["documents"] });
}, [queryClient]);
const titleMut = useMutation({
mutationFn: (title: string) => updateDocumentTitle(doc!.id, title),
onSuccess: () => { invalidateDoc(); setEditingTitle(false); },
});
const typeMut = useMutation({
mutationFn: (document_type: string) => updateDocumentType(doc!.id, document_type),
onSuccess: () => { invalidateDoc(); setEditingType(false); },
});
const addTagMut = useMutation({
mutationFn: (tags: string[]) => updateDocumentTags(doc!.id, tags),
onSuccess: invalidateDoc,
});
const assignCatMut = useMutation({
mutationFn: (catId: string) => assignCategory(doc!.id, catId),
onSuccess: invalidateDoc,
});
const removeCatMut = useMutation({
mutationFn: (catId: string) => removeCategory(doc!.id, catId),
onSuccess: invalidateDoc,
});
const reprocessMut = useMutation({
mutationFn: () => reprocessDocument(doc!.id),
onSuccess: invalidateDoc,
});
const deleteMut = useMutation({
mutationFn: () => deleteDocument(doc!.id),
onSuccess: () => { invalidateDoc(); onDeleted(doc!.id); },
});
const confirmFolderMut = useMutation({
mutationFn: () => confirmFolderSuggestion(doc!.id),
onSuccess: invalidateDoc,
});
const rejectFolderMut = useMutation({
mutationFn: () => rejectFolderSuggestion(doc!.id),
onSuccess: invalidateDoc,
});
const confirmFilenameMut = useMutation({
mutationFn: () => confirmFilenameSuggestion(doc!.id),
onSuccess: invalidateDoc,
});
const rejectFilenameMut = useMutation({
mutationFn: () => rejectFilenameSuggestion(doc!.id),
onSuccess: invalidateDoc,
});
const addShareMut = useMutation({
mutationFn: ({ groupId, canDelete }: { groupId: string; canDelete: boolean }) =>
addDocumentShare(doc!.id, groupId, canDelete),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["document-shares", doc!.id] });
setCanDeleteNew(false);
},
});
const removeShareMut = useMutation({
mutationFn: (groupId: string) => removeDocumentShare(doc!.id, groupId),
onSuccess: () => queryClient.invalidateQueries({ queryKey: ["document-shares", doc!.id] }),
});
if (!doc) return null;
const extractedData = (() => {
try { return doc.extracted_data ? JSON.parse(doc.extracted_data) : null; }
catch { return null; }
})();
const tags: string[] = (() => {
try { return doc.tags ? JSON.parse(doc.tags) : []; }
catch { return []; }
})();
const suggestedCategories: string[] = (() => {
try { return extractedData?.suggested_categories ?? []; }
catch { return []; }
})();
const displayKeys = extractedData
? Object.entries(extractedData).filter(([k]) => !["tags", "suggested_categories"].includes(k))
: [];
const sharedGroupIds = new Set(shares.map((s) => s.group_id));
function addTag() {
const tag = tagInput.trim();
if (!tag || tags.includes(tag)) return;
addTagMut.mutate([...tags, tag]);
setTagInput("");
}
function removeTag(tag: string) {
addTagMut.mutate(tags.filter((t) => t !== tag));
}
return (
<div
className="fixed inset-0 z-40"
onClick={(e) => { if (e.target === e.currentTarget) onClose(); }}
>
{/* Overlay backdrop (subtle) */}
<div className="absolute inset-0 bg-background/20" />
{/* Panel */}
<div className="absolute inset-y-0 right-0 w-[480px] bg-surface border-l border-border shadow-2xl flex flex-col overflow-hidden z-50">
{/* Header */}
<div className="flex items-center gap-3 px-5 py-4 border-b border-border shrink-0">
<div className="flex-1 min-w-0">
<p className="text-xs text-muted truncate">{doc.filename}</p>
</div>
<div className="flex items-center gap-1 shrink-0">
{isOwner && (
<>
<Button
variant="ghost"
size="sm"
onClick={() => viewDocument(doc.id)}
className="h-7 px-2 gap-1 text-xs"
>
<Eye className="h-3.5 w-3.5" /> View
</Button>
<Button
variant="ghost"
size="sm"
onClick={() => downloadDocument(doc.id, doc.filename)}
className="h-7 px-2 gap-1 text-xs"
>
<Download className="h-3.5 w-3.5" /> Download
</Button>
</>
)}
{!isOwner && (
<>
<Button variant="ghost" size="sm" onClick={() => viewDocument(doc.id)} className="h-7 px-2 gap-1 text-xs">
<Eye className="h-3.5 w-3.5" /> View
</Button>
<Button variant="ghost" size="sm" onClick={() => downloadDocument(doc.id, doc.filename)} className="h-7 px-2 gap-1 text-xs">
<Download className="h-3.5 w-3.5" /> Download
</Button>
</>
)}
<button onClick={onClose} className="ml-1 text-muted hover:text-foreground transition-colors">
<X className="h-5 w-5" />
</button>
</div>
</div>
{/* Scrollable body */}
<div className="flex-1 overflow-y-auto px-5 py-4 space-y-5">
{/* Metadata row */}
<div className="flex items-center gap-3 flex-wrap text-xs text-muted">
<span className="flex items-center gap-1.5">
<span className={cn("h-2 w-2 rounded-full", STATUS_COLORS[doc.status] ?? "bg-gray-400")} />
{doc.status}
</span>
<span>{formatBytes(doc.file_size)}</span>
<span>Uploaded {formatDate(doc.created_at)}</span>
{doc.processed_at && <span>Processed {formatDate(doc.processed_at)}</span>}
{doc.source === "watch" && (
<span className="text-primary/60">Watch-ingested</span>
)}
</div>
{/* Error */}
{doc.status === "failed" && doc.error_message && (
<div className="text-xs text-red-500 bg-red-500/10 rounded-md px-3 py-2">
Error: {doc.error_message}
</div>
)}
{/* Title */}
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">Title</p>
{isOwner && editingTitle ? (
<form
onSubmit={(e) => { e.preventDefault(); titleMut.mutate(titleValue); }}
className="flex gap-2"
>
<Input
ref={titleInputRef}
value={titleValue}
onChange={(e) => setTitleValue(e.target.value)}
className="h-8 text-sm flex-1"
disabled={titleMut.isPending}
onKeyDown={(e) => { if (e.key === "Escape") setEditingTitle(false); }}
/>
<button type="submit" disabled={titleMut.isPending} className="text-primary disabled:opacity-50">
<Check className="h-4 w-4" />
</button>
<button type="button" onClick={() => setEditingTitle(false)} className="text-muted hover:text-foreground">
<X className="h-4 w-4" />
</button>
</form>
) : (
<div className="flex items-center gap-2 group">
<span className={cn("text-sm", !doc.title && "text-muted italic")}>
{doc.title ?? "No title"}
</span>
{isOwner && (
<button
onClick={() => { setTitleValue(doc.title ?? ""); setEditingTitle(true); }}
className="text-muted opacity-0 group-hover:opacity-100 hover:text-foreground transition-all"
>
<Pencil className="h-3.5 w-3.5" />
</button>
)}
</div>
)}
</div>
{/* Document type */}
{isOwner && (
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">Type</p>
{editingType ? (
<div className="flex gap-1 flex-wrap">
{DOC_TYPES.map((t) => (
<button
key={t}
onClick={() => typeMut.mutate(t)}
disabled={typeMut.isPending}
className={cn(
"text-xs px-2 py-0.5 rounded border transition-colors",
doc.document_type === t
? "border-primary text-primary bg-primary/10"
: "border-border text-muted hover:border-foreground hover:text-foreground"
)}
>
{t}
</button>
))}
<button onClick={() => setEditingType(false)} className="text-muted hover:text-foreground">
<X className="h-4 w-4" />
</button>
</div>
) : (
<div className="flex items-center gap-2 group">
<span className={cn("text-sm", !doc.document_type && "text-muted italic")}>
{doc.document_type ?? "Unknown"}
</span>
<button
onClick={() => setEditingType(true)}
className="text-muted opacity-0 group-hover:opacity-100 hover:text-foreground transition-all"
>
<Pencil className="h-3.5 w-3.5" />
</button>
</div>
)}
</div>
)}
{/* AI Suggestions */}
{isOwner && (doc.suggested_folder || doc.suggested_filename) && (
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">
AI Suggestions
</p>
<div className="space-y-2">
{doc.suggested_folder && (
<div className="flex items-center gap-2 text-sm bg-amber-500/10 border border-amber-500/20 rounded-md px-3 py-2">
<span className="flex-1">
<span className="text-xs text-muted mr-1">Folder:</span>
{doc.suggested_folder}
</span>
<button
onClick={() => confirmFolderMut.mutate()}
disabled={confirmFolderMut.isPending || rejectFolderMut.isPending}
className="text-xs text-emerald-600 hover:text-emerald-700 font-medium disabled:opacity-50"
>
Apply
</button>
<button
onClick={() => rejectFolderMut.mutate()}
disabled={confirmFolderMut.isPending || rejectFolderMut.isPending}
className="text-muted hover:text-red-500 transition-colors disabled:opacity-50"
>
<X className="h-3.5 w-3.5" />
</button>
</div>
)}
{doc.suggested_filename && (
<div className="flex items-center gap-2 text-sm bg-amber-500/10 border border-amber-500/20 rounded-md px-3 py-2">
<span className="flex-1">
<span className="text-xs text-muted mr-1">Title:</span>
{doc.suggested_filename}
</span>
<button
onClick={() => confirmFilenameMut.mutate()}
disabled={confirmFilenameMut.isPending || rejectFilenameMut.isPending}
className="text-xs text-emerald-600 hover:text-emerald-700 font-medium disabled:opacity-50"
>
Apply
</button>
<button
onClick={() => rejectFilenameMut.mutate()}
disabled={confirmFilenameMut.isPending || rejectFilenameMut.isPending}
className="text-muted hover:text-red-500 transition-colors disabled:opacity-50"
>
<X className="h-3.5 w-3.5" />
</button>
</div>
)}
</div>
</div>
)}
{/* Extracted data */}
{displayKeys.length > 0 && (
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">
Extracted Data
</p>
<table className="w-full text-xs">
<tbody>
{displayKeys.map(([k, v]) => (
<tr key={k} className="border-b border-border/50 last:border-0">
<td className="py-1.5 pr-3 text-muted font-medium w-1/3 align-top">
{k.replace(/_/g, " ")}
</td>
<td className="py-1.5 text-foreground align-top break-words">
{v === null || v === undefined ? (
<span className="text-muted"></span>
) : Array.isArray(v) ? (
<span className="font-mono">{JSON.stringify(v)}</span>
) : (
String(v)
)}
</td>
</tr>
))}
</tbody>
</table>
</div>
)}
{/* Categories */}
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">Categories</p>
<div className="flex flex-wrap gap-1.5">
{doc.categories.map((cat) => (
<span
key={cat.id}
className="flex items-center gap-1 text-xs bg-primary/10 text-primary rounded-full px-2.5 py-0.5"
>
{cat.name}
{isOwner && (
<button
onClick={() => removeCatMut.mutate(cat.id)}
disabled={removeCatMut.isPending}
className="hover:text-red-500 transition-colors disabled:opacity-50"
>
<X className="h-3 w-3" />
</button>
)}
</span>
))}
{isOwner && (
<CategoryCombobox
categories={allCategories}
assigned={doc.categories}
onAssign={(id) => assignCatMut.mutate(id)}
/>
)}
</div>
{/* AI-suggested categories */}
{isOwner && suggestedCategories.length > 0 && (
<div className="mt-2">
<p className="text-xs text-muted mb-1">Suggested by AI:</p>
<div className="flex flex-wrap gap-1.5">
{suggestedCategories
.filter((name) => !doc.categories.some((c) => c.name.toLowerCase() === name.toLowerCase()))
.map((name) => {
const exists = allCategories.find(
(c) => c.name.toLowerCase() === name.toLowerCase()
);
return (
<span
key={name}
className="flex items-center gap-1 text-xs bg-amber-500/10 text-amber-700 dark:text-amber-400 rounded-full px-2.5 py-0.5 border border-amber-500/20"
>
{name}
{exists ? (
<button
onClick={() => assignCatMut.mutate(exists.id)}
className="text-emerald-600 hover:text-emerald-700"
title="Assign"
>
<Check className="h-3 w-3" />
</button>
) : (
<button
onClick={async () => {
// Create the category then assign it
const { createCategory } = await import("@/api/client");
const cat = await createCategory(name);
queryClient.invalidateQueries({ queryKey: ["categories"] });
assignCatMut.mutate(cat.id);
}}
className="text-emerald-600 hover:text-emerald-700 text-[10px] font-medium"
title="Create & assign"
>
+
</button>
)}
</span>
);
})}
</div>
</div>
)}
</div>
{/* Tags */}
{isOwner && (
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">Tags</p>
<div className="flex flex-wrap gap-1.5 mb-2">
{tags.map((tag) => (
<span
key={tag}
className="flex items-center gap-1 text-xs bg-muted/20 rounded-full px-2.5 py-0.5"
>
{tag}
<button onClick={() => removeTag(tag)} className="text-muted hover:text-red-500">
<X className="h-3 w-3" />
</button>
</span>
))}
</div>
<form
onSubmit={(e) => { e.preventDefault(); addTag(); }}
className="flex gap-2"
>
<Input
value={tagInput}
onChange={(e) => setTagInput(e.target.value)}
placeholder="Add tag…"
className="h-7 text-xs flex-1"
disabled={addTagMut.isPending}
/>
<button
type="submit"
disabled={!tagInput.trim() || addTagMut.isPending}
className="text-primary disabled:opacity-50"
>
<Plus className="h-4 w-4" />
</button>
</form>
</div>
)}
{/* Sharing */}
{isOwner && (
<div>
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5">Sharing</p>
{shares.length === 0 && (
<p className="text-xs text-muted mb-1">Not shared with any groups</p>
)}
<div className="space-y-1 mb-1">
{shares.map((share) => {
const group = myGroups.find((g) => g.id === share.group_id);
return (
<div key={share.id} className="flex items-center gap-2 text-sm">
<Users className="h-3.5 w-3.5 text-muted shrink-0" />
<span className="flex-1 text-sm">{group?.name ?? share.group_id}</span>
{share.can_delete && (
<span title="Group members can delete this document">
<Trash2 className="h-3 w-3 text-muted shrink-0" />
</span>
)}
<button
onClick={() => removeShareMut.mutate(share.group_id)}
disabled={removeShareMut.isPending}
className="text-muted hover:text-red-500 transition-colors disabled:opacity-50"
title="Stop sharing"
>
<UserMinus className="h-3.5 w-3.5" />
</button>
</div>
);
})}
</div>
<label className="flex items-center gap-1.5 text-xs text-muted cursor-pointer mt-1 mb-0.5 select-none">
<input
type="checkbox"
checked={canDeleteNew}
onChange={(e) => setCanDeleteNew(e.target.checked)}
className="h-3 w-3 accent-primary"
/>
Allow group members to delete
</label>
<GroupCombobox
groups={myGroups}
sharedGroupIds={sharedGroupIds}
onShare={(id) => addShareMut.mutate({ groupId: id, canDelete: canDeleteNew })}
/>
</div>
)}
{/* Owner/permitted actions */}
{(isOwner || doc.viewer_can_delete) && (
<div className="flex gap-2 pt-1">
{isOwner && (
<Button
variant="outline"
size="sm"
onClick={() => reprocessMut.mutate()}
disabled={reprocessMut.isPending || doc.status === "pending" || doc.status === "processing"}
className="gap-1.5"
>
<RefreshCw className={cn("h-3.5 w-3.5", reprocessMut.isPending && "animate-spin")} />
Re-analyse
</Button>
)}
{doc.viewer_can_delete && (
<Button
variant="outline"
size="sm"
onClick={() => {
if (confirm(`Delete "${doc.title ?? doc.filename}"? This cannot be undone.`)) {
deleteMut.mutate();
}
}}
disabled={deleteMut.isPending}
className="gap-1.5 text-red-500 border-red-200 hover:bg-red-50 dark:hover:bg-red-900/20"
>
<Trash2 className="h-3.5 w-3.5" />
Delete
</Button>
)}
</div>
)}
{/* Raw text */}
{doc.source && (
<div>
<button
onClick={() => setRawOpen((o) => !o)}
className="flex items-center gap-1.5 text-xs text-muted hover:text-foreground transition-colors"
>
{rawOpen ? <ChevronDown className="h-3.5 w-3.5" /> : <ChevronRight className="h-3.5 w-3.5" />}
Extracted text
</button>
{rawOpen && (
<pre className="mt-2 text-xs bg-muted/10 border border-border rounded-md p-3 overflow-y-auto max-h-64 whitespace-pre-wrap break-words font-mono">
{/* raw_text not in DocumentOut — show message */}
(Raw text is stored server-side; use the backend API to retrieve it.)
</pre>
)}
</div>
)}
</div>
</div>
</div>
);
}
@@ -0,0 +1,245 @@
import { useState, useRef, useEffect } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { X, Pencil, Trash2, Check, Lock } from "lucide-react";
import {
listCategories,
renameCategory,
deleteCategory,
getMyGroups,
getMe,
type CategoryOut,
ApiError,
} from "@/api/client";
import { Input } from "@/components/ui/input";
import { Button } from "@/components/ui/button";
interface Props {
onClose: () => void;
}
export default function ManageCategoriesDialog({ onClose }: Props) {
const queryClient = useQueryClient();
const { data: categories = [] } = useQuery({
queryKey: ["categories"],
queryFn: listCategories,
});
const { data: myGroups = [] } = useQuery({
queryKey: ["my-groups"],
queryFn: getMyGroups,
});
const { data: me } = useQuery({ queryKey: ["me"], queryFn: getMe });
const isSuperuser = me?.is_admin ?? false;
// Set of group IDs for which the current user is a group admin
const adminGroupIds = new Set(myGroups.filter((g) => g.is_group_admin).map((g) => g.id));
const [editingId, setEditingId] = useState<string | null>(null);
const [editValue, setEditValue] = useState("");
const [editError, setEditError] = useState<string | null>(null);
const [search, setSearch] = useState("");
const editInputRef = useRef<HTMLInputElement>(null);
useEffect(() => {
if (editingId) editInputRef.current?.focus();
}, [editingId]);
const renameMut = useMutation({
mutationFn: ({ id, name }: { id: string; name: string }) => renameCategory(id, name),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["categories"] });
setEditingId(null);
setEditError(null);
},
onError: (err) => {
setEditError(err instanceof ApiError ? err.message : "Rename failed");
},
});
const deleteMut = useMutation({
mutationFn: deleteCategory,
onSuccess: () => queryClient.invalidateQueries({ queryKey: ["categories"] }),
});
function canManage(cat: CategoryOut): boolean {
if (isSuperuser) return true;
if (cat.scope === "personal") return true;
if (cat.scope === "group") return cat.group_id != null && adminGroupIds.has(cat.group_id);
return false; // system — non-admin cannot manage
}
const filtered = search
? categories.filter((c) => c.name.toLowerCase().includes(search.toLowerCase()))
: categories;
// Group by scope
const personal = filtered.filter((c) => c.scope === "personal");
const system = filtered.filter((c) => c.scope === "system");
// Group-scoped categories grouped by group_id
const groupCats = filtered.filter((c) => c.scope === "group");
const groupMap = new Map<string, { name: string; cats: CategoryOut[] }>();
for (const cat of groupCats) {
if (!cat.group_id) continue;
if (!groupMap.has(cat.group_id)) {
const grp = myGroups.find((g) => g.id === cat.group_id);
groupMap.set(cat.group_id, { name: grp?.name ?? cat.group_id, cats: [] });
}
groupMap.get(cat.group_id)!.cats.push(cat);
}
function startEdit(id: string, name: string) {
setEditingId(id);
setEditValue(name);
setEditError(null);
}
function submitEdit(id: string) {
const name = editValue.trim();
if (!name) return;
if (name === categories.find((c) => c.id === id)?.name) {
setEditingId(null);
return;
}
renameMut.mutate({ id, name });
}
function renderCat(cat: CategoryOut) {
const manageable = canManage(cat);
return (
<div
key={cat.id}
className="flex items-center gap-2 py-1.5 px-2 rounded-md hover:bg-muted/10 group"
>
{editingId === cat.id ? (
<div className="flex flex-col flex-1 gap-1">
<form
className="flex items-center gap-2"
onSubmit={(e) => { e.preventDefault(); submitEdit(cat.id); }}
>
<Input
ref={editInputRef}
value={editValue}
onChange={(e) => { setEditValue(e.target.value); setEditError(null); }}
className="h-7 text-sm flex-1"
disabled={renameMut.isPending}
onKeyDown={(e) => { if (e.key === "Escape") { setEditingId(null); setEditError(null); } }}
/>
<button
type="submit"
disabled={!editValue.trim() || renameMut.isPending}
className="text-primary hover:text-primary/80 disabled:opacity-50"
>
<Check className="h-4 w-4" />
</button>
<button
type="button"
onClick={() => { setEditingId(null); setEditError(null); }}
className="text-muted hover:text-foreground"
>
<X className="h-4 w-4" />
</button>
</form>
{editError && (
<p className="text-xs text-red-500 pl-0.5">{editError}</p>
)}
</div>
) : (
<>
{!manageable && (
<Lock className="h-3.5 w-3.5 text-muted flex-shrink-0" />
)}
<span className="flex-1 text-sm truncate">{cat.name}</span>
{manageable && (
<div className="flex items-center gap-1 opacity-0 group-hover:opacity-100 transition-opacity">
<button
onClick={() => startEdit(cat.id, cat.name)}
className="text-muted hover:text-foreground transition-colors p-0.5"
title="Rename"
>
<Pencil className="h-3.5 w-3.5" />
</button>
<button
onClick={() => {
if (confirm(`Delete category "${cat.name}"? Documents in it will be uncategorised.`)) {
deleteMut.mutate(cat.id);
}
}}
disabled={deleteMut.isPending}
className="text-muted hover:text-red-500 transition-colors p-0.5 disabled:opacity-50"
title="Delete"
>
<Trash2 className="h-3.5 w-3.5" />
</button>
</div>
)}
</>
)}
</div>
);
}
function renderSection(title: string, cats: CategoryOut[]) {
if (cats.length === 0) return null;
return (
<div className="mb-3">
<p className="text-xs font-semibold text-muted uppercase tracking-wider px-2 mb-1">{title}</p>
{cats.map(renderCat)}
</div>
);
}
const hasAny = filtered.length > 0;
return (
<div
className="fixed inset-0 z-50 flex items-center justify-center bg-background/70"
onClick={(e) => { if (e.target === e.currentTarget) onClose(); }}
>
<div className="bg-surface border border-border rounded-lg w-[520px] max-h-[80vh] flex flex-col shadow-xl">
{/* Header */}
<div className="flex items-center justify-between px-5 py-4 border-b border-border">
<h2 className="text-base font-semibold">Manage Categories</h2>
<button onClick={onClose} className="text-muted hover:text-foreground transition-colors">
<X className="h-5 w-5" />
</button>
</div>
{/* Search */}
{categories.length > 6 && (
<div className="px-5 pt-4 pb-2">
<Input
placeholder="Search categories…"
value={search}
onChange={(e) => setSearch(e.target.value)}
className="h-8 text-sm"
/>
</div>
)}
{/* List */}
<div className="flex-1 overflow-y-auto px-5 py-3 min-h-0">
{!hasAny && (
<p className="text-sm text-muted py-4 text-center">
{search ? "No categories match" : "No categories yet"}
</p>
)}
{renderSection("My Categories", personal)}
{Array.from(groupMap.entries()).map(([, { name, cats }]) =>
renderSection(`Group: ${name}`, cats)
)}
{renderSection("System", system)}
</div>
{/* Footer */}
<div className="px-5 py-4 border-t border-border">
<Button variant="outline" size="sm" onClick={onClose}>
Close
</Button>
</div>
</div>
</div>
);
}
+2 -64
View File
@@ -12,7 +12,6 @@ import {
LogOut,
UserCircle,
FileText,
Folder,
Users,
UsersRound,
Palette,
@@ -20,7 +19,7 @@ import {
import { Button } from "@/components/ui/button";
import ThemeToggle from "@/components/ThemeToggle";
import { useAuth } from "@/hooks/useAuth";
import { getMe, listCategories } from "@/api/client";
import { getMe } from "@/api/client";
import { cn } from "@/lib/utils";
export default function Sidebar() {
@@ -30,11 +29,9 @@ export default function Sidebar() {
const { data: user } = useQuery({ queryKey: ["me"], queryFn: getMe });
const isAppsRoute = location.pathname.startsWith("/apps");
const isDocsRoute = location.pathname.startsWith("/apps/documents");
const isAdminRoute = location.pathname.startsWith("/admin");
const [appsOpen, setAppsOpen] = useState(isAppsRoute);
const [docsOpen, setDocsOpen] = useState(isDocsRoute);
const [adminOpen, setAdminOpen] = useState(isAdminRoute);
// Auto-open sections when navigating to their routes
@@ -42,20 +39,10 @@ export default function Sidebar() {
if (isAppsRoute) setAppsOpen(true);
}, [isAppsRoute]);
useEffect(() => {
if (isDocsRoute) setDocsOpen(true);
}, [isDocsRoute]);
useEffect(() => {
if (isAdminRoute) setAdminOpen(true);
}, [isAdminRoute]);
const { data: categories = [] } = useQuery({
queryKey: ["categories"],
queryFn: listCategories,
enabled: appsOpen && docsOpen && !!user,
});
const navItemClass = (isActive: boolean) =>
cn(
"flex items-center rounded-lg transition-colors",
@@ -74,15 +61,6 @@ export default function Sidebar() {
: "text-muted hover:bg-muted/20 hover:text-foreground"
);
const subSubItemClass = (isActive: boolean) =>
cn(
"flex items-center rounded-lg transition-colors text-sm",
"pl-12 pr-3 py-1 gap-2",
isActive
? "bg-primary/10 text-primary"
: "text-muted hover:bg-muted/20 hover:text-foreground"
);
return (
<aside
className={cn(
@@ -147,53 +125,13 @@ export default function Sidebar() {
{/* Apps sub-items — only when sidebar is expanded and appsOpen */}
{sidebarExpanded && appsOpen && (
<div className="mt-0.5 space-y-0.5">
{/* Documents service */}
<div>
<div
className={cn(
"flex items-center rounded-lg transition-colors text-sm",
isDocsRoute
? "bg-primary/10 text-primary"
: "text-muted hover:bg-muted/20 hover:text-foreground"
)}
>
<NavLink
to="/apps/documents"
end
className="flex items-center gap-2 pl-8 pr-2 py-1.5 flex-1 min-w-0"
className={({ isActive }) => subItemClass(isActive)}
>
<FileText className="h-4 w-4 shrink-0" />
<span className="whitespace-nowrap">Documents</span>
</NavLink>
<button
onClick={() => setDocsOpen((o) => !o)}
className="px-2 py-1.5 rounded-r-lg"
aria-label={docsOpen ? "Collapse documents" : "Expand documents"}
>
{docsOpen ? (
<ChevronDown className="h-3.5 w-3.5 shrink-0" />
) : (
<ChevronRight className="h-3.5 w-3.5 shrink-0" />
)}
</button>
</div>
{/* Categories */}
{docsOpen && (
<div className="mt-0.5 space-y-0.5">
{categories.map((cat) => (
<NavLink
key={cat.id}
to={`/apps/documents?category_id=${cat.id}`}
className={({ isActive }) => subSubItemClass(isActive)}
>
<Folder className="h-3.5 w-3.5 shrink-0" />
<span className="whitespace-nowrap truncate">{cat.name}</span>
</NavLink>
))}
</div>
)}
</div>
</div>
)}
</div>
+292
View File
@@ -0,0 +1,292 @@
import { useState, useRef, useEffect } from "react";
import { useSearchParams } from "react-router-dom";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import { Files, User, Users, Folder, Plus, Settings2, Check, X } from "lucide-react";
import {
listCategories,
createCategory,
getMyGroups,
type CategoryOut,
ApiError,
} from "@/api/client";
import { Input } from "@/components/ui/input";
import { cn } from "@/lib/utils";
import ManageCategoriesDialog from "@/components/ManageCategoriesDialog";
// PascalCase-with-dashes naming convention
const NAME_RE = /^[A-Z][a-zA-Z0-9]*(-[A-Z][a-zA-Z0-9]*)*$/;
export default function SourcePanel() {
const [searchParams, setSearchParams] = useSearchParams();
const queryClient = useQueryClient();
const currentView = searchParams.get("view") ?? "all";
const currentCategoryId = searchParams.get("category_id");
const [catSearch, setCatSearch] = useState("");
const [addingCat, setAddingCat] = useState(false);
const [newCatName, setNewCatName] = useState("");
const [newCatGroupId, setNewCatGroupId] = useState<string>("");
const [nameError, setNameError] = useState<string | null>(null);
const [manageOpen, setManageOpen] = useState(false);
const addInputRef = useRef<HTMLInputElement>(null);
const { data: categories = [] } = useQuery({
queryKey: ["categories"],
queryFn: listCategories,
});
const { data: myGroups = [] } = useQuery({
queryKey: ["my-groups"],
queryFn: getMyGroups,
});
const createMut = useMutation({
mutationFn: ({ name, groupId }: { name: string; groupId?: string }) =>
createCategory(name, groupId),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["categories"] });
setNewCatName("");
setNewCatGroupId("");
setAddingCat(false);
setNameError(null);
},
onError: (err) => {
setNameError(err instanceof ApiError ? err.message : "Failed to create category");
},
});
useEffect(() => {
if (addingCat) addInputRef.current?.focus();
}, [addingCat]);
const filteredCats = catSearch
? categories.filter((c) => c.name.toLowerCase().includes(catSearch.toLowerCase()))
: categories;
// Split by scope for display
const personalCats = filteredCats.filter((c) => c.scope === "personal");
const systemCats = filteredCats.filter((c) => c.scope === "system");
const groupCatMap = new Map<string, { name: string; cats: CategoryOut[] }>();
for (const cat of filteredCats.filter((c) => c.scope === "group")) {
if (!cat.group_id) continue;
if (!groupCatMap.has(cat.group_id)) {
const grp = myGroups.find((g) => g.id === cat.group_id);
groupCatMap.set(cat.group_id, { name: grp?.name ?? cat.group_id, cats: [] });
}
groupCatMap.get(cat.group_id)!.cats.push(cat);
}
function setView(view: string) {
setSearchParams((prev) => {
const next = new URLSearchParams(prev);
next.set("view", view);
next.delete("category_id");
next.set("page", "1");
return next;
});
}
function selectCategory(cat: CategoryOut) {
setSearchParams((prev) => {
const next = new URLSearchParams(prev);
next.delete("view");
next.set("category_id", cat.id);
next.set("page", "1");
return next;
});
}
function handleAddSubmit(e: React.FormEvent) {
e.preventDefault();
const name = newCatName.trim();
if (!name) return;
if (!NAME_RE.test(name)) {
setNameError(
"Must start with a capital letter. Join multiple words with dashes, each capitalised (e.g. Vendor-Invoices)."
);
return;
}
setNameError(null);
createMut.mutate({ name, groupId: newCatGroupId || undefined });
}
const viewItemClass = (active: boolean) =>
cn(
"flex items-center gap-2 w-full px-2 py-1.5 rounded-md text-sm transition-colors",
active
? "bg-primary/10 text-primary font-medium"
: "text-muted hover:bg-muted/20 hover:text-foreground"
);
const catItemClass = (active: boolean) =>
cn(
"flex items-center gap-2 w-full px-2 py-1 rounded-md text-sm transition-colors",
active
? "bg-primary/10 text-primary font-medium"
: "text-muted hover:bg-muted/20 hover:text-foreground"
);
function renderCatSection(label: string, cats: CategoryOut[]) {
if (cats.length === 0) return null;
return (
<div className="mb-1">
<p className="text-[10px] font-semibold text-muted uppercase tracking-wider px-2 pt-1.5 pb-0.5">
{label}
</p>
{cats.map((cat) => (
<button
key={cat.id}
className={catItemClass(currentCategoryId === cat.id)}
onClick={() => selectCategory(cat)}
>
<Folder className="h-3.5 w-3.5 shrink-0" />
<span className="truncate">{cat.name}</span>
</button>
))}
</div>
);
}
const hasSections =
personalCats.length > 0 ||
systemCats.length > 0 ||
groupCatMap.size > 0;
return (
<>
<aside className="w-56 flex flex-col border-r border-border bg-surface shrink-0 h-screen">
{/* Views */}
<div className="p-3 border-b border-border">
<p className="text-xs font-semibold text-muted uppercase tracking-wider mb-1.5 px-1">
Views
</p>
<button
className={viewItemClass(!currentCategoryId && (currentView === "all" || !currentView))}
onClick={() => setView("all")}
>
<Files className="h-4 w-4 shrink-0" />
All Documents
</button>
<button
className={viewItemClass(!currentCategoryId && currentView === "mine")}
onClick={() => setView("mine")}
>
<User className="h-4 w-4 shrink-0" />
Mine
</button>
<button
className={viewItemClass(!currentCategoryId && currentView === "shared")}
onClick={() => setView("shared")}
>
<Users className="h-4 w-4 shrink-0" />
Shared with me
</button>
</div>
{/* Categories */}
<div className="flex-1 flex flex-col min-h-0 p-3">
<div className="flex items-center justify-between mb-1.5 px-1">
<p className="text-xs font-semibold text-muted uppercase tracking-wider">
Categories
</p>
<button
onClick={() => setManageOpen(true)}
className="text-muted hover:text-foreground transition-colors"
title="Manage categories"
>
<Settings2 className="h-3.5 w-3.5" />
</button>
</div>
{categories.length > 4 && (
<Input
placeholder="Search…"
value={catSearch}
onChange={(e) => setCatSearch(e.target.value)}
className="h-7 text-xs mb-2"
/>
)}
<div className="flex-1 overflow-y-auto min-h-0">
{renderCatSection("Mine", personalCats)}
{Array.from(groupCatMap.entries()).map(([, { name, cats }]) =>
renderCatSection(name, cats)
)}
{renderCatSection("System", systemCats)}
{!hasSections && catSearch && (
<p className="text-xs text-muted px-2 py-1">No categories match</p>
)}
{categories.length === 0 && !catSearch && (
<p className="text-xs text-muted px-2 py-1">No categories yet</p>
)}
</div>
{/* Add new category */}
<div className="pt-2 border-t border-border mt-2">
{addingCat ? (
<form onSubmit={handleAddSubmit} className="flex flex-col gap-1">
<div className="flex items-center gap-1">
<Input
ref={addInputRef}
value={newCatName}
onChange={(e) => { setNewCatName(e.target.value); setNameError(null); }}
placeholder="Vendor-Invoices"
className="h-7 text-xs flex-1"
disabled={createMut.isPending}
/>
<button
type="submit"
disabled={!newCatName.trim() || createMut.isPending}
className="text-primary hover:text-primary/80 disabled:opacity-50"
>
<Check className="h-4 w-4" />
</button>
<button
type="button"
onClick={() => {
setAddingCat(false);
setNewCatName("");
setNewCatGroupId("");
setNameError(null);
}}
className="text-muted hover:text-foreground"
>
<X className="h-4 w-4" />
</button>
</div>
{myGroups.length > 0 && (
<select
value={newCatGroupId}
onChange={(e) => setNewCatGroupId(e.target.value)}
className="h-7 text-xs rounded border border-border bg-surface text-foreground px-1"
disabled={createMut.isPending}
>
<option value="">Personal</option>
{myGroups.map((g) => (
<option key={g.id} value={g.id}>{g.name}</option>
))}
</select>
)}
{nameError && (
<p className="text-[10px] text-red-500 leading-tight">{nameError}</p>
)}
</form>
) : (
<button
onClick={() => setAddingCat(true)}
className="flex items-center gap-1.5 text-xs text-muted hover:text-foreground transition-colors w-full px-2 py-1"
>
<Plus className="h-3.5 w-3.5" />
New category
</button>
)}
</div>
</div>
</aside>
{manageOpen && (
<ManageCategoriesDialog onClose={() => setManageOpen(false)} />
)}
</>
);
}
+19
View File
@@ -8,6 +8,7 @@ import {
adminUpdateGroup,
adminAddGroupMember,
adminRemoveGroupMember,
adminSetGroupMemberAdmin,
adminGetUsers,
type GroupOut,
type GroupCreate,
@@ -249,6 +250,14 @@ function GroupMembersPanel({ groupId }: { groupId: string }) {
},
});
const adminMutation = useMutation({
mutationFn: ({ uId, isAdmin }: { uId: string; isAdmin: boolean }) =>
adminSetGroupMemberAdmin(groupId, uId, isAdmin),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["admin-group", groupId] });
},
});
if (isLoading) return <p style={{ padding: "8px 16px" }}>Loading members</p>;
if (!group) return null;
@@ -268,6 +277,7 @@ function GroupMembersPanel({ groupId }: { groupId: string }) {
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Email</th>
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Name</th>
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Status</th>
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Group Admin</th>
<th style={{ padding: "4px 0", fontSize: 13 }}></th>
</tr>
</thead>
@@ -279,6 +289,15 @@ function GroupMembersPanel({ groupId }: { groupId: string }) {
<td style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>
{m.is_active ? "Active" : "Inactive"}
</td>
<td style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>
<input
type="checkbox"
checked={m.is_group_admin}
disabled={adminMutation.isPending}
onChange={() => adminMutation.mutate({ uId: m.id, isAdmin: !m.is_group_admin })}
title={m.is_group_admin ? "Remove group admin" : "Make group admin"}
/>
</td>
<td style={{ padding: "4px 0" }}>
<button
style={{ fontSize: 12, color: "red" }}
File diff suppressed because it is too large Load Diff
+2 -7
View File
@@ -1,4 +1,3 @@
import axios from "axios";
import { useState } from "react";
import { Briefcase, LayoutDashboard } from "lucide-react";
import { Button } from "@/components/ui/button";
@@ -22,12 +21,8 @@ export default function LoginPage() {
try {
await login(email, password);
} catch (err) {
if (axios.isAxiosError(err)) {
const detail = err.response?.data?.detail;
setError(typeof detail === "string" ? detail : "Login failed.");
} else {
setError("Login failed.");
}
const detail = err instanceof Error ? err.message : "Login failed.";
setError(detail);
}
};
+381
View File
@@ -0,0 +1,381 @@
import { useState } from "react";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import {
getStorageConfig,
getMigrationStatus,
startStorageMigration,
cancelMigration,
type StorageBackendConfig,
type MigrationStatus,
} from "../api/client";
type Driver = "local" | "s3" | "webdav";
function Section({ title, children }: { title: string; children: React.ReactNode }) {
return (
<section style={{ marginBottom: 36 }}>
<h2 style={{ fontSize: 18, marginBottom: 16 }}>{title}</h2>
{children}
</section>
);
}
function Field({ label, children }: { label: string; children: React.ReactNode }) {
return (
<div style={{ marginBottom: 12 }}>
<label style={{ display: "block", fontSize: 13, marginBottom: 4, color: "var(--color-text-muted)" }}>
{label}
</label>
{children}
</div>
);
}
function inputStyle(disabled = false): React.CSSProperties {
return {
width: "100%",
padding: "6px 10px",
border: "1px solid var(--color-border)",
borderRadius: 6,
fontSize: 14,
background: disabled ? "var(--color-surface)" : "var(--color-background)",
color: "var(--color-text-primary)",
opacity: disabled ? 0.7 : 1,
};
}
function MigrationProgressBar({ status }: { status: MigrationStatus }) {
const pct = status.total > 0 ? Math.round((status.done / status.total) * 100) : 0;
const isBusy = ["validating", "migrating", "switching", "cleaning"].includes(status.state);
const stateLabel: Record<string, string> = {
idle: "Idle",
validating: "Validating new backend…",
migrating: `Migrating — ${status.done} / ${status.total} objects`,
switching: "Switching active backend…",
cleaning: "Cleaning up old backend…",
done: "Migration complete",
failed: "Migration failed",
cancelled: "Migration cancelled",
};
return (
<div style={{ marginTop: 12 }}>
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: 4, fontSize: 13 }}>
<span style={{ color: status.state === "failed" ? "#dc2626" : "var(--color-text-primary)" }}>
{stateLabel[status.state] ?? status.state}
</span>
{isBusy && <span style={{ color: "var(--color-text-muted)" }}>{pct}%</span>}
</div>
{(isBusy || status.state === "done") && (
<div
style={{
height: 8,
background: "var(--color-border)",
borderRadius: 4,
overflow: "hidden",
}}
>
<div
style={{
height: "100%",
width: status.state === "done" ? "100%" : `${pct}%`,
background: status.state === "done" ? "#16a34a" : "var(--color-primary)",
borderRadius: 4,
transition: "width 0.3s",
}}
/>
</div>
)}
{status.errors.length > 0 && (
<div
style={{
marginTop: 8,
padding: "8px 10px",
background: "#fef2f2",
borderRadius: 6,
fontSize: 12,
color: "#991b1b",
maxHeight: 120,
overflowY: "auto",
}}
>
{status.errors.slice(0, 10).map((e, i) => (
<div key={i} style={{ marginBottom: 2 }}>
{e}
</div>
))}
{status.errors.length > 10 && (
<div style={{ opacity: 0.7 }}>and {status.errors.length - 10} more</div>
)}
</div>
)}
</div>
);
}
export default function StorageAdminPage() {
const queryClient = useQueryClient();
const { data: storageStatus } = useQuery({
queryKey: ["storage-config"],
queryFn: getStorageConfig,
refetchInterval: 10_000,
});
const { data: migStatus, refetch: refetchMig } = useQuery({
queryKey: ["migration-status"],
queryFn: getMigrationStatus,
refetchInterval: (query) => {
const state = query.state.data?.state;
return state && ["validating", "migrating", "switching", "cleaning"].includes(state)
? 2_000
: false;
},
});
const isMigrating =
migStatus &&
["validating", "migrating", "switching", "cleaning"].includes(migStatus.state);
// ── New backend form ─────────────────────────────────────────────────────────
const [driver, setDriver] = useState<Driver>("local");
const [s3EndpointUrl, setS3EndpointUrl] = useState("");
const [s3AccessKey, setS3AccessKey] = useState("");
const [s3SecretKey, setS3SecretKey] = useState("");
const [s3Region, setS3Region] = useState("us-east-1");
const [webdavUrl, setWebdavUrl] = useState("");
const [webdavUsername, setWebdavUsername] = useState("");
const [webdavPassword, setWebdavPassword] = useState("");
const [webdavRootPath, setWebdavRootPath] = useState("/");
const [error, setError] = useState("");
function buildConfig(): StorageBackendConfig {
if (driver === "s3") {
return {
driver,
config: {
endpoint_url: s3EndpointUrl,
access_key: s3AccessKey,
secret_key: s3SecretKey,
region: s3Region,
},
};
}
if (driver === "webdav") {
return {
driver,
config: {
url: webdavUrl,
username: webdavUsername,
password: webdavPassword,
root_path: webdavRootPath,
},
};
}
return { driver: "local", config: {} };
}
const migrateMutation = useMutation({
mutationFn: startStorageMigration,
onSuccess: () => {
setError("");
refetchMig();
},
onError: (e: Error) => setError(e.message),
});
const cancelMutation = useMutation({
mutationFn: cancelMigration,
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["migration-status"] });
queryClient.invalidateQueries({ queryKey: ["storage-config"] });
},
onError: (e: Error) => setError(e.message),
});
const currentDriver = storageStatus?.backend ?? "—";
return (
<div style={{ maxWidth: 680, margin: "0 auto", padding: "32px 16px" }}>
<h1 style={{ fontSize: 24, marginBottom: 4 }}>Storage</h1>
<p style={{ color: "var(--color-text-muted)", marginBottom: 32, fontSize: 14 }}>
All uploaded files are stored through the storage-service. Switch between local filesystem,
S3-compatible cloud storage, or WebDAV (Nextcloud).
</p>
<Section title="Current backend">
<div
style={{
display: "inline-flex",
alignItems: "center",
gap: 8,
padding: "6px 12px",
background: "var(--color-surface)",
border: "1px solid var(--color-border)",
borderRadius: 6,
fontSize: 14,
}}
>
<span
style={{
width: 8,
height: 8,
borderRadius: "50%",
background: storageStatus?.status === "ok" ? "#16a34a" : "#dc2626",
flexShrink: 0,
}}
/>
<strong>{currentDriver}</strong>
{storageStatus?.status === "ok" ? " — healthy" : " — unreachable"}
</div>
</Section>
<Section title="Switch backend">
<p style={{ fontSize: 13, color: "var(--color-text-muted)", marginBottom: 16 }}>
When you click <strong>Test &amp; Migrate</strong>, all existing files will be copied to the
new backend, verified, and the switch will happen only after every file is confirmed. The
old backend is cleaned up automatically.
</p>
<Field label="Backend driver">
<select
value={driver}
onChange={(e) => setDriver(e.target.value as Driver)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
>
<option value="local">Local filesystem (default)</option>
<option value="s3">S3-compatible (MinIO / AWS S3 / Backblaze / Cloudflare R2)</option>
<option value="webdav">WebDAV (Nextcloud / )</option>
</select>
</Field>
{driver === "s3" && (
<>
<Field label="Endpoint URL (leave empty for real AWS S3)">
<input
value={s3EndpointUrl}
onChange={(e) => setS3EndpointUrl(e.target.value)}
placeholder="http://minio:9000"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Access key">
<input
value={s3AccessKey}
onChange={(e) => setS3AccessKey(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Secret key">
<input
type="password"
value={s3SecretKey}
onChange={(e) => setS3SecretKey(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Region">
<input
value={s3Region}
onChange={(e) => setS3Region(e.target.value)}
placeholder="us-east-1"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
</>
)}
{driver === "webdav" && (
<>
<Field label="Server URL">
<input
value={webdavUrl}
onChange={(e) => setWebdavUrl(e.target.value)}
placeholder="https://nextcloud.example.com"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Username">
<input
value={webdavUsername}
onChange={(e) => setWebdavUsername(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Password">
<input
type="password"
value={webdavPassword}
onChange={(e) => setWebdavPassword(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="WebDAV root path">
<input
value={webdavRootPath}
onChange={(e) => setWebdavRootPath(e.target.value)}
placeholder="/remote.php/dav/files/username"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
</>
)}
{error && (
<p style={{ color: "#dc2626", fontSize: 13, marginBottom: 8 }}>{error}</p>
)}
<div style={{ display: "flex", gap: 10, marginTop: 16 }}>
<button
onClick={() => migrateMutation.mutate(buildConfig())}
disabled={!!isMigrating || migrateMutation.isPending}
style={{
padding: "8px 18px",
background: "var(--color-primary)",
color: "#fff",
border: "none",
borderRadius: 6,
fontSize: 14,
cursor: isMigrating || migrateMutation.isPending ? "not-allowed" : "pointer",
opacity: isMigrating || migrateMutation.isPending ? 0.6 : 1,
}}
>
{migrateMutation.isPending ? "Starting…" : "Test & Migrate"}
</button>
{isMigrating && (
<button
onClick={() => cancelMutation.mutate()}
disabled={cancelMutation.isPending}
style={{
padding: "8px 18px",
background: "transparent",
color: "#dc2626",
border: "1px solid #dc2626",
borderRadius: 6,
fontSize: 14,
cursor: "pointer",
}}
>
Cancel
</button>
)}
</div>
{migStatus && migStatus.state !== "idle" && (
<MigrationProgressBar status={migStatus} />
)}
</Section>
</div>
);
}
+381
View File
@@ -0,0 +1,381 @@
# ALL_TESTS — Full Test Suite
Complete test suite covering all 20 feature areas. Run tests relevant to the changed area before merging any feature branch into `main`. Service-specific subsets live in separate files:
- `tests/backend_tests.md` — §19, §18 (auth, users, admin, groups, appearance, service health, plugins, AI/doc settings, infra/security)
- `tests/frontend_tests.md` — §19 (UI & routing)
- `tests/doc-service_tests.md` — §1016 (upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory)
- `tests/ai-service_tests.md` — §17 (AI queue & providers)
- `tests/storage-service_tests.md` — §20 (storage-service: objects, backend switching, migration)
Every test describes the exact UI action or API call to perform and the expected outcome.
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
**Admin credentials:** any superuser account created during stack setup.
**Regular user credentials:** a second non-admin account for permission boundary tests.
---
## Legend
| Symbol | Meaning |
|--------|---------|
| ✅ | Pass |
| ❌ | Fail |
| — | N/A for this change |
Mark each row before opening the PR.
---
## 1. Authentication
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 1.1 | Register new account | `POST /api/auth/register` with valid email + strong password | 201; user row created; login works immediately |
| 1.2 | Password policy — too short | Register with 7-char password | 422 with validation error |
| 1.3 | Password policy — no uppercase | Register with all-lowercase password | 422 with validation error |
| 1.4 | Password policy — no special char | Register without special character | 422 with validation error |
| 1.5 | Password policy — common word | Register with password containing "password" | 422 with validation error |
| 1.6 | Duplicate email | Register with an already-used email | 400 |
| 1.7 | Login — valid credentials | `POST /api/auth/login` with correct email + password | 200; `access_token` returned |
| 1.8 | Login — wrong password | `POST /api/auth/login` with wrong password | 401 |
| 1.9 | Login — inactive account | Admin deactivates user; attempt login | 401 |
| 1.10 | JWT expiry respected | Manually craft token with `exp` in the past; call any protected route | 401 |
| 1.11 | Logout clears session | Click Logout in UI; try navigating to `/` | Redirected to `/login` |
| 1.12 | Unauthenticated redirect | Open `/` without a token in `localStorage` | Redirected to `/login` |
---
## 2. User — Profile & Preferences
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 2.1 | Fetch own profile | `GET /api/profile/me` | 200; profile auto-created if first request |
| 2.2 | Update profile fields | `PUT /api/profile/me` with full_name, phone, position, address, date_of_birth | 200; fields persisted; visible on `/profile` page |
| 2.3 | Invalid phone format | `PUT /api/profile/me` with letters in phone field | 422 |
| 2.4 | Future date of birth | `PUT /api/profile/me` with DOB = tomorrow | 422 |
| 2.5 | DOB before 1900 | `PUT /api/profile/me` with DOB = 1899-12-31 | 422 |
| 2.6 | Fetch dashboard preferences | `GET /api/users/me/preferences` | 200; `app_ids` list |
| 2.7 | Pin an app | Dashboard → pencil button → press `+` on a card → save | Card appears in pinned grid on next load |
| 2.8 | Unpin an app | Dashboard → pencil button → press `` on a pinned card → save | Card removed from pinned grid |
| 2.9 | Pin limit (50) | `PATCH /api/users/me/preferences` with 51 app IDs | 422 |
| 2.10 | Color mode — user pref | User sets mode to "dark"; reload page | Dark theme applied; preference persists across sessions |
| 2.11 | Color mode — system fallback | User has NULL color_mode; admin default_mode = "light" | Light theme applied |
---
## 3. Admin — Users
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 3.1 | List all users | Admin → `/admin/users` | All registered users shown in table |
| 3.2 | Create user | Admin clicks "Create user"; fills form | 201; new user appears in list; can log in |
| 3.3 | Toggle user active | Admin clicks toggle on active user | User deactivated; login returns 401 |
| 3.4 | Delete user | Admin deletes a user | 204; user no longer in list; their documents remain (orphaned) |
| 3.5 | Non-admin access | Regular user navigates to `/admin/users` | Redirected to `/login` |
| 3.6 | Admin 404 semantics | Regular user calls `GET /api/admin/users` via curl | 404 (not 403) |
---
## 4. Admin — Groups
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 4.1 | List groups | Admin → `/admin/groups` | All groups shown with member count |
| 4.2 | Create group | Fill name + description → submit | Group appears in list; `{group_name}-admin` bootstrap group also exists (auto-created on service start) |
| 4.3 | Edit group | Click edit on group → change name → save | Name updated |
| 4.4 | Delete group | Delete group | 204; group gone; memberships cascade-deleted |
| 4.5 | Add member | Open group → search user → add | 204; user appears in member list |
| 4.6 | Remove member | Click remove on a member | User removed from group |
| 4.7 | Duplicate group name | Create group with name that already exists | 400 / validation error shown |
| 4.8 | Non-admin access | Regular user calls `GET /api/admin/groups` | 404 |
| 4.9 | Set group admin role | Admin → group detail → tick "Group admin" checkbox on a member → save | `PATCH /api/admin/groups/{id}/members/{user_id}/admin` with `{"is_group_admin": true}` returns 200; badge shown in member list |
| 4.10 | Unset group admin role | Admin unticks "Group admin" on an existing group admin member | Returns 200; badge removed; user loses group-admin privileges |
---
## 5. Admin — Appearance
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 5.1 | List themes | Admin → `/admin/appearance` | Built-in themes + any custom themes shown |
| 5.2 | Switch active theme | Select a different theme → save | All users see the new theme on next load |
| 5.3 | Create custom theme | Admin → create theme. Required fields: `id` (slug), `label`, `light` (CSS vars object), `dark` (CSS vars object) | 201; theme appears in selector; can be activated |
| 5.4 | Edit custom theme | Admin edits colour values on a custom theme | Colours update live after activation |
| 5.5 | Delete custom theme | Admin deletes a custom theme | 204; theme gone from selector; active theme reverts to default |
| 5.6 | Set default mode | `PATCH /api/settings/appearance` with `{"theme": "<id>", "default_mode": "dark"}` (both fields required) | 200; new users without a personal preference see dark mode |
---
## 6. Service Health & Dashboard
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 6.1 | Services endpoint | `GET /api/services` (authenticated) | Returns health status for doc-service and ai-service |
| 6.2 | Healthy service card | Both services running → `/apps` page. API response uses `healthy: true` (boolean), not `status: "healthy"` | Cards show "Available" badge; clicking navigates to the app |
| 6.3 | Unhealthy service card | Stop doc-service container → wait 30s → `/apps` | Doc-service card dimmed, "Unavailable", not clickable |
| 6.4 | Service recovery | Restart stopped container → wait 30s | Card returns to "Available" |
| 6.5 | Dashboard pinned cards | Pin a service → go to `/` | Pinned card appears in home grid |
| 6.6 | Customize mode | Click pencil on dashboard → toggle cards | Pinned list updates after save |
---
## 7. Plugin System
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 7.1 | List plugins | `GET /api/plugins` (authenticated) | Returns accessible plugins for current user |
| 7.2 | Superuser sees all plugins | Log in as admin → `GET /api/plugins` | All registered service plugins returned |
| 7.3 | Group member sees plugin | Add user to `doc-service-admin` group → `GET /api/plugins` | doc-service plugin returned |
| 7.4 | Unpermitted user hidden | Regular user not in any admin group → `GET /api/plugins` | Empty list (plugins hidden, not 403) |
| 7.5 | Manifest fetch | `GET /api/plugins/doc-service/manifest` as permitted user | JSON Schema + access rules returned |
| 7.6 | Settings read | `GET /api/plugins/doc-service/settings` | Current doc-service plugin settings returned |
| 7.7 | Settings write | `PATCH /api/plugins/doc-service/settings` with valid payload | 200; setting persisted to volume |
| 7.8 | Unpermitted settings access | Regular user `GET /api/plugins/doc-service/settings` | 404 |
---
## 8. AI Service Settings
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 8.1 | Read AI config | Admin (or `ai-service-admin` member) → `GET /api/settings/ai` | Config returned; API keys masked |
| 8.2 | Update provider | `PATCH /api/settings/ai` with provider = "anthropic" + valid key | 200; config persisted |
| 8.3 | Test connection | `POST /api/settings/ai/test` with valid config | 200; success response from provider |
| 8.4 | Test connection — bad key | `POST /api/settings/ai/test` with wrong API key | 502 or error detail |
| 8.5 | Read system prompts | `GET /api/settings/system-prompts` | All registered service prompts returned |
| 8.6 | Update system prompt | `PATCH /api/settings/system-prompts/doc-service` with new prompt text | 200; doc-service picks up new prompt within 30s |
| 8.7 | Non-admin access | Regular user calls any `/api/settings/ai` endpoint | 404 |
| 8.8 | `ai-service-admin` delegation | Non-superuser added to `ai-service-admin` group → accesses AI settings page | Page loads; can read and write settings |
---
## 9. Document Service Settings
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 9.1 | Read upload limits | `GET /api/settings/documents/limits` (admin or `doc-service-admin`) | `max_pdf_bytes` returned |
| 9.2 | Update upload limit | `PATCH /api/settings/documents/limits` with new value | 200; upload of oversized PDF now rejected with 413 |
| 9.3 | Non-admin access | Regular user calls `GET /api/settings/documents/limits` | 404 |
| 9.4 | Settings page loads | Admin navigates to `/apps/documents/settings` | Upload limits section + watch directory config visible |
| 9.5 | `doc-service-admin` delegation | Non-superuser added to `doc-service-admin` → navigates to settings page | Page loads; settings editable |
---
## 10. Document Upload & Processing
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 10.1 | Upload valid PDF | Drag-and-drop or file picker → select a PDF under the size limit | 202; document row appears with `status=pending`; transitions to `done` |
| 10.2 | Upload oversized PDF | Upload a PDF exceeding `max_pdf_bytes` | 413; error shown; no row created |
| 10.3 | Upload non-PDF | Upload a `.docx` or `.jpg` | 415; error shown |
| 10.4 | Multi-file upload | Select 3 PDFs at once | All 3 appear in upload queue panel; each processes independently |
| 10.5 | Upload queue panel | During upload → check bottom-right panel | Per-file status indicator; "Review →" link after each completes |
| 10.6 | Drag-and-drop overlay | Drag file over the documents page | Full-page overlay appears; drop uploads file |
| 10.7 | Processing status poll | Upload a large PDF | Table row auto-updates every 3s until status = `done` or `failed` |
| 10.8 | AI extraction result | Open slide-over for a `done` document | title, document_type, tags, extracted_data fields populated |
| 10.9 | Failed extraction | AI service down → upload PDF | Status = `failed`; error_message shown in slide-over |
| 10.10 | Re-analyse | Click "Re-analyse" in slide-over | 202; status resets to `pending`; re-processes through AI |
---
## 11. Document List & Filtering
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 11.1 | Default list | Navigate to `/apps/documents` | Own documents shown, newest first, 20 per page |
| 11.2 | Search | Type in search box (debounced 400ms) | Results filtered by title / filename / tags / type |
| 11.3 | Filter by status | Add filter chip → Status → "done" | Only completed docs shown |
| 11.4 | Filter by type | Add filter chip → Document type → "invoice" | Only invoices shown |
| 11.5 | Filter by category | Add filter chip → Category → pick one | Only docs in that category shown |
| 11.6 | Remove filter chip | Click × on a chip | Filter removed; full list restored |
| 11.7 | Sort by column | Click "Date" column header | List re-ordered; chevron indicates direction; click again reverses |
| 11.8 | Pagination | Upload > 20 docs → scroll to bottom | Page controls appear; page 2 loads next 20 |
| 11.9 | "Mine" view | Click "Mine" in SourcePanel | Only own (uploaded) documents shown |
| 11.10 | "Shared with me" view | Click "Shared with me" | Docs shared by others via groups; own docs excluded |
| 11.11 | Category filter via SourcePanel | Click a category in the left tree | Table filtered to that category's documents |
| 11.12 | URL state preserved | Apply filters → copy URL → open in new tab | Same filters applied |
---
## 12. Document Detail — Slide-over
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 12.1 | Open slide-over | Click any document row | 480px right panel slides in; metadata loaded |
| 12.2 | Inline title edit | Click pencil icon next to title → type new title → confirm | Title saved; updated in table row |
| 12.3 | Change document type | Click a type chip (Invoice, Receipt, etc.) | Type updated immediately |
| 12.4 | Edit tags | Click into tag area → type a tag → press Enter → remove a tag with × | Tags saved correctly |
| 12.5 | Assign category | Categories combobox → search → select | Category badge appears on document; table row updates |
| 12.6 | Remove category | Click × on an assigned category badge | Category removed from document |
| 12.7 | AI category suggestions | Slide-over shows "Suggested categories" | "Assign" and "Create & Assign" buttons present; clicking assigns |
| 12.8 | Confirm folder suggestion | "Confirm" button next to suggested_folder | Category created (if needed) and assigned; `suggested_folder` cleared |
| 12.9 | Reject folder suggestion | "Reject" button next to suggested_folder | `suggested_folder` cleared; no category created |
| 12.10 | Confirm filename suggestion | "Confirm" button next to suggested_filename | `title` updated to suggested value; `suggested_filename` cleared |
| 12.11 | Reject filename suggestion | "Reject" button next to suggested_filename | `suggested_filename` cleared; title unchanged |
| 12.12 | Extracted data section | Open slide-over on `done` doc | Key-value table of AI-extracted fields (vendor, amounts, dates, etc.) |
| 12.13 | Raw text section | Expand raw text collapse | First ~500k chars of extracted PDF text shown |
| 12.14 | Download | Click "Download" | Browser downloads the original PDF file |
| 12.15 | View in new tab | Click "View" | PDF opens in new browser tab; URL auto-revokes after 60s |
| 12.16 | Delete — owner | Click "Delete" → confirm dialog | Document and file removed; table row gone |
| 12.16b | Delete — admin | Admin opens any doc (not their own) → Delete → confirm | Document deleted; 204 returned |
| 12.16c | Delete — can_delete share | Group member whose share has `can_delete=true` → Delete | 204; document removed; `viewer_can_delete` was `true` in `DocumentOut` |
| 12.16d | Delete — group admin | User is group admin for a group the doc is shared with; no explicit `can_delete` flag → Delete | 204; group admin always has delete rights for docs shared with their group |
| 12.16e | Delete — watch document, admin only | Watch-ingested doc (`source=watch`); regular user → Delete | 403 (not owner); admin can delete it |
| 12.17 | Non-owner cannot edit | Recipient of shared doc opens slide-over (no can_delete, not group admin) | Edit controls (type, tags, title, delete) absent; download available |
---
## 13. Document Sharing
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 13.1 | Share from slide-over | Owner opens sharing section → selects a group from combobox → shares | Group appears in shares list; `share_count` in table row increments |
| 13.2 | Only user's own groups shown | Open group picker in share section | Only groups the current user belongs to are listed |
| 13.3 | Recipient sees shared doc | Log in as group member → "Shared with me" view | Shared document appears with primary accent border |
| 13.4 | Recipient download | Recipient clicks Download on shared doc | PDF downloaded successfully |
| 13.4b | Non-owner calls `GET /documents/{id}/shares` | Regular user on a doc they don't own | 404 (doc-service hides existence, consistent with admin 404 semantics — not 403) |
| 13.5 | Recipient cannot delete | Recipient opens slide-over | Delete button absent |
| 13.6 | Recipient cannot re-share | Recipient opens sharing section | Share controls absent |
| 13.7 | Remove share | Owner clicks remove on a group share | Group removed; `share_count` decrements; recipient no longer sees doc |
| 13.8 | Bulk share | Select multiple rows → bulk share → pick group | All selected docs shared with that group |
| 13.9 | Share count indicator | Document shared with 2 groups | Users icon in table row shows "2" |
| 13.10 | Share with non-member group | `POST /api/documents/{id}/shares` with group not in X-User-Groups | 403 / validation error |
| 13.11 | Share with can_delete enabled | Owner opens sharing section → tick "Allow group members to delete" → share | `can_delete=true` stored; trash icon badge appears next to group name in shares list |
| 13.12 | Share without can_delete (default) | Owner shares without ticking delete checkbox | `can_delete=false`; recipient sees the doc but Delete button absent in slide-over |
| 13.13 | can_delete shown in shares list | Share with can_delete=true → inspect shares list in slide-over | Trash2 icon rendered beside the group name; tooltip "Group members can delete this document" |
| 13.14 | viewer_can_delete in document list | Share with can_delete=true; log in as group member → `GET /api/documents` | `viewer_can_delete=true` in the recipient's list response for that doc |
---
## 14. Categories
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 14.1 | Create category | SourcePanel → "New category" form → submit | Category appears in tree |
| 14.2 | Rename category | Manage categories dialog → edit → save | New name reflected everywhere |
| 14.3 | Delete category | Delete category with documents assigned | 204; documents remain; category assignment removed |
| 14.4 | Category search | More than 4 categories → type in search field | Tree filtered in real time |
| 14.5 | Manage categories dialog | Click "Manage categories" | Modal shows categories grouped by scope (Personal / Group / System) with lock icons on group and system categories |
| 14.6 | New category triggers re-analysis | Create category with name similar to AI suggestion | Background re-analysis triggered (check backend logs) |
| 14.7 | Create personal category | SourcePanel → "New category" → no group selected → submit valid PascalCase name | Created with `scope=personal`; visible only to owner |
| 14.8 | Create group-scoped category | SourcePanel → "New category" → select a group → submit | Created with `scope=group`; visible to all members of that group |
| 14.9 | Group category visible to group members | Log in as another group member | Group category appears in their category list and SourcePanel |
| 14.10 | Non-member cannot see group category | Log in as user not in the group | Group category absent from list |
| 14.11 | Only group admin can rename group category | Regular group member → rename group category | 403; group admin can rename it successfully |
| 14.12 | Only group admin can delete group category | Regular group member → delete group category | 403; group admin can delete it |
| 14.13 | System categories read-only for non-admin | Regular user → Manage categories → rename/delete a system category | 403; lock icon shown; action blocked in UI |
| 14.14 | Admin can manage system categories | Superuser → rename or delete a system category | Succeeds; ManageCategoriesDialog shows edit/delete controls for system rows |
| 14.15 | PascalCase naming enforced — invalid | Create category named `my-invoices` or `Invoice Reports` | 422 with message explaining PascalCase-with-dashes format |
| 14.16 | PascalCase naming enforced — valid | Create category named `Vendor-Invoices` | 201; category created successfully |
| 14.17 | SourcePanel scope sections | Categories exist for all three scopes | SourcePanel tree shows "Mine", per-group, and "System" sections separately |
---
## 15. Bulk Actions
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 15.1 | Select rows | Tick checkboxes on multiple rows | Floating bulk actions bar appears at bottom |
| 15.2 | Bulk share | Select docs → Share with group → confirm | All selected docs shared; confirmation |
| 15.3 | Bulk delete | Select docs → Delete → confirm dialog | All selected docs deleted; bar disappears |
| 15.4 | Clear selection | Click "Clear" in bulk bar | All checkboxes deselected; bar hides |
| 15.5 | Bulk bar — "Mine" view only | Switch to "Shared with me" view | Bulk actions bar not shown (no edit rights for shared docs) |
---
## 16. Watch Directory
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 16.1 | Enable watch | Doc settings page → toggle `watch_enabled` on → save | File watcher starts; backend logs confirm |
| 16.2 | Ingest new file | Drop a PDF into the bind-mounted watch directory | Document appears in "All Documents" view with `source=watch` |
| 16.3 | Sub-folder to category | Place PDF in `watch/invoices/` | Document auto-assigned to "invoices" category |
| 16.4 | Startup scan | Restart doc-service with PDFs already in watch dir | Pre-existing PDFs ingested (idempotent — no duplicates) |
| 16.5 | AI folder suggestion | `ai_folder_suggestion` enabled → ingest file | `suggested_folder` populated; confirm/reject buttons visible in slide-over |
| 16.6 | AI rename suggestion | `ai_rename_suggestion` enabled → ingest file | `suggested_filename` populated; confirm/reject buttons visible |
| 16.7 | No-remove policy | Delete PDF from watch dir | Document record remains in DB |
| 16.8 | Disable watch | Toggle `watch_enabled` off → save | Watcher stops; new files dropped are not ingested |
| 16.9 | Watch docs visible to all users | Log in as any authenticated user | Watch-ingested docs (`user_id = "watch"`) appear in "All Documents" |
---
## 17. AI Service — Queue & Providers
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 17.1 | Health check | `GET /health` on ai-service (via backend services endpoint) | `{"status": "ok"}` |
| 17.2 | Provider health | `GET /health/provider` | Active provider name, model, configured=true |
| 17.3 | Unconfigured provider | Set provider to "anthropic" with empty API key → test connection | 503 or 502 with clear error |
| 17.4 | Sync chat | `POST /chat` with valid messages array | Response returned synchronously |
| 17.5 | Queue — async job | `POST /queue/jobs` | `job_id` returned immediately |
| 17.6 | Queue — poll job | `GET /queue/jobs/{id}` after enqueue | Returns status (`pending``done`) and result |
| 17.7 | Queue — cancel job | `DELETE /queue/jobs/{id}` before processing | Job removed; status = cancelled |
| 17.8 | Queue pause | `POST /queue/pause` | 204; current job finishes; no new jobs picked up |
| 17.9 | Queue resume | `POST /queue/resume` after pause | 204; worker resumes; pending jobs process |
| 17.10 | Priority ordering | Enqueue LOW then HIGH job | HIGH job processed first |
| 17.11 | Provider timeout | `POST /chat` when provider is unreachable | 504 returned after timeout |
---
## 18. Infrastructure & Security
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 18.1 | Non-root containers | `docker inspect <container> --format '{{.Config.User}}'` for each service | Returns `1001` (or `70` for db) |
| 18.2 | No host ports in prod | `docker compose up --build -d``docker ps` | Only port 80 (frontend) exposed; no 8000/8001/8010/5432 |
| 18.3 | backend-net isolation | `curl http://localhost:8000` from host in prod | Connection refused |
| 18.4 | Pre-commit hook runs | Stage a file with `eval("x")``git commit` | Commit blocked; security_check.py output shown |
| 18.5 | Pre-commit hook — clean code | Normal commit | Hook passes; commit succeeds |
| 18.6 | JWT algorithm none rejected | Craft token with `"alg": "none"` → call protected route | 401 |
| 18.7 | XSS — input sanitation | Enter `<script>alert(1)</script>` in title/name fields | Value stored as plain text; not executed in UI |
| 18.8 | SQL injection attempt | Pass `'; DROP TABLE documents; --` as search param | 200 with empty results; no DB error |
| 18.9 | CORS | `curl -H "Origin: http://evil.com" http://localhost/api/users/me` | Request blocked or `access-control-allow-origin` not set for that origin |
| 18.10 | Config volume persistence | Restart all containers | AI provider config + doc limits survive restart |
| 18.11 | Migration auto-apply | Start fresh stack | Both `alembic upgrade head` chains run without error; all tables created |
---
## 19. Frontend — UI & Routing
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 19.1 | PrivateRoute redirect | Open any protected route without token | Redirected to `/login` |
| 19.2 | AdminRoute redirect | Log in as non-admin → navigate to `/admin` | Redirected to `/login` |
| 19.3 | ServiceAdminRoute | Non-admin, non-group-member → navigate to `/apps/documents/settings` | Redirected (access denied) |
| 19.4 | Sidebar collapse | Click collapse button | Sidebar shrinks to icon-only; expand restores labels |
| 19.5 | Apps accordion | Click "Apps" in sidebar | Expands to show "Documents" NavLink |
| 19.6 | SourcePanel visibility | Navigate to `/apps` then `/apps/documents` | SourcePanel only visible on `/apps/documents` route |
| 19.7 | Theme toggle | Click sun/moon button | Mode switches; persists on reload |
| 19.8 | Unknown route | Navigate to `/does-not-exist` | Redirected to `/` |
| 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs |
| 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab |
| 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off |
---
## 20. Storage Service
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored |
| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload |
| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 |
| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` |
| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` |
| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 |
| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent |
| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service |
| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone |
| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket |
| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" |
| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` |
| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss |
| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done |
| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active |
| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" |
| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone |
| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent |
| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent |
| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |
+36
View File
@@ -0,0 +1,36 @@
# AI-Service Tests
Tests covering the AI provider intermediary (`ai-service:8010`): health, provider configuration, synchronous chat, the priority queue (enqueue/poll/cancel/pause/resume), and provider timeout.
Full combined suite: `tests/ALL_TESTS.md`
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
**Admin credentials:** any superuser account created during stack setup.
---
## Legend
| Symbol | Meaning |
|--------|---------|
| ✅ | Pass |
| ❌ | Fail |
| — | N/A for this change |
---
## 17. AI Service — Queue & Providers
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 17.1 | Health check | `GET /health` on ai-service (via backend services endpoint) | `{"status": "ok"}` |
| 17.2 | Provider health | `GET /health/provider` | Active provider name, model, configured=true |
| 17.3 | Unconfigured provider | Set provider to "anthropic" with empty API key → test connection | 503 or 502 with clear error |
| 17.4 | Sync chat | `POST /chat` with valid messages array | Response returned synchronously |
| 17.5 | Queue — async job | `POST /queue/jobs` | `job_id` returned immediately |
| 17.6 | Queue — poll job | `GET /queue/jobs/{id}` after enqueue | Returns status (`pending``done`) and result |
| 17.7 | Queue — cancel job | `DELETE /queue/jobs/{id}` before processing | Job removed; status = cancelled |
| 17.8 | Queue pause | `POST /queue/pause` | 204; current job finishes; no new jobs picked up |
| 17.9 | Queue resume | `POST /queue/resume` after pause | 204; worker resumes; pending jobs process |
| 17.10 | Priority ordering | Enqueue LOW then HIGH job | HIGH job processed first |
| 17.11 | Provider timeout | `POST /chat` when provider is unreachable | 504 returned after timeout |
+172
View File
@@ -0,0 +1,172 @@
# Backend Tests
Tests covering the FastAPI gateway (`backend:8000`): auth, user/profile management, admin (users/groups/appearance), service health, plugin system, AI settings, document settings, and infrastructure/security.
Full combined suite: `tests/ALL_TESTS.md`
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
**Admin credentials:** any superuser account created during stack setup.
**Regular user credentials:** a second non-admin account for permission boundary tests.
---
## Legend
| Symbol | Meaning |
|--------|---------|
| ✅ | Pass |
| ❌ | Fail |
| — | N/A for this change |
---
## 1. Authentication
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 1.1 | Register new account | `POST /api/auth/register` with valid email + strong password | 201; user row created; login works immediately |
| 1.2 | Password policy — too short | Register with 7-char password | 422 with validation error |
| 1.3 | Password policy — no uppercase | Register with all-lowercase password | 422 with validation error |
| 1.4 | Password policy — no special char | Register without special character | 422 with validation error |
| 1.5 | Password policy — common word | Register with password containing "password" | 422 with validation error |
| 1.6 | Duplicate email | Register with an already-used email | 400 |
| 1.7 | Login — valid credentials | `POST /api/auth/login` with correct email + password | 200; `access_token` returned |
| 1.8 | Login — wrong password | `POST /api/auth/login` with wrong password | 401 |
| 1.9 | Login — inactive account | Admin deactivates user; attempt login | 401 |
| 1.10 | JWT expiry respected | Manually craft token with `exp` in the past; call any protected route | 401 |
| 1.11 | Logout clears session | Click Logout in UI; try navigating to `/` | Redirected to `/login` |
| 1.12 | Unauthenticated redirect | Open `/` without a token in `localStorage` | Redirected to `/login` |
---
## 2. User — Profile & Preferences
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 2.1 | Fetch own profile | `GET /api/profile/me` | 200; profile auto-created if first request |
| 2.2 | Update profile fields | `PUT /api/profile/me` with full_name, phone, position, address, date_of_birth | 200; fields persisted; visible on `/profile` page |
| 2.3 | Invalid phone format | `PUT /api/profile/me` with letters in phone field | 422 |
| 2.4 | Future date of birth | `PUT /api/profile/me` with DOB = tomorrow | 422 |
| 2.5 | DOB before 1900 | `PUT /api/profile/me` with DOB = 1899-12-31 | 422 |
| 2.6 | Fetch dashboard preferences | `GET /api/users/me/preferences` | 200; `app_ids` list |
| 2.7 | Pin an app | Dashboard → pencil button → press `+` on a card → save | Card appears in pinned grid on next load |
| 2.8 | Unpin an app | Dashboard → pencil button → press `` on a pinned card → save | Card removed from pinned grid |
| 2.9 | Pin limit (50) | `PATCH /api/users/me/preferences` with 51 app IDs | 422 |
| 2.10 | Color mode — user pref | User sets mode to "dark"; reload page | Dark theme applied; preference persists across sessions |
| 2.11 | Color mode — system fallback | User has NULL color_mode; admin default_mode = "light" | Light theme applied |
---
## 3. Admin — Users
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 3.1 | List all users | Admin → `/admin/users` | All registered users shown in table |
| 3.2 | Create user | Admin clicks "Create user"; fills form | 201; new user appears in list; can log in |
| 3.3 | Toggle user active | Admin clicks toggle on active user | User deactivated; login returns 401 |
| 3.4 | Delete user | Admin deletes a user | 204; user no longer in list; their documents remain (orphaned) |
| 3.5 | Non-admin access | Regular user navigates to `/admin/users` | Redirected to `/login` |
| 3.6 | Admin 404 semantics | Regular user calls `GET /api/admin/users` via curl | 404 (not 403) |
---
## 4. Admin — Groups
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 4.1 | List groups | Admin → `/admin/groups` | All groups shown with member count |
| 4.2 | Create group | Fill name + description → submit | Group appears in list; `{group_name}-admin` bootstrap group also exists (auto-created on service start) |
| 4.3 | Edit group | Click edit on group → change name → save | Name updated |
| 4.4 | Delete group | Delete group | 204; group gone; memberships cascade-deleted |
| 4.5 | Add member | Open group → search user → add | 204; user appears in member list |
| 4.6 | Remove member | Click remove on a member | User removed from group |
| 4.7 | Duplicate group name | Create group with name that already exists | 400 / validation error shown |
| 4.8 | Non-admin access | Regular user calls `GET /api/admin/groups` | 404 |
| 4.9 | Set group admin role | Admin → group detail → tick "Group admin" checkbox on a member → save | `PATCH /api/admin/groups/{id}/members/{user_id}/admin` with `{"is_group_admin": true}` returns 200; badge shown in member list |
| 4.10 | Unset group admin role | Admin unticks "Group admin" on an existing group admin member | Returns 200; badge removed; user loses group-admin privileges |
---
## 5. Admin — Appearance
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 5.1 | List themes | Admin → `/admin/appearance` | Built-in themes + any custom themes shown |
| 5.2 | Switch active theme | Select a different theme → save | All users see the new theme on next load |
| 5.3 | Create custom theme | Admin → create theme. Required fields: `id` (slug), `label`, `light` (CSS vars object), `dark` (CSS vars object) | 201; theme appears in selector; can be activated |
| 5.4 | Edit custom theme | Admin edits colour values on a custom theme | Colours update live after activation |
| 5.5 | Delete custom theme | Admin deletes a custom theme | 204; theme gone from selector; active theme reverts to default |
| 5.6 | Set default mode | `PATCH /api/settings/appearance` with `{"theme": "<id>", "default_mode": "dark"}` (both fields required) | 200; new users without a personal preference see dark mode |
---
## 6. Service Health & Dashboard
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 6.1 | Services endpoint | `GET /api/services` (authenticated) | Returns health status for doc-service and ai-service |
| 6.2 | Healthy service card | Both services running → `/apps` page. API response uses `healthy: true` (boolean), not `status: "healthy"` | Cards show "Available" badge; clicking navigates to the app |
| 6.3 | Unhealthy service card | Stop doc-service container → wait 30s → `/apps` | Doc-service card dimmed, "Unavailable", not clickable |
| 6.4 | Service recovery | Restart stopped container → wait 30s | Card returns to "Available" |
| 6.5 | Dashboard pinned cards | Pin a service → go to `/` | Pinned card appears in home grid |
| 6.6 | Customize mode | Click pencil on dashboard → toggle cards | Pinned list updates after save |
---
## 7. Plugin System
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 7.1 | List plugins | `GET /api/plugins` (authenticated) | Returns accessible plugins for current user |
| 7.2 | Superuser sees all plugins | Log in as admin → `GET /api/plugins` | All registered service plugins returned |
| 7.3 | Group member sees plugin | Add user to `doc-service-admin` group → `GET /api/plugins` | doc-service plugin returned |
| 7.4 | Unpermitted user hidden | Regular user not in any admin group → `GET /api/plugins` | Empty list (plugins hidden, not 403) |
| 7.5 | Manifest fetch | `GET /api/plugins/doc-service/manifest` as permitted user | JSON Schema + access rules returned |
| 7.6 | Settings read | `GET /api/plugins/doc-service/settings` | Current doc-service plugin settings returned |
| 7.7 | Settings write | `PATCH /api/plugins/doc-service/settings` with valid payload | 200; setting persisted to volume |
| 7.8 | Unpermitted settings access | Regular user `GET /api/plugins/doc-service/settings` | 404 |
---
## 8. AI Service Settings
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 8.1 | Read AI config | Admin (or `ai-service-admin` member) → `GET /api/settings/ai` | Config returned; API keys masked |
| 8.2 | Update provider | `PATCH /api/settings/ai` with provider = "anthropic" + valid key | 200; config persisted |
| 8.3 | Test connection | `POST /api/settings/ai/test` with valid config | 200; success response from provider |
| 8.4 | Test connection — bad key | `POST /api/settings/ai/test` with wrong API key | 502 or error detail |
| 8.5 | Read system prompts | `GET /api/settings/system-prompts` | All registered service prompts returned |
| 8.6 | Update system prompt | `PATCH /api/settings/system-prompts/doc-service` with new prompt text | 200; doc-service picks up new prompt within 30s |
| 8.7 | Non-admin access | Regular user calls any `/api/settings/ai` endpoint | 404 |
| 8.8 | `ai-service-admin` delegation | Non-superuser added to `ai-service-admin` group → accesses AI settings page | Page loads; can read and write settings |
---
## 9. Document Service Settings
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 9.1 | Read upload limits | `GET /api/settings/documents/limits` (admin or `doc-service-admin`) | `max_pdf_bytes` returned |
| 9.2 | Update upload limit | `PATCH /api/settings/documents/limits` with new value | 200; upload of oversized PDF now rejected with 413 |
| 9.3 | Non-admin access | Regular user calls `GET /api/settings/documents/limits` | 404 |
| 9.4 | Settings page loads | Admin navigates to `/apps/documents/settings` | Upload limits section + watch directory config visible |
| 9.5 | `doc-service-admin` delegation | Non-superuser added to `doc-service-admin` → navigates to settings page | Page loads; settings editable |
---
## 18. Infrastructure & Security
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 18.1 | Non-root containers | `docker inspect <container> --format '{{.Config.User}}'` for each service | Returns `1001` (or `70` for db) |
| 18.2 | No host ports in prod | `docker compose up --build -d``docker ps` | Only port 80 (frontend) exposed; no 8000/8001/8010/5432 |
| 18.3 | backend-net isolation | `curl http://localhost:8000` from host in prod | Connection refused |
| 18.4 | Pre-commit hook runs | Stage a file with `eval("x")``git commit` | Commit blocked; security_check.py output shown |
| 18.5 | Pre-commit hook — clean code | Normal commit | Hook passes; commit succeeds |
| 18.6 | JWT algorithm none rejected | Craft token with `"alg": "none"` → call protected route | 401 |
| 18.7 | XSS — input sanitation | Enter `<script>alert(1)</script>` in title/name fields | Value stored as plain text; not executed in UI |
| 18.8 | SQL injection attempt | Pass `'; DROP TABLE documents; --` as search param | 200 with empty results; no DB error |
| 18.9 | CORS | `curl -H "Origin: http://evil.com" http://localhost/api/users/me` | Request blocked or `access-control-allow-origin` not set for that origin |
| 18.10 | Config volume persistence | Restart all containers | AI provider config + doc limits survive restart |
| 18.11 | Migration auto-apply | Start fresh stack | Both `alembic upgrade head` chains run without error; all tables created |
+157
View File
@@ -0,0 +1,157 @@
# Doc-Service Tests
Tests covering the PDF extraction microservice (`doc-service:8001`): document upload/processing, list/filtering, slide-over detail view, sharing, categories, bulk actions, and the file watcher.
Full combined suite: `tests/ALL_TESTS.md`
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
**Admin credentials:** any superuser account created during stack setup.
**Regular user credentials:** a second non-admin account for permission boundary tests.
---
## Legend
| Symbol | Meaning |
|--------|---------|
| ✅ | Pass |
| ❌ | Fail |
| — | N/A for this change |
---
## 10. Document Upload & Processing
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 10.1 | Upload valid PDF | Drag-and-drop or file picker → select a PDF under the size limit | 202; document row appears with `status=pending`; transitions to `done` |
| 10.2 | Upload oversized PDF | Upload a PDF exceeding `max_pdf_bytes` | 413; error shown; no row created |
| 10.3 | Upload non-PDF | Upload a `.docx` or `.jpg` | 415; error shown |
| 10.4 | Multi-file upload | Select 3 PDFs at once | All 3 appear in upload queue panel; each processes independently |
| 10.5 | Upload queue panel | During upload → check bottom-right panel | Per-file status indicator; "Review →" link after each completes |
| 10.6 | Drag-and-drop overlay | Drag file over the documents page | Full-page overlay appears; drop uploads file |
| 10.7 | Processing status poll | Upload a large PDF | Table row auto-updates every 3s until status = `done` or `failed` |
| 10.8 | AI extraction result | Open slide-over for a `done` document | title, document_type, tags, extracted_data fields populated |
| 10.9 | Failed extraction | AI service down → upload PDF | Status = `failed`; error_message shown in slide-over |
| 10.10 | Re-analyse | Click "Re-analyse" in slide-over | 202; status resets to `pending`; re-processes through AI |
---
## 11. Document List & Filtering
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 11.1 | Default list | Navigate to `/apps/documents` | Own documents shown, newest first, 20 per page |
| 11.2 | Search | Type in search box (debounced 400ms) | Results filtered by title / filename / tags / type |
| 11.3 | Filter by status | Add filter chip → Status → "done" | Only completed docs shown |
| 11.4 | Filter by type | Add filter chip → Document type → "invoice" | Only invoices shown |
| 11.5 | Filter by category | Add filter chip → Category → pick one | Only docs in that category shown |
| 11.6 | Remove filter chip | Click × on a chip | Filter removed; full list restored |
| 11.7 | Sort by column | Click "Date" column header | List re-ordered; chevron indicates direction; click again reverses |
| 11.8 | Pagination | Upload > 20 docs → scroll to bottom | Page controls appear; page 2 loads next 20 |
| 11.9 | "Mine" view | Click "Mine" in SourcePanel | Only own (uploaded) documents shown |
| 11.10 | "Shared with me" view | Click "Shared with me" | Docs shared by others via groups; own docs excluded |
| 11.11 | Category filter via SourcePanel | Click a category in the left tree | Table filtered to that category's documents |
| 11.12 | URL state preserved | Apply filters → copy URL → open in new tab | Same filters applied |
---
## 12. Document Detail — Slide-over
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 12.1 | Open slide-over | Click any document row | 480px right panel slides in; metadata loaded |
| 12.2 | Inline title edit | Click pencil icon next to title → type new title → confirm | Title saved; updated in table row |
| 12.3 | Change document type | Click a type chip (Invoice, Receipt, etc.) | Type updated immediately |
| 12.4 | Edit tags | Click into tag area → type a tag → press Enter → remove a tag with × | Tags saved correctly |
| 12.5 | Assign category | Categories combobox → search → select | Category badge appears on document; table row updates |
| 12.6 | Remove category | Click × on an assigned category badge | Category removed from document |
| 12.7 | AI category suggestions | Slide-over shows "Suggested categories" | "Assign" and "Create & Assign" buttons present; clicking assigns |
| 12.8 | Confirm folder suggestion | "Confirm" button next to suggested_folder | Category created (if needed) and assigned; `suggested_folder` cleared |
| 12.9 | Reject folder suggestion | "Reject" button next to suggested_folder | `suggested_folder` cleared; no category created |
| 12.10 | Confirm filename suggestion | "Confirm" button next to suggested_filename | `title` updated to suggested value; `suggested_filename` cleared |
| 12.11 | Reject filename suggestion | "Reject" button next to suggested_filename | `suggested_filename` cleared; title unchanged |
| 12.12 | Extracted data section | Open slide-over on `done` doc | Key-value table of AI-extracted fields (vendor, amounts, dates, etc.) |
| 12.13 | Raw text section | Expand raw text collapse | First ~500k chars of extracted PDF text shown |
| 12.14 | Download | Click "Download" | Browser downloads the original PDF file |
| 12.15 | View in new tab | Click "View" | PDF opens in new browser tab; URL auto-revokes after 60s |
| 12.16 | Delete — owner | Click "Delete" → confirm dialog | Document and file removed; table row gone |
| 12.16b | Delete — admin | Admin opens any doc (not their own) → Delete → confirm | Document deleted; 204 returned |
| 12.16c | Delete — can_delete share | Group member whose share has `can_delete=true` → Delete | 204; document removed; `viewer_can_delete` was `true` in `DocumentOut` |
| 12.16d | Delete — group admin | User is group admin for a group the doc is shared with; no explicit `can_delete` flag → Delete | 204; group admin always has delete rights for docs shared with their group |
| 12.16e | Delete — watch document, admin only | Watch-ingested doc (`source=watch`); regular user → Delete | 403 (not owner); admin can delete it |
| 12.17 | Non-owner cannot edit | Recipient of shared doc opens slide-over (no can_delete, not group admin) | Edit controls (type, tags, title, delete) absent; download available |
---
## 13. Document Sharing
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 13.1 | Share from slide-over | Owner opens sharing section → selects a group from combobox → shares | Group appears in shares list; `share_count` in table row increments |
| 13.2 | Only user's own groups shown | Open group picker in share section | Only groups the current user belongs to are listed |
| 13.3 | Recipient sees shared doc | Log in as group member → "Shared with me" view | Shared document appears with primary accent border |
| 13.4 | Recipient download | Recipient clicks Download on shared doc | PDF downloaded successfully |
| 13.4b | Non-owner calls `GET /documents/{id}/shares` | Regular user on a doc they don't own | 404 (doc-service hides existence, consistent with admin 404 semantics — not 403) |
| 13.5 | Recipient cannot delete | Recipient opens slide-over | Delete button absent |
| 13.6 | Recipient cannot re-share | Recipient opens sharing section | Share controls absent |
| 13.7 | Remove share | Owner clicks remove on a group share | Group removed; `share_count` decrements; recipient no longer sees doc |
| 13.8 | Bulk share | Select multiple rows → bulk share → pick group | All selected docs shared with that group |
| 13.9 | Share count indicator | Document shared with 2 groups | Users icon in table row shows "2" |
| 13.10 | Share with non-member group | `POST /api/documents/{id}/shares` with group not in X-User-Groups | 403 / validation error |
| 13.11 | Share with can_delete enabled | Owner opens sharing section → tick "Allow group members to delete" → share | `can_delete=true` stored; trash icon badge appears next to group name in shares list |
| 13.12 | Share without can_delete (default) | Owner shares without ticking delete checkbox | `can_delete=false`; recipient sees the doc but Delete button absent in slide-over |
| 13.13 | can_delete shown in shares list | Share with can_delete=true → inspect shares list in slide-over | Trash2 icon rendered beside the group name; tooltip "Group members can delete this document" |
| 13.14 | viewer_can_delete in document list | Share with can_delete=true; log in as group member → `GET /api/documents` | `viewer_can_delete=true` in the recipient's list response for that doc |
---
## 14. Categories
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 14.1 | Create category | SourcePanel → "New category" form → submit | Category appears in tree |
| 14.2 | Rename category | Manage categories dialog → edit → save | New name reflected everywhere |
| 14.3 | Delete category | Delete category with documents assigned | 204; documents remain; category assignment removed |
| 14.4 | Category search | More than 4 categories → type in search field | Tree filtered in real time |
| 14.5 | Manage categories dialog | Click "Manage categories" | Modal shows categories grouped by scope (Personal / Group / System) with lock icons on group and system categories |
| 14.6 | New category triggers re-analysis | Create category with name similar to AI suggestion | Background re-analysis triggered (check backend logs) |
| 14.7 | Create personal category | SourcePanel → "New category" → no group selected → submit valid PascalCase name | Created with `scope=personal`; visible only to owner |
| 14.8 | Create group-scoped category | SourcePanel → "New category" → select a group → submit | Created with `scope=group`; visible to all members of that group |
| 14.9 | Group category visible to group members | Log in as another group member | Group category appears in their category list and SourcePanel |
| 14.10 | Non-member cannot see group category | Log in as user not in the group | Group category absent from list |
| 14.11 | Only group admin can rename group category | Regular group member → rename group category | 403; group admin can rename it successfully |
| 14.12 | Only group admin can delete group category | Regular group member → delete group category | 403; group admin can delete it |
| 14.13 | System categories read-only for non-admin | Regular user → Manage categories → rename/delete a system category | 403; lock icon shown; action blocked in UI |
| 14.14 | Admin can manage system categories | Superuser → rename or delete a system category | Succeeds; ManageCategoriesDialog shows edit/delete controls for system rows |
| 14.15 | PascalCase naming enforced — invalid | Create category named `my-invoices` or `Invoice Reports` | 422 with message explaining PascalCase-with-dashes format |
| 14.16 | PascalCase naming enforced — valid | Create category named `Vendor-Invoices` | 201; category created successfully |
| 14.17 | SourcePanel scope sections | Categories exist for all three scopes | SourcePanel tree shows "Mine", per-group, and "System" sections separately |
---
## 15. Bulk Actions
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 15.1 | Select rows | Tick checkboxes on multiple rows | Floating bulk actions bar appears at bottom |
| 15.2 | Bulk share | Select docs → Share with group → confirm | All selected docs shared; confirmation |
| 15.3 | Bulk delete | Select docs → Delete → confirm dialog | All selected docs deleted; bar disappears |
| 15.4 | Clear selection | Click "Clear" in bulk bar | All checkboxes deselected; bar hides |
| 15.5 | Bulk bar — "Mine" view only | Switch to "Shared with me" view | Bulk actions bar not shown (no edit rights for shared docs) |
---
## 16. Watch Directory
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 16.1 | Enable watch | Doc settings page → toggle `watch_enabled` on → save | File watcher starts; backend logs confirm |
| 16.2 | Ingest new file | Drop a PDF into the bind-mounted watch directory | Document appears in "All Documents" view with `source=watch` |
| 16.3 | Sub-folder to category | Place PDF in `watch/invoices/` | Document auto-assigned to "invoices" category |
| 16.4 | Startup scan | Restart doc-service with PDFs already in watch dir | Pre-existing PDFs ingested (idempotent — no duplicates) |
| 16.5 | AI folder suggestion | `ai_folder_suggestion` enabled → ingest file | `suggested_folder` populated; confirm/reject buttons visible in slide-over |
| 16.6 | AI rename suggestion | `ai_rename_suggestion` enabled → ingest file | `suggested_filename` populated; confirm/reject buttons visible |
| 16.7 | No-remove policy | Delete PDF from watch dir | Document record remains in DB |
| 16.8 | Disable watch | Toggle `watch_enabled` off → save | Watcher stops; new files dropped are not ingested |
| 16.9 | Watch docs visible to all users | Log in as any authenticated user | Watch-ingested docs (`user_id = "watch"`) appear in "All Documents" |
+37
View File
@@ -0,0 +1,37 @@
# Frontend Tests
Tests covering the React SPA (`frontend`, port 5173 dev / 80 prod): route guards, navigation, theme, component behaviour, and TanStack Query patterns.
Full combined suite: `tests/ALL_TESTS.md`
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
**Admin credentials:** any superuser account created during stack setup.
**Regular user credentials:** a second non-admin account for permission boundary tests.
---
## Legend
| Symbol | Meaning |
|--------|---------|
| ✅ | Pass |
| ❌ | Fail |
| — | N/A for this change |
---
## 19. Frontend — UI & Routing
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 19.1 | PrivateRoute redirect | Open any protected route without token | Redirected to `/login` |
| 19.2 | AdminRoute redirect | Log in as non-admin → navigate to `/admin` | Redirected to `/login` |
| 19.3 | ServiceAdminRoute | Non-admin, non-group-member → navigate to `/apps/documents/settings` | Redirected (access denied) |
| 19.4 | Sidebar collapse | Click collapse button | Sidebar shrinks to icon-only; expand restores labels |
| 19.5 | Apps accordion | Click "Apps" in sidebar | Expands to show "Documents" NavLink |
| 19.6 | SourcePanel visibility | Navigate to `/apps` then `/apps/documents` | SourcePanel only visible on `/apps/documents` route |
| 19.7 | Theme toggle | Click sun/moon button | Mode switches; persists on reload |
| 19.8 | Unknown route | Navigate to `/does-not-exist` | Redirected to `/` |
| 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs |
| 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab |
| 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off |
+32
View File
@@ -0,0 +1,32 @@
# Storage Service Tests — §20
Storage-service tests. Run these before merging any change that touches `features/storage-service/`, `docker-compose.yml` storage volumes, or storage-related backend/doc-service code.
See `tests/ALL_TESTS.md` for the full suite and legend.
---
## 20. Storage Service
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored |
| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload |
| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 |
| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` |
| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` |
| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 |
| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent |
| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service |
| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone |
| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket |
| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" |
| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` |
| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss |
| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done |
| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active |
| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" |
| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone |
| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent |
| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent |
| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |