Compare commits
20 Commits
75b7ae6062
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 curo1305
|
0f760c379d | ||
|
curo1305
|
f13ef88711 | ||
|
curo1305
|
0d8e0366c6 | ||
|
curo1305
|
3a66aeeec5 | ||
|
curo1305
|
248b2bb9d7 | ||
|
curo1305
|
cfec3bb906 | ||
|
curo1305
|
4c35d7a2a4 | ||
|
curo1305
|
2f3efb9bf9 | ||
|
curo1305
|
5349f21752 | ||
|
curo1305
|
50d2348b36 | ||
|
curo1305
|
d345ace86d | ||
|
curo1305
|
c59718171c | ||
|
curo1305
|
99d22660f9 | ||
|
curo1305
|
fcfc06cda9 | ||
 curo
|
f5bc28cda2 | ||
|
curo1305
|
1c8b35399c | ||
|
curo1305
|
ebf97b6f4a | ||
|
curo1305
|
fec3953009 | ||
|
curo1305
|
6e5e5c08bf | ||
|
curo1305
|
05d79d3d21 |
@@ -5,8 +5,21 @@
|
||||
"Bash(git add:*)",
|
||||
"Bash(git commit -m ':*)",
|
||||
"Bash(git push:*)",
|
||||
"Bash(git pull:*)",
|
||||
"Bash(git checkout:*)",
|
||||
"Bash(git merge:*)",
|
||||
"Bash(git branch:*)",
|
||||
"Bash(git log:*)",
|
||||
"Bash(git diff:*)",
|
||||
"Bash(git status:*)",
|
||||
"Bash(git config:*)",
|
||||
"Bash(git mv:*)",
|
||||
"Bash(docker compose:*)",
|
||||
"Bash(docker run:*)"
|
||||
"Bash(docker run:*)",
|
||||
"Bash(docker inspect:*)",
|
||||
"Bash(docker ps:*)",
|
||||
"Bash(curl:*)",
|
||||
"Bash(lsof:*)"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,12 +6,21 @@ This file provides permanent, authoritative guidance to Claude Code for every se
|
||||
- `frontend/CLAUDE.md` — routes, components, API client patterns, XSS prevention
|
||||
- `features/ai-service/CLAUDE.md` — /chat, /health, /queue endpoints; queue service
|
||||
- `features/doc-service/CLAUDE.md` — document/category/share endpoints; DB models; PDF limits; file watcher
|
||||
- `features/storage-service/CLAUDE.md` — storage API, pluggable backend drivers (local/S3/WebDAV), migration
|
||||
|
||||
---
|
||||
|
||||
## Merge checklist
|
||||
|
||||
Before merging any feature branch into `main`, every test relevant to the changed area in `tests/MERGE_CHECKLIST.md` must be marked passing. The checklist covers all 19 feature areas (auth, users, admin, groups, appearance, service health, plugins, AI settings, doc settings, upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory, AI queue, infrastructure/security, and frontend routing). Do not merge without it.
|
||||
Before merging any feature branch into `main`, every test relevant to the changed area in `tests/ALL_TESTS.md` (and the relevant service-specific file) must be marked passing. The test suite covers all 20 feature areas across five service files:
|
||||
|
||||
- `tests/backend_tests.md` — §1–9, §18
|
||||
- `tests/frontend_tests.md` — §19
|
||||
- `tests/doc-service_tests.md` — §10–16
|
||||
- `tests/ai-service_tests.md` — §17
|
||||
- `tests/storage-service_tests.md` — §20
|
||||
|
||||
Do not merge without it.
|
||||
|
||||
---
|
||||
|
||||
@@ -28,6 +37,8 @@ Before merging any feature branch into `main`, every test relevant to the change
|
||||
- New Docker service, volume, network, or env var → update **Docker Infrastructure** in this file
|
||||
- Stack version changed → update **Stack** in this file
|
||||
|
||||
- New feature or endpoint added → add test rows to **both** `tests/ALL_TESTS.md` (in the relevant section) **and** the matching service-specific file (`tests/backend_tests.md`, `tests/frontend_tests.md`, `tests/doc-service_tests.md`, `tests/ai-service_tests.md`, or `tests/storage-service_tests.md`). Use the same test number and format as existing rows.
|
||||
|
||||
This check is mandatory — treat it the same as updating STATUS.md.
|
||||
|
||||
---
|
||||
@@ -77,7 +88,11 @@ For service-specific commands (migrations, lint), see `backend/CLAUDE.md` and `f
|
||||
├── .githooks/pre-commit ← Runs scripts/security_check.py before every commit
|
||||
├── scripts/security_check.py ← Static analysis: secrets, weak crypto, SQLi, JWT
|
||||
├── changelog/YYYY-MM-DD_<slug>.md ← Per-date change logs
|
||||
├── tests/MERGE_CHECKLIST.md ← 148-test pre-merge checklist (all features); must pass before merging to main
|
||||
├── tests/ALL_TESTS.md ← Full test suite (all 19 areas); must pass before merging to main
|
||||
├── tests/backend_tests.md ← Backend-only tests (§1–9, §18)
|
||||
├── tests/frontend_tests.md ← Frontend-only tests (§19)
|
||||
├── tests/doc-service_tests.md ← Doc-service tests (§10–16)
|
||||
├── tests/ai-service_tests.md ← AI-service tests (§17)
|
||||
├── dev-watch/ ← Dev bind-mount for file watcher testing (.gitkeep only)
|
||||
│
|
||||
├── backend/ ← FastAPI gateway (port 8000, internal); see backend/CLAUDE.md
|
||||
@@ -130,7 +145,17 @@ These standards are **non-negotiable**. Every change must comply. Implementation
|
||||
|
||||
- `backend-net`: all containers except frontend; not reachable from host in prod.
|
||||
- `frontend-net`: only frontend; single host port (80 prod / 5173 dev).
|
||||
- DB, backend, doc-service, ai-service have **no** host port bindings in prod.
|
||||
- DB, backend, doc-service, ai-service, storage-service have **no** host port bindings in prod.
|
||||
|
||||
### Storage rule (non-negotiable)
|
||||
|
||||
**No service may write to a filesystem path for persistent data.** All file/blob storage must go through the storage-service HTTP API (`PUT/GET/DELETE /objects/{bucket}/{key}`). Config JSON files must be stored in the `config` bucket. Uploaded files must be stored in the `documents` bucket. Violation is a security and architecture defect.
|
||||
|
||||
The only two persistent storage mechanisms in the project are:
|
||||
1. **PostgreSQL** — structured/relational data
|
||||
2. **storage-service** — all file/blob/config data (local filesystem by default; switchable to S3-compatible or WebDAV)
|
||||
|
||||
New services and features must follow this pattern. See `features/storage-service/CLAUDE.md` for the API reference.
|
||||
|
||||
### Pre-commit security hook
|
||||
|
||||
@@ -166,9 +191,10 @@ All other per-service defaults are in the relevant sub-CLAUDE.md file.
|
||||
| Service | Image base | Internal port | User | Volumes | Network |
|
||||
|---------|-----------|---------------|------|---------|---------|
|
||||
| `db` | postgres:16-alpine | 5432 | 70:70 | `postgres_data` | backend-net |
|
||||
| `backend` | python:3.12-slim | 8000 | 1001:1001 | `app_config` | backend-net |
|
||||
| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | `app_config` | backend-net |
|
||||
| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `doc_data`, `watch_data`, `app_config` | backend-net |
|
||||
| `backend` | python:3.12-slim | 8000 | 1001:1001 | — | backend-net |
|
||||
| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | — | backend-net |
|
||||
| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `watch_data` | backend-net |
|
||||
| `storage-service` | python:3.12-slim | 8020 | 1001:1001 | `storage_data` | backend-net |
|
||||
| `frontend` | nginx-unprivileged:alpine | 8080 | 1001:1001 | — | backend-net, frontend-net |
|
||||
|
||||
### Volumes
|
||||
@@ -176,15 +202,14 @@ All other per-service defaults are in the relevant sub-CLAUDE.md file.
|
||||
| Volume | Mount path | Contains |
|
||||
|--------|-----------|---------|
|
||||
| `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL data |
|
||||
| `doc_data` | `/data/documents` | Uploaded PDF files |
|
||||
| `storage_data` | `/data/storage` | All file/blob storage: PDFs (`documents/`) and config JSONs (`config/`) |
|
||||
| `watch_data` | `/data/watch` | Watch directory (bind-mount NAS/Nextcloud via docker-compose.override.yml) |
|
||||
| `app_config` | `/config` | Per-service runtime config JSON files |
|
||||
|
||||
### Networks
|
||||
|
||||
| Network | Host-accessible | Members |
|
||||
|---------|----------------|---------|
|
||||
| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, frontend |
|
||||
| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, storage-service, frontend |
|
||||
| `frontend-net` | Yes (port 80 → frontend:8080) | frontend |
|
||||
|
||||
### Environment variables (required in `backend/.env`)
|
||||
@@ -200,6 +225,7 @@ Injected by docker-compose (not in `.env`):
|
||||
```
|
||||
DOC_SERVICE_URL=http://doc-service:8001
|
||||
AI_SERVICE_URL=http://ai-service:8010
|
||||
STORAGE_SERVICE_URL=http://storage-service:8020
|
||||
```
|
||||
|
||||
---
|
||||
@@ -266,8 +292,9 @@ Each entry must include:
|
||||
4. Add router in `backend/app/routers/`, mount it in `main.py`
|
||||
5. Add API function(s) to `frontend/src/api/client.ts`
|
||||
6. Add page component in `frontend/src/pages/`, register route in `App.tsx`
|
||||
7. Update `STATUS.md` for affected services
|
||||
8. Add changelog entry
|
||||
7. If the resource involves file or blob data: store it via `PUT /objects/{bucket}/{key}` on `storage-service:8020`. Never write to the local filesystem. See `features/storage-service/CLAUDE.md` for the API.
|
||||
8. Update `STATUS.md` for affected services
|
||||
9. Add changelog entry
|
||||
|
||||
---
|
||||
|
||||
@@ -303,23 +330,18 @@ git checkout -b feat/<descriptive-title> # e.g. feat/user-profile-avatar-uploa
|
||||
```
|
||||
|
||||
#### 2 — Spin up an isolated Docker stack for the feature
|
||||
A dedicated compose stack runs alongside the main dev stack so both can be tested independently.
|
||||
The feature stack always uses port `5173` (same as the main dev stack). Stop the main stack before starting a feature stack, and restart it when done.
|
||||
|
||||
**Find the next free port** (main dev stack owns 5173):
|
||||
**Stop the main dev stack first:**
|
||||
```bash
|
||||
for port in $(seq 5174 5200); do
|
||||
lsof -iTCP:$port -sTCP:LISTEN -t &>/dev/null || { echo "$port"; break; }
|
||||
done
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
|
||||
```
|
||||
Use the first free port returned (call it `$PORT`).
|
||||
|
||||
**Create a per-feature override file** at `docker-compose.feat-<slug>.yml` (gitignored):
|
||||
```yaml
|
||||
# docker-compose.feat-<slug>.yml — feature test stack, never committed to main
|
||||
services:
|
||||
frontend:
|
||||
ports:
|
||||
- "$PORT:8080" # e.g. 5174:8080
|
||||
container_name: frontend-<slug>
|
||||
backend:
|
||||
container_name: backend-<slug>
|
||||
@@ -345,8 +367,7 @@ docker compose -f docker-compose.yml \
|
||||
--project-name <slug> up --build
|
||||
```
|
||||
|
||||
The feature frontend is now reachable at `http://localhost:$PORT`.
|
||||
The main dev stack continues running unaffected on `:5173`.
|
||||
The feature frontend is now reachable at `http://localhost:5173`.
|
||||
|
||||
#### 3 — Develop on the feature branch
|
||||
All code changes happen on `feat/<slug>`. Commit and push normally:
|
||||
@@ -357,7 +378,7 @@ git push -u origin feat/<slug>
|
||||
```
|
||||
|
||||
#### 4 — Confirm functionality
|
||||
Before merging, verify all of the following on `http://localhost:$PORT`:
|
||||
Before merging, verify all of the following on `http://localhost:5173`:
|
||||
- [ ] Login and registration work end-to-end
|
||||
- [ ] The specific feature works as intended
|
||||
- [ ] No regressions visible in the UI
|
||||
@@ -374,13 +395,16 @@ git branch -d feat/<slug>
|
||||
git push origin --delete feat/<slug>
|
||||
```
|
||||
|
||||
#### 6 — Tear down the feature stack
|
||||
#### 6 — Tear down the feature stack and restart main dev stack
|
||||
```bash
|
||||
docker compose -f docker-compose.yml \
|
||||
-f docker-compose.dev.yml \
|
||||
-f docker-compose.feat-<slug>.yml \
|
||||
--project-name <slug> down --volumes --remove-orphans
|
||||
rm docker-compose.feat-<slug>.yml
|
||||
|
||||
# Restart the main dev stack on :5173
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -20,10 +20,11 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
|
||||
- All input sanitized before reaching the DB (null-byte rejection, length caps, format validation)
|
||||
- **PDF Documents app** (`/apps/documents`): upload PDFs, async text extraction (pdfplumber), AI classification via ai-service, per-user categories, file download
|
||||
- **AI Service** (`ai-service:8010`): shared AI intermediary container; routes prompts to Anthropic / Ollama / LM Studio; stateless; all feature containers talk to it via `POST /chat`
|
||||
- Admin settings: `/apps/ai/settings/admin` (provider, credentials, test connection); `/apps/documents/settings/admin` (upload limits only)
|
||||
- Config stored in shared Docker volume: `/config/ai_service_config.json` and `/config/doc_service_config.json`
|
||||
- **Storage Service** (`storage-service:8020`): unified file/blob storage with pluggable backends (local filesystem default; S3-compatible and WebDAV built in); backend switchable via admin UI with zero-data-loss migration
|
||||
- Admin settings: AI provider, doc upload limits, storage backend switching with live migration progress
|
||||
- Config stored in storage-service (`config` bucket); PDFs stored in storage-service (`documents` bucket) — no shared filesystem volumes
|
||||
- `/apps` launcher hub — one card per installed app with Open + Settings links
|
||||
- 5 separate Docker containers: `db`, `backend`, `ai-service`, `doc-service`, `frontend`
|
||||
- 6 separate Docker containers: `db`, `backend`, `ai-service`, `doc-service`, `storage-service`, `frontend`
|
||||
- All containers run as non-root users (UID 1001 for app containers, UID 70 for db)
|
||||
- Network-isolated: only the frontend exposes a host port (80/5173); all backend services are unreachable from outside Docker
|
||||
- Dev environment seeds a test user automatically on startup (`test@example.com` / `Test123!`)
|
||||
@@ -38,6 +39,7 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
|
||||
| `backend` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | FastAPI management API + proxy to doc-service |
|
||||
| `ai-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | Shared AI intermediary (routes to LM Studio / Ollama / Anthropic) |
|
||||
| `doc-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | PDF extraction microservice (calls ai-service) |
|
||||
| `storage-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | Unified file/blob storage (local / S3-compatible / WebDAV) |
|
||||
| `frontend` | custom (nginxinc/nginx-unprivileged:alpine) | 80 (prod) / 5173 (dev) | backend-net + frontend-net | 1001:1001 | React UI + nginx reverse proxy |
|
||||
|
||||
**Networks:**
|
||||
@@ -46,8 +48,8 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
|
||||
|
||||
**Volumes:**
|
||||
- `postgres_data` — PostgreSQL data files
|
||||
- `doc_data` — uploaded PDF files (mounted into doc-service at `/data/documents`)
|
||||
- `app_config` — per-service runtime config JSON files (mounted into backend, ai-service, and doc-service at `/config`)
|
||||
- `storage_data` — all file/blob storage: uploaded PDFs (`documents/` bucket) and service config JSON files (`config/` bucket); mounted into storage-service at `/data/storage`
|
||||
- `watch_data` — file watcher input directory; mounted into doc-service at `/data/watch`
|
||||
|
||||
The frontend nginx proxies `/api/*` to `backend:8000` via `backend-net`. The backend proxies `/api/documents/*` and `/api/documents/categories/*` to `doc-service:8001`. The backend test-connection endpoint proxies to `ai-service:8010`. No backend service or database port is ever exposed to the host.
|
||||
|
||||
@@ -126,10 +128,8 @@ Copy `.env.example` to `backend/.env` and adjust:
|
||||
| `JWT_PRIVATE_KEY` | — | RS256 private key PEM (generate with `scripts/generate_jwt_keys.py`) |
|
||||
| `JWT_PUBLIC_KEY` | — | RS256 public key PEM (generate with `scripts/generate_jwt_keys.py`) |
|
||||
| `CORS_ORIGINS` | `["http://localhost:5173"]` | Allowed frontend origins |
|
||||
| `APP_CONFIG_DIR` | `/config` | Directory for per-service runtime config JSON files |
|
||||
| `DOC_SERVICE_URL` | `http://doc-service:8001` | Internal URL of the doc-service (set by docker-compose) |
|
||||
|
||||
`doc-service` reads `DATABASE_URL`, `DATA_DIR`, and `CONFIG_PATH` from its own environment (set in `docker-compose.yml`).
|
||||
| `DOC_SERVICE_URL` | `http://doc-service:8001` | Internal URL of doc-service (set by docker-compose) |
|
||||
| `STORAGE_SERVICE_URL` | `http://storage-service:8020` | Internal URL of storage-service (set by docker-compose) |
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
+18
-3
@@ -36,7 +36,8 @@ backend/
|
||||
│ │ ├── config.py ← All settings via pydantic-settings (reads .env)
|
||||
│ │ ├── security.py ← JWT sign/verify (RS256), bcrypt hash/verify
|
||||
│ │ ├── sanitize.py ← Input sanitization helpers (see Security Standards)
|
||||
│ │ └── app_config.py ← Per-service config load/save to /config volume; theme files in /config/themes/
|
||||
│ │ ├── app_config.py ← Per-service config load/save via storage-service; theme files in config/themes/{id}.json
|
||||
│ │ └── config_storage.py ← Thin async HTTP helpers: read_json/write_json/delete_key/list_keys → storage-service config bucket
|
||||
│ ├── models/
|
||||
│ │ ├── __init__.py ← Imports all models (required for Alembic autogenerate)
|
||||
│ │ ├── user.py ← User model
|
||||
@@ -56,7 +57,8 @@ backend/
|
||||
│ │ ├── services.py ← GET /services (health status)
|
||||
│ │ ├── plugins.py ← Generic plugin proxy (GET/PATCH /api/plugins/*)
|
||||
│ │ ├── categories_proxy.py ← Transparent proxy → doc-service /categories/*
|
||||
│ │ └── documents_proxy.py ← Transparent proxy → doc-service /documents/*
|
||||
│ │ ├── documents_proxy.py ← Transparent proxy → doc-service /documents/*
|
||||
│ │ └── storage_config.py ← Admin proxy → storage-service config + migration endpoints
|
||||
│ └── services/
|
||||
│ ├── service_health.py ← Background 30s health-check loop; caches /plugin/manifest per service
|
||||
│ └── group_bootstrap.py ← Ensures {service-id}-admin group exists for every registered service at startup
|
||||
@@ -115,6 +117,7 @@ Relationship: `profile` (one-to-one, cascade all+delete-orphan)
|
||||
| `id` | String | PK, UUID |
|
||||
| `group_id` | String | FK→groups.id, indexed, CASCADE |
|
||||
| `user_id` | String | FK→users.id, indexed, CASCADE |
|
||||
| `is_group_admin` | Boolean | NOT NULL, default=false | grants group-admin rights (manage group categories, delete shared docs) |
|
||||
| `joined_at` | DateTime(tz) | server_default=now() |
|
||||
|
||||
Unique constraint: `(group_id, user_id)`
|
||||
@@ -128,6 +131,7 @@ Unique constraint: `(group_id, user_id)`
|
||||
| `a3f9c2d14e87` | `add_groups_and_group_memberships` |
|
||||
| `c7e8f9a0b1d2` | `add_dashboard_app_ids_to_users` |
|
||||
| `dd6ad2f2c211` | `add_color_mode_to_users` |
|
||||
| `e1f2a3b4c5d6` | `add_group_member_is_admin` |
|
||||
|
||||
---
|
||||
|
||||
@@ -177,6 +181,7 @@ Unique constraint: `(group_id, user_id)`
|
||||
| DELETE | `/api/admin/groups/{id}` | Delete (cascades memberships) |
|
||||
| POST | `/api/admin/groups/{id}/members/{user_id}` | Add member |
|
||||
| DELETE | `/api/admin/groups/{id}/members/{user_id}` | Remove member |
|
||||
| PATCH | `/api/admin/groups/{id}/members/{user_id}/admin` | Set/unset group admin role (body: `{ is_group_admin: bool }`) |
|
||||
|
||||
### Settings (`/api/settings`) — admin-only
|
||||
|
||||
@@ -213,9 +218,19 @@ Unique constraint: `(group_id, user_id)`
|
||||
|
||||
Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence.
|
||||
|
||||
### Admin — Storage (`/api/admin`) — admin-only
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/admin/storage-config` | Current backend driver + health → proxied from storage-service `/health` |
|
||||
| PATCH | `/api/admin/storage-config` | Reconfigure backend without data migration (same-backend credential update) |
|
||||
| POST | `/api/admin/storage-config/migrate` | Start async migration to a new backend (copy → verify → switch → cleanup) |
|
||||
| GET | `/api/admin/storage-config/migrate/status` | Poll migration progress: `{state, total, done, failed, errors[]}` |
|
||||
| DELETE | `/api/admin/storage-config/migrate` | Cancel a running migration; old backend remains active |
|
||||
|
||||
### Documents and Categories — proxied
|
||||
|
||||
`/api/documents/*` and `/api/documents/categories/*` are transparently proxied to `doc-service:8001`. The backend injects `x-user-id` and `x-user-groups` headers. See `features/doc-service/CLAUDE.md` for the internal endpoint list.
|
||||
`/api/documents/*` and `/api/documents/categories/*` are transparently proxied to `doc-service:8001`. The backend injects `x-user-id`, `x-user-groups`, and `x-user-is-admin` headers. See `features/doc-service/CLAUDE.md` for the internal endpoint list.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+16
-3
@@ -75,10 +75,20 @@ A background task (`service_health.py`) polls each service's `/health` endpoint
|
||||
| `GET` | `/api/settings/system-prompts` | All editable system prompts — superuser OR `ai-service-admin` member |
|
||||
| `PATCH` | `/api/settings/system-prompts/{id}` | Update system prompt — same access |
|
||||
|
||||
Settings are persisted to JSON files on the `app_config` Docker named volume and read by the respective feature services.
|
||||
Settings are persisted to the `config` bucket of `storage-service:8020` via `core/config_storage.py`. All config I/O is async HTTP; no filesystem volumes are used.
|
||||
|
||||
Access to service-specific settings endpoints is enforced by `get_service_admin(service_id)` in `deps.py` — grants access to superusers OR members of the `{service_id}-admin` group.
|
||||
|
||||
### Storage config (`/api/admin`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/admin/storage-config` | Current backend driver + health (proxied from storage-service) |
|
||||
| `PATCH` | `/api/admin/storage-config` | Reconfigure backend without migration |
|
||||
| `POST` | `/api/admin/storage-config/migrate` | Start async migration to a new backend |
|
||||
| `GET` | `/api/admin/storage-config/migrate/status` | Poll migration progress |
|
||||
| `DELETE` | `/api/admin/storage-config/migrate` | Cancel running migration |
|
||||
|
||||
### Feature proxies
|
||||
|
||||
All `/api/documents/*` and `/api/documents/categories/*` requests are transparently proxied to `doc-service:8001` via `httpx.AsyncClient`. The proxy:
|
||||
@@ -129,8 +139,11 @@ Browser (port 5173 dev / 80 prod)
|
||||
┌───────────┼────────────┬──────────────┐
|
||||
/auth /settings /documents/* /services
|
||||
/users (JSON │ │
|
||||
/admin volume) └── proxy → health-check loop
|
||||
/profile doc-service:8001 (30s poll)
|
||||
/admin /storage- └── proxy → health-check loop
|
||||
/profile config doc-service:8001 (30s poll)
|
||||
(proxy)
|
||||
│
|
||||
storage-service:8020
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""add is_group_admin to group_memberships
|
||||
|
||||
Revision ID: e1f2a3b4c5d6
|
||||
Revises: dd6ad2f2c211
|
||||
Create Date: 2026-04-18
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "e1f2a3b4c5d6"
|
||||
down_revision: Union[str, None] = "dd6ad2f2c211"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"group_memberships",
|
||||
sa.Column(
|
||||
"is_group_admin",
|
||||
sa.Boolean(),
|
||||
nullable=False,
|
||||
server_default=sa.text("false"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("group_memberships", "is_group_admin")
|
||||
@@ -1,21 +1,25 @@
|
||||
"""
|
||||
Per-service runtime config helpers.
|
||||
|
||||
Config files live on the shared `app_config` Docker volume at /config/.
|
||||
Each service has its own JSON file.
|
||||
All config files are stored in the 'config' bucket of the storage-service.
|
||||
Every function is async — callers must await them.
|
||||
|
||||
Atomic write pattern: write to .tmp in same dir, then os.replace() so
|
||||
services never read a partial file.
|
||||
Key layout in the config bucket:
|
||||
ai_service_config.json
|
||||
doc_service_config.json
|
||||
appearance_config.json
|
||||
themes/{id}.json
|
||||
"""
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
from copy import deepcopy
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
_CONFIG_DIR = Path(os.environ.get("APP_CONFIG_DIR", "/config"))
|
||||
from app.core import config_storage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── AI service config schemas ──────────────────────────────────────────────────
|
||||
|
||||
@@ -108,59 +112,50 @@ def _mask_ai_config(data: dict) -> dict:
|
||||
|
||||
# ── Load / Save ────────────────────────────────────────────────────────────────
|
||||
|
||||
def _config_path(service: str) -> Path:
|
||||
return _CONFIG_DIR / f"{service}_config.json"
|
||||
|
||||
|
||||
def load_service_config(service: str) -> dict:
|
||||
path = _config_path(service)
|
||||
if not path.exists():
|
||||
async def load_service_config(service: str) -> dict:
|
||||
data = await config_storage.read_json(f"{service}_config.json")
|
||||
if data is None:
|
||||
if service == "ai_service":
|
||||
return AIServiceConfig().model_dump()
|
||||
if service == "doc_service":
|
||||
return DocServiceConfig().model_dump()
|
||||
return {}
|
||||
with path.open() as f:
|
||||
return json.load(f)
|
||||
return data
|
||||
|
||||
|
||||
def save_service_config(service: str, data: dict) -> None:
|
||||
path = _config_path(service)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(data, indent=2))
|
||||
os.replace(tmp, path)
|
||||
async def save_service_config(service: str, data: dict) -> None:
|
||||
await config_storage.write_json(f"{service}_config.json", data)
|
||||
|
||||
|
||||
# AI service helpers
|
||||
|
||||
def load_ai_service_config() -> AIServiceConfig:
|
||||
raw = load_service_config("ai_service")
|
||||
async def load_ai_service_config() -> AIServiceConfig:
|
||||
raw = await load_service_config("ai_service")
|
||||
return AIServiceConfig.model_validate(raw)
|
||||
|
||||
|
||||
def save_ai_service_config(config: AIServiceConfig) -> None:
|
||||
save_service_config("ai_service", config.model_dump())
|
||||
async def save_ai_service_config(config: AIServiceConfig) -> None:
|
||||
await save_service_config("ai_service", config.model_dump())
|
||||
|
||||
|
||||
def load_ai_service_config_masked() -> dict:
|
||||
raw = load_service_config("ai_service")
|
||||
async def load_ai_service_config_masked() -> dict:
|
||||
raw = await load_service_config("ai_service")
|
||||
return _mask_ai_config(raw)
|
||||
|
||||
|
||||
# Doc service helpers
|
||||
|
||||
def load_doc_service_config() -> DocServiceConfig:
|
||||
raw = load_service_config("doc_service")
|
||||
async def load_doc_service_config() -> DocServiceConfig:
|
||||
raw = await load_service_config("doc_service")
|
||||
return DocServiceConfig.model_validate(raw)
|
||||
|
||||
|
||||
def save_doc_service_config(config: DocServiceConfig) -> None:
|
||||
save_service_config("doc_service", config.model_dump())
|
||||
async def save_doc_service_config(config: DocServiceConfig) -> None:
|
||||
await save_service_config("doc_service", config.model_dump())
|
||||
|
||||
|
||||
def load_doc_service_config_masked() -> dict:
|
||||
return load_service_config("doc_service")
|
||||
async def load_doc_service_config_masked() -> dict:
|
||||
return await load_service_config("doc_service")
|
||||
|
||||
|
||||
def _merge_api_key(new_key: str, existing_key: str) -> str:
|
||||
@@ -172,18 +167,16 @@ def _merge_api_key(new_key: str, existing_key: str) -> str:
|
||||
|
||||
# ── System prompts helpers ─────────────────────────────────────────────────────
|
||||
|
||||
# Registry of all services that have editable system prompts.
|
||||
# key = service identifier, value = human-readable label
|
||||
SYSTEM_PROMPT_SERVICES: dict[str, str] = {
|
||||
"doc_service": "Document Service",
|
||||
}
|
||||
|
||||
|
||||
def load_all_system_prompts() -> dict:
|
||||
async def load_all_system_prompts() -> dict:
|
||||
"""Return {service_id: {label, system, user_template, default_system, default_user_template}}."""
|
||||
result: dict = {}
|
||||
for service_id, label in SYSTEM_PROMPT_SERVICES.items():
|
||||
config = load_service_config(service_id)
|
||||
config = await load_service_config(service_id)
|
||||
prompts = config.get("system_prompts", {})
|
||||
defaults = _get_service_prompt_defaults(service_id)
|
||||
result[service_id] = {
|
||||
@@ -196,15 +189,14 @@ def load_all_system_prompts() -> dict:
|
||||
return result
|
||||
|
||||
|
||||
def save_service_system_prompts(service_id: str, system: str, user_template: str) -> None:
|
||||
"""Persist updated system prompts into the service's config file."""
|
||||
async def save_service_system_prompts(service_id: str, system: str, user_template: str) -> None:
|
||||
if service_id not in SYSTEM_PROMPT_SERVICES:
|
||||
raise ValueError(f"Unknown service: {service_id!r}")
|
||||
config = load_service_config(service_id)
|
||||
config = await load_service_config(service_id)
|
||||
config.setdefault("system_prompts", {})
|
||||
config["system_prompts"]["system"] = system
|
||||
config["system_prompts"]["user_template"] = user_template
|
||||
save_service_config(service_id, config)
|
||||
await save_service_config(service_id, config)
|
||||
|
||||
|
||||
def _get_service_prompt_defaults(service_id: str) -> dict:
|
||||
@@ -221,26 +213,19 @@ class AppearanceConfig(BaseModel):
|
||||
default_mode: str = "system"
|
||||
|
||||
|
||||
def load_appearance_config() -> AppearanceConfig:
|
||||
path = _CONFIG_DIR / "appearance_config.json"
|
||||
if not path.exists():
|
||||
async def load_appearance_config() -> AppearanceConfig:
|
||||
data = await config_storage.read_json("appearance_config.json")
|
||||
if data is None:
|
||||
return AppearanceConfig()
|
||||
with path.open() as f:
|
||||
return AppearanceConfig.model_validate(json.load(f))
|
||||
return AppearanceConfig.model_validate(data)
|
||||
|
||||
|
||||
def save_appearance_config(config: AppearanceConfig) -> None:
|
||||
path = _CONFIG_DIR / "appearance_config.json"
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(config.model_dump(), indent=2))
|
||||
os.replace(tmp, path)
|
||||
async def save_appearance_config(config: AppearanceConfig) -> None:
|
||||
await config_storage.write_json("appearance_config.json", config.model_dump())
|
||||
|
||||
|
||||
# ── Theme file management ──────────────────────────────────────────────────────
|
||||
|
||||
_THEMES_DIR = _CONFIG_DIR / "themes"
|
||||
|
||||
# 9 required colour tokens per mode
|
||||
_REQUIRED_TOKENS = frozenset({
|
||||
"primary", "primary_hover", "accent", "accent_hover",
|
||||
@@ -361,36 +346,57 @@ _BUILTIN_THEMES: list[dict] = [
|
||||
]
|
||||
|
||||
|
||||
def seed_builtin_themes() -> None:
|
||||
"""Create /config/themes/ and write built-in theme files if missing."""
|
||||
_THEMES_DIR.mkdir(parents=True, exist_ok=True)
|
||||
async def seed_builtin_themes() -> None:
|
||||
"""Write built-in theme files to storage-service if they are not already there."""
|
||||
existing_keys = await config_storage.list_keys(prefix="themes/")
|
||||
existing_ids = {k.removeprefix("themes/").removesuffix(".json") for k in existing_keys}
|
||||
for theme in _BUILTIN_THEMES:
|
||||
path = _THEMES_DIR / f"{theme['id']}.json"
|
||||
if not path.exists():
|
||||
path.write_text(json.dumps(theme, indent=2))
|
||||
if theme["id"] not in existing_ids:
|
||||
await config_storage.write_json(f"themes/{theme['id']}.json", theme)
|
||||
logger.info("Built-in themes seeded (%d themes)", len(_BUILTIN_THEMES))
|
||||
|
||||
|
||||
def load_all_themes() -> list[dict]:
|
||||
"""Return all themes from /config/themes/*.json, built-ins first."""
|
||||
if not _THEMES_DIR.exists():
|
||||
seed_builtin_themes()
|
||||
themes = []
|
||||
for f in sorted(_THEMES_DIR.glob("*.json")):
|
||||
try:
|
||||
themes.append(json.loads(f.read_text()))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
# Sort: built-ins first (preserving their original order), then custom by label
|
||||
async def load_all_themes() -> list[dict]:
|
||||
"""Return all themes from storage-service, built-ins first then custom by label."""
|
||||
keys = await config_storage.list_keys(prefix="themes/")
|
||||
themes: list[dict] = []
|
||||
for key in keys:
|
||||
data = await config_storage.read_json(key)
|
||||
if data:
|
||||
themes.append(data)
|
||||
|
||||
builtin_ids = [t["id"] for t in _BUILTIN_THEMES]
|
||||
|
||||
def sort_key(t: dict) -> tuple:
|
||||
tid = t.get("id", "")
|
||||
try:
|
||||
return (0, builtin_ids.index(tid))
|
||||
except ValueError:
|
||||
return (1, t.get("label", tid).lower())
|
||||
|
||||
return sorted(themes, key=sort_key)
|
||||
|
||||
|
||||
async def load_theme_by_id(theme_id: str) -> dict | None:
|
||||
"""Return a single theme dict, or None if not found."""
|
||||
return await config_storage.read_json(f"themes/{theme_id}.json")
|
||||
|
||||
|
||||
async def save_theme(theme: dict) -> None:
|
||||
"""Write a theme to storage-service."""
|
||||
await config_storage.write_json(f"themes/{theme['id']}.json", theme)
|
||||
|
||||
|
||||
async def delete_theme(theme_id: str) -> None:
|
||||
"""Delete a custom theme. Raises ValueError for built-ins, KeyError if not found."""
|
||||
data = await config_storage.read_json(f"themes/{theme_id}.json")
|
||||
if data is None:
|
||||
raise FileNotFoundError(theme_id)
|
||||
if data.get("builtin"):
|
||||
raise ValueError("Cannot delete a built-in theme")
|
||||
await config_storage.delete_key(f"themes/{theme_id}.json")
|
||||
|
||||
|
||||
def validate_theme_tokens(colors: dict) -> list[str]:
|
||||
"""Return a list of validation error messages, empty if valid."""
|
||||
errors = []
|
||||
@@ -401,23 +407,3 @@ def validate_theme_tokens(colors: dict) -> list[str]:
|
||||
if key in _REQUIRED_TOKENS and not _RGB_RE.match(str(val)):
|
||||
errors.append(f"Token '{key}' must be an RGB triplet like '37 99 235', got: {val!r}")
|
||||
return errors
|
||||
|
||||
|
||||
def save_theme(theme: dict) -> None:
|
||||
"""Write a theme file atomically."""
|
||||
_THEMES_DIR.mkdir(parents=True, exist_ok=True)
|
||||
path = _THEMES_DIR / f"{theme['id']}.json"
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(theme, indent=2))
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def delete_theme(theme_id: str) -> None:
|
||||
"""Delete a custom theme file. Raises ValueError for built-ins, FileNotFoundError if missing."""
|
||||
path = _THEMES_DIR / f"{theme_id}.json"
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(theme_id)
|
||||
data = json.loads(path.read_text())
|
||||
if data.get("builtin"):
|
||||
raise ValueError("Cannot delete a built-in theme")
|
||||
path.unlink()
|
||||
|
||||
@@ -17,6 +17,7 @@ class Settings(BaseSettings):
|
||||
|
||||
DOC_SERVICE_URL: str = "http://doc-service:8001"
|
||||
AI_SERVICE_URL: str = "http://ai-service:8010"
|
||||
STORAGE_SERVICE_URL: str = "http://storage-service:8020"
|
||||
|
||||
@field_validator("JWT_PRIVATE_KEY", "JWT_PUBLIC_KEY", mode="before")
|
||||
@classmethod
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
"""
|
||||
Async HTTP client for the 'config' bucket in storage-service.
|
||||
|
||||
All JSON config files (AI settings, doc settings, appearance, themes, …) are stored
|
||||
in the 'config' bucket under the storage-service. This module provides thin
|
||||
async helpers so app_config.py does not depend on the filesystem at all.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_BUCKET = "config"
|
||||
_TIMEOUT = 10.0
|
||||
|
||||
|
||||
def _url(key: str) -> str:
|
||||
return f"{settings.STORAGE_SERVICE_URL}/objects/{_BUCKET}/{key}"
|
||||
|
||||
|
||||
async def read_json(key: str) -> dict | None:
|
||||
"""Return parsed JSON from the config bucket, or None if the key does not exist."""
|
||||
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
|
||||
resp = await client.get(_url(key))
|
||||
if resp.status_code == 404:
|
||||
return None
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
async def write_json(key: str, data: dict) -> None:
|
||||
"""Serialise *data* to JSON and PUT it into the config bucket."""
|
||||
payload = json.dumps(data, indent=2).encode()
|
||||
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
|
||||
resp = await client.put(
|
||||
_url(key),
|
||||
content=payload,
|
||||
headers={"Content-Type": "application/octet-stream"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
|
||||
async def delete_key(key: str) -> None:
|
||||
"""Delete a key from the config bucket. No-op if it does not exist."""
|
||||
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
|
||||
resp = await client.delete(_url(key))
|
||||
if resp.status_code not in (204, 404):
|
||||
resp.raise_for_status()
|
||||
|
||||
|
||||
async def list_keys(prefix: str = "") -> list[str]:
|
||||
"""List all keys in the config bucket, optionally filtered by *prefix*."""
|
||||
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
|
||||
resp = await client.get(f"{settings.STORAGE_SERVICE_URL}/objects/{_BUCKET}")
|
||||
resp.raise_for_status()
|
||||
keys: list[str] = resp.json().get("keys", [])
|
||||
if prefix:
|
||||
keys = [k for k in keys if k.startswith(prefix)]
|
||||
return keys
|
||||
+4
-1
@@ -9,16 +9,18 @@ from app.core.config import settings
|
||||
from app.database import AsyncSessionLocal
|
||||
from app.routers import admin, auth, categories_proxy, documents_proxy, groups, plugins, profile, services, users
|
||||
from app.routers import settings as settings_router
|
||||
from app.routers import storage_config
|
||||
from app.services.group_bootstrap import ensure_service_admin_groups
|
||||
from app.services.service_health import check_all, health_check_loop, register_services
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
await asyncio.to_thread(seed_builtin_themes)
|
||||
await seed_builtin_themes()
|
||||
register_services(
|
||||
doc_service_url=settings.DOC_SERVICE_URL,
|
||||
ai_service_url=settings.AI_SERVICE_URL,
|
||||
storage_service_url=settings.STORAGE_SERVICE_URL,
|
||||
)
|
||||
# Create <service-id>-admin groups for every registered service (idempotent)
|
||||
async with AsyncSessionLocal() as db:
|
||||
@@ -51,6 +53,7 @@ app.include_router(admin.router, prefix="/api/admin", tags=["admin"])
|
||||
app.include_router(groups.router, prefix="/api/admin/groups", tags=["admin"])
|
||||
app.include_router(settings_router.router, prefix="/api/settings", tags=["settings"])
|
||||
app.include_router(services.router, prefix="/api/services", tags=["services"])
|
||||
app.include_router(storage_config.router, prefix="/api/admin", tags=["admin"])
|
||||
app.include_router(plugins.router, prefix="/api/plugins", tags=["plugins"])
|
||||
# categories_proxy MUST be registered before documents_proxy —
|
||||
# otherwise /api/documents/{path:path} swallows /api/documents/categories/*
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import DateTime, ForeignKey, String, UniqueConstraint
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, String, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.database import Base
|
||||
@@ -35,6 +35,9 @@ class GroupMembership(Base):
|
||||
user_id: Mapped[str] = mapped_column(
|
||||
String, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
is_group_admin: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=False, server_default="false"
|
||||
)
|
||||
joined_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=lambda: datetime.now(timezone.utc),
|
||||
|
||||
@@ -39,18 +39,26 @@ _HOP_BY_HOP = frozenset([
|
||||
_STRIP_RESPONSE = frozenset([*_HOP_BY_HOP, "content-length", "content-type"])
|
||||
|
||||
|
||||
async def _forward_headers(request: Request, user_id: str, db: AsyncSession) -> dict:
|
||||
async def _forward_headers(
|
||||
request: Request, user_id: str, is_admin: bool, db: AsyncSession
|
||||
) -> dict:
|
||||
headers = {
|
||||
k: v
|
||||
for k, v in request.headers.items()
|
||||
if k.lower() not in _HOP_BY_HOP
|
||||
}
|
||||
headers["x-user-id"] = user_id
|
||||
result = await db.execute(
|
||||
select(GroupMembership.group_id).where(GroupMembership.user_id == user_id)
|
||||
headers["x-user-is-admin"] = "true" if is_admin else "false"
|
||||
|
||||
mem_result = await db.execute(
|
||||
select(GroupMembership.group_id, GroupMembership.is_group_admin)
|
||||
.where(GroupMembership.user_id == user_id)
|
||||
)
|
||||
group_ids = [row[0] for row in result.all()]
|
||||
rows = mem_result.all()
|
||||
group_ids = [row[0] for row in rows]
|
||||
admin_group_ids = [row[0] for row in rows if row[1]]
|
||||
headers["x-user-groups"] = ",".join(group_ids)
|
||||
headers["x-user-admin-groups"] = ",".join(admin_group_ids)
|
||||
return headers
|
||||
|
||||
|
||||
@@ -63,7 +71,7 @@ async def proxy_categories(
|
||||
path: str = "",
|
||||
) -> Response:
|
||||
url = f"/categories/{path}" if path else "/categories"
|
||||
headers = await _forward_headers(request, str(current_user.id), db)
|
||||
headers = await _forward_headers(request, str(current_user.id), current_user.is_superuser, db)
|
||||
body = await request.body()
|
||||
|
||||
try:
|
||||
|
||||
@@ -50,21 +50,28 @@ _HOP_BY_HOP = frozenset([
|
||||
_STRIP_RESPONSE = frozenset([*_HOP_BY_HOP, "content-length", "content-type"])
|
||||
|
||||
|
||||
async def _forward_headers(request: Request, user_id: str, db: AsyncSession) -> dict:
|
||||
async def _forward_headers(
|
||||
request: Request, user_id: str, is_admin: bool, db: AsyncSession
|
||||
) -> dict:
|
||||
headers = {
|
||||
k: v
|
||||
for k, v in request.headers.items()
|
||||
if k.lower() not in _HOP_BY_HOP
|
||||
}
|
||||
headers["x-user-id"] = user_id
|
||||
headers["x-user-is-admin"] = "true" if is_admin else "false"
|
||||
|
||||
# Inject the user's group memberships so the doc-service can evaluate
|
||||
# group-shared document access without querying the backend DB.
|
||||
result = await db.execute(
|
||||
select(GroupMembership.group_id).where(GroupMembership.user_id == user_id)
|
||||
# Inject group memberships and group-admin status so the doc-service can
|
||||
# evaluate ownership, sharing access, and group-admin permissions.
|
||||
mem_result = await db.execute(
|
||||
select(GroupMembership.group_id, GroupMembership.is_group_admin)
|
||||
.where(GroupMembership.user_id == user_id)
|
||||
)
|
||||
group_ids = [row[0] for row in result.all()]
|
||||
rows = mem_result.all()
|
||||
group_ids = [row[0] for row in rows]
|
||||
admin_group_ids = [row[0] for row in rows if row[1]]
|
||||
headers["x-user-groups"] = ",".join(group_ids)
|
||||
headers["x-user-admin-groups"] = ",".join(admin_group_ids)
|
||||
|
||||
return headers
|
||||
|
||||
@@ -78,7 +85,7 @@ async def proxy_documents(
|
||||
path: str = "",
|
||||
) -> Response:
|
||||
url = f"/documents/{path}" if path else "/documents"
|
||||
headers = await _forward_headers(request, str(current_user.id), db)
|
||||
headers = await _forward_headers(request, str(current_user.id), current_user.is_superuser, db)
|
||||
body = await request.body()
|
||||
|
||||
try:
|
||||
|
||||
@@ -7,7 +7,7 @@ from app.database import get_db
|
||||
from app.deps import get_current_admin
|
||||
from app.models.group import Group, GroupMembership
|
||||
from app.models.user import User
|
||||
from app.schemas.group import GroupCreate, GroupDetailOut, GroupOut, GroupUpdate, GroupMemberOut
|
||||
from app.schemas.group import GroupCreate, GroupDetailOut, GroupMemberAdminUpdate, GroupMemberOut, GroupOut, GroupUpdate
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -111,6 +111,7 @@ async def get_group(
|
||||
email=user.email,
|
||||
full_name=user.full_name,
|
||||
is_active=user.is_active,
|
||||
is_group_admin=membership.is_group_admin,
|
||||
joined_at=membership.joined_at,
|
||||
)
|
||||
for membership, user in rows
|
||||
@@ -197,6 +198,26 @@ async def add_member(
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.patch("/{group_id}/members/{user_id}/admin", status_code=status.HTTP_204_NO_CONTENT)
|
||||
async def set_member_admin(
|
||||
group_id: str,
|
||||
user_id: str,
|
||||
body: GroupMemberAdminUpdate,
|
||||
_admin: User = Depends(get_current_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> None:
|
||||
result = await db.execute(
|
||||
select(GroupMembership).where(
|
||||
GroupMembership.group_id == group_id, GroupMembership.user_id == user_id
|
||||
)
|
||||
)
|
||||
membership = result.scalar_one_or_none()
|
||||
if not membership:
|
||||
raise HTTPException(status_code=404, detail="User is not a member of this group")
|
||||
membership.is_group_admin = body.is_group_admin
|
||||
await db.commit()
|
||||
|
||||
|
||||
@router.delete("/{group_id}/members/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
|
||||
async def remove_member(
|
||||
group_id: str,
|
||||
|
||||
@@ -2,10 +2,9 @@
|
||||
Admin-only settings API for per-service runtime configuration.
|
||||
|
||||
All endpoints require the caller to be an admin (Depends(get_current_admin)).
|
||||
Config files live on the shared app_config volume (/config/).
|
||||
Config files are stored in the 'config' bucket of storage-service.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import re as _re
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
@@ -21,10 +20,11 @@ from app.core.app_config import (
|
||||
load_all_system_prompts,
|
||||
load_all_themes,
|
||||
load_appearance_config,
|
||||
save_appearance_config,
|
||||
load_doc_service_config,
|
||||
load_doc_service_config_masked,
|
||||
load_theme_by_id,
|
||||
save_ai_service_config,
|
||||
save_appearance_config,
|
||||
save_doc_service_config,
|
||||
save_service_system_prompts,
|
||||
save_theme,
|
||||
@@ -36,6 +36,8 @@ from app.models.user import User
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
_THEME_ID_RE = _re.compile(r"^[a-z0-9_-]{1,64}$")
|
||||
|
||||
|
||||
# ── Pydantic request bodies ────────────────────────────────────────────────────
|
||||
|
||||
@@ -98,7 +100,7 @@ class ThemeUpdate(BaseModel):
|
||||
async def get_ai_settings(
|
||||
_: User = Depends(get_service_admin("ai-service")),
|
||||
) -> dict:
|
||||
return load_ai_service_config_masked()
|
||||
return await load_ai_service_config_masked()
|
||||
|
||||
|
||||
@router.patch("/ai")
|
||||
@@ -110,7 +112,7 @@ async def update_ai_settings(
|
||||
if body.provider not in valid_providers:
|
||||
raise HTTPException(status_code=422, detail=f"provider must be one of {valid_providers}")
|
||||
|
||||
config = load_ai_service_config()
|
||||
config = await load_ai_service_config()
|
||||
config.provider = body.provider
|
||||
|
||||
# Anthropic
|
||||
@@ -139,8 +141,8 @@ async def update_ai_settings(
|
||||
body.lmstudio_api_key, config.lmstudio.api_key
|
||||
)
|
||||
|
||||
await asyncio.to_thread(save_ai_service_config, config)
|
||||
return load_ai_service_config_masked()
|
||||
await save_ai_service_config(config)
|
||||
return await load_ai_service_config_masked()
|
||||
|
||||
|
||||
@router.post("/ai/test")
|
||||
@@ -173,7 +175,7 @@ async def test_ai_connection(
|
||||
async def get_documents_limits(
|
||||
_: User = Depends(get_service_admin("doc-service")),
|
||||
) -> dict:
|
||||
return load_doc_service_config_masked()
|
||||
return await load_doc_service_config_masked()
|
||||
|
||||
|
||||
@router.patch("/documents/limits")
|
||||
@@ -184,10 +186,10 @@ async def update_documents_limits(
|
||||
if body.max_pdf_mb < 1 or body.max_pdf_mb > 200:
|
||||
raise HTTPException(status_code=422, detail="max_pdf_mb must be between 1 and 200")
|
||||
|
||||
config = load_doc_service_config()
|
||||
config = await load_doc_service_config()
|
||||
config.documents.max_pdf_bytes = body.max_pdf_mb * 1024 * 1024
|
||||
await asyncio.to_thread(save_doc_service_config, config)
|
||||
return load_doc_service_config_masked()
|
||||
await save_doc_service_config(config)
|
||||
return await load_doc_service_config_masked()
|
||||
|
||||
|
||||
# ── System prompts ─────────────────────────────────────────────────────────────
|
||||
@@ -197,8 +199,7 @@ async def update_documents_limits(
|
||||
async def get_system_prompts(
|
||||
_: User = Depends(get_service_admin("ai-service")),
|
||||
) -> dict:
|
||||
"""Return all editable system prompts, keyed by service id."""
|
||||
return await asyncio.to_thread(load_all_system_prompts)
|
||||
return await load_all_system_prompts()
|
||||
|
||||
|
||||
@router.patch("/system-prompts/{service_id}")
|
||||
@@ -207,26 +208,20 @@ async def update_system_prompt(
|
||||
body: SystemPromptUpdate,
|
||||
_: User = Depends(get_service_admin("ai-service")),
|
||||
) -> dict:
|
||||
"""Update the system prompts for a single service."""
|
||||
if service_id not in SYSTEM_PROMPT_SERVICES:
|
||||
raise HTTPException(status_code=404, detail=f"No system prompts registered for {service_id!r}")
|
||||
await asyncio.to_thread(
|
||||
save_service_system_prompts, service_id, body.system, body.user_template
|
||||
)
|
||||
return await asyncio.to_thread(load_all_system_prompts)
|
||||
await save_service_system_prompts(service_id, body.system, body.user_template)
|
||||
return await load_all_system_prompts()
|
||||
|
||||
|
||||
# ── Appearance (global default — auth read, admin write) ───────────────────────
|
||||
|
||||
import re as _re
|
||||
_THEME_ID_RE = _re.compile(r"^[a-z0-9_-]{1,64}$")
|
||||
|
||||
|
||||
@router.get("/appearance")
|
||||
async def get_appearance(
|
||||
_: User = Depends(get_current_user),
|
||||
) -> dict:
|
||||
config = await asyncio.to_thread(load_appearance_config)
|
||||
config = await load_appearance_config()
|
||||
return config.model_dump()
|
||||
|
||||
|
||||
@@ -237,12 +232,12 @@ async def update_appearance(
|
||||
) -> dict:
|
||||
if body.default_mode not in ("light", "dark", "system"):
|
||||
raise HTTPException(status_code=422, detail="default_mode must be 'light', 'dark', or 'system'")
|
||||
themes = await asyncio.to_thread(load_all_themes)
|
||||
themes = await load_all_themes()
|
||||
theme_ids = {t["id"] for t in themes}
|
||||
if body.theme not in theme_ids:
|
||||
raise HTTPException(status_code=422, detail=f"Unknown theme: {body.theme!r}")
|
||||
config = AppearanceConfig(theme=body.theme, default_mode=body.default_mode)
|
||||
await asyncio.to_thread(save_appearance_config, config)
|
||||
await save_appearance_config(config)
|
||||
return config.model_dump()
|
||||
|
||||
|
||||
@@ -253,7 +248,7 @@ async def update_appearance(
|
||||
async def list_themes(
|
||||
_: User = Depends(get_current_user),
|
||||
) -> list:
|
||||
return await asyncio.to_thread(load_all_themes)
|
||||
return await load_all_themes()
|
||||
|
||||
|
||||
@router.post("/themes", status_code=201)
|
||||
@@ -263,7 +258,7 @@ async def create_theme(
|
||||
) -> dict:
|
||||
if not _THEME_ID_RE.match(body.id):
|
||||
raise HTTPException(status_code=422, detail="Theme ID must match [a-z0-9_-]{1,64}")
|
||||
existing = {t["id"] for t in await asyncio.to_thread(load_all_themes)}
|
||||
existing = {t["id"] for t in await load_all_themes()}
|
||||
if body.id in existing:
|
||||
raise HTTPException(status_code=400, detail=f"Theme ID already in use: {body.id!r}")
|
||||
light = body.light.model_dump()
|
||||
@@ -273,7 +268,7 @@ async def create_theme(
|
||||
if errors:
|
||||
raise HTTPException(status_code=422, detail=f"{mode}: {'; '.join(errors)}")
|
||||
theme = {"id": body.id, "label": body.label, "builtin": False, "light": light, "dark": dark}
|
||||
await asyncio.to_thread(save_theme, theme)
|
||||
await save_theme(theme)
|
||||
return theme
|
||||
|
||||
|
||||
@@ -283,11 +278,9 @@ async def update_theme(
|
||||
body: ThemeUpdate,
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> dict:
|
||||
from app.core.app_config import _THEMES_DIR
|
||||
path = _THEMES_DIR / f"{theme_id}.json"
|
||||
if not path.exists():
|
||||
theme = await load_theme_by_id(theme_id)
|
||||
if theme is None:
|
||||
raise HTTPException(status_code=404, detail="Theme not found")
|
||||
theme = json.loads(path.read_text())
|
||||
if theme.get("builtin"):
|
||||
raise HTTPException(status_code=400, detail="Cannot edit a built-in theme")
|
||||
if body.label is not None:
|
||||
@@ -304,7 +297,7 @@ async def update_theme(
|
||||
if errors:
|
||||
raise HTTPException(status_code=422, detail=f"dark: {'; '.join(errors)}")
|
||||
theme["dark"] = dark
|
||||
await asyncio.to_thread(save_theme, theme)
|
||||
await save_theme(theme)
|
||||
return theme
|
||||
|
||||
|
||||
@@ -314,7 +307,7 @@ async def remove_theme(
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> None:
|
||||
try:
|
||||
await asyncio.to_thread(delete_theme, theme_id)
|
||||
await delete_theme(theme_id)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="Theme not found")
|
||||
except ValueError as exc:
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""
|
||||
Admin-only endpoints for storage-service backend configuration.
|
||||
|
||||
GET /admin/storage-config — current backend driver + health
|
||||
PATCH /admin/storage-config — update backend config (no data migration)
|
||||
POST /admin/storage-config/migrate — start migration to a new backend
|
||||
GET /admin/storage-config/migrate/status — poll migration progress
|
||||
DELETE /admin/storage-config/migrate — cancel in-progress migration
|
||||
|
||||
All endpoints proxy to storage-service:8020.
|
||||
"""
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.config import settings
|
||||
from app.deps import get_current_admin
|
||||
from app.models.user import User
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_STORAGE_BASE = settings.STORAGE_SERVICE_URL
|
||||
|
||||
|
||||
class BackendConfigUpdate(BaseModel):
|
||||
driver: str
|
||||
config: dict = {}
|
||||
|
||||
|
||||
class MigrateRequest(BaseModel):
|
||||
driver: str
|
||||
config: dict = {}
|
||||
|
||||
|
||||
def _storage_url(path: str) -> str:
|
||||
return f"{_STORAGE_BASE}{path}"
|
||||
|
||||
|
||||
async def _proxy_get(path: str) -> dict:
|
||||
async with httpx.AsyncClient(timeout=15.0) as client:
|
||||
resp = await client.get(_storage_url(path))
|
||||
if resp.status_code == 404:
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
@router.get("/storage-config")
|
||||
async def get_storage_config(
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> dict:
|
||||
"""Return current backend driver and health status."""
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.get(_storage_url("/health"))
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
@router.patch("/storage-config", status_code=204)
|
||||
async def update_storage_config(
|
||||
body: BackendConfigUpdate,
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> None:
|
||||
"""
|
||||
Reconfigure the active backend without migrating data.
|
||||
Use when changing credentials for the same backend type, or reverting to local.
|
||||
To move data to a new backend, use POST /admin/storage-config/migrate instead.
|
||||
"""
|
||||
async with httpx.AsyncClient(timeout=15.0) as client:
|
||||
resp = await client.patch(
|
||||
_storage_url("/backend-config"),
|
||||
json={"driver": body.driver, "config": body.config},
|
||||
)
|
||||
if resp.status_code == 400:
|
||||
raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed"))
|
||||
if resp.status_code == 409:
|
||||
raise HTTPException(status_code=409, detail="Migration in progress — cannot reconfigure now")
|
||||
resp.raise_for_status()
|
||||
|
||||
|
||||
@router.post("/storage-config/migrate", status_code=202)
|
||||
async def start_migration(
|
||||
body: MigrateRequest,
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> dict:
|
||||
"""
|
||||
Start an async migration to a new backend.
|
||||
|
||||
Flow: validate new backend → copy all objects → verify → switch → delete old objects.
|
||||
The old backend stays active until 100% of objects are verified on the new one.
|
||||
Poll GET /admin/storage-config/migrate/status to track progress.
|
||||
"""
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
resp = await client.post(
|
||||
_storage_url("/migrate"),
|
||||
json={"driver": body.driver, "config": body.config},
|
||||
)
|
||||
if resp.status_code == 400:
|
||||
raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed"))
|
||||
if resp.status_code == 409:
|
||||
raise HTTPException(status_code=409, detail="A migration is already in progress")
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
@router.get("/storage-config/migrate/status")
|
||||
async def migration_status(
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> dict:
|
||||
"""Poll migration progress. State: idle → validating → migrating → switching → cleaning → done."""
|
||||
return await _proxy_get("/migrate/status")
|
||||
|
||||
|
||||
@router.delete("/storage-config/migrate", status_code=204)
|
||||
async def cancel_migration(
|
||||
_: User = Depends(get_current_admin),
|
||||
) -> None:
|
||||
"""Cancel a running migration. The old backend remains active."""
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.delete(_storage_url("/migrate"))
|
||||
if resp.status_code == 409:
|
||||
raise HTTPException(status_code=409, detail="No cancellable migration in progress")
|
||||
resp.raise_for_status()
|
||||
@@ -39,14 +39,17 @@ async def get_my_groups(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Return all groups the current user belongs to."""
|
||||
"""Return all groups the current user belongs to, including their admin status."""
|
||||
result = await db.execute(
|
||||
select(Group)
|
||||
select(Group, GroupMembership.is_group_admin)
|
||||
.join(GroupMembership, GroupMembership.group_id == Group.id)
|
||||
.where(GroupMembership.user_id == current_user.id)
|
||||
.order_by(Group.name)
|
||||
)
|
||||
return result.scalars().all()
|
||||
return [
|
||||
UserGroupOut(id=g.id, name=g.name, description=g.description, is_group_admin=is_admin)
|
||||
for g, is_admin in result.all()
|
||||
]
|
||||
|
||||
|
||||
@router.patch("/me/color-mode", response_model=UserOut)
|
||||
|
||||
@@ -18,11 +18,16 @@ class GroupMemberOut(BaseModel):
|
||||
email: str
|
||||
full_name: str | None
|
||||
is_active: bool
|
||||
is_group_admin: bool = False
|
||||
joined_at: datetime
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
|
||||
class GroupMemberAdminUpdate(BaseModel):
|
||||
is_group_admin: bool
|
||||
|
||||
|
||||
class GroupOut(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
|
||||
@@ -122,6 +122,7 @@ class UserGroupOut(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
description: str | None
|
||||
is_group_admin: bool = False
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ _health: dict[str, bool | None] = {}
|
||||
_manifests: dict[str, dict | None] = {}
|
||||
|
||||
|
||||
def register_services(doc_service_url: str, ai_service_url: str) -> None:
|
||||
def register_services(doc_service_url: str, ai_service_url: str, storage_service_url: str) -> None:
|
||||
"""Called once during app startup to populate the registry from config."""
|
||||
global _REGISTRY, _health, _manifests
|
||||
|
||||
@@ -63,6 +63,15 @@ def register_services(doc_service_url: str, ai_service_url: str) -> None:
|
||||
app_path="",
|
||||
settings_path="/apps/ai/settings",
|
||||
),
|
||||
ServiceDefinition(
|
||||
id="storage-service",
|
||||
name="Storage",
|
||||
description="Unified file storage. Manages all uploaded files with pluggable backends (local, S3, WebDAV).",
|
||||
internal_url=storage_service_url,
|
||||
health_path="/health",
|
||||
app_path="",
|
||||
settings_path="/admin/storage",
|
||||
),
|
||||
]
|
||||
|
||||
_health = {svc.id: None for svc in _REGISTRY}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# 2026-04-18 — Category scopes, group admin role, and permission model
|
||||
|
||||
**Timestamp:** 2026-04-18T00:00:00Z
|
||||
|
||||
## Summary
|
||||
|
||||
Introduces three category scopes (personal / group / system), a PascalCase-with-dashes naming convention, a group-admin role on group memberships, and a full permission model for who can create, rename, and delete categories and documents.
|
||||
|
||||
## Files Added
|
||||
|
||||
- `backend/alembic/versions/e1f2a3b4c5d6_add_group_member_is_admin.py` — adds `is_group_admin BOOLEAN` to `group_memberships`
|
||||
- `features/doc-service/alembic/versions/0005_add_share_can_delete.py` — adds `can_delete BOOLEAN` to `document_shares` (backfill from feat/document-delete-permissions)
|
||||
- `features/doc-service/alembic/versions/0006_add_category_scope.py` — adds `scope VARCHAR(16)` and `group_id VARCHAR` to `document_categories`; data-migrates watch categories to scope='system'
|
||||
|
||||
## Files Modified
|
||||
|
||||
- `backend/app/models/group.py` — added `is_group_admin` to `GroupMembership`
|
||||
- `backend/app/schemas/group.py` — added `is_group_admin` to `GroupMemberOut`; new `GroupMemberAdminUpdate`
|
||||
- `backend/app/schemas/user.py` — added `is_group_admin` to `UserGroupOut`
|
||||
- `backend/app/routers/users.py` — `get_my_groups` now joins `GroupMembership` to include `is_group_admin`
|
||||
- `backend/app/routers/groups.py` — `get_group` includes `is_group_admin`; new `PATCH /{id}/members/{user_id}/admin` endpoint
|
||||
- `backend/app/routers/categories_proxy.py` — injects `x-user-is-admin` and `x-user-admin-groups` headers
|
||||
- `backend/app/routers/documents_proxy.py` — injects `x-user-admin-groups` header (was already injecting `x-user-is-admin`)
|
||||
- `features/doc-service/app/models/category.py` — added `scope`, `group_id` columns
|
||||
- `features/doc-service/app/schemas/category.py` — `CategoryOut` includes `scope`/`group_id`; `CategoryCreate` accepts `group_id`
|
||||
- `features/doc-service/app/deps.py` — added `get_user_is_admin`, `get_user_admin_groups`
|
||||
- `features/doc-service/app/routers/categories.py` — full rewrite: name validation regex, scope-based list/create, `_check_can_manage_cat` permission helper, scope-aware rename/delete
|
||||
- `features/doc-service/app/routers/documents.py` — `delete_document` enforces is_admin/can_delete/group-admin hierarchy; `remove_category` requires doc ownership; `assign_category` accepts group/system categories
|
||||
- `frontend/src/api/client.ts` — `CategoryOut` gains `scope`/`group_id`; `createCategory` accepts optional `groupId`; `UserGroupOut`/`GroupMemberOut` gain `is_group_admin`; new `adminSetGroupMemberAdmin()`; `ApiError` exported
|
||||
- `frontend/src/components/ManageCategoriesDialog.tsx` — categories grouped by scope; lock icons for unmanageable categories; rename/delete gated by scope permissions; inline rename error display
|
||||
- `frontend/src/components/SourcePanel.tsx` — categories shown in sections (Mine / Group name / System); scope picker on new category form; client-side name validation
|
||||
- `frontend/src/pages/AdminGroupsPage.tsx` — group admin checkbox column in members table
|
||||
- `backend/CLAUDE.md` — updated `group_memberships` model, migration chain, endpoints
|
||||
- `features/doc-service/CLAUDE.md` — updated `document_categories` model, `document_shares` model, migration chain, deps note
|
||||
@@ -0,0 +1,25 @@
|
||||
# 2026-04-18 — Document delete permissions + three-dots menu fix
|
||||
|
||||
**Timestamp:** 2026-04-18T00:00:00Z
|
||||
|
||||
## Summary
|
||||
|
||||
Added a proper permission model for document deletion: owners and admins can always delete; group members can delete only when the share was explicitly granted `can_delete=true`. Fixed silent delete failures (watch docs returning 404 with no user feedback) and fixed the three-dots context menu being clipped by `overflow-hidden` on the table container.
|
||||
|
||||
## Files Added / Modified / Deleted
|
||||
|
||||
### Added
|
||||
- `features/doc-service/alembic/versions/0005_add_share_can_delete.py` — migration: adds `can_delete BOOLEAN NOT NULL DEFAULT false` to `document_shares`
|
||||
|
||||
### Modified
|
||||
- `features/doc-service/app/models/document_share.py` — added `can_delete: Mapped[bool]` column
|
||||
- `features/doc-service/app/schemas/share.py` — added `can_delete` to `DocumentShareOut` and `DocumentShareCreate`; added `viewer_can_delete` to `SharedDocumentOut`
|
||||
- `features/doc-service/app/schemas/document.py` — added `viewer_can_delete: bool = False` to `DocumentOut`
|
||||
- `features/doc-service/app/deps.py` — added `get_user_is_admin()` dep reading `x-user-is-admin` header
|
||||
- `features/doc-service/app/routers/documents.py` — added `_get_deletable_doc_ids()` helper; updated list/get/delete endpoints with permission logic; updated `add_share` to store `can_delete`; updated shared-with-me to include `viewer_can_delete`
|
||||
- `backend/app/routers/documents_proxy.py` — `_forward_headers()` now injects `x-user-is-admin` header
|
||||
- `frontend/src/api/client.ts` — `DocumentOut`: added `viewer_can_delete`; `DocumentShareOut`: added `can_delete`; `addDocumentShare`: accepts `canDelete` param
|
||||
- `frontend/src/pages/DocumentsPage.tsx` — `RowActionsMenu`: replaced absolute dropdown with `createPortal` to fix clipping; delete button now uses `doc.viewer_can_delete`; added `onError` handler for silent failures
|
||||
- `frontend/src/components/DocumentSlideOver.tsx` — sharing section: shows trash icon badge on shares with `can_delete=true`; added "Allow group members to delete" checkbox before group picker; delete button uses `doc.viewer_can_delete`
|
||||
- `features/doc-service/CLAUDE.md` — updated `document_shares` table docs + migration chain
|
||||
- `backend/CLAUDE.md` — noted `x-user-is-admin` header injection
|
||||
@@ -0,0 +1,12 @@
|
||||
# 2026-04-19 — Merge checklist update + admin delete bug fix
|
||||
|
||||
**Timestamp:** 2026-04-19T00:15:00Z
|
||||
|
||||
## Summary
|
||||
|
||||
Updated `tests/MERGE_CHECKLIST.md` with all new tests for the two recently merged features (document delete permissions and category scopes / group-admin role). While running the new test 12.16b, discovered and fixed a bug where the doc-service delete endpoint returned 404 for admins deleting non-owned documents.
|
||||
|
||||
## Files Added / Modified / Deleted
|
||||
|
||||
- **Modified** `tests/MERGE_CHECKLIST.md` — added 18 new tests: 4.9–4.10 (group admin role), 12.16b–12.16e (delete permissions), 13.11–13.14 (can_delete sharing), 14.7–14.17 (category scopes, PascalCase naming), 19.11 (three-dots portal fix); updated 12.16 and 14.5 descriptions
|
||||
- **Modified** `features/doc-service/app/routers/documents.py` — fixed `delete_document` to bypass group-membership filter for admins; previously admins got 404 on any document they didn't own or that wasn't a watch doc
|
||||
@@ -0,0 +1,60 @@
|
||||
# 2026-04-20 — Dedicated storage-service with pluggable backends
|
||||
|
||||
**Timestamp:** 2026-04-20T00:00:00Z
|
||||
|
||||
## Summary
|
||||
|
||||
Introduced a dedicated `storage-service` container (port 8020) as the single file/blob persistence layer for the entire stack. All services now route file and config I/O through this service's HTTP API. The service supports pluggable storage backends (local filesystem by default; S3-compatible and WebDAV built in) with a zero-data-loss migration flow. The `doc_data` and `app_config` Docker volumes were removed.
|
||||
|
||||
## Files Added
|
||||
|
||||
- `features/storage-service/app/main.py` — FastAPI app, lifespan (backend init)
|
||||
- `features/storage-service/app/core/config.py` — Settings (DATA_DIR, STORAGE_BACKEND, S3_*, WEBDAV_*)
|
||||
- `features/storage-service/app/routers/health.py` — GET /health
|
||||
- `features/storage-service/app/routers/objects.py` — PUT/GET/DELETE /objects/{bucket}/{key:path}, GET /objects/{bucket}
|
||||
- `features/storage-service/app/routers/migrate.py` — POST/GET/DELETE /migrate, PATCH /backend-config
|
||||
- `features/storage-service/app/services/backend_manager.py` — Driver factory, singleton, atomic switch
|
||||
- `features/storage-service/app/services/migration.py` — Async migration: copy → verify → switch → cleanup
|
||||
- `features/storage-service/app/services/backends/base.py` — AbstractStorageBackend ABC
|
||||
- `features/storage-service/app/services/backends/local.py` — LocalFSBackend (path traversal guard)
|
||||
- `features/storage-service/app/services/backends/s3.py` — S3Backend (aiobotocore, endpoint_url configurable)
|
||||
- `features/storage-service/app/services/backends/webdav.py` — WebDAVBackend (aiohttp + defusedxml)
|
||||
- `features/storage-service/scripts/start.sh` — prod uvicorn start
|
||||
- `features/storage-service/scripts/start_dev.sh` — dev uvicorn --reload start
|
||||
- `features/storage-service/pyproject.toml` — Dependencies
|
||||
- `features/storage-service/Dockerfile` — python:3.12-slim, non-root user 1001, port 8020
|
||||
- `features/storage-service/CLAUDE.md` — API reference, bucket docs, driver docs
|
||||
- `features/storage-service/STATUS.md` — Service status
|
||||
- `backend/app/core/config_storage.py` — Thin async helpers: read_json/write_json/delete_key/list_keys
|
||||
- `backend/app/routers/storage_config.py` — Admin proxy endpoints for storage config + migration
|
||||
- `features/doc-service/alembic/versions/0008_rename_file_path_to_storage_key.py` — DB migration
|
||||
- `frontend/src/pages/StorageAdminPage.tsx` — Admin UI: backend status, driver form, migration progress
|
||||
- `tests/storage-service_tests.md` — §20 storage-service test suite
|
||||
|
||||
## Files Modified
|
||||
|
||||
- `docker-compose.yml` — Added storage-service, storage_data volume; removed doc_data, app_config; added depends_on service_healthy
|
||||
- `docker-compose.dev.yml` — Added storage-service dev override
|
||||
- `backend/app/core/config.py` — Added STORAGE_SERVICE_URL
|
||||
- `backend/app/core/app_config.py` — Full async rewrite using config_storage HTTP helpers (no filesystem)
|
||||
- `backend/app/routers/settings.py` — Removed all asyncio.to_thread wrappers; direct await calls
|
||||
- `backend/app/main.py` — Register storage_config router; update register_services call
|
||||
- `backend/app/services/service_health.py` — Register storage-service
|
||||
- `features/doc-service/app/core/config.py` — Added STORAGE_SERVICE_URL
|
||||
- `features/doc-service/app/models/document.py` — file_path → storage_key
|
||||
- `features/doc-service/app/services/storage.py` — Complete rewrite: HTTP client calls to storage-service
|
||||
- `features/doc-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket
|
||||
- `features/doc-service/app/services/file_watcher.py` — Uses save_upload() → storage-service
|
||||
- `features/doc-service/app/routers/documents.py` — storage_key refs, pdfplumber(io.BytesIO), streaming from storage-service
|
||||
- `features/ai-service/app/core/config.py` — Added STORAGE_SERVICE_URL; removed CONFIG_PATH
|
||||
- `features/ai-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket
|
||||
- `frontend/src/api/client.ts` — Added StorageStatus, MigrationStatus, StorageBackendConfig interfaces + 5 API functions
|
||||
- `frontend/src/App.tsx` — Added /admin/storage route (AdminRoute → StorageAdminPage)
|
||||
- `tests/ALL_TESTS.md` — Updated to 20 feature areas; added §20 storage-service tests
|
||||
- `CLAUDE.md` — Added storage-service to Services/Volumes/Networks tables; storage enforcement rule; §20 test file
|
||||
- `backend/CLAUDE.md` — Added config_storage.py, storage_config.py to tree; added admin storage endpoints
|
||||
- `frontend/CLAUDE.md` — Added StorageAdminPage to tree; added /admin/storage route
|
||||
- `features/doc-service/CLAUDE.md` — Updated storage.py description; file_path → storage_key; added migration 0008
|
||||
- `features/ai-service/CLAUDE.md` — Added config_reader.py description
|
||||
- `backend/STATUS.md` — Added storage-config endpoints; updated settings persistence note
|
||||
- `frontend/STATUS.md` — Added /admin/storage route; added StorageAdminPage description
|
||||
@@ -29,6 +29,11 @@ services:
|
||||
volumes:
|
||||
- ./features/ai-service:/app
|
||||
|
||||
storage-service:
|
||||
command: sh scripts/start_dev.sh
|
||||
volumes:
|
||||
- ./features/storage-service:/app
|
||||
|
||||
doc-service:
|
||||
command: sh scripts/start_dev.sh
|
||||
env_file: ./features/doc-service/.env
|
||||
|
||||
+32
-11
@@ -19,6 +19,27 @@ services:
|
||||
networks:
|
||||
- backend-net
|
||||
|
||||
# ── Storage service (unified blob storage) ──────────────────────────────────
|
||||
storage-service:
|
||||
build:
|
||||
context: ./features/storage-service
|
||||
dockerfile: Dockerfile
|
||||
network: host
|
||||
user: "1001:1001"
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
STORAGE_BACKEND: local
|
||||
DATA_DIR: /data/storage
|
||||
volumes:
|
||||
- storage_data:/data/storage
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8020/health')\""]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- backend-net
|
||||
|
||||
# ── Backend (management) ────────────────────────────────────────────────────
|
||||
backend:
|
||||
build:
|
||||
@@ -32,11 +53,12 @@ services:
|
||||
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-password}@db:5432/${POSTGRES_DB:-destroying_sap}
|
||||
DOC_SERVICE_URL: http://doc-service:8001
|
||||
AI_SERVICE_URL: http://ai-service:8010
|
||||
volumes:
|
||||
- app_config:/config
|
||||
STORAGE_SERVICE_URL: http://storage-service:8020
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
storage-service:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- backend-net
|
||||
|
||||
@@ -49,9 +71,10 @@ services:
|
||||
user: "1001:1001"
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
CONFIG_PATH: /config/ai_service_config.json
|
||||
volumes:
|
||||
- app_config:/config
|
||||
STORAGE_SERVICE_URL: http://storage-service:8020
|
||||
depends_on:
|
||||
storage-service:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- backend-net
|
||||
|
||||
@@ -65,18 +88,17 @@ services:
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-password}@db:5432/${POSTGRES_DB:-destroying_sap}
|
||||
DATA_DIR: /data/documents
|
||||
CONFIG_PATH: /config/doc_service_config.json
|
||||
AI_SERVICE_URL: http://ai-service:8010
|
||||
STORAGE_SERVICE_URL: http://storage-service:8020
|
||||
volumes:
|
||||
- doc_data:/data/documents
|
||||
- watch_data:/data/watch
|
||||
- app_config:/config
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
ai-service:
|
||||
condition: service_started
|
||||
storage-service:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- backend-net
|
||||
|
||||
@@ -98,9 +120,8 @@ services:
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
doc_data: # PDF files persisted across restarts
|
||||
storage_data: # All file/blob storage — managed by storage-service (documents + config)
|
||||
watch_data: # Watch directory — bind-mount your NAS/Nextcloud here via docker-compose.override.yml
|
||||
app_config: # Per-service runtime config JSON files
|
||||
|
||||
networks:
|
||||
# backend-net: db ↔ backend ↔ doc-service. No host ports bound.
|
||||
|
||||
@@ -22,7 +22,8 @@ features/ai-service/
|
||||
│ │ ├── queue.py ← GET /queue/status, /pause, /resume, /cancel/{id}
|
||||
│ │ └── plugin.py ← GET /plugin/manifest (access rules for ai-service-admin group)
|
||||
│ └── services/
|
||||
│ └── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL)
|
||||
│ ├── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL)
|
||||
│ └── config_reader.py ← Reads ai_service_config.json from storage-service config bucket (30 s TTL cache)
|
||||
├── Dockerfile ← python:3.12-slim, non-root user 1001
|
||||
└── STATUS.md
|
||||
```
|
||||
|
||||
@@ -15,8 +15,7 @@ FROM python:3.12-slim
|
||||
RUN groupadd --gid 1001 appuser && \
|
||||
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
|
||||
|
||||
# Pre-create the config directory with correct ownership
|
||||
RUN mkdir -p /config && chown -R appuser:appuser /config
|
||||
# No filesystem directories needed — all config goes through storage-service.
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@ from pydantic_settings import BaseSettings
|
||||
|
||||
class Settings(BaseSettings):
|
||||
PROJECT_NAME: str = "ai-service"
|
||||
CONFIG_PATH: str = "/config/ai_service_config.json"
|
||||
STORAGE_SERVICE_URL: str = "http://storage-service:8020"
|
||||
|
||||
model_config = {"env_file": ".env", "extra": "ignore"}
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""
|
||||
Reads ai_service_config.json from the shared config volume.
|
||||
Reads ai_service_config.json from the storage-service config bucket.
|
||||
30-second TTL cache + env var overrides (dev credentials stay out of git).
|
||||
|
||||
Env var overrides (all optional):
|
||||
@@ -8,15 +8,17 @@ Env var overrides (all optional):
|
||||
OLLAMA_BASE_URL, OLLAMA_MODEL, OLLAMA_API_KEY
|
||||
ANTHROPIC_API_KEY, ANTHROPIC_MODEL
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
_CONFIG_KEY = "ai_service_config.json"
|
||||
|
||||
_DEFAULT_CONFIG: dict = {
|
||||
"provider": "lmstudio",
|
||||
"timeout_seconds": 60,
|
||||
@@ -31,12 +33,18 @@ _cache_at: float = 0.0
|
||||
_CACHE_TTL = 30.0
|
||||
|
||||
|
||||
def _read_config_sync() -> dict:
|
||||
path = Path(settings.CONFIG_PATH)
|
||||
if not path.exists():
|
||||
return _apply_env_overrides(deepcopy(_DEFAULT_CONFIG))
|
||||
with open(path) as f:
|
||||
return _apply_env_overrides(json.load(f))
|
||||
def _storage_url() -> str:
|
||||
return f"{settings.STORAGE_SERVICE_URL}/objects/config/{_CONFIG_KEY}"
|
||||
|
||||
|
||||
async def _fetch_config() -> dict:
|
||||
"""Fetch config from storage-service. Returns defaults if not found."""
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.get(_storage_url())
|
||||
if resp.status_code == 404:
|
||||
return deepcopy(_DEFAULT_CONFIG)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
def _apply_env_overrides(config: dict) -> dict:
|
||||
@@ -75,7 +83,8 @@ async def load_ai_config() -> dict:
|
||||
now = time.monotonic()
|
||||
if _cache is not None and (now - _cache_at) < _CACHE_TTL:
|
||||
return _cache
|
||||
data = await asyncio.to_thread(_read_config_sync)
|
||||
raw = await _fetch_config()
|
||||
data = _apply_env_overrides(raw)
|
||||
_cache = data
|
||||
_cache_at = now
|
||||
return data
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# doc-service — Claude context
|
||||
|
||||
PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
|
||||
PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. All file/blob storage goes through `storage-service:8020` — no files are written directly to the filesystem. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
|
||||
|
||||
---
|
||||
|
||||
@@ -23,7 +23,7 @@ features/doc-service/
|
||||
├── app/
|
||||
│ ├── main.py ← FastAPI, lifespan (file watcher start/stop)
|
||||
│ ├── database.py ← Same PostgreSQL instance as backend
|
||||
│ ├── deps.py ← get_user_id (x-user-id), get_user_groups (x-user-groups)
|
||||
│ ├── deps.py ← get_user_id, get_user_groups, get_user_is_admin, get_user_admin_groups (injected headers)
|
||||
│ ├── models/
|
||||
│ │ ├── document.py ← Document model
|
||||
│ │ ├── category.py ← DocumentCategory model
|
||||
@@ -38,13 +38,14 @@ features/doc-service/
|
||||
│ │ ├── categories.py ← Category CRUD (includes watch-owned categories)
|
||||
│ │ └── plugin.py ← GET /plugin/manifest, GET+PATCH /plugin/settings
|
||||
│ └── services/
|
||||
│ ├── storage.py ← File I/O
|
||||
│ ├── storage.py ← Storage client: save_upload/download_file/delete_file → storage-service:8020 documents bucket
|
||||
│ ├── ai_client.py ← classify_document() → ai-service:8010/chat
|
||||
│ ├── config_reader.py ← Config load/save including storage/watch settings
|
||||
│ ├── config_reader.py ← Config load/save via storage-service config bucket (doc_service_config.json)
|
||||
│ └── file_watcher.py ← watchdog-based PDF watcher + startup scan + ingestion
|
||||
├── alembic/versions/ ← Migration chain
|
||||
│ ├── 0003_add_watch_columns.py ← source, watch_path, suggested_folder, suggested_filename
|
||||
│ └── 0004_add_document_shares.py ← document_shares table (group-based sharing)
|
||||
│ ├── 0004_add_document_shares.py ← document_shares table (group-based sharing)
|
||||
│ └── 0008_rename_file_path_to_storage_key.py ← file_path → storage_key; strips /data/documents/ prefix from existing rows
|
||||
├── Dockerfile ← python:3.12-slim, non-root user 1001
|
||||
└── STATUS.md
|
||||
```
|
||||
@@ -60,7 +61,7 @@ features/doc-service/
|
||||
| `id` | String | PK, UUID | |
|
||||
| `user_id` | String | indexed | not FK — trusts x-user-id header |
|
||||
| `filename` | String | NOT NULL | |
|
||||
| `file_path` | String | NOT NULL | absolute path under /data/documents |
|
||||
| `storage_key` | String | NOT NULL | storage-service key: `{user_id}/{doc_id}.pdf` (documents bucket) |
|
||||
| `file_size` | Integer | NOT NULL | bytes |
|
||||
| `status` | String | default="pending" | pending / processing / done / failed |
|
||||
| `title` | String(500) | nullable | AI-extracted |
|
||||
@@ -78,12 +79,14 @@ features/doc-service/
|
||||
|
||||
### `document_categories`
|
||||
|
||||
| Column | Type | Constraints |
|
||||
|--------|------|-------------|
|
||||
| `id` | String | PK, UUID |
|
||||
| `user_id` | String | indexed |
|
||||
| `name` | String(128) | NOT NULL |
|
||||
| `created_at` | DateTime(tz) | server_default=now() |
|
||||
| Column | Type | Constraints | Notes |
|
||||
|--------|------|-------------|-------|
|
||||
| `id` | String | PK, UUID | |
|
||||
| `user_id` | String | indexed | owner; "watch" for system categories |
|
||||
| `name` | String(128) | NOT NULL | PascalCase-with-dashes convention enforced on create/rename |
|
||||
| `scope` | String(16) | NOT NULL, default="personal" | "personal" / "group" / "system" |
|
||||
| `group_id` | String | nullable, indexed | set when scope="group" |
|
||||
| `created_at` | DateTime(tz) | server_default=now() | |
|
||||
|
||||
### `document_category_assignments` (composite PK)
|
||||
|
||||
@@ -100,6 +103,7 @@ features/doc-service/
|
||||
| `document_id` | String | indexed, NOT NULL | not FK — trusts proxy |
|
||||
| `group_id` | String | indexed, NOT NULL | group from backend |
|
||||
| `shared_by_user_id` | String | NOT NULL | owner who shared |
|
||||
| `can_delete` | Boolean | NOT NULL, default=false | allows group members to delete the doc |
|
||||
| `created_at` | DateTime(tz) | server_default=now() | |
|
||||
|
||||
Unique constraint: `(document_id, group_id)`
|
||||
@@ -112,6 +116,10 @@ Unique constraint: `(document_id, group_id)`
|
||||
| `0002` | `add_document_title` |
|
||||
| `0003` | `add_watch_columns` |
|
||||
| `0004` | `add_document_shares` |
|
||||
| `0005` | `add_share_can_delete` |
|
||||
| `0006` | `add_category_scope` |
|
||||
| `0007` | `capitalize_system_category_names` |
|
||||
| `0008` | `rename_file_path_to_storage_key` |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -15,9 +15,9 @@ FROM python:3.12-slim
|
||||
RUN groupadd --gid 1001 appuser && \
|
||||
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
|
||||
|
||||
# Pre-create data and config dirs with correct ownership.
|
||||
# Named volumes mounted over these paths will inherit ownership on first creation.
|
||||
RUN mkdir -p /data/documents /data/watch /config && chown -R appuser:appuser /data /config
|
||||
# Pre-create watch dir with correct ownership.
|
||||
# /data/documents and /config are no longer used — all file/config storage goes through storage-service.
|
||||
RUN mkdir -p /data/watch && chown -R appuser:appuser /data
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""add can_delete to document_shares
|
||||
|
||||
Revision ID: 0005
|
||||
Revises: 0004
|
||||
Create Date: 2026-04-18
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0005"
|
||||
down_revision: Union[str, None] = "0004"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"document_shares",
|
||||
sa.Column(
|
||||
"can_delete",
|
||||
sa.Boolean(),
|
||||
nullable=False,
|
||||
server_default=sa.text("false"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("document_shares", "can_delete")
|
||||
@@ -0,0 +1,42 @@
|
||||
"""add scope and group_id to document_categories
|
||||
|
||||
Revision ID: 0006
|
||||
Revises: 0005
|
||||
Create Date: 2026-04-18
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0006"
|
||||
down_revision: Union[str, None] = "0005"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"document_categories",
|
||||
sa.Column(
|
||||
"scope",
|
||||
sa.String(16),
|
||||
nullable=False,
|
||||
server_default="personal",
|
||||
),
|
||||
)
|
||||
op.add_column(
|
||||
"document_categories",
|
||||
sa.Column("group_id", sa.String(), nullable=True),
|
||||
)
|
||||
op.create_index("ix_document_categories_group_id", "document_categories", ["group_id"])
|
||||
|
||||
# Migrate existing watch-owned categories to system scope
|
||||
op.execute("UPDATE document_categories SET scope = 'system' WHERE user_id = 'watch'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_document_categories_group_id", table_name="document_categories")
|
||||
op.drop_column("document_categories", "group_id")
|
||||
op.drop_column("document_categories", "scope")
|
||||
@@ -0,0 +1,44 @@
|
||||
"""capitalize existing system category names to PascalCase-with-dashes
|
||||
|
||||
Revision ID: 0007
|
||||
Revises: 0006
|
||||
Create Date: 2026-04-18
|
||||
|
||||
Converts names like "invoices" → "Invoices", "vendor-invoices" → "Vendor-Invoices"
|
||||
for all categories with scope='system' (watch-ingested).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import re
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
revision: str = "0007"
|
||||
down_revision: Union[str, None] = "0006"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _to_pascal(name: str) -> str:
|
||||
return "-".join(p.capitalize() for p in re.split(r"[-_\s]+", name) if p)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
rows = conn.execute(
|
||||
text("SELECT id, name FROM document_categories WHERE scope = 'system'")
|
||||
).fetchall()
|
||||
for row in rows:
|
||||
new_name = _to_pascal(row.name)
|
||||
if new_name != row.name:
|
||||
conn.execute(
|
||||
text("UPDATE document_categories SET name = :name WHERE id = :id"),
|
||||
{"name": new_name[:128], "id": row.id},
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
pass # names before migration are unknown; downgrade is a no-op
|
||||
@@ -0,0 +1,56 @@
|
||||
"""rename file_path to storage_key and strip filesystem prefix from existing rows
|
||||
|
||||
Revision ID: 0008
|
||||
Revises: 0007
|
||||
Create Date: 2026-04-20
|
||||
|
||||
Renames the documents.file_path column to storage_key.
|
||||
Existing rows have paths like '/data/documents/{user_id}/{doc_id}.pdf' or
|
||||
'/data/documents/watch/{doc_id}.pdf'. The migration strips the leading
|
||||
'/data/documents/' prefix so the value becomes a plain storage key
|
||||
(e.g. '{user_id}/{doc_id}.pdf') that the storage-service uses as the object key.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "0008"
|
||||
down_revision: Union[str, None] = "0007"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("documents") as batch_op:
|
||||
batch_op.alter_column(
|
||||
"file_path",
|
||||
new_column_name="storage_key",
|
||||
existing_type=sa.String(),
|
||||
existing_nullable=False,
|
||||
)
|
||||
|
||||
# Strip the '/data/documents/' filesystem prefix from pre-migration rows.
|
||||
op.execute(
|
||||
sa.text(
|
||||
"UPDATE documents SET storage_key = REPLACE(storage_key, '/data/documents/', '')"
|
||||
" WHERE storage_key LIKE '/data/documents/%'"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Restore the filesystem prefix so old code can still find the files.
|
||||
op.execute(
|
||||
sa.text(
|
||||
"UPDATE documents SET storage_key = '/data/documents/' || storage_key"
|
||||
" WHERE storage_key NOT LIKE '/data/documents/%'"
|
||||
)
|
||||
)
|
||||
with op.batch_alter_table("documents") as batch_op:
|
||||
batch_op.alter_column(
|
||||
"storage_key",
|
||||
new_column_name="file_path",
|
||||
existing_type=sa.String(),
|
||||
existing_nullable=False,
|
||||
)
|
||||
@@ -7,6 +7,7 @@ class Settings(BaseSettings):
|
||||
DATA_DIR: str = "/data/documents"
|
||||
CONFIG_PATH: str = "/config/doc_service_config.json"
|
||||
AI_SERVICE_URL: str = "http://ai-service:8010"
|
||||
STORAGE_SERVICE_URL: str = "http://storage-service:8020"
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
|
||||
@@ -21,3 +21,22 @@ async def get_user_groups(x_user_groups: str = Header(default="")) -> list[str]:
|
||||
if not x_user_groups:
|
||||
return []
|
||||
return [g.strip() for g in x_user_groups.split(",") if g.strip()]
|
||||
|
||||
|
||||
async def get_user_is_admin(x_user_is_admin: str = Header(default="false")) -> bool:
|
||||
"""
|
||||
Extract the superuser flag injected by the main backend proxy.
|
||||
Returns True only when the header value is exactly "true".
|
||||
"""
|
||||
return x_user_is_admin.lower() == "true"
|
||||
|
||||
|
||||
async def get_user_admin_groups(x_user_admin_groups: str = Header(default="")) -> list[str]:
|
||||
"""
|
||||
Extract the group IDs for which the current user is a group admin.
|
||||
Injected by the main backend proxy from GroupMembership.is_group_admin.
|
||||
Returns an empty list if absent or empty.
|
||||
"""
|
||||
if not x_user_admin_groups:
|
||||
return []
|
||||
return [g.strip() for g in x_user_admin_groups.split(",") if g.strip()]
|
||||
|
||||
@@ -13,6 +13,8 @@ class DocumentCategory(Base):
|
||||
id: Mapped[str] = mapped_column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
user_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
|
||||
name: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
scope: Mapped[str] = mapped_column(String(16), nullable=False, default="personal", server_default="personal")
|
||||
group_id: Mapped[str | None] = mapped_column(String, nullable=True, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), nullable=False
|
||||
)
|
||||
|
||||
@@ -13,7 +13,7 @@ class Document(Base):
|
||||
id: Mapped[str] = mapped_column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
|
||||
user_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
|
||||
filename: Mapped[str] = mapped_column(String, nullable=False)
|
||||
file_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||
storage_key: Mapped[str] = mapped_column(String, nullable=False)
|
||||
file_size: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
status: Mapped[str] = mapped_column(String, nullable=False, default="pending")
|
||||
title: Mapped[str | None] = mapped_column(String(500), nullable=True)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import DateTime, String, UniqueConstraint, func
|
||||
from sqlalchemy import Boolean, DateTime, String, UniqueConstraint, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.database import Base
|
||||
@@ -14,6 +14,7 @@ class DocumentShare(Base):
|
||||
document_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
|
||||
group_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
|
||||
shared_by_user_id: Mapped[str] = mapped_column(String, nullable=False)
|
||||
can_delete: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), nullable=False
|
||||
)
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import difflib
|
||||
import json
|
||||
import re
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
|
||||
from sqlalchemy import select
|
||||
@@ -8,7 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import or_
|
||||
|
||||
from app.database import AsyncSessionLocal, get_db
|
||||
from app.deps import get_user_id
|
||||
from app.deps import get_user_admin_groups, get_user_groups, get_user_id, get_user_is_admin
|
||||
from app.models.category import DocumentCategory
|
||||
from app.models.category_assignment import CategoryAssignment
|
||||
from app.models.document import Document
|
||||
@@ -22,14 +23,31 @@ _WATCH_USER_ID = "watch"
|
||||
|
||||
_SIMILARITY_THRESHOLD = 0.4
|
||||
|
||||
# PascalCase-with-dashes: each word starts with a capital, words joined by '-'
|
||||
# Valid: Invoices, Vendor-Invoices, Q1-Reports
|
||||
# Invalid: invoices, Invoice Reports, Invoice-reports
|
||||
_NAME_RE = re.compile(r'^[A-Z][a-zA-Z0-9]*(-[A-Z][a-zA-Z0-9]*)*$')
|
||||
|
||||
|
||||
def _validate_name(name: str) -> None:
|
||||
if not _NAME_RE.match(name):
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail=(
|
||||
"Category name must start with a capital letter. "
|
||||
"Multiple words are joined with dashes, each starting with a capital "
|
||||
"(e.g. Invoices, Vendor-Invoices, Q1-Reports)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _name_similarity(a: str, b: str) -> float:
|
||||
"""Return similarity score (0–1) between two category names."""
|
||||
a_low = a.lower()
|
||||
b_low = b.lower()
|
||||
# Word overlap is a strong signal
|
||||
a_words = set(a_low.split())
|
||||
b_words = set(b_low.split())
|
||||
a_words = set(a_low.split("-"))
|
||||
b_words = set(b_low.split("-"))
|
||||
if a_words & b_words:
|
||||
return 0.9
|
||||
# Fallback: character sequence ratio
|
||||
@@ -81,15 +99,44 @@ async def _reanalyze_documents_for_new_category(
|
||||
pass
|
||||
|
||||
|
||||
async def _check_can_manage_cat(
|
||||
cat: DocumentCategory,
|
||||
user_id: str,
|
||||
is_admin: bool,
|
||||
user_admin_groups: list[str],
|
||||
) -> None:
|
||||
"""Raise 403/404 if the current user may not rename or delete this category."""
|
||||
if is_admin:
|
||||
return # superuser can manage anything
|
||||
if cat.scope == "system":
|
||||
raise HTTPException(status_code=403, detail="Only admins can modify system categories")
|
||||
if cat.scope == "personal" and cat.user_id != user_id:
|
||||
raise HTTPException(status_code=404, detail="Category not found")
|
||||
if cat.scope == "group" and cat.group_id not in user_admin_groups:
|
||||
raise HTTPException(status_code=403, detail="Only group admins can modify group categories")
|
||||
|
||||
|
||||
@router.get("", response_model=list[CategoryOut])
|
||||
async def list_categories(
|
||||
user_id: str = Depends(get_user_id),
|
||||
user_groups: list[str] = Depends(get_user_groups),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> list[DocumentCategory]:
|
||||
# Include watch-ingested categories so they appear in the sidebar/filter
|
||||
"""
|
||||
Return all categories visible to the current user:
|
||||
- personal (owned by the user)
|
||||
- group (any group the user belongs to)
|
||||
- system (watch-ingested, scope='system')
|
||||
"""
|
||||
result = await db.execute(
|
||||
select(DocumentCategory)
|
||||
.where(or_(DocumentCategory.user_id == user_id, DocumentCategory.user_id == _WATCH_USER_ID))
|
||||
.where(
|
||||
or_(
|
||||
DocumentCategory.user_id == user_id, # personal
|
||||
DocumentCategory.group_id.in_(user_groups), # group
|
||||
DocumentCategory.scope == "system", # system
|
||||
)
|
||||
)
|
||||
.order_by(DocumentCategory.name)
|
||||
)
|
||||
return result.scalars().all()
|
||||
@@ -100,18 +147,32 @@ async def create_category(
|
||||
body: CategoryCreate,
|
||||
background_tasks: BackgroundTasks,
|
||||
user_id: str = Depends(get_user_id),
|
||||
user_groups: list[str] = Depends(get_user_groups),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> DocumentCategory:
|
||||
name = body.name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=422, detail="Category name cannot be empty")
|
||||
_validate_name(name)
|
||||
|
||||
if body.group_id:
|
||||
# User must be a member of the target group
|
||||
if body.group_id not in user_groups:
|
||||
raise HTTPException(status_code=403, detail="You are not a member of that group")
|
||||
cat = DocumentCategory(
|
||||
user_id=user_id,
|
||||
name=name[:128],
|
||||
scope="group",
|
||||
group_id=body.group_id,
|
||||
)
|
||||
else:
|
||||
cat = DocumentCategory(user_id=user_id, name=name[:128], scope="personal")
|
||||
|
||||
cat = DocumentCategory(user_id=user_id, name=name[:128])
|
||||
db.add(cat)
|
||||
await db.commit()
|
||||
await db.refresh(cat)
|
||||
|
||||
# Find existing categories with similar names
|
||||
# Find existing personal categories with similar names for background AI reanalysis
|
||||
result = await db.execute(
|
||||
select(DocumentCategory)
|
||||
.where(DocumentCategory.user_id == user_id)
|
||||
@@ -140,12 +201,18 @@ async def rename_category(
|
||||
cat_id: str,
|
||||
body: CategoryUpdate,
|
||||
user_id: str = Depends(get_user_id),
|
||||
is_admin: bool = Depends(get_user_is_admin),
|
||||
user_admin_groups: list[str] = Depends(get_user_admin_groups),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> DocumentCategory:
|
||||
cat = await _get_user_cat(cat_id, user_id, db)
|
||||
cat = await _fetch_visible_cat(cat_id, user_id, db)
|
||||
await _check_can_manage_cat(cat, user_id, is_admin, user_admin_groups)
|
||||
|
||||
name = body.name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=422, detail="Category name cannot be empty")
|
||||
_validate_name(name)
|
||||
|
||||
cat.name = name[:128]
|
||||
await db.commit()
|
||||
await db.refresh(cat)
|
||||
@@ -156,19 +223,20 @@ async def rename_category(
|
||||
async def delete_category(
|
||||
cat_id: str,
|
||||
user_id: str = Depends(get_user_id),
|
||||
is_admin: bool = Depends(get_user_is_admin),
|
||||
user_admin_groups: list[str] = Depends(get_user_admin_groups),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> None:
|
||||
cat = await _get_user_cat(cat_id, user_id, db)
|
||||
cat = await _fetch_visible_cat(cat_id, user_id, db)
|
||||
await _check_can_manage_cat(cat, user_id, is_admin, user_admin_groups)
|
||||
await db.delete(cat)
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def _get_user_cat(cat_id: str, user_id: str, db: AsyncSession) -> DocumentCategory:
|
||||
async def _fetch_visible_cat(cat_id: str, user_id: str, db: AsyncSession) -> DocumentCategory:
|
||||
"""Fetch a category that the user can see (personal/group/system). 404 if absent."""
|
||||
result = await db.execute(
|
||||
select(DocumentCategory).where(
|
||||
DocumentCategory.id == cat_id,
|
||||
DocumentCategory.user_id == user_id,
|
||||
)
|
||||
select(DocumentCategory).where(DocumentCategory.id == cat_id)
|
||||
)
|
||||
cat = result.scalar_one_or_none()
|
||||
if cat is None:
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
import asyncio
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import aiofiles
|
||||
import pdfplumber
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
@@ -13,7 +13,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from app.database import AsyncSessionLocal, get_db
|
||||
from app.deps import get_user_groups, get_user_id
|
||||
from app.deps import get_user_admin_groups, get_user_groups, get_user_id, get_user_is_admin
|
||||
from app.models.category import DocumentCategory
|
||||
from app.models.category_assignment import CategoryAssignment
|
||||
from app.models.document import Document
|
||||
@@ -29,7 +29,7 @@ from app.schemas.document import (
|
||||
from app.schemas.share import DocumentShareCreate, DocumentShareOut, SharedDocumentOut
|
||||
from app.services.ai_client import AIServiceError, classify_document
|
||||
from app.services.config_reader import load_doc_config
|
||||
from app.services.storage import delete_file, get_upload_path, save_upload
|
||||
from app.services.storage import delete_file, download_file as storage_download, save_upload
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -73,7 +73,26 @@ async def _get_share_counts(doc_ids: list[str], db: AsyncSession) -> dict[str, i
|
||||
return {row[0]: row[1] for row in rows.all()}
|
||||
|
||||
|
||||
def _doc_with_categories(doc: Document, share_count: int = 0) -> DocumentOut:
|
||||
async def _get_deletable_doc_ids(
|
||||
doc_ids: list[str], user_groups: list[str], db: AsyncSession
|
||||
) -> set[str]:
|
||||
"""Return doc IDs for which the user has delete permission via a group share."""
|
||||
if not doc_ids or not user_groups:
|
||||
return set()
|
||||
rows = await db.execute(
|
||||
select(DocumentShare.document_id)
|
||||
.where(
|
||||
DocumentShare.document_id.in_(doc_ids),
|
||||
DocumentShare.group_id.in_(user_groups),
|
||||
DocumentShare.can_delete.is_(True),
|
||||
)
|
||||
)
|
||||
return {row[0] for row in rows.all()}
|
||||
|
||||
|
||||
def _doc_with_categories(
|
||||
doc: Document, share_count: int = 0, viewer_can_delete: bool = False
|
||||
) -> DocumentOut:
|
||||
from app.schemas.document import CategoryOut
|
||||
cats = [CategoryOut(id=a.category.id, name=a.category.name) for a in doc.category_assignments]
|
||||
return DocumentOut(
|
||||
@@ -95,13 +114,14 @@ def _doc_with_categories(doc: Document, share_count: int = 0) -> DocumentOut:
|
||||
suggested_folder=doc.suggested_folder,
|
||||
suggested_filename=doc.suggested_filename,
|
||||
share_count=share_count,
|
||||
viewer_can_delete=viewer_can_delete,
|
||||
)
|
||||
|
||||
|
||||
def _extract_pdf_text(file_path: str) -> str:
|
||||
def _extract_pdf_text(pdf_bytes: bytes) -> str:
|
||||
"""Synchronous — must be called via asyncio.to_thread."""
|
||||
text_parts = []
|
||||
with pdfplumber.open(file_path) as pdf:
|
||||
with pdfplumber.open(io.BytesIO(pdf_bytes)) as pdf:
|
||||
for page in pdf.pages:
|
||||
page_text = page.extract_text()
|
||||
if page_text:
|
||||
@@ -126,7 +146,8 @@ async def process_document(doc_id: str) -> None:
|
||||
await db.commit()
|
||||
|
||||
try:
|
||||
text = await asyncio.to_thread(_extract_pdf_text, doc.file_path)
|
||||
pdf_bytes = await storage_download(doc.storage_key)
|
||||
text = await asyncio.to_thread(_extract_pdf_text, pdf_bytes)
|
||||
result = await classify_document(text)
|
||||
|
||||
doc.raw_text = text[:500_000] # cap stored text at 500k chars
|
||||
@@ -167,13 +188,13 @@ async def upload_document(
|
||||
)
|
||||
|
||||
doc_id = str(uuid.uuid4())
|
||||
dest = await save_upload(file_data, user_id, doc_id)
|
||||
storage_key = await save_upload(file_data, user_id, doc_id)
|
||||
|
||||
doc = Document(
|
||||
id=doc_id,
|
||||
user_id=user_id,
|
||||
filename=file.filename or "upload.pdf",
|
||||
file_path=str(dest),
|
||||
storage_key=storage_key,
|
||||
file_size=len(file_data),
|
||||
status="pending",
|
||||
)
|
||||
@@ -209,6 +230,8 @@ async def list_documents(
|
||||
search: str | None = Query(default=None),
|
||||
category_id: str | None = Query(default=None),
|
||||
user_id: str = Depends(get_user_id),
|
||||
user_groups: list[str] = Depends(get_user_groups),
|
||||
is_admin: bool = Depends(get_user_is_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> DocumentPage:
|
||||
sort_col = _SORT_COLUMNS.get(sort, Document.created_at)
|
||||
@@ -254,8 +277,19 @@ async def list_documents(
|
||||
)
|
||||
docs = items_result.scalars().all()
|
||||
|
||||
share_counts = await _get_share_counts([d.id for d in docs], db)
|
||||
items = [_doc_with_categories(d, share_counts.get(d.id, 0)) for d in docs]
|
||||
doc_ids = [d.id for d in docs]
|
||||
share_counts = await _get_share_counts(doc_ids, db)
|
||||
|
||||
if is_admin:
|
||||
deletable_ids = set(doc_ids)
|
||||
else:
|
||||
deletable_ids = {d.id for d in docs if d.user_id == user_id}
|
||||
deletable_ids |= await _get_deletable_doc_ids(doc_ids, user_groups, db)
|
||||
|
||||
items = [
|
||||
_doc_with_categories(d, share_counts.get(d.id, 0), viewer_can_delete=d.id in deletable_ids)
|
||||
for d in docs
|
||||
]
|
||||
|
||||
return DocumentPage(
|
||||
items=items,
|
||||
@@ -367,6 +401,7 @@ async def list_shared_with_me(
|
||||
source=doc.source,
|
||||
shared_by_user_id=share.shared_by_user_id if share else "",
|
||||
shared_via_group_id=share.group_id if share else "",
|
||||
viewer_can_delete=bool(share and share.can_delete),
|
||||
)
|
||||
)
|
||||
|
||||
@@ -382,11 +417,19 @@ async def list_shared_with_me(
|
||||
async def get_document(
|
||||
doc_id: str,
|
||||
user_id: str = Depends(get_user_id),
|
||||
user_groups: list[str] = Depends(get_user_groups),
|
||||
is_admin: bool = Depends(get_user_is_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> DocumentOut:
|
||||
doc = await _get_user_doc(doc_id, user_id, db)
|
||||
counts = await _get_share_counts([doc.id], db)
|
||||
return _doc_with_categories(doc, counts.get(doc.id, 0))
|
||||
if is_admin:
|
||||
viewer_can_delete = True
|
||||
elif doc.user_id == user_id:
|
||||
viewer_can_delete = True
|
||||
else:
|
||||
viewer_can_delete = bool(await _get_deletable_doc_ids([doc.id], user_groups, db))
|
||||
return _doc_with_categories(doc, counts.get(doc.id, 0), viewer_can_delete=viewer_can_delete)
|
||||
|
||||
|
||||
@router.get("/{doc_id}/status", response_model=DocumentStatusOut)
|
||||
@@ -479,15 +522,64 @@ async def reprocess_document(
|
||||
async def delete_document(
|
||||
doc_id: str,
|
||||
user_id: str = Depends(get_user_id),
|
||||
is_admin: bool = Depends(get_user_is_admin),
|
||||
user_groups: list[str] = Depends(get_user_groups),
|
||||
user_admin_groups: list[str] = Depends(get_user_admin_groups),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> None:
|
||||
if is_admin:
|
||||
# Admins can delete any document — fetch unconditionally.
|
||||
result = await db.execute(select(Document).where(Document.id == doc_id))
|
||||
else:
|
||||
# Fetch the document (owner, watch, or shared with user's groups)
|
||||
result = await db.execute(
|
||||
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
|
||||
select(Document).where(
|
||||
Document.id == doc_id,
|
||||
or_(
|
||||
Document.user_id == user_id,
|
||||
Document.user_id == _WATCH_USER_ID,
|
||||
Document.id.in_(
|
||||
select(DocumentShare.document_id).where(
|
||||
DocumentShare.group_id.in_(user_groups)
|
||||
)
|
||||
) if user_groups else False,
|
||||
),
|
||||
)
|
||||
)
|
||||
doc = result.scalar_one_or_none()
|
||||
if doc is None:
|
||||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
delete_file(doc.file_path)
|
||||
|
||||
is_owner = doc.user_id == user_id
|
||||
|
||||
if not is_owner and not is_admin:
|
||||
# Check: shared with a group where the user has can_delete=True
|
||||
can_delete_via_share = False
|
||||
if user_groups:
|
||||
share_result = await db.execute(
|
||||
select(DocumentShare).where(
|
||||
DocumentShare.document_id == doc_id,
|
||||
DocumentShare.group_id.in_(user_groups),
|
||||
DocumentShare.can_delete.is_(True),
|
||||
)
|
||||
)
|
||||
can_delete_via_share = share_result.scalar_one_or_none() is not None
|
||||
|
||||
# Check: user is a group admin for a group the doc is shared with
|
||||
can_delete_as_group_admin = False
|
||||
if user_admin_groups:
|
||||
admin_share_result = await db.execute(
|
||||
select(DocumentShare).where(
|
||||
DocumentShare.document_id == doc_id,
|
||||
DocumentShare.group_id.in_(user_admin_groups),
|
||||
)
|
||||
)
|
||||
can_delete_as_group_admin = admin_share_result.scalar_one_or_none() is not None
|
||||
|
||||
if not can_delete_via_share and not can_delete_as_group_admin:
|
||||
raise HTTPException(status_code=403, detail="Not allowed to delete this document")
|
||||
|
||||
await delete_file(doc.storage_key)
|
||||
await db.delete(doc)
|
||||
await db.commit()
|
||||
|
||||
@@ -518,13 +610,13 @@ async def download_file(
|
||||
if doc is None:
|
||||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
|
||||
async def file_generator():
|
||||
async with aiofiles.open(doc.file_path, "rb") as f:
|
||||
while chunk := await f.read(64 * 1024):
|
||||
yield chunk
|
||||
try:
|
||||
pdf_bytes = await storage_download(doc.storage_key)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="File not found in storage")
|
||||
|
||||
return StreamingResponse(
|
||||
file_generator(),
|
||||
iter([pdf_bytes]),
|
||||
media_type="application/pdf",
|
||||
headers={"Content-Disposition": f'inline; filename="{doc.filename}"'},
|
||||
)
|
||||
@@ -537,6 +629,7 @@ async def assign_category(
|
||||
doc_id: str,
|
||||
cat_id: str,
|
||||
user_id: str = Depends(get_user_id),
|
||||
user_groups: list[str] = Depends(get_user_groups),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> None:
|
||||
doc_result = await db.execute(
|
||||
@@ -548,9 +641,15 @@ async def assign_category(
|
||||
if doc_result.scalar_one_or_none() is None:
|
||||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
|
||||
# Accept personal, group (user is a member), or system categories
|
||||
cat_result = await db.execute(
|
||||
select(DocumentCategory).where(
|
||||
DocumentCategory.id == cat_id, DocumentCategory.user_id == user_id
|
||||
DocumentCategory.id == cat_id,
|
||||
or_(
|
||||
DocumentCategory.user_id == user_id,
|
||||
DocumentCategory.group_id.in_(user_groups) if user_groups else False,
|
||||
DocumentCategory.scope == "system",
|
||||
),
|
||||
)
|
||||
)
|
||||
if cat_result.scalar_one_or_none() is None:
|
||||
@@ -574,6 +673,13 @@ async def remove_category(
|
||||
user_id: str = Depends(get_user_id),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> None:
|
||||
# Only the document owner may remove a category assignment
|
||||
doc_result = await db.execute(
|
||||
select(Document).where(Document.id == doc_id, Document.user_id == user_id)
|
||||
)
|
||||
if doc_result.scalar_one_or_none() is None:
|
||||
raise HTTPException(status_code=403, detail="Only the document owner can remove category assignments")
|
||||
|
||||
result = await db.execute(
|
||||
select(CategoryAssignment).where(
|
||||
CategoryAssignment.document_id == doc_id,
|
||||
@@ -718,6 +824,7 @@ async def add_document_share(
|
||||
document_id=doc_id,
|
||||
group_id=body.group_id,
|
||||
shared_by_user_id=user_id,
|
||||
can_delete=body.can_delete,
|
||||
)
|
||||
db.add(share)
|
||||
await db.commit()
|
||||
|
||||
@@ -7,6 +7,8 @@ class CategoryOut(BaseModel):
|
||||
id: str
|
||||
user_id: str
|
||||
name: str
|
||||
scope: str = "personal"
|
||||
group_id: str | None = None
|
||||
created_at: datetime
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
@@ -14,6 +16,7 @@ class CategoryOut(BaseModel):
|
||||
|
||||
class CategoryCreate(BaseModel):
|
||||
name: str
|
||||
group_id: str | None = None
|
||||
|
||||
|
||||
class CategoryUpdate(BaseModel):
|
||||
|
||||
@@ -28,6 +28,7 @@ class DocumentOut(BaseModel):
|
||||
suggested_folder: str | None = None
|
||||
suggested_filename: str | None = None
|
||||
share_count: int = 0
|
||||
viewer_can_delete: bool = False
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ class DocumentShareOut(BaseModel):
|
||||
document_id: str
|
||||
group_id: str
|
||||
shared_by_user_id: str
|
||||
can_delete: bool
|
||||
created_at: datetime
|
||||
|
||||
model_config = {"from_attributes": True}
|
||||
@@ -15,6 +16,7 @@ class DocumentShareOut(BaseModel):
|
||||
|
||||
class DocumentShareCreate(BaseModel):
|
||||
group_id: str
|
||||
can_delete: bool = False
|
||||
|
||||
|
||||
class SharedDocumentOut(BaseModel):
|
||||
@@ -34,6 +36,7 @@ class SharedDocumentOut(BaseModel):
|
||||
categories: list = []
|
||||
source: str = "upload"
|
||||
share_count: int = 0
|
||||
viewer_can_delete: bool = False
|
||||
# Sharing context
|
||||
shared_by_user_id: str
|
||||
shared_via_group_id: str
|
||||
|
||||
@@ -1,19 +1,20 @@
|
||||
"""
|
||||
Reads doc_service_config.json from the shared config volume.
|
||||
Reads doc_service_config.json from the storage-service config bucket.
|
||||
30-second TTL cache + env var overrides.
|
||||
|
||||
Env var overrides (all optional):
|
||||
DOC_MAX_PDF_MB — max upload size in megabytes (e.g. "50")
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
_CONFIG_KEY = "doc_service_config.json"
|
||||
|
||||
_DEFAULT_STORAGE_CONFIG: dict = {
|
||||
"watch_enabled": False,
|
||||
"watch_path": "/data/watch",
|
||||
@@ -63,33 +64,30 @@ _cache_at: float = 0.0
|
||||
_CACHE_TTL = 30.0
|
||||
|
||||
|
||||
def _read_config_sync() -> dict:
|
||||
path = Path(settings.CONFIG_PATH)
|
||||
if not path.exists():
|
||||
base = deepcopy(_DEFAULT_CONFIG)
|
||||
else:
|
||||
with open(path) as f:
|
||||
base = json.load(f)
|
||||
return _apply_env_overrides(base)
|
||||
def _storage_url() -> str:
|
||||
return f"{settings.STORAGE_SERVICE_URL}/objects/config/{_CONFIG_KEY}"
|
||||
|
||||
|
||||
def _read_config_sync_raw() -> dict:
|
||||
"""Read without env overrides — used when we need to write back to disk."""
|
||||
path = Path(settings.CONFIG_PATH)
|
||||
if not path.exists():
|
||||
async def _fetch_config() -> dict:
|
||||
"""Fetch config from storage-service. Returns defaults if not found."""
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.get(_storage_url())
|
||||
if resp.status_code == 404:
|
||||
return deepcopy(_DEFAULT_CONFIG)
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
def _write_config_sync(config: dict) -> None:
|
||||
"""Atomically write config JSON to disk."""
|
||||
path = Path(settings.CONFIG_PATH)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
os.replace(tmp, path)
|
||||
async def _write_config(data: dict) -> None:
|
||||
import json
|
||||
payload = json.dumps(data, indent=2).encode()
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.put(
|
||||
_storage_url(),
|
||||
content=payload,
|
||||
headers={"Content-Type": "application/octet-stream"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
|
||||
def _apply_env_overrides(config: dict) -> dict:
|
||||
@@ -108,7 +106,8 @@ async def load_doc_config() -> dict:
|
||||
now = time.monotonic()
|
||||
if _cache is not None and (now - _cache_at) < _CACHE_TTL:
|
||||
return _cache
|
||||
data = await asyncio.to_thread(_read_config_sync)
|
||||
raw = await _fetch_config()
|
||||
data = _apply_env_overrides(raw)
|
||||
_cache = data
|
||||
_cache_at = now
|
||||
return data
|
||||
@@ -123,11 +122,10 @@ async def get_storage_config() -> dict:
|
||||
|
||||
|
||||
async def save_storage_config(data: dict) -> None:
|
||||
"""Merge data into the storage config block and persist to disk."""
|
||||
"""Merge data into the storage config block and persist to storage-service."""
|
||||
global _cache, _cache_at
|
||||
raw = await asyncio.to_thread(_read_config_sync_raw)
|
||||
raw = await _fetch_config()
|
||||
raw.setdefault("storage", {}).update(data)
|
||||
await asyncio.to_thread(_write_config_sync, raw)
|
||||
# Invalidate cache so next read picks up the new values
|
||||
await _write_config(raw)
|
||||
_cache = None
|
||||
_cache_at = 0.0
|
||||
|
||||
@@ -3,7 +3,7 @@ File-system watcher for the watch directory.
|
||||
|
||||
Uses the watchdog library to monitor a configured directory for new PDF files.
|
||||
When a PDF is detected, it is automatically ingested into the document service
|
||||
(copied to /data/documents, a DB record is created, and the AI pipeline runs).
|
||||
(uploaded to storage-service, a DB record is created, and the AI pipeline runs).
|
||||
|
||||
Key design decisions:
|
||||
- No-remove policy: on_deleted and on_moved events are intentionally ignored.
|
||||
@@ -11,7 +11,8 @@ Key design decisions:
|
||||
- Watch documents use user_id="watch" as a sentinel so they are visible to
|
||||
all authenticated users in the document list.
|
||||
- Subfolder names map to categories: a file at invoices/bill.pdf is assigned
|
||||
to a "invoices" category (auto-created if needed).
|
||||
to an "Invoices" category (auto-created if needed; folder name is converted
|
||||
to PascalCase-with-dashes: "vendor-invoices" → "Vendor-Invoices").
|
||||
- Suggestions: if ai_folder_suggestion or ai_rename_suggestion are enabled,
|
||||
the relevant fields are set on the document after AI processing so users
|
||||
can confirm/reject from the UI.
|
||||
@@ -21,6 +22,7 @@ Key design decisions:
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
@@ -66,7 +68,10 @@ async def ingest_file(path_str: str, watch_root: Path, config: dict) -> None:
|
||||
# Determine category from the first subfolder component
|
||||
try:
|
||||
rel = path.relative_to(watch_root)
|
||||
folder_name = rel.parts[0] if len(rel.parts) > 1 else None
|
||||
folder_name = (
|
||||
"-".join(p.capitalize() for p in re.split(r"[-_\s]+", rel.parts[0]) if p)
|
||||
if len(rel.parts) > 1 else None
|
||||
)
|
||||
except ValueError:
|
||||
folder_name = None
|
||||
|
||||
@@ -77,13 +82,13 @@ async def ingest_file(path_str: str, watch_root: Path, config: dict) -> None:
|
||||
logger.warning("[watcher] Cannot read %s: %s", path_str, exc)
|
||||
return
|
||||
|
||||
# Save a copy to /data/documents/watch/{doc_id}.pdf
|
||||
# Upload to storage-service under documents/watch/{doc_id}.pdf
|
||||
doc_id = existing.id if existing is not None else str(uuid.uuid4())
|
||||
dest = await save_upload(file_data, WATCH_USER_ID, doc_id)
|
||||
storage_key = await save_upload(file_data, WATCH_USER_ID, doc_id)
|
||||
|
||||
if existing is not None:
|
||||
# Re-ingest a previously failed document
|
||||
existing.file_path = str(dest)
|
||||
existing.storage_key = storage_key
|
||||
existing.file_size = len(file_data)
|
||||
existing.status = "pending"
|
||||
existing.error_message = None
|
||||
@@ -95,7 +100,7 @@ async def ingest_file(path_str: str, watch_root: Path, config: dict) -> None:
|
||||
source="watch",
|
||||
watch_path=path_str,
|
||||
filename=path.name,
|
||||
file_path=str(dest),
|
||||
storage_key=storage_key,
|
||||
file_size=len(file_data),
|
||||
status="pending",
|
||||
)
|
||||
|
||||
@@ -1,27 +1,61 @@
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
"""
|
||||
Storage client for the storage-service HTTP API.
|
||||
|
||||
import aiofiles
|
||||
All persistent file I/O goes through storage-service:8020.
|
||||
The bucket for all document PDFs is 'documents'.
|
||||
Keys follow the pattern:
|
||||
uploaded: {user_id}/{doc_id}.pdf
|
||||
watch-ingested: watch/{doc_id}.pdf
|
||||
"""
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def get_upload_path(user_id: str, doc_id: str) -> Path:
|
||||
"""Return /data/documents/{user_id}/{doc_id}.pdf, creating the directory if needed."""
|
||||
user_dir = Path(settings.DATA_DIR) / user_id
|
||||
user_dir.mkdir(parents=True, exist_ok=True)
|
||||
return user_dir / f"{doc_id}.pdf"
|
||||
_BUCKET = "documents"
|
||||
|
||||
|
||||
async def save_upload(file_data: bytes, user_id: str, doc_id: str) -> Path:
|
||||
dest = get_upload_path(user_id, doc_id)
|
||||
async with aiofiles.open(dest, "wb") as f:
|
||||
await f.write(file_data)
|
||||
return dest
|
||||
def _storage_url(key: str) -> str:
|
||||
return f"{settings.STORAGE_SERVICE_URL}/objects/{_BUCKET}/{key}"
|
||||
|
||||
|
||||
def delete_file(file_path: str) -> None:
|
||||
def build_storage_key(user_id: str, doc_id: str) -> str:
|
||||
"""Return the canonical storage key for a document."""
|
||||
return f"{user_id}/{doc_id}.pdf"
|
||||
|
||||
|
||||
async def save_upload(file_data: bytes, user_id: str, doc_id: str) -> str:
|
||||
"""Upload bytes to storage-service. Returns the storage key."""
|
||||
key = build_storage_key(user_id, doc_id)
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
resp = await client.put(
|
||||
_storage_url(key),
|
||||
content=file_data,
|
||||
headers={"Content-Type": "application/octet-stream"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return key
|
||||
|
||||
|
||||
async def download_file(storage_key: str) -> bytes:
|
||||
"""Download bytes from storage-service by storage key."""
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
resp = await client.get(_storage_url(storage_key))
|
||||
if resp.status_code == 404:
|
||||
raise FileNotFoundError(f"Object not found: {storage_key}")
|
||||
resp.raise_for_status()
|
||||
return resp.content
|
||||
|
||||
|
||||
async def delete_file(storage_key: str) -> None:
|
||||
"""Delete an object from storage-service. Swallows errors — deletion failure must not 500."""
|
||||
try:
|
||||
Path(file_path).unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass # log but do not raise — deletion failure must not 500
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.delete(_storage_url(storage_key))
|
||||
if resp.status_code not in (204, 404):
|
||||
logger.warning("storage-service DELETE returned %s for key %s", resp.status_code, storage_key)
|
||||
except Exception as exc:
|
||||
logger.warning("Could not delete %s from storage-service: %s", storage_key, exc)
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
# storage-service — Claude context
|
||||
|
||||
Unified file/blob storage microservice, port 8020 (internal). All services must use this service's
|
||||
HTTP API for any file persistence — no service may write to a Docker volume directly. See root
|
||||
`CLAUDE.md` for architecture, Docker, and project-wide workflows.
|
||||
|
||||
---
|
||||
|
||||
## Architecture rule (enforced)
|
||||
|
||||
**No service may write to a filesystem path for persistent data.**
|
||||
All file/blob storage must go through the storage-service HTTP API.
|
||||
Violation is a security/architecture defect.
|
||||
|
||||
---
|
||||
|
||||
## File & Folder Tree
|
||||
|
||||
```
|
||||
features/storage-service/
|
||||
├── app/
|
||||
│ ├── main.py ← FastAPI, lifespan (backend init)
|
||||
│ ├── core/config.py ← Settings (DATA_DIR, STORAGE_BACKEND, S3_*, WEBDAV_*)
|
||||
│ ├── routers/
|
||||
│ │ ├── health.py ← GET /health
|
||||
│ │ ├── objects.py ← PUT/GET/DELETE /objects/{bucket}/{key:path}, GET /objects/{bucket}
|
||||
│ │ └── migrate.py ← POST /migrate, GET /migrate/status, DELETE /migrate, PATCH /backend-config
|
||||
│ └── services/
|
||||
│ ├── backend_manager.py ← build_backend(), initialize_backend(), get_backend(), switch_backend()
|
||||
│ ├── migration.py ← run_migration(), get_status(), cancel(); KNOWN_BUCKETS
|
||||
│ └── backends/
|
||||
│ ├── base.py ← AbstractStorageBackend (ABC)
|
||||
│ ├── local.py ← LocalFSBackend — /data/storage/{bucket}/{key}
|
||||
│ ├── s3.py ← S3Backend — aiobotocore, endpoint_url configurable
|
||||
│ └── webdav.py ← WebDAVBackend — aiohttp + WebDAV PROPFIND/PUT/GET/DELETE
|
||||
├── scripts/
|
||||
│ ├── start.sh ← prod start (uvicorn port 8020)
|
||||
│ └── start_dev.sh ← dev start (uvicorn --reload)
|
||||
├── Dockerfile ← python:3.12-slim, non-root user 1001
|
||||
└── STATUS.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HTTP API
|
||||
|
||||
### Objects
|
||||
|
||||
| Method | Path | Body | Response |
|
||||
|--------|------|------|----------|
|
||||
| PUT | `/objects/{bucket}/{key:path}` | Raw bytes | 204 |
|
||||
| GET | `/objects/{bucket}/{key:path}` | — | 200 Raw bytes / 404 |
|
||||
| DELETE | `/objects/{bucket}/{key:path}` | — | 204 |
|
||||
| GET | `/objects/{bucket}` | — | `{"bucket": "...", "keys": [...]}` |
|
||||
|
||||
Keys may contain `/` (e.g. `user123/abc.pdf`). Path traversal (`..`) returns 400.
|
||||
|
||||
### Migration
|
||||
|
||||
| Method | Path | Body | Response |
|
||||
|--------|------|------|----------|
|
||||
| POST | `/migrate` | `{"driver": "s3", "config": {...}}` | 202 / 400 / 409 |
|
||||
| GET | `/migrate/status` | — | `{state, total, done, failed, errors[]}` |
|
||||
| DELETE | `/migrate` | — | 204 / 409 |
|
||||
| PATCH | `/backend-config` | `{"driver": "...", "config": {...}}` | 204 / 400 / 409 |
|
||||
|
||||
Migration states: `idle → validating → migrating → switching → cleaning → done` (or `failed`/`cancelled`)
|
||||
|
||||
### Health
|
||||
|
||||
| Method | Path | Response |
|
||||
|--------|------|----------|
|
||||
| GET | `/health` | `{"status": "ok", "backend": "local"}` |
|
||||
|
||||
---
|
||||
|
||||
## Buckets
|
||||
|
||||
| Bucket | Contents | Key format |
|
||||
|--------|----------|------------|
|
||||
| `documents` | Uploaded PDFs | `{user_id}/{doc_id}.pdf` or `watch/{doc_id}.pdf` |
|
||||
| `config` | JSON config files | `{service_name}_config.json` |
|
||||
|
||||
To add a new bucket: add it to `KNOWN_BUCKETS` in `services/migration.py` so it is included in migrations.
|
||||
|
||||
---
|
||||
|
||||
## Backend drivers
|
||||
|
||||
| Driver | Config fields | Notes |
|
||||
|--------|---------------|-------|
|
||||
| `local` | `data_dir` (optional) | Default. Files under `/data/storage/`. Zero external deps. |
|
||||
| `s3` | `endpoint_url`, `access_key`, `secret_key`, `region` | Works with MinIO, AWS S3, Backblaze B2, Cloudflare R2. Set `endpoint_url=""` for real AWS. |
|
||||
| `webdav` | `url`, `username`, `password`, `root_path` | Nextcloud: set root_path to `/remote.php/dav/files/{username}` |
|
||||
|
||||
---
|
||||
|
||||
## Adding a new backend driver
|
||||
|
||||
1. Create `app/services/backends/your_driver.py` implementing `AbstractStorageBackend`
|
||||
2. Add a branch in `build_backend()` in `backend_manager.py`
|
||||
3. Add config fields to `app/core/config.py` if env-based config is needed
|
||||
4. Document driver name + config fields in this file
|
||||
|
||||
---
|
||||
|
||||
## Default Values & Limits
|
||||
|
||||
| Parameter | Value | Location |
|
||||
|-----------|-------|----------|
|
||||
| Default backend | `local` | `STORAGE_BACKEND` env var |
|
||||
| Local data dir | `/data/storage` | `DATA_DIR` env var |
|
||||
| S3 region default | `us-east-1` | `S3_REGION` env var |
|
||||
| Migration error cap in response | 50 | `migration.py` |
|
||||
| Port | 8020 | `scripts/start.sh` |
|
||||
@@ -0,0 +1,32 @@
|
||||
# ── Stage 1: dependency installation ─────────────────────────────────────────
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN pip install --upgrade pip
|
||||
|
||||
COPY pyproject.toml .
|
||||
RUN pip install --prefix=/install .
|
||||
|
||||
# ── Stage 2: runtime ──────────────────────────────────────────────────────────
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Create non-root user (UID/GID 1001)
|
||||
RUN groupadd --gid 1001 appuser && \
|
||||
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
|
||||
|
||||
# Pre-create data dir with correct ownership.
|
||||
# Named volume mounted over this path will inherit ownership on first creation.
|
||||
RUN mkdir -p /data/storage && chown -R appuser:appuser /data
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=builder /install /usr/local
|
||||
COPY --chown=appuser:appuser app ./app
|
||||
COPY --chown=appuser:appuser scripts ./scripts
|
||||
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8020
|
||||
|
||||
CMD ["sh", "scripts/start.sh"]
|
||||
@@ -0,0 +1,83 @@
|
||||
# Storage Service — Status
|
||||
|
||||
## What it is
|
||||
Unified file/blob storage microservice, port 8020 (internal). All services store and retrieve files
|
||||
through its HTTP API — no service writes to a Docker volume directly. Uses a pluggable backend
|
||||
driver (local FS by default; S3-compatible and WebDAV available). Backend is switchable at runtime
|
||||
via admin settings with automatic data migration.
|
||||
|
||||
## Current functionality
|
||||
|
||||
### Object API (`/objects`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| PUT | `/objects/{bucket}/{key}` | Upload raw bytes |
|
||||
| GET | `/objects/{bucket}/{key}` | Download raw bytes |
|
||||
| DELETE | `/objects/{bucket}/{key}` | Delete object |
|
||||
| GET | `/objects/{bucket}` | List all keys in bucket |
|
||||
|
||||
### Migration API (`/migrate`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| POST | `/migrate` | Start migration to a new backend (validates, copies, switches, cleans) |
|
||||
| GET | `/migrate/status` | Poll migration progress |
|
||||
| DELETE | `/migrate` | Cancel in-progress migration |
|
||||
| PATCH | `/backend-config` | Reconfigure backend without migrating data |
|
||||
|
||||
### Health
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/health` | `{"status": "ok", "backend": "<driver>"}` |
|
||||
|
||||
### Buckets
|
||||
|
||||
| Bucket | Contents |
|
||||
|--------|----------|
|
||||
| `documents` | Uploaded PDFs (keyed as `{user_id}/{doc_id}.pdf` or `watch/{doc_id}.pdf`) |
|
||||
| `config` | JSON config files (replaces `app_config` volume — Phase 3) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
backend / doc-service / future-svc
|
||||
│ HTTP
|
||||
▼
|
||||
storage-service:8020
|
||||
│
|
||||
backend_manager
|
||||
│
|
||||
┌──────┴──────────────────┐
|
||||
│ │
|
||||
LocalFSBackend S3Backend / WebDAVBackend
|
||||
/data/storage/ (configured via admin UI)
|
||||
{bucket}/{key}
|
||||
```
|
||||
|
||||
Migration flow:
|
||||
```
|
||||
POST /migrate { driver, config }
|
||||
→ test_connection() (validate)
|
||||
→ list all objects in KNOWN_BUCKETS (enumerate)
|
||||
→ GET old / PUT new / exists verify (copy + verify, per object)
|
||||
→ if 0 failures: switch_backend() (atomic switch)
|
||||
→ DELETE old objects (cleanup)
|
||||
```
|
||||
|
||||
## Known limitations / not implemented
|
||||
|
||||
- Migration state is in-memory — a container restart during migration loses progress (restart restarts from scratch)
|
||||
- No presigned URL support (direct client downloads go through the API)
|
||||
- rclone backends (Google Drive, OneDrive, Dropbox) not yet implemented
|
||||
- No per-object metadata or content-type headers
|
||||
- No multipart upload for very large files (> available RAM)
|
||||
|
||||
## Future work
|
||||
|
||||
- [ ] rclone-based backend adapter (GDrive, OneDrive, Dropbox)
|
||||
- [ ] Presigned URL generation for direct browser downloads
|
||||
- [ ] Persist migration state to DB so restarts can resume
|
||||
- [ ] Streaming upload/download to avoid buffering entire file in memory
|
||||
- [ ] Per-bucket access policies
|
||||
@@ -0,0 +1,23 @@
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
|
||||
DATA_DIR: str = "/data/storage"
|
||||
STORAGE_BACKEND: str = "local" # local | s3 | webdav
|
||||
|
||||
# S3-compatible (MinIO, AWS S3, Backblaze B2, Cloudflare R2, …)
|
||||
S3_ENDPOINT_URL: str = "" # leave empty for real AWS S3
|
||||
S3_ACCESS_KEY: str = ""
|
||||
S3_SECRET_KEY: str = ""
|
||||
S3_REGION: str = "us-east-1"
|
||||
|
||||
# WebDAV (Nextcloud, …)
|
||||
WEBDAV_URL: str = ""
|
||||
WEBDAV_USERNAME: str = ""
|
||||
WEBDAV_PASSWORD: str = ""
|
||||
WEBDAV_ROOT_PATH: str = "/"
|
||||
|
||||
|
||||
settings = Settings()
|
||||
@@ -0,0 +1,29 @@
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.core.config import settings
|
||||
from app.routers import health, objects, migrate
|
||||
from app.services.backend_manager import initialize_backend
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
initialize_backend()
|
||||
logger.info("storage-service started (backend=%s)", settings.STORAGE_BACKEND)
|
||||
yield
|
||||
logger.info("storage-service shutting down")
|
||||
|
||||
|
||||
app = FastAPI(title="Storage Service", lifespan=lifespan)
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(objects.router)
|
||||
app.include_router(migrate.router)
|
||||
@@ -0,0 +1,10 @@
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.services.backend_manager import get_backend
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok", "backend": get_backend().driver_name}
|
||||
@@ -0,0 +1,88 @@
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.services import migration
|
||||
from app.services.backend_manager import build_backend, switch_backend
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MigrateRequest(BaseModel):
|
||||
driver: str
|
||||
config: dict = {}
|
||||
|
||||
|
||||
class BackendConfigRequest(BaseModel):
|
||||
driver: str
|
||||
config: dict = {}
|
||||
|
||||
|
||||
@router.post("/migrate", status_code=202)
|
||||
async def start_migration(body: MigrateRequest, background_tasks: BackgroundTasks):
|
||||
"""
|
||||
Validate the new backend, then start an async migration job that:
|
||||
1. Copies all objects from the current backend to the new one
|
||||
2. Verifies every object
|
||||
3. Atomically switches the active backend
|
||||
4. Deletes all objects from the old backend
|
||||
|
||||
Returns 409 if a migration is already in progress.
|
||||
Returns 400 if the new backend config fails validation.
|
||||
"""
|
||||
if migration.is_in_progress():
|
||||
raise HTTPException(status_code=409, detail="A migration is already in progress")
|
||||
|
||||
# Reset status and enter validating state before any async work
|
||||
migration._status.state = "validating"
|
||||
migration._status.total = 0
|
||||
migration._status.done = 0
|
||||
migration._status.failed = 0
|
||||
migration._status.errors.clear()
|
||||
|
||||
try:
|
||||
new_backend = build_backend(body.driver, body.config)
|
||||
await new_backend.test_connection()
|
||||
except Exception as exc:
|
||||
migration._status.state = "idle"
|
||||
raise HTTPException(status_code=400, detail=f"Backend validation failed: {exc}")
|
||||
|
||||
background_tasks.add_task(migration.run_migration, new_backend)
|
||||
return {"status": "started", "driver": body.driver}
|
||||
|
||||
|
||||
@router.get("/migrate/status")
|
||||
async def migration_status():
|
||||
"""Poll this to track migration progress."""
|
||||
return migration.get_status()
|
||||
|
||||
|
||||
@router.delete("/migrate", status_code=204)
|
||||
async def cancel_migration():
|
||||
"""
|
||||
Request cancellation of a running migration.
|
||||
The old backend remains active. Returns 409 if no migration is running.
|
||||
"""
|
||||
cancelled = await migration.cancel()
|
||||
if not cancelled:
|
||||
raise HTTPException(status_code=409, detail="No cancellable migration in progress")
|
||||
|
||||
|
||||
@router.patch("/backend-config", status_code=204)
|
||||
async def update_backend_config(body: BackendConfigRequest):
|
||||
"""
|
||||
Reconfigure the active backend without migrating data (e.g. update S3 credentials
|
||||
for the same endpoint, or switch back to local after a failed migration).
|
||||
|
||||
Use POST /migrate when you need data to be moved to the new backend.
|
||||
"""
|
||||
if migration.is_in_progress():
|
||||
raise HTTPException(status_code=409, detail="Cannot reconfigure while migration is in progress")
|
||||
try:
|
||||
new_backend = build_backend(body.driver, body.config)
|
||||
await new_backend.test_connection()
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Backend validation failed: {exc}")
|
||||
switch_backend(new_backend)
|
||||
@@ -0,0 +1,60 @@
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi.responses import Response
|
||||
|
||||
from app.services.backend_manager import get_backend
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _validate_key(key: str) -> str:
|
||||
"""Reject path traversal. Key may contain '/' for nested objects (e.g. user/doc.pdf)."""
|
||||
parts = key.split("/")
|
||||
if ".." in parts:
|
||||
raise HTTPException(status_code=400, detail="Invalid key: path traversal not allowed")
|
||||
return key
|
||||
|
||||
|
||||
@router.put("/objects/{bucket}/{key:path}", status_code=204)
|
||||
async def put_object(bucket: str, key: str, request: Request):
|
||||
"""Upload raw bytes. Body is read as-is (application/octet-stream)."""
|
||||
_validate_key(key)
|
||||
data = await request.body()
|
||||
try:
|
||||
await get_backend().put(bucket, key, data)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc))
|
||||
|
||||
|
||||
@router.get("/objects/{bucket}/{key:path}")
|
||||
async def get_object(bucket: str, key: str):
|
||||
"""Download raw bytes."""
|
||||
_validate_key(key)
|
||||
try:
|
||||
data = await get_backend().get(bucket, key)
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail="Object not found")
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc))
|
||||
return Response(content=data, media_type="application/octet-stream")
|
||||
|
||||
|
||||
@router.delete("/objects/{bucket}/{key:path}", status_code=204)
|
||||
async def delete_object(bucket: str, key: str):
|
||||
"""Delete an object. No-op if it does not exist."""
|
||||
_validate_key(key)
|
||||
try:
|
||||
await get_backend().delete(bucket, key)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc))
|
||||
|
||||
|
||||
@router.get("/objects/{bucket}")
|
||||
async def list_objects(bucket: str):
|
||||
"""List all keys in a bucket."""
|
||||
try:
|
||||
keys = await get_backend().list_keys(bucket)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc))
|
||||
return {"bucket": bucket, "keys": keys}
|
||||
@@ -0,0 +1,70 @@
|
||||
import logging
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.backends.base import AbstractStorageBackend
|
||||
from app.services.backends.local import LocalFSBackend
|
||||
from app.services.backends.s3 import S3Backend
|
||||
from app.services.backends.webdav import WebDAVBackend
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_active_backend: AbstractStorageBackend | None = None
|
||||
|
||||
|
||||
def build_backend(driver: str, config: dict) -> AbstractStorageBackend:
|
||||
"""Construct a backend instance from a driver name + config dict."""
|
||||
if driver == "local":
|
||||
return LocalFSBackend(data_dir=config.get("data_dir", settings.DATA_DIR))
|
||||
if driver == "s3":
|
||||
return S3Backend(
|
||||
endpoint_url=config.get("endpoint_url", ""),
|
||||
access_key=config.get("access_key", ""),
|
||||
secret_key=config.get("secret_key", ""),
|
||||
region=config.get("region", "us-east-1"),
|
||||
)
|
||||
if driver == "webdav":
|
||||
return WebDAVBackend(
|
||||
url=config.get("url", ""),
|
||||
username=config.get("username", ""),
|
||||
password=config.get("password", ""),
|
||||
root_path=config.get("root_path", "/"),
|
||||
)
|
||||
raise ValueError(f"Unknown driver: {driver!r}. Valid options: local, s3, webdav")
|
||||
|
||||
|
||||
def initialize_backend() -> None:
|
||||
"""Build the initial backend from environment variables at startup."""
|
||||
global _active_backend
|
||||
driver = settings.STORAGE_BACKEND
|
||||
config: dict = {}
|
||||
if driver == "s3":
|
||||
config = {
|
||||
"endpoint_url": settings.S3_ENDPOINT_URL,
|
||||
"access_key": settings.S3_ACCESS_KEY,
|
||||
"secret_key": settings.S3_SECRET_KEY,
|
||||
"region": settings.S3_REGION,
|
||||
}
|
||||
elif driver == "webdav":
|
||||
config = {
|
||||
"url": settings.WEBDAV_URL,
|
||||
"username": settings.WEBDAV_USERNAME,
|
||||
"password": settings.WEBDAV_PASSWORD,
|
||||
"root_path": settings.WEBDAV_ROOT_PATH,
|
||||
}
|
||||
# local needs no extra config — DATA_DIR is read from settings inside build_backend
|
||||
_active_backend = build_backend(driver, config)
|
||||
logger.info("Storage backend initialized: %s", driver)
|
||||
|
||||
|
||||
def get_backend() -> AbstractStorageBackend:
|
||||
if _active_backend is None:
|
||||
raise RuntimeError("Backend not initialized — call initialize_backend() at startup")
|
||||
return _active_backend
|
||||
|
||||
|
||||
def switch_backend(new_backend: AbstractStorageBackend) -> None:
|
||||
"""Replace the active backend. Called by the migration job after all data is verified."""
|
||||
global _active_backend
|
||||
old_name = _active_backend.driver_name if _active_backend else "none"
|
||||
_active_backend = new_backend
|
||||
logger.info("Storage backend switched: %s → %s", old_name, new_backend.driver_name)
|
||||
@@ -0,0 +1,34 @@
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
|
||||
class AbstractStorageBackend(ABC):
|
||||
"""Common interface every storage backend must implement."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def driver_name(self) -> str:
|
||||
"""Short identifier returned in /health: 'local', 's3', or 'webdav'."""
|
||||
|
||||
@abstractmethod
|
||||
async def put(self, bucket: str, key: str, data: bytes) -> None:
|
||||
"""Store *data* under bucket/key. Creates bucket/intermediate dirs as needed."""
|
||||
|
||||
@abstractmethod
|
||||
async def get(self, bucket: str, key: str) -> bytes:
|
||||
"""Return the stored bytes. Raises KeyError if the object does not exist."""
|
||||
|
||||
@abstractmethod
|
||||
async def delete(self, bucket: str, key: str) -> None:
|
||||
"""Delete the object. No-op if it does not exist."""
|
||||
|
||||
@abstractmethod
|
||||
async def list_keys(self, bucket: str) -> list[str]:
|
||||
"""Return all keys stored in *bucket*. Returns [] if bucket is empty/absent."""
|
||||
|
||||
@abstractmethod
|
||||
async def exists(self, bucket: str, key: str) -> bool:
|
||||
"""Return True if the object exists."""
|
||||
|
||||
@abstractmethod
|
||||
async def test_connection(self) -> None:
|
||||
"""Verify the backend is reachable and writable. Raise on failure."""
|
||||
@@ -0,0 +1,67 @@
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
import aiofiles
|
||||
|
||||
from .base import AbstractStorageBackend
|
||||
|
||||
|
||||
class LocalFSBackend(AbstractStorageBackend):
|
||||
"""Stores objects as files under <data_dir>/<bucket>/<key>."""
|
||||
|
||||
def __init__(self, data_dir: str) -> None:
|
||||
self._root = Path(data_dir)
|
||||
self._root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@property
|
||||
def driver_name(self) -> str:
|
||||
return "local"
|
||||
|
||||
def _resolve(self, bucket: str, key: str) -> Path:
|
||||
safe_key = key.lstrip("/")
|
||||
if ".." in safe_key.split("/"):
|
||||
raise ValueError(f"Invalid key: {key!r}")
|
||||
return self._root / bucket / safe_key
|
||||
|
||||
async def put(self, bucket: str, key: str, data: bytes) -> None:
|
||||
dest = self._resolve(bucket, key)
|
||||
await asyncio.to_thread(dest.parent.mkdir, parents=True, exist_ok=True)
|
||||
async with aiofiles.open(dest, "wb") as f:
|
||||
await f.write(data)
|
||||
|
||||
async def get(self, bucket: str, key: str) -> bytes:
|
||||
path = self._resolve(bucket, key)
|
||||
if not path.exists():
|
||||
raise KeyError(f"{bucket}/{key}")
|
||||
async with aiofiles.open(path, "rb") as f:
|
||||
return await f.read()
|
||||
|
||||
async def delete(self, bucket: str, key: str) -> None:
|
||||
path = self._resolve(bucket, key)
|
||||
try:
|
||||
await asyncio.to_thread(path.unlink, missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
async def list_keys(self, bucket: str) -> list[str]:
|
||||
bucket_dir = self._root / bucket
|
||||
if not bucket_dir.exists():
|
||||
return []
|
||||
|
||||
def _scan() -> list[str]:
|
||||
return [
|
||||
str(p.relative_to(bucket_dir))
|
||||
for p in bucket_dir.rglob("*")
|
||||
if p.is_file()
|
||||
]
|
||||
|
||||
return await asyncio.to_thread(_scan)
|
||||
|
||||
async def exists(self, bucket: str, key: str) -> bool:
|
||||
return self._resolve(bucket, key).exists()
|
||||
|
||||
async def test_connection(self) -> None:
|
||||
self._root.mkdir(parents=True, exist_ok=True)
|
||||
probe = self._root / ".health_probe"
|
||||
probe.write_bytes(b"ok")
|
||||
probe.unlink()
|
||||
@@ -0,0 +1,99 @@
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from aiobotocore.session import get_session
|
||||
|
||||
from .base import AbstractStorageBackend
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class S3Backend(AbstractStorageBackend):
|
||||
"""
|
||||
S3-compatible backend. Works with AWS S3, MinIO, Backblaze B2, Cloudflare R2, etc.
|
||||
Set endpoint_url to the service URL for non-AWS providers; leave empty for real AWS.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
endpoint_url: str,
|
||||
access_key: str,
|
||||
secret_key: str,
|
||||
region: str = "us-east-1",
|
||||
) -> None:
|
||||
self._endpoint_url = endpoint_url or None
|
||||
self._access_key = access_key
|
||||
self._secret_key = secret_key
|
||||
self._region = region
|
||||
self._session = get_session()
|
||||
|
||||
@property
|
||||
def driver_name(self) -> str:
|
||||
return "s3"
|
||||
|
||||
@asynccontextmanager
|
||||
async def _client(self):
|
||||
async with self._session.create_client(
|
||||
"s3",
|
||||
endpoint_url=self._endpoint_url,
|
||||
aws_access_key_id=self._access_key,
|
||||
aws_secret_access_key=self._secret_key,
|
||||
region_name=self._region,
|
||||
) as client:
|
||||
yield client
|
||||
|
||||
async def _ensure_bucket(self, client, bucket: str) -> None:
|
||||
try:
|
||||
await client.head_bucket(Bucket=bucket)
|
||||
except Exception:
|
||||
try:
|
||||
if self._region == "us-east-1":
|
||||
await client.create_bucket(Bucket=bucket)
|
||||
else:
|
||||
await client.create_bucket(
|
||||
Bucket=bucket,
|
||||
CreateBucketConfiguration={"LocationConstraint": self._region},
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug("Bucket create skipped (may already exist): %s", exc)
|
||||
|
||||
async def put(self, bucket: str, key: str, data: bytes) -> None:
|
||||
async with self._client() as client:
|
||||
await self._ensure_bucket(client, bucket)
|
||||
await client.put_object(Bucket=bucket, Key=key, Body=data)
|
||||
|
||||
async def get(self, bucket: str, key: str) -> bytes:
|
||||
async with self._client() as client:
|
||||
try:
|
||||
response = await client.get_object(Bucket=bucket, Key=key)
|
||||
return await response["Body"].read()
|
||||
except Exception as exc:
|
||||
raise KeyError(f"{bucket}/{key}") from exc
|
||||
|
||||
async def delete(self, bucket: str, key: str) -> None:
|
||||
async with self._client() as client:
|
||||
await client.delete_object(Bucket=bucket, Key=key)
|
||||
|
||||
async def list_keys(self, bucket: str) -> list[str]:
|
||||
async with self._client() as client:
|
||||
try:
|
||||
paginator = client.get_paginator("list_objects_v2")
|
||||
keys: list[str] = []
|
||||
async for page in paginator.paginate(Bucket=bucket):
|
||||
for obj in page.get("Contents", []):
|
||||
keys.append(obj["Key"])
|
||||
return keys
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
async def exists(self, bucket: str, key: str) -> bool:
|
||||
async with self._client() as client:
|
||||
try:
|
||||
await client.head_object(Bucket=bucket, Key=key)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
async def test_connection(self) -> None:
|
||||
async with self._client() as client:
|
||||
await client.list_buckets()
|
||||
@@ -0,0 +1,121 @@
|
||||
import base64
|
||||
from urllib.parse import quote
|
||||
|
||||
import defusedxml.ElementTree as ET
|
||||
|
||||
import aiohttp
|
||||
|
||||
from .base import AbstractStorageBackend
|
||||
|
||||
|
||||
class WebDAVBackend(AbstractStorageBackend):
|
||||
"""
|
||||
WebDAV backend. Compatible with Nextcloud and any standard WebDAV server.
|
||||
root_path should be the WebDAV root on the server, e.g. '/remote.php/dav/files/username'.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
url: str,
|
||||
username: str,
|
||||
password: str,
|
||||
root_path: str = "/",
|
||||
) -> None:
|
||||
self._base = url.rstrip("/")
|
||||
self._root = root_path.rstrip("/")
|
||||
creds = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
self._auth = f"Basic {creds}"
|
||||
|
||||
@property
|
||||
def driver_name(self) -> str:
|
||||
return "webdav"
|
||||
|
||||
def _url(self, *parts: str) -> str:
|
||||
encoded = "/".join(quote(p, safe="") for p in parts)
|
||||
return f"{self._base}{self._root}/{encoded}"
|
||||
|
||||
def _headers(self, extra: dict | None = None) -> dict[str, str]:
|
||||
h = {"Authorization": self._auth}
|
||||
if extra:
|
||||
h.update(extra)
|
||||
return h
|
||||
|
||||
async def _ensure_collection(self, session: aiohttp.ClientSession, *parts: str) -> None:
|
||||
"""MKCOL is idempotent — ignore 405 (already exists)."""
|
||||
url = self._url(*parts)
|
||||
async with session.request("MKCOL", url, headers=self._headers()) as resp:
|
||||
if resp.status not in (200, 201, 405):
|
||||
pass # best-effort; PUT will fail if directory is truly missing
|
||||
|
||||
async def put(self, bucket: str, key: str, data: bytes) -> None:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
await self._ensure_collection(session, bucket)
|
||||
parts = key.split("/")
|
||||
for i in range(1, len(parts)):
|
||||
await self._ensure_collection(session, bucket, *parts[:i])
|
||||
url = self._url(bucket, key)
|
||||
async with session.put(url, data=data, headers=self._headers()) as resp:
|
||||
if resp.status not in (200, 201, 204):
|
||||
raise OSError(f"WebDAV PUT {url} → {resp.status}")
|
||||
|
||||
async def get(self, bucket: str, key: str) -> bytes:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
url = self._url(bucket, key)
|
||||
async with session.get(url, headers=self._headers()) as resp:
|
||||
if resp.status == 404:
|
||||
raise KeyError(f"{bucket}/{key}")
|
||||
if resp.status != 200:
|
||||
raise OSError(f"WebDAV GET {url} → {resp.status}")
|
||||
return await resp.read()
|
||||
|
||||
async def delete(self, bucket: str, key: str) -> None:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
url = self._url(bucket, key)
|
||||
async with session.delete(url, headers=self._headers()) as resp:
|
||||
if resp.status not in (200, 204, 404):
|
||||
raise OSError(f"WebDAV DELETE {url} → {resp.status}")
|
||||
|
||||
async def list_keys(self, bucket: str) -> list[str]:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
url = self._url(bucket)
|
||||
headers = self._headers({"Depth": "infinity", "Content-Type": "application/xml"})
|
||||
body = '<?xml version="1.0"?><d:propfind xmlns:d="DAV:"><d:prop><d:resourcetype/></d:prop></d:propfind>'
|
||||
async with session.request("PROPFIND", url, headers=headers, data=body) as resp:
|
||||
if resp.status == 404:
|
||||
return []
|
||||
if resp.status != 207:
|
||||
return []
|
||||
xml_body = await resp.text()
|
||||
|
||||
ns = {"d": "DAV:"}
|
||||
try:
|
||||
root = ET.fromstring(xml_body)
|
||||
except ET.ParseError:
|
||||
return []
|
||||
|
||||
prefix = f"{self._base}{self._root}/{quote(bucket, safe='')}/"
|
||||
keys: list[str] = []
|
||||
for response in root.findall("d:response", ns):
|
||||
href = response.findtext("d:href", namespaces=ns) or ""
|
||||
prop = response.find(".//d:prop", ns)
|
||||
if prop is not None:
|
||||
rt = prop.find("d:resourcetype", ns)
|
||||
if rt is not None and rt.find("d:collection", ns) is not None:
|
||||
continue # skip directories
|
||||
if href.startswith(prefix):
|
||||
keys.append(href[len(prefix):])
|
||||
return keys
|
||||
|
||||
async def exists(self, bucket: str, key: str) -> bool:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
url = self._url(bucket, key)
|
||||
async with session.request("HEAD", url, headers=self._headers()) as resp:
|
||||
return resp.status == 200
|
||||
|
||||
async def test_connection(self) -> None:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
root_url = f"{self._base}{self._root}/"
|
||||
headers = self._headers({"Depth": "0"})
|
||||
async with session.request("PROPFIND", root_url, headers=headers) as resp:
|
||||
if resp.status not in (200, 207):
|
||||
raise ConnectionError(f"WebDAV root PROPFIND → {resp.status}")
|
||||
@@ -0,0 +1,139 @@
|
||||
"""
|
||||
Backend migration service.
|
||||
|
||||
Flow:
|
||||
1. POST /migrate → validate new backend (test_connection)
|
||||
2. Background task enumerates all objects in all known buckets
|
||||
3. Each object is copied old → new, then verified
|
||||
4. Only after 100 % success: atomically switch active backend
|
||||
5. Delete all objects from old backend
|
||||
6. If any copy fails: old backend stays active; state = "failed"
|
||||
7. DELETE /migrate cancels a running migration (old backend stays active)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal
|
||||
|
||||
from app.services.backends.base import AbstractStorageBackend
|
||||
from app.services.backend_manager import get_backend, switch_backend
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# All logical buckets the service knows about — enumerated during migration.
|
||||
KNOWN_BUCKETS = ["documents", "config"]
|
||||
|
||||
MigrationState = Literal[
|
||||
"idle", "validating", "migrating", "switching", "cleaning", "done", "failed", "cancelled"
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class _MigrationStatus:
|
||||
state: MigrationState = "idle"
|
||||
total: int = 0
|
||||
done: int = 0
|
||||
failed: int = 0
|
||||
errors: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
_status = _MigrationStatus()
|
||||
_cancel_requested: bool = False
|
||||
|
||||
|
||||
def get_status() -> dict:
|
||||
return {
|
||||
"state": _status.state,
|
||||
"total": _status.total,
|
||||
"done": _status.done,
|
||||
"failed": _status.failed,
|
||||
"errors": _status.errors[:50], # cap to avoid huge responses
|
||||
}
|
||||
|
||||
|
||||
def is_in_progress() -> bool:
|
||||
return _status.state in ("validating", "migrating", "switching", "cleaning")
|
||||
|
||||
|
||||
async def cancel() -> bool:
|
||||
global _cancel_requested
|
||||
if _status.state == "migrating":
|
||||
_cancel_requested = True
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def run_migration(new_backend: AbstractStorageBackend) -> None:
|
||||
"""
|
||||
Background task: copy all objects to new_backend, verify, switch, clean old.
|
||||
Called after the caller has already validated new_backend.test_connection().
|
||||
"""
|
||||
global _cancel_requested
|
||||
_cancel_requested = False
|
||||
old_backend = get_backend()
|
||||
|
||||
_status.state = "migrating"
|
||||
_status.done = 0
|
||||
_status.failed = 0
|
||||
_status.errors.clear()
|
||||
|
||||
try:
|
||||
# Collect all objects across every known bucket
|
||||
all_objects: list[tuple[str, str]] = []
|
||||
for bucket in KNOWN_BUCKETS:
|
||||
try:
|
||||
keys = await old_backend.list_keys(bucket)
|
||||
for key in keys:
|
||||
all_objects.append((bucket, key))
|
||||
except Exception as exc:
|
||||
logger.warning("Could not list bucket %r: %s", bucket, exc)
|
||||
|
||||
_status.total = len(all_objects)
|
||||
logger.info("Migration: %d objects to migrate across %d buckets", len(all_objects), len(KNOWN_BUCKETS))
|
||||
|
||||
for bucket, key in all_objects:
|
||||
if _cancel_requested:
|
||||
_status.state = "cancelled"
|
||||
logger.info("Migration cancelled (%d/%d done)", _status.done, _status.total)
|
||||
return
|
||||
|
||||
try:
|
||||
data = await old_backend.get(bucket, key)
|
||||
await new_backend.put(bucket, key, data)
|
||||
if not await new_backend.exists(bucket, key):
|
||||
raise OSError("Verification failed: object absent after PUT")
|
||||
_status.done += 1
|
||||
except Exception as exc:
|
||||
_status.failed += 1
|
||||
entry = f"{bucket}/{key}: {exc}"
|
||||
_status.errors.append(entry)
|
||||
logger.warning("Migration copy failed — %s", entry)
|
||||
|
||||
if _status.failed > 0:
|
||||
_status.state = "failed"
|
||||
logger.error(
|
||||
"Migration failed: %d/%d objects could not be copied; old backend remains active",
|
||||
_status.failed,
|
||||
_status.total,
|
||||
)
|
||||
return
|
||||
|
||||
# All objects verified — atomically switch
|
||||
_status.state = "switching"
|
||||
switch_backend(new_backend)
|
||||
|
||||
# Remove all objects from old backend (best-effort)
|
||||
_status.state = "cleaning"
|
||||
for bucket, key in all_objects:
|
||||
try:
|
||||
await old_backend.delete(bucket, key)
|
||||
except Exception as exc:
|
||||
logger.warning("Cleanup failed for %s/%s: %s", bucket, key, exc)
|
||||
|
||||
_status.state = "done"
|
||||
logger.info("Migration complete: %d objects moved to %s", _status.total, new_backend.driver_name)
|
||||
|
||||
except Exception as exc:
|
||||
_status.state = "failed"
|
||||
_status.errors.append(f"Unexpected error: {exc}")
|
||||
logger.exception("Migration aborted with unexpected error")
|
||||
@@ -0,0 +1,25 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=45"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "storage-service"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.111",
|
||||
"uvicorn[standard]>=0.29",
|
||||
"pydantic-settings>=2.2",
|
||||
"aiofiles>=23.0",
|
||||
"aiobotocore>=2.13",
|
||||
"aiohttp>=3.9",
|
||||
"defusedxml>=0.7",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"ruff>=0.4",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
echo "[storage-service] starting uvicorn..."
|
||||
exec uvicorn app.main:app --host 0.0.0.0 --port 8020
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
echo "[storage-service] starting uvicorn (dev)..."
|
||||
exec uvicorn app.main:app --host 0.0.0.0 --port 8020 --reload
|
||||
+3
-1
@@ -37,7 +37,8 @@ frontend/
|
||||
│ │ └── ui/ ← shadcn/ui components (Button, Input, …)
|
||||
│ ├── pages/ ← One file per route
|
||||
│ │ ├── DocServiceSettingsPage.tsx ← Combined doc-service settings: upload limits + watch directory
|
||||
│ │ └── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest
|
||||
│ │ ├── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest
|
||||
│ │ └── StorageAdminPage.tsx ← Admin storage backend config + live migration progress
|
||||
│ ├── lib/utils.ts ← cn() = clsx + tailwind-merge
|
||||
│ └── styles/theme.css ← CSS custom properties, Tailwind setup
|
||||
├── vite.config.ts ← /api/* proxied to backend:8000
|
||||
@@ -66,6 +67,7 @@ frontend/
|
||||
| `/admin/users` | `AdminUsersPage` | AdminRoute |
|
||||
| `/admin/groups` | `AdminGroupsPage` | AdminRoute |
|
||||
| `/admin/appearance` | `AdminAppearancePage` | AdminRoute |
|
||||
| `/admin/storage` | `StorageAdminPage` | AdminRoute |
|
||||
| `*` | redirect to `/` | — |
|
||||
|
||||
`PrivateRoute` — checks `token` from `useAuth`, redirects to `/login` if absent.
|
||||
|
||||
@@ -21,6 +21,7 @@ All API calls go through `src/api/client.ts` (single Axios instance, JWT injecte
|
||||
| `/admin` | `AdminPage` (redirects to `/admin/users`) | Admin only |
|
||||
| `/admin/users` | `AdminUsersPage` | Admin only |
|
||||
| `/admin/groups` | `AdminGroupsPage` | Admin only |
|
||||
| `/admin/storage` | `StorageAdminPage` | Admin only |
|
||||
| `/profile` | `ProfilePage` | Required |
|
||||
| `/settings` | `SettingsPage` (placeholder) | Required |
|
||||
| `/settings/plugins/:id` | `PluginSettingsPage` | Required (per-plugin access control) |
|
||||
@@ -114,6 +115,10 @@ Provider selector, per-provider fields, Test Connection, Save.
|
||||
|
||||
Upload limits + watch directory config.
|
||||
|
||||
### Admin — Storage page (`/admin/storage`)
|
||||
|
||||
Current backend status (green/red health dot). Driver selector (local/S3/WebDAV) with conditional credential fields. "Test & Migrate" button triggers an async migration that copies all objects to the new backend, verifies, then switches atomically. Live progress bar with 2s polling (states: validating → migrating → switching → cleaning → done). Cancel button during in-progress migrations.
|
||||
|
||||
### Admin — Users page (`/admin/users`)
|
||||
|
||||
User list, toggle active, create user, delete user.
|
||||
@@ -202,6 +207,7 @@ Key document-related functions:
|
||||
- [x] AI suggestion confirm/reject UI (folder + filename)
|
||||
- [x] Groups admin UI
|
||||
- [x] Replace Axios with native fetch; add global 401 → `/login` redirect for expired sessions
|
||||
- [x] Admin storage page with live migration progress bar
|
||||
- [ ] Toast notification system
|
||||
- [ ] Loading skeletons
|
||||
- [ ] Cmd+K global search (`CommandDialog`)
|
||||
|
||||
@@ -16,6 +16,7 @@ import DocServiceSettingsPage from "./pages/DocServiceSettingsPage";
|
||||
import AIAdminSettingsPage from "./pages/AIAdminSettingsPage";
|
||||
import SettingsPage from "./pages/SettingsPage";
|
||||
import PluginSettingsPage from "./pages/PluginSettingsPage";
|
||||
import StorageAdminPage from "./pages/StorageAdminPage";
|
||||
|
||||
function PrivateRoute({ children }: { children: React.ReactNode }) {
|
||||
const { token } = useAuth();
|
||||
@@ -102,6 +103,7 @@ export default function App() {
|
||||
<Route path="/admin/users" element={<AdminRoute><AdminUsersPage /></AdminRoute>} />
|
||||
<Route path="/admin/groups" element={<AdminRoute><AdminGroupsPage /></AdminRoute>} />
|
||||
<Route path="/admin/appearance" element={<AdminRoute><AdminAppearancePage /></AdminRoute>} />
|
||||
<Route path="/admin/storage" element={<AdminRoute><StorageAdminPage /></AdminRoute>} />
|
||||
|
||||
{/* Catch-all */}
|
||||
<Route path="*" element={<Navigate to="/" replace />} />
|
||||
|
||||
@@ -4,7 +4,7 @@ const BASE = "/api";
|
||||
// Core fetch wrapper
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
class ApiError extends Error {
|
||||
export class ApiError extends Error {
|
||||
status: number;
|
||||
detail: string;
|
||||
|
||||
@@ -58,8 +58,9 @@ async function request<T>(
|
||||
// Global 401 handler — expired or invalid token
|
||||
if (response.status === 401) {
|
||||
localStorage.removeItem("token");
|
||||
if (window.location.pathname !== "/login") {
|
||||
window.location.href = "/login";
|
||||
// Throw so callers don't try to process the response
|
||||
}
|
||||
throw new ApiError(401, "Session expired");
|
||||
}
|
||||
|
||||
@@ -198,6 +199,8 @@ export type DocumentStatus = "pending" | "processing" | "done" | "failed";
|
||||
export interface CategoryOut {
|
||||
id: string;
|
||||
name: string;
|
||||
scope: "personal" | "group" | "system";
|
||||
group_id: string | null;
|
||||
}
|
||||
|
||||
export interface DocumentOut {
|
||||
@@ -219,6 +222,7 @@ export interface DocumentOut {
|
||||
suggested_folder: string | null;
|
||||
suggested_filename: string | null;
|
||||
share_count: number;
|
||||
viewer_can_delete: boolean;
|
||||
}
|
||||
|
||||
export interface SharedDocumentOut extends DocumentOut {
|
||||
@@ -231,6 +235,7 @@ export interface DocumentShareOut {
|
||||
document_id: string;
|
||||
group_id: string;
|
||||
shared_by_user_id: string;
|
||||
can_delete: boolean;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
@@ -269,8 +274,8 @@ export const listSharedWithMe = (params: DocumentListParams = {}) =>
|
||||
export const getDocumentShares = (docId: string) =>
|
||||
api.get<DocumentShareOut[]>(`/documents/${docId}/shares`);
|
||||
|
||||
export const addDocumentShare = (docId: string, groupId: string) =>
|
||||
api.post<DocumentShareOut>(`/documents/${docId}/shares`, { group_id: groupId });
|
||||
export const addDocumentShare = (docId: string, groupId: string, canDelete = false) =>
|
||||
api.post<DocumentShareOut>(`/documents/${docId}/shares`, { group_id: groupId, can_delete: canDelete });
|
||||
|
||||
export const removeDocumentShare = (docId: string, groupId: string) =>
|
||||
api.delete(`/documents/${docId}/shares/${groupId}`);
|
||||
@@ -331,8 +336,8 @@ export const removeCategory = (docId: string, catId: string) =>
|
||||
|
||||
export const listCategories = () => api.get<CategoryOut[]>("/documents/categories");
|
||||
|
||||
export const createCategory = (name: string) =>
|
||||
api.post<CategoryOut>("/documents/categories", { name });
|
||||
export const createCategory = (name: string, groupId?: string) =>
|
||||
api.post<CategoryOut>("/documents/categories", { name, group_id: groupId ?? null });
|
||||
|
||||
export const renameCategory = (id: string, name: string) =>
|
||||
api.patch<CategoryOut>(`/documents/categories/${id}`, { name });
|
||||
@@ -430,6 +435,7 @@ export interface UserGroupOut {
|
||||
id: string;
|
||||
name: string;
|
||||
description: string | null;
|
||||
is_group_admin: boolean;
|
||||
}
|
||||
|
||||
export const getMyGroups = () => api.get<UserGroupOut[]>("/users/me/groups");
|
||||
@@ -452,6 +458,7 @@ export interface GroupMemberOut {
|
||||
full_name: string | null;
|
||||
is_active: boolean;
|
||||
joined_at: string;
|
||||
is_group_admin: boolean;
|
||||
}
|
||||
|
||||
export interface GroupDetailOut extends GroupOut {
|
||||
@@ -488,6 +495,9 @@ export const adminAddGroupMember = (groupId: string, userId: string) =>
|
||||
export const adminRemoveGroupMember = (groupId: string, userId: string) =>
|
||||
api.delete(`/admin/groups/${groupId}/members/${userId}`);
|
||||
|
||||
export const adminSetGroupMemberAdmin = (groupId: string, userId: string, isAdmin: boolean) =>
|
||||
api.patch(`/admin/groups/${groupId}/members/${userId}/admin`, { is_group_admin: isAdmin });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Services
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -575,6 +585,49 @@ export interface PluginManifest {
|
||||
};
|
||||
}
|
||||
|
||||
// ── Storage admin ──────────────────────────────────────────────────────────────
|
||||
|
||||
export interface StorageStatus {
|
||||
status: string;
|
||||
backend: string;
|
||||
}
|
||||
|
||||
export interface MigrationStatus {
|
||||
state:
|
||||
| "idle"
|
||||
| "validating"
|
||||
| "migrating"
|
||||
| "switching"
|
||||
| "cleaning"
|
||||
| "done"
|
||||
| "failed"
|
||||
| "cancelled";
|
||||
total: number;
|
||||
done: number;
|
||||
failed: number;
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
export interface StorageBackendConfig {
|
||||
driver: string;
|
||||
config: Record<string, string>;
|
||||
}
|
||||
|
||||
export const getStorageConfig = () => api.get<StorageStatus>("/admin/storage-config");
|
||||
|
||||
export const updateStorageConfig = (body: StorageBackendConfig) =>
|
||||
api.patch<void>("/admin/storage-config", body);
|
||||
|
||||
export const startStorageMigration = (body: StorageBackendConfig) =>
|
||||
api.post<{ status: string; driver: string }>("/admin/storage-config/migrate", body);
|
||||
|
||||
export const getMigrationStatus = () =>
|
||||
api.get<MigrationStatus>("/admin/storage-config/migrate/status");
|
||||
|
||||
export const cancelMigration = () => api.delete<void>("/admin/storage-config/migrate");
|
||||
|
||||
// ── Plugins ────────────────────────────────────────────────────────────────────
|
||||
|
||||
export const getPlugins = () => api.get<PluginOut[]>("/plugins");
|
||||
|
||||
export const getPluginManifest = (id: string) =>
|
||||
|
||||
@@ -186,6 +186,7 @@ export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }:
|
||||
const [titleValue, setTitleValue] = useState("");
|
||||
const [editingType, setEditingType] = useState(false);
|
||||
const [tagInput, setTagInput] = useState("");
|
||||
const [canDeleteNew, setCanDeleteNew] = useState(false);
|
||||
const titleInputRef = useRef<HTMLInputElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
@@ -279,8 +280,12 @@ export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }:
|
||||
});
|
||||
|
||||
const addShareMut = useMutation({
|
||||
mutationFn: (groupId: string) => addDocumentShare(doc!.id, groupId),
|
||||
onSuccess: () => queryClient.invalidateQueries({ queryKey: ["document-shares", doc!.id] }),
|
||||
mutationFn: ({ groupId, canDelete }: { groupId: string; canDelete: boolean }) =>
|
||||
addDocumentShare(doc!.id, groupId, canDelete),
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ["document-shares", doc!.id] });
|
||||
setCanDeleteNew(false);
|
||||
},
|
||||
});
|
||||
|
||||
const removeShareMut = useMutation({
|
||||
@@ -694,6 +699,11 @@ export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }:
|
||||
<div key={share.id} className="flex items-center gap-2 text-sm">
|
||||
<Users className="h-3.5 w-3.5 text-muted shrink-0" />
|
||||
<span className="flex-1 text-sm">{group?.name ?? share.group_id}</span>
|
||||
{share.can_delete && (
|
||||
<span title="Group members can delete this document">
|
||||
<Trash2 className="h-3 w-3 text-muted shrink-0" />
|
||||
</span>
|
||||
)}
|
||||
<button
|
||||
onClick={() => removeShareMut.mutate(share.group_id)}
|
||||
disabled={removeShareMut.isPending}
|
||||
@@ -706,17 +716,27 @@ export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }:
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
<label className="flex items-center gap-1.5 text-xs text-muted cursor-pointer mt-1 mb-0.5 select-none">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={canDeleteNew}
|
||||
onChange={(e) => setCanDeleteNew(e.target.checked)}
|
||||
className="h-3 w-3 accent-primary"
|
||||
/>
|
||||
Allow group members to delete
|
||||
</label>
|
||||
<GroupCombobox
|
||||
groups={myGroups}
|
||||
sharedGroupIds={sharedGroupIds}
|
||||
onShare={(id) => addShareMut.mutate(id)}
|
||||
onShare={(id) => addShareMut.mutate({ groupId: id, canDelete: canDeleteNew })}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Owner actions */}
|
||||
{isOwner && (
|
||||
{/* Owner/permitted actions */}
|
||||
{(isOwner || doc.viewer_can_delete) && (
|
||||
<div className="flex gap-2 pt-1">
|
||||
{isOwner && (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
@@ -727,6 +747,8 @@ export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }:
|
||||
<RefreshCw className={cn("h-3.5 w-3.5", reprocessMut.isPending && "animate-spin")} />
|
||||
Re-analyse
|
||||
</Button>
|
||||
)}
|
||||
{doc.viewer_can_delete && (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
@@ -741,6 +763,7 @@ export default function DocumentSlideOver({ doc, isOwner, onClose, onDeleted }:
|
||||
<Trash2 className="h-3.5 w-3.5" />
|
||||
Delete
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
|
||||
@@ -1,7 +1,15 @@
|
||||
import { useState, useRef, useEffect } from "react";
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { X, Pencil, Trash2, Check } from "lucide-react";
|
||||
import { listCategories, renameCategory, deleteCategory } from "@/api/client";
|
||||
import { X, Pencil, Trash2, Check, Lock } from "lucide-react";
|
||||
import {
|
||||
listCategories,
|
||||
renameCategory,
|
||||
deleteCategory,
|
||||
getMyGroups,
|
||||
getMe,
|
||||
type CategoryOut,
|
||||
ApiError,
|
||||
} from "@/api/client";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Button } from "@/components/ui/button";
|
||||
|
||||
@@ -11,13 +19,26 @@ interface Props {
|
||||
|
||||
export default function ManageCategoriesDialog({ onClose }: Props) {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
const { data: categories = [] } = useQuery({
|
||||
queryKey: ["categories"],
|
||||
queryFn: listCategories,
|
||||
});
|
||||
|
||||
const { data: myGroups = [] } = useQuery({
|
||||
queryKey: ["my-groups"],
|
||||
queryFn: getMyGroups,
|
||||
});
|
||||
|
||||
const { data: me } = useQuery({ queryKey: ["me"], queryFn: getMe });
|
||||
const isSuperuser = me?.is_admin ?? false;
|
||||
|
||||
// Set of group IDs for which the current user is a group admin
|
||||
const adminGroupIds = new Set(myGroups.filter((g) => g.is_group_admin).map((g) => g.id));
|
||||
|
||||
const [editingId, setEditingId] = useState<string | null>(null);
|
||||
const [editValue, setEditValue] = useState("");
|
||||
const [editError, setEditError] = useState<string | null>(null);
|
||||
const [search, setSearch] = useState("");
|
||||
const editInputRef = useRef<HTMLInputElement>(null);
|
||||
|
||||
@@ -30,6 +51,10 @@ export default function ManageCategoriesDialog({ onClose }: Props) {
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ["categories"] });
|
||||
setEditingId(null);
|
||||
setEditError(null);
|
||||
},
|
||||
onError: (err) => {
|
||||
setEditError(err instanceof ApiError ? err.message : "Rename failed");
|
||||
},
|
||||
});
|
||||
|
||||
@@ -38,24 +63,136 @@ export default function ManageCategoriesDialog({ onClose }: Props) {
|
||||
onSuccess: () => queryClient.invalidateQueries({ queryKey: ["categories"] }),
|
||||
});
|
||||
|
||||
function canManage(cat: CategoryOut): boolean {
|
||||
if (isSuperuser) return true;
|
||||
if (cat.scope === "personal") return true;
|
||||
if (cat.scope === "group") return cat.group_id != null && adminGroupIds.has(cat.group_id);
|
||||
return false; // system — non-admin cannot manage
|
||||
}
|
||||
|
||||
const filtered = search
|
||||
? categories.filter((c) => c.name.toLowerCase().includes(search.toLowerCase()))
|
||||
: categories;
|
||||
|
||||
// Group by scope
|
||||
const personal = filtered.filter((c) => c.scope === "personal");
|
||||
const system = filtered.filter((c) => c.scope === "system");
|
||||
|
||||
// Group-scoped categories grouped by group_id
|
||||
const groupCats = filtered.filter((c) => c.scope === "group");
|
||||
const groupMap = new Map<string, { name: string; cats: CategoryOut[] }>();
|
||||
for (const cat of groupCats) {
|
||||
if (!cat.group_id) continue;
|
||||
if (!groupMap.has(cat.group_id)) {
|
||||
const grp = myGroups.find((g) => g.id === cat.group_id);
|
||||
groupMap.set(cat.group_id, { name: grp?.name ?? cat.group_id, cats: [] });
|
||||
}
|
||||
groupMap.get(cat.group_id)!.cats.push(cat);
|
||||
}
|
||||
|
||||
function startEdit(id: string, name: string) {
|
||||
setEditingId(id);
|
||||
setEditValue(name);
|
||||
setEditError(null);
|
||||
}
|
||||
|
||||
function submitEdit(id: string) {
|
||||
const name = editValue.trim();
|
||||
if (name && name !== categories.find((c) => c.id === id)?.name) {
|
||||
renameMut.mutate({ id, name });
|
||||
} else {
|
||||
if (!name) return;
|
||||
if (name === categories.find((c) => c.id === id)?.name) {
|
||||
setEditingId(null);
|
||||
return;
|
||||
}
|
||||
renameMut.mutate({ id, name });
|
||||
}
|
||||
|
||||
function renderCat(cat: CategoryOut) {
|
||||
const manageable = canManage(cat);
|
||||
return (
|
||||
<div
|
||||
key={cat.id}
|
||||
className="flex items-center gap-2 py-1.5 px-2 rounded-md hover:bg-muted/10 group"
|
||||
>
|
||||
{editingId === cat.id ? (
|
||||
<div className="flex flex-col flex-1 gap-1">
|
||||
<form
|
||||
className="flex items-center gap-2"
|
||||
onSubmit={(e) => { e.preventDefault(); submitEdit(cat.id); }}
|
||||
>
|
||||
<Input
|
||||
ref={editInputRef}
|
||||
value={editValue}
|
||||
onChange={(e) => { setEditValue(e.target.value); setEditError(null); }}
|
||||
className="h-7 text-sm flex-1"
|
||||
disabled={renameMut.isPending}
|
||||
onKeyDown={(e) => { if (e.key === "Escape") { setEditingId(null); setEditError(null); } }}
|
||||
/>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={!editValue.trim() || renameMut.isPending}
|
||||
className="text-primary hover:text-primary/80 disabled:opacity-50"
|
||||
>
|
||||
<Check className="h-4 w-4" />
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => { setEditingId(null); setEditError(null); }}
|
||||
className="text-muted hover:text-foreground"
|
||||
>
|
||||
<X className="h-4 w-4" />
|
||||
</button>
|
||||
</form>
|
||||
{editError && (
|
||||
<p className="text-xs text-red-500 pl-0.5">{editError}</p>
|
||||
)}
|
||||
</div>
|
||||
) : (
|
||||
<>
|
||||
{!manageable && (
|
||||
<Lock className="h-3.5 w-3.5 text-muted flex-shrink-0" />
|
||||
)}
|
||||
<span className="flex-1 text-sm truncate">{cat.name}</span>
|
||||
{manageable && (
|
||||
<div className="flex items-center gap-1 opacity-0 group-hover:opacity-100 transition-opacity">
|
||||
<button
|
||||
onClick={() => startEdit(cat.id, cat.name)}
|
||||
className="text-muted hover:text-foreground transition-colors p-0.5"
|
||||
title="Rename"
|
||||
>
|
||||
<Pencil className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
<button
|
||||
onClick={() => {
|
||||
if (confirm(`Delete category "${cat.name}"? Documents in it will be uncategorised.`)) {
|
||||
deleteMut.mutate(cat.id);
|
||||
}
|
||||
}}
|
||||
disabled={deleteMut.isPending}
|
||||
className="text-muted hover:text-red-500 transition-colors p-0.5 disabled:opacity-50"
|
||||
title="Delete"
|
||||
>
|
||||
<Trash2 className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function renderSection(title: string, cats: CategoryOut[]) {
|
||||
if (cats.length === 0) return null;
|
||||
return (
|
||||
<div className="mb-3">
|
||||
<p className="text-xs font-semibold text-muted uppercase tracking-wider px-2 mb-1">{title}</p>
|
||||
{cats.map(renderCat)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
const hasAny = filtered.length > 0;
|
||||
|
||||
return (
|
||||
<div
|
||||
className="fixed inset-0 z-50 flex items-center justify-center bg-background/70"
|
||||
@@ -83,73 +220,17 @@ export default function ManageCategoriesDialog({ onClose }: Props) {
|
||||
)}
|
||||
|
||||
{/* List */}
|
||||
<div className="flex-1 overflow-y-auto px-5 py-3 space-y-1 min-h-0">
|
||||
{filtered.length === 0 && (
|
||||
<div className="flex-1 overflow-y-auto px-5 py-3 min-h-0">
|
||||
{!hasAny && (
|
||||
<p className="text-sm text-muted py-4 text-center">
|
||||
{search ? "No categories match" : "No categories yet"}
|
||||
</p>
|
||||
)}
|
||||
{filtered.map((cat) => (
|
||||
<div
|
||||
key={cat.id}
|
||||
className="flex items-center gap-2 py-1.5 px-2 rounded-md hover:bg-muted/10 group"
|
||||
>
|
||||
{editingId === cat.id ? (
|
||||
<form
|
||||
className="flex items-center gap-2 flex-1"
|
||||
onSubmit={(e) => { e.preventDefault(); submitEdit(cat.id); }}
|
||||
>
|
||||
<Input
|
||||
ref={editInputRef}
|
||||
value={editValue}
|
||||
onChange={(e) => setEditValue(e.target.value)}
|
||||
className="h-7 text-sm flex-1"
|
||||
disabled={renameMut.isPending}
|
||||
onKeyDown={(e) => { if (e.key === "Escape") setEditingId(null); }}
|
||||
/>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={!editValue.trim() || renameMut.isPending}
|
||||
className="text-primary hover:text-primary/80 disabled:opacity-50"
|
||||
>
|
||||
<Check className="h-4 w-4" />
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setEditingId(null)}
|
||||
className="text-muted hover:text-foreground"
|
||||
>
|
||||
<X className="h-4 w-4" />
|
||||
</button>
|
||||
</form>
|
||||
) : (
|
||||
<>
|
||||
<span className="flex-1 text-sm truncate">{cat.name}</span>
|
||||
<div className="flex items-center gap-1 opacity-0 group-hover:opacity-100 transition-opacity">
|
||||
<button
|
||||
onClick={() => startEdit(cat.id, cat.name)}
|
||||
className="text-muted hover:text-foreground transition-colors p-0.5"
|
||||
title="Rename"
|
||||
>
|
||||
<Pencil className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
<button
|
||||
onClick={() => {
|
||||
if (confirm(`Delete category "${cat.name}"? Documents in it will be uncategorised.`)) {
|
||||
deleteMut.mutate(cat.id);
|
||||
}
|
||||
}}
|
||||
disabled={deleteMut.isPending}
|
||||
className="text-muted hover:text-red-500 transition-colors p-0.5 disabled:opacity-50"
|
||||
title="Delete"
|
||||
>
|
||||
<Trash2 className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
</div>
|
||||
</>
|
||||
{renderSection("My Categories", personal)}
|
||||
{Array.from(groupMap.entries()).map(([, { name, cats }]) =>
|
||||
renderSection(`Group: ${name}`, cats)
|
||||
)}
|
||||
</div>
|
||||
))}
|
||||
{renderSection("System", system)}
|
||||
</div>
|
||||
|
||||
{/* Footer */}
|
||||
|
||||
@@ -2,11 +2,20 @@ import { useState, useRef, useEffect } from "react";
|
||||
import { useSearchParams } from "react-router-dom";
|
||||
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import { Files, User, Users, Folder, Plus, Settings2, Check, X } from "lucide-react";
|
||||
import { listCategories, createCategory, CategoryOut } from "@/api/client";
|
||||
import {
|
||||
listCategories,
|
||||
createCategory,
|
||||
getMyGroups,
|
||||
type CategoryOut,
|
||||
ApiError,
|
||||
} from "@/api/client";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { cn } from "@/lib/utils";
|
||||
import ManageCategoriesDialog from "@/components/ManageCategoriesDialog";
|
||||
|
||||
// PascalCase-with-dashes naming convention
|
||||
const NAME_RE = /^[A-Z][a-zA-Z0-9]*(-[A-Z][a-zA-Z0-9]*)*$/;
|
||||
|
||||
export default function SourcePanel() {
|
||||
const [searchParams, setSearchParams] = useSearchParams();
|
||||
const queryClient = useQueryClient();
|
||||
@@ -16,6 +25,8 @@ export default function SourcePanel() {
|
||||
const [catSearch, setCatSearch] = useState("");
|
||||
const [addingCat, setAddingCat] = useState(false);
|
||||
const [newCatName, setNewCatName] = useState("");
|
||||
const [newCatGroupId, setNewCatGroupId] = useState<string>("");
|
||||
const [nameError, setNameError] = useState<string | null>(null);
|
||||
const [manageOpen, setManageOpen] = useState(false);
|
||||
const addInputRef = useRef<HTMLInputElement>(null);
|
||||
|
||||
@@ -24,12 +35,23 @@ export default function SourcePanel() {
|
||||
queryFn: listCategories,
|
||||
});
|
||||
|
||||
const { data: myGroups = [] } = useQuery({
|
||||
queryKey: ["my-groups"],
|
||||
queryFn: getMyGroups,
|
||||
});
|
||||
|
||||
const createMut = useMutation({
|
||||
mutationFn: createCategory,
|
||||
mutationFn: ({ name, groupId }: { name: string; groupId?: string }) =>
|
||||
createCategory(name, groupId),
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ["categories"] });
|
||||
setNewCatName("");
|
||||
setNewCatGroupId("");
|
||||
setAddingCat(false);
|
||||
setNameError(null);
|
||||
},
|
||||
onError: (err) => {
|
||||
setNameError(err instanceof ApiError ? err.message : "Failed to create category");
|
||||
},
|
||||
});
|
||||
|
||||
@@ -41,6 +63,19 @@ export default function SourcePanel() {
|
||||
? categories.filter((c) => c.name.toLowerCase().includes(catSearch.toLowerCase()))
|
||||
: categories;
|
||||
|
||||
// Split by scope for display
|
||||
const personalCats = filteredCats.filter((c) => c.scope === "personal");
|
||||
const systemCats = filteredCats.filter((c) => c.scope === "system");
|
||||
const groupCatMap = new Map<string, { name: string; cats: CategoryOut[] }>();
|
||||
for (const cat of filteredCats.filter((c) => c.scope === "group")) {
|
||||
if (!cat.group_id) continue;
|
||||
if (!groupCatMap.has(cat.group_id)) {
|
||||
const grp = myGroups.find((g) => g.id === cat.group_id);
|
||||
groupCatMap.set(cat.group_id, { name: grp?.name ?? cat.group_id, cats: [] });
|
||||
}
|
||||
groupCatMap.get(cat.group_id)!.cats.push(cat);
|
||||
}
|
||||
|
||||
function setView(view: string) {
|
||||
setSearchParams((prev) => {
|
||||
const next = new URLSearchParams(prev);
|
||||
@@ -61,6 +96,20 @@ export default function SourcePanel() {
|
||||
});
|
||||
}
|
||||
|
||||
function handleAddSubmit(e: React.FormEvent) {
|
||||
e.preventDefault();
|
||||
const name = newCatName.trim();
|
||||
if (!name) return;
|
||||
if (!NAME_RE.test(name)) {
|
||||
setNameError(
|
||||
"Must start with a capital letter. Join multiple words with dashes, each capitalised (e.g. Vendor-Invoices)."
|
||||
);
|
||||
return;
|
||||
}
|
||||
setNameError(null);
|
||||
createMut.mutate({ name, groupId: newCatGroupId || undefined });
|
||||
}
|
||||
|
||||
const viewItemClass = (active: boolean) =>
|
||||
cn(
|
||||
"flex items-center gap-2 w-full px-2 py-1.5 rounded-md text-sm transition-colors",
|
||||
@@ -77,6 +126,32 @@ export default function SourcePanel() {
|
||||
: "text-muted hover:bg-muted/20 hover:text-foreground"
|
||||
);
|
||||
|
||||
function renderCatSection(label: string, cats: CategoryOut[]) {
|
||||
if (cats.length === 0) return null;
|
||||
return (
|
||||
<div className="mb-1">
|
||||
<p className="text-[10px] font-semibold text-muted uppercase tracking-wider px-2 pt-1.5 pb-0.5">
|
||||
{label}
|
||||
</p>
|
||||
{cats.map((cat) => (
|
||||
<button
|
||||
key={cat.id}
|
||||
className={catItemClass(currentCategoryId === cat.id)}
|
||||
onClick={() => selectCategory(cat)}
|
||||
>
|
||||
<Folder className="h-3.5 w-3.5 shrink-0" />
|
||||
<span className="truncate">{cat.name}</span>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
const hasSections =
|
||||
personalCats.length > 0 ||
|
||||
systemCats.length > 0 ||
|
||||
groupCatMap.size > 0;
|
||||
|
||||
return (
|
||||
<>
|
||||
<aside className="w-56 flex flex-col border-r border-border bg-surface shrink-0 h-screen">
|
||||
@@ -132,18 +207,13 @@ export default function SourcePanel() {
|
||||
/>
|
||||
)}
|
||||
|
||||
<div className="flex-1 overflow-y-auto space-y-0.5 min-h-0">
|
||||
{filteredCats.map((cat) => (
|
||||
<button
|
||||
key={cat.id}
|
||||
className={catItemClass(currentCategoryId === cat.id)}
|
||||
onClick={() => selectCategory(cat)}
|
||||
>
|
||||
<Folder className="h-3.5 w-3.5 shrink-0" />
|
||||
<span className="truncate">{cat.name}</span>
|
||||
</button>
|
||||
))}
|
||||
{filteredCats.length === 0 && catSearch && (
|
||||
<div className="flex-1 overflow-y-auto min-h-0">
|
||||
{renderCatSection("Mine", personalCats)}
|
||||
{Array.from(groupCatMap.entries()).map(([, { name, cats }]) =>
|
||||
renderCatSection(name, cats)
|
||||
)}
|
||||
{renderCatSection("System", systemCats)}
|
||||
{!hasSections && catSearch && (
|
||||
<p className="text-xs text-muted px-2 py-1">No categories match</p>
|
||||
)}
|
||||
{categories.length === 0 && !catSearch && (
|
||||
@@ -154,18 +224,13 @@ export default function SourcePanel() {
|
||||
{/* Add new category */}
|
||||
<div className="pt-2 border-t border-border mt-2">
|
||||
{addingCat ? (
|
||||
<form
|
||||
onSubmit={(e) => {
|
||||
e.preventDefault();
|
||||
if (newCatName.trim()) createMut.mutate(newCatName.trim());
|
||||
}}
|
||||
className="flex items-center gap-1"
|
||||
>
|
||||
<form onSubmit={handleAddSubmit} className="flex flex-col gap-1">
|
||||
<div className="flex items-center gap-1">
|
||||
<Input
|
||||
ref={addInputRef}
|
||||
value={newCatName}
|
||||
onChange={(e) => setNewCatName(e.target.value)}
|
||||
placeholder="Category name"
|
||||
onChange={(e) => { setNewCatName(e.target.value); setNameError(null); }}
|
||||
placeholder="Vendor-Invoices"
|
||||
className="h-7 text-xs flex-1"
|
||||
disabled={createMut.isPending}
|
||||
/>
|
||||
@@ -178,11 +243,33 @@ export default function SourcePanel() {
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => { setAddingCat(false); setNewCatName(""); }}
|
||||
onClick={() => {
|
||||
setAddingCat(false);
|
||||
setNewCatName("");
|
||||
setNewCatGroupId("");
|
||||
setNameError(null);
|
||||
}}
|
||||
className="text-muted hover:text-foreground"
|
||||
>
|
||||
<X className="h-4 w-4" />
|
||||
</button>
|
||||
</div>
|
||||
{myGroups.length > 0 && (
|
||||
<select
|
||||
value={newCatGroupId}
|
||||
onChange={(e) => setNewCatGroupId(e.target.value)}
|
||||
className="h-7 text-xs rounded border border-border bg-surface text-foreground px-1"
|
||||
disabled={createMut.isPending}
|
||||
>
|
||||
<option value="">Personal</option>
|
||||
{myGroups.map((g) => (
|
||||
<option key={g.id} value={g.id}>{g.name}</option>
|
||||
))}
|
||||
</select>
|
||||
)}
|
||||
{nameError && (
|
||||
<p className="text-[10px] text-red-500 leading-tight">{nameError}</p>
|
||||
)}
|
||||
</form>
|
||||
) : (
|
||||
<button
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
adminUpdateGroup,
|
||||
adminAddGroupMember,
|
||||
adminRemoveGroupMember,
|
||||
adminSetGroupMemberAdmin,
|
||||
adminGetUsers,
|
||||
type GroupOut,
|
||||
type GroupCreate,
|
||||
@@ -249,6 +250,14 @@ function GroupMembersPanel({ groupId }: { groupId: string }) {
|
||||
},
|
||||
});
|
||||
|
||||
const adminMutation = useMutation({
|
||||
mutationFn: ({ uId, isAdmin }: { uId: string; isAdmin: boolean }) =>
|
||||
adminSetGroupMemberAdmin(groupId, uId, isAdmin),
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ["admin-group", groupId] });
|
||||
},
|
||||
});
|
||||
|
||||
if (isLoading) return <p style={{ padding: "8px 16px" }}>Loading members…</p>;
|
||||
if (!group) return null;
|
||||
|
||||
@@ -268,6 +277,7 @@ function GroupMembersPanel({ groupId }: { groupId: string }) {
|
||||
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Email</th>
|
||||
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Name</th>
|
||||
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Status</th>
|
||||
<th style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>Group Admin</th>
|
||||
<th style={{ padding: "4px 0", fontSize: 13 }}></th>
|
||||
</tr>
|
||||
</thead>
|
||||
@@ -279,6 +289,15 @@ function GroupMembersPanel({ groupId }: { groupId: string }) {
|
||||
<td style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>
|
||||
{m.is_active ? "Active" : "Inactive"}
|
||||
</td>
|
||||
<td style={{ padding: "4px 12px 4px 0", fontSize: 13 }}>
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={m.is_group_admin}
|
||||
disabled={adminMutation.isPending}
|
||||
onChange={() => adminMutation.mutate({ uId: m.id, isAdmin: !m.is_group_admin })}
|
||||
title={m.is_group_admin ? "Remove group admin" : "Make group admin"}
|
||||
/>
|
||||
</td>
|
||||
<td style={{ padding: "4px 0" }}>
|
||||
<button
|
||||
style={{ fontSize: 12, color: "red" }}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { useState, useRef, useCallback, useEffect } from "react";
|
||||
import { createPortal } from "react-dom";
|
||||
import { useSearchParams } from "react-router-dom";
|
||||
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import {
|
||||
@@ -47,43 +48,70 @@ function formatDate(iso: string) {
|
||||
// ── Row actions dropdown ──────────────────────────────────────────────────────
|
||||
|
||||
function RowActionsMenu({
|
||||
doc, isOwner, onSelect,
|
||||
}: { doc: DocumentOut; isOwner: boolean; onSelect: () => void }) {
|
||||
doc, onSelect,
|
||||
}: { doc: DocumentOut; onSelect: () => void }) {
|
||||
const [open, setOpen] = useState(false);
|
||||
const ref = useRef<HTMLDivElement>(null);
|
||||
const [menuPos, setMenuPos] = useState({ top: 0, right: 0 });
|
||||
const triggerRef = useRef<HTMLButtonElement>(null);
|
||||
const menuRef = useRef<HTMLDivElement>(null);
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
useEffect(() => {
|
||||
const handler = (e: MouseEvent) => {
|
||||
if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false);
|
||||
const close = (e: MouseEvent) => {
|
||||
if (
|
||||
triggerRef.current && !triggerRef.current.contains(e.target as Node) &&
|
||||
menuRef.current && !menuRef.current.contains(e.target as Node)
|
||||
) setOpen(false);
|
||||
};
|
||||
const closeOnScroll = () => setOpen(false);
|
||||
document.addEventListener("mousedown", close);
|
||||
window.addEventListener("scroll", closeOnScroll, true);
|
||||
window.addEventListener("resize", closeOnScroll);
|
||||
return () => {
|
||||
document.removeEventListener("mousedown", close);
|
||||
window.removeEventListener("scroll", closeOnScroll, true);
|
||||
window.removeEventListener("resize", closeOnScroll);
|
||||
};
|
||||
document.addEventListener("mousedown", handler);
|
||||
return () => document.removeEventListener("mousedown", handler);
|
||||
}, []);
|
||||
|
||||
const deleteMut = useMutation({
|
||||
mutationFn: () => deleteDocument(doc.id),
|
||||
onSuccess: () => queryClient.invalidateQueries({ queryKey: ["documents"] }),
|
||||
onError: () => alert("Failed to delete document. Please try again."),
|
||||
});
|
||||
|
||||
const handleToggle = (e: React.MouseEvent) => {
|
||||
e.stopPropagation();
|
||||
if (!open && triggerRef.current) {
|
||||
const rect = triggerRef.current.getBoundingClientRect();
|
||||
setMenuPos({ top: rect.bottom + 4, right: window.innerWidth - rect.right });
|
||||
}
|
||||
setOpen((o) => !o);
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="relative" ref={ref} onClick={(e) => e.stopPropagation()}>
|
||||
<div onClick={(e) => e.stopPropagation()}>
|
||||
<button
|
||||
onClick={() => setOpen((o) => !o)}
|
||||
ref={triggerRef}
|
||||
onClick={handleToggle}
|
||||
className="p-1 rounded text-muted hover:text-foreground hover:bg-muted/20 transition-colors opacity-0 group-hover:opacity-100"
|
||||
title="Actions"
|
||||
>
|
||||
<MoreHorizontal className="h-4 w-4" />
|
||||
</button>
|
||||
{open && (
|
||||
<div className="absolute right-0 top-full mt-1 z-20 bg-surface border border-border rounded-lg shadow-lg w-36 py-1">
|
||||
{open && createPortal(
|
||||
<div
|
||||
ref={menuRef}
|
||||
style={{ position: "fixed", top: menuPos.top, right: menuPos.right, zIndex: 9999 }}
|
||||
className="bg-surface border border-border rounded-lg shadow-lg w-36 py-1"
|
||||
>
|
||||
<button
|
||||
className="w-full text-left px-3 py-1.5 text-sm hover:bg-muted/20 transition-colors"
|
||||
onClick={() => { onSelect(); setOpen(false); }}
|
||||
>
|
||||
Open details
|
||||
</button>
|
||||
{isOwner && (
|
||||
{doc.viewer_can_delete && (
|
||||
<button
|
||||
className="w-full text-left px-3 py-1.5 text-sm hover:bg-muted/20 transition-colors text-red-500"
|
||||
onClick={() => {
|
||||
@@ -94,7 +122,8 @@ function RowActionsMenu({
|
||||
Delete
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</div>,
|
||||
document.body
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
@@ -470,7 +499,7 @@ function DocumentRow({
|
||||
</td>
|
||||
|
||||
<td className="px-3 py-2.5 w-8 text-right">
|
||||
<RowActionsMenu doc={doc} isOwner={isOwner} onSelect={onOpen} />
|
||||
<RowActionsMenu doc={doc} onSelect={onOpen} />
|
||||
</td>
|
||||
</tr>
|
||||
);
|
||||
|
||||
@@ -0,0 +1,381 @@
|
||||
import { useState } from "react";
|
||||
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import {
|
||||
getStorageConfig,
|
||||
getMigrationStatus,
|
||||
startStorageMigration,
|
||||
cancelMigration,
|
||||
type StorageBackendConfig,
|
||||
type MigrationStatus,
|
||||
} from "../api/client";
|
||||
|
||||
type Driver = "local" | "s3" | "webdav";
|
||||
|
||||
function Section({ title, children }: { title: string; children: React.ReactNode }) {
|
||||
return (
|
||||
<section style={{ marginBottom: 36 }}>
|
||||
<h2 style={{ fontSize: 18, marginBottom: 16 }}>{title}</h2>
|
||||
{children}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
function Field({ label, children }: { label: string; children: React.ReactNode }) {
|
||||
return (
|
||||
<div style={{ marginBottom: 12 }}>
|
||||
<label style={{ display: "block", fontSize: 13, marginBottom: 4, color: "var(--color-text-muted)" }}>
|
||||
{label}
|
||||
</label>
|
||||
{children}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function inputStyle(disabled = false): React.CSSProperties {
|
||||
return {
|
||||
width: "100%",
|
||||
padding: "6px 10px",
|
||||
border: "1px solid var(--color-border)",
|
||||
borderRadius: 6,
|
||||
fontSize: 14,
|
||||
background: disabled ? "var(--color-surface)" : "var(--color-background)",
|
||||
color: "var(--color-text-primary)",
|
||||
opacity: disabled ? 0.7 : 1,
|
||||
};
|
||||
}
|
||||
|
||||
function MigrationProgressBar({ status }: { status: MigrationStatus }) {
|
||||
const pct = status.total > 0 ? Math.round((status.done / status.total) * 100) : 0;
|
||||
const isBusy = ["validating", "migrating", "switching", "cleaning"].includes(status.state);
|
||||
|
||||
const stateLabel: Record<string, string> = {
|
||||
idle: "Idle",
|
||||
validating: "Validating new backend…",
|
||||
migrating: `Migrating — ${status.done} / ${status.total} objects`,
|
||||
switching: "Switching active backend…",
|
||||
cleaning: "Cleaning up old backend…",
|
||||
done: "Migration complete",
|
||||
failed: "Migration failed",
|
||||
cancelled: "Migration cancelled",
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={{ marginTop: 12 }}>
|
||||
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: 4, fontSize: 13 }}>
|
||||
<span style={{ color: status.state === "failed" ? "#dc2626" : "var(--color-text-primary)" }}>
|
||||
{stateLabel[status.state] ?? status.state}
|
||||
</span>
|
||||
{isBusy && <span style={{ color: "var(--color-text-muted)" }}>{pct}%</span>}
|
||||
</div>
|
||||
{(isBusy || status.state === "done") && (
|
||||
<div
|
||||
style={{
|
||||
height: 8,
|
||||
background: "var(--color-border)",
|
||||
borderRadius: 4,
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
height: "100%",
|
||||
width: status.state === "done" ? "100%" : `${pct}%`,
|
||||
background: status.state === "done" ? "#16a34a" : "var(--color-primary)",
|
||||
borderRadius: 4,
|
||||
transition: "width 0.3s",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
{status.errors.length > 0 && (
|
||||
<div
|
||||
style={{
|
||||
marginTop: 8,
|
||||
padding: "8px 10px",
|
||||
background: "#fef2f2",
|
||||
borderRadius: 6,
|
||||
fontSize: 12,
|
||||
color: "#991b1b",
|
||||
maxHeight: 120,
|
||||
overflowY: "auto",
|
||||
}}
|
||||
>
|
||||
{status.errors.slice(0, 10).map((e, i) => (
|
||||
<div key={i} style={{ marginBottom: 2 }}>
|
||||
{e}
|
||||
</div>
|
||||
))}
|
||||
{status.errors.length > 10 && (
|
||||
<div style={{ opacity: 0.7 }}>…and {status.errors.length - 10} more</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function StorageAdminPage() {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
const { data: storageStatus } = useQuery({
|
||||
queryKey: ["storage-config"],
|
||||
queryFn: getStorageConfig,
|
||||
refetchInterval: 10_000,
|
||||
});
|
||||
|
||||
const { data: migStatus, refetch: refetchMig } = useQuery({
|
||||
queryKey: ["migration-status"],
|
||||
queryFn: getMigrationStatus,
|
||||
refetchInterval: (query) => {
|
||||
const state = query.state.data?.state;
|
||||
return state && ["validating", "migrating", "switching", "cleaning"].includes(state)
|
||||
? 2_000
|
||||
: false;
|
||||
},
|
||||
});
|
||||
|
||||
const isMigrating =
|
||||
migStatus &&
|
||||
["validating", "migrating", "switching", "cleaning"].includes(migStatus.state);
|
||||
|
||||
// ── New backend form ─────────────────────────────────────────────────────────
|
||||
const [driver, setDriver] = useState<Driver>("local");
|
||||
const [s3EndpointUrl, setS3EndpointUrl] = useState("");
|
||||
const [s3AccessKey, setS3AccessKey] = useState("");
|
||||
const [s3SecretKey, setS3SecretKey] = useState("");
|
||||
const [s3Region, setS3Region] = useState("us-east-1");
|
||||
const [webdavUrl, setWebdavUrl] = useState("");
|
||||
const [webdavUsername, setWebdavUsername] = useState("");
|
||||
const [webdavPassword, setWebdavPassword] = useState("");
|
||||
const [webdavRootPath, setWebdavRootPath] = useState("/");
|
||||
const [error, setError] = useState("");
|
||||
|
||||
function buildConfig(): StorageBackendConfig {
|
||||
if (driver === "s3") {
|
||||
return {
|
||||
driver,
|
||||
config: {
|
||||
endpoint_url: s3EndpointUrl,
|
||||
access_key: s3AccessKey,
|
||||
secret_key: s3SecretKey,
|
||||
region: s3Region,
|
||||
},
|
||||
};
|
||||
}
|
||||
if (driver === "webdav") {
|
||||
return {
|
||||
driver,
|
||||
config: {
|
||||
url: webdavUrl,
|
||||
username: webdavUsername,
|
||||
password: webdavPassword,
|
||||
root_path: webdavRootPath,
|
||||
},
|
||||
};
|
||||
}
|
||||
return { driver: "local", config: {} };
|
||||
}
|
||||
|
||||
const migrateMutation = useMutation({
|
||||
mutationFn: startStorageMigration,
|
||||
onSuccess: () => {
|
||||
setError("");
|
||||
refetchMig();
|
||||
},
|
||||
onError: (e: Error) => setError(e.message),
|
||||
});
|
||||
|
||||
const cancelMutation = useMutation({
|
||||
mutationFn: cancelMigration,
|
||||
onSuccess: () => {
|
||||
queryClient.invalidateQueries({ queryKey: ["migration-status"] });
|
||||
queryClient.invalidateQueries({ queryKey: ["storage-config"] });
|
||||
},
|
||||
onError: (e: Error) => setError(e.message),
|
||||
});
|
||||
|
||||
const currentDriver = storageStatus?.backend ?? "—";
|
||||
|
||||
return (
|
||||
<div style={{ maxWidth: 680, margin: "0 auto", padding: "32px 16px" }}>
|
||||
<h1 style={{ fontSize: 24, marginBottom: 4 }}>Storage</h1>
|
||||
<p style={{ color: "var(--color-text-muted)", marginBottom: 32, fontSize: 14 }}>
|
||||
All uploaded files are stored through the storage-service. Switch between local filesystem,
|
||||
S3-compatible cloud storage, or WebDAV (Nextcloud).
|
||||
</p>
|
||||
|
||||
<Section title="Current backend">
|
||||
<div
|
||||
style={{
|
||||
display: "inline-flex",
|
||||
alignItems: "center",
|
||||
gap: 8,
|
||||
padding: "6px 12px",
|
||||
background: "var(--color-surface)",
|
||||
border: "1px solid var(--color-border)",
|
||||
borderRadius: 6,
|
||||
fontSize: 14,
|
||||
}}
|
||||
>
|
||||
<span
|
||||
style={{
|
||||
width: 8,
|
||||
height: 8,
|
||||
borderRadius: "50%",
|
||||
background: storageStatus?.status === "ok" ? "#16a34a" : "#dc2626",
|
||||
flexShrink: 0,
|
||||
}}
|
||||
/>
|
||||
<strong>{currentDriver}</strong>
|
||||
{storageStatus?.status === "ok" ? " — healthy" : " — unreachable"}
|
||||
</div>
|
||||
</Section>
|
||||
|
||||
<Section title="Switch backend">
|
||||
<p style={{ fontSize: 13, color: "var(--color-text-muted)", marginBottom: 16 }}>
|
||||
When you click <strong>Test & Migrate</strong>, all existing files will be copied to the
|
||||
new backend, verified, and the switch will happen only after every file is confirmed. The
|
||||
old backend is cleaned up automatically.
|
||||
</p>
|
||||
|
||||
<Field label="Backend driver">
|
||||
<select
|
||||
value={driver}
|
||||
onChange={(e) => setDriver(e.target.value as Driver)}
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
>
|
||||
<option value="local">Local filesystem (default)</option>
|
||||
<option value="s3">S3-compatible (MinIO / AWS S3 / Backblaze / Cloudflare R2)</option>
|
||||
<option value="webdav">WebDAV (Nextcloud / …)</option>
|
||||
</select>
|
||||
</Field>
|
||||
|
||||
{driver === "s3" && (
|
||||
<>
|
||||
<Field label="Endpoint URL (leave empty for real AWS S3)">
|
||||
<input
|
||||
value={s3EndpointUrl}
|
||||
onChange={(e) => setS3EndpointUrl(e.target.value)}
|
||||
placeholder="http://minio:9000"
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="Access key">
|
||||
<input
|
||||
value={s3AccessKey}
|
||||
onChange={(e) => setS3AccessKey(e.target.value)}
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="Secret key">
|
||||
<input
|
||||
type="password"
|
||||
value={s3SecretKey}
|
||||
onChange={(e) => setS3SecretKey(e.target.value)}
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="Region">
|
||||
<input
|
||||
value={s3Region}
|
||||
onChange={(e) => setS3Region(e.target.value)}
|
||||
placeholder="us-east-1"
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
</>
|
||||
)}
|
||||
|
||||
{driver === "webdav" && (
|
||||
<>
|
||||
<Field label="Server URL">
|
||||
<input
|
||||
value={webdavUrl}
|
||||
onChange={(e) => setWebdavUrl(e.target.value)}
|
||||
placeholder="https://nextcloud.example.com"
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="Username">
|
||||
<input
|
||||
value={webdavUsername}
|
||||
onChange={(e) => setWebdavUsername(e.target.value)}
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="Password">
|
||||
<input
|
||||
type="password"
|
||||
value={webdavPassword}
|
||||
onChange={(e) => setWebdavPassword(e.target.value)}
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="WebDAV root path">
|
||||
<input
|
||||
value={webdavRootPath}
|
||||
onChange={(e) => setWebdavRootPath(e.target.value)}
|
||||
placeholder="/remote.php/dav/files/username"
|
||||
disabled={!!isMigrating}
|
||||
style={inputStyle(!!isMigrating)}
|
||||
/>
|
||||
</Field>
|
||||
</>
|
||||
)}
|
||||
|
||||
{error && (
|
||||
<p style={{ color: "#dc2626", fontSize: 13, marginBottom: 8 }}>{error}</p>
|
||||
)}
|
||||
|
||||
<div style={{ display: "flex", gap: 10, marginTop: 16 }}>
|
||||
<button
|
||||
onClick={() => migrateMutation.mutate(buildConfig())}
|
||||
disabled={!!isMigrating || migrateMutation.isPending}
|
||||
style={{
|
||||
padding: "8px 18px",
|
||||
background: "var(--color-primary)",
|
||||
color: "#fff",
|
||||
border: "none",
|
||||
borderRadius: 6,
|
||||
fontSize: 14,
|
||||
cursor: isMigrating || migrateMutation.isPending ? "not-allowed" : "pointer",
|
||||
opacity: isMigrating || migrateMutation.isPending ? 0.6 : 1,
|
||||
}}
|
||||
>
|
||||
{migrateMutation.isPending ? "Starting…" : "Test & Migrate"}
|
||||
</button>
|
||||
|
||||
{isMigrating && (
|
||||
<button
|
||||
onClick={() => cancelMutation.mutate()}
|
||||
disabled={cancelMutation.isPending}
|
||||
style={{
|
||||
padding: "8px 18px",
|
||||
background: "transparent",
|
||||
color: "#dc2626",
|
||||
border: "1px solid #dc2626",
|
||||
borderRadius: 6,
|
||||
fontSize: 14,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Cancel
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{migStatus && migStatus.state !== "idle" && (
|
||||
<MigrationProgressBar status={migStatus} />
|
||||
)}
|
||||
</Section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,6 +1,13 @@
|
||||
# Merge Checklist — Pre-merge Test Suite
|
||||
# ALL_TESTS — Full Test Suite
|
||||
|
||||
Complete test suite covering all 20 feature areas. Run tests relevant to the changed area before merging any feature branch into `main`. Service-specific subsets live in separate files:
|
||||
|
||||
- `tests/backend_tests.md` — §1–9, §18 (auth, users, admin, groups, appearance, service health, plugins, AI/doc settings, infra/security)
|
||||
- `tests/frontend_tests.md` — §19 (UI & routing)
|
||||
- `tests/doc-service_tests.md` — §10–16 (upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory)
|
||||
- `tests/ai-service_tests.md` — §17 (AI queue & providers)
|
||||
- `tests/storage-service_tests.md` — §20 (storage-service: objects, backend switching, migration)
|
||||
|
||||
Run all tests relevant to the changed area before merging any feature branch into `main`.
|
||||
Every test describes the exact UI action or API call to perform and the expected outcome.
|
||||
|
||||
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
|
||||
@@ -83,6 +90,8 @@ Mark each row before opening the PR.
|
||||
| 4.6 | Remove member | Click remove on a member | User removed from group |
|
||||
| 4.7 | Duplicate group name | Create group with name that already exists | 400 / validation error shown |
|
||||
| 4.8 | Non-admin access | Regular user calls `GET /api/admin/groups` | 404 |
|
||||
| 4.9 | Set group admin role | Admin → group detail → tick "Group admin" checkbox on a member → save | `PATCH /api/admin/groups/{id}/members/{user_id}/admin` with `{"is_group_admin": true}` returns 200; badge shown in member list |
|
||||
| 4.10 | Unset group admin role | Admin unticks "Group admin" on an existing group admin member | Returns 200; badge removed; user loses group-admin privileges |
|
||||
|
||||
---
|
||||
|
||||
@@ -209,8 +218,12 @@ Mark each row before opening the PR.
|
||||
| 12.13 | Raw text section | Expand raw text collapse | First ~500k chars of extracted PDF text shown |
|
||||
| 12.14 | Download | Click "Download" | Browser downloads the original PDF file |
|
||||
| 12.15 | View in new tab | Click "View" | PDF opens in new browser tab; URL auto-revokes after 60s |
|
||||
| 12.16 | Delete | Click "Delete" → confirm dialog | Document and file removed; table row gone |
|
||||
| 12.17 | Non-owner cannot edit | Recipient of shared doc opens slide-over | Edit controls (type, tags, title, delete) absent; download available |
|
||||
| 12.16 | Delete — owner | Click "Delete" → confirm dialog | Document and file removed; table row gone |
|
||||
| 12.16b | Delete — admin | Admin opens any doc (not their own) → Delete → confirm | Document deleted; 204 returned |
|
||||
| 12.16c | Delete — can_delete share | Group member whose share has `can_delete=true` → Delete | 204; document removed; `viewer_can_delete` was `true` in `DocumentOut` |
|
||||
| 12.16d | Delete — group admin | User is group admin for a group the doc is shared with; no explicit `can_delete` flag → Delete | 204; group admin always has delete rights for docs shared with their group |
|
||||
| 12.16e | Delete — watch document, admin only | Watch-ingested doc (`source=watch`); regular user → Delete | 403 (not owner); admin can delete it |
|
||||
| 12.17 | Non-owner cannot edit | Recipient of shared doc opens slide-over (no can_delete, not group admin) | Edit controls (type, tags, title, delete) absent; download available |
|
||||
|
||||
---
|
||||
|
||||
@@ -229,6 +242,10 @@ Mark each row before opening the PR.
|
||||
| 13.8 | Bulk share | Select multiple rows → bulk share → pick group | All selected docs shared with that group |
|
||||
| 13.9 | Share count indicator | Document shared with 2 groups | Users icon in table row shows "2" |
|
||||
| 13.10 | Share with non-member group | `POST /api/documents/{id}/shares` with group not in X-User-Groups | 403 / validation error |
|
||||
| 13.11 | Share with can_delete enabled | Owner opens sharing section → tick "Allow group members to delete" → share | `can_delete=true` stored; trash icon badge appears next to group name in shares list |
|
||||
| 13.12 | Share without can_delete (default) | Owner shares without ticking delete checkbox | `can_delete=false`; recipient sees the doc but Delete button absent in slide-over |
|
||||
| 13.13 | can_delete shown in shares list | Share with can_delete=true → inspect shares list in slide-over | Trash2 icon rendered beside the group name; tooltip "Group members can delete this document" |
|
||||
| 13.14 | viewer_can_delete in document list | Share with can_delete=true; log in as group member → `GET /api/documents` | `viewer_can_delete=true` in the recipient's list response for that doc |
|
||||
|
||||
---
|
||||
|
||||
@@ -240,8 +257,19 @@ Mark each row before opening the PR.
|
||||
| 14.2 | Rename category | Manage categories dialog → edit → save | New name reflected everywhere |
|
||||
| 14.3 | Delete category | Delete category with documents assigned | 204; documents remain; category assignment removed |
|
||||
| 14.4 | Category search | More than 4 categories → type in search field | Tree filtered in real time |
|
||||
| 14.5 | Manage categories dialog | Click "Manage categories" | Modal shows all categories with rename/delete actions |
|
||||
| 14.5 | Manage categories dialog | Click "Manage categories" | Modal shows categories grouped by scope (Personal / Group / System) with lock icons on group and system categories |
|
||||
| 14.6 | New category triggers re-analysis | Create category with name similar to AI suggestion | Background re-analysis triggered (check backend logs) |
|
||||
| 14.7 | Create personal category | SourcePanel → "New category" → no group selected → submit valid PascalCase name | Created with `scope=personal`; visible only to owner |
|
||||
| 14.8 | Create group-scoped category | SourcePanel → "New category" → select a group → submit | Created with `scope=group`; visible to all members of that group |
|
||||
| 14.9 | Group category visible to group members | Log in as another group member | Group category appears in their category list and SourcePanel |
|
||||
| 14.10 | Non-member cannot see group category | Log in as user not in the group | Group category absent from list |
|
||||
| 14.11 | Only group admin can rename group category | Regular group member → rename group category | 403; group admin can rename it successfully |
|
||||
| 14.12 | Only group admin can delete group category | Regular group member → delete group category | 403; group admin can delete it |
|
||||
| 14.13 | System categories read-only for non-admin | Regular user → Manage categories → rename/delete a system category | 403; lock icon shown; action blocked in UI |
|
||||
| 14.14 | Admin can manage system categories | Superuser → rename or delete a system category | Succeeds; ManageCategoriesDialog shows edit/delete controls for system rows |
|
||||
| 14.15 | PascalCase naming enforced — invalid | Create category named `my-invoices` or `Invoice Reports` | 422 with message explaining PascalCase-with-dashes format |
|
||||
| 14.16 | PascalCase naming enforced — valid | Create category named `Vendor-Invoices` | 201; category created successfully |
|
||||
| 14.17 | SourcePanel scope sections | Categories exist for all three scopes | SourcePanel tree shows "Mine", per-group, and "System" sections separately |
|
||||
|
||||
---
|
||||
|
||||
@@ -323,3 +351,31 @@ Mark each row before opening the PR.
|
||||
| 19.8 | Unknown route | Navigate to `/does-not-exist` | Redirected to `/` |
|
||||
| 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs |
|
||||
| 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab |
|
||||
| 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off |
|
||||
|
||||
---
|
||||
|
||||
## 20. Storage Service
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored |
|
||||
| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload |
|
||||
| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 |
|
||||
| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` |
|
||||
| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` |
|
||||
| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 |
|
||||
| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent |
|
||||
| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service |
|
||||
| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone |
|
||||
| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket |
|
||||
| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" |
|
||||
| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` |
|
||||
| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss |
|
||||
| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done |
|
||||
| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active |
|
||||
| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" |
|
||||
| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone |
|
||||
| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent |
|
||||
| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent |
|
||||
| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |
|
||||
@@ -0,0 +1,36 @@
|
||||
# AI-Service Tests
|
||||
|
||||
Tests covering the AI provider intermediary (`ai-service:8010`): health, provider configuration, synchronous chat, the priority queue (enqueue/poll/cancel/pause/resume), and provider timeout.
|
||||
|
||||
Full combined suite: `tests/ALL_TESTS.md`
|
||||
|
||||
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
|
||||
**Admin credentials:** any superuser account created during stack setup.
|
||||
|
||||
---
|
||||
|
||||
## Legend
|
||||
|
||||
| Symbol | Meaning |
|
||||
|--------|---------|
|
||||
| ✅ | Pass |
|
||||
| ❌ | Fail |
|
||||
| — | N/A for this change |
|
||||
|
||||
---
|
||||
|
||||
## 17. AI Service — Queue & Providers
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 17.1 | Health check | `GET /health` on ai-service (via backend services endpoint) | `{"status": "ok"}` |
|
||||
| 17.2 | Provider health | `GET /health/provider` | Active provider name, model, configured=true |
|
||||
| 17.3 | Unconfigured provider | Set provider to "anthropic" with empty API key → test connection | 503 or 502 with clear error |
|
||||
| 17.4 | Sync chat | `POST /chat` with valid messages array | Response returned synchronously |
|
||||
| 17.5 | Queue — async job | `POST /queue/jobs` | `job_id` returned immediately |
|
||||
| 17.6 | Queue — poll job | `GET /queue/jobs/{id}` after enqueue | Returns status (`pending` → `done`) and result |
|
||||
| 17.7 | Queue — cancel job | `DELETE /queue/jobs/{id}` before processing | Job removed; status = cancelled |
|
||||
| 17.8 | Queue pause | `POST /queue/pause` | 204; current job finishes; no new jobs picked up |
|
||||
| 17.9 | Queue resume | `POST /queue/resume` after pause | 204; worker resumes; pending jobs process |
|
||||
| 17.10 | Priority ordering | Enqueue LOW then HIGH job | HIGH job processed first |
|
||||
| 17.11 | Provider timeout | `POST /chat` when provider is unreachable | 504 returned after timeout |
|
||||
@@ -0,0 +1,172 @@
|
||||
# Backend Tests
|
||||
|
||||
Tests covering the FastAPI gateway (`backend:8000`): auth, user/profile management, admin (users/groups/appearance), service health, plugin system, AI settings, document settings, and infrastructure/security.
|
||||
|
||||
Full combined suite: `tests/ALL_TESTS.md`
|
||||
|
||||
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
|
||||
**Admin credentials:** any superuser account created during stack setup.
|
||||
**Regular user credentials:** a second non-admin account for permission boundary tests.
|
||||
|
||||
---
|
||||
|
||||
## Legend
|
||||
|
||||
| Symbol | Meaning |
|
||||
|--------|---------|
|
||||
| ✅ | Pass |
|
||||
| ❌ | Fail |
|
||||
| — | N/A for this change |
|
||||
|
||||
---
|
||||
|
||||
## 1. Authentication
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 1.1 | Register new account | `POST /api/auth/register` with valid email + strong password | 201; user row created; login works immediately |
|
||||
| 1.2 | Password policy — too short | Register with 7-char password | 422 with validation error |
|
||||
| 1.3 | Password policy — no uppercase | Register with all-lowercase password | 422 with validation error |
|
||||
| 1.4 | Password policy — no special char | Register without special character | 422 with validation error |
|
||||
| 1.5 | Password policy — common word | Register with password containing "password" | 422 with validation error |
|
||||
| 1.6 | Duplicate email | Register with an already-used email | 400 |
|
||||
| 1.7 | Login — valid credentials | `POST /api/auth/login` with correct email + password | 200; `access_token` returned |
|
||||
| 1.8 | Login — wrong password | `POST /api/auth/login` with wrong password | 401 |
|
||||
| 1.9 | Login — inactive account | Admin deactivates user; attempt login | 401 |
|
||||
| 1.10 | JWT expiry respected | Manually craft token with `exp` in the past; call any protected route | 401 |
|
||||
| 1.11 | Logout clears session | Click Logout in UI; try navigating to `/` | Redirected to `/login` |
|
||||
| 1.12 | Unauthenticated redirect | Open `/` without a token in `localStorage` | Redirected to `/login` |
|
||||
|
||||
---
|
||||
|
||||
## 2. User — Profile & Preferences
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 2.1 | Fetch own profile | `GET /api/profile/me` | 200; profile auto-created if first request |
|
||||
| 2.2 | Update profile fields | `PUT /api/profile/me` with full_name, phone, position, address, date_of_birth | 200; fields persisted; visible on `/profile` page |
|
||||
| 2.3 | Invalid phone format | `PUT /api/profile/me` with letters in phone field | 422 |
|
||||
| 2.4 | Future date of birth | `PUT /api/profile/me` with DOB = tomorrow | 422 |
|
||||
| 2.5 | DOB before 1900 | `PUT /api/profile/me` with DOB = 1899-12-31 | 422 |
|
||||
| 2.6 | Fetch dashboard preferences | `GET /api/users/me/preferences` | 200; `app_ids` list |
|
||||
| 2.7 | Pin an app | Dashboard → pencil button → press `+` on a card → save | Card appears in pinned grid on next load |
|
||||
| 2.8 | Unpin an app | Dashboard → pencil button → press `−` on a pinned card → save | Card removed from pinned grid |
|
||||
| 2.9 | Pin limit (50) | `PATCH /api/users/me/preferences` with 51 app IDs | 422 |
|
||||
| 2.10 | Color mode — user pref | User sets mode to "dark"; reload page | Dark theme applied; preference persists across sessions |
|
||||
| 2.11 | Color mode — system fallback | User has NULL color_mode; admin default_mode = "light" | Light theme applied |
|
||||
|
||||
---
|
||||
|
||||
## 3. Admin — Users
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 3.1 | List all users | Admin → `/admin/users` | All registered users shown in table |
|
||||
| 3.2 | Create user | Admin clicks "Create user"; fills form | 201; new user appears in list; can log in |
|
||||
| 3.3 | Toggle user active | Admin clicks toggle on active user | User deactivated; login returns 401 |
|
||||
| 3.4 | Delete user | Admin deletes a user | 204; user no longer in list; their documents remain (orphaned) |
|
||||
| 3.5 | Non-admin access | Regular user navigates to `/admin/users` | Redirected to `/login` |
|
||||
| 3.6 | Admin 404 semantics | Regular user calls `GET /api/admin/users` via curl | 404 (not 403) |
|
||||
|
||||
---
|
||||
|
||||
## 4. Admin — Groups
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 4.1 | List groups | Admin → `/admin/groups` | All groups shown with member count |
|
||||
| 4.2 | Create group | Fill name + description → submit | Group appears in list; `{group_name}-admin` bootstrap group also exists (auto-created on service start) |
|
||||
| 4.3 | Edit group | Click edit on group → change name → save | Name updated |
|
||||
| 4.4 | Delete group | Delete group | 204; group gone; memberships cascade-deleted |
|
||||
| 4.5 | Add member | Open group → search user → add | 204; user appears in member list |
|
||||
| 4.6 | Remove member | Click remove on a member | User removed from group |
|
||||
| 4.7 | Duplicate group name | Create group with name that already exists | 400 / validation error shown |
|
||||
| 4.8 | Non-admin access | Regular user calls `GET /api/admin/groups` | 404 |
|
||||
| 4.9 | Set group admin role | Admin → group detail → tick "Group admin" checkbox on a member → save | `PATCH /api/admin/groups/{id}/members/{user_id}/admin` with `{"is_group_admin": true}` returns 200; badge shown in member list |
|
||||
| 4.10 | Unset group admin role | Admin unticks "Group admin" on an existing group admin member | Returns 200; badge removed; user loses group-admin privileges |
|
||||
|
||||
---
|
||||
|
||||
## 5. Admin — Appearance
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 5.1 | List themes | Admin → `/admin/appearance` | Built-in themes + any custom themes shown |
|
||||
| 5.2 | Switch active theme | Select a different theme → save | All users see the new theme on next load |
|
||||
| 5.3 | Create custom theme | Admin → create theme. Required fields: `id` (slug), `label`, `light` (CSS vars object), `dark` (CSS vars object) | 201; theme appears in selector; can be activated |
|
||||
| 5.4 | Edit custom theme | Admin edits colour values on a custom theme | Colours update live after activation |
|
||||
| 5.5 | Delete custom theme | Admin deletes a custom theme | 204; theme gone from selector; active theme reverts to default |
|
||||
| 5.6 | Set default mode | `PATCH /api/settings/appearance` with `{"theme": "<id>", "default_mode": "dark"}` (both fields required) | 200; new users without a personal preference see dark mode |
|
||||
|
||||
---
|
||||
|
||||
## 6. Service Health & Dashboard
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 6.1 | Services endpoint | `GET /api/services` (authenticated) | Returns health status for doc-service and ai-service |
|
||||
| 6.2 | Healthy service card | Both services running → `/apps` page. API response uses `healthy: true` (boolean), not `status: "healthy"` | Cards show "Available" badge; clicking navigates to the app |
|
||||
| 6.3 | Unhealthy service card | Stop doc-service container → wait 30s → `/apps` | Doc-service card dimmed, "Unavailable", not clickable |
|
||||
| 6.4 | Service recovery | Restart stopped container → wait 30s | Card returns to "Available" |
|
||||
| 6.5 | Dashboard pinned cards | Pin a service → go to `/` | Pinned card appears in home grid |
|
||||
| 6.6 | Customize mode | Click pencil on dashboard → toggle cards | Pinned list updates after save |
|
||||
|
||||
---
|
||||
|
||||
## 7. Plugin System
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 7.1 | List plugins | `GET /api/plugins` (authenticated) | Returns accessible plugins for current user |
|
||||
| 7.2 | Superuser sees all plugins | Log in as admin → `GET /api/plugins` | All registered service plugins returned |
|
||||
| 7.3 | Group member sees plugin | Add user to `doc-service-admin` group → `GET /api/plugins` | doc-service plugin returned |
|
||||
| 7.4 | Unpermitted user hidden | Regular user not in any admin group → `GET /api/plugins` | Empty list (plugins hidden, not 403) |
|
||||
| 7.5 | Manifest fetch | `GET /api/plugins/doc-service/manifest` as permitted user | JSON Schema + access rules returned |
|
||||
| 7.6 | Settings read | `GET /api/plugins/doc-service/settings` | Current doc-service plugin settings returned |
|
||||
| 7.7 | Settings write | `PATCH /api/plugins/doc-service/settings` with valid payload | 200; setting persisted to volume |
|
||||
| 7.8 | Unpermitted settings access | Regular user `GET /api/plugins/doc-service/settings` | 404 |
|
||||
|
||||
---
|
||||
|
||||
## 8. AI Service Settings
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 8.1 | Read AI config | Admin (or `ai-service-admin` member) → `GET /api/settings/ai` | Config returned; API keys masked |
|
||||
| 8.2 | Update provider | `PATCH /api/settings/ai` with provider = "anthropic" + valid key | 200; config persisted |
|
||||
| 8.3 | Test connection | `POST /api/settings/ai/test` with valid config | 200; success response from provider |
|
||||
| 8.4 | Test connection — bad key | `POST /api/settings/ai/test` with wrong API key | 502 or error detail |
|
||||
| 8.5 | Read system prompts | `GET /api/settings/system-prompts` | All registered service prompts returned |
|
||||
| 8.6 | Update system prompt | `PATCH /api/settings/system-prompts/doc-service` with new prompt text | 200; doc-service picks up new prompt within 30s |
|
||||
| 8.7 | Non-admin access | Regular user calls any `/api/settings/ai` endpoint | 404 |
|
||||
| 8.8 | `ai-service-admin` delegation | Non-superuser added to `ai-service-admin` group → accesses AI settings page | Page loads; can read and write settings |
|
||||
|
||||
---
|
||||
|
||||
## 9. Document Service Settings
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 9.1 | Read upload limits | `GET /api/settings/documents/limits` (admin or `doc-service-admin`) | `max_pdf_bytes` returned |
|
||||
| 9.2 | Update upload limit | `PATCH /api/settings/documents/limits` with new value | 200; upload of oversized PDF now rejected with 413 |
|
||||
| 9.3 | Non-admin access | Regular user calls `GET /api/settings/documents/limits` | 404 |
|
||||
| 9.4 | Settings page loads | Admin navigates to `/apps/documents/settings` | Upload limits section + watch directory config visible |
|
||||
| 9.5 | `doc-service-admin` delegation | Non-superuser added to `doc-service-admin` → navigates to settings page | Page loads; settings editable |
|
||||
|
||||
---
|
||||
|
||||
## 18. Infrastructure & Security
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 18.1 | Non-root containers | `docker inspect <container> --format '{{.Config.User}}'` for each service | Returns `1001` (or `70` for db) |
|
||||
| 18.2 | No host ports in prod | `docker compose up --build -d` → `docker ps` | Only port 80 (frontend) exposed; no 8000/8001/8010/5432 |
|
||||
| 18.3 | backend-net isolation | `curl http://localhost:8000` from host in prod | Connection refused |
|
||||
| 18.4 | Pre-commit hook runs | Stage a file with `eval("x")` → `git commit` | Commit blocked; security_check.py output shown |
|
||||
| 18.5 | Pre-commit hook — clean code | Normal commit | Hook passes; commit succeeds |
|
||||
| 18.6 | JWT algorithm none rejected | Craft token with `"alg": "none"` → call protected route | 401 |
|
||||
| 18.7 | XSS — input sanitation | Enter `<script>alert(1)</script>` in title/name fields | Value stored as plain text; not executed in UI |
|
||||
| 18.8 | SQL injection attempt | Pass `'; DROP TABLE documents; --` as search param | 200 with empty results; no DB error |
|
||||
| 18.9 | CORS | `curl -H "Origin: http://evil.com" http://localhost/api/users/me` | Request blocked or `access-control-allow-origin` not set for that origin |
|
||||
| 18.10 | Config volume persistence | Restart all containers | AI provider config + doc limits survive restart |
|
||||
| 18.11 | Migration auto-apply | Start fresh stack | Both `alembic upgrade head` chains run without error; all tables created |
|
||||
@@ -0,0 +1,157 @@
|
||||
# Doc-Service Tests
|
||||
|
||||
Tests covering the PDF extraction microservice (`doc-service:8001`): document upload/processing, list/filtering, slide-over detail view, sharing, categories, bulk actions, and the file watcher.
|
||||
|
||||
Full combined suite: `tests/ALL_TESTS.md`
|
||||
|
||||
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
|
||||
**Admin credentials:** any superuser account created during stack setup.
|
||||
**Regular user credentials:** a second non-admin account for permission boundary tests.
|
||||
|
||||
---
|
||||
|
||||
## Legend
|
||||
|
||||
| Symbol | Meaning |
|
||||
|--------|---------|
|
||||
| ✅ | Pass |
|
||||
| ❌ | Fail |
|
||||
| — | N/A for this change |
|
||||
|
||||
---
|
||||
|
||||
## 10. Document Upload & Processing
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 10.1 | Upload valid PDF | Drag-and-drop or file picker → select a PDF under the size limit | 202; document row appears with `status=pending`; transitions to `done` |
|
||||
| 10.2 | Upload oversized PDF | Upload a PDF exceeding `max_pdf_bytes` | 413; error shown; no row created |
|
||||
| 10.3 | Upload non-PDF | Upload a `.docx` or `.jpg` | 415; error shown |
|
||||
| 10.4 | Multi-file upload | Select 3 PDFs at once | All 3 appear in upload queue panel; each processes independently |
|
||||
| 10.5 | Upload queue panel | During upload → check bottom-right panel | Per-file status indicator; "Review →" link after each completes |
|
||||
| 10.6 | Drag-and-drop overlay | Drag file over the documents page | Full-page overlay appears; drop uploads file |
|
||||
| 10.7 | Processing status poll | Upload a large PDF | Table row auto-updates every 3s until status = `done` or `failed` |
|
||||
| 10.8 | AI extraction result | Open slide-over for a `done` document | title, document_type, tags, extracted_data fields populated |
|
||||
| 10.9 | Failed extraction | AI service down → upload PDF | Status = `failed`; error_message shown in slide-over |
|
||||
| 10.10 | Re-analyse | Click "Re-analyse" in slide-over | 202; status resets to `pending`; re-processes through AI |
|
||||
|
||||
---
|
||||
|
||||
## 11. Document List & Filtering
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 11.1 | Default list | Navigate to `/apps/documents` | Own documents shown, newest first, 20 per page |
|
||||
| 11.2 | Search | Type in search box (debounced 400ms) | Results filtered by title / filename / tags / type |
|
||||
| 11.3 | Filter by status | Add filter chip → Status → "done" | Only completed docs shown |
|
||||
| 11.4 | Filter by type | Add filter chip → Document type → "invoice" | Only invoices shown |
|
||||
| 11.5 | Filter by category | Add filter chip → Category → pick one | Only docs in that category shown |
|
||||
| 11.6 | Remove filter chip | Click × on a chip | Filter removed; full list restored |
|
||||
| 11.7 | Sort by column | Click "Date" column header | List re-ordered; chevron indicates direction; click again reverses |
|
||||
| 11.8 | Pagination | Upload > 20 docs → scroll to bottom | Page controls appear; page 2 loads next 20 |
|
||||
| 11.9 | "Mine" view | Click "Mine" in SourcePanel | Only own (uploaded) documents shown |
|
||||
| 11.10 | "Shared with me" view | Click "Shared with me" | Docs shared by others via groups; own docs excluded |
|
||||
| 11.11 | Category filter via SourcePanel | Click a category in the left tree | Table filtered to that category's documents |
|
||||
| 11.12 | URL state preserved | Apply filters → copy URL → open in new tab | Same filters applied |
|
||||
|
||||
---
|
||||
|
||||
## 12. Document Detail — Slide-over
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 12.1 | Open slide-over | Click any document row | 480px right panel slides in; metadata loaded |
|
||||
| 12.2 | Inline title edit | Click pencil icon next to title → type new title → confirm | Title saved; updated in table row |
|
||||
| 12.3 | Change document type | Click a type chip (Invoice, Receipt, etc.) | Type updated immediately |
|
||||
| 12.4 | Edit tags | Click into tag area → type a tag → press Enter → remove a tag with × | Tags saved correctly |
|
||||
| 12.5 | Assign category | Categories combobox → search → select | Category badge appears on document; table row updates |
|
||||
| 12.6 | Remove category | Click × on an assigned category badge | Category removed from document |
|
||||
| 12.7 | AI category suggestions | Slide-over shows "Suggested categories" | "Assign" and "Create & Assign" buttons present; clicking assigns |
|
||||
| 12.8 | Confirm folder suggestion | "Confirm" button next to suggested_folder | Category created (if needed) and assigned; `suggested_folder` cleared |
|
||||
| 12.9 | Reject folder suggestion | "Reject" button next to suggested_folder | `suggested_folder` cleared; no category created |
|
||||
| 12.10 | Confirm filename suggestion | "Confirm" button next to suggested_filename | `title` updated to suggested value; `suggested_filename` cleared |
|
||||
| 12.11 | Reject filename suggestion | "Reject" button next to suggested_filename | `suggested_filename` cleared; title unchanged |
|
||||
| 12.12 | Extracted data section | Open slide-over on `done` doc | Key-value table of AI-extracted fields (vendor, amounts, dates, etc.) |
|
||||
| 12.13 | Raw text section | Expand raw text collapse | First ~500k chars of extracted PDF text shown |
|
||||
| 12.14 | Download | Click "Download" | Browser downloads the original PDF file |
|
||||
| 12.15 | View in new tab | Click "View" | PDF opens in new browser tab; URL auto-revokes after 60s |
|
||||
| 12.16 | Delete — owner | Click "Delete" → confirm dialog | Document and file removed; table row gone |
|
||||
| 12.16b | Delete — admin | Admin opens any doc (not their own) → Delete → confirm | Document deleted; 204 returned |
|
||||
| 12.16c | Delete — can_delete share | Group member whose share has `can_delete=true` → Delete | 204; document removed; `viewer_can_delete` was `true` in `DocumentOut` |
|
||||
| 12.16d | Delete — group admin | User is group admin for a group the doc is shared with; no explicit `can_delete` flag → Delete | 204; group admin always has delete rights for docs shared with their group |
|
||||
| 12.16e | Delete — watch document, admin only | Watch-ingested doc (`source=watch`); regular user → Delete | 403 (not owner); admin can delete it |
|
||||
| 12.17 | Non-owner cannot edit | Recipient of shared doc opens slide-over (no can_delete, not group admin) | Edit controls (type, tags, title, delete) absent; download available |
|
||||
|
||||
---
|
||||
|
||||
## 13. Document Sharing
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 13.1 | Share from slide-over | Owner opens sharing section → selects a group from combobox → shares | Group appears in shares list; `share_count` in table row increments |
|
||||
| 13.2 | Only user's own groups shown | Open group picker in share section | Only groups the current user belongs to are listed |
|
||||
| 13.3 | Recipient sees shared doc | Log in as group member → "Shared with me" view | Shared document appears with primary accent border |
|
||||
| 13.4 | Recipient download | Recipient clicks Download on shared doc | PDF downloaded successfully |
|
||||
| 13.4b | Non-owner calls `GET /documents/{id}/shares` | Regular user on a doc they don't own | 404 (doc-service hides existence, consistent with admin 404 semantics — not 403) |
|
||||
| 13.5 | Recipient cannot delete | Recipient opens slide-over | Delete button absent |
|
||||
| 13.6 | Recipient cannot re-share | Recipient opens sharing section | Share controls absent |
|
||||
| 13.7 | Remove share | Owner clicks remove on a group share | Group removed; `share_count` decrements; recipient no longer sees doc |
|
||||
| 13.8 | Bulk share | Select multiple rows → bulk share → pick group | All selected docs shared with that group |
|
||||
| 13.9 | Share count indicator | Document shared with 2 groups | Users icon in table row shows "2" |
|
||||
| 13.10 | Share with non-member group | `POST /api/documents/{id}/shares` with group not in X-User-Groups | 403 / validation error |
|
||||
| 13.11 | Share with can_delete enabled | Owner opens sharing section → tick "Allow group members to delete" → share | `can_delete=true` stored; trash icon badge appears next to group name in shares list |
|
||||
| 13.12 | Share without can_delete (default) | Owner shares without ticking delete checkbox | `can_delete=false`; recipient sees the doc but Delete button absent in slide-over |
|
||||
| 13.13 | can_delete shown in shares list | Share with can_delete=true → inspect shares list in slide-over | Trash2 icon rendered beside the group name; tooltip "Group members can delete this document" |
|
||||
| 13.14 | viewer_can_delete in document list | Share with can_delete=true; log in as group member → `GET /api/documents` | `viewer_can_delete=true` in the recipient's list response for that doc |
|
||||
|
||||
---
|
||||
|
||||
## 14. Categories
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 14.1 | Create category | SourcePanel → "New category" form → submit | Category appears in tree |
|
||||
| 14.2 | Rename category | Manage categories dialog → edit → save | New name reflected everywhere |
|
||||
| 14.3 | Delete category | Delete category with documents assigned | 204; documents remain; category assignment removed |
|
||||
| 14.4 | Category search | More than 4 categories → type in search field | Tree filtered in real time |
|
||||
| 14.5 | Manage categories dialog | Click "Manage categories" | Modal shows categories grouped by scope (Personal / Group / System) with lock icons on group and system categories |
|
||||
| 14.6 | New category triggers re-analysis | Create category with name similar to AI suggestion | Background re-analysis triggered (check backend logs) |
|
||||
| 14.7 | Create personal category | SourcePanel → "New category" → no group selected → submit valid PascalCase name | Created with `scope=personal`; visible only to owner |
|
||||
| 14.8 | Create group-scoped category | SourcePanel → "New category" → select a group → submit | Created with `scope=group`; visible to all members of that group |
|
||||
| 14.9 | Group category visible to group members | Log in as another group member | Group category appears in their category list and SourcePanel |
|
||||
| 14.10 | Non-member cannot see group category | Log in as user not in the group | Group category absent from list |
|
||||
| 14.11 | Only group admin can rename group category | Regular group member → rename group category | 403; group admin can rename it successfully |
|
||||
| 14.12 | Only group admin can delete group category | Regular group member → delete group category | 403; group admin can delete it |
|
||||
| 14.13 | System categories read-only for non-admin | Regular user → Manage categories → rename/delete a system category | 403; lock icon shown; action blocked in UI |
|
||||
| 14.14 | Admin can manage system categories | Superuser → rename or delete a system category | Succeeds; ManageCategoriesDialog shows edit/delete controls for system rows |
|
||||
| 14.15 | PascalCase naming enforced — invalid | Create category named `my-invoices` or `Invoice Reports` | 422 with message explaining PascalCase-with-dashes format |
|
||||
| 14.16 | PascalCase naming enforced — valid | Create category named `Vendor-Invoices` | 201; category created successfully |
|
||||
| 14.17 | SourcePanel scope sections | Categories exist for all three scopes | SourcePanel tree shows "Mine", per-group, and "System" sections separately |
|
||||
|
||||
---
|
||||
|
||||
## 15. Bulk Actions
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 15.1 | Select rows | Tick checkboxes on multiple rows | Floating bulk actions bar appears at bottom |
|
||||
| 15.2 | Bulk share | Select docs → Share with group → confirm | All selected docs shared; confirmation |
|
||||
| 15.3 | Bulk delete | Select docs → Delete → confirm dialog | All selected docs deleted; bar disappears |
|
||||
| 15.4 | Clear selection | Click "Clear" in bulk bar | All checkboxes deselected; bar hides |
|
||||
| 15.5 | Bulk bar — "Mine" view only | Switch to "Shared with me" view | Bulk actions bar not shown (no edit rights for shared docs) |
|
||||
|
||||
---
|
||||
|
||||
## 16. Watch Directory
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 16.1 | Enable watch | Doc settings page → toggle `watch_enabled` on → save | File watcher starts; backend logs confirm |
|
||||
| 16.2 | Ingest new file | Drop a PDF into the bind-mounted watch directory | Document appears in "All Documents" view with `source=watch` |
|
||||
| 16.3 | Sub-folder to category | Place PDF in `watch/invoices/` | Document auto-assigned to "invoices" category |
|
||||
| 16.4 | Startup scan | Restart doc-service with PDFs already in watch dir | Pre-existing PDFs ingested (idempotent — no duplicates) |
|
||||
| 16.5 | AI folder suggestion | `ai_folder_suggestion` enabled → ingest file | `suggested_folder` populated; confirm/reject buttons visible in slide-over |
|
||||
| 16.6 | AI rename suggestion | `ai_rename_suggestion` enabled → ingest file | `suggested_filename` populated; confirm/reject buttons visible |
|
||||
| 16.7 | No-remove policy | Delete PDF from watch dir | Document record remains in DB |
|
||||
| 16.8 | Disable watch | Toggle `watch_enabled` off → save | Watcher stops; new files dropped are not ingested |
|
||||
| 16.9 | Watch docs visible to all users | Log in as any authenticated user | Watch-ingested docs (`user_id = "watch"`) appear in "All Documents" |
|
||||
@@ -0,0 +1,37 @@
|
||||
# Frontend Tests
|
||||
|
||||
Tests covering the React SPA (`frontend`, port 5173 dev / 80 prod): route guards, navigation, theme, component behaviour, and TanStack Query patterns.
|
||||
|
||||
Full combined suite: `tests/ALL_TESTS.md`
|
||||
|
||||
**Test environment:** Feature stack at `http://localhost:$PORT` (see CLAUDE.md §Feature branch workflow).
|
||||
**Admin credentials:** any superuser account created during stack setup.
|
||||
**Regular user credentials:** a second non-admin account for permission boundary tests.
|
||||
|
||||
---
|
||||
|
||||
## Legend
|
||||
|
||||
| Symbol | Meaning |
|
||||
|--------|---------|
|
||||
| ✅ | Pass |
|
||||
| ❌ | Fail |
|
||||
| — | N/A for this change |
|
||||
|
||||
---
|
||||
|
||||
## 19. Frontend — UI & Routing
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 19.1 | PrivateRoute redirect | Open any protected route without token | Redirected to `/login` |
|
||||
| 19.2 | AdminRoute redirect | Log in as non-admin → navigate to `/admin` | Redirected to `/login` |
|
||||
| 19.3 | ServiceAdminRoute | Non-admin, non-group-member → navigate to `/apps/documents/settings` | Redirected (access denied) |
|
||||
| 19.4 | Sidebar collapse | Click collapse button | Sidebar shrinks to icon-only; expand restores labels |
|
||||
| 19.5 | Apps accordion | Click "Apps" in sidebar | Expands to show "Documents" NavLink |
|
||||
| 19.6 | SourcePanel visibility | Navigate to `/apps` then `/apps/documents` | SourcePanel only visible on `/apps/documents` route |
|
||||
| 19.7 | Theme toggle | Click sun/moon button | Mode switches; persists on reload |
|
||||
| 19.8 | Unknown route | Navigate to `/does-not-exist` | Redirected to `/` |
|
||||
| 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs |
|
||||
| 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab |
|
||||
| 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off |
|
||||
@@ -0,0 +1,32 @@
|
||||
# Storage Service Tests — §20
|
||||
|
||||
Storage-service tests. Run these before merging any change that touches `features/storage-service/`, `docker-compose.yml` storage volumes, or storage-related backend/doc-service code.
|
||||
|
||||
See `tests/ALL_TESTS.md` for the full suite and legend.
|
||||
|
||||
---
|
||||
|
||||
## 20. Storage Service
|
||||
|
||||
| # | Test | Steps | Expected |
|
||||
|---|------|-------|----------|
|
||||
| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored |
|
||||
| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload |
|
||||
| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 |
|
||||
| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` |
|
||||
| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` |
|
||||
| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 |
|
||||
| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent |
|
||||
| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service |
|
||||
| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone |
|
||||
| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket |
|
||||
| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" |
|
||||
| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` |
|
||||
| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss |
|
||||
| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done |
|
||||
| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active |
|
||||
| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" |
|
||||
| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone |
|
||||
| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent |
|
||||
| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent |
|
||||
| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |
|
||||
Reference in New Issue
Block a user