diff --git a/CLAUDE.md b/CLAUDE.md index 0345fc0..0d879e0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,17 +6,19 @@ This file provides permanent, authoritative guidance to Claude Code for every se - `frontend/CLAUDE.md` — routes, components, API client patterns, XSS prevention - `features/ai-service/CLAUDE.md` — /chat, /health, /queue endpoints; queue service - `features/doc-service/CLAUDE.md` — document/category/share endpoints; DB models; PDF limits; file watcher +- `features/storage-service/CLAUDE.md` — storage API, pluggable backend drivers (local/S3/WebDAV), migration --- ## Merge checklist -Before merging any feature branch into `main`, every test relevant to the changed area in `tests/ALL_TESTS.md` (and the relevant service-specific file) must be marked passing. The test suite covers all 19 feature areas across four service files: +Before merging any feature branch into `main`, every test relevant to the changed area in `tests/ALL_TESTS.md` (and the relevant service-specific file) must be marked passing. The test suite covers all 20 feature areas across five service files: - `tests/backend_tests.md` — §1–9, §18 - `tests/frontend_tests.md` — §19 - `tests/doc-service_tests.md` — §10–16 - `tests/ai-service_tests.md` — §17 +- `tests/storage-service_tests.md` — §20 Do not merge without it. @@ -35,7 +37,7 @@ Do not merge without it. - New Docker service, volume, network, or env var → update **Docker Infrastructure** in this file - Stack version changed → update **Stack** in this file -- New feature or endpoint added → add test rows to **both** `tests/ALL_TESTS.md` (in the relevant section) **and** the matching service-specific file (`tests/backend_tests.md`, `tests/frontend_tests.md`, `tests/doc-service_tests.md`, or `tests/ai-service_tests.md`). Use the same test number and format as existing rows. +- New feature or endpoint added → add test rows to **both** `tests/ALL_TESTS.md` (in the relevant section) **and** the matching service-specific file (`tests/backend_tests.md`, `tests/frontend_tests.md`, `tests/doc-service_tests.md`, `tests/ai-service_tests.md`, or `tests/storage-service_tests.md`). Use the same test number and format as existing rows. This check is mandatory — treat it the same as updating STATUS.md. @@ -143,7 +145,17 @@ These standards are **non-negotiable**. Every change must comply. Implementation - `backend-net`: all containers except frontend; not reachable from host in prod. - `frontend-net`: only frontend; single host port (80 prod / 5173 dev). -- DB, backend, doc-service, ai-service have **no** host port bindings in prod. +- DB, backend, doc-service, ai-service, storage-service have **no** host port bindings in prod. + +### Storage rule (non-negotiable) + +**No service may write to a filesystem path for persistent data.** All file/blob storage must go through the storage-service HTTP API (`PUT/GET/DELETE /objects/{bucket}/{key}`). Config JSON files must be stored in the `config` bucket. Uploaded files must be stored in the `documents` bucket. Violation is a security and architecture defect. + +The only two persistent storage mechanisms in the project are: +1. **PostgreSQL** — structured/relational data +2. **storage-service** — all file/blob/config data (local filesystem by default; switchable to S3-compatible or WebDAV) + +New services and features must follow this pattern. See `features/storage-service/CLAUDE.md` for the API reference. ### Pre-commit security hook @@ -179,9 +191,10 @@ All other per-service defaults are in the relevant sub-CLAUDE.md file. | Service | Image base | Internal port | User | Volumes | Network | |---------|-----------|---------------|------|---------|---------| | `db` | postgres:16-alpine | 5432 | 70:70 | `postgres_data` | backend-net | -| `backend` | python:3.12-slim | 8000 | 1001:1001 | `app_config` | backend-net | -| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | `app_config` | backend-net | -| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `doc_data`, `watch_data`, `app_config` | backend-net | +| `backend` | python:3.12-slim | 8000 | 1001:1001 | — | backend-net | +| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | — | backend-net | +| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `watch_data` | backend-net | +| `storage-service` | python:3.12-slim | 8020 | 1001:1001 | `storage_data` | backend-net | | `frontend` | nginx-unprivileged:alpine | 8080 | 1001:1001 | — | backend-net, frontend-net | ### Volumes @@ -189,15 +202,14 @@ All other per-service defaults are in the relevant sub-CLAUDE.md file. | Volume | Mount path | Contains | |--------|-----------|---------| | `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL data | -| `doc_data` | `/data/documents` | Uploaded PDF files | +| `storage_data` | `/data/storage` | All file/blob storage: PDFs (`documents/`) and config JSONs (`config/`) | | `watch_data` | `/data/watch` | Watch directory (bind-mount NAS/Nextcloud via docker-compose.override.yml) | -| `app_config` | `/config` | Per-service runtime config JSON files | ### Networks | Network | Host-accessible | Members | |---------|----------------|---------| -| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, frontend | +| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, storage-service, frontend | | `frontend-net` | Yes (port 80 → frontend:8080) | frontend | ### Environment variables (required in `backend/.env`) @@ -213,6 +225,7 @@ Injected by docker-compose (not in `.env`): ``` DOC_SERVICE_URL=http://doc-service:8001 AI_SERVICE_URL=http://ai-service:8010 +STORAGE_SERVICE_URL=http://storage-service:8020 ``` --- diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 49ba78a..0da5456 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -36,7 +36,8 @@ backend/ │ │ ├── config.py ← All settings via pydantic-settings (reads .env) │ │ ├── security.py ← JWT sign/verify (RS256), bcrypt hash/verify │ │ ├── sanitize.py ← Input sanitization helpers (see Security Standards) -│ │ └── app_config.py ← Per-service config load/save to /config volume; theme files in /config/themes/ +│ │ ├── app_config.py ← Per-service config load/save via storage-service; theme files in config/themes/{id}.json +│ │ └── config_storage.py ← Thin async HTTP helpers: read_json/write_json/delete_key/list_keys → storage-service config bucket │ ├── models/ │ │ ├── __init__.py ← Imports all models (required for Alembic autogenerate) │ │ ├── user.py ← User model @@ -56,7 +57,8 @@ backend/ │ │ ├── services.py ← GET /services (health status) │ │ ├── plugins.py ← Generic plugin proxy (GET/PATCH /api/plugins/*) │ │ ├── categories_proxy.py ← Transparent proxy → doc-service /categories/* -│ │ └── documents_proxy.py ← Transparent proxy → doc-service /documents/* +│ │ ├── documents_proxy.py ← Transparent proxy → doc-service /documents/* +│ │ └── storage_config.py ← Admin proxy → storage-service config + migration endpoints │ └── services/ │ ├── service_health.py ← Background 30s health-check loop; caches /plugin/manifest per service │ └── group_bootstrap.py ← Ensures {service-id}-admin group exists for every registered service at startup @@ -216,6 +218,16 @@ Unique constraint: `(group_id, user_id)` Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence. +### Admin — Storage (`/api/admin`) — admin-only + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/admin/storage-config` | Current backend driver + health → proxied from storage-service `/health` | +| PATCH | `/api/admin/storage-config` | Reconfigure backend without data migration (same-backend credential update) | +| POST | `/api/admin/storage-config/migrate` | Start async migration to a new backend (copy → verify → switch → cleanup) | +| GET | `/api/admin/storage-config/migrate/status` | Poll migration progress: `{state, total, done, failed, errors[]}` | +| DELETE | `/api/admin/storage-config/migrate` | Cancel a running migration; old backend remains active | + ### Documents and Categories — proxied `/api/documents/*` and `/api/documents/categories/*` are transparently proxied to `doc-service:8001`. The backend injects `x-user-id`, `x-user-groups`, and `x-user-is-admin` headers. See `features/doc-service/CLAUDE.md` for the internal endpoint list. diff --git a/backend/STATUS.md b/backend/STATUS.md index 963a870..808e3e3 100644 --- a/backend/STATUS.md +++ b/backend/STATUS.md @@ -75,10 +75,20 @@ A background task (`service_health.py`) polls each service's `/health` endpoint | `GET` | `/api/settings/system-prompts` | All editable system prompts — superuser OR `ai-service-admin` member | | `PATCH` | `/api/settings/system-prompts/{id}` | Update system prompt — same access | -Settings are persisted to JSON files on the `app_config` Docker named volume and read by the respective feature services. +Settings are persisted to the `config` bucket of `storage-service:8020` via `core/config_storage.py`. All config I/O is async HTTP; no filesystem volumes are used. Access to service-specific settings endpoints is enforced by `get_service_admin(service_id)` in `deps.py` — grants access to superusers OR members of the `{service_id}-admin` group. +### Storage config (`/api/admin`) + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/admin/storage-config` | Current backend driver + health (proxied from storage-service) | +| `PATCH` | `/api/admin/storage-config` | Reconfigure backend without migration | +| `POST` | `/api/admin/storage-config/migrate` | Start async migration to a new backend | +| `GET` | `/api/admin/storage-config/migrate/status` | Poll migration progress | +| `DELETE` | `/api/admin/storage-config/migrate` | Cancel running migration | + ### Feature proxies All `/api/documents/*` and `/api/documents/categories/*` requests are transparently proxied to `doc-service:8001` via `httpx.AsyncClient`. The proxy: @@ -129,8 +139,11 @@ Browser (port 5173 dev / 80 prod) ┌───────────┼────────────┬──────────────┐ /auth /settings /documents/* /services /users (JSON │ │ - /admin volume) └── proxy → health-check loop - /profile doc-service:8001 (30s poll) + /admin /storage- └── proxy → health-check loop + /profile config doc-service:8001 (30s poll) + (proxy) + │ + storage-service:8020 ``` --- diff --git a/backend/app/main.py b/backend/app/main.py index f6913a9..95e99dd 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -9,6 +9,7 @@ from app.core.config import settings from app.database import AsyncSessionLocal from app.routers import admin, auth, categories_proxy, documents_proxy, groups, plugins, profile, services, users from app.routers import settings as settings_router +from app.routers import storage_config from app.services.group_bootstrap import ensure_service_admin_groups from app.services.service_health import check_all, health_check_loop, register_services @@ -52,6 +53,7 @@ app.include_router(admin.router, prefix="/api/admin", tags=["admin"]) app.include_router(groups.router, prefix="/api/admin/groups", tags=["admin"]) app.include_router(settings_router.router, prefix="/api/settings", tags=["settings"]) app.include_router(services.router, prefix="/api/services", tags=["services"]) +app.include_router(storage_config.router, prefix="/api/admin", tags=["admin"]) app.include_router(plugins.router, prefix="/api/plugins", tags=["plugins"]) # categories_proxy MUST be registered before documents_proxy — # otherwise /api/documents/{path:path} swallows /api/documents/categories/* diff --git a/backend/app/routers/storage_config.py b/backend/app/routers/storage_config.py new file mode 100644 index 0000000..bfd1728 --- /dev/null +++ b/backend/app/routers/storage_config.py @@ -0,0 +1,126 @@ +""" +Admin-only endpoints for storage-service backend configuration. + +GET /admin/storage-config — current backend driver + health +PATCH /admin/storage-config — update backend config (no data migration) +POST /admin/storage-config/migrate — start migration to a new backend +GET /admin/storage-config/migrate/status — poll migration progress +DELETE /admin/storage-config/migrate — cancel in-progress migration + +All endpoints proxy to storage-service:8020. +""" +import logging + +import httpx +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel + +from app.core.config import settings +from app.deps import get_current_admin +from app.models.user import User + +router = APIRouter() +logger = logging.getLogger(__name__) + +_STORAGE_BASE = settings.STORAGE_SERVICE_URL + + +class BackendConfigUpdate(BaseModel): + driver: str + config: dict = {} + + +class MigrateRequest(BaseModel): + driver: str + config: dict = {} + + +def _storage_url(path: str) -> str: + return f"{_STORAGE_BASE}{path}" + + +async def _proxy_get(path: str) -> dict: + async with httpx.AsyncClient(timeout=15.0) as client: + resp = await client.get(_storage_url(path)) + if resp.status_code == 404: + raise HTTPException(status_code=404, detail="Not found") + resp.raise_for_status() + return resp.json() + + +@router.get("/storage-config") +async def get_storage_config( + _: User = Depends(get_current_admin), +) -> dict: + """Return current backend driver and health status.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.get(_storage_url("/health")) + resp.raise_for_status() + return resp.json() + + +@router.patch("/storage-config", status_code=204) +async def update_storage_config( + body: BackendConfigUpdate, + _: User = Depends(get_current_admin), +) -> None: + """ + Reconfigure the active backend without migrating data. + Use when changing credentials for the same backend type, or reverting to local. + To move data to a new backend, use POST /admin/storage-config/migrate instead. + """ + async with httpx.AsyncClient(timeout=15.0) as client: + resp = await client.patch( + _storage_url("/backend-config"), + json={"driver": body.driver, "config": body.config}, + ) + if resp.status_code == 400: + raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed")) + if resp.status_code == 409: + raise HTTPException(status_code=409, detail="Migration in progress — cannot reconfigure now") + resp.raise_for_status() + + +@router.post("/storage-config/migrate", status_code=202) +async def start_migration( + body: MigrateRequest, + _: User = Depends(get_current_admin), +) -> dict: + """ + Start an async migration to a new backend. + + Flow: validate new backend → copy all objects → verify → switch → delete old objects. + The old backend stays active until 100% of objects are verified on the new one. + Poll GET /admin/storage-config/migrate/status to track progress. + """ + async with httpx.AsyncClient(timeout=30.0) as client: + resp = await client.post( + _storage_url("/migrate"), + json={"driver": body.driver, "config": body.config}, + ) + if resp.status_code == 400: + raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed")) + if resp.status_code == 409: + raise HTTPException(status_code=409, detail="A migration is already in progress") + resp.raise_for_status() + return resp.json() + + +@router.get("/storage-config/migrate/status") +async def migration_status( + _: User = Depends(get_current_admin), +) -> dict: + """Poll migration progress. State: idle → validating → migrating → switching → cleaning → done.""" + return await _proxy_get("/migrate/status") + + +@router.delete("/storage-config/migrate", status_code=204) +async def cancel_migration( + _: User = Depends(get_current_admin), +) -> None: + """Cancel a running migration. The old backend remains active.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.delete(_storage_url("/migrate")) + if resp.status_code == 409: + raise HTTPException(status_code=409, detail="No cancellable migration in progress") + resp.raise_for_status() diff --git a/changelog/2026-04-20_storage-service.md b/changelog/2026-04-20_storage-service.md new file mode 100644 index 0000000..e57c501 --- /dev/null +++ b/changelog/2026-04-20_storage-service.md @@ -0,0 +1,60 @@ +# 2026-04-20 — Dedicated storage-service with pluggable backends + +**Timestamp:** 2026-04-20T00:00:00Z + +## Summary + +Introduced a dedicated `storage-service` container (port 8020) as the single file/blob persistence layer for the entire stack. All services now route file and config I/O through this service's HTTP API. The service supports pluggable storage backends (local filesystem by default; S3-compatible and WebDAV built in) with a zero-data-loss migration flow. The `doc_data` and `app_config` Docker volumes were removed. + +## Files Added + +- `features/storage-service/app/main.py` — FastAPI app, lifespan (backend init) +- `features/storage-service/app/core/config.py` — Settings (DATA_DIR, STORAGE_BACKEND, S3_*, WEBDAV_*) +- `features/storage-service/app/routers/health.py` — GET /health +- `features/storage-service/app/routers/objects.py` — PUT/GET/DELETE /objects/{bucket}/{key:path}, GET /objects/{bucket} +- `features/storage-service/app/routers/migrate.py` — POST/GET/DELETE /migrate, PATCH /backend-config +- `features/storage-service/app/services/backend_manager.py` — Driver factory, singleton, atomic switch +- `features/storage-service/app/services/migration.py` — Async migration: copy → verify → switch → cleanup +- `features/storage-service/app/services/backends/base.py` — AbstractStorageBackend ABC +- `features/storage-service/app/services/backends/local.py` — LocalFSBackend (path traversal guard) +- `features/storage-service/app/services/backends/s3.py` — S3Backend (aiobotocore, endpoint_url configurable) +- `features/storage-service/app/services/backends/webdav.py` — WebDAVBackend (aiohttp + defusedxml) +- `features/storage-service/scripts/start.sh` — prod uvicorn start +- `features/storage-service/scripts/start_dev.sh` — dev uvicorn --reload start +- `features/storage-service/pyproject.toml` — Dependencies +- `features/storage-service/Dockerfile` — python:3.12-slim, non-root user 1001, port 8020 +- `features/storage-service/CLAUDE.md` — API reference, bucket docs, driver docs +- `features/storage-service/STATUS.md` — Service status +- `backend/app/core/config_storage.py` — Thin async helpers: read_json/write_json/delete_key/list_keys +- `backend/app/routers/storage_config.py` — Admin proxy endpoints for storage config + migration +- `features/doc-service/alembic/versions/0008_rename_file_path_to_storage_key.py` — DB migration +- `frontend/src/pages/StorageAdminPage.tsx` — Admin UI: backend status, driver form, migration progress +- `tests/storage-service_tests.md` — §20 storage-service test suite + +## Files Modified + +- `docker-compose.yml` — Added storage-service, storage_data volume; removed doc_data, app_config; added depends_on service_healthy +- `docker-compose.dev.yml` — Added storage-service dev override +- `backend/app/core/config.py` — Added STORAGE_SERVICE_URL +- `backend/app/core/app_config.py` — Full async rewrite using config_storage HTTP helpers (no filesystem) +- `backend/app/routers/settings.py` — Removed all asyncio.to_thread wrappers; direct await calls +- `backend/app/main.py` — Register storage_config router; update register_services call +- `backend/app/services/service_health.py` — Register storage-service +- `features/doc-service/app/core/config.py` — Added STORAGE_SERVICE_URL +- `features/doc-service/app/models/document.py` — file_path → storage_key +- `features/doc-service/app/services/storage.py` — Complete rewrite: HTTP client calls to storage-service +- `features/doc-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket +- `features/doc-service/app/services/file_watcher.py` — Uses save_upload() → storage-service +- `features/doc-service/app/routers/documents.py` — storage_key refs, pdfplumber(io.BytesIO), streaming from storage-service +- `features/ai-service/app/core/config.py` — Added STORAGE_SERVICE_URL; removed CONFIG_PATH +- `features/ai-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket +- `frontend/src/api/client.ts` — Added StorageStatus, MigrationStatus, StorageBackendConfig interfaces + 5 API functions +- `frontend/src/App.tsx` — Added /admin/storage route (AdminRoute → StorageAdminPage) +- `tests/ALL_TESTS.md` — Updated to 20 feature areas; added §20 storage-service tests +- `CLAUDE.md` — Added storage-service to Services/Volumes/Networks tables; storage enforcement rule; §20 test file +- `backend/CLAUDE.md` — Added config_storage.py, storage_config.py to tree; added admin storage endpoints +- `frontend/CLAUDE.md` — Added StorageAdminPage to tree; added /admin/storage route +- `features/doc-service/CLAUDE.md` — Updated storage.py description; file_path → storage_key; added migration 0008 +- `features/ai-service/CLAUDE.md` — Added config_reader.py description +- `backend/STATUS.md` — Added storage-config endpoints; updated settings persistence note +- `frontend/STATUS.md` — Added /admin/storage route; added StorageAdminPage description diff --git a/features/ai-service/CLAUDE.md b/features/ai-service/CLAUDE.md index 75582ed..26734cf 100644 --- a/features/ai-service/CLAUDE.md +++ b/features/ai-service/CLAUDE.md @@ -22,7 +22,8 @@ features/ai-service/ │ │ ├── queue.py ← GET /queue/status, /pause, /resume, /cancel/{id} │ │ └── plugin.py ← GET /plugin/manifest (access rules for ai-service-admin group) │ └── services/ -│ └── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL) +│ ├── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL) +│ └── config_reader.py ← Reads ai_service_config.json from storage-service config bucket (30 s TTL cache) ├── Dockerfile ← python:3.12-slim, non-root user 1001 └── STATUS.md ``` diff --git a/features/doc-service/CLAUDE.md b/features/doc-service/CLAUDE.md index 3dc60a3..d8d82a5 100644 --- a/features/doc-service/CLAUDE.md +++ b/features/doc-service/CLAUDE.md @@ -1,6 +1,6 @@ # doc-service — Claude context -PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows. +PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. All file/blob storage goes through `storage-service:8020` — no files are written directly to the filesystem. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows. --- @@ -38,13 +38,14 @@ features/doc-service/ │ │ ├── categories.py ← Category CRUD (includes watch-owned categories) │ │ └── plugin.py ← GET /plugin/manifest, GET+PATCH /plugin/settings │ └── services/ -│ ├── storage.py ← File I/O +│ ├── storage.py ← Storage client: save_upload/download_file/delete_file → storage-service:8020 documents bucket │ ├── ai_client.py ← classify_document() → ai-service:8010/chat -│ ├── config_reader.py ← Config load/save including storage/watch settings +│ ├── config_reader.py ← Config load/save via storage-service config bucket (doc_service_config.json) │ └── file_watcher.py ← watchdog-based PDF watcher + startup scan + ingestion ├── alembic/versions/ ← Migration chain │ ├── 0003_add_watch_columns.py ← source, watch_path, suggested_folder, suggested_filename -│ └── 0004_add_document_shares.py ← document_shares table (group-based sharing) +│ ├── 0004_add_document_shares.py ← document_shares table (group-based sharing) +│ └── 0008_rename_file_path_to_storage_key.py ← file_path → storage_key; strips /data/documents/ prefix from existing rows ├── Dockerfile ← python:3.12-slim, non-root user 1001 └── STATUS.md ``` @@ -60,7 +61,7 @@ features/doc-service/ | `id` | String | PK, UUID | | | `user_id` | String | indexed | not FK — trusts x-user-id header | | `filename` | String | NOT NULL | | -| `file_path` | String | NOT NULL | absolute path under /data/documents | +| `storage_key` | String | NOT NULL | storage-service key: `{user_id}/{doc_id}.pdf` (documents bucket) | | `file_size` | Integer | NOT NULL | bytes | | `status` | String | default="pending" | pending / processing / done / failed | | `title` | String(500) | nullable | AI-extracted | @@ -118,6 +119,7 @@ Unique constraint: `(document_id, group_id)` | `0005` | `add_share_can_delete` | | `0006` | `add_category_scope` | | `0007` | `capitalize_system_category_names` | +| `0008` | `rename_file_path_to_storage_key` | --- diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index ec69e2f..874b586 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -37,7 +37,8 @@ frontend/ │ │ └── ui/ ← shadcn/ui components (Button, Input, …) │ ├── pages/ ← One file per route │ │ ├── DocServiceSettingsPage.tsx ← Combined doc-service settings: upload limits + watch directory -│ │ └── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest +│ │ ├── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest +│ │ └── StorageAdminPage.tsx ← Admin storage backend config + live migration progress │ ├── lib/utils.ts ← cn() = clsx + tailwind-merge │ └── styles/theme.css ← CSS custom properties, Tailwind setup ├── vite.config.ts ← /api/* proxied to backend:8000 @@ -66,6 +67,7 @@ frontend/ | `/admin/users` | `AdminUsersPage` | AdminRoute | | `/admin/groups` | `AdminGroupsPage` | AdminRoute | | `/admin/appearance` | `AdminAppearancePage` | AdminRoute | +| `/admin/storage` | `StorageAdminPage` | AdminRoute | | `*` | redirect to `/` | — | `PrivateRoute` — checks `token` from `useAuth`, redirects to `/login` if absent. diff --git a/frontend/STATUS.md b/frontend/STATUS.md index 5facebb..2c23acb 100644 --- a/frontend/STATUS.md +++ b/frontend/STATUS.md @@ -21,6 +21,7 @@ All API calls go through `src/api/client.ts` (single Axios instance, JWT injecte | `/admin` | `AdminPage` (redirects to `/admin/users`) | Admin only | | `/admin/users` | `AdminUsersPage` | Admin only | | `/admin/groups` | `AdminGroupsPage` | Admin only | +| `/admin/storage` | `StorageAdminPage` | Admin only | | `/profile` | `ProfilePage` | Required | | `/settings` | `SettingsPage` (placeholder) | Required | | `/settings/plugins/:id` | `PluginSettingsPage` | Required (per-plugin access control) | @@ -114,6 +115,10 @@ Provider selector, per-provider fields, Test Connection, Save. Upload limits + watch directory config. +### Admin — Storage page (`/admin/storage`) + +Current backend status (green/red health dot). Driver selector (local/S3/WebDAV) with conditional credential fields. "Test & Migrate" button triggers an async migration that copies all objects to the new backend, verifies, then switches atomically. Live progress bar with 2s polling (states: validating → migrating → switching → cleaning → done). Cancel button during in-progress migrations. + ### Admin — Users page (`/admin/users`) User list, toggle active, create user, delete user. @@ -202,6 +207,7 @@ Key document-related functions: - [x] AI suggestion confirm/reject UI (folder + filename) - [x] Groups admin UI - [x] Replace Axios with native fetch; add global 401 → `/login` redirect for expired sessions +- [x] Admin storage page with live migration progress bar - [ ] Toast notification system - [ ] Loading skeletons - [ ] Cmd+K global search (`CommandDialog`) diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 561c405..a24e6a6 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -16,6 +16,7 @@ import DocServiceSettingsPage from "./pages/DocServiceSettingsPage"; import AIAdminSettingsPage from "./pages/AIAdminSettingsPage"; import SettingsPage from "./pages/SettingsPage"; import PluginSettingsPage from "./pages/PluginSettingsPage"; +import StorageAdminPage from "./pages/StorageAdminPage"; function PrivateRoute({ children }: { children: React.ReactNode }) { const { token } = useAuth(); @@ -102,6 +103,7 @@ export default function App() { } /> } /> } /> + } /> {/* Catch-all */} } /> diff --git a/frontend/src/api/client.ts b/frontend/src/api/client.ts index 3205c11..742b24f 100644 --- a/frontend/src/api/client.ts +++ b/frontend/src/api/client.ts @@ -585,6 +585,49 @@ export interface PluginManifest { }; } +// ── Storage admin ────────────────────────────────────────────────────────────── + +export interface StorageStatus { + status: string; + backend: string; +} + +export interface MigrationStatus { + state: + | "idle" + | "validating" + | "migrating" + | "switching" + | "cleaning" + | "done" + | "failed" + | "cancelled"; + total: number; + done: number; + failed: number; + errors: string[]; +} + +export interface StorageBackendConfig { + driver: string; + config: Record; +} + +export const getStorageConfig = () => api.get("/admin/storage-config"); + +export const updateStorageConfig = (body: StorageBackendConfig) => + api.patch("/admin/storage-config", body); + +export const startStorageMigration = (body: StorageBackendConfig) => + api.post<{ status: string; driver: string }>("/admin/storage-config/migrate", body); + +export const getMigrationStatus = () => + api.get("/admin/storage-config/migrate/status"); + +export const cancelMigration = () => api.delete("/admin/storage-config/migrate"); + +// ── Plugins ──────────────────────────────────────────────────────────────────── + export const getPlugins = () => api.get("/plugins"); export const getPluginManifest = (id: string) => diff --git a/frontend/src/pages/StorageAdminPage.tsx b/frontend/src/pages/StorageAdminPage.tsx new file mode 100644 index 0000000..224da06 --- /dev/null +++ b/frontend/src/pages/StorageAdminPage.tsx @@ -0,0 +1,382 @@ +import { useState, useEffect, useRef } from "react"; +import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"; +import { + getStorageConfig, + getMigrationStatus, + startStorageMigration, + cancelMigration, + updateStorageConfig, + type StorageBackendConfig, + type MigrationStatus, +} from "../api/client"; + +type Driver = "local" | "s3" | "webdav"; + +function Section({ title, children }: { title: string; children: React.ReactNode }) { + return ( +
+

{title}

+ {children} +
+ ); +} + +function Field({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
+ + {children} +
+ ); +} + +function inputStyle(disabled = false): React.CSSProperties { + return { + width: "100%", + padding: "6px 10px", + border: "1px solid var(--color-border)", + borderRadius: 6, + fontSize: 14, + background: disabled ? "var(--color-surface)" : "var(--color-background)", + color: "var(--color-text-primary)", + opacity: disabled ? 0.7 : 1, + }; +} + +function MigrationProgressBar({ status }: { status: MigrationStatus }) { + const pct = status.total > 0 ? Math.round((status.done / status.total) * 100) : 0; + const isBusy = ["validating", "migrating", "switching", "cleaning"].includes(status.state); + + const stateLabel: Record = { + idle: "Idle", + validating: "Validating new backend…", + migrating: `Migrating — ${status.done} / ${status.total} objects`, + switching: "Switching active backend…", + cleaning: "Cleaning up old backend…", + done: "Migration complete", + failed: "Migration failed", + cancelled: "Migration cancelled", + }; + + return ( +
+
+ + {stateLabel[status.state] ?? status.state} + + {isBusy && {pct}%} +
+ {(isBusy || status.state === "done") && ( +
+
+
+ )} + {status.errors.length > 0 && ( +
+ {status.errors.slice(0, 10).map((e, i) => ( +
+ {e} +
+ ))} + {status.errors.length > 10 && ( +
…and {status.errors.length - 10} more
+ )} +
+ )} +
+ ); +} + +export default function StorageAdminPage() { + const queryClient = useQueryClient(); + + const { data: storageStatus } = useQuery({ + queryKey: ["storage-config"], + queryFn: getStorageConfig, + refetchInterval: 10_000, + }); + + const { data: migStatus, refetch: refetchMig } = useQuery({ + queryKey: ["migration-status"], + queryFn: getMigrationStatus, + refetchInterval: (query) => { + const state = query.state.data?.state; + return state && ["validating", "migrating", "switching", "cleaning"].includes(state) + ? 2_000 + : false; + }, + }); + + const isMigrating = + migStatus && + ["validating", "migrating", "switching", "cleaning"].includes(migStatus.state); + + // ── New backend form ───────────────────────────────────────────────────────── + const [driver, setDriver] = useState("local"); + const [s3EndpointUrl, setS3EndpointUrl] = useState(""); + const [s3AccessKey, setS3AccessKey] = useState(""); + const [s3SecretKey, setS3SecretKey] = useState(""); + const [s3Region, setS3Region] = useState("us-east-1"); + const [webdavUrl, setWebdavUrl] = useState(""); + const [webdavUsername, setWebdavUsername] = useState(""); + const [webdavPassword, setWebdavPassword] = useState(""); + const [webdavRootPath, setWebdavRootPath] = useState("/"); + const [error, setError] = useState(""); + + function buildConfig(): StorageBackendConfig { + if (driver === "s3") { + return { + driver, + config: { + endpoint_url: s3EndpointUrl, + access_key: s3AccessKey, + secret_key: s3SecretKey, + region: s3Region, + }, + }; + } + if (driver === "webdav") { + return { + driver, + config: { + url: webdavUrl, + username: webdavUsername, + password: webdavPassword, + root_path: webdavRootPath, + }, + }; + } + return { driver: "local", config: {} }; + } + + const migrateMutation = useMutation({ + mutationFn: startStorageMigration, + onSuccess: () => { + setError(""); + refetchMig(); + }, + onError: (e: Error) => setError(e.message), + }); + + const cancelMutation = useMutation({ + mutationFn: cancelMigration, + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: ["migration-status"] }); + queryClient.invalidateQueries({ queryKey: ["storage-config"] }); + }, + onError: (e: Error) => setError(e.message), + }); + + const currentDriver = storageStatus?.backend ?? "—"; + + return ( +
+

Storage

+

+ All uploaded files are stored through the storage-service. Switch between local filesystem, + S3-compatible cloud storage, or WebDAV (Nextcloud). +

+ +
+
+ + {currentDriver} + {storageStatus?.status === "ok" ? " — healthy" : " — unreachable"} +
+
+ +
+

+ When you click Test & Migrate, all existing files will be copied to the + new backend, verified, and the switch will happen only after every file is confirmed. The + old backend is cleaned up automatically. +

+ + + + + + {driver === "s3" && ( + <> + + setS3EndpointUrl(e.target.value)} + placeholder="http://minio:9000" + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + setS3AccessKey(e.target.value)} + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + setS3SecretKey(e.target.value)} + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + setS3Region(e.target.value)} + placeholder="us-east-1" + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + )} + + {driver === "webdav" && ( + <> + + setWebdavUrl(e.target.value)} + placeholder="https://nextcloud.example.com" + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + setWebdavUsername(e.target.value)} + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + setWebdavPassword(e.target.value)} + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + setWebdavRootPath(e.target.value)} + placeholder="/remote.php/dav/files/username" + disabled={!!isMigrating} + style={inputStyle(!!isMigrating)} + /> + + + )} + + {error && ( +

{error}

+ )} + +
+ + + {isMigrating && ( + + )} +
+ + {migStatus && migStatus.state !== "idle" && ( + + )} +
+
+ ); +} diff --git a/tests/ALL_TESTS.md b/tests/ALL_TESTS.md index 3f439fe..a56c3f4 100644 --- a/tests/ALL_TESTS.md +++ b/tests/ALL_TESTS.md @@ -1,11 +1,12 @@ # ALL_TESTS — Full Test Suite -Complete test suite covering all 19 feature areas. Run tests relevant to the changed area before merging any feature branch into `main`. Service-specific subsets live in separate files: +Complete test suite covering all 20 feature areas. Run tests relevant to the changed area before merging any feature branch into `main`. Service-specific subsets live in separate files: - `tests/backend_tests.md` — §1–9, §18 (auth, users, admin, groups, appearance, service health, plugins, AI/doc settings, infra/security) - `tests/frontend_tests.md` — §19 (UI & routing) - `tests/doc-service_tests.md` — §10–16 (upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory) - `tests/ai-service_tests.md` — §17 (AI queue & providers) +- `tests/storage-service_tests.md` — §20 (storage-service: objects, backend switching, migration) Every test describes the exact UI action or API call to perform and the expected outcome. @@ -351,3 +352,30 @@ Mark each row before opening the PR. | 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs | | 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab | | 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off | + +--- + +## 20. Storage Service + +| # | Test | Steps | Expected | +|---|------|-------|----------| +| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored | +| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload | +| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 | +| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` | +| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` | +| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 | +| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent | +| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service | +| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone | +| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket | +| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" | +| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` | +| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss | +| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done | +| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active | +| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" | +| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone | +| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent | +| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent | +| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it | diff --git a/tests/storage-service_tests.md b/tests/storage-service_tests.md new file mode 100644 index 0000000..926c76f --- /dev/null +++ b/tests/storage-service_tests.md @@ -0,0 +1,32 @@ +# Storage Service Tests — §20 + +Storage-service tests. Run these before merging any change that touches `features/storage-service/`, `docker-compose.yml` storage volumes, or storage-related backend/doc-service code. + +See `tests/ALL_TESTS.md` for the full suite and legend. + +--- + +## 20. Storage Service + +| # | Test | Steps | Expected | +|---|------|-------|----------| +| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored | +| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload | +| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 | +| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` | +| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` | +| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 | +| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent | +| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service | +| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone | +| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket | +| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" | +| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` | +| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss | +| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done | +| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active | +| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" | +| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone | +| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent | +| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent | +| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |