feat: Phase 4+5 — admin storage UI, backend proxy, CLAUDE.md enforcement

- backend/app/routers/storage_config.py: 5 admin-only endpoints proxying
  storage-service config + migration API (GET/PATCH/POST/DELETE)
- backend/app/main.py: register storage_config router
- frontend/src/api/client.ts: StorageStatus, MigrationStatus,
  StorageBackendConfig interfaces + 5 API functions
- frontend/src/pages/StorageAdminPage.tsx: full admin UI — backend health
  dot, driver selector (local/S3/WebDAV), conditional credential fields,
  Test & Migrate button, live 2s-poll migration progress bar, Cancel
- frontend/src/App.tsx: /admin/storage route (AdminRoute guard)
- CLAUDE.md: storage enforcement rule, updated Docker tables (6 services,
  3 volumes), §20 in merge checklist
- backend/CLAUDE.md, frontend/CLAUDE.md, doc-service/CLAUDE.md,
  ai-service/CLAUDE.md: updated to reflect storage-service integration
- tests/ALL_TESTS.md + tests/storage-service_tests.md: §20 (20 tests)
- backend/STATUS.md, frontend/STATUS.md: updated with new endpoints/routes
- changelog/2026-04-20_storage-service.md: full change log

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
2026-04-20 16:13:05 +02:00
parent 4c35d7a2a4
commit cfec3bb906
15 changed files with 746 additions and 22 deletions
+22 -9
View File
@@ -6,17 +6,19 @@ This file provides permanent, authoritative guidance to Claude Code for every se
- `frontend/CLAUDE.md` — routes, components, API client patterns, XSS prevention - `frontend/CLAUDE.md` — routes, components, API client patterns, XSS prevention
- `features/ai-service/CLAUDE.md` — /chat, /health, /queue endpoints; queue service - `features/ai-service/CLAUDE.md` — /chat, /health, /queue endpoints; queue service
- `features/doc-service/CLAUDE.md` — document/category/share endpoints; DB models; PDF limits; file watcher - `features/doc-service/CLAUDE.md` — document/category/share endpoints; DB models; PDF limits; file watcher
- `features/storage-service/CLAUDE.md` — storage API, pluggable backend drivers (local/S3/WebDAV), migration
--- ---
## Merge checklist ## Merge checklist
Before merging any feature branch into `main`, every test relevant to the changed area in `tests/ALL_TESTS.md` (and the relevant service-specific file) must be marked passing. The test suite covers all 19 feature areas across four service files: Before merging any feature branch into `main`, every test relevant to the changed area in `tests/ALL_TESTS.md` (and the relevant service-specific file) must be marked passing. The test suite covers all 20 feature areas across five service files:
- `tests/backend_tests.md` — §19, §18 - `tests/backend_tests.md` — §19, §18
- `tests/frontend_tests.md` — §19 - `tests/frontend_tests.md` — §19
- `tests/doc-service_tests.md` — §1016 - `tests/doc-service_tests.md` — §1016
- `tests/ai-service_tests.md` — §17 - `tests/ai-service_tests.md` — §17
- `tests/storage-service_tests.md` — §20
Do not merge without it. Do not merge without it.
@@ -35,7 +37,7 @@ Do not merge without it.
- New Docker service, volume, network, or env var → update **Docker Infrastructure** in this file - New Docker service, volume, network, or env var → update **Docker Infrastructure** in this file
- Stack version changed → update **Stack** in this file - Stack version changed → update **Stack** in this file
- New feature or endpoint added → add test rows to **both** `tests/ALL_TESTS.md` (in the relevant section) **and** the matching service-specific file (`tests/backend_tests.md`, `tests/frontend_tests.md`, `tests/doc-service_tests.md`, or `tests/ai-service_tests.md`). Use the same test number and format as existing rows. - New feature or endpoint added → add test rows to **both** `tests/ALL_TESTS.md` (in the relevant section) **and** the matching service-specific file (`tests/backend_tests.md`, `tests/frontend_tests.md`, `tests/doc-service_tests.md`, `tests/ai-service_tests.md`, or `tests/storage-service_tests.md`). Use the same test number and format as existing rows.
This check is mandatory — treat it the same as updating STATUS.md. This check is mandatory — treat it the same as updating STATUS.md.
@@ -143,7 +145,17 @@ These standards are **non-negotiable**. Every change must comply. Implementation
- `backend-net`: all containers except frontend; not reachable from host in prod. - `backend-net`: all containers except frontend; not reachable from host in prod.
- `frontend-net`: only frontend; single host port (80 prod / 5173 dev). - `frontend-net`: only frontend; single host port (80 prod / 5173 dev).
- DB, backend, doc-service, ai-service have **no** host port bindings in prod. - DB, backend, doc-service, ai-service, storage-service have **no** host port bindings in prod.
### Storage rule (non-negotiable)
**No service may write to a filesystem path for persistent data.** All file/blob storage must go through the storage-service HTTP API (`PUT/GET/DELETE /objects/{bucket}/{key}`). Config JSON files must be stored in the `config` bucket. Uploaded files must be stored in the `documents` bucket. Violation is a security and architecture defect.
The only two persistent storage mechanisms in the project are:
1. **PostgreSQL** — structured/relational data
2. **storage-service** — all file/blob/config data (local filesystem by default; switchable to S3-compatible or WebDAV)
New services and features must follow this pattern. See `features/storage-service/CLAUDE.md` for the API reference.
### Pre-commit security hook ### Pre-commit security hook
@@ -179,9 +191,10 @@ All other per-service defaults are in the relevant sub-CLAUDE.md file.
| Service | Image base | Internal port | User | Volumes | Network | | Service | Image base | Internal port | User | Volumes | Network |
|---------|-----------|---------------|------|---------|---------| |---------|-----------|---------------|------|---------|---------|
| `db` | postgres:16-alpine | 5432 | 70:70 | `postgres_data` | backend-net | | `db` | postgres:16-alpine | 5432 | 70:70 | `postgres_data` | backend-net |
| `backend` | python:3.12-slim | 8000 | 1001:1001 | `app_config` | backend-net | | `backend` | python:3.12-slim | 8000 | 1001:1001 | | backend-net |
| `ai-service` | python:3.12-slim | 8010 | 1001:1001 | `app_config` | backend-net | | `ai-service` | python:3.12-slim | 8010 | 1001:1001 | | backend-net |
| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `doc_data`, `watch_data`, `app_config` | backend-net | | `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `watch_data` | backend-net |
| `storage-service` | python:3.12-slim | 8020 | 1001:1001 | `storage_data` | backend-net |
| `frontend` | nginx-unprivileged:alpine | 8080 | 1001:1001 | — | backend-net, frontend-net | | `frontend` | nginx-unprivileged:alpine | 8080 | 1001:1001 | — | backend-net, frontend-net |
### Volumes ### Volumes
@@ -189,15 +202,14 @@ All other per-service defaults are in the relevant sub-CLAUDE.md file.
| Volume | Mount path | Contains | | Volume | Mount path | Contains |
|--------|-----------|---------| |--------|-----------|---------|
| `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL data | | `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL data |
| `doc_data` | `/data/documents` | Uploaded PDF files | | `storage_data` | `/data/storage` | All file/blob storage: PDFs (`documents/`) and config JSONs (`config/`) |
| `watch_data` | `/data/watch` | Watch directory (bind-mount NAS/Nextcloud via docker-compose.override.yml) | | `watch_data` | `/data/watch` | Watch directory (bind-mount NAS/Nextcloud via docker-compose.override.yml) |
| `app_config` | `/config` | Per-service runtime config JSON files |
### Networks ### Networks
| Network | Host-accessible | Members | | Network | Host-accessible | Members |
|---------|----------------|---------| |---------|----------------|---------|
| `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, frontend | | `backend-net` | No (no host ports in prod) | db, backend, ai-service, doc-service, storage-service, frontend |
| `frontend-net` | Yes (port 80 → frontend:8080) | frontend | | `frontend-net` | Yes (port 80 → frontend:8080) | frontend |
### Environment variables (required in `backend/.env`) ### Environment variables (required in `backend/.env`)
@@ -213,6 +225,7 @@ Injected by docker-compose (not in `.env`):
``` ```
DOC_SERVICE_URL=http://doc-service:8001 DOC_SERVICE_URL=http://doc-service:8001
AI_SERVICE_URL=http://ai-service:8010 AI_SERVICE_URL=http://ai-service:8010
STORAGE_SERVICE_URL=http://storage-service:8020
``` ```
--- ---
+14 -2
View File
@@ -36,7 +36,8 @@ backend/
│ │ ├── config.py ← All settings via pydantic-settings (reads .env) │ │ ├── config.py ← All settings via pydantic-settings (reads .env)
│ │ ├── security.py ← JWT sign/verify (RS256), bcrypt hash/verify │ │ ├── security.py ← JWT sign/verify (RS256), bcrypt hash/verify
│ │ ├── sanitize.py ← Input sanitization helpers (see Security Standards) │ │ ├── sanitize.py ← Input sanitization helpers (see Security Standards)
│ │ ── app_config.py ← Per-service config load/save to /config volume; theme files in /config/themes/ │ │ ── app_config.py ← Per-service config load/save via storage-service; theme files in config/themes/{id}.json
│ │ └── config_storage.py ← Thin async HTTP helpers: read_json/write_json/delete_key/list_keys → storage-service config bucket
│ ├── models/ │ ├── models/
│ │ ├── __init__.py ← Imports all models (required for Alembic autogenerate) │ │ ├── __init__.py ← Imports all models (required for Alembic autogenerate)
│ │ ├── user.py ← User model │ │ ├── user.py ← User model
@@ -56,7 +57,8 @@ backend/
│ │ ├── services.py ← GET /services (health status) │ │ ├── services.py ← GET /services (health status)
│ │ ├── plugins.py ← Generic plugin proxy (GET/PATCH /api/plugins/*) │ │ ├── plugins.py ← Generic plugin proxy (GET/PATCH /api/plugins/*)
│ │ ├── categories_proxy.py ← Transparent proxy → doc-service /categories/* │ │ ├── categories_proxy.py ← Transparent proxy → doc-service /categories/*
│ │ ── documents_proxy.py ← Transparent proxy → doc-service /documents/* │ │ ── documents_proxy.py ← Transparent proxy → doc-service /documents/*
│ │ └── storage_config.py ← Admin proxy → storage-service config + migration endpoints
│ └── services/ │ └── services/
│ ├── service_health.py ← Background 30s health-check loop; caches /plugin/manifest per service │ ├── service_health.py ← Background 30s health-check loop; caches /plugin/manifest per service
│ └── group_bootstrap.py ← Ensures {service-id}-admin group exists for every registered service at startup │ └── group_bootstrap.py ← Ensures {service-id}-admin group exists for every registered service at startup
@@ -216,6 +218,16 @@ Unique constraint: `(group_id, user_id)`
Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence. Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence.
### Admin — Storage (`/api/admin`) — admin-only
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/storage-config` | Current backend driver + health → proxied from storage-service `/health` |
| PATCH | `/api/admin/storage-config` | Reconfigure backend without data migration (same-backend credential update) |
| POST | `/api/admin/storage-config/migrate` | Start async migration to a new backend (copy → verify → switch → cleanup) |
| GET | `/api/admin/storage-config/migrate/status` | Poll migration progress: `{state, total, done, failed, errors[]}` |
| DELETE | `/api/admin/storage-config/migrate` | Cancel a running migration; old backend remains active |
### Documents and Categories — proxied ### Documents and Categories — proxied
`/api/documents/*` and `/api/documents/categories/*` are transparently proxied to `doc-service:8001`. The backend injects `x-user-id`, `x-user-groups`, and `x-user-is-admin` headers. See `features/doc-service/CLAUDE.md` for the internal endpoint list. `/api/documents/*` and `/api/documents/categories/*` are transparently proxied to `doc-service:8001`. The backend injects `x-user-id`, `x-user-groups`, and `x-user-is-admin` headers. See `features/doc-service/CLAUDE.md` for the internal endpoint list.
+16 -3
View File
@@ -75,10 +75,20 @@ A background task (`service_health.py`) polls each service's `/health` endpoint
| `GET` | `/api/settings/system-prompts` | All editable system prompts — superuser OR `ai-service-admin` member | | `GET` | `/api/settings/system-prompts` | All editable system prompts — superuser OR `ai-service-admin` member |
| `PATCH` | `/api/settings/system-prompts/{id}` | Update system prompt — same access | | `PATCH` | `/api/settings/system-prompts/{id}` | Update system prompt — same access |
Settings are persisted to JSON files on the `app_config` Docker named volume and read by the respective feature services. Settings are persisted to the `config` bucket of `storage-service:8020` via `core/config_storage.py`. All config I/O is async HTTP; no filesystem volumes are used.
Access to service-specific settings endpoints is enforced by `get_service_admin(service_id)` in `deps.py` — grants access to superusers OR members of the `{service_id}-admin` group. Access to service-specific settings endpoints is enforced by `get_service_admin(service_id)` in `deps.py` — grants access to superusers OR members of the `{service_id}-admin` group.
### Storage config (`/api/admin`)
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/admin/storage-config` | Current backend driver + health (proxied from storage-service) |
| `PATCH` | `/api/admin/storage-config` | Reconfigure backend without migration |
| `POST` | `/api/admin/storage-config/migrate` | Start async migration to a new backend |
| `GET` | `/api/admin/storage-config/migrate/status` | Poll migration progress |
| `DELETE` | `/api/admin/storage-config/migrate` | Cancel running migration |
### Feature proxies ### Feature proxies
All `/api/documents/*` and `/api/documents/categories/*` requests are transparently proxied to `doc-service:8001` via `httpx.AsyncClient`. The proxy: All `/api/documents/*` and `/api/documents/categories/*` requests are transparently proxied to `doc-service:8001` via `httpx.AsyncClient`. The proxy:
@@ -129,8 +139,11 @@ Browser (port 5173 dev / 80 prod)
┌───────────┼────────────┬──────────────┐ ┌───────────┼────────────┬──────────────┐
/auth /settings /documents/* /services /auth /settings /documents/* /services
/users (JSON │ │ /users (JSON │ │
/admin volume) └── proxy → health-check loop /admin /storage- └── proxy → health-check loop
/profile doc-service:8001 (30s poll) /profile config doc-service:8001 (30s poll)
(proxy)
storage-service:8020
``` ```
--- ---
+2
View File
@@ -9,6 +9,7 @@ from app.core.config import settings
from app.database import AsyncSessionLocal from app.database import AsyncSessionLocal
from app.routers import admin, auth, categories_proxy, documents_proxy, groups, plugins, profile, services, users from app.routers import admin, auth, categories_proxy, documents_proxy, groups, plugins, profile, services, users
from app.routers import settings as settings_router from app.routers import settings as settings_router
from app.routers import storage_config
from app.services.group_bootstrap import ensure_service_admin_groups from app.services.group_bootstrap import ensure_service_admin_groups
from app.services.service_health import check_all, health_check_loop, register_services from app.services.service_health import check_all, health_check_loop, register_services
@@ -52,6 +53,7 @@ app.include_router(admin.router, prefix="/api/admin", tags=["admin"])
app.include_router(groups.router, prefix="/api/admin/groups", tags=["admin"]) app.include_router(groups.router, prefix="/api/admin/groups", tags=["admin"])
app.include_router(settings_router.router, prefix="/api/settings", tags=["settings"]) app.include_router(settings_router.router, prefix="/api/settings", tags=["settings"])
app.include_router(services.router, prefix="/api/services", tags=["services"]) app.include_router(services.router, prefix="/api/services", tags=["services"])
app.include_router(storage_config.router, prefix="/api/admin", tags=["admin"])
app.include_router(plugins.router, prefix="/api/plugins", tags=["plugins"]) app.include_router(plugins.router, prefix="/api/plugins", tags=["plugins"])
# categories_proxy MUST be registered before documents_proxy — # categories_proxy MUST be registered before documents_proxy —
# otherwise /api/documents/{path:path} swallows /api/documents/categories/* # otherwise /api/documents/{path:path} swallows /api/documents/categories/*
+126
View File
@@ -0,0 +1,126 @@
"""
Admin-only endpoints for storage-service backend configuration.
GET /admin/storage-config — current backend driver + health
PATCH /admin/storage-config — update backend config (no data migration)
POST /admin/storage-config/migrate — start migration to a new backend
GET /admin/storage-config/migrate/status — poll migration progress
DELETE /admin/storage-config/migrate — cancel in-progress migration
All endpoints proxy to storage-service:8020.
"""
import logging
import httpx
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from app.core.config import settings
from app.deps import get_current_admin
from app.models.user import User
router = APIRouter()
logger = logging.getLogger(__name__)
_STORAGE_BASE = settings.STORAGE_SERVICE_URL
class BackendConfigUpdate(BaseModel):
driver: str
config: dict = {}
class MigrateRequest(BaseModel):
driver: str
config: dict = {}
def _storage_url(path: str) -> str:
return f"{_STORAGE_BASE}{path}"
async def _proxy_get(path: str) -> dict:
async with httpx.AsyncClient(timeout=15.0) as client:
resp = await client.get(_storage_url(path))
if resp.status_code == 404:
raise HTTPException(status_code=404, detail="Not found")
resp.raise_for_status()
return resp.json()
@router.get("/storage-config")
async def get_storage_config(
_: User = Depends(get_current_admin),
) -> dict:
"""Return current backend driver and health status."""
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.get(_storage_url("/health"))
resp.raise_for_status()
return resp.json()
@router.patch("/storage-config", status_code=204)
async def update_storage_config(
body: BackendConfigUpdate,
_: User = Depends(get_current_admin),
) -> None:
"""
Reconfigure the active backend without migrating data.
Use when changing credentials for the same backend type, or reverting to local.
To move data to a new backend, use POST /admin/storage-config/migrate instead.
"""
async with httpx.AsyncClient(timeout=15.0) as client:
resp = await client.patch(
_storage_url("/backend-config"),
json={"driver": body.driver, "config": body.config},
)
if resp.status_code == 400:
raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed"))
if resp.status_code == 409:
raise HTTPException(status_code=409, detail="Migration in progress — cannot reconfigure now")
resp.raise_for_status()
@router.post("/storage-config/migrate", status_code=202)
async def start_migration(
body: MigrateRequest,
_: User = Depends(get_current_admin),
) -> dict:
"""
Start an async migration to a new backend.
Flow: validate new backend → copy all objects → verify → switch → delete old objects.
The old backend stays active until 100% of objects are verified on the new one.
Poll GET /admin/storage-config/migrate/status to track progress.
"""
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.post(
_storage_url("/migrate"),
json={"driver": body.driver, "config": body.config},
)
if resp.status_code == 400:
raise HTTPException(status_code=400, detail=resp.json().get("detail", "Validation failed"))
if resp.status_code == 409:
raise HTTPException(status_code=409, detail="A migration is already in progress")
resp.raise_for_status()
return resp.json()
@router.get("/storage-config/migrate/status")
async def migration_status(
_: User = Depends(get_current_admin),
) -> dict:
"""Poll migration progress. State: idle → validating → migrating → switching → cleaning → done."""
return await _proxy_get("/migrate/status")
@router.delete("/storage-config/migrate", status_code=204)
async def cancel_migration(
_: User = Depends(get_current_admin),
) -> None:
"""Cancel a running migration. The old backend remains active."""
async with httpx.AsyncClient(timeout=10.0) as client:
resp = await client.delete(_storage_url("/migrate"))
if resp.status_code == 409:
raise HTTPException(status_code=409, detail="No cancellable migration in progress")
resp.raise_for_status()
+60
View File
@@ -0,0 +1,60 @@
# 2026-04-20 — Dedicated storage-service with pluggable backends
**Timestamp:** 2026-04-20T00:00:00Z
## Summary
Introduced a dedicated `storage-service` container (port 8020) as the single file/blob persistence layer for the entire stack. All services now route file and config I/O through this service's HTTP API. The service supports pluggable storage backends (local filesystem by default; S3-compatible and WebDAV built in) with a zero-data-loss migration flow. The `doc_data` and `app_config` Docker volumes were removed.
## Files Added
- `features/storage-service/app/main.py` — FastAPI app, lifespan (backend init)
- `features/storage-service/app/core/config.py` — Settings (DATA_DIR, STORAGE_BACKEND, S3_*, WEBDAV_*)
- `features/storage-service/app/routers/health.py` — GET /health
- `features/storage-service/app/routers/objects.py` — PUT/GET/DELETE /objects/{bucket}/{key:path}, GET /objects/{bucket}
- `features/storage-service/app/routers/migrate.py` — POST/GET/DELETE /migrate, PATCH /backend-config
- `features/storage-service/app/services/backend_manager.py` — Driver factory, singleton, atomic switch
- `features/storage-service/app/services/migration.py` — Async migration: copy → verify → switch → cleanup
- `features/storage-service/app/services/backends/base.py` — AbstractStorageBackend ABC
- `features/storage-service/app/services/backends/local.py` — LocalFSBackend (path traversal guard)
- `features/storage-service/app/services/backends/s3.py` — S3Backend (aiobotocore, endpoint_url configurable)
- `features/storage-service/app/services/backends/webdav.py` — WebDAVBackend (aiohttp + defusedxml)
- `features/storage-service/scripts/start.sh` — prod uvicorn start
- `features/storage-service/scripts/start_dev.sh` — dev uvicorn --reload start
- `features/storage-service/pyproject.toml` — Dependencies
- `features/storage-service/Dockerfile` — python:3.12-slim, non-root user 1001, port 8020
- `features/storage-service/CLAUDE.md` — API reference, bucket docs, driver docs
- `features/storage-service/STATUS.md` — Service status
- `backend/app/core/config_storage.py` — Thin async helpers: read_json/write_json/delete_key/list_keys
- `backend/app/routers/storage_config.py` — Admin proxy endpoints for storage config + migration
- `features/doc-service/alembic/versions/0008_rename_file_path_to_storage_key.py` — DB migration
- `frontend/src/pages/StorageAdminPage.tsx` — Admin UI: backend status, driver form, migration progress
- `tests/storage-service_tests.md` — §20 storage-service test suite
## Files Modified
- `docker-compose.yml` — Added storage-service, storage_data volume; removed doc_data, app_config; added depends_on service_healthy
- `docker-compose.dev.yml` — Added storage-service dev override
- `backend/app/core/config.py` — Added STORAGE_SERVICE_URL
- `backend/app/core/app_config.py` — Full async rewrite using config_storage HTTP helpers (no filesystem)
- `backend/app/routers/settings.py` — Removed all asyncio.to_thread wrappers; direct await calls
- `backend/app/main.py` — Register storage_config router; update register_services call
- `backend/app/services/service_health.py` — Register storage-service
- `features/doc-service/app/core/config.py` — Added STORAGE_SERVICE_URL
- `features/doc-service/app/models/document.py` — file_path → storage_key
- `features/doc-service/app/services/storage.py` — Complete rewrite: HTTP client calls to storage-service
- `features/doc-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket
- `features/doc-service/app/services/file_watcher.py` — Uses save_upload() → storage-service
- `features/doc-service/app/routers/documents.py` — storage_key refs, pdfplumber(io.BytesIO), streaming from storage-service
- `features/ai-service/app/core/config.py` — Added STORAGE_SERVICE_URL; removed CONFIG_PATH
- `features/ai-service/app/services/config_reader.py` — Complete rewrite: reads/writes via storage-service config bucket
- `frontend/src/api/client.ts` — Added StorageStatus, MigrationStatus, StorageBackendConfig interfaces + 5 API functions
- `frontend/src/App.tsx` — Added /admin/storage route (AdminRoute → StorageAdminPage)
- `tests/ALL_TESTS.md` — Updated to 20 feature areas; added §20 storage-service tests
- `CLAUDE.md` — Added storage-service to Services/Volumes/Networks tables; storage enforcement rule; §20 test file
- `backend/CLAUDE.md` — Added config_storage.py, storage_config.py to tree; added admin storage endpoints
- `frontend/CLAUDE.md` — Added StorageAdminPage to tree; added /admin/storage route
- `features/doc-service/CLAUDE.md` — Updated storage.py description; file_path → storage_key; added migration 0008
- `features/ai-service/CLAUDE.md` — Added config_reader.py description
- `backend/STATUS.md` — Added storage-config endpoints; updated settings persistence note
- `frontend/STATUS.md` — Added /admin/storage route; added StorageAdminPage description
+2 -1
View File
@@ -22,7 +22,8 @@ features/ai-service/
│ │ ├── queue.py ← GET /queue/status, /pause, /resume, /cancel/{id} │ │ ├── queue.py ← GET /queue/status, /pause, /resume, /cancel/{id}
│ │ └── plugin.py ← GET /plugin/manifest (access rules for ai-service-admin group) │ │ └── plugin.py ← GET /plugin/manifest (access rules for ai-service-admin group)
│ └── services/ │ └── services/
── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL) ── queue.py ← Priority queue (CRITICAL > HIGH > NORMAL)
│ └── config_reader.py ← Reads ai_service_config.json from storage-service config bucket (30 s TTL cache)
├── Dockerfile ← python:3.12-slim, non-root user 1001 ├── Dockerfile ← python:3.12-slim, non-root user 1001
└── STATUS.md └── STATUS.md
``` ```
+7 -5
View File
@@ -1,6 +1,6 @@
# doc-service — Claude context # doc-service — Claude context
PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows. PDF extraction microservice, port 8001 (internal). Shares the same PostgreSQL instance as the backend. Receives proxied requests from `backend:8000`, which injects `x-user-id` and `x-user-groups` headers — doc-service trusts these headers directly. Calls `ai-service:8010` for document classification. All file/blob storage goes through `storage-service:8020` — no files are written directly to the filesystem. See root `CLAUDE.md` for architecture, Docker, and project-wide workflows.
--- ---
@@ -38,13 +38,14 @@ features/doc-service/
│ │ ├── categories.py ← Category CRUD (includes watch-owned categories) │ │ ├── categories.py ← Category CRUD (includes watch-owned categories)
│ │ └── plugin.py ← GET /plugin/manifest, GET+PATCH /plugin/settings │ │ └── plugin.py ← GET /plugin/manifest, GET+PATCH /plugin/settings
│ └── services/ │ └── services/
│ ├── storage.py ← File I/O │ ├── storage.py ← Storage client: save_upload/download_file/delete_file → storage-service:8020 documents bucket
│ ├── ai_client.py ← classify_document() → ai-service:8010/chat │ ├── ai_client.py ← classify_document() → ai-service:8010/chat
│ ├── config_reader.py ← Config load/save including storage/watch settings │ ├── config_reader.py ← Config load/save via storage-service config bucket (doc_service_config.json)
│ └── file_watcher.py ← watchdog-based PDF watcher + startup scan + ingestion │ └── file_watcher.py ← watchdog-based PDF watcher + startup scan + ingestion
├── alembic/versions/ ← Migration chain ├── alembic/versions/ ← Migration chain
│ ├── 0003_add_watch_columns.py ← source, watch_path, suggested_folder, suggested_filename │ ├── 0003_add_watch_columns.py ← source, watch_path, suggested_folder, suggested_filename
── 0004_add_document_shares.py ← document_shares table (group-based sharing) ── 0004_add_document_shares.py ← document_shares table (group-based sharing)
│ └── 0008_rename_file_path_to_storage_key.py ← file_path → storage_key; strips /data/documents/ prefix from existing rows
├── Dockerfile ← python:3.12-slim, non-root user 1001 ├── Dockerfile ← python:3.12-slim, non-root user 1001
└── STATUS.md └── STATUS.md
``` ```
@@ -60,7 +61,7 @@ features/doc-service/
| `id` | String | PK, UUID | | | `id` | String | PK, UUID | |
| `user_id` | String | indexed | not FK — trusts x-user-id header | | `user_id` | String | indexed | not FK — trusts x-user-id header |
| `filename` | String | NOT NULL | | | `filename` | String | NOT NULL | |
| `file_path` | String | NOT NULL | absolute path under /data/documents | | `storage_key` | String | NOT NULL | storage-service key: `{user_id}/{doc_id}.pdf` (documents bucket) |
| `file_size` | Integer | NOT NULL | bytes | | `file_size` | Integer | NOT NULL | bytes |
| `status` | String | default="pending" | pending / processing / done / failed | | `status` | String | default="pending" | pending / processing / done / failed |
| `title` | String(500) | nullable | AI-extracted | | `title` | String(500) | nullable | AI-extracted |
@@ -118,6 +119,7 @@ Unique constraint: `(document_id, group_id)`
| `0005` | `add_share_can_delete` | | `0005` | `add_share_can_delete` |
| `0006` | `add_category_scope` | | `0006` | `add_category_scope` |
| `0007` | `capitalize_system_category_names` | | `0007` | `capitalize_system_category_names` |
| `0008` | `rename_file_path_to_storage_key` |
--- ---
+3 -1
View File
@@ -37,7 +37,8 @@ frontend/
│ │ └── ui/ ← shadcn/ui components (Button, Input, …) │ │ └── ui/ ← shadcn/ui components (Button, Input, …)
│ ├── pages/ ← One file per route │ ├── pages/ ← One file per route
│ │ ├── DocServiceSettingsPage.tsx ← Combined doc-service settings: upload limits + watch directory │ │ ├── DocServiceSettingsPage.tsx ← Combined doc-service settings: upload limits + watch directory
│ │ ── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest │ │ ── PluginSettingsPage.tsx ← Generic plugin settings page driven by manifest
│ │ └── StorageAdminPage.tsx ← Admin storage backend config + live migration progress
│ ├── lib/utils.ts ← cn() = clsx + tailwind-merge │ ├── lib/utils.ts ← cn() = clsx + tailwind-merge
│ └── styles/theme.css ← CSS custom properties, Tailwind setup │ └── styles/theme.css ← CSS custom properties, Tailwind setup
├── vite.config.ts ← /api/* proxied to backend:8000 ├── vite.config.ts ← /api/* proxied to backend:8000
@@ -66,6 +67,7 @@ frontend/
| `/admin/users` | `AdminUsersPage` | AdminRoute | | `/admin/users` | `AdminUsersPage` | AdminRoute |
| `/admin/groups` | `AdminGroupsPage` | AdminRoute | | `/admin/groups` | `AdminGroupsPage` | AdminRoute |
| `/admin/appearance` | `AdminAppearancePage` | AdminRoute | | `/admin/appearance` | `AdminAppearancePage` | AdminRoute |
| `/admin/storage` | `StorageAdminPage` | AdminRoute |
| `*` | redirect to `/` | — | | `*` | redirect to `/` | — |
`PrivateRoute` — checks `token` from `useAuth`, redirects to `/login` if absent. `PrivateRoute` — checks `token` from `useAuth`, redirects to `/login` if absent.
+6
View File
@@ -21,6 +21,7 @@ All API calls go through `src/api/client.ts` (single Axios instance, JWT injecte
| `/admin` | `AdminPage` (redirects to `/admin/users`) | Admin only | | `/admin` | `AdminPage` (redirects to `/admin/users`) | Admin only |
| `/admin/users` | `AdminUsersPage` | Admin only | | `/admin/users` | `AdminUsersPage` | Admin only |
| `/admin/groups` | `AdminGroupsPage` | Admin only | | `/admin/groups` | `AdminGroupsPage` | Admin only |
| `/admin/storage` | `StorageAdminPage` | Admin only |
| `/profile` | `ProfilePage` | Required | | `/profile` | `ProfilePage` | Required |
| `/settings` | `SettingsPage` (placeholder) | Required | | `/settings` | `SettingsPage` (placeholder) | Required |
| `/settings/plugins/:id` | `PluginSettingsPage` | Required (per-plugin access control) | | `/settings/plugins/:id` | `PluginSettingsPage` | Required (per-plugin access control) |
@@ -114,6 +115,10 @@ Provider selector, per-provider fields, Test Connection, Save.
Upload limits + watch directory config. Upload limits + watch directory config.
### Admin — Storage page (`/admin/storage`)
Current backend status (green/red health dot). Driver selector (local/S3/WebDAV) with conditional credential fields. "Test & Migrate" button triggers an async migration that copies all objects to the new backend, verifies, then switches atomically. Live progress bar with 2s polling (states: validating → migrating → switching → cleaning → done). Cancel button during in-progress migrations.
### Admin — Users page (`/admin/users`) ### Admin — Users page (`/admin/users`)
User list, toggle active, create user, delete user. User list, toggle active, create user, delete user.
@@ -202,6 +207,7 @@ Key document-related functions:
- [x] AI suggestion confirm/reject UI (folder + filename) - [x] AI suggestion confirm/reject UI (folder + filename)
- [x] Groups admin UI - [x] Groups admin UI
- [x] Replace Axios with native fetch; add global 401 → `/login` redirect for expired sessions - [x] Replace Axios with native fetch; add global 401 → `/login` redirect for expired sessions
- [x] Admin storage page with live migration progress bar
- [ ] Toast notification system - [ ] Toast notification system
- [ ] Loading skeletons - [ ] Loading skeletons
- [ ] Cmd+K global search (`CommandDialog`) - [ ] Cmd+K global search (`CommandDialog`)
+2
View File
@@ -16,6 +16,7 @@ import DocServiceSettingsPage from "./pages/DocServiceSettingsPage";
import AIAdminSettingsPage from "./pages/AIAdminSettingsPage"; import AIAdminSettingsPage from "./pages/AIAdminSettingsPage";
import SettingsPage from "./pages/SettingsPage"; import SettingsPage from "./pages/SettingsPage";
import PluginSettingsPage from "./pages/PluginSettingsPage"; import PluginSettingsPage from "./pages/PluginSettingsPage";
import StorageAdminPage from "./pages/StorageAdminPage";
function PrivateRoute({ children }: { children: React.ReactNode }) { function PrivateRoute({ children }: { children: React.ReactNode }) {
const { token } = useAuth(); const { token } = useAuth();
@@ -102,6 +103,7 @@ export default function App() {
<Route path="/admin/users" element={<AdminRoute><AdminUsersPage /></AdminRoute>} /> <Route path="/admin/users" element={<AdminRoute><AdminUsersPage /></AdminRoute>} />
<Route path="/admin/groups" element={<AdminRoute><AdminGroupsPage /></AdminRoute>} /> <Route path="/admin/groups" element={<AdminRoute><AdminGroupsPage /></AdminRoute>} />
<Route path="/admin/appearance" element={<AdminRoute><AdminAppearancePage /></AdminRoute>} /> <Route path="/admin/appearance" element={<AdminRoute><AdminAppearancePage /></AdminRoute>} />
<Route path="/admin/storage" element={<AdminRoute><StorageAdminPage /></AdminRoute>} />
{/* Catch-all */} {/* Catch-all */}
<Route path="*" element={<Navigate to="/" replace />} /> <Route path="*" element={<Navigate to="/" replace />} />
+43
View File
@@ -585,6 +585,49 @@ export interface PluginManifest {
}; };
} }
// ── Storage admin ──────────────────────────────────────────────────────────────
export interface StorageStatus {
status: string;
backend: string;
}
export interface MigrationStatus {
state:
| "idle"
| "validating"
| "migrating"
| "switching"
| "cleaning"
| "done"
| "failed"
| "cancelled";
total: number;
done: number;
failed: number;
errors: string[];
}
export interface StorageBackendConfig {
driver: string;
config: Record<string, string>;
}
export const getStorageConfig = () => api.get<StorageStatus>("/admin/storage-config");
export const updateStorageConfig = (body: StorageBackendConfig) =>
api.patch<void>("/admin/storage-config", body);
export const startStorageMigration = (body: StorageBackendConfig) =>
api.post<{ status: string; driver: string }>("/admin/storage-config/migrate", body);
export const getMigrationStatus = () =>
api.get<MigrationStatus>("/admin/storage-config/migrate/status");
export const cancelMigration = () => api.delete<void>("/admin/storage-config/migrate");
// ── Plugins ────────────────────────────────────────────────────────────────────
export const getPlugins = () => api.get<PluginOut[]>("/plugins"); export const getPlugins = () => api.get<PluginOut[]>("/plugins");
export const getPluginManifest = (id: string) => export const getPluginManifest = (id: string) =>
+382
View File
@@ -0,0 +1,382 @@
import { useState, useEffect, useRef } from "react";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import {
getStorageConfig,
getMigrationStatus,
startStorageMigration,
cancelMigration,
updateStorageConfig,
type StorageBackendConfig,
type MigrationStatus,
} from "../api/client";
type Driver = "local" | "s3" | "webdav";
function Section({ title, children }: { title: string; children: React.ReactNode }) {
return (
<section style={{ marginBottom: 36 }}>
<h2 style={{ fontSize: 18, marginBottom: 16 }}>{title}</h2>
{children}
</section>
);
}
function Field({ label, children }: { label: string; children: React.ReactNode }) {
return (
<div style={{ marginBottom: 12 }}>
<label style={{ display: "block", fontSize: 13, marginBottom: 4, color: "var(--color-text-muted)" }}>
{label}
</label>
{children}
</div>
);
}
function inputStyle(disabled = false): React.CSSProperties {
return {
width: "100%",
padding: "6px 10px",
border: "1px solid var(--color-border)",
borderRadius: 6,
fontSize: 14,
background: disabled ? "var(--color-surface)" : "var(--color-background)",
color: "var(--color-text-primary)",
opacity: disabled ? 0.7 : 1,
};
}
function MigrationProgressBar({ status }: { status: MigrationStatus }) {
const pct = status.total > 0 ? Math.round((status.done / status.total) * 100) : 0;
const isBusy = ["validating", "migrating", "switching", "cleaning"].includes(status.state);
const stateLabel: Record<string, string> = {
idle: "Idle",
validating: "Validating new backend…",
migrating: `Migrating — ${status.done} / ${status.total} objects`,
switching: "Switching active backend…",
cleaning: "Cleaning up old backend…",
done: "Migration complete",
failed: "Migration failed",
cancelled: "Migration cancelled",
};
return (
<div style={{ marginTop: 12 }}>
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: 4, fontSize: 13 }}>
<span style={{ color: status.state === "failed" ? "#dc2626" : "var(--color-text-primary)" }}>
{stateLabel[status.state] ?? status.state}
</span>
{isBusy && <span style={{ color: "var(--color-text-muted)" }}>{pct}%</span>}
</div>
{(isBusy || status.state === "done") && (
<div
style={{
height: 8,
background: "var(--color-border)",
borderRadius: 4,
overflow: "hidden",
}}
>
<div
style={{
height: "100%",
width: status.state === "done" ? "100%" : `${pct}%`,
background: status.state === "done" ? "#16a34a" : "var(--color-primary)",
borderRadius: 4,
transition: "width 0.3s",
}}
/>
</div>
)}
{status.errors.length > 0 && (
<div
style={{
marginTop: 8,
padding: "8px 10px",
background: "#fef2f2",
borderRadius: 6,
fontSize: 12,
color: "#991b1b",
maxHeight: 120,
overflowY: "auto",
}}
>
{status.errors.slice(0, 10).map((e, i) => (
<div key={i} style={{ marginBottom: 2 }}>
{e}
</div>
))}
{status.errors.length > 10 && (
<div style={{ opacity: 0.7 }}>and {status.errors.length - 10} more</div>
)}
</div>
)}
</div>
);
}
export default function StorageAdminPage() {
const queryClient = useQueryClient();
const { data: storageStatus } = useQuery({
queryKey: ["storage-config"],
queryFn: getStorageConfig,
refetchInterval: 10_000,
});
const { data: migStatus, refetch: refetchMig } = useQuery({
queryKey: ["migration-status"],
queryFn: getMigrationStatus,
refetchInterval: (query) => {
const state = query.state.data?.state;
return state && ["validating", "migrating", "switching", "cleaning"].includes(state)
? 2_000
: false;
},
});
const isMigrating =
migStatus &&
["validating", "migrating", "switching", "cleaning"].includes(migStatus.state);
// ── New backend form ─────────────────────────────────────────────────────────
const [driver, setDriver] = useState<Driver>("local");
const [s3EndpointUrl, setS3EndpointUrl] = useState("");
const [s3AccessKey, setS3AccessKey] = useState("");
const [s3SecretKey, setS3SecretKey] = useState("");
const [s3Region, setS3Region] = useState("us-east-1");
const [webdavUrl, setWebdavUrl] = useState("");
const [webdavUsername, setWebdavUsername] = useState("");
const [webdavPassword, setWebdavPassword] = useState("");
const [webdavRootPath, setWebdavRootPath] = useState("/");
const [error, setError] = useState("");
function buildConfig(): StorageBackendConfig {
if (driver === "s3") {
return {
driver,
config: {
endpoint_url: s3EndpointUrl,
access_key: s3AccessKey,
secret_key: s3SecretKey,
region: s3Region,
},
};
}
if (driver === "webdav") {
return {
driver,
config: {
url: webdavUrl,
username: webdavUsername,
password: webdavPassword,
root_path: webdavRootPath,
},
};
}
return { driver: "local", config: {} };
}
const migrateMutation = useMutation({
mutationFn: startStorageMigration,
onSuccess: () => {
setError("");
refetchMig();
},
onError: (e: Error) => setError(e.message),
});
const cancelMutation = useMutation({
mutationFn: cancelMigration,
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["migration-status"] });
queryClient.invalidateQueries({ queryKey: ["storage-config"] });
},
onError: (e: Error) => setError(e.message),
});
const currentDriver = storageStatus?.backend ?? "—";
return (
<div style={{ maxWidth: 680, margin: "0 auto", padding: "32px 16px" }}>
<h1 style={{ fontSize: 24, marginBottom: 4 }}>Storage</h1>
<p style={{ color: "var(--color-text-muted)", marginBottom: 32, fontSize: 14 }}>
All uploaded files are stored through the storage-service. Switch between local filesystem,
S3-compatible cloud storage, or WebDAV (Nextcloud).
</p>
<Section title="Current backend">
<div
style={{
display: "inline-flex",
alignItems: "center",
gap: 8,
padding: "6px 12px",
background: "var(--color-surface)",
border: "1px solid var(--color-border)",
borderRadius: 6,
fontSize: 14,
}}
>
<span
style={{
width: 8,
height: 8,
borderRadius: "50%",
background: storageStatus?.status === "ok" ? "#16a34a" : "#dc2626",
flexShrink: 0,
}}
/>
<strong>{currentDriver}</strong>
{storageStatus?.status === "ok" ? " — healthy" : " — unreachable"}
</div>
</Section>
<Section title="Switch backend">
<p style={{ fontSize: 13, color: "var(--color-text-muted)", marginBottom: 16 }}>
When you click <strong>Test &amp; Migrate</strong>, all existing files will be copied to the
new backend, verified, and the switch will happen only after every file is confirmed. The
old backend is cleaned up automatically.
</p>
<Field label="Backend driver">
<select
value={driver}
onChange={(e) => setDriver(e.target.value as Driver)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
>
<option value="local">Local filesystem (default)</option>
<option value="s3">S3-compatible (MinIO / AWS S3 / Backblaze / Cloudflare R2)</option>
<option value="webdav">WebDAV (Nextcloud / )</option>
</select>
</Field>
{driver === "s3" && (
<>
<Field label="Endpoint URL (leave empty for real AWS S3)">
<input
value={s3EndpointUrl}
onChange={(e) => setS3EndpointUrl(e.target.value)}
placeholder="http://minio:9000"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Access key">
<input
value={s3AccessKey}
onChange={(e) => setS3AccessKey(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Secret key">
<input
type="password"
value={s3SecretKey}
onChange={(e) => setS3SecretKey(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Region">
<input
value={s3Region}
onChange={(e) => setS3Region(e.target.value)}
placeholder="us-east-1"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
</>
)}
{driver === "webdav" && (
<>
<Field label="Server URL">
<input
value={webdavUrl}
onChange={(e) => setWebdavUrl(e.target.value)}
placeholder="https://nextcloud.example.com"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Username">
<input
value={webdavUsername}
onChange={(e) => setWebdavUsername(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="Password">
<input
type="password"
value={webdavPassword}
onChange={(e) => setWebdavPassword(e.target.value)}
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
<Field label="WebDAV root path">
<input
value={webdavRootPath}
onChange={(e) => setWebdavRootPath(e.target.value)}
placeholder="/remote.php/dav/files/username"
disabled={!!isMigrating}
style={inputStyle(!!isMigrating)}
/>
</Field>
</>
)}
{error && (
<p style={{ color: "#dc2626", fontSize: 13, marginBottom: 8 }}>{error}</p>
)}
<div style={{ display: "flex", gap: 10, marginTop: 16 }}>
<button
onClick={() => migrateMutation.mutate(buildConfig())}
disabled={!!isMigrating || migrateMutation.isPending}
style={{
padding: "8px 18px",
background: "var(--color-primary)",
color: "#fff",
border: "none",
borderRadius: 6,
fontSize: 14,
cursor: isMigrating || migrateMutation.isPending ? "not-allowed" : "pointer",
opacity: isMigrating || migrateMutation.isPending ? 0.6 : 1,
}}
>
{migrateMutation.isPending ? "Starting…" : "Test & Migrate"}
</button>
{isMigrating && (
<button
onClick={() => cancelMutation.mutate()}
disabled={cancelMutation.isPending}
style={{
padding: "8px 18px",
background: "transparent",
color: "#dc2626",
border: "1px solid #dc2626",
borderRadius: 6,
fontSize: 14,
cursor: "pointer",
}}
>
Cancel
</button>
)}
</div>
{migStatus && migStatus.state !== "idle" && (
<MigrationProgressBar status={migStatus} />
)}
</Section>
</div>
);
}
+29 -1
View File
@@ -1,11 +1,12 @@
# ALL_TESTS — Full Test Suite # ALL_TESTS — Full Test Suite
Complete test suite covering all 19 feature areas. Run tests relevant to the changed area before merging any feature branch into `main`. Service-specific subsets live in separate files: Complete test suite covering all 20 feature areas. Run tests relevant to the changed area before merging any feature branch into `main`. Service-specific subsets live in separate files:
- `tests/backend_tests.md` — §19, §18 (auth, users, admin, groups, appearance, service health, plugins, AI/doc settings, infra/security) - `tests/backend_tests.md` — §19, §18 (auth, users, admin, groups, appearance, service health, plugins, AI/doc settings, infra/security)
- `tests/frontend_tests.md` — §19 (UI & routing) - `tests/frontend_tests.md` — §19 (UI & routing)
- `tests/doc-service_tests.md` — §1016 (upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory) - `tests/doc-service_tests.md` — §1016 (upload/processing, list/filtering, slide-over, sharing, categories, bulk actions, watch directory)
- `tests/ai-service_tests.md` — §17 (AI queue & providers) - `tests/ai-service_tests.md` — §17 (AI queue & providers)
- `tests/storage-service_tests.md` — §20 (storage-service: objects, backend switching, migration)
Every test describes the exact UI action or API call to perform and the expected outcome. Every test describes the exact UI action or API call to perform and the expected outcome.
@@ -351,3 +352,30 @@ Mark each row before opening the PR.
| 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs | | 19.9 | TanStack Query cache | Navigate away from docs → back | List loads from cache instantly; background refetch runs |
| 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab | | 19.10 | 30s service poll | Leave `/apps` open for 30s | `GET /api/services` fires again in network tab |
| 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off | | 19.11 | Three-dots menu not clipped | Scroll document table → open three-dot actions on any row | Dropdown renders above the table's overflow-hidden container; not cut off |
---
## 20. Storage Service
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored |
| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload |
| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 |
| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` |
| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` |
| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 |
| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent |
| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service |
| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone |
| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket |
| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" |
| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` |
| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss |
| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done |
| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active |
| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" |
| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone |
| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent |
| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent |
| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |
+32
View File
@@ -0,0 +1,32 @@
# Storage Service Tests — §20
Storage-service tests. Run these before merging any change that touches `features/storage-service/`, `docker-compose.yml` storage volumes, or storage-related backend/doc-service code.
See `tests/ALL_TESTS.md` for the full suite and legend.
---
## 20. Storage Service
| # | Test | Steps | Expected |
|---|------|-------|----------|
| 20.1 | Upload object | `PUT /objects/documents/test/file.pdf` with binary body | 204; object stored |
| 20.2 | Download object | `GET /objects/documents/test/file.pdf` after 20.1 | 200; binary content matches upload |
| 20.3 | Delete object | `DELETE /objects/documents/test/file.pdf` | 204; subsequent GET returns 404 |
| 20.4 | List bucket | `GET /objects/documents` | 200; JSON array of keys includes `test/file.pdf` |
| 20.5 | Health endpoint | `GET /health` | `{"status":"ok","backend":"local"}` |
| 20.6 | Path traversal rejected | `PUT /objects/documents/../etc/passwd` | 400 |
| 20.7 | PDF upload via UI | Upload a PDF document | File stored in storage-service under `documents/{user_id}/{doc_id}.pdf`; `doc_data` volume absent |
| 20.8 | PDF download via UI | Download a previously uploaded PDF | File streams correctly from storage-service |
| 20.9 | Document delete via UI | Delete a document | `DELETE /objects/documents/{key}` called; storage-service key gone |
| 20.10 | Config persistence | Restart all containers | `doc_service_config.json` and AI config survive restart in storage-service config bucket |
| 20.11 | Admin storage page | Navigate to `/admin/storage` as admin | Page loads; current backend shows "local — healthy" |
| 20.12 | Non-admin storage page blocked | Navigate to `/admin/storage` as non-admin | Redirected to `/login` |
| 20.13 | Start migration — local to local | Select "Local filesystem" and click "Test & Migrate" | 400 or migration completes instantly; no data loss |
| 20.14 | Migration progress poll | Start a migration | Status badge updates every ~2 s: validating → migrating → done |
| 20.15 | Cancel migration | Start migration; immediately click Cancel | Migration state becomes "cancelled"; old backend remains active |
| 20.16 | Migration conflict | Start a migration while one is running | 409 "A migration is already in progress" |
| 20.17 | Migration — switch to S3 | Configure MinIO credentials; click "Test & Migrate" | All objects copied to S3 bucket; `GET /health` reports `backend: s3`; old local files gone |
| 20.18 | No doc_data volume | `docker volume ls` after full stack up | `doc_data` volume absent |
| 20.19 | No app_config volume | `docker volume ls` after full stack up | `app_config` volume absent |
| 20.20 | Only storage_data volume | Verify `storage_data` volume exists | `docker volume ls` shows `storage_data`; all config and documents in it |