Compare commits

...

5 Commits

Author SHA1 Message Date
curo1305 0f760c379d fix: remove obsolete /data/documents and /config dirs from Dockerfiles
doc-service and ai-service no longer use local filesystem directories —
all file and config I/O goes through storage-service. Update README and
CLAUDE.md to reflect 6-service architecture, new volumes, and add
storage-service step to the "Adding a new resource" checklist.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 13:45:12 +02:00
curo1305 f13ef88711 Merge feat/storage-service: dedicated storage service with pluggable backends
All file/blob persistence now routes through storage-service (port 8020).
Replaces doc_data and app_config volumes. Supports local (default), S3-compatible,
and WebDAV backends with zero-data-loss migration flow.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 13:41:26 +02:00
curo1305 0d8e0366c6 docs: always use port 5173 for feature stacks (no per-branch ports)
Update feature branch workflow: stop main stack before starting feature
stack, always use :5173. Simplify feat override template — no port
remapping needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 11:48:22 +02:00
curo1305 3a66aeeec5 fix: rename download_file import to storage_download to avoid shadow
The route handler async def download_file() shadowed the storage import
of the same name, causing the endpoint to call itself recursively.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 11:48:04 +02:00
curo1305 248b2bb9d7 fix: remove unused imports in StorageAdminPage
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 11:25:59 +02:00
6 changed files with 29 additions and 33 deletions
+12 -14
View File
@@ -292,8 +292,9 @@ Each entry must include:
4. Add router in `backend/app/routers/`, mount it in `main.py`
5. Add API function(s) to `frontend/src/api/client.ts`
6. Add page component in `frontend/src/pages/`, register route in `App.tsx`
7. Update `STATUS.md` for affected services
8. Add changelog entry
7. If the resource involves file or blob data: store it via `PUT /objects/{bucket}/{key}` on `storage-service:8020`. Never write to the local filesystem. See `features/storage-service/CLAUDE.md` for the API.
8. Update `STATUS.md` for affected services
9. Add changelog entry
---
@@ -329,23 +330,18 @@ git checkout -b feat/<descriptive-title> # e.g. feat/user-profile-avatar-uploa
```
#### 2 — Spin up an isolated Docker stack for the feature
A dedicated compose stack runs alongside the main dev stack so both can be tested independently.
The feature stack always uses port `5173` (same as the main dev stack). Stop the main stack before starting a feature stack, and restart it when done.
**Find the next free port** (main dev stack owns 5173):
**Stop the main dev stack first:**
```bash
for port in $(seq 5174 5200); do
lsof -iTCP:$port -sTCP:LISTEN -t &>/dev/null || { echo "$port"; break; }
done
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
```
Use the first free port returned (call it `$PORT`).
**Create a per-feature override file** at `docker-compose.feat-<slug>.yml` (gitignored):
```yaml
# docker-compose.feat-<slug>.yml — feature test stack, never committed to main
services:
frontend:
ports:
- "$PORT:8080" # e.g. 5174:8080
container_name: frontend-<slug>
backend:
container_name: backend-<slug>
@@ -371,8 +367,7 @@ docker compose -f docker-compose.yml \
--project-name <slug> up --build
```
The feature frontend is now reachable at `http://localhost:$PORT`.
The main dev stack continues running unaffected on `:5173`.
The feature frontend is now reachable at `http://localhost:5173`.
#### 3 — Develop on the feature branch
All code changes happen on `feat/<slug>`. Commit and push normally:
@@ -383,7 +378,7 @@ git push -u origin feat/<slug>
```
#### 4 — Confirm functionality
Before merging, verify all of the following on `http://localhost:$PORT`:
Before merging, verify all of the following on `http://localhost:5173`:
- [ ] Login and registration work end-to-end
- [ ] The specific feature works as intended
- [ ] No regressions visible in the UI
@@ -400,13 +395,16 @@ git branch -d feat/<slug>
git push origin --delete feat/<slug>
```
#### 6 — Tear down the feature stack
#### 6 — Tear down the feature stack and restart main dev stack
```bash
docker compose -f docker-compose.yml \
-f docker-compose.dev.yml \
-f docker-compose.feat-<slug>.yml \
--project-name <slug> down --volumes --remove-orphans
rm docker-compose.feat-<slug>.yml
# Restart the main dev stack on :5173
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build -d
```
---
+9 -9
View File
@@ -20,10 +20,11 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
- All input sanitized before reaching the DB (null-byte rejection, length caps, format validation)
- **PDF Documents app** (`/apps/documents`): upload PDFs, async text extraction (pdfplumber), AI classification via ai-service, per-user categories, file download
- **AI Service** (`ai-service:8010`): shared AI intermediary container; routes prompts to Anthropic / Ollama / LM Studio; stateless; all feature containers talk to it via `POST /chat`
- Admin settings: `/apps/ai/settings/admin` (provider, credentials, test connection); `/apps/documents/settings/admin` (upload limits only)
- Config stored in shared Docker volume: `/config/ai_service_config.json` and `/config/doc_service_config.json`
- **Storage Service** (`storage-service:8020`): unified file/blob storage with pluggable backends (local filesystem default; S3-compatible and WebDAV built in); backend switchable via admin UI with zero-data-loss migration
- Admin settings: AI provider, doc upload limits, storage backend switching with live migration progress
- Config stored in storage-service (`config` bucket); PDFs stored in storage-service (`documents` bucket) — no shared filesystem volumes
- `/apps` launcher hub — one card per installed app with Open + Settings links
- 5 separate Docker containers: `db`, `backend`, `ai-service`, `doc-service`, `frontend`
- 6 separate Docker containers: `db`, `backend`, `ai-service`, `doc-service`, `storage-service`, `frontend`
- All containers run as non-root users (UID 1001 for app containers, UID 70 for db)
- Network-isolated: only the frontend exposes a host port (80/5173); all backend services are unreachable from outside Docker
- Dev environment seeds a test user automatically on startup (`test@example.com` / `Test123!`)
@@ -38,6 +39,7 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
| `backend` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | FastAPI management API + proxy to doc-service |
| `ai-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | Shared AI intermediary (routes to LM Studio / Ollama / Anthropic) |
| `doc-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | PDF extraction microservice (calls ai-service) |
| `storage-service` | custom (python:3.12-slim) | none | backend-net | 1001:1001 | Unified file/blob storage (local / S3-compatible / WebDAV) |
| `frontend` | custom (nginxinc/nginx-unprivileged:alpine) | 80 (prod) / 5173 (dev) | backend-net + frontend-net | 1001:1001 | React UI + nginx reverse proxy |
**Networks:**
@@ -46,8 +48,8 @@ A fullstack SaaS web application built with FastAPI, React, and PostgreSQL.
**Volumes:**
- `postgres_data` — PostgreSQL data files
- `doc_data`uploaded PDF files (mounted into doc-service at `/data/documents`)
- `app_config` — per-service runtime config JSON files (mounted into backend, ai-service, and doc-service at `/config`)
- `storage_data`all file/blob storage: uploaded PDFs (`documents/` bucket) and service config JSON files (`config/` bucket); mounted into storage-service at `/data/storage`
- `watch_data` — file watcher input directory; mounted into doc-service at `/data/watch`
The frontend nginx proxies `/api/*` to `backend:8000` via `backend-net`. The backend proxies `/api/documents/*` and `/api/documents/categories/*` to `doc-service:8001`. The backend test-connection endpoint proxies to `ai-service:8010`. No backend service or database port is ever exposed to the host.
@@ -126,10 +128,8 @@ Copy `.env.example` to `backend/.env` and adjust:
| `JWT_PRIVATE_KEY` | — | RS256 private key PEM (generate with `scripts/generate_jwt_keys.py`) |
| `JWT_PUBLIC_KEY` | — | RS256 public key PEM (generate with `scripts/generate_jwt_keys.py`) |
| `CORS_ORIGINS` | `["http://localhost:5173"]` | Allowed frontend origins |
| `APP_CONFIG_DIR` | `/config` | Directory for per-service runtime config JSON files |
| `DOC_SERVICE_URL` | `http://doc-service:8001` | Internal URL of the doc-service (set by docker-compose) |
`doc-service` reads `DATABASE_URL`, `DATA_DIR`, and `CONFIG_PATH` from its own environment (set in `docker-compose.yml`).
| `DOC_SERVICE_URL` | `http://doc-service:8001` | Internal URL of doc-service (set by docker-compose) |
| `STORAGE_SERVICE_URL` | `http://storage-service:8020` | Internal URL of storage-service (set by docker-compose) |
## Development
+1 -2
View File
@@ -15,8 +15,7 @@ FROM python:3.12-slim
RUN groupadd --gid 1001 appuser && \
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
# Pre-create the config directory with correct ownership
RUN mkdir -p /config && chown -R appuser:appuser /config
# No filesystem directories needed — all config goes through storage-service.
WORKDIR /app
+3 -3
View File
@@ -15,9 +15,9 @@ FROM python:3.12-slim
RUN groupadd --gid 1001 appuser && \
useradd --uid 1001 --gid 1001 --no-create-home --shell /bin/sh appuser
# Pre-create data and config dirs with correct ownership.
# Named volumes mounted over these paths will inherit ownership on first creation.
RUN mkdir -p /data/documents /data/watch /config && chown -R appuser:appuser /data /config
# Pre-create watch dir with correct ownership.
# /data/documents and /config are no longer used — all file/config storage goes through storage-service.
RUN mkdir -p /data/watch && chown -R appuser:appuser /data
WORKDIR /app
@@ -29,7 +29,7 @@ from app.schemas.document import (
from app.schemas.share import DocumentShareCreate, DocumentShareOut, SharedDocumentOut
from app.services.ai_client import AIServiceError, classify_document
from app.services.config_reader import load_doc_config
from app.services.storage import delete_file, download_file, save_upload
from app.services.storage import delete_file, download_file as storage_download, save_upload
router = APIRouter()
@@ -146,7 +146,7 @@ async def process_document(doc_id: str) -> None:
await db.commit()
try:
pdf_bytes = await download_file(doc.storage_key)
pdf_bytes = await storage_download(doc.storage_key)
text = await asyncio.to_thread(_extract_pdf_text, pdf_bytes)
result = await classify_document(text)
@@ -611,7 +611,7 @@ async def download_file(
raise HTTPException(status_code=404, detail="Document not found")
try:
pdf_bytes = await download_file(doc.storage_key)
pdf_bytes = await storage_download(doc.storage_key)
except FileNotFoundError:
raise HTTPException(status_code=404, detail="File not found in storage")
+1 -2
View File
@@ -1,11 +1,10 @@
import { useState, useEffect, useRef } from "react";
import { useState } from "react";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import {
getStorageConfig,
getMigrationStatus,
startStorageMigration,
cancelMigration,
updateStorageConfig,
type StorageBackendConfig,
type MigrationStatus,
} from "../api/client";