Files

T

curo1305 5349f21752 feat: add storage-service container with pluggable backends (Phase 1)

New FastAPI microservice (port 8020) providing unified blob storage via
PUT/GET/DELETE/LIST HTTP API. Local filesystem backend is the default (zero
extra deps). S3-compatible and WebDAV backends are built in. Backend is
switchable at runtime via POST /migrate, which copies all objects to the new
backend, verifies each one, atomically switches, then cleans up the old backend.

WebDAV XML parsing uses defusedxml to prevent XXE attacks.

Wired into docker-compose (storage_data volume) and registered in the backend
service-health poller as 'storage-service'.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-20 15:50:31 +02:00

2.9 KiB

Raw Blame History

Storage Service — Status

What it is

Unified file/blob storage microservice, port 8020 (internal). All services store and retrieve files through its HTTP API — no service writes to a Docker volume directly. Uses a pluggable backend driver (local FS by default; S3-compatible and WebDAV available). Backend is switchable at runtime via admin settings with automatic data migration.

Current functionality

Object API (`/objects`)

Method	Path	Description
PUT	`/objects/{bucket}/{key}`	Upload raw bytes
GET	`/objects/{bucket}/{key}`	Download raw bytes
DELETE	`/objects/{bucket}/{key}`	Delete object
GET	`/objects/{bucket}`	List all keys in bucket

Migration API (`/migrate`)

Method	Path	Description
POST	`/migrate`	Start migration to a new backend (validates, copies, switches, cleans)
GET	`/migrate/status`	Poll migration progress
DELETE	`/migrate`	Cancel in-progress migration
PATCH	`/backend-config`	Reconfigure backend without migrating data

Health

Method	Path	Description
GET	`/health`	`{"status": "ok", "backend": "<driver>"}`

Buckets

Bucket	Contents
`documents`	Uploaded PDFs (keyed as `{user_id}/{doc_id}.pdf` or `watch/{doc_id}.pdf`)
`config`	JSON config files (replaces `app_config` volume — Phase 3)

Architecture

backend / doc-service / future-svc
           │  HTTP
           ▼
    storage-service:8020
           │
    backend_manager
           │
    ┌──────┴──────────────────┐
    │                         │
  LocalFSBackend         S3Backend / WebDAVBackend
  /data/storage/         (configured via admin UI)
  {bucket}/{key}

Migration flow:

POST /migrate { driver, config }
  → test_connection()                  (validate)
  → list all objects in KNOWN_BUCKETS  (enumerate)
  → GET old / PUT new / exists verify  (copy + verify, per object)
  → if 0 failures: switch_backend()    (atomic switch)
  → DELETE old objects                 (cleanup)

Known limitations / not implemented

Migration state is in-memory — a container restart during migration loses progress (restart restarts from scratch)
No presigned URL support (direct client downloads go through the API)
rclone backends (Google Drive, OneDrive, Dropbox) not yet implemented
No per-object metadata or content-type headers
No multipart upload for very large files (> available RAM)

Future work

rclone-based backend adapter (GDrive, OneDrive, Dropbox)
Presigned URL generation for direct browser downloads
Persist migration state to DB so restarts can resume
Streaming upload/download to avoid buffering entire file in memory
Per-bucket access policies

2.9 KiB Raw Blame History