curo1305
5349f21752
New FastAPI microservice (port 8020) providing unified blob storage via PUT/GET/DELETE/LIST HTTP API. Local filesystem backend is the default (zero extra deps). S3-compatible and WebDAV backends are built in. Backend is switchable at runtime via POST /migrate, which copies all objects to the new backend, verifies each one, atomically switches, then cleans up the old backend. WebDAV XML parsing uses defusedxml to prevent XXE attacks. Wired into docker-compose (storage_data volume) and registered in the backend service-health poller as 'storage-service'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
84 lines
2.9 KiB
Markdown
84 lines
2.9 KiB
Markdown
# Storage Service — Status
|
|
|
|
## What it is
|
|
Unified file/blob storage microservice, port 8020 (internal). All services store and retrieve files
|
|
through its HTTP API — no service writes to a Docker volume directly. Uses a pluggable backend
|
|
driver (local FS by default; S3-compatible and WebDAV available). Backend is switchable at runtime
|
|
via admin settings with automatic data migration.
|
|
|
|
## Current functionality
|
|
|
|
### Object API (`/objects`)
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| PUT | `/objects/{bucket}/{key}` | Upload raw bytes |
|
|
| GET | `/objects/{bucket}/{key}` | Download raw bytes |
|
|
| DELETE | `/objects/{bucket}/{key}` | Delete object |
|
|
| GET | `/objects/{bucket}` | List all keys in bucket |
|
|
|
|
### Migration API (`/migrate`)
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| POST | `/migrate` | Start migration to a new backend (validates, copies, switches, cleans) |
|
|
| GET | `/migrate/status` | Poll migration progress |
|
|
| DELETE | `/migrate` | Cancel in-progress migration |
|
|
| PATCH | `/backend-config` | Reconfigure backend without migrating data |
|
|
|
|
### Health
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| GET | `/health` | `{"status": "ok", "backend": "<driver>"}` |
|
|
|
|
### Buckets
|
|
|
|
| Bucket | Contents |
|
|
|--------|----------|
|
|
| `documents` | Uploaded PDFs (keyed as `{user_id}/{doc_id}.pdf` or `watch/{doc_id}.pdf`) |
|
|
| `config` | JSON config files (replaces `app_config` volume — Phase 3) |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
backend / doc-service / future-svc
|
|
│ HTTP
|
|
▼
|
|
storage-service:8020
|
|
│
|
|
backend_manager
|
|
│
|
|
┌──────┴──────────────────┐
|
|
│ │
|
|
LocalFSBackend S3Backend / WebDAVBackend
|
|
/data/storage/ (configured via admin UI)
|
|
{bucket}/{key}
|
|
```
|
|
|
|
Migration flow:
|
|
```
|
|
POST /migrate { driver, config }
|
|
→ test_connection() (validate)
|
|
→ list all objects in KNOWN_BUCKETS (enumerate)
|
|
→ GET old / PUT new / exists verify (copy + verify, per object)
|
|
→ if 0 failures: switch_backend() (atomic switch)
|
|
→ DELETE old objects (cleanup)
|
|
```
|
|
|
|
## Known limitations / not implemented
|
|
|
|
- Migration state is in-memory — a container restart during migration loses progress (restart restarts from scratch)
|
|
- No presigned URL support (direct client downloads go through the API)
|
|
- rclone backends (Google Drive, OneDrive, Dropbox) not yet implemented
|
|
- No per-object metadata or content-type headers
|
|
- No multipart upload for very large files (> available RAM)
|
|
|
|
## Future work
|
|
|
|
- [ ] rclone-based backend adapter (GDrive, OneDrive, Dropbox)
|
|
- [ ] Presigned URL generation for direct browser downloads
|
|
- [ ] Persist migration state to DB so restarts can resume
|
|
- [ ] Streaming upload/download to avoid buffering entire file in memory
|
|
- [ ] Per-bucket access policies
|