Add generic plugin architecture and watch-directory feature

Introduces a manifest contract so feature containers self-describe their settings (JSON Schema + access rules). Backend and frontend gain generic plugin proxy and dynamic Extensions UI with zero feature-specific code. Doc-service is the first plugin consumer: exposes /plugin/manifest and /plugin/settings, adds a watchdog-based file watcher that auto-ingests PDFs from a mounted directory, maps subfolders to categories, supports AI-suggested folder/filename (user-confirmed), and enforces a no-remove policy. Access is gated by is_superuser or doc-service-admin group. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 02:09:50 +02:00
parent 2d7207b62f
commit 00466a9801
29 changed files with 1373 additions and 52 deletions
@@ -78,12 +78,13 @@ docker compose up --build -d
 ├── .githooks/pre-commit                   ← Runs scripts/security_check.py before every commit
 ├── scripts/security_check.py             ← Static analysis: secrets, weak crypto, SQLi, JWT
 ├── changelog/YYYY-MM-DD_<slug>.md        ← Per-date change logs
+├── dev-watch/                             ← Dev bind-mount for file watcher testing (.gitkeep only)
 │
 ├── backend/                               ← FastAPI gateway (port 8000, internal)
 │   ├── app/
 │   │   ├── main.py                        ← App factory, router registration, lifespan (health loop)
 │   │   ├── database.py                    ← AsyncEngine, AsyncSessionLocal, Base
-│   │   ├── deps.py                        ← get_current_user, get_current_admin
+│   │   ├── deps.py                        ← get_current_user, get_current_admin, check_plugin_access
 │   │   ├── core/
 │   │   │   ├── config.py                  ← All settings via pydantic-settings (reads .env)
 │   │   │   ├── security.py                ← JWT sign/verify (RS256), bcrypt hash/verify
@@ -106,10 +107,11 @@ docker compose up --build -d
 │   │   │   ├── groups.py                  ← Group CRUD + member management (admin-only)
 │   │   │   ├── settings.py                ← AI, doc limits, system prompts, appearance, themes (admin-only)
 │   │   │   ├── services.py                ← GET /services (health status)
+│   │   │   ├── plugins.py                 ← Generic plugin proxy (GET/PATCH /api/plugins/*)
 │   │   │   ├── categories_proxy.py        ← Transparent proxy → doc-service /categories/*
 │   │   │   └── documents_proxy.py         ← Transparent proxy → doc-service /documents/*
 │   │   └── services/
-│   │       └── service_health.py          ← Background 30s health-check loop
+│   │       └── service_health.py          ← Background 30s health-check loop; caches /plugin/manifest per service
 │   ├── alembic/
 │   │   ├── env.py                         ← Async migration runner
 │   │   └── versions/                      ← Migration chain (see Migrations section)
@@ -133,7 +135,7 @@ docker compose up --build -d
 │   │
 │   └── doc-service/                       ← PDF extraction microservice (port 8001, internal)
 │       ├── app/
-│       │   ├── main.py
+│       │   ├── main.py                    ← FastAPI, lifespan (file watcher start/stop)
 │       │   ├── database.py                ← Same PostgreSQL instance as backend
 │       │   ├── deps.py                    ← get_user_id (reads x-user-id header)
 │       │   ├── models/
@@ -144,13 +146,16 @@ docker compose up --build -d
 │       │   │   ├── document.py            ← DocumentOut, DocumentPage, DocumentStatusOut, etc.
 │       │   │   └── category.py            ← CategoryOut, CategoryCreate, CategoryUpdate
 │       │   ├── routers/
-│       │   │   ├── documents.py           ← Full document CRUD + file serving + reprocess
-│       │   │   └── categories.py          ← Category CRUD
+│       │   │   ├── documents.py           ← Full document CRUD + file serving + reprocess + suggestion endpoints
+│       │   │   ├── categories.py          ← Category CRUD (includes watch-owned categories)
+│       │   │   └── plugin.py              ← GET /plugin/manifest, GET+PATCH /plugin/settings
 │       │   └── services/
 │       │       ├── storage.py             ← File I/O
 │       │       ├── ai_client.py           ← classify_document() → ai-service:8010/chat
-│       │       └── config_reader.py
+│       │       ├── config_reader.py       ← Config load/save including storage/watch settings
+│       │       └── file_watcher.py        ← watchdog-based PDF watcher + startup scan + ingestion
 │       ├── alembic/versions/              ← Doc-service migration chain
+│       │   └── 0003_add_watch_columns.py  ← source, watch_path, suggested_folder, suggested_filename
 │       ├── Dockerfile
 │       └── STATUS.md
 │
@@ -164,10 +169,12 @@ docker compose up --build -d
    │   │   └── useTheme.ts                ← Theme toggle
    │   ├── components/
    │   │   ├── AppShell.tsx               ← Layout: Sidebar + scrollable main
-    │   │   ├── Sidebar.tsx                ← Collapsible nav (icons ↔ icons+labels)
+    │   │   ├── Sidebar.tsx                ← Collapsible nav; "Extensions" section auto-populated from /api/plugins
    │   │   ├── ThemeToggle.tsx            ← Light/dark mode toggle
+    │   │   ├── PluginSchemaForm.tsx       ← JSON Schema → React form (boolean/string/number/readOnly)
    │   │   └── ui/                        ← shadcn/ui components (Button, Input, …)
    │   ├── pages/                         ← One file per route (see Routes section)
+    │   │   └── PluginSettingsPage.tsx     ← Generic plugin settings page driven by manifest
    │   ├── lib/utils.ts                   ← cn() = clsx + tailwind-merge
    │   └── styles/theme.css               ← CSS custom properties, Tailwind setup
    ├── vite.config.ts                     ← /api/* proxied to backend:8000
@@ -283,6 +290,10 @@ Unique constraint: `(group_id, user_id)`
 | `error_message` | String(500) | nullable | |
 | `created_at` | DateTime(tz) | server_default=now() | |
 | `processed_at` | DateTime(tz) | nullable | |
+| `source` | String(16) | default="upload" | "upload" or "watch" |
+| `watch_path` | String | nullable | original absolute path in watch directory |
+| `suggested_folder` | String(128) | nullable | AI-suggested category (pending user confirm) |
+| `suggested_filename` | String(500) | nullable | AI-suggested title/rename (pending user confirm) |

 **`document_categories`**

@@ -318,6 +329,7 @@ Unique constraint: `(group_id, user_id)`
 |--------|------|
 | `0001` | `create_doc_tables` |
 | `0002` | `add_document_title` |
+| `0003` | `add_watch_columns` |

 ---

@@ -407,6 +419,10 @@ Unique constraint: `(group_id, user_id)`
 | GET | `/api/documents/{id}/file` | Download PDF (streaming) |
 | POST | `/api/documents/{id}/categories/{cat_id}` | Assign category |
 | DELETE | `/api/documents/{id}/categories/{cat_id}` | Remove category |
+| POST | `/api/documents/{id}/suggestions/folder/confirm` | Confirm AI folder suggestion |
+| POST | `/api/documents/{id}/suggestions/folder/reject` | Reject AI folder suggestion |
+| POST | `/api/documents/{id}/suggestions/filename/confirm` | Confirm AI filename suggestion |
+| POST | `/api/documents/{id}/suggestions/filename/reject` | Reject AI filename suggestion |

 ### Categories (`/api/documents/categories/*`) — authenticated, proxied to doc-service

@@ -417,6 +433,17 @@ Unique constraint: `(group_id, user_id)`
 | PATCH | `/api/documents/categories/{id}` | Rename |
 | DELETE | `/api/documents/categories/{id}` | Delete (204) |

+### Plugins (`/api/plugins`) — authenticated, auth-per-plugin
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/plugins` | List plugins accessible to current user |
+| GET | `/api/plugins/{id}/manifest` | Plugin manifest with settings JSON Schema (auth-gated) |
+| GET | `/api/plugins/{id}/settings` | Proxy to feature `/plugin/settings` (auth-gated) |
+| PATCH | `/api/plugins/{id}/settings` | Proxy to feature `/plugin/settings` (auth-gated) |
+
+Auth: is_superuser OR member of group listed in manifest `required_groups`. Returns 404 (not 403) to hide existence.
+
 ### AI-service (internal only — not exposed to browser)

 | Method | Path | Description |
@@ -442,6 +469,7 @@ Unique constraint: `(group_id, user_id)`
 | `/apps/ai/settings/admin` | `AIAdminSettingsPage` | AdminRoute |
 | `/profile` | `ProfilePage` | PrivateRoute |
 | `/settings` | `SettingsPage` | PrivateRoute |
+| `/settings/plugins/:id` | `PluginSettingsPage` | PrivateRoute (auth enforced per-plugin by backend) |
 | `/admin` | `AdminPage` (→ `/admin/users`) | AdminRoute |
 | `/admin/users` | `AdminUsersPage` | AdminRoute |
 | `/admin/groups` | `AdminGroupsPage` | AdminRoute |
@@ -566,6 +594,9 @@ Adding a new API call:
 ["categories"]            // document categories
 ["documents", params]     // document list (params object for cache isolation)
 ["document", id]          // single document
+["plugins"]               // accessible plugin list (filtered by user access)
+["plugin-manifest", id]   // plugin manifest (cached)
+["plugin-settings", id]   // plugin current settings
 ```

 **Mutation pattern**:
@@ -708,7 +739,7 @@ Use `validation_alias` when the ORM field name differs from the JSON key (e.g.,
 | `db` | postgres:16-alpine | 5432 | 70:70 | `postgres_data` | backend-net |
 | `backend` | python:3.12-slim | 8000 | 1001:1001 | `app_config` | backend-net |
 | `ai-service` | python:3.12-slim | 8010 | 1001:1001 | `app_config` | backend-net |
-| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `doc_data`, `app_config` | backend-net |
+| `doc-service` | python:3.12-slim | 8001 | 1001:1001 | `doc_data`, `watch_data`, `app_config` | backend-net |
 | `frontend` | nginx-unprivileged:alpine | 8080 | 1001:1001 | — | backend-net, frontend-net |

 ### Volumes
@@ -717,6 +748,7 @@ Use `validation_alias` when the ORM field name differs from the JSON key (e.g.,
 |--------|-----------|---------|
 | `postgres_data` | `/var/lib/postgresql/data` | PostgreSQL data |
 | `doc_data` | `/data/documents` | Uploaded PDF files |
+| `watch_data` | `/data/watch` | Watch directory (bind-mount NAS/Nextcloud via docker-compose.override.yml) |
 | `app_config` | `/config` | Per-service runtime config JSON files |

 ### Networks
@@ -816,6 +848,100 @@ Always run `git push` immediately after every `git commit`.

 ---

+### Feature branch & isolated test environment
+
+Every non-trivial implementation (anything beyond a one-line fix or doc change) **must** follow this workflow:
+
+#### 1 — Create a feature branch
+After the planning phase is approved, branch off `main`:
+```bash
+git checkout main && git pull
+git checkout -b feat/<slug>   # e.g. feat/color-mode, feat/admin-appearance
+```
+
+#### 2 — Spin up an isolated Docker stack for the feature
+A dedicated compose stack runs alongside the main dev stack so both can be tested independently.
+
+**Find the next free port** (main dev stack owns 5173):
+```bash
+for port in $(seq 5174 5200); do
+  lsof -iTCP:$port -sTCP:LISTEN -t &>/dev/null || { echo "$port"; break; }
+done
+```
+Use the first free port returned (call it `$PORT`).
+
+**Create a per-feature override file** at `docker-compose.feat-<slug>.yml` (gitignored):
+```yaml
+# docker-compose.feat-<slug>.yml  — feature test stack, never committed to main
+services:
+  frontend:
+    ports:
+      - "$PORT:8080"        # e.g. 5174:8080
+    container_name: frontend-<slug>
+  backend:
+    container_name: backend-<slug>
+  doc-service:
+    container_name: doc-service-<slug>
+  ai-service:
+    container_name: ai-service-<slug>
+  db:
+    container_name: db-<slug>
+
+networks:
+  backend-net:
+    name: backend-net-<slug>
+  frontend-net:
+    name: frontend-net-<slug>
+```
+
+**Start the feature stack**:
+```bash
+docker compose -f docker-compose.yml \
+               -f docker-compose.dev.yml \
+               -f docker-compose.feat-<slug>.yml \
+  --project-name <slug> up --build
+```
+
+The feature frontend is now reachable at `http://localhost:$PORT`.
+The main dev stack continues running unaffected on `:5173`.
+
+#### 3 — Develop on the feature branch
+All code changes happen on `feat/<slug>`. Commit and push normally:
+```bash
+git add <files>
+git commit -m "feat: <description>"
+git push -u origin feat/<slug>
+```
+
+#### 4 — Confirm functionality
+Before merging, verify all of the following on `http://localhost:$PORT`:
+- [ ] Login and registration work end-to-end
+- [ ] The specific feature works as intended
+- [ ] No regressions visible in the UI
+- [ ] Backend logs show no unexpected errors: `docker compose -p <slug> logs backend`
+- [ ] Migrations (if any) applied cleanly: `docker compose -p <slug> exec backend alembic upgrade head`
+
+#### 5 — Merge to main
+Once all checks pass:
+```bash
+git checkout main
+git merge --no-ff feat/<slug> -m "Merge feat/<slug>: <description>"
+git push
+git branch -d feat/<slug>
+git push origin --delete feat/<slug>
+```
+
+#### 6 — Tear down the feature stack
+```bash
+docker compose -f docker-compose.yml \
+               -f docker-compose.dev.yml \
+               -f docker-compose.feat-<slug>.yml \
+  --project-name <slug> down --volumes --remove-orphans
+rm docker-compose.feat-<slug>.yml
+```
+
+---
+
 ### Infrastructure change protocol

 After **any** change to Dockerfiles, `docker-compose*.yml`, `nginx.conf`, or setup scripts: