Business-Management/features/doc-service/STATUS.md

# Doc Service — Status

## What it is

PDF document management microservice. Handles upload, storage, async AI-powered extraction, tagging, categorisation, and retrieval of PDF documents on a per-user basis. Also supports automatic ingestion from a mounted watch directory (NAS, Nextcloud, Syncthing, etc.).

Port: `8001` (internal only, not exposed to host). All traffic arrives via the backend proxy (`backend/app/routers/documents_proxy.py`), which injects the authenticated `x-user-id` header.

Database: shared PostgreSQL instance, isolated via `alembic_version_doc_service` Alembic version table. Storage: `/data/documents/` (Docker named volume `doc_data`). Watch directory: `/data/watch` (named volume `watch_data` in prod; bind-mount in dev via `docker-compose.dev.yml`).

---

## Current functionality

### Document lifecycle

1. `POST /documents/upload` — validate PDF, persist file to `/data/documents/{user_id}/{doc_id}.pdf`, create DB row with `status=pending`, enqueue background extraction
2. Background task: extract text with `pdfplumber` → POST to ai-service `/chat` → parse JSON result → update `status=done` (or `failed`)
3. AI extracts: `title`, `document_type`, `tags`, `suggested_categories`, plus domain fields (vendor, customer, dates, amounts, etc.) into `extracted_data` (JSON string)

### Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/documents/upload` | Upload PDF; returns 202 with initial doc row |
| `GET` | `/documents` | Paginated list with filters, sort, and optional `category_id` filter |
| `GET` | `/documents/{id}` | Single document |
| `GET` | `/documents/{id}/status` | Lightweight status poll |
| `GET` | `/documents/{id}/download` | Stream file bytes |
| `DELETE` | `/documents/{id}` | Delete document and file |
| `PATCH` | `/documents/{id}/type` | Update document type |
| `PATCH` | `/documents/{id}/tags` | Replace tag list (dedup, preserve order) |
| `PATCH` | `/documents/{id}/title` | Update editable title |
| `GET` | `/documents/categories` | List all categories (user + watch) |
| `POST` | `/documents/categories` | Create a category; triggers re-analysis of documents in similar categories |
| `POST` | `/documents/{id}/reprocess` | Reset status to pending and re-run AI extraction; 409 if already pending/processing |
| `PATCH` | `/documents/categories/{id}` | Rename a category |
| `DELETE` | `/documents/categories/{id}` | Delete a category |
| `POST` | `/documents/{id}/categories/{cat_id}` | Assign category to document |
| `DELETE` | `/documents/{id}/categories/{cat_id}` | Remove category from document |
| `POST` | `/documents/{id}/suggestions/folder/confirm` | Apply AI folder suggestion → create/find category + assign |
| `POST` | `/documents/{id}/suggestions/folder/reject` | Clear AI folder suggestion |
| `POST` | `/documents/{id}/suggestions/filename/confirm` | Apply AI filename suggestion → set title |
| `POST` | `/documents/{id}/suggestions/filename/reject` | Clear AI filename suggestion |

### Plugin endpoints (internal — backend calls only)

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/plugin/manifest` | Static manifest: metadata, JSON Schema for settings, access rules |
| `GET` | `/plugin/settings` | Current watch/storage config values |
| `PATCH` | `/plugin/settings` | Update watch/storage config (persisted to `/config/doc_service_config.json`) |

### Pagination & filtering (`GET /documents`)

Query params:

| Param | Default | Notes |
|-------|---------|-------|
| `page` | 1 | ≥ 1 |
| `per_page` | 20 | 1–100 |
| `sort` | `created_at` | `created_at`, `processed_at`, `filename`, `title`, `file_size`, `status`, `document_type` |
| `order` | `desc` | `asc` \| `desc` |
| `status` | — | filter by status string |
| `document_type` | — | filter by document type |
| `search` | — | case-insensitive ILIKE on `title`, `filename`, `tags`, `document_type` |
| `category_id` | — | filter to documents assigned to this category UUID |

Response: `{ items: [...], total: N, page: N, pages: N }`

### Document schema

```
id                UUID
user_id           string (from x-user-id header; "watch" for watch-ingested docs)
filename          original filename
title             AI-suggested editable title (nullable)
file_size         bytes
status            pending | processing | done | failed
document_type     AI-classified type (nullable)
extracted_data    JSON string — all AI-extracted fields
tags              JSON array string — editable tags
error_message     set if status=failed
created_at        upload timestamp
processed_at      when extraction finished
source            "upload" (default) or "watch"
watch_path        original absolute path in watch directory (nullable)
suggested_folder  AI-suggested category name, pending user confirm (nullable)
suggested_filename AI-suggested title/rename, pending user confirm (nullable)
categories        many-to-many via category_assignments
```

Watch-ingested documents (`user_id = "watch"`) are visible to all authenticated users.

### AI extraction (via ai-service)

System prompt and user prompt template are loaded at runtime from `doc_service_config.json` (`system_prompts` key). Defaults are built into the service and used as fallback if the config key is absent. Changes made via the AI Settings UI take effect within 30 seconds (config cache TTL).

Prompt sends the first 50 000 chars of extracted text. Expected JSON response includes:
- `title` — suggested human-readable title
- `document_type` — invoice / bill / receipt / order / expense / revenue / unknown
- `tags` — list of keyword tags
- `suggested_categories` — list of category names to suggest in the UI
- Domain fields: `vendor`, `customer`, `invoice_number`, `due_date`, `total_amount`, `currency`, etc.

### Config (runtime, persisted to shared volume)

`/config/doc_service_config.json`:
```json
{ "documents": { "max_pdf_bytes": 20971520 } }
```
Env override: `DOC_MAX_PDF_MB`

### Watch directory feature

Controlled via plugin settings (UI accessible to superusers and `doc-service-admin` group members):

- `watch_enabled` — toggle file watching (default: false)
- `watch_path` — mount point (read-only, `/data/watch`; override via Docker volume)
- `ai_folder_suggestion` — AI suggests a category for each ingested doc (user confirms)
- `ai_folder_default` — default category when AI suggestion is disabled
- `ai_rename_suggestion` — AI suggests a title for each ingested doc (user confirms)

On startup scan, the watcher walks the watch directory and ingests any PDFs not already in the database (idempotency check by `watch_path`). Subfolders are automatically mapped to categories (e.g. `watch/invoices/bill.pdf` → category "invoices"). No-remove policy: deleting a file from the watch directory does not delete the document record.

### Database migrations

| Revision | Description |
|----------|-------------|
| 0001 | Initial schema (documents, categories, category_assignments) |
| 0002 | Add `title` column to documents |
| 0003 | Add `source`, `watch_path`, `suggested_folder`, `suggested_filename` columns |

Run automatically on container start via `alembic upgrade head`.

---

## Architecture

```
backend (proxy)  →  doc-service:8001
                        │
           ┌────────────┼────────────────────┐
      documents.py    categories.py        plugin.py
           │               │             (internal only)
  ┌────────┴────────┐
upload           list/get/patch/suggest
  │
save_upload()        pdfplumber extraction
  │                    │
Document(status=pending)   ai_client.classify_document()
  │                    │
BackgroundTask         ai-service:8010/chat
  │                    │
process_document()   JSON result → update doc row

file_watcher.py (watchdog Observer, daemon thread)
  │
  ├── _PdfEventHandler.on_created / on_moved
  │       └── asyncio.run_coroutine_threadsafe(ingest_file, loop)
  │
  └── _scan_existing() on startup (catches offline gaps)
```

---

## Known limitations / not implemented

- **Re-process** — no endpoint to re-trigger AI extraction on an existing document (e.g. after changing the AI model or prompt)
- **Advanced field-level search** — `search` param matches text fields via ILIKE but does not query into `extracted_data` JSON (e.g. filter by `vendor` or `due_date`)
- **Bulk operations** — no bulk category assign/remove, no bulk delete
- **Document sharing** — documents are strictly per-user; no group sharing yet
- **Pagination in categories** — categories are returned as a full list (no pagination)
- **File type** — only PDF supported
- **Concurrent uploads** — no rate limiting per user

---

## Future work

- [x] `POST /documents/{id}/reprocess` — re-run AI extraction
- [x] Watch directory feature with file watcher, startup scan, folder-to-category mapping, AI suggestion toggles
- [x] Plugin manifest endpoint (`/plugin/manifest`, `/plugin/settings`) for generic settings UI
- [ ] Advanced filter: query `extracted_data` JSON fields (vendor, due_date, amount) — requires PostgreSQL `jsonb` column or indexed virtual columns
- [ ] Bulk operations endpoint
- [ ] Document sharing via groups (blocked on groups/permissions system in backend)
- [ ] Support additional file types (images via OCR, DOCX)
- [ ] Rate limiting on upload endpoint
- [ ] Soft delete with restore
- [ ] Category rename / delete with cascade handling
- [ ] Frontend UI for suggestion badges (suggested_folder / suggested_filename confirm/reject buttons)