Business-Management/features/doc-service/STATUS.md

Doc Service — Status

What it is

PDF document management microservice. Handles upload, storage, async AI-powered extraction, tagging, categorisation, and retrieval of PDF documents on a per-user basis.

Port: 8001 (internal only, not exposed to host). All traffic arrives via the backend proxy (backend/app/routers/documents_proxy.py), which injects the authenticated x-user-id header.

Database: shared PostgreSQL instance, isolated via alembic_version_doc_service Alembic version table. Storage: /data/documents/ (Docker named volume doc_data).

---

Current functionality

Document lifecycle

1. POST /documents/upload — validate PDF, persist file to /data/documents/{user_id}/{doc_id}.pdf, create DB row with status=pending, enqueue background extraction
2. Background task: extract text with pdfplumber → POST to ai-service /chat → parse JSON result → update status=done (or failed)
3. AI extracts: title, document_type, tags, suggested_categories, plus domain fields (vendor, customer, dates, amounts, etc.) into extracted_data (JSON string)

Endpoints

Method	Path	Description
POST	/documents/upload	Upload PDF; returns 202 with initial doc row
GET	/documents	Paginated list with filters and sort
GET	/documents/{id}	Single document
GET	/documents/{id}/status	Lightweight status poll
GET	/documents/{id}/download	Stream file bytes
DELETE	/documents/{id}	Delete document and file
PATCH	/documents/{id}/type	Update document type
PATCH	/documents/{id}/tags	Replace tag list (dedup, preserve order)
PATCH	/documents/{id}/title	Update editable title
GET	/documents/categories	List all categories for the user
POST	/documents/categories	Create a category
POST	/documents/{id}/categories/{cat_id}	Assign category to document
DELETE	/documents/{id}/categories/{cat_id}	Remove category from document

Pagination & filtering (GET /documents)

Query params:

Param	Default	Notes
page	1	≥ 1
per_page	20	1–100
sort	created_at	created_at, processed_at, filename, title, file_size, status, document_type
order	desc	asc | desc
status	—	filter by status string
document_type	—	filter by document type
search	—	case-insensitive ILIKE on title, filename, tags, document_type

Response: { items: [...], total: N, page: N, pages: N }

Document schema

id            UUID
user_id       string (from x-user-id header)
filename      original filename
title         AI-suggested editable title (nullable)
file_size     bytes
status        pending | processing | done | failed
document_type AI-classified type (nullable)
extracted_data JSON string — all AI-extracted fields
tags          JSON array string — editable tags
error_message set if status=failed
created_at    upload timestamp
processed_at  when extraction finished
categories    many-to-many via category_assignments

AI extraction (via ai-service)

Prompt sends the first 50 000 chars of extracted text. Expected JSON response includes:

• title — suggested human-readable title
• document_type — invoice / bill / receipt / order / expense / revenue / unknown
• tags — list of keyword tags
• suggested_categories — list of category names to suggest in the UI
• Domain fields: vendor, customer, invoice_number, due_date, total_amount, currency, etc.

Config (runtime, persisted to shared volume)

/config/doc_service_config.json:

{ "documents": { "max_pdf_bytes": 20971520 } }

Env override: DOC_MAX_PDF_MB

Database migrations

Revision	Description
0001	Initial schema (documents, categories, category_assignments)
0002	Add title column to documents

Run automatically on container start via alembic upgrade head.

---

Architecture

backend (proxy)  →  doc-service:8001
                        │
                   documents.py router
                        │
               ┌────────┴────────┐
          upload              list/get/patch
               │
        save_upload()        pdfplumber extraction
               │                    │
         Document(status=pending)   ai_client.classify_document()
               │                    │
        BackgroundTask         ai-service:8010/chat
               │                    │
         process_document()   JSON result → update doc row

---

Known limitations / not implemented

• Re-process — no endpoint to re-trigger AI extraction on an existing document (e.g. after changing the AI model or prompt)
• Advanced field-level search — search param matches text fields via ILIKE but does not query into extracted_data JSON (e.g. filter by vendor or due_date)
• Bulk operations — no bulk category assign/remove, no bulk delete
• Document sharing — documents are strictly per-user; no group sharing yet
• Pagination in categories — categories are returned as a full list (no pagination)
• File type — only PDF supported
• Concurrent uploads — no rate limiting per user

---

Future work

• POST /documents/{id}/reprocess — re-run AI extraction
• Advanced filter: query extracted_data JSON fields (vendor, due_date, amount) — requires PostgreSQL jsonb column or indexed virtual columns
• Bulk operations endpoint
• Document sharing via groups (blocked on groups/permissions system in backend)
• Support additional file types (images via OCR, DOCX)
• Rate limiting on upload endpoint
• Soft delete with restore
• Category rename / delete with cascade handling