Alembic op.execute GIN index must not collide with ORM model table_args
ix_documents_fts
from
to
via
pattern
backend/tasks/audit_tasks.py
backend/storage/minio_backend.py
put_object_raw called by daily export task
put_object_raw
Create Alembic migration 0004 and add MinIOBackend.put_object_raw(). This plan has no
dependency on the test scaffolds plan (04-01) and can run in parallel.
Purpose: Establish the database schema additions (pdf_open_mode, FTS index) and storage
method (put_object_raw for audit CSV upload) that later plans depend on.
Output: Migration file 0004 + extended MinIOBackend.
@.planning/phases/04-folders-sharing-quotas-document-ux/04-CONTEXT.md
@.planning/phases/04-folders-sharing-quotas-document-ux/04-PATTERNS.md
@backend/migrations/versions/0003_multi_user_isolation.py
@backend/storage/minio_backend.py
@backend/db/models.py
Task 1: Alembic migration 0004 — pdf_open_mode column + GIN FTS index + audit-logs bucket
backend/migrations/versions/0004_phase4_pdf_open_mode_tsvector.py
backend/migrations/versions/0003_multi_user_isolation.py — read the entire file; extract the batch_alter_table pattern for adding columns, the MinIO bucket creation pattern gated on MINIO_ENDPOINT env var, the module docstring format, and the downgrade() function pattern
backend/db/models.py — read lines 44-70 (User model) to confirm pdf_open_mode does NOT already exist before adding it
Create backend/migrations/versions/0004_phase4_pdf_open_mode_tsvector.py.
Module docstring (follow 0003 format): list three changes in order — (1) users.pdf_open_mode column, (2) GIN expression index on documents.extracted_text, (3) audit-logs MinIO bucket creation.
Set: revision = "0004", down_revision = "0003", branch_labels = None, depends_on = None.
upgrade() function — three steps in this exact order:
Step 1: Add users.pdf_open_mode column using batch_alter_table (per D-10, for SQLite compat):
`with op.batch_alter_table("users") as batch_op: batch_op.add_column(sa.Column("pdf_open_mode", sa.String(), nullable=False, server_default="in_app"))`
Step 2: Create GIN expression index MANUALLY — do NOT use Column Computed() or Index() — use op.execute with raw SQL (per D-11, Pitfall 2, PATTERNS.md). The exact SQL is:
`CREATE INDEX ix_documents_fts ON documents USING GIN (to_tsvector('english', coalesce(extracted_text, '')))`
Add a comment above this line: `# managed manually — do not autogenerate (Alembic issue #1390)`.
Step 3: Create the audit-logs MinIO bucket gated on MINIO_ENDPOINT env var (per Pitfall 8, PATTERNS.md). Follow the exact same guard pattern from migration 0003: `if os.environ.get("MINIO_ENDPOINT"):` then instantiate Minio client using env vars MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY with secure=False. Check `client.bucket_exists("audit-logs")` and call `client.make_bucket("audit-logs")` only if it does not exist.
downgrade() function: (1) `op.execute("DROP INDEX IF EXISTS ix_documents_fts")`, (2) `with op.batch_alter_table("users") as batch_op: batch_op.drop_column("pdf_open_mode")`. Add a comment: `# MinIO bucket NOT reversed — bucket may contain audit data`.
Imports needed at module top: `from __future__ import annotations`, `import os`, `import sqlalchemy as sa`, `from alembic import op`.
cd /Users/nik/Documents/Progamming/document_scanner/backend && python -c "from migrations.versions import 0004_phase4_pdf_open_mode_tsvector" 2>&1 || python -c "import importlib.util; spec = importlib.util.spec_from_file_location('m', 'migrations/versions/0004_phase4_pdf_open_mode_tsvector.py'); m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m); print('OK')"
- File exists at backend/migrations/versions/0004_phase4_pdf_open_mode_tsvector.py
- `revision = "0004"` and `down_revision = "0003"` present in file
- File contains `batch_alter_table("users")` with pdf_open_mode column, server_default="in_app"
- File contains `CREATE INDEX ix_documents_fts ON documents USING GIN` (grep confirms)
- File contains `# managed manually — do not autogenerate` comment
- File contains `if os.environ.get("MINIO_ENDPOINT")` MinIO gate for audit-logs bucket
- File contains `audit-logs` bucket name (grep confirms)
- File contains `DROP INDEX IF EXISTS ix_documents_fts` in downgrade()
- `python -c "import py_compile; py_compile.compile('migrations/versions/0004_phase4_pdf_open_mode_tsvector.py')"` exits 0 (no syntax errors)
Migration file is syntactically valid and contains all three upgrade steps + downgrade reversal.
Task 2: Add put_object_raw() to MinIOBackend
backend/storage/minio_backend.py
backend/storage/minio_backend.py — read the entire file; identify the existing put_object() method signature and asyncio.to_thread() call pattern; identify the class definition and existing imports including io import
backend/storage/base.py — read to see if StorageBackend ABC needs a corresponding abstract method added (it does NOT — put_object_raw is concrete only on MinIOBackend, not on the ABC, because only MinIOBackend has an audit-logs bucket)
Add the put_object_raw() async method to the MinIOBackend class in backend/storage/minio_backend.py.
Place it immediately after the existing put_object() method.
Method signature: `async def put_object_raw(self, bucket: str, key: str, data: io.BytesIO, length: int, content_type: str) -> None:`
Docstring: "Upload bytes to an arbitrary bucket+key (used for audit-logs CSV export). Unlike put_object(), does NOT apply the document key schema — the caller supplies the complete key. The main documents bucket is NOT used."
Implementation: call `await asyncio.to_thread(self._client.put_object, bucket, key, data, length=length, content_type=content_type)`. This mirrors the exact pattern of the existing put_object() method (which uses self._client.put_object via asyncio.to_thread).
Do NOT add io import if it already exists. Do NOT add this method to StorageBackend ABC (base.py) — this is MinIOBackend-only.
cd /Users/nik/Documents/Progamming/document_scanner/backend && python -c "from storage.minio_backend import MinIOBackend; import inspect; src = inspect.getsource(MinIOBackend.put_object_raw); print('OK')"
- MinIOBackend.put_object_raw exists as an async method (grep: `async def put_object_raw`)
- Method accepts: bucket: str, key: str, data: io.BytesIO, length: int, content_type: str
- Method body calls `asyncio.to_thread(self._client.put_object, bucket, key, data, length=length, content_type=content_type)`
- `python -c "from storage.minio_backend import MinIOBackend"` exits 0 (no import errors)
- StorageBackend ABC (base.py) is NOT modified (grep: `put_object_raw` absent from base.py)
MinIOBackend.put_object_raw() is importable and callable; base.py unchanged.
<threat_model>
Trust Boundaries
Boundary
Description
Migration → MinIO
Bucket creation uses env var credentials; no credentials in code
put_object_raw → MinIO SDK
Caller supplies bucket name; method does not validate bucket against allowlist
STRIDE Threat Register
Threat ID
Category
Component
Disposition
Mitigation Plan
T-04-02-01
Tampering
migration 0004 GIN index
mitigate
Index created via raw SQL (not autogenerate) to prevent Alembic repeat-generation bug; comment documents the decision
T-04-02-02
Information Disclosure
audit-logs MinIO bucket
mitigate
Bucket creation gated on MINIO_ENDPOINT env var (no hardcoded credentials); bucket policy is private-by-default (MinIO default)
T-04-02-03
Tampering
put_object_raw caller-supplied key
accept
put_object_raw is called only from trusted application code (Celery task); key is constructed from application logic, not user input
curo1305
747303246a