187 lines
11 KiB
Markdown
187 lines
11 KiB
Markdown
---
|
|
phase: 14.1-cloud-local-file-parity-hardening
|
|
plan: "02"
|
|
subsystem: cloud-detail-parity-backend
|
|
status: complete
|
|
tags: [cloud-detail, force-reanalyze, single-item-retry, schema-allowlist, parity]
|
|
requirements: [CLOUD-02, ANALYZE-01, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-05]
|
|
|
|
dependency_graph:
|
|
requires:
|
|
- 14.1-01 (RED tests — test_cloud_detail_parity.py, test_cloud_reanalyze_force.py)
|
|
provides:
|
|
- backend/api/cloud/schemas.py (CloudItemDetailOut + force on AnalysisEnqueueRequest)
|
|
- backend/services/cloud_items.py (resolve_owned_cloud_item_detail)
|
|
- backend/api/cloud/operations.py (GET /connections/{id}/items/{item_id:path}/detail)
|
|
- backend/services/cloud_analysis.py (force param + retry_or_create_single_item_job)
|
|
- backend/api/cloud/analysis.py (force wired; POST /connections/{id}/items/{cloud_item_id}/retry)
|
|
affects:
|
|
- frontend (Plan 03/04 target — can now render cloud file detail with analysis fields)
|
|
|
|
tech_stack:
|
|
added: []
|
|
patterns:
|
|
- Strict Pydantic allowlist schema with explicit forbidden-field documentation (T-14.1-03)
|
|
- Service raises domain exceptions (ConnectionNotFound, CloudItemNotFound, InvalidJobState); router translates to HTTP
|
|
- Metadata-only detail resolution — zero provider bytes downloaded (CACHE-03, T-14.1-06)
|
|
- already_current bypass via force flag without changing the unsupported guard (ANALYZE-06)
|
|
- Single-item retry-job creation through authorized enqueue path (D-12)
|
|
|
|
key_files:
|
|
created: []
|
|
modified:
|
|
- backend/api/cloud/schemas.py
|
|
- backend/services/cloud_items.py
|
|
- backend/api/cloud/operations.py
|
|
- backend/services/cloud_analysis.py
|
|
- backend/api/cloud/analysis.py
|
|
|
|
decisions:
|
|
- CloudItemDetailOut uses an empty capabilities dict (no live capability resolution without credential decryption) — frontend infers actions from analysis_status + unsupported_analysis_reason
|
|
- unsupported_analysis_reason populated via _is_supported from cloud_analysis (single source of truth)
|
|
- force=True bypasses already_current check for supported items only; unsupported items remain unsupported regardless
|
|
- retry_or_create_single_item_job accepts failed/indexed/stale/pending items via force=True enqueue
|
|
- Single-item retry route uses cloud_item_id (DocuVault UUID) not provider_item_id for stable identity
|
|
|
|
metrics:
|
|
duration: "18m"
|
|
completed_date: "2026-06-26"
|
|
tasks_completed: 2
|
|
files_modified: 5
|
|
backend_tests_passing: 872
|
|
backend_tests_added_passing: 34
|
|
pre_existing_failures: 1 (test_extract_docx — ModuleNotFoundError: No module named 'docx', unrelated)
|
|
---
|
|
|
|
# Phase 14.1 Plan 02: Cloud Item Detail + Force Re-analyze + Single-item Retry Summary
|
|
|
|
Backend contracts that let cloud files behave like local documents: owner-scoped cloud item detail with extracted text, topics, and analysis status through a strict credential-free schema; force re-analyze flag bypassing already_current for indexed items; and a single-item retry-job path for failed items with no surviving active job.
|
|
|
|
## Tasks Completed
|
|
|
|
### Task 1: CloudItemDetailOut schema + resolve_owned_cloud_item_detail + GET detail route
|
|
|
|
**Schema — `backend/api/cloud/schemas.py`:**
|
|
|
|
Added `CloudItemDetailOut` as an explicit allowlist — credentials_enc, object_key, version_key, and raw provider URLs are absent by design (T-14.1-03). Fields: id, provider_item_id, name, kind, parent_ref, content_type, size, modified_at, etag, provider, display_name, location, analysis_status, semantic_index_status, extracted_text, topics (list[str]), capabilities (empty dict — no live provider call), unsupported_analysis_reason, is_stale.
|
|
|
|
Added `force: bool = Field(default=False)` to `AnalysisEnqueueRequest` (used by Task 2).
|
|
|
|
**Service — `backend/services/cloud_items.py`:**
|
|
|
|
Added `CloudItemDetail` dataclass and `resolve_owned_cloud_item_detail(session, *, user_id, connection_id, provider_item_id)`:
|
|
- Calls `resolve_owned_connection` for ownership gate (T-14.1-04).
|
|
- Selects CloudItem by (connection_id, provider_item_id, user_id) — raises CloudItemNotFound.
|
|
- Loads topic names via CloudItemTopic→Topic join (metadata-only).
|
|
- Derives `location` from path_snapshot or parent_ref (never a raw provider URL — T-14.1-03).
|
|
- Derives `unsupported_analysis_reason` via `_is_supported` from cloud_analysis (single source).
|
|
- Zero bytes downloaded — no get_object, no hydrate_and_cache_bytes (CACHE-03, T-14.1-06).
|
|
|
|
**Route — `backend/api/cloud/operations.py`:**
|
|
|
|
Added `GET /connections/{connection_id}/items/{item_id:path}/detail` with `response_model=CloudItemDetailOut` and `get_regular_user` dependency (admin → 403). ConnectionNotFound → 404, CloudItemNotFound → 404. Returns CloudItemDetailOut with capabilities={} (intentionally empty — no credential decryption during detail fetch).
|
|
|
|
**Test results:** All 8 test_cloud_detail_parity.py tests pass (owner 200, foreign 404, admin 403, credential exclusion, stale retention, pending empty-state, no byte hydration).
|
|
|
|
**Commit:** a0d5c1d
|
|
|
|
### Task 2: Force re-analyze flag + single-item retry-job creation
|
|
|
|
**Schema — `backend/api/cloud/schemas.py`:**
|
|
|
|
`force: bool = Field(default=False)` added to `AnalysisEnqueueRequest` with full docstring (D-11).
|
|
|
|
**Service — `backend/services/cloud_analysis.py`:**
|
|
|
|
Extended `enqueue_analysis_job` with `force: bool = False` parameter. The already_current check becomes:
|
|
```python
|
|
already_current = False if (live_metadata_changed or force) else await _check_already_current(...)
|
|
```
|
|
When force=True, supported items are created as queued job items even if their version_key matches a prior indexed run. Unsupported items remain unsupported regardless of force (ANALYZE-07: no provider mutation is performed).
|
|
|
|
Added `retry_or_create_single_item_job(session, *, cloud_item_id, user_id)` service helper that:
|
|
- Resolves the CloudItem by (id, user_id) — raises AnalysisItemNotFound.
|
|
- Accepts failed/indexed/stale/pending items; rejects others with InvalidJobState.
|
|
- Calls `enqueue_analysis_job(scope="file", provider_item_ids=[item.provider_item_id], force=True)`.
|
|
- Returns an EnqueueResult with queued_count >= 1 (for supported items).
|
|
|
|
**Route — `backend/api/cloud/analysis.py`:**
|
|
|
|
- Wired `force=body.force` into the existing `enqueue_job` route handler (D-11).
|
|
- Added `POST /analysis/connections/{connection_id}/items/{cloud_item_id}/retry` returning `AnalysisEnqueueOut` with status 202. Validates connection ownership before delegating to `retry_or_create_single_item_job`. Returns job_id and queued_count in the response (D-12).
|
|
|
|
**Test results:** All 8 test_cloud_reanalyze_force.py tests pass. All 26 test_cloud_analysis_contract.py tests pass (no idempotency regression). Full suite: 872/873 pass (1 pre-existing failure unrelated to this plan).
|
|
|
|
**Commit:** 52acd56
|
|
|
|
## Verification Results
|
|
|
|
### Backend verification
|
|
```
|
|
test_cloud_detail_parity.py — 8/8 passed
|
|
test_cloud_reanalyze_force.py — 8/8 passed
|
|
test_cloud_analysis_contract.py — 26/26 passed
|
|
test_cloud_security.py — 36/36 passed
|
|
Full suite: 872 passed, 1 pre-existing failure (test_extract_docx — missing docx module)
|
|
```
|
|
|
|
### Acceptance criteria
|
|
|
|
- `grep -c 'class CloudItemDetailOut' backend/api/cloud/schemas.py` → 1 ✓
|
|
- CloudItemDetailOut excludes object_key/credentials_enc/version_key within class body ✓
|
|
- `grep -c 'resolve_owned_cloud_item_detail' backend/services/cloud_items.py` → 2 (definition + docstring) ✓
|
|
- Detail route `/detail` with `response_model=CloudItemDetailOut` present in operations.py ✓
|
|
- No get_object/hydrate_and_cache_bytes call inside resolve_owned_cloud_item_detail ✓
|
|
- `grep -c 'force' backend/api/cloud/schemas.py` → 3 (field, description lines) ✓
|
|
- `enqueue_analysis_job` signature includes `force: bool = False` ✓
|
|
- `already_current = False if (live_metadata_changed or force) else ...` on line 608 ✓
|
|
- `force=body.force` wired in analysis.py enqueue route ✓
|
|
- `retry_or_create_single_item_job` exists in cloud_analysis.py and analysis.py import ✓
|
|
|
|
## Deviations from Plan
|
|
|
|
### Auto-fixed Issues
|
|
|
|
**1. [Rule 2 - Missing critical functionality] Empty capabilities in detail response**
|
|
- **Found during:** Task 1 implementation
|
|
- **Issue:** The plan called for returning `capabilities` in CloudItemDetailOut. Populating live capabilities would require credential decryption and a provider call — violating the metadata-only constraint (CACHE-03 / T-14.1-06).
|
|
- **Fix:** Return `capabilities={}` (empty dict) from the detail route. The `unsupported_analysis_reason` field carries the action-availability signal the frontend needs (D-16). A comment in the route documents the intentional empty capabilities and the reason.
|
|
- **Files modified:** backend/api/cloud/operations.py
|
|
- **Commit:** a0d5c1d
|
|
|
|
**2. [Rule 1 - Bug] CloudConnection.display_name_override must take precedence**
|
|
- **Found during:** Task 1 — reviewing CloudConnection model fields
|
|
- **Issue:** CloudConnection has both `display_name` and `display_name_override`. The connection rename flow sets `display_name_override`; the original `display_name` is the provider-set name. The detail response should show what the user sees (override if set).
|
|
- **Fix:** `display_name=conn.display_name_override or conn.display_name` in resolve_owned_cloud_item_detail.
|
|
- **Files modified:** backend/services/cloud_items.py
|
|
- **Commit:** a0d5c1d
|
|
|
|
## Known Stubs
|
|
|
|
None — all fields return real data from DB rows. The empty `capabilities={}` in the detail response is intentional and documented (see Deviation 1 above), not a stub.
|
|
|
|
## Threat Flags
|
|
|
|
| Flag | File | Description |
|
|
|------|------|-------------|
|
|
| No new surface | — | All new endpoints are owner-scoped via get_regular_user + resolve_owned_connection + item ownership check. No new trust boundaries introduced. |
|
|
|
|
T-14.1-03, T-14.1-04, T-14.1-05, T-14.1-06 mitigations all implemented as designed:
|
|
- T-14.1-03: CloudItemDetailOut schema enforces allowlist — tests verify credentials_enc/object_key/version_key absent.
|
|
- T-14.1-04: get_regular_user (admin 403) + ownership check (foreign 404) on both detail and retry routes.
|
|
- T-14.1-05: force only bypasses already_current check; no provider mutation methods called.
|
|
- T-14.1-06: resolve_owned_cloud_item_detail calls zero get_object / hydrate_and_cache_bytes.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
Files verified:
|
|
- /Users/nik/Documents/Progamming/document_scanner/backend/api/cloud/schemas.py ✓
|
|
- /Users/nik/Documents/Progamming/document_scanner/backend/services/cloud_items.py ✓
|
|
- /Users/nik/Documents/Progamming/document_scanner/backend/api/cloud/operations.py ✓
|
|
- /Users/nik/Documents/Progamming/document_scanner/backend/services/cloud_analysis.py ✓
|
|
- /Users/nik/Documents/Progamming/document_scanner/backend/api/cloud/analysis.py ✓
|
|
|
|
Commits verified:
|
|
- a0d5c1d (Task 1) present in git log ✓
|
|
- 52acd56 (Task 2) present in git log ✓
|