105 lines
4.8 KiB
Markdown
105 lines
4.8 KiB
Markdown
---
|
|
phase: "14"
|
|
plan: "05"
|
|
type: execute
|
|
wave: 4
|
|
depends_on:
|
|
- "14-03"
|
|
- "14-04"
|
|
files_modified:
|
|
- backend/tasks/cloud_analysis_tasks.py
|
|
- backend/celery_app.py
|
|
- backend/services/cloud_analysis_processing.py
|
|
- backend/services/cloud_byte_cache.py
|
|
- backend/services/classifier.py
|
|
- backend/tests/test_cloud_analysis_contract.py
|
|
- backend/tests/test_cloud_cache.py
|
|
autonomous: true
|
|
requirements:
|
|
- ANALYZE-04
|
|
- ANALYZE-05
|
|
- ANALYZE-06
|
|
- ANALYZE-07
|
|
- CACHE-03
|
|
- CACHE-04
|
|
---
|
|
|
|
<objective>
|
|
Implement Celery-backed cloud analysis processing: hydrate bytes on demand, retain/reuse cache entries, extract text, classify topics onto `CloudItem`, update per-item and aggregate states, and honor cancellation/retry semantics.
|
|
|
|
Scope fence: do not build frontend controls in this plan.
|
|
</objective>
|
|
|
|
<context>
|
|
@.planning/phases/14-selective-analysis-and-byte-cache/14-PATTERNS.md
|
|
@backend/tasks/cloud_tasks.py
|
|
@backend/tasks/document_tasks.py
|
|
@backend/services/extractor.py
|
|
@backend/services/classifier.py
|
|
@backend/storage/cloud_backend_factory.py
|
|
@backend/services/cloud_byte_cache.py
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: Add processing service for one job item</name>
|
|
<files>backend/services/cloud_analysis_processing.py, backend/services/cloud_byte_cache.py, backend/tests/test_cloud_analysis_contract.py, backend/tests/test_cloud_cache.py</files>
|
|
<read_first>
|
|
- backend/services/extractor.py
|
|
- backend/services/classifier.py
|
|
- backend/services/cloud_byte_cache.py
|
|
- backend/services/cloud_analysis_versioning.py
|
|
</read_first>
|
|
<action>
|
|
Implement a service that processes a single job item by revalidating owner/connection/item, checking cancellation, checking version-key currentness, reusing or hydrating cache bytes, extracting text, persisting `CloudItem.extracted_text`, classifying topics through existing classifier logic adapted for cloud items, updating status to indexed/failed/cancelled/stale, and releasing cache pins. Compute content hash only while bytes are already available. Never create a local `Document` row and never call provider mutation methods.
|
|
</action>
|
|
<acceptance_criteria>
|
|
- Status transitions include downloading, extracting, classifying, indexed, cancelled, failed, stale.
|
|
- Unchanged version skips processing before provider byte fetch.
|
|
- Provider byte fetch occurs only for actionable analysis/open/preview work.
|
|
- Cache pins are released on success, failure, and cancellation.
|
|
</acceptance_criteria>
|
|
<verify><automated>cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py -k "processing or idempotent or cancellation"</automated></verify>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 2: Add Celery task and retry harness</name>
|
|
<files>backend/tasks/cloud_analysis_tasks.py, backend/celery_app.py, backend/tests/test_cloud_analysis_contract.py</files>
|
|
<read_first>
|
|
- backend/tasks/cloud_tasks.py
|
|
- backend/tasks/document_tasks.py
|
|
- backend/celery_app.py
|
|
</read_first>
|
|
<action>
|
|
Add sync Celery tasks that bridge to async processing with the existing sentinel retry pattern. Task payloads include job id/item id only. Decrypt credentials inside the worker after owner revalidation. Support bounded transient provider/classifier retry, terminal auth failures, max-retry final failure status, and cooperative cancellation checks between stages.
|
|
</action>
|
|
<acceptance_criteria>
|
|
- Celery broker payload contains no credentials, provider URLs, filenames, or bytes.
|
|
- `self.retry()` is raised only in the outer sync task layer.
|
|
- Max retry writes controlled failure status.
|
|
- Aggregate job counters update after item transitions.
|
|
</acceptance_criteria>
|
|
<verify><automated>cd backend && pytest -q tests/test_cloud_analysis_contract.py -k "celery or retry or broker"</automated></verify>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<threat_model>
|
|
| Threat ID | Threat | Mitigation |
|
|
|-----------|--------|------------|
|
|
| T-14-10 | Credentials in broker payload | IDs only in Celery task args; decrypt in worker |
|
|
| T-14-11 | Provider file mutation | Processing service uses content-read adapter path only and tests fake mutation counters |
|
|
| T-14-12 | Cache pin leak | finally block releases pins; tests cover failure/cancel paths |
|
|
| T-14-13 | Stale result presented as current | Final version/fingerprint comparison marks item stale if provider metadata changed |
|
|
</threat_model>
|
|
|
|
<verification>
|
|
1. `cd backend && pytest -q tests/test_cloud_analysis_contract.py tests/test_cloud_cache.py`
|
|
2. `rg "credentials|credentials_enc|file_bytes" backend/tasks/cloud_analysis_tasks.py` and confirm no broker payload exposure.
|
|
3. `rg "Document\\(" backend/services/cloud_analysis_processing.py backend/tasks/cloud_analysis_tasks.py` returns no local import creation.
|
|
</verification>
|
|
|
|
<output>Create `.planning/phases/14-selective-analysis-and-byte-cache/14-05-SUMMARY.md` when complete.</output>
|
|
|