docs(13-06): complete upload follow-through plan

This commit is contained in:
curo1305
2026-06-22 19:31:38 +02:00
parent d959e0cf42
commit fc28e032be
3 changed files with 189 additions and 7 deletions
+3 -3
View File
@@ -24,7 +24,7 @@ Before any phase is marked complete:
| Phase | Name | Goal | Requirements | | Phase | Name | Goal | Requirements |
|------:|------|------|--------------| |------:|------|------|--------------|
| 12 | 6/6 | Complete | 2026-06-21 | | 12 | 6/6 | Complete | 2026-06-21 |
| 13 | 5/11 | In Progress| | | 13 | 6/11 | In Progress| |
| 14 | Selective Analysis and Byte Cache | Analyze selected cloud scopes through cancellable jobs backed by bounded, private, on-demand byte caching | ANALYZE-01..07, CACHE-03..05 | | 14 | Selective Analysis and Byte Cache | Analyze selected cloud scopes through cancellable jobs backed by bounded, private, on-demand byte caching | ANALYZE-01..07, CACHE-03..05 |
| 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 |
| 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 |
@@ -80,7 +80,7 @@ Plans:
**Depends on:** Phase 12 **Depends on:** Phase 12
**Requirements:** CONN-01, CONN-02, CONN-03, CLOUD-02, CLOUD-03, CLOUD-04, CLOUD-05, CLOUD-06, CLOUD-07, CLOUD-09 **Requirements:** CONN-01, CONN-02, CONN-03, CLOUD-02, CLOUD-03, CLOUD-04, CLOUD-05, CLOUD-06, CLOUD-07, CLOUD-09
**Plans:** 5/11 plans executed **Plans:** 6/11 plans executed
**Execution waves:** Wave 0: 13-01 and 13-02 in parallel; Wave 1: 13-03 after 13-01; Wave 2: 13-04 after 13-01 and 13-03; Wave 3: 13-05 after 13-01, 13-03, and 13-04; Wave 4: 13-06 after 13-05; Wave 5: 13-07 and 13-08 in parallel (13-07 after 13-02, 13-04, and 13-06; 13-08 after 13-03, 13-04, and 13-06); Wave 6: 13-09 after 13-08; Wave 7: 13-10 after 13-02, 13-04, 13-07, and 13-09; Wave 8: 13-11 after 13-03 through 13-10. **Execution waves:** Wave 0: 13-01 and 13-02 in parallel; Wave 1: 13-03 after 13-01; Wave 2: 13-04 after 13-01 and 13-03; Wave 3: 13-05 after 13-01, 13-03, and 13-04; Wave 4: 13-06 after 13-05; Wave 5: 13-07 and 13-08 in parallel (13-07 after 13-02, 13-04, and 13-06; 13-08 after 13-03, 13-04, and 13-06); Wave 6: 13-09 after 13-08; Wave 7: 13-10 after 13-02, 13-04, 13-07, and 13-09; Wave 8: 13-11 after 13-03 through 13-10.
Plans: Plans:
@@ -90,7 +90,7 @@ Plans:
- [x] 13-03-PLAN.md — Build the mutable cloud contract and orchestration seam without breaking centralized reconciliation. - [x] 13-03-PLAN.md — Build the mutable cloud contract and orchestration seam without breaking centralized reconciliation.
- [x] 13-04-PLAN.md — Implement connection-ID reconnect, explicit health test, broader Drive scope handling, and authorized content routes. - [x] 13-04-PLAN.md — Implement connection-ID reconnect, explicit health test, broader Drive scope handling, and authorized content routes.
- [x] 13-05-PLAN.md — Implement backend upload provider mechanics, typed route results, and refreshed-credential handoff. - [x] 13-05-PLAN.md — Implement backend upload provider mechanics, typed route results, and refreshed-credential handoff.
- [ ] 13-06-PLAN.md — Complete upload reconcile, freshness, and metadata-only audit follow-through before frontend queue wiring. - [x] 13-06-PLAN.md — Complete upload reconcile, freshness, and metadata-only audit follow-through before frontend queue wiring.
- [ ] 13-07-PLAN.md — Wire the shared browser queue and binary-only preview or download fallback through thin cloud view handlers. - [ ] 13-07-PLAN.md — Wire the shared browser queue and binary-only preview or download fallback through thin cloud view handlers.
- [ ] 13-08-PLAN.md — Implement backend create-folder and rename semantics with collision, stale, and stable-identity safeguards. - [ ] 13-08-PLAN.md — Implement backend create-folder and rename semantics with collision, stale, and stable-identity safeguards.
- [ ] 13-09-PLAN.md — Implement backend move and delete semantics with same-connection, disclosure, security, and audit safeguards. - [ ] 13-09-PLAN.md — Implement backend move and delete semantics with same-connection, disclosure, security, and audit safeguards.
+6 -4
View File
@@ -6,14 +6,14 @@ current_phase: 13
current_phase_name: virtual-local-cloud-operations current_phase_name: virtual-local-cloud-operations
status: executing status: executing
stopped_at: Phase 13 context gathered stopped_at: Phase 13 context gathered
last_updated: "2026-06-22T17:22:26.784Z" last_updated: "2026-06-22T17:31:22.350Z"
last_activity: 2026-06-22 last_activity: 2026-06-22
last_activity_desc: Phase 13 execution started last_activity_desc: Phase 13 execution started
progress: progress:
total_phases: 6 total_phases: 6
completed_phases: 2 completed_phases: 2
total_plans: 21 total_plans: 21
completed_plans: 15 completed_plans: 16
percent: 33 percent: 33
--- ---
@@ -26,7 +26,7 @@ progress:
## Current Position ## Current Position
Phase: 13 (virtual-local-cloud-operations) — EXECUTING Phase: 13 (virtual-local-cloud-operations) — EXECUTING
Plan: 6 of 11 Plan: 7 of 11
Status: Ready to execute Status: Ready to execute
Last activity: 2026-06-22 — Phase 13 execution started Last activity: 2026-06-22 — Phase 13 execution started
@@ -56,6 +56,7 @@ Last activity: 2026-06-22 — Phase 13 execution started
| Phase 13 P02 | 30m | 2 tasks | 6 files | | Phase 13 P02 | 30m | 2 tasks | 6 files |
| Phase 13 P03 | 134 | 2 tasks | 8 files | | Phase 13 P03 | 134 | 2 tasks | 8 files |
| Phase 13 P04 | 180m | 2 tasks | 7 files | | Phase 13 P04 | 180m | 2 tasks | 7 files |
| Phase 13 P06 | 5m | 2 tasks | 3 files |
## Accumulated Context ## Accumulated Context
@@ -99,7 +100,7 @@ _Updated at each phase transition._
| Field | Value | | Field | Value |
|---|---| |---|---|
| Last session | 2026-06-22T17:22:26.779Z | | Last session | 2026-06-22T17:31:12.657Z |
| Next action | Execute Phase 12.1 Plan 01 | | Next action | Execute Phase 12.1 Plan 01 |
| Pending decisions | None | | Pending decisions | None |
| Resume file | None | | Resume file | None |
@@ -119,3 +120,4 @@ _Updated at each phase transition._
- [Phase ?]: D-17: Google Drive OAuth uses drive scope (not drive.file) for Phase 13 full-access mutations - [Phase ?]: D-17: Google Drive OAuth uses drive scope (not drive.file) for Phase 13 full-access mutations
- [Phase ?]: D-18: Preview is binary-only (PDF/images); Office/Workspace formats use typed unsupported_preview fallback to authorized download endpoint - [Phase ?]: D-18: Preview is binary-only (PDF/images); Office/Workspace formats use typed unsupported_preview fallback to authorized download endpoint
- [Phase ?]: Phase 13 mutation errors use JSONResponse (not HTTPException) so kind/reason appear at top level of response body, not nested under detail - [Phase ?]: Phase 13 mutation errors use JSONResponse (not HTTPException) so kind/reason appear at top level of response body, not nested under detail
- [Phase ?]: upload_mutated folder state code: upload success marks parent folder as warning/upload_mutated for reconcile-before-return
@@ -0,0 +1,180 @@
---
phase: "13"
plan: "06"
subsystem: cloud-operations
status: complete
tags:
- cloud
- upload
- reconciliation
- audit
- cloud-items
- tdd
dependency_graph:
requires:
- "13-01 (RED mutation contract tests)"
- "13-03 (cloud operations seam + mutable provider implementations)"
- "13-04 (operations.py route layer + typed result vocabulary)"
- "13-05 (keep_both_name + upload mechanics tests)"
provides:
- "Upload success routes through upsert_cloud_item for stable row identity"
- "Parent folder state invalidated (warning/upload_mutated) after success"
- "cloud.file_uploaded audit row written in same transaction as reconciliation"
- "3 Task 1 reconciliation tests in test_cloud_mutations.py"
- "1 Task 2 upload audit test promoted from xfail in test_cloud_audit.py"
affects:
- "13-07 through 13-11 (rename, move, delete, folder audit and reconciliation plans)"
tech_stack:
added:
- "upsert_cloud_item called from upload route on MUT_KIND_UPLOADED success"
- "update_folder_state called with warning/upload_mutated on upload success"
- "write_audit_log called with cloud.file_uploaded event on upload success"
- "get_client_ip imported from deps.utils in operations.py"
- "write_audit_log imported from services.audit in operations.py"
patterns:
- "Reconcile-before-return: upsert + folder invalidation complete before response"
- "Audit-in-transaction: write_audit_log flushes in caller transaction, caller commits"
- "Non-success bypass: conflict/offline/reauth paths never reach reconciliation or audit"
- "upload_mutated folder state: controlled code signaling re-list without a full provider scan"
key_files:
modified:
- path: "backend/api/cloud/operations.py"
change: "Upload success path now calls upsert_cloud_item, update_folder_state, write_audit_log before commit"
- path: "backend/tests/test_cloud_mutations.py"
change: "Added 3 reconciliation behavioral tests for Task 1 (RED then GREEN)"
- path: "backend/tests/test_cloud_audit.py"
change: "Promoted test_upload_success_writes_metadata_only_audit_row from xfail to real test (RED then GREEN)"
decisions:
- "upload_mutated is the controlled error_code for folder invalidation after upload — avoids apply_listing_and_finalize which needs a full provider listing"
- "CloudResource.id assigned uuid4() at route layer before upsert — cloud_items.upsert_cloud_item uses provider_item_id as the stable identity key, not this id"
- "Audit write uses flush (not commit) per services.audit contract; one commit covers upsert + folder state + audit atomically"
- "Non-success upload paths (conflict, offline, reauth_required) do not call write_audit_log, satisfying T-13-21 (no false success events)"
metrics:
duration: "~5 minutes"
completed: "2026-06-22"
tasks_completed: 2
tasks_planned: 2
files_changed: 3
files_created: 0
tests_added: 4
tests_passing: 745
---
# Phase 13 Plan 06: Upload Follow-Through Slice Summary
**One-liner:** Upload success routes through `upsert_cloud_item` + folder state invalidation + metadata-only `cloud.file_uploaded` audit row before the response is returned.
## Tasks Completed
| Task | Name | Commit | Key Files |
|------|------|--------|-----------|
| 1 (RED) | Add failing upload reconciliation tests | 4cd6499 | test_cloud_mutations.py |
| 1 (GREEN) | Route upload success through centralized reconciliation | 7ecbec7 | api/cloud/operations.py |
| 2 (RED) | Promote upload audit test from xfail | 33f0498 | test_cloud_audit.py |
| 2 (GREEN) | Emit metadata-only audit row on upload success | d959e0c | api/cloud/operations.py |
## What Was Built
### Task 1: Reconcile-before-return on upload success
**RED phase** added 3 tests to `test_cloud_mutations.py`:
- `test_upload_success_upserts_cloud_item_before_returning` — verifies that after a 200 upload response, a CloudItem row exists in the DB with the correct `provider_item_id` and `name`. This was the core reconcile-before-return behavioral test.
- `test_upload_success_marks_folder_freshness_stale` — verifies that a `CloudFolderState` row exists for the parent folder after success (folder state created/updated to signal re-list needed).
- `test_upload_failed_does_not_mutate_cloud_items` — verifies offline/error upload results do not create phantom CloudItem rows.
**GREEN phase** implemented in `backend/api/cloud/operations.py`:
Inside the `MUT_KIND_UPLOADED` success branch of `upload_cloud_file`:
```python
resource = CloudResource(
id=uuid.uuid4(),
provider_item_id=provider_item_id,
connection_id=connection_id,
user_id=current_user.id,
name=resolved_name,
kind="file",
parent_ref=resolved_parent_ref,
content_type=content_type,
size=resolved_size,
)
await upsert_cloud_item(session, user_id=str(current_user.id), resource=resource)
await update_folder_state(
session,
user_id=str(current_user.id),
connection_id=str(connection_id),
parent_ref=invalidate_parent,
refresh_state="warning",
error_code="upload_mutated",
error_message="Folder contents changed by upload — re-listing required.",
)
```
The folder state is set to `warning` with `error_code="upload_mutated"` (not `apply_listing_and_finalize`) because we do not have a full provider listing — we only know one new item arrived. The browse endpoint will detect the non-fresh state and trigger a provider re-list on the next navigation.
### Task 2: Metadata-only upload audit row
**RED phase** promoted `test_upload_success_writes_metadata_only_audit_row` from `xfail` to a real test. The test now uses a mock adapter so it reliably triggers the 200 success path and checks for a `cloud.file_uploaded` audit row.
**GREEN phase** added audit write in `backend/api/cloud/operations.py`:
```python
await write_audit_log(
session,
event_type="cloud.file_uploaded",
user_id=current_user.id,
actor_id=current_user.id,
resource_id=None,
ip_address=get_client_ip(request),
metadata_={
"connection_id": str(connection_id),
"provider_item_id": provider_item_id,
"filename": resolved_name,
"size_bytes": resolved_size,
"parent_ref": resolved_parent_ref,
},
)
await session.commit()
```
The audit payload is metadata-only: no provider URLs, access tokens, refresh tokens, document content, or raw bytes. The `session.commit()` is called once after upsert + folder state + audit — all three writes land in a single atomic transaction.
Non-success paths (`MUT_KIND_CONFLICT`, `MUT_KIND_OFFLINE`, `MUT_KIND_REAUTH`) branch before this code is reached, so they cannot produce false `cloud.file_uploaded` events (T-13-21).
## Deviations from Plan
### Auto-fixed Issues
None. The plan was executed exactly as written.
**Scope note:** The plan listed `backend/api/cloud/operations.py`, `backend/services/cloud_operations.py`, `backend/services/cloud_items.py`, `backend/tests/test_cloud_mutations.py`, and `backend/tests/test_cloud_audit.py` as files modified. In practice, `cloud_operations.py` and `cloud_items.py` were not modified — the existing `upsert_cloud_item`, `update_folder_state`, and `write_audit_log` helpers were sufficient. The route layer in `operations.py` consumed them directly per the CLAUDE.md shared module map.
## Known Stubs
None. Upload reconciliation and audit are fully functional. Remaining xfail tests in `test_cloud_audit.py` cover rename, move, delete, and folder operations which are deferred to Plans 0711.
## Threat Flags
No new security surfaces introduced. T-13-19, T-13-20, and T-13-21 are mitigated:
- **T-13-19** (upload reconciliation bypass): Tests verify reconcile-before-return; non-success paths bypass the upsert/folder-state path.
- **T-13-20** (audit payload secrecy): Audit metadata contains only controlled metadata fields; no provider token, URL, or content.
- **T-13-21** (false success audit events): Conflict, offline, and reauth_required branches never reach the audit write call.
## Self-Check: PASSED
- FOUND: `backend/api/cloud/operations.py` with `upsert_cloud_item`, `update_folder_state`, `write_audit_log` calls in MUT_KIND_UPLOADED branch
- FOUND: `backend/tests/test_cloud_mutations.py` with 3 new reconciliation tests
- FOUND: `backend/tests/test_cloud_audit.py` with `test_upload_success_writes_metadata_only_audit_row` promoted from xfail
- FOUND commit: `4cd6499` (Task 1 RED)
- FOUND commit: `7ecbec7` (Task 1 GREEN)
- FOUND commit: `33f0498` (Task 2 RED)
- FOUND commit: `d959e0c` (Task 2 GREEN)
- Full suite: 745 passed, 17 skipped, 4 deselected, 12 xfailed