diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 252526c..74aa2a3 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -24,7 +24,7 @@ Before any phase is marked complete: | Phase | Name | Goal | Requirements | |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | -| 13 | 5/11 | In Progress| | +| 13 | 6/11 | In Progress| | | 14 | Selective Analysis and Byte Cache | Analyze selected cloud scopes through cancellable jobs backed by bounded, private, on-demand byte caching | ANALYZE-01..07, CACHE-03..05 | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -80,7 +80,7 @@ Plans: **Depends on:** Phase 12 **Requirements:** CONN-01, CONN-02, CONN-03, CLOUD-02, CLOUD-03, CLOUD-04, CLOUD-05, CLOUD-06, CLOUD-07, CLOUD-09 -**Plans:** 5/11 plans executed +**Plans:** 6/11 plans executed **Execution waves:** Wave 0: 13-01 and 13-02 in parallel; Wave 1: 13-03 after 13-01; Wave 2: 13-04 after 13-01 and 13-03; Wave 3: 13-05 after 13-01, 13-03, and 13-04; Wave 4: 13-06 after 13-05; Wave 5: 13-07 and 13-08 in parallel (13-07 after 13-02, 13-04, and 13-06; 13-08 after 13-03, 13-04, and 13-06); Wave 6: 13-09 after 13-08; Wave 7: 13-10 after 13-02, 13-04, 13-07, and 13-09; Wave 8: 13-11 after 13-03 through 13-10. Plans: @@ -90,7 +90,7 @@ Plans: - [x] 13-03-PLAN.md — Build the mutable cloud contract and orchestration seam without breaking centralized reconciliation. - [x] 13-04-PLAN.md — Implement connection-ID reconnect, explicit health test, broader Drive scope handling, and authorized content routes. - [x] 13-05-PLAN.md — Implement backend upload provider mechanics, typed route results, and refreshed-credential handoff. -- [ ] 13-06-PLAN.md — Complete upload reconcile, freshness, and metadata-only audit follow-through before frontend queue wiring. +- [x] 13-06-PLAN.md — Complete upload reconcile, freshness, and metadata-only audit follow-through before frontend queue wiring. - [ ] 13-07-PLAN.md — Wire the shared browser queue and binary-only preview or download fallback through thin cloud view handlers. - [ ] 13-08-PLAN.md — Implement backend create-folder and rename semantics with collision, stale, and stable-identity safeguards. - [ ] 13-09-PLAN.md — Implement backend move and delete semantics with same-connection, disclosure, security, and audit safeguards. diff --git a/.planning/STATE.md b/.planning/STATE.md index bef6d95..66274d7 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,14 +6,14 @@ current_phase: 13 current_phase_name: virtual-local-cloud-operations status: executing stopped_at: Phase 13 context gathered -last_updated: "2026-06-22T17:22:26.784Z" +last_updated: "2026-06-22T17:31:22.350Z" last_activity: 2026-06-22 last_activity_desc: Phase 13 execution started progress: total_phases: 6 completed_phases: 2 total_plans: 21 - completed_plans: 15 + completed_plans: 16 percent: 33 --- @@ -26,7 +26,7 @@ progress: ## Current Position Phase: 13 (virtual-local-cloud-operations) — EXECUTING -Plan: 6 of 11 +Plan: 7 of 11 Status: Ready to execute Last activity: 2026-06-22 — Phase 13 execution started @@ -56,6 +56,7 @@ Last activity: 2026-06-22 — Phase 13 execution started | Phase 13 P02 | 30m | 2 tasks | 6 files | | Phase 13 P03 | 134 | 2 tasks | 8 files | | Phase 13 P04 | 180m | 2 tasks | 7 files | +| Phase 13 P06 | 5m | 2 tasks | 3 files | ## Accumulated Context @@ -99,7 +100,7 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-22T17:22:26.779Z | +| Last session | 2026-06-22T17:31:12.657Z | | Next action | Execute Phase 12.1 Plan 01 | | Pending decisions | None | | Resume file | None | @@ -119,3 +120,4 @@ _Updated at each phase transition._ - [Phase ?]: D-17: Google Drive OAuth uses drive scope (not drive.file) for Phase 13 full-access mutations - [Phase ?]: D-18: Preview is binary-only (PDF/images); Office/Workspace formats use typed unsupported_preview fallback to authorized download endpoint - [Phase ?]: Phase 13 mutation errors use JSONResponse (not HTTPException) so kind/reason appear at top level of response body, not nested under detail +- [Phase ?]: upload_mutated folder state code: upload success marks parent folder as warning/upload_mutated for reconcile-before-return diff --git a/.planning/phases/13-virtual-local-cloud-operations/13-06-SUMMARY.md b/.planning/phases/13-virtual-local-cloud-operations/13-06-SUMMARY.md new file mode 100644 index 0000000..362e1d7 --- /dev/null +++ b/.planning/phases/13-virtual-local-cloud-operations/13-06-SUMMARY.md @@ -0,0 +1,180 @@ +--- +phase: "13" +plan: "06" +subsystem: cloud-operations +status: complete +tags: + - cloud + - upload + - reconciliation + - audit + - cloud-items + - tdd + +dependency_graph: + requires: + - "13-01 (RED mutation contract tests)" + - "13-03 (cloud operations seam + mutable provider implementations)" + - "13-04 (operations.py route layer + typed result vocabulary)" + - "13-05 (keep_both_name + upload mechanics tests)" + provides: + - "Upload success routes through upsert_cloud_item for stable row identity" + - "Parent folder state invalidated (warning/upload_mutated) after success" + - "cloud.file_uploaded audit row written in same transaction as reconciliation" + - "3 Task 1 reconciliation tests in test_cloud_mutations.py" + - "1 Task 2 upload audit test promoted from xfail in test_cloud_audit.py" + affects: + - "13-07 through 13-11 (rename, move, delete, folder audit and reconciliation plans)" + +tech_stack: + added: + - "upsert_cloud_item called from upload route on MUT_KIND_UPLOADED success" + - "update_folder_state called with warning/upload_mutated on upload success" + - "write_audit_log called with cloud.file_uploaded event on upload success" + - "get_client_ip imported from deps.utils in operations.py" + - "write_audit_log imported from services.audit in operations.py" + patterns: + - "Reconcile-before-return: upsert + folder invalidation complete before response" + - "Audit-in-transaction: write_audit_log flushes in caller transaction, caller commits" + - "Non-success bypass: conflict/offline/reauth paths never reach reconciliation or audit" + - "upload_mutated folder state: controlled code signaling re-list without a full provider scan" + +key_files: + modified: + - path: "backend/api/cloud/operations.py" + change: "Upload success path now calls upsert_cloud_item, update_folder_state, write_audit_log before commit" + - path: "backend/tests/test_cloud_mutations.py" + change: "Added 3 reconciliation behavioral tests for Task 1 (RED then GREEN)" + - path: "backend/tests/test_cloud_audit.py" + change: "Promoted test_upload_success_writes_metadata_only_audit_row from xfail to real test (RED then GREEN)" + +decisions: + - "upload_mutated is the controlled error_code for folder invalidation after upload — avoids apply_listing_and_finalize which needs a full provider listing" + - "CloudResource.id assigned uuid4() at route layer before upsert — cloud_items.upsert_cloud_item uses provider_item_id as the stable identity key, not this id" + - "Audit write uses flush (not commit) per services.audit contract; one commit covers upsert + folder state + audit atomically" + - "Non-success upload paths (conflict, offline, reauth_required) do not call write_audit_log, satisfying T-13-21 (no false success events)" + +metrics: + duration: "~5 minutes" + completed: "2026-06-22" + tasks_completed: 2 + tasks_planned: 2 + files_changed: 3 + files_created: 0 + tests_added: 4 + tests_passing: 745 +--- + +# Phase 13 Plan 06: Upload Follow-Through Slice Summary + +**One-liner:** Upload success routes through `upsert_cloud_item` + folder state invalidation + metadata-only `cloud.file_uploaded` audit row before the response is returned. + +## Tasks Completed + +| Task | Name | Commit | Key Files | +|------|------|--------|-----------| +| 1 (RED) | Add failing upload reconciliation tests | 4cd6499 | test_cloud_mutations.py | +| 1 (GREEN) | Route upload success through centralized reconciliation | 7ecbec7 | api/cloud/operations.py | +| 2 (RED) | Promote upload audit test from xfail | 33f0498 | test_cloud_audit.py | +| 2 (GREEN) | Emit metadata-only audit row on upload success | d959e0c | api/cloud/operations.py | + +## What Was Built + +### Task 1: Reconcile-before-return on upload success + +**RED phase** added 3 tests to `test_cloud_mutations.py`: + +- `test_upload_success_upserts_cloud_item_before_returning` — verifies that after a 200 upload response, a CloudItem row exists in the DB with the correct `provider_item_id` and `name`. This was the core reconcile-before-return behavioral test. +- `test_upload_success_marks_folder_freshness_stale` — verifies that a `CloudFolderState` row exists for the parent folder after success (folder state created/updated to signal re-list needed). +- `test_upload_failed_does_not_mutate_cloud_items` — verifies offline/error upload results do not create phantom CloudItem rows. + +**GREEN phase** implemented in `backend/api/cloud/operations.py`: + +Inside the `MUT_KIND_UPLOADED` success branch of `upload_cloud_file`: + +```python +resource = CloudResource( + id=uuid.uuid4(), + provider_item_id=provider_item_id, + connection_id=connection_id, + user_id=current_user.id, + name=resolved_name, + kind="file", + parent_ref=resolved_parent_ref, + content_type=content_type, + size=resolved_size, +) +await upsert_cloud_item(session, user_id=str(current_user.id), resource=resource) + +await update_folder_state( + session, + user_id=str(current_user.id), + connection_id=str(connection_id), + parent_ref=invalidate_parent, + refresh_state="warning", + error_code="upload_mutated", + error_message="Folder contents changed by upload — re-listing required.", +) +``` + +The folder state is set to `warning` with `error_code="upload_mutated"` (not `apply_listing_and_finalize`) because we do not have a full provider listing — we only know one new item arrived. The browse endpoint will detect the non-fresh state and trigger a provider re-list on the next navigation. + +### Task 2: Metadata-only upload audit row + +**RED phase** promoted `test_upload_success_writes_metadata_only_audit_row` from `xfail` to a real test. The test now uses a mock adapter so it reliably triggers the 200 success path and checks for a `cloud.file_uploaded` audit row. + +**GREEN phase** added audit write in `backend/api/cloud/operations.py`: + +```python +await write_audit_log( + session, + event_type="cloud.file_uploaded", + user_id=current_user.id, + actor_id=current_user.id, + resource_id=None, + ip_address=get_client_ip(request), + metadata_={ + "connection_id": str(connection_id), + "provider_item_id": provider_item_id, + "filename": resolved_name, + "size_bytes": resolved_size, + "parent_ref": resolved_parent_ref, + }, +) +await session.commit() +``` + +The audit payload is metadata-only: no provider URLs, access tokens, refresh tokens, document content, or raw bytes. The `session.commit()` is called once after upsert + folder state + audit — all three writes land in a single atomic transaction. + +Non-success paths (`MUT_KIND_CONFLICT`, `MUT_KIND_OFFLINE`, `MUT_KIND_REAUTH`) branch before this code is reached, so they cannot produce false `cloud.file_uploaded` events (T-13-21). + +## Deviations from Plan + +### Auto-fixed Issues + +None. The plan was executed exactly as written. + +**Scope note:** The plan listed `backend/api/cloud/operations.py`, `backend/services/cloud_operations.py`, `backend/services/cloud_items.py`, `backend/tests/test_cloud_mutations.py`, and `backend/tests/test_cloud_audit.py` as files modified. In practice, `cloud_operations.py` and `cloud_items.py` were not modified — the existing `upsert_cloud_item`, `update_folder_state`, and `write_audit_log` helpers were sufficient. The route layer in `operations.py` consumed them directly per the CLAUDE.md shared module map. + +## Known Stubs + +None. Upload reconciliation and audit are fully functional. Remaining xfail tests in `test_cloud_audit.py` cover rename, move, delete, and folder operations which are deferred to Plans 07–11. + +## Threat Flags + +No new security surfaces introduced. T-13-19, T-13-20, and T-13-21 are mitigated: + +- **T-13-19** (upload reconciliation bypass): Tests verify reconcile-before-return; non-success paths bypass the upsert/folder-state path. +- **T-13-20** (audit payload secrecy): Audit metadata contains only controlled metadata fields; no provider token, URL, or content. +- **T-13-21** (false success audit events): Conflict, offline, and reauth_required branches never reach the audit write call. + +## Self-Check: PASSED + +- FOUND: `backend/api/cloud/operations.py` with `upsert_cloud_item`, `update_folder_state`, `write_audit_log` calls in MUT_KIND_UPLOADED branch +- FOUND: `backend/tests/test_cloud_mutations.py` with 3 new reconciliation tests +- FOUND: `backend/tests/test_cloud_audit.py` with `test_upload_success_writes_metadata_only_audit_row` promoted from xfail +- FOUND commit: `4cd6499` (Task 1 RED) +- FOUND commit: `7ecbec7` (Task 1 GREEN) +- FOUND commit: `33f0498` (Task 2 RED) +- FOUND commit: `d959e0c` (Task 2 GREEN) +- Full suite: 745 passed, 17 skipped, 4 deselected, 12 xfailed