From cad5ef5de2a4068e2f1284a548af2b58f867a91c Mon Sep 17 00:00:00 2001 From: curo1305 Date: Mon, 22 Jun 2026 19:22:40 +0200 Subject: [PATCH] docs(13-05): complete upload mechanics plan --- .planning/ROADMAP.md | 8 +- .planning/STATE.md | 12 +- .../13-05-SUMMARY.md | 157 ++++++++++++++++++ CLAUDE.md | 3 +- 4 files changed, 171 insertions(+), 9 deletions(-) create mode 100644 .planning/phases/13-virtual-local-cloud-operations/13-05-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index d099cf1..252526c 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -24,7 +24,7 @@ Before any phase is marked complete: | Phase | Name | Goal | Requirements | |------:|------|------|--------------| | 12 | 6/6 | Complete | 2026-06-21 | -| 13 | 3/11 | In Progress| | +| 13 | 5/11 | In Progress| | | 14 | Selective Analysis and Byte Cache | Analyze selected cloud scopes through cancellable jobs backed by bounded, private, on-demand byte caching | ANALYZE-01..07, CACHE-03..05 | | 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 | | 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 | @@ -80,7 +80,7 @@ Plans: **Depends on:** Phase 12 **Requirements:** CONN-01, CONN-02, CONN-03, CLOUD-02, CLOUD-03, CLOUD-04, CLOUD-05, CLOUD-06, CLOUD-07, CLOUD-09 -**Plans:** 3/11 plans executed +**Plans:** 5/11 plans executed **Execution waves:** Wave 0: 13-01 and 13-02 in parallel; Wave 1: 13-03 after 13-01; Wave 2: 13-04 after 13-01 and 13-03; Wave 3: 13-05 after 13-01, 13-03, and 13-04; Wave 4: 13-06 after 13-05; Wave 5: 13-07 and 13-08 in parallel (13-07 after 13-02, 13-04, and 13-06; 13-08 after 13-03, 13-04, and 13-06); Wave 6: 13-09 after 13-08; Wave 7: 13-10 after 13-02, 13-04, 13-07, and 13-09; Wave 8: 13-11 after 13-03 through 13-10. Plans: @@ -88,8 +88,8 @@ Plans: - [x] 13-01-PLAN.md — Create red backend and provider-contract suites for reconnect, content, mutations, and audit secrecy. - [x] 13-02-PLAN.md — Create red frontend/store suites for queue, preview, health UX, broader Drive consent, and no-probe-on-navigation. - [x] 13-03-PLAN.md — Build the mutable cloud contract and orchestration seam without breaking centralized reconciliation. -- [ ] 13-04-PLAN.md — Implement connection-ID reconnect, explicit health test, broader Drive scope handling, and authorized content routes. -- [ ] 13-05-PLAN.md — Implement backend upload provider mechanics, typed route results, and refreshed-credential handoff. +- [x] 13-04-PLAN.md — Implement connection-ID reconnect, explicit health test, broader Drive scope handling, and authorized content routes. +- [x] 13-05-PLAN.md — Implement backend upload provider mechanics, typed route results, and refreshed-credential handoff. - [ ] 13-06-PLAN.md — Complete upload reconcile, freshness, and metadata-only audit follow-through before frontend queue wiring. - [ ] 13-07-PLAN.md — Wire the shared browser queue and binary-only preview or download fallback through thin cloud view handlers. - [ ] 13-08-PLAN.md — Implement backend create-folder and rename semantics with collision, stale, and stable-identity safeguards. diff --git a/.planning/STATE.md b/.planning/STATE.md index e31f05b..bef6d95 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,14 +6,14 @@ current_phase: 13 current_phase_name: virtual-local-cloud-operations status: executing stopped_at: Phase 13 context gathered -last_updated: "2026-06-22T16:45:34.553Z" +last_updated: "2026-06-22T17:22:26.784Z" last_activity: 2026-06-22 last_activity_desc: Phase 13 execution started progress: total_phases: 6 completed_phases: 2 total_plans: 21 - completed_plans: 13 + completed_plans: 15 percent: 33 --- @@ -26,7 +26,7 @@ progress: ## Current Position Phase: 13 (virtual-local-cloud-operations) — EXECUTING -Plan: 4 of 11 +Plan: 6 of 11 Status: Ready to execute Last activity: 2026-06-22 — Phase 13 execution started @@ -55,6 +55,7 @@ Last activity: 2026-06-22 — Phase 13 execution started | Phase 12.1 P04 | 15m | 4 tasks | 10 files | | Phase 13 P02 | 30m | 2 tasks | 6 files | | Phase 13 P03 | 134 | 2 tasks | 8 files | +| Phase 13 P04 | 180m | 2 tasks | 7 files | ## Accumulated Context @@ -98,7 +99,7 @@ _Updated at each phase transition._ | Field | Value | |---|---| -| Last session | 2026-06-22T16:45:34.548Z | +| Last session | 2026-06-22T17:22:26.779Z | | Next action | Execute Phase 12.1 Plan 01 | | Pending decisions | None | | Resume file | None | @@ -115,3 +116,6 @@ _Updated at each phase transition._ - [Phase ?]: file-open must call openCloudFile API; window.open() to provider URL is forbidden (D-02/T-13-07) - [Phase ?]: connectionHealth store translation is single source for browser compact status and Settings diagnostics (D-12) - [Phase ?]: testCloudConnection never called as side effect of folder browse navigation (D-13) +- [Phase ?]: D-17: Google Drive OAuth uses drive scope (not drive.file) for Phase 13 full-access mutations +- [Phase ?]: D-18: Preview is binary-only (PDF/images); Office/Workspace formats use typed unsupported_preview fallback to authorized download endpoint +- [Phase ?]: Phase 13 mutation errors use JSONResponse (not HTTPException) so kind/reason appear at top level of response body, not nested under detail diff --git a/.planning/phases/13-virtual-local-cloud-operations/13-05-SUMMARY.md b/.planning/phases/13-virtual-local-cloud-operations/13-05-SUMMARY.md new file mode 100644 index 0000000..dd25016 --- /dev/null +++ b/.planning/phases/13-virtual-local-cloud-operations/13-05-SUMMARY.md @@ -0,0 +1,157 @@ +--- +phase: "13" +plan: "05" +subsystem: cloud-operations +status: complete +tags: + - cloud + - upload + - conflict + - keep-both + - typed-results + - tdd + +dependency_graph: + requires: + - "13-01 (RED mutable adapter contract tests)" + - "13-03 (cloud operations seam + mutable provider implementations)" + - "13-04 (operations.py route layer + typed result vocabulary)" + provides: + - "keep_both_name() helper in services/cloud_operations.py for D-03/D-05 collision naming" + - "TestUploadConflictSemantics: 16 provider-level upload behavioral tests" + - "TestKeepBothNaming: 7 keep-both naming format tests" + - "7 typed upload route mechanics tests in test_cloud_mutations.py" + - "All 4 providers verified to return typed upload results (never raw provider errors)" + affects: + - "13-06 through 13-11 (reconcile, audit, frontend upload queue plans)" + +tech_stack: + added: + - "keep_both_name(filename, counter) in services/cloud_operations.py" + patterns: + - "Counter-before-extension naming: Report.pdf → Report (1).pdf" + - "Typed upload result vocabulary: MUT_KIND_UPLOADED / MUT_KIND_CONFLICT / MUT_KIND_OFFLINE / MUT_KIND_REAUTH" + - "Provider-neutral upload mechanics: all 4 providers normalize errors into shared kinds" + - "Queue-control semantics from backend-authored typed bodies (not Vue-side HTTP status inference)" + +key_files: + modified: + - path: "backend/services/cloud_operations.py" + change: "Added keep_both_name(filename, counter) helper for D-03/D-05 collision naming" + - path: "backend/tests/test_cloud_backends.py" + change: "Added TestUploadConflictSemantics (16 tests) and TestKeepBothNaming (7 tests)" + - path: "backend/tests/test_cloud_mutations.py" + change: "Added 7 typed upload route mechanics tests (Task 2 behaviors)" + +decisions: + - "keep_both_name lives in services/cloud_operations (not a provider) — naming is queue/service concern, not provider concern" + - "Counter inserted before first dot to preserve compound extensions (archive.tar.gz → archive (1).tar.gz)" + - "Providers already had upload_file implementations from Plan 03 — Plan 05 adds behavioral tests and the naming utility only" + - "Upload route already had typed result handling from Plan 04 — Plan 05 confirms test coverage of all outcome paths" + - "Reconcile and audit follow-through deferred to Plans 06+ per PLAN.md bounded scope" + +metrics: + duration: "~45 minutes" + completed: "2026-06-22" + tasks_completed: 2 + tasks_planned: 2 + files_changed: 3 + files_created: 0 + tests_added: 30 + tests_passing: 741 +--- + +# Phase 13 Plan 05: Upload Mechanics Slice Summary + +**One-liner:** Provider-normalized upload conflict/retry/reauth typed mechanics with `keep_both_name()` helper for D-03/D-05 queue-level collision resolution. + +## Tasks Completed + +| Task | Name | Commit | Key Files | +|------|------|--------|-----------| +| 1 (RED) | Add failing upload conflict and keep-both naming tests | f47e36d | test_cloud_backends.py | +| 1 (GREEN) | Implement keep_both_name() helper | e1ce4cb | services/cloud_operations.py | +| 2 | Add typed upload route mechanics tests | 3ddf4da | test_cloud_mutations.py | + +## What Was Built + +### Task 1: Provider-normalized upload contract tests + keep_both_name() helper + +**RED phase:** Added 23 tests (16 provider behavioral + 7 keep-both naming) to `test_cloud_backends.py`: + +**TestUploadConflictSemantics:** +- `test_google_drive_upload_success_returns_typed_result` — kind='uploaded' with provider_item_id, name, size; no credentials in result +- `test_google_drive_upload_transient_error_returns_offline_kind` — HTTP 503 → kind='offline' (retryable) +- `test_google_drive_upload_auth_error_returns_reauth_kind` — HTTP 401 → kind='reauth_required' +- `test_onedrive_upload_success_returns_typed_result` — resumable session success → kind='uploaded' +- `test_onedrive_upload_conflict_returns_typed_conflict_kind` — conflictBehavior=fail 409 → kind='conflict', reason='name_collision' +- `test_onedrive_upload_never_writes_cloud_items` — source scan confirms no cloud_items import +- `test_webdav_upload_success_returns_typed_result` — PUT success → kind='uploaded' with correct path +- `test_webdav_upload_provider_error_returns_error_kind` — connection timeout → kind='offline' +- `test_upload_result_never_contains_credentials` — parametrized over 4 providers; no access_token/refresh_token in result +- `test_upload_result_kind_in_known_kinds` — parametrized over 4 providers; kind must be in MUT_KINDS constant set + +**TestKeepBothNaming:** +- `test_keep_both_naming_module_exists` — services.cloud_operations exposes keep_both_name +- `test_keep_both_basic_pdf` — Report.pdf + 1 → Report (1).pdf +- `test_keep_both_counter_increments` — Report.pdf + 2 → Report (2).pdf +- `test_keep_both_no_extension` — README + 1 → README (1) +- `test_keep_both_compound_extension` — archive.tar.gz + 1 → archive (1).tar.gz +- `test_keep_both_preserves_existing_counter` — Report (1).pdf + 2 → Report (1) (2).pdf +- `test_keep_both_spaces_in_name` — My Document.docx + 1 → My Document (1).docx + +**GREEN phase:** Implemented `keep_both_name(filename, counter)` in `services/cloud_operations.py`: + +```python +def keep_both_name(filename: str, counter: int) -> str: + """Insert collision counter before first file extension. + + Report.pdf + 1 → Report (1).pdf + archive.tar.gz + 1 → archive (1).tar.gz + README + 1 → README (1) + """ +``` + +The counter is inserted before the first dot (not last), so compound extensions are preserved intact. + +### Task 2: Typed upload route mechanics tests + +Added 7 tests to `test_cloud_mutations.py` verifying Task 2 behavioral contract: + +1. `test_upload_success_returns_typed_uploaded_body` — success body has kind/provider_item_id/name/size +2. `test_upload_provider_retryable_error_returns_offline_kind` — offline kind from provider error, no bare 500 +3. `test_upload_provider_conflict_returns_conflict_kind` — provider-detected conflict when cache misses +4. `test_upload_reauth_result_is_typed_and_not_500` — CONN-02 token expiry surfaces as typed body +5. `test_upload_queue_control_semantics_are_backend_authored` — all 4 required queue fields present +6. `test_upload_keep_both_name_format` — service utility accessible to route layer +7. `test_upload_foreign_user_blocked` — IDOR protection on upload endpoint + +All 7 tests pass because the upload route in `operations.py` (Plan 04) already implements the typed result dispatch correctly. Plan 05 adds the test coverage that was missing. + +## Deviations from Plan + +### Auto-fixed Issues + +None. The plan stated providers should "implement D-03 and D-04 inside the provider layer" but providers already had complete `upload_file` implementations from Plan 03. Plan 05 correctly adds behavioral tests and the naming utility that were genuinely missing. + +**Note on scope:** The providers already normalized upload results into typed kinds from Plan 03. Plan 05's contribution is: +1. Behavioral tests that prove the normalization is correct +2. The `keep_both_name()` service helper that the upload queue needs for conflict resolution + +## Known Stubs + +None. The upload mechanics are fully functional. Reconcile and audit follow-through are intentionally deferred to Plans 06+. + +## Threat Flags + +No new security surfaces introduced. Upload route is covered by T-13-16 (typed conflict bodies forbid silent overwrite), T-13-17 (no provider URLs/tokens in response), T-13-18 (WebDAV SSRF guard preserved via __init__). + +## Self-Check: PASSED + +- FOUND: `backend/services/cloud_operations.py` with `keep_both_name()` function +- FOUND: `backend/tests/test_cloud_backends.py` (modified — 23 new tests) +- FOUND: `backend/tests/test_cloud_mutations.py` (modified — 7 new tests) +- FOUND commit: `f47e36d` (Task 1 RED) +- FOUND commit: `e1ce4cb` (Task 1 GREEN — keep_both_name) +- FOUND commit: `3ddf4da` (Task 2 — route mechanics tests) +- Full suite: 741 passed, 17 skipped, 4 deselected, 13 xfailed diff --git a/CLAUDE.md b/CLAUDE.md index bcddc6a..bb1ff5c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ DocuVault is a multi-user SaaS document management platform built on FastAPI (Python) + Vue 3. It handles document upload, text extraction (PDF/DOCX/image/text), AI-based topic classification, per-user isolated storage, folder organization, document sharing, and pluggable cloud storage backends (OneDrive, Google Drive, Nextcloud, WebDAV). -**Current state:** v0.2.6 — Phase 12.1 complete 2026-06-22. Nextcloud root listing functional: legacy `list_folder` signature override removed; incomplete listings blocked from `fresh`; frontend uses `kind` and `provider_item_id`; server freshness mapped verbatim. Live smoke test suite (opt-in, read-only) with owner-confirmed exact-name fixture. Full validation, security, documentation, versioning, and push complete. Not cleared for public internet deployment. +**Current state:** v0.2.6 — Phase 13 Plan 05 complete 2026-06-22. Upload mechanics slice: `keep_both_name()` helper in `services/cloud_operations.py`; typed upload conflict/retry/reauth results across all 4 providers; 37 upload-specific tests pass. Not cleared for public internet deployment. ## Stack @@ -43,6 +43,7 @@ Before adding a helper, check if it belongs in an existing shared module: | `backend/storage/cloud_base.py` | `CloudResourceAdapter`, `CloudListing`, `CloudResource`, `CloudCapability` — Phase 12 browse contract; all 4 providers implement this | | `backend/storage/cloud_utils.py` | `validate_cloud_url`, `normalize_nextcloud_url`, `encrypt_credentials`, `decrypt_credentials` — Phase 12.1 adds `normalize_nextcloud_url` | | `backend/services/cloud_items.py` | `resolve_owned_connection`, `upsert_cloud_item`, `reconcile_cloud_listing`, `get_or_create_folder_state`, `apply_listing_and_finalize`, `ListingResult` — owner-scoped cloud metadata service; `apply_listing_and_finalize` is the single shared freshness gate | +| `backend/services/cloud_operations.py` | `keep_both_name(filename, counter)` — canonical D-03/D-05 keep-both collision naming (counter before first extension); upload queue uses this for conflict resolution | | `backend/api/cloud/schemas.py` | `CloudBrowseResponse`, `CloudItemOut`, `FolderFreshnessOut`, `ConnectionRenameRequest` — credential-free cloud response schemas | **Rules:**