docs(05-05): complete cloud API endpoints plan — SUMMARY and STATE
- Created 05-05-SUMMARY.md: cloud.py (7 endpoints), main.py (router registration), admin.py (SEC-09 cleanup) - Updated STATE.md: plan advanced to 5/8, session log updated, decisions recorded - Updated ROADMAP.md: 05-03, 05-04, 05-05 marked complete - Updated REQUIREMENTS.md: SEC-09 marked complete (cloud credential purge on account deletion)
This commit is contained in:
curo1305
@@ -25,7 +25,7 @@ _Last updated: 2026-05-21_
|
||||
- [ ] **SEC-06**: Constant-time comparison used for all token and code verification (prevents timing attacks)
|
||||
- [ ] **SEC-07**: Admin role verified on every admin endpoint request; admin cannot access document content, extracted text, or cloud credentials in any response
|
||||
- [ ] **SEC-08**: Cloud credential ciphertext (`credentials_enc`) excluded from all API serializers by default — admin and user responses return only `provider, display_name, connected_at, status`
|
||||
- [ ] **SEC-09**: Account deletion triggers `delete_user_files()` on every active cloud connection before removing DB records (prevents orphaned cloud data and satisfies GDPR Article 17)
|
||||
- [x] **SEC-09**: Account deletion triggers `delete_user_files()` on every active cloud connection before removing DB records (prevents orphaned cloud data and satisfies GDPR Article 17)
|
||||
|
||||
### Users & Admin (ADMIN)
|
||||
|
||||
|
||||
@@ -231,12 +231,12 @@ Before any phase is marked complete, all three gates must pass:
|
||||
|
||||
**Wave 3** — Cloud backends (parallel, both blocked on Wave 2 / Plan 05-02)
|
||||
|
||||
- [ ] 05-03-PLAN.md — GoogleDriveBackend + OneDriveBackend (all 7 StorageBackend methods)
|
||||
- [ ] 05-04-PLAN.md — NextcloudBackend + WebDAVBackend (all 7 StorageBackend methods)
|
||||
- [x] 05-03-PLAN.md — GoogleDriveBackend + OneDriveBackend (all 7 StorageBackend methods)
|
||||
- [x] 05-04-PLAN.md — NextcloudBackend + WebDAVBackend (all 7 StorageBackend methods)
|
||||
|
||||
**Wave 4** — Cloud API
|
||||
|
||||
- [ ] 05-05-PLAN.md — All /api/cloud/* endpoints + /api/users/me/default-storage + main.py router registration
|
||||
- [x] 05-05-PLAN.md — All /api/cloud/* endpoints + /api/users/me/default-storage + main.py router registration
|
||||
|
||||
**Wave 5** — Document routing + full test suite
|
||||
|
||||
|
||||
+12
-8
@@ -4,13 +4,13 @@ milestone: v1.0
|
||||
milestone_name: milestone
|
||||
current_phase: 5
|
||||
status: executing
|
||||
last_updated: "2026-05-28T19:15:00.000Z"
|
||||
last_updated: "2026-05-29T09:21:57.000Z"
|
||||
progress:
|
||||
total_phases: 5
|
||||
completed_phases: 4
|
||||
total_plans: 32
|
||||
completed_plans: 27
|
||||
percent: 84
|
||||
completed_plans: 28
|
||||
percent: 87
|
||||
---
|
||||
|
||||
# Project State
|
||||
@@ -28,13 +28,13 @@ progress:
|
||||
| 2 | Users & Authentication | ✓ Complete (5/5 plans) |
|
||||
| 3 | Document Migration & Multi-User Isolation | ✓ Complete (5/5 plans, UAT passed, security gate passed) |
|
||||
| 4 | Folders, Sharing, Quotas & Document UX | ✓ Complete (9/9 plans, UAT 14/15 passed, 1 bug fixed) |
|
||||
| 5 | Cloud Storage Backends | In Progress (4/8 plans complete) |
|
||||
| 5 | Cloud Storage Backends | In Progress (5/8 plans complete) |
|
||||
|
||||
## Current Position
|
||||
|
||||
**Phase:** 05-cloud-storage-backends — In Progress
|
||||
**Plan:** 4/8
|
||||
**Progress:** [████████░░] 84%
|
||||
**Plan:** 5/8
|
||||
**Progress:** [████████░░] 87%
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
@@ -131,6 +131,9 @@ progress:
|
||||
| WebDAVBackend SSRF double guard pattern | validate_cloud_url in __init__ (construct-time) AND before every asyncio.to_thread() call — mirrors D-17 requirement for DNS-rebinding mitigation |
|
||||
| nextcloud/webdav dispatch to distinct classes | NextcloudBackend for 'nextcloud' provider (has list_folder); WebDAVBackend for 'webdav' — identical constructor signatures |
|
||||
| webdavclient3 upload_to/download_from confirmed | A1 assumption in RESEARCH.md was correct; verified via runtime dir(Client) inspection before use |
|
||||
| OAuth callback not authenticated via JWT | OAuth redirect flow cannot carry Bearer header; state token (256 bits, TTL 1800s, single-use) provides equivalent security |
|
||||
| Cloud cleanup added to admin delete_user only | auth.py has no DELETE /api/users/me; admin-initiated deletion is the only account deletion code path |
|
||||
| Cloud cleanup runs before MinIO cleanup | credentials still in DB when get_storage_backend_for_document is called; sessions.flush() after conn deletes |
|
||||
|
||||
### Open Questions
|
||||
|
||||
@@ -178,6 +181,7 @@ _Updated at each phase transition._
|
||||
| Last session | 2026-05-28 — Plan 05-02 executed: cloud_utils.py (SSRF+HKDF), cloud_cache.py (TTLCache), storage factory extended; 199 passed / 43 xfailed / 1 pre-existing failure |
|
||||
| Last session | 2026-05-28 — Plan 05-03 executed: GoogleDriveBackend (Drive v3, cache_discovery=False, asyncio.to_thread) + OneDriveBackend (MSAL, resumable upload, CHUNK_SIZE=10MB); 262 passed / 43 xfailed / 1 pre-existing failure |
|
||||
| Last session | 2026-05-28 — Plan 05-04 executed: WebDAVBackend + NextcloudBackend (SSRF double-guard, asyncio.to_thread, list_folder); 262 passed / 43 xfailed / 1 pre-existing failure |
|
||||
| Next action | Execute Plan 05-05: Cloud API Endpoints |
|
||||
| Last session | 2026-05-29 — Plan 05-05 executed: cloud.py (7 endpoints), main.py (routers registered), admin.py (SEC-09 cloud cleanup); 262 passed / 43 xfailed / 1 pre-existing failure |
|
||||
| Next action | Execute Plan 05-06: Cloud Document Upload/Download |
|
||||
| Pending decisions | None |
|
||||
| Resume file | `.planning/phases/05-cloud-storage-backends/05-05-PLAN.md` |
|
||||
| Resume file | `.planning/phases/05-cloud-storage-backends/05-06-PLAN.md` |
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
phase: 05-cloud-storage-backends
|
||||
plan: 05
|
||||
subsystem: api
|
||||
tags: [oauth, google-drive, onedrive, webdav, nextcloud, redis, cloud-storage, audit-log, sec-09]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 03
|
||||
provides: "GoogleDriveBackend, OneDriveBackend, CloudConnectionError — used in cloud.py endpoints"
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 04
|
||||
provides: "WebDAVBackend, NextcloudBackend, validate_cloud_url — WebDAV connect endpoint"
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 02
|
||||
provides: "encrypt_credentials, decrypt_credentials, get_storage_backend_for_document, get_cloud_folders_cached"
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 01
|
||||
provides: "CloudConnection model, settings (cloud_creds_key, google_*, onedrive_*, backend_url, frontend_url)"
|
||||
provides:
|
||||
- "backend/api/cloud.py: 6 cloud endpoints on /api/cloud/* + 1 endpoint on /api/users/me/default-storage"
|
||||
- "backend/main.py: cloud_router and cloud_users_router registered"
|
||||
- "backend/api/admin.py: cloud credential cleanup on admin user deletion (SEC-09)"
|
||||
affects: [05-06, 05-07, 05-08]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "_call_cloud_op helper: transparent token refresh (on token_expired) + REQUIRES_REAUTH (on invalid_grant) — wraps all cloud ops"
|
||||
- "OAuth state token: secrets.token_urlsafe(32) stored in Redis as 'oauth_state:{token}' with TTL 1800, single-use delete"
|
||||
- "CloudConnectionOut imported from api.admin — never redefined; same whitelist enforces credentials_enc exclusion everywhere"
|
||||
- "OAuth callback not authenticated via JWT — state token binds callback to user session"
|
||||
- "cloud.credentials_purged audit event on account deletion with providers list metadata"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- backend/api/cloud.py
|
||||
modified:
|
||||
- backend/main.py
|
||||
- backend/api/admin.py
|
||||
|
||||
key-decisions:
|
||||
- "OAuth callback handler does not require JWT auth — the state token (Redis-bound, TTL 1800, single-use) is the auth mechanism that ties the callback to the initiating user (T-05-05-01, T-05-05-02)"
|
||||
- "DELETE /connections/{connection_id} returns 404 for wrong-owner connections — prevents connection ID enumeration (D-19, T-05-05-04)"
|
||||
- "Cloud cleanup added to admin delete_user (not self-deletion) — auth.py has no DELETE /api/users/me; admin-initiated deletion is the account deletion code path in the codebase"
|
||||
- "_call_cloud_op retries op_fn once on token_expired; builds a new backend instance from refreshed credentials after updating DB"
|
||||
- "Cloud object deletion for cloud-stored documents runs BEFORE MinIO cleanup in delete_user: credentials still in DB when get_storage_backend_for_document is called"
|
||||
|
||||
patterns-established:
|
||||
- "_call_cloud_op pattern: zero-argument async lambda captures backend in closure; helper retries after token refresh"
|
||||
- "invalidate_provider_cache() called on disconnect to invalidate TTLCache entries for that user+provider"
|
||||
- "Folder listing: google_drive uses Drive v3 files.list with cache_discovery=False; onedrive uses httpx against Graph API; nextcloud/webdav use NextcloudBackend.list_folder()"
|
||||
|
||||
requirements-completed:
|
||||
- CLOUD-01
|
||||
- CLOUD-02
|
||||
- CLOUD-03
|
||||
- CLOUD-04
|
||||
- CLOUD-05
|
||||
- CLOUD-06
|
||||
- SEC-09
|
||||
|
||||
# Metrics
|
||||
duration: 12min
|
||||
completed: 2026-05-29
|
||||
---
|
||||
|
||||
# Phase 5 Plan 05: Cloud Connection Management API Summary
|
||||
|
||||
**OAuth initiate/callback for Google Drive and OneDrive, WebDAV/Nextcloud credential connect, connection list/delete, TTL-cached folder listing, default-storage PATCH, and cloud credential purge on account deletion — all 7 endpoints using get_regular_user dep with credentials_enc excluded from all responses**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 12 min
|
||||
- **Started:** 2026-05-29T09:09:57Z
|
||||
- **Completed:** 2026-05-29T09:21:57Z
|
||||
- **Tasks:** 3
|
||||
- **Files modified:** 3
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Created `backend/api/cloud.py` with all 7 endpoints: OAuth initiate and callback (Google Drive + OneDrive), WebDAV/Nextcloud credential connect, connection list, connection delete, folder listing (TTL-cached), and default-storage update. All handlers use `Depends(get_regular_user)` — admin gets 403. `CloudConnectionOut` imported from `api.admin` so the credentials_enc exclusion whitelist is never duplicated.
|
||||
- Implemented `_call_cloud_op` helper for transparent token refresh: retries op_fn once on `token_expired` (decrypt → refresh via provider → encrypt → update DB → rebuild backend → retry), sets `status="REQUIRES_REAUTH"` and writes audit log on `invalid_grant`.
|
||||
- Registered both routers in `main.py` (Phase 5 section after audit router). All 6 cloud routes + `/api/users/me/default-storage` visible in `app.routes`.
|
||||
- Added cloud credential cleanup to `admin.py delete_user` (SEC-09): queries all `CloudConnection` rows, deletes cloud-stored documents via `get_storage_backend_for_document + delete_object` (best-effort), explicitly deletes `CloudConnection` rows with `session.flush()`, writes `cloud.credentials_purged` audit event with providers list.
|
||||
|
||||
## Task Commits
|
||||
|
||||
1. **Task 1: Create cloud.py** - `2424f52` (feat)
|
||||
2. **Task 2: Register cloud routers in main.py** - `f509c37` (feat)
|
||||
3. **Task 3: Cloud cleanup on admin user deletion** - `d85a097` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `/Users/nik/Documents/Progamming/document_scanner/backend/api/cloud.py` — All 7 cloud connection management endpoints + `_call_cloud_op` helper + `_upsert_cloud_connection` helper
|
||||
- `/Users/nik/Documents/Progamming/document_scanner/backend/main.py` — `cloud_router` and `cloud_users_router` registered (Phase 5 section)
|
||||
- `/Users/nik/Documents/Progamming/document_scanner/backend/api/admin.py` — `CloudConnection` + `get_storage_backend_for_document` imported; cloud cleanup block added to `delete_user` before MinIO cleanup
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- OAuth callback endpoint does not require a JWT Bearer token. The OAuth redirect flow happens in the browser: the provider redirects back with `code` and `state`, and the backend validates the `state` against Redis to identify the user. Adding a JWT dep would break the flow (browser doesn't carry the Bearer header in redirect responses). The state token provides equivalent security: 256 bits of entropy, TTL 1800s, single-use deletion.
|
||||
- Cloud cleanup was added to `admin.py delete_user` (not `auth.py`). The `auth.py` module does not implement account self-deletion — there is no `DELETE /api/users/me` endpoint. The admin-initiated deletion (`DELETE /api/admin/users/{id}`) is the only account deletion code path.
|
||||
- Cloud object deletion runs before MinIO document deletion in `delete_user` so that `get_storage_backend_for_document` can decrypt credentials and build the correct backend. After `session.delete(conn)` + `session.flush()`, the credentials would be gone.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None — plan executed exactly as written. `cloud.py` was already partially written (untracked file) from a prior aborted attempt; it was inspected and verified complete, then committed. All three tasks executed as specified.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
`cloud.py` already existed as an untracked file from a prior failed execution attempt. Verified it contained all required endpoints, routes, and security invariants before committing. No rework needed.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None. All endpoints have real implementations. The `test_cloud.py` stubs remain xfail because they call `pytest.xfail()` unconditionally inside the test body — they are scaffolding tests that will be replaced by real integration tests in Plan 05-06 or later.
|
||||
|
||||
## Threat Surface Scan
|
||||
|
||||
New network-facing endpoints introduced (all on `router` and `users_router`):
|
||||
|
||||
- `GET /api/cloud/oauth/initiate/{provider}` — initiates OAuth redirect; all providers validated against allowlist
|
||||
- `GET /api/cloud/oauth/callback/{provider}` — receives OAuth code from provider; state validated before any DB write
|
||||
- `POST /api/cloud/connections/webdav` — user-supplied URL validated via `validate_cloud_url` (SSRF) before any outbound call
|
||||
- `GET /api/cloud/connections` — read-only, returns CloudConnectionOut (credentials_enc excluded)
|
||||
- `DELETE /api/cloud/connections/{connection_id}` — returns 404 for wrong-owner (enumeration prevention)
|
||||
- `GET /api/cloud/folders/{provider}/{folder_id}` — makes outbound call to cloud provider using decrypted credentials; TTL-cached
|
||||
- `PATCH /api/users/me/default-storage` — updates a single user column; no outbound calls
|
||||
|
||||
All STRIDE threats from the plan's threat model are mitigated as implemented:
|
||||
- T-05-05-01: `secrets.token_urlsafe(32)` state token in Redis — confirmed
|
||||
- T-05-05-02: Redis key deleted immediately on callback validation — confirmed
|
||||
- T-05-05-03: `CloudConnectionOut` imported from admin.py — same whitelist — confirmed
|
||||
- T-05-05-04: DELETE returns 404 for wrong-owner — confirmed
|
||||
- T-05-05-05: `validate_cloud_url` called before WebDAV backend instantiation — confirmed
|
||||
- T-05-05-06: `Depends(get_regular_user)` on all endpoints — confirmed
|
||||
- T-05-05-08: audit metadata = `{"provider": provider}` only — confirmed
|
||||
|
||||
No threat flags raised beyond those already documented in the plan.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- All cloud connection management endpoints are live and importable. Plans 05-06 through 05-08 can begin building on top of the API layer.
|
||||
- `_call_cloud_op` is available for use in upload/download handlers in Plan 05-06.
|
||||
- `_build_backend` helper in `cloud.py` reconstructs any backend from provider name + credentials dict.
|
||||
- 262 tests pass / 43 xfailed / 1 pre-existing failure (`test_extract_docx` — python-docx not installed).
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
Files verified present:
|
||||
- `backend/api/cloud.py`: FOUND
|
||||
- `backend/main.py`: FOUND (cloud_router registered)
|
||||
- `backend/api/admin.py`: FOUND (cloud cleanup present)
|
||||
|
||||
Commits verified:
|
||||
- 2424f52: feat(05-05): implement cloud.py — FOUND
|
||||
- f509c37: feat(05-05): register cloud and users routers in main.py — FOUND
|
||||
- d85a097: feat(05-05): add cloud credential cleanup on admin user deletion — FOUND
|
||||
|
||||
---
|
||||
*Phase: 05-cloud-storage-backends*
|
||||
*Completed: 2026-05-29*
|
||||
Reference in New Issue
Block a user