docs(05-05): complete cloud API endpoints plan — SUMMARY and STATE
- Created 05-05-SUMMARY.md: cloud.py (7 endpoints), main.py (router registration), admin.py (SEC-09 cleanup) - Updated STATE.md: plan advanced to 5/8, session log updated, decisions recorded - Updated ROADMAP.md: 05-03, 05-04, 05-05 marked complete - Updated REQUIREMENTS.md: SEC-09 marked complete (cloud credential purge on account deletion)
This commit is contained in:
curo1305
@@ -0,0 +1,163 @@
|
||||
---
|
||||
phase: 05-cloud-storage-backends
|
||||
plan: 05
|
||||
subsystem: api
|
||||
tags: [oauth, google-drive, onedrive, webdav, nextcloud, redis, cloud-storage, audit-log, sec-09]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 03
|
||||
provides: "GoogleDriveBackend, OneDriveBackend, CloudConnectionError — used in cloud.py endpoints"
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 04
|
||||
provides: "WebDAVBackend, NextcloudBackend, validate_cloud_url — WebDAV connect endpoint"
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 02
|
||||
provides: "encrypt_credentials, decrypt_credentials, get_storage_backend_for_document, get_cloud_folders_cached"
|
||||
- phase: 05-cloud-storage-backends
|
||||
plan: 01
|
||||
provides: "CloudConnection model, settings (cloud_creds_key, google_*, onedrive_*, backend_url, frontend_url)"
|
||||
provides:
|
||||
- "backend/api/cloud.py: 6 cloud endpoints on /api/cloud/* + 1 endpoint on /api/users/me/default-storage"
|
||||
- "backend/main.py: cloud_router and cloud_users_router registered"
|
||||
- "backend/api/admin.py: cloud credential cleanup on admin user deletion (SEC-09)"
|
||||
affects: [05-06, 05-07, 05-08]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "_call_cloud_op helper: transparent token refresh (on token_expired) + REQUIRES_REAUTH (on invalid_grant) — wraps all cloud ops"
|
||||
- "OAuth state token: secrets.token_urlsafe(32) stored in Redis as 'oauth_state:{token}' with TTL 1800, single-use delete"
|
||||
- "CloudConnectionOut imported from api.admin — never redefined; same whitelist enforces credentials_enc exclusion everywhere"
|
||||
- "OAuth callback not authenticated via JWT — state token binds callback to user session"
|
||||
- "cloud.credentials_purged audit event on account deletion with providers list metadata"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- backend/api/cloud.py
|
||||
modified:
|
||||
- backend/main.py
|
||||
- backend/api/admin.py
|
||||
|
||||
key-decisions:
|
||||
- "OAuth callback handler does not require JWT auth — the state token (Redis-bound, TTL 1800, single-use) is the auth mechanism that ties the callback to the initiating user (T-05-05-01, T-05-05-02)"
|
||||
- "DELETE /connections/{connection_id} returns 404 for wrong-owner connections — prevents connection ID enumeration (D-19, T-05-05-04)"
|
||||
- "Cloud cleanup added to admin delete_user (not self-deletion) — auth.py has no DELETE /api/users/me; admin-initiated deletion is the account deletion code path in the codebase"
|
||||
- "_call_cloud_op retries op_fn once on token_expired; builds a new backend instance from refreshed credentials after updating DB"
|
||||
- "Cloud object deletion for cloud-stored documents runs BEFORE MinIO cleanup in delete_user: credentials still in DB when get_storage_backend_for_document is called"
|
||||
|
||||
patterns-established:
|
||||
- "_call_cloud_op pattern: zero-argument async lambda captures backend in closure; helper retries after token refresh"
|
||||
- "invalidate_provider_cache() called on disconnect to invalidate TTLCache entries for that user+provider"
|
||||
- "Folder listing: google_drive uses Drive v3 files.list with cache_discovery=False; onedrive uses httpx against Graph API; nextcloud/webdav use NextcloudBackend.list_folder()"
|
||||
|
||||
requirements-completed:
|
||||
- CLOUD-01
|
||||
- CLOUD-02
|
||||
- CLOUD-03
|
||||
- CLOUD-04
|
||||
- CLOUD-05
|
||||
- CLOUD-06
|
||||
- SEC-09
|
||||
|
||||
# Metrics
|
||||
duration: 12min
|
||||
completed: 2026-05-29
|
||||
---
|
||||
|
||||
# Phase 5 Plan 05: Cloud Connection Management API Summary
|
||||
|
||||
**OAuth initiate/callback for Google Drive and OneDrive, WebDAV/Nextcloud credential connect, connection list/delete, TTL-cached folder listing, default-storage PATCH, and cloud credential purge on account deletion — all 7 endpoints using get_regular_user dep with credentials_enc excluded from all responses**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 12 min
|
||||
- **Started:** 2026-05-29T09:09:57Z
|
||||
- **Completed:** 2026-05-29T09:21:57Z
|
||||
- **Tasks:** 3
|
||||
- **Files modified:** 3
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Created `backend/api/cloud.py` with all 7 endpoints: OAuth initiate and callback (Google Drive + OneDrive), WebDAV/Nextcloud credential connect, connection list, connection delete, folder listing (TTL-cached), and default-storage update. All handlers use `Depends(get_regular_user)` — admin gets 403. `CloudConnectionOut` imported from `api.admin` so the credentials_enc exclusion whitelist is never duplicated.
|
||||
- Implemented `_call_cloud_op` helper for transparent token refresh: retries op_fn once on `token_expired` (decrypt → refresh via provider → encrypt → update DB → rebuild backend → retry), sets `status="REQUIRES_REAUTH"` and writes audit log on `invalid_grant`.
|
||||
- Registered both routers in `main.py` (Phase 5 section after audit router). All 6 cloud routes + `/api/users/me/default-storage` visible in `app.routes`.
|
||||
- Added cloud credential cleanup to `admin.py delete_user` (SEC-09): queries all `CloudConnection` rows, deletes cloud-stored documents via `get_storage_backend_for_document + delete_object` (best-effort), explicitly deletes `CloudConnection` rows with `session.flush()`, writes `cloud.credentials_purged` audit event with providers list.
|
||||
|
||||
## Task Commits
|
||||
|
||||
1. **Task 1: Create cloud.py** - `2424f52` (feat)
|
||||
2. **Task 2: Register cloud routers in main.py** - `f509c37` (feat)
|
||||
3. **Task 3: Cloud cleanup on admin user deletion** - `d85a097` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `/Users/nik/Documents/Progamming/document_scanner/backend/api/cloud.py` — All 7 cloud connection management endpoints + `_call_cloud_op` helper + `_upsert_cloud_connection` helper
|
||||
- `/Users/nik/Documents/Progamming/document_scanner/backend/main.py` — `cloud_router` and `cloud_users_router` registered (Phase 5 section)
|
||||
- `/Users/nik/Documents/Progamming/document_scanner/backend/api/admin.py` — `CloudConnection` + `get_storage_backend_for_document` imported; cloud cleanup block added to `delete_user` before MinIO cleanup
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- OAuth callback endpoint does not require a JWT Bearer token. The OAuth redirect flow happens in the browser: the provider redirects back with `code` and `state`, and the backend validates the `state` against Redis to identify the user. Adding a JWT dep would break the flow (browser doesn't carry the Bearer header in redirect responses). The state token provides equivalent security: 256 bits of entropy, TTL 1800s, single-use deletion.
|
||||
- Cloud cleanup was added to `admin.py delete_user` (not `auth.py`). The `auth.py` module does not implement account self-deletion — there is no `DELETE /api/users/me` endpoint. The admin-initiated deletion (`DELETE /api/admin/users/{id}`) is the only account deletion code path.
|
||||
- Cloud object deletion runs before MinIO document deletion in `delete_user` so that `get_storage_backend_for_document` can decrypt credentials and build the correct backend. After `session.delete(conn)` + `session.flush()`, the credentials would be gone.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None — plan executed exactly as written. `cloud.py` was already partially written (untracked file) from a prior aborted attempt; it was inspected and verified complete, then committed. All three tasks executed as specified.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
`cloud.py` already existed as an untracked file from a prior failed execution attempt. Verified it contained all required endpoints, routes, and security invariants before committing. No rework needed.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None. All endpoints have real implementations. The `test_cloud.py` stubs remain xfail because they call `pytest.xfail()` unconditionally inside the test body — they are scaffolding tests that will be replaced by real integration tests in Plan 05-06 or later.
|
||||
|
||||
## Threat Surface Scan
|
||||
|
||||
New network-facing endpoints introduced (all on `router` and `users_router`):
|
||||
|
||||
- `GET /api/cloud/oauth/initiate/{provider}` — initiates OAuth redirect; all providers validated against allowlist
|
||||
- `GET /api/cloud/oauth/callback/{provider}` — receives OAuth code from provider; state validated before any DB write
|
||||
- `POST /api/cloud/connections/webdav` — user-supplied URL validated via `validate_cloud_url` (SSRF) before any outbound call
|
||||
- `GET /api/cloud/connections` — read-only, returns CloudConnectionOut (credentials_enc excluded)
|
||||
- `DELETE /api/cloud/connections/{connection_id}` — returns 404 for wrong-owner (enumeration prevention)
|
||||
- `GET /api/cloud/folders/{provider}/{folder_id}` — makes outbound call to cloud provider using decrypted credentials; TTL-cached
|
||||
- `PATCH /api/users/me/default-storage` — updates a single user column; no outbound calls
|
||||
|
||||
All STRIDE threats from the plan's threat model are mitigated as implemented:
|
||||
- T-05-05-01: `secrets.token_urlsafe(32)` state token in Redis — confirmed
|
||||
- T-05-05-02: Redis key deleted immediately on callback validation — confirmed
|
||||
- T-05-05-03: `CloudConnectionOut` imported from admin.py — same whitelist — confirmed
|
||||
- T-05-05-04: DELETE returns 404 for wrong-owner — confirmed
|
||||
- T-05-05-05: `validate_cloud_url` called before WebDAV backend instantiation — confirmed
|
||||
- T-05-05-06: `Depends(get_regular_user)` on all endpoints — confirmed
|
||||
- T-05-05-08: audit metadata = `{"provider": provider}` only — confirmed
|
||||
|
||||
No threat flags raised beyond those already documented in the plan.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- All cloud connection management endpoints are live and importable. Plans 05-06 through 05-08 can begin building on top of the API layer.
|
||||
- `_call_cloud_op` is available for use in upload/download handlers in Plan 05-06.
|
||||
- `_build_backend` helper in `cloud.py` reconstructs any backend from provider name + credentials dict.
|
||||
- 262 tests pass / 43 xfailed / 1 pre-existing failure (`test_extract_docx` — python-docx not installed).
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
Files verified present:
|
||||
- `backend/api/cloud.py`: FOUND
|
||||
- `backend/main.py`: FOUND (cloud_router registered)
|
||||
- `backend/api/admin.py`: FOUND (cloud cleanup present)
|
||||
|
||||
Commits verified:
|
||||
- 2424f52: feat(05-05): implement cloud.py — FOUND
|
||||
- f509c37: feat(05-05): register cloud and users routers in main.py — FOUND
|
||||
- d85a097: feat(05-05): add cloud credential cleanup on admin user deletion — FOUND
|
||||
|
||||
---
|
||||
*Phase: 05-cloud-storage-backends*
|
||||
*Completed: 2026-05-29*
|
||||
Reference in New Issue
Block a user