diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index b8d029d..963044c 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -25,7 +25,7 @@ _Last updated: 2026-05-21_ - [ ] **SEC-06**: Constant-time comparison used for all token and code verification (prevents timing attacks) - [ ] **SEC-07**: Admin role verified on every admin endpoint request; admin cannot access document content, extracted text, or cloud credentials in any response - [ ] **SEC-08**: Cloud credential ciphertext (`credentials_enc`) excluded from all API serializers by default — admin and user responses return only `provider, display_name, connected_at, status` -- [ ] **SEC-09**: Account deletion triggers `delete_user_files()` on every active cloud connection before removing DB records (prevents orphaned cloud data and satisfies GDPR Article 17) +- [x] **SEC-09**: Account deletion triggers `delete_user_files()` on every active cloud connection before removing DB records (prevents orphaned cloud data and satisfies GDPR Article 17) ### Users & Admin (ADMIN) diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 298a785..2c40209 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -231,12 +231,12 @@ Before any phase is marked complete, all three gates must pass: **Wave 3** — Cloud backends (parallel, both blocked on Wave 2 / Plan 05-02) -- [ ] 05-03-PLAN.md — GoogleDriveBackend + OneDriveBackend (all 7 StorageBackend methods) -- [ ] 05-04-PLAN.md — NextcloudBackend + WebDAVBackend (all 7 StorageBackend methods) +- [x] 05-03-PLAN.md — GoogleDriveBackend + OneDriveBackend (all 7 StorageBackend methods) +- [x] 05-04-PLAN.md — NextcloudBackend + WebDAVBackend (all 7 StorageBackend methods) **Wave 4** — Cloud API -- [ ] 05-05-PLAN.md — All /api/cloud/* endpoints + /api/users/me/default-storage + main.py router registration +- [x] 05-05-PLAN.md — All /api/cloud/* endpoints + /api/users/me/default-storage + main.py router registration **Wave 5** — Document routing + full test suite diff --git a/.planning/STATE.md b/.planning/STATE.md index fa44c7e..13ffe40 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,13 +4,13 @@ milestone: v1.0 milestone_name: milestone current_phase: 5 status: executing -last_updated: "2026-05-28T19:15:00.000Z" +last_updated: "2026-05-29T09:21:57.000Z" progress: total_phases: 5 completed_phases: 4 total_plans: 32 - completed_plans: 27 - percent: 84 + completed_plans: 28 + percent: 87 --- # Project State @@ -28,13 +28,13 @@ progress: | 2 | Users & Authentication | ✓ Complete (5/5 plans) | | 3 | Document Migration & Multi-User Isolation | ✓ Complete (5/5 plans, UAT passed, security gate passed) | | 4 | Folders, Sharing, Quotas & Document UX | ✓ Complete (9/9 plans, UAT 14/15 passed, 1 bug fixed) | -| 5 | Cloud Storage Backends | In Progress (4/8 plans complete) | +| 5 | Cloud Storage Backends | In Progress (5/8 plans complete) | ## Current Position **Phase:** 05-cloud-storage-backends — In Progress -**Plan:** 4/8 -**Progress:** [████████░░] 84% +**Plan:** 5/8 +**Progress:** [████████░░] 87% ## Performance Metrics @@ -131,6 +131,9 @@ progress: | WebDAVBackend SSRF double guard pattern | validate_cloud_url in __init__ (construct-time) AND before every asyncio.to_thread() call — mirrors D-17 requirement for DNS-rebinding mitigation | | nextcloud/webdav dispatch to distinct classes | NextcloudBackend for 'nextcloud' provider (has list_folder); WebDAVBackend for 'webdav' — identical constructor signatures | | webdavclient3 upload_to/download_from confirmed | A1 assumption in RESEARCH.md was correct; verified via runtime dir(Client) inspection before use | +| OAuth callback not authenticated via JWT | OAuth redirect flow cannot carry Bearer header; state token (256 bits, TTL 1800s, single-use) provides equivalent security | +| Cloud cleanup added to admin delete_user only | auth.py has no DELETE /api/users/me; admin-initiated deletion is the only account deletion code path | +| Cloud cleanup runs before MinIO cleanup | credentials still in DB when get_storage_backend_for_document is called; sessions.flush() after conn deletes | ### Open Questions @@ -178,6 +181,7 @@ _Updated at each phase transition._ | Last session | 2026-05-28 — Plan 05-02 executed: cloud_utils.py (SSRF+HKDF), cloud_cache.py (TTLCache), storage factory extended; 199 passed / 43 xfailed / 1 pre-existing failure | | Last session | 2026-05-28 — Plan 05-03 executed: GoogleDriveBackend (Drive v3, cache_discovery=False, asyncio.to_thread) + OneDriveBackend (MSAL, resumable upload, CHUNK_SIZE=10MB); 262 passed / 43 xfailed / 1 pre-existing failure | | Last session | 2026-05-28 — Plan 05-04 executed: WebDAVBackend + NextcloudBackend (SSRF double-guard, asyncio.to_thread, list_folder); 262 passed / 43 xfailed / 1 pre-existing failure | -| Next action | Execute Plan 05-05: Cloud API Endpoints | +| Last session | 2026-05-29 — Plan 05-05 executed: cloud.py (7 endpoints), main.py (routers registered), admin.py (SEC-09 cloud cleanup); 262 passed / 43 xfailed / 1 pre-existing failure | +| Next action | Execute Plan 05-06: Cloud Document Upload/Download | | Pending decisions | None | -| Resume file | `.planning/phases/05-cloud-storage-backends/05-05-PLAN.md` | +| Resume file | `.planning/phases/05-cloud-storage-backends/05-06-PLAN.md` | diff --git a/.planning/phases/05-cloud-storage-backends/05-05-SUMMARY.md b/.planning/phases/05-cloud-storage-backends/05-05-SUMMARY.md new file mode 100644 index 0000000..4a40490 --- /dev/null +++ b/.planning/phases/05-cloud-storage-backends/05-05-SUMMARY.md @@ -0,0 +1,163 @@ +--- +phase: 05-cloud-storage-backends +plan: 05 +subsystem: api +tags: [oauth, google-drive, onedrive, webdav, nextcloud, redis, cloud-storage, audit-log, sec-09] + +# Dependency graph +requires: + - phase: 05-cloud-storage-backends + plan: 03 + provides: "GoogleDriveBackend, OneDriveBackend, CloudConnectionError — used in cloud.py endpoints" + - phase: 05-cloud-storage-backends + plan: 04 + provides: "WebDAVBackend, NextcloudBackend, validate_cloud_url — WebDAV connect endpoint" + - phase: 05-cloud-storage-backends + plan: 02 + provides: "encrypt_credentials, decrypt_credentials, get_storage_backend_for_document, get_cloud_folders_cached" + - phase: 05-cloud-storage-backends + plan: 01 + provides: "CloudConnection model, settings (cloud_creds_key, google_*, onedrive_*, backend_url, frontend_url)" +provides: + - "backend/api/cloud.py: 6 cloud endpoints on /api/cloud/* + 1 endpoint on /api/users/me/default-storage" + - "backend/main.py: cloud_router and cloud_users_router registered" + - "backend/api/admin.py: cloud credential cleanup on admin user deletion (SEC-09)" +affects: [05-06, 05-07, 05-08] + +# Tech tracking +tech-stack: + added: [] + patterns: + - "_call_cloud_op helper: transparent token refresh (on token_expired) + REQUIRES_REAUTH (on invalid_grant) — wraps all cloud ops" + - "OAuth state token: secrets.token_urlsafe(32) stored in Redis as 'oauth_state:{token}' with TTL 1800, single-use delete" + - "CloudConnectionOut imported from api.admin — never redefined; same whitelist enforces credentials_enc exclusion everywhere" + - "OAuth callback not authenticated via JWT — state token binds callback to user session" + - "cloud.credentials_purged audit event on account deletion with providers list metadata" + +key-files: + created: + - backend/api/cloud.py + modified: + - backend/main.py + - backend/api/admin.py + +key-decisions: + - "OAuth callback handler does not require JWT auth — the state token (Redis-bound, TTL 1800, single-use) is the auth mechanism that ties the callback to the initiating user (T-05-05-01, T-05-05-02)" + - "DELETE /connections/{connection_id} returns 404 for wrong-owner connections — prevents connection ID enumeration (D-19, T-05-05-04)" + - "Cloud cleanup added to admin delete_user (not self-deletion) — auth.py has no DELETE /api/users/me; admin-initiated deletion is the account deletion code path in the codebase" + - "_call_cloud_op retries op_fn once on token_expired; builds a new backend instance from refreshed credentials after updating DB" + - "Cloud object deletion for cloud-stored documents runs BEFORE MinIO cleanup in delete_user: credentials still in DB when get_storage_backend_for_document is called" + +patterns-established: + - "_call_cloud_op pattern: zero-argument async lambda captures backend in closure; helper retries after token refresh" + - "invalidate_provider_cache() called on disconnect to invalidate TTLCache entries for that user+provider" + - "Folder listing: google_drive uses Drive v3 files.list with cache_discovery=False; onedrive uses httpx against Graph API; nextcloud/webdav use NextcloudBackend.list_folder()" + +requirements-completed: + - CLOUD-01 + - CLOUD-02 + - CLOUD-03 + - CLOUD-04 + - CLOUD-05 + - CLOUD-06 + - SEC-09 + +# Metrics +duration: 12min +completed: 2026-05-29 +--- + +# Phase 5 Plan 05: Cloud Connection Management API Summary + +**OAuth initiate/callback for Google Drive and OneDrive, WebDAV/Nextcloud credential connect, connection list/delete, TTL-cached folder listing, default-storage PATCH, and cloud credential purge on account deletion — all 7 endpoints using get_regular_user dep with credentials_enc excluded from all responses** + +## Performance + +- **Duration:** 12 min +- **Started:** 2026-05-29T09:09:57Z +- **Completed:** 2026-05-29T09:21:57Z +- **Tasks:** 3 +- **Files modified:** 3 + +## Accomplishments + +- Created `backend/api/cloud.py` with all 7 endpoints: OAuth initiate and callback (Google Drive + OneDrive), WebDAV/Nextcloud credential connect, connection list, connection delete, folder listing (TTL-cached), and default-storage update. All handlers use `Depends(get_regular_user)` — admin gets 403. `CloudConnectionOut` imported from `api.admin` so the credentials_enc exclusion whitelist is never duplicated. +- Implemented `_call_cloud_op` helper for transparent token refresh: retries op_fn once on `token_expired` (decrypt → refresh via provider → encrypt → update DB → rebuild backend → retry), sets `status="REQUIRES_REAUTH"` and writes audit log on `invalid_grant`. +- Registered both routers in `main.py` (Phase 5 section after audit router). All 6 cloud routes + `/api/users/me/default-storage` visible in `app.routes`. +- Added cloud credential cleanup to `admin.py delete_user` (SEC-09): queries all `CloudConnection` rows, deletes cloud-stored documents via `get_storage_backend_for_document + delete_object` (best-effort), explicitly deletes `CloudConnection` rows with `session.flush()`, writes `cloud.credentials_purged` audit event with providers list. + +## Task Commits + +1. **Task 1: Create cloud.py** - `2424f52` (feat) +2. **Task 2: Register cloud routers in main.py** - `f509c37` (feat) +3. **Task 3: Cloud cleanup on admin user deletion** - `d85a097` (feat) + +## Files Created/Modified + +- `/Users/nik/Documents/Progamming/document_scanner/backend/api/cloud.py` — All 7 cloud connection management endpoints + `_call_cloud_op` helper + `_upsert_cloud_connection` helper +- `/Users/nik/Documents/Progamming/document_scanner/backend/main.py` — `cloud_router` and `cloud_users_router` registered (Phase 5 section) +- `/Users/nik/Documents/Progamming/document_scanner/backend/api/admin.py` — `CloudConnection` + `get_storage_backend_for_document` imported; cloud cleanup block added to `delete_user` before MinIO cleanup + +## Decisions Made + +- OAuth callback endpoint does not require a JWT Bearer token. The OAuth redirect flow happens in the browser: the provider redirects back with `code` and `state`, and the backend validates the `state` against Redis to identify the user. Adding a JWT dep would break the flow (browser doesn't carry the Bearer header in redirect responses). The state token provides equivalent security: 256 bits of entropy, TTL 1800s, single-use deletion. +- Cloud cleanup was added to `admin.py delete_user` (not `auth.py`). The `auth.py` module does not implement account self-deletion — there is no `DELETE /api/users/me` endpoint. The admin-initiated deletion (`DELETE /api/admin/users/{id}`) is the only account deletion code path. +- Cloud object deletion runs before MinIO document deletion in `delete_user` so that `get_storage_backend_for_document` can decrypt credentials and build the correct backend. After `session.delete(conn)` + `session.flush()`, the credentials would be gone. + +## Deviations from Plan + +None — plan executed exactly as written. `cloud.py` was already partially written (untracked file) from a prior aborted attempt; it was inspected and verified complete, then committed. All three tasks executed as specified. + +## Issues Encountered + +`cloud.py` already existed as an untracked file from a prior failed execution attempt. Verified it contained all required endpoints, routes, and security invariants before committing. No rework needed. + +## Known Stubs + +None. All endpoints have real implementations. The `test_cloud.py` stubs remain xfail because they call `pytest.xfail()` unconditionally inside the test body — they are scaffolding tests that will be replaced by real integration tests in Plan 05-06 or later. + +## Threat Surface Scan + +New network-facing endpoints introduced (all on `router` and `users_router`): + +- `GET /api/cloud/oauth/initiate/{provider}` — initiates OAuth redirect; all providers validated against allowlist +- `GET /api/cloud/oauth/callback/{provider}` — receives OAuth code from provider; state validated before any DB write +- `POST /api/cloud/connections/webdav` — user-supplied URL validated via `validate_cloud_url` (SSRF) before any outbound call +- `GET /api/cloud/connections` — read-only, returns CloudConnectionOut (credentials_enc excluded) +- `DELETE /api/cloud/connections/{connection_id}` — returns 404 for wrong-owner (enumeration prevention) +- `GET /api/cloud/folders/{provider}/{folder_id}` — makes outbound call to cloud provider using decrypted credentials; TTL-cached +- `PATCH /api/users/me/default-storage` — updates a single user column; no outbound calls + +All STRIDE threats from the plan's threat model are mitigated as implemented: +- T-05-05-01: `secrets.token_urlsafe(32)` state token in Redis — confirmed +- T-05-05-02: Redis key deleted immediately on callback validation — confirmed +- T-05-05-03: `CloudConnectionOut` imported from admin.py — same whitelist — confirmed +- T-05-05-04: DELETE returns 404 for wrong-owner — confirmed +- T-05-05-05: `validate_cloud_url` called before WebDAV backend instantiation — confirmed +- T-05-05-06: `Depends(get_regular_user)` on all endpoints — confirmed +- T-05-05-08: audit metadata = `{"provider": provider}` only — confirmed + +No threat flags raised beyond those already documented in the plan. + +## Next Phase Readiness + +- All cloud connection management endpoints are live and importable. Plans 05-06 through 05-08 can begin building on top of the API layer. +- `_call_cloud_op` is available for use in upload/download handlers in Plan 05-06. +- `_build_backend` helper in `cloud.py` reconstructs any backend from provider name + credentials dict. +- 262 tests pass / 43 xfailed / 1 pre-existing failure (`test_extract_docx` — python-docx not installed). + +## Self-Check: PASSED + +Files verified present: +- `backend/api/cloud.py`: FOUND +- `backend/main.py`: FOUND (cloud_router registered) +- `backend/api/admin.py`: FOUND (cloud cleanup present) + +Commits verified: +- 2424f52: feat(05-05): implement cloud.py — FOUND +- f509c37: feat(05-05): register cloud and users routers in main.py — FOUND +- d85a097: feat(05-05): add cloud credential cleanup on admin user deletion — FOUND + +--- +*Phase: 05-cloud-storage-backends* +*Completed: 2026-05-29*