Commit Graph
21 Commits
Author SHA1 Message Date
curo1305 dfc335022f feat(14-06): route preview/download bytes through durable byte cache
- Add hydrate_and_cache_bytes to services/cloud_cache.py (Plan 06 cache lifecycle)
  - Cache-hit path: pin existing entry, read from MinIO, release pin in finally
  - Cache-miss path: fetch from provider, store in MinIO, create/reactivate entry, pin during response, release pin in finally
  - CacheQuotaExceeded and MinIO failures are non-fatal (bytes returned from provider path)
  - evict_lru_entries run after pin release to stay within user cache limit
- Update preview_cloud_file and download_cloud_file in operations.py to call hydrate_and_cache_bytes
  - Version key computed from CloudItem metadata (no provider call required for key resolution)
  - Shared _make_minio_cache_wrapper helper eliminates duplicate adapter code
  - Falls through to direct provider fetch when no CloudItem row found
- Add _make_minio_cache_wrapper helper to operations.py (DRY, shared by preview and download)
- Add 5 new tests to test_cloud_cache.py: cache-miss fetches provider, cache-hit skips provider, pin released on hit, pin released on miss, user isolation
- Add 3 new security tests to test_cloud_security.py: preview response excludes object_key, download response excludes object_key, foreign-user IDOR via download blocked
2026-06-23 16:28:06 +02:00
curo1305 ba48d625dd feat(14-04): add analysis API routes and typed schemas
- POST /analysis/connections/{id}/estimate: scope estimate without bytes (T-14-04)
- POST /analysis/connections/{id}/jobs: enqueue job, returns job_id (ANALYZE-01)
- GET /analysis/jobs, GET /analysis/jobs/{id}: job status list and detail
- POST /analysis/jobs/{id}/cancel: batch cancel (queued items immediate; running cooperative)
- POST /analysis/jobs/{id}/items/{item_id}/cancel|skip|retry: per-item controls (ANALYZE-05)
- AnalysisEstimateRequest/Out, AnalysisEnqueueRequest/Out, AnalysisJobOut: typed schemas
- AnalysisJobOut: both simple (waiting/working/done/skipped/failed) and per-stage counts
- All routes use get_regular_user — admin blocked (T-14-01)
- Response schemas exclude credentials_enc, object_key, raw provider URLs (T-14-02)
- 14/14 test_cloud_analysis_api.py pass; 7 new analysis security tests pass (33/33 total)
2026-06-23 15:50:37 +02:00
curo1305 299d4b523d feat(14-03): cache settings/status API and security tests
- api/cloud/cache.py: GET /analysis/cache, PATCH /analysis/cache/settings
- api/cloud/analysis.py: analysis router aggregator (Phase 14 parent)
- schemas.py: CacheStatusOut, CacheSettingsUpdateRequest (explicit allowlists)
- __init__.py: register analysis_router under /api/cloud
- 7 new security tests: admin block, unauthenticated block, object_key exclusion,
  response structure, enum validation, tier cap enforcement (T-14-01..02, T-14-08)
- All 47 plan target tests pass (test_cloud_cache.py + test_cloud_security.py)
2026-06-23 15:32:12 +02:00
curo1305andClaude Sonnet 4.6 ed046752e4 test(13): add Nyquist validation — promote 3 xfail audit tests to green
Implemented write_audit_log calls for open_cloud_file, create_cloud_folder,
and rename_cloud_item in api/cloud/operations.py. Removed xfail markers from
test_cloud_audit.py. All 11 audit tests pass; 360 cloud tests pass total.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 08:17:24 +02:00
curo1305 60df8552b5 fix(13): CR-05 commit audit log after write_audit_log in reconnect endpoint so event is not silently lost 2026-06-23 00:29:34 +02:00
curo1305 af0de30e93 fix(13): CR-04 soft-delete CloudItem row on successful cloud delete to prevent orphaned metadata 2026-06-23 00:29:11 +02:00
curo1305 405c7a6610 fix(13): CR-03 wrap post-upload DB block in try/except to handle provider-success/DB-fail divergence 2026-06-23 00:28:34 +02:00
curo1305 d4b2697109 fix(13): CR-01 use RFC 6266 filename*=UTF-8'' encoding in Content-Disposition to prevent header injection 2026-06-23 00:26:59 +02:00
curo1305 ebc74f4abe fix(13): CR-06 re-raise HTTPException in preview_cloud_file instead of masking as unsupported_preview 2026-06-23 00:26:23 +02:00
curo1305 508d27643b feat(13-09): move stale-guard, descendant safety, reconciliation, delete disclosure, and metadata-only auditing
Move endpoint (D-07, D-08, D-09, T-13-27, T-13-28, T-13-29):
- Descendant destination detection via cloud_items ancestor-chain walk (D-09)
- Stale provider result marks source folder non-fresh, returns typed stale body (D-07)
- Success: upsert_cloud_item with new parent_ref before returning
- Success: update_folder_state for both source and destination folders
- Success: write_audit_log 'cloud.item_moved' with metadata-only payload

Delete endpoint (D-10, D-11, T-13-29):
- Pre-fetch item kind for D-10 folder-vs-file disclosure
- Success: update_folder_state for parent folder before returning
- Success: write_audit_log 'cloud.item_deleted' with delete_kind metadata
- Response carries is_folder/item_kind for frontend disclosure
- Failed deletes never emit audit rows

Promote xfail markers in test_cloud_audit.py for move and delete audit rows
2026-06-22 20:02:09 +02:00
curo1305 aaa63c19e4 feat(13-08): implement create-folder and rename collision retry, stale guard, and reconciliation
- create_cloud_folder: bounded collision retry (up to 5 attempts) with keep_both_name counter suffix (D-05, D-06)
- create_cloud_folder: stale precondition marks parent folder non-fresh and returns typed stale result (D-07)
- rename_cloud_item: stale precondition marks parent folder non-fresh via folder state update (D-07)
- create_cloud_folder: reconcile-before-return upserts new folder in cloud_items and invalidates parent (T-13-26)
- rename_cloud_item: reconcile-before-return upserts renamed item and invalidates parent folder (T-13-26)
- Import upsert_cloud_item, update_folder_state, keep_both_name, CloudResource from service modules
2026-06-22 19:51:57 +02:00
curo1305 d959e0cf42 feat(13-06): emit metadata-only audit rows on authoritative upload success
- Write cloud.file_uploaded audit row in same transaction as reconciliation
- Audit payload is metadata-only: filename, size_bytes, connection_id, provider_item_id, parent_ref
- No provider URLs, tokens, bytes, or document text in audit metadata (T-13-02)
- Non-success paths (conflict, offline, reauth_required) bypass audit write (T-13-21)
- Import get_client_ip from deps.utils and write_audit_log from services.audit
2026-06-22 19:29:46 +02:00
curo1305 7ecbec7df9 feat(13-06): route upload success through centralized reconciliation before returning
- Upsert uploaded item into cloud_items for stable row identity and navigation
- Invalidate parent folder state (warning/upload_mutated) so next browse re-fetches
- Reconcile-before-return: session committed before response returned
- Failed/conflict uploads bypass reconciliation path (T-13-19)
2026-06-22 19:26:41 +02:00
curo1305 94a0617b9b feat(13-04): add authorized cloud content and mutation routes with typed bodies
- backend/api/cloud/operations.py: owner-scoped open, preview, download, create-folder,
  rename, move, delete, upload routes with typed kind/reason response bodies (D-02, D-03,
  D-05, D-07, D-08, D-09, D-10, D-11, D-18, T-13-01, T-13-14)
- backend/api/cloud/__init__.py: register operations_router on /api/cloud
- backend/api/cloud/connections.py: Google Drive OAuth scope broadened to 'drive' (D-17)
- backend/api/cloud/schemas.py: ConnectionHealthOut, ReconnectOut, ContentResultOut,
  MutationResultOut, CreateFolderRequest, RenameItemRequest, MoveItemRequest typed schemas
- frontend/src/api/cloud.js: centralized helpers for all Phase 13 routes
- backend/tests/test_cloud_mutations.py: mock adapter + settings key fixture; 21 tests pass
- backend/tests/test_cloud_audit.py: mark 6 RED audit tests as xfail (audit writes are
  T-13-05 scope for a later plan); update credential fixture to use settings key
2026-06-22 19:09:38 +02:00
curo1305 5c0bc2a3b4 feat(13-04): upgrade Google Drive scope to drive, add health/reconnect/content API helpers
- connections.py: Update Google Drive OAuth to broader `drive` scope (D-17)
  instead of `drive.file` so authorized users can operate on all existing Drive
  items, not just files created by this app. Both initiation and callback flows
  updated to keep scopes consistent.
- schemas.py: Add Phase 13 typed response schemas — ConnectionHealthOut, ReconnectOut,
  ContentResultOut, MutationResultOut, and request schemas CreateFolderRequest,
  RenameItemRequest, MoveItemRequest. All credential-free (T-13-14, CONN-03).
- api/cloud.js: Add centralized frontend helpers for getConnectionHealth,
  reconnectCloudConnection, testCloudConnection (D-12, D-13), plus authorized
  cloud content and mutation helpers: openCloudFile, previewCloudFile,
  createCloudFolder, renameCloudItem, moveCloudItem, deleteCloudItem, uploadCloudFile.

All reconnect/health/test tests pass (14/14). Security suite passes (19/19).
2026-06-22 18:49:47 +02:00
curo1305 6784d3bdb7 feat(13-03): add cloud operations service and reconnect/health/test routes
- Create backend/services/cloud_operations.py as the single Phase 13 orchestration seam:
  get_connection_health (D-12), test_connection (D-13), reconnect_connection (CONN-01/02/03,
  D-14), disconnect_connection (D-16, explicit CloudItem cascade for SQLite compatibility)
- Add POST /connections/{id}/reconnect route (CONN-01/02/03, D-14)
- Add GET /connections/{id}/health route (D-12, T-13-02)
- Add POST /connections/{id}/test route (D-13, T-13-02)
- Update DELETE /connections/{id} to use service-layer disconnect with explicit
  CloudItem deletion (D-16; covers FK-cascade-less environments like SQLite tests)
- Fix (Rule 2 - T-13-02): add _scrub_audit_metadata() to admin audit log API to
  remove credential fields before returning audit rows to admin callers
- All 14 reconnect tests pass + 110 provider-contract tests pass
- Pre-existing failures in test_cloud_mutations and test_extractor are unrelated
2026-06-22 18:43:50 +02:00
curo1305 bfa4dc502a feat(12.1-02): centralize listing freshness gate — apply_listing_and_finalize
- Add ListingResult dataclass to cloud_items.py
- Add apply_listing_and_finalize() — single shared gate for complete/incomplete semantics
- complete=True: upserts, soft-deletes unseen children, marks fresh, advances last_refreshed_at
- complete=False: upserts seen items only, never deletes unseen, preserves prior timestamp, warns
- browse.py: replace unconditional update_folder_state('fresh') with apply_listing_and_finalize
- cloud_tasks.py: replace unconditional fresh transition with apply_listing_and_finalize
- Celery task returns structured warning status on incomplete (no credentials/provider item names)
- Fix timezone naive/aware comparison in test (SQLite strips tzinfo)
2026-06-22 08:29:41 +02:00
curo1305 731b65ecdd feat(12-06): connection-ID native lifecycle — always INSERT, PUT credentials, connectionsFor multi-account
- Replace _upsert_cloud_connection with _insert_cloud_connection (always inserts new UUID row)
- Add PUT /connections/{id}/credentials endpoint (owner-scoped, SSRF + health-check, password-preserve)
- SettingsCloudTab: connectionsFor() renders all same-provider connections with Add account row
- CloudCredentialModal.submit: calls updateWebDavCredentials on edit, connectWebDav on create
- utils.js: FastAPI validation arrays normalised to concise field messages (Rule 2)
- Backend tests: same-provider independence + IDOR negative test for credential update
2026-06-21 22:29:59 +02:00
curo1305andClaude Sonnet 4.6 b6911fb4ed fix(12): resolve critical code review findings (CR-01 through CR-04)
- CR-01: fix browse URL from /folders/{id} to /items?parent_ref= (broken feature)
- CR-02: remove raw provider exception text from _TerminalProviderError messages
- CR-03: add user_id predicate to get_or_create_folder_state query (ownership boundary)
- CR-04: add max_length=255 to ConnectionRenameRequest.display_name

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-19 02:00:05 +02:00
curo1305 c6237ca57f feat(12-02): add refresh_cloud_folder Celery task, staleness trigger, version 0.1.5
- Add tasks/cloud_tasks.py: durable refresh_cloud_folder task with 3-retry
  bounded backoff (30s/90s/270s +jitter), credential decryption in worker only
- Register tasks.cloud_tasks.* on documents queue in celery_app.py
- Add stale-while-revalidate staleness trigger in browse.py (5-min threshold)
- Add 4 Task 3 tests: idempotency, cached-row retention on failure, task structure,
  no-byte-download contract; add background-refresh scheduling integration test
- Bump backend version 0.1.4 → 0.1.5, frontend package.json 0.1.4 → 0.1.5
- Update AGENTS.md with Phase 12 Plan 02 state and new shared module map entries
- Update README with connection-ID browse API table and v0.1.5
2026-06-18 23:17:34 +02:00
curo1305 e186019066 feat(12-02): decompose api/cloud.py into package with connection-ID browse endpoint
- Add api/cloud/ package: connections.py, browse.py, schemas.py, __init__.py
- Add GET /api/cloud/connections/{connection_id}/items (T-12-01 IDOR, T-12-03 cred-free)
- Add PATCH /api/cloud/connections/{id} for display_name_override rename
- Add display_name_override ORM field to CloudConnection model
- Add CloudResourceAdapter service layer with str/UUID coercion
- Fix UUID type compatibility: test_cloud_items.py now uses UUID(as_uuid=True)
  matching conftest — removes String(36) patch that caused type incompatibility
- Add 11 Phase 12 integration tests (IDOR, admin block, credential exclusion,
  duplicate providers, rename, malformed UUID)
- Remove deleted api/cloud.py (replaced by api/cloud/ package)
2026-06-18 23:10:30 +02:00