Commit Graph
809 Commits
Author SHA1 Message Date
curo1305 a2b7928ba4 docs(14-06): complete byte cache for open/preview/download plan 2026-06-23 16:33:01 +02:00
curo1305 b84b912acf test(14-06): add cache transparency assertion to frontend open/preview tests
- Add cache_backed_response_has_no_object_key describe block to CloudFolderOpenPreview.test.js
- Verifies openCloudFile response processed by CloudFolderView never exposes object_key, MinIO paths, or credentials_enc in rendered HTML or API call arguments (T-14-02)
- All 9 tests pass (8 pre-existing + 1 new)
2026-06-23 16:29:11 +02:00
curo1305 dfc335022f feat(14-06): route preview/download bytes through durable byte cache
- Add hydrate_and_cache_bytes to services/cloud_cache.py (Plan 06 cache lifecycle)
  - Cache-hit path: pin existing entry, read from MinIO, release pin in finally
  - Cache-miss path: fetch from provider, store in MinIO, create/reactivate entry, pin during response, release pin in finally
  - CacheQuotaExceeded and MinIO failures are non-fatal (bytes returned from provider path)
  - evict_lru_entries run after pin release to stay within user cache limit
- Update preview_cloud_file and download_cloud_file in operations.py to call hydrate_and_cache_bytes
  - Version key computed from CloudItem metadata (no provider call required for key resolution)
  - Shared _make_minio_cache_wrapper helper eliminates duplicate adapter code
  - Falls through to direct provider fetch when no CloudItem row found
- Add _make_minio_cache_wrapper helper to operations.py (DRY, shared by preview and download)
- Add 5 new tests to test_cloud_cache.py: cache-miss fetches provider, cache-hit skips provider, pin released on hit, pin released on miss, user isolation
- Add 3 new security tests to test_cloud_security.py: preview response excludes object_key, download response excludes object_key, foreign-user IDOR via download blocked
2026-06-23 16:28:06 +02:00
curo1305 5a9f3a628e docs(14-05): complete cloud analysis processing plan 2026-06-23 16:10:42 +02:00
curo1305 f235d894fb feat(14-05): add Celery cloud analysis task with retry harness (Task 2)
- Implement tasks/cloud_analysis_tasks.py: sync Celery bridge to async processing
- Broker payload contains IDs only — credentials decrypted in worker (T-14-10)
- _TransientError and _ClassificationRetry sentinels for bounded retry (max=3)
- Auth errors are terminal (not retried); MaxRetriesExceeded writes failure status
- celery_app.py: add cloud_analysis_tasks route and import registration
- Add 6 contract tests: task registration, broker payload shape, queue routing,
  bounded retries, sentinel pattern, connection-not-found graceful drop
2026-06-23 16:06:07 +02:00
curo1305 cea9980aaf feat(14-05): add cloud analysis processing service (Task 1)
- Implement services/cloud_analysis_processing.py with full pipeline:
  queued -> downloading -> extracting -> classifying -> indexed
- Stale guard: version key comparison fires before provider byte fetch (T-14-13)
- Cache pin lifecycle: pin acquired before bytes used, released in finally block (T-14-12)
- No provider mutation methods called: adapter only used via get_object (T-14-03)
- No local Document row created: classification targets CloudItem directly
- Cooperative cancellation check at job/item level before each stage
- ProcessingResult dataclass with status, cache_entry_id, error_code fields
- Add 4 contract tests: status_transitions, stale_detection, pin_release, no_mutations
2026-06-23 16:02:44 +02:00
curo1305 1ae9930ca7 docs(14-04): complete analysis orchestration and API plan 2026-06-23 15:54:49 +02:00
curo1305 ba48d625dd feat(14-04): add analysis API routes and typed schemas
- POST /analysis/connections/{id}/estimate: scope estimate without bytes (T-14-04)
- POST /analysis/connections/{id}/jobs: enqueue job, returns job_id (ANALYZE-01)
- GET /analysis/jobs, GET /analysis/jobs/{id}: job status list and detail
- POST /analysis/jobs/{id}/cancel: batch cancel (queued items immediate; running cooperative)
- POST /analysis/jobs/{id}/items/{item_id}/cancel|skip|retry: per-item controls (ANALYZE-05)
- AnalysisEstimateRequest/Out, AnalysisEnqueueRequest/Out, AnalysisJobOut: typed schemas
- AnalysisJobOut: both simple (waiting/working/done/skipped/failed) and per-stage counts
- All routes use get_regular_user — admin blocked (T-14-01)
- Response schemas exclude credentials_enc, object_key, raw provider URLs (T-14-02)
- 14/14 test_cloud_analysis_api.py pass; 7 new analysis security tests pass (33/33 total)
2026-06-23 15:50:37 +02:00
curo1305 3f26cd2059 feat(14-04): add cloud_analysis orchestration service
- estimate_scope: file/selection/folder/connection scope from cached metadata only
- enqueue_analysis_job: create CloudAnalysisJob + items; already_current skip without bytes
- already_current detection: indexed analysis_status + live metadata check (metadata-only, T-14-04)
- cancel_job, cancel_job_item, skip_job_item, retry_job_item: full control operations
- check_scan_quota: seam for future tier enforcement (always True in Phase 14)
- _is_supported: content type + extension list; explicit unsupported-override for .exe etc
- get_adapter / resolve_provider_metadata: stubs for Plan 05 provider integration
- No provider mutation methods called (T-14-03); no bytes downloaded (T-14-04)
- 16/16 test_cloud_analysis_contract.py tests pass
2026-06-23 15:50:24 +02:00
curo1305 4e02b04ba8 docs(14-03): complete cache service and API plan 2026-06-23 15:36:24 +02:00
curo1305 299d4b523d feat(14-03): cache settings/status API and security tests
- api/cloud/cache.py: GET /analysis/cache, PATCH /analysis/cache/settings
- api/cloud/analysis.py: analysis router aggregator (Phase 14 parent)
- schemas.py: CacheStatusOut, CacheSettingsUpdateRequest (explicit allowlists)
- __init__.py: register analysis_router under /api/cloud
- 7 new security tests: admin block, unauthenticated block, object_key exclusion,
  response structure, enum validation, tier cap enforcement (T-14-01..02, T-14-08)
- All 47 plan target tests pass (test_cloud_cache.py + test_cloud_security.py)
2026-06-23 15:32:12 +02:00
curo1305 5bdd23f3bc feat(14-03): add cache orchestration functions and tests
- retain_or_reuse_cache_entry: reuse non-evicted entry or reactivate evicted row
- pin_cache_entry / release_cache_entry: pin_count lifecycle management
- update_cache_access: touch last_accessed_at for LRU ordering
- Ownership assertions in all new service functions (T-14-06)
- 9 new integration tests covering retain/reuse, pin/release, and isolation
2026-06-23 15:24:13 +02:00
curo1305 54cc78eb96 docs(14-02): complete Phase 14 Plan 02 — schema and service helpers
- 14-02-SUMMARY.md: documents migration 0007, ORM models, version-key
  precedence helper, settings service, and Phase 14 cache functions
- STATE.md: advance to Plan 3 of 9, record P02 metrics and decisions
2026-06-23 15:17:05 +02:00
curo1305 1df61040c6 feat(14-02): add version-key, settings helpers and Phase 14 cache service functions
- cloud_analysis_versioning.py: compute_version_key with version>etag>fingerprint
  precedence (D-19); compute_fingerprint helper; content_hash is optional post-
  hydration only (D-20); raises ValueError only, no HTTPException
- cloud_analysis_settings.py: get_or_create_user_analysis_settings (upsert on
  first access), update_user_analysis_settings with enum validation and tier-capped
  cache limit, build_default_settings for API responses; raises ValueError only
- cloud_cache.py: Phase 14 durable cache functions added alongside existing Phase 12
  in-memory TTL cache — create_cache_entry, list_cache_entries (owner-scoped),
  evict_lru_entries (LRU, skips pin_count>0 and active_job_count>0), increment/
  decrement_quota_for_cache (atomic UPDATE pattern); re-exports compute_version_key
  from cloud_analysis_versioning so tests have one import site
2026-06-23 15:14:19 +02:00
curo1305 fd76d06b3a feat(14-02): add migration 0007 and ORM models for cloud analysis cache schema
- 0007_cloud_analysis_cache.py: new Alembic migration adding 4 Phase 14 tables
  - cloud_byte_cache_entries: durable MinIO cache with version_key uniqueness,
    pin_count/active_job_count guards, LRU eviction indexes
  - cloud_analysis_jobs: batch job rows (scope_kind, status, aggregate counters,
    provider_bytes_estimate, failure_behavior)
  - cloud_analysis_job_items: per-item idempotency rows (status vocabulary,
    controlled error fields, cache_entry_id, retry_count, version_key fingerprint)
  - user_analysis_settings: per-user preferences (progress_detail, failure_behavior,
    cloud_cache_limit_bytes)
  - Reversible downgrade drops Phase 14 objects only
- db/models.py: CloudByteCacheEntry, CloudAnalysisJob, CloudAnalysisJobItem,
  UserAnalysisSettings ORM models mirroring migration
2026-06-23 15:06:24 +02:00
curo1305 493d348ba6 docs(14-01): complete Phase 14 Plan 01 red test contract 2026-06-23 15:00:57 +02:00
curo1305andClaude Sonnet 4.6 7f2f570582 test(14-01): add RED frontend tests for analysis queue UX and store actions
- StorageBrowser.analysis-queue.test.js: row-level Analyze action (D-01),
  toolbar Analyze Selection/Folder/Connection (D-01..03), estimate modal
  with D-03 required fields (supported_count/bytes/unsupported, Start action),
  aggregate progress with simplified/detailed labels (D-05..08), expandable
  queue item list, per-item cancel/retry/skip controls (D-09..10),
  batch Cancel all (D-09), architecture no-parallel-grid guard (CLAUDE.md)
- cloudConnections.analysis.test.js: analysis state existence, translateAnalysisStatus
  single-source (D-06/D-12), API call assertions without credentials (T-14-02),
  cancel/retry/skip action coverage (ANALYZE-05), failure_behavior default=pause_batch
  (D-11), detailedAnalysisProgress preference (D-06), cache limit exposure
  (CACHE-04/D-15), credential-free localStorage guard

Requirements: ANALYZE-01..05, CACHE-04..05
All tests fail against missing Phase 14 implementation — no import errors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 14:56:39 +02:00
curo1305andClaude Sonnet 4.6 f67559c7f0 test(14-01): add RED contract tests for cloud analysis and byte cache (backend)
- test_cloud_analysis_contract.py: estimate (single/selection/connection/folder),
  already-current skip without byte fetch, changed-etag requeue, no-mutation
  invariant, IDOR/admin-negative access (T-14-01..04), cancel/retry transitions
- test_cloud_cache.py: CloudByteCacheEntry schema, LRU eviction with pin
  protection, quota increment/decrement seam (T-14-05), owner isolation
  (CACHE-05), object-key secrecy (T-14-02), version-key precedence helper
- test_cloud_analysis_api.py: route registration gates, job status shape,
  simplified/detailed progress labels (D-06), skip/cancel controls, scan
  quota seam (D-13), failure_behavior field acceptance, unauthenticated block

Requirements: ANALYZE-01..07, CACHE-03..05
All tests fail against missing Phase 14 modules — no syntax/import errors in tests themselves

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 14:56:29 +02:00
curo1305 b528b51881 docs(phase-14): plan selective analysis and byte cache 2026-06-23 14:24:43 +02:00
curo1305 3514fdcf3a docs(workflow): require gsd worktrees before edits 2026-06-23 14:10:41 +02:00
curo1305 fac7721dd9 docs(state): record phase 14 context session 2026-06-23 14:04:48 +02:00
curo1305 d3700e20c4 docs(14): capture phase context 2026-06-23 13:59:53 +02:00
curo1305andClaude Sonnet 4.6 2f00e122d3 docs(phase-13): update VALIDATION.md — all tasks green, wave-0 complete
Nyquist audit 2026-06-23: 3 gaps found and resolved (open/create-folder/rename
audit writes). All 10 tasks marked green, wave_0_complete: true, status: complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 08:17:29 +02:00
curo1305andClaude Sonnet 4.6 ed046752e4 test(13): add Nyquist validation — promote 3 xfail audit tests to green
Implemented write_audit_log calls for open_cloud_file, create_cloud_folder,
and rename_cloud_item in api/cloud/operations.py. Removed xfail markers from
test_cloud_audit.py. All 11 audit tests pass; 360 cloud tests pass total.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 08:17:24 +02:00
curo1305andClaude Sonnet 4.6 fd48c5b928 docs(phase-13): add security threat verification — 34/34 threats CLOSED
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-23 07:52:31 +02:00
curo1305 0c4f9ccba3 test(13): persist human verification items as UAT 2026-06-23 07:41:48 +02:00
curo1305 40b1572582 test(13): persist human verification items as UAT 2026-06-23 00:43:01 +02:00
curo1305 f92270fda4 test(13): persist human verification items as UAT 2026-06-23 00:42:48 +02:00
curo1305 b1a9f436c4 fix(13): WR-06 read result.status (not result.state) in testConnection to reflect actual server field name 2026-06-23 00:30:01 +02:00
curo1305 60df8552b5 fix(13): CR-05 commit audit log after write_audit_log in reconnect endpoint so event is not silently lost 2026-06-23 00:29:34 +02:00
curo1305 af0de30e93 fix(13): CR-04 soft-delete CloudItem row on successful cloud delete to prevent orphaned metadata 2026-06-23 00:29:11 +02:00
curo1305 405c7a6610 fix(13): CR-03 wrap post-upload DB block in try/except to handle provider-success/DB-fail divergence 2026-06-23 00:28:34 +02:00
curo1305 a1d1c3ba2e fix(13): CR-02 apply PurePosixPath.name guard in WebDAV upload_file and rename to prevent path traversal 2026-06-23 00:27:52 +02:00
curo1305 d4b2697109 fix(13): CR-01 use RFC 6266 filename*=UTF-8'' encoding in Content-Disposition to prevent header injection 2026-06-23 00:26:59 +02:00
curo1305 ebc74f4abe fix(13): CR-06 re-raise HTTPException in preview_cloud_file instead of masking as unsupported_preview 2026-06-23 00:26:23 +02:00
curo1305 fcb38c50b1 docs(13): add code review report 2026-06-23 00:24:20 +02:00
curo1305 cd5bd82260 docs(13-11): update ROADMAP.md via gsd roadmap.update-plan-progress — Phase 13 11/11 summaries verified 2026-06-23 00:19:40 +02:00
curo1305 f09387a9d4 docs(13-11): add self-check result to summary 2026-06-23 00:18:52 +02:00
curo1305 25cc2537bb docs(13-11): complete Phase 13 closeout plan summary and state update
- 13-11-SUMMARY.md: gate evidence (766 backend + 429 frontend tests pass, bandit 0 HIGH, npm audit clean, gitleaks 3 pre-existing), version 0.3.0, phase complete
- STATE.md: Phase 13 marked complete, progress updated, key decisions carried forward
2026-06-23 00:18:31 +02:00
curo1305 e68faf3051 chore(13-11): bump to v0.3.0, update docs, roadmap, and Phase 13 security gate evidence
- Version bump: 0.2.6 → 0.3.0 (Phase 13 complete — full cloud mutation surface shipped)
- CLAUDE.md: update current-state, shared module map, and Phase 13 non-negotiable rules
- README.md: add cloud file management, connection health, and authorized preview features; add Phase 13 mutation API table; mark Phase 13 complete
- ROADMAP.md: mark Phase 13 11/11 plans complete
- SECURITY.md: add Phase 13 threat register (T-13-01 through T-13-34), gate evidence (766 backend + 429 frontend tests pass, bandit 0 HIGH, npm audit 0 high/critical, gitleaks 3 pre-existing)
2026-06-23 00:14:57 +02:00
curo1305 e809df9f51 docs(13-10): complete frontend health UX and rendered-flow plan summary 2026-06-22 23:59:08 +02:00
curo1305 a7e55d1bb0 feat(13-10): reactive mock store in rendered-flow test surfaces health banner and no-probe assertions
- CloudFolderRenderedFlow.test.js: mock store uses Vue refs so folderFreshness propagates
  reactively to StorageBrowser via CloudFolderView's template binding (D-12 GREEN)
- setBrowseState in mock now updates _mockFolderFreshness so cloud-health-banner appears
  when API returns warning freshness (T-13-30 mitigated)
- Added testCloudConnection to API mock so D-13 no-probe assertion can verify it was not called
- beforeEach resets reactive refs for test isolation
- 11 rendered-flow tests pass; 70 total across health/store/view/rendered-flow suites
2026-06-22 23:56:13 +02:00
curo1305 60afb028bf feat(13-10): store-backed health mapping, reconnect UX, no-probe-on-navigation, and Google Drive consent copy
- D-12: connectionHealth ref as single source for browser compact status and Settings diagnostics
- D-13: handleHealthFailure schedules pendingHealthRetest; navigation never triggers probe
- D-13: testConnection method in store (explicit action only, not navigation side-effect)
- D-14: markReconnecting preserves cached browseItems as stale; markReconnectRefreshPending flag
- D-15: degraded vs requires_reauth health states distinguished in store vocabulary
- D-17: Google Drive scope notice in SettingsCloudTab for all connect/reconnect paths
- StorageBrowser: cloud-health-banner (warning/stale) and requires-reauth prompt with reconnect action
- 59 tests passing: store, Settings, and CloudFolderView health/reconnect suites
2026-06-22 23:55:58 +02:00
curo1305 a77b7324aa docs(13-09): complete move and delete mutation plan summary 2026-06-22 20:04:38 +02:00
curo1305 508d27643b feat(13-09): move stale-guard, descendant safety, reconciliation, delete disclosure, and metadata-only auditing
Move endpoint (D-07, D-08, D-09, T-13-27, T-13-28, T-13-29):
- Descendant destination detection via cloud_items ancestor-chain walk (D-09)
- Stale provider result marks source folder non-fresh, returns typed stale body (D-07)
- Success: upsert_cloud_item with new parent_ref before returning
- Success: update_folder_state for both source and destination folders
- Success: write_audit_log 'cloud.item_moved' with metadata-only payload

Delete endpoint (D-10, D-11, T-13-29):
- Pre-fetch item kind for D-10 folder-vs-file disclosure
- Success: update_folder_state for parent folder before returning
- Success: write_audit_log 'cloud.item_deleted' with delete_kind metadata
- Response carries is_folder/item_kind for frontend disclosure
- Failed deletes never emit audit rows

Promote xfail markers in test_cloud_audit.py for move and delete audit rows
2026-06-22 20:02:09 +02:00
curo1305 af7d45f3c4 test(13-09): add failing RED tests for move stale-guard, reconciliation, and delete disclosure/audit
- Move: stale etag returns 409 + refreshes source folder state
- Move: descendant destination rejected (D-09)
- Move: success upserts cloud_item with new parent_ref (reconcile-before-return)
- Move: success marks source + destination folders non-fresh
- Move: success writes metadata-only 'cloud.item_moved' audit row
- Delete: success marks parent folder non-fresh
- Delete: folder disclosure stronger than file (is_folder=True, D-10)
- Delete: success writes metadata-only 'cloud.item_deleted' audit row
- Delete: failed delete must not write false audit event
2026-06-22 19:58:00 +02:00
curo1305 3e9355f42d docs(13-08): complete create-folder and rename collision, stale guard, reconciliation plan summary 2026-06-22 19:53:24 +02:00
curo1305 aaa63c19e4 feat(13-08): implement create-folder and rename collision retry, stale guard, and reconciliation
- create_cloud_folder: bounded collision retry (up to 5 attempts) with keep_both_name counter suffix (D-05, D-06)
- create_cloud_folder: stale precondition marks parent folder non-fresh and returns typed stale result (D-07)
- rename_cloud_item: stale precondition marks parent folder non-fresh via folder state update (D-07)
- create_cloud_folder: reconcile-before-return upserts new folder in cloud_items and invalidates parent (T-13-26)
- rename_cloud_item: reconcile-before-return upserts renamed item and invalidates parent folder (T-13-26)
- Import upsert_cloud_item, update_folder_state, keep_both_name, CloudResource from service modules
2026-06-22 19:51:57 +02:00
curo1305 9ad99461bf test(13-08): add RED tests for create-folder and rename collision retry, stale guard, and reconciliation
- Test 1: collision auto-retry with bounded window (D-05, D-06)
- Test 2: stale guard updates folder state and returns typed stale kind (D-07)
- Test 3: create-folder and rename success upserts CloudItem and invalidates parent folder
- Test 4: failed rename does not mutate CloudItem rows
2026-06-22 19:49:41 +02:00
curo1305 561a40908d docs(13-07): complete cloud queue and preview plan summary 2026-06-22 19:45:55 +02:00