221 lines
17 KiB
Markdown
221 lines
17 KiB
Markdown
# DocuVault Roadmap: v0.3 Reimagining Cloud Storage integration
|
|
|
|
**Status:** Proposed
|
|
**Phases:** 12-16 plus inserted 12.1, 14.1, and 14.2
|
|
**Requirements:** 36
|
|
**Defined:** 2026-06-17
|
|
|
|
## Milestone Goal
|
|
|
|
Make connected cloud storage behave like first-class DocuVault storage: users can navigate and manage provider-owned files as if they were local, selectively analyze them, and find them through unified keyword and semantic search while DocuVault minimizes locally cached file bytes.
|
|
|
|
## Mandatory Cross-Cutting Gates
|
|
|
|
Before any phase is marked complete:
|
|
|
|
1. Backend and frontend test suites pass with zero failures; every new function, endpoint, store, and component has focused coverage.
|
|
2. Security gate verifies user/connection/item ownership, credential secrecy, SSRF defenses, metadata-only audit logs, cache isolation, and no admin document-content access.
|
|
3. Dependency audits report no high/critical vulnerabilities.
|
|
4. Provider integration tests cover supported capabilities and explicit unsupported behavior.
|
|
5. User-facing changes update `AGENTS.md`, relevant README sections, and application patch/minor versions according to project protocol.
|
|
|
|
## Phase Summary
|
|
|
|
| Phase | Name | Goal | Requirements |
|
|
|------:|------|------|--------------|
|
|
| 12 | 6/6 | Complete | 2026-06-21 |
|
|
| 13 | 11/11 | Complete | 2026-06-23 |
|
|
| 14 | 9/9 | Complete | 2026-06-23 |
|
|
| 14.1 | 4/5 | In Progress| |
|
|
| 14.2 | Cross-Codebase Review and Cleanup | Cross-reference backend/frontend code, remove duplication and dead code, consolidate shared paths, and fix inefficiencies without behavior changes | Cross-cutting quality gates |
|
|
| 15 | Unified Smart Search | Search local and analyzed cloud documents by exact text or semantic ideas without downloading files during queries | SEARCH-01..07 |
|
|
| 16 | Change Tracking and Reliability | Detect external provider changes, mark stale indexes, remove deleted items, and harden refresh behavior | SYNC-02..04 |
|
|
|
|
## Phase Details
|
|
|
|
### Phase 12: Cloud Resource Foundation
|
|
|
|
**Goal:** DocuVault has a provider-neutral cloud file capability contract and durable per-user cloud item index, and the shared browser can render cloud resources and supported actions without copying full provider files into local storage.
|
|
|
|
**Depends on:** Phase 11
|
|
**Requirements:** CONN-04, CLOUD-01, CLOUD-08, CACHE-01, CACHE-02, SYNC-01
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Every supported provider reports normalized capabilities for browse, open, preview, upload, folder creation, rename, move, delete, and change tracking.
|
|
2. Browsing any connected provider persists normalized, owner-scoped cloud item metadata including provider ID/path, parent, type, size, modification time, and version/etag.
|
|
3. `StorageBrowser.vue` remains the single file browser and renders cloud actions from capabilities, including disabled actions with provider-specific explanations.
|
|
4. Provider-owned bytes remain outside DocuVault while metadata, extracted text placeholders, topic links, and semantic-index state can be persisted independently.
|
|
5. Ownership and admin-negative tests prove cloud connection and cloud item metadata cannot cross user boundaries.
|
|
|
|
### Phase 12.1: Fix Nextcloud root listing and sync visibility (INSERTED)
|
|
|
|
**Goal:** Restore visible, truthful, metadata-only cloud browsing by repairing Nextcloud's adapter contract, applying one completeness/freshness contract across all providers, and aligning the shared frontend browser with normalized provider identities.
|
|
**Requirements:** CONN-04, CLOUD-01, CACHE-01, SYNC-01
|
|
**Depends on:** Phase 12
|
|
**Plans:** 4/4 plans complete
|
|
**Execution waves:** Wave 1: 12.1-01; Wave 2: 12.1-02 after 01; Wave 3: 12.1-03 after 02; Wave 4: 12.1-04 after 01-03.
|
|
|
|
Plans:
|
|
|
|
- [x] 12.1-01-PLAN.md
|
|
- [x] 12.1-02-PLAN.md
|
|
- [x] 12.1-03-PLAN.md
|
|
- [x] 12.1-04-PLAN.md
|
|
- [x] 12.1-01 (Wave 1) — Restore the shared four-provider adapter contract, secure Nextcloud URL/redirect handling, and metadata-only listing behavior.
|
|
- [x] 12.1-02 (Wave 2; after 12.1-01) — Centralize complete/incomplete reconciliation so incomplete provider results cannot become fresh or delete cached metadata.
|
|
- [x] 12.1-03 (Wave 3; after 12.1-02) — Align the shared browser and trees with `kind`, opaque `provider_item_id` routing, lineage-based breadcrumbs, and server freshness.
|
|
- [ ] 12.1-04 (Wave 4; after 12.1-01 through 12.1-03) — Run sanitized two-mode live Nextcloud acceptance, rendered/security/secret gates, and phase closeout.
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Nextcloud and generic WebDAV expose one canonical `CloudResourceAdapter.list_folder` implementation, and all four providers pass the same runtime identity/completeness/no-byte/no-mutation contract with provider-specific pagination fixtures.
|
|
2. Every outbound Nextcloud/WebDAV target, including redirects, is rejected unless each hop is HTTPS, same-origin, and passes URL plus resolved-address SSRF validation; automatic redirects are disabled.
|
|
3. `complete=False` never marks a folder fresh, advances `last_refreshed_at`, or authorizes unseen-item deletion in synchronous or worker paths; cached rows remain usable with a controlled warning.
|
|
4. The frontend classifies by `kind`, navigates through named routes/query serialization with opaque `provider_item_id` values, builds breadcrumbs from explicit navigation lineage rather than splitting provider IDs, and renders backend freshness verbatim.
|
|
5. Live Nextcloud validation first runs a nonblocking sanitized diagnostic; exact names/kinds become a blocking gate only after the owner-confirmed manifest is stored in a tracked, non-secret fixture. Unexpected live names are never printed or persisted.
|
|
6. Automated backend/frontend/rendered-flow tests, dedicated per-plan security reviews, dependency audits, an executable secret scan, documentation/version updates, and one atomic commit/push per plan pass without exposing credentials or provider bytes.
|
|
|
|
### Phase 13: Virtual-Local Cloud Operations
|
|
|
|
**Goal:** Users can maintain healthy cloud connections and perform the main file-management workflows in connected storage with the same shared browser interactions used for local files.
|
|
|
|
**Depends on:** Phase 12
|
|
**Requirements:** CONN-01, CONN-02, CONN-03, CLOUD-02, CLOUD-03, CLOUD-04, CLOUD-05, CLOUD-06, CLOUD-07, CLOUD-09
|
|
**Plans:** 11/11 plans complete
|
|
**Execution waves:** Wave 0: 13-01 and 13-02 in parallel; Wave 1: 13-03 after 13-01; Wave 2: 13-04 after 13-01 and 13-03; Wave 3: 13-05 after 13-01, 13-03, and 13-04; Wave 4: 13-06 after 13-05; Wave 5: 13-07 and 13-08 in parallel (13-07 after 13-02, 13-04, and 13-06; 13-08 after 13-03, 13-04, and 13-06); Wave 6: 13-09 after 13-08; Wave 7: 13-10 after 13-02, 13-04, 13-07, and 13-09; Wave 8: 13-11 after 13-03 through 13-10.
|
|
|
|
Plans:
|
|
|
|
- [x] 13-01-PLAN.md — Create red backend and provider-contract suites for reconnect, content, mutations, and audit secrecy.
|
|
- [x] 13-02-PLAN.md — Create red frontend/store suites for queue, preview, health UX, broader Drive consent, and no-probe-on-navigation.
|
|
- [x] 13-03-PLAN.md — Build the mutable cloud contract and orchestration seam without breaking centralized reconciliation.
|
|
- [x] 13-04-PLAN.md — Implement connection-ID reconnect, explicit health test, broader Drive scope handling, and authorized content routes.
|
|
- [x] 13-05-PLAN.md — Implement backend upload provider mechanics, typed route results, and refreshed-credential handoff.
|
|
- [x] 13-06-PLAN.md — Complete upload reconcile, freshness, and metadata-only audit follow-through before frontend queue wiring.
|
|
- [x] 13-07-PLAN.md — Wire the shared browser queue and binary-only preview or download fallback through thin cloud view handlers.
|
|
- [x] 13-08-PLAN.md — Implement backend create-folder and rename semantics with collision, stale, and stable-identity safeguards.
|
|
- [x] 13-09-PLAN.md — Implement backend move and delete semantics with same-connection, disclosure, security, and audit safeguards.
|
|
- [x] 13-10-PLAN.md — Finish shared-browser mutation UX, store-backed health behavior, reconnect copy, and the no-probe invariant.
|
|
- [x] 13-11-PLAN.md — Run closeout-only docs, versions, full gates, explicit secret scan, and ship-readiness checks.
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Users can connect, test, reconnect, and disconnect providers while seeing current health and actionable reauthentication/error states.
|
|
2. Users can open or preview supported cloud documents and upload files into the current cloud folder through authorized DocuVault endpoints.
|
|
3. Users can create folders, rename items, move items within one connection, and delete items after confirmation wherever the provider supports each action.
|
|
4. Successful mutations immediately invalidate affected listings/index entries, refresh the shared browser, and write metadata-only audit events.
|
|
5. Provider-specific tests verify operation semantics, conflict/error responses, token refresh persistence, SSRF protection, and explicit unsupported capabilities.
|
|
|
|
### Phase 14: Selective Analysis and Byte Cache
|
|
|
|
**Goal:** Users can analyze individual files, selected scopes, or whole connections through observable and controllable background jobs, while file bytes are hydrated only on demand and evicted safely.
|
|
|
|
**Depends on:** Phase 13
|
|
**Requirements:** ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05
|
|
**Plans:** 9/9 plans complete
|
|
**Execution waves:** Wave 0: 14-01; Wave 1: 14-02 after 14-01; Wave 2: 14-03 after 14-02; Wave 3: 14-04 after 14-02 and 14-03; Wave 4: 14-05 after 14-03 and 14-04; Wave 5: 14-06 after 14-03 and 14-05; Wave 6: 14-07 after 14-04 and 14-05; Wave 7: 14-08 after 14-03 and 14-07; Wave 8: 14-09 after 14-01 through 14-08.
|
|
|
|
Plans:
|
|
|
|
- [x] 14-01-PLAN.md — Create red backend/frontend tests for analysis scopes, estimates, cache invariants, queue states, and security negatives.
|
|
- [x] 14-02-PLAN.md — Add durable cache/job/settings schema plus version-key and settings helpers.
|
|
- [x] 14-03-PLAN.md — Implement owner-scoped byte cache service, quota accounting, pinning, eviction, and cache settings/status API.
|
|
- [x] 14-04-PLAN.md — Implement estimate/enqueue/status/cancel/skip/retry analysis APIs with idempotent job creation.
|
|
- [x] 14-05-PLAN.md — Implement Celery-backed cloud analysis processing, cache hydration, extraction, classification, and cancellation.
|
|
- [x] 14-06-PLAN.md — Route existing cloud open/preview/download byte hydration through the bounded cache lifecycle.
|
|
- [x] 14-07-PLAN.md — Wire shared-browser analysis actions, estimates, aggregate progress, expandable queue, and controls.
|
|
- [x] 14-08-PLAN.md — Add Settings controls for cache limit, progress detail, and failure behavior.
|
|
- [x] 14-09-PLAN.md — Run closeout docs, versions, full test/security gates, validation, commit, and push.
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Users can analyze one file, a multi-selection, a folder tree, or a whole connection after reviewing recursive file-count and byte-size estimates.
|
|
2. The UI exposes aggregate and per-item queued, downloading, extracting, classifying, indexed, cancelled, and failed states with retry and queued-work cancellation.
|
|
3. Analysis jobs are idempotent by cloud item and provider version/etag, and they never mutate the provider-owned file.
|
|
4. Cloud bytes are cached only for active opening, preview, or analysis work and are evicted by configurable limits without interrupting pinned active jobs.
|
|
5. Cache ownership, cache-key versioning, cancellation, duplicate-job, failure-retry, and eviction behavior have dedicated tests.
|
|
|
|
### Phase 14.1: Cloud/Local File Parity Hardening (INSERTED)
|
|
|
|
**Goal:** Cloud files opened, viewed, downloaded, analyzed, and displayed in results use the same user-facing behavior as local files, while preserving provider ownership, cache boundaries, authorization, and no-provider-mutation guarantees.
|
|
**Requirements:** CLOUD-02, ANALYZE-01, ANALYZE-02, ANALYZE-03, ANALYZE-04, ANALYZE-05, ANALYZE-06, ANALYZE-07, CACHE-03, CACHE-04, CACHE-05
|
|
**Depends on:** Phase 14
|
|
**Plans:** 4/5 plans executed
|
|
**Execution waves:** Wave 1: 14.1-01 (RED tests); Wave 2: 14.1-02 after 01 (backend detail/force/retry); Wave 3: 14.1-03 after 01-02 (shared detail surface + cloud route/view); Wave 4: 14.1-04 after 01-03 (browser row parity + cloud row navigation); Wave 5: 14.1-05 after 01-04 (gates, security, docs, version, commit).
|
|
|
|
Plans:
|
|
|
|
- [x] 14.1-01-PLAN.md — RED backend + frontend parity tests (cloud detail endpoint, force re-analyze, single-item retry, route + row parity).
|
|
- [x] 14.1-02-PLAN.md — Backend cloud detail endpoint (CloudItemDetailOut + resolve_owned_cloud_item_detail), force re-analyze flag, single-item retry-job creation.
|
|
- [x] 14.1-03-PLAN.md — Shared DocumentDetailSurface extraction, DocumentView refactor (Re-analyze copy), CloudDetailView + cloud-file-detail route + getCloudItemDetail.
|
|
- [x] 14.1-04-PLAN.md — Browse row topics/analysis_status/stale, StorageBrowser row parity via store translator, cloud row click → cloud detail route (no auto preview/download).
|
|
- [ ] 14.1-05-PLAN.md — Full test suites, security gate, dependency/secret audits, CLAUDE.md/README updates, version bump, atomic commit.
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Cloud and local file rows/cards expose equivalent open, view, download, analyze, retry, and reanalyze states through `StorageBrowser.vue`.
|
|
2. Analyzed cloud files surface extracted text, topics, analysis status, stale/current state, and retry/reanalyze affordances wherever equivalent local file data appears.
|
|
3. Authorized cloud open, preview, and download never expose provider URLs or credentials and hydrate bytes only through the existing cache lifecycle.
|
|
4. Unsupported or provider-limited cloud actions use typed responses and shared UI states instead of creating local/cloud UX forks.
|
|
5. End-to-end parity tests cover local versus cloud workflows for open, preview/download fallback, analyze, status display, retry, and ownership negatives.
|
|
|
|
### Phase 14.2: Cross-Codebase Review and Cleanup (INSERTED)
|
|
|
|
**Goal:** Cross-reference the full codebase, remove duplicate logic, consolidate shared helpers/components/services, improve inefficient paths, delete dead code, and preserve behavior.
|
|
**Requirements:** Cross-cutting quality gates
|
|
**Depends on:** Phase 14.1
|
|
**Plans:** 0 plans
|
|
|
|
Plans:
|
|
|
|
- [ ] TBD (run /gsd-plan-phase 14.2 to break down)
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Backend routers, services, providers, and tasks are audited for duplicated helper logic, raw inline orchestration, repeated parsing/formatting, and inefficient query/cache paths.
|
|
2. Frontend views, components, stores, and utilities are audited for duplicated browser logic, formatters, provider styling, tree behavior, and local/cloud branching.
|
|
3. Shared module maps and non-negotiable rules in `AGENTS.md` are updated for any newly centralized helpers or components.
|
|
4. Unused files, dead imports, stale tests, obsolete planning references, and unreachable code are removed in the same cleanup work.
|
|
5. Full backend, frontend, security, audit, dependency, secret-scan, and rendered UI gates pass after behavior-preserving cleanup.
|
|
|
|
### Phase 15: Unified Smart Search
|
|
|
|
**Goal:** One search experience finds local and analyzed cloud documents by keywords, sentences, or semantic ideas and opens cloud results through on-demand hydration.
|
|
|
|
**Depends on:** Phase 14.2
|
|
**Requirements:** SEARCH-01, SEARCH-02, SEARCH-03, SEARCH-04, SEARCH-05, SEARCH-06, SEARCH-07
|
|
|
|
**Success Criteria:**
|
|
|
|
1. A single query returns authorized local and analyzed cloud results ranked through full-text and semantic relevance.
|
|
2. Keyword and sentence searches match persisted extracted text, while idea searches use persisted embeddings or equivalent semantic indexes.
|
|
3. Results clearly show source, provider, cloud location, topics, and current analysis/stale status and can be filtered by those fields.
|
|
4. Search execution performs no provider file download; only opening or previewing a result may hydrate bytes.
|
|
5. Search isolation, ranking, filtering, stale-result handling, and no-download-on-query invariants have automated coverage.
|
|
|
|
### Phase 16: Change Tracking and Reliability
|
|
|
|
**Goal:** DocuVault remains accurate when users or other applications modify connected cloud storage outside DocuVault.
|
|
|
|
**Depends on:** Phase 15
|
|
**Requirements:** SYNC-02, SYNC-03, SYNC-04
|
|
|
|
**Success Criteria:**
|
|
|
|
1. Provider delta feeds or bounded metadata refreshes detect changed versions and mark previously analyzed items stale until reanalysis.
|
|
2. Items deleted outside DocuVault disappear from navigation and are excluded from search without deleting unrelated user data.
|
|
3. Refresh jobs are idempotent, cursor-aware where supported, rate-limited, and retry transient failures with bounded exponential backoff.
|
|
4. Users can see refresh/stale/error state and deliberately reanalyze changed items.
|
|
5. End-to-end UAT verifies cloud-as-local operations, selective analysis, cache offload, unified smart search, and external-change recovery across supported provider classes.
|
|
|
|
## Coverage
|
|
|
|
- v0.3 requirements: 36
|
|
- Mapped to phases: 36
|
|
- Unmapped: 0
|
|
- Duplicate mappings: 0
|
|
|
|
---
|
|
*Roadmap proposed: 2026-06-17*
|