docs(12.1): add phase 12.1 planning artifacts — Nextcloud root listing fix
Plans 01–04, CONTEXT, PATTERNS, RESEARCH, UAT, and .gitkeep for the fix-nextcloud-root-listing-and-sync-visibility sub-phase. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
ab31c1344c
commit
73328eee0b
@@ -0,0 +1,211 @@
|
||||
---
|
||||
phase: "12.1"
|
||||
plan: "01"
|
||||
type: tdd
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- backend/storage/cloud_base.py
|
||||
- backend/storage/cloud_utils.py
|
||||
- backend/storage/webdav_backend.py
|
||||
- backend/storage/nextcloud_backend.py
|
||||
- backend/storage/google_drive_backend.py
|
||||
- backend/storage/onedrive_backend.py
|
||||
- backend/storage/cloud_backend_factory.py
|
||||
- backend/tests/test_cloud_provider_contract.py
|
||||
- backend/tests/test_cloud_backends.py
|
||||
- backend/tests/test_webdav_backend.py
|
||||
- backend/tests/fixtures/cloud/nextcloud_root.xml
|
||||
- backend/tests/fixtures/cloud/webdav_root.xml
|
||||
- backend/tests/fixtures/cloud/google_drive_pages.json
|
||||
- backend/tests/fixtures/cloud/onedrive_pages.json
|
||||
- backend/main.py
|
||||
- frontend/package.json
|
||||
- frontend/package-lock.json
|
||||
- AGENTS.md
|
||||
- README.md
|
||||
- RUNBOOK.md
|
||||
- SECURITY.md
|
||||
autonomous: true
|
||||
requirements:
|
||||
- CLOUD-01
|
||||
- CACHE-01
|
||||
- SYNC-01
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Nextcloud accepts the exact four-argument CloudResourceAdapter.list_folder contract and returns CloudListing rather than a legacy list of dictionaries"
|
||||
- "One shared contract suite runs against Nextcloud, generic WebDAV, Google Drive, and OneDrive"
|
||||
- "Every provider normalizes root and nested children with trusted connection/user identity, opaque provider navigation identity, parent reference, kind, and nullable metadata"
|
||||
- "A provider reports complete=True only after every page or complete DAV multistatus has been parsed successfully"
|
||||
- "Browse contract tests prove zero byte-download and zero provider-mutation calls"
|
||||
artifacts:
|
||||
- path: "backend/tests/test_cloud_provider_contract.py"
|
||||
provides: "Reusable four-provider adapter contract and forbidden-operation spies"
|
||||
contains: "test_provider_list_folder_contract"
|
||||
- path: "backend/storage/nextcloud_backend.py"
|
||||
provides: "Nextcloud specialization that preserves the canonical WebDAV browse method"
|
||||
contains: "NextcloudBackend"
|
||||
- path: "backend/tests/fixtures/cloud/nextcloud_root.xml"
|
||||
provides: "Credential-free DAV root fixture with folders, files, spaces, root self-entry, and optional metadata"
|
||||
contains: "multistatus"
|
||||
key_links:
|
||||
- from: "backend/storage/nextcloud_backend.py"
|
||||
to: "backend/storage/webdav_backend.py"
|
||||
via: "inheritance or a narrow normalization hook; no incompatible public override"
|
||||
pattern: 'class NextcloudBackend\(WebDAVBackend\)'
|
||||
- from: "backend/tests/test_cloud_provider_contract.py"
|
||||
to: "backend/storage/cloud_base.py"
|
||||
via: "the same parametrized assertions for every provider factory"
|
||||
pattern: "CloudListing|CloudResourceAdapter"
|
||||
- from: "provider fixtures"
|
||||
to: "CloudResource.provider_item_id"
|
||||
via: "provider-specific response normalization"
|
||||
pattern: "provider_item_id"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Repair the provider boundary test-first. Reproduce the Nextcloud signature failure, replace the legacy browse override with the shared adapter contract, and establish one reusable listing contract for all four providers with provider-specific root, nested, pagination, malformed-response, and no-byte/no-mutation fixtures.
|
||||
|
||||
This plan changes metadata listing only. It does not add upload, create-folder, rename, move, delete, preview, download, or byte caching.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-PATTERNS.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-02-PLAN.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md
|
||||
@backend/storage/cloud_base.py
|
||||
@backend/storage/nextcloud_backend.py
|
||||
@backend/storage/webdav_backend.py
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Write the shared four-provider contract and red tests</name>
|
||||
<files>backend/tests/test_cloud_provider_contract.py, backend/tests/test_cloud_backends.py, backend/tests/test_webdav_backend.py, backend/tests/fixtures/cloud/nextcloud_root.xml, backend/tests/fixtures/cloud/webdav_root.xml, backend/tests/fixtures/cloud/google_drive_pages.json, backend/tests/fixtures/cloud/onedrive_pages.json</files>
|
||||
<read_first>
|
||||
- backend/storage/cloud_base.py — exact CloudListing and CloudResource invariants
|
||||
- backend/storage/cloud_backend_factory.py — supported-provider construction paths
|
||||
- backend/tests/test_cloud_backends.py — existing provider mocks and missing behavioral coverage
|
||||
- backend/tests/test_webdav_backend.py — existing webdavclient3 test conventions
|
||||
- .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md — confirmed signatures and provider fixture matrix
|
||||
</read_first>
|
||||
<action>
|
||||
Create a reusable parametrized contract harness whose provider cases supply only fixture setup and response stubs. Add concrete tests named `test_provider_list_folder_contract`, `test_provider_root_and_nested_identity`, `test_provider_metadata_normalization`, `test_provider_consumes_all_pages_before_complete`, `test_provider_page_failure_is_incomplete`, and `test_provider_listing_never_downloads_or_mutates`. Invoke each adapter through the canonical positional/keyword signature `(connection_id, user_id, parent_ref=None, page_token=None)` and assert the result is CloudListing; this test must fail against the current Nextcloud override.
|
||||
|
||||
For every provider assert: trusted caller UUIDs are copied into resources; `provider_item_id` is the opaque navigation reference; `parent_ref` is the requested folder reference; `kind` is exactly file/folder; optional size/content-type/modified/version metadata becomes `None` when absent; and no provider response can override owner/connection IDs. Install spies that fail on GET-media/download/get_object/put/upload/create/move/copy/rename/delete methods and assert quota/MinIO are not imported or called by listing.
|
||||
|
||||
Add credential-free provider fixtures: Nextcloud and WebDAV Depth-1 multistatus with root self-entry, encoded spaces/unicode, collections, files, absent optional properties, absolute/relative hrefs, and a foreign/escaping href; Google Drive with root/nested queries, native files with nullable size, trashed exclusion, and multiple nextPageToken pages; OneDrive with root/nested endpoints, folder/file facets, encoded names, and multiple @odata.nextLink pages. Fixture names must be synthetic—not copied from the live account—and contain no URL, username, tokens, or provider-owned unexpected names.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- the Nextcloud canonical-signature regression fails before implementation and passes afterward
|
||||
- the same six contract tests execute for nextcloud, webdav, google_drive, and onedrive
|
||||
- fixtures cover root, nested, missing optional metadata, malformed/foreign data, and provider pagination
|
||||
- no contract fixture or pytest ID contains credentials, a full live URL, or live unexpected names
|
||||
- forbidden byte and mutation spies remain untouched for every provider
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_cloud_provider_contract.py tests/test_cloud_backends.py tests/test_webdav_backend.py</automated></verify>
|
||||
<done>The shared contract is red for the known Nextcloud violation and fully specifies normalized, complete, read-only behavior for all providers.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Repair Nextcloud/WebDAV listing without splitting the public contract</name>
|
||||
<files>backend/storage/cloud_utils.py, backend/storage/webdav_backend.py, backend/storage/nextcloud_backend.py, backend/storage/cloud_backend_factory.py, backend/tests/test_cloud_provider_contract.py, backend/tests/test_webdav_backend.py</files>
|
||||
<read_first>
|
||||
- backend/storage/nextcloud_backend.py — incompatible legacy list_folder override to remove
|
||||
- backend/storage/webdav_backend.py — canonical list_folder implementation and SSRF revalidation points
|
||||
- backend/storage/cloud_utils.py — canonical URL/SSRF helpers
|
||||
- backend/storage/cloud_backend_factory.py — credential-to-adapter construction
|
||||
- backend/tests/test_cloud_security.py — existing SSRF invariants that must remain green
|
||||
</read_first>
|
||||
<action>
|
||||
Remove the incompatible `NextcloudBackend.list_folder(folder_path="") -> list[dict]` public override. Let Nextcloud inherit the canonical WebDAV implementation, or add only a protected Nextcloud endpoint/path-normalization hook consumed by that implementation. Keep exactly one public DAV `list_folder` algorithm.
|
||||
|
||||
Normalize a Nextcloud base URL and username to its canonical DAV root in one pure shared helper used at adapter construction, while accepting an already canonical endpoint idempotently. Percent-encode the username as one path segment; preserve deployment subpaths; reject userinfo, query, fragment, non-HTTPS, malformed URLs, and URLs rejected by `validate_cloud_url`; never log the normalized URL because it contains the username path segment.
|
||||
|
||||
Disable automatic redirects in the DAV HTTP transport. If redirect support is required, follow it explicitly with a strict bounded hop count and, before every hop, parse and validate the target URL, require HTTPS and the original normalized origin, resolve the hostname, and reject loopback, private, link-local, multicast, unspecified, reserved, or changed/untrusted addresses. Revalidate immediately before dispatch to limit DNS rebinding. Add automated redirect tests for loopback, RFC1918/private, link-local, cross-origin, non-HTTPS, excessive-hop, and a permitted same-origin target; assert rejected targets receive no second request and no URL is leaked.
|
||||
|
||||
Make the shared DAV listing parser authoritative for one Depth-1 response: exclude exactly the normalized self href, preserve a canonical child provider reference, use DAV collection resource type for kind, decode display path segments exactly once, reject hrefs outside the configured root, and return complete=False on malformed/ambiguous responses. Do not silently convert per-item metadata failures into a complete listing. If the existing SDK cannot expose a safe one-response parser without a new dependency, retain its calls but enforce the same completeness and no-byte contract; document the tradeoff in the summary.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- `inspect.signature(NextcloudBackend.list_folder)` matches the canonical adapter method
|
||||
- root and nested Nextcloud/WebDAV fixtures return normalized CloudListing values
|
||||
- malformed XML, foreign hrefs, or interrupted metadata parsing cannot return complete=True
|
||||
- Nextcloud URL normalization is idempotent and remains behind canonical SSRF validation
|
||||
- automatic redirects are disabled or each bounded hop is same-origin and URL/DNS revalidated; loopback/private/link-local/cross-origin redirect tests pass
|
||||
- no second public Nextcloud listing path or raw dictionary response remains
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_cloud_provider_contract.py tests/test_webdav_backend.py tests/test_cloud_security.py -k 'provider or webdav or nextcloud or ssrf or redirect'</automated></verify>
|
||||
<done>Nextcloud and generic WebDAV share one secure canonical listing path and the known signature defect is eliminated.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: Bring Drive and OneDrive to the same completion contract</name>
|
||||
<files>backend/storage/google_drive_backend.py, backend/storage/onedrive_backend.py, backend/tests/test_cloud_provider_contract.py, backend/tests/test_cloud_backends.py</files>
|
||||
<read_first>
|
||||
- backend/storage/google_drive_backend.py — root query, nextPageToken, metadata and error behavior
|
||||
- backend/storage/onedrive_backend.py — root/items endpoints, @odata.nextLink, metadata and error behavior
|
||||
- backend/storage/cloud_base.py — complete/next_page_token semantics
|
||||
- backend/tests/fixtures/cloud/google_drive_pages.json — required fixture cases
|
||||
- backend/tests/fixtures/cloud/onedrive_pages.json — required fixture cases
|
||||
</read_first>
|
||||
<action>
|
||||
Make Google Drive and OneDrive pass the same contract without creating provider-specific API schemas. Google Drive must translate root internally, follow every nextPageToken before complete=True, exclude trashed items, recognize folder MIME type, preserve native Google files with nullable size, and map controlled auth/scope failures without raw provider responses. OneDrive must translate root internally, follow every trusted @odata.nextLink, identify folders by facet, preserve nullable metadata, and reject cross-origin/untrusted continuation URLs. Any page failure returns complete=False with already normalized items retained and no deletion authority.
|
||||
|
||||
Keep provider IDs opaque. Do not turn DocuVault CloudItem UUIDs into provider references and do not call media/download endpoints. Extend concrete provider tests where behavior is too provider-specific for the shared assertions.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- all pages are consumed before complete=True for both providers
|
||||
- page failure or untrusted continuation returns complete=False
|
||||
- Drive native files and OneDrive folders normalize without fabricated metadata
|
||||
- raw auth/provider bodies and continuation URLs are absent from raised/displayable messages
|
||||
- the complete four-provider focused suite passes
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_cloud_provider_contract.py tests/test_cloud_backends.py tests/test_webdav_backend.py tests/test_cloud_security.py</automated></verify>
|
||||
<done>All four providers satisfy one listing/completeness/identity/no-byte contract with provider-specific pagination evidence.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4: Document, security-review, commit, and push Plan 01</name>
|
||||
<files>backend/main.py, frontend/package.json, frontend/package-lock.json, AGENTS.md, README.md, RUNBOOK.md, SECURITY.md, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-01-SUMMARY.md</files>
|
||||
<action>Update the required docs for shipped Plan 01 behavior, including shared helpers and redirect policy. Because this repairs user-visible cloud browsing, bump the backend/frontend patch version once and keep package-lock metadata synchronized. Run a dedicated security agent against this plan's diff and resolve or escalate every finding. Re-run focused/full backend tests, Bandit and dependency gates applicable to the changed backend, `docker compose config --quiet`, and `git diff --check`. Stage only Plan 01 files, verify `.env` and unrelated changes are absent, create one atomic `fix(12.1): restore cloud adapter contract` commit, and push immediately according to AGENTS.md.</action>
|
||||
<verify><automated>docker compose config --quiet && cd backend && pytest -v && bandit -r . && pip audit && cd .. && git diff --check</automated></verify>
|
||||
<done>Plan 01 has its own docs, clean security-agent verdict, green gates, atomic commit, and push.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
| Threat ID | Threat | Mitigation and evidence |
|
||||
|-----------|--------|-------------------------|
|
||||
| T-12.1-01 | Provider response forges owner or connection identity | Contract copies trusted caller UUIDs and tests hostile fixture fields |
|
||||
| T-12.1-02 | SSRF through Nextcloud normalization, redirects, or DAV hrefs | Canonical validator before construction/request; reject userinfo/query/fragment/private targets/escaping hrefs |
|
||||
| T-12.1-03 | Partial page authorizes metadata deletion | complete=False on any page/parser/trust failure; contract tests every provider |
|
||||
| T-12.1-04 | Browse downloads or mutates provider content | Failing spies on byte and mutation methods; metadata requests only |
|
||||
| T-12.1-05 | Secrets/provider data leak through fixtures or failures | Synthetic fixtures, controlled exceptions, no live URL/user/token/unexpected-name output |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
1. Observe the Nextcloud signature test fail before implementation and pass after repair.
|
||||
2. Run the shared contract for all four providers, not four copied suites.
|
||||
3. Run provider-specific root, nested, pagination, malformed-response, and no-byte tests.
|
||||
4. Run existing cloud security tests to prove SSRF and credential boundaries remain intact.
|
||||
5. Search confirms no incompatible Nextcloud public list_folder override remains.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Nextcloud can be invoked through CloudResourceAdapter exactly like every other provider.
|
||||
- Every provider has executable evidence for normalized identity, kind, metadata, pagination/completeness, and read-only behavior.
|
||||
- Provider failures cannot masquerade as an authoritative empty collection.
|
||||
- No Phase 13 mutation or Phase 14 byte behavior enters the contract.
|
||||
</success_criteria>
|
||||
|
||||
<output>Create `.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-01-SUMMARY.md` when complete.</output>
|
||||
@@ -0,0 +1,168 @@
|
||||
---
|
||||
phase: "12.1"
|
||||
plan: "02"
|
||||
type: tdd
|
||||
wave: 2
|
||||
depends_on:
|
||||
- "12.1-01"
|
||||
files_modified:
|
||||
- backend/services/cloud_items.py
|
||||
- backend/api/cloud/browse.py
|
||||
- backend/tasks/cloud_tasks.py
|
||||
- backend/api/cloud/schemas.py
|
||||
- backend/tests/test_cloud_items.py
|
||||
- backend/tests/test_cloud.py
|
||||
- backend/tests/test_cloud_security.py
|
||||
- backend/main.py
|
||||
- frontend/package.json
|
||||
- frontend/package-lock.json
|
||||
- AGENTS.md
|
||||
- README.md
|
||||
- RUNBOOK.md
|
||||
- SECURITY.md
|
||||
autonomous: true
|
||||
requirements:
|
||||
- CLOUD-01
|
||||
- CACHE-01
|
||||
- SYNC-01
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "CloudListing.complete=False can never transition a folder to fresh in either synchronous browse or Celery refresh"
|
||||
- "A complete authoritative empty listing may become fresh and reconcile deletions"
|
||||
- "Incomplete/error refresh retains cached children and the previous last successful refresh timestamp"
|
||||
- "The synchronous API and background worker use one service-level reconciliation/completion gate"
|
||||
- "Connection-ID ownership, credential secrecy, zero-byte browse, and zero quota mutation remain enforced"
|
||||
artifacts:
|
||||
- path: "backend/services/cloud_items.py"
|
||||
provides: "Single listing application gate shared by API and worker"
|
||||
contains: "reconcile_cloud_listing"
|
||||
- path: "backend/tests/test_cloud_items.py"
|
||||
provides: "Complete-versus-incomplete reconciliation and timestamp regression tests"
|
||||
contains: "incomplete_listing"
|
||||
- path: "backend/tests/test_cloud.py"
|
||||
provides: "Connection-ID browse freshness integration tests"
|
||||
contains: "complete_false"
|
||||
key_links:
|
||||
- from: "backend/api/cloud/browse.py"
|
||||
to: "backend/services/cloud_items.py"
|
||||
via: "shared listing application/finalization service"
|
||||
pattern: "reconcile_cloud_listing"
|
||||
- from: "backend/tasks/cloud_tasks.py"
|
||||
to: "backend/services/cloud_items.py"
|
||||
via: "the same service gate used by synchronous browse"
|
||||
pattern: "reconcile_cloud_listing"
|
||||
- from: "CloudListing.complete"
|
||||
to: "CloudFolderState.refresh_state"
|
||||
via: "fresh only for authoritative completion"
|
||||
pattern: "complete.*fresh"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Make backend freshness truthful. Centralize how normalized listings are reconciled and finalized so incomplete, ambiguous, or failed provider results retain metadata and produce a controlled warning rather than a false fresh state.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-SECURITY.md
|
||||
@backend/services/cloud_items.py
|
||||
@backend/api/cloud/browse.py
|
||||
@backend/tasks/cloud_tasks.py
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Specify complete, incomplete, and empty reconciliation semantics</name>
|
||||
<files>backend/tests/test_cloud_items.py, backend/tests/test_cloud.py, backend/tests/test_cloud_security.py</files>
|
||||
<read_first>
|
||||
- backend/services/cloud_items.py — current reconciliation and update_folder_state behavior
|
||||
- backend/api/cloud/browse.py — first-visit synchronous fresh transition
|
||||
- backend/tasks/cloud_tasks.py — worker fresh transition and retry paths
|
||||
- backend/tests/test_cloud_items.py — existing incomplete-retention tests
|
||||
- backend/tests/test_cloud.py — browse response/freshness fixtures
|
||||
</read_first>
|
||||
<action>
|
||||
Add red tests named `test_incomplete_listing_never_marks_folder_fresh`, `test_incomplete_listing_retains_cached_rows_and_last_success`, `test_complete_empty_listing_is_authoritative_and_fresh`, `test_partial_items_upsert_without_deleting_unseen_children`, `test_sync_browse_returns_warning_for_complete_false`, and `test_worker_returns_warning_for_complete_false`. Seed a previous successful timestamp and prove it is unchanged on incomplete/error results. Distinguish a provider-confirmed `CloudListing(items=(), complete=True)` from `complete=False`; only the former may soft-delete unseen children and update last_refreshed_at.
|
||||
|
||||
Add API/security assertions that responses remain whitelisted, warning messages are controlled, raw provider errors/URLs are absent, cached items remain owner scoped, and no get_object/download/MinIO/quota mutation occurs. Exercise both root `parent_ref=None` and a nested opaque provider reference.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- tests fail on the current unconditional fresh transitions
|
||||
- tests cover service, synchronous API, and Celery worker paths
|
||||
- complete-empty and incomplete-empty results have explicitly different outcomes
|
||||
- cached rows and prior last_refreshed_at survive incomplete/error results
|
||||
- owner/admin/credential/no-byte/no-quota assertions remain in the focused run
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_cloud_items.py tests/test_cloud.py tests/test_cloud_security.py -k 'incomplete or complete_false or complete_empty or refresh or no_byte or quota'</automated></verify>
|
||||
<done>Executable red tests define truthful completion semantics at every backend entry point.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Centralize listing application and freshness finalization</name>
|
||||
<files>backend/services/cloud_items.py, backend/api/cloud/browse.py, backend/tasks/cloud_tasks.py, backend/api/cloud/schemas.py, backend/tests/test_cloud_items.py, backend/tests/test_cloud.py</files>
|
||||
<read_first>
|
||||
- backend/services/cloud_items.py — sole reconciliation entry point and service exception conventions
|
||||
- backend/api/cloud/schemas.py — credential-free freshness response contract
|
||||
- backend/api/cloud/browse.py — cached-first behavior and controlled warning copy
|
||||
- backend/tasks/cloud_tasks.py — terminal/transient classification and retry behavior
|
||||
- AGENTS.md — service layer must not raise HTTPException
|
||||
</read_first>
|
||||
<action>
|
||||
Add one service-level operation in `services/cloud_items.py` that calls `reconcile_cloud_listing` and finalizes CloudFolderState according to listing completeness. Both browse.py and cloud_tasks.py must call it; neither caller may independently mark a listing fresh. Preserve `reconcile_cloud_listing` as the sole metadata reconciliation entry point. A complete listing sets fresh, clears controlled errors, and advances last_refreshed_at. An incomplete listing may upsert seen items but cannot delete unseen items, cannot advance last_refreshed_at, and sets warning with a stable code such as `incomplete_listing` and credential-free message.
|
||||
|
||||
Keep provider exceptions controlled: auth/credential failures remain terminal warnings; transient transport errors retain cached data and bounded retry behavior. Do not serialize raw exceptions, response bodies, full URLs, or credentials. Return a structured service result/domain exception—not HTTPException—so the API can still return cached metadata and authoritative FolderFreshnessOut. Ensure task return status differentiates complete success from incomplete warning without placing credentials or provider item names in broker/task results.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- no unconditional `refresh_state="fresh"` remains after adapter.list_folder in API or worker
|
||||
- both paths use one service-level listing completion gate
|
||||
- incomplete results expose controlled warning state and retain the previous successful timestamp
|
||||
- complete empty results are accepted as fresh and reconcile correctly
|
||||
- task arguments/results and API schemas contain no credentials or raw provider details
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_cloud_items.py tests/test_cloud.py tests/test_cloud_security.py && ! rg -U 'listing = await adapter\.list_folder[\s\S]{0,800}refresh_state="fresh"' api/cloud/browse.py tasks/cloud_tasks.py</automated></verify>
|
||||
<done>Synchronous and background refreshes share one authoritative completion gate and cannot report an incomplete listing as fresh.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: Document, security-review, commit, and push Plan 02</name>
|
||||
<files>backend/main.py, frontend/package.json, frontend/package-lock.json, AGENTS.md, README.md, RUNBOOK.md, SECURITY.md, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-02-SUMMARY.md</files>
|
||||
<action>Update required documentation for truthful freshness and the shared listing-application service. Bump the backend/frontend patch version once because synchronization status is user-visible, keeping package-lock metadata synchronized. Run a dedicated security agent over this plan's diff, resolve or escalate every finding, then run focused/full backend tests, Bandit, pip audit, Compose validation, and diff checks. Stage only Plan 02 files, exclude `.env` and unrelated work, commit atomically as `fix(12.1): make cloud freshness authoritative`, and push immediately.</action>
|
||||
<verify><automated>docker compose config --quiet && cd backend && pytest -v && bandit -r . && pip audit && cd .. && git diff --check</automated></verify>
|
||||
<done>Plan 02 has independent documentation, security approval, green gates, commit, and push.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
| Threat ID | Threat | Mitigation and evidence |
|
||||
|-----------|--------|-------------------------|
|
||||
| T-12.1-06 | False fresh state hides provider failure | Shared completion gate; complete=False warning tests in API and worker |
|
||||
| T-12.1-07 | Incomplete listing deletes cached metadata | Reconcile deletion remains guarded by complete=True |
|
||||
| T-12.1-08 | Error overwrites last known-good evidence | Prior timestamp preserved on incomplete/error refresh |
|
||||
| T-12.1-09 | Provider errors leak secrets or URLs | Stable codes and controlled messages only |
|
||||
| T-12.1-10 | Refresh changes bytes/quota | Existing and expanded no-byte/no-MinIO/no-quota assertions |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
1. Demonstrate the new incomplete tests fail before implementation.
|
||||
2. Run service tests for complete, complete-empty, partial, incomplete-empty, and exception paths.
|
||||
3. Run connection-ID API and Celery task tests for the same state transitions.
|
||||
4. Run owner/admin/credential/no-byte/no-quota security negatives.
|
||||
5. Search proves callers no longer set fresh independently after list_folder.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Backend freshness means the provider supplied a complete authoritative listing.
|
||||
- Cached metadata and last successful refresh evidence survive ambiguity/failure.
|
||||
- Complete empty folders remain valid and reconcile correctly.
|
||||
- Shared architecture and security boundaries remain intact.
|
||||
</success_criteria>
|
||||
|
||||
<output>Create `.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-02-SUMMARY.md` when complete.</output>
|
||||
@@ -0,0 +1,205 @@
|
||||
---
|
||||
phase: "12.1"
|
||||
plan: "03"
|
||||
type: tdd
|
||||
wave: 3
|
||||
depends_on:
|
||||
- "12.1-02"
|
||||
files_modified:
|
||||
- frontend/src/views/CloudFolderView.vue
|
||||
- frontend/src/stores/cloudConnections.js
|
||||
- frontend/src/components/cloud/CloudProviderTreeItem.vue
|
||||
- frontend/src/components/cloud/CloudFolderTreeItem.vue
|
||||
- frontend/src/components/storage/StorageBrowser.vue
|
||||
- frontend/src/views/__tests__/CloudFolderView.test.js
|
||||
- frontend/src/stores/__tests__/cloudConnections.test.js
|
||||
- frontend/src/components/cloud/__tests__/CloudProviderTreeItem.test.js
|
||||
- frontend/src/components/cloud/__tests__/CloudFolderTreeItem.test.js
|
||||
- frontend/src/components/storage/__tests__/StorageBrowser.skeleton.test.js
|
||||
- frontend/src/components/cloud/__tests__/CloudBreadcrumbNavigation.test.js
|
||||
- backend/main.py
|
||||
- frontend/package.json
|
||||
- frontend/package-lock.json
|
||||
- AGENTS.md
|
||||
- README.md
|
||||
- RUNBOOK.md
|
||||
- SECURITY.md
|
||||
autonomous: true
|
||||
requirements:
|
||||
- CLOUD-01
|
||||
- CONN-04
|
||||
- SYNC-01
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Cloud UI classifies items only with kind and never relies on the absent is_dir field"
|
||||
- "Folder navigation sends provider_item_id while stable DocuVault id remains the row key/metadata identity"
|
||||
- "The frontend renders response.freshness and never manufactures fresh state or a browser-clock refresh timestamp after HTTP 200"
|
||||
- "Cached rows remain visible and navigable while freshness is refreshing or warning"
|
||||
- "Opaque provider references, including slashes, question marks, hashes, percent signs, spaces, and Unicode, are transported without path interpretation"
|
||||
- "Breadcrumbs come from explicit navigation lineage and never split provider_item_id into path segments"
|
||||
- "CloudFolderView remains a thin data provider and StorageBrowser remains the single file browser"
|
||||
artifacts:
|
||||
- path: "frontend/src/views/CloudFolderView.vue"
|
||||
provides: "Normalized item split, provider-reference navigation, and server freshness mapping"
|
||||
contains: "provider_item_id"
|
||||
- path: "frontend/src/views/__tests__/CloudFolderView.test.js"
|
||||
provides: "kind/provider_item_id/truthful-freshness regressions"
|
||||
contains: "warning"
|
||||
- path: "frontend/src/components/cloud/__tests__/CloudFolderTreeItem.test.js"
|
||||
provides: "Nested folder kind and provider reference tests"
|
||||
contains: "provider_item_id"
|
||||
key_links:
|
||||
- from: "CloudBrowseResponse.items[].kind"
|
||||
to: "StorageBrowser folders/files props"
|
||||
via: "CloudFolderView computed classification"
|
||||
pattern: "kind === 'folder'"
|
||||
- from: "folder click/tree expansion"
|
||||
to: "GET /api/cloud/connections/{id}/items?parent_ref="
|
||||
via: "provider_item_id, never CloudItem.id"
|
||||
pattern: "provider_item_id"
|
||||
- from: "CloudBrowseResponse.freshness"
|
||||
to: "StorageBrowser/BreadcrumbBar freshness props"
|
||||
via: "Pinia browse state"
|
||||
pattern: "refresh_state|last_refreshed_at"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Align the cloud frontend with the normalized API. Use `kind` for classification, `provider_item_id` for provider navigation, stable `id` for DocuVault row identity, and backend freshness as the only source of synchronization truth.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-PATTERNS.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-03-PLAN.md
|
||||
@frontend/src/views/CloudFolderView.vue
|
||||
@frontend/src/stores/cloudConnections.js
|
||||
@frontend/src/components/storage/StorageBrowser.vue
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Add red normalized-item and navigation regressions</name>
|
||||
<files>frontend/src/views/__tests__/CloudFolderView.test.js, frontend/src/components/cloud/__tests__/CloudProviderTreeItem.test.js, frontend/src/components/cloud/__tests__/CloudFolderTreeItem.test.js, frontend/src/components/storage/__tests__/StorageBrowser.skeleton.test.js, frontend/src/components/cloud/__tests__/CloudBreadcrumbNavigation.test.js</files>
|
||||
<read_first>
|
||||
- frontend/src/views/CloudFolderView.vue — current is_dir filtering, item.id navigation, and fabricated freshness
|
||||
- frontend/src/components/cloud/CloudProviderTreeItem.vue — root folder filtering
|
||||
- frontend/src/components/cloud/CloudFolderTreeItem.vue — nested filtering and navigation
|
||||
- frontend/src/components/storage/StorageBrowser.vue — shared row keys/events and freshness props
|
||||
- frontend/src/components/cloud/__tests__/CloudBreadcrumbNavigation.test.js — explicit lineage and opaque-reference breadcrumb regressions
|
||||
- backend/api/cloud/schemas.py — canonical response shape
|
||||
</read_first>
|
||||
<action>
|
||||
Replace legacy-shaped fixtures with realistic CloudItemOut objects containing distinct `id` and `provider_item_id`, `kind`, nullable metadata, and no `is_dir`. Add tests named `renders_kind_folder_and_file_in_shared_browser`, `folder_click_uses_provider_item_id_not_id`, `tree_expansion_filters_kind_folder`, `nested_tree_uses_provider_item_id`, and `stable_docuvault_id_remains_row_key`. Include duplicate display names with distinct IDs/references so tests cannot pass accidentally by name.
|
||||
|
||||
Add opaque-reference fixtures containing `/`, `?`, `#`, `%`, spaces, and Unicode. Assert CloudFolderView only passes props/emits events through StorageBrowser and does not add another grid/tree implementation. Verify root sentinel remains `parent_ref=null` at the API boundary and nested folders pass their opaque provider reference unchanged through named Vue Router navigation and the API client's query serializer—never manual route-string interpolation.
|
||||
|
||||
Add breadcrumb tests proving labels and navigation refs come from an explicit session-only lineage of visited `{name, provider_item_id}` folder nodes (or equivalent API-provided ancestors), not `split('/')` or any other parsing of provider references. Breadcrumb clicks must restore the exact stored opaque reference.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- tests fail against current is_dir and item.id behavior
|
||||
- fixtures omit is_dir and distinguish id from provider_item_id
|
||||
- root and nested provider navigation are both covered
|
||||
- StorageBrowser remains the rendered browser component
|
||||
- no test relies on provider-specific path reconstruction in Vue
|
||||
- reserved-character provider references survive route/query transport and breadcrumb navigation unchanged
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd frontend && npm test -- --run src/views/__tests__/CloudFolderView.test.js src/components/cloud/__tests__/CloudProviderTreeItem.test.js src/components/cloud/__tests__/CloudFolderTreeItem.test.js src/components/storage/__tests__/StorageBrowser.skeleton.test.js src/components/cloud/__tests__/CloudBreadcrumbNavigation.test.js</automated></verify>
|
||||
<done>Red frontend tests reproduce invisible folders and wrong-reference navigation using the actual API shape.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Make backend freshness authoritative in Pinia and the view</name>
|
||||
<files>frontend/src/stores/cloudConnections.js, frontend/src/views/CloudFolderView.vue, frontend/src/stores/__tests__/cloudConnections.test.js, frontend/src/views/__tests__/CloudFolderView.test.js</files>
|
||||
<read_first>
|
||||
- frontend/src/stores/cloudConnections.js — folderFreshness and lastRefreshedAt state API
|
||||
- frontend/src/views/CloudFolderView.vue — unconditional fresh/browser-clock assignment
|
||||
- frontend/src/components/ui/BreadcrumbBar.vue — expected refreshing/fresh/warning values and timestamp rendering
|
||||
- frontend/src/views/__tests__/CloudFolderView.test.js — existing fetch/error behavior
|
||||
- frontend/src/stores/__tests__/cloudConnections.test.js — store reset/persistence conventions
|
||||
</read_first>
|
||||
<action>
|
||||
Add tests `maps_server_refreshing_freshness`, `maps_server_warning_and_last_success`, `http_200_does_not_imply_fresh`, `does_not_use_browser_clock_as_refresh_evidence`, and `cached_items_remain_visible_during_warning`. Map `response.freshness.refresh_state`, `last_refreshed_at`, `error_code`, and controlled `error_message` into Pinia. Keep a request-in-flight UI state only if it cannot overwrite the last authoritative server state; after the response, render the server object verbatim. A successful cached browse response is not proof of provider synchronization.
|
||||
|
||||
Remove `new Date().toISOString()` and any unconditional `fresh` assignment from CloudFolderView. On request failure without a server freshness payload, preserve cached items and prior last successful timestamp while showing a controlled client transport warning; never invent a provider refresh time.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- HTTP 200 plus warning freshness renders warning, not fresh
|
||||
- server last_refreshed_at is preserved exactly and browser Date is not called to create evidence
|
||||
- error code/message remain controlled display data and do not affect item ownership/navigation
|
||||
- cached items remain usable during refreshing/warning states
|
||||
- store reset clears cloud browse state without persisting secrets
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd frontend && npm test -- --run src/stores/__tests__/cloudConnections.test.js src/views/__tests__/CloudFolderView.test.js</automated></verify>
|
||||
<done>The UI reports backend freshness truthfully and preserves cached navigation through warnings.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: Implement normalized item use across cloud view and trees</name>
|
||||
<files>frontend/src/views/CloudFolderView.vue, frontend/src/components/cloud/CloudProviderTreeItem.vue, frontend/src/components/cloud/CloudFolderTreeItem.vue, frontend/src/components/storage/StorageBrowser.vue, frontend/src/views/__tests__/CloudFolderView.test.js, frontend/src/components/cloud/__tests__/CloudProviderTreeItem.test.js, frontend/src/components/cloud/__tests__/CloudFolderTreeItem.test.js, frontend/src/components/cloud/__tests__/CloudBreadcrumbNavigation.test.js</files>
|
||||
<read_first>
|
||||
- frontend/src/components/ui/TreeItem.vue — shared expand/collapse state contract
|
||||
- frontend/src/components/storage/StorageBrowser.vue — single browser row/event contract
|
||||
- frontend/src/views/CloudFolderView.vue — thin view boundary
|
||||
- frontend/src/components/cloud/CloudProviderTreeItem.vue — root folder source
|
||||
- frontend/src/components/cloud/CloudFolderTreeItem.vue — nested folder source
|
||||
</read_first>
|
||||
<action>
|
||||
Replace all cloud `is_dir` checks with `item.kind === "folder"` and file checks with `item.kind === "file"`. Route and browse nested folders with `provider_item_id`; retain `id` for Vue keys, selection, and DocuVault metadata identity. Do not add an alias field, provider-specific mapper, alternate browser, or duplicate expand/collapse state. Keep CloudProviderTreeItem and CloudFolderTreeItem wrapped by TreeItem.vue.
|
||||
|
||||
Use named-route objects for connection routing and keep `parent_ref` in serialized query/API state; do not interpolate it into a route path. Ensure folder references are encoded only by Vue Router/API query serializers, not manually concatenated or decoded. Maintain explicit session-only breadcrumb lineage from root to visited folders; never reconstruct hierarchy by splitting `provider_item_id`, because Drive/OneDrive IDs and DAV references are opaque. Preserve connection UUID routing from Phase 12. Remove dead compatibility logic in the same change.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- no active cloud component references item.is_dir
|
||||
- every provider browse/navigation call uses provider_item_id for nested parent_ref
|
||||
- stable id remains the rendering/metadata identity and is never sent to a provider adapter
|
||||
- CloudFolderView contains no grid/layout duplication
|
||||
- focused and full cloud frontend suites pass
|
||||
- opaque IDs containing reserved characters and breadcrumb back-navigation round-trip exactly
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd frontend && npm test -- --run src/views/__tests__/CloudFolderView.test.js src/components/cloud/__tests__/CloudProviderTreeItem.test.js src/components/cloud/__tests__/CloudFolderTreeItem.test.js src/components/storage/__tests__/StorageBrowser.skeleton.test.js src/stores/__tests__/cloudConnections.test.js src/components/cloud/__tests__/CloudBreadcrumbNavigation.test.js && ! rg '\.is_dir' src/views/CloudFolderView.vue src/components/cloud</automated></verify>
|
||||
<done>Cloud folders render and navigate with the canonical API fields while shared component boundaries remain intact.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4: Document, security-review, commit, and push Plan 03</name>
|
||||
<files>AGENTS.md, README.md, RUNBOOK.md, SECURITY.md, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-03-SUMMARY.md</files>
|
||||
<action>Update required docs for normalized cloud UI routing, breadcrumb lineage, and server freshness. Bump the backend/frontend patch version once and synchronize package-lock metadata because the plan changes user-visible behavior. Run a dedicated security agent against this plan's diff, resolving or escalating findings. Run focused/full frontend tests, production build, npm audit, Compose validation, and diff checks. Stage only Plan 03 files, exclude `.env` and unrelated work, commit atomically as `fix(12.1): align cloud browser contract`, and push immediately.</action>
|
||||
<verify><automated>docker compose config --quiet && cd frontend && npm test && npm run build && npm audit --audit-level=high && cd .. && git diff --check</automated></verify>
|
||||
<done>Plan 03 has independent documentation, security approval, green gates, commit, and push.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
| Threat ID | Threat | Mitigation and evidence |
|
||||
|-----------|--------|-------------------------|
|
||||
| T-12.1-11 | DocuVault UUID sent as provider path | Distinct-ID fixtures and provider_item_id navigation assertions |
|
||||
| T-12.1-12 | UI falsely reports provider synchronization | Server freshness mapping; no browser-clock evidence |
|
||||
| T-12.1-13 | Provider reference injected into provider-specific URL | API query serialization; adapters own provider translation/validation |
|
||||
| T-12.1-14 | Secrets persisted in frontend state | Existing memory/session rules; tests store folder reference only |
|
||||
| T-12.1-15 | Parallel browser/tree logic drifts | StorageBrowser and TreeItem remain canonical shared components |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
1. Demonstrate actual API-shaped fixtures fail current is_dir/item.id code.
|
||||
2. Verify kind-based root and nested classification.
|
||||
3. Verify provider_item_id navigation while id remains row identity.
|
||||
4. Verify warning/refreshing/fresh server payloads and exact timestamps.
|
||||
5. Search for legacy is_dir usage in active cloud components and run the focused frontend suite.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Users can see folders/files returned by every normalized provider adapter.
|
||||
- Folder clicks and tree expansion use valid provider references.
|
||||
- Synchronization wording reflects backend state rather than HTTP status or client time.
|
||||
- Shared StorageBrowser/TreeItem architecture remains unchanged.
|
||||
</success_criteria>
|
||||
|
||||
<output>Create `.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-03-SUMMARY.md` when complete.</output>
|
||||
@@ -0,0 +1,226 @@
|
||||
---
|
||||
phase: "12.1"
|
||||
plan: "04"
|
||||
type: execute
|
||||
wave: 4
|
||||
depends_on:
|
||||
- "12.1-01"
|
||||
- "12.1-02"
|
||||
- "12.1-03"
|
||||
files_modified:
|
||||
- backend/tests/test_nextcloud_live.py
|
||||
- backend/tests/fixtures/cloud/nextcloud_expected_root.json
|
||||
- backend/pytest.ini
|
||||
- frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js
|
||||
- .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-VALIDATION.md
|
||||
- backend/main.py
|
||||
- frontend/package.json
|
||||
- frontend/package-lock.json
|
||||
- AGENTS.md
|
||||
- README.md
|
||||
- RUNBOOK.md
|
||||
- SECURITY.md
|
||||
autonomous: false
|
||||
requirements:
|
||||
- CONN-04
|
||||
- CLOUD-01
|
||||
- CACHE-01
|
||||
- SYNC-01
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "An opt-in read-only live test browses the configured Nextcloud root through the owner-scoped connection-ID DocuVault API as testuser@docuvault.example"
|
||||
- "Live execution never prints secrets, the full URL, or unexpected provider names and never downloads/mutates provider data"
|
||||
- "Live validation has a nonblocking sanitized diagnostic mode and a separate exact acceptance mode enabled only by an owner-confirmed tracked manifest"
|
||||
- "Mocked and live evidence together prove folders/files, kind, provider identity, freshness, zero-byte, and owner isolation"
|
||||
- "Full tests, security/dependency/secret gates, docs, version, atomic commit, and push complete before Phase 12.1 closes"
|
||||
artifacts:
|
||||
- path: "backend/tests/test_nextcloud_live.py"
|
||||
provides: "Opt-in sanitized live Nextcloud adapter and connection-ID API smoke"
|
||||
contains: "test_nextcloud_root_diagnostic"
|
||||
- path: "backend/tests/fixtures/cloud/nextcloud_expected_root.json"
|
||||
provides: "Owner-confirmed, non-secret exact root name/kind acceptance manifest"
|
||||
contains: "owner_confirmed"
|
||||
- path: ".planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-VALIDATION.md"
|
||||
provides: "Nyquist matrix and live/mock/security evidence"
|
||||
contains: "7 of 10"
|
||||
- path: "SECURITY.md"
|
||||
provides: "Phase 12.1 threat and gate evidence without secrets/provider content"
|
||||
contains: "Phase 12.1"
|
||||
key_links:
|
||||
- from: "NEXTCLOUD_URL/NEXTCLOUD_USER/NEXTCLOUD_APP_PASSWORD"
|
||||
to: "backend/tests/test_nextcloud_live.py"
|
||||
via: "ignored environment read in process memory only"
|
||||
pattern: 'os\.environ|getenv'
|
||||
- from: "testuser@docuvault.example"
|
||||
to: "GET /api/cloud/connections/{connection_id}/items"
|
||||
via: "isolated test DB user, encrypted connection, authenticated API request"
|
||||
pattern: 'testuser@docuvault\.example'
|
||||
- from: "supplied expected manifest"
|
||||
to: "exact live signoff"
|
||||
via: "explicit discrepancy reconciliation gate"
|
||||
pattern: 'Documents|Reasons to use Nextcloud\.pdf'
|
||||
---
|
||||
|
||||
<objective>
|
||||
Prove the repaired flow against the designated test account without risking credentials or provider data, reconcile the known manifest discrepancy before exact-name acceptance, then close Phase 12.1 with full validation, security, documentation, versioning, commit, and push protocol.
|
||||
|
||||
The live test is intentionally opt-in and read-only. Missing credentials skip with a prerequisite message; a manifest discrepancy blocks exact-name signoff rather than mutating the provider or silently changing expectations.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md
|
||||
@.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-PATTERNS.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-SECURITY.md
|
||||
@.planning/phases/12-cloud-resource-foundation/12-VERIFICATION.md
|
||||
@AGENTS.md
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Build and run the nonblocking sanitized live diagnostic</name>
|
||||
<files>backend/tests/test_nextcloud_live.py, backend/pytest.ini, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-VALIDATION.md</files>
|
||||
<read_first>
|
||||
- .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-CONTEXT.md — credentials, user, expected manifest, and read-only rules
|
||||
- .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md — sanitized 207/10/7-of-10 probe evidence
|
||||
- backend/tests/conftest.py — isolated DB/auth fixture conventions
|
||||
- backend/tests/test_cloud.py — connection creation and connection-ID browse patterns
|
||||
- backend/tests/test_cloud_security.py — owner/admin/credential/no-byte assertions
|
||||
- backend/storage/cloud_utils.py — credential encryption and URL validation
|
||||
</read_first>
|
||||
<action>
|
||||
Add a `live_nextcloud` pytest marker excluded from ordinary test runs unless explicitly selected. Read only `NEXTCLOUD_URL`, `NEXTCLOUD_USER`, and `NEXTCLOUD_APP_PASSWORD` from the ignored environment at runtime; if any are absent, skip with a generic prerequisite message naming variable keys only. Never load or print `.env`, and never place values in parametrization IDs, assertion diffs, exceptions, snapshots, logs, artifacts, or command lines.
|
||||
|
||||
Create `test_nextcloud_adapter_read_only_root_metadata`, `test_nextcloud_root_diagnostic`, and an authenticated `test_nextcloud_connection_id_browse_as_designated_user`. In an isolated test database, provision/resolve the regular DocuVault user exactly `testuser@docuvault.example`, encrypt the live connection credentials through production helpers, then browse root through `GET /api/cloud/connections/{connection_id}/items`; also assert foreign-user/admin denial. The test may issue metadata-only DAV requests required for listing. Install a request guard that rejects byte GET/media and PUT/POST/PATCH/MKCOL/MOVE/COPY/DELETE methods before network dispatch; assert quota and MinIO/object storage are unchanged.
|
||||
|
||||
In diagnostic mode compare in memory against the ten owner-supplied candidate names, but do not assert exact equality and do not claim acceptance. Emit only total count, expected-match count, missing supplied names, unexpected count, completion state, and kind counts; never emit unexpected names. The diagnostic must exit successfully when transport/security/read-only invariants pass even if the candidate manifest is 7/10, while recording `manifest_status: unconfirmed`. Catch/redact transport errors into stable codes so neither full URL nor credentials enter pytest output. Assert captured logs/output contain none of the secret values or full URL.
|
||||
|
||||
Record the contract, commands, skip behavior, sanitized result fields, and threat references in 12.1-VALIDATION.md. Do not record unexpected live names.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- ordinary pytest does not contact Nextcloud and missing variables produce an explicit safe skip
|
||||
- selected live tests authenticate only as the designated regular app user and use connection-ID browse
|
||||
- network guard permits metadata listing but rejects download and every mutation method
|
||||
- output contains no credential, full URL, raw response body, or unexpected provider name
|
||||
- manifest mismatch is diagnostic-only and nonblocking before owner confirmation
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_nextcloud_live.py -m live_nextcloud</automated></verify>
|
||||
<done>The opt-in diagnostic safely proves connectivity/listing invariants and reports sanitized drift without claiming exact acceptance.</done>
|
||||
</task>
|
||||
|
||||
<task type="checkpoint:human-verify" gate="blocking">
|
||||
<name>Task 2: Reconcile the known 7-of-10 fixture discrepancy before exact signoff</name>
|
||||
<files>backend/tests/fixtures/cloud/nextcloud_expected_root.json, backend/tests/test_nextcloud_live.py, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-VALIDATION.md</files>
|
||||
<read_first>
|
||||
- .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md — three supplied names not exactly matched in sanitized probe
|
||||
- backend/tests/test_nextcloud_live.py — supplied expected manifest and safe mismatch report
|
||||
</read_first>
|
||||
<action>
|
||||
Run the live smoke and inspect only its sanitized count/missing-expected output. The prior probe found 10 children but only 7 exact supplied-name matches; therefore do not declare the ten-name assertion accepted merely because item count is 10. If the same mismatch remains, stop and ask the project owner to compare the three supplied expected names (`Manual Nextcloud.png`, `Nextcloud Intro.mp4`, `Template credits.md`) with the Nextcloud UI and state which source is authoritative. Do not reveal the three unexpected provider names and do not rename/delete/upload anything.
|
||||
|
||||
After explicit owner confirmation, store the authoritative names and kinds in tracked `backend/tests/fixtures/cloud/nextcloud_expected_root.json` with no URL, username, credential, provider ID, timestamp, or unexpected-name discovery data; include `owner_confirmed: true` and a short non-secret provenance note. Only then add/enable `test_nextcloud_expected_root_manifest`, loading that fixture. If the original supplied expectation is authoritative, leave its names unchanged and treat the provider account as requiring external fixture correction. Re-run until exact set equality and exact kind equality pass. Record the owner-confirmed disposition and only sanitized counts in 12.1-VALIDATION.md.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- exact-name signoff is blocked while only 7/10 supplied names match
|
||||
- no unexpected provider names are printed or persisted during reconciliation
|
||||
- any expected-manifest change has explicit owner confirmation, not inference from live output
|
||||
- final live result is exact set and kind equality, not count-only acceptance
|
||||
- provider content remains unchanged
|
||||
</acceptance_criteria>
|
||||
<verify><automated>cd backend && pytest -q tests/test_nextcloud_live.py -m live_nextcloud -k expected_root_manifest</automated></verify>
|
||||
<done>The owner has reconciled the fixture discrepancy and the live account passes exact root-name and kind acceptance without provider mutation.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: Run full regression, security, environment, and rendered-flow gates</name>
|
||||
<files>frontend/src/views/__tests__/CloudFolderRenderedFlow.test.js, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-VALIDATION.md, SECURITY.md, RUNBOOK.md</files>
|
||||
<read_first>
|
||||
- AGENTS.md — mandatory tests, security checks, environment contract, and deployment exclusions
|
||||
- .planning/phases/12-cloud-resource-foundation/12-VALIDATION.md — baseline Nyquist matrix and commands
|
||||
- .planning/phases/12-cloud-resource-foundation/12-SECURITY.md — inherited threat evidence
|
||||
- SECURITY.md — project security evidence format
|
||||
- RUNBOOK.md — operational cloud/migration guidance
|
||||
</read_first>
|
||||
<action>
|
||||
Add an executable Vue Test Utils rendered-flow test `CloudFolderRenderedFlow.test.js` that mounts the real CloudFolderView with StorageBrowser/BreadcrumbBar (stubbing only network/router boundaries), renders mixed root folders/files, clicks a folder whose opaque reference contains reserved characters, navigates back via lineage breadcrumbs, and renders refreshing/warning/fresh server states while cached rows remain visible. Run it in the normal Vitest command. Use the live account only for the separate sanitized API smoke; do not place provider names or credentials in rendered snapshots.
|
||||
|
||||
Run `bandit -r backend/`, `pip audit`, `npm audit --audit-level=high`, security-invariant tests, and `gitleaks detect --source . --no-banner --redact` as the concrete repository/history secret scanner (install the pinned approved binary before execution if absent; absence blocks the gate). Also run `git ls-files '.env' '.env.*'` and require only approved example files. Verify no new high/critical findings, no tracked `.env`, no secret values in current history/diff, no raw provider URL/error leakage, no admin/foreign-user access, and no provider byte/mutation/quota activity. Resolve introduced findings without skips, suppressions, or broad exception swallowing. Update validation/security/runbook evidence with commands and sanitized outcomes only.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- four-provider focused contract, live exact acceptance, full suites, and build pass
|
||||
- Compose config resolves every required variable non-empty without printing values
|
||||
- Bandit and pip/npm audits have zero introduced high/critical findings
|
||||
- secret scan and git diff contain no credentials/full live URL
|
||||
- rendered root/nested/freshness behavior matches API truth and uses StorageBrowser
|
||||
</acceptance_criteria>
|
||||
<verify><automated>docker compose config --quiet && gitleaks detect --source . --no-banner --redact && test -z "$(git ls-files '.env' '.env.*' | grep -v '^\.env\.example$')" && cd backend && pytest -q tests/test_cloud_provider_contract.py tests/test_cloud_backends.py tests/test_webdav_backend.py tests/test_cloud_items.py tests/test_cloud.py tests/test_cloud_security.py && pytest -q tests/test_nextcloud_live.py -m live_nextcloud && pytest -v && bandit -r . && pip audit && cd ../frontend && npm test -- --run src/views/__tests__/CloudFolderRenderedFlow.test.js && npm test && npm run build && npm audit --audit-level=high</automated></verify>
|
||||
<done>Automated, live, rendered, security, dependency, secret, and environment gates pass with sanitized evidence.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4: Close documentation, version, commit, and push protocol</name>
|
||||
<files>backend/main.py, frontend/package.json, frontend/package-lock.json, AGENTS.md, README.md, RUNBOOK.md, SECURITY.md, .planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-VALIDATION.md</files>
|
||||
<read_first>
|
||||
- backend/main.py — backend version source of truth
|
||||
- frontend/package.json — frontend version source of truth
|
||||
- frontend/package-lock.json — lockfile root/package version entries
|
||||
- AGENTS.md — current state, shared module map, documentation/version/Git protocols
|
||||
- README.md — cloud browse feature and environment documentation
|
||||
- .planning/ROADMAP.md — Phase 12.1 boundary and Phase 13/14 exclusions
|
||||
</read_first>
|
||||
<action>
|
||||
Update AGENTS.md current state to Phase 12.1 complete and add any new canonical listing/freshness helper to the backend shared-module map with a no-duplication rule. Update README with truthful provider-neutral browse behavior and opt-in live test variable names only; never add values. Update RUNBOOK with safe live-smoke invocation, redaction expectations, warning semantics, and fixture-reconciliation procedure. Update SECURITY.md and 12.1-VALIDATION.md with final evidence and inherited controls.
|
||||
|
||||
Increment the current backend/frontend patch version exactly once for Plan 04 in backend/main.py, frontend/package.json, and both relevant package-lock entries; earlier plans have already performed their own required per-plan bumps. Use source values at execution time and keep all versions identical. Do not claim Phase 13 mutations or Phase 14 byte cache behavior.
|
||||
|
||||
Run a dedicated security agent on Plan 04's diff and resolve or escalate every finding. Re-run focused/full tests, rendered test, gitleaks, build, security gates, and `docker compose config --quiet` after docs/version changes. Review `git diff --check`, `git status`, and staged diff for secrets/unrelated user changes. Stage only Plan 04 live-test/fixture/rendered-test/planning/docs/version files explicitly, commit atomically as `test(12.1): verify live cloud browse integration`, and push immediately. Never amend or accumulate Plans 01-03, stage `.env`, or overwrite unrelated work.
|
||||
</action>
|
||||
<acceptance_criteria>
|
||||
- AGENTS/README/RUNBOOK/SECURITY describe shipped behavior and explicit Phase 13/14 exclusions
|
||||
- documentation names environment keys only and contains no credential, full URL, or unexpected live provider name
|
||||
- backend/package/lock versions match after one patch bump
|
||||
- final test/build/security/config gates remain green
|
||||
- Plan 04 has its own atomic commit and push without unrelated files or `.env`; Plans 01-03 remain separate commits
|
||||
</acceptance_criteria>
|
||||
<verify><automated>docker compose config --quiet && git diff --check && cd backend && pytest -v && cd ../frontend && npm test && npm run build && npm audit --audit-level=high</automated></verify>
|
||||
<done>Phase 12.1 is documented, versioned, fully verified, atomically committed, and pushed with no secret or scope leakage.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
| Threat ID | Threat | Mitigation and evidence |
|
||||
|-----------|--------|-------------------------|
|
||||
| T-12.1-16 | Live credentials/full URL leak through pytest/logs | Runtime env only, generic failures, capture assertions, no values in IDs/commands/artifacts |
|
||||
| T-12.1-17 | Live smoke mutates/downloads user data | Network method guard; metadata-only requests; no-byte/no-quota assertions |
|
||||
| T-12.1-18 | Live connection crosses user/admin boundary | Designated regular user plus foreign/admin negatives through connection-ID API |
|
||||
| T-12.1-19 | Count-only test accepts wrong root | Exact set and kind equality; blocking 7/10 reconciliation checkpoint |
|
||||
| T-12.1-20 | Unexpected provider names become persisted sensitive metadata | Unexpected count only; no names in output/docs/fixtures |
|
||||
| T-12.1-SC | Supply-chain or committed-secret regression | Bandit, pip/npm audits, secret scan, staged diff review before commit/push |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
1. Ordinary tests skip live access safely; selected live tests use only the three named environment keys.
|
||||
2. Live adapter and connection-ID API smoke pass as the designated user with exact names/kinds after explicit discrepancy reconciliation.
|
||||
3. Network guard and DB assertions prove no provider mutation, download, MinIO, or quota change.
|
||||
4. Shared four-provider, backend/frontend full, build, security, audit, secret, and Compose gates pass.
|
||||
5. Docs/version are synchronized; atomic commit and push succeed without `.env` or unrelated changes.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- The connected Nextcloud root is visible through DocuVault and matches an owner-confirmed exact manifest.
|
||||
- The test design validates all providers through one contract while preserving provider-specific fixtures and pagination.
|
||||
- Freshness is truthful from adapter completion through API, store, and rendered browser.
|
||||
- Credentials/provider data remain private and provider content remains untouched.
|
||||
- Phase 12.1 closes without importing Phase 13 mutations or Phase 14 byte caching.
|
||||
</success_criteria>
|
||||
|
||||
<output>Create `.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-04-SUMMARY.md` when complete, then update roadmap/state through the normal GSD phase-close workflow.</output>
|
||||
@@ -0,0 +1,90 @@
|
||||
# Phase 12.1: Fix Nextcloud Root Listing and Sync Visibility - Context
|
||||
|
||||
**Gathered:** 2026-06-22
|
||||
**Status:** Ready for planning
|
||||
**Source:** User-reported integration failure and test-account manifest
|
||||
|
||||
<domain>
|
||||
## Phase Boundary
|
||||
|
||||
Diagnose and fix the cloud browse path that can report a successful/fresh synchronization while showing no provider files or folders. Prove the repair against the configured Nextcloud test account and add provider-neutral listing contracts for Nextcloud, generic WebDAV, Google Drive, and OneDrive. This phase repairs browse/list correctness and truthful freshness; cloud mutations remain Phase 13 scope.
|
||||
|
||||
</domain>
|
||||
|
||||
<decisions>
|
||||
## Implementation Decisions
|
||||
|
||||
### Live Nextcloud acceptance contract
|
||||
- D-01: The ignored repository `.env` provides the opt-in live-test variables `NEXTCLOUD_URL`, `NEXTCLOUD_USER`, and `NEXTCLOUD_APP_PASSWORD`; credentials must never be printed, logged, committed, placed in test IDs, or copied into planning artifacts.
|
||||
- D-02: The DocuVault application identity used for the live integration flow is `testuser@docuvault.example`.
|
||||
- D-03: A successful root browse must expose exactly these expected entry names, with folder/file kind checked separately: `Documents`, `docuvault`, `Photos`, `Templates`, `Nextcloud.png`, `Nextcloud Intro.mp4`, `Manual Nextcloud.png`, `Readme.md`, `Reasons to use Nextcloud.pdf`, `Template credits.md`.
|
||||
- D-04: Live provider tests are opt-in, read-only, and skip with an explicit prerequisite message when credentials are absent. They may list metadata but must not upload, create, rename, move, delete, download bytes, or change quota.
|
||||
|
||||
### Provider-wide correctness
|
||||
- D-05: The fix must establish one provider-neutral listing correctness contract for Nextcloud, generic WebDAV, Google Drive, and OneDrive rather than adding a Nextcloud-only API or UI path.
|
||||
- D-06: Every adapter contract test must prove normalized child identity, parent reference, name, file/folder kind, metadata completeness, pagination/completeness semantics, and zero byte-download/mutation calls.
|
||||
- D-07: A refresh may transition to `fresh` only after a valid listing has been normalized and reconciled. Empty listings are valid only when the provider positively returned an empty collection; parser/transport/root-resolution ambiguity must become a controlled warning/error, not “synced”.
|
||||
- D-08: Cached metadata remains visible during refresh and after provider failure. A failed or ambiguous refresh must not delete cached children or claim a successful empty root.
|
||||
|
||||
### Security and architecture
|
||||
- D-09: Preserve owner and connection scoping at every API/service boundary; the live connection belongs only to the designated regular test user and admin access remains blocked.
|
||||
- D-10: Provider credentials are decrypted only at the provider/task boundary and never enter broker payloads, API responses, browser storage, logs, snapshots, fixtures, or committed files.
|
||||
- D-11: All outbound WebDAV/Nextcloud requests continue through canonical SSRF validation; redirects and resolved addresses must not bypass the allowlist/private-address controls.
|
||||
- D-12: `CloudResourceAdapter`, `reconcile_cloud_listing`, the connection-ID browse API, and `StorageBrowser.vue` remain the shared contracts. No parallel provider-specific browser, reconciliation path, or response schema may be introduced.
|
||||
|
||||
### Claude's Discretion
|
||||
- Exact fixture organization and helper names.
|
||||
- Whether listing validation is represented as a result type, domain exception, or validated `CloudListing` constructor, provided controlled errors and existing shared contracts are preserved.
|
||||
- Whether the live test calls the adapter directly, the refresh task, or both, provided there is at least one end-to-end connection-ID browse assertion through DocuVault.
|
||||
|
||||
</decisions>
|
||||
|
||||
<canonical_refs>
|
||||
## Canonical References
|
||||
|
||||
### Phase 12 architecture and validation
|
||||
- `.planning/phases/12-cloud-resource-foundation/12-CONTEXT.md` — locked cloud resource architecture decisions.
|
||||
- `.planning/phases/12-cloud-resource-foundation/12-VALIDATION.md` — existing provider, browse, security, and no-byte verification map.
|
||||
- `.planning/phases/12-cloud-resource-foundation/12-SECURITY.md` — Phase 12 threat model and security evidence.
|
||||
- `AGENTS.md` — non-negotiable shared-module, testing, security, environment, documentation, and Git rules.
|
||||
|
||||
### Shared cloud implementation
|
||||
- `backend/storage/cloud_base.py` — normalized resource/listing/capability contract.
|
||||
- `backend/storage/nextcloud_backend.py` — Nextcloud listing implementation.
|
||||
- `backend/storage/webdav_backend.py` — generic WebDAV listing implementation.
|
||||
- `backend/storage/google_drive_backend.py` — Google Drive listing implementation.
|
||||
- `backend/storage/onedrive_backend.py` — OneDrive listing implementation.
|
||||
- `backend/services/cloud_items.py` — sole metadata reconciliation entry point.
|
||||
- `backend/tasks/cloud_tasks.py` — background refresh and freshness transitions.
|
||||
- `backend/api/cloud/browse.py` — owner-scoped connection-ID cached-first browse API.
|
||||
|
||||
### Existing verification
|
||||
- `backend/tests/test_cloud_backends.py` — provider normalization contract fixtures.
|
||||
- `backend/tests/test_cloud_items.py` — reconciliation, refresh, retention, and no-byte tests.
|
||||
- `backend/tests/test_cloud.py` — connection and browse API integration tests.
|
||||
- `backend/tests/test_cloud_security.py` — owner/admin/credential/SSRF/no-byte negative suite.
|
||||
|
||||
</canonical_refs>
|
||||
|
||||
<specifics>
|
||||
## Specific Ideas
|
||||
|
||||
- The observed symptom is internally contradictory: the UI says synchronization succeeded, yet the connected Nextcloud root renders no folders or files.
|
||||
- The expected manifest intentionally includes both folders and files, names with spaces, multiple extensions, and mixed media/document types; it should catch root self-entry filtering, href decoding, and kind-normalization defects.
|
||||
- The eventual test output should report only sanitized status, entry counts, and expected/missing/unexpected basenames.
|
||||
|
||||
</specifics>
|
||||
|
||||
<deferred>
|
||||
## Deferred Ideas
|
||||
|
||||
- Upload, create-folder, rename, move, and delete behavior remain Phase 13.
|
||||
- Byte download/cache and document analysis remain Phase 14.
|
||||
- Production monitoring beyond truthful per-folder freshness remains Phase 16.
|
||||
|
||||
</deferred>
|
||||
|
||||
---
|
||||
|
||||
*Phase: 12.1-fix-nextcloud-root-listing-and-sync-visibility*
|
||||
*Context gathered: 2026-06-22*
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
# Phase 12.1 Pattern Map: Cloud Listing Correctness
|
||||
|
||||
## Purpose
|
||||
|
||||
Map the existing shared patterns and exact change seams for making root and nested cloud listings correct across Nextcloud, generic WebDAV, Google Drive, and OneDrive. This is a static repository map only: no `.env` access, credentials, provider calls, or network operations were used.
|
||||
|
||||
The supplied Nextcloud root names are acceptance evidence, not configuration or test secrets:
|
||||
|
||||
- Folders: `Documents`, `docuvault`, `Photos`, `Templates`
|
||||
- Files: `Nextcloud.png`, `Nextcloud Intro.mp4`, `Manual Nextcloud.png`, `Readme.md`, `Reasons to use Nextcloud.pdf`, `Template credits.md`
|
||||
|
||||
## Root-Cause Map
|
||||
|
||||
### 1. Nextcloud breaks the shared adapter contract
|
||||
|
||||
`backend/storage/cloud_base.py` defines the canonical interface:
|
||||
|
||||
```python
|
||||
await adapter.list_folder(connection_id, user_id, parent_ref=None, page_token=None)
|
||||
-> CloudListing
|
||||
```
|
||||
|
||||
`backend/storage/webdav_backend.py` implements that interface and already contains the reusable WebDAV/Nextcloud PROPFIND normalization into `CloudResource` and `CloudListing`.
|
||||
|
||||
`backend/storage/nextcloud_backend.py`, however, overrides `list_folder` with the legacy signature `list_folder(folder_path="") -> list[dict]`. The canonical API and Celery task call it with connection/user UUIDs plus `parent_ref`, so Nextcloud can raise a signature error before listing. It also returns the legacy shape instead of `CloudListing`.
|
||||
|
||||
**Fix pattern:** do not add another Nextcloud listing implementation. Remove the incompatible override and inherit `WebDAVBackend.list_folder`, or move the legacy behavior behind an explicitly named compatibility method used only by the legacy route. The canonical `CloudResourceAdapter.list_folder` must have one signature and one return type for every provider.
|
||||
|
||||
### 2. The frontend consumes the legacy item shape
|
||||
|
||||
`backend/api/cloud/schemas.py` exposes canonical items as:
|
||||
|
||||
```text
|
||||
id DocuVault stable UUID
|
||||
provider_item_id provider folder/file reference
|
||||
kind "folder" or "file"
|
||||
```
|
||||
|
||||
`frontend/src/views/CloudFolderView.vue` still filters by `is_dir` and navigates with `item.id`. Consequently:
|
||||
|
||||
- canonical folders are not recognized as folders;
|
||||
- nested navigation sends the DocuVault metadata UUID as `parent_ref` instead of the provider reference;
|
||||
- this fails for every provider, not just Nextcloud.
|
||||
|
||||
**Fix pattern:** normalize once at the API/view boundary or consume the canonical schema directly. In `CloudFolderView.vue`, derive folders/files from `item.kind`, and navigate using `item.provider_item_id`. Do not teach `StorageBrowser.vue` provider-specific IDs or add provider branches.
|
||||
|
||||
### 3. The UI invents freshness instead of rendering provider state
|
||||
|
||||
`backend/api/cloud/browse.py` returns `freshness` from durable `CloudFolderState`, including `refresh_state`, `last_refreshed_at`, and controlled errors. `frontend/src/views/CloudFolderView.vue` ignores it and sets `freshness: 'fresh'` plus the browser's current time after every HTTP 200.
|
||||
|
||||
An empty listing caused by a provider failure can therefore appear synchronized.
|
||||
|
||||
**Fix pattern:** map `data.freshness.refresh_state` into the store's UI vocabulary and use `data.freshness.last_refreshed_at`; preserve `error_code/error_message` as the user-visible warning source. Never infer provider freshness from HTTP success.
|
||||
|
||||
### 4. Empty complete listings and failed listings must remain distinguishable
|
||||
|
||||
`CloudListing.complete` in `backend/storage/cloud_base.py` is the authoritative distinction:
|
||||
|
||||
- `complete=True`: full listing; reconciliation may soft-delete unseen children.
|
||||
- `complete=False`: partial/failed listing; reconciliation must retain cached children.
|
||||
|
||||
`backend/services/cloud_items.py::reconcile_cloud_listing` already implements this correctly. Keep it as the only metadata reconciliation entry point. Adapter/API fixes must not bypass it or update `CloudItem` rows directly.
|
||||
|
||||
The synchronous first-load path in `backend/api/cloud/browse.py` should only mark a folder fresh after an authoritative result. A `CloudListing(complete=False)` must produce a controlled warning/stale state and must not be reported as fresh. Apply the same decision in `backend/tasks/cloud_tasks.py` so synchronous and background refreshes cannot disagree.
|
||||
|
||||
## Shared Pipeline and Exact Files
|
||||
|
||||
### Adapter contract and normalization
|
||||
|
||||
- `backend/storage/cloud_base.py`
|
||||
- Preserve `CloudResourceAdapter.list_folder` as the sole browse contract.
|
||||
- Preserve normalized `CloudResource` identity fields and `CloudListing.complete` semantics.
|
||||
- If validation is strengthened, add provider-neutral checks here rather than in four adapters (for example, reject duplicate provider IDs within one authoritative page/listing).
|
||||
- `backend/storage/webdav_backend.py`
|
||||
- Keep one DAV listing implementation for both WebDAV and Nextcloud.
|
||||
- Root is `parent_ref=None`, translated internally to the SDK's empty relative path.
|
||||
- Continue returning provider-relative paths as `provider_item_id` and the requested parent reference as `parent_ref`.
|
||||
- Keep SSRF revalidation before outbound calls and the no-byte-download invariant.
|
||||
- `backend/storage/nextcloud_backend.py`
|
||||
- Remove/rename the legacy `list_folder(folder_path)` override so the inherited canonical implementation is used.
|
||||
- Retain only genuinely Nextcloud-specific behavior such as root health-check path conventions.
|
||||
- Do not duplicate the PROPFIND normalization loop.
|
||||
- `backend/storage/google_drive_backend.py`
|
||||
- Preserve root translation `None -> "root"`, provider IDs for navigation, full pagination, and `complete=False` on provider failure.
|
||||
- `backend/storage/onedrive_backend.py`
|
||||
- Preserve root translation to `/me/drive/root/children`, item IDs for navigation, `@odata.nextLink` exhaustion, and incomplete failure semantics.
|
||||
- `backend/storage/cloud_backend_factory.py`
|
||||
- Continue constructing all providers behind `build_cloud_resource_adapter`.
|
||||
- Add/retain a defensive contract assertion, but do not special-case list behavior by provider.
|
||||
|
||||
### Service and task patterns
|
||||
|
||||
- `backend/services/cloud_items.py`
|
||||
- `resolve_owned_connection`: owner boundary before adapter access.
|
||||
- `upsert_cloud_item`: stable metadata identity by `(connection_id, provider_item_id)`.
|
||||
- `reconcile_cloud_listing`: only reconciliation entry point; soft-delete only for complete listings.
|
||||
- `list_cloud_children`: query by the exact canonical `parent_ref` used during reconciliation.
|
||||
- `update_folder_state`: canonical controlled freshness/error state; never raw provider exception text.
|
||||
- `backend/tasks/cloud_tasks.py`
|
||||
- Keep credential decryption inside the worker.
|
||||
- Call only the canonical adapter method.
|
||||
- Treat `complete=False` as refresh failure/warning rather than successful freshness.
|
||||
- Preserve bounded retries, owner revalidation, metadata-only operation, and no quota mutation/download.
|
||||
|
||||
### API patterns
|
||||
|
||||
- `backend/api/cloud/browse.py`
|
||||
- Keep `/api/cloud/connections/{connection_id}/items` as the canonical owner-scoped endpoint.
|
||||
- Preserve whitelisted response conversion through `_item_out`, `_capability_out`, and `_freshness_out`.
|
||||
- First visit may fetch synchronously, but must not mark an incomplete listing fresh.
|
||||
- Cached visits continue stale-while-revalidate through `refresh_cloud_folder.delay`.
|
||||
- Retire or isolate `_fetch_*_folders` compatibility helpers; do not let the legacy route define new listing semantics.
|
||||
- `backend/api/cloud/connections.py`
|
||||
- The `/folders/{provider}/{folder_id}` route is legacy. Migrate remaining consumers/tests to the connection-ID endpoint, then remove it when unreferenced rather than maintaining two provider listing pipelines.
|
||||
- `backend/api/cloud/schemas.py`
|
||||
- Keep the explicit credential-free schema.
|
||||
- `kind` and `provider_item_id` are canonical; avoid restoring `is_dir` solely to accommodate stale frontend code unless a deliberate transitional compatibility field is documented and tested.
|
||||
|
||||
### Frontend patterns
|
||||
|
||||
- `frontend/src/api/cloud.js`
|
||||
- Keep `getCloudFoldersByConnectionId(connectionId, parentRef)` as the single browse client.
|
||||
- Preserve URL encoding of opaque/path-like provider references.
|
||||
- Remove deprecated `getCloudFolders(provider, folderId)` once no active consumer remains.
|
||||
- `frontend/src/views/CloudFolderView.vue`
|
||||
- Remain a thin data provider.
|
||||
- Split items using `kind` and navigate folders using `provider_item_id`.
|
||||
- Feed backend freshness and controlled warning data into the store; do not synthesize “fresh”.
|
||||
- Keep connection UUID routing and session-only folder memory.
|
||||
- Do not add provider switches or listing layout.
|
||||
- `frontend/src/stores/cloudConnections.js`
|
||||
- Keep shared browse state here.
|
||||
- If freshness translation is non-trivial, add one shared mapper/action here and test it once; views should not each invent status semantics.
|
||||
- `frontend/src/components/storage/StorageBrowser.vue`
|
||||
- Remain the only local/cloud file browser.
|
||||
- Receive already normalized folders/files and capability/freshness props.
|
||||
- No provider-specific path parsing or Nextcloud-only rendering.
|
||||
|
||||
## Test Design
|
||||
|
||||
### Provider contract suite: one parametrized contract, provider fixtures only
|
||||
|
||||
Extend `backend/tests/test_cloud_backends.py` with a shared adapter contract harness covering Google Drive, OneDrive, WebDAV, and Nextcloud. Each provider supplies mocked SDK/HTTP responses; assertions stay provider-neutral:
|
||||
|
||||
1. Root call accepts `(connection_id, user_id, parent_ref=None)` and returns `CloudListing`.
|
||||
2. Every child is a `CloudResource` with the supplied connection/user IDs.
|
||||
3. Folder/file classification, names, provider IDs, parent refs, sizes, MIME types, timestamps, and etags normalize correctly.
|
||||
4. Nested listing accepts a returned folder's `provider_item_id` as the next `parent_ref`.
|
||||
5. Pagination is exhausted before `complete=True` (where applicable).
|
||||
6. Provider/network failure returns or is translated to `complete=False` without byte downloads or mutation calls.
|
||||
7. No adapter invokes `get_object`, upload, delete, move, rename, or quota code while browsing.
|
||||
|
||||
Add an explicit Nextcloud regression in `backend/tests/test_cloud_backends.py`:
|
||||
|
||||
- instantiate `NextcloudBackend` with mocked `webdav3` client;
|
||||
- call the canonical four-argument adapter method;
|
||||
- return the ten supplied root entries from mocked PROPFIND metadata;
|
||||
- assert exactly four normalized folders and six normalized files by the expected names;
|
||||
- assert root children have `parent_ref is None`;
|
||||
- select `Documents` and prove its `provider_item_id` can be passed back for a nested listing.
|
||||
|
||||
This test uses only expected names as fixture evidence. It must not contain a URL, username, app-password, encrypted credentials, or network marker.
|
||||
|
||||
### Service reconciliation suite
|
||||
|
||||
Extend `backend/tests/test_cloud_items.py`:
|
||||
|
||||
- authoritative root listing upserts all ten unique Nextcloud fixture names;
|
||||
- repeating the listing is idempotent;
|
||||
- renamed/moved metadata preserves DocuVault item UUID by provider ID;
|
||||
- `complete=False` retains cached rows and records no deletions;
|
||||
- empty `complete=True` is allowed to remove known children, proving failure and true-empty semantics remain distinct;
|
||||
- root (`None`) and nested parent refs never collide across connection/user boundaries.
|
||||
|
||||
### Task suite
|
||||
|
||||
Extend `backend/tests/test_cloud_items.py` or create `backend/tests/test_cloud_tasks.py` if task behavior becomes substantial:
|
||||
|
||||
- Nextcloud adapter receives canonical arguments through `_run`;
|
||||
- complete listing reconciles and marks fresh;
|
||||
- incomplete listing marks warning/stale and does not delete cached items;
|
||||
- transient exception retries, terminal auth failure does not retry;
|
||||
- credentials stay out of broker payload and logs;
|
||||
- no byte or quota APIs are called.
|
||||
|
||||
### API integration suite
|
||||
|
||||
Extend `backend/tests/test_cloud.py` and `backend/tests/test_cloud_security.py`:
|
||||
|
||||
- first root browse with a mocked provider contract returns all normalized items;
|
||||
- response exposes `kind` and `provider_item_id`, never credentials;
|
||||
- incomplete first fetch returns controlled non-fresh status rather than a false fresh status;
|
||||
- cached listing is returned while refresh is scheduled;
|
||||
- nested browse forwards the exact provider reference;
|
||||
- all four provider values pass through the same endpoint;
|
||||
- wrong-owner/admin requests remain 404/403 and never invoke adapters;
|
||||
- browse never downloads bytes or mutates quota.
|
||||
|
||||
### Frontend suite
|
||||
|
||||
Extend `frontend/src/views/__tests__/CloudFolderView.test.js`:
|
||||
|
||||
- canonical `kind: 'folder'` and `kind: 'file'` entries are passed to `StorageBrowser` in the right props;
|
||||
- clicking `Documents` routes with its `provider_item_id`, not its DocuVault `id`;
|
||||
- root request omits/translates the `root` sentinel consistently to canonical `parent_ref=None` at the API boundary;
|
||||
- backend `freshness.refresh_state`, timestamp, and warning message populate store/UI state;
|
||||
- HTTP 200 with warning/incomplete state never becomes “fresh” merely because the request resolved;
|
||||
- expected ten-name Nextcloud fixture is rendered through the same shared browser as every provider.
|
||||
|
||||
Extend `frontend/src/stores/__tests__/cloudConnections.test.js` if freshness mapping is placed in the store. Preserve existing connection UUID and session-storage secrecy tests.
|
||||
|
||||
## Acceptance Matrix
|
||||
|
||||
| Layer | Provider-neutral acceptance |
|
||||
|---|---|
|
||||
| Adapter | All four providers implement the exact `CloudResourceAdapter` signature and return `CloudListing` |
|
||||
| Root identity | UI root sentinel becomes backend `parent_ref=None`; adapters translate root internally |
|
||||
| Child identity | `provider_item_id` is used for provider navigation; `id` remains DocuVault metadata identity |
|
||||
| Classification | `kind` is the sole canonical folder/file discriminator |
|
||||
| Reconciliation | Only `reconcile_cloud_listing` writes listing metadata; incomplete results never delete cached rows |
|
||||
| Freshness | Backend folder state is authoritative; incomplete/error results cannot display as fresh |
|
||||
| Security | Owner scoping, credential secrecy, SSRF checks, no byte download, and no quota mutation remain intact |
|
||||
| Nextcloud evidence | Root shows exactly the four supplied folders and six supplied files under mocked acceptance data |
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Add the shared provider contract tests and failing Nextcloud signature/root regression.
|
||||
2. Remove the Nextcloud legacy override in favor of inherited DAV normalization.
|
||||
3. Make incomplete-listing handling consistent in synchronous API and background task paths.
|
||||
4. Fix `CloudFolderView.vue` to use `kind`, `provider_item_id`, and backend freshness.
|
||||
5. Add API and frontend regressions for root, nested navigation, warning truthfulness, and all-provider parity.
|
||||
6. Remove the legacy provider-slug browse path/helpers when repository search proves no active consumer remains.
|
||||
|
||||
No provider-specific file grid, reconciliation function, freshness mapper, or Nextcloud-only listing parser should be introduced.
|
||||
+333
@@ -0,0 +1,333 @@
|
||||
# Phase 12.1: Fix Nextcloud Root Listing and Sync Visibility — Research
|
||||
|
||||
**Researched:** 2026-06-22
|
||||
**Status:** Complete
|
||||
**Scope:** Secure live diagnosis, provider-neutral browse contracts, reconciliation/freshness correctness, and cross-provider validation design
|
||||
|
||||
## Research Question
|
||||
|
||||
Why can a healthy Nextcloud connection report a successful sync while its root appears empty, and what contract, integration, live-smoke, and UI tests are required to make browse correctness observable across Nextcloud, generic WebDAV, Google Drive, and OneDrive without downloading provider bytes or mutating provider content?
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The live Nextcloud account is reachable and its root is not empty. A credential-redacted, read-only `PROPFIND` with `Depth: 1` returned HTTP `207 Multi-Status` and 10 direct children. Seven names exactly matched the supplied expectation and three returned names differed from the supplied expectation; no unexpected names were printed or persisted. This proves that the configured account and WebDAV endpoint can return root metadata and that the DocuVault empty-root behavior is primarily an application defect, not an empty account or basic connectivity failure. The three name differences should be treated as test-fixture drift until confirmed in the Nextcloud UI; they do not explain DocuVault returning no items.
|
||||
|
||||
The immediate backend root cause is deterministic: `NextcloudBackend` overrides the provider-neutral `WebDAVBackend.list_folder(connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing` with a legacy `list_folder(folder_path="") -> list[dict]`. The canonical browse endpoint invokes the four-argument contract, so a Nextcloud adapter raises `TypeError` before performing a listing. Existing tests only assert that Nextcloud is a `CloudResourceAdapter` subclass and that `list_folder` is async; they never invoke the Nextcloud instance through the canonical contract. Python inheritance therefore made a structurally invalid adapter look valid.
|
||||
|
||||
Three additional defects create false-sync or invisible-navigation behavior across providers:
|
||||
|
||||
1. The synchronous browse path and Celery refresh path mark a folder `fresh` even when an adapter returns `CloudListing(complete=False)`. An incomplete or failed fetch is therefore eligible to look successfully synchronized.
|
||||
2. `CloudFolderView.vue` ignores `response.freshness`, unconditionally sets freshness to `fresh`, and invents `lastRefreshedAt` from the browser clock after any HTTP 200. A safe HTTP response containing an empty incomplete/warning listing can therefore be shown as current.
|
||||
3. The API returns `kind: "folder" | "file"`, `provider_item_id`, and a stable DocuVault `id`, but cloud views and tree components still filter on `is_dir` and navigate using `item.id`. Folders are consequently treated as files, and folder navigation sends the DocuVault metadata UUID where adapters require the provider folder reference/path.
|
||||
|
||||
Phase 12.1 should fix the inherited adapter contract first, then make listing completeness control freshness, then align the frontend with the normalized API. The verification architecture should apply one reusable provider contract suite to all four adapters and add opt-in, read-only live smoke tests whose credentials remain in ignored environment variables.
|
||||
|
||||
## Secure Live Probe Evidence
|
||||
|
||||
### Guardrails Used
|
||||
|
||||
- Read only the ignored `.env` variables `NEXTCLOUD_URL`, `NEXTCLOUD_USER`, and `NEXTCLOUD_APP_PASSWORD` in process memory.
|
||||
- Performed only one-level `PROPFIND` metadata requests; no `GET`, download, upload, `MKCOL`, `PUT`, `MOVE`, `COPY`, or `DELETE` request was made.
|
||||
- Did not edit `.env` or any environment file.
|
||||
- Did not print, echo, log, quote, persist, or include credentials or the full URL in this artifact.
|
||||
- Reported only response status, direct-child count, expected-match count, missing expected names, and unexpected-name count. Unexpected provider names were deliberately not emitted.
|
||||
|
||||
### Sanitized Result
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Endpoint reachable | Yes |
|
||||
| Authentication accepted | Yes |
|
||||
| Method | `PROPFIND` |
|
||||
| Depth | `1` |
|
||||
| HTTP status | `207 Multi-Status` |
|
||||
| Direct root children returned | 10 |
|
||||
| Exact supplied-name matches | 7 of 10 |
|
||||
| Supplied names not exactly matched | `Manual Nextcloud.png`, `Nextcloud Intro.mp4`, `Template credits.md` |
|
||||
| Additional returned-name count | 3; names withheld |
|
||||
| Byte download or provider mutation | None |
|
||||
|
||||
The root-list response confirms that URL derivation to the user WebDAV root works when a base Nextcloud URL is normalized to the standard `/remote.php/dav/files/{encoded-user}/` endpoint. Repeated percent-decoding and plus-decoding did not turn the three differences into exact matches, so those differences are not explained by a simple href percent-encoding bug.
|
||||
|
||||
## Root-Cause Analysis
|
||||
|
||||
### P0 — Nextcloud Violates the Provider-Neutral Adapter Signature
|
||||
|
||||
Current runtime signatures:
|
||||
|
||||
```text
|
||||
NextcloudBackend.list_folder(self, folder_path: str = "") -> list[dict]
|
||||
WebDAVBackend.list_folder(self, connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing
|
||||
```
|
||||
|
||||
`build_cloud_resource_adapter("nextcloud", credentials)` passes the `isinstance(..., CloudResourceAdapter)` check because `NextcloudBackend` inherits from `WebDAVBackend`. However, the override replaces the abstract-contract implementation with the old pre-Phase-12 method. The canonical browse endpoint calls:
|
||||
|
||||
```python
|
||||
await adapter.list_folder(connection_id, current_user.id, parent_ref=parent_ref)
|
||||
```
|
||||
|
||||
For Nextcloud this raises before the provider request. The most direct fix is to delete the duplicate Nextcloud `list_folder` override and inherit the shared WebDAV implementation. If Nextcloud-specific href normalization is required, it should be a narrowly scoped protected hook used by the shared WebDAV listing algorithm, not a second public method with a different contract.
|
||||
|
||||
This is exactly the project rule “things that look the same to the user are the same in code”: Nextcloud is a WebDAV specialization and must not own a parallel browse API.
|
||||
|
||||
### P0 — Incomplete Listings Are Marked Fresh
|
||||
|
||||
Both first-visit browse reconciliation and `refresh_cloud_folder` do the following regardless of `listing.complete`:
|
||||
|
||||
1. reconcile the listing;
|
||||
2. set folder state to `fresh`;
|
||||
3. update `last_refreshed_at`.
|
||||
|
||||
`reconcile_cloud_listing` correctly avoids soft deletion for `complete=False`, but freshness semantics are not enforced by the service boundary. An incomplete empty result can therefore preserve cached rows yet still claim synchronization succeeded; on first visit it can claim an empty root is fresh.
|
||||
|
||||
The planner should introduce one service-level completion gate shared by synchronous browse and Celery refresh. A listing may transition to `fresh` only when `complete=True` and every required page/response was parsed successfully. `complete=False` must retain the previous `last_refreshed_at`, set a controlled warning/error code, and schedule bounded retry where appropriate. An empty `complete=True` listing remains a valid authoritative empty folder.
|
||||
|
||||
### P0 — Frontend Discards Server Freshness
|
||||
|
||||
`CloudFolderView.vue` currently changes state to:
|
||||
|
||||
```text
|
||||
freshness = "fresh"
|
||||
refreshedAt = new Date().toISOString()
|
||||
```
|
||||
|
||||
after every successful HTTP response. It does not use `data.freshness.refresh_state`, `last_refreshed_at`, `error_code`, or `error_message`. This independently explains the user-visible “everything is synced” report even when backend refresh failed or was incomplete.
|
||||
|
||||
The view must map the server freshness object verbatim into the store and render controlled warning copy. The client clock must never manufacture provider refresh evidence. A 200 response means the cached browse request succeeded; it does not mean provider synchronization succeeded.
|
||||
|
||||
### P0 — API/UI Item Shape and Navigation Reference Disagree
|
||||
|
||||
The backend whitelisted response uses:
|
||||
|
||||
```text
|
||||
kind: "folder" | "file"
|
||||
id: stable DocuVault CloudItem UUID
|
||||
provider_item_id: provider folder/file reference
|
||||
```
|
||||
|
||||
The frontend still expects `is_dir` and uses `item.id` as the next `parent_ref`. Consequences:
|
||||
|
||||
- `items.filter(i => i.is_dir)` yields no folders because `is_dir` is absent;
|
||||
- folders fall into the file collection;
|
||||
- sidebar trees also filter on the absent field;
|
||||
- clicking a folder, where possible, sends a DocuVault UUID to WebDAV/Drive/Graph instead of the provider reference.
|
||||
|
||||
Prefer one normalized frontend shape rather than adding duplicate backend fields. Views/components should use `item.kind === "folder"`, and provider navigation should use `item.provider_item_id`. The stable DocuVault `id` remains the row identity/key for persistence and future analysis. If the team chooses to expose an explicit `navigation_ref`, it must be populated centrally in `CloudItemOut`; do not reconstruct provider-specific navigation values in Vue.
|
||||
|
||||
### P1 — Nextcloud URL Normalization Is UI-Only
|
||||
|
||||
The credential modal constructs the standard Nextcloud WebDAV root from a base URL and encoded username, but the backend accepts and stores an arbitrary `server_url` without provider-specific normalization. This creates inconsistent behavior among UI-created connections, API clients, imported configuration, edited credentials, and live tests.
|
||||
|
||||
Add one pure shared helper for Nextcloud endpoint normalization, used by connection create/update and backend construction. Required cases:
|
||||
|
||||
- bare origin with or without trailing slash;
|
||||
- origin with a deployment subpath;
|
||||
- already canonical `/remote.php/dav/files/{user}/` endpoint;
|
||||
- username requiring path-segment percent encoding;
|
||||
- trailing-slash idempotence;
|
||||
- rejection of query strings, fragments, userinfo, non-HTTPS, private/loopback/link-local targets, and malformed endpoints;
|
||||
- no double append of the WebDAV suffix;
|
||||
- no logging of the normalized URL when it contains a username path segment.
|
||||
|
||||
SSRF validation must run against the normalized URL before client construction and before every outbound request. Redirects must not bypass host/IP revalidation.
|
||||
|
||||
### P1 — WebDAV Listing Uses N+1 Requests and Weak Error Semantics
|
||||
|
||||
The shared implementation calls `client.list()` and then `client.info()` once per item. Per-item `info()` errors are swallowed and converted to defaults, potentially misclassifying a folder as a file. A standards-level `PROPFIND Depth: 1` can return the root self-response and all direct children with `resourcetype`, length, type, mtime, and etag in one response.
|
||||
|
||||
Phase 12.1 should preferably centralize a single Depth-1 parser for Nextcloud and generic WebDAV. It must:
|
||||
|
||||
- send explicit `Depth: 1`;
|
||||
- request an allowlist of metadata properties only;
|
||||
- identify and exclude exactly the collection self-response by normalized href, not by a display-name coincidence;
|
||||
- decode each href path segment exactly once for display while retaining a canonical provider reference for subsequent requests;
|
||||
- accept absolute and relative hrefs but reject a child href that escapes the configured root;
|
||||
- distinguish folders from the DAV `collection` resource type, not size or trailing slash alone;
|
||||
- treat missing optional properties as `None` without dropping the item;
|
||||
- mark the entire listing incomplete on malformed multistatus, request failure, untrusted href, or interrupted pagination/response processing;
|
||||
- never call byte-content methods.
|
||||
|
||||
## Provider-Neutral Correctness Contract
|
||||
|
||||
Every provider adapter must pass the same behavioral suite. Provider-specific fixtures are inputs; normalized `CloudListing` and `CloudResource` behavior is the contract.
|
||||
|
||||
### Adapter Contract
|
||||
|
||||
For each of Nextcloud, WebDAV, Google Drive, and OneDrive:
|
||||
|
||||
1. `list_folder` has the canonical callable signature and accepts root plus nested `parent_ref`.
|
||||
2. It returns `CloudListing`, never a provider dictionary/list.
|
||||
3. Every resource contains the requested `connection_id` and `user_id` supplied by the trusted caller.
|
||||
4. `provider_item_id` is a usable opaque navigation reference; `id` is a generated/stable-in-DB DocuVault identity and is not sent back to the provider.
|
||||
5. `kind`, `name`, parent, size, content type, modified time, and etag/version normalize consistently, with absent provider metadata represented as `None`.
|
||||
6. All pages are consumed before `complete=True`; an error on any page yields `complete=False` and must not authorize deletion.
|
||||
7. Authentication/scope failures map to typed, controlled domain reasons; raw provider bodies and URLs never reach API responses or audit logs.
|
||||
8. Listing and capability discovery invoke no upload, create-folder, move, rename, copy, delete, media/content, or download method.
|
||||
9. Listing changes neither `quotas.used_bytes` nor MinIO/object storage.
|
||||
|
||||
### Provider-Specific Fixtures
|
||||
|
||||
| Provider | Required fixture assertions |
|
||||
|---|---|
|
||||
| Nextcloud | Canonical and base URL normalization; `207` multistatus; explicit Depth 1; root self href removed; encoded spaces/unicode; DAV collection detection; nested path; malformed/foreign href incomplete; canonical four-argument method inherited or implemented exactly once |
|
||||
| Generic WebDAV | Absolute/relative hrefs; missing optional props; servers with/without trailing slashes; 401/403 vs 5xx mapping; root and nested Depth-1 listings; no N+1 metadata calls if direct parser is adopted |
|
||||
| Google Drive | Root query uses `root`; every `nextPageToken` followed; folder MIME type; native Google files have nullable size; trashed items excluded; insufficient `drive.file` visibility represented as scope limitation where detectable; no `get_media` |
|
||||
| OneDrive | Root and item children endpoints; every `@odata.nextLink` followed; folder/file facets; parent reference and drive context retained; eTag/cTag normalization; expired token refresh failure controlled; no `/content` request |
|
||||
|
||||
## Test Design
|
||||
|
||||
### 1. Static and Runtime Contract Regression
|
||||
|
||||
Add a parametrized test over all factory providers that constructs the adapter, inspects `list_folder`, and invokes it with `(connection_id, user_id, parent_ref=None, page_token=None)` against mocked provider responses. This would have caught the current Nextcloud override immediately. Merely testing `issubclass` or `iscoroutinefunction` is insufficient.
|
||||
|
||||
Suggested location: `backend/tests/test_cloud_provider_contract.py`.
|
||||
|
||||
Core assertions:
|
||||
|
||||
```text
|
||||
factory returns CloudResourceAdapter
|
||||
canonical invocation does not raise TypeError
|
||||
return type is CloudListing
|
||||
item ownership IDs equal trusted call arguments
|
||||
listing uses provider metadata only
|
||||
no byte or mutation spy was called
|
||||
```
|
||||
|
||||
### 2. WebDAV/Nextcloud Parser Tests
|
||||
|
||||
Use recorded synthetic XML fixtures with no real credentials or hostnames:
|
||||
|
||||
- root self + 4 folders + 6 files produces exactly 10 children;
|
||||
- spaces, `%` characters, unicode, trailing slash, absolute href, and relative href;
|
||||
- child folder recognized from `<d:collection/>` even with size absent;
|
||||
- same basename as root is not incorrectly filtered when it is a genuine child;
|
||||
- href outside the configured DAV root is rejected and listing is incomplete;
|
||||
- missing property status blocks do not drop a child;
|
||||
- malformed XML, non-207 response, timeout, and mid-parse failure return/raise a controlled incomplete result;
|
||||
- request method is `PROPFIND`, `Depth` is exactly `1`, and body requests no content bytes.
|
||||
|
||||
### 3. Freshness/Reconciliation Tests
|
||||
|
||||
Add a shared service such as `apply_cloud_listing_result` or an equivalent explicit branch and test:
|
||||
|
||||
- `complete=True`, non-empty: upsert, delete unseen siblings, state fresh, advance `last_refreshed_at`;
|
||||
- `complete=True`, empty: authoritative empty folder, delete unseen siblings, state fresh;
|
||||
- `complete=False`, empty or partial: retain all unseen rows, state warning, preserve previous `last_refreshed_at`;
|
||||
- provider exception: retain cache, controlled warning, bounded retry only for transient errors;
|
||||
- first visit + incomplete empty listing: response is empty with warning, never fresh;
|
||||
- repeated complete listing is idempotent;
|
||||
- no branch changes quota or calls byte storage.
|
||||
|
||||
Exercise both the synchronous first-visit endpoint and the Celery `_run` path so they cannot drift.
|
||||
|
||||
### 4. API and Frontend Contract Tests
|
||||
|
||||
Backend API tests should return a mixed folder/file fixture and assert `kind`, stable `id`, provider navigation reference, freshness, credential exclusion, owner scoping, and admin denial.
|
||||
|
||||
Frontend tests should use the real response shape, not `{items: []}`:
|
||||
|
||||
- mixed items split correctly using `kind`;
|
||||
- folder click navigates with `provider_item_id` or centralized `navigation_ref`, never stable metadata `id`;
|
||||
- sidebar tree applies the same rule;
|
||||
- warning/refreshing/fresh is copied from `data.freshness`;
|
||||
- `last_refreshed_at` comes from the server and is not replaced with `new Date()`;
|
||||
- HTTP 200 + warning + empty items renders a synchronization warning, not “This folder is empty” with a fresh indicator;
|
||||
- cached items remain visible during warning;
|
||||
- filenames render as text with no `v-html`.
|
||||
|
||||
Suggested files:
|
||||
|
||||
- `frontend/src/views/__tests__/CloudFolderView.test.js`
|
||||
- `frontend/src/components/cloud/__tests__/CloudProviderTreeItem.test.js`
|
||||
- `frontend/src/components/cloud/__tests__/CloudFolderTreeItem.test.js`
|
||||
- `frontend/src/stores/__tests__/cloudConnections.test.js`
|
||||
|
||||
### 5. Opt-In Read-Only Live Tests
|
||||
|
||||
Live tests should be marked, excluded from the default deterministic suite, and enabled explicitly, for example with `RUN_CLOUD_LIVE_TESTS=1`. They must skip when provider-specific variables are absent. Test output must contain provider name, status category, counts, and sanitized mismatch summaries only—never credentials, tokens, usernames, response headers, response bodies, or full URLs.
|
||||
|
||||
Nextcloud variables already available:
|
||||
|
||||
```text
|
||||
NEXTCLOUD_URL
|
||||
NEXTCLOUD_USER
|
||||
NEXTCLOUD_APP_PASSWORD
|
||||
```
|
||||
|
||||
The Nextcloud live test should normalize the endpoint through production code, invoke `build_cloud_resource_adapter("nextcloud", ...)`, call the canonical root listing, and assert:
|
||||
|
||||
- `CloudListing.complete is True`;
|
||||
- exactly 10 direct children for the controlled fixture account, once fixture drift is reconciled;
|
||||
- the agreed expected-name set matches exactly;
|
||||
- at least the expected folder/file kinds are correct;
|
||||
- no content or mutation method is available to the test harness;
|
||||
- a quota snapshot before/after is unchanged if exercised through the DocuVault API.
|
||||
|
||||
For Google Drive, OneDrive, and generic WebDAV, define equivalent optional environment contracts only when dedicated revocable test accounts are supplied. OAuth live tests should list a controlled root folder with least-privilege test credentials, follow pagination, and compare an agreed fixture set. Never make the live suite create its own marker file: provider mutations are outside Phase 12.1 research and are prohibited for these smoke tests.
|
||||
|
||||
### 6. Security-Negative Tests
|
||||
|
||||
- foreign user connection ID returns indistinguishable 404;
|
||||
- admin cannot browse user cloud metadata;
|
||||
- credentials and normalized full endpoint never appear in response, logs, task payload, exception strings, or snapshots;
|
||||
- worker decrypts credentials only after owner-scoped connection resolution;
|
||||
- WebDAV/Nextcloud root and every redirect/request pass SSRF checks;
|
||||
- malicious href cannot escape configured root or cause a request to another origin;
|
||||
- provider error text is mapped to controlled codes/messages;
|
||||
- listing performs no `get_object`, Google `get_media`, OneDrive `/content`, MinIO operation, quota update, or mutation verb.
|
||||
|
||||
## Recommended Implementation Shape
|
||||
|
||||
### Plan 12.1-01 — Restore the Shared Adapter Contract
|
||||
|
||||
- Add failing cross-provider runtime contract tests first.
|
||||
- Remove the legacy Nextcloud public `list_folder` override so Nextcloud inherits the shared canonical WebDAV implementation.
|
||||
- Resolve the deprecated provider-folder helper: route it through the canonical connection-ID adapter or delete the dead compatibility route if no active route/import depends on it. Do not retain two public listing shapes.
|
||||
- Add backend Nextcloud URL normalization in one shared helper and test idempotence/security cases.
|
||||
|
||||
### Plan 12.1-02 — Make WebDAV Metadata Listing Authoritative
|
||||
|
||||
- Implement or encapsulate a single Depth-1 PROPFIND parser shared by Nextcloud and generic WebDAV.
|
||||
- Normalize root self filtering, href containment/decoding, kind, metadata, and controlled errors.
|
||||
- Add fixture tests for the supplied 10-item root plus encoding, missing-property, malformed-response, and SSRF cases.
|
||||
- Keep listing metadata-only and mutation-free.
|
||||
|
||||
### Plan 12.1-03 — Eliminate False Freshness
|
||||
|
||||
- Centralize complete/incomplete listing state transitions.
|
||||
- Use the same transition in synchronous browse and Celery refresh.
|
||||
- Ensure incomplete results preserve cache and last success, set warning, and retry only when transient.
|
||||
- Add reconciliation, endpoint, worker, quota, and no-byte tests.
|
||||
|
||||
### Plan 12.1-04 — Align the Shared Browser Contract
|
||||
|
||||
- Update cloud views/trees to use normalized `kind` and provider navigation reference.
|
||||
- Consume server freshness instead of manufacturing success.
|
||||
- Add mixed-item navigation and warning-state frontend regressions while keeping `StorageBrowser.vue` as the sole browser.
|
||||
- Run the read-only Nextcloud live smoke, then full backend/frontend/security gates and documentation/version closeout required by project protocol.
|
||||
|
||||
## Acceptance Evidence
|
||||
|
||||
Phase 12.1 is complete only when all of the following hold:
|
||||
|
||||
1. The production Nextcloud adapter, invoked through `build_cloud_resource_adapter`, returns the agreed root fixture through the canonical contract.
|
||||
2. Nextcloud and generic WebDAV share one public listing implementation/signature.
|
||||
3. All four providers pass one runtime adapter contract suite and provider-specific normalization/pagination tests.
|
||||
4. `complete=False` cannot produce `fresh` or advance `last_refreshed_at` in either request or worker paths.
|
||||
5. The frontend renders folders/files from the API's normalized shape, navigates with provider references, and displays backend freshness faithfully.
|
||||
6. Browse/live tests use metadata requests only and prove zero byte downloads, provider mutations, MinIO writes, and quota changes.
|
||||
7. Owner/admin/credential/SSRF/href-containment negative tests pass.
|
||||
8. The controlled Nextcloud live smoke reports a complete listing and the final agreed exact-name fixture; any fixture drift is explicitly reconciled rather than hidden.
|
||||
|
||||
## Risks and Planner Notes
|
||||
|
||||
- Do not “fix” Nextcloud by adding an adapter-specific branch in `browse.py` or Vue. The public contract must be identical across providers.
|
||||
- Do not interpret `HTTP 200` from DocuVault as provider freshness; cached browse success and provider refresh success are separate states.
|
||||
- Do not use a root item count alone as general production correctness. Exact names are appropriate only for dedicated controlled live accounts; generic folders may legitimately be empty.
|
||||
- Do not mark an empty listing incomplete merely because it is empty. Completeness comes from successful authoritative traversal, not item count.
|
||||
- Google `drive.file` can produce a valid but intentionally restricted view. Provider-neutral tests should distinguish transport completeness from authorization scope and expose `insufficient_scope` where known.
|
||||
- OneDrive continuation URLs are provider-supplied. Follow them only for the expected Graph origin and never accept arbitrary client-supplied pagination URLs.
|
||||
- Keep unexpected live filenames out of CI logs; report hashed/set counts or controlled expected-name diffs only.
|
||||
- The current worktree contains unrelated user changes and untracked files. Implementation plans must stage only Phase 12.1 files plus required documentation/version updates.
|
||||
|
||||
## RESEARCH COMPLETE
|
||||
@@ -61,7 +61,12 @@ blocked: 0
|
||||
reason: "User reported: All 'modified' rows are empty and just with a '-' filled. The native cloud storage browser does show a modified time."
|
||||
severity: major
|
||||
test: 6
|
||||
root_cause: ""
|
||||
artifacts: []
|
||||
missing: []
|
||||
root_cause: "StorageBrowser.vue renders formatDate(folder.created_at) and formatDate(file.created_at) (lines 170, 239). Cloud items have modified_at (not created_at) — the field exists in CloudItemOut and is mapped by the backend, but the template reads the wrong key."
|
||||
artifacts:
|
||||
- path: "frontend/src/components/storage/StorageBrowser.vue"
|
||||
issue: "Lines 170 and 239 use created_at; cloud items populate modified_at instead"
|
||||
- path: "backend/api/cloud/schemas.py"
|
||||
issue: "CloudItemOut.modified_at exists and is correctly populated — no backend change needed"
|
||||
missing:
|
||||
- "StorageBrowser must render modified_at for cloud items (prop or computed field)"
|
||||
debug_session: ""
|
||||
|
||||
Reference in New Issue
Block a user