kite/.planning/phases/12.1-fix-nextcloud-root-listing-and-sync-visibility/12.1-RESEARCH.md

Phase 12.1: Fix Nextcloud Root Listing and Sync Visibility — Research

Researched: 2026-06-22
Status: Complete
Scope: Secure live diagnosis, provider-neutral browse contracts, reconciliation/freshness correctness, and cross-provider validation design

Research Question

Why can a healthy Nextcloud connection report a successful sync while its root appears empty, and what contract, integration, live-smoke, and UI tests are required to make browse correctness observable across Nextcloud, generic WebDAV, Google Drive, and OneDrive without downloading provider bytes or mutating provider content?

Executive Summary

The live Nextcloud account is reachable and its root is not empty. A credential-redacted, read-only PROPFIND with Depth: 1 returned HTTP 207 Multi-Status and 10 direct children. Seven names exactly matched the supplied expectation and three returned names differed from the supplied expectation; no unexpected names were printed or persisted. This proves that the configured account and WebDAV endpoint can return root metadata and that the DocuVault empty-root behavior is primarily an application defect, not an empty account or basic connectivity failure. The three name differences should be treated as test-fixture drift until confirmed in the Nextcloud UI; they do not explain DocuVault returning no items.

The immediate backend root cause is deterministic: NextcloudBackend overrides the provider-neutral WebDAVBackend.list_folder(connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing with a legacy list_folder(folder_path="") -> list[dict]. The canonical browse endpoint invokes the four-argument contract, so a Nextcloud adapter raises TypeError before performing a listing. Existing tests only assert that Nextcloud is a CloudResourceAdapter subclass and that list_folder is async; they never invoke the Nextcloud instance through the canonical contract. Python inheritance therefore made a structurally invalid adapter look valid.

Three additional defects create false-sync or invisible-navigation behavior across providers:

1. The synchronous browse path and Celery refresh path mark a folder fresh even when an adapter returns CloudListing(complete=False). An incomplete or failed fetch is therefore eligible to look successfully synchronized.
2. CloudFolderView.vue ignores response.freshness, unconditionally sets freshness to fresh, and invents lastRefreshedAt from the browser clock after any HTTP 200. A safe HTTP response containing an empty incomplete/warning listing can therefore be shown as current.
3. The API returns kind: "folder" | "file", provider_item_id, and a stable DocuVault id, but cloud views and tree components still filter on is_dir and navigate using item.id. Folders are consequently treated as files, and folder navigation sends the DocuVault metadata UUID where adapters require the provider folder reference/path.

Phase 12.1 should fix the inherited adapter contract first, then make listing completeness control freshness, then align the frontend with the normalized API. The verification architecture should apply one reusable provider contract suite to all four adapters and add opt-in, read-only live smoke tests whose credentials remain in ignored environment variables.

Secure Live Probe Evidence

Guardrails Used

• Read only the ignored .env variables NEXTCLOUD_URL, NEXTCLOUD_USER, and NEXTCLOUD_APP_PASSWORD in process memory.
• Performed only one-level PROPFIND metadata requests; no GET, download, upload, MKCOL, PUT, MOVE, COPY, or DELETE request was made.
• Did not edit .env or any environment file.
• Did not print, echo, log, quote, persist, or include credentials or the full URL in this artifact.
• Reported only response status, direct-child count, expected-match count, missing expected names, and unexpected-name count. Unexpected provider names were deliberately not emitted.

Sanitized Result

Check	Result
Endpoint reachable	Yes
Authentication accepted	Yes
Method	PROPFIND
Depth	1
HTTP status	207 Multi-Status
Direct root children returned	10
Exact supplied-name matches	7 of 10
Supplied names not exactly matched	Manual Nextcloud.png, Nextcloud Intro.mp4, Template credits.md
Additional returned-name count	3; names withheld
Byte download or provider mutation	None

The root-list response confirms that URL derivation to the user WebDAV root works when a base Nextcloud URL is normalized to the standard /remote.php/dav/files/{encoded-user}/ endpoint. Repeated percent-decoding and plus-decoding did not turn the three differences into exact matches, so those differences are not explained by a simple href percent-encoding bug.

Root-Cause Analysis

P0 — Nextcloud Violates the Provider-Neutral Adapter Signature

Current runtime signatures:

NextcloudBackend.list_folder(self, folder_path: str = "") -> list[dict]
WebDAVBackend.list_folder(self, connection_id, user_id, parent_ref=None, page_token=None) -> CloudListing

build_cloud_resource_adapter("nextcloud", credentials) passes the isinstance(..., CloudResourceAdapter) check because NextcloudBackend inherits from WebDAVBackend. However, the override replaces the abstract-contract implementation with the old pre-Phase-12 method. The canonical browse endpoint calls:

await adapter.list_folder(connection_id, current_user.id, parent_ref=parent_ref)

For Nextcloud this raises before the provider request. The most direct fix is to delete the duplicate Nextcloud list_folder override and inherit the shared WebDAV implementation. If Nextcloud-specific href normalization is required, it should be a narrowly scoped protected hook used by the shared WebDAV listing algorithm, not a second public method with a different contract.

This is exactly the project rule “things that look the same to the user are the same in code”: Nextcloud is a WebDAV specialization and must not own a parallel browse API.

P0 — Incomplete Listings Are Marked Fresh

Both first-visit browse reconciliation and refresh_cloud_folder do the following regardless of listing.complete:

1. reconcile the listing;
2. set folder state to fresh;
3. update last_refreshed_at.

reconcile_cloud_listing correctly avoids soft deletion for complete=False, but freshness semantics are not enforced by the service boundary. An incomplete empty result can therefore preserve cached rows yet still claim synchronization succeeded; on first visit it can claim an empty root is fresh.

The planner should introduce one service-level completion gate shared by synchronous browse and Celery refresh. A listing may transition to fresh only when complete=True and every required page/response was parsed successfully. complete=False must retain the previous last_refreshed_at, set a controlled warning/error code, and schedule bounded retry where appropriate. An empty complete=True listing remains a valid authoritative empty folder.

P0 — Frontend Discards Server Freshness

CloudFolderView.vue currently changes state to:

freshness = "fresh"
refreshedAt = new Date().toISOString()

after every successful HTTP response. It does not use data.freshness.refresh_state, last_refreshed_at, error_code, or error_message. This independently explains the user-visible “everything is synced” report even when backend refresh failed or was incomplete.

The view must map the server freshness object verbatim into the store and render controlled warning copy. The client clock must never manufacture provider refresh evidence. A 200 response means the cached browse request succeeded; it does not mean provider synchronization succeeded.

P0 — API/UI Item Shape and Navigation Reference Disagree

The backend whitelisted response uses:

kind: "folder" | "file"
id: stable DocuVault CloudItem UUID
provider_item_id: provider folder/file reference

The frontend still expects is_dir and uses item.id as the next parent_ref. Consequences:

• items.filter(i => i.is_dir) yields no folders because is_dir is absent;
• folders fall into the file collection;
• sidebar trees also filter on the absent field;
• clicking a folder, where possible, sends a DocuVault UUID to WebDAV/Drive/Graph instead of the provider reference.

Prefer one normalized frontend shape rather than adding duplicate backend fields. Views/components should use item.kind === "folder", and provider navigation should use item.provider_item_id. The stable DocuVault id remains the row identity/key for persistence and future analysis. If the team chooses to expose an explicit navigation_ref, it must be populated centrally in CloudItemOut; do not reconstruct provider-specific navigation values in Vue.

P1 — Nextcloud URL Normalization Is UI-Only

The credential modal constructs the standard Nextcloud WebDAV root from a base URL and encoded username, but the backend accepts and stores an arbitrary server_url without provider-specific normalization. This creates inconsistent behavior among UI-created connections, API clients, imported configuration, edited credentials, and live tests.

Add one pure shared helper for Nextcloud endpoint normalization, used by connection create/update and backend construction. Required cases:

• bare origin with or without trailing slash;
• origin with a deployment subpath;
• already canonical /remote.php/dav/files/{user}/ endpoint;
• username requiring path-segment percent encoding;
• trailing-slash idempotence;
• rejection of query strings, fragments, userinfo, non-HTTPS, private/loopback/link-local targets, and malformed endpoints;
• no double append of the WebDAV suffix;
• no logging of the normalized URL when it contains a username path segment.

SSRF validation must run against the normalized URL before client construction and before every outbound request. Redirects must not bypass host/IP revalidation.

P1 — WebDAV Listing Uses N+1 Requests and Weak Error Semantics

The shared implementation calls client.list() and then client.info() once per item. Per-item info() errors are swallowed and converted to defaults, potentially misclassifying a folder as a file. A standards-level PROPFIND Depth: 1 can return the root self-response and all direct children with resourcetype, length, type, mtime, and etag in one response.

Phase 12.1 should preferably centralize a single Depth-1 parser for Nextcloud and generic WebDAV. It must:

• send explicit Depth: 1;
• request an allowlist of metadata properties only;
• identify and exclude exactly the collection self-response by normalized href, not by a display-name coincidence;
• decode each href path segment exactly once for display while retaining a canonical provider reference for subsequent requests;
• accept absolute and relative hrefs but reject a child href that escapes the configured root;
• distinguish folders from the DAV collection resource type, not size or trailing slash alone;
• treat missing optional properties as None without dropping the item;
• mark the entire listing incomplete on malformed multistatus, request failure, untrusted href, or interrupted pagination/response processing;
• never call byte-content methods.

Provider-Neutral Correctness Contract

Every provider adapter must pass the same behavioral suite. Provider-specific fixtures are inputs; normalized CloudListing and CloudResource behavior is the contract.

Adapter Contract

For each of Nextcloud, WebDAV, Google Drive, and OneDrive:

1. list_folder has the canonical callable signature and accepts root plus nested parent_ref.
2. It returns CloudListing, never a provider dictionary/list.
3. Every resource contains the requested connection_id and user_id supplied by the trusted caller.
4. provider_item_id is a usable opaque navigation reference; id is a generated/stable-in-DB DocuVault identity and is not sent back to the provider.
5. kind, name, parent, size, content type, modified time, and etag/version normalize consistently, with absent provider metadata represented as None.
6. All pages are consumed before complete=True; an error on any page yields complete=False and must not authorize deletion.
7. Authentication/scope failures map to typed, controlled domain reasons; raw provider bodies and URLs never reach API responses or audit logs.
8. Listing and capability discovery invoke no upload, create-folder, move, rename, copy, delete, media/content, or download method.
9. Listing changes neither quotas.used_bytes nor MinIO/object storage.

Provider-Specific Fixtures

Provider	Required fixture assertions
Nextcloud	Canonical and base URL normalization; 207 multistatus; explicit Depth 1; root self href removed; encoded spaces/unicode; DAV collection detection; nested path; malformed/foreign href incomplete; canonical four-argument method inherited or implemented exactly once
Generic WebDAV	Absolute/relative hrefs; missing optional props; servers with/without trailing slashes; 401/403 vs 5xx mapping; root and nested Depth-1 listings; no N+1 metadata calls if direct parser is adopted
Google Drive	Root query uses root; every nextPageToken followed; folder MIME type; native Google files have nullable size; trashed items excluded; insufficient drive.file visibility represented as scope limitation where detectable; no get_media
OneDrive	Root and item children endpoints; every @odata.nextLink followed; folder/file facets; parent reference and drive context retained; eTag/cTag normalization; expired token refresh failure controlled; no /content request

Test Design

1. Static and Runtime Contract Regression

Add a parametrized test over all factory providers that constructs the adapter, inspects list_folder, and invokes it with (connection_id, user_id, parent_ref=None, page_token=None) against mocked provider responses. This would have caught the current Nextcloud override immediately. Merely testing issubclass or iscoroutinefunction is insufficient.

Suggested location: backend/tests/test_cloud_provider_contract.py.

Core assertions:

factory returns CloudResourceAdapter
canonical invocation does not raise TypeError
return type is CloudListing
item ownership IDs equal trusted call arguments
listing uses provider metadata only
no byte or mutation spy was called

2. WebDAV/Nextcloud Parser Tests

Use recorded synthetic XML fixtures with no real credentials or hostnames:

• root self + 4 folders + 6 files produces exactly 10 children;
• spaces, % characters, unicode, trailing slash, absolute href, and relative href;
• child folder recognized from <d:collection/> even with size absent;
• same basename as root is not incorrectly filtered when it is a genuine child;
• href outside the configured DAV root is rejected and listing is incomplete;
• missing property status blocks do not drop a child;
• malformed XML, non-207 response, timeout, and mid-parse failure return/raise a controlled incomplete result;
• request method is PROPFIND, Depth is exactly 1, and body requests no content bytes.

3. Freshness/Reconciliation Tests

Add a shared service such as apply_cloud_listing_result or an equivalent explicit branch and test:

• complete=True, non-empty: upsert, delete unseen siblings, state fresh, advance last_refreshed_at;
• complete=True, empty: authoritative empty folder, delete unseen siblings, state fresh;
• complete=False, empty or partial: retain all unseen rows, state warning, preserve previous last_refreshed_at;
• provider exception: retain cache, controlled warning, bounded retry only for transient errors;
• first visit + incomplete empty listing: response is empty with warning, never fresh;
• repeated complete listing is idempotent;
• no branch changes quota or calls byte storage.

Exercise both the synchronous first-visit endpoint and the Celery _run path so they cannot drift.

4. API and Frontend Contract Tests

Backend API tests should return a mixed folder/file fixture and assert kind, stable id, provider navigation reference, freshness, credential exclusion, owner scoping, and admin denial.

Frontend tests should use the real response shape, not {items: []}:

• mixed items split correctly using kind;
• folder click navigates with provider_item_id or centralized navigation_ref, never stable metadata id;
• sidebar tree applies the same rule;
• warning/refreshing/fresh is copied from data.freshness;
• last_refreshed_at comes from the server and is not replaced with new Date();
• HTTP 200 + warning + empty items renders a synchronization warning, not “This folder is empty” with a fresh indicator;
• cached items remain visible during warning;
• filenames render as text with no v-html.

Suggested files:

• frontend/src/views/__tests__/CloudFolderView.test.js
• frontend/src/components/cloud/__tests__/CloudProviderTreeItem.test.js
• frontend/src/components/cloud/__tests__/CloudFolderTreeItem.test.js
• frontend/src/stores/__tests__/cloudConnections.test.js

5. Opt-In Read-Only Live Tests

Live tests should be marked, excluded from the default deterministic suite, and enabled explicitly, for example with RUN_CLOUD_LIVE_TESTS=1. They must skip when provider-specific variables are absent. Test output must contain provider name, status category, counts, and sanitized mismatch summaries only—never credentials, tokens, usernames, response headers, response bodies, or full URLs.

Nextcloud variables already available:

NEXTCLOUD_URL
NEXTCLOUD_USER
NEXTCLOUD_APP_PASSWORD

The Nextcloud live test should normalize the endpoint through production code, invoke build_cloud_resource_adapter("nextcloud", ...), call the canonical root listing, and assert:

• CloudListing.complete is True;
• exactly 10 direct children for the controlled fixture account, once fixture drift is reconciled;
• the agreed expected-name set matches exactly;
• at least the expected folder/file kinds are correct;
• no content or mutation method is available to the test harness;
• a quota snapshot before/after is unchanged if exercised through the DocuVault API.

For Google Drive, OneDrive, and generic WebDAV, define equivalent optional environment contracts only when dedicated revocable test accounts are supplied. OAuth live tests should list a controlled root folder with least-privilege test credentials, follow pagination, and compare an agreed fixture set. Never make the live suite create its own marker file: provider mutations are outside Phase 12.1 research and are prohibited for these smoke tests.

6. Security-Negative Tests

• foreign user connection ID returns indistinguishable 404;
• admin cannot browse user cloud metadata;
• credentials and normalized full endpoint never appear in response, logs, task payload, exception strings, or snapshots;
• worker decrypts credentials only after owner-scoped connection resolution;
• WebDAV/Nextcloud root and every redirect/request pass SSRF checks;
• malicious href cannot escape configured root or cause a request to another origin;
• provider error text is mapped to controlled codes/messages;
• listing performs no get_object, Google get_media, OneDrive /content, MinIO operation, quota update, or mutation verb.

Recommended Implementation Shape

Plan 12.1-01 — Restore the Shared Adapter Contract

• Add failing cross-provider runtime contract tests first.
• Remove the legacy Nextcloud public list_folder override so Nextcloud inherits the shared canonical WebDAV implementation.
• Resolve the deprecated provider-folder helper: route it through the canonical connection-ID adapter or delete the dead compatibility route if no active route/import depends on it. Do not retain two public listing shapes.
• Add backend Nextcloud URL normalization in one shared helper and test idempotence/security cases.

Plan 12.1-02 — Make WebDAV Metadata Listing Authoritative

• Implement or encapsulate a single Depth-1 PROPFIND parser shared by Nextcloud and generic WebDAV.
• Normalize root self filtering, href containment/decoding, kind, metadata, and controlled errors.
• Add fixture tests for the supplied 10-item root plus encoding, missing-property, malformed-response, and SSRF cases.
• Keep listing metadata-only and mutation-free.

Plan 12.1-03 — Eliminate False Freshness

• Centralize complete/incomplete listing state transitions.
• Use the same transition in synchronous browse and Celery refresh.
• Ensure incomplete results preserve cache and last success, set warning, and retry only when transient.
• Add reconciliation, endpoint, worker, quota, and no-byte tests.

Plan 12.1-04 — Align the Shared Browser Contract

• Update cloud views/trees to use normalized kind and provider navigation reference.
• Consume server freshness instead of manufacturing success.
• Add mixed-item navigation and warning-state frontend regressions while keeping StorageBrowser.vue as the sole browser.
• Run the read-only Nextcloud live smoke, then full backend/frontend/security gates and documentation/version closeout required by project protocol.

Acceptance Evidence

Phase 12.1 is complete only when all of the following hold:

1. The production Nextcloud adapter, invoked through build_cloud_resource_adapter, returns the agreed root fixture through the canonical contract.
2. Nextcloud and generic WebDAV share one public listing implementation/signature.
3. All four providers pass one runtime adapter contract suite and provider-specific normalization/pagination tests.
4. complete=False cannot produce fresh or advance last_refreshed_at in either request or worker paths.
5. The frontend renders folders/files from the API's normalized shape, navigates with provider references, and displays backend freshness faithfully.
6. Browse/live tests use metadata requests only and prove zero byte downloads, provider mutations, MinIO writes, and quota changes.
7. Owner/admin/credential/SSRF/href-containment negative tests pass.
8. The controlled Nextcloud live smoke reports a complete listing and the final agreed exact-name fixture; any fixture drift is explicitly reconciled rather than hidden.

Risks and Planner Notes

• Do not “fix” Nextcloud by adding an adapter-specific branch in browse.py or Vue. The public contract must be identical across providers.
• Do not interpret HTTP 200 from DocuVault as provider freshness; cached browse success and provider refresh success are separate states.
• Do not use a root item count alone as general production correctness. Exact names are appropriate only for dedicated controlled live accounts; generic folders may legitimately be empty.
• Do not mark an empty listing incomplete merely because it is empty. Completeness comes from successful authoritative traversal, not item count.
• Google drive.file can produce a valid but intentionally restricted view. Provider-neutral tests should distinguish transport completeness from authorization scope and expose insufficient_scope where known.
• OneDrive continuation URLs are provider-supplied. Follow them only for the expected Graph origin and never accept arbitrary client-supplied pagination URLs.
• Keep unexpected live filenames out of CI logs; report hashed/set counts or controlled expected-name diffs only.
• The current worktree contains unrelated user changes and untracked files. Implementation plans must stage only Phase 12.1 files plus required documentation/version updates.

RESEARCH COMPLETE