curo/kite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(05): create phase 5 plan — cloud storage backends (8 plans, 7 waves)

Browse Source 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
curo1305
2026-05-28 19:43:12 +02:00
parent 141e582eab
commit baa5bed7e2
9 changed files with 2677 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/phases/05-cloud-storage-backends/05-02-PLAN.md
+279
View File
@@ -0,0 +1,279 @@
---
phase: 05-cloud-storage-backends
plan: 02
type: execute
wave: 2
depends_on:
- "05-01"
files_modified:
- backend/storage/cloud_utils.py
- backend/services/cloud_cache.py
- backend/storage/__init__.py
autonomous: true
requirements:
- CLOUD-02
- CLOUD-07
must_haves:
truths:
- "validate_cloud_url() blocks all RFC-1918, loopback, and link-local addresses"
- "encrypt_credentials / decrypt_credentials produce a correct round-trip for any dict"
- "get_storage_backend_for_document() factory returns the correct backend type from document.storage_backend"
- "TTLCache singleton is module-level in cloud_cache.py with maxsize=1000, ttl=60"
artifacts:
- path: "backend/storage/cloud_utils.py"
provides: "SSRF validation + HKDF credential encryption"
contains: "def validate_cloud_url"
- path: "backend/services/cloud_cache.py"
provides: "TTLCache singleton for cloud folder listings"
contains: "get_cloud_folders_cached"
- path: "backend/storage/__init__.py"
provides: "Extended factory for cloud backends"
contains: "get_storage_backend_for_document"
key_links:
- from: "backend/storage/cloud_utils.py"
to: "backend/config.py"
via: "settings.cloud_creds_key"
pattern: "cloud_creds_key"
- from: "backend/storage/__init__.py"
to: "backend/storage/cloud_utils.py"
via: "decrypt_credentials import"
pattern: "decrypt_credentials"
---
<objective>
Create the shared utilities layer for Phase 5: SSRF-safe URL validation, HKDF+Fernet credential encryption/decryption, TTLCache for folder listings, and the extended storage backend factory.
Purpose: All cloud backends and API handlers depend on these primitives. Establishing them before the backends prevents duplication and ensures security invariants are enforced in one place.
Output: cloud_utils.py (validate_cloud_url, encrypt_credentials, decrypt_credentials), cloud_cache.py (TTLCache singleton), updated storage/__init__.py (get_storage_backend_for_document factory).
</objective>
<execution_context>
@/Users/nik/.claude/get-shit-done/workflows/execute-plan.md
@/Users/nik/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/05-cloud-storage-backends/05-CONTEXT.md
@.planning/phases/05-cloud-storage-backends/05-RESEARCH.md
@.planning/phases/05-cloud-storage-backends/05-01-SUMMARY.md
</context>
<interfaces>
<!-- From backend/storage/__init__.py — current factory -->
From backend/storage/__init__.py:
def get_storage_backend() -> StorageBackend:
"""Returns MinIOBackend configured from settings."""
From backend/db/models.py:
Document: storage_backend (String, nullable=False, default="minio"), user_id (UUID nullable)
CloudConnection: id (UUID), user_id (UUID FK), provider (String), credentials_enc (Text),
status (String), connected_at (TIMESTAMP)
User: id (UUID), default_storage_backend (String, default="minio")
From backend/config.py (after Plan 01):
settings.cloud_creds_key: str
settings.minio_endpoint, minio_access_key, minio_secret_key, minio_bucket, minio_public_endpoint
From backend/storage/minio_backend.py:
class MinIOBackend(StorageBackend): -- reference asyncio.to_thread() pattern
RESEARCH.md Pattern 6: SSRF validation using ipaddress + socket.getaddrinfo.
RESEARCH.md Pattern 2: HKDF+Fernet — fresh HKDF instance per call (AlreadyFinalized pitfall).
RESEARCH.md Pattern 8: TTLCache thread safety — threading.Lock required for concurrent access.
RESEARCH.md Pattern 9: get_storage_backend_for_document factory extension.
</interfaces>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Create cloud_utils.py — SSRF validation + HKDF credential encryption</name>
<files>backend/storage/cloud_utils.py</files>
<read_first>
- backend/storage/base.py — StorageBackend ABC, 7 method signatures
- backend/config.py — settings.cloud_creds_key field name
- .planning/phases/05-cloud-storage-backends/05-RESEARCH.md — Pattern 2 (HKDF+Fernet) and Pattern 6 (SSRF)
</read_first>
<behavior>
- validate_cloud_url(url: str) -> None raises ValueError for: localhost, 127.0.0.0/8, 169.254.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, ::1/128, fc00::/7
- validate_cloud_url resolves DNS via socket.getaddrinfo before checking IP (anti-DNS-rebinding: resolves hostname to IP then checks IP against blocked networks)
- validate_cloud_url raises ValueError for non-http/https schemes
- validate_cloud_url raises ValueError for URLs with no hostname
- _derive_fernet_key(master_key: bytes, user_id: str) -> Fernet: creates a fresh HKDF instance on every call (never reuses); uses algorithm=hashes.SHA256(), length=32, salt=user_id.encode("utf-8"), info=b"cloud-credentials"
- encrypt_credentials(master_key: bytes, user_id: str, credentials: dict) -> str: returns Fernet-encrypted JSON string (not plaintext)
- decrypt_credentials(master_key: bytes, user_id: str, credentials_enc: str) -> dict: returns original dict
- Round-trip: decrypt_credentials(master_key, uid, encrypt_credentials(master_key, uid, creds)) == creds
</behavior>
<action>
Create backend/storage/cloud_utils.py with module docstring explaining SSRF prevention and HKDF pattern.
Implement validate_cloud_url(url: str) -> None:
- Import: ipaddress, socket, urllib.parse.urlparse
- Parse URL; reject non-http/https schemes; reject missing hostname
- Define BLOCKED_NETS list: ip_network("127.0.0.0/8"), ip_network("169.254.0.0/16"),
ip_network("10.0.0.0/8"), ip_network("172.16.0.0/12"), ip_network("192.168.0.0/16"),
ip_network("::1/128"), ip_network("fc00::/7")
- Also explicitly block hostname == "localhost" string before IP resolution
- Try ipaddress.ip_address(hostname) — if that fails (not a raw IP), use
socket.getaddrinfo(hostname, None)[0][4][0] to resolve; wrap socket.gaierror
- Check resolved IP against each BLOCKED_NETS entry using addr in net
Implement _derive_fernet_key(master_key: bytes, user_id: str) -> Fernet:
- Import: base64, cryptography.hazmat.primitives.hashes, cryptography.hazmat.primitives.kdf.hkdf.HKDF, cryptography.fernet.Fernet
- Create new HKDF(...) instance each call — do NOT cache or store the instance
- Call hkdf.derive(master_key) → 32 raw bytes
- Return Fernet(base64.urlsafe_b64encode(raw_key))
Implement encrypt_credentials(master_key: bytes, user_id: str, credentials: dict) -> str:
- import json inside function body (or at top)
- Call _derive_fernet_key to get a Fernet instance
- Return f.encrypt(json.dumps(credentials).encode("utf-8")).decode("utf-8")
Implement decrypt_credentials(master_key: bytes, user_id: str, credentials_enc: str) -> dict:
- Call _derive_fernet_key to get a Fernet instance
- Return json.loads(f.decrypt(credentials_enc.encode("utf-8")))
</action>
<verify>
<automated>cd /Users/nik/Documents/Progamming/document_scanner/backend && python -c "
from storage.cloud_utils import validate_cloud_url, encrypt_credentials, decrypt_credentials
import pytest
# SSRF check
try:
validate_cloud_url('http://127.0.0.1/dav')
print('FAIL: loopback should raise')
except ValueError:
print('OK: loopback blocked')
try:
validate_cloud_url('http://10.0.0.1/dav')
print('FAIL: RFC-1918 should raise')
except ValueError:
print('OK: RFC-1918 blocked')
# Round-trip
mk = b'test-master-key-32bytes-padded!!'
uid = '550e8400-e29b-41d4-a716-446655440000'
creds = {'access_token': 'ya29.xxx', 'refresh_token': '1//xxx'}
enc = encrypt_credentials(mk, uid, creds)
assert enc != str(creds)
dec = decrypt_credentials(mk, uid, enc)
assert dec == creds, f'Round-trip failed: {dec}'
print('OK: encryption round-trip')
"</automated>
</verify>
<acceptance_criteria>
- backend/storage/cloud_utils.py contains def validate_cloud_url, def encrypt_credentials, def decrypt_credentials, def _derive_fernet_key
- validate_cloud_url("http://127.0.0.1/dav") raises ValueError
- validate_cloud_url("http://10.0.0.1/dav") raises ValueError
- validate_cloud_url("http://169.254.169.254/dav") raises ValueError
- validate_cloud_url("http://192.168.1.1/dav") raises ValueError
- validate_cloud_url("http://localhost/dav") raises ValueError
- Encryption round-trip: decrypt_credentials(key, uid, encrypt_credentials(key, uid, creds)) == creds
- "access_token" plaintext does NOT appear in the encrypted string
</acceptance_criteria>
<done>cloud_utils.py created; SSRF validation blocks all 5 network categories; HKDF round-trip verified via python -c invocation</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Create cloud_cache.py and extend storage factory</name>
<files>backend/services/cloud_cache.py, backend/storage/__init__.py</files>
<read_first>
- backend/storage/__init__.py — current get_storage_backend() factory
- backend/storage/base.py — StorageBackend ABC
- backend/storage/minio_backend.py — MinIOBackend constructor signature
- backend/db/models.py — CloudConnection, Document, User model fields
- .planning/phases/05-cloud-storage-backends/05-RESEARCH.md — Pattern 8 (TTLCache), Pattern 9 (factory extension)
</read_first>
<behavior>
- backend/services/cloud_cache.py exports a module-level _folder_cache = TTLCache(maxsize=1000, ttl=60) and a threading.Lock()
- get_cloud_folders_cached(user_id: str, provider: str, folder_id: str, fetch_fn: Awaitable) is an async function that checks cache before calling fetch_fn
- get_storage_backend_for_document(document, user, session) is an async function added to backend/storage/__init__.py that returns MinIOBackend for storage_backend=="minio" and raises HTTPException(503) for unknown or inactive cloud connections
- existing get_storage_backend() function in __init__.py is NOT modified (existing callers unaffected)
- get_storage_backend_for_document raises HTTPException(503, detail="Cloud connection not found or inactive") when CloudConnection is missing or status != "ACTIVE"
</behavior>
<action>
Create backend/services/cloud_cache.py:
- Import: threading, cachetools.TTLCache, typing.Callable, typing.Awaitable
- Module-level: _folder_cache: TTLCache = TTLCache(maxsize=1000, ttl=60)
- Module-level: _folder_cache_lock = threading.Lock()
- async function get_cloud_folders_cached(user_id: str, provider: str, folder_id: str, fetch_fn) -> list:
cache_key = f"{user_id}:{provider}:{folder_id}"
with _folder_cache_lock: check if cache_key in _folder_cache; return cached if found
result = await fetch_fn() # called OUTSIDE the lock to not block event loop
with _folder_cache_lock: store result in cache
return result
- Function invalidate_provider_cache(user_id: str, provider: str) -> None: iterates
_folder_cache with lock and deletes all keys starting with f"{user_id}:{provider}:"
Extend backend/storage/__init__.py (add after existing get_storage_backend()):
- Import at top of file: select from sqlalchemy, HTTPException from fastapi, AsyncSession from sqlalchemy.ext.asyncio, Optional from typing
- Import: from db.models import CloudConnection, Document, User
- Import: from config import settings
- Import: from storage.cloud_utils import decrypt_credentials
- Add async function get_storage_backend_for_document(document, user, session: AsyncSession) -> StorageBackend:
If document.storage_backend == "minio": return get_storage_backend() (existing factory)
Otherwise: query CloudConnection where user_id=user.id AND provider=document.storage_backend AND status="ACTIVE"
If not found: raise HTTPException(status_code=503, detail="Cloud connection not found or inactive")
Decrypt credentials: master_key = settings.cloud_creds_key.encode(); credentials = decrypt_credentials(master_key, str(user.id), conn.credentials_enc)
If provider == "google_drive": import GoogleDriveBackend; return GoogleDriveBackend(credentials)
Elif provider == "onedrive": import OneDriveBackend; return OneDriveBackend(credentials)
Elif provider in ("nextcloud", "webdav"): import WebDAVBackend; return WebDAVBackend(credentials["server_url"], credentials["username"], credentials["password"])
Else: raise ValueError(f"Unknown storage backend: {document.storage_backend}")
Use lazy imports (inside the function) for cloud backends to avoid circular imports at module load time.
</action>
<verify>
<automated>cd /Users/nik/Documents/Progamming/document_scanner/backend && python -c "
from services.cloud_cache import get_cloud_folders_cached, _folder_cache, _folder_cache_lock, invalidate_provider_cache
from storage import get_storage_backend, get_storage_backend_for_document
print('cloud_cache imports OK')
print('factory extension imports OK')
print(f'TTLCache maxsize={_folder_cache.maxsize}, ttl={_folder_cache.ttl}')
"</automated>
</verify>
<acceptance_criteria>
- backend/services/cloud_cache.py exists and exports _folder_cache (TTLCache), _folder_cache_lock (Lock), get_cloud_folders_cached (async), invalidate_provider_cache
- _folder_cache.maxsize == 1000 and _folder_cache.ttl == 60
- backend/storage/__init__.py exports get_storage_backend_for_document (async function)
- `from storage import get_storage_backend_for_document` imports without error
- Existing `from storage import get_storage_backend` still works (no regression)
- `python -m pytest -v --tb=short` passes with 0 failures (no import regressions)
</acceptance_criteria>
<done>cloud_cache.py created with TTLCache singleton and cache/invalidate helpers; storage/__init__.py has get_storage_backend_for_document; full pytest suite passes</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| user-supplied URL → validate_cloud_url | Untrusted URL must be checked against SSRF blocklist before any HTTP call |
| credentials dict → Fernet ciphertext | Credentials must never appear in plaintext after this layer |
| DNS resolution → IP check | DNS-based SSRF bypass: hostname resolves to internal IP after validation |
## STRIDE Threat Register
| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-05-02-01 | Tampering | validate_cloud_url — DNS resolution | mitigate | socket.getaddrinfo resolves hostname to IP before network check; validate_cloud_url called immediately before each request (not only at connect-time) per D-17; DNS rebinding window is minimized |
| T-05-02-02 | Information Disclosure | _derive_fernet_key — HKDF instance reuse | mitigate | New HKDF(...) instance created on every _derive_fernet_key call; AlreadyFinalized pitfall (RESEARCH.md Pitfall 3) prevented by construction |
| T-05-02-03 | Information Disclosure | cloud_creds_key default value | mitigate | Default "CHANGEME-32-bytes-padded!!" is clearly a placeholder; production deployment requires CLOUD_CREDS_KEY env var; docstring on Settings field documents the requirement |
| T-05-02-04 | Elevation of Privilege | get_storage_backend_for_document — cross-user | mitigate | Function receives user object from get_regular_user dep; CloudConnection query includes user_id=user.id filter; cross-user access impossible via this function |
| T-05-02-SC | Tampering | cachetools package install | mitigate | cachetools verified [OK] in RESEARCH.md slopcheck audit |
</threat_model>
<verification>
cd /Users/nik/Documents/Progamming/document_scanner/backend && python -m pytest tests/test_cloud.py -v && python -m pytest -v --tb=short 2>&1 | tail -10
</verification>
<success_criteria>
- cloud_utils.py: validate_cloud_url blocks RFC-1918/loopback/link-local; HKDF round-trip correct
- cloud_cache.py: TTLCache(maxsize=1000, ttl=60) with thread-safe lock; get_cloud_folders_cached works
- storage/__init__.py: get_storage_backend_for_document added alongside existing get_storage_backend()
- pytest -v exits 0, 0 failures; test_cloud.py still all xfailed
</success_criteria>
<output>
Create `.planning/phases/05-cloud-storage-backends/05-02-SUMMARY.md` when done
</output>