From 800b1e949460c1cdab134d38788d90704ab8dc5a Mon Sep 17 00:00:00 2001 From: curo1305 Date: Mon, 18 May 2026 15:28:06 +0200 Subject: [PATCH] docs: mark Stage 3 complete, update architecture and code inventory Co-Authored-By: Claude Sonnet 4.6 --- CLAUDE.md | 42 +++++++++++++++++++++++++++++------------- 1 file changed, 29 insertions(+), 13 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1dcd267..661c3ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,8 +7,8 @@ a plugin/integration system (Stage 2+) and an encrypted vault (Stage 3+). ## Current Status -**Stage 2 — Plugin Framework: complete** (2026-05-18) -Next: Stage 3 — Memory Database +**Stage 3 — Memory Database: complete** (2026-05-18) +Next: Stage 4 — Vault Encryption ## Project Roadmap @@ -25,10 +25,13 @@ memory in `~/.pyra/memory/`, and hard security boundaries around the vault. - Chat session: AI tool-use loop (up to 10 iterations), approval gate, plugin slash commands - CLI: `pyra plugin list/install/enable/disable/setup`, `pyra daemon *` stubs -### Stage 3 — Memory Database (next) -Replace the flat `.md` file scanner with SQLite + FTS5 for fast full-text search. -Schema designed to add a vector column later for semantic (embedding-based) search. -Backwards-compatible: existing `.md` memory files are migrated on first run. +### Stage 3 — Memory Database ✅ COMPLETE +- `src/pyra/memory/database.py`: SQLite + FTS5 via `memory_meta` + `memory_fts` tables +- `memory_meta` columns: `path`, `category`, `size_bytes`, `modified`, `summary`, `keywords`, `embedding BLOB` (reserved for Stage 8) +- `list_memories()` queries DB; `lookup_memories()` uses FTS5 with JSON-index fallback +- `write_memory()` / `append_memory()` upsert to DB on every write +- `bootstrap()` calls `init_db()` + `migrate_from_files()` (one-shot migration of existing `.md` files) +- `.md` files remain the canonical store; DB is the search index ### Stage 4 — Vault Encryption Encrypt `~/.pyra/vault/secrets/` using `age` (or GPG fallback). Pyra decrypts in memory @@ -97,9 +100,10 @@ the vault under namespaced keys (`plugin:{name}:{key}`). | `chat/session.py` | prompt_toolkit REPL loop, AI tool-use loop, plugin slash commands | | `chat/renderer.py` | Streaming + non-streaming markdown via rich, injection warning panel | | `chat/history.py` | Conversation list, token budget trimming, tool message support | -| `memory/reader.py` | `list_memories()`, `read_memory()`, `load_context_for_session()` | -| `memory/writer.py` | `write_memory()`, `append_memory()` — relative names only, no traversal | -| `memory/index.py` | Auto-regenerate `MEMORY_INDEX.md` on every write | +| `memory/database.py` | SQLite+FTS5 — `init_db()`, `upsert()`, `remove()`, `search()`, `list_all()`, `migrate_from_files()` | +| `memory/reader.py` | `list_memories()` (DB-backed), `read_memory()`, `lookup_memories()` (FTS5), `load_context_for_session()` | +| `memory/writer.py` | `write_memory()`, `append_memory()` — writes file + upserts to DB | +| `memory/index.py` | Auto-regenerate `MEMORY_INDEX.md` + `memory_index.json` on every write | | `vault/reader.py` | `get_key(key)` — sole accessor of `vault/secrets/api_keys.json` | | `vault/writer.py` | `set_key()`, `delete_key()` — only called from setup wizard + plugin setup | | `security/boundaries.py` | `assert_safe_path()`, `check_vault_lock()`, `BLOCKED_PREFIXES` | @@ -368,12 +372,24 @@ Dataclass: `InjectionWarning(pattern_label: str, matched_text: str)` | `set_key` | `vault.writer` | `(provider_id: str, api_key: str) -> None` | Stores or overwrites a key in the vault | | `delete_key` | `vault.writer` | `(provider_id: str) -> bool` | Removes a key; returns `True` if it existed | +#### `memory.database` + +| Function | Signature | Purpose | +|----------|-----------|---------| +| `init_db` | `() -> None` | Creates `memory.db` with `memory_meta` + `memory_fts` tables; chmod 600 | +| `upsert` | `(path, *, content, category, size_bytes, modified, summary, keywords) -> None` | Insert or replace one entry in both tables | +| `remove` | `(path: str) -> None` | Delete entry from both tables | +| `search` | `(query: str, limit: int = 20) -> list[dict]` | FTS5 MATCH search; returns `[{file, summary, keywords, snippet}]` | +| `list_all` | `() -> list[dict]` | All rows from `memory_meta` ordered by path | +| `migrate_from_files` | `() -> None` | One-shot: populate DB from existing `.md` files if DB is empty | + #### `memory.reader` | Function | Signature | Purpose | |----------|-----------|---------| -| `list_memories` | `() -> list[MemoryFile]` | Scans `~/.pyra/memory/**/*.md`; each entry is a `MemoryFile` dataclass | +| `list_memories` | `() -> list[MemoryFile]` | Queries DB (`memory_meta`); falls back to file scan if DB empty | | `read_memory` | `(name: str) -> str` | Reads memory file by relative path; validates against vault/traversal | +| `lookup_memories` | `(query: str) -> list[dict]` | FTS5 full-text search; falls back to JSON index substring search | | `load_context_for_session` | `() -> str` | Concatenates all memory files into a system-prompt block | Dataclass: `MemoryFile(name, path, category, size_bytes, modified)` @@ -382,14 +398,14 @@ Dataclass: `MemoryFile(name, path, category, size_bytes, modified)` | Function | Signature | Purpose | |----------|-----------|---------| -| `write_memory` | `(name: str, content: str) -> Path` | Creates/overwrites a memory `.md` file, updates index | -| `append_memory` | `(name: str, content: str) -> Path` | Appends to a memory file (creates if missing), updates index | +| `write_memory` | `(name: str, content: str, summary: str, keywords: list[str]) -> Path` | Creates/overwrites a memory `.md` file, updates index and DB | +| `append_memory` | `(name: str, content: str) -> Path` | Appends to a memory file (creates if missing), updates index and DB | #### `memory.index` | Function | Signature | Purpose | |----------|-----------|---------| -| `update_index` | `() -> None` | Regenerates `MEMORY_INDEX.md` — called automatically by writer functions | +| `update_index` | `() -> None` | Regenerates `MEMORY_INDEX.md` and `memory_index.json` — called automatically by writer functions | #### `setup.providers`