commit 145523e918b986aa3cb6478a954dd9ea6bbf7bd5 Author: Lucas Winther Date: Mon Sep 28 17:35:00 2026 +0200 docs: define local RAG desktop application spec diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..b5bf97c --- /dev/null +++ b/SPEC.md @@ -0,0 +1,142 @@ +# Local RAG — Desktop search, collections, MCP, and updates + +Status: draft for review · 2026-09-28 + +This is a new application in this repository. `/home/user/recoll-modern` is a reference for proven Recoll behavior and possible code reuse, subject to ownership and licensing review. Its numbered specs, server API, Docker deployment, and migration history are not contracts for this app. This spec defines the first desktop product independently. + +## Goal and release criteria + +One installed app lets a local user add folders, organize documents into collections, search within an explicit collection scope, read extracted text, and open originals. The user can temporarily start a local MCP endpoint for agents. Installed builds receive signed updates. No terminal, Docker daemon, or separately installed Python or Recoll is required. + +Linux x86_64 and Windows x86_64 are the first release targets. Both must pass the same user flows before this desktop release is called complete. macOS is a later target, preceded by a packaging spike. + +Acceptance checks: + +1. On clean Linux and Windows machines: install, launch, add a folder, create a collection, add files or a folder to it, search and see snippets, read extracted text, open an original, and reveal it in the file manager. No terminal step is needed. +2. Start MCP displays a copyable, version-checked configuration for each supported client. After configuration, a client can list collections and search the same scoped documents as the UI. +3. Stop MCP or quit the app closes its listener and invalidates its bearer token. Every new start issues a new token. +4. A running older build detects a newer signed release and offers to apply it on restart after jobs finish. An unsigned or modified artifact is rejected. +5. Core invariants hold: documents are never written to, identity is by content hash, collection IDs are opaque, and every search names at least one collection. + +## Decisions + +| ID | Decision | +|---|---| +| D1 | Tauri v2 Rust shell. Rust owns process supervision, native dialogs, app lifecycle, and updates. Enable only the Tauri plugins and capabilities each flow needs. | +| D2 | Install and pin a stable Rust toolchain for each build environment; document `cargo` commands in this repository's `AGENTS.md`. | +| D3 | Linux x86_64 and Windows x86_64 first. Linux AppImage is the self-updating package; a `.deb` may be supplied with package-manager/manual upgrades. Windows uses a Tauri-supported signed updater artifact, selected by the Windows packaging spike. macOS follows later. | +| D4 | Bun for JavaScript package management and scripts; Vite, TypeScript, and React for the UI. Commit `bun.lock`. Build static assets served by the Python service. | +| D5 | Multiple library roots from the first schema version, with root-aware path and membership-source provenance. No legacy single-root migration is needed. | +| D6 | A fresh 256-bit bearer token for each MCP listener start. The UI listener accepts cookies only; the MCP listener accepts bearer tokens only. No persistent MCP token in this release. | +| D7 | Official Python `mcp` SDK using Streamable HTTP on a separate loopback listener in the service process. | +| D8 | Bundle a pinned python-build-standalone runtime, a relocatable locked environment, Recoll, its Python binding, Xapian, the C helper, and selected filters. This is conditional on the two-platform phase 0 spike. No runtime dependency is resolved through host `PATH`. | +| D9 | Gitea Actions builds and publishes Linux. A Windows runner (Gitea or another CI service) builds Windows and uploads to the same release. Both builds and signing must be automated before release. | +| D10 | License original project code under MIT **if the dependency and distribution review permits the intended packaging**. Recoll and Xapian retain their own GPL terms. The published bundle must not be described as wholly MIT; license notices, corresponding source, build scripts, and any required source offer are release gates. Existing `recoll-modern` code may be copied only after confirming rights to relicense it. If the planned bundle cannot meet the chosen license policy, resolve the architecture or license before publishing binaries. | + +Xapian currently describes itself as GPL v2+ ([Xapian licensing](https://trac.xapian.org/wiki/Licensing)); Recoll describes itself as GPL licensed ([Recoll](https://www.recoll.org/)). The boundary between separate programs and a combined work depends on how they communicate and are packaged ([GNU GPL FAQ](https://www.gnu.org/licenses/gpl-faq.en.html#MereAggregation)). D10 therefore records an intended project license, not a conclusion that one MIT notice covers the distributable. + +## Architecture and boundaries + +```text +Tauri Rust shell: window, single instance, dialogs, sidecar supervisor, updates + │ starts bundled service; loads local URL + ▼ +Python FastAPI service: static React UI + local API + durable worker + SQLite + │ │ optional listener + │ └── 127.0.0.1: MCP and read-only API + ▼ +Recoll adapter and indexer: bundled Recoll, Xapian, filters, C membership helper +``` + +The React UI calls this app's HTTP API and Tauri commands for native operations. MCP tools call the same service/domain operations used by HTTP routes; neither MCP nor React calls the Recoll adapter directly. Only `engine/` imports the Recoll binding or invokes `recollindex`. Rust holds no collection, indexing, or search logic. + +The main listener binds only to `127.0.0.1` on an OS-assigned port. The MCP listener is absent until requested. The app ignores proxy headers, has no password login, and serves the static UI. A server deployment is outside this release. + +Proposed files: `web/` (Bun/React/Vite), `desktop/src-tauri/` (Rust shell), `desktop/sidecars/` (per-target build scripts), and `src/local_rag/` with `api/`, `domain/`, `storage/`, `indexing/`, `engine/`, `mcp/`, and `static/`. The UI build output is packaged into `src/local_rag/static/`. + +## Process lifecycle and desktop authentication + +1. A second launch focuses the existing window. The shell chooses the app-data and log directories, generates a 32-byte launch secret, and starts the bundled service with desktop mode, loopback host, port `0`, state paths, and bundled binary/filter paths. The secret goes over a private stdin pipe, never argv or the environment. The shell owns the child process group on Linux and a job object on Windows. +2. After binding, the service emits one machine-readable stdout line: `{"event":"ready","ui_port":N,"version":"x.y.z","schema":K}`. Diagnostics go to rotating log files and stderr. A 30-second timeout shows a retry screen and the last 200 sanitized log lines. +3. The shell opens `/auth/desktop?token=` in the webview. The service validates and consumes the token, creates an expiring HttpOnly cookie session and CSRF token, then redirects to `/`. The token is never logged, is excluded from referrers, and cannot be replayed. There is no password login or seeded administrator. +4. The cookie is HttpOnly and scoped to the loopback UI origin. Unsafe UI requests require CSRF. The launch secret authenticates shell-only desktop control calls. Unauthenticated local processes get no access to document, collection, or control routes. +5. A sidecar crash shows a service-stopped screen with logs and a Restart action. At most three automatic retries occur in 60 seconds. There is no endless restart loop. +6. On quit the shell stops MCP, requests graceful worker shutdown, waits for the current file and SQLite close for up to 10 seconds, then terminates the process tree. Interrupted durable jobs recover on next launch. A Reveal logs menu item opens the log directory. + +## Library roots and document identity + +Library roots are absolute directories selected through the native picker. `library_roots` stores an opaque ID matching `^r[0-9a-f]+$`, canonical path, optional label, added time, and status (`ok`, `missing`, `unreadable`). Root addition rejects a duplicate, ancestor, descendant, or symlink alias of an existing root with `root_overlap`. Root paths are local configuration and are never written into document contents. + +Every path in storage and requests is `(root_id, relative_path)`. `document_paths` has a composite uniqueness constraint on those values and retains its content-hash document reference. Rules, rule exclusions, file sources, snapshots, job replay paths, and file errors carry root IDs from the first schema version. + +Direct membership sources retain which selected root and path contributed a document hash. Effective collection membership remains the union of all live direct and rule-derived sources for a content-hash document. If two roots contain identical bytes, they remain one document with multiple paths. Removing a root removes only its paths and sources; another root's path and sources remain. If the removed path was canonical, a surviving available path is promoted and reindexed. If none remains, the document becomes missing and drops out of search. Root removal never deletes a user file or the entire index. + +Canonical selection is deterministic across `(root_id, relative_path)` and prefers an available path. A missing or unreadable root retains its paths and membership provenance, but cannot supply search results, text, or originals until it returns. The UI displays that state. Rescan reconciles only an available root. It must not interpret an unplugged drive as an empty root and purge its documents. Recoll `topdirs` is generated from active roots; root addition/removal has a durable, retryable index job with an explicit stale-entry purge strategy verified against real Recoll. No silent whole-index rebuild is allowed. + +The per-root path policy uses no-follow opening, containment, and symlink checks. `GET /roots`, `POST /roots`, `PATCH /roots/{id}`, and `DELETE /roots/{id}` manage the local library. `GET /files`, membership source requests, rules, search paths, and retrieval links use root IDs where paths are exposed. The native picker is the UI's folder-entry method; the API still validates every submitted absolute root path, because an HTTP request alone cannot prove picker use. + +Collections have opaque IDs and unique normalized names. Membership sources are individual files, one-time directory snapshots, and persistent directory rules with exclusions. A document may belong to several collections. Direct sources remain additive; a per-file exclusion suppresses rule-derived membership only. Collection deletion removes its metadata and membership through a durable cleanup job, never user files. Jobs persist state, progress, per-file errors, and retry data across restart. + +The local API provides collections CRUD; membership and rule CRUD; root and file browsing; jobs, reindex and retry; scoped search; extracted text; ranged original retrieval; health; and authenticated OpenAPI. `GET /search` requires a nonempty query and at least one collection ID, combines scopes into one ranked Recoll query, then verifies every hit against current SQLite membership and path state. It supports repeated field selection, `limit` and nonnegative `offset`, with bounded inline text and an approximate total when the engine estimates it. Text and original-file routes resolve a current canonical path under an available root. Errors use stable codes in a JSON envelope; the phase 1 plan must write the endpoint and error matrix before implementation. + +## React UI + +Run `/impeccable init` before the first UI design to create `PRODUCT.md`. Use `shape` before each screen, write `DESIGN.md` with the first build, and use `harden` before release. Bun installs and runs the UI toolchain with the committed lockfile. + +Minimum screens: + +- **First run:** add a folder, create a collection, add files or a folder, then search without reading documentation. +- **Search:** required collection picker, query, field chips, snippets, paging, detail text, Open original, and Reveal in folder. An empty scope cannot submit a query. +- **Collections:** create, rename, delete, list documents with direct/rule provenance, edit rules and exclusions, and add sources using a picker constrained to roots. +- **Library:** root status, add/remove, and Rescan with clear handling for disconnected drives. +- **Activity:** durable jobs, progress, failures, per-file errors, and retry. +- **Agents:** MCP start/stop, status, connected-client count, token visibility/copy, and checked client configurations. +- **Settings/About:** version, update channel, check for updates, logs, license notices, and source information. + +Native Open original and Reveal in folder resolve the canonical root/path in the service, then use a narrowly scoped Tauri command. No arbitrary path received from web content is opened without resolving it against a registered root and a current document. + +## Temporary MCP and read-only agent API + +MCP is off by default. Start binds `127.0.0.1:8765`, or an OS-assigned free port if occupied. Each start creates a new 256-bit token. Stop, quit, or the selected idle timeout (none by default; 15 minutes, 1 hour, or 4 hours) closes the listener and erases the token. An optional Start MCP when app opens preference may start the listener, but still creates a fresh token; copied client configuration must be refreshed. Stable agent credentials are outside this release. + +The listener exposes Streamable HTTP at `/mcp` and authenticated, read-only `/api/*` routes plus `/openapi.json`. Every request checks `Authorization: Bearer`, an exact loopback Host with the listener's port, and rejects any browser `Origin`. Cookies never authenticate this listener. The UI listener never accepts bearer tokens. The SDK's session-manager lifespan is owned by the hosting ASGI app. MCP tool handlers reuse the existing domain services, including scoped search checks. + +| Tool | Behavior | +|---|---| +| `list_collections` | IDs, names, document counts | +| `search` | Required nonempty collection IDs, query, fields, `limit` at most 50, nonnegative `offset`; same hits and scope as `GET /search` | +| `get_document_text` | Paged extracted text with the API's bounds | +| `get_document_info` | Metadata and effective memberships; define and test a shared domain operation and HTTP representation | +| `list_collection_documents` | Paged document list and provenance | + +No write tools ship initially. Tool responses may show root-relative paths for added roots, never paths outside registered roots. The Agents screen generates Claude Code, Codex, Antigravity, and generic configurations. Check exact syntax against current client documentation at implementation time and record the checked versions in `docs/DECISIONS.md`. + +## Bundling, update, and release + +For each first-release target, a build script pins Python 3.13, Recoll, Xapian, filters, and the C helper; builds the Recoll binding against that bundled Python; and assembles a relocatable runtime from `uv.lock`. It must index, search, and extract text from a generated corpus after installation at a different path, with no host Python, Recoll, or filter lookup. Windows Recoll and binding packaging is a phase 0 go/no-go result, not an assumption borrowed from Debian. + +The Tauri updater verifies its mandatory Ed25519 signature using a public key compiled into the shell. The private key exists only in release CI. A signed `latest.json`-format manifest names a signed artifact per supported target. Linux self-update uses the AppImage artifact; `.deb` upgrades follow the package's documented installation path. Windows uses the updater-supported installer artifact proven by the spike. The app checks at launch and every six hours, downloads in the background, and offers Restart to update. Jobs block application of an update; the app never restarts itself during a job. + +Gitea is the initial release forge. A build-time release source may also resolve GitHub, GitLab, or a direct HTTPS manifest. Stable and beta channels have separate manifest selection, so a latest-stable endpoint is never assumed to discover prereleases. Network/parse failures are logged and displayed as a quiet unavailable state in Settings. An invalid signature is a visible update failure. The user cannot disable signature verification. + +Before a migration the service creates an SQLite-consistent backup, including pending WAL state, and retains the last three versioned backups. Migrations are forward-only. A failure shows recovery instructions; rollback behavior is specified per package and tested. A release records its Recoll/Xapian major versions. A change requiring an index rebuild asks the user first; until then, use the old index read-only only if compatibility is demonstrated. + +Release CI builds and signs both platforms, generates the manifest, runs installed-bundle smoke tests, and publishes artifacts plus third-party license notices and corresponding source material. The legal/distribution review in D10 must approve the binary bundle before the first public release. + +## Phases and verification + +Each phase gets an unnumbered plan under `docs/plans/desktop-phase-*.md` and review. Stop and re-plan if it grows materially beyond its plan. + +| Phase | Deliverable | Exit check | +|---|---|---| +| 0 | Throwaway Linux and Windows packaging spikes: bundled Python/Recoll/Xapian/helper/filters, installed and relocated; minimal Tauri launch | Generated text and PDF corpus indexed, searched, and extracted on clean machines; decide D8 and Windows packaging from evidence | +| 1 | Root-aware storage/API, indexing, jobs, bootstrap auth | Root overlap, duplicate content, unplugged root, removal, job recovery, and bootstrap auth tests pass | +| 2 | Tauri lifecycle and supervision on both targets | Launch, handshake, UI load, crash/retry, quit, and no child processes left | +| 3 | Bun/React UI through impeccable | Acceptance check 1 on both targets | +| 4 | MCP listener and Agents screen | Checks 2–3, including bearer, Host, Origin, replay, and required scope | +| 5 | Signed updater and release CI | Check 4 against a local release server and a real Gitea release; D10 release gate complete | +| 6 | macOS spike and packaging | Checks 1–4 on macOS before adding it to releases | + +Python tests use the deterministic engine fake for routine tests and a real bundled Recoll run for integration. Rust tests cover handshake parsing, supervisor retry limits, manifest resolution, and channel selection. Bun runs type checking, UI tests for the first-run and explicit-scope flows, and the production build. Desktop end-to-end tests use the installed app, temporary state, generated corpus, and clean shutdown. Update tests verify a valid higher-version artifact, tamper rejection, backup, and failed-migration recovery. + +During implementation, create and maintain `docs/ARCHITECTURE.md` and append choices to `docs/DECISIONS.md`; document Rust/Bun commands and platform gates in `AGENTS.md`; create `PRODUCT.md` through impeccable; and record deferred hot UI updates, MCP write tools, filesystem watching, and OCR in `docs/TODO.md`.