# Architecture How `recoll-modern` is built today. Update in the same commit as any structural change. ## Overview **Nothing is built yet.** This file records intent, the design questions that are now closed, and the Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing code. The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern web UI and (b) an MCP server, running in Docker, with several independent collections such as `techbooks` and `cookbooks`. ## Map | Path | Responsibility | |---|---| | `AGENTS.md` | Agent rules; single source of truth | | `PRODUCT.md` | Durable product context for the `impeccable` skill | | `docs/specs/`, `docs/plans/` | Approved specs and implementation plans | No source directories exist yet. Add rows here as they appear. ## Delivery order The product is four specs, built in order. Each gets its own spec, plan and review cycle. | Spec | Scope | |---|---| | 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication | | 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` | | 003 MCP server | Thin wrapper over 001's HTTP API | | 004 Multi-user management | Roles, user CRUD, per-collection permissions | ## Intended shape - **One** Recoll index for all documents, not one per collection. Collection membership is per-file metadata supplied by the service at index time, not directory layout, so two files in the same directory can belong to different collections and the documents are never modified. - A Python service owns Recoll through its native Python bindings (`from recoll import recoll`). - SQLite holds collections, membership, indexing jobs, users and sessions. - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge. - A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage. ## Verified Recoll mechanics Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted **read-only**. Re-verify if the Recoll major version changes. - **Documents are never written to.** Membership reaches the index through `metadatacmds` in `recoll.conf` (`metadatacmds = ; collection = %f`), which Recoll runs per file at index time to fetch external metadata. The helper answers from our own store. Indexing a read-only mount succeeds; `touch` and `setfattr` on it both fail, as they must. - **Extended attributes are NOT usable.** An earlier design carried membership in a `user.collection` xattr. It works only when the documents are writable, so it is superseded — see `docs/DECISIONS.md`. - **Multi-membership works.** A space-separated helper response puts one file in several collections. An `OR` across collections returns it **once**, in a single coherently ranked set: Xapian does the ranking, so no application-side merging is needed. - **Membership changes need a forced per-file reindex.** Neither a plain incremental run nor `recollindex -i ` notices a change that did not touch the file — Recoll's change signature is size + mtime (observed as `sig` = `''`), and a membership change alters neither. `recollindex -i -Z ` does pick it up, at roughly 15ms per file, and accepts several paths. - **The field value is tokenised and matched case-insensitively.** A value of `cook-books` is matched by `collection:cook` *and* `collection:books`; `TechBooks` by `collection:techbooks`. Collection identifiers must therefore be opaque single tokens with no separators. Tested clean with `^c[0-9a-f]+$`. - **Recoll has no content hash.** `md5` is empty and `sig` is only size + mtime, so document identity by content is ours to compute and store. - **An unscoped query returns every document**, including ones belonging to no collection. Recoll will not enforce collection scoping; the API layer must. - **The helper must start fast.** Recoll spawns it once per file per index run. Over 400 files: 0.09s with no helper, 0.13s for a shell+awk helper, 0.82s for a `python3` helper — interpreter startup dominates. The helper reads a flat map file exported from SQLite; it is not a Python process. ## Open design questions Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning: 1. ~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by `metadatacmds` at index time; human names live only in SQLite; documents are never written to. 2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's. 3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite. 4. ~~Stack~~ — Python + FastAPI, managed with `uv`. 5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004. 6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate endpoints. Still open, to be closed by the remaining sections of the 001 design: - Component boundaries and the SQLite schema. - API surface, pagination, and error semantics. - Test strategy, including how tests skip when the container is down. ## External dependencies | Dependency | What breaks without it | |---|---| | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. | | Docker + compose | The only supported way to run the service. | | Document volumes | Nothing to index; collections resolve to empty. | | A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. | ## Invariants - A query never silently spans collections: the scope is an explicit, required parameter, and the API refuses a search that does not name one. - Collection identity in the index is an opaque single-token ID, never a user-supplied name, so collections cannot bleed into one another and renaming never touches the index. - The service never writes to the documents it indexes. The document volume is treated as read-only even when it is not, and the deployment mounts it `:ro`. - Document identity is the SHA-256 of file content, not its path or name: two files with the same name and different content are two documents, and one file reachable by several paths is one document. - Indexed content is never written into the repository. ## Known rough edges None yet — nothing has been built. Cross-linked backlog: `docs/TODO.md`.