recoll-modern/docs/ARCHITECTURE.md

Architecture

How recoll-modern is built today. Update in the same commit as any structural change.

Overview

The Python service scaffold, SQLite schema, collection and membership repositories, document discovery/identity modules, and durable job worker are built. The API and Recoll adapter remain intended structure. Recoll behaviour below was verified against a real container.

The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern web UI and (b) an MCP server, running in Docker, with several independent collections such as techbooks and cookbooks.

Map

Path	Responsibility
AGENTS.md	Agent rules; single source of truth
PRODUCT.md	Durable product context for the impeccable skill
docs/specs/, docs/plans/	Approved specs and implementation plans
src/recoll_modern/	Python service package; configuration, application factory and CLI boundary
src/recoll_modern/indexing/	Root-relative path policy, read-only discovery, stable file hashing
src/recoll_modern/storage/	SQLite migrations, collection metadata and content-hash identity
src/recoll_modern/domain/membership.py	Validates sources and applies direct additions and rule changes
src/recoll_modern/indexing/reconcile.py	Refreshes persistent directory rules while retaining excluded or unreadable files as present
src/recoll_modern/indexing/worker.py	Claims one durable job at a time and runs blocking handlers off the event loop
src/recoll_modern/indexing/handlers.py	Scans membership sources before strict-mode mutations and records file errors
tests/	Automated service tests

Discovery produces content hashes and root-relative paths without writing to the document volume. The document repository records these in one transaction, chooses the smallest live path as canonical, and marks paths missing only when a caller explicitly reconciles them.

Delivery order

The product is five specs, built in order. Each gets its own spec, plan and review cycle. Spec 005 is a future distribution track and starts only after the hosted product through 004 is delivered.

Spec	Scope
001 Search engine + API	Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication
002 Web UI	Blocked on /impeccable init replacing the PRODUCT.md stub, then DESIGN.md
003 MCP server	Thin wrapper over 001's HTTP API
004 Multi-user management	Roles, user CRUD, per-collection permissions
005 Cross-platform desktop distribution	Installable shell reusing the web UI and launching the API plus Recoll as bundled sidecars

Intended shape

• One Recoll index for all documents, not one per collection. Collection membership is per-file metadata supplied by the service at index time, not directory layout, so two files in the same directory can belong to different collections and the documents are never modified.
• A Python service owns Recoll through its native Python bindings (from recoll import recoll).
• SQLite holds collections, membership, indexing jobs, users and sessions.
• An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
• A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
• A packaging seam for a future desktop shell: application logic does not depend on Docker, state paths and listen addresses are configurable, process startup/shutdown is explicit, and every UI operation crosses the same HTTP API. Docker remains the only supported distribution through spec 004.

Component boundaries

Module	Responsibility
api/	HTTP transport, validation, auth dependencies and serialization
domain/	Framework-independent collection, document, membership, rule and job behavior
storage/	SQLite migrations, transactions and repositories
indexing/	Discovery, hashing, directory-rule reconciliation, map export and queue worker
engine/	The search/index Protocol, fake implementation and Recoll adapter
app.py	Configuration, dependency wiring and lifecycle

Only engine/ may import Recoll or invoke recollindex. Membership changes first establish desired state in SQLite, atomically export the helper's flat map, then force-index affected canonical paths. Engine retries are therefore idempotent. Persistent directory rules reconcile at startup and on an explicit reindex, not through live filesystem monitoring.

Storage model

collections, documents and document_paths hold identity and canonical paths. Membership provenance is normalized: direct_memberships holds explicit files and snapshot expansions; directory_rules, directory_rule_exclusions and rule_memberships hold persistent additions; and membership_exclusions holds per-file overrides against rule-derived membership. Effective membership is direct membership union non-excluded rule membership, so direct additions remain additive.

jobs and job_file_errors make work and partial failures durable. At startup, the worker requeues well-formed interrupted jobs, clears errors from the prior attempt, and fails unreadable requests. users and hashed-token sessions provide authentication. schema_migrations is forward-only. The exact columns and constraints are the contract in docs/specs/001-search-engine-api.md.

HTTP boundary

The resource groups are auth, health, read-only file browsing, collections and membership sources, directory rules, scoped search, document text/original retrieval, and indexing jobs. Search callers select returned fields; only document_id is unconditional, and inline full text is bounded. Every route, cookie, redirect, OpenAPI URL and generated link supports a configured mount prefix. Forwarded prefix and origin data are used only from trusted proxies. The exact endpoints and error codes are the contract in docs/specs/001-search-engine-api.md.

Verified Recoll mechanics

Established empirically on 2026-09-18 in a throwaway debian:trixie-slim container running Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted read-only. Re-verify if the Recoll major version changes.

• Documents are never written to. Membership reaches the index through metadatacmds in recoll.conf (metadatacmds = ; collection = <helper> %f), which Recoll runs per file at index time to fetch external metadata. The helper answers from our own store. Indexing a read-only mount succeeds; touch and setfattr on it both fail, as they must.
• Extended attributes are NOT usable. An earlier design carried membership in a user.collection xattr. It works only when the documents are writable, so it is superseded — see docs/DECISIONS.md.
• Multi-membership works. A space-separated helper response puts one file in several collections. An OR across collections returns it once, in a single coherently ranked set: Xapian does the ranking, so no application-side merging is needed.
• Membership changes need a forced per-file reindex. Neither a plain incremental run nor recollindex -i <file> notices a change that did not touch the file — Recoll's change signature is size + mtime (observed as sig = '<size><mtime>'), and a membership change alters neither. recollindex -i -Z <file> does pick it up, at roughly 15ms per file, and accepts several paths.
• The field value is tokenised and matched case-insensitively. A value of cook-books is matched by collection:cook and collection:books; TechBooks by collection:techbooks. Collection identifiers must therefore be opaque single tokens with no separators. Tested clean with ^c[0-9a-f]+$.
• Recoll has no content hash. md5 is empty and sig is only size + mtime, so document identity by content is ours to compute and store.
• An unscoped query returns every document, including ones belonging to no collection. Recoll will not enforce collection scoping; the API layer must.
• The helper must start fast. Recoll spawns it once per file per index run. Over 400 files: 0.09s with no helper, 0.13s for a shell+awk helper, 0.82s for a python3 helper — interpreter startup dominates. The helper reads a flat map file exported from SQLite; it is not a Python process.

Open design questions

Closed by the 001 brainstorm (2026-09-18) — see docs/DECISIONS.md for the reasoning:

1. Collection isolation — one index; membership is an opaque collection ID supplied to Recoll by metadatacmds at index time; human names live only in SQLite; documents are never written to.
2. Cross-collection search — allowed, but only when explicitly requested. Ranking is Xapian's.
3. Indexing trigger — an in-process job queue, backed by SQLite.
4. Stack — Python + FastAPI, managed with uv.
5. Auth and exposure — real authentication in 001; user administration deferred to 004.
6. Result presentation — callers select result fields; metadata plus snippets is the default, inline text is bounded, and full text and file download are separate endpoints.

The remaining 001 design questions — component boundaries, schema, HTTP surface, pagination, error semantics and test strategy — are closed in docs/specs/001-search-engine-api.md. That spec remains draft until maintainer sign-off.

External dependencies

Dependency	What breaks without it
Recoll (in-container)	All indexing and search. Not installed on the host — see AGENTS.md Rules.
Docker + compose	The only supported way to run the service.
Document volumes	Nothing to index; collections resolve to empty.
A writable volume for the index and SQLite	All state. The document volume itself may be mounted read-only.

Docker is the only supported deployment for specs 001–004, not a permanent application-layer dependency. Spec 005 will design cross-platform packaging of the web UI, API service, Recoll and its document filters. A Tauri-style shell with bundled sidecars is the intended approach; choosing Tauri itself and resolving per-platform Recoll licensing, signing, updates and filter coverage are decisions for that spec.

Invariants

• A query never silently spans collections: the scope is an explicit, required parameter, and the API refuses a search that does not name one.
• Collection identity in the index is an opaque single-token ID, never a user-supplied name, so collections cannot bleed into one another and renaming never touches the index.
• The service never writes to the documents it indexes. The document volume is treated as read-only even when it is not, and the deployment mounts it :ro.
• Document identity is the SHA-256 of file content, not its path or name: two files with the same name and different content are two documents, and one file reachable by several paths is one document.
• Indexed content is never written into the repository.

Known rough edges

None yet — nothing has been built. Cross-linked backlog: docs/TODO.md.