recoll-modern/docs/ARCHITECTURE.md

# Architecture

How `recoll-modern` is built today. Update in the same commit as any structural change.

## Overview

**Nothing is built yet.** This file records intent, the design questions that are now closed, and the
Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing
code.

The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
web UI and (b) an MCP server, running in Docker, with several independent collections such as
`techbooks` and `cookbooks`.

## Map

| Path | Responsibility |
|---|---|
| `AGENTS.md` | Agent rules; single source of truth |
| `PRODUCT.md` | Durable product context for the `impeccable` skill |
| `docs/specs/`, `docs/plans/` | Approved specs and implementation plans |

No source directories exist yet. Add rows here as they appear.

## Delivery order

The product is four specs, built in order. Each gets its own spec, plan and review cycle.

| Spec | Scope |
|---|---|
| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication |
| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` |
| 003 MCP server | Thin wrapper over 001's HTTP API |
| 004 Multi-user management | Roles, user CRUD, per-collection permissions |

## Intended shape

- **One** Recoll index for all documents, not one per collection. Collection membership is per-file
  metadata supplied by the service at index time, not directory layout, so two files in the same
  directory can belong to different collections and the documents are never modified.
- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`).
- SQLite holds collections, membership, indexing jobs, users and sessions.
- An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.

### Component boundaries

| Module | Responsibility |
|---|---|
| `api/` | HTTP transport, validation, auth dependencies and serialization |
| `domain/` | Framework-independent collection, document, membership, rule and job behavior |
| `storage/` | SQLite migrations, transactions and repositories |
| `indexing/` | Discovery, hashing, directory-rule reconciliation, map export and queue worker |
| `engine/` | The search/index Protocol, fake implementation and Recoll adapter |
| `app.py` | Configuration, dependency wiring and lifecycle |

Only `engine/` may import Recoll or invoke `recollindex`. Membership changes first establish desired
state in SQLite, atomically export the helper's flat map, then force-index affected canonical paths.
Engine retries are therefore idempotent. Persistent directory rules reconcile at startup and on an
explicit reindex, not through live filesystem monitoring.

### Storage model

`collections`, `documents` and `document_paths` hold identity and canonical paths. Membership provenance
is normalized: `direct_memberships` holds explicit files and snapshot expansions; `directory_rules`,
`directory_rule_exclusions` and `rule_memberships` hold persistent additions; and
`membership_exclusions` holds per-file overrides against rule-derived membership. Effective membership
is direct membership union non-excluded rule membership, so direct additions remain additive.

`jobs` and `job_file_errors` make work and partial failures durable. `users` and hashed-token `sessions`
provide authentication. `schema_migrations` is forward-only. The exact columns and constraints are the
contract in `docs/specs/001-search-engine-api.md`.

### HTTP boundary

The resource groups are auth, health, read-only file browsing, collections and membership sources,
directory rules, scoped search, document text/original retrieval, and indexing jobs. Search callers
select returned fields; only `document_id` is unconditional, and inline full text is bounded. Every
route, cookie, redirect, OpenAPI URL and generated link supports a configured mount prefix. Forwarded
prefix and origin data are used only from trusted proxies. The exact endpoints and error codes are the
contract in `docs/specs/001-search-engine-api.md`.

## Verified Recoll mechanics

Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running
Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted **read-only**. Re-verify if the
Recoll major version changes.

- **Documents are never written to.** Membership reaches the index through `metadatacmds` in
  `recoll.conf` (`metadatacmds = ; collection = <helper> %f`), which Recoll runs per file at index time
  to fetch external metadata. The helper answers from our own store. Indexing a read-only mount
  succeeds; `touch` and `setfattr` on it both fail, as they must.
- **Extended attributes are NOT usable.** An earlier design carried membership in a `user.collection`
  xattr. It works only when the documents are writable, so it is superseded — see `docs/DECISIONS.md`.
- **Multi-membership works.** A space-separated helper response puts one file in several collections.
  An `OR` across collections returns it **once**, in a single coherently ranked set: Xapian does the
  ranking, so no application-side merging is needed.
- **Membership changes need a forced per-file reindex.** Neither a plain incremental run nor
  `recollindex -i <file>` notices a change that did not touch the file — Recoll's change signature is
  size + mtime (observed as `sig` = `'<size><mtime>'`), and a membership change alters neither.
  `recollindex -i -Z <file>` does pick it up, at roughly 15ms per file, and accepts several paths.
- **The field value is tokenised and matched case-insensitively.** A value of `cook-books` is matched by
  `collection:cook` *and* `collection:books`; `TechBooks` by `collection:techbooks`. Collection
  identifiers must therefore be opaque single tokens with no separators. Tested clean with `^c[0-9a-f]+$`.
- **Recoll has no content hash.** `md5` is empty and `sig` is only size + mtime, so document identity by
  content is ours to compute and store.
- **An unscoped query returns every document**, including ones belonging to no collection. Recoll will
  not enforce collection scoping; the API layer must.
- **The helper must start fast.** Recoll spawns it once per file per index run. Over 400 files: 0.09s
  with no helper, 0.13s for a shell+awk helper, 0.82s for a `python3` helper — interpreter startup
  dominates. The helper reads a flat map file exported from SQLite; it is not a Python process.

## Open design questions

Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning:

1. ~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by
   `metadatacmds` at index time; human names live only in SQLite; documents are never written to.
2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's.
3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite.
4. ~~Stack~~ — Python + FastAPI, managed with `uv`.
5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004.
6. ~~Result presentation~~ — callers select result fields; metadata plus snippets is the default, inline
   text is bounded, and full text and file download are separate endpoints.

The remaining 001 design questions — component boundaries, schema, HTTP surface, pagination, error
semantics and test strategy — are closed in `docs/specs/001-search-engine-api.md`. That spec remains
draft until maintainer sign-off.

## External dependencies

| Dependency | What breaks without it |
|---|---|
| Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. |
| Docker + compose | The only supported way to run the service. |
| Document volumes | Nothing to index; collections resolve to empty. |
| A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. |

## Invariants

- A query never silently spans collections: the scope is an explicit, required parameter, and the API
  refuses a search that does not name one.
- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so
  collections cannot bleed into one another and renaming never touches the index.
- The service never writes to the documents it indexes. The document volume is treated as read-only
  even when it is not, and the deployment mounts it `:ro`.
- Document identity is the SHA-256 of file content, not its path or name: two files with the same name
  and different content are two documents, and one file reachable by several paths is one document.
- Indexed content is never written into the repository.

## Known rough edges

None yet — nothing has been built. Cross-linked backlog: `docs/TODO.md`.