Archived

This repository has been archived on 2026-10-08. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

Lucas WintherandClaude Opus 5 883db6a59d docs: supersede the xattr membership design for read-only documents

The service must work with read-only access to the documents it indexes.
Writing a user.collection extended attribute needs write access, so the
previous membership design fails on a :ro mount; setfattr returns "Read-only
file system". Verified in the spike container.

Membership now reaches Recoll through metadatacmds in recoll.conf, a documented
facility for external metadata: Recoll runs a helper per file at index time and
the helper answers from our own store. Verified end to end against a read-only
bind mount. Everything previously established about opaque collection IDs,
multi-membership and single-ranked-set OR queries survives unchanged, because
only the transport differs. SQLite becomes the only store of membership rather
than the authoritative one of two, so drift is impossible and the reconcile path
is gone. Extended attributes are no longer a deployment requirement.

Document identity becomes the SHA-256 of file content. Files sharing a name but
differing in content are separate documents; one file reachable by several paths
is a single document with one canonical path, and only that path is reported as
belonging to a collection. Recoll cannot help here: md5 is empty and sig is only
size plus mtime.

Two measured constraints are recorded. A membership change is invisible to both
a plain incremental run and recollindex -i, because the change signature is size
plus mtime; recollindex -i -Z on the affected path picks it up in about 15ms.
And the helper is spawned once per file per index run, so it must start fast: a
400-file index took 0.09s with no helper, 0.13s with shell and awk, and 0.82s
with python3.

The superseded entry in docs/DECISIONS.md is left intact, per that file's
append-only rule, and the new entry links back to it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-18 17:06:29 +02:00

6.6 KiB

Raw Blame History

Architecture

How recoll-modern is built today. Update in the same commit as any structural change.

Overview

Nothing is built yet. This file records intent, the design questions that are now closed, and the Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing code.

The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern web UI and (b) an MCP server, running in Docker, with several independent collections such as techbooks and cookbooks.

Map

Path	Responsibility
`AGENTS.md`	Agent rules; single source of truth
`PRODUCT.md`	Durable product context for the `impeccable` skill
`docs/specs/`, `docs/plans/`	Approved specs and implementation plans

No source directories exist yet. Add rows here as they appear.

Delivery order

The product is four specs, built in order. Each gets its own spec, plan and review cycle.

Spec	Scope
001 Search engine + API	Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication
002 Web UI	Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md`
003 MCP server	Thin wrapper over 001's HTTP API
004 Multi-user management	Roles, user CRUD, per-collection permissions

Intended shape

One Recoll index for all documents, not one per collection. Collection membership is per-file metadata supplied by the service at index time, not directory layout, so two files in the same directory can belong to different collections and the documents are never modified.
A Python service owns Recoll through its native Python bindings (from recoll import recoll).
SQLite holds collections, membership, indexing jobs, users and sessions.
An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.

Verified Recoll mechanics

Established empirically on 2026-09-18 in a throwaway debian:trixie-slim container running Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted read-only. Re-verify if the Recoll major version changes.

Documents are never written to. Membership reaches the index through metadatacmds in recoll.conf (metadatacmds = ; collection = <helper> %f), which Recoll runs per file at index time to fetch external metadata. The helper answers from our own store. Indexing a read-only mount succeeds; touch and setfattr on it both fail, as they must.
Extended attributes are NOT usable. An earlier design carried membership in a user.collection xattr. It works only when the documents are writable, so it is superseded — see docs/DECISIONS.md.
Multi-membership works. A space-separated helper response puts one file in several collections. An OR across collections returns it once, in a single coherently ranked set: Xapian does the ranking, so no application-side merging is needed.
Membership changes need a forced per-file reindex. Neither a plain incremental run nor recollindex -i <file> notices a change that did not touch the file — Recoll's change signature is size + mtime (observed as sig = '<size><mtime>'), and a membership change alters neither. recollindex -i -Z <file> does pick it up, at roughly 15ms per file, and accepts several paths.
The field value is tokenised and matched case-insensitively. A value of cook-books is matched by collection:cook and collection:books; TechBooks by collection:techbooks. Collection identifiers must therefore be opaque single tokens with no separators. Tested clean with ^c[0-9a-f]+$.
Recoll has no content hash. md5 is empty and sig is only size + mtime, so document identity by content is ours to compute and store.
An unscoped query returns every document, including ones belonging to no collection. Recoll will not enforce collection scoping; the API layer must.
The helper must start fast. Recoll spawns it once per file per index run. Over 400 files: 0.09s with no helper, 0.13s for a shell+awk helper, 0.82s for a python3 helper — interpreter startup dominates. The helper reads a flat map file exported from SQLite; it is not a Python process.

Open design questions

Closed by the 001 brainstorm (2026-09-18) — see docs/DECISIONS.md for the reasoning:

~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by metadatacmds at index time; human names live only in SQLite; documents are never written to.
~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's.
~~Indexing trigger~~ — an in-process job queue, backed by SQLite.
~~Stack~~ — Python + FastAPI, managed with uv.
~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004.
~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate endpoints.

Still open, to be closed by the remaining sections of the 001 design:

Component boundaries and the SQLite schema.
API surface, pagination, and error semantics.
Test strategy, including how tests skip when the container is down.

External dependencies

Dependency	What breaks without it
Recoll (in-container)	All indexing and search. Not installed on the host — see `AGENTS.md` Rules.
Docker + compose	The only supported way to run the service.
Document volumes	Nothing to index; collections resolve to empty.
A writable volume for the index and SQLite	All state. The document volume itself may be mounted read-only.

Invariants

A query never silently spans collections: the scope is an explicit, required parameter, and the API refuses a search that does not name one.
Collection identity in the index is an opaque single-token ID, never a user-supplied name, so collections cannot bleed into one another and renaming never touches the index.
The service never writes to the documents it indexes. The document volume is treated as read-only even when it is not, and the deployment mounts it :ro.
Document identity is the SHA-256 of file content, not its path or name: two files with the same name and different content are two documents, and one file reachable by several paths is one document.
Indexed content is never written into the repository.

Known rough edges

None yet — nothing has been built. Cross-linked backlog: docs/TODO.md.

6.6 KiB Raw Blame History