Archived
Lucas WintherandClaude Opus 5
def811bab4
Records the outcome of the 001 brainstorm and the Recoll spike that backed it. Decisions: Python + FastAPI managed with uv; the product split into four specs built in order; collections managed at runtime through the API with SQLite as the store of record; indexing as an in-process job queue; search scope as an explicit required parameter; results carrying metadata and snippets; real authentication in 001 with user administration deferred to 004. The load-bearing one is collection membership. A collection cannot be a directory, because files sharing a directory must be able to belong to different collections, so every document is indexed once into a single Recoll index and membership is a user.collection extended attribute mapped to a queryable field. Verified against Recoll 1.43.0 in a throwaway container: the tag must be an opaque single-token ID rather than the collection name, because Recoll tokenises the field value and matches case-insensitively, so a tag of cook-books is also matched by collection:cook and collection:books. Names as tags would let collections silently bleed into each other. Opaque IDs also make renaming free, since the index is never touched. Two operational consequences are recorded in docs/TODO.md: documents must live on a volume with extended-attribute support, checked loudly at startup, and a plain incremental recollindex does not notice xattr-only changes, so retagging must go through recollindex -i on the affected file. uv 0.12.16 was installed on this host as part of the stack decision; AGENTS.md is corrected in the same commit, since it previously listed uv as absent. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
5.4 KiB
5.4 KiB
Architecture
How recoll-modern is built today. Update in the same commit as any structural change.
Overview
Nothing is built yet. This file records intent, the design questions that are now closed, and the Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing code.
The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
web UI and (b) an MCP server, running in Docker, with several independent collections such as
techbooks and cookbooks.
Map
| Path | Responsibility |
|---|---|
AGENTS.md |
Agent rules; single source of truth |
PRODUCT.md |
Durable product context for the impeccable skill |
docs/specs/, docs/plans/ |
Approved specs and implementation plans |
No source directories exist yet. Add rows here as they appear.
Delivery order
The product is four specs, built in order. Each gets its own spec, plan and review cycle.
| Spec | Scope |
|---|---|
| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication |
| 002 Web UI | Blocked on /impeccable init replacing the PRODUCT.md stub, then DESIGN.md |
| 003 MCP server | Thin wrapper over 001's HTTP API |
| 004 Multi-user management | Roles, user CRUD, per-collection permissions |
Intended shape
- One Recoll index for all documents, not one per collection. Collection membership is per-file metadata, not directory layout, so two files in the same directory can belong to different collections.
- A Python service owns Recoll through its native Python bindings (
from recoll import recoll). - SQLite holds collections, membership, indexing jobs, users and sessions.
- An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
Verified Recoll mechanics
Established empirically on 2026-09-18 in a throwaway debian:trixie-slim container running
Recoll 1.43.0 / Xapian 1.4.29. Re-verify if the Recoll major version changes.
- Per-file metadata is indexable and queryable. A
fieldsfile declaring[xattrtofields] collection = collection,[prefixes] collection = XYCOLLand[stored] collection =makes theuser.collectionextended attribute a queryable field (collection:<value>), returned on each result. Recoll strips theuser.prefix itself. - Multi-membership works. A space-separated xattr value puts one file in several collections. An
ORacross collections returns it once, in a single coherently ranked result set — Xapian does the ranking, so no application-side merging is needed. - A plain incremental
recollindexdoes not see xattr-only changes. It compares mtime, andsetfattrdoes not change mtime;touchdoes not help either.recollindex -i <file>does pick the change up, in roughly 20ms for one file. Retagging therefore costs one file, never a rebuild. - The field value is tokenised and matched case-insensitively. A tag of
cook-booksis matched bycollection:cookandcollection:books;TechBooksis matched bycollection:techbooks. Using human-readable collection names as tags would let collections silently bleed into one another. - An unscoped query returns every document, including untagged ones. Recoll will not enforce collection scoping; the API layer must.
Open design questions
Closed by the 001 brainstorm (2026-09-18) — see docs/DECISIONS.md for the reasoning:
Collection isolation— one index; membership is an opaque collection ID in auser.collectionxattr; human names live only in SQLite.Cross-collection search— allowed, but only when explicitly requested. Ranking is Xapian's.Indexing trigger— an in-process job queue, backed by SQLite.Stack— Python + FastAPI, managed withuv.Auth and exposure— real authentication in 001; user administration deferred to 004.Result presentation— metadata plus Recoll snippets; full text and file download are separate endpoints.
Still open, to be closed by the remaining sections of the 001 design:
- Component boundaries and the SQLite schema.
- API surface, pagination, and error semantics.
- Test strategy, including how tests skip when the container is down.
External dependencies
| Dependency | What breaks without it |
|---|---|
| Recoll (in-container) | All indexing and search. Not installed on the host — see AGENTS.md Rules. |
| Docker + compose | The only supported way to run the service. |
| Document volumes | Nothing to index; collections resolve to empty. |
| Extended-attribute support on the document volume | Collection membership. Without xattrs every document is untagged and every scoped search returns nothing — a silent failure, so it must be checked at startup. |
Invariants
- A query never silently spans collections: the scope is an explicit, required parameter, and the API refuses a search that does not name one.
- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so collections cannot bleed into one another and renaming never touches the index.
- Indexed content is never written into the repository.
Known rough edges
None yet — nothing has been built. Cross-linked backlog: docs/TODO.md.