Archived
Lucas WintherandClaude Opus 5
883db6a59d
The service must work with read-only access to the documents it indexes. Writing a user.collection extended attribute needs write access, so the previous membership design fails on a :ro mount; setfattr returns "Read-only file system". Verified in the spike container. Membership now reaches Recoll through metadatacmds in recoll.conf, a documented facility for external metadata: Recoll runs a helper per file at index time and the helper answers from our own store. Verified end to end against a read-only bind mount. Everything previously established about opaque collection IDs, multi-membership and single-ranked-set OR queries survives unchanged, because only the transport differs. SQLite becomes the only store of membership rather than the authoritative one of two, so drift is impossible and the reconcile path is gone. Extended attributes are no longer a deployment requirement. Document identity becomes the SHA-256 of file content. Files sharing a name but differing in content are separate documents; one file reachable by several paths is a single document with one canonical path, and only that path is reported as belonging to a collection. Recoll cannot help here: md5 is empty and sig is only size plus mtime. Two measured constraints are recorded. A membership change is invisible to both a plain incremental run and recollindex -i, because the change signature is size plus mtime; recollindex -i -Z on the affected path picks it up in about 15ms. And the helper is spawned once per file per index run, so it must start fast: a 400-file index took 0.09s with no helper, 0.13s with shell and awk, and 0.82s with python3. The superseded entry in docs/DECISIONS.md is left intact, per that file's append-only rule, and the new entry links back to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
119 lines
6.6 KiB
Markdown
119 lines
6.6 KiB
Markdown
# Architecture
|
|
|
|
How `recoll-modern` is built today. Update in the same commit as any structural change.
|
|
|
|
## Overview
|
|
|
|
**Nothing is built yet.** This file records intent, the design questions that are now closed, and the
|
|
Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing
|
|
code.
|
|
|
|
The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
|
|
web UI and (b) an MCP server, running in Docker, with several independent collections such as
|
|
`techbooks` and `cookbooks`.
|
|
|
|
## Map
|
|
|
|
| Path | Responsibility |
|
|
|---|---|
|
|
| `AGENTS.md` | Agent rules; single source of truth |
|
|
| `PRODUCT.md` | Durable product context for the `impeccable` skill |
|
|
| `docs/specs/`, `docs/plans/` | Approved specs and implementation plans |
|
|
|
|
No source directories exist yet. Add rows here as they appear.
|
|
|
|
## Delivery order
|
|
|
|
The product is four specs, built in order. Each gets its own spec, plan and review cycle.
|
|
|
|
| Spec | Scope |
|
|
|---|---|
|
|
| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication |
|
|
| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` |
|
|
| 003 MCP server | Thin wrapper over 001's HTTP API |
|
|
| 004 Multi-user management | Roles, user CRUD, per-collection permissions |
|
|
|
|
## Intended shape
|
|
|
|
- **One** Recoll index for all documents, not one per collection. Collection membership is per-file
|
|
metadata supplied by the service at index time, not directory layout, so two files in the same
|
|
directory can belong to different collections and the documents are never modified.
|
|
- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`).
|
|
- SQLite holds collections, membership, indexing jobs, users and sessions.
|
|
- An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
|
|
- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
|
|
|
|
## Verified Recoll mechanics
|
|
|
|
Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running
|
|
Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted **read-only**. Re-verify if the
|
|
Recoll major version changes.
|
|
|
|
- **Documents are never written to.** Membership reaches the index through `metadatacmds` in
|
|
`recoll.conf` (`metadatacmds = ; collection = <helper> %f`), which Recoll runs per file at index time
|
|
to fetch external metadata. The helper answers from our own store. Indexing a read-only mount
|
|
succeeds; `touch` and `setfattr` on it both fail, as they must.
|
|
- **Extended attributes are NOT usable.** An earlier design carried membership in a `user.collection`
|
|
xattr. It works only when the documents are writable, so it is superseded — see `docs/DECISIONS.md`.
|
|
- **Multi-membership works.** A space-separated helper response puts one file in several collections.
|
|
An `OR` across collections returns it **once**, in a single coherently ranked set: Xapian does the
|
|
ranking, so no application-side merging is needed.
|
|
- **Membership changes need a forced per-file reindex.** Neither a plain incremental run nor
|
|
`recollindex -i <file>` notices a change that did not touch the file — Recoll's change signature is
|
|
size + mtime (observed as `sig` = `'<size><mtime>'`), and a membership change alters neither.
|
|
`recollindex -i -Z <file>` does pick it up, at roughly 15ms per file, and accepts several paths.
|
|
- **The field value is tokenised and matched case-insensitively.** A value of `cook-books` is matched by
|
|
`collection:cook` *and* `collection:books`; `TechBooks` by `collection:techbooks`. Collection
|
|
identifiers must therefore be opaque single tokens with no separators. Tested clean with `^c[0-9a-f]+$`.
|
|
- **Recoll has no content hash.** `md5` is empty and `sig` is only size + mtime, so document identity by
|
|
content is ours to compute and store.
|
|
- **An unscoped query returns every document**, including ones belonging to no collection. Recoll will
|
|
not enforce collection scoping; the API layer must.
|
|
- **The helper must start fast.** Recoll spawns it once per file per index run. Over 400 files: 0.09s
|
|
with no helper, 0.13s for a shell+awk helper, 0.82s for a `python3` helper — interpreter startup
|
|
dominates. The helper reads a flat map file exported from SQLite; it is not a Python process.
|
|
|
|
## Open design questions
|
|
|
|
Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning:
|
|
|
|
1. ~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by
|
|
`metadatacmds` at index time; human names live only in SQLite; documents are never written to.
|
|
2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's.
|
|
3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite.
|
|
4. ~~Stack~~ — Python + FastAPI, managed with `uv`.
|
|
5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004.
|
|
6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate
|
|
endpoints.
|
|
|
|
Still open, to be closed by the remaining sections of the 001 design:
|
|
|
|
- Component boundaries and the SQLite schema.
|
|
- API surface, pagination, and error semantics.
|
|
- Test strategy, including how tests skip when the container is down.
|
|
|
|
## External dependencies
|
|
|
|
| Dependency | What breaks without it |
|
|
|---|---|
|
|
| Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. |
|
|
| Docker + compose | The only supported way to run the service. |
|
|
| Document volumes | Nothing to index; collections resolve to empty. |
|
|
| A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. |
|
|
|
|
## Invariants
|
|
|
|
- A query never silently spans collections: the scope is an explicit, required parameter, and the API
|
|
refuses a search that does not name one.
|
|
- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so
|
|
collections cannot bleed into one another and renaming never touches the index.
|
|
- The service never writes to the documents it indexes. The document volume is treated as read-only
|
|
even when it is not, and the deployment mounts it `:ro`.
|
|
- Document identity is the SHA-256 of file content, not its path or name: two files with the same name
|
|
and different content are two documents, and one file reachable by several paths is one document.
|
|
- Indexed content is never written into the repository.
|
|
|
|
## Known rough edges
|
|
|
|
None yet — nothing has been built. Cross-linked backlog: `docs/TODO.md`.
|