From def811bab46748bd65fb605e04f6ccb88a98caff Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 18 Sep 2026 16:57:54 +0200 Subject: [PATCH 1/3] docs: close the six open design questions for spec 001 Records the outcome of the 001 brainstorm and the Recoll spike that backed it. Decisions: Python + FastAPI managed with uv; the product split into four specs built in order; collections managed at runtime through the API with SQLite as the store of record; indexing as an in-process job queue; search scope as an explicit required parameter; results carrying metadata and snippets; real authentication in 001 with user administration deferred to 004. The load-bearing one is collection membership. A collection cannot be a directory, because files sharing a directory must be able to belong to different collections, so every document is indexed once into a single Recoll index and membership is a user.collection extended attribute mapped to a queryable field. Verified against Recoll 1.43.0 in a throwaway container: the tag must be an opaque single-token ID rather than the collection name, because Recoll tokenises the field value and matches case-insensitively, so a tag of cook-books is also matched by collection:cook and collection:books. Names as tags would let collections silently bleed into each other. Opaque IDs also make renaming free, since the index is never touched. Two operational consequences are recorded in docs/TODO.md: documents must live on a volume with extended-attribute support, checked loudly at startup, and a plain incremental recollindex does not notice xattr-only changes, so retagging must go through recollindex -i on the affected file. uv 0.12.16 was installed on this host as part of the stack decision; AGENTS.md is corrected in the same commit, since it previously listed uv as absent. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 5 +- docs/ARCHITECTURE.md | 83 ++++++++++++++++++++-------- docs/DECISIONS.md | 125 +++++++++++++++++++++++++++++++++++++++++++ docs/TODO.md | 17 +++--- 4 files changed, 201 insertions(+), 29 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d8433d6..55c4d84 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,8 +15,9 @@ TODO(maintainer): no stack yet. Fill install/dev/test/single-test/lint/typecheck commit that scaffolds the app, and delete this line. These must be what CI runs. Installed on this host: `docker` 29.5.2 with `docker compose`, `node` 20.19.2, `npm` 9.2.0, `bun` 1.3.14, -`yarn` 1.22.22, `python3` 3.13.5, `dotnet` 10.0.302. -**Absent: `pnpm`, `uv`, `poetry`, `go`, `cargo`** — do not write commands that assume them. +`yarn` 1.22.22, `python3` 3.13.5, `uv` 0.12.16 (in `~/.local/bin`, already on `PATH`), `dotnet` 10.0.302. +**Absent: `pnpm`, `poetry`, `go`, `cargo`** — do not write commands that assume them. +`uv` is the package manager for this project; do not use bare `pip`, `venv`, or `poetry`. ## Rules diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 4b88116..584d6a6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -4,8 +4,9 @@ How `recoll-modern` is built today. Update in the same commit as any structural ## Overview -**Nothing is built yet.** This file currently records intent and the open questions that the first -spec must close. Do not read anything below "Intended shape" as describing existing code. +**Nothing is built yet.** This file records intent, the design questions that are now closed, and the +Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing +code. The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern web UI and (b) an MCP server, running in Docker, with several independent collections such as @@ -21,30 +22,66 @@ web UI and (b) an MCP server, running in Docker, with several independent collec No source directories exist yet. Add rows here as they appear. +## Delivery order + +The product is four specs, built in order. Each gets its own spec, plan and review cycle. + +| Spec | Scope | +|---|---| +| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication | +| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` | +| 003 MCP server | Thin wrapper over 001's HTTP API | +| 004 Multi-user management | Roles, user CRUD, per-collection permissions | + ## Intended shape -Not yet decided or validated. The pieces the spec has to place: - -- A Recoll layer that owns indexing and querying, per collection. +- **One** Recoll index for all documents, not one per collection. Collection membership is per-file + metadata, not directory layout, so two files in the same directory can belong to different + collections. +- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`). +- SQLite holds collections, membership, indexing jobs, users and sessions. - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge. -- An MCP server exposing search (and likely collection listing) as tools. -- A Docker image that contains Recoll itself, plus compose wiring for document volumes and index storage. +- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage. + +## Verified Recoll mechanics + +Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running +Recoll 1.43.0 / Xapian 1.4.29. Re-verify if the Recoll major version changes. + +- **Per-file metadata is indexable and queryable.** A `fields` file declaring + `[xattrtofields] collection = collection`, `[prefixes] collection = XYCOLL` and `[stored] collection =` + makes the `user.collection` extended attribute a queryable field (`collection:`), returned on + each result. Recoll strips the `user.` prefix itself. +- **Multi-membership works.** A space-separated xattr value puts one file in several collections. An + `OR` across collections returns it **once**, in a single coherently ranked result set — Xapian does + the ranking, so no application-side merging is needed. +- **A plain incremental `recollindex` does not see xattr-only changes.** It compares mtime, and + `setfattr` does not change mtime; `touch` does not help either. `recollindex -i ` does pick the + change up, in roughly 20ms for one file. Retagging therefore costs one file, never a rebuild. +- **The field value is tokenised and matched case-insensitively.** A tag of `cook-books` is matched by + `collection:cook` *and* `collection:books`; `TechBooks` is matched by `collection:techbooks`. Using + human-readable collection names as tags would let collections silently bleed into one another. +- **An unscoped query returns every document, including untagged ones.** Recoll will not enforce + collection scoping; the API layer must. ## Open design questions -The first spec must answer these. None are settled. +Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning: -1. **Collection isolation.** Recoll's model is a config directory per index (`RECOLL_CONFDIR`). - Confirm against Recoll's own docs, then decide: one config dir per collection, and what owns the - mapping from collection name to config dir and document root. -2. **Cross-collection search.** Allowed at all? If yes, merged ranking across separately-scored indexes - is not free — decide before building the UI around it. -3. **Indexing trigger.** Cron inside the container, Recoll's real-time monitor, or API-triggered runs. - Affects whether the container needs a long-running process beyond the web server. -4. **Stack.** Unchosen. See `AGENTS.md` Commands for what this host actually has installed. -5. **Auth and exposure.** Whether the service authenticates at all, or assumes a reverse proxy does it. -6. **Result presentation.** Recoll returns snippets and can extract/preview documents; decide how much - of that the API surfaces, since it drives both the UI and the MCP tool shape. +1. ~~Collection isolation~~ — one index; membership is an opaque collection ID in a `user.collection` + xattr; human names live only in SQLite. +2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's. +3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite. +4. ~~Stack~~ — Python + FastAPI, managed with `uv`. +5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004. +6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate + endpoints. + +Still open, to be closed by the remaining sections of the 001 design: + +- Component boundaries and the SQLite schema. +- API surface, pagination, and error semantics. +- Test strategy, including how tests skip when the container is down. ## External dependencies @@ -53,11 +90,15 @@ The first spec must answer these. None are settled. | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. | | Docker + compose | The only supported way to run the service. | | Document volumes | Nothing to index; collections resolve to empty. | +| Extended-attribute support on the document volume | Collection membership. Without xattrs every document is untagged and every scoped search returns nothing — a silent failure, so it must be checked at startup. | ## Invariants -To be established with the first code. Candidates the spec should confirm and then record here: -a query never silently spans collections; indexed content is never written into the repository. +- A query never silently spans collections: the scope is an explicit, required parameter, and the API + refuses a search that does not name one. +- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so + collections cannot bleed into one another and renaming never touches the index. +- Indexed content is never written into the repository. ## Known rough edges diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index cefdd71..f19ffc5 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -3,6 +3,131 @@ Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back. One entry per decision a future maintainer would otherwise have to reverse-engineer. +## 2026-09-18 — Search results carry metadata and snippets, not full text +**Status:** accepted +**Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the +original file. The response shape drives both the web UI and the MCP tool. +**Decision:** a search hit carries title, author, path, MIME type, size, date, score, its collections and +Recoll's snippets. Extracted text and file download are separate endpoints. +**Alternatives:** metadata only (a results list without context is markedly worse, and an MCP client +would fetch every candidate to judge it, costing more overall); metadata plus full extracted text +(responses become enormous for book-length PDFs and blow an agent's context window). +**Consequences:** one search call is enough for an agent to judge relevance. The UI needs a second call +to open a document, which is the right trade. + +--- + +## 2026-09-18 — Authentication in 001, user administration in 004 +**Status:** accepted +**Context:** the service needs real auth, but roles, user CRUD and per-collection permissions are a +subsystem in their own right, and `PRODUCT.md` currently describes a single technical operator. +**Decision:** 001 ships genuine authentication — argon2 password hashing, server-side sessions, an admin +account seeded from the environment at first boot, every endpoint behind an auth dependency. User +administration and permissions are deferred to spec 004. +**Alternatives:** no auth in the service, trusting a reverse proxy (a misconfigured proxy or an exposed +Docker port would leave the destructive endpoints naked); the full identity system in 001 (a large +addition to an already-grown spec, building for users who do not exist yet). +**Consequences:** 001 is safe to expose behind a proxy. 004 must extend the auth layer without +reshaping endpoints, so the auth dependency is designed as a seam from the start. + +--- + +## 2026-09-18 — Search scope is an explicit, required parameter +**Status:** accepted +**Context:** `docs/ARCHITECTURE.md` proposed the invariant that a query never silently spans +collections. A Recoll query with no collection clause returns every document, including untagged ones. +**Decision:** every search names one or more collections; omitting the parameter is an error, not an +implicit "all". Multiple collections are OR-ed into one Xapian query so ranking stays coherent and each +hit reports which collections it belongs to. +**Alternatives:** single-collection-only searches (the UI and MCP would be built around a model they +later outgrow); a searchable-by-default flag (reintroduces the implicit scope the invariant forbids); +per-collection fan-out and application-side merging (scores from separate databases are not comparable, +so the merged order would be strictly worse than letting Xapian rank). +**Consequences:** the API layer enforces the invariant, because Recoll will not. Every search path needs +a test proving an unscoped request is rejected. + +--- + +## 2026-09-18 — Indexing runs as an in-process job queue +**Status:** accepted +**Context:** collections are created at runtime, so something must index a collection the moment it is +created, and report progress back to the UI and to MCP clients. +**Decision:** creating a collection, changing membership, or calling the reindex endpoint enqueues a job. +A background worker in the same FastAPI process runs `recollindex` and records status, progress and +errors in SQLite. One container process, no broker. +**Alternatives:** `recollindex -m` real-time monitoring (one supervised process per collection, inotify +watch limits, and a new collection still needs a first full index from somewhere — revisit as an opt-in +mode later); cron-scheduled full runs (new collections sit empty until the next tick, and there is no +way to force a refresh). +**Consequences:** jobs die with the process, so the queue must recover interrupted runs at startup. +Job state is queryable, so "indexing, 40%" is available to every consumer. + +--- + +## 2026-09-18 — Collections are runtime-managed through the API +**Status:** accepted +**Context:** collections could be declared in a mounted config file or managed at runtime. +**Decision:** collections are created, edited and deleted through the HTTP API, with SQLite as the store +of record. The service creates and maintains the index state on demand. +**Alternatives:** a declarative mounted `collections.yaml` (reproducible and git-friendly, but no +management UI without adding a write path later); one collection per subdirectory by convention (forces +the on-disk layout to match the collection layout, which the per-file membership requirement rules out). +**Consequences:** 001 grows a persistence layer, validation and migrations. `AGENTS.md` makes deleting +or rebuilding an index an ask-first operation, so the delete path needs an explicit guard rather than a +bare `DELETE`. + +--- + +## 2026-09-18 — One index; collection membership is an opaque per-file tag +**Status:** accepted +**Context:** a collection cannot be "a directory", because files sharing a directory must be able to +belong to different collections. Verified in a throwaway Recoll 1.43.0 container — details and the +exact config in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics". +**Decision:** index every document once into a single Recoll index. Membership is a `user.collection` +extended attribute mapped to a queryable Recoll field. The tag is an **opaque single-token ID**; the +human-readable name lives only in SQLite. +**Alternatives:** one index per collection with rule-based selection (a file in two collections is +stored and indexed twice and can be returned twice, and hand-picking files in a shared directory is +awkward); an explicit file list driving indexing (does not map onto Recoll's topdirs-walking model). +**Why opaque IDs:** Recoll tokenises the field value and matches case-insensitively, so a tag of +`cook-books` is also matched by `collection:cook` and `collection:books`, and `TechBooks` by +`collection:techbooks`. Names-as-tags would let collections silently bleed into each other — exactly +what the no-silent-spanning invariant forbids. Opaque IDs tested clean, with no partial matches. +**Consequences:** collection names need no character restrictions and renaming never touches the index. +Documents must sit on a volume supporting extended attributes, checked at startup — without it every +scoped search silently returns nothing. Retagging costs `recollindex -i `, not a rebuild, because +a plain incremental run does not notice xattr-only changes. + +--- + +## 2026-09-18 — Stack: Python + FastAPI, managed with uv +**Status:** accepted +**Context:** the stack was unchosen and is an ask-first decision. Recoll ships first-class Python +bindings; every other language must drive `recollq` as a subprocess and parse its output. +**Decision:** Python + FastAPI for the engine and HTTP API, with `uv` as the package manager. `uv` +0.12.16 was installed on this host as part of this decision, and `AGENTS.md` updated in the same commit. +**Alternatives:** TypeScript on Node/Bun or C# on ASP.NET Core (one language across UI, API and MCP, but +both make a fragile output parser load-bearing); a split Python engine behind a TypeScript API (two +runtimes and an extra hop inside one container). +**Consequences:** structured results with no parsing layer, and OpenAPI for free, which specs 002 and 003 +can generate clients from. The web UI will likely be a second language; that boundary is the HTTP API. + +--- + +## 2026-09-18 — The product is four specs, built in order +**Status:** accepted +**Context:** "a web UI and an MCP server over Recoll" is four subsystems. One spec covering all of them +would be too large to plan or review, and `PRODUCT.md` is still an uninterviewed stub, so any UI spec +written now would design against fiction. +**Decision:** 001 search engine + API, 002 web UI, 003 MCP server, 004 multi-user management. Each gets +its own spec, plan and review cycle. The table lives in `docs/ARCHITECTURE.md`. +**Alternatives:** one thin vertical slice through all four (would lock in a stack and a UI before +`PRODUCT.md` is real); engine plus MCP together (defensible, but it widens 001, which has already grown). +**Consequences:** 002 is blocked on `/impeccable init`. 001's HTTP API is the contract 002 and 003 are +both built against, so it is the piece that must be got right. + +--- + ## 2026-09-18 — Override the superpowers artefact paths **Status:** accepted **Context:** superpowers 6.3.0 writes specs to `docs/superpowers/specs/YYYY-MM-DD--design.md` diff --git a/docs/TODO.md b/docs/TODO.md index 8476ec3..9c07a39 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -4,17 +4,22 @@ Durable backlog. Agents append here instead of leaving `TODO` comments in source Small, specific, independently actionable. Remove only when actually done. ## Now -- [ ] Answer the open questions in the bootstrap report, then fill the `Commands` section of `AGENTS.md` - and delete its `TODO(maintainer)` line. -- [ ] Brainstorm -> spec `docs/specs/001-*.md` covering the open design questions in - `docs/ARCHITECTURE.md`, starting with collection isolation and the stack choice. +- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in + `docs/DECISIONS.md`. +- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics, + test strategy. Then write `docs/specs/001-search-engine-api.md`. +- [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in + the same commit that scaffolds the app, with every command actually run. ## Next -- [ ] Scaffold the chosen stack; verify install/dev/test/lint/typecheck/build commands by running them, +- [ ] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them, then record the verified ones in `AGENTS.md`. - [ ] Dockerfile + compose: Recoll in the image, volumes for documents and indexes. +- [ ] Startup check: refuse to start, loudly, if the document volume does not support extended + attributes — without them every scoped search silently returns nothing. +- [ ] Recover interrupted indexing jobs at startup; they do not survive a process restart on their own. - [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial - brief, not an interview. + brief, not an interview. Blocks spec 002. ## Someday - [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/` From 883db6a59dd26ce029da5f981f6f4c4821e75979 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 18 Sep 2026 17:06:29 +0200 Subject: [PATCH 2/3] docs: supersede the xattr membership design for read-only documents The service must work with read-only access to the documents it indexes. Writing a user.collection extended attribute needs write access, so the previous membership design fails on a :ro mount; setfattr returns "Read-only file system". Verified in the spike container. Membership now reaches Recoll through metadatacmds in recoll.conf, a documented facility for external metadata: Recoll runs a helper per file at index time and the helper answers from our own store. Verified end to end against a read-only bind mount. Everything previously established about opaque collection IDs, multi-membership and single-ranked-set OR queries survives unchanged, because only the transport differs. SQLite becomes the only store of membership rather than the authoritative one of two, so drift is impossible and the reconcile path is gone. Extended attributes are no longer a deployment requirement. Document identity becomes the SHA-256 of file content. Files sharing a name but differing in content are separate documents; one file reachable by several paths is a single document with one canonical path, and only that path is reported as belonging to a collection. Recoll cannot help here: md5 is empty and sig is only size plus mtime. Two measured constraints are recorded. A membership change is invisible to both a plain incremental run and recollindex -i, because the change signature is size plus mtime; recollindex -i -Z on the affected path picks it up in about 15ms. And the helper is spawned once per file per index run, so it must start fast: a 400-file index took 0.09s with no helper, 0.13s with shell and awk, and 0.82s with python3. The superseded entry in docs/DECISIONS.md is left intact, per that file's append-only rule, and the new entry links back to it. Co-Authored-By: Claude Opus 5 (1M context) --- docs/ARCHITECTURE.md | 55 +++++++++++++++++++++++++++----------------- docs/DECISIONS.md | 44 +++++++++++++++++++++++++++++++++++ docs/TODO.md | 12 +++++++--- 3 files changed, 87 insertions(+), 24 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 584d6a6..fb6286c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -36,8 +36,8 @@ The product is four specs, built in order. Each gets its own spec, plan and revi ## Intended shape - **One** Recoll index for all documents, not one per collection. Collection membership is per-file - metadata, not directory layout, so two files in the same directory can belong to different - collections. + metadata supplied by the service at index time, not directory layout, so two files in the same + directory can belong to different collections and the documents are never modified. - A Python service owns Recoll through its native Python bindings (`from recoll import recoll`). - SQLite holds collections, membership, indexing jobs, users and sessions. - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge. @@ -46,30 +46,39 @@ The product is four specs, built in order. Each gets its own spec, plan and revi ## Verified Recoll mechanics Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running -Recoll 1.43.0 / Xapian 1.4.29. Re-verify if the Recoll major version changes. +Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted **read-only**. Re-verify if the +Recoll major version changes. -- **Per-file metadata is indexable and queryable.** A `fields` file declaring - `[xattrtofields] collection = collection`, `[prefixes] collection = XYCOLL` and `[stored] collection =` - makes the `user.collection` extended attribute a queryable field (`collection:`), returned on - each result. Recoll strips the `user.` prefix itself. -- **Multi-membership works.** A space-separated xattr value puts one file in several collections. An - `OR` across collections returns it **once**, in a single coherently ranked result set — Xapian does - the ranking, so no application-side merging is needed. -- **A plain incremental `recollindex` does not see xattr-only changes.** It compares mtime, and - `setfattr` does not change mtime; `touch` does not help either. `recollindex -i ` does pick the - change up, in roughly 20ms for one file. Retagging therefore costs one file, never a rebuild. -- **The field value is tokenised and matched case-insensitively.** A tag of `cook-books` is matched by - `collection:cook` *and* `collection:books`; `TechBooks` is matched by `collection:techbooks`. Using - human-readable collection names as tags would let collections silently bleed into one another. -- **An unscoped query returns every document, including untagged ones.** Recoll will not enforce - collection scoping; the API layer must. +- **Documents are never written to.** Membership reaches the index through `metadatacmds` in + `recoll.conf` (`metadatacmds = ; collection = %f`), which Recoll runs per file at index time + to fetch external metadata. The helper answers from our own store. Indexing a read-only mount + succeeds; `touch` and `setfattr` on it both fail, as they must. +- **Extended attributes are NOT usable.** An earlier design carried membership in a `user.collection` + xattr. It works only when the documents are writable, so it is superseded — see `docs/DECISIONS.md`. +- **Multi-membership works.** A space-separated helper response puts one file in several collections. + An `OR` across collections returns it **once**, in a single coherently ranked set: Xapian does the + ranking, so no application-side merging is needed. +- **Membership changes need a forced per-file reindex.** Neither a plain incremental run nor + `recollindex -i ` notices a change that did not touch the file — Recoll's change signature is + size + mtime (observed as `sig` = `''`), and a membership change alters neither. + `recollindex -i -Z ` does pick it up, at roughly 15ms per file, and accepts several paths. +- **The field value is tokenised and matched case-insensitively.** A value of `cook-books` is matched by + `collection:cook` *and* `collection:books`; `TechBooks` by `collection:techbooks`. Collection + identifiers must therefore be opaque single tokens with no separators. Tested clean with `^c[0-9a-f]+$`. +- **Recoll has no content hash.** `md5` is empty and `sig` is only size + mtime, so document identity by + content is ours to compute and store. +- **An unscoped query returns every document**, including ones belonging to no collection. Recoll will + not enforce collection scoping; the API layer must. +- **The helper must start fast.** Recoll spawns it once per file per index run. Over 400 files: 0.09s + with no helper, 0.13s for a shell+awk helper, 0.82s for a `python3` helper — interpreter startup + dominates. The helper reads a flat map file exported from SQLite; it is not a Python process. ## Open design questions Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning: -1. ~~Collection isolation~~ — one index; membership is an opaque collection ID in a `user.collection` - xattr; human names live only in SQLite. +1. ~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by + `metadatacmds` at index time; human names live only in SQLite; documents are never written to. 2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's. 3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite. 4. ~~Stack~~ — Python + FastAPI, managed with `uv`. @@ -90,7 +99,7 @@ Still open, to be closed by the remaining sections of the 001 design: | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. | | Docker + compose | The only supported way to run the service. | | Document volumes | Nothing to index; collections resolve to empty. | -| Extended-attribute support on the document volume | Collection membership. Without xattrs every document is untagged and every scoped search returns nothing — a silent failure, so it must be checked at startup. | +| A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. | ## Invariants @@ -98,6 +107,10 @@ Still open, to be closed by the remaining sections of the 001 design: refuses a search that does not name one. - Collection identity in the index is an opaque single-token ID, never a user-supplied name, so collections cannot bleed into one another and renaming never touches the index. +- The service never writes to the documents it indexes. The document volume is treated as read-only + even when it is not, and the deployment mounts it `:ro`. +- Document identity is the SHA-256 of file content, not its path or name: two files with the same name + and different content are two documents, and one file reachable by several paths is one document. - Indexed content is never written into the repository. ## Known rough edges diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index f19ffc5..73d61a5 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -3,6 +3,50 @@ Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back. One entry per decision a future maintainer would otherwise have to reverse-engineer. +## 2026-09-18 — Document identity is a content hash, not a path +**Status:** accepted +**Context:** the document set may contain files that share a name but differ in content, and files that +differ in name but are byte-identical. Keying membership on path merges neither and distinguishes +neither correctly. +**Decision:** a document is identified by the SHA-256 of its bytes. `documents` is keyed by that hash, +`document_paths` maps many paths to one document, and membership attaches to the document. Each document +has one **canonical path**, and only that path is reported to Recoll as belonging to any collection. +**Alternatives:** path as identity (byte-identical copies appear as separate hits for the same content); +relying on Recoll to deduplicate (it cannot — `md5` is empty and `sig` is only size + mtime). +**Consequences:** duplicate copies are indexed by Recoll but belong to no collection, so the required +collection scope makes them unreachable — verified: a scoped search over a tree with one duplicated file +returned it once. Deleting a canonical path must promote another path and force a reindex, or the +document silently leaves every collection. + +--- + +## 2026-09-18 — Membership is supplied at index time; documents are never written to +**Status:** accepted +**Supersedes:** "One index; collection membership is an opaque per-file tag" (below, same date), which +carried membership in a `user.collection` extended attribute. That entry's xattr mechanism must not be +implemented. +**Context:** the service must work with **read-only** access to the documents it indexes. Writing an +xattr requires write access, so the superseded design fails outright on a `:ro` mount — verified: +`setfattr` returns "Read-only file system". +**Decision:** membership reaches Recoll through `metadatacmds` in `recoll.conf` +(`metadatacmds = ; collection = %f`), a documented Recoll facility for external metadata. +Recoll runs the helper per file at index time; the helper answers from our own store. Everything the +superseded entry established about opaque IDs, multi-membership and single-ranked-set OR queries still +holds — only the transport changes. +**Alternatives:** filtering Recoll's results against SQLite after the query (breaks paging and total +counts); one index per collection (cannot express two files in one directory belonging to different +collections); requiring a writable document volume (contradicts the requirement). +**Consequences:** SQLite becomes the *only* store of membership rather than the authoritative copy of +two, so drift between store and index is now impossible and no reconcile path is needed. Extended +attributes are no longer a deployment requirement. A membership change needs +`recollindex -i -Z ` — verified that neither a plain incremental run nor `recollindex -i` notices +it, because Recoll's change signature is size + mtime and a membership change touches neither. +The helper is spawned once per file per index run, so it must start fast: over 400 files, indexing took +0.09s with no helper, 0.13s with a shell+awk helper and 0.82s with a `python3` helper. The service +exports a flat map from SQLite for a small non-Python helper to read. + +--- + ## 2026-09-18 — Search results carry metadata and snippets, not full text **Status:** accepted **Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the diff --git a/docs/TODO.md b/docs/TODO.md index 9c07a39..6778149 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -14,9 +14,15 @@ Small, specific, independently actionable. Remove only when actually done. ## Next - [ ] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them, then record the verified ones in `AGENTS.md`. -- [ ] Dockerfile + compose: Recoll in the image, volumes for documents and indexes. -- [ ] Startup check: refuse to start, loudly, if the document volume does not support extended - attributes — without them every scoped search silently returns nothing. +- [ ] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for + the index and SQLite. +- [ ] Write the `metadatacmds` membership helper as a fast-starting non-Python program, fed by a flat + map exported from SQLite. A `python3` helper made a 400-file index 9x slower; Recoll spawns it + once per file per index run. +- [ ] Re-measure indexing overhead against real documents (PDF, EPUB) rather than small text files — + extraction cost should dwarf the helper, but that is an assumption until measured. +- [ ] Promote a new canonical path and force a reindex when a canonical path disappears, or the + document silently leaves every collection it is in. - [ ] Recover interrupted indexing jobs at startup; they do not survive a process restart on their own. - [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial brief, not an interview. Blocks spec 002. From 3d6067f4facf338517319ec7813c66652b7289b2 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 18 Sep 2026 17:09:26 +0200 Subject: [PATCH 3/3] docs: add spec 001 and a resume note for the next session Writes docs/specs/001-search-engine-api.md covering the search engine and HTTP API: collections managed at runtime, per-file membership supplied to Recoll at index time, content-hash document identity with canonical paths, an in-process indexing job queue, scoped search, and authentication. Twenty-four numbered requirements, each with an acceptance criterion, plus the constraints the design must hold: read-only documents, per-file membership, content-hash identity, and no silently unscoped search. The spec is status: draft. It has had its self-review pass but has not been signed off, so no plan and no code may be written against it yet. Self-review changed three things. Deleting a collection now states that it enqueues reindexing of the affected canonical paths, because the previous wording claimed no index work while leaving the deleted id in the index. The search scope parameter is specified as collection ids rather than names, since names are mutable and would break saved queries on rename. And the reported match count is documented as approximate, because Xapian estimates it for large result sets. Three known gaps are recorded rather than left implicit: there is no way to read a matched document until spec 002, the indexing overhead figures come from small text files rather than real PDFs, and every authenticated user sees every collection until spec 004. docs/TODO.md gains a "Resume here" section stating where the workflow stands, what the next action is, and the traps a resuming agent would otherwise hit -- chiefly that the extended-attribute membership design was superseded mid-session and survives in the append-only decision log. Co-Authored-By: Claude Opus 5 (1M context) --- docs/TODO.md | 31 +++++ docs/specs/001-search-engine-api.md | 175 ++++++++++++++++++++++++++++ 2 files changed, 206 insertions(+) create mode 100644 docs/specs/001-search-engine-api.md diff --git a/docs/TODO.md b/docs/TODO.md index 6778149..ff46f6b 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -3,6 +3,37 @@ Durable backlog. Agents append here instead of leaving `TODO` comments in source. Small, specific, independently actionable. Remove only when actually done. +## Resume here + +Last touched 2026-09-18. Still greenfield: **no application code exists.** + +Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review): + +- Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with + reasoning in `docs/DECISIONS.md`. +- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** It has had its self-review + pass but **has not been signed off by the maintainer.** Do not start a plan or write code against it + until its front matter says `approved`. +- Plan `docs/plans/001-search-engine-api.md`: **not written.** It is the next artefact, and only after + sign-off. + +The next action is to ask the maintainer to review the spec. If they approve it, set `status: approved` +and invoke the superpowers `writing-plans` skill to produce `docs/plans/001-search-engine-api.md`, +numbered to match. + +Things a resuming agent will otherwise get wrong: + +- The design carried collection membership in a `user.collection` extended attribute for part of this + session. **That is superseded** — the documents are read-only. `docs/DECISIONS.md` has both entries, + newest first; the older one is left intact by that file's append-only rule, so read the newer one. +- The Recoll facts in `docs/ARCHITECTURE.md` -> "Verified Recoll mechanics" were measured in a container, + not recalled. Trust them over intuition, and re-verify only if Recoll's major version moves. +- The throwaway spike container is not part of the repo and does not survive the session. Everything + worth keeping from it is already in `docs/`. +- `uv` was installed on this host during this session and `AGENTS.md` was corrected accordingly. The + `Commands` section of `AGENTS.md` is still a `TODO(maintainer)` stub and must be filled in the same + commit that scaffolds the app, with every command actually run first. + ## Now - [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in `docs/DECISIONS.md`. diff --git a/docs/specs/001-search-engine-api.md b/docs/specs/001-search-engine-api.md new file mode 100644 index 0000000..7b05241 --- /dev/null +++ b/docs/specs/001-search-engine-api.md @@ -0,0 +1,175 @@ +--- +status: draft +plan: docs/plans/001-search-engine-api.md (not yet written) +supersedes: none +--- + +# 001 — Search engine and HTTP API + +The first of four specs (see `docs/ARCHITECTURE.md`, "Delivery order"). It delivers the Recoll layer and +the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI. + +## Why + +Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold +several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run +as a container on a server. The API is the contract two later consumers depend on, so it is the piece +that must be got right first. + +## Scope + +**In:** Recoll in Docker · one index over all documents · collections created and managed at runtime · +per-file collection membership · content-hash document identity · an indexing job queue · a scoped +search endpoint · authentication · a read-only file browse endpoint · health. + +**Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles +and per-collection permissions (004) · fetching a document's extracted text or downloading the original +file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth. + +## Constraints + +These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0 +and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; the reasoning is in +`docs/DECISIONS.md`. + +- **C1. The documents are read-only.** The service must operate with read-only access to everything it + indexes and must never write to a document, its directory, or its extended attributes. The deployment + mounts the document volume `:ro`. +- **C2. Membership is per file, not per directory.** Two files in the same directory may belong to + different collections. +- **C3. Documents identical in content are one document.** Documents sharing a name but differing in + content are separate documents. +- **C4. A search never silently spans collections.** + +## Requirements + +### Collections + +- **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and + an optional description. The id is generated by the service and is never supplied by a client. +- **R2.** Names are unique case-insensitively and may contain any characters, because a name never + reaches the index. +- **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the + database and enqueues no indexing work. +- **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion + removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so + the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds + the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless, + because the API resolves ids against the database and drops any it no longer knows. + +### Documents and membership + +- **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one + document; each document has exactly one canonical path. +- **R6.** Membership attaches to a document, not a path. A document may belong to any number of + collections, including none. +- **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection. + Non-canonical duplicate paths remain in the index but belong to no collection and are therefore + unreachable by any scoped search. +- **R8.** If a canonical path disappears, the service promotes another known path for that document and + forces a reindex of it. A document with no remaining path is marked missing, not deleted. +- **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work + for the affected canonical paths only. + +### Indexing + +- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time. + The helper is a fast-starting non-Python program reading a flat map exported from the database. +- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths. + Neither a plain incremental run nor `recollindex -i` detects such a change. +- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection, + changing membership, and the reindex endpoint all enqueue jobs. +- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while + it runs. +- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either + re-enqueued or failed with a reason, never left claiming to be in progress. + +### Search + +- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection + **ids**, not names — names are mutable and would break saved queries on rename. A request without the + parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is + rejected with `collection_not_found` rather than silently ignored. +- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently + ranked set rather than merged per collection. +- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it + belongs to, and Recoll's keyword-in-context snippets. +- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports. + Xapian estimates this figure for large result sets, so it is documented as approximate rather than + presented as exact. + +### Authentication + +- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session. +- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an + HttpOnly cookie. +- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are + supplied and no user exists, the service refuses to start rather than starting unauthenticated. + +### Operations + +- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring + authentication. +- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add. +- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the + table below. + +| Code | Status | +|---|---| +| `scope_required` | 400 | +| `unauthenticated` | 401 | +| `collection_not_found` | 404 | +| `collection_name_taken` | 409 | +| `confirmation_required` | 409 | +| `validation_failed` | 422 | +| `engine_unavailable` | 503 | + +## Shape + +Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The +one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** +Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run +without a container. + +## Testing + +- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. +- One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when + the container is unreachable; it never fails, and it never silently passes. +- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it. +- Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a + decision first, because `AGENTS.md` forbids committing indexed documents. + +## Acceptance criteria + +- [ ] A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4). +- [ ] Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4). +- [ ] Two files in the same directory can belong to different collections, and each is found only by its + own collection's scope (C2, R6). +- [ ] Two byte-identical files under different names are one document and appear once in a scoped + search (C3, R5, R7). +- [ ] Two files sharing a name with different content are two documents and both are findable (C3). +- [ ] Removing a canonical path promotes another path and the document remains findable (R8). +- [ ] A search without `collections` is rejected with `scope_required` (C4, R15). +- [ ] A search across two collections returns one ranked set, with each hit reporting its collections + (R16, R17). +- [ ] A membership change is visible to search after its job completes, without the document being + written to (C1, R10, R11). +- [ ] Indexing a document volume mounted `:ro` succeeds (C1). +- [ ] A job interrupted by a restart is not left claiming to be in progress (R14). +- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19). +- [ ] The service refuses to start with no user and no seeded credentials (R21). +- [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than + fails with the container down. + +## Known gaps + +Accepted deliberately, recorded so they are not mistaken for oversights. + +- **No way to read a document.** Search reports that a file matched but offers no extracted text and no + download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is + 002's first requirement. +- **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected + to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is + tracked in `docs/TODO.md`. +- **No per-user access control.** Every authenticated user sees every collection until 004.