diff --git a/AGENTS.md b/AGENTS.md index d8433d6..55c4d84 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,8 +15,9 @@ TODO(maintainer): no stack yet. Fill install/dev/test/single-test/lint/typecheck commit that scaffolds the app, and delete this line. These must be what CI runs. Installed on this host: `docker` 29.5.2 with `docker compose`, `node` 20.19.2, `npm` 9.2.0, `bun` 1.3.14, -`yarn` 1.22.22, `python3` 3.13.5, `dotnet` 10.0.302. -**Absent: `pnpm`, `uv`, `poetry`, `go`, `cargo`** — do not write commands that assume them. +`yarn` 1.22.22, `python3` 3.13.5, `uv` 0.12.16 (in `~/.local/bin`, already on `PATH`), `dotnet` 10.0.302. +**Absent: `pnpm`, `poetry`, `go`, `cargo`** — do not write commands that assume them. +`uv` is the package manager for this project; do not use bare `pip`, `venv`, or `poetry`. ## Rules diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 4b88116..fb6286c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -4,8 +4,9 @@ How `recoll-modern` is built today. Update in the same commit as any structural ## Overview -**Nothing is built yet.** This file currently records intent and the open questions that the first -spec must close. Do not read anything below "Intended shape" as describing existing code. +**Nothing is built yet.** This file records intent, the design questions that are now closed, and the +Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing +code. The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern web UI and (b) an MCP server, running in Docker, with several independent collections such as @@ -21,30 +22,75 @@ web UI and (b) an MCP server, running in Docker, with several independent collec No source directories exist yet. Add rows here as they appear. +## Delivery order + +The product is four specs, built in order. Each gets its own spec, plan and review cycle. + +| Spec | Scope | +|---|---| +| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication | +| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` | +| 003 MCP server | Thin wrapper over 001's HTTP API | +| 004 Multi-user management | Roles, user CRUD, per-collection permissions | + ## Intended shape -Not yet decided or validated. The pieces the spec has to place: - -- A Recoll layer that owns indexing and querying, per collection. +- **One** Recoll index for all documents, not one per collection. Collection membership is per-file + metadata supplied by the service at index time, not directory layout, so two files in the same + directory can belong to different collections and the documents are never modified. +- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`). +- SQLite holds collections, membership, indexing jobs, users and sessions. - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge. -- An MCP server exposing search (and likely collection listing) as tools. -- A Docker image that contains Recoll itself, plus compose wiring for document volumes and index storage. +- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage. + +## Verified Recoll mechanics + +Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running +Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted **read-only**. Re-verify if the +Recoll major version changes. + +- **Documents are never written to.** Membership reaches the index through `metadatacmds` in + `recoll.conf` (`metadatacmds = ; collection = %f`), which Recoll runs per file at index time + to fetch external metadata. The helper answers from our own store. Indexing a read-only mount + succeeds; `touch` and `setfattr` on it both fail, as they must. +- **Extended attributes are NOT usable.** An earlier design carried membership in a `user.collection` + xattr. It works only when the documents are writable, so it is superseded — see `docs/DECISIONS.md`. +- **Multi-membership works.** A space-separated helper response puts one file in several collections. + An `OR` across collections returns it **once**, in a single coherently ranked set: Xapian does the + ranking, so no application-side merging is needed. +- **Membership changes need a forced per-file reindex.** Neither a plain incremental run nor + `recollindex -i ` notices a change that did not touch the file — Recoll's change signature is + size + mtime (observed as `sig` = `''`), and a membership change alters neither. + `recollindex -i -Z ` does pick it up, at roughly 15ms per file, and accepts several paths. +- **The field value is tokenised and matched case-insensitively.** A value of `cook-books` is matched by + `collection:cook` *and* `collection:books`; `TechBooks` by `collection:techbooks`. Collection + identifiers must therefore be opaque single tokens with no separators. Tested clean with `^c[0-9a-f]+$`. +- **Recoll has no content hash.** `md5` is empty and `sig` is only size + mtime, so document identity by + content is ours to compute and store. +- **An unscoped query returns every document**, including ones belonging to no collection. Recoll will + not enforce collection scoping; the API layer must. +- **The helper must start fast.** Recoll spawns it once per file per index run. Over 400 files: 0.09s + with no helper, 0.13s for a shell+awk helper, 0.82s for a `python3` helper — interpreter startup + dominates. The helper reads a flat map file exported from SQLite; it is not a Python process. ## Open design questions -The first spec must answer these. None are settled. +Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning: -1. **Collection isolation.** Recoll's model is a config directory per index (`RECOLL_CONFDIR`). - Confirm against Recoll's own docs, then decide: one config dir per collection, and what owns the - mapping from collection name to config dir and document root. -2. **Cross-collection search.** Allowed at all? If yes, merged ranking across separately-scored indexes - is not free — decide before building the UI around it. -3. **Indexing trigger.** Cron inside the container, Recoll's real-time monitor, or API-triggered runs. - Affects whether the container needs a long-running process beyond the web server. -4. **Stack.** Unchosen. See `AGENTS.md` Commands for what this host actually has installed. -5. **Auth and exposure.** Whether the service authenticates at all, or assumes a reverse proxy does it. -6. **Result presentation.** Recoll returns snippets and can extract/preview documents; decide how much - of that the API surfaces, since it drives both the UI and the MCP tool shape. +1. ~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by + `metadatacmds` at index time; human names live only in SQLite; documents are never written to. +2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's. +3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite. +4. ~~Stack~~ — Python + FastAPI, managed with `uv`. +5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004. +6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate + endpoints. + +Still open, to be closed by the remaining sections of the 001 design: + +- Component boundaries and the SQLite schema. +- API surface, pagination, and error semantics. +- Test strategy, including how tests skip when the container is down. ## External dependencies @@ -53,11 +99,19 @@ The first spec must answer these. None are settled. | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. | | Docker + compose | The only supported way to run the service. | | Document volumes | Nothing to index; collections resolve to empty. | +| A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. | ## Invariants -To be established with the first code. Candidates the spec should confirm and then record here: -a query never silently spans collections; indexed content is never written into the repository. +- A query never silently spans collections: the scope is an explicit, required parameter, and the API + refuses a search that does not name one. +- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so + collections cannot bleed into one another and renaming never touches the index. +- The service never writes to the documents it indexes. The document volume is treated as read-only + even when it is not, and the deployment mounts it `:ro`. +- Document identity is the SHA-256 of file content, not its path or name: two files with the same name + and different content are two documents, and one file reachable by several paths is one document. +- Indexed content is never written into the repository. ## Known rough edges diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index cefdd71..73d61a5 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -3,6 +3,175 @@ Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back. One entry per decision a future maintainer would otherwise have to reverse-engineer. +## 2026-09-18 — Document identity is a content hash, not a path +**Status:** accepted +**Context:** the document set may contain files that share a name but differ in content, and files that +differ in name but are byte-identical. Keying membership on path merges neither and distinguishes +neither correctly. +**Decision:** a document is identified by the SHA-256 of its bytes. `documents` is keyed by that hash, +`document_paths` maps many paths to one document, and membership attaches to the document. Each document +has one **canonical path**, and only that path is reported to Recoll as belonging to any collection. +**Alternatives:** path as identity (byte-identical copies appear as separate hits for the same content); +relying on Recoll to deduplicate (it cannot — `md5` is empty and `sig` is only size + mtime). +**Consequences:** duplicate copies are indexed by Recoll but belong to no collection, so the required +collection scope makes them unreachable — verified: a scoped search over a tree with one duplicated file +returned it once. Deleting a canonical path must promote another path and force a reindex, or the +document silently leaves every collection. + +--- + +## 2026-09-18 — Membership is supplied at index time; documents are never written to +**Status:** accepted +**Supersedes:** "One index; collection membership is an opaque per-file tag" (below, same date), which +carried membership in a `user.collection` extended attribute. That entry's xattr mechanism must not be +implemented. +**Context:** the service must work with **read-only** access to the documents it indexes. Writing an +xattr requires write access, so the superseded design fails outright on a `:ro` mount — verified: +`setfattr` returns "Read-only file system". +**Decision:** membership reaches Recoll through `metadatacmds` in `recoll.conf` +(`metadatacmds = ; collection = %f`), a documented Recoll facility for external metadata. +Recoll runs the helper per file at index time; the helper answers from our own store. Everything the +superseded entry established about opaque IDs, multi-membership and single-ranked-set OR queries still +holds — only the transport changes. +**Alternatives:** filtering Recoll's results against SQLite after the query (breaks paging and total +counts); one index per collection (cannot express two files in one directory belonging to different +collections); requiring a writable document volume (contradicts the requirement). +**Consequences:** SQLite becomes the *only* store of membership rather than the authoritative copy of +two, so drift between store and index is now impossible and no reconcile path is needed. Extended +attributes are no longer a deployment requirement. A membership change needs +`recollindex -i -Z ` — verified that neither a plain incremental run nor `recollindex -i` notices +it, because Recoll's change signature is size + mtime and a membership change touches neither. +The helper is spawned once per file per index run, so it must start fast: over 400 files, indexing took +0.09s with no helper, 0.13s with a shell+awk helper and 0.82s with a `python3` helper. The service +exports a flat map from SQLite for a small non-Python helper to read. + +--- + +## 2026-09-18 — Search results carry metadata and snippets, not full text +**Status:** accepted +**Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the +original file. The response shape drives both the web UI and the MCP tool. +**Decision:** a search hit carries title, author, path, MIME type, size, date, score, its collections and +Recoll's snippets. Extracted text and file download are separate endpoints. +**Alternatives:** metadata only (a results list without context is markedly worse, and an MCP client +would fetch every candidate to judge it, costing more overall); metadata plus full extracted text +(responses become enormous for book-length PDFs and blow an agent's context window). +**Consequences:** one search call is enough for an agent to judge relevance. The UI needs a second call +to open a document, which is the right trade. + +--- + +## 2026-09-18 — Authentication in 001, user administration in 004 +**Status:** accepted +**Context:** the service needs real auth, but roles, user CRUD and per-collection permissions are a +subsystem in their own right, and `PRODUCT.md` currently describes a single technical operator. +**Decision:** 001 ships genuine authentication — argon2 password hashing, server-side sessions, an admin +account seeded from the environment at first boot, every endpoint behind an auth dependency. User +administration and permissions are deferred to spec 004. +**Alternatives:** no auth in the service, trusting a reverse proxy (a misconfigured proxy or an exposed +Docker port would leave the destructive endpoints naked); the full identity system in 001 (a large +addition to an already-grown spec, building for users who do not exist yet). +**Consequences:** 001 is safe to expose behind a proxy. 004 must extend the auth layer without +reshaping endpoints, so the auth dependency is designed as a seam from the start. + +--- + +## 2026-09-18 — Search scope is an explicit, required parameter +**Status:** accepted +**Context:** `docs/ARCHITECTURE.md` proposed the invariant that a query never silently spans +collections. A Recoll query with no collection clause returns every document, including untagged ones. +**Decision:** every search names one or more collections; omitting the parameter is an error, not an +implicit "all". Multiple collections are OR-ed into one Xapian query so ranking stays coherent and each +hit reports which collections it belongs to. +**Alternatives:** single-collection-only searches (the UI and MCP would be built around a model they +later outgrow); a searchable-by-default flag (reintroduces the implicit scope the invariant forbids); +per-collection fan-out and application-side merging (scores from separate databases are not comparable, +so the merged order would be strictly worse than letting Xapian rank). +**Consequences:** the API layer enforces the invariant, because Recoll will not. Every search path needs +a test proving an unscoped request is rejected. + +--- + +## 2026-09-18 — Indexing runs as an in-process job queue +**Status:** accepted +**Context:** collections are created at runtime, so something must index a collection the moment it is +created, and report progress back to the UI and to MCP clients. +**Decision:** creating a collection, changing membership, or calling the reindex endpoint enqueues a job. +A background worker in the same FastAPI process runs `recollindex` and records status, progress and +errors in SQLite. One container process, no broker. +**Alternatives:** `recollindex -m` real-time monitoring (one supervised process per collection, inotify +watch limits, and a new collection still needs a first full index from somewhere — revisit as an opt-in +mode later); cron-scheduled full runs (new collections sit empty until the next tick, and there is no +way to force a refresh). +**Consequences:** jobs die with the process, so the queue must recover interrupted runs at startup. +Job state is queryable, so "indexing, 40%" is available to every consumer. + +--- + +## 2026-09-18 — Collections are runtime-managed through the API +**Status:** accepted +**Context:** collections could be declared in a mounted config file or managed at runtime. +**Decision:** collections are created, edited and deleted through the HTTP API, with SQLite as the store +of record. The service creates and maintains the index state on demand. +**Alternatives:** a declarative mounted `collections.yaml` (reproducible and git-friendly, but no +management UI without adding a write path later); one collection per subdirectory by convention (forces +the on-disk layout to match the collection layout, which the per-file membership requirement rules out). +**Consequences:** 001 grows a persistence layer, validation and migrations. `AGENTS.md` makes deleting +or rebuilding an index an ask-first operation, so the delete path needs an explicit guard rather than a +bare `DELETE`. + +--- + +## 2026-09-18 — One index; collection membership is an opaque per-file tag +**Status:** accepted +**Context:** a collection cannot be "a directory", because files sharing a directory must be able to +belong to different collections. Verified in a throwaway Recoll 1.43.0 container — details and the +exact config in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics". +**Decision:** index every document once into a single Recoll index. Membership is a `user.collection` +extended attribute mapped to a queryable Recoll field. The tag is an **opaque single-token ID**; the +human-readable name lives only in SQLite. +**Alternatives:** one index per collection with rule-based selection (a file in two collections is +stored and indexed twice and can be returned twice, and hand-picking files in a shared directory is +awkward); an explicit file list driving indexing (does not map onto Recoll's topdirs-walking model). +**Why opaque IDs:** Recoll tokenises the field value and matches case-insensitively, so a tag of +`cook-books` is also matched by `collection:cook` and `collection:books`, and `TechBooks` by +`collection:techbooks`. Names-as-tags would let collections silently bleed into each other — exactly +what the no-silent-spanning invariant forbids. Opaque IDs tested clean, with no partial matches. +**Consequences:** collection names need no character restrictions and renaming never touches the index. +Documents must sit on a volume supporting extended attributes, checked at startup — without it every +scoped search silently returns nothing. Retagging costs `recollindex -i `, not a rebuild, because +a plain incremental run does not notice xattr-only changes. + +--- + +## 2026-09-18 — Stack: Python + FastAPI, managed with uv +**Status:** accepted +**Context:** the stack was unchosen and is an ask-first decision. Recoll ships first-class Python +bindings; every other language must drive `recollq` as a subprocess and parse its output. +**Decision:** Python + FastAPI for the engine and HTTP API, with `uv` as the package manager. `uv` +0.12.16 was installed on this host as part of this decision, and `AGENTS.md` updated in the same commit. +**Alternatives:** TypeScript on Node/Bun or C# on ASP.NET Core (one language across UI, API and MCP, but +both make a fragile output parser load-bearing); a split Python engine behind a TypeScript API (two +runtimes and an extra hop inside one container). +**Consequences:** structured results with no parsing layer, and OpenAPI for free, which specs 002 and 003 +can generate clients from. The web UI will likely be a second language; that boundary is the HTTP API. + +--- + +## 2026-09-18 — The product is four specs, built in order +**Status:** accepted +**Context:** "a web UI and an MCP server over Recoll" is four subsystems. One spec covering all of them +would be too large to plan or review, and `PRODUCT.md` is still an uninterviewed stub, so any UI spec +written now would design against fiction. +**Decision:** 001 search engine + API, 002 web UI, 003 MCP server, 004 multi-user management. Each gets +its own spec, plan and review cycle. The table lives in `docs/ARCHITECTURE.md`. +**Alternatives:** one thin vertical slice through all four (would lock in a stack and a UI before +`PRODUCT.md` is real); engine plus MCP together (defensible, but it widens 001, which has already grown). +**Consequences:** 002 is blocked on `/impeccable init`. 001's HTTP API is the contract 002 and 003 are +both built against, so it is the piece that must be got right. + +--- + ## 2026-09-18 — Override the superpowers artefact paths **Status:** accepted **Context:** superpowers 6.3.0 writes specs to `docs/superpowers/specs/YYYY-MM-DD--design.md` diff --git a/docs/TODO.md b/docs/TODO.md index 8476ec3..ff46f6b 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -3,18 +3,60 @@ Durable backlog. Agents append here instead of leaving `TODO` comments in source. Small, specific, independently actionable. Remove only when actually done. +## Resume here + +Last touched 2026-09-18. Still greenfield: **no application code exists.** + +Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review): + +- Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with + reasoning in `docs/DECISIONS.md`. +- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** It has had its self-review + pass but **has not been signed off by the maintainer.** Do not start a plan or write code against it + until its front matter says `approved`. +- Plan `docs/plans/001-search-engine-api.md`: **not written.** It is the next artefact, and only after + sign-off. + +The next action is to ask the maintainer to review the spec. If they approve it, set `status: approved` +and invoke the superpowers `writing-plans` skill to produce `docs/plans/001-search-engine-api.md`, +numbered to match. + +Things a resuming agent will otherwise get wrong: + +- The design carried collection membership in a `user.collection` extended attribute for part of this + session. **That is superseded** — the documents are read-only. `docs/DECISIONS.md` has both entries, + newest first; the older one is left intact by that file's append-only rule, so read the newer one. +- The Recoll facts in `docs/ARCHITECTURE.md` -> "Verified Recoll mechanics" were measured in a container, + not recalled. Trust them over intuition, and re-verify only if Recoll's major version moves. +- The throwaway spike container is not part of the repo and does not survive the session. Everything + worth keeping from it is already in `docs/`. +- `uv` was installed on this host during this session and `AGENTS.md` was corrected accordingly. The + `Commands` section of `AGENTS.md` is still a `TODO(maintainer)` stub and must be filled in the same + commit that scaffolds the app, with every command actually run first. + ## Now -- [ ] Answer the open questions in the bootstrap report, then fill the `Commands` section of `AGENTS.md` - and delete its `TODO(maintainer)` line. -- [ ] Brainstorm -> spec `docs/specs/001-*.md` covering the open design questions in - `docs/ARCHITECTURE.md`, starting with collection isolation and the stack choice. +- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in + `docs/DECISIONS.md`. +- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics, + test strategy. Then write `docs/specs/001-search-engine-api.md`. +- [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in + the same commit that scaffolds the app, with every command actually run. ## Next -- [ ] Scaffold the chosen stack; verify install/dev/test/lint/typecheck/build commands by running them, +- [ ] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them, then record the verified ones in `AGENTS.md`. -- [ ] Dockerfile + compose: Recoll in the image, volumes for documents and indexes. +- [ ] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for + the index and SQLite. +- [ ] Write the `metadatacmds` membership helper as a fast-starting non-Python program, fed by a flat + map exported from SQLite. A `python3` helper made a 400-file index 9x slower; Recoll spawns it + once per file per index run. +- [ ] Re-measure indexing overhead against real documents (PDF, EPUB) rather than small text files — + extraction cost should dwarf the helper, but that is an assumption until measured. +- [ ] Promote a new canonical path and force a reindex when a canonical path disappears, or the + document silently leaves every collection it is in. +- [ ] Recover interrupted indexing jobs at startup; they do not survive a process restart on their own. - [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial - brief, not an interview. + brief, not an interview. Blocks spec 002. ## Someday - [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/` diff --git a/docs/specs/001-search-engine-api.md b/docs/specs/001-search-engine-api.md new file mode 100644 index 0000000..7b05241 --- /dev/null +++ b/docs/specs/001-search-engine-api.md @@ -0,0 +1,175 @@ +--- +status: draft +plan: docs/plans/001-search-engine-api.md (not yet written) +supersedes: none +--- + +# 001 — Search engine and HTTP API + +The first of four specs (see `docs/ARCHITECTURE.md`, "Delivery order"). It delivers the Recoll layer and +the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI. + +## Why + +Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold +several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run +as a container on a server. The API is the contract two later consumers depend on, so it is the piece +that must be got right first. + +## Scope + +**In:** Recoll in Docker · one index over all documents · collections created and managed at runtime · +per-file collection membership · content-hash document identity · an indexing job queue · a scoped +search endpoint · authentication · a read-only file browse endpoint · health. + +**Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles +and per-collection permissions (004) · fetching a document's extracted text or downloading the original +file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth. + +## Constraints + +These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0 +and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; the reasoning is in +`docs/DECISIONS.md`. + +- **C1. The documents are read-only.** The service must operate with read-only access to everything it + indexes and must never write to a document, its directory, or its extended attributes. The deployment + mounts the document volume `:ro`. +- **C2. Membership is per file, not per directory.** Two files in the same directory may belong to + different collections. +- **C3. Documents identical in content are one document.** Documents sharing a name but differing in + content are separate documents. +- **C4. A search never silently spans collections.** + +## Requirements + +### Collections + +- **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and + an optional description. The id is generated by the service and is never supplied by a client. +- **R2.** Names are unique case-insensitively and may contain any characters, because a name never + reaches the index. +- **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the + database and enqueues no indexing work. +- **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion + removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so + the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds + the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless, + because the API resolves ids against the database and drops any it no longer knows. + +### Documents and membership + +- **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one + document; each document has exactly one canonical path. +- **R6.** Membership attaches to a document, not a path. A document may belong to any number of + collections, including none. +- **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection. + Non-canonical duplicate paths remain in the index but belong to no collection and are therefore + unreachable by any scoped search. +- **R8.** If a canonical path disappears, the service promotes another known path for that document and + forces a reindex of it. A document with no remaining path is marked missing, not deleted. +- **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work + for the affected canonical paths only. + +### Indexing + +- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time. + The helper is a fast-starting non-Python program reading a flat map exported from the database. +- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths. + Neither a plain incremental run nor `recollindex -i` detects such a change. +- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection, + changing membership, and the reindex endpoint all enqueue jobs. +- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while + it runs. +- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either + re-enqueued or failed with a reason, never left claiming to be in progress. + +### Search + +- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection + **ids**, not names — names are mutable and would break saved queries on rename. A request without the + parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is + rejected with `collection_not_found` rather than silently ignored. +- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently + ranked set rather than merged per collection. +- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it + belongs to, and Recoll's keyword-in-context snippets. +- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports. + Xapian estimates this figure for large result sets, so it is documented as approximate rather than + presented as exact. + +### Authentication + +- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session. +- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an + HttpOnly cookie. +- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are + supplied and no user exists, the service refuses to start rather than starting unauthenticated. + +### Operations + +- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring + authentication. +- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add. +- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the + table below. + +| Code | Status | +|---|---| +| `scope_required` | 400 | +| `unauthenticated` | 401 | +| `collection_not_found` | 404 | +| `collection_name_taken` | 409 | +| `confirmation_required` | 409 | +| `validation_failed` | 422 | +| `engine_unavailable` | 503 | + +## Shape + +Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The +one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** +Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run +without a container. + +## Testing + +- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. +- One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when + the container is unreachable; it never fails, and it never silently passes. +- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it. +- Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a + decision first, because `AGENTS.md` forbids committing indexed documents. + +## Acceptance criteria + +- [ ] A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4). +- [ ] Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4). +- [ ] Two files in the same directory can belong to different collections, and each is found only by its + own collection's scope (C2, R6). +- [ ] Two byte-identical files under different names are one document and appear once in a scoped + search (C3, R5, R7). +- [ ] Two files sharing a name with different content are two documents and both are findable (C3). +- [ ] Removing a canonical path promotes another path and the document remains findable (R8). +- [ ] A search without `collections` is rejected with `scope_required` (C4, R15). +- [ ] A search across two collections returns one ranked set, with each hit reporting its collections + (R16, R17). +- [ ] A membership change is visible to search after its job completes, without the document being + written to (C1, R10, R11). +- [ ] Indexing a document volume mounted `:ro` succeeds (C1). +- [ ] A job interrupted by a restart is not left claiming to be in progress (R14). +- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19). +- [ ] The service refuses to start with no user and no seeded credentials (R21). +- [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than + fails with the container down. + +## Known gaps + +Accepted deliberately, recorded so they are not mistaken for oversights. + +- **No way to read a document.** Search reports that a file matched but offers no extracted text and no + download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is + 002's first requirement. +- **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected + to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is + tracked in `docs/TODO.md`. +- **No per-user access control.** Every authenticated user sees every collection until 004.