Merge branch 'feat/001-search-engine-api'

Brainstorm, verified Recoll spike findings, decision log and draft spec 001.
No application code; spec 001 is status: draft and awaiting sign-off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -15,8 +15,9 @@ TODO(maintainer): no stack yet. Fill install/dev/test/single-test/lint/typecheck
 commit that scaffolds the app, and delete this line. These must be what CI runs.
 
 Installed on this host: `docker` 29.5.2 with `docker compose`, `node` 20.19.2, `npm` 9.2.0, `bun` 1.3.14,
-`yarn` 1.22.22, `python3` 3.13.5, `dotnet` 10.0.302.
-**Absent: `pnpm`, `uv`, `poetry`, `go`, `cargo`** — do not write commands that assume them.
+`yarn` 1.22.22, `python3` 3.13.5, `uv` 0.12.16 (in `~/.local/bin`, already on `PATH`), `dotnet` 10.0.302.
+**Absent: `pnpm`, `poetry`, `go`, `cargo`** — do not write commands that assume them.
+`uv` is the package manager for this project; do not use bare `pip`, `venv`, or `poetry`.
 
 ## Rules
 

docs/ARCHITECTURE.md

@@ -4,8 +4,9 @@ How `recoll-modern` is built today. Update in the same commit as any structural
 
 ## Overview
 
-**Nothing is built yet.** This file currently records intent and the open questions that the first
-spec must close. Do not read anything below "Intended shape" as describing existing code.
+**Nothing is built yet.** This file records intent, the design questions that are now closed, and the
+Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing
+code.
 
 The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
 web UI and (b) an MCP server, running in Docker, with several independent collections such as
@@ -21,30 +22,75 @@ web UI and (b) an MCP server, running in Docker, with several independent collec
 
 No source directories exist yet. Add rows here as they appear.
 
+## Delivery order
+
+The product is four specs, built in order. Each gets its own spec, plan and review cycle.
+
+| Spec | Scope |
+|---|---|
+| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication |
+| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` |
+| 003 MCP server | Thin wrapper over 001's HTTP API |
+| 004 Multi-user management | Roles, user CRUD, per-collection permissions |
+
 ## Intended shape
 
-Not yet decided or validated. The pieces the spec has to place:
-
-- A Recoll layer that owns indexing and querying, per collection.
+- **One** Recoll index for all documents, not one per collection. Collection membership is per-file
+  metadata supplied by the service at index time, not directory layout, so two files in the same
+  directory can belong to different collections and the documents are never modified.
+- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`).
+- SQLite holds collections, membership, indexing jobs, users and sessions.
 - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
-- An MCP server exposing search (and likely collection listing) as tools.
-- A Docker image that contains Recoll itself, plus compose wiring for document volumes and index storage.
+- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
+
+## Verified Recoll mechanics
+
+Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running
+Recoll 1.43.0 / Xapian 1.4.29, with the document tree bind-mounted **read-only**. Re-verify if the
+Recoll major version changes.
+
+- **Documents are never written to.** Membership reaches the index through `metadatacmds` in
+  `recoll.conf` (`metadatacmds = ; collection = <helper> %f`), which Recoll runs per file at index time
+  to fetch external metadata. The helper answers from our own store. Indexing a read-only mount
+  succeeds; `touch` and `setfattr` on it both fail, as they must.
+- **Extended attributes are NOT usable.** An earlier design carried membership in a `user.collection`
+  xattr. It works only when the documents are writable, so it is superseded — see `docs/DECISIONS.md`.
+- **Multi-membership works.** A space-separated helper response puts one file in several collections.
+  An `OR` across collections returns it **once**, in a single coherently ranked set: Xapian does the
+  ranking, so no application-side merging is needed.
+- **Membership changes need a forced per-file reindex.** Neither a plain incremental run nor
+  `recollindex -i <file>` notices a change that did not touch the file — Recoll's change signature is
+  size + mtime (observed as `sig` = `'<size><mtime>'`), and a membership change alters neither.
+  `recollindex -i -Z <file>` does pick it up, at roughly 15ms per file, and accepts several paths.
+- **The field value is tokenised and matched case-insensitively.** A value of `cook-books` is matched by
+  `collection:cook` *and* `collection:books`; `TechBooks` by `collection:techbooks`. Collection
+  identifiers must therefore be opaque single tokens with no separators. Tested clean with `^c[0-9a-f]+$`.
+- **Recoll has no content hash.** `md5` is empty and `sig` is only size + mtime, so document identity by
+  content is ours to compute and store.
+- **An unscoped query returns every document**, including ones belonging to no collection. Recoll will
+  not enforce collection scoping; the API layer must.
+- **The helper must start fast.** Recoll spawns it once per file per index run. Over 400 files: 0.09s
+  with no helper, 0.13s for a shell+awk helper, 0.82s for a `python3` helper — interpreter startup
+  dominates. The helper reads a flat map file exported from SQLite; it is not a Python process.
 
 ## Open design questions
 
-The first spec must answer these. None are settled.
+Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning:
 
-1. **Collection isolation.** Recoll's model is a config directory per index (`RECOLL_CONFDIR`).
-   Confirm against Recoll's own docs, then decide: one config dir per collection, and what owns the
-   mapping from collection name to config dir and document root.
-2. **Cross-collection search.** Allowed at all? If yes, merged ranking across separately-scored indexes
-   is not free — decide before building the UI around it.
-3. **Indexing trigger.** Cron inside the container, Recoll's real-time monitor, or API-triggered runs.
-   Affects whether the container needs a long-running process beyond the web server.
-4. **Stack.** Unchosen. See `AGENTS.md` Commands for what this host actually has installed.
-5. **Auth and exposure.** Whether the service authenticates at all, or assumes a reverse proxy does it.
-6. **Result presentation.** Recoll returns snippets and can extract/preview documents; decide how much
-   of that the API surfaces, since it drives both the UI and the MCP tool shape.
+1. ~~Collection isolation~~ — one index; membership is an opaque collection ID supplied to Recoll by
+   `metadatacmds` at index time; human names live only in SQLite; documents are never written to.
+2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's.
+3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite.
+4. ~~Stack~~ — Python + FastAPI, managed with `uv`.
+5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004.
+6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate
+   endpoints.
+
+Still open, to be closed by the remaining sections of the 001 design:
+
+- Component boundaries and the SQLite schema.
+- API surface, pagination, and error semantics.
+- Test strategy, including how tests skip when the container is down.
 
 ## External dependencies
 
@@ -53,11 +99,19 @@ The first spec must answer these. None are settled.
 | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. |
 | Docker + compose | The only supported way to run the service. |
 | Document volumes | Nothing to index; collections resolve to empty. |
+| A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. |
 
 ## Invariants
 
-To be established with the first code. Candidates the spec should confirm and then record here:
-a query never silently spans collections; indexed content is never written into the repository.
+- A query never silently spans collections: the scope is an explicit, required parameter, and the API
+  refuses a search that does not name one.
+- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so
+  collections cannot bleed into one another and renaming never touches the index.
+- The service never writes to the documents it indexes. The document volume is treated as read-only
+  even when it is not, and the deployment mounts it `:ro`.
+- Document identity is the SHA-256 of file content, not its path or name: two files with the same name
+  and different content are two documents, and one file reachable by several paths is one document.
+- Indexed content is never written into the repository.
 
 ## Known rough edges
 

docs/DECISIONS.md

@@ -3,6 +3,175 @@
 Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back.
 One entry per decision a future maintainer would otherwise have to reverse-engineer.
 
+## 2026-09-18 — Document identity is a content hash, not a path
+**Status:** accepted
+**Context:** the document set may contain files that share a name but differ in content, and files that
+differ in name but are byte-identical. Keying membership on path merges neither and distinguishes
+neither correctly.
+**Decision:** a document is identified by the SHA-256 of its bytes. `documents` is keyed by that hash,
+`document_paths` maps many paths to one document, and membership attaches to the document. Each document
+has one **canonical path**, and only that path is reported to Recoll as belonging to any collection.
+**Alternatives:** path as identity (byte-identical copies appear as separate hits for the same content);
+relying on Recoll to deduplicate (it cannot — `md5` is empty and `sig` is only size + mtime).
+**Consequences:** duplicate copies are indexed by Recoll but belong to no collection, so the required
+collection scope makes them unreachable — verified: a scoped search over a tree with one duplicated file
+returned it once. Deleting a canonical path must promote another path and force a reindex, or the
+document silently leaves every collection.
+
+---
+
+## 2026-09-18 — Membership is supplied at index time; documents are never written to
+**Status:** accepted
+**Supersedes:** "One index; collection membership is an opaque per-file tag" (below, same date), which
+carried membership in a `user.collection` extended attribute. That entry's xattr mechanism must not be
+implemented.
+**Context:** the service must work with **read-only** access to the documents it indexes. Writing an
+xattr requires write access, so the superseded design fails outright on a `:ro` mount — verified:
+`setfattr` returns "Read-only file system".
+**Decision:** membership reaches Recoll through `metadatacmds` in `recoll.conf`
+(`metadatacmds = ; collection = <helper> %f`), a documented Recoll facility for external metadata.
+Recoll runs the helper per file at index time; the helper answers from our own store. Everything the
+superseded entry established about opaque IDs, multi-membership and single-ranked-set OR queries still
+holds — only the transport changes.
+**Alternatives:** filtering Recoll's results against SQLite after the query (breaks paging and total
+counts); one index per collection (cannot express two files in one directory belonging to different
+collections); requiring a writable document volume (contradicts the requirement).
+**Consequences:** SQLite becomes the *only* store of membership rather than the authoritative copy of
+two, so drift between store and index is now impossible and no reconcile path is needed. Extended
+attributes are no longer a deployment requirement. A membership change needs
+`recollindex -i -Z <path>` — verified that neither a plain incremental run nor `recollindex -i` notices
+it, because Recoll's change signature is size + mtime and a membership change touches neither.
+The helper is spawned once per file per index run, so it must start fast: over 400 files, indexing took
+0.09s with no helper, 0.13s with a shell+awk helper and 0.82s with a `python3` helper. The service
+exports a flat map from SQLite for a small non-Python helper to read.
+
+---
+
+## 2026-09-18 — Search results carry metadata and snippets, not full text
+**Status:** accepted
+**Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the
+original file. The response shape drives both the web UI and the MCP tool.
+**Decision:** a search hit carries title, author, path, MIME type, size, date, score, its collections and
+Recoll's snippets. Extracted text and file download are separate endpoints.
+**Alternatives:** metadata only (a results list without context is markedly worse, and an MCP client
+would fetch every candidate to judge it, costing more overall); metadata plus full extracted text
+(responses become enormous for book-length PDFs and blow an agent's context window).
+**Consequences:** one search call is enough for an agent to judge relevance. The UI needs a second call
+to open a document, which is the right trade.
+
+---
+
+## 2026-09-18 — Authentication in 001, user administration in 004
+**Status:** accepted
+**Context:** the service needs real auth, but roles, user CRUD and per-collection permissions are a
+subsystem in their own right, and `PRODUCT.md` currently describes a single technical operator.
+**Decision:** 001 ships genuine authentication — argon2 password hashing, server-side sessions, an admin
+account seeded from the environment at first boot, every endpoint behind an auth dependency. User
+administration and permissions are deferred to spec 004.
+**Alternatives:** no auth in the service, trusting a reverse proxy (a misconfigured proxy or an exposed
+Docker port would leave the destructive endpoints naked); the full identity system in 001 (a large
+addition to an already-grown spec, building for users who do not exist yet).
+**Consequences:** 001 is safe to expose behind a proxy. 004 must extend the auth layer without
+reshaping endpoints, so the auth dependency is designed as a seam from the start.
+
+---
+
+## 2026-09-18 — Search scope is an explicit, required parameter
+**Status:** accepted
+**Context:** `docs/ARCHITECTURE.md` proposed the invariant that a query never silently spans
+collections. A Recoll query with no collection clause returns every document, including untagged ones.
+**Decision:** every search names one or more collections; omitting the parameter is an error, not an
+implicit "all". Multiple collections are OR-ed into one Xapian query so ranking stays coherent and each
+hit reports which collections it belongs to.
+**Alternatives:** single-collection-only searches (the UI and MCP would be built around a model they
+later outgrow); a searchable-by-default flag (reintroduces the implicit scope the invariant forbids);
+per-collection fan-out and application-side merging (scores from separate databases are not comparable,
+so the merged order would be strictly worse than letting Xapian rank).
+**Consequences:** the API layer enforces the invariant, because Recoll will not. Every search path needs
+a test proving an unscoped request is rejected.
+
+---
+
+## 2026-09-18 — Indexing runs as an in-process job queue
+**Status:** accepted
+**Context:** collections are created at runtime, so something must index a collection the moment it is
+created, and report progress back to the UI and to MCP clients.
+**Decision:** creating a collection, changing membership, or calling the reindex endpoint enqueues a job.
+A background worker in the same FastAPI process runs `recollindex` and records status, progress and
+errors in SQLite. One container process, no broker.
+**Alternatives:** `recollindex -m` real-time monitoring (one supervised process per collection, inotify
+watch limits, and a new collection still needs a first full index from somewhere — revisit as an opt-in
+mode later); cron-scheduled full runs (new collections sit empty until the next tick, and there is no
+way to force a refresh).
+**Consequences:** jobs die with the process, so the queue must recover interrupted runs at startup.
+Job state is queryable, so "indexing, 40%" is available to every consumer.
+
+---
+
+## 2026-09-18 — Collections are runtime-managed through the API
+**Status:** accepted
+**Context:** collections could be declared in a mounted config file or managed at runtime.
+**Decision:** collections are created, edited and deleted through the HTTP API, with SQLite as the store
+of record. The service creates and maintains the index state on demand.
+**Alternatives:** a declarative mounted `collections.yaml` (reproducible and git-friendly, but no
+management UI without adding a write path later); one collection per subdirectory by convention (forces
+the on-disk layout to match the collection layout, which the per-file membership requirement rules out).
+**Consequences:** 001 grows a persistence layer, validation and migrations. `AGENTS.md` makes deleting
+or rebuilding an index an ask-first operation, so the delete path needs an explicit guard rather than a
+bare `DELETE`.
+
+---
+
+## 2026-09-18 — One index; collection membership is an opaque per-file tag
+**Status:** accepted
+**Context:** a collection cannot be "a directory", because files sharing a directory must be able to
+belong to different collections. Verified in a throwaway Recoll 1.43.0 container — details and the
+exact config in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics".
+**Decision:** index every document once into a single Recoll index. Membership is a `user.collection`
+extended attribute mapped to a queryable Recoll field. The tag is an **opaque single-token ID**; the
+human-readable name lives only in SQLite.
+**Alternatives:** one index per collection with rule-based selection (a file in two collections is
+stored and indexed twice and can be returned twice, and hand-picking files in a shared directory is
+awkward); an explicit file list driving indexing (does not map onto Recoll's topdirs-walking model).
+**Why opaque IDs:** Recoll tokenises the field value and matches case-insensitively, so a tag of
+`cook-books` is also matched by `collection:cook` and `collection:books`, and `TechBooks` by
+`collection:techbooks`. Names-as-tags would let collections silently bleed into each other — exactly
+what the no-silent-spanning invariant forbids. Opaque IDs tested clean, with no partial matches.
+**Consequences:** collection names need no character restrictions and renaming never touches the index.
+Documents must sit on a volume supporting extended attributes, checked at startup — without it every
+scoped search silently returns nothing. Retagging costs `recollindex -i <file>`, not a rebuild, because
+a plain incremental run does not notice xattr-only changes.
+
+---
+
+## 2026-09-18 — Stack: Python + FastAPI, managed with uv
+**Status:** accepted
+**Context:** the stack was unchosen and is an ask-first decision. Recoll ships first-class Python
+bindings; every other language must drive `recollq` as a subprocess and parse its output.
+**Decision:** Python + FastAPI for the engine and HTTP API, with `uv` as the package manager. `uv`
+0.12.16 was installed on this host as part of this decision, and `AGENTS.md` updated in the same commit.
+**Alternatives:** TypeScript on Node/Bun or C# on ASP.NET Core (one language across UI, API and MCP, but
+both make a fragile output parser load-bearing); a split Python engine behind a TypeScript API (two
+runtimes and an extra hop inside one container).
+**Consequences:** structured results with no parsing layer, and OpenAPI for free, which specs 002 and 003
+can generate clients from. The web UI will likely be a second language; that boundary is the HTTP API.
+
+---
+
+## 2026-09-18 — The product is four specs, built in order
+**Status:** accepted
+**Context:** "a web UI and an MCP server over Recoll" is four subsystems. One spec covering all of them
+would be too large to plan or review, and `PRODUCT.md` is still an uninterviewed stub, so any UI spec
+written now would design against fiction.
+**Decision:** 001 search engine + API, 002 web UI, 003 MCP server, 004 multi-user management. Each gets
+its own spec, plan and review cycle. The table lives in `docs/ARCHITECTURE.md`.
+**Alternatives:** one thin vertical slice through all four (would lock in a stack and a UI before
+`PRODUCT.md` is real); engine plus MCP together (defensible, but it widens 001, which has already grown).
+**Consequences:** 002 is blocked on `/impeccable init`. 001's HTTP API is the contract 002 and 003 are
+both built against, so it is the piece that must be got right.
+
+---
+
 ## 2026-09-18 — Override the superpowers artefact paths
 **Status:** accepted
 **Context:** superpowers 6.3.0 writes specs to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`

docs/TODO.md

@@ -3,18 +3,60 @@
 Durable backlog. Agents append here instead of leaving `TODO` comments in source.
 Small, specific, independently actionable. Remove only when actually done.
 
+## Resume here
+
+Last touched 2026-09-18. Still greenfield: **no application code exists.**
+
+Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review):
+
+- Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with
+  reasoning in `docs/DECISIONS.md`.
+- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** It has had its self-review
+  pass but **has not been signed off by the maintainer.** Do not start a plan or write code against it
+  until its front matter says `approved`.
+- Plan `docs/plans/001-search-engine-api.md`: **not written.** It is the next artefact, and only after
+  sign-off.
+
+The next action is to ask the maintainer to review the spec. If they approve it, set `status: approved`
+and invoke the superpowers `writing-plans` skill to produce `docs/plans/001-search-engine-api.md`,
+numbered to match.
+
+Things a resuming agent will otherwise get wrong:
+
+- The design carried collection membership in a `user.collection` extended attribute for part of this
+  session. **That is superseded** — the documents are read-only. `docs/DECISIONS.md` has both entries,
+  newest first; the older one is left intact by that file's append-only rule, so read the newer one.
+- The Recoll facts in `docs/ARCHITECTURE.md` -> "Verified Recoll mechanics" were measured in a container,
+  not recalled. Trust them over intuition, and re-verify only if Recoll's major version moves.
+- The throwaway spike container is not part of the repo and does not survive the session. Everything
+  worth keeping from it is already in `docs/`.
+- `uv` was installed on this host during this session and `AGENTS.md` was corrected accordingly. The
+  `Commands` section of `AGENTS.md` is still a `TODO(maintainer)` stub and must be filled in the same
+  commit that scaffolds the app, with every command actually run first.
+
 ## Now
-- [ ] Answer the open questions in the bootstrap report, then fill the `Commands` section of `AGENTS.md`
-      and delete its `TODO(maintainer)` line.
-- [ ] Brainstorm -> spec `docs/specs/001-*.md` covering the open design questions in
-      `docs/ARCHITECTURE.md`, starting with collection isolation and the stack choice.
+- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in
+      `docs/DECISIONS.md`.
+- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics,
+      test strategy. Then write `docs/specs/001-search-engine-api.md`.
+- [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in
+      the same commit that scaffolds the app, with every command actually run.
 
 ## Next
-- [ ] Scaffold the chosen stack; verify install/dev/test/lint/typecheck/build commands by running them,
+- [ ] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them,
       then record the verified ones in `AGENTS.md`.
-- [ ] Dockerfile + compose: Recoll in the image, volumes for documents and indexes.
+- [ ] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for
+      the index and SQLite.
+- [ ] Write the `metadatacmds` membership helper as a fast-starting non-Python program, fed by a flat
+      map exported from SQLite. A `python3` helper made a 400-file index 9x slower; Recoll spawns it
+      once per file per index run.
+- [ ] Re-measure indexing overhead against real documents (PDF, EPUB) rather than small text files —
+      extraction cost should dwarf the helper, but that is an assumption until measured.
+- [ ] Promote a new canonical path and force a reindex when a canonical path disappears, or the
+      document silently leaves every collection it is in.
+- [ ] Recover interrupted indexing jobs at startup; they do not survive a process restart on their own.
 - [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial
-      brief, not an interview.
+      brief, not an interview. Blocks spec 002.
 
 ## Someday
 - [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/`

docs/specs/001-search-engine-api.md

@@ -0,0 +1,175 @@
+---
+status: draft
+plan: docs/plans/001-search-engine-api.md (not yet written)
+supersedes: none
+---
+
+# 001 — Search engine and HTTP API
+
+The first of four specs (see `docs/ARCHITECTURE.md`, "Delivery order"). It delivers the Recoll layer and
+the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI.
+
+## Why
+
+Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold
+several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run
+as a container on a server. The API is the contract two later consumers depend on, so it is the piece
+that must be got right first.
+
+## Scope
+
+**In:** Recoll in Docker · one index over all documents · collections created and managed at runtime ·
+per-file collection membership · content-hash document identity · an indexing job queue · a scoped
+search endpoint · authentication · a read-only file browse endpoint · health.
+
+**Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles
+and per-collection permissions (004) · fetching a document's extracted text or downloading the original
+file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth.
+
+## Constraints
+
+These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0
+and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; the reasoning is in
+`docs/DECISIONS.md`.
+
+- **C1. The documents are read-only.** The service must operate with read-only access to everything it
+  indexes and must never write to a document, its directory, or its extended attributes. The deployment
+  mounts the document volume `:ro`.
+- **C2. Membership is per file, not per directory.** Two files in the same directory may belong to
+  different collections.
+- **C3. Documents identical in content are one document.** Documents sharing a name but differing in
+  content are separate documents.
+- **C4. A search never silently spans collections.**
+
+## Requirements
+
+### Collections
+
+- **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and
+  an optional description. The id is generated by the service and is never supplied by a client.
+- **R2.** Names are unique case-insensitively and may contain any characters, because a name never
+  reaches the index.
+- **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the
+  database and enqueues no indexing work.
+- **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion
+  removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so
+  the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds
+  the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless,
+  because the API resolves ids against the database and drops any it no longer knows.
+
+### Documents and membership
+
+- **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one
+  document; each document has exactly one canonical path.
+- **R6.** Membership attaches to a document, not a path. A document may belong to any number of
+  collections, including none.
+- **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection.
+  Non-canonical duplicate paths remain in the index but belong to no collection and are therefore
+  unreachable by any scoped search.
+- **R8.** If a canonical path disappears, the service promotes another known path for that document and
+  forces a reindex of it. A document with no remaining path is marked missing, not deleted.
+- **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work
+  for the affected canonical paths only.
+
+### Indexing
+
+- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
+  The helper is a fast-starting non-Python program reading a flat map exported from the database.
+- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
+  Neither a plain incremental run nor `recollindex -i` detects such a change.
+- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection,
+  changing membership, and the reindex endpoint all enqueue jobs.
+- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while
+  it runs.
+- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either
+  re-enqueued or failed with a reason, never left claiming to be in progress.
+
+### Search
+
+- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
+  **ids**, not names — names are mutable and would break saved queries on rename. A request without the
+  parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is
+  rejected with `collection_not_found` rather than silently ignored.
+- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently
+  ranked set rather than merged per collection.
+- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it
+  belongs to, and Recoll's keyword-in-context snippets.
+- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
+  Xapian estimates this figure for large result sets, so it is documented as approximate rather than
+  presented as exact.
+
+### Authentication
+
+- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
+- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
+  HttpOnly cookie.
+- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are
+  supplied and no user exists, the service refuses to start rather than starting unauthenticated.
+
+### Operations
+
+- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
+  authentication.
+- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add.
+- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
+  table below.
+
+| Code | Status |
+|---|---|
+| `scope_required` | 400 |
+| `unauthenticated` | 401 |
+| `collection_not_found` | 404 |
+| `collection_name_taken` | 409 |
+| `confirmation_required` | 409 |
+| `validation_failed` | 422 |
+| `engine_unavailable` | 503 |
+
+## Shape
+
+Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The
+one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.**
+Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run
+without a container.
+
+## Testing
+
+- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container.
+- One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when
+  the container is unreachable; it never fails, and it never silently passes.
+- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it.
+- Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a
+  decision first, because `AGENTS.md` forbids committing indexed documents.
+
+## Acceptance criteria
+
+- [ ] A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4).
+- [ ] Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4).
+- [ ] Two files in the same directory can belong to different collections, and each is found only by its
+      own collection's scope (C2, R6).
+- [ ] Two byte-identical files under different names are one document and appear once in a scoped
+      search (C3, R5, R7).
+- [ ] Two files sharing a name with different content are two documents and both are findable (C3).
+- [ ] Removing a canonical path promotes another path and the document remains findable (R8).
+- [ ] A search without `collections` is rejected with `scope_required` (C4, R15).
+- [ ] A search across two collections returns one ranked set, with each hit reporting its collections
+      (R16, R17).
+- [ ] A membership change is visible to search after its job completes, without the document being
+      written to (C1, R10, R11).
+- [ ] Indexing a document volume mounted `:ro` succeeds (C1).
+- [ ] A job interrupted by a restart is not left claiming to be in progress (R14).
+- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19).
+- [ ] The service refuses to start with no user and no seeded credentials (R21).
+- [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than
+      fails with the container down.
+
+## Known gaps
+
+Accepted deliberately, recorded so they are not mistaken for oversights.
+
+- **No way to read a document.** Search reports that a file matched but offers no extracted text and no
+  download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is
+  002's first requirement.
+- **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected
+  to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is
+  tracked in `docs/TODO.md`.
+- **No per-user access control.** Every authenticated user sees every collection until 004.