recoll-modern/docs/specs/001-search-engine-api.md

---
status: draft
plan: docs/plans/001-search-engine-api.md (not yet written)
supersedes: none
---

# 001 — Search engine and HTTP API

The first of four specs (see `docs/ARCHITECTURE.md`, "Delivery order"). It delivers the Recoll layer and
the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI.

## Why

Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold
several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run
as a container on a server. The API is the contract two later consumers depend on, so it is the piece
that must be got right first.

## Scope

**In:** Recoll in Docker · one index over all documents · collections created and managed at runtime ·
per-file collection membership · content-hash document identity · an indexing job queue · a scoped
search endpoint · authentication · a read-only file browse endpoint · health.

**Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles
and per-collection permissions (004) · fetching a document's extracted text or downloading the original
file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth.

## Constraints

These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0
and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; the reasoning is in
`docs/DECISIONS.md`.

- **C1. The documents are read-only.** The service must operate with read-only access to everything it
  indexes and must never write to a document, its directory, or its extended attributes. The deployment
  mounts the document volume `:ro`.
- **C2. Membership is per file, not per directory.** Two files in the same directory may belong to
  different collections.
- **C3. Documents identical in content are one document.** Documents sharing a name but differing in
  content are separate documents.
- **C4. A search never silently spans collections.**

## Requirements

### Collections

- **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and
  an optional description. The id is generated by the service and is never supplied by a client.
- **R2.** Names are unique case-insensitively and may contain any characters, because a name never
  reaches the index.
- **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the
  database and enqueues no indexing work.
- **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion
  removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so
  the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds
  the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless,
  because the API resolves ids against the database and drops any it no longer knows.

### Documents and membership

- **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one
  document; each document has exactly one canonical path.
- **R6.** Membership attaches to a document, not a path. A document may belong to any number of
  collections, including none.
- **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection.
  Non-canonical duplicate paths remain in the index but belong to no collection and are therefore
  unreachable by any scoped search.
- **R8.** If a canonical path disappears, the service promotes another known path for that document and
  forces a reindex of it. A document with no remaining path is marked missing, not deleted.
- **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work
  for the affected canonical paths only.

### Indexing

- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
  The helper is a fast-starting non-Python program reading a flat map exported from the database.
- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
  Neither a plain incremental run nor `recollindex -i` detects such a change.
- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection,
  changing membership, and the reindex endpoint all enqueue jobs.
- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while
  it runs.
- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either
  re-enqueued or failed with a reason, never left claiming to be in progress.

### Search

- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
  **ids**, not names — names are mutable and would break saved queries on rename. A request without the
  parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is
  rejected with `collection_not_found` rather than silently ignored.
- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently
  ranked set rather than merged per collection.
- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it
  belongs to, and Recoll's keyword-in-context snippets.
- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
  Xapian estimates this figure for large result sets, so it is documented as approximate rather than
  presented as exact.

### Authentication

- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
  HttpOnly cookie.
- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are
  supplied and no user exists, the service refuses to start rather than starting unauthenticated.

### Operations

- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
  authentication.
- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add.
- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
  table below.

| Code | Status |
|---|---|
| `scope_required` | 400 |
| `unauthenticated` | 401 |
| `collection_not_found` | 404 |
| `collection_name_taken` | 409 |
| `confirmation_required` | 409 |
| `validation_failed` | 422 |
| `engine_unavailable` | 503 |

## Shape

Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The
one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.**
Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run
without a container.

## Testing

- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container.
- One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when
  the container is unreachable; it never fails, and it never silently passes.
- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it.
- Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a
  decision first, because `AGENTS.md` forbids committing indexed documents.

## Acceptance criteria

- [ ] A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4).
- [ ] Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4).
- [ ] Two files in the same directory can belong to different collections, and each is found only by its
      own collection's scope (C2, R6).
- [ ] Two byte-identical files under different names are one document and appear once in a scoped
      search (C3, R5, R7).
- [ ] Two files sharing a name with different content are two documents and both are findable (C3).
- [ ] Removing a canonical path promotes another path and the document remains findable (R8).
- [ ] A search without `collections` is rejected with `scope_required` (C4, R15).
- [ ] A search across two collections returns one ranked set, with each hit reporting its collections
      (R16, R17).
- [ ] A membership change is visible to search after its job completes, without the document being
      written to (C1, R10, R11).
- [ ] Indexing a document volume mounted `:ro` succeeds (C1).
- [ ] A job interrupted by a restart is not left claiming to be in progress (R14).
- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19).
- [ ] The service refuses to start with no user and no seeded credentials (R21).
- [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than
      fails with the container down.

## Known gaps

Accepted deliberately, recorded so they are not mistaken for oversights.

- **No way to read a document.** Search reports that a file matched but offers no extracted text and no
  download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is
  002's first requirement.
- **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected
  to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is
  tracked in `docs/TODO.md`.
- **No per-user access control.** Every authenticated user sees every collection until 004.