recoll-modern/docs/specs/001-search-engine-api.md

status, plan, supersedes

status	plan	supersedes
draft	docs/plans/001-search-engine-api.md (not yet written)	none

001 — Search engine and HTTP API

The first of four specs (see docs/ARCHITECTURE.md, "Delivery order"). It delivers the Recoll layer and the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI.

Why

Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run as a container on a server. The API is the contract two later consumers depend on, so it is the piece that must be got right first.

Scope

In: Recoll in Docker · one index over all documents · collections created and managed at runtime · per-file collection membership · content-hash document identity · an indexing job queue · a scoped search endpoint · authentication · a read-only file browse endpoint · health.

Out, deferred to a named spec: the web UI (002) · the MCP server (003) · user administration, roles and per-collection permissions (004) · fetching a document's extracted text or downloading the original file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth.

Constraints

These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0 and are recorded in docs/ARCHITECTURE.md under "Verified Recoll mechanics"; the reasoning is in docs/DECISIONS.md.

• C1. The documents are read-only. The service must operate with read-only access to everything it indexes and must never write to a document, its directory, or its extended attributes. The deployment mounts the document volume :ro.
• C2. Membership is per file, not per directory. Two files in the same directory may belong to different collections.
• C3. Documents identical in content are one document. Documents sharing a name but differing in content are separate documents.
• C4. A search never silently spans collections.

Requirements

Collections

• R1. A collection has an opaque immutable id matching ^c[0-9a-f]{10}$, a human-readable name, and an optional description. The id is generated by the service and is never supplied by a client.
• R2. Names are unique case-insensitively and may contain any characters, because a name never reaches the index.
• R3. Collections are created, read, renamed and deleted over the API. Renaming touches only the database and enqueues no indexing work.
• R4. Deleting a collection requires a confirmation value matching the collection's name. Deletion removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless, because the API resolves ids against the database and drops any it no longer knows.

Documents and membership

• R5. A document is identified by the SHA-256 of its bytes. Several paths may resolve to one document; each document has exactly one canonical path.
• R6. Membership attaches to a document, not a path. A document may belong to any number of collections, including none.
• R7. Only a document's canonical path is reported to Recoll as belonging to a collection. Non-canonical duplicate paths remain in the index but belong to no collection and are therefore unreachable by any scoped search.
• R8. If a canonical path disappears, the service promotes another known path for that document and forces a reindex of it. A document with no remaining path is marked missing, not deleted.
• R9. Adding or removing documents from a collection updates the database and enqueues indexing work for the affected canonical paths only.

Indexing

• R10. Membership reaches Recoll through metadatacmds, which Recoll runs per file at index time. The helper is a fast-starting non-Python program reading a flat map exported from the database.
• R11. A membership change is applied with recollindex -i -Z on the affected canonical paths. Neither a plain incremental run nor recollindex -i detects such a change.
• R12. Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection, changing membership, and the reindex endpoint all enqueue jobs.
• R13. A job records kind, state, progress, total and any error, and is queryable over the API while it runs.
• R14. Jobs interrupted by a process restart are recovered at startup: a job left running is either re-enqueued or failed with a reason, never left claiming to be in progress.

Search

• R15. GET /search requires a non-empty collections parameter holding one or more collection ids, not names — names are mutable and would break saved queries on rename. A request without the parameter is rejected with scope_required; there is no implicit "all". An id that does not exist is rejected with collection_not_found rather than silently ignored.
• R16. Several ids are combined into a single Recoll query, so results come back as one coherently ranked set rather than merged per collection.
• R17. A hit carries title, author, path, MIME type, size, date, relevance score, the collections it belongs to, and Recoll's keyword-in-context snippets.
• R18. Results are paginated by limit and offset and report the match count Recoll reports. Xapian estimates this figure for large result sets, so it is documented as approximate rather than presented as exact.

Authentication

• R19. Every endpoint except /health and /auth/login requires an authenticated session.
• R20. Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an HttpOnly cookie.
• R21. An initial administrator is seeded from the environment at first boot. If no credentials are supplied and no user exists, the service refuses to start rather than starting unauthenticated.

Operations

• R22. GET /health reports whether Recoll is reachable and the index readable, without requiring authentication.
• R23. GET /files lists the document volume read-only so a client can choose files to add.
• R24. Errors use one envelope, {"error": {"code", "message", "details"}}, with the codes in the table below.

Code	Status
scope_required	400
unauthenticated	401
collection_not_found	404
collection_name_taken	409
confirmation_required	409
validation_failed	422
engine_unavailable	503

Shape

Module boundaries, the SQLite schema and the endpoint list are recorded in docs/ARCHITECTURE.md. The one rule worth restating here: only recoll_modern/engine/ may import recoll or run recollindex. Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run without a container.

Testing

• Domain, API and job-queue tests run against a fake engine and real SQLite, with no container.
• One contract suite runs twice: against the fake, and against real Recoll. The real run skips when the container is unreachable; it never fails, and it never silently passes.
• An end-to-end test indexes a generated corpus on a :ro mount and searches it.
• Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a decision first, because AGENTS.md forbids committing indexed documents.

Acceptance criteria

• A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4).
• Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4).
• Two files in the same directory can belong to different collections, and each is found only by its own collection's scope (C2, R6).
• Two byte-identical files under different names are one document and appear once in a scoped search (C3, R5, R7).
• Two files sharing a name with different content are two documents and both are findable (C3).
• Removing a canonical path promotes another path and the document remains findable (R8).
• A search without collections is rejected with scope_required (C4, R15).
• A search across two collections returns one ranked set, with each hit reporting its collections (R16, R17).
• A membership change is visible to search after its job completes, without the document being written to (C1, R10, R11).
• Indexing a document volume mounted :ro succeeds (C1).
• A job interrupted by a restart is not left claiming to be in progress (R14).
• An unauthenticated request to any endpoint except /health and /auth/login is refused (R19).
• The service refuses to start with no user and no seeded credentials (R21).
• uv run pytest, uv run ruff check . and uv run mypy src pass, and the suite skips rather than fails with the container down.

Known gaps

Accepted deliberately, recorded so they are not mistaken for oversights.

• No way to read a document. Search reports that a file matched but offers no extracted text and no download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is 002's first requirement.
• Indexing overhead is measured only on small text files. The metadatacmds helper cost is expected to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is tracked in docs/TODO.md.
• No per-user access control. Every authenticated user sees every collection until 004.