recoll-modern/docs/specs/001-search-engine-api.md

status, plan, supersedes

status	plan	supersedes
implemented	docs/plans/001-search-engine-api.md	none

001 — Search engine and HTTP API

The first of five specs (see docs/ARCHITECTURE.md, "Delivery order"). It delivers the Recoll layer and the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI.

Why

Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run as a container on a server. The API is the contract two later consumers depend on, so it is the piece that must be got right first.

Scope

In: Recoll in Docker · one index over all documents · collections created and managed at runtime · per-file collection membership · snapshot and persistent directory sources · content-hash document identity · an indexing job queue · a scoped search endpoint with caller-selected fields · authenticated text and original-file retrieval · authentication · a read-only file browse endpoint · configurable subpath hosting · health.

Out, deferred to a named spec: the web UI (002) · the MCP server (003) · user administration, roles and per-collection permissions (004) · a cross-platform installable desktop distribution (005, after 004) · real-time index monitoring · OIDC or forward-auth.

Constraints

These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0 and are recorded in docs/ARCHITECTURE.md under "Verified Recoll mechanics"; the reasoning is in docs/DECISIONS.md.

• C1. The documents are read-only. The service must operate with read-only access to everything it indexes and must never write to a document, its directory, or its extended attributes. The deployment mounts the document volume :ro.
• C2. Membership is per file, not per directory. Two files in the same directory may belong to different collections.
• C3. Documents identical in content are one document. Documents sharing a name but differing in content are separate documents.
• C4. A search never silently spans collections.

Requirements

Collections

• R1. A collection has an opaque immutable id matching ^c[0-9a-f]{10}$, a human-readable name, and an optional description. The id is generated by the service and is never supplied by a client.
• R2. Names contain at least one non-whitespace character, are unique after Unicode normalization and case-folding, and otherwise accept arbitrary Unicode because a name never reaches the index.
• R3. Collections are created, read, renamed and deleted over the API. Renaming touches only the database and enqueues no indexing work.
• R4. Deleting a collection requires a confirmation value matching the collection's name. Deletion removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless, because the API resolves ids against the database and drops any it no longer knows.

Documents and membership

• R5. A document is identified by the SHA-256 of its bytes. Several paths may resolve to one document; each present document has exactly one canonical path and a missing document has none.
• R6. Membership attaches to a document, not a path. A document may belong to any number of collections, including none.
• R7. Only a document's canonical path is reported to Recoll as belonging to a collection. Non-canonical duplicate paths remain in the index but belong to no collection and are therefore unreachable by any scoped search.
• R8. If a canonical path disappears, the service promotes another known path for that document and forces a reindex of it. A document with no remaining path is marked missing, not deleted.
• R9. Adding or removing documents from a collection updates the database and enqueues indexing work for the affected canonical paths only.
• R10. Membership sources may be individual files or directories. A directory source is either a one-time recursive snapshot or a persistent recursive rule. Persistent rules reconcile at startup and during explicit reindex jobs; they do not watch the filesystem in real time.
• R11. A directory source may name exact descendant directories to exclude recursively. Exclusions on persistent sources are scoped to that rule. Removing an individual document that is still selected by a rule creates a per-file override; direct membership remains additive and takes precedence.
• R12. Recursive discovery does not follow directory symlinks. A file symlink is accepted only when its resolved target stays beneath the configured document root.

Indexing

• R13. Membership reaches Recoll through metadatacmds, which Recoll runs per file at index time. The helper is a fast-starting non-Python program reading a flat map exported from the database.
• R14. A membership change is applied with recollindex -i -Z on the affected canonical paths. Neither a plain incremental run nor recollindex -i detects such a change.
• R15. Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection with initial sources, changing membership, changing a persistent rule, and the reindex endpoint enqueue jobs. Creating an empty collection does not enqueue a no-op job.
• R16. A job records kind, state, progress, total and any error, and is queryable over the API while it runs.
• R17. Jobs interrupted by a process restart are recovered at startup: a well-formed job left running is re-enqueued for idempotent replay; an unreadable request is failed with a reason. No job remains claiming to be in progress.
• R18. Membership jobs accept fail_on_file_error, defaulting to false. Best-effort jobs apply readable files and finish as completed_with_errors. Strict jobs discover and validate every source first; any file error fails the job without applying any membership change from that request.

Search

• R19. GET /search requires a non-empty collections parameter holding one or more collection ids, not names — names are mutable and would break saved queries on rename. A request without the parameter is rejected with scope_required; there is no implicit "all". An id that does not exist is rejected with collection_not_found rather than silently ignored.
• R20. Several ids are combined into a single Recoll query, so results come back as one coherently ranked set rather than merged per collection.
• R21. Every hit carries document_id. Callers select any other returned fields from title, author, path, MIME type, size, date, relevance score, collections, snippets, extracted text and document links. Omitting the selection returns metadata, collections and snippets, but not text or links.
• R22. Inline extracted text is bounded by text_limit, defaulting to 20,000 characters per hit and capped by configuration; the response says when it was truncated. Authenticated endpoints expose the complete extracted text and stream the canonical original separately.
• R23. Results are paginated by limit and offset and report the match count Recoll reports. Xapian estimates this figure for large result sets, so it is documented as approximate rather than presented as exact.

Authentication

• R24. Every endpoint except /health and /auth/login requires an authenticated session.
• R25. Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an HttpOnly, SameSite=Lax cookie. Unsafe cookie-authenticated requests require a CSRF token.
• R26. An initial administrator is seeded from the environment at first boot. If no credentials are supplied and no user exists, the service refuses to start rather than starting unauthenticated.

Operations

• R27. GET /health reports whether Recoll is reachable and the index readable, without requiring authentication.
• R28. GET /files lists one directory of the document volume read-only so a client can choose files or directories to add. API and database paths are root-relative and never expose host paths.
• R29. The service can be mounted at a configured subpath. Routes, OpenAPI, redirects, cookie paths and generated links honor it. Forwarded scheme, host and prefix headers are trusted only from explicitly configured proxies.
• R30. Errors use one envelope, {"error": {"code", "message", "details"}}, with the codes in the table below.
• R31. Docker is packaging, not an application-layer dependency. Document, index, database and membership-map paths plus listen host, port and mount prefix are configuration. Application modules never invoke Docker, and shutdown leaves durable jobs recoverable on the next start.

Code	Status
invalid_query, scope_required	400
invalid_credentials, unauthenticated	401
csrf_failed	403
collection_not_found, directory_rule_not_found, document_not_found, job_not_found, path_not_found	404
collection_name_taken, confirmation_required, job_not_retryable	409
document_missing	410
invalid_field, invalid_source, path_outside_root, validation_failed	422
engine_unavailable	503

Component boundaries

Module	Responsibility
api/	HTTP routes, request validation, authentication dependencies and serialization
domain/	Collection, document, membership, directory-rule and job behavior; no framework imports
storage/	SQLite migrations, transactions and repositories
indexing/	Discovery, hashing, reconciliation, membership-map export and the durable worker
engine/	Search/index Protocol, fake engine and real Recoll adapter
app.py	Configuration, dependency wiring, startup recovery and shutdown

Only recoll_modern/engine/ may import recoll or run recollindex. Every other module depends on its narrow Protocol. A membership job resolves safe paths and hashes content before a transaction changes desired state. It then atomically replaces the flat membership map and asks the engine to force-reindex affected canonical paths. An engine failure leaves desired state intact; retrying export and indexing is idempotent.

Docker is the delivery target for 001, but not an application-layer dependency: state paths and listen addresses are configurable, lifecycle is explicit, and the future web UI communicates only through the HTTP API. These seams allow spec 005 to package the unchanged UI with the API and Recoll as local sidecars; 001 does not select or build a desktop framework.

SQLite schema

SQLite foreign keys are enabled. Timestamps are UTC. Collection and username comparison keys are Unicode-normalized and case-folded by the application before their unique constraints are checked.

Table	Material columns and constraints
collections	id PK, name, name_key UNIQUE, description, timestamps
documents	content_hash PK, byte_size, state (present/missing), timestamps
document_paths	id PK, document_hash FK, relative_path UNIQUE, is_canonical, last_seen_at, missing_at; partial unique index permits one canonical path per document
direct_memberships	(collection_id, document_hash) PK, timestamp; explicit files and snapshot results
directory_rules	id PK, collection_id FK, relative_path, timestamps; collection and path are unique together
directory_rule_exclusions	(rule_id, relative_path) PK; exact recursive exclusions
rule_memberships	(rule_id, document_hash) PK, last_seen_at; latest materialized reconciliation
membership_exclusions	(collection_id, document_hash) PK, timestamp; per-file overrides against rules
jobs	id PK, kind, state, request_json, fail_on_file_error, progress, total, error fields and timestamps
job_file_errors	(job_id, relative_path) PK, stable error code and safe message
users	id PK, username, username_key UNIQUE, password_hash, created_at, disabled_at
sessions	token_hash PK, csrf_hash, user_id FK, expiry and activity timestamps; raw tokens are not stored
schema_migrations	version PK, applied_at; forward-only migrations

Effective membership is the union of direct membership and non-excluded rule membership. Direct membership wins over both per-file and directory-rule exclusions. Job requests retain JSON snapshots rather than foreign-keying every target, so deleting a collection cannot invalidate queued cleanup.

HTTP API

All paths below are relative to the configured mount prefix. All except /health and /auth/login require a session. Collection parameters on search repeat rather than using mutable names.

Method and path	Behavior
POST /auth/login, POST /auth/logout, GET /auth/session	Session lifecycle
GET /health	Public readiness; 503 when Recoll or the index is unavailable
GET /files?path=...	Browse one directory beneath the document root
GET /collections, GET /collections/{id}	List or read collections
POST /collections	Create, optionally with initial membership sources
PATCH /collections/{id}	Rename or edit description without indexing
DELETE /collections/{id}?confirmation=...	Delete metadata and enqueue affected-path cleanup
POST /collections/{id}/memberships	Add file, snapshot-directory or persistent-directory sources
GET /collections/{id}/documents	Page through effective members and their provenance
DELETE /collections/{id}/documents/{document_hash}	Remove direct membership and suppress matching rules
GET /collections/{id}/directory-rules	List persistent directory sources
PATCH /collections/{id}/directory-rules/{rule_id}	Replace exact descendant-directory exclusions and reconcile
DELETE /collections/{id}/directory-rules/{rule_id}	Remove one source and reconcile
GET /search	Scoped, paginated search with selectable result fields
GET /documents/{document_hash}/text	Return complete extracted text
GET /documents/{document_hash}/file	Stream the canonical original with range support
POST /jobs/reindex	Reconcile and reindex optional paths/collections or the whole root
GET /jobs, GET /jobs/{id}, POST /jobs/{id}/retry	Inspect and retry durable jobs

Membership sources have an explicit type. Directory sources also carry mode (snapshot or persistent) and optional exact descendant excluded_directories. Mutations default fail_on_file_error to false and return 202 with a job. Empty collection creation is synchronous and returns 201; creation with initial sources returns 202 with the collection and job.

{
  "sources": [
    {"type": "file", "path": "books/a.epub"},
    {
      "type": "directory",
      "path": "books",
      "mode": "persistent",
      "excluded_directories": ["books/.calibre"]
    }
  ],
  "fail_on_file_error": false
}

GET /search requires non-empty q and repeated collections. It accepts repeated fields selected from title, author, path, mime_type, size, date, score, collections, snippets, text and links; document_id is unconditional. Omitting fields returns every metadata field plus collections and snippets. text_limit defaults to 20,000 characters per hit, is capped by configuration and accompanies text_truncated. Links point at the authenticated text and file routes and include the configured mount prefix.

Search defaults to limit=20 and caps it at 100. Other list endpoints default to 50 and cap at 200; all use non-negative offset. Invalid bounds use validation_failed. Job file errors use the stable codes file_vanished, file_unreadable, file_changed_during_scan, file_outside_root and file_hash_failed; messages never reveal host paths.

Login returns an opaque session cookie and a CSRF token. Unsafe cookie-authenticated requests require X-CSRF-Token; login uses strict origin validation. Production cookies are Secure, HttpOnly, SameSite=Lax and scoped to the mount prefix. Authentication failures do not reveal whether a username exists. Unexpected errors return a request ID rather than exception text or filesystem details.

Testing

• Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. They cover Unicode name uniqueness, canonical promotion, overlapping rules, exclusion precedence, and strict versus best-effort jobs.
• Filesystem tests generate trees with duplicate content, same-name/different-content files, vanished and unreadable entries, directory symlinks, and file symlinks resolving both inside and outside the document root.
• Queue tests interrupt each durable transition and verify startup recovery and idempotent replay.
• API contract tests cover every error envelope, CSRF rule, selectable search field and pagination boundary under both / and a non-root mount prefix.
• One contract suite runs twice: against the fake, and against real Recoll. The real run skips when the container is unreachable; it never fails, and it never silently passes.
• An end-to-end test indexes a generated corpus on a :ro mount, exercises snapshot and persistent sources with exclusions, searches with different field selections, and retrieves full text plus a byte range from an original file.
• Security tests cover path traversal, untrusted forwarded headers, cookie scope, CSRF, response-header injection and safe file disposition.
• Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a decision first, because AGENTS.md forbids committing indexed documents.

Acceptance criteria

• A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4).
• Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4).
• Two files in the same directory can belong to different collections, and each is found only by its own collection's scope (C2, R6).
• Two byte-identical files under different names are one document and appear once in a scoped search (C3, R5, R7).
• Two files sharing a name with different content are two documents and both are findable (C3).
• Removing a canonical path promotes another path and the document remains findable (R8).
• Snapshot directories add their current documents once; persistent directories reconcile at startup and on reindex, honoring scoped exact-directory exclusions (R10–R12).
• A direct file addition remains a member despite a rule exclusion, and removing a rule never removes membership contributed by another source (R11).
• A strict job applies no membership changes after any discovery failure; the same best-effort job applies valid files and reports completed_with_errors (R18).
• A search without collections is rejected with scope_required (C4, R19).
• A search across two collections returns one ranked set, with each hit reporting its collections (R20, R21).
• Search field selection omits unrequested data; inline text is bounded and full authenticated text and ranged original-file retrieval work (R21–R22).
• A membership change is visible to search after its job completes, without the document being written to (C1, R13, R14).
• Indexing a document volume mounted :ro succeeds (C1).
• A job interrupted by a restart is not left claiming to be in progress (R17).
• An unauthenticated request to any endpoint except /health and /auth/login is refused (R24).
• The service refuses to start with no user and no seeded credentials (R26).
• Hosting under /recoll produces working routes, cookies, OpenAPI and document links, while untrusted forwarded-prefix headers are ignored (R29).
• The service uses non-default state paths and listen settings without code changes, contains no runtime Docker invocation, and recovers durable work after a cleanly handled shutdown (R31).
• uv run pytest, uv run ruff check . and uv run mypy src pass, and the suite skips rather than fails with the container down.

Known gaps

Accepted deliberately, recorded so they are not mistaken for oversights.

• Indexing overhead is measured only on small text files. The metadatacmds helper cost is expected to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is tracked in docs/TODO.md.
• No per-user access control. Every authenticated user sees every collection until 004.