recoll-modern/docs/specs/001-search-engine-api.md

---
status: implemented
plan: docs/plans/001-search-engine-api.md
supersedes: none
---

# 001 — Search engine and HTTP API

The first of five specs (see `docs/ARCHITECTURE.md`, "Delivery order"). It delivers the Recoll layer and
the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI.

## Why

Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold
several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run
as a container on a server. The API is the contract two later consumers depend on, so it is the piece
that must be got right first.

## Scope

**In:** Recoll in Docker · one index over all documents · collections created and managed at runtime ·
per-file collection membership · snapshot and persistent directory sources · content-hash document
identity · an indexing job queue · a scoped search endpoint with caller-selected fields · authenticated
text and original-file retrieval · authentication · a read-only file browse endpoint · configurable
subpath hosting · health.

**Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles
and per-collection permissions (004) · a cross-platform installable desktop distribution (005, after
004) · real-time index monitoring · OIDC or forward-auth.

## Constraints

These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0
and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; the reasoning is in
`docs/DECISIONS.md`.

- **C1. The documents are read-only.** The service must operate with read-only access to everything it
  indexes and must never write to a document, its directory, or its extended attributes. The deployment
  mounts the document volume `:ro`.
- **C2. Membership is per file, not per directory.** Two files in the same directory may belong to
  different collections.
- **C3. Documents identical in content are one document.** Documents sharing a name but differing in
  content are separate documents.
- **C4. A search never silently spans collections.**

## Requirements

### Collections

- **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and
  an optional description. The id is generated by the service and is never supplied by a client.
- **R2.** Names contain at least one non-whitespace character, are unique after Unicode normalization
  and case-folding, and otherwise accept arbitrary Unicode because a name never reaches the index.
- **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the
  database and enqueues no indexing work.
- **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion
  removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so
  the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds
  the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless,
  because the API resolves ids against the database and drops any it no longer knows.

### Documents and membership

- **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one
  document; each present document has exactly one canonical path and a missing document has none.
- **R6.** Membership attaches to a document, not a path. A document may belong to any number of
  collections, including none.
- **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection.
  Non-canonical duplicate paths remain in the index but belong to no collection and are therefore
  unreachable by any scoped search.
- **R8.** If a canonical path disappears, the service promotes another known path for that document and
  forces a reindex of it. A document with no remaining path is marked missing, not deleted.
- **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work
  for the affected canonical paths only.
- **R10.** Membership sources may be individual files or directories. A directory source is either a
  one-time recursive snapshot or a persistent recursive rule. Persistent rules reconcile at startup and
  during explicit reindex jobs; they do not watch the filesystem in real time.
- **R11.** A directory source may name exact descendant directories to exclude recursively. Exclusions
  on persistent sources are scoped to that rule. Removing an individual document that is still selected
  by a rule creates a per-file override; direct membership remains additive and takes precedence.
- **R12.** Recursive discovery does not follow directory symlinks. A file symlink is accepted only when
  its resolved target stays beneath the configured document root.

### Indexing

- **R13.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
  The helper is a fast-starting non-Python program reading a flat map exported from the database.
- **R14.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
  Neither a plain incremental run nor `recollindex -i` detects such a change.
- **R15.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection with
  initial sources, changing membership, changing a persistent rule, and the reindex endpoint enqueue
  jobs. Creating an empty collection does not enqueue a no-op job.
- **R16.** A job records kind, state, progress, total and any error, and is queryable over the API while
  it runs.
- **R17.** Jobs interrupted by a process restart are recovered at startup: a well-formed job left
  running is re-enqueued for idempotent replay; an unreadable request is failed with a reason. No job
  remains claiming to be in progress.
- **R18.** Membership jobs accept `fail_on_file_error`, defaulting to `false`. Best-effort jobs apply
  readable files and finish as `completed_with_errors`. Strict jobs discover and validate every source
  first; any file error fails the job without applying any membership change from that request.

### Search

- **R19.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
  **ids**, not names — names are mutable and would break saved queries on rename. A request without the
  parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is
  rejected with `collection_not_found` rather than silently ignored.
- **R20.** Several ids are combined into a single Recoll query, so results come back as one coherently
  ranked set rather than merged per collection.
- **R21.** Every hit carries `document_id`. Callers select any other returned fields from title, author,
  path, MIME type, size, date, relevance score, collections, snippets, extracted text and document
  links. Omitting the selection returns metadata, collections and snippets, but not text or links.
- **R22.** Inline extracted text is bounded by `text_limit`, defaulting to 20,000 characters per hit and
  capped by configuration; the response says when it was truncated. Authenticated endpoints expose the
  complete extracted text and stream the canonical original separately.
- **R23.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
  Xapian estimates this figure for large result sets, so it is documented as approximate rather than
  presented as exact.

### Authentication

- **R24.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
- **R25.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
  HttpOnly, `SameSite=Lax` cookie. Unsafe cookie-authenticated requests require a CSRF token.
- **R26.** An initial administrator is seeded from the environment at first boot. If no credentials are
  supplied and no user exists, the service refuses to start rather than starting unauthenticated.

### Operations

- **R27.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
  authentication.
- **R28.** `GET /files` lists one directory of the document volume read-only so a client can choose
  files or directories to add. API and database paths are root-relative and never expose host paths.
- **R29.** The service can be mounted at a configured subpath. Routes, OpenAPI, redirects, cookie paths
  and generated links honor it. Forwarded scheme, host and prefix headers are trusted only from
  explicitly configured proxies.
- **R30.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
  table below.
- **R31.** Docker is packaging, not an application-layer dependency. Document, index, database and
  membership-map paths plus listen host, port and mount prefix are configuration. Application modules
  never invoke Docker, and shutdown leaves durable jobs recoverable on the next start.

| Code | Status |
|---|---|
| `invalid_query`, `scope_required` | 400 |
| `invalid_credentials`, `unauthenticated` | 401 |
| `csrf_failed` | 403 |
| `collection_not_found`, `directory_rule_not_found`, `document_not_found`, `job_not_found`, `path_not_found` | 404 |
| `collection_name_taken`, `confirmation_required`, `job_not_retryable` | 409 |
| `document_missing` | 410 |
| `invalid_field`, `invalid_source`, `path_outside_root`, `validation_failed` | 422 |
| `engine_unavailable` | 503 |

## Component boundaries

| Module | Responsibility |
|---|---|
| `api/` | HTTP routes, request validation, authentication dependencies and serialization |
| `domain/` | Collection, document, membership, directory-rule and job behavior; no framework imports |
| `storage/` | SQLite migrations, transactions and repositories |
| `indexing/` | Discovery, hashing, reconciliation, membership-map export and the durable worker |
| `engine/` | Search/index Protocol, fake engine and real Recoll adapter |
| `app.py` | Configuration, dependency wiring, startup recovery and shutdown |

**Only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** Every other module depends on
its narrow Protocol. A membership job resolves safe paths and hashes content before a transaction
changes desired state. It then atomically replaces the flat membership map and asks the engine to
force-reindex affected canonical paths. An engine failure leaves desired state intact; retrying export
and indexing is idempotent.

Docker is the delivery target for 001, but not an application-layer dependency: state paths and listen
addresses are configurable, lifecycle is explicit, and the future web UI communicates only through the
HTTP API. These seams allow spec 005 to package the unchanged UI with the API and Recoll as local
sidecars; 001 does not select or build a desktop framework.

## SQLite schema

SQLite foreign keys are enabled. Timestamps are UTC. Collection and username comparison keys are
Unicode-normalized and case-folded by the application before their unique constraints are checked.

| Table | Material columns and constraints |
|---|---|
| `collections` | `id` PK, `name`, `name_key` UNIQUE, `description`, timestamps |
| `documents` | `content_hash` PK, `byte_size`, `state` (`present`/`missing`), timestamps |
| `document_paths` | `id` PK, `document_hash` FK, `relative_path` UNIQUE, `is_canonical`, `last_seen_at`, `missing_at`; partial unique index permits one canonical path per document |
| `direct_memberships` | `(collection_id, document_hash)` PK, timestamp; explicit files and snapshot results |
| `directory_rules` | `id` PK, `collection_id` FK, `relative_path`, timestamps; collection and path are unique together |
| `directory_rule_exclusions` | `(rule_id, relative_path)` PK; exact recursive exclusions |
| `rule_memberships` | `(rule_id, document_hash)` PK, `last_seen_at`; latest materialized reconciliation |
| `membership_exclusions` | `(collection_id, document_hash)` PK, timestamp; per-file overrides against rules |
| `jobs` | `id` PK, `kind`, `state`, `request_json`, `fail_on_file_error`, progress, total, error fields and timestamps |
| `job_file_errors` | `(job_id, relative_path)` PK, stable error code and safe message |
| `users` | `id` PK, `username`, `username_key` UNIQUE, `password_hash`, `created_at`, `disabled_at` |
| `sessions` | `token_hash` PK, `csrf_hash`, `user_id` FK, expiry and activity timestamps; raw tokens are not stored |
| `schema_migrations` | `version` PK, `applied_at`; forward-only migrations |

Effective membership is the union of direct membership and non-excluded rule membership. Direct
membership wins over both per-file and directory-rule exclusions. Job requests retain JSON snapshots
rather than foreign-keying every target, so deleting a collection cannot invalidate queued cleanup.

## HTTP API

All paths below are relative to the configured mount prefix. All except `/health` and `/auth/login`
require a session. Collection parameters on search repeat rather than using mutable names.

| Method and path | Behavior |
|---|---|
| `POST /auth/login`, `POST /auth/logout`, `GET /auth/session` | Session lifecycle |
| `GET /health` | Public readiness; `503` when Recoll or the index is unavailable |
| `GET /files?path=...` | Browse one directory beneath the document root |
| `GET /collections`, `GET /collections/{id}` | List or read collections |
| `POST /collections` | Create, optionally with initial membership sources |
| `PATCH /collections/{id}` | Rename or edit description without indexing |
| `DELETE /collections/{id}?confirmation=...` | Delete metadata and enqueue affected-path cleanup |
| `POST /collections/{id}/memberships` | Add file, snapshot-directory or persistent-directory sources |
| `GET /collections/{id}/documents` | Page through effective members and their provenance |
| `DELETE /collections/{id}/documents/{document_hash}` | Remove direct membership and suppress matching rules |
| `GET /collections/{id}/directory-rules` | List persistent directory sources |
| `PATCH /collections/{id}/directory-rules/{rule_id}` | Replace exact descendant-directory exclusions and reconcile |
| `DELETE /collections/{id}/directory-rules/{rule_id}` | Remove one source and reconcile |
| `GET /search` | Scoped, paginated search with selectable result fields |
| `GET /documents/{document_hash}/text` | Return complete extracted text |
| `GET /documents/{document_hash}/file` | Stream the canonical original with range support |
| `POST /jobs/reindex` | Reconcile and reindex optional paths/collections or the whole root |
| `GET /jobs`, `GET /jobs/{id}`, `POST /jobs/{id}/retry` | Inspect and retry durable jobs |

Membership sources have an explicit `type`. Directory sources also carry `mode` (`snapshot` or
`persistent`) and optional exact descendant `excluded_directories`. Mutations default
`fail_on_file_error` to `false` and return `202` with a job. Empty collection creation is synchronous
and returns `201`; creation with initial sources returns `202` with the collection and job.

```json
{
  "sources": [
    {"type": "file", "path": "books/a.epub"},
    {
      "type": "directory",
      "path": "books",
      "mode": "persistent",
      "excluded_directories": ["books/.calibre"]
    }
  ],
  "fail_on_file_error": false
}
```

`GET /search` requires non-empty `q` and repeated `collections`. It accepts repeated `fields` selected
from `title`, `author`, `path`, `mime_type`, `size`, `date`, `score`, `collections`, `snippets`, `text`
and `links`; `document_id` is unconditional. Omitting `fields` returns every metadata field plus
collections and snippets. `text_limit` defaults to 20,000 characters per hit, is capped by
configuration and accompanies `text_truncated`. Links point at the authenticated text and file routes
and include the configured mount prefix.

Search defaults to `limit=20` and caps it at 100. Other list endpoints default to 50 and cap at 200;
all use non-negative `offset`. Invalid bounds use `validation_failed`. Job file errors use the stable
codes `file_vanished`, `file_unreadable`, `file_changed_during_scan`, `file_outside_root` and
`file_hash_failed`; messages never reveal host paths.

Login returns an opaque session cookie and a CSRF token. Unsafe cookie-authenticated requests require
`X-CSRF-Token`; login uses strict origin validation. Production cookies are `Secure`, `HttpOnly`,
`SameSite=Lax` and scoped to the mount prefix. Authentication failures do not reveal whether a username
exists. Unexpected errors return a request ID rather than exception text or filesystem details.

## Testing

- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. They
  cover Unicode name uniqueness, canonical promotion, overlapping rules, exclusion precedence, and
  strict versus best-effort jobs.
- Filesystem tests generate trees with duplicate content, same-name/different-content files, vanished
  and unreadable entries, directory symlinks, and file symlinks resolving both inside and outside the
  document root.
- Queue tests interrupt each durable transition and verify startup recovery and idempotent replay.
- API contract tests cover every error envelope, CSRF rule, selectable search field and pagination
  boundary under both `/` and a non-root mount prefix.
- One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when
  the container is unreachable; it never fails, and it never silently passes.
- An end-to-end test indexes a generated corpus on a `:ro` mount, exercises snapshot and persistent
  sources with exclusions, searches with different field selections, and retrieves full text plus a
  byte range from an original file.
- Security tests cover path traversal, untrusted forwarded headers, cookie scope, CSRF, response-header
  injection and safe file disposition.
- Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a
  decision first, because `AGENTS.md` forbids committing indexed documents.

## Acceptance criteria

- [x] A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4).
- [x] Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4).
- [x] Two files in the same directory can belong to different collections, and each is found only by its
      own collection's scope (C2, R6).
- [x] Two byte-identical files under different names are one document and appear once in a scoped
      search (C3, R5, R7).
- [x] Two files sharing a name with different content are two documents and both are findable (C3).
- [x] Removing a canonical path promotes another path and the document remains findable (R8).
- [x] Snapshot directories add their current documents once; persistent directories reconcile at
      startup and on reindex, honoring scoped exact-directory exclusions (R10–R12).
- [x] A direct file addition remains a member despite a rule exclusion, and removing a rule never
      removes membership contributed by another source (R11).
- [x] A strict job applies no membership changes after any discovery failure; the same best-effort job
      applies valid files and reports `completed_with_errors` (R18).
- [x] A search without `collections` is rejected with `scope_required` (C4, R19).
- [x] A search across two collections returns one ranked set, with each hit reporting its collections
      (R20, R21).
- [x] Search field selection omits unrequested data; inline text is bounded and full authenticated text
      and ranged original-file retrieval work (R21–R22).
- [x] A membership change is visible to search after its job completes, without the document being
      written to (C1, R13, R14).
- [x] Indexing a document volume mounted `:ro` succeeds (C1).
- [x] A job interrupted by a restart is not left claiming to be in progress (R17).
- [x] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R24).
- [x] The service refuses to start with no user and no seeded credentials (R26).
- [x] Hosting under `/recoll` produces working routes, cookies, OpenAPI and document links, while
      untrusted forwarded-prefix headers are ignored (R29).
- [x] The service uses non-default state paths and listen settings without code changes, contains no
      runtime Docker invocation, and recovers durable work after a cleanly handled shutdown (R31).
- [x] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than
      fails with the container down.

## Known gaps

Accepted deliberately, recorded so they are not mistaken for oversights.

- **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected
  to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is
  tracked in `docs/TODO.md`.
- **No per-user access control.** Every authenticated user sees every collection until 004.