docs: complete search engine API design

docs/specs/001-search-engine-api.md

@@ -19,12 +19,13 @@ that must be got right first.
 ## Scope
 
 **In:** Recoll in Docker · one index over all documents · collections created and managed at runtime ·
-per-file collection membership · content-hash document identity · an indexing job queue · a scoped
-search endpoint · authentication · a read-only file browse endpoint · health.
+per-file collection membership · snapshot and persistent directory sources · content-hash document
+identity · an indexing job queue · a scoped search endpoint with caller-selected fields · authenticated
+text and original-file retrieval · authentication · a read-only file browse endpoint · configurable
+subpath hosting · health.
 
 **Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles
-and per-collection permissions (004) · fetching a document's extracted text or downloading the original
-file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth.
+and per-collection permissions (004) · real-time index monitoring · OIDC or forward-auth.
 
 ## Constraints
 
@@ -47,8 +48,8 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th
 
 - **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and
   an optional description. The id is generated by the service and is never supplied by a client.
-- **R2.** Names are unique case-insensitively and may contain any characters, because a name never
-  reaches the index.
+- **R2.** Names contain at least one non-whitespace character, are unique after Unicode normalization
+  and case-folding, and otherwise accept arbitrary Unicode because a name never reaches the index.
 - **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the
   database and enqueues no indexing work.
 - **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion
@@ -60,7 +61,7 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th
 ### Documents and membership
 
 - **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one
-  document; each document has exactly one canonical path.
+  document; each present document has exactly one canonical path and a missing document has none.
 - **R6.** Membership attaches to a document, not a path. A document may belong to any number of
   collections, including none.
 - **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection.
@@ -70,73 +71,205 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th
   forces a reindex of it. A document with no remaining path is marked missing, not deleted.
 - **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work
   for the affected canonical paths only.
+- **R10.** Membership sources may be individual files or directories. A directory source is either a
+  one-time recursive snapshot or a persistent recursive rule. Persistent rules reconcile at startup and
+  during explicit reindex jobs; they do not watch the filesystem in real time.
+- **R11.** A directory source may name exact descendant directories to exclude recursively. Exclusions
+  on persistent sources are scoped to that rule. Removing an individual document that is still selected
+  by a rule creates a per-file override; direct membership remains additive and takes precedence.
+- **R12.** Recursive discovery does not follow directory symlinks. A file symlink is accepted only when
+  its resolved target stays beneath the configured document root.
 
 ### Indexing
 
-- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
+- **R13.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
   The helper is a fast-starting non-Python program reading a flat map exported from the database.
-- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
+- **R14.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
   Neither a plain incremental run nor `recollindex -i` detects such a change.
-- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection,
-  changing membership, and the reindex endpoint all enqueue jobs.
-- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while
+- **R15.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection with
+  initial sources, changing membership, changing a persistent rule, and the reindex endpoint enqueue
+  jobs. Creating an empty collection does not enqueue a no-op job.
+- **R16.** A job records kind, state, progress, total and any error, and is queryable over the API while
   it runs.
-- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either
-  re-enqueued or failed with a reason, never left claiming to be in progress.
+- **R17.** Jobs interrupted by a process restart are recovered at startup: a well-formed job left
+  running is re-enqueued for idempotent replay; an unreadable request is failed with a reason. No job
+  remains claiming to be in progress.
+- **R18.** Membership jobs accept `fail_on_file_error`, defaulting to `false`. Best-effort jobs apply
+  readable files and finish as `completed_with_errors`. Strict jobs discover and validate every source
+  first; any file error fails the job without applying any membership change from that request.
 
 ### Search
 
-- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
+- **R19.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
   **ids**, not names — names are mutable and would break saved queries on rename. A request without the
   parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is
   rejected with `collection_not_found` rather than silently ignored.
-- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently
+- **R20.** Several ids are combined into a single Recoll query, so results come back as one coherently
   ranked set rather than merged per collection.
-- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it
-  belongs to, and Recoll's keyword-in-context snippets.
-- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
+- **R21.** Every hit carries `document_id`. Callers select any other returned fields from title, author,
+  path, MIME type, size, date, relevance score, collections, snippets, extracted text and document
+  links. Omitting the selection returns metadata, collections and snippets, but not text or links.
+- **R22.** Inline extracted text is bounded by `text_limit`, defaulting to 20,000 characters per hit and
+  capped by configuration; the response says when it was truncated. Authenticated endpoints expose the
+  complete extracted text and stream the canonical original separately.
+- **R23.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
   Xapian estimates this figure for large result sets, so it is documented as approximate rather than
   presented as exact.
 
 ### Authentication
 
-- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
-- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
-  HttpOnly cookie.
-- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are
+- **R24.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
+- **R25.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
+  HttpOnly, `SameSite=Lax` cookie. Unsafe cookie-authenticated requests require a CSRF token.
+- **R26.** An initial administrator is seeded from the environment at first boot. If no credentials are
   supplied and no user exists, the service refuses to start rather than starting unauthenticated.
 
 ### Operations
 
-- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
+- **R27.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
   authentication.
-- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add.
-- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
+- **R28.** `GET /files` lists one directory of the document volume read-only so a client can choose
+  files or directories to add. API and database paths are root-relative and never expose host paths.
+- **R29.** The service can be mounted at a configured subpath. Routes, OpenAPI, redirects, cookie paths
+  and generated links honor it. Forwarded scheme, host and prefix headers are trusted only from
+  explicitly configured proxies.
+- **R30.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
   table below.
 
 | Code | Status |
 |---|---|
-| `scope_required` | 400 |
-| `unauthenticated` | 401 |
-| `collection_not_found` | 404 |
-| `collection_name_taken` | 409 |
-| `confirmation_required` | 409 |
-| `validation_failed` | 422 |
+| `invalid_query`, `scope_required` | 400 |
+| `invalid_credentials`, `unauthenticated` | 401 |
+| `csrf_failed` | 403 |
+| `collection_not_found`, `directory_rule_not_found`, `document_not_found`, `job_not_found`, `path_not_found` | 404 |
+| `collection_name_taken`, `confirmation_required`, `job_not_retryable` | 409 |
+| `document_missing` | 410 |
+| `invalid_field`, `invalid_source`, `path_outside_root`, `validation_failed` | 422 |
 | `engine_unavailable` | 503 |
 
-## Shape
+## Component boundaries
 
-Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The
-one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.**
-Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run
-without a container.
+| Module | Responsibility |
+|---|---|
+| `api/` | HTTP routes, request validation, authentication dependencies and serialization |
+| `domain/` | Collection, document, membership, directory-rule and job behavior; no framework imports |
+| `storage/` | SQLite migrations, transactions and repositories |
+| `indexing/` | Discovery, hashing, reconciliation, membership-map export and the durable worker |
+| `engine/` | Search/index Protocol, fake engine and real Recoll adapter |
+| `app.py` | Configuration, dependency wiring, startup recovery and shutdown |
+
+**Only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** Every other module depends on
+its narrow Protocol. A membership job resolves safe paths and hashes content before a transaction
+changes desired state. It then atomically replaces the flat membership map and asks the engine to
+force-reindex affected canonical paths. An engine failure leaves desired state intact; retrying export
+and indexing is idempotent.
+
+## SQLite schema
+
+SQLite foreign keys are enabled. Timestamps are UTC. Collection and username comparison keys are
+Unicode-normalized and case-folded by the application before their unique constraints are checked.
+
+| Table | Material columns and constraints |
+|---|---|
+| `collections` | `id` PK, `name`, `name_key` UNIQUE, `description`, timestamps |
+| `documents` | `content_hash` PK, `byte_size`, `state` (`present`/`missing`), timestamps |
+| `document_paths` | `id` PK, `document_hash` FK, `relative_path` UNIQUE, `is_canonical`, `last_seen_at`, `missing_at`; partial unique index permits one canonical path per document |
+| `direct_memberships` | `(collection_id, document_hash)` PK, timestamp; explicit files and snapshot results |
+| `directory_rules` | `id` PK, `collection_id` FK, `relative_path`, timestamps; collection and path are unique together |
+| `directory_rule_exclusions` | `(rule_id, relative_path)` PK; exact recursive exclusions |
+| `rule_memberships` | `(rule_id, document_hash)` PK, `last_seen_at`; latest materialized reconciliation |
+| `membership_exclusions` | `(collection_id, document_hash)` PK, timestamp; per-file overrides against rules |
+| `jobs` | `id` PK, `kind`, `state`, `request_json`, `fail_on_file_error`, progress, total, error fields and timestamps |
+| `job_file_errors` | `(job_id, relative_path)` PK, stable error code and safe message |
+| `users` | `id` PK, `username`, `username_key` UNIQUE, `password_hash`, `created_at`, `disabled_at` |
+| `sessions` | `token_hash` PK, `csrf_hash`, `user_id` FK, expiry and activity timestamps; raw tokens are not stored |
+| `schema_migrations` | `version` PK, `applied_at`; forward-only migrations |
+
+Effective membership is the union of direct membership and non-excluded rule membership. Direct
+membership wins over both per-file and directory-rule exclusions. Job requests retain JSON snapshots
+rather than foreign-keying every target, so deleting a collection cannot invalidate queued cleanup.
+
+## HTTP API
+
+All paths below are relative to the configured mount prefix. All except `/health` and `/auth/login`
+require a session. Collection parameters on search repeat rather than using mutable names.
+
+| Method and path | Behavior |
+|---|---|
+| `POST /auth/login`, `POST /auth/logout`, `GET /auth/session` | Session lifecycle |
+| `GET /health` | Public readiness; `503` when Recoll or the index is unavailable |
+| `GET /files?path=...` | Browse one directory beneath the document root |
+| `GET /collections`, `GET /collections/{id}` | List or read collections |
+| `POST /collections` | Create, optionally with initial membership sources |
+| `PATCH /collections/{id}` | Rename or edit description without indexing |
+| `DELETE /collections/{id}?confirmation=...` | Delete metadata and enqueue affected-path cleanup |
+| `POST /collections/{id}/memberships` | Add file, snapshot-directory or persistent-directory sources |
+| `GET /collections/{id}/documents` | Page through effective members and their provenance |
+| `DELETE /collections/{id}/documents/{document_hash}` | Remove direct membership and suppress matching rules |
+| `GET /collections/{id}/directory-rules` | List persistent directory sources |
+| `PATCH /collections/{id}/directory-rules/{rule_id}` | Replace exact descendant-directory exclusions and reconcile |
+| `DELETE /collections/{id}/directory-rules/{rule_id}` | Remove one source and reconcile |
+| `GET /search` | Scoped, paginated search with selectable result fields |
+| `GET /documents/{document_hash}/text` | Return complete extracted text |
+| `GET /documents/{document_hash}/file` | Stream the canonical original with range support |
+| `POST /jobs/reindex` | Reconcile and reindex optional paths/collections or the whole root |
+| `GET /jobs`, `GET /jobs/{id}`, `POST /jobs/{id}/retry` | Inspect and retry durable jobs |
+
+Membership sources have an explicit `type`. Directory sources also carry `mode` (`snapshot` or
+`persistent`) and optional exact descendant `excluded_directories`. Mutations default
+`fail_on_file_error` to `false` and return `202` with a job. Empty collection creation is synchronous
+and returns `201`; creation with initial sources returns `202` with the collection and job.
+
+```json
+{
+  "sources": [
+    {"type": "file", "path": "books/a.epub"},
+    {
+      "type": "directory",
+      "path": "books",
+      "mode": "persistent",
+      "excluded_directories": ["books/.calibre"]
+    }
+  ],
+  "fail_on_file_error": false
+}
+```
+
+`GET /search` requires non-empty `q` and repeated `collections`. It accepts repeated `fields` selected
+from `title`, `author`, `path`, `mime_type`, `size`, `date`, `score`, `collections`, `snippets`, `text`
+and `links`; `document_id` is unconditional. Omitting `fields` returns every metadata field plus
+collections and snippets. `text_limit` defaults to 20,000 characters per hit, is capped by
+configuration and accompanies `text_truncated`. Links point at the authenticated text and file routes
+and include the configured mount prefix.
+
+Search defaults to `limit=20` and caps it at 100. Other list endpoints default to 50 and cap at 200;
+all use non-negative `offset`. Invalid bounds use `validation_failed`. Job file errors use the stable
+codes `file_vanished`, `file_unreadable`, `file_changed_during_scan`, `file_outside_root` and
+`file_hash_failed`; messages never reveal host paths.
+
+Login returns an opaque session cookie and a CSRF token. Unsafe cookie-authenticated requests require
+`X-CSRF-Token`; login uses strict origin validation. Production cookies are `Secure`, `HttpOnly`,
+`SameSite=Lax` and scoped to the mount prefix. Authentication failures do not reveal whether a username
+exists. Unexpected errors return a request ID rather than exception text or filesystem details.
 
 ## Testing
 
-- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container.
+- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. They
+  cover Unicode name uniqueness, canonical promotion, overlapping rules, exclusion precedence, and
+  strict versus best-effort jobs.
+- Filesystem tests generate trees with duplicate content, same-name/different-content files, vanished
+  and unreadable entries, directory symlinks, and file symlinks resolving both inside and outside the
+  document root.
+- Queue tests interrupt each durable transition and verify startup recovery and idempotent replay.
+- API contract tests cover every error envelope, CSRF rule, selectable search field and pagination
+  boundary under both `/` and a non-root mount prefix.
 - One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when
   the container is unreachable; it never fails, and it never silently passes.
-- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it.
+- An end-to-end test indexes a generated corpus on a `:ro` mount, exercises snapshot and persistent
+  sources with exclusions, searches with different field selections, and retrieves full text plus a
+  byte range from an original file.
+- Security tests cover path traversal, untrusted forwarded headers, cookie scope, CSRF, response-header
+  injection and safe file disposition.
 - Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a
   decision first, because `AGENTS.md` forbids committing indexed documents.
 
@@ -150,15 +283,25 @@ without a container.
       search (C3, R5, R7).
 - [ ] Two files sharing a name with different content are two documents and both are findable (C3).
 - [ ] Removing a canonical path promotes another path and the document remains findable (R8).
-- [ ] A search without `collections` is rejected with `scope_required` (C4, R15).
+- [ ] Snapshot directories add their current documents once; persistent directories reconcile at
+      startup and on reindex, honoring scoped exact-directory exclusions (R10–R12).
+- [ ] A direct file addition remains a member despite a rule exclusion, and removing a rule never
+      removes membership contributed by another source (R11).
+- [ ] A strict job applies no membership changes after any discovery failure; the same best-effort job
+      applies valid files and reports `completed_with_errors` (R18).
+- [ ] A search without `collections` is rejected with `scope_required` (C4, R19).
 - [ ] A search across two collections returns one ranked set, with each hit reporting its collections
-      (R16, R17).
+      (R20, R21).
+- [ ] Search field selection omits unrequested data; inline text is bounded and full authenticated text
+      and ranged original-file retrieval work (R21–R22).
 - [ ] A membership change is visible to search after its job completes, without the document being
-      written to (C1, R10, R11).
+      written to (C1, R13, R14).
 - [ ] Indexing a document volume mounted `:ro` succeeds (C1).
-- [ ] A job interrupted by a restart is not left claiming to be in progress (R14).
-- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19).
-- [ ] The service refuses to start with no user and no seeded credentials (R21).
+- [ ] A job interrupted by a restart is not left claiming to be in progress (R17).
+- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R24).
+- [ ] The service refuses to start with no user and no seeded credentials (R26).
+- [ ] Hosting under `/recoll` produces working routes, cookies, OpenAPI and document links, while
+      untrusted forwarded-prefix headers are ignored (R29).
 - [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than
       fails with the container down.
 
@@ -166,9 +309,6 @@ without a container.
 
 Accepted deliberately, recorded so they are not mistaken for oversights.
 
-- **No way to read a document.** Search reports that a file matched but offers no extracted text and no
-  download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is
-  002's first requirement.
 - **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected
   to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is
   tracked in `docs/TODO.md`.