docs: complete search engine API design

PRODUCT.md

@@ -12,8 +12,9 @@ web
 
 ## Stack
 
-Undecided. Greenfield: no framework, no scaffold. The stack is a user decision and has not been made.
-See `AGENTS.md` for the toolchains actually installed on the host.
+The backend stack was confirmed during the 001 design: Python + FastAPI, managed with `uv`, because
+Recoll provides native Python bindings. The frontend stack remains undecided. No application has been
+scaffolded yet. See `docs/DECISIONS.md` for the decision and `AGENTS.md` for installed toolchains.
 
 ## Users
 

docs/ARCHITECTURE.md

@@ -43,6 +43,43 @@ The product is four specs, built in order. Each gets its own spec, plan and revi
 - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
 - A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
 
+### Component boundaries
+
+| Module | Responsibility |
+|---|---|
+| `api/` | HTTP transport, validation, auth dependencies and serialization |
+| `domain/` | Framework-independent collection, document, membership, rule and job behavior |
+| `storage/` | SQLite migrations, transactions and repositories |
+| `indexing/` | Discovery, hashing, directory-rule reconciliation, map export and queue worker |
+| `engine/` | The search/index Protocol, fake implementation and Recoll adapter |
+| `app.py` | Configuration, dependency wiring and lifecycle |
+
+Only `engine/` may import Recoll or invoke `recollindex`. Membership changes first establish desired
+state in SQLite, atomically export the helper's flat map, then force-index affected canonical paths.
+Engine retries are therefore idempotent. Persistent directory rules reconcile at startup and on an
+explicit reindex, not through live filesystem monitoring.
+
+### Storage model
+
+`collections`, `documents` and `document_paths` hold identity and canonical paths. Membership provenance
+is normalized: `direct_memberships` holds explicit files and snapshot expansions; `directory_rules`,
+`directory_rule_exclusions` and `rule_memberships` hold persistent additions; and
+`membership_exclusions` holds per-file overrides against rule-derived membership. Effective membership
+is direct membership union non-excluded rule membership, so direct additions remain additive.
+
+`jobs` and `job_file_errors` make work and partial failures durable. `users` and hashed-token `sessions`
+provide authentication. `schema_migrations` is forward-only. The exact columns and constraints are the
+contract in `docs/specs/001-search-engine-api.md`.
+
+### HTTP boundary
+
+The resource groups are auth, health, read-only file browsing, collections and membership sources,
+directory rules, scoped search, document text/original retrieval, and indexing jobs. Search callers
+select returned fields; only `document_id` is unconditional, and inline full text is bounded. Every
+route, cookie, redirect, OpenAPI URL and generated link supports a configured mount prefix. Forwarded
+prefix and origin data are used only from trusted proxies. The exact endpoints and error codes are the
+contract in `docs/specs/001-search-engine-api.md`.
+
 ## Verified Recoll mechanics
 
 Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running
@@ -83,14 +120,12 @@ Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the re
 3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite.
 4. ~~Stack~~ — Python + FastAPI, managed with `uv`.
 5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004.
-6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate
-   endpoints.
+6. ~~Result presentation~~ — callers select result fields; metadata plus snippets is the default, inline
+   text is bounded, and full text and file download are separate endpoints.
 
-Still open, to be closed by the remaining sections of the 001 design:
-
-- Component boundaries and the SQLite schema.
-- API surface, pagination, and error semantics.
-- Test strategy, including how tests skip when the container is down.
+The remaining 001 design questions — component boundaries, schema, HTTP surface, pagination, error
+semantics and test strategy — are closed in `docs/specs/001-search-engine-api.md`. That spec remains
+draft until maintainer sign-off.
 
 ## External dependencies
 

docs/DECISIONS.md

@@ -3,6 +3,53 @@
 Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back.
 One entry per decision a future maintainer would otherwise have to reverse-engineer.
 
+## 2026-09-18 — Deployment supports a configured URL subpath
+**Status:** accepted
+**Context:** the service may sit behind an existing reverse proxy under a path such as `/recoll`, not at
+the origin root. Hard-coded root URLs break redirects, cookies, OpenAPI and document links there.
+**Decision:** one configured external mount prefix drives routing, generated URLs and cookie scope.
+Forwarded scheme, host and prefix values are honored only from explicitly trusted proxies.
+**Alternatives:** root-only hosting (needlessly constrains deployment); trust every forwarded header
+(lets a direct client forge external URLs and cookie paths).
+**Consequences:** API and end-to-end tests run at both `/` and a non-root prefix. Proxy configuration is
+part of deployment documentation, and all URL generation goes through one boundary.
+
+---
+
+## 2026-09-18 — Search result fields are caller-selected
+**Status:** accepted
+**Supersedes:** the fixed result shape in "Search results carry metadata and snippets, not full text"
+(below, same date). The metadata-plus-snippets shape remains the default, not the only representation.
+**Context:** browser and MCP consumers have different response-size needs, and users need to choose
+whether results include metadata, snippets, extracted text or retrieval links.
+**Decision:** `document_id` is unconditional; callers select all other result fields. Inline extracted
+text is opt-in, bounded and marked when truncated. Authenticated endpoints return full extracted text
+and stream the canonical original separately.
+**Alternatives:** one fixed rich shape (wastes bandwidth and agent context); metadata-only search with a
+follow-up per hit (too many round trips); unbounded full text inline (a book can dominate the response).
+**Consequences:** field projection is part of the engine/API contract. Generated retrieval links honor
+the configured mount prefix, and original-file responses require range and safe-disposition handling.
+
+---
+
+## 2026-09-18 — Directory membership has normalized additive provenance
+**Status:** accepted
+**Context:** users need both one-time directory expansion and persistent directory membership, including
+the ability to omit metadata subdirectories such as `.calibre`. Overlapping sources must not make
+removal order-dependent.
+**Decision:** direct file additions and snapshot expansions are durable direct membership. Persistent
+directory rules materialize their own contributed documents and hold exact recursive exclusions scoped
+to the rule. Per-file removal creates an override against rule-derived membership; an explicit direct
+addition still wins. Persistent rules reconcile at startup and explicit reindex, not through watching.
+**Alternatives:** one polymorphic source ledger (weaker relational constraints); recompute every rule on
+demand (slow and nondeterministic); persistent rules without provenance (removing one rule can erase
+membership contributed by another).
+**Consequences:** the schema has separate direct, rule-derived and exclusion tables. Directory symlinks
+are never followed; file symlinks are accepted only inside the document root. Jobs default to
+best-effort, while strict jobs scan first and apply no membership changes if any file fails.
+
+---
+
 ## 2026-09-18 — Document identity is a content hash, not a path
 **Status:** accepted
 **Context:** the document set may contain files that share a name but differ in content, and files that

docs/TODO.md

@@ -11,9 +11,10 @@ Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -
 
 - Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with
   reasoning in `docs/DECISIONS.md`.
-- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** It has had its self-review
-  pass but **has not been signed off by the maintainer.** Do not start a plan or write code against it
-  until its front matter says `approved`.
+- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** Its missing schema, component,
+  endpoint and test designs were completed after maintainer-guided review, including directory rules,
+  selectable search fields and subpath hosting. It **has not received final maintainer sign-off.** Do
+  not start a plan or write code against it until its front matter says `approved`.
 - Plan `docs/plans/001-search-engine-api.md`: **not written.** It is the next artefact, and only after
   sign-off.
 
@@ -37,7 +38,7 @@ Things a resuming agent will otherwise get wrong:
 ## Now
 - [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in
       `docs/DECISIONS.md`.
-- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics,
+- [x] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics,
       test strategy. Then write `docs/specs/001-search-engine-api.md`.
 - [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in
       the same commit that scaffolds the app, with every command actually run.

docs/specs/001-search-engine-api.md

@@ -19,12 +19,13 @@ that must be got right first.
 ## Scope
 
 **In:** Recoll in Docker · one index over all documents · collections created and managed at runtime ·
-per-file collection membership · content-hash document identity · an indexing job queue · a scoped
-search endpoint · authentication · a read-only file browse endpoint · health.
+per-file collection membership · snapshot and persistent directory sources · content-hash document
+identity · an indexing job queue · a scoped search endpoint with caller-selected fields · authenticated
+text and original-file retrieval · authentication · a read-only file browse endpoint · configurable
+subpath hosting · health.
 
 **Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles
-and per-collection permissions (004) · fetching a document's extracted text or downloading the original
-file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth.
+and per-collection permissions (004) · real-time index monitoring · OIDC or forward-auth.
 
 ## Constraints
 
@@ -47,8 +48,8 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th
 
 - **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and
   an optional description. The id is generated by the service and is never supplied by a client.
-- **R2.** Names are unique case-insensitively and may contain any characters, because a name never
-  reaches the index.
+- **R2.** Names contain at least one non-whitespace character, are unique after Unicode normalization
+  and case-folding, and otherwise accept arbitrary Unicode because a name never reaches the index.
 - **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the
   database and enqueues no indexing work.
 - **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion
@@ -60,7 +61,7 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th
 ### Documents and membership
 
 - **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one
-  document; each document has exactly one canonical path.
+  document; each present document has exactly one canonical path and a missing document has none.
 - **R6.** Membership attaches to a document, not a path. A document may belong to any number of
   collections, including none.
 - **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection.
@@ -70,73 +71,205 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th
   forces a reindex of it. A document with no remaining path is marked missing, not deleted.
 - **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work
   for the affected canonical paths only.
+- **R10.** Membership sources may be individual files or directories. A directory source is either a
+  one-time recursive snapshot or a persistent recursive rule. Persistent rules reconcile at startup and
+  during explicit reindex jobs; they do not watch the filesystem in real time.
+- **R11.** A directory source may name exact descendant directories to exclude recursively. Exclusions
+  on persistent sources are scoped to that rule. Removing an individual document that is still selected
+  by a rule creates a per-file override; direct membership remains additive and takes precedence.
+- **R12.** Recursive discovery does not follow directory symlinks. A file symlink is accepted only when
+  its resolved target stays beneath the configured document root.
 
 ### Indexing
 
-- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
+- **R13.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time.
   The helper is a fast-starting non-Python program reading a flat map exported from the database.
-- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
+- **R14.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths.
   Neither a plain incremental run nor `recollindex -i` detects such a change.
-- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection,
-  changing membership, and the reindex endpoint all enqueue jobs.
-- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while
+- **R15.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection with
+  initial sources, changing membership, changing a persistent rule, and the reindex endpoint enqueue
+  jobs. Creating an empty collection does not enqueue a no-op job.
+- **R16.** A job records kind, state, progress, total and any error, and is queryable over the API while
   it runs.
-- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either
-  re-enqueued or failed with a reason, never left claiming to be in progress.
+- **R17.** Jobs interrupted by a process restart are recovered at startup: a well-formed job left
+  running is re-enqueued for idempotent replay; an unreadable request is failed with a reason. No job
+  remains claiming to be in progress.
+- **R18.** Membership jobs accept `fail_on_file_error`, defaulting to `false`. Best-effort jobs apply
+  readable files and finish as `completed_with_errors`. Strict jobs discover and validate every source
+  first; any file error fails the job without applying any membership change from that request.
 
 ### Search
 
-- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
+- **R19.** `GET /search` requires a non-empty `collections` parameter holding one or more collection
   **ids**, not names — names are mutable and would break saved queries on rename. A request without the
   parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is
   rejected with `collection_not_found` rather than silently ignored.
-- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently
+- **R20.** Several ids are combined into a single Recoll query, so results come back as one coherently
   ranked set rather than merged per collection.
-- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it
-  belongs to, and Recoll's keyword-in-context snippets.
-- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
+- **R21.** Every hit carries `document_id`. Callers select any other returned fields from title, author,
+  path, MIME type, size, date, relevance score, collections, snippets, extracted text and document
+  links. Omitting the selection returns metadata, collections and snippets, but not text or links.
+- **R22.** Inline extracted text is bounded by `text_limit`, defaulting to 20,000 characters per hit and
+  capped by configuration; the response says when it was truncated. Authenticated endpoints expose the
+  complete extracted text and stream the canonical original separately.
+- **R23.** Results are paginated by `limit` and `offset` and report the match count Recoll reports.
   Xapian estimates this figure for large result sets, so it is documented as approximate rather than
   presented as exact.
 
 ### Authentication
 
-- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
-- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
-  HttpOnly cookie.
-- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are
+- **R24.** Every endpoint except `/health` and `/auth/login` requires an authenticated session.
+- **R25.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an
+  HttpOnly, `SameSite=Lax` cookie. Unsafe cookie-authenticated requests require a CSRF token.
+- **R26.** An initial administrator is seeded from the environment at first boot. If no credentials are
   supplied and no user exists, the service refuses to start rather than starting unauthenticated.
 
 ### Operations
 
-- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
+- **R27.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring
   authentication.
-- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add.
-- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
+- **R28.** `GET /files` lists one directory of the document volume read-only so a client can choose
+  files or directories to add. API and database paths are root-relative and never expose host paths.
+- **R29.** The service can be mounted at a configured subpath. Routes, OpenAPI, redirects, cookie paths
+  and generated links honor it. Forwarded scheme, host and prefix headers are trusted only from
+  explicitly configured proxies.
+- **R30.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the
   table below.
 
 | Code | Status |
 |---|---|
-| `scope_required` | 400 |
-| `unauthenticated` | 401 |
-| `collection_not_found` | 404 |
-| `collection_name_taken` | 409 |
-| `confirmation_required` | 409 |
-| `validation_failed` | 422 |
+| `invalid_query`, `scope_required` | 400 |
+| `invalid_credentials`, `unauthenticated` | 401 |
+| `csrf_failed` | 403 |
+| `collection_not_found`, `directory_rule_not_found`, `document_not_found`, `job_not_found`, `path_not_found` | 404 |
+| `collection_name_taken`, `confirmation_required`, `job_not_retryable` | 409 |
+| `document_missing` | 410 |
+| `invalid_field`, `invalid_source`, `path_outside_root`, `validation_failed` | 422 |
 | `engine_unavailable` | 503 |
 
-## Shape
+## Component boundaries
 
-Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The
-one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.**
-Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run
-without a container.
+| Module | Responsibility |
+|---|---|
+| `api/` | HTTP routes, request validation, authentication dependencies and serialization |
+| `domain/` | Collection, document, membership, directory-rule and job behavior; no framework imports |
+| `storage/` | SQLite migrations, transactions and repositories |
+| `indexing/` | Discovery, hashing, reconciliation, membership-map export and the durable worker |
+| `engine/` | Search/index Protocol, fake engine and real Recoll adapter |
+| `app.py` | Configuration, dependency wiring, startup recovery and shutdown |
+
+**Only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** Every other module depends on
+its narrow Protocol. A membership job resolves safe paths and hashes content before a transaction
+changes desired state. It then atomically replaces the flat membership map and asks the engine to
+force-reindex affected canonical paths. An engine failure leaves desired state intact; retrying export
+and indexing is idempotent.
+
+## SQLite schema
+
+SQLite foreign keys are enabled. Timestamps are UTC. Collection and username comparison keys are
+Unicode-normalized and case-folded by the application before their unique constraints are checked.
+
+| Table | Material columns and constraints |
+|---|---|
+| `collections` | `id` PK, `name`, `name_key` UNIQUE, `description`, timestamps |
+| `documents` | `content_hash` PK, `byte_size`, `state` (`present`/`missing`), timestamps |
+| `document_paths` | `id` PK, `document_hash` FK, `relative_path` UNIQUE, `is_canonical`, `last_seen_at`, `missing_at`; partial unique index permits one canonical path per document |
+| `direct_memberships` | `(collection_id, document_hash)` PK, timestamp; explicit files and snapshot results |
+| `directory_rules` | `id` PK, `collection_id` FK, `relative_path`, timestamps; collection and path are unique together |
+| `directory_rule_exclusions` | `(rule_id, relative_path)` PK; exact recursive exclusions |
+| `rule_memberships` | `(rule_id, document_hash)` PK, `last_seen_at`; latest materialized reconciliation |
+| `membership_exclusions` | `(collection_id, document_hash)` PK, timestamp; per-file overrides against rules |
+| `jobs` | `id` PK, `kind`, `state`, `request_json`, `fail_on_file_error`, progress, total, error fields and timestamps |
+| `job_file_errors` | `(job_id, relative_path)` PK, stable error code and safe message |
+| `users` | `id` PK, `username`, `username_key` UNIQUE, `password_hash`, `created_at`, `disabled_at` |
+| `sessions` | `token_hash` PK, `csrf_hash`, `user_id` FK, expiry and activity timestamps; raw tokens are not stored |
+| `schema_migrations` | `version` PK, `applied_at`; forward-only migrations |
+
+Effective membership is the union of direct membership and non-excluded rule membership. Direct
+membership wins over both per-file and directory-rule exclusions. Job requests retain JSON snapshots
+rather than foreign-keying every target, so deleting a collection cannot invalidate queued cleanup.
+
+## HTTP API
+
+All paths below are relative to the configured mount prefix. All except `/health` and `/auth/login`
+require a session. Collection parameters on search repeat rather than using mutable names.
+
+| Method and path | Behavior |
+|---|---|
+| `POST /auth/login`, `POST /auth/logout`, `GET /auth/session` | Session lifecycle |
+| `GET /health` | Public readiness; `503` when Recoll or the index is unavailable |
+| `GET /files?path=...` | Browse one directory beneath the document root |
+| `GET /collections`, `GET /collections/{id}` | List or read collections |
+| `POST /collections` | Create, optionally with initial membership sources |
+| `PATCH /collections/{id}` | Rename or edit description without indexing |
+| `DELETE /collections/{id}?confirmation=...` | Delete metadata and enqueue affected-path cleanup |
+| `POST /collections/{id}/memberships` | Add file, snapshot-directory or persistent-directory sources |
+| `GET /collections/{id}/documents` | Page through effective members and their provenance |
+| `DELETE /collections/{id}/documents/{document_hash}` | Remove direct membership and suppress matching rules |
+| `GET /collections/{id}/directory-rules` | List persistent directory sources |
+| `PATCH /collections/{id}/directory-rules/{rule_id}` | Replace exact descendant-directory exclusions and reconcile |
+| `DELETE /collections/{id}/directory-rules/{rule_id}` | Remove one source and reconcile |
+| `GET /search` | Scoped, paginated search with selectable result fields |
+| `GET /documents/{document_hash}/text` | Return complete extracted text |
+| `GET /documents/{document_hash}/file` | Stream the canonical original with range support |
+| `POST /jobs/reindex` | Reconcile and reindex optional paths/collections or the whole root |
+| `GET /jobs`, `GET /jobs/{id}`, `POST /jobs/{id}/retry` | Inspect and retry durable jobs |
+
+Membership sources have an explicit `type`. Directory sources also carry `mode` (`snapshot` or
+`persistent`) and optional exact descendant `excluded_directories`. Mutations default
+`fail_on_file_error` to `false` and return `202` with a job. Empty collection creation is synchronous
+and returns `201`; creation with initial sources returns `202` with the collection and job.
+
+```json
+{
+  "sources": [
+    {"type": "file", "path": "books/a.epub"},
+    {
+      "type": "directory",
+      "path": "books",
+      "mode": "persistent",
+      "excluded_directories": ["books/.calibre"]
+    }
+  ],
+  "fail_on_file_error": false
+}
+```
+
+`GET /search` requires non-empty `q` and repeated `collections`. It accepts repeated `fields` selected
+from `title`, `author`, `path`, `mime_type`, `size`, `date`, `score`, `collections`, `snippets`, `text`
+and `links`; `document_id` is unconditional. Omitting `fields` returns every metadata field plus
+collections and snippets. `text_limit` defaults to 20,000 characters per hit, is capped by
+configuration and accompanies `text_truncated`. Links point at the authenticated text and file routes
+and include the configured mount prefix.
+
+Search defaults to `limit=20` and caps it at 100. Other list endpoints default to 50 and cap at 200;
+all use non-negative `offset`. Invalid bounds use `validation_failed`. Job file errors use the stable
+codes `file_vanished`, `file_unreadable`, `file_changed_during_scan`, `file_outside_root` and
+`file_hash_failed`; messages never reveal host paths.
+
+Login returns an opaque session cookie and a CSRF token. Unsafe cookie-authenticated requests require
+`X-CSRF-Token`; login uses strict origin validation. Production cookies are `Secure`, `HttpOnly`,
+`SameSite=Lax` and scoped to the mount prefix. Authentication failures do not reveal whether a username
+exists. Unexpected errors return a request ID rather than exception text or filesystem details.
 
 ## Testing
 
-- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container.
+- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. They
+  cover Unicode name uniqueness, canonical promotion, overlapping rules, exclusion precedence, and
+  strict versus best-effort jobs.
+- Filesystem tests generate trees with duplicate content, same-name/different-content files, vanished
+  and unreadable entries, directory symlinks, and file symlinks resolving both inside and outside the
+  document root.
+- Queue tests interrupt each durable transition and verify startup recovery and idempotent replay.
+- API contract tests cover every error envelope, CSRF rule, selectable search field and pagination
+  boundary under both `/` and a non-root mount prefix.
 - One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when
   the container is unreachable; it never fails, and it never silently passes.
-- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it.
+- An end-to-end test indexes a generated corpus on a `:ro` mount, exercises snapshot and persistent
+  sources with exclusions, searches with different field selections, and retrieves full text plus a
+  byte range from an original file.
+- Security tests cover path traversal, untrusted forwarded headers, cookie scope, CSRF, response-header
+  injection and safe file disposition.
 - Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a
   decision first, because `AGENTS.md` forbids committing indexed documents.
 
@@ -150,15 +283,25 @@ without a container.
       search (C3, R5, R7).
 - [ ] Two files sharing a name with different content are two documents and both are findable (C3).
 - [ ] Removing a canonical path promotes another path and the document remains findable (R8).
-- [ ] A search without `collections` is rejected with `scope_required` (C4, R15).
+- [ ] Snapshot directories add their current documents once; persistent directories reconcile at
+      startup and on reindex, honoring scoped exact-directory exclusions (R10–R12).
+- [ ] A direct file addition remains a member despite a rule exclusion, and removing a rule never
+      removes membership contributed by another source (R11).
+- [ ] A strict job applies no membership changes after any discovery failure; the same best-effort job
+      applies valid files and reports `completed_with_errors` (R18).
+- [ ] A search without `collections` is rejected with `scope_required` (C4, R19).
 - [ ] A search across two collections returns one ranked set, with each hit reporting its collections
-      (R16, R17).
+      (R20, R21).
+- [ ] Search field selection omits unrequested data; inline text is bounded and full authenticated text
+      and ranged original-file retrieval work (R21–R22).
 - [ ] A membership change is visible to search after its job completes, without the document being
-      written to (C1, R10, R11).
+      written to (C1, R13, R14).
 - [ ] Indexing a document volume mounted `:ro` succeeds (C1).
-- [ ] A job interrupted by a restart is not left claiming to be in progress (R14).
-- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19).
-- [ ] The service refuses to start with no user and no seeded credentials (R21).
+- [ ] A job interrupted by a restart is not left claiming to be in progress (R17).
+- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R24).
+- [ ] The service refuses to start with no user and no seeded credentials (R26).
+- [ ] Hosting under `/recoll` produces working routes, cookies, OpenAPI and document links, while
+      untrusted forwarded-prefix headers are ignored (R29).
 - [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than
       fails with the container down.
 
@@ -166,9 +309,6 @@ without a container.
 
 Accepted deliberately, recorded so they are not mistaken for oversights.
 
-- **No way to read a document.** Search reports that a file matched but offers no extracted text and no
-  download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is
-  002's first requirement.
 - **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected
   to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is
   tracked in `docs/TODO.md`.