From b776ee603e2ee3d658ea141b664786270f3328bb Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 18 Sep 2026 17:34:16 +0200 Subject: [PATCH] docs: complete search engine API design --- PRODUCT.md | 5 +- docs/ARCHITECTURE.md | 49 +++++- docs/DECISIONS.md | 47 ++++++ docs/TODO.md | 9 +- docs/specs/001-search-engine-api.md | 236 ++++++++++++++++++++++------ 5 files changed, 285 insertions(+), 61 deletions(-) diff --git a/PRODUCT.md b/PRODUCT.md index 54081ff..5951673 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -12,8 +12,9 @@ web ## Stack -Undecided. Greenfield: no framework, no scaffold. The stack is a user decision and has not been made. -See `AGENTS.md` for the toolchains actually installed on the host. +The backend stack was confirmed during the 001 design: Python + FastAPI, managed with `uv`, because +Recoll provides native Python bindings. The frontend stack remains undecided. No application has been +scaffolded yet. See `docs/DECISIONS.md` for the decision and `AGENTS.md` for installed toolchains. ## Users diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fb6286c..938c808 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -43,6 +43,43 @@ The product is four specs, built in order. Each gets its own spec, plan and revi - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge. - A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage. +### Component boundaries + +| Module | Responsibility | +|---|---| +| `api/` | HTTP transport, validation, auth dependencies and serialization | +| `domain/` | Framework-independent collection, document, membership, rule and job behavior | +| `storage/` | SQLite migrations, transactions and repositories | +| `indexing/` | Discovery, hashing, directory-rule reconciliation, map export and queue worker | +| `engine/` | The search/index Protocol, fake implementation and Recoll adapter | +| `app.py` | Configuration, dependency wiring and lifecycle | + +Only `engine/` may import Recoll or invoke `recollindex`. Membership changes first establish desired +state in SQLite, atomically export the helper's flat map, then force-index affected canonical paths. +Engine retries are therefore idempotent. Persistent directory rules reconcile at startup and on an +explicit reindex, not through live filesystem monitoring. + +### Storage model + +`collections`, `documents` and `document_paths` hold identity and canonical paths. Membership provenance +is normalized: `direct_memberships` holds explicit files and snapshot expansions; `directory_rules`, +`directory_rule_exclusions` and `rule_memberships` hold persistent additions; and +`membership_exclusions` holds per-file overrides against rule-derived membership. Effective membership +is direct membership union non-excluded rule membership, so direct additions remain additive. + +`jobs` and `job_file_errors` make work and partial failures durable. `users` and hashed-token `sessions` +provide authentication. `schema_migrations` is forward-only. The exact columns and constraints are the +contract in `docs/specs/001-search-engine-api.md`. + +### HTTP boundary + +The resource groups are auth, health, read-only file browsing, collections and membership sources, +directory rules, scoped search, document text/original retrieval, and indexing jobs. Search callers +select returned fields; only `document_id` is unconditional, and inline full text is bounded. Every +route, cookie, redirect, OpenAPI URL and generated link supports a configured mount prefix. Forwarded +prefix and origin data are used only from trusted proxies. The exact endpoints and error codes are the +contract in `docs/specs/001-search-engine-api.md`. + ## Verified Recoll mechanics Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running @@ -83,14 +120,12 @@ Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the re 3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite. 4. ~~Stack~~ — Python + FastAPI, managed with `uv`. 5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004. -6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate - endpoints. +6. ~~Result presentation~~ — callers select result fields; metadata plus snippets is the default, inline + text is bounded, and full text and file download are separate endpoints. -Still open, to be closed by the remaining sections of the 001 design: - -- Component boundaries and the SQLite schema. -- API surface, pagination, and error semantics. -- Test strategy, including how tests skip when the container is down. +The remaining 001 design questions — component boundaries, schema, HTTP surface, pagination, error +semantics and test strategy — are closed in `docs/specs/001-search-engine-api.md`. That spec remains +draft until maintainer sign-off. ## External dependencies diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index 73d61a5..0c2ab3c 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -3,6 +3,53 @@ Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back. One entry per decision a future maintainer would otherwise have to reverse-engineer. +## 2026-09-18 — Deployment supports a configured URL subpath +**Status:** accepted +**Context:** the service may sit behind an existing reverse proxy under a path such as `/recoll`, not at +the origin root. Hard-coded root URLs break redirects, cookies, OpenAPI and document links there. +**Decision:** one configured external mount prefix drives routing, generated URLs and cookie scope. +Forwarded scheme, host and prefix values are honored only from explicitly trusted proxies. +**Alternatives:** root-only hosting (needlessly constrains deployment); trust every forwarded header +(lets a direct client forge external URLs and cookie paths). +**Consequences:** API and end-to-end tests run at both `/` and a non-root prefix. Proxy configuration is +part of deployment documentation, and all URL generation goes through one boundary. + +--- + +## 2026-09-18 — Search result fields are caller-selected +**Status:** accepted +**Supersedes:** the fixed result shape in "Search results carry metadata and snippets, not full text" +(below, same date). The metadata-plus-snippets shape remains the default, not the only representation. +**Context:** browser and MCP consumers have different response-size needs, and users need to choose +whether results include metadata, snippets, extracted text or retrieval links. +**Decision:** `document_id` is unconditional; callers select all other result fields. Inline extracted +text is opt-in, bounded and marked when truncated. Authenticated endpoints return full extracted text +and stream the canonical original separately. +**Alternatives:** one fixed rich shape (wastes bandwidth and agent context); metadata-only search with a +follow-up per hit (too many round trips); unbounded full text inline (a book can dominate the response). +**Consequences:** field projection is part of the engine/API contract. Generated retrieval links honor +the configured mount prefix, and original-file responses require range and safe-disposition handling. + +--- + +## 2026-09-18 — Directory membership has normalized additive provenance +**Status:** accepted +**Context:** users need both one-time directory expansion and persistent directory membership, including +the ability to omit metadata subdirectories such as `.calibre`. Overlapping sources must not make +removal order-dependent. +**Decision:** direct file additions and snapshot expansions are durable direct membership. Persistent +directory rules materialize their own contributed documents and hold exact recursive exclusions scoped +to the rule. Per-file removal creates an override against rule-derived membership; an explicit direct +addition still wins. Persistent rules reconcile at startup and explicit reindex, not through watching. +**Alternatives:** one polymorphic source ledger (weaker relational constraints); recompute every rule on +demand (slow and nondeterministic); persistent rules without provenance (removing one rule can erase +membership contributed by another). +**Consequences:** the schema has separate direct, rule-derived and exclusion tables. Directory symlinks +are never followed; file symlinks are accepted only inside the document root. Jobs default to +best-effort, while strict jobs scan first and apply no membership changes if any file fails. + +--- + ## 2026-09-18 — Document identity is a content hash, not a path **Status:** accepted **Context:** the document set may contain files that share a name but differ in content, and files that diff --git a/docs/TODO.md b/docs/TODO.md index ff46f6b..f2a20b9 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -11,9 +11,10 @@ Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan - - Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with reasoning in `docs/DECISIONS.md`. -- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** It has had its self-review - pass but **has not been signed off by the maintainer.** Do not start a plan or write code against it - until its front matter says `approved`. +- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** Its missing schema, component, + endpoint and test designs were completed after maintainer-guided review, including directory rules, + selectable search fields and subpath hosting. It **has not received final maintainer sign-off.** Do + not start a plan or write code against it until its front matter says `approved`. - Plan `docs/plans/001-search-engine-api.md`: **not written.** It is the next artefact, and only after sign-off. @@ -37,7 +38,7 @@ Things a resuming agent will otherwise get wrong: ## Now - [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in `docs/DECISIONS.md`. -- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics, +- [x] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics, test strategy. Then write `docs/specs/001-search-engine-api.md`. - [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in the same commit that scaffolds the app, with every command actually run. diff --git a/docs/specs/001-search-engine-api.md b/docs/specs/001-search-engine-api.md index 7b05241..b8eb29e 100644 --- a/docs/specs/001-search-engine-api.md +++ b/docs/specs/001-search-engine-api.md @@ -19,12 +19,13 @@ that must be got right first. ## Scope **In:** Recoll in Docker · one index over all documents · collections created and managed at runtime · -per-file collection membership · content-hash document identity · an indexing job queue · a scoped -search endpoint · authentication · a read-only file browse endpoint · health. +per-file collection membership · snapshot and persistent directory sources · content-hash document +identity · an indexing job queue · a scoped search endpoint with caller-selected fields · authenticated +text and original-file retrieval · authentication · a read-only file browse endpoint · configurable +subpath hosting · health. **Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles -and per-collection permissions (004) · fetching a document's extracted text or downloading the original -file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth. +and per-collection permissions (004) · real-time index monitoring · OIDC or forward-auth. ## Constraints @@ -47,8 +48,8 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th - **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and an optional description. The id is generated by the service and is never supplied by a client. -- **R2.** Names are unique case-insensitively and may contain any characters, because a name never - reaches the index. +- **R2.** Names contain at least one non-whitespace character, are unique after Unicode normalization + and case-folding, and otherwise accept arbitrary Unicode because a name never reaches the index. - **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the database and enqueues no indexing work. - **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion @@ -60,7 +61,7 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th ### Documents and membership - **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one - document; each document has exactly one canonical path. + document; each present document has exactly one canonical path and a missing document has none. - **R6.** Membership attaches to a document, not a path. A document may belong to any number of collections, including none. - **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection. @@ -70,73 +71,205 @@ and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; th forces a reindex of it. A document with no remaining path is marked missing, not deleted. - **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work for the affected canonical paths only. +- **R10.** Membership sources may be individual files or directories. A directory source is either a + one-time recursive snapshot or a persistent recursive rule. Persistent rules reconcile at startup and + during explicit reindex jobs; they do not watch the filesystem in real time. +- **R11.** A directory source may name exact descendant directories to exclude recursively. Exclusions + on persistent sources are scoped to that rule. Removing an individual document that is still selected + by a rule creates a per-file override; direct membership remains additive and takes precedence. +- **R12.** Recursive discovery does not follow directory symlinks. A file symlink is accepted only when + its resolved target stays beneath the configured document root. ### Indexing -- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time. +- **R13.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time. The helper is a fast-starting non-Python program reading a flat map exported from the database. -- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths. +- **R14.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths. Neither a plain incremental run nor `recollindex -i` detects such a change. -- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection, - changing membership, and the reindex endpoint all enqueue jobs. -- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while +- **R15.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection with + initial sources, changing membership, changing a persistent rule, and the reindex endpoint enqueue + jobs. Creating an empty collection does not enqueue a no-op job. +- **R16.** A job records kind, state, progress, total and any error, and is queryable over the API while it runs. -- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either - re-enqueued or failed with a reason, never left claiming to be in progress. +- **R17.** Jobs interrupted by a process restart are recovered at startup: a well-formed job left + running is re-enqueued for idempotent replay; an unreadable request is failed with a reason. No job + remains claiming to be in progress. +- **R18.** Membership jobs accept `fail_on_file_error`, defaulting to `false`. Best-effort jobs apply + readable files and finish as `completed_with_errors`. Strict jobs discover and validate every source + first; any file error fails the job without applying any membership change from that request. ### Search -- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection +- **R19.** `GET /search` requires a non-empty `collections` parameter holding one or more collection **ids**, not names — names are mutable and would break saved queries on rename. A request without the parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is rejected with `collection_not_found` rather than silently ignored. -- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently +- **R20.** Several ids are combined into a single Recoll query, so results come back as one coherently ranked set rather than merged per collection. -- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it - belongs to, and Recoll's keyword-in-context snippets. -- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports. +- **R21.** Every hit carries `document_id`. Callers select any other returned fields from title, author, + path, MIME type, size, date, relevance score, collections, snippets, extracted text and document + links. Omitting the selection returns metadata, collections and snippets, but not text or links. +- **R22.** Inline extracted text is bounded by `text_limit`, defaulting to 20,000 characters per hit and + capped by configuration; the response says when it was truncated. Authenticated endpoints expose the + complete extracted text and stream the canonical original separately. +- **R23.** Results are paginated by `limit` and `offset` and report the match count Recoll reports. Xapian estimates this figure for large result sets, so it is documented as approximate rather than presented as exact. ### Authentication -- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session. -- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an - HttpOnly cookie. -- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are +- **R24.** Every endpoint except `/health` and `/auth/login` requires an authenticated session. +- **R25.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an + HttpOnly, `SameSite=Lax` cookie. Unsafe cookie-authenticated requests require a CSRF token. +- **R26.** An initial administrator is seeded from the environment at first boot. If no credentials are supplied and no user exists, the service refuses to start rather than starting unauthenticated. ### Operations -- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring +- **R27.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring authentication. -- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add. -- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the +- **R28.** `GET /files` lists one directory of the document volume read-only so a client can choose + files or directories to add. API and database paths are root-relative and never expose host paths. +- **R29.** The service can be mounted at a configured subpath. Routes, OpenAPI, redirects, cookie paths + and generated links honor it. Forwarded scheme, host and prefix headers are trusted only from + explicitly configured proxies. +- **R30.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the table below. | Code | Status | |---|---| -| `scope_required` | 400 | -| `unauthenticated` | 401 | -| `collection_not_found` | 404 | -| `collection_name_taken` | 409 | -| `confirmation_required` | 409 | -| `validation_failed` | 422 | +| `invalid_query`, `scope_required` | 400 | +| `invalid_credentials`, `unauthenticated` | 401 | +| `csrf_failed` | 403 | +| `collection_not_found`, `directory_rule_not_found`, `document_not_found`, `job_not_found`, `path_not_found` | 404 | +| `collection_name_taken`, `confirmation_required`, `job_not_retryable` | 409 | +| `document_missing` | 410 | +| `invalid_field`, `invalid_source`, `path_outside_root`, `validation_failed` | 422 | | `engine_unavailable` | 503 | -## Shape +## Component boundaries -Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The -one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** -Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run -without a container. +| Module | Responsibility | +|---|---| +| `api/` | HTTP routes, request validation, authentication dependencies and serialization | +| `domain/` | Collection, document, membership, directory-rule and job behavior; no framework imports | +| `storage/` | SQLite migrations, transactions and repositories | +| `indexing/` | Discovery, hashing, reconciliation, membership-map export and the durable worker | +| `engine/` | Search/index Protocol, fake engine and real Recoll adapter | +| `app.py` | Configuration, dependency wiring, startup recovery and shutdown | + +**Only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** Every other module depends on +its narrow Protocol. A membership job resolves safe paths and hashes content before a transaction +changes desired state. It then atomically replaces the flat membership map and asks the engine to +force-reindex affected canonical paths. An engine failure leaves desired state intact; retrying export +and indexing is idempotent. + +## SQLite schema + +SQLite foreign keys are enabled. Timestamps are UTC. Collection and username comparison keys are +Unicode-normalized and case-folded by the application before their unique constraints are checked. + +| Table | Material columns and constraints | +|---|---| +| `collections` | `id` PK, `name`, `name_key` UNIQUE, `description`, timestamps | +| `documents` | `content_hash` PK, `byte_size`, `state` (`present`/`missing`), timestamps | +| `document_paths` | `id` PK, `document_hash` FK, `relative_path` UNIQUE, `is_canonical`, `last_seen_at`, `missing_at`; partial unique index permits one canonical path per document | +| `direct_memberships` | `(collection_id, document_hash)` PK, timestamp; explicit files and snapshot results | +| `directory_rules` | `id` PK, `collection_id` FK, `relative_path`, timestamps; collection and path are unique together | +| `directory_rule_exclusions` | `(rule_id, relative_path)` PK; exact recursive exclusions | +| `rule_memberships` | `(rule_id, document_hash)` PK, `last_seen_at`; latest materialized reconciliation | +| `membership_exclusions` | `(collection_id, document_hash)` PK, timestamp; per-file overrides against rules | +| `jobs` | `id` PK, `kind`, `state`, `request_json`, `fail_on_file_error`, progress, total, error fields and timestamps | +| `job_file_errors` | `(job_id, relative_path)` PK, stable error code and safe message | +| `users` | `id` PK, `username`, `username_key` UNIQUE, `password_hash`, `created_at`, `disabled_at` | +| `sessions` | `token_hash` PK, `csrf_hash`, `user_id` FK, expiry and activity timestamps; raw tokens are not stored | +| `schema_migrations` | `version` PK, `applied_at`; forward-only migrations | + +Effective membership is the union of direct membership and non-excluded rule membership. Direct +membership wins over both per-file and directory-rule exclusions. Job requests retain JSON snapshots +rather than foreign-keying every target, so deleting a collection cannot invalidate queued cleanup. + +## HTTP API + +All paths below are relative to the configured mount prefix. All except `/health` and `/auth/login` +require a session. Collection parameters on search repeat rather than using mutable names. + +| Method and path | Behavior | +|---|---| +| `POST /auth/login`, `POST /auth/logout`, `GET /auth/session` | Session lifecycle | +| `GET /health` | Public readiness; `503` when Recoll or the index is unavailable | +| `GET /files?path=...` | Browse one directory beneath the document root | +| `GET /collections`, `GET /collections/{id}` | List or read collections | +| `POST /collections` | Create, optionally with initial membership sources | +| `PATCH /collections/{id}` | Rename or edit description without indexing | +| `DELETE /collections/{id}?confirmation=...` | Delete metadata and enqueue affected-path cleanup | +| `POST /collections/{id}/memberships` | Add file, snapshot-directory or persistent-directory sources | +| `GET /collections/{id}/documents` | Page through effective members and their provenance | +| `DELETE /collections/{id}/documents/{document_hash}` | Remove direct membership and suppress matching rules | +| `GET /collections/{id}/directory-rules` | List persistent directory sources | +| `PATCH /collections/{id}/directory-rules/{rule_id}` | Replace exact descendant-directory exclusions and reconcile | +| `DELETE /collections/{id}/directory-rules/{rule_id}` | Remove one source and reconcile | +| `GET /search` | Scoped, paginated search with selectable result fields | +| `GET /documents/{document_hash}/text` | Return complete extracted text | +| `GET /documents/{document_hash}/file` | Stream the canonical original with range support | +| `POST /jobs/reindex` | Reconcile and reindex optional paths/collections or the whole root | +| `GET /jobs`, `GET /jobs/{id}`, `POST /jobs/{id}/retry` | Inspect and retry durable jobs | + +Membership sources have an explicit `type`. Directory sources also carry `mode` (`snapshot` or +`persistent`) and optional exact descendant `excluded_directories`. Mutations default +`fail_on_file_error` to `false` and return `202` with a job. Empty collection creation is synchronous +and returns `201`; creation with initial sources returns `202` with the collection and job. + +```json +{ + "sources": [ + {"type": "file", "path": "books/a.epub"}, + { + "type": "directory", + "path": "books", + "mode": "persistent", + "excluded_directories": ["books/.calibre"] + } + ], + "fail_on_file_error": false +} +``` + +`GET /search` requires non-empty `q` and repeated `collections`. It accepts repeated `fields` selected +from `title`, `author`, `path`, `mime_type`, `size`, `date`, `score`, `collections`, `snippets`, `text` +and `links`; `document_id` is unconditional. Omitting `fields` returns every metadata field plus +collections and snippets. `text_limit` defaults to 20,000 characters per hit, is capped by +configuration and accompanies `text_truncated`. Links point at the authenticated text and file routes +and include the configured mount prefix. + +Search defaults to `limit=20` and caps it at 100. Other list endpoints default to 50 and cap at 200; +all use non-negative `offset`. Invalid bounds use `validation_failed`. Job file errors use the stable +codes `file_vanished`, `file_unreadable`, `file_changed_during_scan`, `file_outside_root` and +`file_hash_failed`; messages never reveal host paths. + +Login returns an opaque session cookie and a CSRF token. Unsafe cookie-authenticated requests require +`X-CSRF-Token`; login uses strict origin validation. Production cookies are `Secure`, `HttpOnly`, +`SameSite=Lax` and scoped to the mount prefix. Authentication failures do not reveal whether a username +exists. Unexpected errors return a request ID rather than exception text or filesystem details. ## Testing -- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. +- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. They + cover Unicode name uniqueness, canonical promotion, overlapping rules, exclusion precedence, and + strict versus best-effort jobs. +- Filesystem tests generate trees with duplicate content, same-name/different-content files, vanished + and unreadable entries, directory symlinks, and file symlinks resolving both inside and outside the + document root. +- Queue tests interrupt each durable transition and verify startup recovery and idempotent replay. +- API contract tests cover every error envelope, CSRF rule, selectable search field and pagination + boundary under both `/` and a non-root mount prefix. - One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when the container is unreachable; it never fails, and it never silently passes. -- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it. +- An end-to-end test indexes a generated corpus on a `:ro` mount, exercises snapshot and persistent + sources with exclusions, searches with different field selections, and retrieves full text plus a + byte range from an original file. +- Security tests cover path traversal, untrusted forwarded headers, cookie scope, CSRF, response-header + injection and safe file disposition. - Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a decision first, because `AGENTS.md` forbids committing indexed documents. @@ -150,15 +283,25 @@ without a container. search (C3, R5, R7). - [ ] Two files sharing a name with different content are two documents and both are findable (C3). - [ ] Removing a canonical path promotes another path and the document remains findable (R8). -- [ ] A search without `collections` is rejected with `scope_required` (C4, R15). +- [ ] Snapshot directories add their current documents once; persistent directories reconcile at + startup and on reindex, honoring scoped exact-directory exclusions (R10–R12). +- [ ] A direct file addition remains a member despite a rule exclusion, and removing a rule never + removes membership contributed by another source (R11). +- [ ] A strict job applies no membership changes after any discovery failure; the same best-effort job + applies valid files and reports `completed_with_errors` (R18). +- [ ] A search without `collections` is rejected with `scope_required` (C4, R19). - [ ] A search across two collections returns one ranked set, with each hit reporting its collections - (R16, R17). + (R20, R21). +- [ ] Search field selection omits unrequested data; inline text is bounded and full authenticated text + and ranged original-file retrieval work (R21–R22). - [ ] A membership change is visible to search after its job completes, without the document being - written to (C1, R10, R11). + written to (C1, R13, R14). - [ ] Indexing a document volume mounted `:ro` succeeds (C1). -- [ ] A job interrupted by a restart is not left claiming to be in progress (R14). -- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19). -- [ ] The service refuses to start with no user and no seeded credentials (R21). +- [ ] A job interrupted by a restart is not left claiming to be in progress (R17). +- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R24). +- [ ] The service refuses to start with no user and no seeded credentials (R26). +- [ ] Hosting under `/recoll` produces working routes, cookies, OpenAPI and document links, while + untrusted forwarded-prefix headers are ignored (R29). - [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than fails with the container down. @@ -166,9 +309,6 @@ without a container. Accepted deliberately, recorded so they are not mistaken for oversights. -- **No way to read a document.** Search reports that a file matched but offers no extracted text and no - download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is - 002's first requirement. - **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is tracked in `docs/TODO.md`.