diff --git a/docs/TODO.md b/docs/TODO.md index 6778149..ff46f6b 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -3,6 +3,37 @@ Durable backlog. Agents append here instead of leaving `TODO` comments in source. Small, specific, independently actionable. Remove only when actually done. +## Resume here + +Last touched 2026-09-18. Still greenfield: **no application code exists.** + +Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review): + +- Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with + reasoning in `docs/DECISIONS.md`. +- Spec `docs/specs/001-search-engine-api.md`: **written, `status: draft`.** It has had its self-review + pass but **has not been signed off by the maintainer.** Do not start a plan or write code against it + until its front matter says `approved`. +- Plan `docs/plans/001-search-engine-api.md`: **not written.** It is the next artefact, and only after + sign-off. + +The next action is to ask the maintainer to review the spec. If they approve it, set `status: approved` +and invoke the superpowers `writing-plans` skill to produce `docs/plans/001-search-engine-api.md`, +numbered to match. + +Things a resuming agent will otherwise get wrong: + +- The design carried collection membership in a `user.collection` extended attribute for part of this + session. **That is superseded** — the documents are read-only. `docs/DECISIONS.md` has both entries, + newest first; the older one is left intact by that file's append-only rule, so read the newer one. +- The Recoll facts in `docs/ARCHITECTURE.md` -> "Verified Recoll mechanics" were measured in a container, + not recalled. Trust them over intuition, and re-verify only if Recoll's major version moves. +- The throwaway spike container is not part of the repo and does not survive the session. Everything + worth keeping from it is already in `docs/`. +- `uv` was installed on this host during this session and `AGENTS.md` was corrected accordingly. The + `Commands` section of `AGENTS.md` is still a `TODO(maintainer)` stub and must be filled in the same + commit that scaffolds the app, with every command actually run first. + ## Now - [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in `docs/DECISIONS.md`. diff --git a/docs/specs/001-search-engine-api.md b/docs/specs/001-search-engine-api.md new file mode 100644 index 0000000..7b05241 --- /dev/null +++ b/docs/specs/001-search-engine-api.md @@ -0,0 +1,175 @@ +--- +status: draft +plan: docs/plans/001-search-engine-api.md (not yet written) +supersedes: none +--- + +# 001 — Search engine and HTTP API + +The first of four specs (see `docs/ARCHITECTURE.md`, "Delivery order"). It delivers the Recoll layer and +the HTTP API that specs 002 (web UI) and 003 (MCP server) are both built against. It ships no UI. + +## Why + +Recoll's own interfaces are single-index, single-machine and desktop-shaped. This service exists to hold +several independent collections at once, serve them over HTTP to both a browser and an AI agent, and run +as a container on a server. The API is the contract two later consumers depend on, so it is the piece +that must be got right first. + +## Scope + +**In:** Recoll in Docker · one index over all documents · collections created and managed at runtime · +per-file collection membership · content-hash document identity · an indexing job queue · a scoped +search endpoint · authentication · a read-only file browse endpoint · health. + +**Out, deferred to a named spec:** the web UI (002) · the MCP server (003) · user administration, roles +and per-collection permissions (004) · fetching a document's extracted text or downloading the original +file (002 — see Known gaps) · real-time index monitoring · OIDC or forward-auth. + +## Constraints + +These are requirements, not preferences. The mechanics behind them were verified against Recoll 1.43.0 +and are recorded in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics"; the reasoning is in +`docs/DECISIONS.md`. + +- **C1. The documents are read-only.** The service must operate with read-only access to everything it + indexes and must never write to a document, its directory, or its extended attributes. The deployment + mounts the document volume `:ro`. +- **C2. Membership is per file, not per directory.** Two files in the same directory may belong to + different collections. +- **C3. Documents identical in content are one document.** Documents sharing a name but differing in + content are separate documents. +- **C4. A search never silently spans collections.** + +## Requirements + +### Collections + +- **R1.** A collection has an opaque immutable id matching `^c[0-9a-f]{10}$`, a human-readable name, and + an optional description. The id is generated by the service and is never supplied by a client. +- **R2.** Names are unique case-insensitively and may contain any characters, because a name never + reaches the index. +- **R3.** Collections are created, read, renamed and deleted over the API. Renaming touches only the + database and enqueues no indexing work. +- **R4.** Deleting a collection requires a confirmation value matching the collection's name. Deletion + removes the collection and its memberships, and enqueues reindexing of the affected canonical paths so + the index stops carrying the deleted id. It never deletes documents, and it never deletes or rebuilds + the Recoll index. Until those jobs complete the index still holds the dead id; this is harmless, + because the API resolves ids against the database and drops any it no longer knows. + +### Documents and membership + +- **R5.** A document is identified by the SHA-256 of its bytes. Several paths may resolve to one + document; each document has exactly one canonical path. +- **R6.** Membership attaches to a document, not a path. A document may belong to any number of + collections, including none. +- **R7.** Only a document's canonical path is reported to Recoll as belonging to a collection. + Non-canonical duplicate paths remain in the index but belong to no collection and are therefore + unreachable by any scoped search. +- **R8.** If a canonical path disappears, the service promotes another known path for that document and + forces a reindex of it. A document with no remaining path is marked missing, not deleted. +- **R9.** Adding or removing documents from a collection updates the database and enqueues indexing work + for the affected canonical paths only. + +### Indexing + +- **R10.** Membership reaches Recoll through `metadatacmds`, which Recoll runs per file at index time. + The helper is a fast-starting non-Python program reading a flat map exported from the database. +- **R11.** A membership change is applied with `recollindex -i -Z` on the affected canonical paths. + Neither a plain incremental run nor `recollindex -i` detects such a change. +- **R12.** Indexing runs as jobs on an in-process queue backed by SQLite. Creating a collection, + changing membership, and the reindex endpoint all enqueue jobs. +- **R13.** A job records kind, state, progress, total and any error, and is queryable over the API while + it runs. +- **R14.** Jobs interrupted by a process restart are recovered at startup: a job left running is either + re-enqueued or failed with a reason, never left claiming to be in progress. + +### Search + +- **R15.** `GET /search` requires a non-empty `collections` parameter holding one or more collection + **ids**, not names — names are mutable and would break saved queries on rename. A request without the + parameter is rejected with `scope_required`; there is no implicit "all". An id that does not exist is + rejected with `collection_not_found` rather than silently ignored. +- **R16.** Several ids are combined into a single Recoll query, so results come back as one coherently + ranked set rather than merged per collection. +- **R17.** A hit carries title, author, path, MIME type, size, date, relevance score, the collections it + belongs to, and Recoll's keyword-in-context snippets. +- **R18.** Results are paginated by `limit` and `offset` and report the match count Recoll reports. + Xapian estimates this figure for large result sets, so it is documented as approximate rather than + presented as exact. + +### Authentication + +- **R19.** Every endpoint except `/health` and `/auth/login` requires an authenticated session. +- **R20.** Passwords are hashed with argon2. Sessions are server-side, expiring, and carried in an + HttpOnly cookie. +- **R21.** An initial administrator is seeded from the environment at first boot. If no credentials are + supplied and no user exists, the service refuses to start rather than starting unauthenticated. + +### Operations + +- **R22.** `GET /health` reports whether Recoll is reachable and the index readable, without requiring + authentication. +- **R23.** `GET /files` lists the document volume read-only so a client can choose files to add. +- **R24.** Errors use one envelope, `{"error": {"code", "message", "details"}}`, with the codes in the + table below. + +| Code | Status | +|---|---| +| `scope_required` | 400 | +| `unauthenticated` | 401 | +| `collection_not_found` | 404 | +| `collection_name_taken` | 409 | +| `confirmation_required` | 409 | +| `validation_failed` | 422 | +| `engine_unavailable` | 503 | + +## Shape + +Module boundaries, the SQLite schema and the endpoint list are recorded in `docs/ARCHITECTURE.md`. The +one rule worth restating here: **only `recoll_modern/engine/` may import `recoll` or run `recollindex`.** +Every other module depends on a narrow Protocol. That boundary is what allows the test suite to run +without a container. + +## Testing + +- Domain, API and job-queue tests run against a fake engine and real SQLite, with no container. +- One contract suite runs twice: against the fake, and against real Recoll. The real run **skips** when + the container is unreachable; it never fails, and it never silently passes. +- An end-to-end test indexes a generated corpus on a `:ro` mount and searches it. +- Fixture corpora are generated at test time rather than committed. Committing a binary fixture needs a + decision first, because `AGENTS.md` forbids committing indexed documents. + +## Acceptance criteria + +- [ ] A collection can be created, renamed and deleted over the API; renaming enqueues no job (R1–R4). +- [ ] Deleting without a matching confirmation value is refused, and no index is deleted or rebuilt (R4). +- [ ] Two files in the same directory can belong to different collections, and each is found only by its + own collection's scope (C2, R6). +- [ ] Two byte-identical files under different names are one document and appear once in a scoped + search (C3, R5, R7). +- [ ] Two files sharing a name with different content are two documents and both are findable (C3). +- [ ] Removing a canonical path promotes another path and the document remains findable (R8). +- [ ] A search without `collections` is rejected with `scope_required` (C4, R15). +- [ ] A search across two collections returns one ranked set, with each hit reporting its collections + (R16, R17). +- [ ] A membership change is visible to search after its job completes, without the document being + written to (C1, R10, R11). +- [ ] Indexing a document volume mounted `:ro` succeeds (C1). +- [ ] A job interrupted by a restart is not left claiming to be in progress (R14). +- [ ] An unauthenticated request to any endpoint except `/health` and `/auth/login` is refused (R19). +- [ ] The service refuses to start with no user and no seeded credentials (R21). +- [ ] `uv run pytest`, `uv run ruff check .` and `uv run mypy src` pass, and the suite skips rather than + fails with the container down. + +## Known gaps + +Accepted deliberately, recorded so they are not mistaken for oversights. + +- **No way to read a document.** Search reports that a file matched but offers no extracted text and no + download. 001 is API-only so this is coherent, but 002 cannot build a usable UI without it, and it is + 002's first requirement. +- **Indexing overhead is measured only on small text files.** The `metadatacmds` helper cost is expected + to be dwarfed by extraction cost for real PDFs and EPUBs; that is an assumption until measured, and is + tracked in `docs/TODO.md`. +- **No per-user access control.** Every authenticated user sees every collection until 004.