From def811bab46748bd65fb605e04f6ccb88a98caff Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 18 Sep 2026 16:57:54 +0200 Subject: [PATCH] docs: close the six open design questions for spec 001 Records the outcome of the 001 brainstorm and the Recoll spike that backed it. Decisions: Python + FastAPI managed with uv; the product split into four specs built in order; collections managed at runtime through the API with SQLite as the store of record; indexing as an in-process job queue; search scope as an explicit required parameter; results carrying metadata and snippets; real authentication in 001 with user administration deferred to 004. The load-bearing one is collection membership. A collection cannot be a directory, because files sharing a directory must be able to belong to different collections, so every document is indexed once into a single Recoll index and membership is a user.collection extended attribute mapped to a queryable field. Verified against Recoll 1.43.0 in a throwaway container: the tag must be an opaque single-token ID rather than the collection name, because Recoll tokenises the field value and matches case-insensitively, so a tag of cook-books is also matched by collection:cook and collection:books. Names as tags would let collections silently bleed into each other. Opaque IDs also make renaming free, since the index is never touched. Two operational consequences are recorded in docs/TODO.md: documents must live on a volume with extended-attribute support, checked loudly at startup, and a plain incremental recollindex does not notice xattr-only changes, so retagging must go through recollindex -i on the affected file. uv 0.12.16 was installed on this host as part of the stack decision; AGENTS.md is corrected in the same commit, since it previously listed uv as absent. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 5 +- docs/ARCHITECTURE.md | 83 ++++++++++++++++++++-------- docs/DECISIONS.md | 125 +++++++++++++++++++++++++++++++++++++++++++ docs/TODO.md | 17 +++--- 4 files changed, 201 insertions(+), 29 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d8433d6..55c4d84 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,8 +15,9 @@ TODO(maintainer): no stack yet. Fill install/dev/test/single-test/lint/typecheck commit that scaffolds the app, and delete this line. These must be what CI runs. Installed on this host: `docker` 29.5.2 with `docker compose`, `node` 20.19.2, `npm` 9.2.0, `bun` 1.3.14, -`yarn` 1.22.22, `python3` 3.13.5, `dotnet` 10.0.302. -**Absent: `pnpm`, `uv`, `poetry`, `go`, `cargo`** — do not write commands that assume them. +`yarn` 1.22.22, `python3` 3.13.5, `uv` 0.12.16 (in `~/.local/bin`, already on `PATH`), `dotnet` 10.0.302. +**Absent: `pnpm`, `poetry`, `go`, `cargo`** — do not write commands that assume them. +`uv` is the package manager for this project; do not use bare `pip`, `venv`, or `poetry`. ## Rules diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 4b88116..584d6a6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -4,8 +4,9 @@ How `recoll-modern` is built today. Update in the same commit as any structural ## Overview -**Nothing is built yet.** This file currently records intent and the open questions that the first -spec must close. Do not read anything below "Intended shape" as describing existing code. +**Nothing is built yet.** This file records intent, the design questions that are now closed, and the +Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing +code. The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern web UI and (b) an MCP server, running in Docker, with several independent collections such as @@ -21,30 +22,66 @@ web UI and (b) an MCP server, running in Docker, with several independent collec No source directories exist yet. Add rows here as they appear. +## Delivery order + +The product is four specs, built in order. Each gets its own spec, plan and review cycle. + +| Spec | Scope | +|---|---| +| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication | +| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` | +| 003 MCP server | Thin wrapper over 001's HTTP API | +| 004 Multi-user management | Roles, user CRUD, per-collection permissions | + ## Intended shape -Not yet decided or validated. The pieces the spec has to place: - -- A Recoll layer that owns indexing and querying, per collection. +- **One** Recoll index for all documents, not one per collection. Collection membership is per-file + metadata, not directory layout, so two files in the same directory can belong to different + collections. +- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`). +- SQLite holds collections, membership, indexing jobs, users and sessions. - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge. -- An MCP server exposing search (and likely collection listing) as tools. -- A Docker image that contains Recoll itself, plus compose wiring for document volumes and index storage. +- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage. + +## Verified Recoll mechanics + +Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running +Recoll 1.43.0 / Xapian 1.4.29. Re-verify if the Recoll major version changes. + +- **Per-file metadata is indexable and queryable.** A `fields` file declaring + `[xattrtofields] collection = collection`, `[prefixes] collection = XYCOLL` and `[stored] collection =` + makes the `user.collection` extended attribute a queryable field (`collection:`), returned on + each result. Recoll strips the `user.` prefix itself. +- **Multi-membership works.** A space-separated xattr value puts one file in several collections. An + `OR` across collections returns it **once**, in a single coherently ranked result set — Xapian does + the ranking, so no application-side merging is needed. +- **A plain incremental `recollindex` does not see xattr-only changes.** It compares mtime, and + `setfattr` does not change mtime; `touch` does not help either. `recollindex -i ` does pick the + change up, in roughly 20ms for one file. Retagging therefore costs one file, never a rebuild. +- **The field value is tokenised and matched case-insensitively.** A tag of `cook-books` is matched by + `collection:cook` *and* `collection:books`; `TechBooks` is matched by `collection:techbooks`. Using + human-readable collection names as tags would let collections silently bleed into one another. +- **An unscoped query returns every document, including untagged ones.** Recoll will not enforce + collection scoping; the API layer must. ## Open design questions -The first spec must answer these. None are settled. +Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning: -1. **Collection isolation.** Recoll's model is a config directory per index (`RECOLL_CONFDIR`). - Confirm against Recoll's own docs, then decide: one config dir per collection, and what owns the - mapping from collection name to config dir and document root. -2. **Cross-collection search.** Allowed at all? If yes, merged ranking across separately-scored indexes - is not free — decide before building the UI around it. -3. **Indexing trigger.** Cron inside the container, Recoll's real-time monitor, or API-triggered runs. - Affects whether the container needs a long-running process beyond the web server. -4. **Stack.** Unchosen. See `AGENTS.md` Commands for what this host actually has installed. -5. **Auth and exposure.** Whether the service authenticates at all, or assumes a reverse proxy does it. -6. **Result presentation.** Recoll returns snippets and can extract/preview documents; decide how much - of that the API surfaces, since it drives both the UI and the MCP tool shape. +1. ~~Collection isolation~~ — one index; membership is an opaque collection ID in a `user.collection` + xattr; human names live only in SQLite. +2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's. +3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite. +4. ~~Stack~~ — Python + FastAPI, managed with `uv`. +5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004. +6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate + endpoints. + +Still open, to be closed by the remaining sections of the 001 design: + +- Component boundaries and the SQLite schema. +- API surface, pagination, and error semantics. +- Test strategy, including how tests skip when the container is down. ## External dependencies @@ -53,11 +90,15 @@ The first spec must answer these. None are settled. | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. | | Docker + compose | The only supported way to run the service. | | Document volumes | Nothing to index; collections resolve to empty. | +| Extended-attribute support on the document volume | Collection membership. Without xattrs every document is untagged and every scoped search returns nothing — a silent failure, so it must be checked at startup. | ## Invariants -To be established with the first code. Candidates the spec should confirm and then record here: -a query never silently spans collections; indexed content is never written into the repository. +- A query never silently spans collections: the scope is an explicit, required parameter, and the API + refuses a search that does not name one. +- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so + collections cannot bleed into one another and renaming never touches the index. +- Indexed content is never written into the repository. ## Known rough edges diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index cefdd71..f19ffc5 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -3,6 +3,131 @@ Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back. One entry per decision a future maintainer would otherwise have to reverse-engineer. +## 2026-09-18 — Search results carry metadata and snippets, not full text +**Status:** accepted +**Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the +original file. The response shape drives both the web UI and the MCP tool. +**Decision:** a search hit carries title, author, path, MIME type, size, date, score, its collections and +Recoll's snippets. Extracted text and file download are separate endpoints. +**Alternatives:** metadata only (a results list without context is markedly worse, and an MCP client +would fetch every candidate to judge it, costing more overall); metadata plus full extracted text +(responses become enormous for book-length PDFs and blow an agent's context window). +**Consequences:** one search call is enough for an agent to judge relevance. The UI needs a second call +to open a document, which is the right trade. + +--- + +## 2026-09-18 — Authentication in 001, user administration in 004 +**Status:** accepted +**Context:** the service needs real auth, but roles, user CRUD and per-collection permissions are a +subsystem in their own right, and `PRODUCT.md` currently describes a single technical operator. +**Decision:** 001 ships genuine authentication — argon2 password hashing, server-side sessions, an admin +account seeded from the environment at first boot, every endpoint behind an auth dependency. User +administration and permissions are deferred to spec 004. +**Alternatives:** no auth in the service, trusting a reverse proxy (a misconfigured proxy or an exposed +Docker port would leave the destructive endpoints naked); the full identity system in 001 (a large +addition to an already-grown spec, building for users who do not exist yet). +**Consequences:** 001 is safe to expose behind a proxy. 004 must extend the auth layer without +reshaping endpoints, so the auth dependency is designed as a seam from the start. + +--- + +## 2026-09-18 — Search scope is an explicit, required parameter +**Status:** accepted +**Context:** `docs/ARCHITECTURE.md` proposed the invariant that a query never silently spans +collections. A Recoll query with no collection clause returns every document, including untagged ones. +**Decision:** every search names one or more collections; omitting the parameter is an error, not an +implicit "all". Multiple collections are OR-ed into one Xapian query so ranking stays coherent and each +hit reports which collections it belongs to. +**Alternatives:** single-collection-only searches (the UI and MCP would be built around a model they +later outgrow); a searchable-by-default flag (reintroduces the implicit scope the invariant forbids); +per-collection fan-out and application-side merging (scores from separate databases are not comparable, +so the merged order would be strictly worse than letting Xapian rank). +**Consequences:** the API layer enforces the invariant, because Recoll will not. Every search path needs +a test proving an unscoped request is rejected. + +--- + +## 2026-09-18 — Indexing runs as an in-process job queue +**Status:** accepted +**Context:** collections are created at runtime, so something must index a collection the moment it is +created, and report progress back to the UI and to MCP clients. +**Decision:** creating a collection, changing membership, or calling the reindex endpoint enqueues a job. +A background worker in the same FastAPI process runs `recollindex` and records status, progress and +errors in SQLite. One container process, no broker. +**Alternatives:** `recollindex -m` real-time monitoring (one supervised process per collection, inotify +watch limits, and a new collection still needs a first full index from somewhere — revisit as an opt-in +mode later); cron-scheduled full runs (new collections sit empty until the next tick, and there is no +way to force a refresh). +**Consequences:** jobs die with the process, so the queue must recover interrupted runs at startup. +Job state is queryable, so "indexing, 40%" is available to every consumer. + +--- + +## 2026-09-18 — Collections are runtime-managed through the API +**Status:** accepted +**Context:** collections could be declared in a mounted config file or managed at runtime. +**Decision:** collections are created, edited and deleted through the HTTP API, with SQLite as the store +of record. The service creates and maintains the index state on demand. +**Alternatives:** a declarative mounted `collections.yaml` (reproducible and git-friendly, but no +management UI without adding a write path later); one collection per subdirectory by convention (forces +the on-disk layout to match the collection layout, which the per-file membership requirement rules out). +**Consequences:** 001 grows a persistence layer, validation and migrations. `AGENTS.md` makes deleting +or rebuilding an index an ask-first operation, so the delete path needs an explicit guard rather than a +bare `DELETE`. + +--- + +## 2026-09-18 — One index; collection membership is an opaque per-file tag +**Status:** accepted +**Context:** a collection cannot be "a directory", because files sharing a directory must be able to +belong to different collections. Verified in a throwaway Recoll 1.43.0 container — details and the +exact config in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics". +**Decision:** index every document once into a single Recoll index. Membership is a `user.collection` +extended attribute mapped to a queryable Recoll field. The tag is an **opaque single-token ID**; the +human-readable name lives only in SQLite. +**Alternatives:** one index per collection with rule-based selection (a file in two collections is +stored and indexed twice and can be returned twice, and hand-picking files in a shared directory is +awkward); an explicit file list driving indexing (does not map onto Recoll's topdirs-walking model). +**Why opaque IDs:** Recoll tokenises the field value and matches case-insensitively, so a tag of +`cook-books` is also matched by `collection:cook` and `collection:books`, and `TechBooks` by +`collection:techbooks`. Names-as-tags would let collections silently bleed into each other — exactly +what the no-silent-spanning invariant forbids. Opaque IDs tested clean, with no partial matches. +**Consequences:** collection names need no character restrictions and renaming never touches the index. +Documents must sit on a volume supporting extended attributes, checked at startup — without it every +scoped search silently returns nothing. Retagging costs `recollindex -i `, not a rebuild, because +a plain incremental run does not notice xattr-only changes. + +--- + +## 2026-09-18 — Stack: Python + FastAPI, managed with uv +**Status:** accepted +**Context:** the stack was unchosen and is an ask-first decision. Recoll ships first-class Python +bindings; every other language must drive `recollq` as a subprocess and parse its output. +**Decision:** Python + FastAPI for the engine and HTTP API, with `uv` as the package manager. `uv` +0.12.16 was installed on this host as part of this decision, and `AGENTS.md` updated in the same commit. +**Alternatives:** TypeScript on Node/Bun or C# on ASP.NET Core (one language across UI, API and MCP, but +both make a fragile output parser load-bearing); a split Python engine behind a TypeScript API (two +runtimes and an extra hop inside one container). +**Consequences:** structured results with no parsing layer, and OpenAPI for free, which specs 002 and 003 +can generate clients from. The web UI will likely be a second language; that boundary is the HTTP API. + +--- + +## 2026-09-18 — The product is four specs, built in order +**Status:** accepted +**Context:** "a web UI and an MCP server over Recoll" is four subsystems. One spec covering all of them +would be too large to plan or review, and `PRODUCT.md` is still an uninterviewed stub, so any UI spec +written now would design against fiction. +**Decision:** 001 search engine + API, 002 web UI, 003 MCP server, 004 multi-user management. Each gets +its own spec, plan and review cycle. The table lives in `docs/ARCHITECTURE.md`. +**Alternatives:** one thin vertical slice through all four (would lock in a stack and a UI before +`PRODUCT.md` is real); engine plus MCP together (defensible, but it widens 001, which has already grown). +**Consequences:** 002 is blocked on `/impeccable init`. 001's HTTP API is the contract 002 and 003 are +both built against, so it is the piece that must be got right. + +--- + ## 2026-09-18 — Override the superpowers artefact paths **Status:** accepted **Context:** superpowers 6.3.0 writes specs to `docs/superpowers/specs/YYYY-MM-DD--design.md` diff --git a/docs/TODO.md b/docs/TODO.md index 8476ec3..9c07a39 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -4,17 +4,22 @@ Durable backlog. Agents append here instead of leaving `TODO` comments in source Small, specific, independently actionable. Remove only when actually done. ## Now -- [ ] Answer the open questions in the bootstrap report, then fill the `Commands` section of `AGENTS.md` - and delete its `TODO(maintainer)` line. -- [ ] Brainstorm -> spec `docs/specs/001-*.md` covering the open design questions in - `docs/ARCHITECTURE.md`, starting with collection isolation and the stack choice. +- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in + `docs/DECISIONS.md`. +- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics, + test strategy. Then write `docs/specs/001-search-engine-api.md`. +- [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in + the same commit that scaffolds the app, with every command actually run. ## Next -- [ ] Scaffold the chosen stack; verify install/dev/test/lint/typecheck/build commands by running them, +- [ ] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them, then record the verified ones in `AGENTS.md`. - [ ] Dockerfile + compose: Recoll in the image, volumes for documents and indexes. +- [ ] Startup check: refuse to start, loudly, if the document volume does not support extended + attributes — without them every scoped search silently returns nothing. +- [ ] Recover interrupted indexing jobs at startup; they do not survive a process restart on their own. - [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial - brief, not an interview. + brief, not an interview. Blocks spec 002. ## Someday - [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/`