docs: close the six open design questions for spec 001

Records the outcome of the 001 brainstorm and the Recoll spike that backed it.

Decisions: Python + FastAPI managed with uv; the product split into four specs
built in order; collections managed at runtime through the API with SQLite as
the store of record; indexing as an in-process job queue; search scope as an
explicit required parameter; results carrying metadata and snippets; real
authentication in 001 with user administration deferred to 004.

The load-bearing one is collection membership. A collection cannot be a
directory, because files sharing a directory must be able to belong to
different collections, so every document is indexed once into a single Recoll
index and membership is a user.collection extended attribute mapped to a
queryable field. Verified against Recoll 1.43.0 in a throwaway container: the
tag must be an opaque single-token ID rather than the collection name, because
Recoll tokenises the field value and matches case-insensitively, so a tag of
cook-books is also matched by collection:cook and collection:books. Names as
tags would let collections silently bleed into each other. Opaque IDs also make
renaming free, since the index is never touched.

Two operational consequences are recorded in docs/TODO.md: documents must live
on a volume with extended-attribute support, checked loudly at startup, and a
plain incremental recollindex does not notice xattr-only changes, so retagging
must go through recollindex -i on the affected file.

uv 0.12.16 was installed on this host as part of the stack decision; AGENTS.md
is corrected in the same commit, since it previously listed uv as absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -15,8 +15,9 @@ TODO(maintainer): no stack yet. Fill install/dev/test/single-test/lint/typecheck
 commit that scaffolds the app, and delete this line. These must be what CI runs.
 
 Installed on this host: `docker` 29.5.2 with `docker compose`, `node` 20.19.2, `npm` 9.2.0, `bun` 1.3.14,
-`yarn` 1.22.22, `python3` 3.13.5, `dotnet` 10.0.302.
-**Absent: `pnpm`, `uv`, `poetry`, `go`, `cargo`** — do not write commands that assume them.
+`yarn` 1.22.22, `python3` 3.13.5, `uv` 0.12.16 (in `~/.local/bin`, already on `PATH`), `dotnet` 10.0.302.
+**Absent: `pnpm`, `poetry`, `go`, `cargo`** — do not write commands that assume them.
+`uv` is the package manager for this project; do not use bare `pip`, `venv`, or `poetry`.
 
 ## Rules
 

docs/ARCHITECTURE.md

@@ -4,8 +4,9 @@ How `recoll-modern` is built today. Update in the same commit as any structural
 
 ## Overview
 
-**Nothing is built yet.** This file currently records intent and the open questions that the first
-spec must close. Do not read anything below "Intended shape" as describing existing code.
+**Nothing is built yet.** This file records intent, the design questions that are now closed, and the
+Recoll behaviour verified against a real container. Do not read "Intended shape" as describing existing
+code.
 
 The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
 web UI and (b) an MCP server, running in Docker, with several independent collections such as
@@ -21,30 +22,66 @@ web UI and (b) an MCP server, running in Docker, with several independent collec
 
 No source directories exist yet. Add rows here as they appear.
 
+## Delivery order
+
+The product is four specs, built in order. Each gets its own spec, plan and review cycle.
+
+| Spec | Scope |
+|---|---|
+| 001 Search engine + API | Recoll in Docker, single index, per-file collection membership, collection CRUD, indexing job queue, search endpoint, authentication |
+| 002 Web UI | Blocked on `/impeccable init` replacing the `PRODUCT.md` stub, then `DESIGN.md` |
+| 003 MCP server | Thin wrapper over 001's HTTP API |
+| 004 Multi-user management | Roles, user CRUD, per-collection permissions |
+
 ## Intended shape
 
-Not yet decided or validated. The pieces the spec has to place:
-
-- A Recoll layer that owns indexing and querying, per collection.
+- **One** Recoll index for all documents, not one per collection. Collection membership is per-file
+  metadata, not directory layout, so two files in the same directory can belong to different
+  collections.
+- A Python service owns Recoll through its native Python bindings (`from recoll import recoll`).
+- SQLite holds collections, membership, indexing jobs, users and sessions.
 - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
-- An MCP server exposing search (and likely collection listing) as tools.
-- A Docker image that contains Recoll itself, plus compose wiring for document volumes and index storage.
+- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
+
+## Verified Recoll mechanics
+
+Established empirically on 2026-09-18 in a throwaway `debian:trixie-slim` container running
+Recoll 1.43.0 / Xapian 1.4.29. Re-verify if the Recoll major version changes.
+
+- **Per-file metadata is indexable and queryable.** A `fields` file declaring
+  `[xattrtofields] collection = collection`, `[prefixes] collection = XYCOLL` and `[stored] collection =`
+  makes the `user.collection` extended attribute a queryable field (`collection:<value>`), returned on
+  each result. Recoll strips the `user.` prefix itself.
+- **Multi-membership works.** A space-separated xattr value puts one file in several collections. An
+  `OR` across collections returns it **once**, in a single coherently ranked result set — Xapian does
+  the ranking, so no application-side merging is needed.
+- **A plain incremental `recollindex` does not see xattr-only changes.** It compares mtime, and
+  `setfattr` does not change mtime; `touch` does not help either. `recollindex -i <file>` does pick the
+  change up, in roughly 20ms for one file. Retagging therefore costs one file, never a rebuild.
+- **The field value is tokenised and matched case-insensitively.** A tag of `cook-books` is matched by
+  `collection:cook` *and* `collection:books`; `TechBooks` is matched by `collection:techbooks`. Using
+  human-readable collection names as tags would let collections silently bleed into one another.
+- **An unscoped query returns every document, including untagged ones.** Recoll will not enforce
+  collection scoping; the API layer must.
 
 ## Open design questions
 
-The first spec must answer these. None are settled.
+Closed by the 001 brainstorm (2026-09-18) — see `docs/DECISIONS.md` for the reasoning:
 
-1. **Collection isolation.** Recoll's model is a config directory per index (`RECOLL_CONFDIR`).
-   Confirm against Recoll's own docs, then decide: one config dir per collection, and what owns the
-   mapping from collection name to config dir and document root.
-2. **Cross-collection search.** Allowed at all? If yes, merged ranking across separately-scored indexes
-   is not free — decide before building the UI around it.
-3. **Indexing trigger.** Cron inside the container, Recoll's real-time monitor, or API-triggered runs.
-   Affects whether the container needs a long-running process beyond the web server.
-4. **Stack.** Unchosen. See `AGENTS.md` Commands for what this host actually has installed.
-5. **Auth and exposure.** Whether the service authenticates at all, or assumes a reverse proxy does it.
-6. **Result presentation.** Recoll returns snippets and can extract/preview documents; decide how much
-   of that the API surfaces, since it drives both the UI and the MCP tool shape.
+1. ~~Collection isolation~~ — one index; membership is an opaque collection ID in a `user.collection`
+   xattr; human names live only in SQLite.
+2. ~~Cross-collection search~~ — allowed, but only when explicitly requested. Ranking is Xapian's.
+3. ~~Indexing trigger~~ — an in-process job queue, backed by SQLite.
+4. ~~Stack~~ — Python + FastAPI, managed with `uv`.
+5. ~~Auth and exposure~~ — real authentication in 001; user administration deferred to 004.
+6. ~~Result presentation~~ — metadata plus Recoll snippets; full text and file download are separate
+   endpoints.
+
+Still open, to be closed by the remaining sections of the 001 design:
+
+- Component boundaries and the SQLite schema.
+- API surface, pagination, and error semantics.
+- Test strategy, including how tests skip when the container is down.
 
 ## External dependencies
 
@@ -53,11 +90,15 @@ The first spec must answer these. None are settled.
 | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. |
 | Docker + compose | The only supported way to run the service. |
 | Document volumes | Nothing to index; collections resolve to empty. |
+| Extended-attribute support on the document volume | Collection membership. Without xattrs every document is untagged and every scoped search returns nothing — a silent failure, so it must be checked at startup. |
 
 ## Invariants
 
-To be established with the first code. Candidates the spec should confirm and then record here:
-a query never silently spans collections; indexed content is never written into the repository.
+- A query never silently spans collections: the scope is an explicit, required parameter, and the API
+  refuses a search that does not name one.
+- Collection identity in the index is an opaque single-token ID, never a user-supplied name, so
+  collections cannot bleed into one another and renaming never touches the index.
+- Indexed content is never written into the repository.
 
 ## Known rough edges
 

docs/DECISIONS.md

@@ -3,6 +3,131 @@
 Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back.
 One entry per decision a future maintainer would otherwise have to reverse-engineer.
 
+## 2026-09-18 — Search results carry metadata and snippets, not full text
+**Status:** accepted
+**Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the
+original file. The response shape drives both the web UI and the MCP tool.
+**Decision:** a search hit carries title, author, path, MIME type, size, date, score, its collections and
+Recoll's snippets. Extracted text and file download are separate endpoints.
+**Alternatives:** metadata only (a results list without context is markedly worse, and an MCP client
+would fetch every candidate to judge it, costing more overall); metadata plus full extracted text
+(responses become enormous for book-length PDFs and blow an agent's context window).
+**Consequences:** one search call is enough for an agent to judge relevance. The UI needs a second call
+to open a document, which is the right trade.
+
+---
+
+## 2026-09-18 — Authentication in 001, user administration in 004
+**Status:** accepted
+**Context:** the service needs real auth, but roles, user CRUD and per-collection permissions are a
+subsystem in their own right, and `PRODUCT.md` currently describes a single technical operator.
+**Decision:** 001 ships genuine authentication — argon2 password hashing, server-side sessions, an admin
+account seeded from the environment at first boot, every endpoint behind an auth dependency. User
+administration and permissions are deferred to spec 004.
+**Alternatives:** no auth in the service, trusting a reverse proxy (a misconfigured proxy or an exposed
+Docker port would leave the destructive endpoints naked); the full identity system in 001 (a large
+addition to an already-grown spec, building for users who do not exist yet).
+**Consequences:** 001 is safe to expose behind a proxy. 004 must extend the auth layer without
+reshaping endpoints, so the auth dependency is designed as a seam from the start.
+
+---
+
+## 2026-09-18 — Search scope is an explicit, required parameter
+**Status:** accepted
+**Context:** `docs/ARCHITECTURE.md` proposed the invariant that a query never silently spans
+collections. A Recoll query with no collection clause returns every document, including untagged ones.
+**Decision:** every search names one or more collections; omitting the parameter is an error, not an
+implicit "all". Multiple collections are OR-ed into one Xapian query so ranking stays coherent and each
+hit reports which collections it belongs to.
+**Alternatives:** single-collection-only searches (the UI and MCP would be built around a model they
+later outgrow); a searchable-by-default flag (reintroduces the implicit scope the invariant forbids);
+per-collection fan-out and application-side merging (scores from separate databases are not comparable,
+so the merged order would be strictly worse than letting Xapian rank).
+**Consequences:** the API layer enforces the invariant, because Recoll will not. Every search path needs
+a test proving an unscoped request is rejected.
+
+---
+
+## 2026-09-18 — Indexing runs as an in-process job queue
+**Status:** accepted
+**Context:** collections are created at runtime, so something must index a collection the moment it is
+created, and report progress back to the UI and to MCP clients.
+**Decision:** creating a collection, changing membership, or calling the reindex endpoint enqueues a job.
+A background worker in the same FastAPI process runs `recollindex` and records status, progress and
+errors in SQLite. One container process, no broker.
+**Alternatives:** `recollindex -m` real-time monitoring (one supervised process per collection, inotify
+watch limits, and a new collection still needs a first full index from somewhere — revisit as an opt-in
+mode later); cron-scheduled full runs (new collections sit empty until the next tick, and there is no
+way to force a refresh).
+**Consequences:** jobs die with the process, so the queue must recover interrupted runs at startup.
+Job state is queryable, so "indexing, 40%" is available to every consumer.
+
+---
+
+## 2026-09-18 — Collections are runtime-managed through the API
+**Status:** accepted
+**Context:** collections could be declared in a mounted config file or managed at runtime.
+**Decision:** collections are created, edited and deleted through the HTTP API, with SQLite as the store
+of record. The service creates and maintains the index state on demand.
+**Alternatives:** a declarative mounted `collections.yaml` (reproducible and git-friendly, but no
+management UI without adding a write path later); one collection per subdirectory by convention (forces
+the on-disk layout to match the collection layout, which the per-file membership requirement rules out).
+**Consequences:** 001 grows a persistence layer, validation and migrations. `AGENTS.md` makes deleting
+or rebuilding an index an ask-first operation, so the delete path needs an explicit guard rather than a
+bare `DELETE`.
+
+---
+
+## 2026-09-18 — One index; collection membership is an opaque per-file tag
+**Status:** accepted
+**Context:** a collection cannot be "a directory", because files sharing a directory must be able to
+belong to different collections. Verified in a throwaway Recoll 1.43.0 container — details and the
+exact config in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics".
+**Decision:** index every document once into a single Recoll index. Membership is a `user.collection`
+extended attribute mapped to a queryable Recoll field. The tag is an **opaque single-token ID**; the
+human-readable name lives only in SQLite.
+**Alternatives:** one index per collection with rule-based selection (a file in two collections is
+stored and indexed twice and can be returned twice, and hand-picking files in a shared directory is
+awkward); an explicit file list driving indexing (does not map onto Recoll's topdirs-walking model).
+**Why opaque IDs:** Recoll tokenises the field value and matches case-insensitively, so a tag of
+`cook-books` is also matched by `collection:cook` and `collection:books`, and `TechBooks` by
+`collection:techbooks`. Names-as-tags would let collections silently bleed into each other — exactly
+what the no-silent-spanning invariant forbids. Opaque IDs tested clean, with no partial matches.
+**Consequences:** collection names need no character restrictions and renaming never touches the index.
+Documents must sit on a volume supporting extended attributes, checked at startup — without it every
+scoped search silently returns nothing. Retagging costs `recollindex -i <file>`, not a rebuild, because
+a plain incremental run does not notice xattr-only changes.
+
+---
+
+## 2026-09-18 — Stack: Python + FastAPI, managed with uv
+**Status:** accepted
+**Context:** the stack was unchosen and is an ask-first decision. Recoll ships first-class Python
+bindings; every other language must drive `recollq` as a subprocess and parse its output.
+**Decision:** Python + FastAPI for the engine and HTTP API, with `uv` as the package manager. `uv`
+0.12.16 was installed on this host as part of this decision, and `AGENTS.md` updated in the same commit.
+**Alternatives:** TypeScript on Node/Bun or C# on ASP.NET Core (one language across UI, API and MCP, but
+both make a fragile output parser load-bearing); a split Python engine behind a TypeScript API (two
+runtimes and an extra hop inside one container).
+**Consequences:** structured results with no parsing layer, and OpenAPI for free, which specs 002 and 003
+can generate clients from. The web UI will likely be a second language; that boundary is the HTTP API.
+
+---
+
+## 2026-09-18 — The product is four specs, built in order
+**Status:** accepted
+**Context:** "a web UI and an MCP server over Recoll" is four subsystems. One spec covering all of them
+would be too large to plan or review, and `PRODUCT.md` is still an uninterviewed stub, so any UI spec
+written now would design against fiction.
+**Decision:** 001 search engine + API, 002 web UI, 003 MCP server, 004 multi-user management. Each gets
+its own spec, plan and review cycle. The table lives in `docs/ARCHITECTURE.md`.
+**Alternatives:** one thin vertical slice through all four (would lock in a stack and a UI before
+`PRODUCT.md` is real); engine plus MCP together (defensible, but it widens 001, which has already grown).
+**Consequences:** 002 is blocked on `/impeccable init`. 001's HTTP API is the contract 002 and 003 are
+both built against, so it is the piece that must be got right.
+
+---
+
 ## 2026-09-18 — Override the superpowers artefact paths
 **Status:** accepted
 **Context:** superpowers 6.3.0 writes specs to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`

docs/TODO.md

@@ -4,17 +4,22 @@ Durable backlog. Agents append here instead of leaving `TODO` comments in source
 Small, specific, independently actionable. Remove only when actually done.
 
 ## Now
-- [ ] Answer the open questions in the bootstrap report, then fill the `Commands` section of `AGENTS.md`
-      and delete its `TODO(maintainer)` line.
-- [ ] Brainstorm -> spec `docs/specs/001-*.md` covering the open design questions in
-      `docs/ARCHITECTURE.md`, starting with collection isolation and the stack choice.
+- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in
+      `docs/DECISIONS.md`.
+- [ ] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics,
+      test strategy. Then write `docs/specs/001-search-engine-api.md`.
+- [ ] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in
+      the same commit that scaffolds the app, with every command actually run.
 
 ## Next
-- [ ] Scaffold the chosen stack; verify install/dev/test/lint/typecheck/build commands by running them,
+- [ ] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them,
       then record the verified ones in `AGENTS.md`.
 - [ ] Dockerfile + compose: Recoll in the image, volumes for documents and indexes.
+- [ ] Startup check: refuse to start, loudly, if the document volume does not support extended
+      attributes — without them every scoped search silently returns nothing.
+- [ ] Recover interrupted indexing jobs at startup; they do not survive a process restart on their own.
 - [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial
-      brief, not an interview.
+      brief, not an interview. Blocks spec 002.
 
 ## Someday
 - [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/`