Archived
258 lines
17 KiB
Markdown
258 lines
17 KiB
Markdown
# Decisions
|
|
|
|
Append-only, newest first. Never edit or delete an entry — supersede it with a new one that links back.
|
|
One entry per decision a future maintainer would otherwise have to reverse-engineer.
|
|
|
|
## 2026-09-18 — Deployment supports a configured URL subpath
|
|
**Status:** accepted
|
|
**Context:** the service may sit behind an existing reverse proxy under a path such as `/recoll`, not at
|
|
the origin root. Hard-coded root URLs break redirects, cookies, OpenAPI and document links there.
|
|
**Decision:** one configured external mount prefix drives routing, generated URLs and cookie scope.
|
|
Forwarded scheme, host and prefix values are honored only from explicitly trusted proxies.
|
|
**Alternatives:** root-only hosting (needlessly constrains deployment); trust every forwarded header
|
|
(lets a direct client forge external URLs and cookie paths).
|
|
**Consequences:** API and end-to-end tests run at both `/` and a non-root prefix. Proxy configuration is
|
|
part of deployment documentation, and all URL generation goes through one boundary.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Search result fields are caller-selected
|
|
**Status:** accepted
|
|
**Supersedes:** the fixed result shape in "Search results carry metadata and snippets, not full text"
|
|
(below, same date). The metadata-plus-snippets shape remains the default, not the only representation.
|
|
**Context:** browser and MCP consumers have different response-size needs, and users need to choose
|
|
whether results include metadata, snippets, extracted text or retrieval links.
|
|
**Decision:** `document_id` is unconditional; callers select all other result fields. Inline extracted
|
|
text is opt-in, bounded and marked when truncated. Authenticated endpoints return full extracted text
|
|
and stream the canonical original separately.
|
|
**Alternatives:** one fixed rich shape (wastes bandwidth and agent context); metadata-only search with a
|
|
follow-up per hit (too many round trips); unbounded full text inline (a book can dominate the response).
|
|
**Consequences:** field projection is part of the engine/API contract. Generated retrieval links honor
|
|
the configured mount prefix, and original-file responses require range and safe-disposition handling.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Directory membership has normalized additive provenance
|
|
**Status:** accepted
|
|
**Context:** users need both one-time directory expansion and persistent directory membership, including
|
|
the ability to omit metadata subdirectories such as `.calibre`. Overlapping sources must not make
|
|
removal order-dependent.
|
|
**Decision:** direct file additions and snapshot expansions are durable direct membership. Persistent
|
|
directory rules materialize their own contributed documents and hold exact recursive exclusions scoped
|
|
to the rule. Per-file removal creates an override against rule-derived membership; an explicit direct
|
|
addition still wins. Persistent rules reconcile at startup and explicit reindex, not through watching.
|
|
**Alternatives:** one polymorphic source ledger (weaker relational constraints); recompute every rule on
|
|
demand (slow and nondeterministic); persistent rules without provenance (removing one rule can erase
|
|
membership contributed by another).
|
|
**Consequences:** the schema has separate direct, rule-derived and exclusion tables. Directory symlinks
|
|
are never followed; file symlinks are accepted only inside the document root. Jobs default to
|
|
best-effort, while strict jobs scan first and apply no membership changes if any file fails.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Document identity is a content hash, not a path
|
|
**Status:** accepted
|
|
**Context:** the document set may contain files that share a name but differ in content, and files that
|
|
differ in name but are byte-identical. Keying membership on path merges neither and distinguishes
|
|
neither correctly.
|
|
**Decision:** a document is identified by the SHA-256 of its bytes. `documents` is keyed by that hash,
|
|
`document_paths` maps many paths to one document, and membership attaches to the document. Each document
|
|
has one **canonical path**, and only that path is reported to Recoll as belonging to any collection.
|
|
**Alternatives:** path as identity (byte-identical copies appear as separate hits for the same content);
|
|
relying on Recoll to deduplicate (it cannot — `md5` is empty and `sig` is only size + mtime).
|
|
**Consequences:** duplicate copies are indexed by Recoll but belong to no collection, so the required
|
|
collection scope makes them unreachable — verified: a scoped search over a tree with one duplicated file
|
|
returned it once. Deleting a canonical path must promote another path and force a reindex, or the
|
|
document silently leaves every collection.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Membership is supplied at index time; documents are never written to
|
|
**Status:** accepted
|
|
**Supersedes:** "One index; collection membership is an opaque per-file tag" (below, same date), which
|
|
carried membership in a `user.collection` extended attribute. That entry's xattr mechanism must not be
|
|
implemented.
|
|
**Context:** the service must work with **read-only** access to the documents it indexes. Writing an
|
|
xattr requires write access, so the superseded design fails outright on a `:ro` mount — verified:
|
|
`setfattr` returns "Read-only file system".
|
|
**Decision:** membership reaches Recoll through `metadatacmds` in `recoll.conf`
|
|
(`metadatacmds = ; collection = <helper> %f`), a documented Recoll facility for external metadata.
|
|
Recoll runs the helper per file at index time; the helper answers from our own store. Everything the
|
|
superseded entry established about opaque IDs, multi-membership and single-ranked-set OR queries still
|
|
holds — only the transport changes.
|
|
**Alternatives:** filtering Recoll's results against SQLite after the query (breaks paging and total
|
|
counts); one index per collection (cannot express two files in one directory belonging to different
|
|
collections); requiring a writable document volume (contradicts the requirement).
|
|
**Consequences:** SQLite becomes the *only* store of membership rather than the authoritative copy of
|
|
two, so drift between store and index is now impossible and no reconcile path is needed. Extended
|
|
attributes are no longer a deployment requirement. A membership change needs
|
|
`recollindex -i -Z <path>` — verified that neither a plain incremental run nor `recollindex -i` notices
|
|
it, because Recoll's change signature is size + mtime and a membership change touches neither.
|
|
The helper is spawned once per file per index run, so it must start fast: over 400 files, indexing took
|
|
0.09s with no helper, 0.13s with a shell+awk helper and 0.82s with a `python3` helper. The service
|
|
exports a flat map from SQLite for a small non-Python helper to read.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Search results carry metadata and snippets, not full text
|
|
**Status:** accepted
|
|
**Context:** Recoll can return metadata, keyword-in-context snippets, extracted plain text and the
|
|
original file. The response shape drives both the web UI and the MCP tool.
|
|
**Decision:** a search hit carries title, author, path, MIME type, size, date, score, its collections and
|
|
Recoll's snippets. Extracted text and file download are separate endpoints.
|
|
**Alternatives:** metadata only (a results list without context is markedly worse, and an MCP client
|
|
would fetch every candidate to judge it, costing more overall); metadata plus full extracted text
|
|
(responses become enormous for book-length PDFs and blow an agent's context window).
|
|
**Consequences:** one search call is enough for an agent to judge relevance. The UI needs a second call
|
|
to open a document, which is the right trade.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Authentication in 001, user administration in 004
|
|
**Status:** accepted
|
|
**Context:** the service needs real auth, but roles, user CRUD and per-collection permissions are a
|
|
subsystem in their own right, and `PRODUCT.md` currently describes a single technical operator.
|
|
**Decision:** 001 ships genuine authentication — argon2 password hashing, server-side sessions, an admin
|
|
account seeded from the environment at first boot, every endpoint behind an auth dependency. User
|
|
administration and permissions are deferred to spec 004.
|
|
**Alternatives:** no auth in the service, trusting a reverse proxy (a misconfigured proxy or an exposed
|
|
Docker port would leave the destructive endpoints naked); the full identity system in 001 (a large
|
|
addition to an already-grown spec, building for users who do not exist yet).
|
|
**Consequences:** 001 is safe to expose behind a proxy. 004 must extend the auth layer without
|
|
reshaping endpoints, so the auth dependency is designed as a seam from the start.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Search scope is an explicit, required parameter
|
|
**Status:** accepted
|
|
**Context:** `docs/ARCHITECTURE.md` proposed the invariant that a query never silently spans
|
|
collections. A Recoll query with no collection clause returns every document, including untagged ones.
|
|
**Decision:** every search names one or more collections; omitting the parameter is an error, not an
|
|
implicit "all". Multiple collections are OR-ed into one Xapian query so ranking stays coherent and each
|
|
hit reports which collections it belongs to.
|
|
**Alternatives:** single-collection-only searches (the UI and MCP would be built around a model they
|
|
later outgrow); a searchable-by-default flag (reintroduces the implicit scope the invariant forbids);
|
|
per-collection fan-out and application-side merging (scores from separate databases are not comparable,
|
|
so the merged order would be strictly worse than letting Xapian rank).
|
|
**Consequences:** the API layer enforces the invariant, because Recoll will not. Every search path needs
|
|
a test proving an unscoped request is rejected.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Indexing runs as an in-process job queue
|
|
**Status:** accepted
|
|
**Context:** collections are created at runtime, so something must index a collection the moment it is
|
|
created, and report progress back to the UI and to MCP clients.
|
|
**Decision:** creating a collection, changing membership, or calling the reindex endpoint enqueues a job.
|
|
A background worker in the same FastAPI process runs `recollindex` and records status, progress and
|
|
errors in SQLite. One container process, no broker.
|
|
**Alternatives:** `recollindex -m` real-time monitoring (one supervised process per collection, inotify
|
|
watch limits, and a new collection still needs a first full index from somewhere — revisit as an opt-in
|
|
mode later); cron-scheduled full runs (new collections sit empty until the next tick, and there is no
|
|
way to force a refresh).
|
|
**Consequences:** jobs die with the process, so the queue must recover interrupted runs at startup.
|
|
Job state is queryable, so "indexing, 40%" is available to every consumer.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Collections are runtime-managed through the API
|
|
**Status:** accepted
|
|
**Context:** collections could be declared in a mounted config file or managed at runtime.
|
|
**Decision:** collections are created, edited and deleted through the HTTP API, with SQLite as the store
|
|
of record. The service creates and maintains the index state on demand.
|
|
**Alternatives:** a declarative mounted `collections.yaml` (reproducible and git-friendly, but no
|
|
management UI without adding a write path later); one collection per subdirectory by convention (forces
|
|
the on-disk layout to match the collection layout, which the per-file membership requirement rules out).
|
|
**Consequences:** 001 grows a persistence layer, validation and migrations. `AGENTS.md` makes deleting
|
|
or rebuilding an index an ask-first operation, so the delete path needs an explicit guard rather than a
|
|
bare `DELETE`.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — One index; collection membership is an opaque per-file tag
|
|
**Status:** accepted
|
|
**Context:** a collection cannot be "a directory", because files sharing a directory must be able to
|
|
belong to different collections. Verified in a throwaway Recoll 1.43.0 container — details and the
|
|
exact config in `docs/ARCHITECTURE.md` under "Verified Recoll mechanics".
|
|
**Decision:** index every document once into a single Recoll index. Membership is a `user.collection`
|
|
extended attribute mapped to a queryable Recoll field. The tag is an **opaque single-token ID**; the
|
|
human-readable name lives only in SQLite.
|
|
**Alternatives:** one index per collection with rule-based selection (a file in two collections is
|
|
stored and indexed twice and can be returned twice, and hand-picking files in a shared directory is
|
|
awkward); an explicit file list driving indexing (does not map onto Recoll's topdirs-walking model).
|
|
**Why opaque IDs:** Recoll tokenises the field value and matches case-insensitively, so a tag of
|
|
`cook-books` is also matched by `collection:cook` and `collection:books`, and `TechBooks` by
|
|
`collection:techbooks`. Names-as-tags would let collections silently bleed into each other — exactly
|
|
what the no-silent-spanning invariant forbids. Opaque IDs tested clean, with no partial matches.
|
|
**Consequences:** collection names need no character restrictions and renaming never touches the index.
|
|
Documents must sit on a volume supporting extended attributes, checked at startup — without it every
|
|
scoped search silently returns nothing. Retagging costs `recollindex -i <file>`, not a rebuild, because
|
|
a plain incremental run does not notice xattr-only changes.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Stack: Python + FastAPI, managed with uv
|
|
**Status:** accepted
|
|
**Context:** the stack was unchosen and is an ask-first decision. Recoll ships first-class Python
|
|
bindings; every other language must drive `recollq` as a subprocess and parse its output.
|
|
**Decision:** Python + FastAPI for the engine and HTTP API, with `uv` as the package manager. `uv`
|
|
0.12.16 was installed on this host as part of this decision, and `AGENTS.md` updated in the same commit.
|
|
**Alternatives:** TypeScript on Node/Bun or C# on ASP.NET Core (one language across UI, API and MCP, but
|
|
both make a fragile output parser load-bearing); a split Python engine behind a TypeScript API (two
|
|
runtimes and an extra hop inside one container).
|
|
**Consequences:** structured results with no parsing layer, and OpenAPI for free, which specs 002 and 003
|
|
can generate clients from. The web UI will likely be a second language; that boundary is the HTTP API.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — The product is four specs, built in order
|
|
**Status:** accepted
|
|
**Context:** "a web UI and an MCP server over Recoll" is four subsystems. One spec covering all of them
|
|
would be too large to plan or review, and `PRODUCT.md` is still an uninterviewed stub, so any UI spec
|
|
written now would design against fiction.
|
|
**Decision:** 001 search engine + API, 002 web UI, 003 MCP server, 004 multi-user management. Each gets
|
|
its own spec, plan and review cycle. The table lives in `docs/ARCHITECTURE.md`.
|
|
**Alternatives:** one thin vertical slice through all four (would lock in a stack and a UI before
|
|
`PRODUCT.md` is real); engine plus MCP together (defensible, but it widens 001, which has already grown).
|
|
**Consequences:** 002 is blocked on `/impeccable init`. 001's HTTP API is the contract 002 and 003 are
|
|
both built against, so it is the piece that must be got right.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Override the superpowers artefact paths
|
|
**Status:** accepted
|
|
**Context:** superpowers 6.3.0 writes specs to `docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md`
|
|
and plans to `docs/superpowers/plans/YYYY-MM-DD-<feature>.md` (verified in the installed skill files).
|
|
That nests a second docs tree and dates artefacts that we want numbered and paired.
|
|
**Decision:** specs -> `docs/specs/NNN-slug.md`, plans -> `docs/plans/NNN-slug.md`, plan number matching
|
|
its spec number. Recorded in `AGENTS.md` so every session picks it up.
|
|
**Alternatives:** accept the defaults (two docs trees, no spec/plan pairing); symlink (fragile on Windows
|
|
and in containers).
|
|
**Consequences:** the override must be restated if `AGENTS.md` is ever trimmed; a superpowers upgrade
|
|
that changes its defaults needs this entry re-checked.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — Adopt AGENTS.md as single source of truth
|
|
**Status:** accepted
|
|
**Context:** The repository was empty — no commits, no stack, no agent context. Agents had nothing
|
|
durable to read and would re-explore from scratch every session.
|
|
**Decision:** `AGENTS.md` is canonical; `CLAUDE.md` and `GEMINI.md` are thin pointers with no rules of
|
|
their own. Enforceable things live in `.claude/settings.json` and `.gitignore` rather than in prose.
|
|
**Alternatives:** per-tool rule files (drift and synchronisation debt); no rules file (measurably more
|
|
steps and tokens per task).
|
|
**Consequences:** rule changes happen in one file; the pointer files must stay thin; `AGENTS.md` is an
|
|
executable-context and prompt-injection surface, so changes to it get reviewed like CI config.
|
|
|
|
---
|
|
|
|
## 2026-09-18 — No DESIGN.md until the first UI exists
|
|
**Status:** accepted
|
|
**Context:** The `impeccable` skill reads `PRODUCT.md` and `DESIGN.md`. `PRODUCT.md` could be written
|
|
from the stated product intent; `DESIGN.md` could not — there is no UI, no tokens, no visual world.
|
|
**Decision:** write `PRODUCT.md` now; leave `DESIGN.md` absent. The first UI build produces it
|
|
(`/impeccable document` derives it from shipped code).
|
|
**Alternatives:** a placeholder `DESIGN.md` — rejected, it would assert a visual system that does not
|
|
exist and the skill would design against fiction.
|
|
**Consequences:** the first UI task must end by writing `DESIGN.md`, not skip it.
|