Brainstorm, verified Recoll spike findings, decision log and draft spec 001.
No application code; spec 001 is status: draft and awaiting sign-off.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Writes docs/specs/001-search-engine-api.md covering the search engine and HTTP
API: collections managed at runtime, per-file membership supplied to Recoll at
index time, content-hash document identity with canonical paths, an in-process
indexing job queue, scoped search, and authentication. Twenty-four numbered
requirements, each with an acceptance criterion, plus the constraints the design
must hold: read-only documents, per-file membership, content-hash identity, and
no silently unscoped search.
The spec is status: draft. It has had its self-review pass but has not been
signed off, so no plan and no code may be written against it yet.
Self-review changed three things. Deleting a collection now states that it
enqueues reindexing of the affected canonical paths, because the previous
wording claimed no index work while leaving the deleted id in the index. The
search scope parameter is specified as collection ids rather than names, since
names are mutable and would break saved queries on rename. And the reported
match count is documented as approximate, because Xapian estimates it for large
result sets.
Three known gaps are recorded rather than left implicit: there is no way to read
a matched document until spec 002, the indexing overhead figures come from small
text files rather than real PDFs, and every authenticated user sees every
collection until spec 004.
docs/TODO.md gains a "Resume here" section stating where the workflow stands,
what the next action is, and the traps a resuming agent would otherwise hit --
chiefly that the extended-attribute membership design was superseded mid-session
and survives in the append-only decision log.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The service must work with read-only access to the documents it indexes.
Writing a user.collection extended attribute needs write access, so the
previous membership design fails on a :ro mount; setfattr returns "Read-only
file system". Verified in the spike container.
Membership now reaches Recoll through metadatacmds in recoll.conf, a documented
facility for external metadata: Recoll runs a helper per file at index time and
the helper answers from our own store. Verified end to end against a read-only
bind mount. Everything previously established about opaque collection IDs,
multi-membership and single-ranked-set OR queries survives unchanged, because
only the transport differs. SQLite becomes the only store of membership rather
than the authoritative one of two, so drift is impossible and the reconcile path
is gone. Extended attributes are no longer a deployment requirement.
Document identity becomes the SHA-256 of file content. Files sharing a name but
differing in content are separate documents; one file reachable by several paths
is a single document with one canonical path, and only that path is reported as
belonging to a collection. Recoll cannot help here: md5 is empty and sig is only
size plus mtime.
Two measured constraints are recorded. A membership change is invisible to both
a plain incremental run and recollindex -i, because the change signature is size
plus mtime; recollindex -i -Z on the affected path picks it up in about 15ms.
And the helper is spawned once per file per index run, so it must start fast: a
400-file index took 0.09s with no helper, 0.13s with shell and awk, and 0.82s
with python3.
The superseded entry in docs/DECISIONS.md is left intact, per that file's
append-only rule, and the new entry links back to it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records the outcome of the 001 brainstorm and the Recoll spike that backed it.
Decisions: Python + FastAPI managed with uv; the product split into four specs
built in order; collections managed at runtime through the API with SQLite as
the store of record; indexing as an in-process job queue; search scope as an
explicit required parameter; results carrying metadata and snippets; real
authentication in 001 with user administration deferred to 004.
The load-bearing one is collection membership. A collection cannot be a
directory, because files sharing a directory must be able to belong to
different collections, so every document is indexed once into a single Recoll
index and membership is a user.collection extended attribute mapped to a
queryable field. Verified against Recoll 1.43.0 in a throwaway container: the
tag must be an opaque single-token ID rather than the collection name, because
Recoll tokenises the field value and matches case-insensitively, so a tag of
cook-books is also matched by collection:cook and collection:books. Names as
tags would let collections silently bleed into each other. Opaque IDs also make
renaming free, since the index is never touched.
Two operational consequences are recorded in docs/TODO.md: documents must live
on a volume with extended-attribute support, checked loudly at startup, and a
plain incremental recollindex does not notice xattr-only changes, so retagging
must go through recollindex -i on the affected file.
uv 0.12.16 was installed on this host as part of the stack decision; AGENTS.md
is corrected in the same commit, since it previously listed uv as absent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Empty repository: no commits, no stack. Establishes AGENTS.md as the single
source of truth, thin CLAUDE.md/GEMINI.md pointers, permission config, and the
docs/ progressive-disclosure tree.
Commands section is deliberately unfilled — no stack exists to verify commands
against.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
