diff --git a/PRODUCT.md b/PRODUCT.md index 3681287..87285a8 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -12,9 +12,9 @@ web ## Stack -The backend stack was confirmed during the 001 design: Python + FastAPI, managed with `uv`, because -Recoll provides native Python bindings. The frontend stack remains undecided. No application has been -scaffolded yet. See `docs/DECISIONS.md` for the decision and `AGENTS.md` for installed toolchains. +The backend stack was confirmed during the 001 design and implemented: Python + FastAPI, managed with +`uv`, because Recoll provides native Python bindings. The frontend stack remains undecided, and no web +UI has been built. See `docs/DECISIONS.md` for the decision and `AGENTS.md` for installed toolchains. ## Users @@ -50,8 +50,9 @@ Confirmed from the brief: - Web UI, served from a Docker deployment. - MCP server exposing the same search capability. -Explicitly undecided: authentication, cross-collection search, indexing schedule, result preview depth, -supported document types beyond Recoll's own. See `docs/ARCHITECTURE.md` for the full list. +Spec 001 settled authentication, cross-collection search, indexing triggers, and API result fields. +UI presentation and filter coverage beyond the shipped container remain undecided. See +`docs/ARCHITECTURE.md` for the implemented shape. ## Brand Commitments diff --git a/docs/TODO.md b/docs/TODO.md index b0993bd..7f8ee7f 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1,96 +1,28 @@ # TODO -Durable backlog. Agents append here instead of leaving `TODO` comments in source. -Small, specific, independently actionable. Remove only when actually done. +Open work only. Record completed work in its spec, plan, commits, and `docs/DECISIONS.md`; do not +leave `TODO` comments in source. -## Resume here +Spec 001 is implemented on `main`. The next product step is spec 002 (web UI). Its first step is the +`/impeccable init` interview to replace the `PRODUCT.md` stub; the first UI build then creates +`DESIGN.md`. Follow the spec → plan → TDD → review workflow in `AGENTS.md`. -Last touched 2026-09-28. Spec 001 implementation has completed its review fixes and acceptance matrix. +## Product and engineering backlog -Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review): - -- Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with - reasoning in `docs/DECISIONS.md`. -- Spec `docs/specs/001-search-engine-api.md`: **approved 2026-09-18.** It includes directory rules, - selectable search fields, subpath hosting and the packaging seam for deferred desktop spec 005. -- Plan `docs/plans/001-search-engine-api.md`: **approved 2026-09-18.** Its listed Python runtime and - development dependencies are approved. Execution follows the plan task by task. -- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–14 add discovery, content-hash - identity, additive membership, durable jobs, the atomic membership map, the engine boundary, - session authentication, the HTTP shell, membership/rule/job routes, scoped search and document - retrieval, Docker packaging and acceptance coverage. Their checkboxes are current in the plan. - The first code review found replay, reconciliation, and download races. A follow-up review found - generated documentation exposed without a session, incomplete unfiltered reindexing, stale tags - after an earlier duplicate became canonical, and lost membership on best-effort rule resubmission. - These are fixed on `feat/001-document-discovery`; branch handoff remains. - -The next action is branch integration by the maintainer's choice. Do not repeat Tasks 1–14. The review -fixes include idempotent rule creation, persisted replay paths, atomic collection cleanup enqueue, -unreadable-file preservation, departed-path reindexing, directory-rule pagination, recovery request -validation, descriptor-pinned downloads, authenticated API documentation, complete unfiltered -reindexing, canonical-path retagging, and best-effort rule resubmission. The generated real-container -lifecycle test covers the major flows through Compose. On 2026-09-28, the host suite passed with 160 -passed and 8 container-dependent skips, and the required-container suite passed all 9 checks. Lint, -typecheck and build passed. The isolated acceptance Compose project was stopped with -`docker compose down` and its named volumes were preserved. - -Things a resuming agent will otherwise get wrong: - -- The design carried collection membership in a `user.collection` extended attribute for part of this - session. **That is superseded** — the documents are read-only. `docs/DECISIONS.md` has both entries, - newest first; the older one is left intact by that file's append-only rule, so read the newer one. -- The Recoll facts in `docs/ARCHITECTURE.md` -> "Verified Recoll mechanics" were measured in a container, - not recalled. Trust them over intuition, and re-verify only if Recoll's major version moves. -- The throwaway spike container is not part of the repo and does not survive the session. Everything - worth keeping from it is already in `docs/`. -- `uv` was installed on this host during this session and `AGENTS.md` was corrected accordingly. Task 1 - replaced its command stub in the scaffold commit after running every listed command. - -## Now -- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in - `docs/DECISIONS.md`. -- [x] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics, - test strategy. Then write `docs/specs/001-search-engine-api.md`. -- [x] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in - the same commit that scaffolds the app, with every command actually run. - -## Next -- [ ] Verify the Recoll adapter against a generated corpus in the Compose container, including - exact-path text retrieval and whether the Python API exposes a native numeric relevance score. -- [x] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them, - then record the verified ones in `AGENTS.md`. -- [x] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for - the index and SQLite. -- [x] Write the `metadatacmds` membership helper as a fast-starting non-Python program, fed by a flat - map exported from SQLite. A `python3` helper made a 400-file index 9x slower; Recoll spawns it - once per file per index run. -- [ ] Re-measure indexing overhead against real documents (PDF, EPUB) rather than small text files — - extraction cost should dwarf the helper, but that is an assumption until measured. -- [x] Promote a new canonical path and force a reindex when a canonical path disappears, or the - document silently leaves every collection it is in. -- [x] Recover interrupted indexing jobs at startup; Task 6 requeues valid running jobs and fails - unreadable requests. -- [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial - brief, not an interview. Blocks spec 002. - -## Someday -- [ ] After delivery 004, write spec 005 for a cross-platform installable desktop distribution. Start - from a Tauri-style shell that reuses the web UI and bundles the API and Recoll as sidecars; decide - the actual framework, licensing, signing, updates and per-platform filter coverage in that spec. -- [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/` - paths exist, and fail if `AGENTS.md` exceeds 150 lines. +- [ ] Decide whether a persistent directory rule may target the document root. Current sources require + a named child directory. +- [ ] Investigate whether Recoll's Python binding exposes a native numeric relevance score. The API + currently returns a rank-based proxy; exact-path text retrieval was verified in the container. +- [ ] Measure membership-helper indexing overhead on representative PDF and EPUB documents. Current + measurements cover small text files only. +- [ ] After spec 004, design spec 005 for an installable desktop distribution, including framework, + Recoll packaging and licensing, signing, updates, and filter coverage. +- [ ] Consider a Gitea Actions check for required agent/docs files and the `AGENTS.md` length limit. ## Known issues -- The API test suite reports deprecation warnings from the installed FastAPI/Starlette TestClient - dependency. Replace its client integration when the dependency stack supports the successor API. -- Directory sources currently require a named child of the document root; decide whether the root - itself should be accepted as a persistent rule before finishing membership endpoints. -- Task 3's repository ID test checks the generated collection ID prefix and length rather than asserting - the full `^c[0-9a-f]{10}$` regex. The implementation uses `secrets.token_hex(5)` and is correct; add - the stronger regression assertion when work resumes. -## Recurring -- [ ] Add an `AGENTS.md` rule only in response to a real, observed agent failure — one line, ideally with - a code example. Never pre-empt hypothetical mistakes. -- [ ] Update `AGENTS.md` in the same commit as the convention it describes. A stale rules file misleads. -- [ ] Prune quarterly: delete rules whose convention changed or whose mistake no longer occurs. +- The API tests report deprecation warnings from FastAPI/Starlette's TestClient dependencies. Update + the client integration when a supported replacement is available. +- The repository ID test checks the collection ID prefix and length, rather than the full + `^c[0-9a-f]{10}$` pattern. The implementation uses `secrets.token_hex(5)`; strengthen this + regression assertion when touching that test. diff --git a/docs/plans/001-search-engine-api.md b/docs/plans/001-search-engine-api.md index 2e11b94..480d89b 100644 --- a/docs/plans/001-search-engine-api.md +++ b/docs/plans/001-search-engine-api.md @@ -2,7 +2,7 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Status:** approved 2026-09-18 +**Status:** done 2026-09-28 (approved 2026-09-18) **Goal:** Build the authenticated, collection-scoped Recoll service defined by spec 001, including durable membership/indexing jobs, document retrieval, and its Docker deployment. diff --git a/docs/specs/001-search-engine-api.md b/docs/specs/001-search-engine-api.md index bfe5e20..0dffbfa 100644 --- a/docs/specs/001-search-engine-api.md +++ b/docs/specs/001-search-engine-api.md @@ -1,5 +1,5 @@ --- -status: approved +status: implemented plan: docs/plans/001-search-engine-api.md supersedes: none ---