recoll-modern/docs/TODO.md

# TODO

Durable backlog. Agents append here instead of leaving `TODO` comments in source.
Small, specific, independently actionable. Remove only when actually done.

## Resume here

Last touched 2026-09-28. Spec 001 implementation has completed its review fixes and acceptance matrix.

Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review):

- Brainstorm for spec 001: **done.** All six design questions in `docs/ARCHITECTURE.md` are closed, with
  reasoning in `docs/DECISIONS.md`.
- Spec `docs/specs/001-search-engine-api.md`: **approved 2026-09-18.** It includes directory rules,
  selectable search fields, subpath hosting and the packaging seam for deferred desktop spec 005.
- Plan `docs/plans/001-search-engine-api.md`: **approved 2026-09-18.** Its listed Python runtime and
  development dependencies are approved. Execution follows the plan task by task.
- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–14 add discovery, content-hash
  identity, additive membership, durable jobs, the atomic membership map, the engine boundary,
  session authentication, the HTTP shell, membership/rule/job routes, scoped search and document
  retrieval, Docker packaging and acceptance coverage. Their checkboxes are current in the plan.
  The first code review found replay, reconciliation, and download races. A follow-up review found
  generated documentation exposed without a session, incomplete unfiltered reindexing, stale tags
  after an earlier duplicate became canonical, and lost membership on best-effort rule resubmission.
  These are fixed on `feat/001-document-discovery`; branch handoff remains.

The next action is branch integration by the maintainer's choice. Do not repeat Tasks 1–14. The review
fixes include idempotent rule creation, persisted replay paths, atomic collection cleanup enqueue,
unreadable-file preservation, departed-path reindexing, directory-rule pagination, recovery request
validation, descriptor-pinned downloads, authenticated API documentation, complete unfiltered
reindexing, canonical-path retagging, and best-effort rule resubmission. The generated real-container
lifecycle test covers the major flows through Compose. On 2026-09-28, the host suite passed with 160
passed and 8 container-dependent skips, and the required-container suite passed all 9 checks. Lint,
typecheck and build passed. The isolated acceptance Compose project was stopped with
`docker compose down` and its named volumes were preserved.

Things a resuming agent will otherwise get wrong:

- The design carried collection membership in a `user.collection` extended attribute for part of this
  session. **That is superseded** — the documents are read-only. `docs/DECISIONS.md` has both entries,
  newest first; the older one is left intact by that file's append-only rule, so read the newer one.
- The Recoll facts in `docs/ARCHITECTURE.md` -> "Verified Recoll mechanics" were measured in a container,
  not recalled. Trust them over intuition, and re-verify only if Recoll's major version moves.
- The throwaway spike container is not part of the repo and does not survive the session. Everything
  worth keeping from it is already in `docs/`.
- `uv` was installed on this host during this session and `AGENTS.md` was corrected accordingly. Task 1
  replaced its command stub in the scaffold commit after running every listed command.

## Now
- [x] Answer the six open design questions in `docs/ARCHITECTURE.md` — closed 2026-09-18, reasoning in
      `docs/DECISIONS.md`.
- [x] Finish the 001 design: component boundaries and SQLite schema, API surface and error semantics,
      test strategy. Then write `docs/specs/001-search-engine-api.md`.
- [x] Fill the `Commands` section of `AGENTS.md` and delete its `TODO(maintainer)` line — must happen in
      the same commit that scaffolds the app, with every command actually run.

## Next
- [ ] Verify the Recoll adapter against a generated corpus in the Compose container, including
      exact-path text retrieval and whether the Python API exposes a native numeric relevance score.
- [x] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them,
      then record the verified ones in `AGENTS.md`.
- [x] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for
      the index and SQLite.
- [x] Write the `metadatacmds` membership helper as a fast-starting non-Python program, fed by a flat
      map exported from SQLite. A `python3` helper made a 400-file index 9x slower; Recoll spawns it
      once per file per index run.
- [ ] Re-measure indexing overhead against real documents (PDF, EPUB) rather than small text files —
      extraction cost should dwarf the helper, but that is an assumption until measured.
- [x] Promote a new canonical path and force a reindex when a canonical path disappears, or the
      document silently leaves every collection it is in.
- [x] Recover interrupted indexing jobs at startup; Task 6 requeues valid running jobs and fails
      unreadable requests.
- [ ] Write `PRODUCT.md` properly via `/impeccable init` — the current file is a stub from the initial
      brief, not an interview. Blocks spec 002.

## Someday
- [ ] After delivery 004, write spec 005 for a cross-platform installable desktop distribution. Start
      from a Tauri-style shell that reuses the web UI and bundles the API and Recoll as sidecars; decide
      the actual framework, licensing, signing, updates and per-platform filter coverage in that spec.
- [ ] Propose a CI check (Gitea Actions): assert `AGENTS.md`, `CLAUDE.md`, `GEMINI.md` and the `docs/`
      paths exist, and fail if `AGENTS.md` exceeds 150 lines.

## Known issues
- The API test suite reports deprecation warnings from the installed FastAPI/Starlette TestClient
  dependency. Replace its client integration when the dependency stack supports the successor API.
- Directory sources currently require a named child of the document root; decide whether the root
  itself should be accepted as a persistent rule before finishing membership endpoints.
- Task 3's repository ID test checks the generated collection ID prefix and length rather than asserting
  the full `^c[0-9a-f]{10}$` regex. The implementation uses `secrets.token_hex(5)` and is correct; add
  the stronger regression assertion when work resumes.

## Recurring
- [ ] Add an `AGENTS.md` rule only in response to a real, observed agent failure — one line, ideally with
      a code example. Never pre-empt hypothetical mistakes.
- [ ] Update `AGENTS.md` in the same commit as the convention it describes. A stale rules file misleads.
- [ ] Prune quarterly: delete rules whose convention changed or whose mistake no longer occurs.