feat: add durable indexing jobs

docs/ARCHITECTURE.md

@@ -4,9 +4,9 @@ How `recoll-modern` is built today. Update in the same commit as any structural
 
 ## Overview
 
-The Python service scaffold, SQLite schema, collection and membership repositories, and document
-discovery/identity modules are built. The API, job queue, and Recoll adapter remain intended structure.
-Recoll behaviour below was verified against a real container.
+The Python service scaffold, SQLite schema, collection and membership repositories, document
+discovery/identity modules, and durable job worker are built. The API and Recoll adapter remain
+intended structure. Recoll behaviour below was verified against a real container.
 
 The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
 web UI and (b) an MCP server, running in Docker, with several independent collections such as
@@ -24,6 +24,8 @@ web UI and (b) an MCP server, running in Docker, with several independent collec
 | `src/recoll_modern/storage/` | SQLite migrations, collection metadata and content-hash identity |
 | `src/recoll_modern/domain/membership.py` | Validates sources and applies direct additions and rule changes |
 | `src/recoll_modern/indexing/reconcile.py` | Refreshes persistent directory rules while retaining excluded or unreadable files as present |
+| `src/recoll_modern/indexing/worker.py` | Claims one durable job at a time and runs blocking handlers off the event loop |
+| `src/recoll_modern/indexing/handlers.py` | Scans membership sources before strict-mode mutations and records file errors |
 | `tests/` | Automated service tests |
 
 Discovery produces content hashes and root-relative paths without writing to the document volume.
@@ -80,7 +82,9 @@ is normalized: `direct_memberships` holds explicit files and snapshot expansions
 `membership_exclusions` holds per-file overrides against rule-derived membership. Effective membership
 is direct membership union non-excluded rule membership, so direct additions remain additive.
 
-`jobs` and `job_file_errors` make work and partial failures durable. `users` and hashed-token `sessions`
+`jobs` and `job_file_errors` make work and partial failures durable. At startup, the worker requeues
+well-formed interrupted jobs, clears errors from the prior attempt, and fails unreadable requests.
+`users` and hashed-token `sessions`
 provide authentication. `schema_migrations` is forward-only. The exact columns and constraints are the
 contract in `docs/specs/001-search-engine-api.md`.
 

docs/TODO.md

@@ -15,11 +15,12 @@ Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -
   selectable search fields, subpath hosting and the packaging seam for deferred desktop spec 005.
 - Plan `docs/plans/001-search-engine-api.md`: **approved 2026-09-18.** Its listed Python runtime and
   development dependencies are approved. Execution follows the plan task by task.
-- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–5 add read-only discovery,
-  content-hash identity, additive membership and directory rules. Their checkboxes are current in the
-  plan. The next task is the durable job queue.
+- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–6 add read-only discovery,
+  content-hash identity, additive membership, directory rules and durable jobs. Their checkboxes are
+  current in the plan. The next task is membership-map export and the native helper.
 
-The next action is Task 6, "Add the durable single-worker job queue." Do not repeat Tasks 1–5.
+The next action is Task 7, "Export membership atomically and build the native helper." Do not repeat
+Tasks 1–6.
 
 Things a resuming agent will otherwise get wrong:
 

docs/plans/001-search-engine-api.md

@@ -478,7 +478,7 @@ git commit -m "feat: add membership provenance and rules"
 - Produces: `JobWorker.start()`, `JobWorker.stop()`, `JobWorker.run_once() -> bool`
 - Produces: `MembershipJobHandler.handle(job: Job) -> None`
 
-- [ ] **Step 1: Write failing transition, recovery, and policy tests**
+- [x] **Step 1: Write failing transition, recovery, and policy tests**
 
 ```python
 def test_recover_running_requeues_valid_jobs(repo: JobRepository) -> None:
@@ -498,17 +498,17 @@ def test_best_effort_applies_valid_files_and_records_errors(handler: MembershipJ
     assert jobs.current.state == JobState.COMPLETED_WITH_ERRORS
 ```
 
-- [ ] **Step 2: Run tests and observe failure**
+- [x] **Step 2: Run tests and observe failure**
 
 Run: `uv run pytest tests/storage/test_job_repository.py tests/indexing/test_worker.py tests/indexing/test_job_handlers.py -v`
 
-- [ ] **Step 3: Implement durable transitions and handlers**
+- [x] **Step 3: Implement durable transitions and handlers**
 
 Use opaque `j` plus 32 hex digits. `claim_next` runs under `BEGIN IMMEDIATE`, selects the oldest queued job, and changes it to running atomically. Valid transitions are queued→running, running→completed/completed_with_errors/failed, and failed→queued via retry. Startup requeues running jobs whose JSON decodes to the registered request type; malformed requests become failed with `invalid_job_request`.
 
 The worker owns one asyncio task, sleeps on an `asyncio.Event`, performs blocking discovery/database/engine work with `asyncio.to_thread`, and on shutdown stops claiming new work and leaves an interrupted running job recoverable. Membership handlers scan completely before opening the desired-state transaction in strict mode. Best-effort handlers persist every safe file error and continue.
 
-- [ ] **Step 4: Verify and commit**
+- [x] **Step 4: Verify and commit**
 
 Run: `uv run pytest tests/storage/test_job_repository.py tests/indexing/test_worker.py tests/indexing/test_job_handlers.py -v && uv run ruff check . && uv run mypy src tests`