feat: add Recoll engine boundary

This commit is contained in:
Lucas Winther committed 2026-09-26 10:59:09 +02:00
1 parent f6f03b9128
commit 863f671129
14 files changed
+545 -12

No files matched your search

+2 -1
View File
@@ -5,7 +5,7 @@ How `recoll-modern` is built today. Update in the same commit as any structural
## Overview
The Python service scaffold, SQLite schema, collection and membership repositories, document
discovery/identity modules, and durable job worker are built. The API and Recoll adapter remain
discovery/identity modules, durable job worker, and engine adapter are built. The HTTP API remains
intended structure. Recoll behaviour below was verified against a real container.
The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
@@ -28,6 +28,7 @@ web UI and (b) an MCP server, running in Docker, with several independent collec
| `src/recoll_modern/indexing/handlers.py` | Scans membership sources before strict-mode mutations and records file errors |
| `src/recoll_modern/indexing/map_export.py` | Writes a durable atomic map from canonical paths to opaque collection IDs |
| `helper/` | C reader run by Recoll's `metadatacmds` for each indexed file |
| `src/recoll_modern/engine/` | Shared Protocol, deterministic fake, and isolated Recoll Python/indexer adapter |
| `tests/` | Automated service tests |
Discovery produces content hashes and root-relative paths without writing to the document volume.
+15
View File
@@ -274,3 +274,18 @@ from the stated product intent; `DESIGN.md` could not — there is no UI, no tok
**Alternatives:** a placeholder `DESIGN.md` — rejected, it would assert a visual system that does not
exist and the skill would design against fiction.
**Consequences:** the first UI task must end by writing `DESIGN.md`, not skip it.
---
## 2026-09-26 — Retrieve text by Recoll UDI and expose rank-based score
**Status:** provisional until the real-container contract runs
**Context:** the approved plan proposed an escaped path query for complete text and a relevance score.
Recoll's Python API documents `fsudi.fs_udi(path)` and `Db.getDoc()` for exact path lookup, but does not
document a numeric query weight on returned documents.
**Decision:** derive the UDI from the canonical path for full-text extraction. Set `score` to the
reciprocal result rank, preserving Recoll's order across pages while making no claim that the number
is Xapian's raw weight.
**Alternatives:** a query-language path clause that must safely encode quotes and parentheses; an
undocumented Recoll binding attribute for score.
**Consequences:** unusual filenames do not enter a query parser. Clients can compare score order within
one query, but the value is a rank proxy. Revisit this once the real Recoll container can be tested.
+7 -5
View File
@@ -5,7 +5,7 @@ Small, specific, independently actionable. Remove only when actually done.
## Resume here
Last touched 2026-09-18. Spec 001 implementation is in progress.
Last touched 2026-09-26. Spec 001 implementation is in progress.
Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review):
@@ -15,11 +15,11 @@ Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -
selectable search fields, subpath hosting and the packaging seam for deferred desktop spec 005.
- Plan `docs/plans/001-search-engine-api.md`: **approved 2026-09-18.** Its listed Python runtime and
development dependencies are approved. Execution follows the plan task by task.
- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–7 add discovery, content-hash
identity, additive membership, durable jobs, and the atomic membership map with its C helper. Their
checkboxes are current in the plan. The next task is the engine contract and Recoll adapter.
- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–8 add discovery, content-hash
identity, additive membership, durable jobs, the atomic membership map, and the engine boundary.
Their checkboxes are current in the plan. The next task is authentication.
The next action is Task 8, "Define the engine contract and Recoll adapter." Do not repeat Tasks 1–7.
The next action is Task 9, "Implement users, sessions, seeding, and CSRF." Do not repeat Tasks 1–8.
Things a resuming agent will otherwise get wrong:
@@ -42,6 +42,8 @@ Things a resuming agent will otherwise get wrong:
the same commit that scaffolds the app, with every command actually run.
## Next
- [ ] Verify the Recoll adapter against a generated corpus in the Compose container, including
exact-path text retrieval and whether the Python API exposes a native numeric relevance score.
- [x] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them,
then record the verified ones in `AGENTS.md`.
- [ ] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for
+11 -6
View File
@@ -581,7 +581,7 @@ git commit -m "feat: export Recoll membership metadata"
- Produces: `SearchRequest`, `EngineHit`, `EnginePage`, `EngineHealth`
- Produces: `Engine.health/search/extracted_text/force_reindex`
- [ ] **Step 1: Write one contract suite against the fake**
- [x] **Step 1: Write one contract suite against the fake**
```python
class EngineContract:
@@ -600,11 +600,11 @@ class EngineContract:
Cover metadata, snippets, optional text, offset, total estimate, health, missing text, and forced paths. `test_fake_engine.py` subclasses the contract with deterministic fixtures.
- [ ] **Step 2: Run fake contract and observe failure**
- [x] **Step 2: Run fake contract and observe failure**
Run: `uv run pytest tests/contract/test_fake_engine.py -v`
- [ ] **Step 3: Implement Protocol and fake**
- [x] **Step 3: Implement Protocol and fake**
Define `Engine` as a `Protocol` with exact methods `health() -> EngineHealth`,
`search(SearchRequest) -> EnginePage`, `extracted_text(str) -> str`, and
@@ -612,7 +612,7 @@ Define `Engine` as a `Protocol` with exact methods `health() -> EngineHealth`,
The fake applies collection OR semantics, returns each document once, and logs force-reindex calls.
- [ ] **Step 4: Implement the isolated Recoll adapter**
- [x] **Step 4: Implement the isolated Recoll adapter**
Import `from recoll import recoll` only inside this file. Connect with the configured Recoll config
directory. Build `(<user query>) AND (collection:c0123456789 OR collection:cabcdef012)` using the actual
@@ -622,6 +622,11 @@ decode file URLs, and use `query.rowcount` as approximate total. Use a safely qu
with `fetchtext=True` for `extracted_text`; its contract test includes spaces, quotes, and parentheses in
filenames.
Implementation ruling: exact-path text retrieval uses Recoll's documented `fsudi.fs_udi(path)` plus
`Db.getDoc()` and `rclextract`, which avoids query-language escaping for unusual filenames. The Python
search API does not document a numeric relevance weight, so `score` currently derives from result rank;
verify a native score when the real container contract runs.
Run indexing as an argument list, never a shell string:
```python
@@ -633,11 +638,11 @@ subprocess.run(
Convert import, connection, timeout, and subprocess failures to `EngineUnavailable` without leaking stderr to clients.
- [ ] **Step 5: Make the real contract skip only when unavailable**
- [x] **Step 5: Make the real contract skip only when unavailable**
`test_recoll_engine.py` probes the Compose service/config in a fixture. If unreachable it calls `pytest.skip("Recoll test container is not running")`; otherwise it runs the inherited contract. A separate assertion ensures the fixture did not return without either constructing an engine or recording a skip.
- [ ] **Step 6: Verify and commit**
- [x] **Step 6: Verify and commit**
Run: `uv run pytest tests/contract -v && uv run ruff check . && uv run mypy src tests`
+1
View File
@@ -0,0 +1 @@
"""Narrow search and indexing engine boundary."""
+69
View File
@@ -0,0 +1,69 @@
"""Deterministic in-memory engine for contracts and application tests."""
from collections.abc import Sequence
from dataclasses import dataclass, replace
from recoll_modern.engine.protocol import EngineHealth, EngineHit, EnginePage, SearchRequest
@dataclass(frozen=True, slots=True)
class FakeRecord:
path: str
title: str
text: str
collection_ids: tuple[str, ...]
score: float = 1.0
author: str | None = None
mime_type: str | None = "text/plain"
class FakeEngine:
def __init__(self, records: Sequence[FakeRecord]) -> None:
merged: dict[str, FakeRecord] = {}
for record in records:
previous = merged.get(record.path)
if previous is None:
merged[record.path] = record
else:
selected = record if record.score > previous.score else previous
merged[record.path] = replace(
selected,
collection_ids=tuple(
sorted(set(previous.collection_ids + record.collection_ids))
),
)
self.records = list(merged.values())
self.forced_paths: list[tuple[str, ...]] = []
def health(self) -> EngineHealth:
return EngineHealth(True)
def search(self, request: SearchRequest) -> EnginePage:
selected = [
record for record in self.records
if set(record.collection_ids) & set(request.collection_ids)
and request.query.casefold() in (record.title + " " + record.text).casefold()
]
selected.sort(key=lambda record: (-record.score, record.path))
hits = tuple(
EngineHit(
path=record.path,
title=record.title,
author=record.author,
mime_type=record.mime_type,
size=len(record.text.encode()),
date=None,
score=record.score,
collection_ids=record.collection_ids,
snippets=(record.text[:160],),
text=record.text if request.fetch_text else None,
)
for record in selected[request.offset:request.offset + request.limit]
)
return EnginePage(hits, len(selected), False)
def extracted_text(self, path: str) -> str:
return next((record.text for record in self.records if record.path == path), "")
def force_reindex(self, paths: Sequence[str]) -> None:
self.forced_paths.append(tuple(paths))
+56
View File
@@ -0,0 +1,56 @@
"""Types shared by fake and Recoll-backed search engines."""
from collections.abc import Sequence
from dataclasses import dataclass
from datetime import datetime
from typing import Protocol
@dataclass(frozen=True, slots=True)
class SearchRequest:
query: str
collection_ids: tuple[str, ...]
limit: int
offset: int
fetch_text: bool
@dataclass(frozen=True, slots=True)
class EngineHit:
path: str
title: str
author: str | None
mime_type: str | None
size: int | None
date: datetime | None
score: float
collection_ids: tuple[str, ...]
snippets: tuple[str, ...]
text: str | None = None
@dataclass(frozen=True, slots=True)
class EnginePage:
hits: tuple[EngineHit, ...]
total: int
total_is_approximate: bool = True
@dataclass(frozen=True, slots=True)
class EngineHealth:
available: bool
message: str | None = None
class EngineUnavailable(Exception):
"""Recoll cannot serve a request; details stay in server logs."""
class Engine(Protocol):
def health(self) -> EngineHealth: ...
def search(self, request: SearchRequest) -> EnginePage: ...
def extracted_text(self, path: str) -> str: ...
def force_reindex(self, paths: Sequence[str]) -> None: ...
+163
View File
@@ -0,0 +1,163 @@
"""Recoll Python binding and forced-index adapter, isolated from application code."""
import os
import re
import subprocess
from collections.abc import Sequence
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from recoll_modern.engine.protocol import (
EngineHealth,
EngineHit,
EnginePage,
EngineUnavailable,
SearchRequest,
)
_COLLECTION_ID = re.compile(r"c[0-9a-f]{10}\Z")
class RecollEngine:
def __init__(self, config_dir: Path, index_timeout: float = 300.0) -> None:
self.config_dir = config_dir
self.index_timeout = index_timeout
def _connect(self) -> Any:
try:
from recoll import recoll # type: ignore[import-not-found]
return recoll.connect(confdir=str(self.config_dir))
except Exception as error:
raise EngineUnavailable("Recoll index is unavailable") from error
def health(self) -> EngineHealth:
try:
db = self._connect()
try:
query = db.query()
query.execute("recollmodernhealthprobe")
query.close()
finally:
db.close()
except Exception:
return EngineHealth(False, "Recoll index is unavailable")
return EngineHealth(True)
def search(self, request: SearchRequest) -> EnginePage:
if not _safe_query(request.query):
raise ValueError("query syntax cannot escape collection scope")
if not request.collection_ids or any(
_COLLECTION_ID.fullmatch(collection_id) is None
for collection_id in request.collection_ids
):
raise ValueError("one or more valid collection IDs are required")
scopes = " OR ".join(
f"collection:{collection_id}" for collection_id in request.collection_ids
)
query_text = f"({request.query}) AND ({scopes})"
db = self._connect()
try:
query = db.query()
try:
query.execute(query_text, fetchtext=request.fetch_text)
total = int(query.rowcount)
if request.offset >= total or request.limit == 0:
return EnginePage((), total)
query.scroll(request.offset, "absolute")
docs = query.fetchmany(request.limit)
hits = tuple(
_hit(doc, query, request.fetch_text, request.offset + index)
for index, doc in enumerate(docs)
)
return EnginePage(hits, total)
finally:
query.close()
except Exception as error:
raise EngineUnavailable("Recoll search is unavailable") from error
finally:
db.close()
def extracted_text(self, path: str) -> str:
try:
from recoll import fsudi, rclextract
db = self._connect()
try:
doc = db.getDoc(fsudi.fs_udi(path))
if doc is None:
return ""
extracted = rclextract.Extractor(doc).textextract(doc.ipath)
return str(extracted.text)
finally:
db.close()
except (KeyError, LookupError):
return ""
except Exception as error:
raise EngineUnavailable("Recoll text is unavailable") from error
def force_reindex(self, paths: Sequence[str]) -> None:
if not paths:
return
if any(not Path(path).is_absolute() for path in paths):
raise ValueError("index paths must be absolute")
try:
subprocess.run(
["recollindex", "-c", str(self.config_dir), "-i", "-Z", *paths],
check=True,
capture_output=True,
text=True,
timeout=self.index_timeout,
)
except (OSError, subprocess.CalledProcessError, subprocess.TimeoutExpired) as error:
raise EngineUnavailable("Recoll indexing is unavailable") from error
def _hit(doc: Any, query: Any, fetch_text: bool, rank: int) -> EngineHit:
raw_url = doc.getbinurl()
if isinstance(raw_url, bytes):
path = os.fsdecode(raw_url.removeprefix(b"file://"))
else:
path = str(raw_url).removeprefix("file://")
snippets = tuple(str(fragment[2]) for fragment in query.getsnippets(doc, maxoccs=3))
raw_date = getattr(doc, "fmtime", None)
try:
date = datetime.fromtimestamp(int(raw_date), UTC) if raw_date else None
except (ValueError, TypeError, OverflowError):
date = None
raw_size = getattr(doc, "fbytes", None)
try:
size = int(raw_size) if raw_size else None
except (ValueError, TypeError):
size = None
return EngineHit(
path=path,
title=str(getattr(doc, "title", "")),
author=str(getattr(doc, "author", "")) or None,
mime_type=str(getattr(doc, "mtype", "")) or None,
size=size,
date=date,
score=1.0 / (rank + 1),
collection_ids=tuple(str(getattr(doc, "collection", "")).split()),
snippets=snippets,
text=str(getattr(doc, "text", "")) if fetch_text else None,
)
def _safe_query(query: str) -> bool:
"""Keep user grouping inside the enclosing scope expression."""
if not query.strip() or "\\" in query:
return False
depth = 0
quoted = False
for character in query:
if character == '"':
quoted = not quoted
elif not quoted and character == "(":
depth += 1
elif not quoted and character == ")":
depth -= 1
if depth < 0:
return False
return depth == 0 and not quoted
+1
View File
@@ -0,0 +1 @@
"""Test package for reusable contract cases."""
+1
View File
@@ -0,0 +1 @@
"""Shared engine contract tests."""
+34
View File
@@ -0,0 +1,34 @@
from recoll_modern.engine.protocol import Engine, SearchRequest
class EngineContract:
engine: Engine
def test_search_returns_one_ranked_page(self) -> None:
page = self.engine.search(SearchRequest(
query="python", collection_ids=("c0000000001", "c0000000002"),
limit=20, offset=0, fetch_text=False,
))
assert page.total >= len(page.hits)
assert [hit.score for hit in page.hits] == sorted(
[hit.score for hit in page.hits], reverse=True
)
assert len({hit.path for hit in page.hits}) == len(page.hits)
def test_search_scopes_collection_and_paginates(self) -> None:
first = self.engine.search(SearchRequest(
"python", ("c0000000001",), 1, 0, False
))
second = self.engine.search(SearchRequest(
"python", ("c0000000001",), 1, 1, False
))
assert all("c0000000001" in hit.collection_ids for hit in first.hits + second.hits)
assert not first.hits or not second.hits or first.hits[0].path != second.hits[0].path
def test_optional_text_and_health(self) -> None:
page = self.engine.search(SearchRequest("python", ("c0000000001",), 20, 0, True))
assert self.engine.health().available
assert all(hit.text is not None for hit in page.hits)
def test_missing_extracted_text_is_empty(self) -> None:
assert self.engine.extracted_text("/documents/does-not-exist.txt") == ""
+34
View File
@@ -0,0 +1,34 @@
from recoll_modern.engine.fake import FakeEngine, FakeRecord
from recoll_modern.engine.protocol import SearchRequest
from .engine_contract import EngineContract
class TestFakeEngine(EngineContract):
def setup_method(self) -> None:
self.engine = FakeEngine([
FakeRecord("/documents/a.txt", "Python guide", "A Python guide", (
"c0000000001",), 0.9),
FakeRecord("/documents/b.txt", "Python recipes", "Python recipes", (
"c0000000001", "c0000000002"), 0.8),
FakeRecord("/documents/c.txt", "Python notes", "Python notes", (
"c0000000002",), 0.7),
])
def test_fake_logs_forced_paths() -> None:
engine = FakeEngine([])
engine.force_reindex(("/documents/b.txt", "/documents/a.txt"))
assert engine.forced_paths == [("/documents/b.txt", "/documents/a.txt")]
def test_fake_returns_one_hit_for_duplicate_path() -> None:
engine = FakeEngine([
FakeRecord("/documents/a.txt", "Python", "Python", ("c0000000001",), 0.9),
FakeRecord("/documents/a.txt", "Python", "Python", ("c0000000002",), 0.8),
])
page = engine.search(SearchRequest(
"python", ("c0000000001", "c0000000002"), 20, 0, False
))
assert page.total == 1
assert page.hits[0].collection_ids == ("c0000000001", "c0000000002")
+131
View File
@@ -0,0 +1,131 @@
import subprocess
import sys
from pathlib import Path
from types import ModuleType, SimpleNamespace
import pytest
from recoll_modern.engine.protocol import EngineUnavailable, SearchRequest
from recoll_modern.engine.recoll import RecollEngine
def test_force_reindex_uses_argument_list(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[list[str]] = []
def run(args: list[str], **_kwargs: object) -> None:
calls.append(args)
monkeypatch.setattr(subprocess, "run", run)
engine = RecollEngine(tmp_path)
engine.force_reindex(("/documents/odd name.txt",))
assert calls == [[
"recollindex", "-c", str(tmp_path), "-i", "-Z", "/documents/odd name.txt"
]]
def test_force_reindex_hides_process_stderr(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
def fail(_args: list[str], **_kwargs: object) -> None:
raise subprocess.CalledProcessError(1, "recollindex", stderr="private path")
monkeypatch.setattr(subprocess, "run", fail)
with pytest.raises(EngineUnavailable) as caught:
RecollEngine(tmp_path).force_reindex(("/documents/a.txt",))
assert "private" not in str(caught.value)
def test_empty_search_does_not_scroll(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
class EmptyQuery:
rowcount = 0
def execute(self, *_args: object, **_kwargs: object) -> None:
pass
def scroll(self, *_args: object) -> None:
raise AssertionError("empty result must not scroll")
def close(self) -> None:
pass
query = EmptyQuery()
db = SimpleNamespace(query=lambda: query, close=lambda: None)
monkeypatch.setattr(RecollEngine, "_connect", lambda self: db)
page = RecollEngine(tmp_path).search(SearchRequest(
"python", ("c0000000001",), 20, 0, False
))
assert page.total == 0 and page.hits == ()
def test_query_cannot_escape_collection_scope(tmp_path: Path) -> None:
with pytest.raises(ValueError, match="query"):
RecollEngine(tmp_path).search(SearchRequest(
"python) OR all OR (", ("c0000000001",), 20, 0, False
))
def test_search_uses_one_scoped_query(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
executed: list[str] = []
doc = SimpleNamespace(
getbinurl=lambda: b"file:///documents/odd name.txt",
title="Python guide", author="Ada", mtype="text/plain", fbytes="12",
fmtime="", collection="c0000000001 c0000000002", text="Python guide",
)
class Query:
rowcount = 1
def execute(self, query: str, **_kwargs: object) -> None:
executed.append(query)
def scroll(self, offset: int, mode: str) -> None:
assert (offset, mode) == (0, "absolute")
def fetchmany(self, limit: int) -> list[object]:
assert limit == 20
return [doc]
def getsnippets(self, _doc: object, **_kwargs: object) -> list[tuple[int, str, str]]:
return [(0, "Python", "Python guide")]
def close(self) -> None:
pass
db = SimpleNamespace(query=Query, close=lambda: None)
monkeypatch.setattr(RecollEngine, "_connect", lambda self: db)
page = RecollEngine(tmp_path).search(SearchRequest(
"python", ("c0000000001", "c0000000002"), 20, 0, True
))
assert executed == [
"(python) AND (collection:c0000000001 OR collection:c0000000002)"
]
assert page.hits[0].path == "/documents/odd name.txt"
assert page.hits[0].collection_ids == ("c0000000001", "c0000000002")
assert page.hits[0].text == "Python guide"
def test_extracted_text_uses_exact_path_udi(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
path = '/documents/odd "name" (draft).txt'
seen: list[str] = []
recoll_package = ModuleType("recoll")
def fake_udi(value: str) -> str:
seen.append(value)
return "udi"
recoll_package.fsudi = SimpleNamespace(fs_udi=fake_udi) # type: ignore[attr-defined]
recoll_package.rclextract = SimpleNamespace( # type: ignore[attr-defined]
Extractor=lambda _doc: SimpleNamespace(
textextract=lambda _ipath: SimpleNamespace(text="Complete text")
)
)
monkeypatch.setitem(sys.modules, "recoll", recoll_package)
db = SimpleNamespace(
getDoc=lambda udi: SimpleNamespace(ipath="") if udi == "udi" else None,
close=lambda: None,
)
monkeypatch.setattr(RecollEngine, "_connect", lambda self: db)
assert RecollEngine(tmp_path).extracted_text(path) == "Complete text"
assert seen == [path]
+20
View File
@@ -0,0 +1,20 @@
import os
from pathlib import Path
import pytest
from recoll_modern.engine.recoll import RecollEngine
from .engine_contract import EngineContract
class TestRecollEngine(EngineContract):
@pytest.fixture(autouse=True)
def available_engine(self) -> None:
config = os.environ.get("RECOLL_MODERN_TEST_RECOLL_CONFIG_DIR")
if config is None:
pytest.skip("Recoll test container is not running")
engine = RecollEngine(Path(config))
if not engine.health().available:
pytest.skip("Recoll test container is not running")
self.engine = engine