Archived
feat: add Recoll engine boundary
This commit is contained in:
1 parent
f6f03b9128
commit
863f671129
14 files changed
+545
-12
No files matched your search
@@ -5,7 +5,7 @@ How `recoll-modern` is built today. Update in the same commit as any structural
|
||||
## Overview
|
||||
|
||||
The Python service scaffold, SQLite schema, collection and membership repositories, document
|
||||
discovery/identity modules, and durable job worker are built. The API and Recoll adapter remain
|
||||
discovery/identity modules, durable job worker, and engine adapter are built. The HTTP API remains
|
||||
intended structure. Recoll behaviour below was verified against a real container.
|
||||
|
||||
The goal: a self-hosted service that wraps Recoll's indexing and full-text search behind (a) a modern
|
||||
@@ -28,6 +28,7 @@ web UI and (b) an MCP server, running in Docker, with several independent collec
|
||||
| `src/recoll_modern/indexing/handlers.py` | Scans membership sources before strict-mode mutations and records file errors |
|
||||
| `src/recoll_modern/indexing/map_export.py` | Writes a durable atomic map from canonical paths to opaque collection IDs |
|
||||
| `helper/` | C reader run by Recoll's `metadatacmds` for each indexed file |
|
||||
| `src/recoll_modern/engine/` | Shared Protocol, deterministic fake, and isolated Recoll Python/indexer adapter |
|
||||
| `tests/` | Automated service tests |
|
||||
|
||||
Discovery produces content hashes and root-relative paths without writing to the document volume.
|
||||
|
||||
@@ -274,3 +274,18 @@ from the stated product intent; `DESIGN.md` could not — there is no UI, no tok
|
||||
**Alternatives:** a placeholder `DESIGN.md` — rejected, it would assert a visual system that does not
|
||||
exist and the skill would design against fiction.
|
||||
**Consequences:** the first UI task must end by writing `DESIGN.md`, not skip it.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-26 — Retrieve text by Recoll UDI and expose rank-based score
|
||||
**Status:** provisional until the real-container contract runs
|
||||
**Context:** the approved plan proposed an escaped path query for complete text and a relevance score.
|
||||
Recoll's Python API documents `fsudi.fs_udi(path)` and `Db.getDoc()` for exact path lookup, but does not
|
||||
document a numeric query weight on returned documents.
|
||||
**Decision:** derive the UDI from the canonical path for full-text extraction. Set `score` to the
|
||||
reciprocal result rank, preserving Recoll's order across pages while making no claim that the number
|
||||
is Xapian's raw weight.
|
||||
**Alternatives:** a query-language path clause that must safely encode quotes and parentheses; an
|
||||
undocumented Recoll binding attribute for score.
|
||||
**Consequences:** unusual filenames do not enter a query parser. Clients can compare score order within
|
||||
one query, but the value is a rank proxy. Revisit this once the real Recoll container can be tested.
|
||||
+7
-5
@@ -5,7 +5,7 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
|
||||
## Resume here
|
||||
|
||||
Last touched 2026-09-18. Spec 001 implementation is in progress.
|
||||
Last touched 2026-09-26. Spec 001 implementation is in progress.
|
||||
|
||||
Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -> TDD -> review):
|
||||
|
||||
@@ -15,11 +15,11 @@ Where the workflow stands (`AGENTS.md` -> Workflow: brainstorm -> spec -> plan -
|
||||
selectable search fields, subpath hosting and the packaging seam for deferred desktop spec 005.
|
||||
- Plan `docs/plans/001-search-engine-api.md`: **approved 2026-09-18.** Its listed Python runtime and
|
||||
development dependencies are approved. Execution follows the plan task by task.
|
||||
- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–7 add discovery, content-hash
|
||||
identity, additive membership, durable jobs, and the atomic membership map with its C helper. Their
|
||||
checkboxes are current in the plan. The next task is the engine contract and Recoll adapter.
|
||||
- Plan Tasks 1–3 are complete and independently reviewed. Tasks 4–8 add discovery, content-hash
|
||||
identity, additive membership, durable jobs, the atomic membership map, and the engine boundary.
|
||||
Their checkboxes are current in the plan. The next task is authentication.
|
||||
|
||||
The next action is Task 8, "Define the engine contract and Recoll adapter." Do not repeat Tasks 1–7.
|
||||
The next action is Task 9, "Implement users, sessions, seeding, and CSRF." Do not repeat Tasks 1–8.
|
||||
|
||||
Things a resuming agent will otherwise get wrong:
|
||||
|
||||
@@ -42,6 +42,8 @@ Things a resuming agent will otherwise get wrong:
|
||||
the same commit that scaffolds the app, with every command actually run.
|
||||
|
||||
## Next
|
||||
- [ ] Verify the Recoll adapter against a generated corpus in the Compose container, including
|
||||
exact-path text retrieval and whether the Python API exposes a native numeric relevance score.
|
||||
- [x] Scaffold Python + FastAPI with `uv`; verify install/dev/test/lint/typecheck/build by running them,
|
||||
then record the verified ones in `AGENTS.md`.
|
||||
- [ ] Dockerfile + compose: Recoll in the image, document volume mounted `:ro`, writable volumes for
|
||||
|
||||
@@ -581,7 +581,7 @@ git commit -m "feat: export Recoll membership metadata"
|
||||
- Produces: `SearchRequest`, `EngineHit`, `EnginePage`, `EngineHealth`
|
||||
- Produces: `Engine.health/search/extracted_text/force_reindex`
|
||||
|
||||
- [ ] **Step 1: Write one contract suite against the fake**
|
||||
- [x] **Step 1: Write one contract suite against the fake**
|
||||
|
||||
```python
|
||||
class EngineContract:
|
||||
@@ -600,11 +600,11 @@ class EngineContract:
|
||||
|
||||
Cover metadata, snippets, optional text, offset, total estimate, health, missing text, and forced paths. `test_fake_engine.py` subclasses the contract with deterministic fixtures.
|
||||
|
||||
- [ ] **Step 2: Run fake contract and observe failure**
|
||||
- [x] **Step 2: Run fake contract and observe failure**
|
||||
|
||||
Run: `uv run pytest tests/contract/test_fake_engine.py -v`
|
||||
|
||||
- [ ] **Step 3: Implement Protocol and fake**
|
||||
- [x] **Step 3: Implement Protocol and fake**
|
||||
|
||||
Define `Engine` as a `Protocol` with exact methods `health() -> EngineHealth`,
|
||||
`search(SearchRequest) -> EnginePage`, `extracted_text(str) -> str`, and
|
||||
@@ -612,7 +612,7 @@ Define `Engine` as a `Protocol` with exact methods `health() -> EngineHealth`,
|
||||
|
||||
The fake applies collection OR semantics, returns each document once, and logs force-reindex calls.
|
||||
|
||||
- [ ] **Step 4: Implement the isolated Recoll adapter**
|
||||
- [x] **Step 4: Implement the isolated Recoll adapter**
|
||||
|
||||
Import `from recoll import recoll` only inside this file. Connect with the configured Recoll config
|
||||
directory. Build `(<user query>) AND (collection:c0123456789 OR collection:cabcdef012)` using the actual
|
||||
@@ -622,6 +622,11 @@ decode file URLs, and use `query.rowcount` as approximate total. Use a safely qu
|
||||
with `fetchtext=True` for `extracted_text`; its contract test includes spaces, quotes, and parentheses in
|
||||
filenames.
|
||||
|
||||
Implementation ruling: exact-path text retrieval uses Recoll's documented `fsudi.fs_udi(path)` plus
|
||||
`Db.getDoc()` and `rclextract`, which avoids query-language escaping for unusual filenames. The Python
|
||||
search API does not document a numeric relevance weight, so `score` currently derives from result rank;
|
||||
verify a native score when the real container contract runs.
|
||||
|
||||
Run indexing as an argument list, never a shell string:
|
||||
|
||||
```python
|
||||
@@ -633,11 +638,11 @@ subprocess.run(
|
||||
|
||||
Convert import, connection, timeout, and subprocess failures to `EngineUnavailable` without leaking stderr to clients.
|
||||
|
||||
- [ ] **Step 5: Make the real contract skip only when unavailable**
|
||||
- [x] **Step 5: Make the real contract skip only when unavailable**
|
||||
|
||||
`test_recoll_engine.py` probes the Compose service/config in a fixture. If unreachable it calls `pytest.skip("Recoll test container is not running")`; otherwise it runs the inherited contract. A separate assertion ensures the fixture did not return without either constructing an engine or recording a skip.
|
||||
|
||||
- [ ] **Step 6: Verify and commit**
|
||||
- [x] **Step 6: Verify and commit**
|
||||
|
||||
Run: `uv run pytest tests/contract -v && uv run ruff check . && uv run mypy src tests`
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Narrow search and indexing engine boundary."""
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Deterministic in-memory engine for contracts and application tests."""
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, replace
|
||||
|
||||
from recoll_modern.engine.protocol import EngineHealth, EngineHit, EnginePage, SearchRequest
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class FakeRecord:
|
||||
path: str
|
||||
title: str
|
||||
text: str
|
||||
collection_ids: tuple[str, ...]
|
||||
score: float = 1.0
|
||||
author: str | None = None
|
||||
mime_type: str | None = "text/plain"
|
||||
|
||||
|
||||
class FakeEngine:
|
||||
def __init__(self, records: Sequence[FakeRecord]) -> None:
|
||||
merged: dict[str, FakeRecord] = {}
|
||||
for record in records:
|
||||
previous = merged.get(record.path)
|
||||
if previous is None:
|
||||
merged[record.path] = record
|
||||
else:
|
||||
selected = record if record.score > previous.score else previous
|
||||
merged[record.path] = replace(
|
||||
selected,
|
||||
collection_ids=tuple(
|
||||
sorted(set(previous.collection_ids + record.collection_ids))
|
||||
),
|
||||
)
|
||||
self.records = list(merged.values())
|
||||
self.forced_paths: list[tuple[str, ...]] = []
|
||||
|
||||
def health(self) -> EngineHealth:
|
||||
return EngineHealth(True)
|
||||
|
||||
def search(self, request: SearchRequest) -> EnginePage:
|
||||
selected = [
|
||||
record for record in self.records
|
||||
if set(record.collection_ids) & set(request.collection_ids)
|
||||
and request.query.casefold() in (record.title + " " + record.text).casefold()
|
||||
]
|
||||
selected.sort(key=lambda record: (-record.score, record.path))
|
||||
hits = tuple(
|
||||
EngineHit(
|
||||
path=record.path,
|
||||
title=record.title,
|
||||
author=record.author,
|
||||
mime_type=record.mime_type,
|
||||
size=len(record.text.encode()),
|
||||
date=None,
|
||||
score=record.score,
|
||||
collection_ids=record.collection_ids,
|
||||
snippets=(record.text[:160],),
|
||||
text=record.text if request.fetch_text else None,
|
||||
)
|
||||
for record in selected[request.offset:request.offset + request.limit]
|
||||
)
|
||||
return EnginePage(hits, len(selected), False)
|
||||
|
||||
def extracted_text(self, path: str) -> str:
|
||||
return next((record.text for record in self.records if record.path == path), "")
|
||||
|
||||
def force_reindex(self, paths: Sequence[str]) -> None:
|
||||
self.forced_paths.append(tuple(paths))
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Types shared by fake and Recoll-backed search engines."""
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SearchRequest:
|
||||
query: str
|
||||
collection_ids: tuple[str, ...]
|
||||
limit: int
|
||||
offset: int
|
||||
fetch_text: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EngineHit:
|
||||
path: str
|
||||
title: str
|
||||
author: str | None
|
||||
mime_type: str | None
|
||||
size: int | None
|
||||
date: datetime | None
|
||||
score: float
|
||||
collection_ids: tuple[str, ...]
|
||||
snippets: tuple[str, ...]
|
||||
text: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EnginePage:
|
||||
hits: tuple[EngineHit, ...]
|
||||
total: int
|
||||
total_is_approximate: bool = True
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EngineHealth:
|
||||
available: bool
|
||||
message: str | None = None
|
||||
|
||||
|
||||
class EngineUnavailable(Exception):
|
||||
"""Recoll cannot serve a request; details stay in server logs."""
|
||||
|
||||
|
||||
class Engine(Protocol):
|
||||
def health(self) -> EngineHealth: ...
|
||||
|
||||
def search(self, request: SearchRequest) -> EnginePage: ...
|
||||
|
||||
def extracted_text(self, path: str) -> str: ...
|
||||
|
||||
def force_reindex(self, paths: Sequence[str]) -> None: ...
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Recoll Python binding and forced-index adapter, isolated from application code."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from recoll_modern.engine.protocol import (
|
||||
EngineHealth,
|
||||
EngineHit,
|
||||
EnginePage,
|
||||
EngineUnavailable,
|
||||
SearchRequest,
|
||||
)
|
||||
|
||||
_COLLECTION_ID = re.compile(r"c[0-9a-f]{10}\Z")
|
||||
|
||||
|
||||
class RecollEngine:
|
||||
def __init__(self, config_dir: Path, index_timeout: float = 300.0) -> None:
|
||||
self.config_dir = config_dir
|
||||
self.index_timeout = index_timeout
|
||||
|
||||
def _connect(self) -> Any:
|
||||
try:
|
||||
from recoll import recoll # type: ignore[import-not-found]
|
||||
|
||||
return recoll.connect(confdir=str(self.config_dir))
|
||||
except Exception as error:
|
||||
raise EngineUnavailable("Recoll index is unavailable") from error
|
||||
|
||||
def health(self) -> EngineHealth:
|
||||
try:
|
||||
db = self._connect()
|
||||
try:
|
||||
query = db.query()
|
||||
query.execute("recollmodernhealthprobe")
|
||||
query.close()
|
||||
finally:
|
||||
db.close()
|
||||
except Exception:
|
||||
return EngineHealth(False, "Recoll index is unavailable")
|
||||
return EngineHealth(True)
|
||||
|
||||
def search(self, request: SearchRequest) -> EnginePage:
|
||||
if not _safe_query(request.query):
|
||||
raise ValueError("query syntax cannot escape collection scope")
|
||||
if not request.collection_ids or any(
|
||||
_COLLECTION_ID.fullmatch(collection_id) is None
|
||||
for collection_id in request.collection_ids
|
||||
):
|
||||
raise ValueError("one or more valid collection IDs are required")
|
||||
scopes = " OR ".join(
|
||||
f"collection:{collection_id}" for collection_id in request.collection_ids
|
||||
)
|
||||
query_text = f"({request.query}) AND ({scopes})"
|
||||
db = self._connect()
|
||||
try:
|
||||
query = db.query()
|
||||
try:
|
||||
query.execute(query_text, fetchtext=request.fetch_text)
|
||||
total = int(query.rowcount)
|
||||
if request.offset >= total or request.limit == 0:
|
||||
return EnginePage((), total)
|
||||
query.scroll(request.offset, "absolute")
|
||||
docs = query.fetchmany(request.limit)
|
||||
hits = tuple(
|
||||
_hit(doc, query, request.fetch_text, request.offset + index)
|
||||
for index, doc in enumerate(docs)
|
||||
)
|
||||
return EnginePage(hits, total)
|
||||
finally:
|
||||
query.close()
|
||||
except Exception as error:
|
||||
raise EngineUnavailable("Recoll search is unavailable") from error
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
def extracted_text(self, path: str) -> str:
|
||||
try:
|
||||
from recoll import fsudi, rclextract
|
||||
|
||||
db = self._connect()
|
||||
try:
|
||||
doc = db.getDoc(fsudi.fs_udi(path))
|
||||
if doc is None:
|
||||
return ""
|
||||
extracted = rclextract.Extractor(doc).textextract(doc.ipath)
|
||||
return str(extracted.text)
|
||||
finally:
|
||||
db.close()
|
||||
except (KeyError, LookupError):
|
||||
return ""
|
||||
except Exception as error:
|
||||
raise EngineUnavailable("Recoll text is unavailable") from error
|
||||
|
||||
def force_reindex(self, paths: Sequence[str]) -> None:
|
||||
if not paths:
|
||||
return
|
||||
if any(not Path(path).is_absolute() for path in paths):
|
||||
raise ValueError("index paths must be absolute")
|
||||
try:
|
||||
subprocess.run(
|
||||
["recollindex", "-c", str(self.config_dir), "-i", "-Z", *paths],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=self.index_timeout,
|
||||
)
|
||||
except (OSError, subprocess.CalledProcessError, subprocess.TimeoutExpired) as error:
|
||||
raise EngineUnavailable("Recoll indexing is unavailable") from error
|
||||
|
||||
|
||||
def _hit(doc: Any, query: Any, fetch_text: bool, rank: int) -> EngineHit:
|
||||
raw_url = doc.getbinurl()
|
||||
if isinstance(raw_url, bytes):
|
||||
path = os.fsdecode(raw_url.removeprefix(b"file://"))
|
||||
else:
|
||||
path = str(raw_url).removeprefix("file://")
|
||||
snippets = tuple(str(fragment[2]) for fragment in query.getsnippets(doc, maxoccs=3))
|
||||
raw_date = getattr(doc, "fmtime", None)
|
||||
try:
|
||||
date = datetime.fromtimestamp(int(raw_date), UTC) if raw_date else None
|
||||
except (ValueError, TypeError, OverflowError):
|
||||
date = None
|
||||
raw_size = getattr(doc, "fbytes", None)
|
||||
try:
|
||||
size = int(raw_size) if raw_size else None
|
||||
except (ValueError, TypeError):
|
||||
size = None
|
||||
return EngineHit(
|
||||
path=path,
|
||||
title=str(getattr(doc, "title", "")),
|
||||
author=str(getattr(doc, "author", "")) or None,
|
||||
mime_type=str(getattr(doc, "mtype", "")) or None,
|
||||
size=size,
|
||||
date=date,
|
||||
score=1.0 / (rank + 1),
|
||||
collection_ids=tuple(str(getattr(doc, "collection", "")).split()),
|
||||
snippets=snippets,
|
||||
text=str(getattr(doc, "text", "")) if fetch_text else None,
|
||||
)
|
||||
|
||||
|
||||
def _safe_query(query: str) -> bool:
|
||||
"""Keep user grouping inside the enclosing scope expression."""
|
||||
if not query.strip() or "\\" in query:
|
||||
return False
|
||||
depth = 0
|
||||
quoted = False
|
||||
for character in query:
|
||||
if character == '"':
|
||||
quoted = not quoted
|
||||
elif not quoted and character == "(":
|
||||
depth += 1
|
||||
elif not quoted and character == ")":
|
||||
depth -= 1
|
||||
if depth < 0:
|
||||
return False
|
||||
return depth == 0 and not quoted
|
||||
@@ -0,0 +1 @@
|
||||
"""Test package for reusable contract cases."""
|
||||
@@ -0,0 +1 @@
|
||||
"""Shared engine contract tests."""
|
||||
@@ -0,0 +1,34 @@
|
||||
from recoll_modern.engine.protocol import Engine, SearchRequest
|
||||
|
||||
|
||||
class EngineContract:
|
||||
engine: Engine
|
||||
|
||||
def test_search_returns_one_ranked_page(self) -> None:
|
||||
page = self.engine.search(SearchRequest(
|
||||
query="python", collection_ids=("c0000000001", "c0000000002"),
|
||||
limit=20, offset=0, fetch_text=False,
|
||||
))
|
||||
assert page.total >= len(page.hits)
|
||||
assert [hit.score for hit in page.hits] == sorted(
|
||||
[hit.score for hit in page.hits], reverse=True
|
||||
)
|
||||
assert len({hit.path for hit in page.hits}) == len(page.hits)
|
||||
|
||||
def test_search_scopes_collection_and_paginates(self) -> None:
|
||||
first = self.engine.search(SearchRequest(
|
||||
"python", ("c0000000001",), 1, 0, False
|
||||
))
|
||||
second = self.engine.search(SearchRequest(
|
||||
"python", ("c0000000001",), 1, 1, False
|
||||
))
|
||||
assert all("c0000000001" in hit.collection_ids for hit in first.hits + second.hits)
|
||||
assert not first.hits or not second.hits or first.hits[0].path != second.hits[0].path
|
||||
|
||||
def test_optional_text_and_health(self) -> None:
|
||||
page = self.engine.search(SearchRequest("python", ("c0000000001",), 20, 0, True))
|
||||
assert self.engine.health().available
|
||||
assert all(hit.text is not None for hit in page.hits)
|
||||
|
||||
def test_missing_extracted_text_is_empty(self) -> None:
|
||||
assert self.engine.extracted_text("/documents/does-not-exist.txt") == ""
|
||||
@@ -0,0 +1,34 @@
|
||||
from recoll_modern.engine.fake import FakeEngine, FakeRecord
|
||||
from recoll_modern.engine.protocol import SearchRequest
|
||||
|
||||
from .engine_contract import EngineContract
|
||||
|
||||
|
||||
class TestFakeEngine(EngineContract):
|
||||
def setup_method(self) -> None:
|
||||
self.engine = FakeEngine([
|
||||
FakeRecord("/documents/a.txt", "Python guide", "A Python guide", (
|
||||
"c0000000001",), 0.9),
|
||||
FakeRecord("/documents/b.txt", "Python recipes", "Python recipes", (
|
||||
"c0000000001", "c0000000002"), 0.8),
|
||||
FakeRecord("/documents/c.txt", "Python notes", "Python notes", (
|
||||
"c0000000002",), 0.7),
|
||||
])
|
||||
|
||||
|
||||
def test_fake_logs_forced_paths() -> None:
|
||||
engine = FakeEngine([])
|
||||
engine.force_reindex(("/documents/b.txt", "/documents/a.txt"))
|
||||
assert engine.forced_paths == [("/documents/b.txt", "/documents/a.txt")]
|
||||
|
||||
|
||||
def test_fake_returns_one_hit_for_duplicate_path() -> None:
|
||||
engine = FakeEngine([
|
||||
FakeRecord("/documents/a.txt", "Python", "Python", ("c0000000001",), 0.9),
|
||||
FakeRecord("/documents/a.txt", "Python", "Python", ("c0000000002",), 0.8),
|
||||
])
|
||||
page = engine.search(SearchRequest(
|
||||
"python", ("c0000000001", "c0000000002"), 20, 0, False
|
||||
))
|
||||
assert page.total == 1
|
||||
assert page.hits[0].collection_ids == ("c0000000001", "c0000000002")
|
||||
@@ -0,0 +1,131 @@
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import ModuleType, SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from recoll_modern.engine.protocol import EngineUnavailable, SearchRequest
|
||||
from recoll_modern.engine.recoll import RecollEngine
|
||||
|
||||
|
||||
def test_force_reindex_uses_argument_list(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def run(args: list[str], **_kwargs: object) -> None:
|
||||
calls.append(args)
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", run)
|
||||
engine = RecollEngine(tmp_path)
|
||||
engine.force_reindex(("/documents/odd name.txt",))
|
||||
assert calls == [[
|
||||
"recollindex", "-c", str(tmp_path), "-i", "-Z", "/documents/odd name.txt"
|
||||
]]
|
||||
|
||||
|
||||
def test_force_reindex_hides_process_stderr(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
def fail(_args: list[str], **_kwargs: object) -> None:
|
||||
raise subprocess.CalledProcessError(1, "recollindex", stderr="private path")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", fail)
|
||||
with pytest.raises(EngineUnavailable) as caught:
|
||||
RecollEngine(tmp_path).force_reindex(("/documents/a.txt",))
|
||||
assert "private" not in str(caught.value)
|
||||
|
||||
|
||||
def test_empty_search_does_not_scroll(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
class EmptyQuery:
|
||||
rowcount = 0
|
||||
|
||||
def execute(self, *_args: object, **_kwargs: object) -> None:
|
||||
pass
|
||||
|
||||
def scroll(self, *_args: object) -> None:
|
||||
raise AssertionError("empty result must not scroll")
|
||||
|
||||
def close(self) -> None:
|
||||
pass
|
||||
|
||||
query = EmptyQuery()
|
||||
db = SimpleNamespace(query=lambda: query, close=lambda: None)
|
||||
monkeypatch.setattr(RecollEngine, "_connect", lambda self: db)
|
||||
page = RecollEngine(tmp_path).search(SearchRequest(
|
||||
"python", ("c0000000001",), 20, 0, False
|
||||
))
|
||||
assert page.total == 0 and page.hits == ()
|
||||
|
||||
|
||||
def test_query_cannot_escape_collection_scope(tmp_path: Path) -> None:
|
||||
with pytest.raises(ValueError, match="query"):
|
||||
RecollEngine(tmp_path).search(SearchRequest(
|
||||
"python) OR all OR (", ("c0000000001",), 20, 0, False
|
||||
))
|
||||
|
||||
|
||||
def test_search_uses_one_scoped_query(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
executed: list[str] = []
|
||||
doc = SimpleNamespace(
|
||||
getbinurl=lambda: b"file:///documents/odd name.txt",
|
||||
title="Python guide", author="Ada", mtype="text/plain", fbytes="12",
|
||||
fmtime="", collection="c0000000001 c0000000002", text="Python guide",
|
||||
)
|
||||
|
||||
class Query:
|
||||
rowcount = 1
|
||||
|
||||
def execute(self, query: str, **_kwargs: object) -> None:
|
||||
executed.append(query)
|
||||
|
||||
def scroll(self, offset: int, mode: str) -> None:
|
||||
assert (offset, mode) == (0, "absolute")
|
||||
|
||||
def fetchmany(self, limit: int) -> list[object]:
|
||||
assert limit == 20
|
||||
return [doc]
|
||||
|
||||
def getsnippets(self, _doc: object, **_kwargs: object) -> list[tuple[int, str, str]]:
|
||||
return [(0, "Python", "Python guide")]
|
||||
|
||||
def close(self) -> None:
|
||||
pass
|
||||
|
||||
db = SimpleNamespace(query=Query, close=lambda: None)
|
||||
monkeypatch.setattr(RecollEngine, "_connect", lambda self: db)
|
||||
page = RecollEngine(tmp_path).search(SearchRequest(
|
||||
"python", ("c0000000001", "c0000000002"), 20, 0, True
|
||||
))
|
||||
assert executed == [
|
||||
"(python) AND (collection:c0000000001 OR collection:c0000000002)"
|
||||
]
|
||||
assert page.hits[0].path == "/documents/odd name.txt"
|
||||
assert page.hits[0].collection_ids == ("c0000000001", "c0000000002")
|
||||
assert page.hits[0].text == "Python guide"
|
||||
|
||||
|
||||
def test_extracted_text_uses_exact_path_udi(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
path = '/documents/odd "name" (draft).txt'
|
||||
seen: list[str] = []
|
||||
recoll_package = ModuleType("recoll")
|
||||
|
||||
def fake_udi(value: str) -> str:
|
||||
seen.append(value)
|
||||
return "udi"
|
||||
|
||||
recoll_package.fsudi = SimpleNamespace(fs_udi=fake_udi) # type: ignore[attr-defined]
|
||||
recoll_package.rclextract = SimpleNamespace( # type: ignore[attr-defined]
|
||||
Extractor=lambda _doc: SimpleNamespace(
|
||||
textextract=lambda _ipath: SimpleNamespace(text="Complete text")
|
||||
)
|
||||
)
|
||||
monkeypatch.setitem(sys.modules, "recoll", recoll_package)
|
||||
db = SimpleNamespace(
|
||||
getDoc=lambda udi: SimpleNamespace(ipath="") if udi == "udi" else None,
|
||||
close=lambda: None,
|
||||
)
|
||||
monkeypatch.setattr(RecollEngine, "_connect", lambda self: db)
|
||||
assert RecollEngine(tmp_path).extracted_text(path) == "Complete text"
|
||||
assert seen == [path]
|
||||
@@ -0,0 +1,20 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from recoll_modern.engine.recoll import RecollEngine
|
||||
|
||||
from .engine_contract import EngineContract
|
||||
|
||||
|
||||
class TestRecollEngine(EngineContract):
|
||||
@pytest.fixture(autouse=True)
|
||||
def available_engine(self) -> None:
|
||||
config = os.environ.get("RECOLL_MODERN_TEST_RECOLL_CONFIG_DIR")
|
||||
if config is None:
|
||||
pytest.skip("Recoll test container is not running")
|
||||
engine = RecollEngine(Path(config))
|
||||
if not engine.health().available:
|
||||
pytest.skip("Recoll test container is not running")
|
||||
self.engine = engine
|
||||
Reference in new issue
Block a user