Adds searchPipeline, which ranks candidates shallowly at no API cost, then
enriches a bounded subset (DEFAULT_ENRICH_LIMIT=25, DEFAULT_CONCURRENCY=8)
and re-ranks, taking max(shallow, deep) so a strong title match is never
diluted by unrelated deep content. Auth and rate-limit errors abort the
whole search via isFatalSearchError instead of degrading silently; other
enrichment failures degrade one record and are counted. A recency backfill
spends leftover enrichment budget on the most recently updated candidates
so a deep-only match is reachable at all. Adds the src/search/ barrel
export.
Reconciles the previously uncommitted pipeline.ts (written against an
earlier, defective version of this task's brief) with the corrected brief:
mapWithConcurrency's phase-2 callback is now given an explicit
EnrichedCandidate return type so its `deep: TDeep | undefined` field is a
single type rather than an inferred union, which is what the `entry is
Pair` narrowing predicate requires under TS2677.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
mapWithConcurrency's callback return was inferred as a union of two object
literal types rather than one type with deep: TDeep | undefined, so the
'entry is Pair' predicate was rejected: a type predicate's type must be
assignable to its parameter's type. Added an explicit EnrichedCandidate
annotation on the call. Tests passed but tsc did not, so this was only
visible via typecheck.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds boundedLevenshtein with row-minimum early exit, token scoring
(exact / prefix / contains / shared-prefix / edit distance), and
coverage-weighted field aggregation scaled by BASE_SCALE so that only a
verbatim phrase match reaches 1.0.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The guard tests used 3-char query tokens whose field tokens were literal
continuations ('car'/'carpet'), so the startsWith rule returned 0.9 before
the shared-prefix rule was reached. With MIN_SHARED_PREFIX=4 the rule is
unreachable for any 3-char token, making the assertion impossible to
satisfy. Swapped in cases where neither token is a true prefix of the
other, which actually exercise the guard.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Replace manual exec loop with matchAll to eliminate cross-call mutable
lastIndex state on module-level TOKEN_PATTERN regex. matchAll clones
the regex internally, ensuring no shared state while maintaining
identical behavior and original-text offset semantics.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Hand-computing the plan's own test expectations against its own formula
showed four assertions could not pass:
- Body-only matches maxed at 1.0*1/3 = 0.333, below minScore 0.35, so no
low-weight field could ever return a result. The gate now uses the raw
field score; the weighted score only orders.
- The phrase bonus was dead code whenever every token matched, since the
base already saturated at 1.0. Base is now scaled by 0.85.
- Natural-language queries tanked the coverage multiplier; query tokens
are now stopword-filtered.
- 'decide' vs 'decision' is 4 edits apart and unmatchable; scoreToken
gains a shared-prefix rule.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds an 11-task TDD implementation plan covering the pure search unit
(tokenize, score, rank, snippet, pipeline), per-service integration for
meetings, wikis, work packages and activities, a recall acceptance
suite, and documentation.
Corrects two errors found while writing the plan:
- The spec's behaviour table named an openproject_search_work_packages
tool that does not exist. searchWorkPackages is a service helper that
only tests call, so ranked work package search would have been
unreachable by any LLM. The plan now registers the tool (18 -> 19) and
leaves openproject_list_work_packages as a pure server-side filter.
- Plan Task 6 originally identified matching agenda items by substring
searching the meeting-level snippet, which cannot work because
snippets are trimmed and ellipsized. Agenda items are now ranked
directly.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Specifies replacing exact substring matching with token-aware fuzzy
matching as the default across all search tools, plus relevance ranking
and wider content coverage (wiki body text, work package descriptions
and comments, meeting participants and project names).
Key decisions:
- Hand-rolled zero-dependency scorer in src/search/; fuse.js rejected
because its Bitap implementation caps patterns at 32 chars and targets
short fields, not the long-text matching required here.
- matchMode: 'fuzzy' | 'exact' escape hatch, defaulting to fuzzy, with
exact mode preserving current behaviour byte-for-byte.
- Two-phase pipeline: cheap shallow rank, deep-enrich top 25 with
bounded concurrency, recency backfill for deep-only matches.
Also records two known limitations left out of scope: the two-phase
recall ceiling and the wiki cache's ID 1..50 probe window.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
