Keyword postings in topical order, quicker filtered and exact semantic search, one hit per paper, and fewer results
for agents when few stand out.
Added
- A paper that several collections hold, arXiv and PubMed say, is one hit: the best ranked copy, which lists the
others underalso. Copies share a DOI (front-matter fielddoi), compared in any case and without a link's
prefix.search,vsearchandsimilarlook further down until they havekpapers, andquerykeeps one hit
per paper of the candidates it reranks;similarleaves out the paper's own copies;all_passageskeeps every
copy.sift benchcounts the document sought as found when a hit lists it underalso. adaptive(--adaptive) for every search: up tokhits, those before the largest drop in score and five more
(Adaptive-k, Taguchi et al. 2025). Over 2,000 arXiv titles, vsearch returned 6 of 10 or 50 on average, and kept the
paper sought in 99.2 to 99.6% of the cases it was among them.budget(--budget SECONDS) for keyword search: once spent, and anything found, a search from the postings stops
and returns the best found so far, each hit marked"partial": true. Without it, results stay exact.
Changed
- The keyword postings number passages in a topical order, clusters of their vectors' codes, so that a range of
postings holds passages that share their words and its bound is tight. Over arXiv, questions scored a half to a
twentieth as many ranges, the slowest took 45 ms instead of 90, with the same results, and the postings took 8% less
space; single words took 0.7 ms instead of 0.3. The index layout goes to 9 by itself; the nextsift indexbuilds a
collection's postings anew once it has vectors, and again once a fifth of its passages got vectors since. A sift
1.10 still running, which can't read the order, searches such a collection through FTS5. vsearchranks the 10,000 nearest codes again by the query's own values against their bits, and reads and scores
the best 1,000 exactly instead of the 4,000 nearest: over 3.2 million arXiv abstracts it finds 99.75% of exact
search's top 10 instead of 99.6%, and reads a quarter of the vectors (7 ms instead of 25, warm). Candidates the same
distance away are taken by id, so the result no longer depends on how codes are split among collections.- A filtered
vsearch,queryorsimilarchecks the nearest candidates against the filter, a batch at a time,
instead of listing every passage it lets through, which read every document: a category filter over arXiv that
lets 9% through took 0.6 seconds instead of 6 warm (56 cold). A filter that lets too few through for that, 1% say,
is listed, as before.
Fixed
- A keyword search reads the postings in one transaction, as they were when it began: another sift building them anew
meanwhile made it fail (as in 1.10), and, in a new order, find other passages than it scored.