Several documents or queries per call, the vectors scanned three times as fast by sift serve, and no file left out
for good behind a record's path.
Added
gettakes up to 100 documents at once, each an id or{"id", "lines"}; an unreadable one comes back with its
error, and texts past 100,000 characters in all are left out, each saying why.sift gettakes several ids too.search,vsearchandquerytake a list of up to 20 queries and return a list of results; the queries of
vsearchandqueryare embedded in one pass, those ofsearchshare itsbudget.
Changed
sift servescans the vectors' codes three times as fast. sift now depends on Numba;
the server andsift benchload the compiled scan at start (a second or two, cached inSIFT_HOME/numba). A single
command-line search, or a server that can't load Numba (it says so), uses NumPy.queryreranks in batches of 16 sorted by length: 180 ms a question instead of 331.- A server keeps the embeddings of the last 256 questions and the results of the last 32 searches, so asking again
with anotherkor filter skips the embedding. Results are dropped when anything they read changes; searches with a
time budget, over 1,000 hits, or over an unreadable collection are never kept. queryover collections not embedded yet no longer loads the model.- The documentation: the README is the entry point, and the details are in
docs/(usage, worker, architecture,
benchmarks, contributing, security), each fact in one place.
Fixed
- A semantic search of one collection (
collection,-c) read the codes of every active collection first (half a
minute and 4 GB beside a collection of 30 million passages); it reads only those it searches. - A server embedding through a worker (
SIFT_EMBED_REMOTE) answered nothing while a request waited, up to ten minutes
if the worker's machine slept; cancelled calls now stop waiting. - A file whose path a JSON Lines record or mail attachment had taken (
data.jsonlwith a recorda.md, beside
data/a.md) was skipped for good; it is now read as soon as the path is free. On upgrade, stores re-read once the
files without documents and the JSON Lines files and mails with a file below their names; other left-out records
come back when their file changes. A store sift may only read is upgraded once it may write. sift indexmessages under its progress line are now in its log.- Front-matter JSON strings no longer end at U+2028, U+2029 or NEL; a NUL in a file no longer hides the text after it
(stored as U+FFFD; documents indexed before keep the cut until their text changes);.siftignoreno longer ends
patterns at U+2028, form feeds or NEL;getworks for ids holding a NUL. - A mail dated in year 9999 with a time zone, or in UTF-7 holding half a character, is no longer skipped; a UTF-32
file with a byte order mark is no longer read as UTF-16. getof several documents no longer fails for all when one's store is out of reach.- With the GPU hidden on purpose (
CUDA_VISIBLE_DEVICES=-1), sift no longer says to update the NVIDIA driver. - One damaged store (a table broken though the store opens) stopped
sift index,statusand the server's start for
every collection; it is passed over and said, with how to forget it. A search that reads its damaged tables fails. - Ctrl+C or an error while the keyword postings were built in-process left an undeletable temporary file on Windows.