github tobiaswuerth/sift v1.12.0

one hour ago

Several documents or queries per call, the vectors scanned three times as fast by sift serve, and no file left out
for good behind a record's path.

Added

  • get takes up to 100 documents at once, each an id or {"id", "lines"}; an unreadable one comes back with its
    error, and texts past 100,000 characters in all are left out, each saying why. sift get takes several ids too.
  • search, vsearch and query take a list of up to 20 queries and return a list of results; the queries of
    vsearch and query are embedded in one pass, those of search share its budget.

Changed

  • sift serve scans the vectors' codes three times as fast. sift now depends on Numba;
    the server and sift bench load the compiled scan at start (a second or two, cached in SIFT_HOME/numba). A single
    command-line search, or a server that can't load Numba (it says so), uses NumPy.
  • query reranks in batches of 16 sorted by length: 180 ms a question instead of 331.
  • A server keeps the embeddings of the last 256 questions and the results of the last 32 searches, so asking again
    with another k or filter skips the embedding. Results are dropped when anything they read changes; searches with a
    time budget, over 1,000 hits, or over an unreadable collection are never kept.
  • query over collections not embedded yet no longer loads the model.
  • The documentation: the README is the entry point, and the details are in docs/ (usage, worker, architecture,
    benchmarks, contributing, security), each fact in one place.

Fixed

  • A semantic search of one collection (collection, -c) read the codes of every active collection first (half a
    minute and 4 GB beside a collection of 30 million passages); it reads only those it searches.
  • A server embedding through a worker (SIFT_EMBED_REMOTE) answered nothing while a request waited, up to ten minutes
    if the worker's machine slept; cancelled calls now stop waiting.
  • A file whose path a JSON Lines record or mail attachment had taken (data.jsonl with a record a.md, beside
    data/a.md) was skipped for good; it is now read as soon as the path is free. On upgrade, stores re-read once the
    files without documents and the JSON Lines files and mails with a file below their names; other left-out records
    come back when their file changes. A store sift may only read is upgraded once it may write.
  • sift index messages under its progress line are now in its log.
  • Front-matter JSON strings no longer end at U+2028, U+2029 or NEL; a NUL in a file no longer hides the text after it
    (stored as U+FFFD; documents indexed before keep the cut until their text changes); .siftignore no longer ends
    patterns at U+2028, form feeds or NEL; get works for ids holding a NUL.
  • A mail dated in year 9999 with a time zone, or in UTF-7 holding half a character, is no longer skipped; a UTF-32
    file with a byte order mark is no longer read as UTF-16.
  • get of several documents no longer fails for all when one's store is out of reach.
  • With the GPU hidden on purpose (CUDA_VISIBLE_DEVICES=-1), sift no longer says to update the NVIDIA driver.
  • One damaged store (a table broken though the store opens) stopped sift index, status and the server's start for
    every collection; it is passed over and said, with how to forget it. A search that reads its damaged tables fails.
  • Ctrl+C or an error while the keyword postings were built in-process left an undeletable temporary file on Windows.

Don't miss a new sift release

NewReleases is sending notifications on new releases.