github tobiaswuerth/sift v1.13.0

one hour ago

sift updates itself on a yes, looks up a shortened id instead of reading every path, and indexes long documents and
large JSON Lines files in a fraction of the time and memory.

Added

  • In a terminal, sift says when a newer version is out and, on y, updates itself with the installer it came by (uv,
    pipx or pip) once no sift of the installation runs any more; the next command says how it went. It asks PyPI once a
    day, in the background; SIFT_UPDATE_CHECK=off or HF_HUB_OFFLINE=1 stops it. Never for serve, worker, or
    output to a pipe. A new major version is only told of, with its changelog. While the installer runs, a command
    started stops and says so; an update that a restart of the machine ended is said and offered again. Where sift
    can't update itself, it says the newer version and the command: in an environment without pip, as uv's are (uv pip install --upgrade sift-rag, or uvx sift-rag@latest), and in an installation you may not write.

Changed

  • Logs no longer grow without end: the latest 20 of each command are kept, and a log past 16 MB, as a server's that
    logs every request, goes on in a new part, its last three kept (.log.1 to .log.3).
  • Indexing a JSON Lines file whose records changed takes half the time: the old versions go together, not one by
    one. 50,000 changed records took 11.4 s to store, and take 5.7 s.
  • A document named otherwise than by its whole id is looked up, where every path of its collection was read: a
    shortened id (papers/2603.29315.md for papers/2026/03/2603.29315.md), a record's id alone, a mistyped id, or an
    absolute path in another case, for get, similar, links, mentions, and the links_to and linked_from of
    searches. Over 300,000 documents that took 16 to 32 ms, and 322 ms for an absolute path not indexed, which read every
    document; it grew with the collection. Now 0.03 to 0.05 ms, whatever the collection holds.
  • The index layout is version 10: a store lists the last part of every document's path, a file's name or a record's
    id, in 23 bytes a document; indexing 300,000 records took 38.5 s instead of 36.9. An index of 1.12 gets the list on
    first use, empty, and the next sift index fills it, 3 million documents in 5 seconds; stopped, it goes on where
    it stopped, and other sifts read and change the collection meanwhile. Until then such ids are searched for as
    before. An older sift doesn't open an index so updated.
  • Updating by hand: stop sift serve and sift worker first. On Windows a running sift holds the files an update
    replaces: uv tool upgrade under a running server upgraded the packages but failed on sift.exe.
  • A filter's value matches whether a document writes it as a number or as text: -f year=2024, and {"year": 2024}
    over MCP, find year: 2024 and "year": "2024" alike, and true and false likewise. Before, only the page in
    the browser asked both ways.
  • A search without an active collection is an error that says what to do (sift activate NAME, or sift add FOLDER
    where none is registered), on the command line, over MCP and on the page. A keyword search answered "no matches".
  • Passages end before a Markdown heading only, not before any line starting with #: a comment in a code block, a C
    include or a tag cut a passage in two. Files already indexed keep their passages until they change.

Fixed

  • A large document took time by the square of its size: each of its passages was cut out of its whole text anew. An
    80 MB text file, 32,795 passages, took 11 minutes to index and takes 16 s; an 8 MB file's passages took 20 s to
    read for sift embed and take 0.01 s; 50 passages of it among a search's results took 0.3 s and take 7 ms; a
    phrase is counted in 200 of them in 0.04 s, not 1.7 s. Documents of a few passages are stored as before, and
    nothing in an index changes.
  • A large JSON Lines file was read whole: one of 300,000 records, 69 MB, took ten times its size in memory. Files over
    32 MB are now read as they are stored, a batch of records at a time: storing that file peaks at 297 MB, against
    681 MB, most of it SQLite's page cache, and no longer grows with the file. They are parsed in the indexing process
    rather than in worker processes. Building the keyword postings afterwards still peaks near 700 MB for it.
  • sift embed beside sift index embedded the chunks indexed meanwhile, but its total stayed as it began: the bar
    went past 100% and the time left below zero. The total now follows the chunks indexed or deleted while it runs.
  • sift ui asked for the server that runs through the proxy set for the web, which knows no server of this machine,
    and took a server busy for five seconds for none: it started a second one on the port taken.
  • A browser kept the page's parts, which share the page's address, and showed one alone on the way back to it. Pages
    now forbid keeping them, which also keeps documents' texts out of the browser's cache.
  • sift search ... | head -1 ended with "error: [Errno 22] Invalid argument" on Windows. A reader that leaves now
    ends the command quietly.
  • What sift says on stderr is UTF-8 now where it wasn't, as its results are: in a Windows pipe, file names and titles
    came out unreadable.
  • Every command ended in an AttributeError where stdout or stderr is no file, as within a program that set its own
    (contextlib.redirect_stdout(io.StringIO())), or where there is none, as under pythonw. Such a stream is left as
    it is now, and a command without streams runs and says nothing.
  • Without a stderr but with a stdout, as pythonw -m sift search ... > results.txt from a console has them, what sift
    says went to stdout, among the results: "no matches", a progress line, "starting ..." before the JSON of --json.
    It is said nowhere now; the command's log holds it as before.
  • A store that was locked or damaged, or a call cancelled, was said as "invalid search query".
  • A bug that is a KeyError or an IndexError came out as a bare message (error: 'fields'); it is reported as a
    bug now, with its type, and its traceback on the command line and in the server's log.
  • On Windows, an index on a network share, or on a drive mapped to one, was made and then never opened again ("unable
    to open database file"): SIFT_HOME there, sift add --store, sift move. SQLite refused the address of a file on
    another machine as sift wrote it.
  • A file named by its absolute path through the folder as it was added, a link or junction to it or a drive mapped to
    a network share, was "in no collection" for reindex, sift index --files and get, and the sift add suggested
    was refused: a collection keeps its folder as it resolves. Such a path is resolved now where it isn't found as given.
  • A range on a number took text without a number for 0: -f "year<2000" found every note whose year was unknown,
    empty, or a list inside a list. Text that starts with a number, as a date, still compares as that number. A range
    given as text, -f "created>2024", compared a list inside a list by its JSON, and found it.
  • A scan skipped as SIFT_OCR was off, or as there was no GPU, was read only once its PDF changed: setting
    SIFT_OCR=on, as the skip says, read none of them. The next sift index after SIFT_OCR is turned on, or from
    off to auto, reads them.
  • reindex and sift index --files failed the whole call, every file of it left as it was, when the index refused
    one file's text, as text over SQLite's limit of 1 GB. That file is "unreadable" now, with why, and stays as it was
    indexed; the others are read, as sift index reads them.
  • sift worker failed a request whose sender's address the server didn't give, and took any number of texts in one,
    holding the GPU for as long as a request of millions took. It takes at most 20,000, the most a query reranks.
  • One collection's index damaged, every page of the UI failed: its collection list now says that index is damaged
    and what to do, and the others' pages work.
  • sift embed and sift serve said "loading PyTorch" through a worker, where they never load it, and so did sift similar anywhere, which runs no model. sift serve said it loaded the models into RAM: it now names where they
    are, the worker's address or the device.
  • sift worker wrote its token and certificate in place: one cut short by a crash or a full disk changed the key other
    machines were given, unsaid. Each is written whole or not at all now.
  • SIFT_HOME=~/sift, as an MCP client's config or Windows' setx passes it, unexpanded, was a folder named ~
    wherever a command ran. ~ is the user's folder now, and a relative home is taken from where the command runs, so
    the update's process, which runs at the root of the drive, keeps the state the commands read.
  • A full disk left update.json.PID.part behind in SIFT_HOME, one more for each command that tried to write the
    update check's state. It goes now, as when the state can't be replaced.
  • The docs said sort orders results by a field "instead of score", which read as the newest of every match first.
    It orders the best k; a range filter on the field finds the newest of all. They also said a mail's date is in
    UTC, which it is only where the mail gives its time zone.

Don't miss a new sift release

NewReleases is sending notifications on new releases.