sift updates itself on a yes, looks up a shortened id instead of reading every path, and indexes long documents and
large JSON Lines files in a fraction of the time and memory.
Added
- In a terminal, sift says when a newer version is out and, on
y, updates itself with the installer it came by (uv,
pipx or pip) once no sift of the installation runs any more; the next command says how it went. It asks PyPI once a
day, in the background;SIFT_UPDATE_CHECK=offorHF_HUB_OFFLINE=1stops it. Never forserve,worker, or
output to a pipe. A new major version is only told of, with its changelog. While the installer runs, a command
started stops and says so; an update that a restart of the machine ended is said and offered again. Where sift
can't update itself, it says the newer version and the command: in an environment without pip, as uv's are (uv pip install --upgrade sift-rag, oruvx sift-rag@latest), and in an installation you may not write.
Changed
- Logs no longer grow without end: the latest 20 of each command are kept, and a log past 16 MB, as a server's that
logs every request, goes on in a new part, its last three kept (.log.1to.log.3). - Indexing a JSON Lines file whose records changed takes half the time: the old versions go together, not one by
one. 50,000 changed records took 11.4 s to store, and take 5.7 s. - A document named otherwise than by its whole id is looked up, where every path of its collection was read: a
shortened id (papers/2603.29315.mdforpapers/2026/03/2603.29315.md), a record's id alone, a mistyped id, or an
absolute path in another case, forget,similar,links,mentions, and thelinks_toandlinked_fromof
searches. Over 300,000 documents that took 16 to 32 ms, and 322 ms for an absolute path not indexed, which read every
document; it grew with the collection. Now 0.03 to 0.05 ms, whatever the collection holds. - The index layout is version 10: a store lists the last part of every document's path, a file's name or a record's
id, in 23 bytes a document; indexing 300,000 records took 38.5 s instead of 36.9. An index of 1.12 gets the list on
first use, empty, and the nextsift indexfills it, 3 million documents in 5 seconds; stopped, it goes on where
it stopped, and other sifts read and change the collection meanwhile. Until then such ids are searched for as
before. An older sift doesn't open an index so updated. - Updating by hand: stop
sift serveandsift workerfirst. On Windows a running sift holds the files an update
replaces:uv tool upgradeunder a running server upgraded the packages but failed onsift.exe. - A filter's value matches whether a document writes it as a number or as text:
-f year=2024, and{"year": 2024}
over MCP, findyear: 2024and"year": "2024"alike, andtrueandfalselikewise. Before, only the page in
the browser asked both ways. - A search without an active collection is an error that says what to do (
sift activate NAME, orsift add FOLDER
where none is registered), on the command line, over MCP and on the page. A keyword search answered "no matches". - Passages end before a Markdown heading only, not before any line starting with
#: a comment in a code block, a C
include or a tag cut a passage in two. Files already indexed keep their passages until they change.
Fixed
- A large document took time by the square of its size: each of its passages was cut out of its whole text anew. An
80 MB text file, 32,795 passages, took 11 minutes to index and takes 16 s; an 8 MB file's passages took 20 s to
read forsift embedand take 0.01 s; 50 passages of it among a search's results took 0.3 s and take 7 ms; a
phrase is counted in 200 of them in 0.04 s, not 1.7 s. Documents of a few passages are stored as before, and
nothing in an index changes. - A large JSON Lines file was read whole: one of 300,000 records, 69 MB, took ten times its size in memory. Files over
32 MB are now read as they are stored, a batch of records at a time: storing that file peaks at 297 MB, against
681 MB, most of it SQLite's page cache, and no longer grows with the file. They are parsed in the indexing process
rather than in worker processes. Building the keyword postings afterwards still peaks near 700 MB for it. sift embedbesidesift indexembedded the chunks indexed meanwhile, but its total stayed as it began: the bar
went past 100% and the time left below zero. The total now follows the chunks indexed or deleted while it runs.sift uiasked for the server that runs through the proxy set for the web, which knows no server of this machine,
and took a server busy for five seconds for none: it started a second one on the port taken.- A browser kept the page's parts, which share the page's address, and showed one alone on the way back to it. Pages
now forbid keeping them, which also keeps documents' texts out of the browser's cache. sift search ... | head -1ended with "error: [Errno 22] Invalid argument" on Windows. A reader that leaves now
ends the command quietly.- What sift says on stderr is UTF-8 now where it wasn't, as its results are: in a Windows pipe, file names and titles
came out unreadable. - Every command ended in an
AttributeErrorwhere stdout or stderr is no file, as within a program that set its own
(contextlib.redirect_stdout(io.StringIO())), or where there is none, as underpythonw. Such a stream is left as
it is now, and a command without streams runs and says nothing. - Without a stderr but with a stdout, as
pythonw -m sift search ... > results.txtfrom a console has them, what sift
says went to stdout, among the results: "no matches", a progress line, "starting ..." before the JSON of--json.
It is said nowhere now; the command's log holds it as before. - A store that was locked or damaged, or a call cancelled, was said as "invalid search query".
- A bug that is a
KeyErroror anIndexErrorcame out as a bare message (error: 'fields'); it is reported as a
bug now, with its type, and its traceback on the command line and in the server's log. - On Windows, an index on a network share, or on a drive mapped to one, was made and then never opened again ("unable
to open database file"):SIFT_HOMEthere,sift add --store,sift move. SQLite refused the address of a file on
another machine as sift wrote it. - A file named by its absolute path through the folder as it was added, a link or junction to it or a drive mapped to
a network share, was "in no collection" forreindex,sift index --filesandget, and thesift addsuggested
was refused: a collection keeps its folder as it resolves. Such a path is resolved now where it isn't found as given. - A range on a number took text without a number for 0:
-f "year<2000"found every note whose year wasunknown,
empty, or a list inside a list. Text that starts with a number, as a date, still compares as that number. A range
given as text,-f "created>2024", compared a list inside a list by its JSON, and found it. - A scan skipped as
SIFT_OCRwas off, or as there was no GPU, was read only once its PDF changed: setting
SIFT_OCR=on, as the skip says, read none of them. The nextsift indexafterSIFT_OCRis turned on, or from
offtoauto, reads them. reindexandsift index --filesfailed the whole call, every file of it left as it was, when the index refused
one file's text, as text over SQLite's limit of 1 GB. That file is "unreadable" now, with why, and stays as it was
indexed; the others are read, assift indexreads them.sift workerfailed a request whose sender's address the server didn't give, and took any number of texts in one,
holding the GPU for as long as a request of millions took. It takes at most 20,000, the most aqueryreranks.- One collection's index damaged, every page of the UI failed: its collection list now says that index is damaged
and what to do, and the others' pages work. sift embedandsift servesaid "loading PyTorch" through a worker, where they never load it, and so didsift similaranywhere, which runs no model.sift servesaid it loaded the models into RAM: it now names where they
are, the worker's address or the device.sift workerwrote its token and certificate in place: one cut short by a crash or a full disk changed the key other
machines were given, unsaid. Each is written whole or not at all now.SIFT_HOME=~/sift, as an MCP client's config or Windows'setxpasses it, unexpanded, was a folder named~
wherever a command ran.~is the user's folder now, and a relative home is taken from where the command runs, so
the update's process, which runs at the root of the drive, keeps the state the commands read.- A full disk left
update.json.PID.partbehind inSIFT_HOME, one more for each command that tried to write the
update check's state. It goes now, as when the state can't be replaced. - The docs said
sortorders results by a field "instead of score", which read as the newest of every match first.
It orders the bestk; a range filter on the field finds the newest of all. They also said a mail'sdateis in
UTC, which it is only where the mail gives its time zone.