The search index stays merged, and a common word is found without sorting every match: on a
real account cache of 1.16M messages, tg messages search for a word in 9–19% of the messages
took 180–290 ms with 0.11.1 and now takes about 60 ms, most of it starting tg (#149).
Changed
tg messages searchreturns the newest page of a word in a few percent of the messages
without sorting every match: from a word in about 3% of 1M messages or 1% of 10M, instead of
21% and 7%. It asks the index about runs of the newest messages instead of one message at a
time, and gives up and sorts once it has taken a quarter of what sorting takes, so a word whose
matches are all old takes at most about 1.2 times as long as before. On a made-up cache of 1M
messages a word in 7% of them took 85 ms and now takes 6.7 ms; the fifth page of a common
word (--offset 200) 278 ms and 6.4 ms. Results and their order don't change (#149).
Fixed
- The search index no longer keeps the old entries of re-indexed and deleted messages. After
the one-off re-index of names in 0.11.0, a cache could keep the old entry of every message: an
index twice its size, and asking it whether a message matches took milliseconds instead of
microseconds. The daemon andtg db migratenow merge the index in short batches after
writes, and merge it whole once a tenth of its entries are deleted; a cache left that way is
cleaned once (2.6 s for 1M messages, 202 MB to 104 MB) (#149). - Writes no longer merge the search index themselves, which sometimes held the write lock for
hundreds of ms on a big cache, up to 1 s during the one-off re-index of names. On 1M
messages the longest batch of that re-index took 32 ms instead of 997 ms, and live messages
are written about 45% faster; the merges run in the background between other writes (#142).