Switching between conversations no longer re-reads them: Maya keeps your last few on the SSD.
Update: git pull, then ./setup.sh (Windows: START-MAYA.bat). It recompiles only what changed and starts.
What's new
- Conversation slots: when a request doesn't continue the conversation the engine holds (another chat, or an agent tool's side request such as a title or a sub-task), that conversation is first set aside in a file on the SSD. A later request that continues it takes it back instead of reading its whole prompt again.
- Measured on 1x Tesla V100 with Maya-S, going back to a 3,500-token conversation after another one: prefill (reading the prompt) 10.2 s -> 2.2 s. Setting a conversation aside takes about 0.1 s, taking it back about 0.06 s. With two GPUs, every half's state goes into the file, the MTP block's included.
- Size: about 0.15 GB plus 21 KB a token per conversation (0.2 GB at 3K tokens, 0.8 GB at 30K).
- Checks: a conversation taken back from its file continues token for token as it would from memory. Each conversation still recalls its own details after others ran in between.
- Limits: requests still run one at a time. Up to 4 conversations of 1,024 tokens or more, 16 GB on disk, never below 8 GB free.
STRATA_GLM_SLOTS=0turns it off (README > Tuning).
- Prefill: the disk read-ahead buffer grows to 3% of free RAM, up to 96 experts (was 2%, up to 64). On two GPUs the second one no longer waits on the SSD between layers: 2x Tesla V100 with 30 GB RAM, +1% prefill. With 64 GB RAM (fewer SSD reads) there's no change: 261 / 359 / 351 vs 261 / 358 / 349 tokens/s at 2k / 8k / 16k.
- README: a table of speeds measured by users with
--bench, starting with 2x TITAN RTX. --bench/--report: a prompt-only request's engine line no longer shows a decode (writing the answer) speed in every case where it has none.