github mw00/project-maya v1.0.9
Project Maya v1.0.9

latest releases: v1.1.74, v1.1.73, v1.1.72...
one month ago

Switching between conversations no longer re-reads them: Maya keeps your last few on the SSD.

Update: git pull, then ./setup.sh (Windows: START-MAYA.bat). It recompiles only what changed and starts.

What's new

  • Conversation slots: when a request doesn't continue the conversation the engine holds (another chat, or an agent tool's side request such as a title or a sub-task), that conversation is first set aside in a file on the SSD. A later request that continues it takes it back instead of reading its whole prompt again.
    • Measured on 1x Tesla V100 with Maya-S, going back to a 3,500-token conversation after another one: prefill (reading the prompt) 10.2 s -> 2.2 s. Setting a conversation aside takes about 0.1 s, taking it back about 0.06 s. With two GPUs, every half's state goes into the file, the MTP block's included.
    • Size: about 0.15 GB plus 21 KB a token per conversation (0.2 GB at 3K tokens, 0.8 GB at 30K).
    • Checks: a conversation taken back from its file continues token for token as it would from memory. Each conversation still recalls its own details after others ran in between.
    • Limits: requests still run one at a time. Up to 4 conversations of 1,024 tokens or more, 16 GB on disk, never below 8 GB free. STRATA_GLM_SLOTS=0 turns it off (README > Tuning).
  • Prefill: the disk read-ahead buffer grows to 3% of free RAM, up to 96 experts (was 2%, up to 64). On two GPUs the second one no longer waits on the SSD between layers: 2x Tesla V100 with 30 GB RAM, +1% prefill. With 64 GB RAM (fewer SSD reads) there's no change: 261 / 359 / 351 vs 261 / 358 / 349 tokens/s at 2k / 8k / 16k.
  • README: a table of speeds measured by users with --bench, starting with 2x TITAN RTX.
  • --bench / --report: a prompt-only request's engine line no longer shows a decode (writing the answer) speed in every case where it has none.

Don't miss a new project-maya release

NewReleases is sending notifications on new releases.