Faster prompt reading for DeepSeek on several GPUs, no more re-reading after the thinking budget, and fixes.
- DeepSeek on several GPUs, with some experts read from the disk or RAM: prompts are read in 4K-token pieces by default, which reads an 8K prompt about 24-30% faster on 3x V100 and leaves room for about 3.5% more experts on the GPUs.
--prefill-chunkstill sets it by hand. - DeepSeek's thinking budget: when the budget closes the thinking, the next turn no longer reads the whole thinking again (59 s at 32K tokens before); it continues from the cache (#746).
MAYA_DS4_READ_ORDER=arrival: a running answer keeps writing while another long prompt is read, instead of stopping until the read ends (#751).- Setup screen: no longer stuck at "Loading the model" when the model is already serving (#729).
--engine strata(the old name in older run scripts) starts the real engine again; v1.1.47 started the test engine instead (#770, thanks @cresslank, #771).tools/ds4/slot_keep_test.pyruns on Windows (#743).
Maya is free; if it helps you, you can sponsor it: https://github.com/sponsors/mw00