mw00/project-maya v1.1.26

Project Maya v1.1.26

on GitHub

DeepSeek reads prompts up to 1.5x faster on NVIDIA V100s and faster on every NVIDIA GPU, its long tool calls stream,
and fixes for two-GPU Windows setups and the benchmark.

• DeepSeek prompts on Volta and Turing (V100, RTX 20): the sparse attention now runs on their FP16 tensor cores,
  like RTX 30 and newer since v1.1.24 - twice as fast as before on a V100, as exact as the old path (the held-out
  KL against the original model the same). DS4_ATTN_V7=0 turns it off.
• DeepSeek prompts on every NVIDIA GPU: a prompt's busiest experts are decoded to FP16 once and multiplied by the
  library's GEMM instead of being decoded again for every 64 rows (DS4_FAT_ROWS, 64 rows by default; 0: off), and
  cards of 16-32 GB get wider attention slabs (fewer FP16 decodes of the same weights). Together on three V100s:
  2K-token prompts ~1.3x, 8K and 32K ~1.5x faster; decode the same.
• DeepSeek prompts, more: the expert copies keep going while the host waits for the routing (#360, @OscarSheng);
  prompt skip across a 2-GPU layer split (#343, @Gazelle221B); --tp 1 on one GPU takes the chunked prompt path
  (#320, @pietroMastro92); tensor parallel 2 on Maya's quants, opt-in MAYA_DS4_TP_GGUF_UNEVEN (#408, @pepuscz).
• DeepSeek decode: two cores left free beside the CPU lane where it would take every logical CPU (#359,
  @OscarSheng; with SMT only where it helps, settable with MAYA_DS4_LANE_FREE); the CPU lane prefetches only the
  expert pages not already in RAM - Windows decode +6-7% (#425, @cryptedy).
• GLM prompts on Volta: the prompt attention in one pass (#335, @merbanan) - long prompts ~8% faster on a V100.
• DeepSeek tool calls: a long string argument (a file's content) streams as it is written, so clients with a
  stream-idle timeout no longer give up while the model is still writing (#403).
• Windows, GLM on two GPUs: the RAM tier is pinned in pieces of at most 1 GB, so the second GPU gets its share of
  Windows' pinning budget instead of 0 GB and the start no longer fails (#417). MAYA_GLM_PIN_PIECE_MB=0: one block.
• Fixes: the server's queue no longer stays blocked after an engine restart handed out a request number still in
  use (#330, @jettoblack); --bench sends DeepSeek its own chat format and --report shows DeepSeek's placement and
  speed lines and its pictures setting (#416); --mmap-experts is a setup option kept by later setups (#422).