github mw00/project-maya v1.0.10
Project Maya v1.0.10

latest releases: v1.1.74, v1.1.73, v1.1.72...
one month ago

RTX 20-series cards now read prompts on their tensor cores too.

Update: git pull, then ./setup.sh (Windows: START-MAYA.bat). It recompiles only what changed and starts.

What's new

  • Prefill (reading the prompt) on Turing (RTX 20-series, Titan RTX, Quadro RTX): the tensor-core prompt attention now keeps its context in registers on these cards, so it fits their 64 KB of shared memory. They had been using the slower F32 kernel.
    • 2x TITAN RTX, an 18,476-token prompt: 62 s -> 58 s of prefill. By @dummerjindabin (#10).
    • Other cards keep the kernel they had. This variant's arithmetic is only correct on Turing and newer (a V100 got the attention wrong with it), so Maya uses it only where the other kernel doesn't fit. A V100 gives the same tokens as v1.0.9.
  • Prefill for 4-bit (Q4_K) weights: they're converted for the tensor cores the fast way, as 5- and 6-bit weights already were, with the same values. A test quant with 4-bit attention on 1x V100: 224 / 291 / 285 -> 268 / 368 / 358 tokens/s at 2k / 8k / 16k, the same tokens. Maya-S and Maya-M are unchanged.

Don't miss a new project-maya release

NewReleases is sending notifications on new releases.