The EXL3 quant works with the DFlash2 drafter, and on Windows + AMD a conversation taken back from a slot answers its own question again.
Update: the dashboard's About > Update, or git pull, then ./setup.sh (Windows: START-MAYA.bat). The engine recompiles; nothing is downloaded again.
What's new
- EXL3 with the DFlash2 drafter (#141 by @needmorevram). The drafter's verify window read each token's embedding at the wrong row size for EXL3's BF16 table, so the answers turned to noise (#129, #140, #142), and on one and two GPUs the window had no kernels for EXL3's experts. Both are fixed. A prompt asked again no longer drafts from the previous answer, with every model. On two GPUs EXL3 decodes faster with the MTP block for now, and the speed calibration keeps it there.
- Windows + AMD: a conversation taken back from a slot answered the other conversation's question, then looped (#133). A new conversation's state was zeroed on the GPU's default queue, which nothing made the engine's own queues wait for, and Windows' HIP runtime holds queued work until the program waits on the GPU: the next conversation started from the previous one's state. The state is now zeroed on the engine's compute queue and waited for.
Checked
- EXL3 3.05bpw with the DFlash2 drafter on a Tesla V100: v1.1.3 stopped with "glm window: the batched experts (types 200 / 200)"; v1.1.4 answers, token for token what the model writes without the drafter.
- GGUF models with DFlash2 and with the MTP block: the same answers as v1.1.3 on one GPU, and the same decode speed on two.
- Windows + AMD (Radeon 8060S): the reported slot sequences now give each conversation its own answer, and prompt and decode speed are those of v1.1.3.
- The engine builds on Linux, Windows and AMD; the parity tests and the GitHub checks pass.