turboderp-org/exllamav3 v0.0.27

0.0.27

on GitHub

• New and more robust allocation strategy for non-integer bitrates
• Added -hq argument to quantizer (explanation here)
• Fix bug causing prompt caching to fail on recurrent models for certain combinations of prompt length and chunk size
• Fix broken output when using repetition penalties without decay range (affecting some OAI clients via TabbyAPI)
• Fix issue allowing recurrent state to fall out of sync with K/V cache
• Support more features in Nanochat, for some reason
• Other fixes and QoL improvements

Full Changelog: v0.0.26...v0.0.27