koboldcpp-1.8.1
- Another amazing improvement by @0cc4m, CLBlast now does the 4bit dequantization on GPU! That translates to about a 20% speed increase when using CLBlast for me, and should be a very welcome improvement. To use it, run with
--useclblast [platform_id] [device_id](you may have to figure out the values for your correct GPU through trial and error) - Merged fixes and optimizations from upstream
- Fixed a compile error in OSX
To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
Alternatively, drag and drop a compatible ggml model on top of the .exe, or run it and manually select the model in the popup dialog.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001
For more information, be sure to run the program with the --help flag.
Alternative Options:
Non-AVX2 version now included in the same .exe file, enable with --noavx2 flags