LostRuins/koboldcpp v1.19

koboldcpp-1.19.1

on GitHub

koboldcpp-1.19

• Integrate --usemirostat option for all model types. This must be set at launch, and replaces your normal stochastic samplers with mirostat. Takes 3 params [type][tau][eta], e.g. --usemirostat 2 5.0 0.1 Works on all models, but noticeably bad on smaller ones. Follows the upstream implementation. More info here.

• Added an option --forceversion [ver]. If the model file format detection fails (e.g. A rogue modified model) you can set this to override the detected format (enter desired version, e.g. 401 for GPTNeoX-Type2).

• Added an option --blasthreads, which controls threads when ClBlast is active. Some people wanted to use a different thread count when CLBlast was active and got overall speedups, so now you can experiment. Uses the same value as --threads if not specified.

• Integrated new improvements for RWKV. This provides support for all the new RWKV quantizations, but drops support for Q4_1_O following the upstream - this way I only need to maintain one library. RWKV q5_1 should be much faster than fp16 but perform similarly.

• Bumped up the buffer size slightly to support Chinese alpaca.

• Integrated upstream changes and improvements, various small fixes and optimizations.

• Fixed a bug where GPU device was set incorrectly in clblast

• Special: An experimental Windows 7 Compatible .exe is included for this release, to attempt to provide support for older OS. Let me know if it works (for those still stuck on Win7). Don't expect it to be in every release though.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
Alternatively, drag and drop a compatible ggml model on top of the .exe, or run it and manually select the model in the popup dialog.

and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program with the --help flag.