koboldcpp-1.54

welcome to 2024 edition

• Added logit_bias support (for both OpenAI and Kobold APIs. Accepts a dictionary of key-value pairs, which indicate the token IDs (int) and logit bias (float) to apply for that token. Object format is the same as and compatible with the official OpenAI implementation, though token IDs are model specific. (thanks @DebuggingLife46)
• Updated Lite, added support for custom background images (thanks @Ar57m), and added customizable settings for stepcount and cfgscale for Horde/A1111 image generation.
• Added mouseover tooltips for all labels in the GUI launcher.
• Cleaned up and simplified the UI of the quick launch tab in the GUI launcher, some advanced options moved to other tabs.
• Bug fixes for garbled output in Termux with q5k Phi
• Fixed paged memory fallback when pinned memory alloc fails while not using mmap.
• Attempt to fix on-exit segfault on some Linux systems.
• Updated KAI United class.py, added new parameters.
• Makefile fix for Linux CI build using conda (thanks @henk717)
• Merged new improvements and fixes from upstream llama.cpp (includes VMM pool support)
• Included prebuilt binary for no-cuda Linux as well.
• Various minor fixes.

To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
If you don't need CUDA, you can use koboldcpp_nocuda.exe which is much smaller.
If you're using AMD, you can try koboldcpp_rocm at YellowRoseCx's fork here

Run it from the command line with the desired launch parameters (see --help), or manually select the model in the GUI.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001

For more information, be sure to run the program from command line with the --help flag.