koboldcpp-1.70

mom: we have ChatGPT at home edition

• Updated Kobold Lite:
  • Introducting Corpo Mode: A new beginner friendly UI theme that aims to emulate the ChatGPT look and feel closely, providing a clean, simple and minimalistic interface. It has a limited feature set compared to other UI themes, but should feel very familiar and intuitive for new users. Now available for instruct mode!
  • Settings Menu Rework: The settings menu has also been completely overhauled into 4 distinct panels, and should feel a lot less cramped now, especially on desktop.
  • Sampler Presets and Instruct Presets have been updated and modernized.
  • Added support for importing character cards from aicharactercards.com
  • Added copy for code blocks
  • Added support for dedicated System Tag and System Prompt (you are still encouraged to use the Memory feature instead)
  • Improved accessibility, keyboard tab navigation and screen reader support
• NEW: DRY dynamic N-gram anti-repetition sampler support has been added (credits @pi6am)
• Added --unpack, a new self-extraction feature that allows KoboldCpp binary releases to be unpacked into an empty directory. This allows easy modification and access to the files and contents embedded inside the PyInstaller. Can also be used in the GUI launcher.
• Fix for a Vulkan regression in Q4_K_S mistral models when offloading to GPU (thanks @0cc4m).
• Experimental support for OpenAI tools and function calling API (credits @teddybear082)
• Added a workaround for Deepseek crashing due to unicode decoding issues.
• --chatcompletionsadapter can now be selected on included pre-bundled templates by filename, e.g. Llama-3.json, pre-bundled templates have also been updated for correctness (thanks @xzuyn).
• Default --contextsize is finally increased to 4096, default Chat Completions API output length is also increased.
• Merged fixes and improvements from upstream, including multiple Gemma fixes.

To use on Windows, download and run the koboldcpp_rocm.exe OR download koboldcpp_rocm_files.zip and run python koboldcpp.py from Window Terminal or CMD (additional python pip modules might need installed, like customtkinter and tk or python-tk.

To use on Linux, clone the repo or download Source Code (tar.gz or zip) and build with make LLAMA_HIPBLAS=1 -j4 (-j4 can be adjusted to your number of CPU threads for faster build times)

Run it from the command line with the desired launch parameters (see --help), or use the GUI by launching with python koboldcpp.py ((additional python pip modules might need installed, like customtkinter and tk or python-tk).

Once loaded, you can visit the following URL or use it as the API URL for other front-ends like Silly Tavern: http://localhost:5001/

For more information, be sure to run the program from command line with the --help flag.