koboldcpp-1.7.1
- This release brings an exciting new feature
--smartcontext, this mode provides a way of prompt context manipulation that avoids frequent context recalculation. - Merged optimizations from upstream
- Updated embedded Kobold Lite to v20.
- Edit: A hotfix was deployed that fixed a tiny error in context calculation. The exe has been updated. If you downloaded 1.7 before it, please download it again.
To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
Alternatively, drag and drop a compatible ggml model on top of the .exe, or run it and manually select the model in the popup dialog.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001
For more information, be sure to run the program with the --help flag.
Alternative Options:
Non-AVX2 version now included in the same .exe file, enable with --noavx2 flags