koboldcpp-1.4
- This is an expedited bugfix release because the new model formats were breaking on large contexts.
- Also people have requested mmap to be the default, so now it is, you can disable it with --nommap
To use, download and run the koboldcpp.exe, which is a one-file pyinstaller.
Alternatively, drag and drop a compatible ggml model on top of the .exe, or run it and manually select the model in the popup dialog.
and then once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001
Alternative Options:
None are provided for this release as it is a temporary one.