av/harbor v0.1.25

on GitHub

v0.1.25 - KTransformers integration

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

Starting

# [Optional] Pre-build the image
# This is very large, as it's based on pytorch+cuda
# go grab a coffee!
harbor build ktransformers

# Start the service
harbor up ktransformers

Harbor's version was monkey-patched to be compatible with Open WebUI and will appears as ktransformers in the model selector upon successful start.

https://github.com/av/harbor/wiki/ktransformers-webui.png

• Service configuration guide
• Official configuration examples

Full Changelog: v0.1.24...v0.1.25