A new fit selector strategy for capacity-aware model selection, some UI polish and a fix for the failing Docker builds.
Ohh! A new type of selector that allows loading variations of a model that best
fits in the currently available resources. Sometimes the full fat version of a
model isn't the right choice and you want a smaller version, perhaps with less
context or a lower quant. The new "fit" selector combined with the capacity
router to the rescue! Now llama-swap can choose a variation that can fit in the
currently available resources to serve the request. Thanks to @GitEventhandler
for the feature request!
This release also includes some UI changes:
- Support pasting of images into the Playground chat, thanks @kaczy44
- Improvements to the capture view to split the summary, request and responses. Less scrolling yay.
- Activity token counts adapt to screen size. Great for users who have send more than a hundred million tokens through llama-swap!
Also some docker container build changes. Thanks Opus, you're a better SRE than I ever was!
- PR #1332 ui: reload when a lazy route chunk fails to load: reloads the page once when a lazily loaded route chunk fails, because a tab left open across an upgrade asks for old hashed chunk names, the server 404s them as text/plain and the page hung on its loading spinner (guarded with a 10 s sessionStorage timestamp to avoid reload loops) by @mostlygeek
- a36540b docker/unified: optimize build with ccache: adds ccache to speed up unified image compiles and reduces ARM64 builds to GB10 only, to get the GitHub image builds passing again after they started failing on Nov 6 by @mostlygeek
- PR #1334 ui: support pasting images into Playground chat: pastes screenshots and other images straight into the Playground chat message box with Ctrl+V, previewing them alongside file-picker attachments so they can be reviewed or removed before sending (plain text paste is unchanged) by @kaczy44
- PR #1339 ui: split capture dialog into summary/request/response tabs: reworks the capture dialog into Summary, Request and Response tabs, because stacking request/response headers and bodies in one column always buried the body of interest under the section above it by @mostlygeek
- PR #1340 Add fit selector strategy for capacity-aware model selection: adds a
fitselector strategy that routes each request to the target that is cheapest to run right now, ranked by lowest totalevict_costof the models it would unload, then fewest models unloaded, then readiness, thentargetsorder, along withsettings.no_evict(429 instead of evicting), read-onlyLocalRouter.Evictions/Scheduler.Evictionsqueries, validation,/v1/modelsmetadata and a "no evict" tag in the Models page; it requiresrouting.router.use: capacityand other selectors keep working when it is misconfigured by @mostlygeek - PR #1341 ui: use compact numbers in Activity stats on small screens: shows compact units such as 309.7M (full value on hover) below
lgand lays the stats out in a 2x2 grid belowsm, because large comma-separated token counts overlapped each other on narrow screens by @mostlygeek