Maintainer note: This is the v256 Big-F-Release. I've been very busy and many small improvements have been squatting in main waiting to be unleashed. A few highlights:
Playground Chat UI improvements
- tightened up the margins and cleaned up the design to be more mobile friendly
- more timings and stats than you can shake a stick at
- improved system prompt and added more hyper param controls (do who uses these anymore?)
Thanks to contributors:
- JoC - Started the model idle-time window after models become ready or restart PR #1095.
- Kris - Added live generation statistics to Chat PR #1099.
Changelog
- 6701d0d changelog: v256
- 2409272 CHANGELOG.md,docs: fix pull request link
- b98a8ff changelog: add scripts and automation to generate changelog entries (#1125)
- 40b36d9 ui: make the playground chat responsive and lighter (#1120)
- 0e1f797 internal/server: reduce zstd pool memory consumption (#1123)
- aecf92e internal/store: split repository interfaces from the sqlite backend (#1122)
- 21bc145 process: start TTL idle window when model becomes ready (#1095)
- 41ec321 internal/server: add global concurrency semaphore (#1110)
- e1526c8 cmd/vllm-wrapper: use default timeouts
- 8fa8589 internal/server: add tabbyapi usage data extraction (#1104)
- 1686326 ui: show chat stats in Help agent
- acb7759 ui: show live generation stats in Chat (#1099)
- 6b18115 ui: keep Work status bar stable while streaming (#1101)