Highlights
- Fast JSON Decoding (blog)
- Output logprobs for decoding tokens
- Multiple bug fixes
What's Changed
- Fix no-cache mode by @Ying1123 in #136
- Support Faster JSON decoding for llava by @hnyls2002 in #137
- fix undfined variable by @yaya-sy in #142
- jump-forward rename by @hnyls2002 in #144
- Add warmup to SRT server by @comaniac in #146
- add openai error handler with retry and logger by @ChuyueSun in #148
- Temporary fix OpenAI API for Pydantic v1/v2 by @comaniac in #153
- Add gptq quantization model support by @Arcmoon-Hu in #141
- Support decode token logprobs by @comaniac in #130
- Format code & move functions by @merrymercy in #155
- [Submodule] Change FlashInfer to import by @comaniac in #156
- add
--disable-disk-cacheby @hnyls2002 in #160 - Add Auth Token to RuntimeEndpoint by @nivibilla in #162
- Fix BaseCache metric by @comaniac in #170
- import outlines by @hnyls2002 in #168
- Fix token usage with jump forward by @comaniac in #174
- Support extra field regex in OpenAI API by @comaniac in #172
- Fix the chat template for llava-v1.6-34b & format code by @merrymercy in #177
- Update version to 0.1.12 by @merrymercy in #178
New Contributors
- @yaya-sy made their first contribution in #142
- @ChuyueSun made their first contribution in #148
- @nivibilla made their first contribution in #162
Full Changelog: v0.1.11...v0.1.12