Highlights
- Serve the official release demo of LLaVA v1.6 blog
- Support Yi-VL example
- Faster JSON decoding blog
- Support QWen 2
What's Changed
- Fix the error message and dependency of openai backend by @merrymercy in #71
- Add an async example by @Ying1123 in #37
- Add a note about triton version for older GPUs by @merrymercy in #72
- Support load fine-tuned LLaVA model by @isaac-vidas in #80
- Suppport qwen model and solve some problems by @Arcmoon-Hu in #75
- Fix after QWen support by @merrymercy in #82
- Fix the chat template for QWen by @merrymercy in #83
- Fix SRT endpoint api json syntax by @CSWellesSun in #84
- Return logprob for choices by @merrymercy in #87
- Add health endpoint to SGLang runtime server by @isaac-vidas in #90
- Llava-hd Support by @caoshiyi in #92
- Bump the version to v0.1.8 by @merrymercy in #93
- Improve Chinese character streaming when the last char is half Chinese word. by @haotian-liu in #95
- Handle grayscale images in expand2square by @isaac-vidas in #97
- support speculative execution for openai API by @parasol-aser in #48
- fix batch error for llava-hd by @caoshiyi in #98
- Dynamic model class loading by @comaniac in #101
- Flush Cache API by @hnyls2002 in #103
- Fix Mistral model loading by @comaniac in #108
- Improve the control of streaming and improve the first token latency in streaming by @merrymercy in #117
- Add qwen2 by @JustinLin610 in #114
- Format code by @merrymercy in #118
- Update quick start examples by @merrymercy in #120
- Improve docs & Add JSON decode example by @merrymercy in #121
- [Feature] Adds basic support for image content in OpenAI chat routes by @fozziethebeat in #113
- [Feature] Allow specifying all ports to use in advance by @Ja1Zhou in #116
- Add cache metrics by @comaniac in #119
- Fix model loading & format code by @merrymercy in #125
- Add city doc benchmark mode by @hnyls2002 in #129
- Yi-VL Model by @BabyChouSr in #112
- Fix
is_multimodal_modeljudge by @hnyls2002 in #132 - Add max_prefill_num_token into server arguments by @Ying1123 in #133
- Release 0.1.11 by @Ying1123 in #134
New Contributors
- @isaac-vidas made their first contribution in #80
- @Arcmoon-Hu made their first contribution in #75
- @CSWellesSun made their first contribution in #84
- @haotian-liu made their first contribution in #95
- @parasol-aser made their first contribution in #48
- @JustinLin610 made their first contribution in #114
- @fozziethebeat made their first contribution in #113
- @Ja1Zhou made their first contribution in #116
Full Changelog: v0.1.6...v0.1.11