sgl-project/sglang v0.1.16 on GitHub

Highlight

Support more models: DBRX, Command-R, Gemma
Support llava-video (#423, https://llava-vl.github.io/blog/2024-04-30-llava-next-video/)
Cache performance improvements (#418, #364)
Marlin quantization kernels
Many bug fixes
Update dependencies to be compatible with their latest versions

What's Changed

Fix Runtime missing some ServerArgs options by @Qubitium in #281
adding the triton docker build minimal example by @amirarsalan90 in #242
Fix flashinfer >= 0.0.3 compat by @Qubitium in #282
Fix Incorrect CURL Request Example in README by @amirarsalan90 in #287
enable marlin kernels by @qeternity in #286
Fix env (docker) compat due to file usage by @Qubitium in #288
Fix marlin model loading compat with autogptq by @Liurl21 in #290
Fix outlines-0.0.35 incompatibility by @ZhouGongZaiShi in #291
[Fix/Potential Bugs] Can not correctly import models in python/sglang/srt/models by @Luodian in #311
Use Anthropic messages API by @janimo in #304
Add StableLM model. by @janimo in #301
Support oai in benchmark/mmlu by @merrymercy in #323
Update version to v0.1.14 by @merrymercy in #324
Cleanup codebase: removed unnecessary code/logic by @Qubitium in #298
Update dependencies by @janimo in #326
Openrouter usage example by @janimo in #327
model_rpc style improvement by @hnyls2002 in #293
model_runner simplify by @hnyls2002 in #329
Logprobs Refractor by @hnyls2002 in #331
DBRX support by @hnyls2002 in #337
Add support for new autogptq quant_config.checkpoint_format by @Qubitium in #332
Fix llava parallelism/fork bug by @lockon-n in #315
Eliminate 2 gpu ops during sampling when logit_bias is zero by @hnyls2002 in #343
Revert "Eliminate 2 gpu ops during sampling when logit_bias is zero" by @hnyls2002 in #345
Eliminate 2 gpu ops during sampling when logit_bias is zero by @Qubitium in #338
Add timeout to get_meta_info by @SimoneRaponi in #346
Fix typos in infer_batch.py by @tom-doerr in #354
Time cost utils by @hnyls2002 in #355
Update README.md by @eltociear in #358
support command-r by @ZhouXingg in #369
Fix issue #367 – System message not supported for Anthropic (anthropic.BadRequestError) by @fronx in #368
Update model support in readme by @Ying1123 in #370
Optimize radix tree matching by @ispobock in #364
Reduce overhead when fork(1) by @hnyls2002 in #375
llama3 instruct template by @qeternity in #372
add .isort.cfg by @hnyls2002 in #378
Revert removing the unused imports by @hnyls2002 in #385
Benchmark Updates by @hnyls2002 in #382
Improve performance when running with full parallel by @hnyls2002 in #394
Minor: style improvement of radix_cache and memory_pool by @hnyls2002 in #395
Format Benchmark Code by @hnyls2002 in #399
Fix chatml template by @merrymercy in #406
Adding RAG tracing & eval cookbook using Parea by @joschkabraun in #390
SamplingParams add "spaces_between_special_tokens" argument by @ZhouXingg in #392
Organize Benchmark by @hnyls2002 in #381
Add Cohere Command R chat template by @noah-kim-theori in #411
Fix sync() when fork(1) by @hnyls2002 in #412
Include finish reason in meta info response by @qeternity in #415
Make public APIs more standard. by @hnyls2002 in #416
Compat with latest VLLM 0.4.2 main + fork.number rename + Flashinfer 0.0.4 by @Qubitium in #380
Optimize the memory usage of logits processor by @merrymercy in #420
Clean up by @merrymercy in #422
Fix logit processor bugs by @merrymercy in #427
Minor fix for the import path by @merrymercy in #428
Move openai api server into a separate file by @merrymercy in #429
Fix flashinfer by @merrymercy in #430
Update version to 0.1.15 by @merrymercy in #431
Misc fixes by @merrymercy in #432
Allow input_ids in the input of the /generate endpoint by @lolipopshock in #363
Improve error handling by @merrymercy in #433
Cache optimizations by @hnyls2002 in #418
Update readme by @merrymercy in #434
Raise errors for prompts that are too long by @merrymercy in #436
support llava video by @ZhangYuanhan-AI in #426
Fix streaming by @merrymercy in #437
Update version to 0.1.16 by @merrymercy in #438

New Contributors

@Qubitium made their first contribution in #281
@amirarsalan90 made their first contribution in #242
@Liurl21 made their first contribution in #290
@ZhouGongZaiShi made their first contribution in #291
@Luodian made their first contribution in #311
@janimo made their first contribution in #304
@lockon-n made their first contribution in #315
@SimoneRaponi made their first contribution in #346
@tom-doerr made their first contribution in #354
@ZhouXingg made their first contribution in #369
@fronx made their first contribution in #368
@ispobock made their first contribution in #364
@joschkabraun made their first contribution in #390
@noah-kim-theori made their first contribution in #411
@lolipopshock made their first contribution in #363
@ZhangYuanhan-AI made their first contribution in #426

Full Changelog: v0.1.13...v0.1.16