Highlight
- Refactor attention and optimize GQA(#1258 #1307 #1116), achieving 22+ and 16+ RPS for internlm2-7b and internlm2-20b, about 1.8x faster than vLLM
- Support new models, including Qwen1.5-MOE(#1372), DBRX(#1367), DeepSeek-VL(#1335)
What's Changed
🚀 Features
- Add tensor core GQA dispatch for
[4,5,6,8]by @lzhangzz in #1258 - upgrade turbomind to v2.1 by by @lzhangzz in #1307, #1116
- Support slora to pipeline by @AllentDan in #1286
- Support qwen for pytorch engine by @RunningLeon in #1265
- Support Triton inference server python backend by @ispobock in #1329
- torch engine support dbrx by @grimoire in #1367
- Support qwen2 moe for pytorch engine by @RunningLeon in #1372
- Add deepseek vl by @AllentDan in #1335
💥 Improvements
- rm unused var by @zhyncs in #1256
- Expose cache_block_seq_len to API by @ispobock in #1218
- add chat template for deepseek coder model by @lvhan028 in #1310
- Add more log info for api_server by @AllentDan in #1323
- remove cuda cache after loading vison model by @irexyc in #1325
- Add new chat cli with auto backend feature by @RunningLeon in #1276
- Update rewritings for qwen by @RunningLeon in #1351
- lazy import accelerate.init_empty_weights for vl async engine by @irexyc in #1359
- update lmdeploy pypi packages deps to cuda12 by @irexyc in #1368
- update
max_prefill_token_numfor low gpu memory by @grimoire in #1373 - Optimize pipeline of pytorch engine by @grimoire in #1328
🐞 Bug fixes
- fix different stop/bad words length in batch by @irexyc in #1246
- Fix performance issue of chatbot by @ispobock in #1295
- add missed argument by @irexyc in #1317
- Fix dlpack memory leak by @ispobock in #1344
- Fix invalid context for Internstudio platform by @lzhangzz in #1354
- fix benchmark generation by @grimoire in #1349
- fix window attention by @grimoire in #1341
- fix batchApplyRepetitionPenalty by @irexyc in #1358
- Fix memory leak of DLManagedTensor by @ispobock in #1361
- fix vlm inference hung with tp by @irexyc in #1336
- [Fix] fix the unit test of model name deduce by @AllentDan in #1382
📚 Documentations
- add citation in readme by @RunningLeon in #1308
- Add slora example for pipeline by @AllentDan in #1343
🌐 Other
- Add restful interface regrssion daily test workflow. by @zhulinJulia24 in #1302
- Add offline mode for testcase workflow by @zhulinJulia24 in #1318
- workflow bugfix and add llava-v1.5-13b testcase by @zhulinJulia24 in #1339
- Add benchmark test workflow by @zhulinJulia24 in #1364
- bump version to v0.3.0 by @lvhan028 in #1387
Full Changelog: v0.2.6...v0.3.0