What's Changed
🚀 Features
- Add extra_requires to reduce dependencies by @RunningLeon in #580
- TurboMind 2 by @lzhangzz in #590
- Support loading hf model directly by @irexyc in #685
- convert model with hf repo_id by @irexyc in #774
- Support turbomind bf16 by @grimoire in #803
- support image_embs input by @irexyc in #799
- Add api.py by @AllentDan in #805
💥 Improvements
- Fix Tokenizer encode by @AllentDan in #645
- Optimize for throughput by @lzhangzz in #701
- Replace mmengine with mmengine-lite by @zhouzaida in #715
- Set the default value of
max_context_token_num1 by @lvhan028 in #761 - add triton server test and workflow yml by @RunningLeon in #760
- improvement(build): enable ninja and gold linker by @tpoisonooo in #767
- Report first-token-latency and token-latency percentiles by @lvhan028 in #736
- Unify prefill & decode passes by @lzhangzz in #775
- add cuda12.1 build check ci by @irexyc in #782
- auto upload cuda12.1 python pkg to release when create new tag by @irexyc in #784
- Report the inference benchmark of models with different size by @lvhan028 in #794
- Simplify block manager by @lzhangzz in #812
- Disable attention mask when it is not needed by @lzhangzz in #813
- FIFO pipe strategy for api_server by @AllentDan in #795
- simplify the header of the benchmark table by @lvhan028 in #820
- add encode for opencompass by @AllentDan in #828
- fix: awq should save bin files by @hscspring in #793
- Support building docker image manually in CI by @RunningLeon in #825
🐞 Bug fixes
- Fix init of batch state by @lzhangzz in #682
- fix turbomind stream canceling by @grimoire in #686
- [Fix] Fix load_checkpoint_in_model bug by @HIT-cwh in #690
- Fix wrong eos_id and bos_id obtained through grpc api by @lvhan028 in #644
- Fix cache/output length calculation by @lzhangzz in #738
- [Fix] Skip empty batch by @lzhangzz in #747
- [Fix] build docker image failed since
packagingis missing by @lvhan028 in #753 - [Fix] Rollback the data type of
input_idstoTYPE_UINT32in preprocessor's proto by @lvhan028 in #758 - fix turbomind build on sm<80 by @grimoire in #754
- Fix early-exit condition in attention kernel by @lzhangzz in #788
- Fix missed arguments when benchmark static inference performance by @lvhan028 in #787
- fix extra colon in InternLMChat7B template by @C1rN09 in #796
- Fix local kv head num by @lvhan028 in #806
- Fix out-of-bound access by @lzhangzz in #809
- Set smem size for repetition penalty kernel by @lzhangzz in #818
- Fix cache verification by @lzhangzz in #821
- fix finish_reason by @AllentDan in #816
- fix turbomind awq by @grimoire in #847
- Fix stop requests by await before turbomind queue.get() by @AllentDan in #850
- [Fix] Fix meta tensor error by @pppppM in #848
- Fix cuda reinitialization in a multiprocessing setting by @grimoire in #862
- launch gradio server directly with hf model by @AllentDan in #856
- fix typo by @grimoire in #769
- Add chat template for Yi by @AllentDan in #779
- fix api_server stop_session and end_session by @AllentDan in #835
- Return the iterator after erasing it from a map by @irexyc in #864
📚 Documentations
- [Docs] Update Supported Matrix by @pppppM in #679
- [Docs] Update KV8 Docs by @pppppM in #681
- [Doc] Update restful api doc by @AllentDan in #662
- Check-in user guide about turbomind config by @lvhan028 in #680
- Update benchmark user guide by @lvhan028 in #763
- [Docs] Fix typo in
restful_apiuser guide by @maxchiron in #858 - [Docs] Fix typo in
restful_apiuser guide by @maxchiron in #859
🌐 Other
- bump version to v0.1.0a0 by @lvhan028 in #709
- bump version to 0.1.0a1 by @lvhan028 in #776
- bump version to v0.1.0a2 by @lvhan028 in #807
- bump version to v0.1.0 by @lvhan028 in #834
New Contributors
- @zhouzaida made their first contribution in #715
- @C1rN09 made their first contribution in #796
- @maxchiron made their first contribution in #858
Full Changelog: v0.0.14...v0.1.0