What's Changed
🚀 Features
- support MiniCPM-Llama3-V 2.5 by @irexyc in #1708
- [Feature]: Support llava for pytorch engine by @RunningLeon in #1641
- Device dispatcher by @grimoire in #1775
- Add GLM-4-9B-Chat by @lzhangzz in #1724
- Torch deepseek v2 by @grimoire in #1621
- Support internvl-chat for pytorch engine by @RunningLeon in #1797
- Add interfaces to the pipeline to obtain logits and ppl by @irexyc in #1652
- [Feature]: Support cogvlm-chat by @RunningLeon in #1502
💥 Improvements
- support mistral and llava_mistral in turbomind by @lvhan028 in #1579
- Add health endpoint by @AllentDan in #1679
- upgrade the version of the dependency package peft by @grimoire in #1687
- Follow the conventional model_name by @AllentDan in #1677
- API Image URL fetch timeout by @vody-am in #1684
- Support internlm-xcomposer2-4khd-7b awq by @AllentDan in #1666
- update dockerfile and docs by @RunningLeon in #1715
- lazy import VLAsyncEngine to avoid bringing in VLMs dependencies when deploying LLMs by @lvhan028 in #1714
- feat: align with OpenAI temperature range by @zhyncs in #1733
- feat: align with OpenAI temperature range in api server by @zhyncs in #1734
- Refactor converter about get_input_model_registered_name and get_output_model_registered_name_and_config by @lvhan028 in #1702
- Refine max_new_tokens logic to improve user experience by @AllentDan in #1705
- Refactor loading weights by @grimoire in #1603
- refactor config by @grimoire in #1751
- Add anomaly handler by @lzhangzz in #1780
- Encode raw image file to base64 by @irexyc in #1773
- skip inference for oversized inputs by @grimoire in #1769
- fix: prevent numpy breakage by @zhyncs in #1791
- More accurate time logging for ImageEncoder and fix concurrent image processing corruption by @irexyc in #1765
- Optimize kernel launch for triton2.2.0 and triton2.3.0 by @grimoire in #1499
- feat: auto set awq model_format from hf by @zhyncs in #1799
- check driver mismatch by @grimoire in #1811
- PyTorchEngine adapts to the latest internlm2 modeling. by @grimoire in #1798
- AsyncEngine create cancel task in exception. by @grimoire in #1807
- compat internlm2 for pytorch engine by @RunningLeon in #1825
- Add model revision & download_dir to cli by @irexyc in #1814
- fix image encoder request queue by @irexyc in #1837
- Harden stream callback by @lzhangzz in #1838
- Support Qwen2-1.5b awq by @AllentDan in #1793
- remove chat template config in turbomind engine by @irexyc in #1161
- misc: align PyTorch Engine temprature with TurboMind by @zhyncs in #1850
- docs: update cache-max-entry-count help message by @zhyncs in #1892
🐞 Bug fixes
- fix typos by @irexyc in #1690
- [Bugfix] fix internvl-1.5-chat vision model preprocess and freeze weights by @DefTruth in #1741
- lock setuptools version in dockerfile by @RunningLeon in #1770
- Fix openai package can not use proxy stream mode by @AllentDan in #1692
- Fix finish_reason by @AllentDan in #1768
- fix uncached stop words by @grimoire in #1754
- [side-effect]Fix param
--cache-max-entry-countis not taking effect (#1758) by @QwertyJack in #1778 - support qwen2 1.5b by @lvhan028 in #1782
- fix falcon attention by @grimoire in #1761
- Refine AsyncEngine exception handler by @AllentDan in #1789
- [side-effect] fix weight_type caused by PR #1702 by @lvhan028 in #1795
- fix best_match_model by @irexyc in #1812
- Fix Request completed log by @irexyc in #1821
- fix qwen-vl-chat hung by @irexyc in #1824
- Detokenize with prompt token ids by @AllentDan in #1753
- Update engine.py to fix small typos by @WANGSSSSSSS in #1829
- [side-effect] bring back "--cap" argument in chat cli by @lvhan028 in #1859
- Fix vl session-len by @AllentDan in #1860
- fix gradio vl "stop_words" by @irexyc in #1873
- fix qwen2 cache_position for PyTorch Engine when transformers>4.41.2 by @zhyncs in #1886
- fix model name matching for internvl by @RunningLeon in #1867
📚 Documentations
- docs: add BentoLMDeploy in README by @zhyncs in #1736
- [Doc]: Update docs for internlm2.5 by @RunningLeon in #1887
🌐 Other
- add longtext generation benchmark by @zhulinJulia24 in #1694
- add qwen2 model into testcase by @zhulinJulia24 in #1772
- fix pr test for newest internlm2 model by @zhulinJulia24 in #1806
- react test evaluation config by @zhulinJulia24 in #1861
- bump version to v0.5.0 by @lvhan028 in #1852
New Contributors
- @DefTruth made their first contribution in #1741
- @QwertyJack made their first contribution in #1778
- @WANGSSSSSSS made their first contribution in #1829
Full Changelog: v0.4.2...v0.5.0