dragonflyoss/Dragonfly2 v2.1.0 on GitHub

Dragonfly v2.1.0 is released! 🎉🎉🎉 Thanks to the Xinxin Zhao for helping to refactor the console and the manager provides a new console for users to operate Dragonfly. Welcome to visit d7y.io website.

Features

Console v1.0.0 is released and it provides a new console for users to operate Dragonfly.
Add network topology feature and it can probe the network latency between peers, providing better scheduling capabilities.
Provides the ability to control the features of the scheduler in the manager. If the scheduler preheat feature is not in feature flags, then it will stop providing the preheating in the scheduler.
dfstore adds GetObjectMetadatas and CopyObject to supports using Dragonfly as the JuiceFS backend.
Add personal access tokens feature in the manager and personal access token contains your security credentials for the restful open api.
Add TLS config to manager rest server.
Fix dfdaemon fails to start when there is no available scheduler address.
Add cluster in the manager and the cluster contains a scheduler cluster and a seed peer cluster.
Fix object downloads failed by dfstore when dfdaemon enabled concurrent.
Scheduler adds database field in config and moves the redis config to database field.
Replace net.Dial with grpc health check in dfdaemon.
Fix filtering and evaluation in scheduling. Since the final length of the filter is the candidateParentLimit used, the parents after the filter is wrong.
Fix storage can not write records to file when bufferSize is zero.
Hiding sensitive information in logs, such as the token in the header.
Use unscoped delete when destroying the manager's resources.
Add uk_scheduler index and uk_seed_peer index in the table of the database.
Remove security domain feature and security feature in the manager.
Add advertise port config to manager and scheduler.
Fix fsm changes state failed when register task.

Break Change

The M:N relationship model between the scheduler cluster and the seed peer cluster is no longer supported. In the future, a P2P cluster will be a cluster in the manager, and a cluster will only include a scheduler cluster and a seed peer cluster.

Console

You can see Manager Console for more details.

AI Infrastructure

Triton Inference Server uses Dragonfly to distribute model files, refer to #2185. If there are developers who are interested in the drgaonfly repository agent project, please contact gaius.qi@gmail.com.
TorchServer uses Dragonfly to distribute model files. Developers have already participated in the dragonfly endpoint project, and the feature will be released in v2.1.1.
Fluid downloads data through Dragonfly when running based on JuiceFS, the feature will be released in v2.1.1.
Dragonfly helps Volcano Engine AIGC inference to accelerate image through p2p technology.
There have been many cases in the community, using Dragonfly to distribute data in AI scenarios based on P2P technology. In the inference stage, the concurrent download model of the inference service can effectively relieve the bandwidth pressure of the model registry through Dragonfly, and improving the download speed. Community will share topic 《Dragonfly: Intro, Updates and AI Model Distribution in the Practice of Kuaishou - Wenbo Qi, Ant Group & Zekun Liu, Kuaishou Technology》 with Kuaishou in KubeCon + CloudNativeCon + Open Source Summit China 2023, please follow if interested.

Maintainers

The community has added four new Maintainers, hoping to help more contributors participate in community.

Yiyang Huang: He works for Volcano Engine and will focus on the engineering work for Dragonfly.
Manxiang Wen: He works for Baidu and will focus on the engineering work for Dragonfly.
Mohammed Farooq: He works for Intel and will focus on the engineering work for Dragonfly.
Zhou Xu: He is a PhD student at Dalian University of Technology and will focus on the intelligent scheduling algorithms.

Others

You can see CHANGELOG for more details.