What's Changed
First big feature update of 2024! Brings (still incubating!) object detection using RKNN with a hard-coded YOLOv5 model trained on notes, and an (also new) LimeLight 3 image. Now that features work we'll be changing gears to focus on documenting new things. As always please reach out on Discord with journalctl logs if you encounter any issues!
- Add RKNN / Object Detection Pipeline by @mdurrani808 in #1144
- LL3 by @BytingBulldogs3539 in #1166
The nitty gritty
RKNN is the neural network accelerator hardware built into rockchip CPUs using on SBCs like the Orange Pi 5. We're able to use this hardware to massively accelerate certain math operations like those needed for running ML-based object detection. The code for that lives here, and publishes to Maven. Our pre-trained note model lives here, and NeuralNetworkModelManager deals with extracting that to disk. If a file named "note-640-640-yolov5s.rknn" and labels.txt does not exist at photonvision_config/models/, it'll be extracted from the JAR -- otherwise, the one already on disk will be used. This technically allows power users to replace the model and label files with new ones without rebuilding photon, though this feature is still largely untested, too.
Full Changelog: v2024.1.5...v2024.2.0