Build changes
- Fix TensorsToSegmentationCalculator gpu dependencies.
- Open Source build rules for quantization_util
- Added the binary the converter factory to run the model weight conversion.
- Integrates the kMemoryManagerService into ImageToTensorCalculator and InferenceCalculatorDarwinn.
- Updated iOS OpenCV source build to exclude highgui and videoio
- Open source some BUILD rules for Converter package
Framework and core calculator improvements
- Added Face Landmarker C Tasks API and tests
- Added Pose Landmarker C Tasks API
- Use memcpy now for copying data and indicate how the data is stored
- Remove superfluous glFlush().
- Added Face Detector C Tasks API
- Add mediapipe::file::IsDirectory helper
- Deprecate ImageFrame::ByteDepth
- Added files for the Image Segmenter C Tasks API
- Add general support for PathToResourceAsFile to TfLiteModelLoader
- Add CalculatorGraph::SetErrorCallback to receive errors in case of async graph use cases.
- Add JAX as requirements for MediaPipe python package
- Introduces HardwareBufferPool based on the ReusablePool and MultiPool
- Added the base classes for the LLM weight converter.
- Add stdbool import to C API
- Introduces MemoryManagerService with HardwareBufferPool and integrates it into the Tensor class.
- Added the model writer that writes to the weight binary files.
- Fix GlContext (attachments) cleanup in case of a failing GlContext initialization.
- Add option for using variable XNNPACK operators to MediaPipe XNNPACK flags
- Make InferenceCalculatorDarwinn support float and int32 as input data type.
- Adds VectorToTensorCalculator
- Enable HardwareBufferPool only if MEDIAPIPE_TENSOR_USE_AHWB is enabled
- Enables MultiPool and ReusablePool to pass on absl::Status returns originating from object factory methods.
- Adding TENSOR to InferenceCalculatorCpu to remove vector encumbrances
- Update Clang to version 16
- Add ability to preserve output format to GlScalerCalculator
- Add explicit depedency on XNNPACK & cpuinfo
- Update TensorFlow and Android NDK dependency
- Adds MEDIAPIPE_ANDROID_LINK_NATIVE_WINDOW condition to hardware_buffer_android
- Support interpolate flags in image_to_tensor_converter_opencv.
- Add MODEL_VIEW side input to tflite_model_calculator
MediaPipe Tasks update
This section should highlight the changes that are done specifically for any platform and don't propagate to other platforms.
Android
- Added HolisticLandmarker
- Migrate TextGenerator Java API to C Wrapper
- Add LlmTaskRunner to TextGenerator sources
- Don't cache the JNI environment for async calls
- Simplified api interface
- Updates to LLM JNI Layer
- Handle model loading on Android
- Support custom cache dir.
- Pass cacheDir to LLM engine from Java API
- Add "done" field to the Java LLM API
- Removed backend option from JNI layer
iOS
- Updated supported pixel formats in iOS image classifier Documentation
- Removed support for CVPixelBuffer of type 32RGBA
- Added support for creating CVPixelBuffer from C++ Images to iOS MPPImage Utils
- Updated implementation of MPPImage Utils to reduce lines of code
- Added iOS interactive segmenter options, helpers, implmentation and basic tests
- Added iOS Image Embedder API
- Enabled stream mode on iOS pose landmarker
- Fixed issue with iOS Language Detector Prediction Count
- Added iOS language detector to Cocoapods build
- Added cosine similarity method to iOS MPPImagEmbedder
- Updated method signature of MPPImageEmbedderResult initializer
- Added packet validation in MPPImageEmbedderResult+Helpers
- Added tests for creating MPPImage with source type UIImage from C++ Image
- Renamed methods in MPPImageUtilsTests
- Added a new class for iOS Interctive Segmenter Results
- Added iOS interactive segmenter to cocoapods build
- Added method for initializing MPPImages of all source types from MPPImage+TestUtils
- Added support for creating MPImages of sample buffer source type from C++ Images in MPPImage+Utils
- Added provision to initialize MPImages of source type sample buffer to MPImage+TestUtils
- Added tests for initialization of MPImages with source type sample buffer from MPImage+Utils
- Updated documentation of iOS hand landmarker, image embedder, image segmenter, interactive segmenter, object detector
- Added provision to create graph config from task options that use any proto for iOS tasks
- Added new methods to MPPTaskOptionsProtocol.h
- Updated iOS task runner to initialize tasks using MPPTaskInfo
- Updated iOS text task runner to initialize tasks from task Info.
- Updated iOS vision task runner to use new methods from MPPTaskRunner
- Updated MPPTaskOptionsProtocol
- Updated MPPTaskRunner initializer
- Fixed iOS framework conflicts with TensorFlowLiteC and OpenCV CocoaPods
- Fixed issue in installing iOS tasks text and vision libraries in a single project
Javascript
- Extend verifyGraph to be compatible with proto3.
- Add Holistic Landmarker Web API
- Add export declarations to PoseLandmakerResult
- TypeScript: adding VideoFrame typings support to video input
- Guard WaitOnGpu with extra OpenGL checks.
- Explicitly cast at callsite of WebGL context creation to avoid compilation errors with newer Emscripten versions.
Python
- Added Holistic Landmarker Python API
- Support both proto2 and proto3 in task subgraph options configuration, and revised the Holistic Landmarker API's implementation
- Update holistic_landmarker.py
- Documented HolisticLandmarker
- Fixing delegate passing argument in BaseOptions
- Add model_ckpt_util to Python Build script
- Use pybind_library for GenAI Converter build
MediaPipe Dependencies
- Expose MediaPipe's ABSL and Sentencepiece as shared dependencies
- Remove Sentencepiece's LOG function
- Removed unwanted headers from opencv_ios_xcfraemworl_files.bzl
- Update WASM files for 0.10.10 release
- Upgrade TypeScript to 5.3.3