google-ai-edge/mediapipe v1.1.0

MediaPipe v1.1.0

on GitHub

Build and Bazel changes

• Remove the dependency on internal Bazel Apple rules resources.bzl in mediapipe/metal.bzl, using the metal rule provided by Bazel Apple rules.
• Add Python 3.13 and 3.14 support to the MediaPipe Bazel build.
• Upgrade LiteRT dependency to v2.2.0 in MediaPipe WORKSPACE.
• Move up declaration of kissfft in build configuration.
• Fix xla llvm download URL by using mediapipe-assets mirror.
• Fix Windows build.
• Remove obsolete unistd.h header imports.
• Configure MediaPipe for vendor/no-JNI Android builds.

Framework and core calculator improvements

• Bump MediaPipe version to 1.1.0.
• Add MediaPipe Universal Embedder, Semantic Retriever, and LiteRT-LM integration across C++, C, Python, and Android/Maven APIs.
• MediaPipe Decision: Add EmbeddingGemma 1 & 2 decision backend with multimodal state and option support.
• Implement core TaskPart and EmbeddingProvider interface.
• Refactor TextPart, AudioPart, and ImagePart into tasks.core and unify duplicates.
• Refactor TextEmbedder and ImageEmbedder to use executor abstractions (TextEmbedderExecutor, ImageEmbedderExecutor).
• Add HasFile, GetBundleManifest, and GetBundleId helpers to ModelAssetBundleResources.
• Add noexcept to move ctor/operator and add swap to Packet.
• Add method to create an empty packet in Java API.
• Use ABSL_ASSIGN_OR_RETURN and ABSL_RETURN_IF_ERROR in MediaPipe.
• Rename GeckoPreprocessorCalculator to SentencePiecePreprocessorCalculator.
• Add option to ignore missing encoded_media_bytes in UnpackMediaSequenceCalculator.
• Add LiteRT-LM based C++ Audio Embedder.
• C API for MediaPipe AudioEmbedder.
• Select the MediaPipe OpenGL device by Vulkan device UUID.
• Replace LiteRT delegate with use_litert option in MediaPipe BaseOptions.
• Expose LiteRT and NPU delegate options to MediaPipe Tasks.
• Only use public LiteRT API in LiteRT inference calculator.
• Support passing LiteRtSystemHandles to Environment in LiteRT Inference Calculator.
• Add support for specifying a signature key for LiteRT inference.
• Retain MTLCommandQueue directly instead of DrishtiMetalHelper in InferenceRunnerLiteRt.
• Use runtime layout for rank-0 LiteRT model outputs in InferenceRunnerLiteRt.
• Check GPU service availability in TensorsToDetectionsCalculator.
• Improve MediaPipe Metal GPU delegate error messages for unsupported configurations.
• Allow size mismatched tensors for GPU execution.
• Fix race condition in GpuResources.
• Fix deprecated CreateSourceTexture usage in desktop GPU example.

MediaPipe Tasks update

This section highlights the changes that are done specifically for each platform and don't propagate to other platforms.

Android

• Add plumbing needed for zero-copy AHardwareBuffer (AHWB) output from the GPU.
• Fix OpenGL ES and AHardwareBuffer synchronization when converting between write and read views.
• Wrap VC effects AHWB graph input as AHardwareBuffer (kRGBA32).
• Raise MediaPipeException instead of crashing in surface_output_jni on Android EGL surface creation failure.
• Fix crash in PathToResourceAsFile when TEST_SRCDIR is not set on Android.
• Made AppSearch vector store optional for SemanticRetriever Android SDK.
• Add Semantic Retriever Java API by prebuilding SQLite dependencies.
• Support in-memory audio and image representations in MediaPipe Tasks Android.
• Support multi-part multimodal embedding and vector store serialization in MediaPipe Tasks Android.
• Ensure that JNI methods are exported for Android Tasks.
• Remove legacy LITERT delegate flag from Android API.

iOS

• Update Swift Package Manager (SPM) manifest for MediaPipe Tasks iOS 1.1.0.
• Add iOS Retrieval SPM package (MediaPipeTasksRetrieval).
• Add missing MPPAudioRecord.h to MediaPipeTasksRetrieval XCFramework headers.
• Exclude MediaPipe C API (_Mp*) symbols from the iOS symbol rename map.
• Add inference_calculator_litert dependencies to iOS Tasks targets.
• Return standard NSError instead of crashing when universal embedder model path is invalid.
• Support in-memory audio and image representations in MediaPipe Tasks iOS.
• Support multi-part multimodal embedding and vector store serialization in MediaPipe Tasks iOS.

Javascript

• Add retrieval to MediaPipe Tasks for Web (UniversalEmbedder, SemanticRetriever).
• Export public APIs for Web SemanticRetriever.
• Make the DecisionMaker web API usable from the minified decision_js bundle.
• Refactor task name discovery in MediaPipe Web Tasks.
• Expose per-landmark presence scores on the Web Tasks API.
• Enable GPU usage for UniversalEmbedder on Web.
• Prevent overlapping WASM embedding calls.
• Disable LiteRT-LM extensive logging for UniversalEmbedder on Web.
• Add stats logging to Web Tasks.
• Add unit tests for ImageFrame and YUVImage conversions.
• Add missing await calls to UniversalEmbedder example in web retrieval documentation.

Python

• Python API for MediaPipe AudioEmbedder.
• Add cosine_similarity utility method to UniversalEmbedder.
• Cast all Python types to their explicit C type assignment for ABI stability.
• Add cache_dir option and default to :nocache.
• Add Semantic Retriever Python API with bundled SQLite vector store.

Bug fixes

• Fix HolisticLandmarker crash when face detections are missing.
• Fix HolisticLandmarker Any Builder return type and empty packet MediaPipeExceptions.
• Re-add ImageEmbedder.close() method.
• Fix numpy_view() SIGABRT on non-contiguous float32 masks.
• Fix wrist connection in HAND_PALM_CONNECTIONS.
• Fix landmark drift in LandmarksSmoothingCalculator (OneEuroFilter) when stream resolution dynamically changes between preview and still capture.
• Correct parameter names in core calculators.
• Fix WAV decoder calculations in audio_decoder_calculator_test.

MediaPipe Dependencies

• Upgrade LiteRT dependency to v2.2.0 in WORKSPACE.
• Update WASM asset binaries for 1.1.0 release.