Open3D 0.12.0 Release Notes

Open3D 0.12.0 is out, and it comes with new 3D object detection pipelines and datasets, the newest versions of some of your preferred classic tools, and many bug fixes. Check out our 0.12.0 release video:

Extensions to the Open3D-ML module

The previous release of Open3D introduced an exciting new module dedicated to 3D Machine Learning Open3D-ML, featuring support for 3D semantic segmentation workflows. In this release, we have extended Open3D-ML with the task of 3D object detection. This extension introduces support for new datasets, such as the Waymo Open dataset, Lyft level 5 open data, Argoverse, nuScenes, and KITTI. As always, all these datasets can be visualized out-of-the-box using our visualization tool, from Python or C++. The visualization tool is now equipped with the capability to render 3D bounding boxes along with all the previously existing modalities, e.g. semantic labels, XYZRGB, depth, normals, etc.

Open3D-ML features

PointPillars, the first of the many object detection models to come in the near future. To enable the implementation of PointPillars, we have added a set of new ML operators in Open3D, such as: grid_sampling, NMS, and IOU. These operators are available to the community and can be used to build new models, using our Python and C++ APIs.

import os
import open3d.ml as _ml3d
import open3d.ml.torch as ml3d

cfg_file = "ml3d/configs/pointpillars_kitti.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)

model = ml3d.models.PointPillars(**cfg.model)
cfg.dataset['dataset_path'] = "/path/to/your/dataset"
dataset = ml3d.datasets.KITTI(cfg.dataset.pop('dataset_path', None), **cfg.dataset)
pipeline = ml3d.pipelines.ObjectDetection(model, dataset=dataset, device="gpu", **cfg.pipeline)

... 
# run inference on a single example.
result = pipeline.run_inference(data)

We have also updated our model zoo, providing new pretrained models on KITTI for the task of 3D object detection, and new semantic segmentation models on Paris-Lille3D and Semantic3D.

Remember that all the tools provided in Open3D-ML are compatible with PyTorch and TensorFlow!

Support for RealSense SDK v2

RealSense sensors’ support has been upgraded to leverage the RealSense SDK v2. Users can now capture crisp 3D data from L515 devices. As part of this upgrade, we include support for Bag files format (RSBagReader), and direct streaming from sensors. These operations can now be done through a new sensor class: RealSenseSensor, offering a simple and intuitive way to control your sensors.

import open3d as o3d
bag_reader = o3d.t.io.RSBagReader()
bag_reader.open(bag_filename)
while not bag_reader.is_eof():
    im_rgbd = bag_reader.next_frame()
    # process im_rgbd.depth and im_rgbd.color

bag_reader.close()

import json
import open3d as o3d
with open(config_filename) as cf:
    rs_cfg = o3d.t.io.RealSenseSensorConfig(json.load(cf))

rs = o3d.t.io.RealSenseSensor()
rs.init_sensor(rs_cfg, 0, bag_filename)
rs.start_capture(True)  # true: start recording with capture
for fid in range(150):
    im_rgbd = rs.capture_frame(True, True)  # wait for frames and align them
    # process im_rgbd.depth and im_rgbd.color

rs.stop_capture()

For further information, check this tutorial.

CORE and 3D reconstruction

Open3D 0.12 brings exciting CORE upgrades, including a new Neighbor Search module. This module supports typical neighbor search methods, such as KNN, radius search, and hybrid search, on both CPUs and GPUs, under a common interface!

Furthermore, we have created a new version of the TSDF integration algorithm accelerated on GPU. This version is able to achieve outstanding computational performance, requiring between 2 and 4 ms to integrate a pair of frames.

New rendering functionalities

We have done an important effort over the last months to put out a modern, real-time, rendering API. This effort is still ongoing, and we are committed to bringing top-tier rendering capabilities with a strong emphasis in performance, versatility, ease of use, and beauty. As part of our commitment, in this release we have added relevant extensions to this API:

• Support for Screen-space reflections

• Full programmatic support for headless rendering in Filament (for real)

box = o3d.geometry.TriangleMesh.create_box(2, 2, 1)
render = rendering.OffscreenRenderer(640, 480)
render.scene.add_geometry("box", box, grey)
render.scene.camera.look_at([0, 0, 0], [0, 10, 0], [0, 0, 1])
img = render.render_to_image()

• Support for arbitrary camera intrinsic matrices: A small step for the Camera class; a very anticipated step by the Open3D community

Camera::SetProjection(const Eigen::Matrix3d& intrinsics,
                               double near,
                               double far,
                               double width,
                               double height)

• Support for text rendering: Render text in 3D

Label3D::Label3D(const Eigen::Vector3f& pos, const char* text)

• Full control over the color grading pipeline

class ColorGradingParams {
public:
    ColorGradingParams(Quality q, ToneMapping algorithm);

    void SetTemperature(float temperature);
    void SetTint(float tint);
    void SetContrast(float contrast);
    void SetVibrance(float vibrance);
    void SetSaturation(float saturation);
    void SetChannelMixer(const Eigen::Vector3f& red,
                         const Eigen::Vector3f& green,
                         const Eigen::Vector3f& blue);
    void SetShadowMidtoneHighlights(const Eigen::Vector4f& shadows,
                                    const Eigen::Vector4f& midtones,
                                    const Eigen::Vector4f& highlights,
                                    const Eigen::Vector4f& ranges);
    void SetSlopeOffsetPower(const Eigen::Vector3f& slope,
                             const Eigen::Vector3f& offset,
                             const Eigen::Vector3f& power);

    void SetCurves(const Eigen::Vector3f& shadow_gamma,
                   const Eigen::Vector3f& midpoint,
                   const Eigen::Vector3f& highlight_scale);
}

Control shadow behaviors and post-processing effects:

class View
{
    void SetPostProcessing(bool enabled);
    void SetAmbientOcclusion(bool enabled, bool ssct_enabled);
    void SetAntiAliasing(bool enabled, bool temporal);
    void SetShadowing(bool enabled, ShadowType type);
}

Visualization and GUI: O3DViewer (beta)

The visualization module has been extended, using the new rendering capabilities and the GUI API, to create a unified visualizer displaying all the features contained in previous Open3D visualizers, e.g., camera animation, data selection, support for callbacks, and multiple shading modes.

This new visualizer, codename O3DViewer, will be the official visualization tool in Open3D starting in Open3D 0.14. At that time, previous visualizers will be deprecated.

We hope you find Open3D 0.12.0 exciting and useful. Happy coding!

Remember that you can reach out with questions, requests, or feedback through the following channels:

• e-mail
• Discourse forum
• Discord network

Find the full changelog here.

The Open3D team