Library Usage#

Build you own project with 3D+AI library#

We show how to build a project using the 3D+AI library with CMake. You can use the 3D+AI Demo code as a reference.

Copy the cmake folder inside your repo.

Setup your project, and set the C++ compiler version to C++17.

cmake_minimum_required(VERSION 3.22)

project("Your project")
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_C_STANDARD 17)

set(CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake" ${CMAKE_MODULE_PATH})

find_package(HailoRT 4.15.0 EXACT REQUIRED)
find_package(OpenCV 4.6.0 REQUIRED)

Add your target with its source file.

add_executable(your_target
# Your source files
)

Add the 3D+AI library to the target dependencies.

target_link_libraries(your_target
PRIVATE
3dai
# ...
)

Now you are ready to start writing your code.

How to initialize 3D+AI#

Before starting the 3D+AI library it is strongly recommended to calibrate the stereo camera. You can use the Calibration Tool to produce a JSON file containing the calibration matrices. Otherwise you can calibrate the camera yourself, and then use StereoCameraCalib to initialize the undistort parameters.

To use the 3D+AI library you need to understand a few concepts:

  1. Logging

  2. Presets

  3. Hailo Virtual Device

  4. Config

  5. Callback

  6. System Status

Logging#

The 3D+AI library allow you to setup the log verbosity level and where to print the log, using:

  • Logging level

  • Logging sink

The available logging levels are:

  • OFF: Log is disabled.

  • DEBUG: Debug log level.

  • INFO: Info log level.

  • WARNING: Warning log level.

  • ERR: Error log level.

The available logging sink options are:

  • STDOUT: Print log to standard output.

  • FILE: Print log to the rotating file “log.txt” in the current directory.

  • STDOUT_FILE: Print log to both standard output and the rotating file “log.txt” in the current directory.

By default the library prints INFO level logs to STDOUT. You can change the logging level and sink using the setup_log function:

dv::ai3d::setup_log(dv::ai3d::LogLevel::INFO, dv::ai3d::LogSink::STDOUT);

Presets#

The preset identifies the kind of stereo matching provided by 3D+AI. It contains the resolution and the maximum disparity. The resolution is the image size used in the stereo matching step. If the input stereo images have a resolution different from the preset resolution, both images will be resized to the preset resolution before applying the stereo matching procedure. The disparity is the difference between a pixel in the left image and its corresponding pixel in the right image. Here below is a table of available ModelPreset with the resolution and maximum disparity values.

Preset

Resolution (WxH)

Max disparity

DISP_128_RES_512x384

512x384

128

DISP_128_RES_768x576

768x576

128

Hailo Virtual Device#

Please refer to the official documentation for more information on HailoRT C++ API.

Note

To access the HailoRT C++ API you need to submit a Developer Zone Access Request.

Here is a simple example on how to create a VDevice with the following code:

#include "3dai.h"

//...
auto vdevice = hailort::VDevice::create();
if (!vdevice)
{
    spdlog::error("Failed create vdevice, status = {}", static_cast<int> (vdevice.status()));
    return -1;
}
//...

Config#

The configuration defines the library preset and which functionalities are enabled. Each functionality is defined with its own parameters structure. You can initialize each needed functionality separately, and then add it to the configuration. It’s recommended to initialized them by calling the static method from.

In this chapter we show how to:

  • Set the model preset

  • Undistort input images

  • Compute depth map

  • Compute point cloud

  • Add resized and rectified input images to output

  • Set override parameters

First create a Config, then set the preset and the parameters for each needed functionality.

//...
dv::ai3d::Config config;

dv::ai3d::ModelPreset preset = dv::ai3d::ModelPreset::DISP_128_RES_512x384;
config.set_model_preset(preset);

std::string calibration_filepath ("/<path>/<to>/<calibration_file>.json");

{
    auto undist = dv::ai3d::UndistortParams::from(calibration_filepath, preset);
    if (!undist)
    {
        auto err_code = undist.error();
        spdlog::error("Undistort initialization failed: {}", dv::ai3d::to_string(err_code));
        return false;
    }
    config.set_undistort_params(undist.value());
}

{
    auto depth = dv::ai3d::DepthParams::from(calibration_filepath, preset);
    if (!depth)
    {
        auto err_code = depth.error();
        spdlog::error("Depth initialization failed: {}", dv::ai3d::to_string(err_code));
        return false;
    }
    config.set_depth_params(depth.value());
}

{
    auto pc = dv::ai3d::PointCloudParams::from(calibration_filepath, preset);
    if (!pc)
    {
        auto err_code = pc.error();
        spdlog::error("Point cloud initialization failed: {}", dv::ai3d::to_string(err_code));
        return false;
    }
    config.set_point_cloud_params(pc.value());
}

{
    auto rect = dv::ai3d::RectifiedImagesParams::from(true);
    if (!rect)
    {
        auto err_code = rect.error();
        spdlog::error("Rectified images initialization failed: {}", dv::ai3d::to_string(err_code));
        return false;
    }
    config.set_rectified_images_params(rect.value());
}

{
    const float sensitivity = 0.45f;
    auto override = dv::ai3d::OverrideParams::from(sensitivity);
    if (!override)
    {
        auto err_code = override.error();
        spdlog::error("Override initialization failed: {}", dv::ai3d::to_string(err_code));
        return false;
    }
    config.set_override_params(override.value());
}

dv::ai3d::RetCode code = config.check();
if(code != dv::ai3d::RetCode::OK) {
    spdlog::error("Check configuration failed: {}", dv::ai3d::to_string(code));
    return false;
}

//...

Set the model preset#

After creating a Config, you can specify the preset. Use the Config::set_model_preset to set the ModelPreset. For more information about available presets please refer to Presets.

Undistort input images#

Setting this parameter enables input images rectification. Use the Config::set_undistort_params to set the UndistortParams. The library rectifies the resized input before the 3D reconstruction procedure.

Compute depth map#

Setting this parameter enables depth map computation. Use the Config::set_depth_params to set the DepthParams. The library computes the depth map from the disparity map, and stores it in the Output.

Compute point cloud#

Setting this parameter enables point cloud computation. Use the Config::set_point_cloud_params to set the PointCloudParams. The library computes the point cloud from the disparity map, and stores it in the Output.

Add resized and rectified input images to output#

Setting this parameter enables rectified images (at preset resolution) passthrough. Use the Config::set_rectified_images_params to set the RectifiedImagesParams. The library stores the resized and rectified input images in the Output.

Set override params#

Setting this parameter overrides the library execution parameters. Use the Config::set_override_params to set the OverrideParams. Currently only the 3D reconstruction sensitivity is available. Sensitivity is a float value in range (0,1] which represents the confidence required for each disparity value to be valid, from 0 which indicates to accept any confidence, to 1 which represents the maximum confidence in the result. The default sensitivity value is 0.35.

Initialization#

Once the VDevice is created, and configuration is completed, you can initialize the 3D+AI library.

//...

auto status = dv::ai3d::init(*vdevice.value(), config);
if(status != dv::ai3d::RetCode::OK) {
    spdlog::info("Failed initialization with code {}", dv::ai3d::to_string(status));
    return -1;
}

Callback#

You can retrieve the output of 3D+AI using a callback as defined in the output_callback_t:

//...

void some_callback(const dv::ai3d::Output& hailo_out) {
    //...
}

// library initialization

dv::ai3d::set_out_callback(some_callback);

//...

System Status#

Depending on many factors (chosen preset, input image resolution, cpu usage, etc…), the system may require more resources than available and become congested. This is signaled via logging messages, but for a more programmatic approach you can call the check_system_status function.

This function returns the possible states:

  • OK: The system can process frames at the expected rate.

  • FILLING: The system is moderately congested and frame rate may decrease.

  • OUT_OF_RESOURCES: The system is congested and will discard upcoming frames.

The following snippet shows how to handle system congestion:

//...
dv::ai3d::SystemStatus status = dv::ai3d::check_system_status();
if(status == dv::ai3d::SystemStatus::OK) {
    // Everything is fine
}
if(status == dv::ai3d::SystemStatus::FILLING) {
    // Handle moderate congestion
}
if(status == dv::ai3d::SystemStatus::OUT_OF_RESOURCES) {
    // Handle heavy congestion
}
spdlog::info("Current status: {}", dv::ai3d::to_string(status));
//...

Trial version#

For running the 3D+AI library with the time-limited trial you need to call the check_trial_license function at the start of each run of the library.

const auto status = dv::ai3d::check_trial_license();
if (status != dv::ai3d::RetCode::OK) {
    spdlog::error("Failed trial activation with error: {}", dv::ai3d::to_string(status));
    return -1;
}

If the trial license is not already activated, launch the 3dai_ula_acceptance executable and follow the instructions (see Accept license and activate time-limited trial). If the trial period has expired, a RetCode::LICENSE_ERROR will be returned.