API C++

namespace dv

namespace ai3d

Typedefs

using output_callback_t = std::function<void(const Output &out)>

Callback function template. It asynchronously provides the results of the stereo matching. The results are available according to the initialized configuration.

Enums

enum class RetCode

Return codes for the 3D+AI library.

Values:

enumerator OK

Ok status.

enumerator GENERIC_ERROR

Error not listed below.

enumerator INTERNAL_ERROR

Internal library error. Check the error identifier in the log and ask DV for support.

enumerator ALREADY_INITIALIZED

The library must be initialized exactly once: cannot overwrite initialization.

enumerator NOT_INITIALIZED

The library must be initialized exactly once: not yet initialized.

enumerator CALIBRATION_LOAD_ERROR

The calibration file is missing or corrupted.

enumerator INVALID_INPUT_IMAGE

The input image is empty or has an unexpected number of channels.

enumerator INVALID_CALIBRATION

The provided calibration parameters are not valid.

enumerator VALUE_OUT_OF_RANGE

The provided value is outside the expected range.

enumerator DISCARDED_IMAGE

The provided input image was discarded to throttle processing.

enumerator LICENSE_ERROR

License check failed.

enumerator ULA_OR_TRIAL_NOT_ACCEPTED

3D+AI User License Agreement not accepted or Trial license not activated.

enum class LogLevel

Log verbosity.

Values:

enumerator OFF

Log is disabled.

enumerator DEBUG

Debug log level.

enumerator INFO

Info log level.

enumerator WARNING

Warning log level.

enumerator ERR

Error log level.

enum class LogSink

Sink destination for library logging.

Values:

enumerator STDOUT

Print log to standard output.

enumerator FILE

Print log to the rotating file “log.txt” in the current directory.

enumerator STDOUT_FILE

Print log to both standard output and the rotating file “log.txt” in the current directory.

enum class ModelPreset

List of available model presets.

Values:

enumerator DISP_128_RES_512x384

Preset with disparity 128 and resolution 512x384.

enumerator DISP_128_RES_768x576

Preset with disparity 128 and resolution 768x576.

enum class SystemStatus

Provides an indication of current processing capabilities.

Values:

enumerator OK

The system can process frames at the expected rate.

enumerator FILLING

The system is moderately congested and frame rate may decrease.

enumerator OUT_OF_RESOURCES

The system is congested and will discard upcoming frames.

Functions

RetCode check_trial_license()

Check the trial license for 3D+AI library. It must be called on startup, before initializing 3D+AI library parameters. To launch this function the trial license needs to be already activated. If the trial license is not already activated, launch the 3dai_ula_acceptance executable and follow the given instructions. In case the license has expired calling this function will return a RetCode::LICENSE_ERROR. You need an internet connection to check the license, so if you encounter any license activation error make sure that the device is connected to the internet.

Returns:

License activation status.

RetCode init(hailort::VDevice &vdevice, const Config &config)

Initialize the 3D+AI library with the provided hailort::VDevice and 3D+AI library configuration. It must be called once, before starting any processing.

Parameters:

• vdevice – An already created hailort::VDevice.

• config – Library configuration.

Returns:

Initialization result.

void stop()

Stops the execution of the library and resets it to the initial state. After calling this you need to call the init function with the same or different parameters.

void set_out_callback(output_callback_t cb)

Set the callback function to be called with the library results. After providing a valid pair of images to the function add_stereo_image_3b or add_stereo_image_1b, the pipeline will produce an Output object passed to the callback. It contains optional members that are set according to the initial configuration. It must be called after init.

Parameters:

cb – [in] Callback function

SystemStatus check_system_status()

Check the system processing capabilities.

Returns:

System status.

RetCode add_stereo_image_3b(const cv::Mat3b &left, const cv::Mat3b &right)

Add a pair of stereo BGR images to the system to be processed. Images passed to this function while system state is SystemStatus::OUT_OF_RESOURCES will be dropped.

Parameters:

• left – [in] Left input bgr image.

• right – [in] Right input bgr image.

Returns:

RetCode::OK if the image can be processed, or the error reason.

RetCode add_stereo_image_1b(const cv::Mat1b &left, const cv::Mat1b &right)

Add a pair of stereo single channel images to the system to be processed. Images passed to this function while system state is SystemStatus::OUT_OF_RESOURCES will be dropped.

Parameters:

• left – [in] Left input single channel image.

• right – [in] Right input single channel image.

Returns:

RetCode::OK if the image can be processed, or the error reason.

cv::Size get_preset_size(const ModelPreset &preset)

Get preset resolution.

Parameters:

preset – [in] ModelPreset.

Returns:

Resolution of the preset.

float get_sensitivity()

Get current stereo matching sensitivity.

Returns:

Stereo matching sensitivity.

RetCode set_sensitivity(float sensitivity)

Set current stereo matching sensitivity.

Returns:

RetCode::OK if the sensitivity can be set, or the error reason.

void setup_log(LogLevel log_level = LogLevel::INFO, LogSink log_sink = LogSink::STDOUT)

Setup library logging. Configure the logging level and the log sink. If not called, the default logging will be used.

Parameters:

• log_level – [in] Logging level

• log_sink – [in] Logging sink

void reset()

Stops and drops the current processing images.

std::string to_string(RetCode ret)

Get the string representation of the given code.

Parameters:

ret – RetCode to be converted.

Returns:

RetCode string representation.

std::string to_string(SystemStatus ret)

Get the string representation of the given code.

Parameters:

ret – SystemStatus to be converted.

Returns:

SystemStatus string representation.

std::string get_version()

Get library version.

Returns:

String containing the library version.

Variables

constexpr float INVALID_STEREO_VALUE = -1.f

Invalid value for disparity and depth. It can be used to mask invalid values in user processing.

struct Config

#include <3dai.h>

Struct containing all parameters used during the 3D+AI library execution. Available functionalities are enabled by calling the corresponding set function (e.g. setting undistort parameters will enable the input image rectification). The check function is available to ensure that parameters are valid and consistent. Must be called after check_trial_license.

Public Functions

void set_model_preset(const ModelPreset &preset)

Set 3D+AI library model preset.

Parameters:

preset – [in] The library will use this preset.

void set_undistort_params(const UndistortParams &undistort)

Set 3D+AI library undistort parameters.

Parameters:

undistort – [in] The library will rectify the input images according to the given parameters.

void set_depth_params(const DepthParams &depth)

Set 3D+AI library depth parameters.

Parameters:

depth – [in] The library will compute the depth map according to the given parameters.

void set_point_cloud_params(const PointCloudParams &point_cloud)

Set 3D+AI library point cloud parameters.

Parameters:

point_cloud – [in] The library will compute the point cloud according to the given parameters.

void set_rectified_images_params(const RectifiedImagesParams &rectified_images)

Set 3D+AI library rectified images parameters.

Parameters:

rectified_images – [in] The library will provide the rectified input images.

void set_override_params(const OverrideParams &override)

Set 3D+AI library override parameters.

Parameters:

override – [in] The library will use the user-provided parameters instead of the defaults.

RetCode check() const

Check if the set parameters are valid and consistent.

Returns:

Parameters validity.

Public Members

ModelPreset model_preset

Model preset.

UndistortParams undistort_params

Undistort parameters.

DepthParams depth_params

Depth parameters.

PointCloudParams point_cloud_params

Point cloud parameters.

RectifiedImagesParams rectified_images_params

Rectified images parameters.

OverrideParams override_params

Override parameters.

struct DepthParams

#include <3dai.h>

Struct containing the parameters to transform the disparity map to a depth map. If these parameters are defined during initialization, each disparity produced by the stereo matching will also be used to compute the depth map. The resulting depth map is returned in the Output object as depth. If point cloud computation is enabled, the parameters must be consistent. Must be called after check_trial_license. Use the static methods DepthParams::from to obtain the depth parameters.

Public Members

float fx

Focal length.

float baseline

Baseline.

bool enabled = false

Depth map computation is enabled. This argument is automatically set to true if using DepthParams::from.

Public Static Functions

static Expected<DepthParams> from(const std::string &calibration_path, const ModelPreset &preset)

Create depth parameters from calibration file and preset. The calibration file must be in JSON format and can be produced with the Calibration Tool software.

Parameters:

• calibration_path – [in] Absolute path to the .json calibration file.

• preset – [in] ModelPreset used for the library execution.

static Expected<DepthParams> from(const StereoCameraCalib &stereo_camera_calib, const ModelPreset &preset)

Create depth parameters from stereo calibration and preset. The argument stereo_camera_calib should contain a valid set of stereo calibration matrices.

Parameters:

• stereo_camera_calib – [in] Struct containing stereo camera calibration matrices and original calibration image resolution.

• preset – [in] ModelPreset used for the library execution.

static Expected<DepthParams> from(float fx, float baseline)

Create depth parameters from stereo focal length and baseline.

Parameters:

• fx – [in] Stereo calibration focal length.

• baseline – [in] Stereo camera baseline.

template<class T>
class Expected

#include <3dai.h>

Utility class for simpler return values handling. It provides either the expected object, or an error code.

Public Functions

inline Expected(const T &t)

inline operator bool() const

inline RetCode error() const

inline const T &value() const

Public Static Functions

static inline Expected unexpected(RetCode ret)

struct Output

#include <3dai.h>

Struct containing the library outputs for each pair of stereo images (unless they were discarded). The resolution of the left_undistort, right_undistort, disparity and depth matrices. are defined by the selected ModelPreset.

Public Members

std::optional<cv::Mat3b> left_undistort

Undistorted left frame.

std::optional<cv::Mat3b> right_undistort

Undistorted right frame.

std::optional<cv::Mat1f> disparity

Disparity map.

std::optional<cv::Mat1f> depth

Depth map.

std::optional<std::vector<cv::Point3f>> point_cloud

Point cloud.

struct OverrideParams

#include <3dai.h>

Struct containing the parameters that can be overridden by the user. Now it contains only the sensitivity. Sensitivity is a float value in range (0,1] which represents the confidence required for each disparity value to be valid, from 0 which indicates to accept any confidence, to 1 which represents the maximum confidence in the result. Must be called after check_trial_license. Use the static methods OverrideParams::from to obtain the override parameters.

Public Members

float sensitivity

Disparity sensitivity.

bool enabled = false

Override the default sensitivity value.

Public Static Functions

static Expected<OverrideParams> from(float sensitivity)

Create override parameters.

Parameters:

sensitivity – [in] Sensitivity of the stereo matching disparity map. Default value is 0.35.

struct PointCloudParams

#include <3dai.h>

Struct containing the parameters to transform the disparity map to a vector of 3D points. If these parameters are defined during initialization, each disparity produced by the stereo matching will also be used to compute the point cloud. The resulting point cloud is returned in the Output object as point_cloud. If depth computation is enabled, the parameters must be consistent. Must be called after check_trial_license. Use the static methods PointCloudParams::from to obtain the point cloud parameters.

Public Members

float fx

Focal length.

float baseline

Baseline.

float cx

Principal point x coordinate.

float cy

Principal point y coordinate.

bool enabled = false

Point cloud computation is enabled. This argument is automatically set to true if using PointCloudParams::from.

Public Static Functions

static Expected<PointCloudParams> from(const std::string &calibration_path, const ModelPreset &preset)

Create point cloud parameters from calibration file and preset. The calibration file must be in JSON format and can be produced with the Calibration Tool software.

Parameters:

• calibration_path – [in] Absolute path to the .json calibration file.

• preset – [in] ModelPreset used for the library execution.

static Expected<PointCloudParams> from(const StereoCameraCalib &stereo_camera_calib, const ModelPreset &preset)

Create point cloud parameters from stereo calibration and preset. The argument stereo_camera_calib should contain a valid set of stereo calibration matrices.

Parameters:

• stereo_camera_calib – [in] Struct containing stereo camera calibration matrices and original calibration image resolution.

• preset – [in] ModelPreset used for the library execution.

static Expected<PointCloudParams> from(float fx, float baseline, float cx, float cy)

Create point cloud parameters from stereo focal length, baseline, and camera principal point.

Parameters:

• fx – [in] Stereo calibration focal length.

• baseline – [in] Stereo camera baseline.

• cx – [in] x coordinate of principal point.

• cy – [in] y coordinate of principal point.

static Expected<PointCloudParams> from(const DepthParams &depth_params, float cx, float cy)

Create point cloud parameters from depth parameters and camera principal point.

Parameters:

• depth_params – [in] Depth parameter.

• cx – [in] x coordinate of principal point.

• cy – [in] y coordinate of principal point.

struct RectifiedImagesParams

#include <3dai.h>

Struct containing parameters to return the rectified input images. Defining this parameter during initialization will have two different behaviors depending on whether undistort parameters are defined. If the undistort parameters are defined, enabling this will return the input images resized to the preset resolution and rectified. Otherwise, enabling this will return the input images resized. The resulting images are returned in the Output object as left_undistort and right_undistort. Must be called after check_trial_license. Use the static methods RectifiedImagesParams::from to obtain the rectified images parameters.

Public Members

bool enabled = false

Rectified images return is enabled. This argument is automatically set to true if using RectifiedImagesParams::from.

Public Static Functions

static Expected<RectifiedImagesParams> from(bool enabled)

Create rectified images parameters.

Parameters:

enabled – [in] Enable return of rectified images.

struct StereoCameraCalib

#include <3dai.h>

Struct containing stereo calibration information. This struct is needed only if you want to provide your own stereo camera calibration. For a more in-depth explanation please refer to UndistortParams documentation.

Public Members

cv::Mat K_l

Left camera matrix.

cv::Mat K_r

Right camera matrix.

cv::Mat dist_l

Left camera distortion coefficients.

cv::Mat dist_r

Right camera distortion coefficients.

cv::Mat R

Rotation matrix.

cv::Mat T

Translation vector.

cv::Size image_size

Resolution of the calibration images.

struct UndistortParams

#include <3dai.h>

Struct containing the calibration parameters used to undistort the left/right input frames. If these parameters are defined during initialization, input images provided with add_stereo_image_3b or add_stereo_image_1b will be rectified. Must be called after check_trial_license. Use the static methods UndistortParams::from to obtain the undistort parameters.

Public Members

StereoCameraCalib stereo_camera_calib

Stereo calibration matrices.

bool enabled = false

Undistort input frames enabled. This argument is automatically set to true if using UndistortParams::from.

Public Static Functions

static Expected<UndistortParams> from(const std::string &calibration_path, const ModelPreset &preset)

Create undistort parameters from calibration file and preset. The calibration file must be in JSON format and can be produced with the Calibration Tool software.

Parameters:

• calibration_path – [in] Absolute path to the .json calibration file.

• preset – [in] ModelPreset used for the library execution.

static Expected<UndistortParams> from(const StereoCameraCalib &stereo_camera_calib, const ModelPreset &preset)

Create undistort parameters from stereo calibration and preset. The argument stereo_camera_calib should contain a valid set of stereo calibration matrices.

Parameters:

• stereo_camera_calib – [in] Struct containing stereo camera calibration matrices and original calibration image resolution.

• preset – [in] ModelPreset used for the library execution.