Working with TrackEngine#

TrackEngine is based on face detection and analysis methods provided by the FaceEngine library.

There are two ways of working with TrackEngine:

Using the asynchronous pushFrame or callback method.
It allows you to use a simple API with asynchronous push frames per each stream and get all tracking result events or data in another thread in callbacks. For details, see IStream::pushFrame in the TrackEngine example.
Using the estimator tracking API.
It works like the SDK estimator with an internal state. You should pass a batch of streams or frames to the function and get ready tracking results for input streams. This way is more complex, but flexible for developers. For details, see ITrackEngine::trackin the TrackEngine example.

Using the asynchronous method#

The main advantage of the asynchronous method is simplicity of client side code. You will not have to deal with such tasks as:

Exception handling
Issue multithreading
Creating queues for multiple stream batch gathering
Deferred result processing

This logic is implemented in TrackEngine by callback-mode = 1.

The common solution is:

Create a stream per each frame source.
Set up callback observers.
Submit frames to each stream one by one.
Depending on your app architecture, you can push frames to each stream from different threads. But we recommend that you use one thread per each stream and frame source.

Tracking results are obtained in callbacks from another thread that is created in TrackEngine. This is the place where you should write logic of processing results. When a stream has to be finished, call the IStream::join method to wait all queued frames or callbacks to be processed. After that, the stream cannot be used anymore and can be released.

Using the estimator tracking API#

If you want to control the logic of tracking at most, use the estimator tracking API.

Benefits of using the API:

Minimal memory consumption of TrackEngine. It allows you to achieve better performance, because it does not keep images in any queues. Yet, images are still kept in track data.
No need to configure parameters that regulate buffer sizes or batching settings. For example, tracking-results-buffer-size, frames-buffer-size, callback-buffer-size, min-frames-batch-size, max-frames-batch-gather-timeout. Also, streams should call stop instead of join on finish. In this case, a stream serves only as a state object for tracking.

Important: To use the estimator API, set the callback-mode parameter to 0, otherwise value 1 must be set (default value is 1).

Creating a TrackEngine instance#

To create a TrackEngine instance use the following global factory functions:

ITrackEngine tsdk::createTrackEngine(fsdk::IFaceEngine engine, const char configPath, vsdk::IVehicleEngine vehicleEngine = nullptr, const fsdk::LaunchOptions *launchOptions = nullptr)
- engine - Pointer to FaceEngine instance (should be already initialized).
- configPath - Path to the TrackEngine config file.
- vehicleEngine - Pointer to the VehicleEngine object (if with vehicle logic).
- launchOptions - Launch options for SDK functions.
- return value - Pointer to ITrackEngine.
ITrackEngine tsdk::createTrackEngine(fsdk::IFaceEngine engine, const fsdk::ISettingsProviderPtr& provider, vsdk::IVehicleEngine vehicleEngine = nullptr, const fsdk::LaunchOptions launchOptions = nullptr)
- engine - Pointer to FaceEngine instance (should be already initialized).
- provider - Settings provider with the TrackEngine configuration.
- vehicleEngine - Pointer to the VehicleEngine object (if with vehicle logic).
- launchOptions - Launch options for SDK functions.
- return value - Pointer to ITrackEngine.

Important: We do not recommend that you create multiple TrackEngine instances in one application, unless different configs are required. The reason is that batching improves performance.

Creating the main interface#

The main interface to TrackEngine is stream - an entity to which you submit video frames.

To create a stream use the following TrackEngine methods:

IStream ITrackEngine::createStream(StreamParams params = nullptr)
- params - Pointer to stream specific parameters. It is an optional parameter. If valid, then it overrides configuration parameters for a stream. For details, see StreamParams in the TrackEngine example.
- return value - Pointer to ITrackEngine.
IStream* ITrackEngine::createStreamWithParams(const StreamParamsOpt &params)
- params - Stream specific parameters. Each parameter in the struct is optional. If it is valid, then overrides the same config parameter for the stream. For details, see StreamParamsOpt in the TrackEngine example.
- return value - Pointer to ITrackEngine.

Important: You must own this raw pointer by fsdk::Ref, for example, with fsdk::acquire and reset all references to all streams before TrackEngine object destruction. Otherwise memory leak or/and UB are guaranteed. This is valuable especially in languages where order of objects destruction is not guaranteed, so you should manage objects lifetime manually (for example, python).

Concurrent streams#

You can create multiple streams working concurrently, for example, if you need to track faces from several cameras. In each stream, the engine detects faces and builds their tracks. Each face track has its own unique identifier. It is therefore possible to group face images belonging to the same person with their track IDs. Please note, tracks may break from time to time on either of the reasons:

Due to people leaving the visible area.
Due to the challenging detection conditions. For example, poor image quality, occlusions, extreme head poses, and so on.

Finishing work with TrackEngine#

At the end of work with TrackEngine, call ITrackEngine::stop before the TrackEngine object can be released.

void ITrackEngine::stop() - Stops processing.

Estimator API#

fsdk::ResultValue track(fsdk::Span streams, fsdk::Span frames) - Updates stream tracks by a new frame per each stream and returns ready tracking results data for passed streams (as callbacks compatible data).
- streams - Streams stream identifiers, must contain only unique IDs, otherwise function throws. For details, see IStream::getId.
- frames - Frames input frames per a stream. For details, see IStream::pushFrame and Frame.
- return value - First error code and reference to ready tracking results as callbacks compatible data. For details, see ITrackingResultBatch.

We recommend that you make each frame (from frames) image to be the data owner. Otherwise, performance overhead is possible as TrackEngine internally will clone it to keep in track data. The function returns only ready tracking results per each stream, so it can return tracking results for stream previously passed frames, as well as not return results for the current passed frame. The reason of such delay is that tracking may require several frames to get results. For details, see the Receiving tracking results section. It is thread safe, but call blocking. The function is not exception safe like pushFrame.

Important: Regulating batch size for track is the critical step to achieve high performance of tracking. Higher values lead to better utilization or throughput (especially on GPU), but increase latency of system.

For pre-validation of track inputs, noexcept function validate is useful.

fsdk::Result validate(fsdk::Span streams, fsdk::Span frames, fsdk::Span> outErrors) - Validates input of multiple streams or frames in a single function call.
- streams - Streams an array of stream identifiers array.
- frames - Frames input frames per a stream.
- outErrors - Error output span of errors for each stream or frame.
- return value - Result with the last error code.

When a stream has to be finished, call the IStream stop method before a stream release to get all remaining tracking results. You can use the stream for tracking after calling this function.

fsdk::Ref stop(bool reset = false) - Finishes all current tracks and returns all remaining tracking results.
- reset - If set to true, resets a stream state to initial. Otherwise, keeps internal frame counter, statistics, and so on.
- return value - Remaining tracking results for the stream.

Async API#

bool IStream::pushFrame(const fsdk::Image &frame, uint32_t frameId, tsdk::AdditionalFrameData *data = nullptr) - Pushes a single frame to the stream buffer.
- frame - Frame image input. The format must be R8G8B8 OR R8G8B8X8.
- frameId - Unique identifier for a frame sequence.
- data - Additional data that a developer wants to receive in a callback realization. It must be allocated only with new or be equal to nullptr. Do not use the delete-operator. The garbage collector is implemented inside TrackEngine for this parameter.
- return value - Returns true, if a frame was appended to the queue for processing. Returns false otherwise - a frame was skipped because of the full queue.

We recommend that you make frame to be owner of data, otherwise performance overhead is possible as TrackEngine internally will clone it to keep in track data. Also there are some variations of this method: pushCustomFrame, pushFrameWaitFor, pushCustomFrameWaitFor.

When a stream has to be finished, call the IStream join method before a stream release. You cannot use the stream after joining for processing, only 'getter' functions are available.

void IStream::join() - Blocks the current thread until all frames in this stream are handled and all callbacks are executed.

Important: Ignoring this step may lead to unexpected behavior. TrackEngine writes a warning log in this case.