Skip to content

Working with TrackEngine#

TrackEngine is based on face detection and analysis methods provided by FaceEngine library. This document does not cover FaceEngine usage in detail, for more information please see FaceEngine_Handbook.pdf.

To create a TrackEngine instance use the following global factory functions

  • ITrackEngine tsdk::createTrackEngine(fsdk::IFaceEngine engine, const char configPath, vsdk::IVehicleEngine vehicleEngine = nullptr, const fsdk::LaunchOptions *launchOptions = nullptr)

    • engine - pointer to FaceEngine instance (should be already initialized)
    • configPath - path to TrackEngine config file
    • vehicleEngine - pointer to the VehicleEngine object (if with vehicle logic)
    • launchOptions - launch options for sdk functions
    • return value - pointer to ITrackEngine

  • ITrackEngine tsdk::createTrackEngine(fsdk::IFaceEngine engine, const fsdk::ISettingsProviderPtr& provider, vsdk::IVehicleEngine vehicleEngine = nullptr, const fsdk::LaunchOptions launchOptions = nullptr)

    • engine - pointer to FaceEngine instance (should be already initialized)
    • provider - settings provider with TrackEngine configuration
    • vehicleEngine - pointer to the VehicleEngine object (if with vehicle logic)
    • launchOptions - launch options for sdk functions
    • return value - pointer to ITrackEngine

It is not recommended to create multiple TrackEngine instances in one application, unless different configs are required. The reason is that batching improves performance.

The main interface to TrackEngine is Stream - an entity to which you submit video frames. To create a stream use the following TrackEngine method

  • IStream ITrackEngine::createStream(StreamParams params = nullptr)

    • params - the pointer to stream specific parameters. It's optional parameter, if valid, then it overrides config params for the Stream. Consider StreamParams for details.
    • return value - pointer to IStream

  • IStream* ITrackEngine::createStreamWithParams(const StreamParamsOpt &params)

    • params - stream specific parameters. Each parameter in the struct is optional, so if it's valid, then overrides the same config parameter for the Stream. Consider StreamParamsOpt for details.
    • return value - pointer to IStream

Note: User must own this raw pointer by fsdk::Ref, e.g. with fsdk::acquire and reset all refs to all streams before Track Engine object desctruction, otherwise memory leak or/and UB are guaranteed. This is valuable especially in languages where order of objects desctruction is not guaranteed, so users should manage objects lifetime manually (e.g. python). See examples.

Users can create multiple streams working concurrently (in case when need to track faces from multiple cameras). In each stream the engine detects faces and builds their tracks. Each face track has its own unique identifier. It is therefore possible to group face images belonging to the same person with their track ids. Please note, tracks may break from time to time either due to people leaving the visible area or due to the challenging detection conditions (poor image quality, occlusions, extreme head poses, etc).

There are two ways to work with TE. First one is async pushFrame/callbacks method (See IStream::pushFrame), which allows users to use simple API with async push frames per each Stream and get all tracking result events/data in another thread in callbacks. The second one is more complex, but flexible for developers. It's estimator API (See ITrackEngine::track), that works like SDK estimator with internal state, so users should pass batch of streams/frames to function and get ready tracking results for input streams.

Each of the methods has pros and cons. Main advantage of async method is simplicity of client side code, so users mostly don't have to deal with exceptions handling, multithreading issues and creating queues for multiple stream batches gathering and results deferred processing. All that logic is implemented in TE for callback-mode = 1. The common solution is to create a stream per each frame source, setup callbacks observers, and submit frames to each stream one by one basis. Frames can be pushed to each stream from different threads, depending on architecture of application, but it's recommended to use one thread per each stream and frame source. Tracking results are obtained in the callbacks from another thread (it's created in the TE), so it's the place where users should write logic of processing results. When Stream has to be finished, user should call IStream::join method to wait all queued frames/callbacks to be processed. After that Stream can't be used anymore and can be released. If users want to control logic of tracking maximum, they should use estimator tracking API. One of the key advantages of estimator API is minimal memory consumption of Track Engine (so possibility to achieve better performance), because in this case it doesn't keep images in any queues (images are kept in the tracks data still, though). When users work with estimator tracking API, they don't have to deal with many of the config parameters, regulating any buffer sizes or batching settings, e.g tracking-results-buffer-max-size, frames-buffer-size, callback-buffer-size, min-frames-batch-size, max-frames-batch-gather-timeout. Also, streams should call stop instead of join on finish. In this case Stream serves only as a state object for tracking.

Note: To use estimator API, users should set config parameter callback-mode to 0, otherwise value 1 must be set (default value is 1).

At the end of work with TE, users should call ITrackEngine::stop before TE object can be released.

  • void ITrackEngine::stop() Stops processing.

Estimator API:

  • fsdk::ResultValue track(fsdk::Span streams, fsdk::Span frames) Updates stream tracks by new frame per each stream and returns ready tracking results data for passed streams (as callbacks compatible data).

    • streams - streams stream identifers, must contain only unique id-s, otherwise function throws. See IStream::getId.
    • frames - frames input frames per stream. See also IStream::pushFrame and Frame.
    • return value - First error code and Ref to ready tracking results as callbacks compatible data. Consider ITrackingResultBatch.

It's recommended to make each frame (from frames) image to be owner of data, otherwise performance overhead is possible as TE internally will clone it to keep in track data. Function returns only ready tracking results per each stream, so it can return tracking results for Stream previously passed frames as well as not return results for current passed frame. The reason of such delay is that, generally, tracking may require several frames to get results. See Receiving tracking results section for more details. It's thread safe, but blocking call. The function isn't exception safe like pushFrame.

Note: regulating batch size for track is the critical step to achieve high performance of tracking. Higher values lead to better utilization/throughput (especially on GPU), but increase latency of system.

For prevalidation of track inputs noexcept function validate is useful.

  • fsdk::Result validate(fsdk::Span streams, fsdk::Span frames, fsdk::Span> outErrors) Validate input of multiple streams/frames in a single function call.

    • streams - streams stream identifers array.
    • frames - frames input frames per stream.
    • outErrors - errors output span of errors for each stream/frame.
    • return value - Result with last error code.

When Stream has to be finished, then user should call IStream stop method before Stream release to get all remained tracking results. Stream can be used for tracking after call of this func.

  • fsdk::Ref stop(bool reset = false) Finishes all current tracks and returns all remaining tracking results.

    • reset - reset if set true, then function resets Stream's state to initial (otherwise keep internal frame counter, statistics etc)
    • return value - Remaining tracking results for the Stream.

Async API:

  • bool IStream::pushFrame(const fsdk::Image &frame, uint32_t frameId, tsdk::AdditionalFrameData *data = nullptr) Pushes a single frame to the stream buffer.

    • frame - input frame image. Format must be R8G8B8 OR R8G8B8X8.
    • frameId - unique identifier for frames sequence.
    • data - is any additional data that a developer wants to receive in callbacks-realization. It must be allocated only with new or be equal to nullptr. Do not use the delete-operator. The garbage collector is implemented inside TrackEngine for this param.
    • return value - true if frame was appended to the queue for processing, false otherwise - frame was skipped because of full queue.

It's recommended to make frame to be owner of data, otherwise performance overhead is possible as TE internally will clone it to keep in track data. Also there are some variations of this method: pushCustomFrame, pushFrameWaitFor, pushCustomFrameWaitFor.

When Stream has to be finished, then user should call IStream join method before stream release. Stream can't be used after join for processing, only 'getter' functions are available.

  • void IStream::join() Blocks current thread until all frames in this Stream will be handled and all callbacks will be executed.

Note: Ignoring this step can lead to unexpected behavior (TE writes warning log in this case).

You can set up an observer to receive and react to events. There are two types of observers: per-stream specific single observer and batched observer for all streams. Per-stream observers are deprecated now and remained only for compatibility with old versions.

Note: It's highly recommended to use new batched observers API instead of old per-stream one.

Batched observers have some advantages over per-stream observers:

  • reduce and set fixed number of threads created by TrackEngine itself (see section Threading for details).
  • eliminate performance overhead from multiple concurrently working threads used for per-stream callbacks.
  • allow to easily use batched SDK API without additional aggregation of data from single callbacks. Both for GPU/CPU batched SDK API improves performance (for GPU effect is much more significant).
  • give more information in output (per-stream callbacks functions signatures remain the same because of compatibility with old versions)

Stream observer interfaces:

Per-stream observers:

  • IBestShotObserver

  • IVisualObserver

  • IDebugObserver

Batched event specific observers:

  • IBatchBestShotObserver

  • IBatchVisualObserver

  • IBatchDebugObserver

Batched unified observer:

  • ITrackingResultObserver

ITrackingResultObserver is the most recommended observer to work with, when callback-mode = 1, as it provides all ready tracking results/events in one struct/callback. This observer contains only one function ready which called every time, when tracking results are ready for any streams/frames. Users have assurance that all frames per each stream from ITrackingResultBatch were processed and can write logic based on this knowledge. ready function will be called once per each stream/frame except of last frame of each stream, when trackEnd is called for all remaining tracks (inactive too) on Stream join, because TE can't know which frame actually will be last from user. Note, that it's value type is the same as for track method (used for callback-mode = 0).

There is the next priority of observers use:

  1. ITrackingResultObserver
  2. Batched event specific observers
  3. Per-stream observers

It means, that if 1 is set up then it will be used, otherwise: if 2 is set up then it will be used, otherwise 3.

Note: you have to setup either single per-stream or batched observers for all streams, but not both at the same time.

IBestShotPredicate type defines recognition suitability criteria for face detections. By implementing a custom predicate one may alter the best shot selection logic and, therefore, specify which images will make it to the recognition phase.

Setting per-stream observer API example:

  • void IStream::setBestShotObserver(tsdk::IBestShotObserver* observer) Sets a best shot observer for the Stream.

    • observer - pointer to the observer object, see IBestShotObserver. Don't set to nullptr, if you want disable it, then use IStream::setObserverEnabled with false.

Setting batched observer API example:

  • void ITrackEngine::setBatchBestShotObserver(tsdk::IBatchBestShotObserver *observer) Sets a best shot observer for all streams.

    • observer - pointer to the batched observer object, see IBatchBestShotObserver. Don't set to nullptr, if you want disable it, then use IStream::setObserverEnabled with false.

Setting unified batched observer API:

  • void ITrackEngine::setTrackingResultObserver(tsdk::ITrackingResultObserver *observer) Sets a unified tracking result observer for all streams.

    • observer - pointer to the observer object, see ITrackingResultObserver. If set to nullptr, then per event batched observers will be used.