Observers

You can set up an observer to receive and react on events. There are two types of observers:

• Per-stream specific single observer
  > Per-stream observers are deprecated now and remained only for compatibility with old versions.
• Batched observer for all streams
  > We recommend that you use the new batched observers API instead of old per-stream one.

Batched observers have some advantages over per-stream observers. Batched observers:

• Reduce and set a fixed number of threads created by TrackEngine. For details, see the Threading section.
• Eliminate performance overhead from multiple concurrently working threads used for per-stream callbacks.
• Allow to easily use the batched SDK API without additional aggregation of data from single callbacks. Both for GPU/CPU, the batched SDK API improves performance. For GPU effect is much more significant.
• Give more information in output. Per-stream callback function signatures remain the same because of compatibility with old versions.

Stream observer interfaces:

Per-stream observers	Batched event specific observers	Batched unified observer
IBestShotObserver	IBatchBestShotObserver	ITrackingResultObserver
IVisualObserver	IBatchVisualObserver
IDebugObserver	IBatchDebugObserver

ITrackingResultObserver is the most recommended observer to work with when callback-mode = 1. It provides all ready tracking results or events in one structure or callback. This observer contains only one function ready. It is called every time when tracking results are ready for any streams or frames.

You can be sure that all frames per each stream from ITrackingResultBatch were processed and can write logic based on this knowledge.

The ready function will be called once per each stream or frame. The exception is the last stream frame when trackEnd is called for all remaining tracks (inactive, too) on Stream join. The reason is that TrackEngine cannot define which frame actually will be the last one.

Note, that its value type is the same as for the track method (used for callback-mode = 0).

The priority of observer use is as follows:

1. ITrackingResultObserver
2. Batched event specific observers
3. Per-stream observers

It means, that if 1 is set up, then it will be used. Otherwise, if 2 is set up, then it will be used, otherwise - 3.

Important: You have to setup either single per-stream or batched observers for all streams, but not both at the same time.

The IBestShotPredicate type defines recognition suitability criteria for face detections. By implementing a custom predicate, you can alter the best shot selection logic and, therefore, specify which images will make it to the recognition phase.

Important: Placing images of different resolutions in one stream is not recommended.

Examples

Setting the per-stream observer API

• void IStream::setBestShotObserver(tsdk::IBestShotObserver* observer) - Sets a best shot observer for a stream.

  • observer - Pointer to the observer object. For details, see IBestShotObserver. Do not set to nullptr, if you want disable it. Instead, set IStream::setObserverEnabled to false.

Setting the batched observer API

• void ITrackEngine::setBatchBestShotObserver(tsdk::IBatchBestShotObserver *observer) - Sets a best shot observer for all streams.

  • observer - Pointer to the batched observer object. For details, see IBatchBestShotObserver. Do not set to nullptr, if you want disable it. Instead, set IStream::setObserverEnabled to false.

Setting the unified batched observer API

• void ITrackEngine::setTrackingResultObserver(tsdk::ITrackingResultObserver *observer) Sets a unified tracking result observer for all streams.

  • observer - Pointer to the observer object. For details, see ITrackingResultObserver. If set to nullptr, then batched observers will be used per event.

IBestShotObserver

• void bestShot(const tsdk::DetectionDescr& descr) - Called for each emerged best shot. It provides information on a best shot, including frame number, detection coordinates, cropped still image, and other data. See the DetectionDescr structure definition below for details. Default implementation does nothing.

  • descr - Best shot detection description.

struct TRACK_ENGINE_API DetectionDescr {
    //! Index of the frame
    tsdk::FrameId frameIndex;

    //! Index of the track
    tsdk::TrackId trackId;

    //! Source image
    fsdk::Image image;

    fsdk::Ref<ICustomFrame> customFrame;

    //! Face landmarks
    fsdk::Landmarks5 landmarks;

#ifndef MOBILE_BUILD
    //! Human landmarks
    fsdk::HumanLandmarks17 humanLandmarks;

    //! NOTE: only for internal usage, don't use this field, it isn't valid ptr
    fsdk::IDescriptorPtr descriptor;
#endif

    //! Is it full detection or redetect step
    bool isFullDetect;

    //! Detections flags
    // needed to determine what detections are valid in extraDetections 
    // see EDetectionFlags
    uint32_t detectionsFlags;

    //! Detection
    // always is valid, even when detectionsFlags is combination type
    // useful for one detector case
    // see detectionObject
    fsdk::Detection detection;
};

• void trackEnd(const tsdk::TrackId& trackId) - Indicates that a track with trackId has ended and no more best shots should be expected from it. Default implementation does nothing.

  • trackId - ID of a track.

/** @brief Track status enum. (see human tracking algorithm section in docs for details)
*/
enum class TrackStatus : uint8_t {
    ACTIVE = 0,
    NONACTIVE
};

IVisualObserver

• void visual(const tsdk::FrameId &frameId, const fsdk::Image &image, const tsdk::TrackInfo * trackInfo, const int nTrack) - Allows to visualize the current stream state. It is intended mainly for debugging purposes. The function must be overloaded.

  • frameId - Current frame ID.
  • image - Frame image.
  • trackInfo - Array of currently active tracks.
  • nTrack - Number of tracks.

struct TRACK_ENGINE_API TrackInfo {
        //! Face landmarks
        fsdk::Landmarks5 landmarks;

#if !TE_MOBILE_BUILD
        //! Human landmarks
        fsdk::HumanLandmarks17 humanLandmarks;
#endif

        //! Last detection for track
        fsdk::Rect rect;

        //! Id of track
        TrackId trackId;

        //! Score for last detection in track
        float lastDetectionScore;

        //! Detector id
        TDetectorID m_sourceDetectorId;

        //! number of detections for track (count of frames when track was updated with detect/redetect)
        size_t detectionsCount;

        //! id of frame, when track was created
        tsdk::FrameId firstFrameId;

        //! Is it (re)detected or tracked bounding box
        bool isDetector;
};

IDebugObserver

• void debugDetection(const tsdk::DetectionDebugInfo& descr) - Detector debug callback. Default implementation does nothing.

  • descr - Detection debugging description.

struct DetectionDebugInfo {
    //! Detection description
    DetectionDescr descr;

    //! Is it detected or tracked bounding box
    bool isDetector;

    //! Filtered by user bestShotPredicate or not.
    bool isFiltered;

    //! Best detection for current moment or not
    bool isBestDetection;
};

• void debugForegroundSubtraction(const tsdk::FrameId& frameId, const fsdk::Image& firstMask, const fsdk::Image& secondMask, fsdk::Rect * regions, int nRegions) - Background subtraction debug callback. Default implementation does nothing.

  • frameId - Foreground frame ID.
  • firstMask - Result of the background subtraction operation.
  • secondMask - Result of the background subtraction operation after procedures of erosion and dilation.
  • regions - Regions obtained after the background subtraction operation.
  • nRegions - Number of returned regions.

BestShotPredicate

• bool checkBestShot(const tsdk::DetectionDescr& descr) - Predicate for best shot detection. This is the place to perform any required quality checks (by means of, for example, FaceEngines Estimators). This function must be overloaded.

  • descr - Detection description.
  • return value - true, if descr has passed the check, false otherwise.

VisualPredicate

• bool needRGBImage(const tsdk::FrameId frameId, const tsdk::AdditionalFrameData *data) - Predicate for visual callback. It serves to decide whether to output original image in visual callback or not. This function can be overloaded. Default implementation returns true.

  • frameId - ID of the frame.
  • data - Frame additional data passed by user.
  • return value - true, if the original image or an RGB image for a custom frame needed in output in visual callback, false otherwise.

IBatchBestShotObserver

• void bestShot(const fsdk::Span &streamIDs, const fsdk::Span &data) - Batched version of the bestShot callback.

  • streamIDs - Array of stream IDs.
  • data - Array of callback data for each stream.

struct TRACK_ENGINE_API BestShotCallbackData {
    //! detection description. see 'DetectionDescr' for details
    tsdk::DetectionDescr descr;

    //! additional frame data, passed by user in 'pushFrame'. see 'AdditionalFrameData' for details
    tsdk::AdditionalFrameData *frameData;
};

• void trackEnd(const fsdk::Span &streamIDs, const fsdk::Span &data) - Batched version of the trackEnd callback.

  • streamIDs - Array of stream IDs.
  • data - Array of callback data for each stream.

/**
* @brief Track end reason. See 'TrackEndCallbackData' for details.
*/
enum class TrackEndReason : uint8_t {
    //! not used anymore, deprecated value (may be removed in future releases)
    DEFAULT,
    //! note: deprecated field, not used anymore
    UNKNOWN,
    //! intersection with another track (see "kill-intersected-detections")
    INTERSECTION,
    //! tracker is disabled or failed to update track
    TRACKER_FAIL,
    //! track's gone out of frame
    OUT_OF_FRAME,
    //! `skip-frames` parameter logic (see docs or config comments for details)
    SKIP_FRAMES,
    //! note: deprecated field, not used anymore
    USER,
    //! note: deprecated field, not used anymore
    NONACTIVE_TIMEOUT,
    //! note: deprecated field, not used anymore
    ACTIVE_REID,
    //! note: deprecated field, not used anymore
    NONACTIVE_REID,
    //! all stream tracks end on stream finishing (IStream::join called)
    STREAM_END
};

struct TRACK_ENGINE_API TrackEndCallbackData {
    //! frame id
    tsdk::FrameId frameId;

    //! track id
    tsdk::TrackId trackId;

    //! parameter implies reason of track ending
    // NOTE: now it's using only for human tracking, don't use this for other detectors
    TrackEndReason reason;
};

IBatchVisualObserver

• void visual(const fsdk::Span &streamIDs, const fsdk::Span &data) - Batched version of the visual callback.

  • streamIDs - Array of stream IDs.
  • data - Array of callback data for each stream.

struct TRACK_ENGINE_API VisualCallbackData {
    //! frame id
    tsdk::FrameId frameId;

    //! this is either original image (if 'pushFrame' used) or RGB image got from custom frame convert (is 'pushCustomFrame' used)
    fsdk::Image image;

    //! tracks array raw ptr
    tsdk::TrackInfo *trackInfo;

    //! number of tracks
    int nTrack;

    //! additional frame data, passed by user in 'pushFrame'. See 'AdditionalFrameData' for details.
    tsdk::AdditionalFrameData *frameData;
};

IBatchDebugObserver

• void debugForegroundSubtraction(const fsdk::Span &streamIDs, const fsdk::Span &data) - Batched version of the debugForegroundSubtraction callback.

  • streamIDs - Array of stream IDs.
  • data - Array of callback data for each stream.

• void debugDetection(const fsdk::Span &streamIDs, const fsdk::Span &data) - Batched version of the debugDetection callback.

  • streamIDs - Array of stream IDs.
  • data - Array of callback data for each stream.

struct TRACK_ENGINE_API DebugForegroundSubtractionCallbackData {
    //! frame id
    tsdk::FrameId frameId;

    //! first mask of  the foreground subtraction
    fsdk::Image firstMask;

    //! second mask of  the foreground subtraction
    fsdk::Image secondMask;

    //! regions array raw ptr
    fsdk::Rect *regions;

    //! number of regions
    int nRegions;
};

/** @brief Detection data for debug callback.
    */
struct TRACK_ENGINE_API DebugDetectionCallbackData {
    //! Detection description
    DetectionDescr descr;

    //! Is it detected or tracked bounding box
    bool isDetector;

    //! Filtered by user bestShotPredicate or not.
    bool isFiltered;

    //! Best detection for current moment or not
    bool isBestDetection;
};

ITrackingResultObserver

• void IStream::setObserverEnabled(tsdk::StreamObserverType type, bool enabled) - Enables or disables the observer.

  • type - Observer type.
  • enabled - Flag to enable or disable the observer

For full Stream API, see class IStream from IStream.h header file.

/* @brief Ready tracking result observer interface.
*/
struct TRACK_ENGINE_API ITrackingResultObserver {
    /**
    * @brief Ready tracking result notification
    * @param value tracking result. See 'ITrackingResultBatch' for details
    * @note pure virtual method
    */
    virtual void ready(fsdk::Ref<ITrackingResultBatch> result) = 0;

    virtual ~ITrackingResultObserver() = default;
};

/** @brief Tracking results per stream/frame.
        It involves different tracking data/events per stream/frame
*/
struct TrackingResult {
    //! stream Id
    tsdk::StreamId streamId;

    //! frame Id
    tsdk::FrameId frameId;

    //! Source image
    fsdk::Image image;

    //! Source custom frame
    fsdk::Ref<ICustomFrame> customFrame;

    //! data passed by user
    tsdk::AdditionalFrameData *userData;

    //! array of trackStart (new tracks) events
    fsdk::Span<TrackStartCallbackData> trackStart;

    //! array of trackEnd (finished tracks) events
    fsdk::Span<TrackEndCallbackData> trackEnd;

    //! array of current tracks data (each element of array matches to tracks of specific `EDetectionObject`)
    fsdk::Span<VisualCallbackData> tracks;

    //! array of detections, so this implies only detections, received from Detector (bestshots in case of custom `checkBestShot` func)
    // note: `detections` can be deduced from `tracks` actually (see fields `isDetector`, `isFullDetect`)
    // but can be useful when `checkBestShot` is used or in simple cases, when only detections from Detector are needed
    fsdk::Span<BestShotCallbackData> detections;

    //! array of debug detections data (it, mostly, copies data from `tracks` with some extra debug params like `isFiltered`, `isBestDetection`)
    // note: deprecated field, use `tracks` and `detections` instead for better experience
    fsdk::Span<DebugDetectionCallbackData> debugDetections;

    //! debug foreground subtractions data (mostly, used for debug purposes, but can be used to get FRG info)
    fsdk::Optional<DebugForegroundSubtractionCallbackData> debugForegroundSubtraction;

    //! human tracks, including both face and body (new API version of `tracks`)
    // NOTE: visual observer must be enabled to get this, as it's derived from `tracks`
    fsdk::Span<HumanTrackInfo> humanTracks;
};

/** @brief Tracking results batch as 2D vector of stream/frame.
    It contains tracking results for several frames per each stream
*/
struct ITrackingResultBatch : public fsdk::IRefCounted {
    /**
        * @brief Get array of stream identifiers, tracking results are ready for.
        * @return span of Stream ids.
        * */
    virtual fsdk::Span<tsdk::StreamId> getStreamIds() const = 0;

    /**
        * @brief Get array of frame identifiers for given Stream, tracking results are ready for.
        * @param streamId id of the Stream.
        * @note streamId can be any value (not only from `getStreamIds`), so func returns empty span, if Stream has no ready tracking results yet.
        * @return span of frame ids in tracking result for specific Stream.
        * */
    virtual fsdk::Span<tsdk::FrameId> getStreamFrameIds(tsdk::StreamId streamId) const = 0;

    /**
        * @brief Get tracking result by stream Id and frame Id
        * @param streamId id of the Stream, tracking result requested for. It should be one of the `getStreamIds` array elements.
        * @param frameId id of the Frame, tracking result requested for. It should be one of the `getStreamFrameIds` array elements for `streamId`.
        * @return Tracking result for Stream/Frame. See `TrackingResult` struct for details.
        * @note Return data is valid while the parent ITrackingResultBatch is alive.
        * @throws std::invalid_argument if streamId is not one of the `getStreamIds` array elements.
        * @throws std::invalid_argument if frameId is not one of the `getStreamFrameIds` array elements for passed `streamId`.
        * */
    virtual TrackingResult getTrackingResult(tsdk::StreamId streamId, tsdk::FrameId frameId) const = 0;
};