Observers

IBestShotObserver

• void bestShot(const tsdk::DetectionDescr& descr) called for each emerged best shot. It provides information on a best shot, including frame number, detection coordinates, cropped still image, and other data (see `DetectionDescr structure definition below for details.) Default implementation does nothing.

  • descr - best shot detection description

struct TRACK_ENGINE_API DetectionDescr {
    //! Index of the frame
    tsdk::FrameId frameIndex;

    //! Index of the track
    tsdk::TrackId trackId;

    //! Source image
    fsdk::Image image;

    fsdk::Ref<ICustomFrame> customFrame;

    //! Face landmarks
    fsdk::Landmarks5 landmarks;

#ifndef MOBILE_BUILD
    //! Human landmarks
    fsdk::HumanLandmarks17 humanLandmarks;

    //! NOTE: only for internal usage, don't use this field, it isn't valid ptr
    fsdk::IDescriptorPtr descriptor;
#endif

    //! Is it full detection or redetect step
    bool isFullDetect;

    //! Detections flags
    // needed to determine what detections are valid in extraDetections 
    // see EDetectionFlags
    uint32_t detectionsFlags;

    //! Detection
    // always is valid, even when detectionsFlags is combination type
    // useful for one detector case
    // see detectionObject
    fsdk::Detection detection;
};

• void trackEnd(const tsdk::TrackId& trackId) tells that the track with trackId has ended and no more best shots should be expected from it. Default implementation does nothing.

  • trackId - id of the track

/** @brief Track status enum. (see human tracking algorithm section in docs for details)
*/
enum class TrackStatus : uint8_t {
    ACTIVE = 0,
    NONACTIVE
};

IVisualObserver

• void visual(const tsdk::FrameId &frameId, const fsdk::Image &image, const tsdk::TrackInfo * trackInfo, const int nTrack) allows to visualize current stream state. It is intended mainly for debugging purposes. The function must be overloaded.

  • frameId - current frame id
  • image - frame image
  • trackInfo - array of currently active tracks

struct TRACK_ENGINE_API TrackInfo {
        //! Face landmarks
        fsdk::Landmarks5 landmarks;

#if !TE_MOBILE_BUILD
        //! Human landmarks
        fsdk::HumanLandmarks17 humanLandmarks;
#endif

        //! Last detection for track
        fsdk::Rect rect;

        //! Id of track
        TrackId trackId;

        //! Score for last detection in track
        float lastDetectionScore;

        //! Detector id
        TDetectorID m_sourceDetectorId;

        //! number of detections for track (count of frames when track was updated with detect/redetect)
        size_t detectionsCount;

        //! id of frame, when track was created
        tsdk::FrameId firstFrameId;

        //! Is it (re)detected or tracked bounding box
        bool isDetector;
};

• nTrack - number of tracks

IDebugObserver

• void debugDetection(const tsdk::DetectionDebugInfo& descr) detector debug callback. Default implementation does nothing.

  • descr - detection debugging description

struct DetectionDebugInfo {
    //! Detection description
    DetectionDescr descr;

    //! Is it detected or tracked bounding box
    bool isDetector;

    //! Filtered by user bestShotPredicate or not.
    bool isFiltered;

    //! Best detection for current moment or not
    bool isBestDetection;
};

• void debugForegroundSubtraction(const tsdk::FrameId& frameId, const fsdk::Image& firstMask, const fsdk::Image& secondMask, fsdk::Rect * regions, int nRegions) background subtraction debug callback. Default implementation does nothing.

  • frameId - frame id of foreground
  • firstMask - result of background subtraction operation
  • secondMask - result of background subtraction operation after procedures of erosion and dilation
  • regions - regions obtained after background subtraction operation
  • nRegions - number of returned regions

BestShotPredicate

• bool checkBestShot(const tsdk::DetectionDescr& descr) Predicate for best shot detection. This is the place to perform any required quality checks (by means of, e.g. FaceEngines Estimators). This function must be overloaded.

  • descr - detection description
  • return value - true, if descr has passed the check, false otherwise

VisualPredicate

• bool needRGBImage(const tsdk::FrameId frameId, const tsdk::AdditionalFrameData *data) Predicate for visual callback. It serves to decide whether to output original image in visual callback or not. This function can be overloaded. Default implementation returns true.

  • frameId - id of the frame
  • data - frame additional data, passed by user
  • return value - true, if original image (or rgb image for custom frame) needed in output in visual callback, false otherwise

IBatchBestShotObserver

• void bestShot(const fsdk::Span &streamIDs, const fsdk::Span &data) Batched version of the bestShot callback.

  • streamIDs - array of streams id
  • data - array of callback data for each stream

struct TRACK_ENGINE_API BestShotCallbackData {
    //! detection description. see 'DetectionDescr' for details
    tsdk::DetectionDescr descr;

    //! additional frame data, passed by user in 'pushFrame'. see 'AdditionalFrameData' for details
    tsdk::AdditionalFrameData *frameData;
};

• void trackEnd(const fsdk::Span &streamIDs, const fsdk::Span &data) Batched version of the trackEnd callback.

  • streamIDs - array of streams id
  • data - array of callback data for each stream

/**
* @brief Track end reason. See 'TrackEndCallbackData' for details.
*/
enum class TrackEndReason : uint8_t {
    //! not used anymore, deprecated value (may be removed in future releases)
    DEFAULT,
    //! note: deprecated field, not used anymore
    UNKNOWN,
    //! intersection with another track (see "kill-intersected-detections")
    INTERSECTION,
    //! tracker is disabled or failed to update track
    TRACKER_FAIL,
    //! track's gone out of frame
    OUT_OF_FRAME,
    //! `skip-frames` parameter logic (see docs or config comments for details)
    SKIP_FRAMES,
    //! note: deprecated field, not used anymore
    USER,
    //! note: deprecated field, not used anymore
    NONACTIVE_TIMEOUT,
    //! note: deprecated field, not used anymore
    ACTIVE_REID,
    //! note: deprecated field, not used anymore
    NONACTIVE_REID,
    //! all stream tracks end on stream finishing (IStream::join called)
    STREAM_END
};

struct TRACK_ENGINE_API TrackEndCallbackData {
    //! frame id
    tsdk::FrameId frameId;

    //! track id
    tsdk::TrackId trackId;

    //! parameter implies reason of track ending
    // NOTE: now it's using only for human tracking, don't use this for other detectors
    TrackEndReason reason;
};

IBatchVisualObserver

• void visual(const fsdk::Span &streamIDs, const fsdk::Span &data) Batched version of the visual callback.

  • streamIDs - array of streams id
  • data - array of callback data for each stream

struct TRACK_ENGINE_API VisualCallbackData {
    //! frame id
    tsdk::FrameId frameId;

    //! this is either original image (if 'pushFrame' used) or RGB image got from custom frame convert (is 'pushCustomFrame' used)
    fsdk::Image image;

    //! tracks array raw ptr
    tsdk::TrackInfo *trackInfo;

    //! number of tracks
    int nTrack;

    //! additional frame data, passed by user in 'pushFrame'. See 'AdditionalFrameData' for details.
    tsdk::AdditionalFrameData *frameData;
};

IBatchDebugObserver

• void debugForegroundSubtraction(const fsdk::Span &streamIDs, const fsdk::Span &data) Batched version of the debugForegroundSubtraction callback.

  • streamIDs - array of streams id
  • data - array of callback data for each stream

• void debugDetection(const fsdk::Span &streamIDs, const fsdk::Span &data) Batched version of the debugDetection callback.

  • streamIDs - array of streams id
  • data - array of callback data for each stream

struct TRACK_ENGINE_API DebugForegroundSubtractionCallbackData {
    //! frame id
    tsdk::FrameId frameId;

    //! first mask of  the foreground subtraction
    fsdk::Image firstMask;

    //! second mask of  the foreground subtraction
    fsdk::Image secondMask;

    //! regions array raw ptr
    fsdk::Rect *regions;

    //! number of regions
    int nRegions;
};

/** @brief Detection data for debug callback.
    */
struct TRACK_ENGINE_API DebugDetectionCallbackData {
    //! Detection description
    DetectionDescr descr;

    //! Is it detected or tracked bounding box
    bool isDetector;

    //! Filtered by user bestShotPredicate or not.
    bool isFiltered;

    //! Best detection for current moment or not
    bool isBestDetection;
};

ITrackingResultObserver

/* @brief Ready tracking result observer interface.
*/
struct TRACK_ENGINE_API ITrackingResultObserver {
    /**
    * @brief Ready tracking result notification
    * @param value tracking result. See 'ITrackingResultBatch' for details
    * @note pure virtual method
    */
    virtual void ready(fsdk::Ref<ITrackingResultBatch> result) = 0;

    virtual ~ITrackingResultObserver() = default;
};

/** @brief Tracking results per stream/frame.
        It involves different tracking data/events per stream/frame
*/
struct TrackingResult {
    //! stream Id
    tsdk::StreamId streamId;

    //! frame Id
    tsdk::FrameId frameId;

    //! Source image
    fsdk::Image image;

    //! Source custom frame
    fsdk::Ref<ICustomFrame> customFrame;

    //! data passed by user
    tsdk::AdditionalFrameData *userData;

    //! array of trackStart (new tracks) events
    fsdk::Span<TrackStartCallbackData> trackStart;

    //! array of trackEnd (finished tracks) events
    fsdk::Span<TrackEndCallbackData> trackEnd;

    //! array of current tracks data (each element of array matches to tracks of specific `EDetectionObject`)
    fsdk::Span<VisualCallbackData> tracks;

    //! array of detections, so this implies only detections, received from Detector (bestshots in case of custom `checkBestShot` func)
    // note: `detections` can be deduced from `tracks` actually (see fields `isDetector`, `isFullDetect`)
    // but can be useful when `checkBestShot` is used or in simple cases, when only detections from Detector are needed
    fsdk::Span<BestShotCallbackData> detections;

    //! array of debug detections data (it, mostly, copies data from `tracks` with some extra debug params like `isFiltered`, `isBestDetection`)
    // note: deprecated field, use `tracks` and `detections` instead for better experience
    fsdk::Span<DebugDetectionCallbackData> debugDetections;

    //! debug foreground subtractions data (mostly, used for debug purposes, but can be used to get FRG info)
    fsdk::Optional<DebugForegroundSubtractionCallbackData> debugForegroundSubtraction;

    //! human tracks, including both face and body (new API version of `tracks`)
    // NOTE: visual observer must be enabled to get this, as it's derived from `tracks`
    fsdk::Span<HumanTrackInfo> humanTracks;
};

/** @brief Tracking results batch as 2D vector of stream/frame.
    It contains tracking results for several frames per each stream
*/
struct ITrackingResultBatch : public fsdk::IRefCounted {
    /**
        * @brief Get array of stream identifiers, tracking results are ready for.
        * @return span of Stream ids.
        * */
    virtual fsdk::Span<tsdk::StreamId> getStreamIds() const = 0;

    /**
        * @brief Get array of frame identifiers for given Stream, tracking results are ready for.
        * @param streamId id of the Stream.
        * @note streamId can be any value (not only from `getStreamIds`), so func returns empty span, if Stream has no ready tracking results yet.
        * @return span of frame ids in tracking result for specific Stream.
        * */
    virtual fsdk::Span<tsdk::FrameId> getStreamFrameIds(tsdk::StreamId streamId) const = 0;

    /**
        * @brief Get tracking result by stream Id and frame Id
        * @param streamId id of the Stream, tracking result requested for. It should be one of the `getStreamIds` array elements.
        * @param frameId id of the Frame, tracking result requested for. It should be one of the `getStreamFrameIds` array elements for `streamId`.
        * @return Tracking result for Stream/Frame. See `TrackingResult` struct for details.
        * @note Return data is valid while the parent ITrackingResultBatch is alive.
        * @throws std::invalid_argument if streamId is not one of the `getStreamIds` array elements.
        * @throws std::invalid_argument if frameId is not one of the `getStreamFrameIds` array elements for passed `streamId`.
        * */
    virtual TrackingResult getTrackingResult(tsdk::StreamId streamId, tsdk::FrameId frameId) const = 0;
};

• void IStream::setObserverEnabled(tsdk::StreamObserverType type, bool enabled) Enables or disables observer.

  • type - type of observer
  • enabled - flag to enable/disable observer

For full Stream API see class IStream from IStream.h header file.