Video-Stream Sources#

The application supports simultaneous work with several video-stream sources.

For the normal FaceStream mode, the source configuration is set through the configuration file ./bin/data/input.json. The file is required for FaceStream launch. You should specify full name to the file via the command-line argument -Src (--source-path).

This file is not used in the server mode.

For the server FaceStream mode, the source configuration is set in the Configurator service (see "Set up Configurator to work with FaceStream").

Several types of sources are supported:

stream-sources – real time video sources. It can be both USB-cameras and IP-cameras (via RTSP).
video-sources - video files.
images-sources - a set of frames in the form of separate image files.

All sources to be processed by the application are registered into the source configuration. The description of the configuration file parameters is given below.

Name#

Source identification. It serves to identify the frames sent by the source.

"name": "stream_0",

When sending to LUNA PLATFORM 4, the "name" value and the track ID are used to generate the external_idfield. This field has a size limit of 36 characters. Due to this, you must the size of the "name" parameter to twenty characters!

ID#

This parameter is only used when FaceStream is running in server mode. This parameter is not included in the configuration file for normal FaceStream mode.

This parameter sets the ID of the real-time video source. The parameter is required for connecting video streams specified in the Configurator service when starting FaceStream.

This parameter is also set when sending a request to create a stream source using the FaceStream API.

"id": "8b7678c0-779a-4ec5-8c69-4d55bb4a492d",

Input#

ROI#

ROI specifies the region of interest in which the face detection and tracking are performed.

The specified rectangular area is cut out from the frame and FaceStream performs further processing using this image.

Correct exploitation of the "roi" parameter significantly improves the performance of FaceStream.

Region of interest on the original frame is set in pixels as array [x, y width, height], where (x, y) are the coordinates of the upper-left point of the region of interest. Coordinate system is set in the same way as it is shown on the picture below.

When the values of width and height are set to "0", the entire frame will be the region of interest.

"roi": [0, 0, 0, 0],

DROI#

The parameter specifies the region of interest within the ROI zone. Face detection is performed in ROI, but the best shot is selected only in the DROI area. Face detection must be completely within the DROI zone for the frame to be considered as a best shot.

DROI is recommended to use when working with Access Control Systems and when the\ "use_mask_liveness_filtration" mode is enabled.

For example, it can be used, if there are several turnstiles close to each other and their cameras should find faces only in a small area and simultaneously perform Liveness check. Using DROI enables to limit the area of the best shot selection without losing information about the background.

Region of interest on the original frame is set in pixels as array [x, y width, height], where (x, y) are the coordinates of the upper-left point of the ROI.

Coordinate system is set in the same way as it is shown in the picture below.

When the ROI size is changed and the DROI size remains the default (0, 0, 0, 0), the DROI is not considered. If you change the size of the DROI, it will be considered when choosing the best frame.

"droi": [0, 0, 0, 0],

Rotation#

The rotation angle of the image source. It is used when the incoming video-stream is rotated, for example, if the camera is installed on the ceiling.

"rotation": 0,

Transport#

Transport is used only for stream-sources.

The protocol of video stream transmission. The application can use one of two network protocols to receive video data: TCP or UDP. TCP is set in the application by default.

"transport": "tcp",

TCP Protocol implements an error control mechanism that minimizes the loss of information and the skip of the reference frames at the cost of increasing the network delay. Key frames are the basis of various compression algorithms used in video codecs (for example, h264). Only the reference frames contain enough information to restore (decode) the image completely, while the intermediate frames contain only differences between adjacent reference frames.

In terms of broadcasting on the network, there is a risk of package loss due to imperfect communication channels. In case of loss of the package containing the data keyframe, the video-stream fragment cannot be correctly decoded. Consequently, distinctive artifacts appear, that are easily and visually distinguishable. These artifacts do not allow the face detector to operate in normal mode.

The UDP protocol does not implement an error control mechanism, so the video-stream is not protected from damage. The use of this protocol is recommended only, if there is a high-quality network infrastructure.

Url#

Full path to the source or a USB-device number (for stream-sources).

"url": "rtsp://some_stream_address"

Full path to the video file (for video-sources).

"url": https://127.0.0.1/super_server/

Full path to the directory with the images.

"url": "/example1/path/to/images/"

Number-Of-Ffmpeg-Threads#

The parameter allows to specify the number of threads for decoding video using FFMPEG.

The number of processor cores involved in decoding process increases according to the number of threads. An increase in the number of threads is recommended when processing high-resolution video (4K or higher).

"numberOfFfmpegThreads" : 8

Frame-processing-mode#

This parameter is used for stream-sources and video-sources only.

This parameter is similar to Convert-full-frame, but is set for a specific FaceStream instance.

If the value is set to "full", the frame is immediately converted to RGB image of the required size after decoding. This results in a better image quality and reduces the speed of frames processing.

When set to "scale", the image is scaled according to the settings in the "trackengine.conf" file (standard behavior for releases 3.2.4 and earlier).

The default value is "auto". In this case, one of the two modes is selected automatically.

"frame-processing-mode": "auto",

Real-time-mode-fps#

This parameter is used for video-sources only.

The video is processed with the specified number of FPS.

The video can't be processed with FPS higher than the one specified in this parameter.

If a video has high FPS value and FaceStream cannot work with the specified number of frames per second, the frames are skipped.

Thus, the video file emits a video stream from a real video camera. It can be useful for performance tuning. The video will be played at the specified speed, which is convenient for load testing and subsequent analysis.

"real_time_mode_fps" : 0

Mask#

The Mask parameter is used only for images-sources.

A mask of file names in the directory with images. The mask allows FaceStream to understand which files from the specified folder should be used and in what order.

If you set the mask "Img_%02d.jpg", then FaceStream will take from the folder files which names consist of: Prefix (Img_) + two-digit number (%02d) + format (.jpg)

The following images will be taken in turn:

Img_00.jpg
Img_01.jpg
Img_02.jpg
Img_03.jpg

Another example of a mask is Photo-%09d.jpg. The following images will be taken:

Photo-000000000.jpg
Photo-000000001.jpg
Photo-000000002.jpg
Photo-000000003.jpg

FaceStream processes files in numerical order and does not skip nonexistent files. If there is a missing file in the file sequence FaceStream stops files processing.

The specified mask "example1_%04d.jpg" in the example will result in image processing, which name is composed of an "example1_" prefix and of a sequential frame number, of 4 characters size (for example: example1_0001.jpg, example1_0002.jpg, etc.).

"mask": "example1_%04d.jpg"

Output#

The Login, Password, and Token sections relate to interactions with the LUNA PLATFORM. Detailed information may be found in LUNA PLATFORM documentation.

A login for authorization on the LUNA PLATFORM server.

"login": "loginExample",

Password#

A password for the authorization on the LUNA PLATFORM server.

"password": "passwordExample",

Token#

Token for authorization on the server for LUNA (UUID4) requests can be used with login and password, or without them. For more details, see documentation for LUNA.

"token": "deadbeef-0000-1111-2222-deadbeef0000",

Url#

A full network path for sending images from this source.

"url": "https://127.0.0.1/super_server/"

For LUNA PLATFORM 3, you should specify URL used for the "search" request creation. A list should be already created in LUNA PLATFORM 3.

"url": "http://<luna_adress>:<port>/4/matching/search?list_id="

For LUNA PLATFORM 4, you should specify URL used for the "generate events" request creation. A handler should be already created in LUNA PLATFORM 4.

"url": "http://<luna_adress>:<port>/5/handlers//events"

luna_adress - the address of the server with LUNA PLATFORM API service,

port - the port used by the LUNA PLATFORM API service.

Filtering#

The section describes the filter object parameters and modes of sending the resulting portraits.

"filtering": {
    "min-score": 0.5187,
    "detection-yaw-threshold": 40,
    "detection-pitch-threshold": 40,
    "detection-roll-threshold": 30,
    "yaw-number": 1,
    "yaw-collection-mode": false,
    "mouth-occlusion-threshold" : 0.0
},

Min-score#

Min-score, also known as Approximate Garbage Score (AGS) – score that defines detection quality, threshold for filtering detections sent to the server. All detections with score higher than the value of this parameter can be sent to the server as HTTP-requests, otherwise detections are considered as not appropriate for further analysis.

If a new detection has a higher AGS than those in the existing collection, it will replace the detection with lowest AGS.

Recommended value was established through research and analysis of detections on various face images.

"min-score" : 0.5187,

Detection-yaw-threshold#

Detection-yaw-threshold parameter sets the maximum value of head yaw angle in relation to camera.

If, in a frame, head yaw angle is above the value of this parameter, the frame is considered as not appropriate for further analysis.

"detection-yaw-threshold" : 40,

Detection-pitch-threshold#

Detection-pitch-threshold parameter sets the maximum value of head pitch angle in relation to camera.

If, in a frame, head pitch angle is above the value of this parameter, the frame is considered as not appropriate for further analysis.

"detection-pitch-threshold" : 40,

Detection-roll-threshold#

Detection-roll-threshold parameter sets the maximum value of head yaw angle in relation to camera.

If, in a frame, head roll angle is above the value of this parameter, the frame is considered as not appropriate for further analysis.

"detection-roll-threshold" : 30,

Yaw-number#

Yaw-number parameter defines the number of frames for image filtration based on head tilt angle. This filter removes images where head’s yaw angle is too high.

How it works:

Parameter specifies the number of frames to analyze. A special algorithm analyzes head yaw angles on each of those frames. If on one of them the angle is significantly different from the average value of angles, the frame will not be considered as a candidate for best shot.

Example. Parameter value is set "7", meaning 7 frames will be analyzed. If on six of the frames the rotation angle is in the range between 50-60 degrees and the angle on the seventh frame is estimated at 0, the angle on the seventh frame is, most likely, estimated incorrectly. Reason is: a person cannot turn his head so abruptly in such short period of time. The seventh frame will not be considered for best shot.

By default, the parameter is disabled, the value is "1". The recommended value is "7".

"yaw-number": 1,

Yaw-collection-mode#

Yaw-collection-mode parameter sets the number of frames the system must collect to analyze head yaw angle. Best shot will be selected from those.

If yaw-collection-mode is disabled, the system will analyze the frames sequentially, meaning it analyzes one frame, then two, then three and so on. Maximum number of frames to analyze is set in yaw-mode parameter.

Parameter is disabled by default.

"yaw-collection-mode" : false,

The purpose of utilizing yaw-number and yaw-collection-mode parameters is to increase the accuracy of best shot selection from a track.

Mouth-occlusion-threshold#

Mouth-occlusion-threshold parameter determines how much the mouth can be obscured in the frame.

I.e. when the value is equal to "0.5", 50% of the face can be occluded.

If mouth occlusion of a face in a frame exceeds the value of this threshold, the frame is considered as not appropriate for further analysis.

The filtration is performed when the set value is "0.3" or higher. When the value is lover, the filtration is disabled.

"mouth-occlusion-threshold" : 0.0,

Sending#

Sending – object defines a period within a track during which the analysis for best shot is conducted . In addition, it allows to define parameters connected to forming bestshot collection.

"sending": {
"time-period-of-searching": -1,
"silent-period" : 0,
"type" : "sec",
"number-of-bestshots-to-send": 1
},

Time-period-of-searching#

Time-period-of-searching – interval in track after the end of which a best shot is sent to the server (period starts with the first detection – person appears in the frame) .
Lowering this parameter speeds up recognition but decreases precision.

If the value equals "-1", analysis is conducted on all frames until the end of track. Once the track is over (person leaves the frame), best shot is sent to an external service.

"time-period-of-searching": -1,

Silent-period#

Silent-period – interval between period. Once the analysis period is over, the system holds this silent-period before starting next period of frame analysis.

"silent-period" : 0,

If the value equals "-1", system holds silent period indefinitely.

Type#

Type – sets the measurement metric for analysis periods and intervals between those (per frames or per seconds).

"type" : "sec",

Number-of-bestshots-to-send#

Number-of-bestshots-to-send – number of frames that the user sets to receive from the track or certain periods of this track. This parameter enables collection of best shots from a track or a certain period of a track set in parameter time-period-of-searching-bd. The collection of frames is then sent to the server.

FaceStreamManager can aggregate several warped images from a single track into a LUNA PLATFORM person.

"number-of-bestshots-to-send": 1

Increasing parameter’s value increases the probability of correct face recognition but affects the network load.

Health-check#

The section is used only for the stream-sources and video-sources types.

In this group, you can set the parameters for reconnecting to the video stream when errors occur while the video is streamed.

By default, there is no health_check section in the video-sources configuration, but it can be added there manually. It is required when errors occur during the processing of the video file or the video is interrupted.

Max-error-count#

The maximum number of errors that may occur during a video playback. For example, when it is unable to retrieve or decode a frame. Network problems or inaccessibility of a video can cause the errors.

"max_error_count": 10,

Period#

The parameter represents the period during which the number of errors is calculated. The value is set in seconds. An attempt to reconnect to the video stream is performed when the number of errors, specified in the max_error_count parameter, is reached.

"period": 3600,

Retry-delay#

The parameter specifies the period after which the reconnection attempt is performed. The value is set in seconds.

"retry_delay": 5

Primary-track-policy#

The parameters group is not used for the images-sources type.

This group of parameters is designed to work with Access Control Systems (ACS, turnstiles at the entrances to banks/office buildings) to simplify the control and the introduction of facial recognition technology at the entrance to a protected area.

Use-primary-track-policy#

Use-primary-track-policy parameter is used in cases of Access Control Systems (turnstiles/gates at the office/bank entrances) for easier control and face recognition implementation in a secured area.

If the parameter value is "true", the primary track implementation mode is enabled.

Workflow:

Out of all detections, one of the biggest sizes is selected and its track becomes the primary one. Further analysis is conducted on this track. The best shot from this track is then sent to the server.

All other tracks are processed in regular mode. However, the best shot is sent only from the primary track.

As soon as another face reaches a larger size than the face from the primary track, this face track becomes primary and the processing is performed for it.

While using this parameter at the access control checkpoint, only the best shots of the person who is the closest to the turnstiles will be sent to the server (here the biggest detection size condition is held)

"use_primary_track_policy": false,

Best-shot-min-size#

The parameter is used when "use-primary-track-policy" parameter is enabled

Best-shot-min-size parameter sets the minimal size of detection at which the analysis of frames and bestshot definition begins.

"best_shot_min_size": 70,

Best-shot-proper-size#

The parameter is used when "use-primary-track-policy" parameter is enabled

Best-shot-proper-size – parameter sets the size of detection for Primary Track policy. When a detection reaches the defined value, track immediately sends all its best shots to the server.

"best_shot_proper_size": 140

Liveness#

The parameters group is not used for the images-sources type.

It is recommended to use this functionality only after discussing it with VisionLabs team.

Liveness is used to check whether a person in the frame is real and prevents fraud when printed photos or images on the phone are used to pass the Liveness check.

Liveness can be used at access control checkpoints, to minimize the risk of fraud when someone is trying to enter a secured area using a printed photo or a photo on a phone of someone who has the access rights.

Liveness returns a value, which defines the degree of system’s certainty on whether the person is real in the frame. The value is in the range from 0 to 1.

The following conditions must be met for Liveness check set up:

Face should remain within a frame. The distance from left and right

edges of the frame should be greater than or equal to the width of the face, the distance from the top and bottom edges of the frame should be greater than or equal to the height of the face;
The frame should include chest region;
A camera should be located about waist height and should look

upwards capturing the body and head;
The frame should not include rectangular elements framing the face

area from all four sides (such as doorways or windows).

Use-shoulders-liveness-filtration#

The parameter enables checking the presence of a real person in the frame based on the head and shoulder areas.

"use_shoulders_liveness_filtration": false,

Use-mask-liveness-filtration#

The parameter enables checking the presence of a real person in the frame based on backgrounds.

The check performance depends on the size of the video frames. If the processing speed decreases when the parameter is enabled, it is necessary to reduce the video resolution in the camera settings (e.g., up to 1280x720).

"use_mask_liveness_filtration": false,

Use-flying-faces-liveness-filtration#

The parameter enables checking the presence of a real person in the frame based on the facial surrounding.

"use_flying_faces_liveness_filtration": false,

Liveness-mode#

This parameter enables to specify which frames from a track will undergo Liveness check. There are three options for selecting a frame:

0 - First N frames;
1 - Last N frames before the best shot sending;
2 - All frames in a track.

N value is specified in the number_of_liveness_checks parameter.

"liveness_mode": 0,

Number-of-liveness-checks#

The parameter enables to specify the number of frames to check for Liveness. The specified value is used in the liveness-mode parameter.

"number_of_liveness_checks": 10,

Liveness-threshold#

The liveness_threshold parameter value is used to define the presence of a real person in a frame. The system confirms that it is a real person in the frame, and not a photo, only if Liveness returned a value higher than the one specified in the parameter.

"liveness_threshold": 0.5,

Liveness-weights#

The parameter determines the involvement of each liveness check type (shoulders, mask, and flying_faces) in the resulting estimation of the presence of a human face in the frame.

User must specify three values assigned to different types of liveness. Values are specified in decimals in the following order:

Use_shoulders_liveness_filtration,
Use_mask_liveness_filtration,
Use_flying_faces_liveness_filtration.

In the example present (which is the system default) below 0,05 determines that 5% of liveness estimation will be based on shoulders_liveness, 0,45 - 45% on mask_liveness, and 0,5 - 50% on flying_faces_liveness.

The ratio is always calculated based on liveness_weights values, even if they don’t add up to one, or not all liveness types are active.

"livenesses_weights": [0.05, 0.45, 0.5]

Mask-backgrounds-count#

The number of background frames that are used for the corresponding checks.

Video-Stream Sources#

Name#

ID#

Input#

ROI#

DROI#

Rotation#

Transport#

Url#

Number-Of-Ffmpeg-Threads#

Frame-processing-mode#

Real-time-mode-fps#

Mask#

Output#

Login#

Password#

Token#

Url#

Filtering#

Min-score#

Detection-yaw-threshold#

Detection-pitch-threshold#

Detection-roll-threshold#

Yaw-number#

Yaw-collection-mode#

Mouth-occlusion-threshold#

Sending#

Time-period-of-searching#

Silent-period#

Type#

Number-of-bestshots-to-send#

Health-check#

Max-error-count#

Period#

Retry-delay#

Primary-track-policy#

Use-primary-track-policy#

Best-shot-min-size#

Best-shot-proper-size#

Liveness#

Use-shoulders-liveness-filtration#

Use-mask-liveness-filtration#

Use-flying-faces-liveness-filtration#

Liveness-mode#

Number-of-liveness-checks#

Liveness-threshold#

Liveness-weights#

Mask-backgrounds-count#