Component description#

RSEngine component#

RSEngine provides the interaction of the VisionLabs SDK, RealSense2 SDK, VLS LUNA CAMERA 3D SDK, VLS LUNA CAMERA 2D SDK libraries within the System.

VisionLabs SDK component#

The VisionLabs SDK is a software development kit that includes libraries and neural networks for image analysis to:

detection of faces in images and key points (landmarks) of the face;
choosing the bestshot;
estimation of image attributes for further Liveness check;
estimation of the face in the image using Liveness algorithms.

All estimations described below are performed to ensure that the image meets Liveness requirements. All checks are internal and the results are not transmitted externally. The result of the check can only be shown in case of an error, if any attribute of the image/face is not suitable for the Liveness estimation (see the description of errors in “Appendix 2. Status codes and error descriptions”);

RealSense2 SDK component#

The RealSense2 SDK is a component that allows:

receive incoming images from Intel RealSense cameras;
configure detection parameters;
turn the camera on/off, change various parameters. For example, laser backlight brightness, auto exposure, brightness.
automatically update the connection with the camera. When the connection is updated, the System reconnects to the camera. If the System fails to reconnect to the camera, a soft reset of the connection cable is performed. If the specified operations are unsuccessful, this problem will be reflected in the camera status report in the System logs.

Component VLS LUNA CAMERA 3D SDK#

VLS LUNA CAMERA 3D SDK is a component that allows:

receive incoming images from VLS LUNA CAMERA 3D / VLS LUNA CAMERA 3D Embedded cameras;
configure detection parameters;
turn the camera on/off, change various parameters (e.g. laser illumination brightness, auto exposure, brightness).

Component VLS LUNA CAMERA 2D SDK#

VLS LUNA CAMERA 2D SDK is a component that allows:

receive incoming images from VLS LUNA CAMERA 2D infrared cameras;
configure detection parameters;
turn the camera on/off;
change the rotation angle of the camera video frame.

Camera functions#

Face detection#

The detector uses special face detection algorithms and solves the following problems:

face detection in the image;
determining 5 key points on the face: two for the eyes, one for the tip of the nose and two for the corners of the mouth;
estimation of the detection quality–the degree of probability that the face (not another object) is detected in the image.

Image-quality-estimation#

Image quality is estimated using the following parameters:

Blur–blurriness;;
Light–lightness;
Dark–darkness.

Oral status estimation#

An oral status estimation is performed on the following parameters:

Opened–the mouth is open;
Occluded–the mouth is blocked by an outside object;
Smiling–the presence of a smile.

Eye status estimation#

An estimation of the eye status is performed using the following parameters:

Closed–eyes are closed;
Open–eyes are open;
Occluded–eyes are blocked (e.g. with sunglasses).

Head position estimation#

An estimation of the head position is performed using the following parameters:

Roll–angle of head tilt around the longitudinal axis;
Pitch–angle of head tilt around the transverse axis;
Yaw–angle of head rotation around the vertical axis.

Depth Liveness checking#

The “vitality” of the person in the image is checked using a depth map.

The depth matrix (16 bits) is analyzed. It contains information about the distance of the surfaces of the scene objects (faces) to the viewpoint.

IR Liveness checking#

The “vitality” of the person in the image is checked using infrared image analysis.

The camera must be equipped with infrared illumination to perform the check.

FPR Liveness checking#

FPR is an anagram of the names of the checks– FlyingFaces, Phone и Replay Liveness. The check of "vitality" of a person in the image is performed using:

FlyingFaces Liveness–an algorithm that allows you to detect printed photos and masks.
Phone Liveness–an algorithm that allows you to detect the presence of a phone in an extended BBox;
Replay Liveness–an algorithm that allows you to detect video recording artifacts;

Camera Monitoring Component#

Camera monitoring is used to check the status of the camera.

Camera monitoring queries the following camera parameters:

firmware data;
operating status of infrared cameras–on/off;
RGB camera operating status–on/off;
camera serial number;
operation status of the entire camera–on/off;
camera temperature;
date of the last update.

An example of the contents of the registry in the monitoring section is shown below (Figure 4).

Example of registry contents in the monitoring section

RSE Server component#

The RSE Server is a WebSocket server that processes commands from external systems.

The RSE Server accepts requests and sends responses via WebSocket.

Request Format:

Operation request code (1 byte)
Additional payload (MessagePack or string)

Example of a request:

GET ws://127.0.0.0.1:4444/–establishing a WebSocket connection.

0–content of the message to start the session.

Response Format:

Operation response code (1 byte)
Additional payload (MessagePack or string)

Only one request can be processed at a time.

Depending on the type of integration required (selected at the discretion of the external system developer), you can configure RSE Server in the following ways:

RSE Server expects requests (presented in Table 5) to connect to the camera from the external system–set the cs_communication = msg-pack parameter;
RSE Server starts the process of receiving video stream and face detection process as soon as WebSocket connection is established –set the cs_communication = json parameter.

\newpage

Table 5. Description of requests to RSE Server

Request Name

Request Code

Description

Payload

Possible responses to the request

RSE_START_CAPTURE

0

Starts the process of receiving a video stream and the process of detecting faces

No

RSE_CAPTURE_OK (54),

RSE_CAPTURE_META (55)

RSE_STOP

1

Stops all running processes

No

RSE_STOP_OK (50)

Depending on the type of integration selected (chosen at the discretion of the external system developer), the server response can be presented in two formats:

if the external system developer has set the cs_communication = msg-pack parameter, each response will arrive in msg-pack format and will contain the messageType field with the response code and some additional data fields (payloads) described in Table 6;
if the external system developer has set the cs_communication = json parameter, each response will arrive in json format and will be categorized into the message types described in Table 7.

Table 6. Responses to RSE Server requests with MessagePack response format

Answer title	Code	Description **	Payload
RSE_CAPTURE_OK	54	Captured set of video frames	— rgbFrame—RGB frame in uint8 array format; — rgbFrameWidth—RGB frame width in pixels in int format; — rgbFrameHeight—RGB frame height in pixels in int format; — irFrame—IR frame in uint8 array format; — depthFrame—Depth frame in uint8 array format
RSE_CAPTURE_META	55	Metadata of detected persons	— gotBestshot—indicator of whether bestshot was received, in bool format, returns: true—if bestshot was received; false—if bestshot was not received failureReason—the status or error code for liveness checks in int format (see error description in "Appendix 2. Status codes and error descriptions";) – bestshot—RGB frame, in uint8 array format: if gotBestshot=True, the response is an RGB frame, that passes all checks; if gotBestshot=False, the field is blank
RSE_STOP_OK	50	All processing has stopped. RSE Server is ready for new requests	No payload
RSE_UNKNOWN	51	The request was not recognized	No payload
RSE_INTERNAL_ERROR	52	An error occurred while processing the request	No payload
RSE_BUSY	53	Request denied because server is busy	No payload

\newpage

Table 7. Responses to RSE Server requests with JSON response format

Message type

Description

Payload

visual

The type of response that is used for broadcasting the video stream to the user

– msg_type–the type of message returned (visual);

– img_b64–base64 camera frame;

– metadata–parameters of the returned image:

frame_size–image dimensions:
h–height of the image in pixels;
w–width of the image in pixels;
detections–coordinates of the detected person:
h–height of the frame of the detected face;
w–width of the detected face frame
x–coordinates of the upper left corner of the detected face frame

– y–coordinates of the upper left corner of the detected face frame

progress–displays the stages of Liveness of the detected person's background checks (percentage);
track_id–track identifier

bestshot

The type of response when a person is successfully found.

This frame can be used for subsequent processing (e.g. in an external face recognition system)

– msg_type–the type of message returned (bestshot);

– img_b64–face from camera frame in base64 format;

– metadata–parameters of the returned image:

frame_size–image dimensions:
h–height of the image in pixels;
w–width of the image in pixels;
detections–coordinates of the detected person:
h–height of the frame of the detected face;
w–width of the detected face frame
x–coordinates of the upper left corner of the detected face frame
y–coordinates of the upper left corner of the detected face frame
progress–displays the stages of Liveness of the detected person's background checks (percentage);
track_id–track identifier

WebSocket Client component#

WebSocket Client is an external component for interacting with RSE Server.

WebSocket Client is a JavaScript library for communicating with RSE Server via WebSocket. It uses the minimized binary serialization format MessagePack as a protocol library to encode and decode messages if the server returns responses in MessagePack format.