Glossary#

Introduction#

LUNA PLATFORM is an automated face and body recognition system. LUNA PLATFORM is designed to solve the following problems:

• Images processing and analysis:
  • Face and body detection in photos.
  • Basic attributes (age, gender) and face, body, image parameters estimation (emotions, upper clothing, image ratio and other).
  • Checking images according to ISO/IEC 19794-5:2011 and non-standard conditions.
  • Presentation attacks detection (Liveness check) and image analysis for the use of Deepfake technology.
• Search for similar faces in the database (1 to 1, 1 to many and M to N).
• Storage of the received face and body descriptors in databases.
• Processing and analysis of video files or video streams to solve various analytical problems, such as: counting people, human tracking in a frame, detecting fights, etc.;
• Creation of lists to search in.
• Statistics gathering.
• Flexible request management to meet user data processing requirements.

Some of these features are licensed separately. See the "License information" section.

General concepts#

The following sections explain general LP concepts, describe existing services and created data types. It does not describe all the requests, database structures, and other technical nuances.

Services#

LP consists of several services. Communication between them occurs via HTTP requests: a service receives a request and always returns a response. Some services also communicate with each other via Redis or web sockets.

You can find information about LUNA PLATFORM 5 architecture in the "Interaction of LP Services" section.

All the services can be divided into general and additional. Using the general services, the optimal operation of the LP is ensured. Using of all general services is enabled by default in the API service settings. If necessary, some general services can be disabled (see the "Disableable services" section).

Most of the services have their database or file storage.

General services#

Additional services#

Additional services provide more opportunities for the system to work. Launching additional services is optional.

Below is a diagram of the interaction of the general and additional services.

This diagram does not describe the communication lines between the services. For a full description of the interaction of general services, see the "Interaction of LP services" section.

Third-party services#

There are several third-party services that are usually used with LP.

These services are not described in this document. See the corresponding documentation provided by the services vendors.

Authorization system#

Most requests to the LUNA PLATFORM require a mandatory account, except for requests that do not involve authorization.

Accounts

The account is required to delimit the visibility areas of objects for a particular user. Each created account has its own unique "account_id".

The account can be created using a POST request "create account" to the API service, or by requesting "register account", or using the Admin user interface. When creating the account, you must specify the following data: login (email), password and account type.

The account type determines what data is available to the user.

• user – It is type of account with which you can create objects and use only your account data.
• advanced_user – It is type of account for which rights similar to "user" are available, and there is access to the data of all accounts. Access to data from other accounts means the ability to receive data (GET requests), check their availability (HEAD requests) and perform comparison requests based on data from other accounts.
• admin – It is type of account for which rights similar to "advanced_user" are available, and there is also access to the Admin service.

Using the "Luna-Account-Id" header in the "create account" request, you can set the desired account ID.

Tokens

Token is linked to an existing account with any type and enables you to impose extended restrictions on the requests being made. For example, when creating the token, you can give the user permission only to create and modify all lists and faces, or you can prevent the use of certain handlers by specifying their ID.

The token and all its permissions are stored in the database and linked to the account by the "account_id" parameter.

When creating the token, you can set the following parameters:

• expiration_time – Expiration time of the token in RFC 3339 format. You can specify an infinite token expiration time using the value "null".
• permissions – Permissions that are available to the user.
• visibility_area – Token visibility of data from other accounts.

Authorization types for accessing resources

There are following types of authorization available in LUNA PLATFORM:

• BasicAuth – Authorization by login and password (set during account creation).
• BearerAuth – Authorization by JWT token (issued after the token is created).

Note: Also, as part of backward compatibility with previous versions of LUNA PLATFORM, LunaAccountIdAuth authorization is available. It is not recommended to use this authorization in new projects, it is disabled by default.

See detailed information about the LUNA PLATFORM 5 authorization system in the "Accounts, tokens and authorization types" section.

Approaches at work#

There are three main operations in LUNA PLATFORM:

1․ Detection is the process of recognizing a face or body in a photo and normalizing the image (creating a sample) for further work. At the stage of detection, estimation of face or body parameters also performs, i.e. estimation of emotions, direction of gaze, upper clothes, etc.

2․ Extraction is the process of extracting gender and age from a face image and extracting a set of unique parameters of a face or body (descriptors) from a sample to perform further matching.

3․ Matching is the process of comparing descriptors.

These operations are performed strictly one after the other. It is impossible to match face descriptors without first extracting them.

There are two main approaches when performing the operations described above.

Parallel performing of requests#

The first and main approach is to set the rules of detection, estimation, extraction, matching, etc. in a single handler object. After that, you need to create an object event, which will give a result based on all the rules specified in the handler. Using this approach is the most optimal from the point of view of business logic.

With this approach, the following actions are performed:

• Using the "create handler" request, a handler is created containing information about image processing rules;
• In the "generate events" request, the received handler ID is specified, the processed image is attached and an event is generated containing information obtained by processing handler rules.

Thus, when an event is generated, detection, estimation, extraction, matching, saving to the database, and so on are performed simultaneously.

Examples of common scenarios for using handlers:

• Registration of a reference descriptor with saving to the list.
• Biometric identification of faces on the list without saving.
• Saving faces identified in the list in the database.
• Saving events only for unique faces for later counting.
• Batch identification.
• Batch import.

For some scenarios, it may be necessary to pre-create certain objects in the system using separate requests. For example, to identify a set of faces by a list, you first need to create a list object, link the previously created face objects to it, and then use handler and event.

Advantages of the approach:

• All the rules in one place.
• Flexible management of saving objects to databases, setting up save filters.
• Ability to work with packages of images located in ZIP archive, FTP server, S3-like storage, etc.
• Ability to send notifications via web sockets.
• Collecting statistics.

The classic way to use this approach is paired with the FaceStream module. FaceStream analyzes the video stream and sends the best shots from the video stream to LUNA PLATFORM for further processing. Using the specified handler, LUNA PLATFORM processes images according to the specified rules and saves objects to the specified databases.

Sequential performing of requests#

The second approach is to sequential performing of requests, i.e. in one request you need to perform a face detection and get its result, then use this result in an extraction request and so on.

With this approach, the following actions are performed:

• Normalizing image, as well as face detection and estimation using the "detect faces" request.
• Extracting gender, age and descriptor from a normalized image using the "extract attributes" request.
• Creating a face using the "create face" request.
• Matching of face descriptors using the "matching faces" request.

As a rule, this approach is used when working with faces, but if necessary, you can also make separate requests with bodies, for example, to match body descriptors using the "matching bodies" request.

This approach is recommended for use:

• During the initial acquaintance with the system.
• If you need to measure the speed of each stage of image processing separately.
• If you need to implement a processing scenario that is difficult to implement using handlers and events.

More detailed information about these approaches will be provided below.

Detection#

LP searches for all faces and/or bodies on each incoming photo image.

Detection is performed using neural networks located in the Remote SDK container.

Detailed information about neural networks is described in the "Neural networks" section.

Either the source image or a special format image obtained using the VisionLabs software (sample) can be submitted to the input.

Image format requirements#

Images should be sent only in allowed formats. The image format is specified in the "Content-Type" header.

The following image formats are supported: JPG, PNG, BMP, PORTABLE PIXMAP, TIFF. Each format has its own advantages and is intended for specific tasks.

The most commonly used formats are PNG and JPG. Below is a table with their advantages and disadvantages:

Thus, if it is not intended to save the source images in the Image Store, or the Image Store has a sufficiently large volume, then it is recommended to use PNG format to get the best image processing results.

It is not recommended to send images that are too compressed, as the quality of face and body estimations and matching will be reduced.

LUNA PLATFORM 5 supports many color models (for example, RGB, RGBA, CMYK, HSV, etc.), however, when processing an image, all of them are converted to the RGB color model.

The image can also be encoded in Base64 format.

For more information about the source images, see "Source images".

Face detection and estimation process#

Below is the order of face detection in the image:

Basic steps for face images detection:

• Face detection. The bounding box of the face (height, width, "X" and "Y" coordinates) and five landmarks of the face are determined.
• Normalization. The image is centered in a certain way and cropped to the size of 250x250 pixels required for further work. Thus, all processed images look the same. For example, the left eye is always in a frame defined by some coordinates. Such a normalized image is called a sample. The sample is stored in the Image Store. After creating the sample, it is assigned a special identifier "sample_id", which is used for further image processing.

All objects created in LUNA PLATFORM are assigned identifiers. Sample — "sample_id", face — "face_id", event — "event_id", etc. Identifiers are the main way to transfer data between requests and services.

For more information about samples, see "Sample object".

• Estimation. The following face parameters are estimated:
  • Gaze direction (rotation angles for both eyes).
  • Head pose (yaw, roll and pitch angles).
  • Sixty-eight landmarks. The number of landmarks depends on which face parameters are to be estimated.
  • Image quality (lightness, darkness, blurriness, illumination and specularity).
  • Mouth attributes (overlap, presence of smile).
  • Mask presence (medical or tissue mask on the face, mask is missing, mouth is occluded).
  • Emotions (anger, disgust, fear, happiness, neutrality, sadness, surprise).
  • Others (see the "Face parameters" section).

If necessary, you can configure filtering by head pose angles.

Body detection and estimation process#

Below is the order of body image detection:

Basic steps for body images detection:

• Body detection. The bounding box of the body (height, width, "X" and "Y" coordinates) is determined.
• Normalization. The image is centered in a certain way and cropped to the size of 128x250 pixels required for further work (otherwise, the principle of operation is similar to the process of normalization of the face).
• Estimation. The following body parameters are estimated:
  • Upper body (presence and color of headwear, color of upper clothing).
  • Sleeve length (long sleeves, short sleeves, unknown).
  • Lower body (type of lower garment — trousers, shorts, skirt, unknown; color of lower garment, presence and color of shoes).
  • Accessories.
  • Others (see the "Body parameters" section).

Services interaction during detection#

Below is a diagram of the interaction of services when performing detection.

Requests to perform detection#

Below are the main requests where you can perform face or body detection.

Requests "create handler" and "generate events"#

These requests are used in the approach "Parallel performing of requests".

• Request "create handler":

• Request "generate events":

It is possible to use the "multipart/form-data" schema in the request. Using a schema enables you to send multiple images at the same time, specify additional information for an event, and more. See "Use "multipart/form-data" schema when generating event" for more information.

Request "detect faces"#

The "detect faces" request is used in the approach "Sequential performing of requests". This request cannot be used for the body image.

In the request, you can send an image or specify the URL to the image. You can use a multipart/form-data schema to send several images in the request. Each of the sent images must have a unique name. The content-disposition header must contain the actual filename.

You can also specify a bounding box parameters using the multipart/form-data schema. It enables you to specify the certain zone with a face on the image. You can specify the following properties for the bounding box:

• top left corner "x" and "y" coordinates,
• height,
• width.

Request "sdk"#

The "sdk" request also allows face or body detection on the source image. Unlike other requests, its data cannot be used in the future. For more information, see "Sdk resource".

Requests to perform estimation#

The estimation of the parameters of faces and bodies is performed in conjunction with detection. Most often, the parameters for estimation have a name like estimate_<estimation_name>. For example, estimate_glasses.

The "Estimated data" section contains a list of all possible parameters that can be estimated in LUNA PLATFORM 5.

Detection results#

The following information is returned in responses to detection requests:

• address to the stored face/body sample
• id of the face/body sample
• coordinates of the bounding box of the face/body
• five landmarks of the face

All this data is used for further extraction and matching.

This information is supplemented depending on the enabled face/body estimation parameters.

You can read more about the detection of faces, bodies and the Remote SDK service in the "Remote SDK service" section.

Trusted detections#

In LUNA PLATFORM, you can perform face and body detection. If necessary, you can manually specify the detection by explicitly passing the face or body detection coordinates obtained using VisionLabs algorithms and enabling trusted detection mode. In this case, no detection or redetection will be performed.

Working with trusted detections is available in the following requests:

• "detect faces"
• "sdk"
• "iso"
• "generate events"
• "perform verification"
• "generate stream events (beta)"

The following steps must be performed:

1․ Enable trusted detection mode. This can be done using the "application/json" scheme with the "trusted_detections" parameter or using the "multipart/form-data" scheme with the "X-Luna-Trusted-Detections" header.

2․ Pass the trusted detection coordinates ("x", "y", "width", "height") in the "face_bounding_boxes" or "body_bounding_boxes" parameters.

Note: If you disable standard detection and leave only trusted detection mode active, then using a regular detector becomes impractical.

Note: It is not recommended to specify third-party detections obtained by other algorithms, as this may lead to a loss of accuracy or distortion of the analysis results.

Extraction#

In LUNA PLATFORM, the word "extraction" means:

• Extraction of descriptors and basic attributes (gender, age, ethnicity) of faces.
• Extraction of descriptors of bodies.

It is also estimate to determine a person's gender and age from the body image, but this method of determination is less accurate and occurs at the estimation stage. In addition, it is recommended to estimate the basic attributes of the body together with the extraction of the basic attributes of the face.

Descriptors are sets of characteristics read from faces or bodies in images. Descriptors requires much less memory than the source image.

It is impossible to restore the source image of a face or body from the descriptor due to personal data security reasons.

Descriptors are used to match faces to faces and bodies to bodies. It is impossible to match two faces or bodies in the absence of extracted descriptors. For example, if you want to match a face from a database with an input face image, you need to extract the descriptor for that image.

Descriptor is extracted using a neural network located in the Remote SDK container.

If necessary, you can encrypt the descriptors for additional security. See "Descriptor encryption" for more information.

Detailed information about neural networks is described in the "Neural networks" section.

See "Descriptor" section for more information on descriptors.

Extraction process#

Data extraction requires the following information obtained during detection:

• Sample of the face or body.
• Coordinates of the bounding box for the face or body.
• Five landmarks of the face.

The following is the order in which the extraction is performed:

• In most extraction requests, the descriptor is saved to the database without being returned in the response body (see "Requests to perform extraction").

Specifics of extracting data from faces#

The descriptor and basic attributes of the face derived from the same image are stored in the database as an attribute object. This object may include both descriptor and basic attributes, or only one of these entities.

It is possible to extract descriptor and basic attributes using several samples of the same face at the same time. In this way, an aggregated descriptor and aggregated basic attributes can be obtained. The accuracy of matching and extraction of basic attributes by means of the aggregated descriptor is higher. Aggregation should be used when working with images received from webcams.

Aggregated descriptor can be obtained from images, but not from already created descriptors.

Detailed information about aggregation is described in the "Aggregation" section.

Temporary attributes#

After extracting the attributes using the appropriate requests (see "Requests to perform extraction"), the extracted data is stored in the Redis database as temporary attributes.

Temporary attributes have a TTL (time to live) and are removed from the database after a specified period. In the request, you can specify a period from 1 to 86400 seconds. The default TTL is 5 minutes.

You can get information about a temporary attribute by its identifier until its TTL expires.

In order to save the attributes to the database, you need to create a face object and link the attributes to it. The face is created either by a separate request after the attributes are retrieved (the face is stored in the Faces DB) or during the generation of the event (the face is saved in the Faces DB or in the Events DB). You can create multiple faces and combine them into a list that can be used for matching.

See "Face object" and "List object" for details.

You can store attributes in an external database and use them in LP only when required. See "Create objects using external data".

For details on attributes, see "Attribute object".

Specifics of extracting data from bodies#

There are no attributes for bodies. All body information can only be stored as an event object. Accordingly, the extracted data is not stored in the Redis database. Information about bodies can only be saved to the database of the Events service when the corresponding option is enabled.

Just like faces, events can be aggregated. Detailed information about aggregation is described in the "Aggregation" section.

Services interaction during extraction#

Below is a diagram of the interaction of services when performing extraction.

Requests to perform extraction#

Below are the main requests where you can extract data from images of faces or bodies.

Requests "create handler" and "generate events"#

These requests are used in the approach "Parallel performing of requests".

• Request "create handler":

• Request "generate events":

It is possible to use the "multipart/form-data" schema in the request. Using a schema enables you to send multiple images at the same time, specify additional information for an event, and more. See "Use "multipart/form-data" schema when generating event" for more information.

Request "extract attributes"#

The "extract attributes" request is used in the "Sequential performing of requests" approach. This request cannot be used for body sample.

To save information in the Faces database, you need to link attributes to a face using the "create face" request. Otherwise, after the TTL expires, the face's temporary attributes will be removed from the Redis database. It is not possible to save a face to the Events database using the "extract attributes" + "create face" request combination. To do this, you need to use the "Parallel performing of requests" approach (see above).

Request "sdk"#

The "sdk" request also enables you to extract basic attributes or descriptor. This request returns face or body descriptor in a specific format that can be used when matching "raw" descriptors. See "Sdk resource" for details.

Extraction results#

The following information is returned in responses to extraction requests:

• Identifier of the temporary face attribute (not returned if the "store_attribute" parameter of the "attribute_policy" is disabled in the handler).
• Address to the temporary face attribute stored in the Redis database (not returned if the "store_attribute" parameter of the "attribute_policy" is disabled in the handler).
• Face basic attributes.
• Face/body descriptor quality.
• Set of identifiers of samples for which the extraction was performed.

Matching#

Given the presence of descriptors, LP enables you to search for similar faces or bodies in the database by comparing the given descriptor with descriptors stored in the Redis, Faces, Events databases or with descriptors passed directly.

The matching is done using the Python Matcher service. In response to the matching, the degree of similarity is returned, the value of which lies in the range from 0 to 1. A high degree of similarity means that two descriptors belong to the same person.

There are two types of descriptors — face descriptor and body descriptor. Body descriptor matching is not as accurate as face descriptor matching. With a large number of candidate events, the probability of false determinations of the best matches is higher than with a large number of candidate faces.

The sources of matched objects are represented as references (the objects you want to compare) and candidates (the set of objects that you want to compare with). Each reference will be matched with each of the specified candidates.

Matching cannot be performed if there is no descriptor for any of the compared faces.

You can select the following objects as references and candidates.

References:

• Attributes
• Faces
• External IDs of faces and events
• Events (face or body)
• Event track IDs
• Descriptors

Candidates:

• Faces
• Events (face or body)
• Attributes
• Descriptors

The references are specified using IDs of the corresponding objects. If a non-existent reference is set (for example, a non-existent ID is set in the "event_id" field or the "face_id" field), the corresponding error is returned.

The candidates are specified using filters. Matching results are returned for the candidates that correspond to the specified filters. If none of the candidates corresponds to the filters (for example, a non-existent ID is set in the "event_ids" field or the "face_ids" field), there will be no matching result and no error returned. The result field will be empty.

Matching process#

Below is the process of performing matching using faces as candidates and references:

• Descriptors can also be passed directly using "raw matching" request.

Service interaction during matching#

Below is a diagram of the interaction of services when performing matching.

See "Matching diagrams" for more service interaction scenarios when performing a matching.

Filtering of matching results#

The results of the matching can be filtered. To filter, you need to specify the appropriate request parameters.

Several examples for events and faces filters are given below.

For events filtration one can:

• Specify camera ("source" field) and a period ("create_time__gte" and "create_time__lt" or "end_time__gte" and "end_time__lt" fields).
• Specify tags ("tags" field) for events as filters and perform matching for events with these tags only.
• Specify geographical area and perform matching with the events created in this area only. You can specify direct location (city, area, district, street, house) or a zone specified using geographical coordinates.

For faces filtration one can:

• Specify a list of external face IDs to perform matching with them.
• Specify a list ID to perform matching by list.

See the "matcher" > "matching faces" and "matcher" > "human body matching" sections in the API service reference manual for details about all the available filters.

Requests to perform matching#

Below are the main requests where descriptors can be matched.

Requests "create handler" and "generate events"#

These requests are used in the approach "Parallel performing of requests".

• Request "create handler":

• Request "generate events":

It is possible to use the "multipart/form-data" schema in the request. Using a schema enables you to send multiple images at the same time, specify additional information for an event, and more. See "Use "multipart/form-data" schema when generating event" for more information.

Requests "matching faces" and "matching bodies"#

The "matching faces" and "matching bodies" requests are used in the "Sequential performing of requests".

Request "raw matching"#

You can set descriptors as references and candidates in SDK, Raw and XPK files using the "raw matching" request.

All the descriptors data is provided with the request hence you can use this request when you need to match descriptors that are not stored in LUNA PLATFORM databases.

In LUNA PLATFORM, it is possible to extract descriptor in SDK format (see the "Descriptor formats" section).

Matching results#

The following information is returned in responses to match requests:

• information about the reference
• information about the candidate
• the degree of similarity of each candidate with the reference
• filtered candidates

For more information about descriptor matching services, see the "Matching services" section.

Verification#

You can use the "/verifiers" resource to create a special handler for verification. It includes several policies for the incoming images processing. See the "Verifiers description" section for details on the verifier.

Generally this request is used to match one given object with the incoming object. Use other matching requests for identification tasks.

Matching a large set of descriptors#

Sometimes it is necessary to match a very large set of descriptors (candidates). When matching a large set of descriptors by classical brute-force matching, it is impossible to get a low latency with a high number of requests per second. Therefore, it is required to use approximation methods implemented in LIM that exchange some accuracy for high speed. These methods speed up the matching by building an index containing preprocessing data.

The work of the module is as follows: The index is created using a task containing a "list_id" with linked faces, by which the matching will be made. You can also not specify "list_id", but set the settings for automatic indexing of all lists whose number of faces exceeds the value specified in a certain setting. After the index is created, the user sends a request to the API service, which redirects it to the Python Matcher Proxy service. The Python Matcher Proxy service determines where the matching will be performed — in the Python Matcher service or in the Indexed Matcher service. After matching, the response is returned to the user.

LUNA Index Module is licensed separately and is available on request to VisionLabs. The module delivery contains all the necessary documentation and the Docker Compose script.

Stored data and LP objects#

This section describes data stored by LUNA PLATFORM 5.

This information can be useful when utilizing LUNA PLATFORM 5 according to the European Union legal system and GDPR.

This section does not describe the legal aspects of personal data utilization. You should consider which stored data can be interpreted as personal data according to your local laws and regulations.

Note that combinations of LUNA PLATFORM data may be interpreted by law as personal data, even if data fields separately do not include personal data. For example, a Face object including "user_data" and descriptor can be considered personal data.

Objects and data are stored in LP upon performing certain requests. Hence it is required to disable unnecessary data storage upon the requests execution.

It is recommended to read and understand this section before making a decision on which data should be received and stored in LUNA PLATFORM.

This document considers usage of the resources listed in the OpenAPI specification and creation of LUNA PLATFORM 5 objects only. The data created using Backport 3 and Backport 4 services is not considered in this section.

Source images#

Photo images are general data sources for LP. They are required for samples creation and Liveness check.

You can provide images themselves or URLs to images.

After normalization of the source images, samples are saved in JPG format by default, even if the source image is sent in PNG format. If necessary, you can change the format of the stored samples using the "default_image_extension" setting of the Image Store service.

It is not recommended to send rotated images to LP as they are not processed properly and should be rotated. You can rotate the image using LP in two ways:

• By enabling the "use_exif_info" auto-orientation parameter by EXIF data in the query parameters.
• By enabling the "LUNA_HANDLERS_USE_AUTO_ROTATION" auto-orientation setting in the Configurator settings.

More information about working with rotated images can be found in the Nuances of working with services section.

See the detailed information about the requirements for the source images in the "Image format requirements" section.

Source images usage#

Source images can be specified for processing when performing POST requests on the following resources:

• "/sdk"
• "/detector"
• "/handlers/{handler_id}/events"
• "/verifiers/{verifier_id}/verifications"
• "/liveness"

Source images saving#

Generally, it is not required to store source images after they are processed. They can be optionally stored for system testing purposes or business cases, for example, when the source image should be displayed in a GUI.

Source images can be stored in LP:

• Using the POST request on "/images" resource.
• During the the POST request on "/handlers/{handler_id}/events" resource. Source images are stored if the "store_image" option is enabled for "image_origin_policy" of the "storage_policy" during the handler creation using the "/handlers" resource.

Samples are stored in buckets of the Image Store for an unlimited time.

Samples are stored in buckets:

• "visionlabs-samples" is the name of the bucket for faces samples.
• "visionlabs-bodies-samples" is the name of the bucket for human bodies samples.

Paths to the buckets are specified in the "bucket" parameters of "LUNA_IMAGE_STORE_FACES_SAMPLES_ADDRESS" and "LUNA_IMAGE_STORE_BODIES_SAMPLES_ADDRESS" sections in the Configurator service.

Saving metadata with an image

In the resource "/images", user metadata can be saved along with the image using the header X-Luna-Meta-<user_defined_key> with the value <user_defined_value>. In the Image Store bucket, metadata is saved in a separate file <image_id>.meta.json, which is located next to the source image.

In the response to the "get image" request, you need to specify the header with_meta=1 to get the image metadata in the response header.

To store metadata values for multiple keys, multiple headers must be set.

Source images deletion#

Source images are stored in the bucket for an unlimited period.

You can delete source images from the bucket:

• Using the DELETE request on the "/images/{image_id}" resource.
• Manually, by deleting the corresponding files from the bucket.

Sample object#

Samples usage#

Separate samples are created for face and body.

Samples are required:

• For basic attributes estimation
• For face, body and image parameters estimation
• For face and body descriptors extraction
• When changing the descriptors NN version

When the neural network is changed, you cannot use the descriptors of the previous version. A descriptor of a new version can be extracted if the source sample is preserved only.

Samples can also be used as avatars for faces and events. For example, when it is required to display the avatar in GUI.

Samples are stored in buckets of the Image Store.

Samples should be stored until all the required requests for face, body parameters estimation, basic attributes estimation, and descriptors extraction are finished.

Samples creation and saving#

Samples are created upon the face and/or human body detection in the image. Samples are created during the execution of the following requests:

• POST on the "/detector" resource. Samples are created and stored implicitly. The user does not affect their creation.
• POST on the "/handlers/{handler_id}/events" resource. Samples are stored if the "store_sample" option is enabled for "face_sample_policy" and "body_sample_policy" of the "storage_policy" during the handler creation using the "/handlers" resource.
• POST on the "/verifiers/{verifier_id}/verifications" resource. Samples are stored if the "store_sample" option is enabled for "face_sample_policy" of the "storage_policy" during the verifier creation using the "/verifiers" resource. Human body samples are not stored using this resource.

External samples saving

Samples can be provided to LP directly from external VisionLabs software (e. g., FaceStream). The software itself performs sample creation from the source image.

You can manually store external samples in LP (an external sample should be provided in the requests):

• Using the POST request on the "/samples/{sample_type}" resource.
• When the "warped_image" option is set for the POST request on the "/detector" resource.
• When the "image_type" option is set to "1" ("face sample) or "2" (human body sample) in the query parameters for the POST request on the "/handlers/{handler_id}/events" resource. The "store_sample" option should be enabled for "face_sample_policy" and "body_sample_policy" of the "storage_policy".

Samples saving disabling#

There is no possibility to disable samples saving for the request on the "/detector" resource. You can delete the created samples manually after the request execution.

The "/handlers" resource provides "storage_policy" that allows you to disable saving following objects:

• Face samples. Set "face_sample_policy" > "store_sample" to "0".
• Human body samples. Set "body_sample_policy" > "store_sample" to "0".

The "/verifiers" resource provides "storage_policy" that allows you to disable saving the following objects:

• Face samples. Set "face_sample_policy" > "store_sample" to "0".

Samples deletion#

You can use the following ways to delete face or body samples:

• Perform the DELETE request to "/samples/faces{sample_id}" resource to delete a face sample by its ID.
• Perform the DELETE request to "/samples/bodies{sample_id}" resource to delete a body sample by its ID.
• Manually delete the required face or body samples from their bucket.
• Use "remove_samples" parameter in the "gc task" when deleting events.

Getting information about samples#

You can get face or body sample by ID.

• Perform the GET request to "/samples/faces/{sample_id}" resource to get a face sample by its ID.
• Perform the GET request to "/samples/bodies/{sample_id}" resource to get a body sample by its ID.

If a sample was deleted, then the system will return an error when making a GET request.

Descriptor#

The basic description and use of descriptors is described in the "Extraction" section above.

See "Descriptor encryption" for more information on descriptor encryption.

See the additional information in the "Descriptor formats" section.

Attribute object#

This object is intended to work with faces only.

It is recommended to introduce yourself with the "Extraction" section before reading.

Attributes are temporary objects that include basic attributes and face descriptors. This data is received after the sample processing.

Basic attributes include the following personal data:

• age. The estimated age is returned in years.

• gender. The estimated gender: 0 — female, 1 — male.

• ethnicity. The estimated ethnicity.

You can disable basic attributes extraction to avoid this personal data storage.

A descriptor cannot be considered personal data. You cannot restore the source face using a descriptor.

Attributes usage#

Attribute object can be used in the following cases:

• Matching using attributes can be performed using:

  • "/matcher/faces" resource
  • "/tasks/cross_match" resource
  • "/verifiers/{verifier_id}/verifications" resource

As attributes have TTL (by default, it is set to 5 minutes), it is convenient to use them for verification or identification purpose. They will be deleted soon after you receive the result.

• Attributes may be used for ROC-curves creation using the "/tasks/roc" resource (see the "ROC-curve calculating task" section).

• You can save the data of the existing attribute to a face using the "/faces" resource.

It is not the only way to save descriptors and basic attributes to a face. You can use "/handlers/{handler_id}/events" to create a face and add the extracted descriptor and basic attributes to it.

Basic attributes saved in faces or events objects can be used for filtration of the corresponding objects during requests execution.

Descriptors are required for matching operations. You cannot compare two faces or bodies without their descriptors.

Attributes creation and saving#

Attributes can be created when sending requests on the following resources:

• "/extractor"

The "extract_basic_attributes" and "extract_descriptor" query parameters should be enabled for extraction of the corresponding data. Attributes are implicitly stored after the request execution.

• "/handlers/{handler_id}/events"

The "extract_basic_attributes", "extract_face_descriptor", and "extract_body_descriptor" parameters should be enabled in the handler for extraction of the corresponding data. The "storage_policy" > "attribute_policy" > "store_attribute" option should be enabled in the handler for attributes storage.

• "/verifiers/{verifier_id}/verifications"

The "extract_basic_attributes" parameter should be enabled in the verifier for extraction of the corresponding data. The "storage_policy" > "attribute_policy" > "store_attribute" option should be enabled in the verifier for attributes storage.

Attributes can be created using external descriptors and external basic attributes using the "/attributes" resource.

Attributes time to live#

Attributes have a TTL. After the TTL expiration, attributes are automatically deleted. Hence, it is not required to delete attributes manually.

The default TTL value can be set in the "default_ttl" parameter in the Configurator service. The maximum TTL value can be set in the "max_ttl" parameter in the Configurator service.

TTL can be directly specified in the requests on the following resources:

• "/extractor" in the "ttl" query parameter
• "/handlers" in the "storage_policy" > "attribute_storage" in the "ttl" parameter

Attributes extraction disabling#

You can disable basic attributes extraction by setting the "extract_basic_attributes" parameter to "0" in the following resources:

• "/extractor"
• "/handlers"
• "/verifiers"

You can disable descriptor extraction by setting the "extract_descriptor" parameter to "0" in the following resources:

• "/extractor"
• "/handlers"

Attributes saving disabling#

You can disable the "storage_policy" > "attribute_policy" > "store_attribute" parameter in the "/handlers" resource to disable attributes storage. When this handler is used for the "/handlers/{handler_id}/events" resource, attributes are not saved even for the specified TTL period.

You can disable the "storage_policy" > "attribute_policy" > "store_attribute" parameter in the "/verifiers" resource to disable attributes storage. When this verifier is used for the "/verifiers/{verifier_id}/verifications" resource, attributes are not saved even for the specified TTL period.

Attributes deletion#

Attributes are automatically deleted after the TTL expiration.

Perform the DELETE request to "/attributes/{attribute_id}" resource to delete an attribute by its ID.

Getting information about attributes#

You can receive information about existing temporary attributes before TTL has not expired.

• Perform the GET request to "/attributes/{attribute_id}" resource to receive information about a temporary attribute by its ID.
• Perform the GET request to "/attributes" resource to receive information about previously created attributes by their IDs.
• Perform the GET request to "/attributes/{attribute_id}/samples" resource to receive information about all the temporary attribute samples by the attribute ID.

If any attribute TTL has not expired, the attribute data is returned. Otherwise, no data is returned for this attribute in the response.

Face object#

Faces are changeable objects that include information about a single person.

The following general data is stored in the face object:

• Descriptor ("descriptor")
• Basic attributes ("basic_attributes")
• Avatar ("avatar")
• User data ("user_data")
• External ID ("external_id")
• Event ID ("event_id")
• Sample ID ("sample_id")
• List ID ("list_id")
• Account ID ("account_id")

See the "Faces database description" section for additional information about the face object and the data stored in it.

Attributes data can be stored in a face. Basic attributes data, descriptor data, and information about samples are saved to the Faces database and linked with the face object.

When you delete a face, the linked attributes data is also deleted.

• Descriptor. You should specify a descriptor for the face if you are going to use the face for comparison operations.

You cannot link a face with more than one descriptor.

• Basic attributes: Basic attributes can be used for displaying information in GUI.

The face object can also include IDs of samples used for the creation of attributes.

General event fields description:

• "user_data". This field can include any information about the person.

• "avatar". Avatar is a visual representation of the face that can be used in the user interface.

This field can include a URL to an external image or a sample that is used as an avatar for the face.

• "external_id". The external ID of the face.

You can use the external ID to work with external systems.

You can use the external ID to specify that several faces belong to the same person. You should set the same external ID to these faces upon their creation.

You can get all the faces that have the same external ID using the "faces" > "get faces" request.

• "event_id". This field can include an ID of the event that gave birth to this face.

• "list_id". This field can include an ID of the list to which the face is linked.

A face can be linked to one or several lists.

• "account_id" — the account ID to which the face belongs.

• "sample_id" — One or several samples can be linked to a face. It should be the samples used for attributes extraction. All the samples must belong to the same person. If no samples are save for a face, you cannot update its descriptor to a new NN version.

IDs do not usually include personal information. However, they can be related to the object with personal data.

Faces usage#

Faces usually include information about persons registered in the system. Hence, they are usually required for verification (the incoming descriptor is compared with the face descriptor) and identification (the incoming descriptor is compared with several face descriptors from the specified list) purposes.

If there are no faces stored in your system, you cannot perform the following operations with these objects:

• Matching by faces and lists, when faces are set as candidates or references in the request to the "/matcher/faces" resource.
• Matching by faces and lists, when faces are set as candidates in the matching policy of the request to the "/handlers" resource.
• Cross-matching tasks, when faces are set as candidates or references in the request to the "/tasks/cross_match" resource.
• Clustering tasks, when faces are set as filters for clustering in the request to the "/tasks/clustering" resource.
• Verification requests, when faces IDs are set in query parameters in the request to the "/verifiers/{verifier_id}/verifications" resource.
• ROC curves creation tasks ("/tasks/roc" resource).

Faces creation and saving#

Faces can be created using the following requests:

• "create face"

You can specify attributes for the face using one of several ways:

• By specifying the attribute ID of the temporary attribute.
• By specifying descriptors and basic attributes (with or without samples).
• By specifying descriptors (with or without samples).
• By specifying basic attributes (with or without samples).

The last three ways are used when you need to create a face using data stored in external storage.

• "generate events"

The "storage_policy" > "face_policy" > "store_face" parameter should be enabled in the utilized handler.

Requests for working with faces#

Below are the requests for working with faces:

List object#

A list is an object that can include faces that correspond to a similar category, for example, customers or employees.

Lists include the "description" field that can store any required data to distinguish lists from each other.

Lists usage#

You can add faces to lists for dividing the faces into groups.

Only faces can be added to lists.

Lists IDs are usually specified for matching operations as a filter for faces.

Requests for working with lists#

Below are the requests for working with lists:

Event object#

Events are used in the "Parallel performing of requests" approach.

Events are immutable objects that include information about a single face and/or body. They are received after images processing using handlers or created manually.

Unlike faces, events cannot be changed after creation. The only exception is the event descriptor. It can be updated to the new neural network version.

Generally an event is created for each face/body detected in the image. If the detected face and body belong to the same person they are saved to the same event.

LP also provides possibility to create new events manually without processing using handlers. It is used in the cases when a logic for filling in event fields should differ from the logic using handlers. For example, when you want to extract descriptors only for a part of the detections and not for all the detections.

An event can be linked with a descriptor stored in the Events database.

Since there can be millions of events in the Events database, it is recommended to use a high performance column database. You can also use a relational database, but this will require additional configuration.

The following general data in stored in the event object:

• Field "source". This field can include the source from which the face or human body was received. The value is specified during the "generate events" request execution.

• "location". This group of parameters includes information about the location where the event occurred. The values are specified during the "generate events" request execution. The following fields are included in this group:

  • "city"
  • "area"
  • "district"
  • "street"
  • "house_number"
  • "geo_position" — latitude and longitude.

• Field "tag". This field includes a tag (or several tags) applied during the "conditional_tags_policy" execution. The tag(s) can be specified during the "create handler" request execution or the "generate events" request execution.

• Field "emotion". This field includes the predominant emotion estimated for the face. If necessary, you can disable the "estimate_emotions" parameter in the "/handlers" resource or "create verifier" request.
• Field "insert_time". This field includes the time when the face appeared in the video stream. This data is usually provided by external systems.
• Field "top_similar_object_id". This field includes the top similar object ID.
• Field "top_similar_external_id". This field includes the external ID of the top similar candidate (event or face) with which the face is matched.
• Field "top_matching_candidates_label". This field includes a label applied during the "match_policy" execution. The labels are specified in this policy in the "/handlers" resource.
• Field "face_id". Events include the ID of the face to which the event gave birth.
• Field "list_id". Events include the ID of the list to which the created face was attached.
• Field "external_id". This external ID will be specified for the face created during the event request processing. The value is specified during the "generate events" request execution.
• Field "user_data". This field can include any information. User data will be specified for the face created during the event request processing. The value is specified during the "generate events" request execution.
• Fields "age", "gender", and "ethnic_group". The basic attributes extraction is specified in the "extract_policy" of the "/handlers" resource.
• Face and body descriptors. Events can be used for matching operations as they include descriptors.
• Information about faces, bodies and events that were used for matching with the events.
• Information about faces and human bodies detection results that include:
• IDs of created samples
• Information about bounding boxes of detected faces/bodies
• Time of face/body event detection (field "detect_time"). The time is returned from an external system.
• URL of the source image where the face/body was detected ("image_origin")

IDs do not usually include personal information. However, they can be related to the object with personal data.

See the "Events database description" section for additional information about the event object and the data stored in it.

Attributes aggregation for events

It is possible to enable attributes aggregation for the incoming images when creating an event.

According to the specified settings faces and bodies are detected in the incoming images. If faces are detected and an aggregated descriptor creation is specified, then a single descriptor is created using all the found faces. The same logic is used for the aggregated body descriptor creation. In addition, the basic attributes are aggregated, the obtained values for the definition of Liveness, masks, emotions, upper and lower body and the body accessories.

All the information about the detected face/body and estimated properties is returned in the response separately for each image. The aggregated results are stored when an event is saved in the DB. The information about face/body detection is added to the Events database as a separate record for each image from the request.

When performing aggregation, the images in the request to the "/handlers/{handler_id}/events" resource should include only one face/body and the face/body should belong to the same person.

Events usage#

Events are required to store information about persons' occurrences in a video stream for further analysis. An external system processes a video stream and sends frames or samples with detected faces and bodies.

LP processes these frames or samples and creates events. As events include statistics about face detection time, location, basic attributes, etc., they can be used to collect statistics about the person of interest or gather general statistics using all the saved events.

Events provide the following features:

• Matching by events

As events store descriptors, they can be used for matching. You can match the descriptor of a person with the existing events to receive information about the person's movements.

To perform matching, you should set events as references or candidates for matching operations (see section "Descriptors matching").

• Notifications via web sockets

You can receive notifications about events using web sockets. Notifications are sent according to the specified filters. For example, when an employee is recognized, a notification can be sent to the turnstile application and the turnstile will be opened.

See the "Sending notifications via Sender" section.

• Statistics gathering

You can gather statistics on the existing events using a special request. It can be used to:

• Group events by frequency or time intervals.
• Filter events according to the values of their properties.
• Count the number of created events according to the filters.
• Find the minimum, maximum and average values of properties for the existing events.
• Group existing events according to the specified values of properties.

You should save generated events to the database if you need to collect statistics after the event is created.

See the "events" > "get statistics on events" request in "APIReferenceManual.html" for details.

• User defined data

You can add custom information to events, which can later be used to filter events. The information is passed in JSON format and written to the Events database.

See "Events meta-information" for details.

Events creation and saving#

Events are created during the request to the "/handlers/{handler_id}/events" resource execution. The "event_policy" > "store_event" should be set to "1" when creating a handler using the "/handlers" resource.

For manual event creation and saving use the "/handles/{handler_id}/events/raw" resource.

The format of the generated event is similar to the format returned by the "/handlers/{handler_id}/events" resource. The "event_id" and "url" fields are not specified when creating a request. They are returned in the response after the event is created.

Notifications using web sockets are sent when events are created using this resource.

Events deletion#

Events are only deleted using the garbage collection task. You should specify filters to delete all the events corresponding to the filters.

To delete an event execute the POST request to the "/tasks/gc" resource.

You can also manually delete the required events from the database.

Getting information about events#

You can receive information about existing events.

• The "events" > "get event" request enables you to receive information about an event by its ID.

If the event was deleted, then the system will return an error when making a GET request.

• The "events" > "get events" request enables you to receive information about all previously created events according to the specified filters. By default, events are filtered for the last month from the current date. If any of the following filters are specified, then the default filtering will not be used.

  • List of event IDs (field "event_ids")
  • Lower event ID boundary (field "event_id__gte")
  • Upper event ID boundary (field "event_id__lt")
  • Lower create time boundary (field "create_time__gte")
  • Upper create time boundary (field "create_time__lt")

Use "multipart/form-data" schema when generating event#

In the "generate events"request, you can send an image, provide an image URL, or send a raw descriptor. You can use a multipart/form-data schema to send several images in the request. Each of the sent images must have a unique name. The Content-Disposition header must contain the actual filename.

The following parameters can also be specified in the request using the "multipart/form-data" schema:

• Parameters of the bounding box of the face or body. It enables you to specify the certain zone with a face on the image
• Image detection time
• Time stamp of detection relative to the beginning of videofile
• Source image
• User meta-information

All of the above information will be saved in the event when it is generated.

Handler object#

Handlers are used in the "Parallel performing of requests" approach.

Handler is an object that stores entry points for image processing. The entry points are called policies. They describe how the image is processed hence define the LP services used for the processing. The handler is created using the "create handler" request.

When creating a handler, it is important to ensure that only the required policies are enabled. Unnecessary policies can take into account request execution time and create gigabytes of unnecessary data on disk.

Handler policies

The table below includes all the existing handler policies. Each policy corresponds to one of the LP services listed in the "Service" column.

Table: Handler policies

You can skip or disable policies if they are not required for your case. Skipped policies are executed accorded to the specified default values. For example, you should disable samples storage in the corresponding policy if they are not required. If you just skip the "storage_policy" in the handler, samples will be saved according to the default settings.

All the available policies are described in the "API service reference manual".

Output filtering

Many policies have filters. Filters can be specified either explicitly in the "filters" field or in parameters with the name "‹estimation›_states". If filtering is performed on the first policy, it will cause subsequent policies to stop executing as they are executed sequentially. In this case, the event will not be saved in the database, and the filtered objects will be displayed in the "filtered_detections" subblock in the response body of the event generation request.

Handlers usage

Using handler you can:

• Specify face/body detection parameters and estimated face/body parameters (see "Source image processing and samples creation").
• Specify human body detection execution.
• Enable basic attributes and descriptors extraction (see "Descriptor extraction and attribute creation").
• Perform descriptors comparison (see "Descriptors matching") to receive matching results and use the result as filters for the policies.
• Configure automatic creation of faces using filters. You can specify filters according to the estimated basic attributes and matching results.
• Configure automatic attachment of the created faces to the lists using filters. You can specify filters according to the estimated basic attributes and matching results.
• Specify the objects to be saved after the image processing.
• Configure automatic tag adding for the event using filters. You can specify filters according to the estimated basic attributes and matching results.

In addition, the ability to process a batch of images using a handler is available (see the "Estimator task" section).

There are three types of handlers:

• static
• dynamic
• lambda

Static handler#

Handler parameters of this type are specified when it is created, and then when generating an event or creating an Estimator task, the created handler ID is specified.

Dynamic handler#

Handler parameters of this type are specified when generating an event or creating an Estimator task. Dynamic handlers enable you to separate technical parameters (thresholds and quality parameters that need to be hidden from frontend application users) and business logic. Dynamic handler policies are set using the "multipart/form-data" scheme in the event generation request.

Only static verifiers are available.

Dynamic handlers usage example:

You need to separate head pose thresholds from other handler parameters.

You can save the thresholds to your external database and implement the logic of automatic substitution of this data when creating events (e. g. in your frontend application).

The frontend user sends requests for events creation and specifies the required lists and other parameters. The user does not know about the thresholds and cannot change them.

Lambda handler#

This handler is used when working with the Lambda service, which is intended to work with custom modules that mimic the functionality of a separate service.

All requests will be sent to the lambda handler using "lambda_id".

If the custom Handlers-lambda supports the ability to generate an event, then the format of the body of the request and response to the resources is "/handlers/{handler_id}/events" or "/tasks/estimator" will follow the OpenAPI specification. If Handlers-lambda does not support the ability to generate an event, then a request to the resource "/lambdas/{lambda_id}/proxy" should be used with the corresponding request form.

The logic behind the request behavior is entirely dependent on the lambda written by the user.

See "Lambda service" for details.

Requests for working with handlers#

Below are the requests for working with handlers:

Verifier object#

Verifiers are used in the "Parallel performing of requests" approach.

The Handlers service also processes requests to create verifiers required for the verification process. They are created using the "create verifier" request.

The verifier contains a limited number of handler policies — detect_policy, extract_policy, and storage_policy.

You can specify the "verification_threshold" in the verifier.

The created verifier should be used when sending requests to:

• "/verifiers/{verifier_id}/verifications" resource. You can specify IDs of the objects with which the verification should be performed.
• "/verifiers/{verifier_id}/raw" resource. You can specify raw descriptors as candidates and references for matching. As raw descriptors are processed "verification_threshold" is the general parameter used from the specified verifier.

The response includes the "status" field. It shows, if the verification was successful or not for each pair of matched objects. It is successful if the similarity of two objects is greater than the specified "verification_threshold".

Requests for working with verifiers#

Below are the requests for working with verifiers:

Other objects#

In the Image Store service, you can store any files as objects (for example, a video file).

You can upload files using the "create objects" request. The bytes of the file must be specified in the request body, and the Content-Type header must contain the MIME type of the file (for example, video/mp4). The response to the "get object" request contains the Content-Disposition header. This header contains the file name of the attachment object (for example, video_1.mp4). The file name is generated based on the object_id and the MIME type.

If the MIME type of the file could not be determined, then the file extension will be set as .bin.

Sending notifications via Sender#

You can send notifications about created events to the third-party party applications via web sockets. For example, you can configure LP to send notifications about VIP guests' arrival to your mobile phone.

When LP creates a new event, the event is added to the special DB. Then the event can be sent to the Sender service if the service is enabled.

The third-party party application should be subscribed to Sender via web sockets. The filter for the events of interest should be set for each application. Thus you will receive notifications only for the required events.

The notification about a new event is sent by Sender to the required applications.

See section "Sender service" for details about the Sender service.

Licenses#

The Licenses service provides information about license terms to LP services.

See section "Licenses service" for details.

Admin#

The Admin service implements tools for administrative routines.

All the requests for Admin service are described in OpenAPI specification of the Admin service.

See section "Admin service" for details about the Admin service.

Configurator#

The configurator service includes the settings of all the LP services. Thus it provides the possibility to configure all the services in one place.

See section "Configurator service" for details about the Configurator service.

Tasks#

Tasks requests provide additional possibilities for large data amounts processing.

The greater data array is processed, the longer it takes to finish the task. When a task is created you receive the task ID as the result of the request.

See the "Tasks service" section for details about tasks processing.

Clustering task

The clustering task provides the possibility to group events and faces that belong to the same person to clusters.

You can specify filters for choosing the objects to be processed.

Using the clustering task you can, for example, receive all the IDs of events that belong to the same person and which occurred during the specified period of time.

See the "task processing" > "clustering task" request for details about the Clustering task request creation.

See "Clustering task" for more information about this task.

Reporter task

The reporter task enables you to receive a report in CSV format with extended information about the objects grouped to clusters.

You can select columns that should be added to the report. You can receive images corresponding to each of the IDs in each cluster in the response.

See the "task processing" > "reporter task" request for details about the reporter task request creation.

See "Reporter task" for more information about this task.

Exporter task

The exporter task enables you to collect event and/or face data and export them from LP to a CSV file.

The file rows represent requested objects and corresponding samples (if they were requested).

See the "task processing" > "exporter task" request for details about the exporter task request creation.

See "Clustering task" for more information about this task.

Cross-matching task

Cross-matching means that a large number of references can be matched with a large number of candidates. Thus every reference is matched with every candidate.

Both references and candidates are specified using filters for faces and events.

See the "task processing" > "cross-matching task" request in the API service reference manual for details about the cross-matching task creation.

See "Cross-matching task" for more information about this task.

Linker task

The linker task enables you to:

• link the existing faces to lists
• create faces from the existing events and link the faces to lists

The linked faces are selected according to the specified filters.

See the "task processing" > "linker task" request in the API service reference manual for details about the cross-matching task creation.

See "Linker task" for more information about this task.

Estimator task

The estimator task enables you to perform batch processing of images using the specified policies.

In the request body, you can specify the handler_id of an already existing static or dynamic handler. For the dynamic handler_id, the ability to set the required policies is available. In addition, you can create a static handler specifying policies in the request.

The resource can accept five types of sources with images for processing:

• ZIP archive
• S3-like storage
• Network disk
• FTP server
• Samba network file system

See the "task processing" > "estimator task" request in the API service reference manual for details about the estimator task creation.

See "Estimator task" for more information about this task.

Backports#

In LP 5, Backport is a mechanism of the new platform version mimicry to the older versions.

The backports are implemented for LUNA PLATFORM 3 and LUNA PLATFORM 4 by means of the Backport 3 and Backport 4 services respectively.

The backports enable you to send requests that are similar to the requests from LUNA PLATFORM 3 and LUNA PLATFORM 4 and receive responses in the appropriate format.

There are some nuances when using backports.

• Data migration to Backports is complicated.

  Database structures of LUNA PLATFORM 3 and LUNA PLATFORM 4 differ from the database structure of LUNA PLATFORM 5. Therefore, additional steps should be performed for data migration and errors may occur.

• Backports have many restrictions.

  Not all logic from LP 3 and LP 4 can be supported in Backports. See the "Restrictions for working with Backport services" section for more information.

• Backports are limited by available features and are not developed.

  New features and resources of LUNA PLATFORM 5 are not supported when using Backports and they will never be supported. Thus, these services should only be used if it is not possible to perform a full migration to LUNA PLATFORM 5 and no new features are required.

Use case:

For example, you have a frontend application that sends requests to LUNA PLATFORM 3.

When you use the Backport 3 service, the requests to LP 3 are received by the service. They are formatted and sent to LUNA PLATFORM 5 according to its API format. LP 5 processes the request and sends the response to Backport 3. The Backport 3 service formats all the received results to the LP 3 format and sends the response.

Restrictions for working with Backport services#

As the tables of the saved data and the data itself differs for LP versions there are features and restrictions on the execution of requests. The information about features and restrictions is given in "Backport 3" and "Backport 4" sections of this document.

You can use Backport service and API service of LP 5 simultaneously following these restrictions:

• When you use Backport services, it is strongly recommended not to use the same account for requests to Backport service and the API service of LUNA PLATFORM 5.

• Perform the admin requests that do not use account ID to the LUNA PLATFORM 5 only.

Sdk resource#

The sdk resource allows you to detect faces and/or human bodies and estimate attributes in input images. After the request is performed, the received data is not saved to the database and Image Store, it only returned in the response.

The sdk resource combines the capabilities of other resources and also provides additional options. For example, with "/sdk" request you can do the following features:

• estimate presence of glasses in the image;
• estimate liveness in the image;
• estimate face/body parameter(s);
• aggregate face/body parameter(s);
• create a face sample and return its descriptor in the SDK format encoded in Base64;
• create a human body sample and return its descriptor in the SDK format encoded in Base64;
• extract face and body descriptor(s) and return them in the response;
• set descriptor quality score threshold to filter out images that are poorly suited for further processing;
• filter by mask states;
• other.

Face or body descriptor in the SDK format encoded in Base64 can be used in requests for matching of "raw" descriptors.

Accounts, tokens and authorization types#

Account#

An account is required to delimit the visibility areas of objects for a particular user. The account is necessary to perform most of the requests. Each created account has its own unique "account_id". All account data is stored in the Accounts service database under this ID.

The account can be created using a POST request "create account" to the API service, or using the Admin service . When creating the account, you must specify the following data: login (email), password and account type.

Account type#

The account type determines what data is available to the user.

There are three types of accounts:

• user — the type of account with which you can create objects and use only your account data.
• advanced_user — the type of account for which rights similar to "user" are available, and there is access to the data of all accounts. Access to data from other accounts means the ability to receive data (GET requests), check their availability (HEAD requests) and perform comparison requests based on data from other accounts.
• admin — the type of account for which rights similar to "advanced_user" are available, and there is also access to the Admin service.

In the API service, you can work with all types of accounts, but only "advanced_user" and "user" types of accounts can be created, while in the Admin service you can create all three types.

By default, there is the "admin" type account in the system with the default login and password root@visionlabs.ai/root.

Using the header "Luna-Account-Id" in the "create account" request you can set the desired account ID. It should also be used if it is necessary to preserve the ability to work with data created in LP versions prior to 5.30.0 by specifying the "account_id" in the "Luna-Account-Id" header. Thus, using this parameter will link the old "account_id" to the account being created (See "Account migration" section in the LUNA PLATFORM upgrade manual for details on migration).

In response to a request to create the account, "account_id" is issued. After creating the account, you can use this ID for LunaAccountIdAuth authorization type or use BasicAuth authorization type (login/password authorization).

Token#

Token is linked to an existing account with any type ("user", "advanced_user", "admin") and enables you to impose extended restrictions on the requests being made. For example, when creating the token, you can give the user permission only to create and modify all lists and faces, or you can prevent the use of certain handlers by specifying their ID.

The token is created using the "create token" request to the API service.

An unlimited number of tokens can be created for each account. The token and all its restrictions are stored in the database and linked to the account by the "account_id" parameter. You cannot link one token to different accounts. To create tokens with the same permissions across different accounts, it is recommended to save the request body template for creating the account and use it.

There is no need to use both the token and the account. When working with tokens, you can restrict access to the "/accounts" resource using external means.

You can give additional restrictions on the token at any time using the "replace token" request or you can revoke the token using the "delete token" request. In this case, the token will be deleted from the database and can no longer be used for authorization.

When creating the token, you need to set the following parameters:

• expiration_time – expiration time of the token in the RFC 3339 format. You can specify an infinite token expiration time using the value "null"
• permissions – actions that the user can perform (see "Permissions set in token")

You can also specify whether the token is visible to other accounts data using the "visibility_area" parameter (see the "Viewing other account data" section).

The response to the request is "token_id" and a Base64 encoded JWT token. After creating the token, you can use the received JWT token for BearerAuth authorization type.

Permissions set in token#

The following permissions are available for the generated token:

Custom rights are intended to proxy LUNA Streams requests through the LUNA API. In other cases, using user rights will have no effect. See "Proxying requests to LUNA Streams via LUNA API" in the FaceStream administrator manual.

Resources "/iso", "/sdk", "/liveness" and "/videosdk" do not require any authorization by default.

The value [] means no permissions.

The "emit_events" and "verify" permissions enables you to specify whether requests can be made to the "generate events" and "perform verification" requests, as well as blacklisting or whitelisting handler IDs. If handler IDs or verifier IDs are blacklisted, then only their use will be prohibited. If handler IDs or verifier IDs are present in the white list, then only their use will be allowed. The maximum number of IDs in the lists is 100. When using the "emit_events" permission, the user must not have the "creation" and "modification" Rights to use the handler.

If the "emit_events" permission is granted, then all necessary objects will be created during the generation of the event, regardless of the permissions specified in the token. For example, the "faces" type permission regulates work with faces only in requests to "faces/*" resources, but does not affect the creation of the face during event generation. Thus, when using the handler with the "store_face" parameter and not having the "creation" permission for "faces", the face will still be created.

If the "verify" permission is granted, then all necessary objects will be created during verification regardless of the permissions specified in the token. For example, the "sample" type permission does not affect the creation of a sample during verification. Thus, when using a verifier with the "store_sample" parameter and the absence of the "creation" right for "samples", a sample will still be created.

When setting permissions, the following features should be taken into account:

• The "modification" permission means performing of PATCH and PUT requests.
• Linking/unlinking face to list is the "modification" permission.
• Deleting list with "with_faces" setting enabled ("delete lists" request) requires "face" > "deletion" permission.
• Linking face to list in the "create face" request requires "list" > "modification" permission.
• When performing matching, you should have the "matching" permission for the corresponding objects (event, face, attribute).
• Requests to "get statistics on events" resource require "event" > "view" permission.
• Permission "event" > "create" only grants the right to create an event using the "save event" request. To generate an event, you should use the "emit_events" permission (see above).
• Permission "event" > "create" does not apply to verifiers.

See the token coverage table for specific resources in the developer manual of the API service.

Viewing other account data#

Other account details can be viewed if:

• account type is set to "advanced_user" or "admin"
• when creating the token, the "visibility_area" = "all" parameter was specified.

Account type "user" cannot be set to "visibility_area" = "all".

If the account type is set to "advanced_user"/"admin" and the token is created with the "visibility_area" = "account" parameter set, then when logging in using the token (BearerAuth) you will no longer be able to view data from other accounts, but when logging in using login/password (BasicAuth), this option will remain.

If the account type is set to "advanced_user"/"admin" and the token is created with "visibility_area" set to "all" and then the account type is changed to "user" (using the "patch account" request), then an attempt to perform a request to view the data of other accounts using the token will result in an error.

When using the LunaAccountIdAuth authorization, the visibility area is controlled using the "Luna-Account-Id" header.

For verifiers, the ability to use the visibility area of all accounts is not available. If the "visibility_area" = "all", only the data of your account will be visible.

Authorization types for accessing resources#

There are three types of authorization available in LUNA PLATFORM:

• BasicAuth. Authorization by login and password (set during account creation).
• BearerAuth. Authorization by JWT token (issued after the token is created).
• LunaAccountIdAuth. Authorization by "Luna-Account-Id" header, which specifies the "account_id" generated after creating the account. This method is considered deprecated and is not recommended for use.

LunaAccountIdAuth authorization can be enabled using the "ALLOW_LUNA_ACCOUNT_AUTH_HEADER" setting in the "OTHER" section of the API service settings in the Configurator (disabled by default).

In OpenAPI specification the "Luna-Account-Id" header is marked with the word Deprecated.

There is no need to use all three types of authorization when making requests. It is necessary to choose the preferred method depending on the required tasks.

If the authorization type is not specified in the request, an error with the status code 403 will be issued.

Credentials verifier#

Using the "verify credentials" resource, you can verify existing credentials by one of the types:

• login/password verification. If verification is successful, the "account_id" and account type will be returned.
• token verification. If verification is successful, the account type and all permissions for the token will be returned.
• account ID verification. If verification is successful, the account type will be returned.

Logging account information#

When executing any request, information about the corresponding account is displayed in the logs of the API service. This functionality enables you to determine who exactly executed a particular request. This may be required for information security and system administrators.

If the request was executed with BasicAuth or LunaAccountIdAuth type authorization, the following message will be displayed in the logs:

Request invoked by user (account_id: '270531af-e52e-4538-9181-628d9900a0db')

If the request was executed with BearerAuth type authorization, the following message will be displayed in the logs:

Request invoked by user (account_id: '270531af-e52e-4538-9181-628d9900a0db' token_id: 'd57e16f5-e243-47d2-aa85-8b200c12d86f')

If the request was executed without authorization, the following message will be displayed in the logs:

Request invoked by user (account_id: null)

The logs of the Accounts service additionally display information about the creation of tokens by specific users:

User with account_id: '270531af-e52e-4538-9181-628d9900a0db' create token: 'd57e16f5-e243-47d2-aa85-8b200c12d86f'

Logging information about the creation of tokens enables you to track where the token came from and which user it belonged to, even after it was deleted.

Estimated data#

This section lists the general parameters of faces, bodies and images estimated by LUNA PLATFORM 5 and how to get them.

You can get parameters using a some of tools and resources. Basically, getting parameters is done using the following methods:

1․ Extract gender and age from the image. Gender and age belong to the concept of basic attributes.

The "/extractor", "/sdk" resources and "extract_policy" of "/handlers" and "/verifiers" resources are used to extract basic attributes from a face image.

To extract basic attributes from the face using the "/extractor" resource, you should first create a sample using the "/detector" resource. In response to the "/extractor" request, the gender and age of the person will be returned. The extracted data have a TTL (time to live) and will be removed from the database after the specified period.

See the detailed description of extracting basic attributes in the "Descriptor extraction and attribute creation" section.

When estimating parameters using the "/sdk" resource, you should send the source image to LUNA PLATFORM 5 and specify the "estimate_basic_attributes" parameter in the query parameters. In response to the request, the gender and age of the person will be received. These parameters will not be saved to the database.

To extract this parameters using the "/handlers" and "/verifiers" requests, you should use the "extract_basic_attributes" parameter of the "extract_policy".

Gender and age estimation is also available by body image. This method is not accurate and is performed by body parameters estimation (see "Perform estimation of body parameters" below).

2․ Perform estimation of face and image parameters.

Various resources are used to estimate the parameters. The resources mainly used are "/detector", "/sdk", "/handlers" and "/verifiers".

When estimating parameters using the "/detector" resource, you need to send the source image to LUNA PLATFORM 5 and specify the estimation of the required face or image parameters in the query parameters. In response to request, a face sample will be created and the specified parameters will be given. The estimated parameters will not be saved in the database.

The method for getting parameters using the "/sdk" resource is similar to the method described above, however, the sample will not be created. The estimated parameters will also not be saved to the database.

To estimate parameters using "/handlers" and "/verifiers" resources, you should use the "detect_policy" with required parameters.

3․ Perform estimation of body parameters.

The ability to perform estimation of body parameters is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

The "/sdk" and "/handlers" resources are used to estimate body parameters. Using these resources is similar to performing face and image estimation (see above).

4․ Perform check of face and image parameters according to ISO/IEC 19794-5:2011 or custom conditions.

The ability to perform such checks is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

The "/iso" and "/detector" ("estimate_face_quality" parameter) resources and the "face_quality" check group of the "detect_policy" of the "/handlers" and "/verifiers" requests are used to perform the check.

The responses to requests show the overall result of passing all checks ("0" or "1"), as well as the results of each check.

See the detailed description of this functionality in the "Image check" section.

5․ Perform Liveness estimation.

The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

6․ Perform estimation of number of people in the image

All returned values and the response format depend on the resource where the estimation is performed.

In order to get results when sending requests to "/handlers" or "/verifiers" resources, you need to generate an event and perform verification on the specified handlers. See the Handler object section for more information about working with these resources.

Gender and age by face image#

This estimation determines the basic attributes (gender and age) of a person in the face image.

For details on basic attributes, see Attribute object.

Resources where the estimation is performed:

• "/extractor"

  Estimation name — "extract_basic_attributes".

• "/handlers"

  Estimation name — "policies" > "extract_policy" > "extract_basic_attributes".

• "/verifiers"

  Estimation name — "policies" > "extract_policy" > "extract_basic_attributes".

• "/sdk"

  Estimation name — "estimate_basic_attributes".

Face parameters#

Eyes attributes#

This estimation determines the following state of eyes for each of the eyes:

• "open"
• "closed"
• "occluded"

Poor quality images or ones that depict obscured eyes (for example, eyewear, hair, gestures) fall into the "occluded" category.

Iris landmarks are determined. An array of 34 landmarks is returned for each eye.

In the "/iso", "/detector" (image checking tool) and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_eyes_attributes".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_eyes_attributes".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_eyes_attributes".

• "/sdk"

  Estimation name — "estimate_eyes_attributes".

Eyes attributes estimation using image checking tools:

• "/iso" and "/detector" (see sections "7.2.3 Expression", point "a", "7.2.11 Visibility of pupils and irises" and "7.2.13 Eye patches" of the standard ISO/IEC 19794-5:2011).

  Checks names — "left_eye", "right_eye".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources.

  Checks names — "left_eye", "right_eye".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face landmarks#

There are two estimations of the face landmarks:

• Estimation of 5 face landmarks
• Assessment of 68 face landmarks

Landmarks are used for various purposes, for example, when creating the sample, when extracting the descriptor, etc. See the SDK documentation for a more detailed description of the landmarks.

• "/detector"

  Estimation name — "detect_landmarks68".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "detect_landmarks68".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "detect_landmarks68".

• "/sdk"

  Estimation name — "estimate_landmarks5".

  Estimation name — "estimate_landmarks68".

Distance between eyes#

Note: It is not possible to use a sample as an input image for this estimation.

It is possible to estimate the distance between the centers of the eyes in pixels. The estimated value is also compared with threshold (according to ISO or non-standard threshold).

This estimation can only be performed using image checking tools:

• "/iso" and "/detector" (see section "5.6.5 Eye and nostril centre Landmark Points" of the standard ISO/IEC 19794-5:2011).

  Check name — "eye_distance".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources.

  Check name — "eye_distance".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Red eyes effect#

This estimation determines the presence of the red eyes effect, where:

• "0" — Red eyes effect is not present in the image.
• "1" — Red eyes effect is present in the image.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for image quality:

The table below shows the requirement for natural light:

Resources where the estimation is performed:

Red eyes effect estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.3.4 Unnatural colour" of the standard ISO/IEC 19794-5:2011)

  Check name — "red_eyes".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "red_eyes".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Gaze#

This estimation determines the gaze. The gaze is represented by the following parameters:

• Pitch angle
• Yaw angle

A positive value of the pitch angle means looking up, and a negative value means looking down.

A positive value of yaw angle means looking to the right, and a negative value means looking to the left.

Zero position corresponds to a gaze direction orthogonally to the face plane, with the axis of symmetry parallel to the vertical camera axis.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_gaze".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_gaze".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_gaze".

• "/sdk"

  Estimation name — "estimate_gaze".

Gaze estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression" point "e" of the standard ISO/IEC 19794-5:2011)

  Checks names — "gaze_yaw", "gaze_pitch".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "gaze_yaw", "gaze_pitch".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Glasses#

This estimation determines the predominant state of the glasses from the following states:

• "sun_glasses"
• "glasses"
• "no_glasses"

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/sdk"

  Estimation name — "estimate_glasses".

Glasses estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.9 Eye glasses" of the standard ISO/IEC 19794-5:2011)

  Check name — "glasses".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "glasses".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Eyebrows#

This estimation determines the predominant state of the eyebrows from the following states:

• "neutral" — Eyebrows are in the usual position.
• "raised" — Eyebrows are raised.
• "squinting" — Eyes are narrowed, eyebrows are lowered.
• "frowning" — Eyebrows are frowned.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

It is possible to specify several eyebrow states as acceptable.

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for head pose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Eyebrows estimation is available only using using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression", points "d", "f" and "g" of the standard ISO/IEC 19794-5:2011)

  Check name — "eyebrows_state".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "eyebrows_state".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Mouth attributes#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1]:

• "opened"
• "smile"
• "occluded"

The probability that the mouth is opened is also determined.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_mouth_attributes".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_mouth_attributes".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_mouth_attributes".

• "/sdk"

  Estimation name — "estimate_mouth_attributes".

Mouth attributes estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression" point "a", "b" and "c" of the standard ISO/IEC 19794-5:2011)

  Checks names — "mouth_smiling", "mouth_occluded", "mouth_open".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "mouth_smiling", "mouth_occluded", "mouth_open".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Smile state#

This estimation determines the predominant state of a smile from the following states:

• "none" — No smile found, so no additional parameters are determined.
• "smile_lips" — Usual smile with closed lips.
• "smile_teeth" — Smile with open teeth.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

If necessary, you can specify several smile states as acceptable.

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for head pose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Smile state estimation is only available using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression" point "a", "b" and "c" of the standard ISO/IEC 19794-5:2011)

  Check name — "smile_properties".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "smile_properties".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image quality#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1], where 0 corresponds to low quality and 1 to high quality:

• "dark" — The degree of darkness in the photo.
• "light" — The degree of brightness in the photo.
• "blur" — The degree of blurriness.
• "illumination" — The degree of uniformity in illumination. The lower the difference between light and dark zones on the face, the higher the estimated value. When illumination is evenly distributed across the face, the value is close to "1."
• "specularity" — The degree of absence of glare. The higher the estimated value, the less glare and the better the image quality. If the estimated value is low, there are bright glares on the face.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image quality is determined using a specially trained VisionLabs neural network. If necessary, you can determine the illumination of the face in the image using an algorithm that performs an estimation in accordance with the ICAO standard (see the section "Illumination uniformity according to ICAO standard).

Examples are presented in the images below. Good quality images are shown on the right.

The most important image quality parameters for face recognition are darkness, light, and blur, so you should select them carefully.

The illumination and secularity parameters enable you to select images of better visual quality. Face recognition is not greatly affected by uneven illumination or glares.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_quality".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_quality".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_quality".

• "/sdk"

  Estimation name — "estimate_quality".

Image quality estimation using image checking tools:

• "/iso" and "/detector" (see sections "7.2.7 Subject and scene lighting", "7.3.2 Contrast and saturation", "7.3.3 Focus and depth of field", "7.2.8 Hot spots and specular reflections", "7.2.12 Lighting artefacts", "7.2.7 Subject and scene lighting" and "7.2.12 Lighting artefacts" of the standard ISO/IEC 19794-5:2011)

  Checks names — "illumination_quality", "specularity_quality", "blurriness_quality", "dark_quality", "light_quality".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "illumination_quality", "specularity_quality", "blurriness_quality", "dark_quality", "light_quality".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Illumination uniformity according to ICAO standard#

It is possible to estimate the uniformity of illumination according to the requirements specified in ICAO standard.

The estimated value is also compared with the threshold.

In accordance with the standard, it is recommended to use color images. When using black and white images, the results may be unexpected.

The uniformity of illumination according to the ICAO standard is only available using image checking tool:

• "/detector"

  Check name — "illumination_uniformity".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "illumination_uniformity".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image background#

Background lightness#

Note: It is not possible to use a sample as an input image for these estimations.

This estimation determines background lightness, where:

• [0...0.1] — The background is black.
• [0.1...0.3] — The background is dark.
• [0.3...0.97] — The background is light.
• [0.97...1] — The background is white.

Resources where the estimation is performed:

Background lightness estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "B.2.9 Backgrounds" of the standard ISO/IEC 19794-5:2011)

  Check name — "background_lightness".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "background_lightness".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Background uniformity#

Note: It is not possible to use a sample as an input image for these estimations.

This estimation determines the degree of background uniformity, where:

• "0" — The background is not uniform.
• "1" — The background is uniform.

Resources where the estimation is performed:

Background uniformity estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "B.2.9 Backgrounds" of the standard ISO/IEC 19794-5:2011)

  Check name — "background_uniformity".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "background_uniformity".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Dynamic range according to ICAO standard#

This estimation determines the ratio of brightness of the lightest and darkest areas of the face according to the requirements specified in ICAO standard.

The estimated value is also compared with the threshold.

Dynamic range estimation is only available using image checking tool — "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources.

Check name — "dynamic_range".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Natural light#

This estimation determines whether there is natural lighting on the face, where:

• "0" — The lighting is unnatural.
• "1" — The lighting is natural.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirement for mask:

The table below shows the requirement for image quality:

The table below shows the requirement for glasses:

Resources where the estimation is performed:

Natural light estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.3.4 Unnatural colour" of the standard ISO/IEC 19794-5:2011)

  Check name — "natural_light".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "natural_light".

Acceptable value for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face color type#

This estimation determines the most likely type of face color from the following:

• "color"
• "grayscale"
• "infrared" (near infrared range)

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

Face color type estimation is only available using image checking tools:

• "/iso" and "/detector" (see section "7.4.4 Use of near infra-red cameras" of the standard ISO/IEC 19794-5:2011)

  Check name — "face_color_type".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "face_color_type".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Head pose#

This estimation determines the head pose. The pose is defined by three parameters:

• pitch angle
• roll angle
• yaw angle

The angle values are specified in the range from "-180" to "180".

A positive pitch angle indicates an upward tilt of the head, while a negative angle indicates a downward tilt of the head.

A positive roll angle indicates a rightward deviation of the head, while a negative angle indicates a leftward deviation of the head.

A positive yaw angle indicates a rightward rotation of the head, while a negative angle indicates a leftward rotation of the head.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

In all the resources listed below, with the exception of "/iso", the possibility to filtered out by head pose is availiable.

In the resources "/detector", "/handlers", "/verifiers" and "/sdk", the threshold value is specified from "0" to "180". The default value is "180", which means that the head in the image can be rotated to any angle from "-180" to "180". When you set any other value (for example, "30"), all the detections with the etimated angle less than or equal to "-30" and greater than or equal to "30" will be filtered.

For the "face_quality" field of the "/handlers" and "/verifiers" resources, the minimum and maximum thresholds are set in separate fields.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_head_pose".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_head_pose".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_head_pose".

• "/sdk"

  Estimation name — "estimate_head_pose".

Below are the recommended thresholds for estimation in the "/detector", "/handlers", "/verifiers" and "/sdk" resources.

Recommended maximum thresholds:

The table below shows the recommended maximum thresholds head pose for estimation in cooperative mode:

The table below shows the recommended maximum thresholds head pose for estimation in non-cooperative mode:

Head pose estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.2 Pose" of the standard ISO/IEC 19794-5:2011)

  Checks names — "head_roll", "head_pitch", "head_yaw".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "head_roll", "head_pitch", "head_yaw".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Vertical and horizontal face position#

Note: It is not possible to use a sample as an input image for these estimations.

These estimations determine the position of the center point vertically and horizontally relative to the image.

The estimated values are also compared with thresholds (according to ISO or non-standard thresholds).

Resources where the estimation is performed:

Vertical and horizontal face position estimation is available only using image checking tools:

• "/iso" and "/detector" (see sections "8.3.2 Horizontally centred face" and "8.3.3 Vertical position of the face" of the standard ISO/IEC 19794-5:2011)

  Checks names — "head_horizontal_center", "head_vertical_center".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "head_horizontal_center", "head_vertical_center".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Head width and height#

Note: It is not possible to use a sample as an input image for these estimations.

This estimations determine the vertical and horizontal head size relative to the size of the image. The estimated values are also compared with thresholds (according to ISO or non-standard thresholds).

Resources where the estimation is performed:

Head width and height estimation is available only using image checking tools:

• "/iso" and "/detector" (see sections "8.3.4 Width of head" and "8.3.5 Length of head" of the standard ISO/IEC 19794-5:2011)

  Checks names — "head_width", "head_height".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "head_width", "head_height".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face width and height#

Note: It is not possible to use a sample as an input image for these estimations.

These estimations determine the face width and height in pixels. The estimated values are also compared with the specified thresholds.

Resources where the estimation is performed:

Face width and height estimations are only available using image checking tool:

• "/detector"

  Checks names — "face_width", "face_height".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "face_width", "face_height".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Indents from image edges#

Note: It is not possible to use a sample as an input image for these estimations.

This estimation is determined as the indent from the image border (left, right, top, bottom) to the face border (left, right, top, bottom) in pixels. The estimated values are also compared with the specified thresholds.

Resources where the estimation is performed:

Indents from image edges estimation is only available using image checking tool:

• "/detector"

  Checks names — "indent_upper", "indent_lower", "indent_right", "indent_left".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "indent_upper", "indent_lower", "indent_right", "indent_left".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Mask#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1]:

• "medical_mask" — The probability that the person wears a medical mask.
• "missing" — The probability that the person does not wear a mask.
• "occluded" — The probability that the face is occluded by an object other than a medical mask.

The predominant state of the mask is also determined.

In addition to the three main states, the following additional properties are defined:

• "correct" — There is mask on the face, the mouth and nose are occluded by the mask.
• "mouth" — There is mask on the face and occludes only the mouth.
• "clear" — There is no mask on the face.
• "chin" — There is mask on the face and is located under the chin, without occluding the area from eyes to mouth.
• "partially" — The face is partially occluded, but not by medical mask and not by mask with full face occlusion.
• "full" — There is mask on the face, in which the face is completely occluded, for example, balaclava/ski mask.

Each main mask state corresponds to one of two additional properties. The most likely additional property is returned in the "predominant_occlusion" field:

• The "medical_mask" state corresponds to the "correct" or "mouth" property.
• The "missing" state corresponds to the "clear" or "chin" property.
• The "occluded" state corresponds to the "partially" or "full" property.

For each of the properties, a probabilistic score is returned in the range [0..1].

Additional mask properties are not stored in the database and are not filtered by them.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_mask".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_mask".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_mask".

• "/sdk"

  Estimation name — "estimate_mask".

Emotions#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1]:

• "anger"
• "disgust"
• "fear"
• "happiness"
• "neutral"
• "sadness"
• "surprise"

The predominant emotion is also estimated.

Emotions can be saved in the event object during the event creation.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_emotions".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_emotions".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_emotions".

• "/sdk"

  Estimation name — "estimate_emotions".

Shoulders position#

This estimation determines the most predominant state of the shoulder position from the following:

• "non-parallel"
• "parallel"
• "hidden"

Resources where the estimation is performed:

Shoulders position estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.2.5 Shoulders" of the standard ISO/IEC 19794-5:2011)

  Check name — "shoulders_position".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "shoulders_position".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Headwear#

This estimation determines the most predominant type of headwear from the following:

• "none
• "baseball_cap"
• "beanie"
• "peaked_cap"
• "shawl"
• "hat_with_ear_flaps"
• "helmet"
• "hood"
• "hat"
• "other"

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

It is possible to specify several types of headwear as acceptable.

Image requirements:

The table below shows the requirements for head hose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Headwear estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "B.2.7 Head coverings" of the standard ISO/IEC 19794-5:2011)

  Check name — "headwear_type".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "headwear_type".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Radial distortion (Fisheye effect)#

This estimation determines the presence of the Fisheye effect, where:

• "0" — The Fisheye effect is not present in the image.
• "1" — The Fisheye effect is present in the image.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for head position:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Fisheye effect estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.3.6 Radial distortion of the camera lens" of the standard ISO/IEC 19794-5:2011)

  Check name — "radial_distortion".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "radial_distortion".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face occlusion#

The estimator returns information occlusion of the whole face or its parts.

This estimator returns information about:

• Overall face zone
• Forehead zone
• Nose zone
• Eyes zone (for both eyes)
• Mouth zone
• Lower face zone

Occlusion thresholds

The threshold for the acceptable percentage of the whole face or each of its zones can be specified in the Configurator service. The thresholds can also be set during the handler creation.

In addition, there is the threshold for the hair occlusion estimation. The threshold specifies the acceptable percantage of face occluded by hair. All the hair occlusion above the specified threshold is considered in the overall face occlusion and occlusion of each zone.

• If the hair threshold is set to 0, hair of any length is considered as an occlusion.
• If the hair threshold is set to 1, the hair occlusion is not considered.

The threshoulds can be changed according to your business cases.

Note Moustache and beard are never considered as a face occlusion.

Filtration

Filtration is available. You can specify list of zones that should not be occluded.

The detection is filtered if the occlusion of any of the specified zones exceeds the threshold.

Resources, where the estimation is performed

• "/handlers"

  Estimation name — "estimate_face_occlusion".

  Checks for face_quality:

  • face_occlusion - occlusion of the face zone;
  • lower_face_occlusion - lower face occlusion;
  • forehead_occlusion - forehead zone occlusion;
  • nose_occlusion - nose occlusion.

• "/verifiers"

  Estimation name — "estimate_face_occlusion".

  Checks for face_quality.

• "/sdk"

  Estimation name — "estimate_face_occlusion".

Image parameters#

Image format#

This estimation determines the correspondence of the incoming image format to one of the following formats — "JPEG", "JPEG2000", "PNG". The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

Image format estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.5 Format requirements for the Frontal Image Type" of the standard ISO/IEC 19794-5:2011)

  Check name — "image_format".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "image_format".

Acceptable values for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image size#

This estimation determines the image size in bytes. The estimated value is also compared with the specified threshold.

Resources where the estimation is performed:

Image size estimation is only available using image checking tool:

• "/detector"

  Check name — "image_size".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "image_size".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image width and height#

These estimations determine the image width and height in pixels. The estimated values are also compared with thresholds (according to ISO or non-standard thresholds).

Resources where the estimation is performed:

Image width and height estimations are available only using image checking tools:

• "/iso" and "/detector" (see sections "5.7.4 Width" and "5.7.5 Height" of the standard ISO/IEC 19794-5:2011)

  Checks names — "image_height", "image_width".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "image_height", "image_width".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Aspect ratio#

This estimation determines the proportional ratio of the image width to height. The estimated value is also compared with the specified threshold.

Resources where the estimation is performed:

Aspect ratio estimation is only available using image checking tool:

• "/detector"

  Check name — "aspect_ratio".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "aspect_ratio".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

EXIF metadata#

When the EXIF estimation is enabled, all the tags of the image are parsed and their names and values are outputted. Please refer to JEITA CP-3451 EXIF specification for details. The following data is returned:

• make
• model
• orientation
• latitude
• longitude
• artist
• software
• dateTime
• digitalZoomRatio
• flash
• uid

Resources where the estimation is performed:

• "/detector"

  Estimation name — "extract_exif".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "extract_exif".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "extract_exif".

• "/sdk"

  Estimation name — "use_exif_info".

Body parameters#

Gender and age by body image#

This estimation determines the basic attributes (gender and age) of a person in the body image.

Estimating gender and age from a body image is less accurate than from a face.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_basic_attributes".

• "/sdk"

  Estimation name — "estimate_body_basic_attributes".

Upper body#

This estimation determines the parameters of the following elements of clothing on the upper body:

• "headwear". Type: absent, present, undefined; color: white, black, other, undefined.
• "sleeve". Type: short, long, undefined.
• "upper_clothing". Color: orange, purple, red, white, yellow, pink, brown, beige, khaki, multicolored, undefined.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_upper_body".

• "/sdk"

  Estimation name — "estimate_upper_body".

Lower body#

This estimation determines the parameters of the following elements of clothing on the lower body:

• "lower_garment". Type: trousers, shorts, skirt, undefined; color: orange, purple, red, white, yellow, pink, brown, beige, khaki, multicolored, undefined.
• "shoes". Color: white, black, other, undefined.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_lower_body".

• "/sdk"

  Estimation name — "estimate_lower_body".

Backpack#

This estimation determines the presence of a backpack on the body:

• "0" — There is no backpack on the body image.
• "1" — There is a backpack on the body image.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_accessories".

• "/sdk"

  Estimation name — "estimate_accessories".

Face Image Modification#

The estimator Face Image Modification allows you to automatically determine the fact of image modification/editing by the presence of characteristic artifacts:

• frames (green/black), indicating broken pixels or the fact that the image is a screenshot;
• watermarks/logos;
• superimposed images (compositions of several images);
• superimposed figures (graphic elements over the image);
• superimposed text elements (numbering, inscriptions). The system reacts only to text artificially superimposed during editing; natural elements (tattoos, inscriptions on clothes) are not taken into account.

The system will consider the image to have been modified if the received score (a value from 0 to 1) is below the set threshold. The default threshold is 0.5 (50%).

Image requirements:

• minimum resolution 640 x 480 pixels;
• presence of only one face in the image.

Resources where the check is performed:

• "/handlers"

  Check name — "policies" > "detect_policy" > "face_quality" > "checks" > "image_modification".

• "/verifiers"

  Check name — "policies" > "detect_policy" > "face_quality" > "checks" > "image_modification".

Liveness#

Note: The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

The Liveness technology enables LUNA PLATFORM to detect presentation attacks. To estimate Liveness in the LUNA PLATFORM, the estimator LUNA SDK OneShotLiveness is used.

As a result of the Liveness estimation, one of the following results may be returned:

• "0" — The person is not real.
• "1" — The person is real.
• "2" — Result of the check is unknown.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_liveness".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_liveness".

• "/sdk"

  Estimation name — "estimate_liveness".

• "/videosdk"

  The estimation is performed only for the "human_tracking" analytics. During the analysis, Liveness is estimated per frame for each human track, and the aggregated estimation result is returned in the response.

  Estimation name — "analytics" > "targets" > "aggregated_liveness".

• "/liveness"

See "OneShotLiveness description" for more information about Liveness.

Deepfake#

Note: The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

Note: It is not possible to use a sample as an input image for these estimation.

This estimation enables you to detect facial substitution using DeepFake technology in photo images.

A Deepfake evaluation may return the following results:

• "prediction" = "fake" — The person is not real.
• "prediction" = "real" — The person is real.
• "score" = [0...1] — The degree of reliability of the estimation.

If necessary, you can configure the handler to filter events by the expected result of the Deepfake estimation ("fake" or "real"). To do this, in the request body to create a handler, specify the "deepfake_states" parameter with the value "0" (filter by the value "fake") or "1" (filter by the value "real"). For example, if the "deepfake_states" parameter is "1" (filter by "real"), and the estimator has determined that the result is "fake", then an empty "events" field will be returned in the event, and the results of the check will fall into the "filtered_detections" field.

In the requests "create handler" and "create verifier", it is possible to set the "real_threshold" and the operating "mode". The "sdk" request will use the default values of these parameters (see below) without the possibility of explicit indication.

Threshold

Using the "real_threshold", you can set a value in the range [0...1], below which the system will assume that the person is not real.

For example, if the threshold value is "real_threshold" = "0.5", and the degree of reliability of the estimation is "score" = "0.4", then the result "prediction" = "fake" will be output in the response body. If the threshold value is "real_threshold" = "0.6", and the degree of reliability of the estimation is "score" = "0.7", then the result "prediction" = "real" will be given in the response body.

The default value is "0.5".

Mode

The operating modes are described in the table below.

Image requirements

For correct verification results, the following requirements must be met.

The table below shows the image resolution requirements:

The table below shows the requirements for head pose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_deepfake".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_deepfake".

• "/sdk"

  Estimation name — "estimate_deepfake".

• "/videosdk"

  The estimation is performed only for the "human_tracking" analytics. During the analysis, Deepfake is estimated per frame for each human track, and the aggregated estimation result is returned in the response.

  Estimation name — "analytics" > "targets" > "aggregated_deepfake".

People count#

Note: The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

This estimate determines the number of people in the image.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_people_count".

• "/sdk"

  Estimation name — "estimate_people_count".

If necessary, together with estimating the number of people, you can get the X and Y coordinates of people using the "people_count_coordinates" parameter in the "/handlers" resource and the "people_coordinates" parameter in the "/sdk" resource.

Services interaction#

There are separate databases specified for each service in the images below. This is done for illustration purposes.

It is not required to launch several similar databases for each service. You can launch a single database instance (e. g., PostgreSQL) and use it to store data for all the LP services. Each service will have its own table in this database.

See "General concepts" for details about the databases used by LP services.

API service provides a RESTful interface for faces and bodies detection, extraction and matching procedures.

Requests are sent to LP via HTTP. A common case is when an external service sends requests to LP, receives results and processes the results according to business needs.

Some services can be disabled (see "General concepts").

API receives requests, processes them and checks user authorization using a request to the Accounts service (A0). The Accounts service checks if the user's credentials exist in the Accounts database (Acc1). If the user data is not found in the database, then an authorization error is issued. If the data is found, then the corresponding information is sent to the API service, where it passes the original requests to other services.

The main operations (detection, estimation, extraction, matching) are performed as described below.

Approach "Sequential performing of requests":

• The "detect faces" request, in which face detection is performed, face parameters are estimated and samples are created, is sent from the API service to the Handlers service (A1), and then redirected to Remote SDK service (H1).
• The "extract attributes" request, in which temporary attributes are extracted, is sent from the API service to the Handlers service (A1), and then redirected to the Remote SDK service (H1).
• The request "matching faces" is sent from the API service to the Python Matcher service (A6).

Approach "Parallel performing of requests":

• The "create handler" request with the rules for detecting, estimating, extracting and matching faces and bodies is sent from the API service to the Handlers service (A1).
• The "generate events" request is sent from the API service to the Handlers service (A1), and then redirected to the Remote SDK service (H1).

The API service sends requests to the Faces service to create/modify new faces (requests from the "faces" section), as well as create/modify lists (requests from the "lists") section (A2).

The API service receives information about the current license terms from the Licenses service (A3).

The API service sends samples from external services to the Image Store service (the "save face/body sample" request) (A5).

Handlers creates new handlers and stores them in the Handlers database (H2).

The service also redirects requests for detection and estimation to the Remote SDK (H1) service.

The Handlers service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Remote SDK processes face/body detection and attributes creation requests. It estimates face attributes and face/body parameters.

The Remote SDK service processes the request to detect faces/bodies and estimate parameters from the Handlers (H1) service, sends the result back to Handlers (H1), which sends the received samples to the Image Store service (H3).

The Remote SDK service processes the request to create attributes from the Handlers (H1) service by requesting existing samples from the Handlers service, which it requests from the Image Store service (H3). Next, the Handlers service sends the created temporary attributes to the Faces (H4) service, which saves them in the Redis database (F1).

Requests "/iso", "/sdk", "/liveness" are performed directly to the Remote SDK service (A9). In this case, the license is checked by interacting with the Licenses (RS1) service. To be able to use a set of samples, the Remote SDK service interacts with the Image Store (RS2) service.

Image Store receives samples from the Remote SDK service and stores them on a local storage or in S3-compatible cloud storage and provides access to them (Im1).

The Image Store service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Licenses. The Licenses service receives information about the number of created faces with attributes from the Faces database (Li1).

Faces is responsible for interaction with Faces database, which stores: faces, descriptors for faces, and lists (F2). It provides access to the stored data for API, Python Matcher, and Tasks services.

Faces also stores the created temporary attributes in the Redis database (F1).

Every time a new face is created, the Faces service sends information about the number of created faces with attached attributes to the Licenses service (F3) (see the "Faces limit" section).

Events service stores and provides information about events (the Events section). After an event is created, the Events service receives the created event from the Handlers service (H6) and stores it in the Events database (Ev2).

The Events service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Python Matcher can receive matching requests directly from the API service (A6). If a matching policy is specified in the handler, then a request to match descriptors will come from the Handlers (H7) service. Python Matcher sends matching requests to the Faces database (M3) or Events database (M1). The databases matches descriptors and sends results back to the API service.

Python Matcher can also receive the temporary attributes required for matching from the Redis database (M2).

It is possible to additionally use the Python Matcher Proxy service, which can redirect requests to Python Matcher or matching plugins. Matching plugins can significantly speed up the matching requests execution (see the "Matching plugins" section). This service is not shown in the diagram, but has the same communication lines as the Python Matcher service.

Admin. A user can manage accounts and perform other actions via the user interface of the Admin service (Adm1). The corresponding requests are sent to the Admin service (Adm2).

The Admin service receives information about accounts from the Accounts (Adm3) service.

All information about user accounts is stored in the Accounts database (Acc1).

The Admin service sends requests to Tasks service (Adm4). These tasks are performed for LP databases in general, not for a single account ID.

The Admin service receives information about the current license terms from the Licenses service (Adm5).

The Admin service checks the current number of created faces with linked attributes in the Faces database (Adm6) and calculates the percentage of the database of fullness.

The Admin service performs requests to all LP services, receiving system information (see "/luna_sys_info" request) (Adm7).

Sender. All generated events are received by the Redis DB from the Handlers service (H5). External third-party party software subscribes to receive events according to the specified filters (Se2). Sender service checks Redis channel, receives the required information and sends it to external software (Se1).

The Sender service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Configurator. LP services receive configuration parameters from the Configurator service (Con1). A user can manage configurations in Configurator by means of the user interface (Con2). The changes are sent to Configurator (Con3). All the changes in configuration files are saved in Configurator DB (Con4).

Configurator is utilized for each Luna service by default.

Tasks. Tasks service receives requests from API service (A8). Then Tasks creates and stores a task in the Tasks database (T1).

Next, the Tasks service interacts with its worker via Redis (T2, T3), creating subtasks and merging the results. For more information, see "Task diagrams" section.

The Tasks service receives data from the Faces service for the Clustering, Linker and Additional extraction tasks (T4).

The Tasks service receives data from the Events service for the Clustering and Linker tasks (T9, T10).

The Tasks service receives data from the Handlers service for the Estimator task (T6).

The Tasks service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

The interaction of Tasks workers with other services depends on the task type.

Tasks workers receive data from Faces service for every processed task (T5).

Tasks workers send a request to Python Matcher (T7) for clustering, cross-matching and ROC creation tasks.

Tasks workers store all the reports in the storage of Image Store (T8). Tasks workers also store and receive clusters from Image Store.

Tasks workers send the request for additional extraction task to the Handlers service (T6).

Monitoring. Monitoring system receives requests and errors from each service (Mon1). This data stores in the Influx database (Mon2). Then the monitoring data is sent to Grafana to visualize the monitoring data (Mon3). You can find more information about monitoring in the "Monitoring" section.

Grafana and InfluxDB services have their own interfaces (see the "User interfaces" section).

This diagram does not contain the architecture of Backport 3, Backport 4, Lambda services and video analytics services (Video Agent and Video Manager). See the detailed architecture of the Backport 3 service in "Backport 3 architecture" and the Backport 4 service in "Backport 4 architecture". See the Lambda service interaction diagram in "Lambda diagrams".

Services description#

This section provides more details on functions of the LP services.

Databases can be omitted in the following figures.

See the table with the resource consumption of each of the services listed below in the "Resource consumption by services" section.

General information about services#

Worker processes#

For LUNA PLATFORM services, you can set the number of workers to use additional resources and system memory to process requests to the service. A service will automatically spin up multiple processes and route traffic between the processes.

When starting the service in a Docker container, the number of workers is set using the WORKER_COUNT parameter.

For example, if you set the value WORKER_COUNT=2 for the Faces service, then the service will consume 2 times more resources and memory.

Note the number of available cores on your server when utilizing this feature.

Worker processes utilization is an alternative way for linear service scaling. It is recommended to use additional worker processes when increasing the number of service instances on the same server.

It is not recommended to use additional worker processes for the Remote SDK service when it utilizes GPU. Problems may occur if there is not enough GPU memory, and the workers will interfere with each other.

Automatic configurations reload#

LP services support the auto-reload of configurations. When a setting is changed, it is automatically updated for all the instances of the corresponding services. When this feature is enabled, no manual restart of services is required.

This feature is available for all the settings provided for each Python service. You should enable the feature manually upon each service launching. See the "Enable automatic configuration reload" section.

Starting with version 5.5.0 the configuration reload for Faces and Python Matcher services is done mostly by restarting appropriate processes.

Restrictions#

Service can work incorrectly while new settings are being applied. It is strongly recommended not to send requests to the service when you change important settings (DB setting, work plugins list, and others).

New settings appliance may lead to service restart and caches resetting (e. g., Python Matcher service cache). For example, the default descriptor version changing will lead to the LP restart. Changing the logging level does not cause service restart (if a valid setting value was provided).

Enable automatic configuration reload#

You can enable this feature by specifying a --config-reload option in the command line. In Docker containers, the feature is enabled using the "RELOAD_CONFIG" option.

You can specify the configurations check period in the --pulling-time command line argument. The value is set to 10 seconds by default. In Docker containers, the feature is enabled using the "RELOAD_CONFIG_INTERVAL" option.

Configurations update process#

LP services periodically receive settings from the Configurator service or configuration files. It depends on the way of configurations receiving for a particular service.

Each service compares its existing settings with the received settings:

• If service settings were changed, they will pulled and applied.

  • If the configurations pulling has failed, the service will continue working without applying any changes to the existing configurations.

  • If check connections with new settings have failed, the service will retry new configurations pulling after 5 seconds. The service will shut down after 5 failed attempts.

• If current settings and new pulled settings are the same, the Configurator service will not perform any actions.

API service#

LUNA API is a facial recognition web service. It provides a RESTful interface for interaction with other LUNA PLATFORM services.

Using the API service you can send requests to other LP services and solve the following problems:

• Images processing and analysis:

  • Face/body detection in photos.

  • Face attributes (age, gender, ethnicity) and face parameters (head pose, emotions, gaze direction, eyes attributes, mouth attributes) estimation.

  • Body parameters (age, gender, accessories, headwear, colors of upper and lower clothing, type of sleeves) estimation.

• Search for similar faces/bodies in the database.

• Storage of the received face attributes in databases.

• Creation of lists to search in.

• Statistics gathering.

• Flexible request management to meet user data processing requirements.

Remote SDK service#

The Remote SDK service is used to:

• Perform face detection and face parameters estimation.
• Perform body detection and body parameters estimation.
• Create samples.
• Perform extraction of basic attributes and descriptor, including aggregated ones.
• Process images using handlers and verifiers policies.

Face, body detection, descriptor extraction, estimation of parameters and attributes are performed using neural networks. The algorithm evolves with time and new neural networks appear. They may differ from each other by performance and precision. You should choose a neural network following the business case of your company.

Remote SDK with GPU#

Remote SDK service can utilize GPU instead of CPU for calculations. A single GPU is utilized per Remote SDK service instance.

Attributes extraction on the GPU is engineered for maximum throughput. The input images are processed in batches. This reduces computation cost per image but does not provide the shortest latency per image.

GPU acceleration is designed for high load applications where request counts per second consistently reach thousands. It won’t be beneficial to use GPU acceleration in non-extensively loaded scenarios where latency matters.

Aggregation#

Based on all images transferred in one request, a single set of basic attributes and an aggregated descriptor can be obtained. In addition, during the creation of the event, the aggregation of the received values of Liveness, emotions, medical mask states for faces and upper/lower body, gender, age and the body accessories for bodies is performed.

The matching results are more precise for aggregated descriptor. It is recommended to use aggregation when several images were received from the same camera. It is not guaranteed that aggregated descriptors provide improvements in other cases.

It is considered that each parameter is aggregated from sample. Use the "aggregate_attributes" parameter of the "extract attributes" (only for faces) and "sdk" requests to enable attributes aggregation. Aggregation of liveness, emotion, and mask states for faces and upper body, gender, age and the body accessories for bodies is available using the "aggregate_attributes" parameter in the "generate events", provided that these parameters were estimated earlier in the handler, as well as in the "sdk" request.

An array of "sample_ids" is returned in the response even if there was only a single sample used in the request. In this case, a single sample ID is included in the array.

Descriptor formats#

LUNA PLATFORM supports the following descriptor formats:

Note: Raw and XPK files are deprecated. It is recommended to work with the SDK format.

SDK and Raw formats can be directly linked to a face or stored in a temporary attribute (see "Create objects using external data" below).

In most extraction requests, the descriptor is saved to the database as set of bytes, without being returned in the response body.

There are several requests that can be used to get descriptor in SDK format:

• request "sdk";
• request "get temporary attributes and "get temporary attribute.

With LUNA PLATFORM, it is not possible to get descriptors in Raw and SDK formats. You can use other VisionLabs software to get these formats (eg LUNA SDK). Descriptors obtained using the above resources or using the VisionLabs software are referred to as raw descriptors.

Use raw descriptors for matching

The descriptor formats described above can be used in requests for the use of raw descriptors.

An external raw descriptor can be used as reference in the following resources:

• "/matcher/faces"
• "/matcher/bodies"
• "/matcher/raw"
• "/handlers/{handler_id}/events" when "multipart/form-data" request body schema is set
• "/verifiers/{verifier_id}/verifications" when "multipart/form-data" request body schema is set
• "/verifiers/{verifier_id}/raw"

An external raw descriptor can be used as a candidate in the following resources:

• "/matcher/raw"
• "/verifiers/{verifier_id}/raw"

Create objects using external data#

You can create a temporary attribute of face by sending basic attributes and descriptors to LUNA PLATFORM. Thus you can store this data in external storage and send it to LP for the processing of requests only.

You can create an attribute or face using:

• Basic attributes and their samples.
• Descriptors (raw descriptor in Base64 or SDK descriptor in Base64).
• Both basic attributes and descriptors with the corresponding data.

Samples are optional and are not required for an attribute or face creation.

See the "create temporary attribute" request and "create face request" for details.

Checking images for compliance with standards#

The Remote SDK service enables you to check images according to the ISO/IEC 19794-5:2011 standard or user-specified thresholds using three ways:

• Request "iso".
• Parameter "estimate_face_quality" of the request "detect faces".
• Group of parameters "face_quality" of the policy "detect_policy" of the request "generate events".

For example, it is necessary to check whether the image is of a suitable format, specifying the "JPEG" and "JPEG2000" formats as a satisfactory condition. If the image fits this condition, the system will return the value "1", if the format of the processed image is different from the specified condition, the system will return the value "0". If the conditions are not set, the system will return the estimated value of the image format.

The list of estimations and checks performed is described in the "Image check" section.

The ability to perform check and estimation of image parameters is regulated by a special parameter in the license file.

Enable/disable several estimators and detectors#

By default, the Remote SDK service is launched with all estimators and detectors enabled. If necessary, you can disable the use of some estimators or detectors when launching the Remote SDK container. Disabling unnecessary estimators enables you to save RAM or GPU memory, since when the Remote SDK service launches, the possibility of performing these estimates is checked and neural networks are loaded into memory.

If you disable the estimator or detector, you can also remove its neural network from the Remote SDK container.

Disabling estimators or detectors is possible by transferring documents with the names of estimators to the launch command of the Remote SDK service. Arguments are passed to the container using the "EXTEND_CMD" variable.

List of available estimators:

You can explicitly specify which estimators and detectors are enabled or disabled by passing the appropriate arguments to the "EXTEND_CMD" variable, or you can enable (by default) or disable them all with the enable-all-estimators-by-default argument. You can turn off the use of all estimators and detectors, and then turn on specific estimators by passing the appropriate arguments.

Example of a command to start the Remote SDK service using only a face detector and estimators of a face sample and emotions.

docker run \
...
--env=EXTEND_CMD="--enable-all-estimators-by-default=0 --enable-face-detector=1 --enable-face-warp-estimator=1 --enable-emotions-estimator=1" \
...

Handlers service#

The Handlers service is used to create and store handlers and verifiers.

The data of handlers and verifiers are stored in the Handlers database.

Send events to third-party service#

LUNA PLATFORM provides the ability to send notifications via web sockets or web hooks (HTTP). This is facilitated by the "callbacks" policy.

Sending notifications via web sockets

The "callbacks" policy with the luna-ws-notification parameter provides a notification mechanism based on WebSocket principles. This type of callback allows receiving events through web sockets from the Sender service, which interacts with the Handlers service via the pub/sub mechanism through the Redis channel. This ensures direct, instant data updates using a bidirectional communication channel between the client and server.

Advantages:

• Direct, instant data updates via web sockets.
• Efficient use of an open bidirectional channel.
• Low latency in notification delivery.

In previous versions of LUNA PLATFORM, the "notification_policy" was used. It is now considered deprecated and is not recommended for use. The main advantage of the callback mechanism over the deprecated "notification_policy" is the ability to specify multiple callbacks with different filters in the handler creation request, resulting in only one event being sent.

See detailed information in the "Sender service" section.

Sending notifications via web hooks

The "callbacks" policy with the http parameter provides a notification mechanism based on webhook principles for HTTP. They ensure asynchronous interaction between systems, allowing external services to react to the occurrence of events. Within this policy, you can specify specific parameters such as protocol type, external system address, and authorization parameters and data.

Advantages:

• More flexible notification setup mechanism.
• Easy integration with various external systems.
• Uses familiar HTTP protocols and configurations.

Image Store service#

The Image Store service stores the following data:

• Face and body samples. Samples are stored in Image Store by the Remote SDK service or you can save an external sample using the "samples" > "detect faces" and "samples" > "save face/body sample" requests.
• Reports about tasks. Reports are stored by the Tasks service workers.
• Any objects loaded using the "create objects" request.
• Clusterization information.

Image Store can save data either on a local storage device or in S3-compatible cloud storage.

Buckets description#

The data is stored in special directories called buckets. Each bucket has a unique name. Bucket names should be set in lower case.

The following buckets are used in LP:

• "visionlabs-samples". This bucket stores face samples.
• "visionlabs-bodies-samples". This bucket stores human bodies samples.
• "visionlabs-image-origin". This bucket stores source images.
• "visionlabs-objects". This bucket stores objects.
• "task-result". This bucket stores the results received after tasks processing using the Tasks service.
• "portraits". This bucket stores portraits. The bucket is required for the usage of Backport 3 service.

Buckets creation is described in LP 5 installation manual in the "Buckets creation" section.

After running the Image Store container and the commands for containers creation, the buckets are saved to local storage or S3.

By default, local files are stored in the "/var/lib/luna/current/example-docker/image_store" directory on the server. They are saved in the "/srv/local_storage/" directory in the Image Store container.

Bucket includes directories with samples or other data. The names of the directories correspond to the first four letters of the sample ID. All the samples are distributed to these directories according to their first four ID symbols.

Next to the bucket object is a "*.meta.json" file containing the "account_id" used when performing the request. If the bucket object is not a sample (for example, the bucket object is a JSON file in the "task-result" bucket), then the "Content-Type" will also be specified in this file.

An example of the folders structure in the "visionlabs-samples", "task-result" and "visionlabs-bodies-samples" buckets is given below.

./local_storage/visionlabs-samples/8f4f/
            8f4f0070-c464-460b-sf78-fac234df32e9.jpg
            8f4f0070-c464-460b-sf78-fac234df32e9.meta.json
            8f4f1253-d542-621b-9rf7-ha52111hm5s0.jpg
            8f4f1253-d542-621b-9rf7-ha52111hm5s0.meta.json
./local_storage/task-result/1b03/
            1b0359af-ecd8-4712-8fc0-08401612d39b
            1b0359af-ecd8-4712-8fc0-08401612d39b.meta.json
./local_storage/visionlabs-bodies-samples/6e98/
            6e987e9c-1c9c-4139-9ef4-4a78b8ab6eb6.jpg
            6e987e9c-1c9c-4139-9ef4-4a78b8ab6eb6.meta.json

A significant amount of memory may be required when storing a large number of samples. A single sample takes about 30 Kbytes of the disk space.

It is also recommended to create backups of the samples. Samples are utilized when the NN version is changed or when you need to recover your database of faces.

Use S3-compatible storage#

To enable the use of S3-compatible storage, you must perform the following steps:

• Make sure that the access key has sufficient authority to access the buckets of the S3-compatible storage.
• Launch the Image Store service (see "Image Store" section in the installation manual).
• Set the "S3" value for the "storage_type" setting of the Image Store service settings.
• Fill in the settings for connecting to an S3-compatible storage (host, Access Key and Secret Key, etc.) in the "S3" section of the Image Store service settings.

If necessary, you can disable SSL certificate verification using the "verify_ssl" setting in the "S3" section of the Image Store service settings. This enables you to use a self-signed SSL certificate.

Supported cloud providers#

Image Store supports S3-like storages. Support has been tested on MinIO and standard S3. For other storages, the degree of compatibility depends on their compliance with the S3 API.

Object TTL#

You can set the object time to live (TTL) in buckets (both local and S3). Objects mean:

• Samples of faces or bodies.
• Images or objects created in the resources "/images" or "/objects".
• Source images.
• Task results.

TTL for objects is calculated relative to the GMT time format.

TTL for objects is set in days in the following ways:

• During the creation of a bucket for all objects at once (basic TTL bucket policy).
• After creating a bucket for specific objects using requests to the corresponding resources.

The number of days is selected from the list in the corresponding requests (see below).

In addition to the number of days, the parameter can take the value "-1", meaning that objects should be stored indefinitely.

Configuring basic TTL bucket policy#

The basic TTL bucket policy can be configured in the following ways:

• Using the --bucket-ttl flag for prepare command with the argument lis_bucket. For example:

  docker run \
  --rm \
  --network=host \
  dockerhub.visionlabs.ru/luna/storages:v.0.5.51 \
  bash -c "luna_prepare prepare lis_bucket \
        --bucket-ttl=2"
  

• Using a request to the Image Store service. For example, curl -X POST http://127.0.0.1:5020/1/buckets?bucket=visionlabs-samples?ttl=2.

Configuring TTL for specific objects#

TTL for specific objects can be configured using the "ttl" parameter in the following places:

• In the "storage_policy" > "face_sample_policy", "body_sample_policy" and "image_origin_policy" handler policies.
• In the requests "create object", "create images" and "save sample".
• In the requests to create tasks or schedules in the "result_storage_policy" field.

If the "ttl" parameter is not specified, then the basic policy of the bucket in which the object is located (see above) will be applied.

Adding TTL to existing objects#

You can add a TLL to an existing object using PUT requests to the /objects, /images, /samples/{sample_type} resources of the Image Store service. It is not possible to add TTL of task results to already created and executed tasks. You can add TTL of task results to an already created schedule using the request "replace tasks schedule". For tasks created or running at the time of the request, the TTL of task results will not be applied.

You can add a TTL to an existing local bucket using a PUT request to the Image Store resource /buckets.

To add a TTL for a bucket located in S3, you need to perform a migration using the "base_scripts/migrate_ttl_settings" script from the Image Store service. This is because for TTL objects in S3 is applied via tag related filters. The command to perform S3 bucket migration is given in the installation manual. See "Migration to apply TTL to objects in S3" for details on S3 bucket migration.

Migration to apply TTL to objects in S3#

Lifecycle customization for S3 is applied through filters associated with tags (see official documentation). This assumes that objects have a tag with a limited set of values, and buckets have a set of rules based on the value of that tag.

To add tags and rules, you must perform a migration. Migration is strictly necessary to fully apply lifecycle customization for the following reasons:

• Buckets without rules will not delete objects, even if the user specifies a lifetime for a specific object.
• Objects without tags will never be deleted, even if the user specifies a lifetime for the bucket.

You need to add the following tags and rules:

• To support TTL for buckets, you need to add a vl-expire tag with a default value for all existing objects.
• To support TTL for specific objects, you need to add a set or TTL-related lifecycle rules for existing segments:

{
     "ID": "vl-expire-<ttl>",
     "Expiration": {
         "Days": <ttl>,
     },
     "Filter": {"Tag": {"Key": "vl-expire", "Value": <ttl>}},
     "Status": "Enabled",
}

A set of specific tag values associated with an object's TTL is supported: 1, 2, 3, 4, 5, 6, 7, 14, 30, 60, 90, 180, 365.

The migration process consists of two stages:

• Configuration of the bucket life cycle, expanded with a set of life cycle rules related to TTL.
• Assigning a vl-expire tag to each object in the bucket, if it does not already have one.

Assigning a tag for each object can be skipped if necessary using the -update-tags=0 argument.

See the upgrade manual for example commands to perform the migration.

Permission issues

By default, all S3 resources, including buckets, objects, and lifecycle configuration, are private. If necessary, default rules and tags can be created manually by the resource owner using one of the applicable methods. See the official S3 documentation for details.

Useful links to official documentation:

• Creating a lifecycle configuration
• Put lifecycle configuration
• Managing object tags
• Expiring objects

Expiration of TTL#

When an object's TTL comes to an end, it is marked for deletion. For local buckets, the cleanup task is performed once a day (at 01:00 am). S3 buckets use internal ttl configuration rules. To prevent conflicts or duplication of cleanup tasks when multiple instances or worker processes are involved, a locking mechanism is implemented. This ensures that only one instance or worker process is responsible for performing the local storage cleanup process.

There may be a delay between the expiration date and the date the item is actually deleted. Both for local storage and S3.

Search for expiring objects#

To find out when an object expires, you can use queries with the HEAD methods on the /objects and /images resources. These requests return X-Luna-Expiry-Date response headers, which indicate the date on which the object is no longer eligible for persistence.

External samples#

You can send an external sample to Image Store. The external sample is received using third-party software or the VisionLabs software (e. g., FaceStream).

See the POST request on the "/samples/{sample_type}" resource in "APIReferenceManual.html" for details.

The external sample should correspond to certain standards so that LP could process it. Some of them are listed in the "Sample requirements" section.

The samples received using the VisionLabs software satisfy this requirement.

In case of third-party software, it is not guaranteed that the result of the external sample processing will be the same as for the VisionLabs sample. The sample can be of low quality (too dark, blurry and so on). Low quality leads to incorrect image processing results.

Anyway, it is recommended to consult VisionLabs before using external samples.

Accounts service#

The Accounts service is intended for:

• Creation, management and storage of accounts.
• Creation, management and storage of tokens and their permissions.
• Verification of accounts and tokens.

See "Accounts, tokens and authorization types" section for more information about the authorization system in LUNA PLATFORM 5.

All created accounts, tokens and their permissions are saved in the Accounts service database.

JWT Tokens algorithms#

The JWT (JSON Web Tokens) authentication mechanism supports various algorithms for token signing. This section describes the default algorithm used and the necessary steps to use an alternative algorithm.

Default algorithm#

By default, the service uses the HS256 algorithm to sign JWT tokens. If you want to use asymmetric cryptographic encryption, you can use the ES256 algorithm.

Use ES256 algorithm#

To use the ES256 algorithm, follow these steps:

1․ Generate a private ECDSA key.

First, you need to generate a private ECDSA key using the prime256v1 curve. This can be done using command-line tools such as OpenSSL.

Example command:

openssl ecparam -genkey -name prime256v1 -out ec_private.pem

You can also generate a key protected by a password, for example:

openssl ecparam -genkey -name prime256v1 | openssl ec -aes256 -out ec_private_enc.pem

2․ Encode the private key in Base64.

After generating the private key, encode it in Base64 format. This can be achieved with tools available in most operating systems.

Example command:

base64 -w0 ecdsa_private.pem > ecdsa_private_base64

3․ Set the environment variable.

The encoded private key must be specified in the ACCOUNTS_JWT_ECDSA_KEY environment variable when starting the container. This allows the service to use the key for signing JWT tokens with the ES256 algorithm.

Additionally, if your private key is protected with a password, you can specify the password in the ACCOUNTS_JWT_ECDSA_KEY_PASSWORD environment variable.

Example container run command with environment variables:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=ACCOUNTS_JWT_ECDSA_KEY=jwt_ecdsa_key \
--env=ACCOUNTS_JWT_ECDSA_KEY_PASSWORD=ecdsa_key_password \
...

By following these steps, the service will be able to sign JWT tokens using the ES256 algorithm, providing enhanced security through asymmetric cryptography.

Impact of changing algorithm type#

Switching the signing algorithm from HS256 to ES256 (or vice versa) has a significant impact on token validation. All existing tokens signed with the previous algorithm will become invalid after the changes are made. This happens because the token signature verification mechanism expects the structure and cryptographic base of the token to match the newly specified algorithm.

Faces service#

Faces service is used for:

• Creating temporary attributes.
• Creating faces.
• Creating lists.
• Attaching faces to lists.
• Managing of the general database that stores faces with the attached data and lists.
• Receive information about the existing faces and lists.

Matching services#

Python Matcher has the following features:

• Matching according to the specified filters. This matching is performed directly on the Faces or the Events database. Matching by DB is beneficial when several filters are set.
• Matching by lists. In this case, it is recommended that descriptors are save in the Python Matcher cache.

Python Matcher Proxy is used to route requests to Python Matcher services and matching plugins.

Python Matcher#

Python Matcher utilizes Faces DB for filtration and matching when faces are set as candidates for matching and filters for them are specified. This feature is always enabled for Python Matcher.

Python Matcher utilizes Events DB for filtration and matching when events are set as candidates for matching and filters for them are specified. The matching using the Events DB is optional, and it is not used when the Events service is not utilized.

A VLMatch matching function is required for matching by DB. It should be registered for the Faces DB and the Events DB. The function utilizes a library that should be compiled for your current DB version. You can find information about it in the installation manual in "VLMatch library compilation", "Create VLMatch function for Faces DB", and "Create VLMatch function for Events DB" sections.

Python Matcher service additionally uses workers that process requests.

Python Matcher Proxy#

The API service sends requests to the Python Matcher Proxy if it is configured in the API configuration. Then the Python Matcher Proxy service redirects requests to the Python Matcher service or to matching plugins (if they are used).

If the matching plugins are not used, then the service route requests only to the Python Matcher service. Thus, you don't need to use Python Matcher Proxy unless you intend to use matching plugins. See the "Matching plugins" section for a description of how the matching plugins work.

List caching#

When faces are specified as candidates for matching and list IDs for them are specified as filters, Python Matcher performs a matching by lists.

By default, when the Python Matcher service is launched, all descriptors in all lists are cached in its memory.

Caching is managed by the "DESCRIPTORS_CACHE" section.

The Python Matcher service will not start until it loads all available descriptors into the cache.

When executing a list matching request, the Python Matcher service automatically adds it to the queue, from where it is picked up by the worker and sent to the Cached Matcher entity to perform a matching on cached data.

After performing the matching, the worker takes the results and returns them to the Python Matcher service and the user.

This caching enabling you to significantly increase the performance of the matching.

If necessary, you can process only specific lists using the parameter "cached_data > faces_lists > include" or exclude lists using the parameter "cached_data > faces_lists > exclude". The latter is especially useful when working with the LUNA Index Module to implement the logic of processing parts of lists using Python Matcher, and parts using LIM Indexed Matcher.

For more information about LIM, see "Matching a large set of descriptors".

Workers cache#

When multiple workers are launched for the Python Matcher service, each of the workers uses the same descriptors cache.

This change can both speed up and slow down the service. If you need to ensure that the cache is stored in each of the Python Matcher workers, you should run each of the server instances separately.

Events service#

The Events service is used for:

• Storage of all the created events in the Events database.
• Returning all the events that satisfy filters.
• Gathering statistics on all the existing events according to the specified aggregation and frequency/period.
• Storage of descriptors created for events.

As the event is a report, you can't modify already existing events.

The Events service should be enabled in the API service configuration file. Otherwise, events will not be saved to the database.

Database for Events#

PostgreSQL is used as a database for the Events service.

The speed of request processing is primarily affected by:

• The number of events in the database.
• Lack of indexes for PostgreSQL.

PostgreSQL shows acceptable requests processing speed with the number of events from 1 000 000 to 10 000 000. If the number of events exceeds 10 000 000, the request to PostgreSQL may fail.

The speed of the statistics requests processing in the PostgreSQL database can be increased by configuring the database and creating indexes.

Geo position#

You can add a geo position during event creation.

The geo position is represented as a JSON with GPS coordinates of the geographical point:

• "longitude" — Geographical longitude in degrees.
• "latitude" — Geographical latitude in degrees.

The geo position is specified in the "location" body parameter of the event creation request. See the "Create new events" section of the Events service reference manual.

You can use the geo position filter to receive all the events that occurred in the required area.

Geo position filter#

A geo position filter is a bounding box specified by coordinates of its center (origin) and some delta.

It is specified using the following parameters:

• "origin_longitude"
• "origin_latitude"
• "longitude_delta"
• "latitude_delta"

The geo position filter can be used when you get events, get statistics on events, and perform events matching.

Geo position filter is considered as properly specified if:

• both "origin_longitude" and "origin_latitude" are set.
• neither "origin_longitude", "origin_latitude", "longitude_delta", or "latitude_delta" is set.

If both "origin_longitude" and "origin_latitude" are set and "longitude_delta" is not set — the default value is applied (see the default value in the OpenAPI documentation).

Read the following recommendations before using geo position filters.

The general recommendations and restrictions for geo position filters are:

• Do not create filters with a vertex or a border on the International Date Line (IDL), the North Pole or the South Pole. They are not fully supported due to the features of database spatial index. The filtering result may be unpredictable.
• Geo position filters with edges more than 180 degrees long are not allowed.
• It is highly recommended to use the geo position filter citywide only. If a larger area is specified, the filtration results on the borders of the area can be unexpected due to the spatial features.
• Avoid creating a filter that is too extended along longitude or latitude. It is recommended to set the values of deltas close to each other.

The last two recommendations exist due to the spatial features of the filter. According to these features, when one or two deltas are set to large values, the result may differ from the expected though it will be correct. See the "Filter features" section for details.

Filter performance#

Geo position filter performance depends on the spatial data type used to store event geo position in the database.

Two spatial data types are supported:

• GEOMETRY. Spatial object with coordinates expressed as (longitude, latitude) pairs, defined in the Cartesian plane. All calculations use Cartesian coordinates.
• GEOGRAPHY. Spatial object with coordinates expressed as (longitude, latitude) pairs, defined as on the surface of a perfect sphere, or a spatial object in the WGS84 coordinate system.

For a detailed description, see geometry vs geography.

Geo position filter is based on the ST_Covers PostGIS function supported for both Geometry and Geography type.

Filter features#

Geo position filter has some features caused by PostGIS.

When geography type is used and the geo position filter covers a wide portion of the planet surface, filter result may be unexpected but geographically correct due to some spatial features.

The following example illustrates this case.

An event with the following geo position was added in the database:

{
    "longitude": 16.79,
    "latitude": 64.92,
}

We apply a geo position filter and try to find the required point on the map. The filter is too extended along the longitude:

{
    "origin_longitude": 16.79,
    "origin_latitude": 64.92,
    "longitude_delta": 2,
    "latitude_delta": 0.01,
}

This filter will not return the expected event. The event will be filtered due to spatial features. Here is the illustration showing that the point is outside the filter.

You should consider this feature to create a correct filter.

For details, see Geography.

Events creation#

Events are created using handlers. Handlers are stored in the Handlers database. You should specify the required handler ID in the event creation request. All the data stored in the event will be received according to the handler parameters.

You should perform two separate requests for event creation.

The first request creates a handler. Handler includes policies that describes how the image is processed hence defining the LP services used for the processing.

The second request creates new events using the existing handler. An event is created for each image that has been processed.

You can specify the following additional data for each event creation request:

• external ID (for created faces)
• user data (for created faces)
• source (for created events)
• tags (for created events)

The handler is processed policy after policy. All the data from the request is processed by a policy before going to the next policy. The "detect" policy is performed for all the images from the request, then "multiface" policy is applied, then the "extract" policy is performed for all the received samples, etc. For more information about handlers, see the "Handlers description" section.

General Events#

If it is necessary to save an event with any custom structure generated by video analytics, a plugin, or a client application, the general events mechanism should be used. The size of a general event must not exceed the 2 MB limit.

Since general events have a flexible structure, they must be categorized by type (the "event_type" parameter), which should be specified when saving the event.

When saving a general event, some general information (e.g., the event's creation time, the event's end time, and the account associated with the event) should also be provided.

Although the structure of a general event is flexible, it may include specific fields relevant to LUNA PLATFORM (e.g., location, stream ID, track ID). Searching through these fields can be performed more efficiently thanks to storage optimizations in the Events database.

The user is provided with the following capabilities:

• The ability to save events with a flexible structure.
• The ability to retrieve saved events.
• The ability to filter by the general information of a general event (e.g., event_type, event_create_time, event_end_time, account_id).
• The ability to filter by optional fields specific to LUNA PLATFORM (e.g., location, stream_id, track_id).
• The ability to filter by fields in the custom structure content.

The user is required to:

• Maintain data according to the specified schemas. If the data does not comply, PostgreSQL will not allow inserting a row with a type that cannot be added to an existing index (if one exists).
• Migrate data when necessary.
• Build partial indexes according to event types.
• Specify data types when performing a query (by default, all values are assumed to be strings).
• Pay attention to field names in the custom structure. Fields used for filtering must not contain reserved keywords such as :int, double underscores, special characters, etc.

For detailed information on building indexes and addressing various issues with general events, see the Events service developer manual.

Method for saving general events#

A general event can be saved via the "create new general events" request to the API service.

Accordingly, if the user writes their own plugin or application, they must send data to the endpoint described above.

General events search#

The following requests are used to search for events:

• "get general events"
• "get general event"

Filters can be passed to these requests. There are several conventions for specifying filters:

• To navigate nested objects, use a dot (.), e.g., event.user_info.temperature.
• To specify a comparison operator, use a double underscore suffix (__eq, __like, __in, __gt, __lt, etc.), e.g., event.user_info.temperature__gte.
• To specify a data type, use a colon suffix (:string, :integer, :numeric), e.g., event.user_info.temperature__gte:numeric.

Statistics on general events can also be obtained using the "get statistics on general events" request.

General events matching#

The functionality of matching by general events is implemented through a request general events matching.

This functionality allows you to match specified references - general events or descriptors in binary format - with candidates represented as general events.
The results of the matching are returned as a list of matches for the candidates according to the specified filters, or an empty list if no matches are found.

Events meta-information#

If any additional data needs to be stored along with the event, the "meta" field should be used. The "meta" field stores data in the JSON format. The total size of the data stored in the "meta" field for one event cannot exceed 2 MB. It is assumed that with the help of this functionality, the user will create his own data model (event structure) and will use it to store the necessary data.

Note that you cannot specify field names with spaces in the "meta" field.

Data in the "meta" field can be set in the following ways:

• In the "generate events" request body with the content type application/json or multipart/form-data.
• In the "save event" request body.
• Using a custom plugin or client application.

In the "generate events" request body, it is possible to set the "meta" field both for specific images and for all images at once (mutual meta-information). For requests with aggregation enabled, only mutual meta-information will be used for the aggregated event, and meta-information for specific images will be ignored. See the detailed information in the "generate events" request body in the OpenAPI specification.

Example of recording the "meta" field:

{
    "meta": {
        "user_info": {
            "temperature": 36.6
        }
    }
}

In order to store multiple structures, it is necessary to explicitly separate them to avoid overlapping fields. For example, as follows:

{
    "struct1": {
        ...
    },
    "struct2": {
        ...
    }
}

Search by "meta" field#

You can get the contents of the "meta" field using the appropriate filter in the "get events" request.

The filter should be entered using a specific syntax — meta.<path.to.field>__<operator>:<type>, where:

• meta. — An indication that the "meta" field of the Events database is being accessed.
• <path.to.field> — Path to the object. A dot (.) is used to navigate nested objects. For example, in the string {"user_info":{"temperature":"36.6"}} to refer to the temperature object, use the following filter meta.user_info.temperature.
• __<operator> — One of the following operators — eq (default), neq, like, nlike, in, nin, gt, gte, lt, lte. For example, meta.user_info.temperature__gte;
• :type — one of the following data types — string, integer, numeric. For example, meta.user_info.temperature__gte:numeric.

For each operator, the use of certain data types is available. See the table of operator dependency on data types in the OpenAPI specification.

If necessary, you can build an index to improve the search. See the Events developer manual for details on building an index.

Important notes#

When working with the "meta" field, remember the following:

• you need to keep data consistent with given schemes; in case of a mismatch, PostgreSQL will not allow inserting a row with a type that cannot be added to the existing index (if any);
• if necessary, you can migrate data;
• if necessary, you can build an index;
• specify the data type when performing a request (by default, all values are assumed to be strings);
• you need to pay attention to the names of the fields; fields to be filtered by must not contain reserved keywords like :int, double underscores, special symbols, and so on.

Saving deleted events#

There is the ability to save deleted events (regular and general) to the Events database.

Deleted events are stored in the Events database in the deleted_event and deleted_general_event tables. See the "Events eatabase description" section.

To enable this functionality, you must set the environment variable EVENT_DELETION_LOG=1 in the Events service launch command. For example:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5040 \
--env=EVENT_DELETION_LOG=1 \
--name=luna-events \
...

After starting the service, the IDs of deleted events will be written to the corresponding database tables.

You can get deleted events using the following requests:

• "get deleted events"
• "get deleted general events"

Sender service#

The Sender service is an additional service that is used to send events via web sockets. This service communicates with the Handlers service (in which events are created) through the pub/sub mechanism via the Redis DB channel.

If necessary, you can send notifications over the HTTP protocol. See the "Send events to third-party service" section for more details.

Events are created based on handlers. To receive notifications, the "callbacks" policy with "luna-ws-notification" must be enabled. This policy has filters that enable you to send notifications only under certain conditions, for example, to send only if the candidate is very similar to the reference (the "similarity__lte" parameter).

In previous versions of LUNA PLATFORM, the "notification_policy" was used. It is now considered deprecated and is not recommended for use. The main advantage of the callback mechanism over the deprecated "notification_policy" is the ability to specify multiple callbacks with different filters in the handler creation request, resulting in only one event being sent.

You should configure web sockets connection using special request. It is recommended create web sockets connection using the "/ws" resource of the API service. You can specify filters (query parameters) in the request, i.e. you can configure the Sender service to receive only certain events. See OpenAPI specification for detailed information about the configuration of creating a connection to a web socket.

Configuring web sockets directly via Sender is also available (see "/ws" of the Sender service). It can be used to reduce the load on the API service.

When an event is created it can be:

• Saved to the Events database. The Events service should be enabled to save an event.

• Returned in the response without saving to the database.

In both cases, the event is sent via the Redis DB channel to the Sender service.

In this case, the Redis DB acts as a connection between Sender and Handlers services and does not store transferred events.

The Sender service is independent of the Events service. Events can be sent to Sender even if the Events service is disabled.

Creating handlers and specifying filters for sending notifications

1․ The user sends the "create handler" request to the API service, where it enables the "callbacks" and sets filters according to which events will be sent to the Sender service.

2․ The API service sends a request to the Handlers service.

3․ The Handlers service sends a response to the API service.

4․ The API service sends the "handler_id" to the user.

The user saves the ID "handler_id", which is necessary for creating events.

Activation of subscription to events and filtering of their sending

1․ The user or application sends a request "ws handshake" to the API service and sets filters through which it will be possible to filter the received data from the Handlers service.

2․ The API service sends a request to the Sender service.

3․ The Sender service establishes a connection via web sockets with the user application.

Now, when an event is generated, it will be automatically redirected to the Sender service (see below) in accordance with the specified filters.

Event generation and sending to Sender

The general workflow is as follows:

1․ A user or an application sends the "generate events" request to the API service.

2․ The API service sends the request to the Handlers service.

3․ The Handlers service sends requests to the corresponding LP services.

4․ LP services process the requests and send results. New events are created.

5․ The Handlers service sends an event to the Redis database using the pub/sub model. Redis has a channel to which the Sender service is subscribed, and it is waiting for messages to be received from this channel.

6․ Redis sends the received events to Sender by the channel.

7․ Third-party party applications should be subscribed to the Sender service via web sockets to receive events. If there is a subscribed third-party party application, Sender sends events to it according to the specified filters.

See the OpenAPI documentation for information about the JSON structure returned by the Sender service.

Tasks service#

The Tasks service is used for long tasks processing.

General information about tasks#

As tasks processing takes time, the task ID is returned in the response to the task creation.

After the task processing is finished, you can receive the task results using the "task " > "get task result" request. You should specify the task ID to receive its results.

You can find the examples of tasks processing results in the response section of "task " > "get task result" request. You should select the task type in the Response samples section of documentation.

You should make sure that the task was finished before requesting its results:

• You can check the task status by specifying the task ID in the "tasks" > "get task" request. There are the following task statuses:

• You can receive information about all the tasks using the "tasks" > "get tasks" request. You can set filter to receive information about tasks of interest only.

Clustering task#

As the result of the task a cluster with objects selected according to the specified filters for faces or events is created. Objects corresponding to all of the filters will be added to the cluster. Available filters depend on the object type: events or faces.

You can receive the task status or result using additional requests (see the "General information about tasks").

You can use the reporter task to receive the report about objects added to clusters.

Clustering is performed in several steps:

• Objects with descriptors are collected according to provided filters

• Every object is matched with all the other objects

• Create clusters as groups of "connected components" from the similarity graph.

Here "connected" means that similarity is greater than provided threshold or default "DEFAULT_CLUSTERING_THRESHOLD" from the config.

• If needed, download existing images corresponding to each object: avatar for a face, first sample for an event.

As a result of the task an array of clusters is returned. A cluster includes IDs of objects (faces or events) whose similarity is greater then the specified threshold. You can use the information for further data analysis.

{
    "errors": [],
    "result": {
        "clusters": [
            [
                "6c721b90-f5a0-409a-ab70-bc339a70184c"
            ],
            [
                "8bc6e8df-410b-4065-b592-abc5f0432a1c"
            ],
            [
                "e4e3fc66-53b4-448c-9c88-f430c00cb7ea"
            ],
            [
                "02a3a1c4-93d7-4b69-99ec-21d5ef23852e",
                "144244cb-e10e-478c-bdac-18cd2eb27ee6",
                "1f4cdbcb-7b1e-40cc-873b-3ff7fa6a6cf0"
            ]
        ],
        "total_objects": 6,
        "total_clusters": 4
    }
}

The clustering task result can also include information about errors occurred during the objects processing.

For such a task, you can create a schedule.

Reporter task#

As a result of the task, the report on the clustering task is created. You can select data that should be added to the report. The report has CSV format.

You can receive the task status or result using additional requests (see the "General information about tasks").

You should specify the clustering task ID and the columns that should be added to the report. The selected columns correspond to the general events and faces fields.

Make sure that the selected columns correspond to the objects selected in the clustering task.

You can also receive the images for all the objects in clusters if they are available.

Exporter task#

The task enables you to collect event and/or face data and export them from LP to a CSV file. The file rows represent requested objects and corresponding samples (if they were requested).

This task uses memory when collecting data. So, its possible that Tasks Worker will be killed by OOM (Out-Of-Memory) killer if you request a lot of data.

You can export event or face data using the "/tasks/exporter" request. You should specify what type of object is required by setting objects_type parameter when creating a request. You can also narrow your request by providing filters for faces and events objects. See the "exporter task" request in the API service reference manual.

As a result of the task a zip archive containing a CSV file is returned.

You can receive the task status or result using additional requests (see the "General information about tasks").

When executing the Exporter task with a large number of faces in the Faces database (for example, 90,000,000 faces), the execution time of requests to the Faces service can be significantly increased. To speed up request execution, you can set the PostgreSQL setting "parallel_setup_cost" to 500. However, be aware that changing this setting may have other consequences, so you should be careful when changing the setting.

For such a task, you can create a schedule.

Cross-matching task#

When the task is performed, all the references are matched with all the candidates. References and candidates are set using filters for faces and events.

Matching is performed only for objects that contain extracted descriptors.

You can specify the maximum number of matching candidates returned for every match using the limit field.

You can set a threshold to specify the minimal acceptable value of similarity. If the similarity of two descriptors is lower then the specified value, the matching result will be ignored and not returned in the response. References without matches with any candidates are also ignored.

Cross-matching is performed in several steps:

• collect objects having descriptors using provided filters
• match every reference object with every candidate object
• match results are sorted (lexicographically) and cropped (limit and threshold are applied)

You can receive the task status or results using additional requests (see the "General information about tasks").

As a result an array is returned. Each element of the array includes a reference and top similar candidates for it. Information about errors occurred during the task execution is also returned in the response.

{
"result": [
    {
        "reference_id": "e99d42df-6859-4ab7-98d4-dafd18f47f30",
        "candidates": [
            {
                "candidate_id": "93de0ea1-0d21-4b67-8f3f-d871c159b740",
                "similarity": 0.548252
            },
            {
                "candidate_id": "54860fc6-c726-4521-9c7f-3fa354983e02",
                "similarity": 0.62344
            }
        ]
    },
    {
        "reference_id": "345af6e3-625b-4f09-a54c-3be4c834780d",
        "candidates": [
            {
                "candidate_id": "6ade1494-1138-49ac-bfd3-29e9f5027240",
                "similarity": 0.7123213
            },
            {
                "candidate_id": "e0e3c474-9099-4fad-ac61-d892cd6688bf",
                "similarity": 0.9543
            }
        ]
    }
],
"errors": [
    {
        "error_id": 10,
        "task_id": 123,
        "subtask_id": 5,
        "error_code": 0,
        "description": "Faces not found",
        "detail": "One or more faces not found, including face with id '8f4f0070-c464-460b-bf78-fac225df72e9'",
        "additional_info": "8f4f0070-c464-460b-bf78-fac225df72e9",
        "error_time": "2018-08-11T09:11:41.674Z"
    }
]
}

For such a task, you can create a schedule.

Linker task#

The task enables you to attach faces to lists according to the specified filters.

You can specify creation of a new list or specify the already existing list in the requests.

You can specify filters for faces or events to perform the task. When an event is specified for linking to list a new face is created based on the event.

If the create_time_lt filter is not specified, it will be set to the current time.

As the result of the task you receive IDs of faces linked to the list.

You can receive the task status or result using additional requests (see the "General information about tasks").

Task execution process for faces:

• A list is created (if create_list parameter is set to 1) or the specified list_id existence is checked.
• Face ID boundaries are received. Then one or several subtasks are created with about 1000 face ids per each. The number depends on face ID spreading.

• For each subtask:

  • Face IDs are received. They are specified for the current subtask by filters in the subtask content.
  • The request is sent to the Luna Faces to link specified faces to the specified list.
  • The result for each subtask is saved to the Image Store service.

• After the last subtask is finished, the worker collects results of all the subtasks, merges them and puts them to the Image Store service (as task result).

Task execution process for events:

• A list is created (if create_list parameter is set to 1) or the specified list_id existence is checked.
• Events page numbers are received. Then one or several subtasks are created.

• For each subtask:

  • Event with their descriptors are received from the Events service.
  • Faces are created using the Faces service. Attribute(s) and sample(s) are added to the faces.
  • The request is sent to the Luna Faces to link specified faces to the specified list.
  • The result for each subtask is saved to the Image Store service.

• After the last subtask is finished, the worker collects results of all the subtasks, merges them and puts them to the Image Store service (as task result).

For such a task, you can create a schedule.

Garbage collection task#

During the task processing, faces, events, general events or descriptors can be deleted.

• When descriptors are set as a GC target, you should specify the descriptor version. All the descriptors of the specified version will be deleted.
• When events are set as a GC target, you should specify one or several of the following parameters:
  • Account ID.
  • Upper excluded boundary of event creation time.
  • Upper excluded boundary of the event appearance in the video stream.
  • Handler ID used for the event creation.
• When faces are set as a GC target, you should specify one or several of the following parameters:
  • Upper excluded boundary of face creation time.
  • Lower included boundary of face creation time.
  • User data.
  • List ID.

If necessary, you can delete samples along with faces or events. You can also delete image origins for events.

Using the "tasks" > "garbage collection task" request of API service, you can specify events (events value), general events (general_events value), or faces (faces value) as values for the target field. In contrast, in the Admin or Tasks services, you can set faces, events, general events, and descriptors (descriptors value) as the target. In this case, the specified objects will be deleted for all existing accounts.

You can receive the task status or result using additional requests (see the "General information about tasks").

For such a task, you can create a schedule.

Additional extraction task#

The Additional extraction task re-extracts descriptors extracted using the previous neural network model using a new version of the neural network. This enables you to save previously used descriptors when updating the neural network model. If there is no need to use the old descriptors, then you can not perform this task and only update the neural network model in the Configurator settings.

This section describes how to work with the Additional extraction task. See detailed information about neural networks, the process of updating a neural network to a new model and relevant examples in the "Neural networks" section.

Re-extraction can be performed for face and event objects. You can re-extract the descriptors of faces, descriptors of bodies (for events) or basic attributes if they were not extracted earlier.

The samples for descriptors should be stored for the task execution. If any descriptors do not have source samples, they cannot be updated to a new NN version.

The re-extraction tasks are used for the update to a new neural network for descriptors extraction. All the descriptors of the previous version will be re-extracted using a new NN.

It is highly recommended not to perform any requests changing the state of databases during the descriptor version updates. It can lead to data loss.

Create backups of LP databases and the Image Store storage before launching the additional extraction task.

When processing the task, a new neural network descriptor is extracted for each object (face or event) whose descriptor version matches the version specified in the "DEFAULT_FACE_DESCRIPTOR_VERSION" (for faces) or "DEFAULT_HUMAN_DESCRIPTOR_VERSION" (for bodies) settings. Descriptors whose version does not match the version specified in these settings are not re-extracted. They can be removed using the Garbage collection task.

Request to the Admin service

You need to make a request to the "additional_extract" resource, specifying the following parameters in the request body:

• "content" > "extraction_target" – Face descriptors, body descriptors, basic attributes.
• "content" > "options" > "descriptor_version" – New neural network version (not applicable for basic attributes).
• "content" > "filters" > "object_type" – Faces or events.

If necessary, you can additionally filter the object type by "account_id", "face_id__lt", etc.

See the "create additional extract task" request in the Admin service OpenAPI specification for more information.

You can receive the task status or result using additional requests (see the "General information about tasks").

Admin user interface

You need to do the following:

• Go to the Admin user interface: http://<admin_server_ip>:5010/tasks.

• Run the additional extraction task using the corresponding button.

• In the window that appears, set the object type (face or event), the extraction type (face descriptor, body descriptor or basic attributes), new neural network model (not applicable for basic attributes) and click "Start", confirming the start of the task.

If necessary, you can additionally filter the object type by "account_id".

See the detailed information about the Admin user interface in the "Admin user interface" section.

For such a task, you can create a schedule.

ROC-curve calculating task#

As a result of the task, the Receiver Operating Characteristic curve with TPR (True Positive Rate) against the FPR (False Positive Rate) is created.

See additional information about ROC-curve creation in "TasksDevelopmentManual".

ROC calculation task

ROC (or Receiver Operating Characteristic) is a performance measurement for classification tasks at various thresholds settings. The ROC-curve is plotted with TPR (True Positive Rate) against the FPR (False Positive Rate). TPR is a true positive match pair count divided by a count of total expected positive match pairs, and FPR is a false positive match pair count divided by a count of total expected negative match pairs. Each point (FPR, TPR) of the ROC-cure corresponds to a certain similarity threshold. See more at wiki.

Using ROC the model performance is determined by looking at:

• Area under the ROC-curve (or AUC).
• Type I and type II error rates equal point, i.e. the ROC-curve and the secondary main diagonal intersection point.

The model performance also determined by hit into the top-N probability, i.e. probability of hit a positive match pair into the top-N for any match result group sorted by similarity.

It requires "markup" to make a ROC task. One can optionally specify "threshold_hit_top" (default 0) to calculate hit into the top-N probability, the match "limit" (default 5), "key_FPRs" — list of key FPR values to calculate ROC-curve key points, and "filters" with "account_id". Also, it needs "account_id" for task creation.

You can receive the task status or result using additional requests (see the "General information about tasks").

Markup

Markup is expected in the following format:

[{'face_id': <face_id>, 'label': <label>}]

Label (or group id) can be a number or any string.

Example:

[{'face_id': '94ae2c69-277a-4e46-817d-543f7d3446e2', 'label': 0},
 {'face_id': 'cd6b52be-cdc1-40a8-938b-a97a1f77d196', 'label': 1},
 {'face_id': 'cb9bda07-8e95-4d71-98ee-5905a36ec74a', 'label': 2},
 {'face_id': '4e5e32bb-113d-4c22-ac7f-8f6b48736378', 'label': 3},
 {'face_id': 'c43c0c0f-1368-41c0-b51c-f78a96672900', 'label': 2}]

For such a task, you can create a schedule.

Estimator task#

The estimator task enables you to perform batch processing of images using the specified policies.

As a result of the task performing, JSON is returned with data for each of the processed images and information about the errors that have occurred.

In the request body, you can specify the handler_id of an already existing static or dynamic handler. For the dynamic handler_id, the ability to set the required policies is available. In addition, you can create a static handler specifying policies in the request.

The resource can accept five types of sources with images for processing:

• ZIP archive
• S3-like storage
• Network disk
• FTP server
• Samba network file system

To obtain correct results of image processing using the Estimator task, all processed images should be either in the source format or in the format of samples. The type of transferred images is specified in the request in the "image_type" parameter.

For such a task, you can create a schedule. When creating a schedule, it is not possible to specify a ZIP archive as an image source.

ZIP archive as image source of estimator task

The resource accepts for processing a link to a ZIP archive with images. The size of the archive is set using the "ARCHIVE_MAX_SIZE" parameter in the "config.py" configuration file of the Tasks service. The default size is 100 GB. An external URL or the URL to an archive saved in the Image Store can be used as a link to the archive. In the second case, the archive should first be saved to the LP using a POST request to the "/objects" resource.

When using an external URL, the ZIP archive is first downloaded to the Tasks Worker container storage, where the images are unpacked and processed. After the end of the task, the archive is deleted from the repository along with the unpacked images.

It is necessary to take into account the availability of free space for the above actions.

The archive can be password protected. The password can be passed in the request using the "authorization" -> "password" parameter.

S3-like storage as image source of estimator task

The following parameters can be set for this type of source:

• "bucket_name" — Bucket name/Access Point ARN/Outpost ARN (required).
• "endpoint" — Storage endpoint (only when specifying the bucket name).
• "region" — Bucket region (only when specifying the bucket name).
• "prefix" — File key prefix. It can also be used to load images from a specific folder, such as "2022/January".

The following parameters are used to configure authorization:

• Public access key (required)
• Secret access key (required)
• Authorization signature version ("s3v2"/"s3v4")

It is also possible to recursively download images from nested bucket folders and save original images.

For more information about working with S3-like repositories, see AWS User Guide.

Network disk as image source of estimator task

The following parameters can be set for this type of source:

• "path" — Absolute path to the directory with images in the container (required).
• "follow_links" — Enables/disables symbolic link processing.
• "prefix" — File key prefix.
• "postfix" — File key postfix.

See an example of using prefixes and postfixes in the "/tasks/estimator" resource description.

When using a network disk as an image source and launching Tasks and Tasks Worker services through Docker containers, it is necessary to mount the directory with images from the network disk to the local directory and synchronize it with the specified directory in the container. You can mount a directory from a network disk in any convenient way. After that, you can synchronize the mounted directory with the directory in the container using the following command when launching the Tasks and Tasks Worker services:

docker run \
...
-v /var/lib/luna/current/images:/srv/images
...

Here:

• /var/lib/luna/current/images — Path to the previously mounted directory with images from the network disk.
• /srv/images- Path to the directory with the images in the container where they will be moved from the network disk. This path should be specified in the request body of the Estimator task (the "path" parameter).

As for S3-like storage, the ability to recursively download images from nested bucket folders is available.

FTP server as image source of estimator task

For this type of source, the following parameters can be set in the request body for connecting to the FTP server:

• "host" — FTP server IP address or hostname (required).
• "port" — FTP server port.
• "max_sessions" — Maximum number of allowed sessions on the FTP server.
• "user", "password" — Authorization parameters (required).

As in Estimator tasks using S3-like storage or network disk as image sources, it is possible to set the path to the directory with images, recursively receive images from nested directories, select the type of transferred images, and specify the prefix and postfix.

See an example of using prefixes and postfixes in the "/tasks/estimator" resource description.

Samba as image source of estimator task

For this type of source, the parameters are similar to those of an FTP server, except for the "max_sessions" parameter. Also, if authorization data is not specified, the connection to Samba will be performed as a guest.

Task processing#

The Tasks service includes the Tasks service and Tasks workers. Tasks receives requests to the Tasks service, creates tasks in the DB and sends subtasks to Tasks workers. The workers are implemented as a separate Tasks Worker container. Tasks workers receive subtasks and perform all the required requests to other services to solve the subtasks.

The general approach for working with tasks is listed below.

• User sends the request for creation of a new task.
• Tasks service creates a new task and sends subtasks to workers.
• Tasks workers process subtasks and create reports.
• If several workers have processed subtasks and have created several reports, the worker, which finished the last subtask, gathers all the reports and creates a single report.
• When the task is finished, the last worker updated its status in the Tasks database.
• User can send requests to receive information about tasks and subtasks and number of active tasks. The user can cancel or delete tasks.
• User can receive information about errors that occurred during execution of the tasks.
• After the task is finished the user can send a request to receive results of the task.

See the "Tasks diagrams" section for details about tasks processing.

Running scheduled tasks#

In LUNA PLATFORM, it is possible to set a schedule for Garbage collection, Clusterization, Exporter, Linker, Estimator, Additional extract, Cross-matching and Roc-curve calculating tasks.

To use a filter relative to the current time ("now-time"), the current time will be counted not from the creation of the schedule, but from the creation of the task by the schedule in accordance with the cron expression. See "Now-time filters" for details.

The schedule is created using the request "create tasks schedule" to the API service, which specifies the contents of the task being created and the time interval for its launch. To specify the time interval, Cron expressions are used.

Cron expressions are used to determine the task execution schedule. They consist of five fields separated by spaces. Each field defines a specific time interval in which the task should be completed. The week number starts from Sunday.

For tasks that can only be performed using the Admin service (for example, the task of removing some objects using the GC task), you can assign a schedule only in the Admin service.

In response to the request, a "schedule_id" is issued, which can be used to get information about the status of the task, the time of the next task, etc. (requests "get tasks schedule and "get tasks schedules). The id and all additional information are stored in the "schedule" table of the Tasks database.

If necessary, you can create a delayed schedule, and then activate it using the "action" = "start" parameter of the "patch tasks schedule" request. Similarly, you can stop the scheduled task using "action" = "stop". To delete a schedule, you can use the "delete tasks schedule" request.

Permissions to work with schedules are specified in the token with the "task" permission. This means that if the user has permission to work with tasks, then he will also be able to use the schedule.

The possibility of schedule creation is also available for Lambda tasks.

Examples of Cron expressions#

This section describes various examples of Cron expressions.

1․ Run the task every day at 3 a.m.:

0 3 * * *

2․ Run the task every Friday at 18:30:

30 18 * * 5

3․ Run the task every first day of the month at noon:

0 12 1 * *

4․ Run the task every 15 minutes:

*/15 * * * *

5․ Run the task every morning at 8:00, except weekends (Saturday and Sunday):

0 8 * * 1-5

6․ Run the task at 9:00 am on the first and 15th day of each month, but only if it is Monday:

0 9 1,15 * 1

Send notification about task and subtask status changes#

If necessary, you can send notifications about changes in task and subtask status using the callback mechanism. Callbacks allow you to send data to a third-party system at a specified URL or to Telegram. To configure notifications, you need to configure the "notification_policy" in the request parameters of the corresponding task.

You can also configure sending notifications for tasks and subtask in the schedule settings.

If necessary, you can obtain information about the current state of the notification policy or change some policy data using "get task notification policy" and "replace task notification policy" requests.

Additional protection for passwords and tokens#

Passwords and tokens passed in Estimator task and in "notification_policy" can be additionally encrypted. To do this, pass custom values to the FERNET_PASSPHRASE and SALT environment variables when starting the Tasks service container.

FERNET_PASSPHRASE is the password or key used to encrypt data using the Fernet algorithm.

SALT is a random string added to the password before it is hashed.

Fernet is a symmetric encryption algorithm that provides authentication and data integrity as well as confidentiality. With this algorithm, the same key is used to encrypt and decrypt data.

Salt is added to make it more difficult to crack the password by brute force. Each time a password is hashed, a unique string is used, making identical passwords hashed differently. This increases the security of the system, especially if users have the same passwords.

Example of a container startup command passing environment variables:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=FERNET_PASSPHRASE=security_passphrase.
--env=SALT=salt_for_passwords_and_tokens.
...

Important: When the container is started with the above environment variables specified, the old passwords and tokens will no longer work. Additional migration steps must be performed (see section below).

Add encryption when updating#

In order to add additional protection for already existing passwords and tokens, you need to specify environment variables in the Tasks database migration command (see above). After that, you should start a new Tasks container with environment variables specified to enable the use of encryption when creating new objects.

Admin service#

The Admin service is used to perform general administrative routines:

• Manage user accounts.
• Receive information about objects belonging to different accounts.
• Create garbage collection tasks.
• Create tasks to extract descriptors with a new neural network version.
• Receive reports and errors on processed tasks.
• Cancel and delete existing tasks.

Admin service has access to all the data attached to different accounts.

Three types of accounts can be created in the Admin service — "user", "advanced_user" and "admin". The first two types are created using an account creation request to the API service, but the third type can only be created using the Admin service.

Using the "admin" account type, you can log in to the interface and perform the above tasks. An account with the "admin" type can be created either in the user interface (see above) or by requesting the "/4/accounts" resource of the Admin service. To create an account in the last way, you need to specify a username and password.

If you are creating an account for the first time, you must use the default login and password.

Example of CURL request to the "/4/accounts" resource of the Admin service:

curl --location --request POST 'http://127.0.0.1:5010/4/accounts' \
--header 'Authorization: Basic cm9vdEB2aXNpb25sYWJzLmFpOnJvb3Q=' \
--header 'Content-Type: application/json' \
--data '{
  "login": "mylogin@gmail.com",
  "password": "password",
  "account_type": "admin",
  "description": "description"
}' 

All the requests for Admin service are described in Admin service reference manual.

Admin user interface#

The user interface of the Admin service is designed to simplify the work with administrative tasks.

The interface can be opened in a browser by specifying the address and port of the Admin service: <Admin_server_address>:<Admin_server_port>.

The default Admin service port is 5010.

The default login and password to access the interface are root@visionlabs.ai/root. You can also use default login and password in Base64 format — cm9vdEB2aXNpb25sYWJzLmFpOnJvb3Q=.

You can change the default password for the Admin service using the "Change authorization" request.

There are three tabs on the page:

• Accounts. The tab is designed to provide information about all created accounts and to create new accounts.
• Tasks — The tab is designed for working with Garbage collection and Additional extraction tasks.
• Info — The tab contains information about the user interface and the LUNA PLATFORM license.

Accounts tab#

This tab displays all existing accounts.

You can manage existing accounts using the following buttons:

– View account information.

– Delete account.

Clicking the view info button opens a page containing general information about the account, lists created with that account, and faces.

When you click the "Create account" button, an account creation window opens, containing the standard account creation settings — login, password, account type, description and the desired "account_id".

See "Account" for details on accounts and their types.

Tasks tab#

This tab displays running/completed Garbage collection and Additional extraction tasks.

.

Tasks are displayed in a table whose columns can be sorted and also filtered by the date the tasks were completed.

When you press the "Start Garbage collection" and "Start Additional extraction" buttons, windows for creating the corresponding tasks open.

The "Garbage collection" window contains the following settings, similar to the parameters of the "garbage collecting task" request body to the Tasks service:

• Description — "description" parameter
• Target — "content > target" parameter
• Account ID — "content > filters > account_id" parameter
• Remove sample — "content > remove_samples" parameter
• Remove image origins — "content > remove_image_origins" parameter
• Delete data before — "content > create_time__lt" parameter

See "Garbage collection task" for details.

The "Additional extraction" window contains the following settings, similar to the parameters of the "additional extract task" request body to the Tasks service:

• Objects type — "content > filters > object_type" parameter
• Extraction type — "content > extraction_target" parameter
• Descriptor version — "content > options > descriptor_version" parameter
• Description — "description" parameter
• Account ID — "content > filters > account_id" parameter

See "Additional extraction task" for details.

After creating a task, its execution begins. The progress of the task is displayed by the icon . The task is considered completed when the "Parts done" value matches the "Parts total" value and the icon changes to . If necessary, you can stop the task execution using the icon .

The following buttons are available for each task:

– download the task result as a JSON file.

– go to the page with a detailed description of the task and errors received during its execution.

– delete task.

Tasks are executed by the Tasks service after receiving a request from the Admin service.

Schedules tab#

This tab is intended for working with task scheduling.

The tab displays all created task schedules and all relevant information (status, ID, Cron string, etc.).

When you click on the "Create schedule" button, the schedule creation window opens.

In the window you can specify schedule settings for the Garbage collection task. The parameters in this window correspond to the parameters of the "create tasks schedule" request.

After filling in the parameters and clicking the "Create schedule" button, the schedule will appear in the Schedules tab.

You can control delayed start using the following buttons:

– start the schedule.

– pause the schedule.

Using the button, you can edit the schedule. Using the button you can delete a schedule.

Info tab#

This tab displays complete license information and features that can be performed using the Admin UI.

.

See the detailed license description in the "License information" section.

By clicking on the "Download system info" button, you can also get the following technical information about the LP:

• LUNA PLATFORM version
• Versions of the services
• Number of descriptors
• Configuration files values
• License information
• Requests and estimations statistics (see section "Requests and estimations statistics gathering")

You can also get the above system information using the "get system info" request to the Admin service.

Configurator service#

The Configurator service simplifies the configuration of LP services.

The service stores all the required configurations for all the LP services in a single place. You can edit configurations through the user interface or special limitation files.

You can also store configurations for any third-party party software in Configurator.

The general workflow is as follows:

• User edits configurations in the UI.
• Configurator stores all changed configurations and other data in the database.
• LP services request Configurator service during startup and receive all required configurations. All the services should be configured to use the Configurator service.

During Configurator installation, you can also use your limitation file with all the required fields to create limitations and fill in the Configurator database. You can find more details about this process in the Configurator development manual documentation.

Settings used by several services are updated for each of the services. For example, if you edit the "LUNA_FACES_ADDRESS" section for the Handlers service in the Configurator user interface, the setting will be also updated for API, Admin and Python Matcher services.

Configurator UI#

Open the Configurator interface in your browser: <Configurator_server_address> :5070

This URL may differ. In this example, the Configurator service interface is opened on the Configurator service server.

LP includes the beta version of the Configurator UI. The UI was tested on Chrome and Yandex browser. The recommended screen resolution for working with the UI is 1920 x 1080.

The following tabs are available in the UI of Configurator:

• Settings. All the data in the Configurator service is stored on the Settings tab. The tab displays all the existing settings. It also allows to manage and filter them;
• Limitations. The tab is used to create new limitations for settings. The limitations are templates for JSON files that contain available data type and other rules for the definition of the parameters;
• Groups. The tab allows to group all the required settings. When you select a group on the Settings tab, only the settings corresponding to the group will be displayed. It is possible to get settings by filters and/or tags for a single specific service. For this purpose, the Groups tab is used.
• About. The tab includes information about the Configurator service interface.

Settings#

Each of the Configurator settings contain the following fields:

• "Name" — Name for the setting.
• "Description" — Setting description.
• "ID and Times" — Unique setting ID.
• "Create time" — Setting create time.
• "Last update time" — Setting last update time.
• "Value" — Body of the setting.
• "Schema" — Verification template for the schema body.
• "Tag" — Tags for the setting used to filter settings for the services.

The "Tags" field is not available for the default settings. You should press the Duplicate button and create a new setting on the basis of the existing one.

The following options for the settings are available:

• Create a new setting — press the Create new button, enter required values and press Create. You should also select an already existing limitation for the setting. The Configurator will try to check the value of a setting if the Check on save flag is enabled and there is a limitation selected for the setting;

• Duplicate existing setting — press the Duplicate button on the right side of the setting, change required values and press Create. The Configurator will try to check the setting value if the Check on save flag is enabled on the lower left side of the screen and there is such a possibility;

• Delete existing setting — press the Delete button on the right side of the setting.

• Update existing setting — change name, description, tags, value and press Save button on the right side of the setting.

• Filter existing settings by name, description, tags, service names, groups — use the filters on the left side of the screen and press Enter or click on Search button;

Show limitations. The flags are used to enable displaying of limitations for each of the settings.

JSON editors. The flag enables you to switch the mode of the value field representation. If the flag is disabled, the name of the parameter and a field for its value are displayed. If the flag is enabled, the Value field is displayed as a JSON.

The Filters section on the left side of the window enables you to display all the required settings according to the specified values. You may enter the required name manually or select it from the list:

• Setting. The filter enables you to display the setting with the specified name.
• Description. The filter enables you to display all settings with the specified description or part of description.
• Tags. The filter enables you to display all settings with the specified tag.
• Service filter. The filter enables you to display all settings that belong to the selected service.
• Group. The filter enables you to display all settings that belong to the specified group. For example, you can select to display all the services belonging to LP.

Use tagged settings#

Using tagged settings, you can run several identical services that will use different settings from the Configurator.

To do this, follow these steps:

1․ Duplicate or create a new setting by specifying a tag for it. For example, you can duplicate the "LUNA_EVENTS_DB" setting and assign it the "EVENTS_DB_TAG" tag.

2․ Pass the following arguments to the command "run.py" the corresponding container:

• --luna-config — flag containing the address of the Configurator service
• --<configuration_name> — flag containing the configuration and tag

See the "Service arguments" section of the installation guide for more information on the arguments.

For example, to configure "LUNA_EVENTS_DB" with the tag "EVENTS_DB_TAG", the container launch command will look like this:

docker run \
...
dockerhub.visionlabs.ru/luna/luna-events:v.4.23.3
python3 /srv/luna_events/run.py --luna-config http://127.0.0.1:5070/1 --LUNA_EVENTS_DB EVENTS_DB_TAG

Limitations#

Limitations are used as service settings validation schema.

Settings and limitations have the same names. A new setting is created upon limitation creation.

The limitations are set by default for each of the LP services. You cannot change them.

Each of the limitations includes the following fields:

• Name is the name of the limitation.
• Description is the description of the limitation.
• Service list is the list of services that can use settings of this limitation.
• Schema is the object with JSON schema to validate settings
• Default value is the default value created with the limitation.

The following actions are available for managing limitations:

• Create a new limitation — Press the Create new button, enter required values and press "Create". Also, the setting with default value will be created.
• Duplicate existing limitation — Press the Duplicate button on the right side of the limitation, change required values and press Create. Also, the setting with default value will be created.
• Update limitation values — Change name/description/service list/validation schema/default values and press the Save button on the right side of the limitation.
• Filter existing limitations by names, descriptions, and groups.
• Delete existing limitation — Press the Delete button on the right side of the limitation.

Groups#

Group has a name and a description.

It is possible to:

• Create a new group — Press the Create new button, enter the group name and optionally description and press Create.
• Filter existing groups by group names and/or limitation names — Use filters on the left side and press 'RETURN' or click on Search button.
• Update group description — Update the existing description and press the Save button on the right side of the group.
• Update linked limitation list — to unlink limitation, press "-" button on the right side of the limitation name, to link limitation, enter its name in the field at the bottom of the limitation list and press the "+" button. To accept changes, press the Save button.
• Delete group — Press the Delete button on the right side of the group.

Settings dump#

The dump file includes all the settings of all the LP services.

Receive settings dump#

You can fetch the existing service settings from the Configurator by creating a dump file. This may be useful for saving the current service settings.

To receive a dump file use the following options:

• wget: wget -O settings_dump.json 127.0.0.1:5070/1/dump
• curl: curl 127.0.0.1:5070/1/dump > settings_dump.json
• text editor

The current values, specified in the Configurator service, are received.

Apply settings dump#

To apply the saved settings, you need to run the Storages command to load the dump (load_dump) with the --dump-file argument: --dump-file=/srv/settings_dump.json.

You can apply full settings dump on an empty database only.

If the settings update is required, you should delete the whole "limitations" group from the dump file before applying it.

    "limitations":[
      ...
    ],

Run the following command to upload the dump file to the Configurator:

docker run \
--rm \
--network=host \
-v /var/lib/luna/current/extras/conf/storages_config.conf:/srv/storages_config.conf \
-v /var/lib/luna/current/extras/conf/platform_settings.json:/srv/platform_settings.json \
dockerhub.visionlabs.ru/luna/storages:v.0.4.50 \
bash -c "luna_prepare load_dump \
    --dump-file=/srv/platform_settings.json \
    --config=/srv/storages_config.conf"

Here:

• luna_prepare load_dump — Command "load_dump", which enables you to upload a dump file to the Configurator database.
• -v /var/lib/luna/current/extras/conf/platform_settings.json:/srv/platform_settings.json \ — Command to mount the dump file platform_settings.json.
• --dump-file=/srv/platform_settings.json — Named argument containing the address of the dump file inside the container.
• -v /var/lib/luna/current/extras/conf/storages_config.conf:/srv/storages_config.conf — Command to mount the Storages configuration file.
• --config=/srv/storages_config.conf — Named argument containing the address of the configuration file for use by the Storages service.

Limitations from the existing limitations files are replaced with limitations from the dump file, if limitations names are the same.

Limitations file#

Receive limitation file#

Limitation file includes limitations of the specified service. It does not include existing settings and their values.

To download a limitations file for one or more services, use the following commands:

1․ Enter the Configurator container.

2․ Create the output "base_scripts/results" directory: mkdir base_scripts/results

3․ Run the "base_scripts/get_limitation.py" script: python3 base_scripts/get_limitation.py --service luna-image-store luna-handlers --output base_scripts/results/my_limitations.json.

Note the "base_scripts/get_limitation.py" script parameters:

• --service for specifying one or more service names (required)
• --output for specifying the directory or a file where to save the output. The default value: "current_dir/{timestamp}_limitation.json" (optional)

Authorization#

The Configurator service has the option to use BasicAuth authorization for requests that logically require authorization.

Using authorization provides an additional degree of security for encryption and protection against unauthorized access.

To enable authorization, you must set the following settings from the new "LUNA_CONFIGURATOR_AUTHORIZATION" section of the Configurator service:

• "USE_AUTHORIZATION" - Enable/disable authorization.
• "LUNA_CONFIGURATOR_USER" - Login, which will be used by other services for authorization.
• "LUNA_CONFIGURATOR_PASS" - Password, which will be used by other services for authorization.

You can set the settings in two ways:

• Passing the settings through the VL_SETTINGS environment variable at service startup. For example:

env "VL_SETTINGS.LUNA_CONFIGURATOR.USE_AUTHORIZATION=1"
env "VL_SETTINGS.LUNA_CONFIGURATOR.LUNA_CONFIGURATOR_USER=luna".
env "VL_SETTINGS.LUNA_CONFIGURATOR.LUNA_CONFIGURATOR_PASS=root"

• Transfer of settings to the configuration file of the Configurator service. The Configurator service configuration file is located at the following path in the distribution package: example-docker/luna_configurator/configs/luna_configurator_postgres.conf

In order for the other LUNA PLATFORM services to be able to make requests to the Configurator service, a login and password must be passed to them. This can be done using the following methods:

• Passing the "LUNA_CONFIGURATOR_USER" and "LUNA_CONFIGURATOR" settings through the VL_SETTINGS environment variable at service startup (see example above).
• Transfer of "LUNA_CONFIGURATOR_USER" and "LUNA_CONFIGURATOR" settings to the service configuration file when the service is used.

Licenses service#

General information#

The Licenses service stores information about the available licensed features and their limits.

There are three ways to get license information:

• Using the "get license" request to the Licenses service.
• Using the "get system info" request to the Admin service.
• Using the user interface of the "Admin service".

You can also use the "get platform features" request to the API service, in the response to which you can get information about the license status, the license functions enabled ("face_quality", "body_attributes" and "liveness") and the status of optional services (Image Store, Events, Tasks and Sender) from the "ADDITIONAL_SERVICES_USAGE" configuration of the Configurator service.

If you disable some license feature and try to use a request that requires this function, error 33002 will be returned with the description "License problem Failed to get value of License feature {value}".

License information#

LP license includes the following features:

• License expiration date.
• Maximum number of faces with linked descriptors or basic attributes.
• OneShotLiveness estimation availability.
• OneShotLiveness current balance.
• Deepfake estimation availability.
• Image check according to ISO/IEC 19794-5:2011 standard availability.
• Body parameters estimation availability.
• People count estimation availability.
• Using Lambda service availability.
• Possibility of using the Index Matcher service in the LUNA Index Module.
• Maximum number of simultaneous streams that can be taken for processing at one time from FaceStream.
• Maximum number of simultaneous streams that can be taken for processing at one time from the Video Manager service.

When ordering the license, you need to inform technical support about the need to use any of the above features.

The features "Possibility of using the Index Matcher service in the LUNA Index Module" and "Maximum number of simultaneous streams that can be taken for processing at one time from FaceStream" are described in the LUNA Index Module and FaceStream documentation, respectively.

Notifications are available for some features when approaching the limit. Notifications work in the form of sending messages to the logs of the corresponding service. For example, when approaching the allowable number of created faces with descriptors, the following message will be displayed in the Faces service logs: "License limit exceeded: 8% of the available license limit is used. Please contact VisionLabs for license upgrade or delete redundant faces". Notifications work due to constant monitoring implemented using the Influx database. Monitoring data is stored in the corresponding fields of the Influx database.

See the detailed information in the section "Monitoring".

Expiration date#

When the license expires, you cannot use LUNA PLATFORM.

By default, the notification about the end of the license is sent two weeks before the expiration date.

When the license ends, the following message is returned "License has expired. Please contact VisionLabs for a license extension.".

The Licenses service writes data about the license expiration date to the logs and the Influx database in the "license_period_rest" field.

Faces limit#

The Faces service checks the number of faces left according to the maximum available number of faces received from the Licenses service. The faces with linked descriptors or basic attributes are counted only.

The percentage of the used limit for faces is written in the Faces log and displayed in the Admin GUI.

The Faces service writes data about the created faces to the logs and the Influx database in the "license_faces_limit_rate" field.

The created faces are written in the Faces log and displayed in the Admin GUI as a percentage of the database fullness. You should calculate the number of faces with descriptors left using the current percentage.

You start receiving notifications when there are 15% of available faces left. When you exceed the number of available faces, the message "License limit exceeded. Please contact VisionLabs for license upgrade or delete redundant faces" appears in logs. You cannot attach attributes to faces if the number of faces exceeds 110%.

Consequences of missing a feature

If this feature is disabled, it will be impossible to perform the following requests:

• "create face"
• "generate events" specifying a handler with "policies" > "storage_policy" > "face_policy" > "store_face"

OneShotLiveness#

An unlimited license or a license with a limited number of transactions is available to estimate Liveness using the OneShotLiveness estimator.

Each use of Liveness in requests reduces the transaction count. It is impossible to use the Liveness score in requests after the transaction limit is exhausted. Requests that do not use Liveness and requests where the Liveness estimation is disabled are not affected by the exhaustion of the limit. They continue to work as usual.

The Licenses service stores information about the liveness transactions left. The number of transactions left is returned in the response from the "/license" resource.

The Remote SDK service writes data on the number of available Liveness transactions to the logs and the Influx database in the "liveness_balance" field.

A warning about the exhaustion of the number of available transactions is sent to the monitoring and logs of the Remote SDK service when the remaining 2000 transactions of Liveness are reached (this threshold is set in the system).

See the "OneShotLiveness description" section for more information on how Liveness works.

Consequences of missing a feature

If this feature is disabled, it will be impossible to estimate Liveness (the "estimate_liveness" parameter) in the following requests:

• "sdk"
• "detect faces"
• "generate events" specifying a handler with "policies" > "detect_policy" > "estimate_liveness"
• "perform verification" specifying a handler with "policies" > "detect_policy" > "estimate_liveness"
• "estimator task" specifying a handler with "policies" > "detect_policy" > "estimate_liveness"
• "predict liveness"

Body parameters estimation#

This feature enables you to estimate body parameters. Two values can be set in the license — 0 or 1. Monitoring is not intended for this parameter.

Consequences of missing a feature

If this feature is disabled, it will be impossible to estimate the body parameters (parameters "estimate_upper_body", "estimate_lower_body", "estimate_body_basic_attributes", "estimate_accessories") in the following requests:

• "sdk"
• "generate events" specifying a handler with "policies" > "detect_policy" > "body_attributes"

People count estimation#

This feature enables you to estimate the number of people. Two values can be set in the license — 0 or 1. Monitoring is not intended for this parameter.

Consequences of missing a feature

If this feature is disabled, it will be impossible to estimate the number of people (the "estimate_people_count" parameter) in the following requests:

• "sdk"
• "generate events" specifying a handler with "policies" > "detect_policy" > "estimate_people_count"

Image check by ISO/IEC 19794-5:2011 standard#

This feature enables you to perform various image checks by ISO/IEC 19794-5:2011 standard. Two values can be set in the license — 0 or 1. Monitoring is not intended for this parameter.

Consequences of missing a feature

If this feature is disabled, it will be impossible to perform the following requests:

• "iso"
• "detect faces" with the parameter "estimate_face_quality"
• "generate events" specifying a handler with "policies" > "detect_policy" > "face_quality"
• "estimator task" specifying a handler with "policies" > "detect_policy" > "face_quality"
• "perform verification" specifying a handler with "policies" > "detect_policy" > "face_quality"

Video analytics services#

LUNA PLATFORM offers video analytics services that allow you to analyze the number of people or track a person in a video file or video stream.

There are two video analytics services available:

• Video Agent - a service that implements video analytics algorithms. The service contains all the necessary neural networks to perform video analysis. Without the Video Agent service, the basic functionality of video processing will not work.
• Video Manager - a service that implements the advanced functionality of streams by creating a separate entity stream containing all the information about the video file or video stream being processed and the video analytics that will need to be performed. The Video Manager service sends streams to the Video Agent service for processing. If there are several agents, the service independently determines the agent depending on its load.

Both services are enabled in the "ADDITIONAL_SERVICE_USAGE" setting and are disabled by default.

Video analysis can be performed using a resource "/videosdk" and resource group "/streams".

When performing analytics on the "/videosdk" resource, the Video Agent service can only process the video file and return the result in the response body.

When performing analytics in the "/streams" resource group, the Video Agent service interacts with the Video Manager service, which implements the streams functionality.

See basic information about people count and human tracking video analytics in "Video analytics".

See the section for more information on the functionality of streams "Stream functionality for video analysis".

Streams Retranslator service#

The Streams Retranslator service retransmits video streams, providing them in the HLS format, convenient for displaying in user interfaces. Retransmission is performed using the FFmpeg and MediaMTX utilities running in the service container.

Important: When starting the container, you can forward additional ports for the MediaMTX and HLS stream to provide access to them via non-standard ports (see the example of starting the service in the installation documentation).

Retransmission is started using the "get stream retransmission" request. The stream identifier must be specified in the request. Optionally, you can specify the retransmission quality using the "quality" parameter. The value of the parameter determines the frame height in pixels. For example, to retransmission a stream with a height of 720 pixels instead of the original 1080, set "quality=720". The width will be calculated automatically, preserving the original aspect ratio.

The following data is returned as a result of the request:

• url. Link to the HLS stream (for example, http://127.0.0.1:8888/84406ce5-db60-4be1-b741-781f7979ccc5/index.m3u8, where 127.0.0.1:8888 is the host and port, and 84406ce5-db60-4be1-b741-781f7979ccc5 is the reretransmission identifier).
• type. retransmission type (currently only HLS).
• quality. retransmission quality (in pixels).
• token. JWT token for accessing the HLS stream.

Note: The host and port returned in the response must be configured via the "EXTERNAL_LUNA_STREAMS_HLS_RETRANSMISSION_ADDRESS" parameter in order to use this data in the web interface. The default values ​​will not work if the service is not started in the same place where the HLS stream is consumed, or if a non-standard port for HLS was forwarded when the container was started (for example, 8899 was forwarded instead of 8888).

The retransmission data is stored in Redis and exists until the retransmission is completed.

The retransmission duration is controlled by the "luna_streams_retranslator_ignored_restream_ttl" parameter. This parameter specifies the time (60 seconds by default) during which the stream is considered in demand (for example, if there are viewers). If the stream is not requested within the specified time, the retransmission will be automatically terminated.

When the retransmission is stopped, a request to restart it will create a new link and token. If the request is received before the current retransmission is completed, the user will receive the same link to the retransmission and token, since it is already active.

Operating principle#

1․ The user sends a request to restream the RTSP stream, specifying the quality (optional).

2․ Service: - Starts the retransmission via FFmpeg. - Saves the link to the HLS and the token in Redis. - Returns the link and token to the user.

3․ The user embeds the link and token in the interface (example).

Scaling#

The Streams Retranslator service has certain scaling characteristics.

The correct way to use multiple instances of Streams Retranslator is to distribute requests among services using different ranges of stream_id for each instance.

Important: The address and port on which the load balancer operates must match the values of the parameters from the "EXTERNAL_LUNA_STREAMS_HLS_RETRANSMISSION_ADDRESS" group.

Each stream has a unique identifier stream_id, represented in the uuid4 format. This identifier is randomly generated when the stream is created, ensuring an even distribution.

Examples of dividing uuid4 into ranges:

• From 00000000-0000-4000-8000-000000000000 to 80000000-0000-4000-8000-000000000000

• From 90000000-0000-4000-8000-000000000000 to ffffffff-ffff-4fff-bfff-ffffffffffff

• From 00000000-0000-4000-8000-000000000000 to 30000000-0000-4000-8000-000000000000

• From 40000000-0000-4000-8000-000000000000 to 80000000-0000-4000-8000-000000000000
• From 90000000-0000-4000-8000-000000000000 to a0000000-0000-4000-8000-000000000000
• From c0000000-0000-4000-8000-000000000000 to ffffffff-ffff-4fff-bfff-ffffffffffff

By distributing requests according to the described principle, the load can be evenly balanced across services.

See examples in the "Scaling" section of the Streams Retranslator developer manual.

Example of HTML fragment for retransmission#

Below is an example that allows you to display the stream on a web page. Replace url and token with your values:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>HLS Stream</title>
    <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
</head>
<body>
    <h1>HLS Stream</h1>
    <video id="video" controls></video>
    <script>
        const video = document.getElementById('video');
        const url = "http://127.0.0.1:8888/84406ce5-db60-4be1-b741-781f7979ccc5/index.m3u8";
        const token = "eyJhbGciOiJ";

        if (Hls.isSupported()) {
            const hls = new Hls();
            hls.attachMedia(video);
            hls.loadSource(url);
            hls.on(Hls.Events.MEDIA_ATTACHED, () => {
                hls.config.xhrSetup = xhr => xhr.setRequestHeader('Authorization', 'Bearer ' + token);
            });
            hls.on(Hls.Events.MANIFEST_PARSED, () => video.play());
        } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
            video.src = url;
            video.play();
        } else {
            alert('HLS is not supported in this browser.');
        }
    </script>
</body>
</html>

Lambda service#

The Lambda service is intended to work with user modules that mimic the functionality of a separate service. The service enables you to write and use your own handler or write an external service that will closely interact with the LUNA PLATFORM and immediately have several functions typical of LP services (such as logging, automatic configuration reload, etc.).

The Lambda service creates a Docker image and then runs it in a Kubernetes cluster. It is impossible to manage a custom module without Kubernetes. Full-fledged work with the Lambda service is possible when deploying LUNA PLATFORM services in Kubernetes. To use it, you must independently deploy LUNA PLATFORM services in Kubernetes or consult VisionLabs specialists. If necessary, you can use Minikube for local development and testing, thus providing a Kubernetes-like environment without the need to manage a full production Kubernetes cluster.

This functionality should not be confused with the plugin mechanism. Plugins are designed to implement narrow targeted functionality, while the Lambda service enables you to implement the functionality of full-fledged services.

It is strongly recommended to learn as much as possible about the objects and mechanisms of the LUNA PLATFORM (especially about handlers) before starting to work with this service.

A custom module running in a Kubernetes cluster is called lambda. Information about the created lambda is stored in the Lambda database.

The number of lambda created is unlimited. Each lambda has the option to add its own OpenAPI specification.

To work with the Lambda service, you need a special license feature. If the feature is not available, the corresponding error will be returned when requesting the creation of lambda.

Note: The description given below is intended for general acquaintance with the functionality of the Lambda service. See developer manual for more details. The "Quick start guide" section is available in the developer manual, which enables you to start working with the service.

Before start#

Before you start working with the Lambda service, you need to familiarize yourself with all the requirements and set the service settings correctly.

Code and archive requirements#

The module is written in Python and must be transferred to the Lambda service in a ZIP archive.

The code and archive must meet certain requirements, the main of which are listed below:

• Python version 3.11 or higher must be used.
• Development requires the "luna-lambda-tools" library, available in VisionLabs PyPI.
• Archive should not be password-protected.

Also, the files in the archive must have a certain structure. See the detailed information in the section "Requirements" of the developer manual.

Environment requirements#

To work with the Lambda service, the following environment requirements are required:

• Availability of running Licenses and Configurator services*.
• Availability of S3 bucket for storing archives.
• Availability of Docker registry for storing images.
• Availability of Kubernetes cluster.

* during its operation, lambda will additionally interact with some LUNA PLATFORM services. The list of services depends on the lambda type (see "Lambda types").

If necessary, you can configure TTL for storing archives in S3. See "Migration to apply TTL to objects in S3".

Write/read access to the S3 storage bucket must be provided and certain access rights in the Kubernetes cluster must be configured. You need to transfer the basic Lambda images to your Docker registry. The commands for transferring images are given in the LUNA PLATFORM installation manual.

When the Lambda service starts, the Docker registry and base images are checked.

For more information, see the "Requirements" section of the developer manuals.

Lambda service configuration#

In the Lambda service settings, you must specify the following data:

• Location of the Kubernetes cluster (see setting "CLUSTER_LOCATION"):

• "internal" — Lambda service works in a Kubernetes cluster and does not require other additional settings.

• "remote" — Lambda service works with a remote Kubernetes cluster and correctly defined settings "CLUSTER_CREDENTIALS" (host, token and certificate).
• "local" — Lambda service works in the same place where the Kubernetes cluster is running.

In the classic version of working with the Lambda service, it is assumed to use the "internal" parameter.

• S3 bucket settings (see setting "S3").

• Registry address for storing the Docker image (see setting "LAMBDA_REGISTRY").

• Addresses of insecure registries that may need access during lambda creation (see setting "LAMBDA_INSECURE_REGISTRIES").

For more information, see the "Configuration requirements" section of the developer manual.

Configuring lambda entities#

A specific set of settings is available for running lambda entities. They can be set in the Configurator by filtering the settings by the name "luna-lambda-unit".

The settings for the Lambda service and the settings for lambda entities are different.

The settings contain service addresses, connection timeouts, logging settings, etc., allowing lambda entities to effectively interact with LUNA PLATFORM services. See all available settings in the section "Lambda configuration".

The settings apply to all lambda at the same time.

Lambda types#

Lambda can be of three types:

• Handlers-lambda, intended to replace the functionality of the classic handler.
• Standalone-lambda, intended to implement independent functionality to perform close integration with the LUNA PLATFORM.
• Tasks-lambda, intended to implement additional custom long task types.

Each type has certain requirements for LUNA PLATFORM services, the actual settings of which will be automatically used to process requests. Before starting work, the user must decide which lambda he needs.

Handlers-lambda#

Examples of possible functionality:

• Performing verification with the possibility of saving an event.
• Matching of two images without performing the rest of the functionality of the classic handler.
• Adding your own filtering logic to the matching functionality;
• Circumventing certain limitations of the LUNA PLATFORM (for example, specify the maximum number of candidates greater than 100).
• Embedding the neural network SDK bypassing the LUNA PLATFORM.

During its operation, Handlers-lambda will interact with the following LUNA PLATFORM services:

• Configurator — To get the settings.
• Faces — For working with faces and lists.
• Remote SDK — For performing detections, estimations and extractions.
• Events* — For working with events.
• Python Matcher/Python Matcher Proxy** — For classical/cross-matching of faces or bodies.
• Image Store* — For storing samples and source images of faces or bodies.

The Lambda service will not check the connection to the disabled services and will give an error if the user tries to make a request to the disabled service.

To run the Lambda service, only the presence of the Configurator and Licenses services is required.

* the service can be disabled in the "ADDITIONAL_SERVICES_USAGE" setting.

** the service is disabled by default. To enable the service, see the setting "ADDITIONAL_SERVICES_USAGE".

The Lambda-handler can be used in two cases:

• As a custom handler that has its own response scheme, which may differ from the response of classic handlers and cannot be properly used in other LUNA PLATFORM services.

• As a custom handler that mimics the response of a classic handler. There are some requirements for such a case:

  • The response must match the response scheme of the event generation request.
  • The handler must process incoming data correctly so that other services can use it, otherwise there is no guarantee of compatibility with other services. That is, if such a handler implies face recognition, the module should return information about face recognition in response, if the handler implies body detection, the module should return body detection in response, etc.

  For example, if a Lambda handler satisfies the above conditions, then it can be used in Estimator task as a classic handler.

For more information and code examples for Handlers-lambda, see the "Handlers lambda development" section of the developer manual.

Standalone-lambda#

Examples of possible functionality:

• Filtering incoming images by format for subsequent sending to the Remote SDK service.
• Creation of a service for sending notifications by analogy with the Sender service.
• Creation of a service for recording a video stream and saving it as a video file to the Image Store service for subsequent processing by the FaceStream application.

During its operation, Standalone-lambda will interact at least with the Configurator service, which enables lambda to receive its settings (for example, logging settings).

To run the Lambda service, only the presence of the Configurator and Licenses services is required.

For more information and sample code for Standalone-lambda, see the "Standalone lambda development" section of the developer manual.

Tasks-lambda#

Examples of possible functionality:

• Unlink faces from lists if faces do not similar to the specified face.
• Remove duplicates from list.
• Recursively find events similar to the specified photo.

During its operation, Tasks-lambda will interact with the following LUNA PLATFORM services:

• Configurator
• Faces
• Python Matcher
• Remote SDK
• Tasks
• Events*
• Image Store*
• Handlers*

* the service can be disabled in the "ADDITIONAL_SERVICES_USAGE" setting

After creating lambda, the "lambda task" requests must be executed to create a Lambda task. The results of the task/subtask can be obtained using standard Tasks service requests.

The Lambda task is created according to the general task creation process, except that the Lambda service is used instead of the Tasks worker.

When writing code for Tasks-lambda, it is recommended to break each task into subtasks for the convenience of representing the process and parallelizing the execution of tasks.

You can create schedule for Tasks-lambda.

For more information and code examples for Tasks-lambda, see "Tasks lambda development" in the developer manual.

Create lambda#

To create a lambda, you need to do the following:

1․ Write Python code in accordance with the type of future lambda and code requirements.

2․ Move files to the archive in accordance with archive requirements.

3․ Perform the "create lambda" request, specifying the following mandatory data: * "archive" — The address to the archive with the user module. * "credentials" > "lambda_name" — The name for the lambda being created. * "parameters" > "lambda_type" — The type of lambda being created ("handlers", "standalone" or "tasks").

Also optionally in the "deploy_parameters" section you can set the number of pods, allocate resources to pods, set namespaces, as well as enable GPU use.

It is also possible to specify labels for running lambda using specific Kubernetes nodes using the "deploy_parameters" > "selector" parameter. This enables for more granular control over lambda deployment, allowing administrators to specify the nodes on which lambda should be deployed based on their resource needs. By applying labels to Kubernetes nodes and using the "selector" parameter, administrators can manage resource allocation not only for individual lambdas, but also for lambda sections with similar resource requirements.

If necessary, you can specify a list of additional Docker commands to create a lambda container. See the "Lambda — Archive requirements" section of the developer manual.

In response to a successful request, the "lambda_id" will be issued.

The creation of lambda consists of several stages, namely:

• Creating a Docker image:
  • Getting the provided ZIP archive.
  • Addition of the archive with the necessary files.
  • Saving the archive in S3 storage.
  • Publishing the image to the registry.
• Creating a service in the Kubernetes cluster.

During lambda creation, you can run the following requests:

• "get lambda image creation status" to get the Docker image creation status ("in_progress", "error", "completed", "not_found")
• "get lambda image creation logs" to get Docker image creation logs
• "get lambda status" to get the lambda creation status ("running", "waiting", "terminated", "not_found")

See the lambda creation sequence diagram in "Lambda creation diagram".

See the detailed description of the lambda creation process in the "Creation pipeline" section of the developer manual.

Create handler for Handlers-lambda#

If lambda is supposed to be used as a custom handler simulating the response of a classic handler, then it is necessary to create a handler, specifying "handler_type" = "2" and the resulting "lambda_id".

During the creation of the handler, the Handlers service will perform a health check to the Kubernetes cluster.

The resulting "handler_id" can be used in requests "generate events" or "estimator task".

Use lambda#

The table below shows resources for working with lambda, depending on its type.

See the lambda processing sequence diagram in "Lambda processing diagram".

Each lambda has its own API, the description of which is also available using the "get lambda open api documentation" request.

Each lambda response will contain multiple headers, including:

• "Luna-Request-Id" — Classic external ID of the LUNA PLATFORM request.
• "Lambda-Version" — Contains the current lambda version.

Useful requests when working with lambda:

• "get lambda status" to get lambda creation status ("running", "waiting", "terminated", "not_found")
• "get lambda" to get complete information about the created lambda (creation time, name, status, etc.)
• "get lambda logs" to get lambda creation logs

Create GPU-enabled lambda#

It is possible to create a lambda that can leverage graphics processing unit (GPU) resources. To enable GPU usage, the "deploy_parameters" > "enable_gpu" parameter must be included in the lambda creation request.

Lambda supports only NVIDIA graphics processors. For additional information on GPU usage, refer to the Kubernetes documentation.

If there is only one GPU available but more are required, you can enable shared GPU access (refer to the official documentation).

The Lambda service does not manage cluster resources, including GPU allocation. Resource management is handled by the Kubernetes administrator.

Additionally, when deploying LUNA PLATFORM to Kubernetes, it is recommended to configure the manifest in a specific way for all containers that use the GPU (including the Lambda service container) to avoid problems with device visibility. For detailed information, refer to the "Configuring manifest GPU support" section in the installation manual.

Update lambda#

The base image for lambda containers is updated periodically. This image contains the necessary modules to interact with LUNA PLATFORM services. Applying updates requires that the lambda be recreated so that the container can be rebuilt based on the new image. After updating the base image and the "luna-lambda-tools" library, lambda functionality may be broken.

You can update the lambda using the request "update lambda" using the latest base image. It is recommended to have a backup copy of the archive on S3.

See the "Lambda updates" section of the developer manual for detailed information about the update mechanism, backup strategies, and restoring from a backup.

Backport 3#

The Backport 3 service is used to process the requests for LUNA PLATFORM 3 using LUNA PLATFORM 5.

Although most of the requests are performed in the same way as in LUNA PLATFORM 3, there are still some restrictions. See "Backport 3 features and restrictions" for details.

See Backport 3 OpenAPI specification for details about the Backport 3 API.

Backport 3 new resources#

Liveness estimation#

Backport 3 provides Liveness estimation in addition to the LUNA PLATFORM 3 features. See the "liveness > predict liveness" section in the Backport 3 OpenAPI specification.

Handlers#

The Backport 3 service provides several handlers: "extractor", "identify", "verify". The handlers enable to perform several actions in a single request:

• "handlers" > "face extractor" — Enables you to extract a descriptor from an image, create a person with this descriptor, attach the person to the predefined list.

• "handlers" > "identify face" — Enables you to extract a descriptor from an image and match the descriptor with the predefined list of candidates.

• "handlers" > "verify face" — Enables you to extract a descriptor from an image and match the descriptor with the person's descriptor.

The description of the handlers and all the parameters of the handlers can be found in the following sections:

• "handlers" > "patch extractor handler"
• "handlers" > "patch verify handler"
• "handlers" > "patch identify handler"

The requests are based on handlers and unlike the standard "descriptors" > "extract descriptors", "matching" > "identification", and "matching" > verification" requests the listed above request are more flexible.

You can patch the already existing handlers thus applying additional estimation to the requests. E. g. you can specify head angles thresholds or enable/disable basic attributes estimation.

The Handlers are created for every new account at the moment the account is created. The created handlers include default parameters.

Each of the handlers has the corresponding handler in the Handlers service. The parameters of the handlers are stored in the luna_backport3 database.

Each handler supports GET and PATCH requests thus it is possible to get and update parameters of each handler.

Each handler has its version. The version is incremented with every PATCH request. If the current handler is removed, the version will be reset to 1:

• For the requests with POST and GET methods:

  If the Handlers and/or Backport 3 service has no handler for the specified action, it will be created with default parameters.

• For requests with PATCH methods:

  If Handlers and/or Backport 3 service has no handler for the specified action, a new handler with a mix of default policies and policies from the request will be created.

Backport 3 architecture#

Backport 3 interacts with the API service and sends requests to LUNA PLATFORM 5 using it. In turn, the API service interacts with the Accounts service to check the authentication data.

Backport 3 has its own database (see "Backport 3 database"). Some of its tables are similar to the tables of the Faces database of LP 3. It enables you to create and use the same entities (persons, account tokens and accounts) as in LP 3.

The backport service uses Image Store to store portraits.

You can configure Backport 3 using the Configurator service.

Backport 3 features and restrictions#

The following features have core differences:

For the following resources on method POST default descriptor version to extract from image is 56:

• /storage/descriptors
• /handlers/extractor
• /handlers/verify
• /handlers/identify
• /matching/search

You can still upload the existing descriptors of versions 52, 54, 56. The older descriptor versions are no longer supported. - For resource /storage/descriptors on method POST, estimation of "saturation" property is no longer supported, and the value is always set to 1. - For resource /storage/descriptors on method POST, estimation of "eyeglasses" attribute is no longer supported. The attributes structure in the response will lack the "eyeglasses" member. - For resource /storage/descriptors on method POST, head position angle thresholds can still be sent as float values in range [0, 180], but they will be internally rounded to integer values. As before, thresholds outside the range [0, 180] are not taken into account.

Garbage collection (GC) module#

According to LUNA PLATFORM 3 logic, garbage is the descriptors that are linked neither to a person nor to a list.

For normal system operation, one needs to regularly delete garbage from databases. For this, run the system cleaning script remove_not_linked_descriptors.py from ./base_scripts/gc/ folder.

According to Backport 3 architecture, this script removes faces, which do not have links with any lists or persons from the Backport 3 database, from the Faces service.

Script execution pipeline#

The script execution pipeline consists of several stages:

1․ A temporary table is created in the Faces database. See more info about temporary tables for oracle or postgres.

2․ IDs of faces that are not linked to lists are obtained. The IDs are stored in the temporary table.

3․ While the temporary table is not empty, the following operations are performed:

• The batch of IDs from the temporary table is obtained. First 10k (or less) face ids are received.
• Filtered IDs are obtained. Filtered IDs are ids that do not exist in the "person_face" table of the Backport 3 database.
• Filtered IDs are removed from the Faces database. If some of the faces cannot be removed, the script stops.
• Filtered IDs are removed from the Backport 3 database (foolcheck). A warning will be printed.
• IDs are removed from the temporary table.

Script launching#

docker run --rm -t --network=host --entrypoint bash dockerhub.visionlabs.ru/luna/luna-backport-3:v.0.12.3 -c "python3 ./base_scripts/gc/remove_not_linked_descriptors.py"

The output will include information about the number of removed faces and the number of persons with faces.

Backport 4#

The Backport 4 service is used to process the requests for LUNA PLATFORM 4 using LUNA PLATFORM 5.

Although most of the requests are performed in the same way as in LUNA PLATFORM 4, there are still some restrictions. See "Backport 4 features and restrictions" for details.

See Backport 4 OpenAPI specification for details about the Backport 4 API.

Backport 4 architecture#

Backport 4 interacts with the API service and sends requests to LUNA PLATFORM 5 using it.

Backport 4 directly interacts with the Faces service to receive the number of existing attributes.

Backport 4 directly interacts with the Sender service. All the requests to Sender are sent using the Backport 4 service. See the "ws" > "ws handshake" request in the Backport 4 OpenAPI specification.

You can configure Backport 4 using the Configurator service.

Backport 4 features and restrictions#

The following features have core differences:

The current versions for LUNA PLATFORM services are returned on the request to the /version resource. For example, the versions of the following services are returned:

• "luna-faces"
• "luna-events"
• "luna-image-store"
• "luna-python-matcher" or "luna-matcher-proxy"
• "luna-tasks"
• "luna-handlers"
• "luna-api"
• "LUNA PLATFORM"
• "luna-backport4" — Current service

Resources changelog:

• Resource /attributes/count is available without any query parameters and does not support accounting. The resource works with temporary attributes.

• Resource /attributes on method GET: "attribute_ids" query parameter is allowed instead of "page", "page_size", "time__lt" and "time__gte" query parameters. Thus you can get attributes by their IDs not by filters. The resource works with temporary attributes.

• Resource /attributes/<attribute_id> on methods GET, HEAD, DELETE and resource /attributes/<attribute_id>/samples on method GET interact with temporary attributes and return attribute data if the attribute TTL has not expired. Otherwise, the "Not found" error is returned.

• If you already used the attribute to create a face, use the "face_id" to receive the attribute data. In this case, the "attribute_id" from the request is equal to "face_id".

• Resource /faces enables you to create more than one face with the same "attribute_id".

• Resource /faces/<face_id> on method DELETE enables you to remove face without removing its attribute.

• Resource /faces/<face_id> on method PATCH enables you to patch attribute of the face making the first request to patch "event_id", "external_id", "user_data", "avatar" (if required) and the second request to patch attribute (if required).

• If face attribute_id is to be changed, the service will try to patch it with temporary attribute data if the temporary attribute exists. Otherwise, the service tries to patch it with attribute data from the face with "face_id" = "attribute_id".

• The match policy of resource /handlers now has the default match limit that is configured using the "MATCH_LIMIT" setting from the Backport 4 "config.py" file.

• Resource /events/stats on method POST: "attribute_id" usage in "filters" object was prohibited as this field is no longer stored in the database. The response with the 403 status code will be returned.

• The "attribute_id" in events is not null and is equal to "face_id" for back compatibility. GC task is unavailable because all the attributes are temporary and will be removed automatically. Status code 400 is returned on a request to the /tasks/gc resource.

• The column "attribute_id" is not added to the report of the Reporter task and this column is ignored if specified in the request. Columns "top_similar_face_id", "top_similar_face_list", "top_similar_face_similarity" are replaced by the "top_match" column in the report if any of these columns is passed in the reporter task request.

• Linker task always creates new faces from events and ignores faces created during the event processing request.

• Resource /matcher does not check the presence of provided faces thus error "FacesNotFound" is never returned. If the user has specified a non-existent candidate of type "faces", no error will be reported, and no actual matching against that face will be made.

• Resource /matcher checks whether reference with type attribute has the ID of face attribute or the ID of temporary attribute and performs type substitution. Hence it provides sending references for matching in the way it was done in the previous version.

• Resource /matcher takes matching limits into account. By default, the maximum number of references or candidates is limited to 30. If you need to overcome these limits, configure "reference_limit" and "candidate_limit".

• Resource /ws has been added. There was no /ws resource in the LUNA PLATFORM 4 API as it was a separate resource of the Sender service. This added resource is similar to the Sender service resource, except that "attribute_id" of candidates faces is equal to "face_id".

• Resource /handlers returns the error "Invalid handler with id {handler_id}", if the handler was created in the LUNA PLATFORM 5 API and is not supported in LUNA Backport 4.

Backport 4 User Interface#

The User Interface service is used for the visual representation of LP features. It does not include all the functionality available in LP. User Interface enables you to:

• Download photos and create faces using them.
• Create lists.
• Match existing faces.
• Show existing events.
• Show existing handlers.

All the information in User Interface is displayed according to the account data, specified in the configuration file of the User Interface service ()./luna-ui/browser/env.js.

User Interface works with only one account at a time, which must be of "user" type.

You should open your browser and enter the User Interface address. The default value is: <server_url>:4200.

You can select a page on the left side of the window.

Lists/faces page#

The starting page of User Interface is Lists/Faces. It includes all the faces and lists created using the account specified in the configuration.

The left column of the workspace displays existing lists. You can create a new list by pressing the Add list button. In the appeared window you can specify the user data for the list.

The right column shows all the created faces with pagination.

Use the Add new faces button to create new faces.

On the first step, you should select photos to create faces from. You can select one or several images with one or several faces in them.

After you select images, all the found faces will be shown in a new dialog window.

All the correctly preprocessed images will be marked as "Done". If the image does not correspond to any of the requirements, an error will be displayed for it.

Press the Next step button.

On the next step, you should select the attributes to extract for the faces.

Press the Next step button.

On the next step, you can specify user data and external ID for each of the faces. You can also select lists to which each of the faces will be added. Press Add Faces to create faces.

You can change pages using arrow buttons.

You can change the display of faces and filter them using buttons in the top right corner.

Filer faces. You can filter faces by ID, external ID or list ID.

/

Change view of the existing faces.

Handlers page#

Handlers page displays all handlers created using the account specified in the configuration.

All the information about specified handler policies is displayed when you select a handler.

You can edit or delete a handler using edit

and delete

icons.

Events page#

The events page displays all the events created using the account specified in the configuration.

It also includes filters for displaying of events

.

Common information#

You can edit

or delete

an item (face, list or handler) using special icons. The icons appear when you hover the cursor on an item.

Matching dialog#

The Matching button in the bottom left corner of the window enables you to perform matching.

After pressing the button you can select the number of the received results for each of the references.

On the first step, you should select references for matching. You can select faces and/or events as references.

On the second step, you should select candidates for matching. You can select faces or lists as candidates.

On the last step, you should press the Start matching button to receive results.

Resource consumption by services#

Below is a table describing the most significant resource consumption by services. The Remote SDK and Python Matcher services perform the most resource-intensive operations. 

Additional Information#

OneShotLiveness description#

The Liveness technology enables to detect presentation attacks. A presentation attack is a situation when an imposter tries to use a video or photos of another person to circumvent the recognition system and gain access to the person's private data.

To estimate Liveness in the LUNA PLATFORM, the estimator LUNA SDK OneShotLiveness is used.

There are the following general types of presentation attacks:

• Printed Photo Attack. One or several photos of another person are used.
• Video Replay Attack. A video of another person is used.
• Printed Mask Attack. An imposter cuts out a face from a photo and covers his face with it.
• 3D Mask Attack. An imposer puts on a 3D mask depicting the face of another person.

Liveness has the ability to license the number of performed transactions. You can also choose between an unlimited license and a license with a limited number of transactions (a maximum of 16 777 215 transactions can be specified in the key). After the transaction limit is exhausted, it will be impossible to use the Liveness estimation in requests. Requests that do not use Liveness or with Liveness estimation disabled are not affected by the exhaustion of the limit, they continue to work as usual.

Liveness check results#

The Liveness algorithm uses a single image for processing and returns the following data:

• Liveness probability [0..1]. Here 1 means real person, 0 means spoof. The parameter shows the probability of a live person being present in the image, i.e. it is not a presentation attack. In general, the estimated probability must exceed the Liveness threshold.

• Prediction. Based on the above data, LUNA PLATFORM issues the following prediction:

• 0 (spoof). Check revealed that the person is not real.

• 1 (real). Check revealed that the person is real.

Requests for estimating Liveness#

Liveness is used in the following resources:

• "/Liveness"
• "/sdk"
• "/handlers"
• "/verifiers"

You can filter events by Liveness in the "handlers/{handler_id}\/events" and "/verifiers/{verifier_id}\/verifications" resources, i.e. you can exclude "spoof" or "real" results from image processing.

Filtering by Liveness is available for the following scenarios:

• When performing matching operations.
• When performing tasks.
• When sending data via web sockets.

You can also specify the Liveness estimation parameter when manually creating and saving events in the "handlers/{handler_id}\/events/raw" resource.

For multiple uploaded images, you can aggregate the Liveness results to obtain more accurate data.

Liveness requirements#

The requirements for the processed image and the face in the image are listed below.

Note: When running Liveness and Deepfake checks together, it is necessary to take into account the requirements for the processed image for both estimators.

See the "Image Quality" section for details on image quality thresholds.

Liveness threshold#

The Liveness threshold is used to adjust the Liveness score.

Liveness threshold — the threshold lower which the system will consider the result as a presentation attack ("spoof"). This threshold is set in the "real_threshold" setting from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service. The default threshold value is 0.5 (50%).

For images received from mobile devices, it is recommended to set a threshold 0.5. For images obtained from a webcam, it is recommended to set a threshold 0.364 . At these threshold values, approximately equal results are obtained in terms of the accuracy of the algorithm.

Changing threshold on "/handlers" and "/verifiers" resources#

In the "/handlers" and "/verifiers" there is one additional parameter - "liveness_threshold" which sets the Liveness threshold.

Setting this parameter redefines the value of the "real_threshold" settings from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service.

Changing threshold on «/Liveness», «/sdk» and «/videosdk»#

There are no additional parameters for overriding threshold for the "/Liveness", "/sdk" and "/videosdk" resources. Threshold is set in the "real_threshold" settings from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service.

Video analytics#

LUNA PLATFORM offers the ability to perform video analytics in two ways:

• the "videosdk" resource
• the group of resourses "/streams" with enabled Video Agent and Video Manager services (further - stream functionality for video analysis).

Main features of the resource "/videosdk":

• processing one video file at a time
• processing video files in background
• performing analytics of people count and human tracking
• issuing information only in the body of the response to the request

Main features of the group of resources "/streams", realizing the functionality of streams:

• processing a video file or video stream
• performing analytics for people count and human tracking, as well as external self-written analytics
• saving information about processing rules in a separate entity stream and in the database
• parallel processing of streams
• generating general events and sending notifications to third-party systems using the Callback mechanism

This section describes the basic concepts of video analytics. See the detailed description of working with the "/streams" resource group in the section "Stream functionality for video analysis".

Important: A general event is not related in any way to events generated by handlers.

Since both methods pursue different goals, they may have their own set of specific parameters and values. For example, when using the stream functionality, it is possible to set the moment of event generation (at the beginning of the track, at the end of the track, periodically), while in the "videosdk" resource there is no such possibility. The features of the presence of certain parameters are given in the corresponding sections below.

Important: Currently, the video analytics functionality is in beta testing. Input and output schemas may change in future releases without backward compatibility support.

Video analytics is a set of functions that process frame by frame and evaluate useful data.

A key concept in video analytics is the track. A track represents a long continuous observation in which several events can be generated at certain intervals. A track is needed to organize and aggregate data about groups of people who can appear and move in the video.

If necessary, the video or video stream can be rotated before processing by 90, 180, or 270 degrees.

During video analytics execution, a certain number of events are generated according to some rules, where each event has a beginning and an end. Events contain specific information of the particular video analytics. The response also contains basic meta-information about the video (number of frames, frame rate, video duration).

Important: An event in video analytics is not related to events generated by handlers.

Video processing settings, such as the number of decoder processes, the type of processor for video decoding, the temporary directory for storing videos, etc., can be set in the "LUNA_VIDEO_AGENT_VIDEO_SETTINGS" settings group.

People counting video analytics#

A track starts when the specified number of people (parameter "people_count_threshold") appears on the last frame of the specified number of consecutive frames (parameter "probe_count"). For example, if the minimum number of people is 10 and the number of consecutive frames is 3, then if there are 10 people in 3 frames, the track will start from the 3rd frame. If there are 10 people in 2 frames and 9 people in the 3rd frame, the track will not start.

By default, every 10 frames are processed. If necessary, the frame processing frequency can be adjusted using the "rate" parameter (e.g., process every 3 seconds or every 3 frames).

The logic of generating video analytics events differs for "videosdk" requests and video analytics services.

When performing analytics using the "videosdk" resource, the number of events will equal the number of tracks.

When performing video analytics using the Video Agent service, video analytics events can be generated in the following cases ("event_policy"):

• When the track starts ("trigger" > "start").
• When the track ends ("trigger" > "end").
• Periodically while the track exists ("trigger" > "period" and "interval").

For example, if 100 people are constantly in the frame for an hour, the service can generate events every 5 minutes to track changes or confirm the constant number of people.

In the request body, you can set the "targets" parameter, which accepts the following values:

• "coordinates" — Calculation of people's coordinates.
• "overview" — Obtaining an image with the maximum number of people and enabling image saving. If necessary, you can fine-tune the quality, format, and maximum size of the saved image using the "image_retain_policy" section.

You can set both values or none. If no values are set, a basic analysis will be performed, containing only information about the video recording, the number of people, and the frames associated with them.

For such video analytics, it is also possible to configure a region of interest ROI or a region of interest DROI on the frame (coordinates "x", "y", width, height, units of measurement — percentages or coordinates). The principle of the ROI is similar to FaceStream.

Each event in people count video analytics contains the following information:

• "event_id" — Event identifier.
• "track_id" — Track identifier linking events generated within one continuous observation.
• "people_count" — Maximum number of people detected in the processed video segment.
• "video_segment" — Information about the video segment (event start time, event end time, first frame number, last frame number).
• "frames_estimations" (only "videosdk" resource) — Array with information about estimations on each frame, containing the following information:
  • People coordinates.
  • Time corresponding to the frame.
  • Number of people.
• "overview" — information about the frame in the video where the most people were found:
  • Image.
  • Time corresponding to the frame.
  • People coordinates.

Human tracking video analytics#

For people tracking video analytics, a track is an object containing information about the position of one person in a sequence of frames. A track is created when the detection of a face, body or body with a face (depends on the detector type in the "detector_type" parameter) appears on the last frame of the specified number of consecutive frames ("probe_count" parameter). There can be as many tracks as the number of people detected on the frame.

When performing video analytics using the "videosdk" request, there can only be one event for one track.

For example, if "probe_count" = "3" and there are detections on two consecutive frames and no detections on the third frame, the track and event will not start. If at any moment of tracking a person there were detections on two consecutive frames and no detections on the third frame, the track and event will end.

When performing video analytics using the Stream functionality for video analysis, video analytics events can be generated in the following cases ("event_policy"):

• When the track starts ("trigger" > "start").
• When the track ends ("trigger" > "end").
• Periodically while the track exists ("trigger" > "period" and "interval").

During human tracking, it is possible to estimate attributes of faces, bodies, images, estimations Deepfake and OneShotLiveness (only resource "videosdk"), and also to obtain an image with the best detection. The list of estimations is specified in the "targets" parameter and differs depending on the analytics method ("videosdk" or Stream functionality for video analysis). Estimation can be performed based on one best image from the track or on several (parameter "face_samples"/"body_samples" > "count"). If necessary, you can filter the best images by the "score", "head_pitch", "head_roll", "head_yaw" parameters.

If necessary, you can enable Re-identification of the body (ReID). The ReID feature is used to improve tracking accuracy by combining different tracks of the same person.

For this type of video analytics, you can also configure ROI or DROI regions on the frame (coordinates "x", "y", width, height, units - per cent or coordinates). The principle of ROI region of interest is similar to FaceStream.

Each people count video analytics event contains the following information:

• "event_id" — Event ID.
• "track_id" — Track ID.
• "video_segment" — Information about the video segment (start time of the event, end time of the event, first frame number, last frame number).
• "frames_estimations" — Array with information about estimates on each frame, containing the following information:
  • Frame number.
  • Time corresponding to the frame.
  • Bbox coordinates and score.
• "aggregated_estimations" — Result of the score specified in the "targets" parameter in the request body.
• "track_count" - Number of tracks.

When using the Stream functionality for video analysis, each event also displays the stream ID and location specified in the request to create the stream.

Important: Depending on the length, resolution, and frame per second (fps) of the video, performing people tracking video analytics using the "/videosdk" resource may take longer. In this regard, it may be necessary to increase the values ​​of time intervals that regulate the timeouts of HTTP requests. To do this, you will need to do the following:

• go to the Configurator service interface http://<your_server_ip_adress>:5070/;
• enter the value "LUNA_REMOTE_SDK_TIMEOUTS" in the "Setting name" field and click "Apply Filters";
• in Value, set the values ​​"request": 320, "connect": 20, "sock_connect": 10, "sock_read": 320.

roi#

ROI (Region of Interest) specifies the region of interest in which the estimation is performed.

The specified rectangular area is cut out from the frame and LUNA PLATFORM performs further processing using this image.

Correct exploitation of the "roi" parameter significantly improves the performance of video analysis.

ROI on the source frame is specified by the "x", "y", "width", "height" and "mode" parameters, where:

• "x", "y" – Coordinates of the upper left point of the ROI area of interest.
• "width" and "height" — Width and height of the processed area of the frame.

• "mode" – Mode for specifying "x", "y", "width" and "height". Two modes are available:

  • "abs" — Parameters "x", "y", "width" and "height" are set in pixels.
  • "percent" — Parameters "x", "y", "width" and "height" are set as percentages of the current frame size.

  If the "mode" field is not specified in the request body, then the value "abs" will be used.

With width and height values of "0", the entire frame is considered the region of interest.

The coordinate system on the image is set similarly to the figure below.

When the values of width and height are set to "0", the entire frame will be the region of interest.

Below is an example of calculating ROI as a percentage:

droi#

DROI specifies a region of interest relative to the source frame. If ROI is intended to optimize the estimation of the corresponding features, then DROI works as a filter after the estimation has been completed and is intended to implement business logic. You can use DROI both together with ROI and separately. For example, if after processing by the ROI area the number of people in the frame was equal to N, then after additional filtering by the DROI area the number of people in the frame can be reduced to M.

The DROI on the source frame is specified by the following parameters:

• "area" — Geometry of the region of interest. The parameter is represented as an array of polygons. Each polygon is represented by an array of objects, where each object contains the x and y coordinates of the polygon's vertex. For example, you can define a triangular area of interest.
• "mode" – mode of specifying “x”, “y”. Two modes are available:
  • "abs" — Parameters "x", "y", "width" and "height" are set in pixels.
  • "percent" — Parameters "x", "y", "width" and "height" are set as percentages of the current frame size.
• "form" — Format of the region of interest. Can take the values: "common" - the general format and "wkt" (Well-known text) - the text format for representing geometric objects.

Processing video in background#

To run video analysis based on the specified analytics parameters in the background, you need to make a request to the resource /videosdk with the deferred_task parameter enabled, which will return the task_id to track its status.

You can track the task status and get the result URL (if completed successfully) by task_id using the request get task.

The parameter max_active_tasks in the LUNA_VIDEO_AGENT_SETTINGS setting is responsible for regulating the maximum number of tasks that can be simultaneously executed in the background by one agent. The default value is 2. The created tasks will be executed in the order of the queue if the number of active tasks reaches the specified limit.

Stream functionality for video analysis#

The Video Manager service implements advanced functionality for working with video analytics. It allows you to create separate entities streams containing all the information about the video file or video stream being processed and video analytics that will need to be performed, and also allows you to flexibly manage streams.

The functionality of the streams is based on the interaction of three key components: analytics, agents and Video Manager service.

Note: The ability to create streams requires a licensed feature that defines the maximum number of simultaneous streams that can be created in the Video Manager service. See the "Licenses service" section for details on licensed features.

Analytics

Analytics is a set of algorithms that perform specific tasks for processing streams. Each analytic has a set of parameters that can be configured by the user. The Video Manager stores information about available analytics and passes it to the agents that implement the corresponding algorithms.

Agent

An agent is a component that implements video analytics algorithms. It receives streams, performs the specified analytics, and sends the results through callbacks.

The default agent is the Video Agent service, but you can write your own agent if needed. See the example code in the Video Manager service developer guide.

Video Agent service provides the following analytics:

• People counting video analytics
• Human tracking video analytics

Video Manager

The Video Manager acts as a coordinator between users and agents. It contains information about available analytics and distributes video streams for processing among agents. The Video Manager is also responsible for managing streams, monitoring their status, and coordinating automatic restarts in case of errors.

See detailed information about the interaction between the Video Manager service and the agent in the section "Video manager and agent ideo services

The basic principle of operation is as follows:

1․ The user makes a request to create a stream using the "create stream" request.

2․ The Video Manager service directs the stream to an agent capable of processing all the analytics specified in the request.

3․ The agent processes the stream and sends the processing status and any possible errors to the Video Manager service.

4․ The agent sends the processing results through callbacks according to selected type (see "Send and save analytic results" section).

You can also get the processing results by running the stream events ws handshake request, which allows you to establish a web socket connection to get the processing results of a specific stream in real time by specifying the "stream_id".

The user can also recreate the stream, get the stream processing status, stop and start the stream, etc. See detailed information in the streams request group of the OpenAPI specification.

See detailed information on how the user interacts with video analytics services in the section "User and Video Manager interaction".

See the section "Quick start guide to create stream" below to learn the basic steps for creating a stream.

See detailed information in the section "Video manager and agent intecation".

Send and save analytic results#

An agent can send processing results using the callbacks with the following types:

• "http", after configuring a server to receive messages
• "luna-ws-notification", after establishing a web socket connection
• "luna-event"

The "http" and "luna-ws-notification" types are standard. See the section "Send events to third-party service".

If websockets are used, the Sender service must be enabled in the "ADDITIONAL_SERVICE_USAGE" setting and a connection to the Sender service must be established (see request "ws handshake for general events").

The "luna-event" type is intended for saving events in the Events database using the general events mechanism. Such events can be retrieved using the "get general events" and "get general event" requests by specifying the obtained "stream_id" as a filter.

You can also get estimation results (not events) in real time using the query "stream events ws handshake". For example, if you are analyzing the number of people, you can get the coordinates of people at the moment they appear on the frame.

When creating a stream, you can configure notifications about changes in the stream processing status using the parameter notification_policy. There are two types of callbacks available for sending notifications:
- http — notifications are sent to the specified URL;
- telegram — notifications are sent to the Telegram chat. To do this, you need to specify the chat ID chat_id, and the authorization token token for Telegram.

See create stream.

Analytics#

From the perspective of the Video Manager service, an analytic is an object containing a name, an unknown set of parameters, and, if necessary, a description and documentation. An analytic is created either by an agent or by an administrator by sending a "create analytic" request to the Video Manager service.

There are two analytics available in the Video Agent service - people_count and human_tracking. Analytics are automatically registered when the Video Agent service is launched.

The Video Manager distributes streams among agents based on the knowledge of which agent can perform which analytic. When an agent or administrator creates an analytic, they can specify a set of parameters that will be used when the analytic is started. However, when making a stream processing request, the user can specify other parameters that will be used for that analytic (using the "analytics" > "parameters" parameter in the "create stream" request).

Note: It is important to note that the minimum system requirements are specified exclusively for face and body recognition, without taking into account video analytics tasks (tracking people, counting the number of people, etc.). The resources required to perform these analytics are calculated individually depending on the analytics used.

Detailed information about available parameters and their descriptions varies for each analytic. It is available in the corresponding analytic's documentation and can be obtained in response to a "get analytics" request.

External mode of Video Agent service#

External mode is intended for cases when it is necessary to launch the Video Agent service outside the main platform circuit - directly next to the video stream sources (cameras) in order to reduce video transmission delays and reduce the likelihood of data loss during network failures. In this mode, Video Agent interacts only through the API service.

Limitations when launch the Video Agent service in external mode:

• the resource /videosdk is not available;
• the API and Licenses services must be available and enabled.

Requests available when Video Agent is launching in external mode:

• "post agent streams" - Get agents streams by agent ID and send information about streams processing.
• "post streams feedback" - Post information about streams processing.
• "create analytic" - Create analytic.
• "put analytic" - Update existing analytic. You should specify all the fields required for the stream in the request.
• "register agent" - Register agent.
• "remove agent" - Remove the agent by ID.
• "publish general event" - Publish events generated by the user.

Self-written agents also support this functionality. See the Video Manager developer guide for more details.

See "Launch external mode of the Video Agent service".

User and Video Manager interaction#

To perform stream analytics, the user must create a stream, specifying the desired analytic.

To create a stream, a "create stream" request must be made. To recreate a stream, a "put stream" request must be made, in which case new stream data will be used, and the stream version will be incremented by 1.

Once a new stream appears, it transitions to the status "pending".

When the stream is being processed by one of the available agents (see the section "Stream distribution"), its status will be "in_progress".

While the stream is being processed, the agent sends feedback to the Video Manager service, which creates logs that the user can obtain by making a "get streams logs" request.

Stream processing can be stopped:

• by user request ("patch stream" request with the parameter "action" = "stop"). In this case, the stream status will be "stop".

• by user request ("remove stream" request). In this case, the stream will be deleted.

• when the stream ends. In this case, the stream status will be "done".

• in the event of a fatal error. In this case, the stream status will be "failure".

In all cases, except for manual log deletion using the "delete stream logs" request, stream logs will be available after making a "get streams logs" request.

In all cases, except for stream deletion using the "remove stream" request, stream data will be available after making a "get stream" request.

Video Manager and agent interaction#

The interaction between the Video Manager service and the agent is described below:

1․ The agent or administrator must register an analytic that the agent can handle with the Video Manager service by making a request to the /1/analytic resource of the Video Manager service.

2․ At startup, the agent must make a request to the /1/agent resource of the Video Manager service to register itself. This request specifies the following parameters:

• agent name (required)
• maximum number of streams the agent can handle (required)
• set of analytics the agent can handle (required)
• agent description

In response to the request, the agent will receive a unique "agent_id".

3․ The agent must periodically make requests to the /1/agent/{agent_id}/streams resource of the Video Manager service to obtain/stop a stream for processing.

Once the agent receives a stream for processing, the Video Manager service starts the stream counter being processed by this agent.

Note that if the agent does not make such a request in time, the agent's status will be marked as "not_ready" and the agent will be excluded from the list of agents that can process streams (see the section "Managing agent status in case of no agent request" for details).

4․ During processing, the agent must send feedback on the stream processing status to the /1/agent/{agent_id}/streams/feedback resource.

The report contains the stream identifier, its status, error, stream version, and report generation time. After sending the report, the user can obtain the processing logs using the "get streams logs" request.

Note that a user agent must perform the above actions. See the example code in the "Agent interaction" section of the Video Manager service developer guide.

The Video Agent service automatically performs the above actions.

Streams#

Streams are created by the user using the "create stream" request.

The user can set the name of the stream, its description, add information about the city or street, configure automatic restart of the stream, set the parameters of stream grouping, analytics, as well as the following general data for stream processing:

• "type" - Source type: videofile or videostream
• "reference" - Source reference.

• "orientation_strategy" - Source for determining video rotation.

  The following values are available:

  • static - manual angle setting (set by default). You can set the video rotation angle by 90, 180 and 270 degrees before processing it in the rotation field (angle of camera frame rotation).
  • videoinfo - the rotation angle is determined by the metadata of the video file or stream.

• "downloadable" - Determines whether the video file should be pre-downloaded before it is processed.

  Pre-downloading is necessary for:

  • Successful decoding of video files. Some files may experience decoding errors if the files are not saved first.
  • Long processing processes. If video processing takes a significant amount of time, storing prevents problems associated with connection breaks.

  Note that sometimes security policies prohibit saving video files. In such cases, pre-downloading may not be applicable.

• "timestamp_source" - Specifies where the timestamps for video analysis are to be sourced from.

  The following values are available:

  • "pts" - Uses video timestamps if present for precise playback time representation. Timestamps might not always be accurate (see below).
  • "server" - Utilizes server time for the video stream, ensuring consistency and synchronization with other events.
  • "frame_rate" - Employs frame rate for video files, helping to approximate timestamps.
  • "auto" (default) - Automatically selects the time source, first checking timestamps ("pts"), and then switching to server time ("server") or frame rate ("frame_rate") if timestamps are inaccurate.

  Accurate timestamps for a video file mean that the time stamps (PTS) should be close to zero, that is, the time from the beginning of the video to the tag should be relatively small (the absolute value should not exceed 10^5 seconds).

  Accurate timestamps for a video stream mean that the stream time differs from the server time by less than 1 day.

• "pts" > "start_time" - Sets a custom offset for video timestamps (PTS).

  Specifying an offset allows synchronization of video processing start with a specific point in time. This is useful, for example, if the video is processed in segments and it is necessary for the timestamps of new segments to continue from the previous ones. Consequently, when splitting a large video into smaller segments, the timestamps will appear as though a single continuous video was processed.