Glossary#

Introduction#

LUNA PLATFORM is an automated face and body recognition system. LUNA PLATFORM is designed to solve the following problems:

• Images processing and analysis:
  • Face and body detection in photos.
  • Basic attributes (age, gender) and face, body, image parameters estimation (emotions, upper clothing, image ratio and other).
  • Checking images according to ISO/IEC 19794-5:2011 and non-standard conditions.
  • Presentation attacks detection (Liveness check) and image analysis for the use of Deepfake technology.
• Search for similar faces in the database (1 to 1, 1 to many and M to N).
• Storage of the received face and body descriptors in databases.
• Creation of lists to search in.
• Statistics gathering.
• Flexible request management to meet user data processing requirements.

Some of these features are licensed separately. See the "License information" section.

General concepts#

The following sections explain general LP concepts, describe existing services and created data types. It does not describe all the requests, database structures, and other technical nuances.

Services#

LP consists of several services. Communication between them occurs via HTTP requests: a service receives a request and always returns a response. Some services also communicate with each other via Redis or web sockets.

You can find information about LUNA PLATFORM 5 architecture in the "Interaction of LP Services" section.

All the services can be divided into general and additional. Using the general services, the optimal operation of the LP is ensured. Using of all general services is enabled by default in the API service settings. If necessary, some general services can be disabled (see the "Disableable services" section).

Most of the services have their database or file storage.

General services#

Additional services#

Additional services provide more opportunities for the system to work. Launching additional services is optional.

Below is a diagram of the interaction of the general and additional services.

This diagram does not describe the communication lines between the services. For a full description of the interaction of services, see the "Interaction of LP services" section.

Third-party services#

There are several third-party services that are usually used with LP.

These services are not described in this document. See the corresponding documentation provided by the services vendors.

Authorization system#

Most requests to the LUNA PLATFORM require a mandatory account, except for requests that do not involve authorization.

Accounts

The account is required to delimit the visibility areas of objects for a particular user. Each created account has its own unique "account_id".

The account can be created using a POST request "create account" to the API service, or by requesting "register account", or using the Admin user interface. When creating the account, you must specify the following data: login (email), password and account type.

The account type determines what data is available to the user.

• user – It is type of account with which you can create objects and use only your account data.
• advanced_user – It is type of account for which rights similar to "user" are available, and there is access to the data of all accounts. Access to data from other accounts means the ability to receive data (GET requests), check their availability (HEAD requests) and perform comparison requests based on data from other accounts.
• admin – It is type of account for which rights similar to "advanced_user" are available, and there is also access to the Admin service.

Using the "Luna-Account-Id" header in the "create account" request, you can set the desired account ID.

Tokens

Token is linked to an existing account with any type and enables you to impose extended restrictions on the requests being made. For example, when creating the token, you can give the user permission only to create and modify all lists and faces, or you can prevent the use of certain handlers by specifying their ID.

The token and all its permissions are stored in the database and linked to the account by the "account_id" parameter.

When creating the token, you can set the following parameters:

• expiration_time – Expiration time of the token in RFC 3339 format. You can specify an infinite token expiration time using the value "null".
• permissions – Permissions that are available to the user.
• visibility_area – Token visibility of data from other accounts.

Authorization types for accessing resources

There are three types of authorization available in LUNA PLATFORM:

• BasicAuth – Authorization by login and password (set during account creation).
• BearerAuth – Authorization by JWT token (issued after the token is created).
• LunaAccountIdAuth – Authorization by "Luna-Account-Id" header, which specifies the "account_id" generated after creating the account (this method was adopted as the main one before version 5.30.0).

LunaAccountIdAuth authorization has the lowest priority compared to other methods and can be disabled using the "ALLOW_LUNA_ACCOUNT_AUTH_HEADER" setting in the "OTHER" section of the API service settings in the Configurator (enabled by default). In the OpenAPI specification the "Luna-Account-Id" header is marked with the word Deprecated.

See detailed information about the LUNA PLATFORM 5 authorization system in the "Accounts, tokens and authorization types" section.

Approaches at work#

There are three main operations in LUNA PLATFORM:

1. Detection is the process of recognizing a face or body in a photo and normalizing the image (creating a sample) for further work. At the stage of detection, estimation of face or body parameters also performs, i.e. estimation of emotions, direction of gaze, upper clothes, etc.
2. Extraction is the process of extracting gender and age from a face image and extracting a set of unique parameters of a face or body (descriptors) from a sample to perform further matching.
3. Matching is the process of comparing descriptors.

These operations are performed strictly one after the other. It is impossible to match face descriptors without first extracting them.

There are two main approaches when performing the operations described above.

Parallel performing of requests#

The first and main approach is to set the rules of detection, estimation, extraction, matching, etc. in a single handler object. After that, you need to create an object event, which will give a result based on all the rules specified in the handler. Using this approach is the most optimal from the point of view of business logic.

With this approach, the following actions are performed:

• Using the "create handler" request, a handler is created containing information about image processing rules;
• In the "generate events" request, the received handler ID is specified, the processed image is attached and an event is generated containing information obtained by processing handler rules.

Thus, when an event is generated, detection, estimation, extraction, matching, saving to the database, and so on are performed simultaneously.

Examples of common scenarios for using handlers:

• Registration of a reference descriptor with saving to the list.
• Biometric identification of faces on the list without saving.
• Saving faces identified in the list in the database.
• Saving events only for unique faces for later counting.
• Batch identification.
• Batch import.

For some scenarios, it may be necessary to pre-create certain objects in the system using separate requests. For example, to identify a set of faces by a list, you first need to create a list object, link the previously created face objects to it, and then use handler and event.

Advantages of the approach:

• All the rules in one place.
• Flexible management of saving objects to databases, setting up save filters.
• Ability to work with packages of images located in ZIP archive, FTP server, S3-like storage, etc.
• Ability to send notifications via web sockets.
• Collecting statistics.

The classic way to use this approach is paired with the FaceStream module. FaceStream analyzes the video stream and sends the best shots from the video stream to LUNA PLATFORM for further processing. Using the specified handler, LUNA PLATFORM processes images according to the specified rules and saves objects to the specified databases.

Sequential performing of requests#

The second approach is to sequential performing of requests, i.e. in one request you need to perform a face detection and get its result, then use this result in an extraction request and so on.

With this approach, the following actions are performed:

• Normalizing image, as well as face detection and estimation using the "detect faces" request.
• Extracting gender, age and descriptor from a normalized image using the "extract attributes" request.
• Creating a face using the "create face" request.
• Matching of face descriptors using the "matching faces" request.

As a rule, this approach is used when working with faces, but if necessary, you can also make separate requests with bodies, for example, to match body descriptors using the "matching bodies" request.

This approach is recommended for use:

• During the initial acquaintance with the system.
• If you need to measure the speed of each stage of image processing separately.
• If you need to implement a processing scenario that is difficult to implement using handlers and events.

More detailed information about these approaches will be provided below.

Detection#

LP searches for all faces and/or bodies on each incoming photo image.

Detection is performed using neural networks located in the Remote SDK container.

Detailed information about neural networks is described in the "Neural networks" section.

Either the source image or a special format image obtained using the VisionLabs software (sample) can be submitted to the input.

Image format requirements#

Images should be sent only in allowed formats. The image format is specified in the "Content-Type" header.

The following image formats are supported: JPG, PNG, BMP, PORTABLE PIXMAP, TIFF. Each format has its own advantages and is intended for specific tasks.

The most commonly used formats are PNG and JPG. Below is a table with their advantages and disadvantages:

Thus, if it is not intended to save the source images in the Image Store, or the Image Store has a sufficiently large volume, then it is recommended to use PNG format to get the best image processing results.

It is not recommended to send images that are too compressed, as the quality of face and body estimations and matching will be reduced.

LUNA PLATFORM 5 supports many color models (for example, RGB, RGBA, CMYK, HSV, etc.), however, when processing an image, all of them are converted to the RGB color model.

The image can also be encoded in Base64 format.

For more information about the source images, see "Source images".

Face detection and estimation process#

Below is the order of face detection in the image:

Basic steps for face images detection:

• Face detection. The bounding box of the face (height, width, "X" and "Y" coordinates) and five landmarks of the face are determined.
• Normalization. The image is centered in a certain way and cropped to the size of 250x250 pixels required for further work. Thus, all processed images look the same. For example, the left eye is always in a frame defined by some coordinates. Such a normalized image is called a sample. The sample is stored in the Image Store. After creating the sample, it is assigned a special identifier "sample_id", which is used for further image processing.

All objects created in LUNA PLATFORM are assigned identifiers. Sample — "sample_id", face — "face_id", event — "event_id", etc. Identifiers are the main way to transfer data between requests and services.

For more information about samples, see "Sample object".

• Estimation. The following face parameters are estimated:
  • Gaze direction (rotation angles for both eyes).
  • Head pose (yaw, roll and pitch angles).
  • Sixty-eight landmarks. The number of landmarks depends on which face parameters are to be estimated.
  • Image quality (lightness, darkness, blurriness, illumination and specularity).
  • Mouth attributes (overlap, presence of smile).
  • Mask presence (medical or tissue mask on the face, mask is missing, mouth is occluded).
  • Emotions (anger, disgust, fear, happiness, neutrality, sadness, surprise).
  • Others (see the "Face parameters" section).

If necessary, you can configure filtering by head pose angles.

Body detection and estimation process#

Below is the order of body image detection:

Basic steps for body images detection:

• Body detection. The bounding box of the body (height, width, "X" and "Y" coordinates) is determined.
• Normalization. The image is centered in a certain way and cropped to the size of 128x250 pixels required for further work (otherwise, the principle of operation is similar to the process of normalization of the face).
• Estimation. The following body parameters are estimated:
  • Upper body (presence and color of headwear, color of upper clothing).
  • Sleeve length (long sleeves, short sleeves, unknown).
  • Lower body (type of lower garment — trousers, shorts, skirt, unknown; color of lower garment, presence and color of shoes).
  • Accessories.
  • Others (see the "Body parameters" section).

Services interaction during detection#

Below is a diagram of the interaction of services when performing detection.

Requests to perform detection#

Below are the main requests where you can perform face or body detection.

Requests "create handler" and "generate events"#

These requests are used in the approach "Parallel performing of requests".

• Request "create handler":

• Request "generate events":

It is possible to use the "multipart/form-data" schema in the request. Using a schema enables you to send multiple images at the same time, specify additional information for an event, and more. See "Use "multipart/form-data" schema when generating event" for more information.

Request "detect faces"#

The "detect faces" request is used in the approach "Sequential performing of requests". This request cannot be used for the body image.

In the request, you can send an image or specify the URL to the image. You can use a multipart/form-data schema to send several images in the request. Each of the sent images must have a unique name. The content-disposition header must contain the actual filename.

You can also specify a bounding box parameters using the multipart/form-data schema. It enables you to specify the certain zone with a face on the image. You can specify the following properties for the bounding box:

• top left corner "x" and "y" coordinates,
• height,
• width.

Request "sdk"#

The "sdk" request also allows face or body detection on the source image. Unlike other requests, its data cannot be used in the future. For more information, see "Sdk resource".

Requests to perform estimation#

The estimation of the parameters of faces and bodies is performed in conjunction with detection. Most often, the parameters for estimation have a name like estimate_<estimation_name>. For example, estimate_glasses.

The "Estimated data" section contains a list of all possible parameters that can be estimated in LUNA PLATFORM 5.

Detection results#

The following information is returned in responses to detection requests:

• address to the stored face/body sample
• id of the face/body sample
• coordinates of the bounding box of the face/body
• five landmarks of the face

All this data is used for further extraction and matching.

This information is supplemented depending on the enabled face/body estimation parameters.

You can read more about the detection of faces, bodies and the Remote SDK service in the "Remote SDK service" section.

Redetection#

If necessary, you can specify detections (coordinates "x", "y", "width", "height") rather than images in the request bodies. For example, using the "application/json" schema of the "detect faces" request, you can specify an array of detections in the "face_bounding_boxes" field.

If you explicitly specify detections in the request bod instead of an image, you can indicate to LUNA PLATFORM whether repeated detection (redetection) should be performed or whether the specified detection can be considered trusted. It is assumed that trusted detections were obtained using VisionLabs algorithms. Marking a third-party detection as trusted may affect the estimation results.

You can indicate whether the detection is trusted using the "application/json" scheme (the "trusted_detections" parameter) or the "multipart/form-data" scheme (the "X-Luna-Trusted-Detections" header) in the following requests, where detections are explicitly specified instead of images:

• "detect faces"
• "sdk"
• "iso"
• "generate events"
• "perform verification"
• "generate stream events (beta)"

Extraction#

In LUNA PLATFORM, the word "extraction" means:

• Extraction of descriptors and basic attributes (gender, age, ethnicity) of faces.
• Extraction of descriptors of bodies.

It is also estimate to determine a person's gender and age from the body image, but this method of determination is less accurate and occurs at the estimation stage. In addition, it is recommended to estimate the basic attributes of the body together with the extraction of the basic attributes of the face.

Descriptors are sets of characteristics read from faces or bodies in images. Descriptors requires much less memory than the source image.

It is impossible to restore the source image of a face or body from the descriptor due to personal data security reasons.

Descriptors are used to match faces to faces and bodies to bodies. It is impossible to match two faces or bodies in the absence of extracted descriptors. For example, if you want to match a face from a database with an input face image, you need to extract the descriptor for that image.

Descriptor is extracted using a neural network located in the Remote SDK container.

Detailed information about neural networks is described in the "Neural networks" section.

See "Descriptor" section for more information on descriptors.

Extraction process#

Data extraction requires the following information obtained during detection:

• Sample of the face or body.
• Coordinates of the bounding box for the face or body.
• Five landmarks of the face.

The following is the order in which the extraction is performed:

* In most extraction requests, the descriptor is saved to the database without being returned in the response body (see "Requests to perform extraction").

Specifics of extracting data from faces#

The descriptor and basic attributes of the face derived from the same image are stored in the database as an attribute object. This object may include both descriptor and basic attributes, or only one of these entities.

It is possible to extract descriptor and basic attributes using several samples of the same face at the same time. In this way, an aggregated descriptor and aggregated basic attributes can be obtained. The accuracy of matching and extraction of basic attributes by means of the aggregated descriptor is higher. Aggregation should be used when working with images received from webcams.

Aggregated descriptor can be obtained from images, but not from already created descriptors.

Detailed information about aggregation is described in the "Aggregation" section.

Temporary attributes#

After extracting the attributes using the appropriate requests (see "Requests to perform extraction"), the extracted data is stored in the Redis database as temporary attributes.

Temporary attributes have a TTL (time to live) and are removed from the database after a specified period. In the request, you can specify a period from 1 to 86400 seconds. The default TTL is 5 minutes.

You can get information about a temporary attribute by its identifier until its TTL expires.

In order to save the attributes to the database, you need to create a face object and link the attributes to it. The face is created either by a separate request after the attributes are retrieved (the face is stored in the Faces DB) or during the generation of the event (the face is saved in the Faces DB or in the Events DB). You can create multiple faces and combine them into a list that can be used for matching.

See "Face object" and "List object" for details.

You can store attributes in an external database and use them in LP only when required. See "Create objects using external data".

For details on attributes, see "Attribute object".

Specifics of extracting data from bodies#

There are no attributes for bodies. All body information can only be stored as an event object. Accordingly, the extracted data is not stored in the Redis database. Information about bodies can only be saved to the database of the Events service when the corresponding option is enabled.

Just like faces, events can be aggregated. Detailed information about aggregation is described in the "Aggregation" section.

Services interaction during extraction#

Below is a diagram of the interaction of services when performing extraction.

Requests to perform extraction#

Below are the main requests where you can extract data from images of faces or bodies.

Requests "create handler" and "generate events"#

These requests are used in the approach "Parallel performing of requests".

• Request "create handler":

• Request "generate events":

It is possible to use the "multipart/form-data" schema in the request. Using a schema enables you to send multiple images at the same time, specify additional information for an event, and more. See "Use "multipart/form-data" schema when generating event" for more information.

Request "extract attributes"#

The "extract attributes" request is used in the "Sequential performing of requests" approach. This request cannot be used for body sample.

To save information in the Faces database, you need to link attributes to a face using the "create face" request. Otherwise, after the TTL expires, the face's temporary attributes will be removed from the Redis database. It is not possible to save a face to the Events database using the "extract attributes" + "create face" request combination. To do this, you need to use the "Parallel performing of requests" approach (see above).

Request "sdk"#

The "sdk" request also enables you to extract basic attributes or descriptor. This request returns face or body descriptor in a specific format that can be used when matching "raw" descriptors. See "Sdk resource" for details.

Extraction results#

The following information is returned in responses to extraction requests:

• Identifier of the temporary face attribute (not returned if the "store_attribute" parameter of the "attribute_policy" is disabled in the handler).
• Address to the temporary face attribute stored in the Redis database (not returned if the "store_attribute" parameter of the "attribute_policy" is disabled in the handler).
• Face basic attributes.
• Face/body descriptor quality.
• Set of identifiers of samples for which the extraction was performed.

Matching#

Given the presence of descriptors, LP enables you to search for similar faces or bodies in the database by comparing the given descriptor with descriptors stored in the Redis, Faces, Events databases or with descriptors passed directly.

The matching is done using the Python Matcher service. In response to the matching, the degree of similarity is returned, the value of which lies in the range from 0 to 1. A high degree of similarity means that two descriptors belong to the same person.

There are two types of descriptors — face descriptor and body descriptor. Body descriptor matching is not as accurate as face descriptor matching. With a large number of candidate events, the probability of false determinations of the best matches is higher than with a large number of candidate faces.

The sources of matched objects are represented as references (the objects you want to compare) and candidates (the set of objects that you want to compare with). Each reference will be matched with each of the specified candidates.

Matching cannot be performed if there is no descriptor for any of the compared faces.

You can select the following objects as references and candidates.

References:

• Attributes
• Faces
• External IDs of faces and events
• Events (face or body)
• Event track IDs
• Descriptors

Candidates:

• Faces
• Events (face or body)
• Attributes
• Descriptors

The references are specified using IDs of the corresponding objects. If a non-existent reference is set (for example, a non-existent ID is set in the "event_id" field or the "face_id" field), the corresponding error is returned.

The candidates are specified using filters. Matching results are returned for the candidates that correspond to the specified filters. If none of the candidates corresponds to the filters (for example, a non-existent ID is set in the "event_ids" field or the "face_ids" field), there will be no matching result and no error returned. The result field will be empty.

Matching process#

Below is the process of performing matching using faces as candidates and references:

* Descriptors can also be passed directly using "raw matching" request.

Service interaction during matching#

Below is a diagram of the interaction of services when performing matching.

See "Matching diagrams" for more service interaction scenarios when performing a matching.

Filtering of matching results#

The results of the matching can be filtered. To filter, you need to specify the appropriate request parameters.

Several examples for events and faces filters are given below.

For events filtration one can:

• Specify camera ("source" field) and a period ("create_time__gte" and "create_time__lt" or "end_time__gte" and "end_time__lt" fields).
• Specify tags ("tags" field) for events as filters and perform matching for events with these tags only.
• Specify geographical area and perform matching with the events created in this area only. You can specify direct location (city, area, district, street, house) or a zone specified using geographical coordinates.

For faces filtration one can:

• Specify a list of external face IDs to perform matching with them.
• Specify a list ID to perform matching by list.

See the "matcher" > "matching faces" and "matcher" > "human body matching" sections in the API service reference manual for details about all the available filters.

Requests to perform matching#

Below are the main requests where descriptors can be matched.

Requests "create handler" and "generate events"#

These requests are used in the approach "Parallel performing of requests".

• Request "create handler":

• Request "generate events":

It is possible to use the "multipart/form-data" schema in the request. Using a schema enables you to send multiple images at the same time, specify additional information for an event, and more. See "Use "multipart/form-data" schema when generating event" for more information.

Requests "matching faces" and "matching bodies"#

The "matching faces" and "matching bodies" requests are used in the "Sequential performing of requests".

Request "raw matching"#

You can set descriptors as references and candidates in SDK, Raw and XPK files using the "raw matching" request.

All the descriptors data is provided with the request hence you can use this request when you need to match descriptors that are not stored in LUNA PLATFORM databases.

In LUNA PLATFORM, it is possible to extract descriptor in SDK format (see the "Descriptor formats" section).

Matching results#

The following information is returned in responses to match requests:

• information about the reference
• information about the candidate
• the degree of similarity of each candidate with the reference
• filtered candidates

For more information about descriptor matching services, see the "Matching services" section.

Verification#

You can use the "/verifiers" resource to create a special handler for verification. It includes several policies for the incoming images processing. See the "Verifiers description" section for details on the verifier.

Generally this request is used to match one given object with the incoming object. Use other matching requests for identification tasks.

Matching a large set of descriptors#

Sometimes it is necessary to match a very large set of descriptors (candidates). When matching a large set of descriptors by classical brute-force matching, it is impossible to get a low latency with a high number of requests per second. Therefore, it is required to use approximation methods implemented in LIM that exchange some accuracy for high speed. These methods speed up the matching by building an index containing preprocessing data.

The work of the module is as follows: The index is created using a task containing a "list_id" with linked faces, by which the matching will be made. You can also not specify "list_id", but set the settings for automatic indexing of all lists whose number of faces exceeds the value specified in a certain setting. After the index is created, the user sends a request to the API service, which redirects it to the Python Matcher Proxy service. The Python Matcher Proxy service determines where the matching will be performed — in the Python Matcher service or in the Indexed Matcher service. After matching, the response is returned to the user.

LUNA Index Module is licensed separately and is available on request to VisionLabs. The module delivery contains all the necessary documentation and the Docker Compose script.

Stored data and LP objects#

This section describes data stored by LUNA PLATFORM 5.

This information can be useful when utilizing LUNA PLATFORM 5 according to the European Union legal system and GDPR.

This section does not describe the legal aspects of personal data utilization. You should consider which stored data can be interpreted as personal data according to your local laws and regulations.

Note that combinations of LUNA PLATFORM data may be interpreted by law as personal data, even if data fields separately do not include personal data. For example, a Face object including "user_data" and descriptor can be considered personal data.

Objects and data are stored in LP upon performing certain requests. Hence it is required to disable unnecessary data storage upon the requests execution.

It is recommended to read and understand this section before making a decision on which data should be received and stored in LUNA PLATFORM.

This document considers usage of the resources listed in the OpenAPI specification and creation of LUNA PLATFORM 5 objects only. The data created using Backport 3 and Backport 4 services is not considered in this section.

Source images#

Photo images are general data sources for LP. They are required for samples creation and Liveness check.

You can provide images themselves or URLs to images.

After normalization of the source images, samples are saved in JPG format by default, even if the source image is sent in PNG format. If necessary, you can change the format of the stored samples using the "default_image_extension" setting of the Image Store service.

It is not recommended to send rotated images to LP as they are not processed properly and should be rotated. You can rotate the image using LP in two ways:

• By enabling the "use_exif_info" auto-orientation parameter by EXIF data in the query parameters.
• By enabling the "LUNA_HANDLERS_USE_AUTO_ROTATION" auto-orientation setting in the Configurator settings.

More information about working with rotated images can be found in the Nuances of working with services section.

See the detailed information about the requirements for the source images in the "Image format requirements" section.

Source images usage#

Source images can be specified for processing when performing POST requests on the following resources:

• "/sdk"
• "/detector"
• "/handlers/{handler_id}/events"
• "/verifiers/{verifier_id}/verifications"
• "/liveness"

Source images saving#

Generally, it is not required to store source images after they are processed. They can be optionally stored for system testing purposes or business cases, for example, when the source image should be displayed in a GUI.

Source images can be stored in LP:

• Using the POST request on "/images" resource.
• During the the POST request on "/handlers/{handler_id}/events" resource. Source images are stored if the "store_image" option is enabled for "image_origin_policy" of the "storage_policy" during the handler creation using the "/handlers" resource.

Samples are stored in buckets of the Image Store for an unlimited time.

Samples are stored in buckets:

• "visionlabs-samples" is the name of the bucket for faces samples.
• "visionlabs-bodies-samples" is the name of the bucket for human bodies samples.

Paths to the buckets are specified in the "bucket" parameters of "LUNA_IMAGE_STORE_FACES_SAMPLES_ADDRESS" and "LUNA_IMAGE_STORE_BODIES_SAMPLES_ADDRESS" sections in the Configurator service.

Saving metadata with an image

In the resource "/images", user metadata can be saved along with the image using the header X-Luna-Meta-<user_defined_key> with the value <user_defined_value>. In the Image Store bucket, metadata is saved in a separate file <image_id>.meta.json, which is located next to the source image.

In the response to the "get image" request, you need to specify the header with_meta=1 to get the image metadata in the response header.

To store metadata values for multiple keys, multiple headers must be set.

Source images deletion#

Source images are stored in the bucket for an unlimited period.

You can delete source images from the bucket:

• Using the DELETE request on the "/images/{image_id}" resource.
• Manually, by deleting the corresponding files from the bucket.

Sample object#

Samples usage#

Separate samples are created for face and body.

Samples are required:

• For basic attributes estimation
• For face, body and image parameters estimation
• For face and body descriptors extraction
• When changing the descriptors NN version

When the neural network is changed, you cannot use the descriptors of the previous version. A descriptor of a new version can be extracted if the source sample is preserved only.

Samples can also be used as avatars for faces and events. For example, when it is required to display the avatar in GUI.

Samples are stored in buckets of the Image Store.

Samples should be stored until all the required requests for face, body parameters estimation, basic attributes estimation, and descriptors extraction are finished.

Samples creation and saving#

Samples are created upon the face and/or human body detection in the image. Samples are created during the execution of the following requests:

• POST on the "/detector" resource. Samples are created and stored implicitly. The user does not affect their creation.
• POST on the "/handlers/{handler_id}/events" resource. Samples are stored if the "store_sample" option is enabled for "face_sample_policy" and "body_sample_policy" of the "storage_policy" during the handler creation using the "/handlers" resource.
• POST on the "/verifiers/{verifier_id}/verifications" resource. Samples are stored if the "store_sample" option is enabled for "face_sample_policy" of the "storage_policy" during the verifier creation using the "/verifiers" resource. Human body samples are not stored using this resource.

External samples saving

Samples can be provided to LP directly from external VisionLabs software (e. g., FaceStream). The software itself performs sample creation from the source image.

You can manually store external samples in LP (an external sample should be provided in the requests):

• Using the POST request on the "/samples/{sample_type}" resource.
• When the "warped_image" option is set for the POST request on the "/detector" resource.
• When the "image_type" option is set to "1" ("face sample) or "2" (human body sample) in the query parameters for the POST request on the "/handlers/{handler_id}/events" resource. The "store_sample" option should be enabled for "face_sample_policy" and "body_sample_policy" of the "storage_policy".

Samples saving disabling#

There is no possibility to disable samples saving for the request on the "/detector" resource. You can delete the created samples manually after the request execution.

The "/handlers" resource provides "storage_policy" that allows you to disable saving following objects:

• Face samples. Set "face_sample_policy" > "store_sample" to "0".
• Human body samples. Set "body_sample_policy" > "store_sample" to "0".

The "/verifiers" resource provides "storage_policy" that allows you to disable saving the following objects:

• Face samples. Set "face_sample_policy" > "store_sample" to "0".

Samples deletion#

You can use the following ways to delete face or body samples:

• Perform the DELETE request to "/samples/faces{sample_id}" resource to delete a face sample by its ID.
• Perform the DELETE request to "/samples/bodies{sample_id}" resource to delete a body sample by its ID.
• Manually delete the required face or body samples from their bucket.
• Use "remove_samples" parameter in the "gc task" when deleting events.

Getting information about samples#

You can get face or body sample by ID.

• Perform the GET request to "/samples/faces/{sample_id}" resource to get a face sample by its ID.
• Perform the GET request to "/samples/bodies/{sample_id}" resource to get a body sample by its ID.

If a sample was deleted, then the system will return an error when making a GET request.

Descriptor#

The basic description and use of descriptors is described in the "Extraction" section above.

See the additional information in the "Descriptor formats" section.

Attribute object#

This object is intended to work with faces only.

It is recommended to introduce yourself with the "Extraction" section before reading.

Attributes are temporary objects that include basic attributes and face descriptors. This data is received after the sample processing.

Basic attributes include the following personal data:

• age. The estimated age is returned in years.

• gender. The estimated gender: 0 — female, 1 — male.

• ethnicity. The estimated ethnicity.

You can disable basic attributes extraction to avoid this personal data storage.

A descriptor cannot be considered personal data. You cannot restore the source face using a descriptor.

Attributes usage#

Attribute object can be used in the following cases:

• Matching using attributes can be performed using:

  • "/matcher/faces" resource
  • "/tasks/cross_match" resource
  • "/verifiers/{verifier_id}/verifications" resource

As attributes have TTL (by default, it is set to 5 minutes), it is convenient to use them for verification or identification purpose. They will be deleted soon after you receive the result.

• Attributes may be used for ROC-curves creation using the "/tasks/roc" resource (see the "ROC-curve calculating task" section).

• You can save the data of the existing attribute to a face using the "/faces" resource.

It is not the only way to save descriptors and basic attributes to a face. You can use "/handlers/{handler_id}/events" to create a face and add the extracted descriptor and basic attributes to it.

Basic attributes saved in faces or events objects can be used for filtration of the corresponding objects during requests execution.

Descriptors are required for matching operations. You cannot compare two faces or bodies without their descriptors.

Attributes creation and saving#

Attributes can be created when sending requests on the following resources:

• "/extractor"

The "extract_basic_attributes" and "extract_descriptor" query parameters should be enabled for extraction of the corresponding data. Attributes are implicitly stored after the request execution.

• "/handlers/{handler_id}/events"

The "extract_basic_attributes", "extract_face_descriptor", and "extract_body_descriptor" parameters should be enabled in the handler for extraction of the corresponding data. The "storage_policy" > "attribute_policy" > "store_attribute" option should be enabled in the handler for attributes storage.

• "/verifiers/{verifier_id}/verifications"

The "extract_basic_attributes" parameter should be enabled in the verifier for extraction of the corresponding data. The "storage_policy" > "attribute_policy" > "store_attribute" option should be enabled in the verifier for attributes storage.

Attributes can be created using external descriptors and external basic attributes using the "/attributes" resource.

Attributes time to live#

Attributes have a TTL. After the TTL expiration, attributes are automatically deleted. Hence, it is not required to delete attributes manually.

The default TTL value can be set in the "default_ttl" parameter in the Configurator service. The maximum TTL value can be set in the "max_ttl" parameter in the Configurator service.

TTL can be directly specified in the requests on the following resources:

• "/extractor" in the "ttl" query parameter
• "/handlers" in the "storage_policy" > "attribute_storage" in the "ttl" parameter

Attributes extraction disabling#

You can disable basic attributes extraction by setting the "extract_basic_attributes" parameter to "0" in the following resources:

• "/extractor"
• "/handlers"
• "/verifiers"

You can disable descriptor extraction by setting the "extract_descriptor" parameter to "0" in the following resources:

• "/extractor"
• "/handlers"

Attributes saving disabling#

You can disable the "storage_policy" > "attribute_policy" > "store_attribute" parameter in the "/handlers" resource to disable attributes storage. When this handler is used for the "/handlers/{handler_id}/events" resource, attributes are not saved even for the specified TTL period.

You can disable the "storage_policy" > "attribute_policy" > "store_attribute" parameter in the "/verifiers" resource to disable attributes storage. When this verifier is used for the "/verifiers/{verifier_id}/verifications" resource, attributes are not saved even for the specified TTL period.

Attributes deletion#

Attributes are automatically deleted after the TTL expiration.

Perform the DELETE request to "/attributes/{attribute_id}" resource to delete an attribute by its ID.

Getting information about attributes#

You can receive information about existing temporary attributes before TTL has not expired.

• Perform the GET request to "/attributes/{attribute_id}" resource to receive information about a temporary attribute by its ID.
• Perform the GET request to "/attributes" resource to receive information about previously created attributes by their IDs.
• Perform the GET request to "/attributes/{attribute_id}/samples" resource to receive information about all the temporary attribute samples by the attribute ID.

If any attribute TTL has not expired, the attribute data is returned. Otherwise, no data is returned for this attribute in the response.

Face object#

Faces are changeable objects that include information about a single person.

The following general data is stored in the face object:

• Descriptor ("descriptor")
• Basic attributes ("basic_attributes")
• Avatar ("avatar")
• User data ("user_data")
• External ID ("external_id")
• Event ID ("event_id")
• Sample ID ("sample_id")
• List ID ("list_id")
• Account ID ("account_id")

See the "Faces database description" section for additional information about the face object and the data stored in it.

Attributes data can be stored in a face. Basic attributes data, descriptor data, and information about samples are saved to the Faces database and linked with the face object.

When you delete a face, the linked attributes data is also deleted.

• Descriptor. You should specify a descriptor for the face if you are going to use the face for comparison operations.

You cannot link a face with more than one descriptor.

• Basic attributes: Basic attributes can be used for displaying information in GUI.

The face object can also include IDs of samples used for the creation of attributes.

General event fields description:

• "user_data". This field can include any information about the person.

• "avatar". Avatar is a visual representation of the face that can be used in the user interface.

This field can include a URL to an external image or a sample that is used as an avatar for the face.

• "external_id". The external ID of the face.

You can use the external ID to work with external systems.

You can use the external ID to specify that several faces belong to the same person. You should set the same external ID to these faces upon their creation.

You can get all the faces that have the same external ID using the "faces" > "get faces" request.

• "event_id". This field can include an ID of the event that gave birth to this face.

• "list_id". This field can include an ID of the list to which the face is linked.

A face can be linked to one or several lists.

• "account_id" — the account ID to which the face belongs.

• "sample_id" — One or several samples can be linked to a face. It should be the samples used for attributes extraction. All the samples must belong to the same person. If no samples are save for a face, you cannot update its descriptor to a new NN version.

IDs do not usually include personal information. However, they can be related to the object with personal data.

Faces usage#

Faces usually include information about persons registered in the system. Hence, they are usually required for verification (the incoming descriptor is compared with the face descriptor) and identification (the incoming descriptor is compared with several face descriptors from the specified list) purposes.

If there are no faces stored in your system, you cannot perform the following operations with these objects:

• Matching by faces and lists, when faces are set as candidates or references in the request to the "/matcher/faces" resource.
• Matching by faces and lists, when faces are set as candidates in the matching policy of the request to the "/handlers" resource.
• Cross-matching tasks, when faces are set as candidates or references in the request to the "/tasks/cross_match" resource.
• Clustering tasks, when faces are set as filters for clustering in the request to the "/tasks/clustering" resource.
• Verification requests, when faces IDs are set in query parameters in the request to the "/verifiers/{verifier_id}/verifications" resource.
• ROC curves creation tasks ("/tasks/roc" resource).

Faces creation and saving#

Faces can be created using the following requests:

• "create face"

You can specify attributes for the face using one of several ways:

• By specifying the attribute ID of the temporary attribute.
• By specifying descriptors and basic attributes (with or without samples).
• By specifying descriptors (with or without samples).
• By specifying basic attributes (with or without samples).

The last three ways are used when you need to create a face using data stored in external storage.

• "generate events"

The "storage_policy" > "face_policy" > "store_face" parameter should be enabled in the utilized handler.

Requests for working with faces#

Below are the requests for working with faces:

List object#

A list is an object that can include faces that correspond to a similar category, for example, customers or employees.

Lists include the "description" field that can store any required data to distinguish lists from each other.

Lists usage#

You can add faces to lists for dividing the faces into groups.

Only faces can be added to lists.

Lists IDs are usually specified for matching operations as a filter for faces.

Requests for working with lists#

Below are the requests for working with lists:

Event object#

Events are used in the "Parallel performing of requests" approach.

Events are immutable objects that include information about a single face and/or body. They are received after images processing using handlers or created manually.

Unlike faces, events cannot be changed after creation. The only exception is the event descriptor. It can be updated to the new neural network version.

Generally an event is created for each face/body detected in the image. If the detected face and body belong to the same person they are saved to the same event.

LP also provides possibility to create new events manually without processing using handlers. It is used in the cases when a logic for filling in event fields should differ from the logic using handlers. For example, when you want to extract descriptors only for a part of the detections and not for all the detections.

An event can be linked with a descriptor stored in the Events database.

Since there can be millions of events in the Events database, it is recommended to use a high performance column database. You can also use a relational database, but this will require additional configuration.

The following general data in stored in the event object:

• Field "source". This field can include the source from which the face or human body was received. The value is specified during the "generate events" request execution.

• "location". This group of parameters includes information about the location where the event occurred. The values are specified during the "generate events" request execution. The following fields are included in this group:

  • "city"
  • "area"
  • "district"
  • "street"
  • "house_number"
  • "geo_position" — latitude and longitude.

• Field "tag". This field includes a tag (or several tags) applied during the "conditional_tags_policy" execution. The tag(s) can be specified during the "create handler" request execution or the "generate events" request execution.

• Field "emotion". This field includes the predominant emotion estimated for the face. If necessary, you can disable the "estimate_emotions" parameter in the "/handlers" resource or "create verifier" request.
• Field "insert_time". This field includes the time when the face appeared in the video stream. This data is usually provided by external systems.
• Field "top_similar_object_id". This field includes the top similar object ID.
• Field "top_similar_external_id". This field includes the external ID of the top similar candidate (event or face) with which the face is matched.
• Field "top_matching_candidates_label". This field includes a label applied during the "match_policy" execution. The labels are specified in this policy in the "/handlers" resource.
• Field "face_id". Events include the ID of the face to which the event gave birth.
• Field "list_id". Events include the ID of the list to which the created face was attached.
• Field "external_id". This external ID will be specified for the face created during the event request processing. The value is specified during the "generate events" request execution.
• Field "user_data". This field can include any information. User data will be specified for the face created during the event request processing. The value is specified during the "generate events" request execution.
• Fields "age", "gender", and "ethnic_group". The basic attributes extraction is specified in the "extract_policy" of the "/handlers" resource.
• Face and body descriptors. Events can be used for matching operations as they include descriptors.
• Information about faces, bodies and events that were used for matching with the events.
• Information about faces and human bodies detection results that include:
• IDs of created samples
• Information about bounding boxes of detected faces/bodies
• Time of face/body event detection (field "detect_time"). The time is returned from an external system.
• URL of the source image where the face/body was detected ("image_origin")

IDs do not usually include personal information. However, they can be related to the object with personal data.

See the "Events database description" section for additional information about the event object and the data stored in it.

Attributes aggregation for events

It is possible to enable attributes aggregation for the incoming images when creating an event.

According to the specified settings faces and bodies are detected in the incoming images. If faces are detected and an aggregated descriptor creation is specified, then a single descriptor is created using all the found faces. The same logic is used for the aggregated body descriptor creation. In addition, the basic attributes are aggregated, the obtained values for the definition of Liveness, masks, emotions, upper and lower body and the body accessories.

All the information about the detected face/body and estimated properties is returned in the response separately for each image. The aggregated results are stored when an event is saved in the DB. The information about face/body detection is added to the Events database as a separate record for each image from the request.

When performing aggregation, the images in the request to the "/handlers/{handler_id}/events" resource should include only one face/body and the face/body should belong to the same person.

Events usage#

Events are required to store information about persons' occurrences in a video stream for further analysis. An external system processes a video stream and sends frames or samples with detected faces and bodies.

LP processes these frames or samples and creates events. As events include statistics about face detection time, location, basic attributes, etc., they can be used to collect statistics about the person of interest or gather general statistics using all the saved events.

Events provide the following features:

• Matching by events

As events store descriptors, they can be used for matching. You can match the descriptor of a person with the existing events to receive information about the person's movements.

To perform matching, you should set events as references or candidates for matching operations (see section "Descriptors matching").

• Notifications via web sockets

You can receive notifications about events using web sockets. Notifications are sent according to the specified filters. For example, when an employee is recognized, a notification can be sent to the turnstile application and the turnstile will be opened.

See the "Sending notifications via Sender" section.

• Statistics gathering

You can gather statistics on the existing events using a special request. It can be used to:

• Group events by frequency or time intervals.
• Filter events according to the values of their properties.
• Count the number of created events according to the filters.
• Find the minimum, maximum and average values of properties for the existing events.
• Group existing events according to the specified values of properties.

You should save generated events to the database if you need to collect statistics after the event is created.

See the "events" > "get statistics on events" request in "APIReferenceManual.html" for details.

• User defined data

You can add custom information to events, which can later be used to filter events. The information is passed in JSON format and written to the Events database.

See "Events meta-information" for details.

Events creation and saving#

Events are created during the request to the "/handlers/{handler_id}/events" resource execution. The "event_policy" > "store_event" should be set to "1" when creating a handler using the "/handlers" resource.

For manual event creation and saving use the "/handles/{handler_id}/events/raw" resource.

The format of the generated event is similar to the format returned by the "/handlers/{handler_id}/events" resource. The "event_id" and "url" fields are not specified when creating a request. They are returned in the response after the event is created.

Notifications using web sockets are sent when events are created using this resource.

Events deletion#

Events are only deleted using the garbage collection task. You should specify filters to delete all the events corresponding to the filters.

To delete an event execute the POST request to the "/tasks/gc" resource.

You can also manually delete the required events from the database.

Getting information about events#

You can receive information about existing events.

• The "events" > "get event" request enables you to receive information about an event by its ID.

If the event was deleted, then the system will return an error when making a GET request.

• The "events" > "get events" request enables you to receive information about all previously created events according to the specified filters. By default, events are filtered for the last month from the current date. If any of the following filters are specified, then the default filtering will not be used.

  • List of event IDs (field "event_ids")
  • Lower event ID boundary (field "event_id__gte")
  • Upper event ID boundary (field "event_id__lt")
  • Lower create time boundary (field "create_time__gte")
  • Upper create time boundary (field "create_time__lt")

Use "multipart/form-data" schema when generating event#

In the "generate events"request, you can send an image, provide an image URL, or send a raw descriptor. You can use a multipart/form-data schema to send several images in the request. Each of the sent images must have a unique name. The Content-Disposition header must contain the actual filename.

The following parameters can also be specified in the request using the "multipart/form-data" schema:

• Parameters of the bounding box of the face or body. It enables you to specify the certain zone with a face on the image
• Image detection time
• Time stamp of detection relative to the beginning of videofile
• Source image
• User meta-information

All of the above information will be saved in the event when it is generated.

Handler object#

Handlers are used in the "Parallel performing of requests" approach.

Handler is an object that stores entry points for image processing. The entry points are called policies. They describe how the image is processed hence define the LP services used for the processing. The handler is created using the "create handler" request.

When creating a handler, it is important to ensure that only the required policies are enabled. Unnecessary policies can take into account request execution time and create gigabytes of unnecessary data on disk.

Handler policies

The table below includes all the existing handler policies. Each policy corresponds to one of the LP services listed in the "Service" column.

Handler policies

You can skip or disable policies if they are not required for your case. Skipped policies are executed accorded to the specified default values. For example, you should disable samples storage in the corresponding policy if they are not required. If you just skip the "storage_policy" in the handler, samples will be saved according to the default settings.

All the available policies are described in the "API service reference manual".

Output filtering

Many policies have filters. Filters can be specified either explicitly in the "filters" field or in parameters with the name "‹estimation›_states". If filtering is performed on the first policy, it will cause subsequent policies to stop executing as they are executed sequentially. In this case, the event will not be saved in the database, and the filtered objects will be displayed in the "filtered_detections" subblock in the response body of the event generation request.

Handlers usage

Using handler you can:

• Specify face/body detection parameters and estimated face/body parameters (see "Source image processing and samples creation").
• Specify human body detection execution.
• Enable basic attributes and descriptors extraction (see "Descriptor extraction and attribute creation").
• Perform descriptors comparison (see "Descriptors matching") to receive matching results and use the result as filters for the policies.
• Configure automatic creation of faces using filters. You can specify filters according to the estimated basic attributes and matching results.
• Configure automatic attachment of the created faces to the lists using filters. You can specify filters according to the estimated basic attributes and matching results.
• Specify the objects to be saved after the image processing.
• Configure automatic tag adding for the event using filters. You can specify filters according to the estimated basic attributes and matching results.

In addition, the ability to process a batch of images using a handler is available (see the "Estimator task" section).

There are three types of handlers:

• static
• dynamic
• lambda

Static handler#

Handler parameters of this type are specified when it is created, and then when generating an event or creating an Estimator task, the created handler ID is specified.

Dynamic handler#

Handler parameters of this type are specified when generating an event or creating an Estimator task. Dynamic handlers enable you to separate technical parameters (thresholds and quality parameters that need to be hidden from frontend application users) and business logic. Dynamic handler policies are set using the "multipart/form-data" scheme in the event generation request.

Only static verifiers are available.

Dynamic handlers usage example:

You need to separate head pose thresholds from other handler parameters.

You can save the thresholds to your external database and implement the logic of automatic substitution of this data when creating events (e. g. in your frontend application).

The frontend user sends requests for events creation and specifies the required lists and other parameters. The user does not know about the thresholds and cannot change them.

Lambda handler#

This handler is used when working with the Lambda service, which is intended to work with custom modules that mimic the functionality of a separate service.

All requests will be sent to the lambda handler using "lambda_id".

If the custom Handlers-lambda supports the ability to generate an event, then the format of the body of the request and response to the resources is "/handlers/{handler_id}/events" or "/tasks/estimator" will follow the OpenAPI specification. If Handlers-lambda does not support the ability to generate an event, then a request to the resource "/lambdas/{lambda_id}/proxy" should be used with the corresponding request form.

The logic behind the request behavior is entirely dependent on the lambda written by the user.

See "Lambda service" for details.

Requests for working with handlers#

Below are the requests for working with handlers:

Verifier object#

Verifiers are used in the "Parallel performing of requests" approach.

The Handlers service also processes requests to create verifiers required for the verification process. They are created using the "create verifier" request.

The verifier contains a limited number of handler policies — detect_policy, extract_policy, and storage_policy.

You can specify the "verification_threshold" in the verifier.

The created verifier should be used when sending requests to:

• "/verifiers/{verifier_id}/verifications" resource. You can specify IDs of the objects with which the verification should be performed.
• "/verifiers/{verifier_id}/raw" resource. You can specify raw descriptors as candidates and references for matching. As raw descriptors are processed "verification_threshold" is the general parameter used from the specified verifier.

The response includes the "status" field. It shows, if the verification was successful or not for each pair of matched objects. It is successful if the similarity of two objects is greater than the specified "verification_threshold".

Requests for working with verifiers#

Below are the requests for working with verifiers:

Other objects#

In the Image Store service, you can store any files as objects (for example, a video file).

You can upload files using the "create objects" request. The bytes of the file must be specified in the request body, and the Content-Type header must contain the MIME type of the file (for example, video/mp4). The response to the "get object" request contains the Content-Disposition header. This header contains the file name of the attachment object (for example, video_1.mp4). The file name is generated based on the object_id and the MIME type.

If the MIME type of the file could not be determined, then the file extension will be set as .bin.

Sending notifications via Sender#

You can send notifications about created events to the third-party party applications via web sockets. For example, you can configure LP to send notifications about VIP guests' arrival to your mobile phone.

When LP creates a new event, the event is added to the special DB. Then the event can be sent to the Sender service if the service is enabled.

The third-party party application should be subscribed to Sender via web sockets. The filter for the events of interest should be set for each application. Thus you will receive notifications only for the required events.

The notification about a new event is sent by Sender to the required applications.

See section "Sender service" for details about the Sender service.

Licenses#

The Licenses service provides information about license terms to LP services.

See section "Licenses service" for details.

Admin#

The Admin service implements tools for administrative routines.

All the requests for Admin service are described in OpenAPI specification of the Admin service.

See section "Admin service" for details about the Admin service.

Configurator#

The configurator service includes the settings of all the LP services. Thus it provides the possibility to configure all the services in one place.

See section "Configurator service" for details about the Configurator service.

Tasks#

Tasks requests provide additional possibilities for large data amounts processing.

The greater data array is processed, the longer it takes to finish the task. When a task is created you receive the task ID as the result of the request.

See the "Tasks service" section for details about tasks processing.

Clustering task

The clustering task provides the possibility to group events and faces that belong to the same person to clusters.

You can specify filters for choosing the objects to be processed.

Using the clustering task you can, for example, receive all the IDs of events that belong to the same person and which occurred during the specified period of time.

See the "task processing" > "clustering task" request for details about the Clustering task request creation.

See "Clustering task" for more information about this task.

Reporter task

The reporter task enables you to receive a report in CSV format with extended information about the objects grouped to clusters.

You can select columns that should be added to the report. You can receive images corresponding to each of the IDs in each cluster in the response.

See the "task processing" > "reporter task" request for details about the reporter task request creation.

See "Reporter task" for more information about this task.

Exporter task

The exporter task enables you to collect event and/or face data and export them from LP to a CSV file.

The file rows represent requested objects and corresponding samples (if they were requested).

See the "task processing" > "exporter task" request for details about the exporter task request creation.

See "Clustering task" for more information about this task.

Cross-matching task

Cross-matching means that a large number of references can be matched with a large number of candidates. Thus every reference is matched with every candidate.

Both references and candidates are specified using filters for faces and events.

See the "task processing" > "cross-matching task" request in the API service reference manual for details about the cross-matching task creation.

See "Cross-matching task" for more information about this task.

Linker task

The linker task enables you to:

• link the existing faces to lists
• create faces from the existing events and link the faces to lists

The linked faces are selected according to the specified filters.

See the "task processing" > "linker task" request in the API service reference manual for details about the cross-matching task creation.

See "Linker task" for more information about this task.

Estimator task

The estimator task enables you to perform batch processing of images using the specified policies.

In the request body, you can specify the handler_id of an already existing static or dynamic handler. For the dynamic handler_id, the ability to set the required policies is available. In addition, you can create a static handler specifying policies in the request.

The resource can accept five types of sources with images for processing:

• ZIP archive
• S3-like storage
• Network disk
• FTP server
• Samba network file system

See the "task processing" > "estimator task" request in the API service reference manual for details about the estimator task creation.

See "Estimator task" for more information about this task.

Backports#

In LP 5, Backport is a mechanism of the new platform version mimicry to the older versions.

The backports are implemented for LUNA PLATFORM 3 and LUNA PLATFORM 4 by means of the Backport 3 and Backport 4 services respectively.

The backports enable you to send requests that are similar to the requests from LUNA PLATFORM 3 and LUNA PLATFORM 4 and receive responses in the appropriate format.

There are some nuances when using backports.

• Data migration to Backports is complicated.

  Database structures of LUNA PLATFORM 3 and LUNA PLATFORM 4 differ from the database structure of LUNA PLATFORM 5. Therefore, additional steps should be performed for data migration and errors may occur.

• Backports have many restrictions.

  Not all logic from LP 3 and LP 4 can be supported in Backports. See the "Restrictions for working with Backport services" section for more information.

• Backports are limited by available features and are not developed.

  New features and resources of LUNA PLATFORM 5 are not supported when using Backports and they will never be supported. Thus, these services should only be used if it is not possible to perform a full migration to LUNA PLATFORM 5 and no new features are required.

Use case:

For example, you have a frontend application that sends requests to LUNA PLATFORM 3.

When you use the Backport 3 service, the requests to LP 3 are received by the service. They are formatted and sent to LUNA PLATFORM 5 according to its API format. LP 5 processes the request and sends the response to Backport 3. The Backport 3 service formats all the received results to the LP 3 format and sends the response.

Restrictions for working with Backport services#

As the tables of the saved data and the data itself differs for LP versions there are features and restrictions on the execution of requests. The information about features and restrictions is given in "Backport 3" and "Backport 4" sections of this document.

You can use Backport service and API service of LP 5 simultaneously following these restrictions:

• When you use Backport services, it is strongly recommended not to use the same account for requests to Backport service and the API service of LUNA PLATFORM 5.

• Perform the admin requests that do not use account ID to the LUNA PLATFORM 5 only.

Sdk resource#

The sdk resource allows you to detect faces and/or human bodies and estimate attributes in input images. After the request is performed, the received data is not saved to the database and Image Store, it only returned in the response.

The sdk resource combines the capabilities of other resources and also provides additional options. For example, with "/sdk" request you can do the following features:

• estimate presence of glasses in the image;
• estimate liveness in the image;
• estimate face/body parameter(s);
• aggregate face/body parameter(s);
• create a face sample and return its descriptor in the SDK format encoded in Base64;
• create a human body sample and return its descriptor in the SDK format encoded in Base64;
• extract face and body descriptor(s) and return them in the response;
• set descriptor quality score threshold to filter out images that are poorly suited for further processing;
• filter by mask states;
• other.

Face or body descriptor in the SDK format encoded in Base64 can be used in requests for matching of "raw" descriptors.

Accounts, tokens and authorization types#

Account#

An account is required to delimit the visibility areas of objects for a particular user. The account is necessary to perform most of the requests. Each created account has its own unique "account_id". All account data is stored in the Accounts service database under this ID.

The account can be created using a POST request "create account" to the API service, or using the Admin service . When creating the account, you must specify the following data: login (email), password and account type.

Account type#

The account type determines what data is available to the user.

There are three types of accounts:

• user — the type of account with which you can create objects and use only your account data.
• advanced_user — the type of account for which rights similar to "user" are available, and there is access to the data of all accounts. Access to data from other accounts means the ability to receive data (GET requests), check their availability (HEAD requests) and perform comparison requests based on data from other accounts.
• admin — the type of account for which rights similar to "advanced_user" are available, and there is also access to the Admin service.

In the API service, you can work with all types of accounts, but only "advanced_user" and "user" types of accounts can be created, while in the Admin service you can create all three types.

By default, there is the "admin" type account in the system with the default login and password root@visionlabs.ai/root.

Using the header "Luna-Account-Id" in the "create account" request you can set the desired account ID. It should also be used if it is necessary to preserve the ability to work with data created in LP versions prior to 5.30.0 by specifying the "account_id" in the "Luna-Account-Id" header. Thus, using this parameter will link the old "account_id" to the account being created (See "Account migration" section in the LUNA PLATFORM upgrade manual for details on migration).

In response to a request to create the account, "account_id" is issued. After creating the account, you can use this ID for LunaAccountIdAuth authorization type or use BasicAuth authorization type (login/password authorization).

Token#

Token is linked to an existing account with any type ("user", "advanced_user", "admin") and enables you to impose extended restrictions on the requests being made. For example, when creating the token, you can give the user permission only to create and modify all lists and faces, or you can prevent the use of certain handlers by specifying their ID.

The token is created using the "create token" request to the API service.

An unlimited number of tokens can be created for each account. The token and all its restrictions are stored in the database and linked to the account by the "account_id" parameter. You cannot link one token to different accounts. To create tokens with the same permissions across different accounts, it is recommended to save the request body template for creating the account and use it.

There is no need to use both the token and the account. When working with tokens, you can restrict access to the "/accounts" resource using external means.

You can give additional restrictions on the token at any time using the "replace token" request or you can revoke the token using the "delete token" request. In this case, the token will be deleted from the database and can no longer be used for authorization.

When creating the token, you need to set the following parameters:

• expiration_time – expiration time of the token in the RFC 3339 format. You can specify an infinite token expiration time using the value "null"
• permissions – actions that the user can perform (see "Permissions set in token")

You can also specify whether the token is visible to other accounts data using the "visibility_area" parameter (see the "Viewing other account data" section).

The response to the request is "token_id" and a Base64 encoded JWT token. After creating the token, you can use the received JWT token for BearerAuth authorization type.

Permissions set in token#

The following permissions are available for the generated token:

Custom rights are intended to proxy LUNA Streams requests through the LUNA API. In other cases, using user rights will have no effect. See "Proxying requests to LUNA Streams via LUNA API" in the FaceStream administrator manual.

Resources "/iso", "/sdk" and "/liveness" do not require any authorization by default.

The value [] means no permissions.

The "emit_events" permission enables you to specify whether requests can be made to the "generate events" resource, as well as blacklisting or whitelisting handler IDs. If handler IDs are blacklisted, then only their use will be prohibited. If handler IDs are present in the white list, then only their use will be allowed. The maximum number of IDs in the lists is 100. When using the "emit_events" permission, the user must not have the "creation" and "modification" Rights to use the handler.

If the "emit_events" permission is granted, then all necessary objects will be created during the generation of the event, regardless of the permissions specified in the token. For example, the "faces" type permission regulates work with faces only in requests to "faces/*" resources, but does not affect the creation of the face during event generation. Thus, when using the handler with the "store_face" parameter and not having the "creation" permission for "faces", the face will still be created.

When setting permissions, the following features should be taken into account:

• The "modification" permission means performing of PATCH and PUT requests.
• Linking/unlinking face to list is the "modification" permission.
• Deleting list with "with_faces" setting enabled ("delete lists" request) requires "face" > "deletion" permission.
• Linking face to list in the "create face" request requires "list" > "modification" permission.
• When performing matching, you should have the "matching" permission for the corresponding objects (event, face, attribute).
• Requests to "get statistics on events" resource require "event" > "view" permission.
• Permission "event" > "create" only grants the right to create an event using the "save event" request. To generate an event, you should use the "emit_events" permission (see above).
• Permission "event" > "create" does not apply to verifiers.

See the token coverage table for specific resources in the developer manual of the API service.

Viewing other account data#

Other account details can be viewed if:

• account type is set to "advanced_user" or "admin"
• when creating the token, the "visibility_area" = "all" parameter was specified.

Account type "user" cannot be set to "visibility_area" = "all".

If the account type is set to "advanced_user"/"admin" and the token is created with the "visibility_area" = "account" parameter set, then when logging in using the token (BearerAuth) you will no longer be able to view data from other accounts, but when logging in using login/password (BasicAuth), this option will remain.

If the account type is set to "advanced_user"/"admin" and the token is created with "visibility_area" set to "all" and then the account type is changed to "user" (using the "patch account" request), then an attempt to perform a request to view the data of other accounts using the token will result in an error.

When using the LunaAccountIdAuth authorization, the visibility area is controlled using the "Luna-Account-Id" header.

For verifiers, the ability to use the visibility area of all accounts is not available. If the "visibility_area" = "all", only the data of your account will be visible.

Authorization types for accessing resources#

There are three types of authorization available in LUNA PLATFORM:

• BasicAuth. Authorization by login and password (set during account creation).
• BearerAuth. Authorization by JWT token (issued after the token is created).
• LunaAccountIdAuth. Authorization by "Luna-Account-Id" header, which specifies the "account_id" generated after creating the account (this method was adopted as the main one before version 5.30.0).

LunaAccountIdAuth authorization has the lowest priority compared to other methods and can be disabled using the "ALLOW_LUNA_ACCOUNT_AUTH_HEADER" setting in the "OTHER" section of the API service settings in the Configurator (enabled by default).

In OpenAPI specification the "Luna-Account-Id" header is marked with the word Deprecated.

There is no need to use all three types of authorization when making requests. It is necessary to choose the preferred method depending on the required tasks.

If the authorization type is not specified in the request, an error with the status code 403 will be issued.

Credentials verifier#

Using the "verify credentials" resource, you can verify existing credentials by one of the types:

• login/password verification. If verification is successful, the "account_id" and account type will be returned.
• token verification. If verification is successful, the account type and all permissions for the token will be returned.
• account ID verification. If verification is successful, the account type will be returned.

Logging account information#

When executing any request, information about the corresponding account is displayed in the logs of the API service. This functionality enables you to determine who exactly executed a particular request. This may be required for information security and system administrators.

If the request was executed with BasicAuth or LunaAccountIdAuth type authorization, the following message will be displayed in the logs:

Request invoked by user (account_id: '270531af-e52e-4538-9181-628d9900a0db')

If the request was executed with BearerAuth type authorization, the following message will be displayed in the logs:

Request invoked by user (account_id: '270531af-e52e-4538-9181-628d9900a0db' token_id: 'd57e16f5-e243-47d2-aa85-8b200c12d86f')

If the request was executed without authorization, the following message will be displayed in the logs:

Request invoked by user (account_id: null)

The logs of the Accounts service additionally display information about the creation of tokens by specific users:

User with account_id: '270531af-e52e-4538-9181-628d9900a0db' create token: 'd57e16f5-e243-47d2-aa85-8b200c12d86f'

Logging information about the creation of tokens enables you to track where the token came from and which user it belonged to, even after it was deleted.

Estimated data#

This section lists the general parameters of faces, bodies and images estimated by LUNA PLATFORM 5 and how to get them.

You can get parameters using a some of tools and resources. Basically, getting parameters is done using the following methods:

1. Extract gender and age from the image. Gender and age belong to the concept of basic attributes.

   The "/extractor", "/sdk" resources and "extract_policy" of "/handlers" and "/verifiers" resources are used to extract basic attributes from a face image.

   To extract basic attributes from the face using the "/extractor" resource, you should first create a sample using the "/detector" resource. In response to the "/extractor" request, the gender and age of the person will be returned. The extracted data have a TTL (time to live) and will be removed from the database after the specified period.

   See the detailed description of extracting basic attributes in the "Descriptor extraction and attribute creation" section.

   When estimating parameters using the "/sdk" resource, you should send the source image to LUNA PLATFORM 5 and specify the "estimate_basic_attributes" parameter in the query parameters. In response to the request, the gender and age of the person will be received. These parameters will not be saved to the database.

   To extract this parameters using the "/handlers" and "/verifiers" requests, you should use the "extract_basic_attributes" parameter of the "extract_policy".

   Gender and age estimation is also available by body image. This method is not accurate and is performed by body parameters estimation (see "Perform estimation of body parameters" below).

2. Perform estimation of face and image parameters.

   Various resources are used to estimate the parameters. The resources mainly used are "/detector", "/sdk", "/handlers" and "/verifiers".

   When estimating parameters using the "/detector" resource, you need to send the source image to LUNA PLATFORM 5 and specify the estimation of the required face or image parameters in the query parameters. In response to request, a face sample will be created and the specified parameters will be given. The estimated parameters will not be saved in the database.

   The method for getting parameters using the "/sdk" resource is similar to the method described above, however, the sample will not be created. The estimated parameters will also not be saved to the database.

   To estimate parameters using "/handlers" and "/verifiers" resources, you should use the "detect_policy" with required parameters.

3. Perform estimation of body parameters.

   The ability to perform estimation of body parameters is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

   The "/sdk" and "/handlers" resources are used to estimate body parameters. Using these resources is similar to performing face and image estimation (see above).

4. Perform check of face and image parameters according to ISO/IEC 19794-5:2011 or custom conditions.

   The ability to perform such checks is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

   The "/iso" and "/detector" ("estimate_face_quality" parameter) resources and the "face_quality" check group of the "detect_policy" of the "/handlers" and "/verifiers" requests are used to perform the check.

   The responses to requests show the overall result of passing all checks ("0" or "1"), as well as the results of each check.

   See the detailed description of this functionality in the "Image check" section.

5. Perform Liveness estimation.

   The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

6. Perform estimation of number of people in the image

All returned values and the response format depend on the resource where the estimation is performed.

In order to get results when sending requests to "/handlers" or "/verifiers" resources, you need to generate an event and perform verification on the specified handlers. See the Handler object section for more information about working with these resources.

Gender and age by face image#

This estimation determines the basic attributes (gender and age) of a person in the face image.

For details on basic attributes, see Attribute object.

Resources where the estimation is performed:

• "/extractor"

  Estimation name — "extract_basic_attributes".

• "/handlers"

  Estimation name — "policies" > "extract_policy" > "extract_basic_attributes".

• "/verifiers"

  Estimation name — "policies" > "extract_policy" > "extract_basic_attributes".

• "/sdk"

  Estimation name — "estimate_basic_attributes".

Face parameters#

Eyes attributes#

This estimation determines the following state of eyes for each of the eyes:

• "open"
• "closed"
• "occluded"

Poor quality images or ones that depict obscured eyes (for example, eyewear, hair, gestures) fall into the "occluded" category.

Iris landmarks are determined. An array of 34 landmarks is returned for each eye.

In the "/iso", "/detector" (image checking tool) and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_eyes_attributes".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_eyes_attributes".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_eyes_attributes".

• "/sdk"

  Estimation name — "estimate_eyes_attributes".

Eyes attributes estimation using image checking tools:

• "/iso" and "/detector" (see sections "7.2.3 Expression", point "a", "7.2.11 Visibility of pupils and irises" and "7.2.13 Eye patches" of the standard ISO/IEC 19794-5:2011).

  Checks names — "left_eye", "right_eye".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources.

  Checks names — "left_eye", "right_eye".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face landmarks#

There are two estimations of the face landmarks:

• Estimation of 5 face landmarks
• Assessment of 68 face landmarks

Landmarks are used for various purposes, for example, when creating the sample, when extracting the descriptor, etc. See the SDK documentation for a more detailed description of the landmarks.

• "/detector"

  Estimation name — "detect_landmarks68".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "detect_landmarks68".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "detect_landmarks68".

• "/sdk"

  Estimation name — "estimate_landmarks5".

  Estimation name — "estimate_landmarks68".

Distance between eyes#

Note: It is not possible to use a sample as an input image for this estimation.

It is possible to estimate the distance between the centers of the eyes in pixels. The estimated value is also compared with threshold (according to ISO or non-standard threshold).

This estimation can only be performed using image checking tools:

• "/iso" and "/detector" (see section "5.6.5 Eye and nostril centre Landmark Points" of the standard ISO/IEC 19794-5:2011).

  Check name — "eye_distance".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources.

  Check name — "eye_distance".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Red eyes effect#

This estimation determines the presence of the red eyes effect, where:

• "0" — Red eyes effect is not present in the image.
• "1" — Red eyes effect is present in the image.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for image quality:

The table below shows the requirement for natural light:

Resources where the estimation is performed:

Red eyes effect estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.3.4 Unnatural colour" of the standard ISO/IEC 19794-5:2011)

  Check name — "red_eyes".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "red_eyes".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Gaze#

This estimation determines the gaze. The gaze is represented by the following parameters:

• Pitch angle
• Yaw angle

A positive value of the pitch angle means looking up, and a negative value means looking down.

A positive value of yaw angle means looking to the right, and a negative value means looking to the left.

Zero position corresponds to a gaze direction orthogonally to the face plane, with the axis of symmetry parallel to the vertical camera axis.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_gaze".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_gaze".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_gaze".

• "/sdk"

  Estimation name — "estimate_gaze".

Gaze estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression" point "e" of the standard ISO/IEC 19794-5:2011)

  Checks names — "gaze_yaw", "gaze_pitch".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "gaze_yaw", "gaze_pitch".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Glasses#

This estimation determines the predominant state of the glasses from the following states:

• "sun_glasses"
• "glasses"
• "no_glasses"

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/sdk"

  Estimation name — "estimate_glasses".

Glasses estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.9 Eye glasses" of the standard ISO/IEC 19794-5:2011)

  Check name — "glasses".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "glasses".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Eyebrows#

This estimation determines the predominant state of the eyebrows from the following states:

• "neutral" — Eyebrows are in the usual position.
• "raised" — Eyebrows are raised.
• "squinting" — Eyes are narrowed, eyebrows are lowered.
• "frowning" — Eyebrows are frowned.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

It is possible to specify several eyebrow states as acceptable.

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for head pose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Eyebrows estimation is available only using using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression", points "d", "f" and "g" of the standard ISO/IEC 19794-5:2011)

  Check name — "eyebrows_state".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "eyebrows_state".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Mouth attributes#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1]:

• "opened"
• "smile"
• "occluded"

The probability that the mouth is opened is also determined.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_mouth_attributes".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_mouth_attributes".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_mouth_attributes".

• "/sdk"

  Estimation name — "estimate_mouth_attributes".

Mouth attributes estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression" point "a", "b" and "c" of the standard ISO/IEC 19794-5:2011)

  Checks names — "mouth_smiling", "mouth_occluded", "mouth_open".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "mouth_smiling", "mouth_occluded", "mouth_open".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Smile state#

This estimation determines the predominant state of a smile from the following states:

• "none" — No smile found, so no additional parameters are determined.
• "smile_lips" — Usual smile with closed lips.
• "smile_teeth" — Smile with open teeth.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

If necessary, you can specify several smile states as acceptable.

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for head pose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Smile state estimation is only available using image checking tools:

• "/iso" and "/detector" (see section "7.2.3 Expression" point "a", "b" and "c" of the standard ISO/IEC 19794-5:2011)

  Check name — "smile_properties".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "smile_properties".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image quality#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1], where 0 corresponds to low quality and 1 to high quality:

• "dark" — The degree of darkness in the photo.
• "light" — The degree of brightness in the photo.
• "blur" — The degree of blurriness.
• "illumination" — The degree of uniformity in illumination. The lower the difference between light and dark zones on the face, the higher the estimated value. When illumination is evenly distributed across the face, the value is close to "1."
• "specularity" — The degree of absence of glare. The higher the estimated value, the less glare and the better the image quality. If the estimated value is low, there are bright glares on the face.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image quality is determined using a specially trained VisionLabs neural network. If necessary, you can determine the illumination of the face in the image using an algorithm that performs an estimation in accordance with the ICAO standard (see the section "Illumination uniformity according to ICAO standard).

Examples are presented in the images below. Good quality images are shown on the right.

The most important image quality parameters for face recognition are darkness, light, and blur, so you should select them carefully.

The illumination and secularity parameters enable you to select images of better visual quality. Face recognition is not greatly affected by uneven illumination or glares.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_quality".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_quality".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_quality".

• "/sdk"

  Estimation name — "estimate_quality".

Image quality estimation using image checking tools:

• "/iso" and "/detector" (see sections "7.2.7 Subject and scene lighting", "7.3.2 Contrast and saturation", "7.3.3 Focus and depth of field", "7.2.8 Hot spots and specular reflections", "7.2.12 Lighting artefacts", "7.2.7 Subject and scene lighting" and "7.2.12 Lighting artefacts" of the standard ISO/IEC 19794-5:2011)

  Checks names — "illumination_quality", "specularity_quality", "blurriness_quality", "dark_quality", "light_quality".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "illumination_quality", "specularity_quality", "blurriness_quality", "dark_quality", "light_quality".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Illumination uniformity according to ICAO standard#

It is possible to estimate the uniformity of illumination according to the requirements specified in ICAO standard.

The estimated value is also compared with the threshold.

In accordance with the standard, it is recommended to use color images. When using black and white images, the results may be unexpected.

The uniformity of illumination according to the ICAO standard is only available using image checking tool:

• "/detector"

  Check name — "illumination_uniformity".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "illumination_uniformity".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image background#

Background lightness#

Note: It is not possible to use a sample as an input image for these estimations.

This estimation determines background lightness, where:

• [0...0.1] — The background is black.
• [0.1...0.3] — The background is dark.
• [0.3...0.97] — The background is light.
• [0.97...1] — The background is white.

Resources where the estimation is performed:

Background lightness estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "B.2.9 Backgrounds" of the standard ISO/IEC 19794-5:2011)

  Check name — "background_lightness".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "background_lightness".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Background uniformity#

Note: It is not possible to use a sample as an input image for these estimations.

This estimation determines the degree of background uniformity, where:

• "0" — The background is not uniform.
• "1" — The background is uniform.

Resources where the estimation is performed:

Background uniformity estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "B.2.9 Backgrounds" of the standard ISO/IEC 19794-5:2011)

  Check name — "background_uniformity".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "background_uniformity".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Dynamic range according to ICAO standard#

This estimation determines the ratio of brightness of the lightest and darkest areas of the face according to the requirements specified in ICAO standard.

The estimated value is also compared with the threshold.

Dynamic range estimation is only available using image checking tool — "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources.

Check name — "dynamic_range".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Natural light#

This estimation determines whether there is natural lighting on the face, where:

• "0" — The lighting is unnatural.
• "1" — The lighting is natural.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirement for mask:

The table below shows the requirement for image quality:

The table below shows the requirement for glasses:

Resources where the estimation is performed:

Natural light estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.3.4 Unnatural colour" of the standard ISO/IEC 19794-5:2011)

  Check name — "natural_light".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "natural_light".

Acceptable value for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face color type#

This estimation determines the most likely type of face color from the following:

• "color"
• "grayscale"
• "infrared" (near infrared range)

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

Face color type estimation is only available using image checking tools:

• "/iso" and "/detector" (see section "7.4.4 Use of near infra-red cameras" of the standard ISO/IEC 19794-5:2011)

  Check name — "face_color_type".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "face_color_type".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Head pose#

This estimation determines the head pose. The pose is defined by three parameters:

• pitch angle
• roll angle
• yaw angle

The angle values are specified in the range from "-180" to "180".

A positive pitch angle indicates an upward tilt of the head, while a negative angle indicates a downward tilt of the head.

A positive roll angle indicates a rightward deviation of the head, while a negative angle indicates a leftward deviation of the head.

A positive yaw angle indicates a rightward rotation of the head, while a negative angle indicates a leftward rotation of the head.

In the "/iso" and "detect_policy" > "face_quality" resources, the estimated value is also compared with threshold (according to ISO or non-standard threshold).

In all the resources listed below, with the exception of "/iso", the possibility to filtered out by head pose is availiable.

In the resources "/detector", "/handlers", "/verifiers" and "/sdk", the threshold value is specified from "0" to "180". The default value is "180", which means that the head in the image can be rotated to any angle from "-180" to "180". When you set any other value (for example, "30"), all the detections with the etimated angle less than or equal to "-30" and greater than or equal to "30" will be filtered.

For the "face_quality" field of the "/handlers" and "/verifiers" resources, the minimum and maximum thresholds are set in separate fields.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_head_pose".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_head_pose".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_head_pose".

• "/sdk"

  Estimation name — "estimate_head_pose".

Below are the recommended thresholds for estimation in the "/detector", "/handlers", "/verifiers" and "/sdk" resources.

Recommended maximum thresholds:

The table below shows the recommended maximum thresholds head pose for estimation in cooperative mode:

The table below shows the recommended maximum thresholds head pose for estimation in non-cooperative mode:

Head pose estimation using image checking tools:

• "/iso" and "/detector" (see section "7.2.2 Pose" of the standard ISO/IEC 19794-5:2011)

  Checks names — "head_roll", "head_pitch", "head_yaw".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "head_roll", "head_pitch", "head_yaw".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Vertical and horizontal face position#

Note: It is not possible to use a sample as an input image for these estimations.

These estimations determine the position of the center point vertically and horizontally relative to the image.

The estimated values are also compared with thresholds (according to ISO or non-standard thresholds).

Resources where the estimation is performed:

Vertical and horizontal face position estimation is available only using image checking tools:

• "/iso" and "/detector" (see sections "8.3.2 Horizontally centred face" and "8.3.3 Vertical position of the face" of the standard ISO/IEC 19794-5:2011)

  Checks names — "head_horizontal_center", "head_vertical_center".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "head_horizontal_center", "head_vertical_center".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Head width and height#

Note: It is not possible to use a sample as an input image for these estimations.

This estimations determine the vertical and horizontal head size relative to the size of the image. The estimated values are also compared with thresholds (according to ISO or non-standard thresholds).

Resources where the estimation is performed:

Head width and height estimation is available only using image checking tools:

• "/iso" and "/detector" (see sections "8.3.4 Width of head" and "8.3.5 Length of head" of the standard ISO/IEC 19794-5:2011)

  Checks names — "head_width", "head_height".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "head_width", "head_height".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Face width and height#

Note: It is not possible to use a sample as an input image for these estimations.

These estimations determine the face width and height in pixels. The estimated values are also compared with the specified thresholds.

Resources where the estimation is performed:

Face width and height estimations are only available using image checking tool:

• "/detector"

  Checks names — "face_width", "face_height".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "face_width", "face_height".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Indents from image edges#

Note: It is not possible to use a sample as an input image for these estimations.

This estimation is determined as the indent from the image border (left, right, top, bottom) to the face border (left, right, top, bottom) in pixels. The estimated values are also compared with the specified thresholds.

Resources where the estimation is performed:

Indents from image edges estimation is only available using image checking tool:

• "/detector"

  Checks names — "indent_upper", "indent_lower", "indent_right", "indent_left".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "indent_upper", "indent_lower", "indent_right", "indent_left".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Mask#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1]:

• "medical_mask" — The probability that the person wears a medical mask.
• "missing" — The probability that the person does not wear a mask.
• "occluded" — The probability that the face is occluded by an object other than a medical mask.

The predominant state of the mask is also determined.

In addition to the three main states, the following additional properties are defined:

• "correct" — There is mask on the face, the mouth and nose are occluded by the mask.
• "mouth" — There is mask on the face and occludes only the mouth.
• "clear" — There is no mask on the face.
• "chin" — There is mask on the face and is located under the chin, without occluding the area from eyes to mouth.
• "partially" — The face is partially occluded, but not by medical mask and not by mask with full face occlusion.
• "full" — There is mask on the face, in which the face is completely occluded, for example, balaclava/ski mask.

Each main mask state corresponds to one of two additional properties. The most likely additional property is returned in the "predominant_occlusion" field:

• The "medical_mask" state corresponds to the "correct" or "mouth" property.
• The "missing" state corresponds to the "clear" or "chin" property.
• The "occluded" state corresponds to the "partially" or "full" property.

For each of the properties, a probabilistic score is returned in the range [0..1].

Additional mask properties are not stored in the database and are not filtered by them.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_mask".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_mask".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_mask".

• "/sdk"

  Estimation name — "estimate_mask".

Emotions#

This estimation determines a probabilistic score for each of the following parameters in the range [0..1]:

• "anger"
• "disgust"
• "fear"
• "happiness"
• "neutral"
• "sadness"
• "surprise"

The predominant emotion is also estimated.

Emotions can be saved in the event object during the event creation.

Resources where the estimation is performed:

• "/detector"

  Estimation name — "estimate_emotions".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_emotions".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_emotions".

• "/sdk"

  Estimation name — "estimate_emotions".

Shoulders position#

This estimation determines the most predominant state of the shoulder position from the following:

• "non-parallel"
• "parallel"
• "hidden"

Resources where the estimation is performed:

Shoulders position estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.2.5 Shoulders" of the standard ISO/IEC 19794-5:2011)

  Check name — "shoulders_position".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "shoulders_position".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Headwear#

This estimation determines the most predominant type of headwear from the following:

• "none
• "baseball_cap"
• "beanie"
• "peaked_cap"
• "shawl"
• "hat_with_ear_flaps"
• "helmet"
• "hood"
• "hat"
• "other"

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

It is possible to specify several types of headwear as acceptable.

Image requirements:

The table below shows the requirements for head hose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Headwear estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "B.2.7 Head coverings" of the standard ISO/IEC 19794-5:2011)

  Check name — "headwear_type".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "headwear_type".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Radial distortion (Fisheye effect)#

This estimation determines the presence of the Fisheye effect, where:

• "0" — The Fisheye effect is not present in the image.
• "1" — The Fisheye effect is present in the image.

The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Image requirements:

For correct checking results, the following requirements should be met.

The table below shows the requirements for head position:

The table below shows the requirement for face width:

Resources where the estimation is performed:

Fisheye effect estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.3.6 Radial distortion of the camera lens" of the standard ISO/IEC 19794-5:2011)

  Check name — "radial_distortion".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "radial_distortion".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image parameters#

Image format#

This estimation determines the correspondence of the incoming image format to one of the following formats — "JPEG", "JPEG2000", "PNG". The estimated value is also compared with threshold (according to ISO or non-standard threshold).

Resources where the estimation is performed:

Image format estimation is available only using image checking tools:

• "/iso" and "/detector" (see section "7.5 Format requirements for the Frontal Image Type" of the standard ISO/IEC 19794-5:2011)

  Check name — "image_format".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "image_format".

Acceptable values for passing check:

The image passes the check if it falls within the acceptable value of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image size#

This estimation determines the image size in bytes. The estimated value is also compared with the specified threshold.

Resources where the estimation is performed:

Image size estimation is only available using image checking tool:

• "/detector"

  Check name — "image_size".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "image_size".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Image width and height#

These estimations determine the image width and height in pixels. The estimated values are also compared with thresholds (according to ISO or non-standard thresholds).

Resources where the estimation is performed:

Image width and height estimations are available only using image checking tools:

• "/iso" and "/detector" (see sections "5.7.4 Width" and "5.7.5 Height" of the standard ISO/IEC 19794-5:2011)

  Checks names — "image_height", "image_width".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Checks names — "image_height", "image_width".

Acceptable ranges for passing checks:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

Aspect ratio#

This estimation determines the proportional ratio of the image width to height. The estimated value is also compared with the specified threshold.

Resources where the estimation is performed:

Aspect ratio estimation is only available using image checking tool:

• "/detector"

  Check name — "aspect_ratio".

• "detect_policy" > "face_quality" group of checks in the "/handlers" and "/verifiers" resources

  Check name — "aspect_ratio".

Acceptable range for passing check:

The image passes the check if it falls within the acceptable range of the corresponding threshold:

For the "face_quality" image checking tool, the acceptable value can be set manually, but this will mean a deviation from the standard.

EXIF metadata#

When the EXIF estimation is enabled, all the tags of the image are parsed and their names and values are outputted. Please refer to JEITA CP-3451 EXIF specification for details. The following data is returned:

• make
• model
• orientation
• latitude
• longitude
• artist
• software
• dateTime
• digitalZoomRatio
• flash
• uid

Resources where the estimation is performed:

• "/detector"

  Estimation name — "extract_exif".

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "extract_exif".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "extract_exif".

• "/sdk"

  Estimation name — "use_exif_info".

Body parameters#

Gender and age by body image#

This estimation determines the basic attributes (gender and age) of a person in the body image.

Estimating gender and age from a body image is less accurate than from a face.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_basic_attributes".

• "/sdk"

  Estimation name — "estimate_body_basic_attributes".

Upper body#

This estimation determines the parameters of the following elements of clothing on the upper body:

• "headwear". Type: absent, present, undefined; color: white, black, other, undefined.
• "sleeve". Type: short, long, undefined.
• "upper_clothing". Color: orange, purple, red, white, yellow, pink, brown, beige, khaki, multicolored, undefined.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_upper_body".

• "/sdk"

  Estimation name — "estimate_upper_body".

Lower body#

This estimation determines the parameters of the following elements of clothing on the lower body:

• "lower_garment". Type: trousers, shorts, skirt, undefined; color: orange, purple, red, white, yellow, pink, brown, beige, khaki, multicolored, undefined.
• "shoes". Color: white, black, other, undefined.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_lower_body".

• "/sdk"

  Estimation name — "estimate_lower_body".

Backpack#

This estimation determines the presence of a backpack on the body:

• "0" — There is no backpack on the body image.
• "1" — There is a backpack on the body image.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "body_attributes" > "estimate_accessories".

• "/sdk"

  Estimation name — "estimate_accessories".

Liveness#

Note: The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

The Liveness technology enables LUNA PLATFORM to detect presentation attacks. To estimate Liveness in the LUNA PLATFORM, the estimator LUNA SDK OneShotLiveness is used.

As a result of the Liveness estimation, one of the following results may be returned:

• "0" — The person is not real.
• "1" — The person is real.
• "2" — Result of the check is unknown.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_liveness".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_liveness".

• "/sdk"

  Estimation name — "estimate_liveness".

• "/liveness"

See "OneShotLiveness description" for more information about Liveness.

Deepfake#

Note: The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

Note: It is not possible to use a sample as an input image for these estimation.

This estimation enables you to detect facial substitution using DeepFake technology in photo images.

A Deepfake evaluation may return the following results:

• "prediction" = "fake" — The person is not real.
• "prediction" = "real" — The person is real.
• "score" = [0...1] — The degree of reliability of the estimation.

If necessary, you can configure the handler to filter events by the expected result of the Deepfake estimation ("fake" or "real"). To do this, in the request body to create a handler, specify the "deepfake_states" parameter with the value "0" (filter by the value "fake") or "1" (filter by the value "real"). For example, if the "deepfake_states" parameter is "1" (filter by "real"), and the estimator has determined that the result is "fake", then an empty "events" field will be returned in the event, and the results of the check will fall into the "filtered_detections" field.

In the requests "create handler" and "create verifier", it is possible to set the "real_threshold" and the operating "mode". The "sdk" request will use the default values of these parameters (see below) without the possibility of explicit indication.

Threshold

Using the "real_threshold", you can set a value in the range [0...1], below which the system will assume that the person is not real.

For example, if the threshold value is "real_threshold" = "0.5", and the degree of reliability of the estimation is "score" = "0.4", then the result "prediction" = "fake" will be output in the response body. If the threshold value is "real_threshold" = "0.6", and the degree of reliability of the estimation is "score" = "0.7", then the result "prediction" = "real" will be given in the response body.

The default value is "0.5".

Mode

The operating modes are described in the table below.

Image requirements

For correct verification results, the following requirements must be met.

The table below shows the requirements for head pose:

The table below shows the requirement for face width:

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_deepfake".

• "/verifiers"

  Estimation name — "policies" > "detect_policy" > "estimate_deepfake".

• "/sdk"

  Estimation name — "estimate_deepfake".

People count#

Note: The ability to perform such estimation is adjusted by a special parameter in the LUNA PLATFORM 5 license key.

This estimate determines the number of people in the image.

Resources where the estimation is performed:

• "/handlers"

  Estimation name — "policies" > "detect_policy" > "estimate_people_count".

• "/sdk"

  Estimation name — "estimate_people_count".

If necessary, together with estimating the number of people, you can get the X and Y coordinates of people using the "people_count_coordinates" parameter in the "/handlers" resource and the "people_coordinates" parameter in the "/sdk" resource.

Services interaction#

There are separate databases specified for each service in the images below. This is done for illustration purposes.

It is not required to launch several similar databases for each service. You can launch a single database instance (e. g., PostgreSQL) and use it to store data for all the LP services. Each service will have its own table in this database.

See "General concepts" for details about the databases used by LP services.

API service provides a RESTful interface for faces and bodies detection, extraction and matching procedures.

Requests are sent to LP via HTTP. A common case is when an external service sends requests to LP, receives results and processes the results according to business needs.

Some services can be disabled (see "General concepts").

API receives requests, processes them and checks user authorization using a request to the Accounts service (A0). The Accounts service checks if the user's credentials exist in the Accounts database (Acc1). If the user data is not found in the database, then an authorization error is issued. If the data is found, then the corresponding information is sent to the API service, where it passes the original requests to other services.

The main operations (detection, estimation, extraction, matching) are performed as described below.

Approach "Sequential performing of requests":

• The "detect faces" request, in which face detection is performed, face parameters are estimated and samples are created, is sent from the API service to the Handlers service (A1), and then redirected to Remote SDK service (H1).
• The "extract attributes" request, in which temporary attributes are extracted, is sent from the API service to the Handlers service (A1), and then redirected to the Remote SDK service (H1).
• The request "matching faces" is sent from the API service to the Python Matcher service (A6).

Approach "Parallel performing of requests":

• The "create handler" request with the rules for detecting, estimating, extracting and matching faces and bodies is sent from the API service to the Handlers service (A1).
• The "generate events" request is sent from the API service to the Handlers service (A1), and then redirected to the Remote SDK service (H1).

The API service sends requests to the Faces service to create/modify new faces (requests from the "faces" section), as well as create/modify lists (requests from the "lists") section (A2).

The API service receives information about the current license terms from the Licenses service (A3).

The API service sends samples from external services to the Image Store service (the "save face/body sample" request) (A5).

Handlers creates new handlers and stores them in the Handlers database (H2).

The service also redirects requests for detection and estimation to the Remote SDK (H1) service.

The Handlers service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Remote SDK processes face/body detection and attributes creation requests. It estimates face attributes and face/body parameters.

The Remote SDK service processes the request to detect faces/bodies and estimate parameters from the Handlers (H1) service, sends the result back to Handlers (H1), which sends the received samples to the Image Store service (H3).

The Remote SDK service processes the request to create attributes from the Handlers (H1) service by requesting existing samples from the Handlers service, which it requests from the Image Store service (H3). Next, the Handlers service sends the created temporary attributes to the Faces (H4) service, which saves them in the Redis database (F1).

Requests "/iso", "/sdk", "/liveness" are performed directly to the Remote SDK service (A9). In this case, the license is checked by interacting with the Licenses (RS1) service. To be able to use a set of samples, the Remote SDK service interacts with the Image Store (RS2) service.

Image Store receives samples from the Remote SDK service and stores them on a local storage or in S3-compatible cloud storage and provides access to them (Im1).

The Image Store service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Licenses. The Licenses service receives information about the number of created faces with attributes from the Faces database (Li1).

Faces is responsible for interaction with Faces database, which stores: faces, descriptors for faces, and lists (F2). It provides access to the stored data for API, Python Matcher, and Tasks services.

Faces also stores the created temporary attributes in the Redis database (F1).

Every time a new face is created, the Faces service sends information about the number of created faces with attached attributes to the Licenses service (F3) (see the "Faces limit" section).

Events  service stores and provides information about events (the Events section). After an event is created, the Events service receives the created event from the Handlers service (H6) and stores it in the Events database (Ev2).

The Events service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Python Matcher can receive matching requests directly from the API service (A6). If a matching policy is specified in the handler, then a request to match descriptors will come from the Handlers (H7) service. Python Matcher sends matching requests to the Faces database (M3) or Events database (M1). The databases matches descriptors and sends results back to the API service.

Python Matcher can also receive the temporary attributes required for matching from the Redis database (M2).

It is possible to additionally use the Python Matcher Proxy service, which can redirect requests to Python Matcher or matching plugins. Matching plugins can significantly speed up the matching requests execution (see the "Matching plugins" section). This service is not shown in the diagram, but has the same communication lines as the Python Matcher service.

Admin. A user can manage accounts and perform other actions via the user interface of the Admin service (Adm1). The corresponding requests are sent to the Admin service (Adm2).

The Admin service receives information about accounts from the Accounts (Adm3) service.

All information about user accounts is stored in the Accounts database (Acc1).

The Admin service sends requests to Tasks service (Adm4). These tasks are performed for LP databases in general, not for a single account ID.

The Admin service receives information about the current license terms from the Licenses service (Adm5).

The Admin service checks the current number of created faces with linked attributes in the Faces database (Adm6) and calculates the percentage of the database of fullness.

The Admin service performs requests to all LP services, receiving system information (see "/luna_sys_info" request) (Adm7).

Sender. All generated events are received by the Redis DB from the Handlers service (H5). External third-party party software subscribes to receive events according to the specified filters (Se2). Sender service checks Redis channel, receives the required information and sends it to external software (Se1).

The Sender service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

Configurator. LP services receive configuration parameters from the Configurator service (Con1). A user can manage configurations in Configurator by means of the user interface (Con2). The changes are sent to Configurator (Con3). All the changes in configuration files are saved in Configurator DB (Con4).

Configurator is utilized for each Luna service by default.

Tasks. Tasks service receives requests from API service (A8). Then Tasks creates and stores a task in the Tasks database (T1).

Next, the Tasks service interacts with its worker via Redis (T2, T3), creating subtasks and merging the results. For more information, see "Task diagrams" section.

The Tasks service receives data from the Faces service for the Clustering, Linker and Additional extraction tasks (T4).

The Tasks service receives data from the Events service for the Clustering and Linker tasks (T9, T10).

The Tasks service receives data from the Handlers service for the Estimator task (T6).

The Tasks service is enabled/disabled in the "ADDITIONAL_SERVICE_USAGE" setting.

The interaction of Tasks workers with other services depends on the task type.

Tasks workers receive data from Faces service for every processed task (T5).

Tasks workers send a request to Python Matcher (T7) for clustering, cross-matching and ROC creation tasks.

Tasks workers store all the reports in the storage of Image Store (T8). Tasks workers also store and receive clusters from Image Store.

Tasks workers send the request for additional extraction task to the Handlers service (T6).

Monitoring. Monitoring system receives requests and errors from each service (Mon1). This data stores in the Influx database (Mon2). Then the monitoring data is sent to Grafana to visualize the monitoring data (Mon3). You can find more information about monitoring in the "Monitoring" section.

Grafana and InfluxDB services have their own interfaces (see the "User interfaces" section).

This diagram does not contain the architecture of the Backport 3 and Backport 4 services. See the detailed architecture of the Backport 3 service in "Backport 3 architecture" section and the Backport 4 service in "Backport 4 architecture" section.

Services description#

This section provides more details on functions of the LP services.

Databases can be omitted in the following figures.

See the table with the resource consumption of each of the services listed below in the "Resource consumption by services" section.

General information about services#

Worker processes#

For LUNA PLATFORM services, you can set the number of workers to use additional resources and system memory to process requests to the service. A service will automatically spin up multiple processes and route traffic between the processes.

When starting the service in a Docker container, the number of workers is set using the WORKER_COUNT parameter.

For example, if you set the value WORKER_COUNT=2 for the Faces service, then the service will consume 2 times more resources and memory.

Note the number of available cores on your server when utilizing this feature.

Worker processes utilization is an alternative way for linear service scaling. It is recommended to use additional worker processes when increasing the number of service instances on the same server.

It is not recommended to use additional worker processes for the Remote SDK service when it utilizes GPU. Problems may occur if there is not enough GPU memory, and the workers will interfere with each other.

Automatic configurations reload#

LP services support the auto-reload of configurations. When a setting is changed, it is automatically updated for all the instances of the corresponding services. When this feature is enabled, no manual restart of services is required.

This feature is available for all the settings provided for each Python service. You should enable the feature manually upon each service launching. See the "Enable automatic configuration reload" section.

Starting with version 5.5.0 the configuration reload for Faces and Python Matcher services is done mostly by restarting appropriate processes.

Restrictions#

Service can work incorrectly while new settings are being applied. It is strongly recommended not to send requests to the service when you change important settings (DB setting, work plugins list, and others).

New settings appliance may lead to service restart and caches resetting (e. g., Python Matcher service cache). For example, the default descriptor version changing will lead to the LP restart. Changing the logging level does not cause service restart (if a valid setting value was provided).

Enable automatic configuration reload#

You can enable this feature by specifying a --config-reload option in the command line. In Docker containers, the feature is enabled using the "RELOAD_CONFIG" option.

You can specify the configurations check period in the --pulling-time command line argument. The value is set to 10 seconds by default. In Docker containers, the feature is enabled using the "RELOAD_CONFIG_INTERVAL" option.

Configurations update process#

LP services periodically receive settings from the Configurator service or configuration files. It depends on the way of configurations receiving for a particular service.

Each service compares its existing settings with the received settings:

• If service settings were changed, they will pulled and applied.

  • If the configurations pulling has failed, the service will continue working without applying any changes to the existing configurations.

  • If check connections with new settings have failed, the service will retry new configurations pulling after 5 seconds. The service will shut down after 5 failed attempts.

• If current settings and new pulled settings are the same, the Configurator service will not perform any actions.

Database migration execution#

You should execute migration scripts to update your database structure when upgrading to new LP builds. By default, migrations are automatically applied when running db_create script.

This method may be useful when you need to rollback to the previous LUNA PLATFORM build or upgrade the database structure without changing the stored data. Anyway, it is recommended to create the backup of your database before applying any changes.

You can run migrations from a container or use a single command.

Single command#

The example is given for the Tasks service.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/tasks:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-tasks:v.3.22.1 \
alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

Running from container#

To run migrations from a container follow these steps (the example is given for the Configurator service):

• Go to the service docker container. See the "Enter container" section in LP 5 installation manual.

• Run the migrations.

For most of the services, the configuration parameters should be received from the Configurator service and the command is the following:

alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

-x luna-config=http://127.0.0.1:5070/1 — Specifies that the configuration parameters for migrations should be received from the Configurator service.

For the Configurator service the parameters are received from "srv/luna_configurator/configs/config.conf" file.

You should use the following command for the Configurator service:

alembic upgrade head

• Exit the container. The container will be removed after you exit.

exit

API service#

LUNA API is a facial recognition web service. It provides a RESTful interface for interaction with other LUNA PLATFORM services.

Using the API service you can send requests to other LP services and solve the following problems:

• Images processing and analysis:

  • Face/body detection in photos.

  • Face attributes (age, gender, ethnicity) and face parameters (head pose, emotions, gaze direction, eyes attributes, mouth attributes) estimation.

  • Body parameters (age, gender, accessories, headwear, colors of upper and lower clothing, type of sleeves) estimation.

• Search for similar faces/bodies in the database.

• Storage of the received face attributes in databases.

• Creation of lists to search in.

• Statistics gathering.

• Flexible request management to meet user data processing requirements.

Remote SDK service#

The Remote SDK service is used to:

• Perform face detection and face parameters estimation.
• Perform body detection and body parameters estimation.
• Create samples.
• Perform extraction of basic attributes and descriptor, including aggregated ones.
• Process images using handlers and verifiers policies.

Face, body detection, descriptor extraction, estimation of parameters and attributes are performed using neural networks. The algorithm evolves with time and new neural networks appear. They may differ from each other by performance and precision. You should choose a neural network following the business case of your company.

Remote SDK with GPU#

Remote SDK service can utilize GPU instead of CPU for calculations. A single GPU is utilized per Remote SDK service instance.

Attributes extraction on the GPU is engineered for maximum throughput. The input images are processed in batches. This reduces computation cost per image but does not provide the shortest latency per image.

GPU acceleration is designed for high load applications where request counts per second consistently reach thousands. It won’t be beneficial to use GPU acceleration in non-extensively loaded scenarios where latency matters.

Aggregation#

Based on all images transferred in one request, a single set of basic attributes and an aggregated descriptor can be obtained. In addition, during the creation of the event, the aggregation of the received values of Liveness, emotions, medical mask states for faces and upper/lower body, gender, age and the body accessories for bodies is performed.

The matching results are more precise for aggregated descriptor. It is recommended to use aggregation when several images were received from the same camera. It is not guaranteed that aggregated descriptors provide improvements in other cases.

It is considered that each parameter is aggregated from sample. Use the "aggregate_attributes" parameter of the "extract attributes" (only for faces) and "sdk" requests to enable attributes aggregation. Aggregation of liveness, emotion, and mask states for faces and upper body, gender, age and the body accessories for bodies is available using the "aggregate_attributes" parameter in the "generate events", provided that these parameters were estimated earlier in the handler, as well as in the "sdk" request.

An array of "sample_ids" is returned in the response even if there was only a single sample used in the request. In this case, a single sample ID is included in the array.

Descriptor formats#

LUNA PLATFORM supports the following descriptor formats:

SDK and Raw formats can be directly linked to a face or stored in a temporary attribute (see "Create objects using external data" below).

In most extraction requests, the descriptor is saved to the database as set of bytes, without being returned in the response body.

There are several requests that can be used to get descriptor in SDK format:

• request "sdk";
• request "get temporary attributes and "get temporary attribute.

With LUNA PLATFORM, it is not possible to get descriptors in Raw and SDK formats. You can use other VisionLabs software to get these formats (eg LUNA SDK). Descriptors obtained using the above resources or using the VisionLabs software are referred to as raw descriptors.

Use raw descriptors for matching

The descriptor formats described above can be used in requests for the use of raw descriptors.

An external raw descriptor can be used as reference in the following resources:

• "/matcher/faces"
• "/matcher/bodies"
• "/matcher/raw"
• "/handlers/{handler_id}/events" when "multipart/form-data" request body schema is set
• "/verifiers/{verifier_id}/verifications" when "multipart/form-data" request body schema is set
• "/verifiers/{verifier_id}/raw"

An external raw descriptor can be used as a candidate in the following resources:

• "/matcher/raw"
• "/verifiers/{verifier_id}/raw"

Create objects using external data#

You can create a temporary attribute of face by sending basic attributes and descriptors to LUNA PLATFORM. Thus you can store this data in external storage and send it to LP for the processing of requests only.

You can create an attribute or face using:

• Basic attributes and their samples.
• Descriptors (raw descriptor in Base64 or SDK descriptor in Base64).
• Both basic attributes and descriptors with the corresponding data.

Samples are optional and are not required for an attribute or face creation.

See the "create temporary attribute" request and "create face request" for details.

Checking images for compliance with standards#

The Remote SDK service enables you to check images according to the ISO/IEC 19794-5:2011 standard or user-specified thresholds using three ways:

• Request "iso".
• Parameter "estimate_face_quality" of the request "detect faces".
• Group of parameters "face_quality" of the policy "detect_policy" of the request "generate events".

For example, it is necessary to check whether the image is of a suitable format, specifying the "JPEG" and "JPEG2000" formats as a satisfactory condition. If the image fits this condition, the system will return the value "1", if the format of the processed image is different from the specified condition, the system will return the value "0". If the conditions are not set, the system will return the estimated value of the image format.

The list of estimations and checks performed is described in the "Image check" section.

The ability to perform check and estimation of image parameters is regulated by a special parameter in the license file.

Enable/disable several estimators and detectors#

By default, the Remote SDK service is launched with all estimators and detectors enabled. If necessary, you can disable the use of some estimators or detectors when launching the Remote SDK container. Disabling unnecessary estimators enables you to save RAM or GPU memory, since when the Remote SDK service launches, the possibility of performing these estimates is checked and neural networks are loaded into memory.

If you disable the estimator or detector, you can also remove its neural network from the Remote SDK container.

Disabling estimators or detectors is possible by transferring documents with the names of estimators to the launch command of the Remote SDK service. Arguments are passed to the container using the "EXTEND_CMD" variable.

List of available estimators:

You can explicitly specify which estimators and detectors are enabled or disabled by passing the appropriate arguments to the "EXTEND_CMD" variable, or you can enable (by default) or disable them all with the enable-all-estimators-by-default argument. You can turn off the use of all estimators and detectors, and then turn on specific estimators by passing the appropriate arguments.

Example of a command to start the Remote SDK service using only a face detector and estimators of a face sample and emotions.

docker run \
...
--env=EXTEND_CMD="--enable-all-estimators-by-default=0 --enable-face-detector=1 --enable-face-warp-estimator=1 --enable-emotions-estimator=1" \
...

Handlers service#

The Handlers service is used to create and store handlers and verifiers.

The data of handlers and verifiers are stored in the Handlers database.

Send events to third-party service#

LUNA PLATFORM provides the ability to send notifications using two policies — "notification_policy" and "callbacks".

The descriptions of these policies are provided below.

Image Store service#

The Image Store service stores the following data:

• Face and body samples. Samples are stored in Image Store by the Remote SDK service or you can save an external sample using the "samples" > "detect faces" and "samples" > "save face/body sample" requests.
• Reports about tasks. Reports are stored by the Tasks service workers.
• Any objects loaded using the "create objects" request.
• Clusterization information.

Image Store can save data either on a local storage device or in S3-compatible cloud storage (Amazon S3, etc.).

Buckets description#

The data is stored in special directories called buckets. Each bucket has a unique name. Bucket names should be set in lower case.

The following buckets are used in LP:

• "visionlabs-samples". This bucket stores face samples.
• "visionlabs-bodies-samples". This bucket stores human bodies samples.
• "visionlabs-image-origin". This bucket stores source images.
• "visionlabs-objects". This bucket stores objects.
• "task-result". This bucket stores the results received after tasks processing using the Tasks service.
• "portraits". This bucket stores portraits. The bucket is required for the usage of Backport 3 service.

Buckets creation is described in LP 5 installation manual in the "Buckets creation" section.

After running the Image Store container and the commands for containers creation, the buckets are saved to local storage or S3.

By default, local files are stored in the "/var/lib/luna/current/example-docker/image_store" directory on the server. They are saved in the "/srv/local_storage/" directory in the Image Store container.

Bucket includes directories with samples or other data. The names of the directories correspond to the first four letters of the sample ID. All the samples are distributed to these directories according to their first four ID symbols.

Next to the bucket object is a "*.meta.json" file containing the "account_id" used when performing the request. If the bucket object is not a sample (for example, the bucket object is a JSON file in the "task-result" bucket), then the "Content-Type" will also be specified in this file.

An example of the folders structure in the "visionlabs-samples", "task-result" and "visionlabs-bodies-samples" buckets is given below.

./local_storage/visionlabs-samples/8f4f/
            8f4f0070-c464-460b-sf78-fac234df32e9.jpg
            8f4f0070-c464-460b-sf78-fac234df32e9.meta.json
            8f4f1253-d542-621b-9rf7-ha52111hm5s0.jpg
            8f4f1253-d542-621b-9rf7-ha52111hm5s0.meta.json
./local_storage/task-result/1b03/
            1b0359af-ecd8-4712-8fc0-08401612d39b
            1b0359af-ecd8-4712-8fc0-08401612d39b.meta.json
./local_storage/visionlabs-bodies-samples/6e98/
            6e987e9c-1c9c-4139-9ef4-4a78b8ab6eb6.jpg
            6e987e9c-1c9c-4139-9ef4-4a78b8ab6eb6.meta.json

A significant amount of memory may be required when storing a large number of samples. A single sample takes about 30 Kbytes of the disk space.

It is also recommended to create backups of the samples. Samples are utilized when the NN version is changed or when you need to recover your database of faces.

Use S3-compatible storage#

To enable the use of S3-compatible storage, you must perform the following steps:

• Make sure that the access key has sufficient authority to access the buckets of the S3-compatible storage.
• Launch the Image Store service (see "Image Store" section in the installation manual).
• Set the "S3" value for the "storage_type" setting of the Image Store service settings.
• Fill in the settings for connecting to an S3-compatible storage (host, Access Key and Secret Key, etc.) in the "S3" section of the Image Store service settings.
• Run the script for creating buckets lis_bucket_create.py (see the "Create buckets" section in the installation manual).

If necessary, you can disable SSL certificate verification using the "verify_ssl" setting in the "S3" section of the Image Store service settings. This enables you to use a self-signed SSL certificate.

Object TTL#

You can set the object time to live (TTL) in buckets (both local and S3). Objects mean:

• Samples of faces or bodies.
• Images or objects created in the resources "/images" or "/objects".
• Source images.
• Task results.

TTL for objects is calculated relative to the GMT time format.

TTL for objects is set in days in the following ways:

• During the creation of a bucket for all objects at once (basic TTL bucket policy).
• After creating a bucket for specific objects using requests to the corresponding resources.

The number of days is selected from the list in the corresponding requests (see below).

In addition to the number of days, the parameter can take the value "-1", meaning that objects should be stored indefinitely.

Configuring basic TTL bucket policy#

The basic TTL bucket policy can be configured in the following ways:

• Using the --bucket-ttl flag for the lis_bucket_create.py script. For example, python3 ./base_scripts/lis_bucket_create.py -ii --bucket-ttl=2.
• Using a request to the Image Store service. For example, curl -X POST http://127.0.0.1:5020/1/buckets?bucket=visionlabs-samples?ttl=2.

Configuring TTL for specific objects#

TTL for specific objects can be configured using the "ttl" parameter in the following places:

• In the "storage_policy" > "face_sample_policy", "body_sample_policy" and "image_origin_policy" handler policies.
• In the requests "create object", "create images" and "save sample".
• In the requests to create tasks or schedules in the "result_storage_policy" field.

If the "ttl" parameter is not specified, then the basic policy of the bucket in which the object is located (see above) will be applied.

Adding TTL to existing objects#

You can add a TLL to an existing object using PUT requests to the /objects, /images, /samples/{sample_type} resources of the Image Store service. It is not possible to add TTL of task results to already created and executed tasks. You can add TTL of task results to an already created schedule using the request "replace tasks schedule". For tasks created or running at the time of the request, the TTL of task results will not be applied.

You can add a TTL to an existing local bucket using a PUT request to the Image Store resource /buckets.

To add a TTL for a bucket located in S3, you need to perform a migration using the "base_scripts/migrate_ttl_settings" script from the Image Store service. This is because for TTL objects in S3 is applied via tag related filters. The command to perform S3 bucket migration is given in the installation manual. See "Migration to apply TTL to objects in S3" for details on S3 bucket migration.

Supported cloud providers#

Amazon S3 cloud providers, Yandex cloud storage and MinIO are supported.

Migration to apply TTL to objects in S3#

Lifecycle customization for S3 is applied through filters associated with tags (see official documentation). This assumes that objects have a tag with a limited set of values, and buckets have a set of rules based on the value of that tag.

To add tags and rules, you must perform a migration. Migration is strictly necessary to fully apply lifecycle customization for the following reasons:

• Buckets without rules will not delete objects, even if the user specifies a lifetime for a specific object.
• Objects without tags will never be deleted, even if the user specifies a lifetime for the bucket.

You need to add the following tags and rules:

• To support TTL for buckets, you need to add a vl-expire tag with a default value for all existing objects.
• To support TTL for specific objects, you need to add a set or TTL-related lifecycle rules for existing segments:

{
     "ID": "vl-expire-<ttl>",
     "Expiration": {
         "Days": <ttl>,
     },
     "Filter": {"Tag": {"Key": "vl-expire", "Value": <ttl>}},
     "Status": "Enabled",
}

A set of specific tag values associated with an object's TTL is supported: 1, 2, 3, 4, 5, 6, 7, 14, 30, 60, 90, 180, 365.

The migration process consists of two stages:

• Configuration of the bucket life cycle, expanded with a set of life cycle rules related to TTL.
• Assigning a vl-expire tag to each object in the bucket, if it does not already have one.

Assigning a tag for each object can be skipped if necessary using the -update-tags=0 argument.

See the upgrade manual for example commands to perform the migration.

Permission issues

By default, all S3 resources, including buckets, objects, and lifecycle configuration, are private. If necessary, default rules and tags can be created manually by the resource owner using one of the applicable methods. See the official S3 documentation for details.

Useful links to official documentation:

• Creating a lifecycle configuration
• Put lifecycle configuration
• Managing object tags
• Expiring objects

Expiration of TTL#

When an object's TTL comes to an end, it is marked for deletion. For local buckets, the cleanup task is performed once a day (at 01:00 am). S3 buckets use internal ttl configuration rules. To prevent conflicts or duplication of cleanup tasks when multiple instances or worker processes are involved, a locking mechanism is implemented. This ensures that only one instance or worker process is responsible for performing the local storage cleanup process.

There may be a delay between the expiration date and the date the item is actually deleted. Both for local storage and S3.

Search for expiring objects#

To find out when an object expires, you can use queries with the HEAD methods on the /objects and /images resources. These requests return X-Luna-Expiry-Date response headers, which indicate the date on which the object is no longer eligible for persistence.

External samples#

You can send an external sample to Image Store. The external sample is received using third-party software or the VisionLabs software (e. g., FaceStream).

See the POST request on the "/samples/{sample_type}" resource in "APIReferenceManual.html" for details.

The external sample should correspond to certain standards so that LP could process it. Some of them are listed in the "Sample requirements" section.

The samples received using the VisionLabs software satisfy this requirement.

In case of third-party software, it is not guaranteed that the result of the external sample processing will be the same as for the VisionLabs sample. The sample can be of low quality (too dark, blurry and so on). Low quality leads to incorrect image processing results.

Anyway, it is recommended to consult VisionLabs before using external samples.

Accounts service#

The Accounts service is intended for:

• Creation, management and storage of accounts.
• Creation, management and storage of tokens and their permissions.
• Verification of accounts and tokens.

See "Accounts, tokens and authorization types" section for more information about the authorization system in LUNA PLATFORM 5.

All created accounts, tokens and their permissions are saved in the Accounts service database.

JWT Tokens algorithms#

The JWT (JSON Web Tokens) authentication mechanism supports various algorithms for token signing. This section describes the default algorithm used and the necessary steps to use an alternative algorithm.

Default algorithm#

By default, the service uses the HS256 algorithm to sign JWT tokens. If you want to use asymmetric cryptographic encryption, you can use the ES256 algorithm.

Use ES256 algorithm#

To use the ES256 algorithm, follow these steps:

1. Generate a private ECDSA key.

   First, you need to generate a private ECDSA key using the prime256v1 curve. This can be done using command-line tools such as OpenSSL.

   Example command:

   openssl ecparam -genkey -name prime256v1 -out ec_private.pem

   You can also generate a key protected by a password, for example:

   openssl ecparam -genkey -name prime256v1 | openssl ec -aes256 -out ec_private_enc.pem

2. Encode the private key in Base64.

   After generating the private key, encode it in Base64 format. This can be achieved with tools available in most operating systems.

   Example command:

   base64 -w0 ecdsa_private.pem > ecdsa_private_base64

3. Set the environment variable.

   The encoded private key must be specified in the ACCOUNTS_JWT_ECDSA_KEY environment variable when starting the container. This allows the service to use the key for signing JWT tokens with the ES256 algorithm.

   Additionally, if your private key is protected with a password, you can specify the password in the ACCOUNTS_JWT_ECDSA_KEY_PASSWORD environment variable.

   Example container run command with environment variables:

   docker run \ --env=CONFIGURATOR_HOST=127.0.0.1 \ --env=ACCOUNTS_JWT_ECDSA_KEY=jwt_ecdsa_key \ --env=ACCOUNTS_JWT_ECDSA_KEY_PASSWORD=ecdsa_key_password \ ...

By following these steps, the service will be able to sign JWT tokens using the ES256 algorithm, providing enhanced security through asymmetric cryptography.

Impact of changing algorithm type#

Switching the signing algorithm from HS256 to ES256 (or vice versa) has a significant impact on token validation. All existing tokens signed with the previous algorithm will become invalid after the changes are made. This happens because the token signature verification mechanism expects the structure and cryptographic base of the token to match the newly specified algorithm.

Faces service#

Faces service is used for:

• Creating temporary attributes.
• Creating faces.
• Creating lists.
• Attaching faces to lists.
• Managing of the general database that stores faces with the attached data and lists.
• Receive information about the existing faces and lists.

Matching services#

Python Matcher has the following features:

• Matching according to the specified filters. This matching is performed directly on the Faces or the Events database. Matching by DB is beneficial when several filters are set.
• Matching by lists. In this case, it is recommended that descriptors are save in the Python Matcher cache.

Python Matcher Proxy is used to route requests to Python Matcher services and matching plugins.

Python Matcher#

Python Matcher utilizes Faces DB for filtration and matching when faces are set as candidates for matching and filters for them are specified. This feature is always enabled for Python Matcher.

Python Matcher utilizes Events DB for filtration and matching when events are set as candidates for matching and filters for them are specified. The matching using the Events DB is optional, and it is not used when the Events service is not utilized.

A VLMatch matching function is required for matching by DB. It should be registered for the Faces DB and the Events DB. The function utilizes a library that should be compiled for your current DB version. You can find information about it in the installation manual in "VLMatch library compilation", "Create VLMatch function for Faces DB", and "Create VLMatch function for Events DB" sections.

Python Matcher service additionally uses workers that process requests.

Python Matcher Proxy#

The API service sends requests to the Python Matcher Proxy if it is configured in the API configuration. Then the Python Matcher Proxy service redirects requests to the Python Matcher service or to matching plugins (if they are used).

If the matching plugins are not used, then the service route requests only to the Python Matcher service. Thus, you don't need to use Python Matcher Proxy unless you intend to use matching plugins. See the "Matching plugins" section for a description of how the matching plugins work.

List caching#

When faces are specified as candidates for matching and list IDs for them are specified as filters, Python Matcher performs a matching by lists.

By default, when the Python Matcher service is launched, all descriptors in all lists are cached in its memory.

Caching is managed by the "DESCRIPTORS_CACHE" section.

The Python Matcher service will not start until it loads all available descriptors into the cache.

When executing a list matching request, the Python Matcher service automatically adds it to the queue, from where it is picked up by the worker and sent to the Cached Matcher entity to perform a matching on cached data.

After performing the matching, the worker takes the results and returns them to the Python Matcher service and the user.

This caching enabling you to significantly increase the performance of the matching.

If necessary, you can process only specific lists using the parameter "cached_data > faces_lists > include" or exclude lists using the parameter "cached_data > faces_lists > exclude". The latter is especially useful when working with the LUNA Index Module to implement the logic of processing parts of lists using Python Matcher, and parts using LIM Indexed Matcher.

For more information about LIM, see "Matching a large set of descriptors".

Workers cache#

When multiple workers are launched for the Python Matcher service, each of the workers uses the same descriptors cache.

This change can both speed up and slow down the service. If you need to ensure that the cache is stored in each of the Python Matcher workers, you should run each of the server instances separately.

Events service#

The Events service is used for:

• Storage of all the created events in the Events database.
• Returning all the events that satisfy filters.
• Gathering statistics on all the existing events according to the specified aggregation and frequency/period.
• Storage of descriptors created for events.

As the event is a report, you can't modify already existing events.

The Events service should be enabled in the API service configuration file. Otherwise, events will not be saved to the database.

Database for Events#

PostgreSQL is used as a database for the Events service.

The speed of request processing is primarily affected by:

• The number of events in the database.
• Lack of indexes for PostgreSQL.

PostgreSQL shows acceptable requests processing speed with the number of events from 1 000 000 to 10 000 000. If the number of events exceeds 10 000 000, the request to PostgreSQL may fail.

The speed of the statistics requests processing in the PostgreSQL database can be increased by configuring the database and creating indexes.

Geo position#

You can add a geo position during event creation.

The geo position is represented as a JSON with GPS coordinates of the geographical point:

• "longitude" — Geographical longitude in degrees.
• "latitude" — Geographical latitude in degrees.

The geo position is specified in the "location" body parameter of the event creation request. See the "Create new events" section of the Events service reference manual.

You can use the geo position filter to receive all the events that occurred in the required area.

Geo position filter#

A geo position filter is a bounding box specified by coordinates of its center (origin) and some delta.

It is specified using the following parameters:

• "origin_longitude"
• "origin_latitude"
• "longitude_delta"
• "latitude_delta"

The geo position filter can be used when you get events, get statistics on events, and perform events matching.

Geo position filter is considered as properly specified if:

• both "origin_longitude" and "origin_latitude" are set.
• neither "origin_longitude", "origin_latitude", "longitude_delta", or "latitude_delta" is set.

If both "origin_longitude" and "origin_latitude" are set and "longitude_delta" is not set — the default value is applied (see the default value in the OpenAPI documentation).

Read the following recommendations before using geo position filters.

The general recommendations and restrictions for geo position filters are:

• Do not create filters with a vertex or a border on the International Date Line (IDL), the North Pole or the South Pole. They are not fully supported due to the features of database spatial index. The filtering result may be unpredictable.
• Geo position filters with edges more than 180 degrees long are not allowed.
• It is highly recommended to use the geo position filter citywide only. If a larger area is specified, the filtration results on the borders of the area can be unexpected due to the spatial features.
• Avoid creating a filter that is too extended along longitude or latitude. It is recommended to set the values of deltas close to each other.

The last two recommendations exist due to the spatial features of the filter. According to these features, when one or two deltas are set to large values, the result may differ from the expected though it will be correct. See the "Filter features" section for details.

Filter performance#

Geo position filter performance depends on the spatial data type used to store event geo position in the database.

Two spatial data types are supported:

• GEOMETRY. Spatial object with coordinates expressed as (longitude, latitude) pairs, defined in the Cartesian plane. All calculations use Cartesian coordinates.
• GEOGRAPHY. Spatial object with coordinates expressed as (longitude, latitude) pairs, defined as on the surface of a perfect sphere, or a spatial object in the WGS84 coordinate system.

For a detailed description, see geometry vs geography.

Geo position filter is based on the ST_Covers PostGIS function supported for both Geometry and Geography type.

Filter features#

Geo position filter has some features caused by PostGIS.

When geography type is used and the geo position filter covers a wide portion of the planet surface, filter result may be unexpected but geographically correct due to some spatial features.

The following example illustrates this case.

An event with the following geo position was added in the database:

{
    "longitude": 16.79,
    "latitude": 64.92,
}

We apply a geo position filter and try to find the required point on the map. The filter is too extended along the longitude:

{
    "origin_longitude": 16.79,
    "origin_latitude": 64.92,
    "longitude_delta": 2,
    "latitude_delta": 0.01,
}

This filter will not return the expected event. The event will be filtered due to spatial features. Here is the illustration showing that the point is outside the filter.

You should consider this feature to create a correct filter.

For details, see Geography.

Events creation#

Events are created using handlers. Handlers are stored in the Handlers database. You should specify the required handler ID in the event creation request. All the data stored in the event will be received according to the handler parameters.

You should perform two separate requests for event creation.

The first request creates a handler. Handler includes policies that describes how the image is processed hence defining the LP services used for the processing.

The second request creates new events using the existing handler. An event is created for each image that has been processed.

You can specify the following additional data for each event creation request:

• external ID (for created faces)
• user data (for created faces)
• source (for created events)
• tags (for created events)

The handler is processed policy after policy. All the data from the request is processed by a policy before going to the next policy. The "detect" policy is performed for all the images from the request, then "multiface" policy is applied, then the "extract" policy is performed for all the received samples, etc. For more information about handlers, see the "Handlers description" section.

Events meta-information#

If any additional data needs to be stored along with the event, the "meta" field should be used. The "meta" field stores data in the JSON format. The total size of the data stored in the "meta" field for one event cannot exceed 2 MB. It is assumed that with the help of this functionality, the user will create his own data model (event structure) and will use it to store the necessary data.

Note that you cannot specify field names with spaces in the "meta" field.

Data in the "meta" field can be set in the following ways:

• In the "generate events" request body with the content type application/json or multipart/form-data.
• In the "save event" request body.
• Using a custom plugin or client application.

In the "generate events" request body, it is possible to set the "meta" field both for specific images and for all images at once (mutual meta-information). For requests with aggregation enabled, only mutual meta-information will be used for the aggregated event, and meta-information for specific images will be ignored. See the detailed information in the "generate events" request body in the OpenAPI specification.

Example of recording the "meta" field:

{
    "meta": {
        "user_info": {
            "temperature": 36.6
        }
    }
}

In order to store multiple structures, it is necessary to explicitly separate them to avoid overlapping fields. For example, as follows:

{
    "struct1": {
        ...
    },
    "struct2": {
        ...
    }
}

Search by "meta" field#

You can get the contents of the "meta" field using the appropriate filter in the "get events" request.

The filter should be entered using a specific syntax — meta.<path.to.field>__<operator>:<type>, where:

• meta. — An indication that the "meta" field of the Events database is being accessed.
• <path.to.field> — Path to the object. A dot (.) is used to navigate nested objects. For example, in the string {"user_info":{"temperature":"36.6"}} to refer to the temperature object, use the following filter meta.user_info.temperature.
• __<operator> — One of the following operators — eq (default), neq, like, nlike, in, nin, gt, gte, lt, lte. For example, meta.user_info.temperature__gte;
• :type — one of the following data types — string, integer, numeric. For example, meta.user_info.temperature__gte:numeric.

For each operator, the use of certain data types is available. See the table of operator dependency on data types in the OpenAPI specification.

If necessary, you can build an index to improve the search. See the Events developer manual for details on building an index.

Important notes#

When working with the "meta" field, remember the following:

• you need to keep data consistent with given schemes; in case of a mismatch, PostgreSQL will not allow inserting a row with a type that cannot be added to the existing index (if any);
• if necessary, you can migrate data;
• if necessary, you can build an index;
• specify the data type when performing a request (by default, all values are assumed to be strings);
• you need to pay attention to the names of the fields; fields to be filtered by must not contain reserved keywords like :int, double underscores, special symbols, and so on.

Sender service#

The Sender service is an additional service that is used to send events via web sockets. This service communicates with the Handlers service (in which events are created) through the pub/sub mechanism via the Redis DB channel.

If necessary, you can send notifications over the HTTP protocol. See the "Send events to third-party service" section for more details.

Events are created based on handlers. To receive notifications, the "notification_policy" must be enabled. This policy has filters that enable you to send notifications only under certain conditions, for example, to send only if the candidate is very similar to the reference (the "similarity__lte" parameter).

You should configure web sockets connection using special request. It is recommended create web sockets connection using the "/ws" resource of the API service. You can specify filters (query parameters) in the request, i.e. you can configure the Sender service to receive only certain events. See OpenAPI specification for detailed information about the configuration of creating a connection to a web socket.

Configuring web sockets directly via Sender is also available (see "/ws" of the Sender service). It can be used to reduce the load on the API service.

When an event is created it can be:

• Saved to the Events database. The Events service should be enabled to save an event.

• Returned in the response without saving to the database.

In both cases, the event is sent via the Redis DB channel to the Sender service.

In this case, the Redis DB acts as a connection between Sender and Handlers services and does not store transferred events.

The Sender service is independent of the Events service. Events can be sent to Sender even if the Events service is disabled.

Creating handlers and specifying filters for sending notifications

1. The user sends the "create handler" request to the API service, where it enables the "notification_policy" and sets filters according to which events will be sent to the Sender service.
2. The API service sends a request to the Handlers service.
3. The Handlers service sends a response to the API service.
4. The API service sends the "handler_id" to the user.

The user saves the ID "handler_id", which is necessary for creating events.

Activation of subscription to events and filtering of their sending

1. The user or application sends a request "ws handshake" to the API service and sets filters through which it will be possible to filter the received data from the Handlers service.
2. The API service sends a request to the Sender service.
3. The Sender service establishes a connection via web sockets with the user application.

Now, when an event is generated, it will be automatically redirected to the Sender service (see below) in accordance with the specified filters.

Event generation and sending to Sender

The general workflow is as follows:

1. A user or an application sends the "generate events" request to the API service.
2. The API service sends the request to the Handlers service.
3. The Handlers service sends requests to the corresponding LP services.
4. LP services process the requests and send results. New events are created.
5. The Handlers service sends an event to the Redis database using the pub/sub model. Redis has a channel to which the Sender service is subscribed, and it is waiting for messages to be received from this channel.
6. Redis sends the received events to Sender by the channel.
7. Third-party party applications should be subscribed to the Sender service via web sockets to receive events. If there is a subscribed third-party party application, Sender sends events to it according to the specified filters.

See the OpenAPI documentation for information about the JSON structure returned by the Sender service.

Tasks service#

The Tasks service is used for long tasks processing.

General information about tasks#

As tasks processing takes time, the task ID is returned in the response to the task creation.

After the task processing is finished, you can receive the task results using the "task " > "get task result" request. You should specify the task ID to receive its results.

You can find the examples of tasks processing results in the response section of "task " > "get task result" request. You should select the task type in the Response samples section of documentation.

You should make sure that the task was finished before requesting its results:

• You can check the task status by specifying the task ID in the "tasks" > "get task" request. There are the following task statuses:

• You can receive information about all the tasks using the "tasks" > "get tasks" request. You can set filter to receive information about tasks of interest only.

Clustering task#

As the result of the task a cluster with objects selected according to the specified filters for faces or events is created. Objects corresponding to all of the filters will be added to the cluster. Available filters depend on the object type: events or faces.

You can receive the task status or result using additional requests (see the "General information about tasks").

You can use the reporter task to receive the report about objects added to clusters.

Clustering is performed in several steps:

• Objects with descriptors are collected according to provided filters

• Every object is matched with all the other objects

• Create clusters as groups of "connected components" from the similarity graph.

Here "connected" means that similarity is greater than provided threshold or default "DEFAULT_CLUSTERING_THRESHOLD" from the config.

• If needed, download existing images corresponding to each object: avatar for a face, first sample for an event.

As a result of the task an array of clusters is returned. A cluster includes IDs of objects (faces or events) whose similarity is greater then the specified threshold. You can use the information for further data analysis.

{
    "errors": [],
    "result": {
        "clusters": [
            [
                "6c721b90-f5a0-409a-ab70-bc339a70184c"
            ],
            [
                "8bc6e8df-410b-4065-b592-abc5f0432a1c"
            ],
            [
                "e4e3fc66-53b4-448c-9c88-f430c00cb7ea"
            ],
            [
                "02a3a1c4-93d7-4b69-99ec-21d5ef23852e",
                "144244cb-e10e-478c-bdac-18cd2eb27ee6",
                "1f4cdbcb-7b1e-40cc-873b-3ff7fa6a6cf0"
            ]
        ],
        "total_objects": 6,
        "total_clusters": 4
    }
}

The clustering task result can also include information about errors occurred during the objects processing.

For such a task, you can create a schedule.

Reporter task#

As a result of the task, the report on the clustering task is created. You can select data that should be added to the report. The report has CSV format.

You can receive the task status or result using additional requests (see the "General information about tasks").

You should specify the clustering task ID and the columns that should be added to the report. The selected columns correspond to the general events and faces fields.

Make sure that the selected columns correspond to the objects selected in the clustering task.

You can also receive the images for all the objects in clusters if they are available.

Exporter task#

The task enables you to collect event and/or face data and export them from LP to a CSV file. The file rows represent requested objects and corresponding samples (if they were requested).

This task uses memory when collecting data. So, its possible that Tasks Worker will be killed by OOM (Out-Of-Memory) killer if you request a lot of data.

You can export event or face data using the "/tasks/exporter" request. You should specify what type of object is required by setting objects_type parameter when creating a request. You can also narrow your request by providing filters for faces and events objects. See the "exporter task" request in the API service reference manual.

As a result of the task a zip archive containing a CSV file is returned.

You can receive the task status or result using additional requests (see the "General information about tasks").

When executing the Exporter task with a large number of faces in the Faces database (for example, 90,000,000 faces), the execution time of requests to the Faces service can be significantly increased. To speed up request execution, you can set the PostgreSQL setting "parallel_setup_cost" to 500. However, be aware that changing this setting may have other consequences, so you should be careful when changing the setting.

For such a task, you can create a schedule.

Cross-matching task#

When the task is performed, all the references are matched with all the candidates. References and candidates are set using filters for faces and events.

Matching is performed only for objects that contain extracted descriptors.

You can specify the maximum number of matching candidates returned for every match using the limit field.

You can set a threshold to specify the minimal acceptable value of similarity. If the similarity of two descriptors is lower then the specified value, the matching result will be ignored and not returned in the response. References without matches with any candidates are also ignored.

Cross-matching is performed in several steps:

• collect objects having descriptors using provided filters
• match every reference object with every candidate object
• match results are sorted (lexicographically) and cropped (limit and threshold are applied)

You can receive the task status or results using additional requests (see the "General information about tasks").

As a result an array is returned. Each element of the array includes a reference and top similar candidates for it. Information about errors occurred during the task execution is also returned in the response.

{
"result": [
    {
        "reference_id": "e99d42df-6859-4ab7-98d4-dafd18f47f30",
        "candidates": [
            {
                "candidate_id": "93de0ea1-0d21-4b67-8f3f-d871c159b740",
                "similarity": 0.548252
            },
            {
                "candidate_id": "54860fc6-c726-4521-9c7f-3fa354983e02",
                "similarity": 0.62344
            }
        ]
    },
    {
        "reference_id": "345af6e3-625b-4f09-a54c-3be4c834780d",
        "candidates": [
            {
                "candidate_id": "6ade1494-1138-49ac-bfd3-29e9f5027240",
                "similarity": 0.7123213
            },
            {
                "candidate_id": "e0e3c474-9099-4fad-ac61-d892cd6688bf",
                "similarity": 0.9543
            }
        ]
    }
],
"errors": [
    {
        "error_id": 10,
        "task_id": 123,
        "subtask_id": 5,
        "error_code": 0,
        "description": "Faces not found",
        "detail": "One or more faces not found, including face with id '8f4f0070-c464-460b-bf78-fac225df72e9'",
        "additional_info": "8f4f0070-c464-460b-bf78-fac225df72e9",
        "error_time": "2018-08-11T09:11:41.674Z"
    }
]
}

For such a task, you can create a schedule.

Linker task#

The task enables you to attach faces to lists according to the specified filters.

You can specify creation of a new list or specify the already existing list in the requests.

You can specify filters for faces or events to perform the task. When an event is specified for linking to list a new face is created based on the event.

If the create_time_lt filter is not specified, it will be set to the current time.

As the result of the task you receive IDs of faces linked to the list.

You can receive the task status or result using additional requests (see the "General information about tasks").

Task execution process for faces:

• A list is created (if create_list parameter is set to 1) or the specified list_id existence is checked.
• Face ID boundaries are received. Then one or several subtasks are created with about 1000 face ids per each. The number depends on face ID spreading.

• For each subtask:

  • Face IDs are received. They are specified for the current subtask by filters in the subtask content.
  • The request is sent to the Luna Faces to link specified faces to the specified list.
  • The result for each subtask is saved to the Image Store service.

• After the last subtask is finished, the worker collects results of all the subtasks, merges them and puts them to the Image Store service (as task result).

Task execution process for events:

• A list is created (if create_list parameter is set to 1) or the specified list_id existence is checked.
• Events page numbers are received. Then one or several subtasks are created.

• For each subtask:

  • Event with their descriptors are received from the Events service.
  • Faces are created using the Faces service. Attribute(s) and sample(s) are added to the faces.
  • The request is sent to the Luna Faces to link specified faces to the specified list.
  • The result for each subtask is saved to the Image Store service.

• After the last subtask is finished, the worker collects results of all the subtasks, merges them and puts them to the Image Store service (as task result).

For such a task, you can create a schedule.

Garbage collection task#

During the task processing, faces, events or descriptors can be deleted.

• When descriptors are set as a GC target, you should specify the descriptor version. All the descriptors of the specified version will be deleted.
• When events are set as a GC target, you should specify one or several of the following parameters:
  • Account ID.
  • Upper excluded boundary of event creation time.
  • Upper excluded boundary of the event appearance in the video stream.
  • Handler ID used for the event creation.
• When faces are set as a GC target, you should specify one or several of the following parameters:
  • Upper excluded boundary of face creation time.
  • Lower included boundary of face creation time.
  • User data.
  • List ID.

If necessary, you can delete samples along with faces or events. You can also delete image origins for events.

Garbage collection task with faces or events set as the target can be processed using the API service API, while the Admin or Task services API can be used to set faces, events and descriptors as the target. Thus the specified objects will be deleted for all the existing accounts.

You can receive the task status or result using additional requests (see the "General information about tasks").

For such a task, you can create a schedule.

Additional extraction task#

The Additional extraction task re-extracts descriptors extracted using the previous neural network model using a new version of the neural network. This enables you to save previously used descriptors when updating the neural network model. If there is no need to use the old descriptors, then you can not perform this task and only update the neural network model in the Configurator settings.

This section describes how to work with the Additional extraction task. See detailed information about neural networks, the process of updating a neural network to a new model and relevant examples in the "Neural networks" section.

Re-extraction can be performed for face and event objects. You can re-extract the descriptors of faces, descriptors of bodies (for events) or basic attributes if they were not extracted earlier.

The samples for descriptors should be stored for the task execution. If any descriptors do not have source samples, they cannot be updated to a new NN version.

The re-extraction tasks are used for the update to a new neural network for descriptors extraction. All the descriptors of the previous version will be re-extracted using a new NN.

It is highly recommended not to perform any requests changing the state of databases during the descriptor version updates. It can lead to data loss.

Create backups of LP databases and the Image Store storage before launching the additional extraction task.

When processing the task, a new neural network descriptor is extracted for each object (face or event) whose descriptor version matches the version specified in the "DEFAULT_FACE_DESCRIPTOR_VERSION" (for faces) or "DEFAULT_HUMAN_DESCRIPTOR_VERSION" (for bodies) settings. Descriptors whose version does not match the version specified in these settings are not re-extracted. They can be removed using the Garbage collection task.

Request to the Admin service

You need to make a request to the "additional_extract" resource, specifying the following parameters in the request body:

• "content" > "extraction_target" – Face descriptors, body descriptors, basic attributes.
• "content" > "options" > "descriptor_version" – New neural network version (not applicable for basic attributes).
• "content" > "filters" > "object_type" – Faces or events.

If necessary, you can additionally filter the object type by "account_id", "face_id__lt", etc.

See the "create additional extract task" request in the Admin service OpenAPI specification for more information.

You can receive the task status or result using additional requests (see the "General information about tasks").

Admin user interface

You need to do the following:

• Go to the Admin user interface: http://<admin_server_ip>:5010/tasks.

• Run the additional extraction task using the corresponding button.

• In the window that appears, set the object type (face or event), the extraction type (face descriptor, body descriptor or basic attributes), new neural network model (not applicable for basic attributes) and click "Start", confirming the start of the task.

If necessary, you can additionally filter the object type by "account_id".

See the detailed information about the Admin user interface in the "Admin user interface" section.

For such a task, you can create a schedule.

ROC-curve calculating task#

As a result of the task, the Receiver Operating Characteristic curve with TPR (True Positive Rate) against the FPR (False Positive Rate) is created.

See additional information about ROC-curve creation in "TasksDevelopmentManual".

ROC calculation task

ROC (or Receiver Operating Characteristic) is a performance measurement for classification tasks at various thresholds settings. The ROC-curve is plotted with TPR (True Positive Rate) against the FPR (False Positive Rate). TPR is a true positive match pair count divided by a count of total expected positive match pairs, and FPR is a false positive match pair count divided by a count of total expected negative match pairs. Each point (FPR, TPR) of the ROC-cure corresponds to a certain similarity threshold. See more at wiki.

Using ROC the model performance is determined by looking at:

• Area under the ROC-curve (or AUC).
• Type I and type II error rates equal point, i.e. the ROC-curve and the secondary main diagonal intersection point.

The model performance also determined by hit into the top-N probability, i.e. probability of hit a positive match pair into the top-N for any match result group sorted by similarity.

It requires "markup" to make a ROC task. One can optionally specify "threshold_hit_top" (default 0) to calculate hit into the top-N probability, the match "limit" (default 5), "key_FPRs" — list of key FPR values to calculate ROC-curve key points, and "filters" with "account_id". Also, it needs "account_id" for task creation.

You can receive the task status or result using additional requests (see the "General information about tasks").

Markup

Markup is expected in the following format:

[{'face_id': <face_id>, 'label': <label>}]

Label (or group id) can be a number or any string.

Example:

[{'face_id': '94ae2c69-277a-4e46-817d-543f7d3446e2', 'label': 0},
 {'face_id': 'cd6b52be-cdc1-40a8-938b-a97a1f77d196', 'label': 1},
 {'face_id': 'cb9bda07-8e95-4d71-98ee-5905a36ec74a', 'label': 2},
 {'face_id': '4e5e32bb-113d-4c22-ac7f-8f6b48736378', 'label': 3},
 {'face_id': 'c43c0c0f-1368-41c0-b51c-f78a96672900', 'label': 2}]

For such a task, you can create a schedule.

Estimator task#

The estimator task enables you to perform batch processing of images using the specified policies.

As a result of the task performing, JSON is returned with data for each of the processed images and information about the errors that have occurred.

In the request body, you can specify the handler_id of an already existing static or dynamic handler. For the dynamic handler_id, the ability to set the required policies is available. In addition, you can create a static handler specifying policies in the request.

The resource can accept five types of sources with images for processing:

• ZIP archive
• S3-like storage
• Network disk
• FTP server
• Samba network file system

To obtain correct results of image processing using the Estimator task, all processed images should be either in the source format or in the format of samples. The type of transferred images is specified in the request in the "image_type" parameter.

For such a task, you can create a schedule. When creating a schedule, it is not possible to specify a ZIP archive as an image source.

ZIP archive as image source of estimator task

The resource accepts for processing a link to a ZIP archive with images. The size of the archive is set using the "ARCHIVE_MAX_SIZE" parameter in the "config.py" configuration file of the Tasks service. The default size is 100 GB. An external URL or the URL to an archive saved in the Image Store can be used as a link to the archive. In the second case, the archive should first be saved to the LP using a POST request to the "/objects" resource.

When using an external URL, the ZIP archive is first downloaded to the Tasks Worker container storage, where the images are unpacked and processed. After the end of the task, the archive is deleted from the repository along with the unpacked images.

It is necessary to take into account the availability of free space for the above actions.

The archive can be password protected. The password can be passed in the request using the "authorization" -> "password" parameter.

S3-like storage as image source of estimator task

The following parameters can be set for this type of source:

• "bucket_name" — Bucket name/Access Point ARN/Outpost ARN (required).
• "endpoint" — Storage endpoint (only when specifying the bucket name).
• "region" — Bucket region (only when specifying the bucket name).
• "prefix" — File key prefix. It can also be used to load images from a specific folder, such as "2022/January".

The following parameters are used to configure authorization:

• Public access key (required)
• Secret access key (required)
• Authorization signature version ("s3v2"/"s3v4")

It is also possible to recursively download images from nested bucket folders and save original images.

For more information about working with S3-like repositories, see AWS User Guide.

Network disk as image source of estimator task

The following parameters can be set for this type of source:

• "path" — Absolute path to the directory with images in the container (required).
• "follow_links" — Enables/disables symbolic link processing.
• "prefix" — File key prefix.
• "postfix" — File key postfix.

See an example of using prefixes and postfixes in the "/tasks/estimator" resource description.

When using a network disk as an image source and launching Tasks and Tasks Worker services through Docker containers, it is necessary to mount the directory with images from the network disk to the local directory and synchronize it with the specified directory in the container. You can mount a directory from a network disk in any convenient way. After that, you can synchronize the mounted directory with the directory in the container using the following command when launching the Tasks and Tasks Worker services:

docker run \
...
-v /var/lib/luna/current/images:/srv/images
...

Here:

• /var/lib/luna/current/images — Path to the previously mounted directory with images from the network disk.
• /srv/images- Path to the directory with the images in the container where they will be moved from the network disk. This path should be specified in the request body of the Estimator task (the "path" parameter).

As for S3-like storage, the ability to recursively download images from nested bucket folders is available.

FTP server as image source of estimator task

For this type of source, the following parameters can be set in the request body for connecting to the FTP server:

• "host" — FTP server IP address or hostname (required).
• "port" — FTP server port.
• "max_sessions" — Maximum number of allowed sessions on the FTP server.
• "user", "password" — Authorization parameters (required).

As in Estimator tasks using S3-like storage or network disk as image sources, it is possible to set the path to the directory with images, recursively receive images from nested directories, select the type of transferred images, and specify the prefix and postfix.

See an example of using prefixes and postfixes in the "/tasks/estimator" resource description.

Samba as image source of estimator task

For this type of source, the parameters are similar to those of an FTP server, except for the "max_sessions" parameter. Also, if authorization data is not specified, the connection to Samba will be performed as a guest.

Task processing#

The Tasks service includes the Tasks service and Tasks workers. Tasks receives requests to the Tasks service, creates tasks in the DB and sends subtasks to Tasks workers. The workers are implemented as a separate Tasks Worker container. Tasks workers receive subtasks and perform all the required requests to other services to solve the subtasks.

The general approach for working with tasks is listed below.

• User sends the request for creation of a new task.
• Tasks service creates a new task and sends subtasks to workers.
• Tasks workers process subtasks and create reports.
• If several workers have processed subtasks and have created several reports, the worker, which finished the last subtask, gathers all the reports and creates a single report.
• When the task is finished, the last worker updated its status in the Tasks database.
• User can send requests to receive information about tasks and subtasks and number of active tasks. The user can cancel or delete tasks.
• User can receive information about errors that occurred during execution of the tasks.
• After the task is finished the user can send a request to receive results of the task.

See the "Tasks diagrams" section for details about tasks processing.

Running scheduled tasks#

In LUNA PLATFORM, it is possible to set a schedule for Garbage collection, Clusterization, Exporter, Linker, Estimator, Additional extract, Cross-matching and Roc-curve calculating tasks.

To use a filter relative to the current time ("now-time"), the current time will be counted not from the creation of the schedule, but from the creation of the task by the schedule in accordance with the cron expression. See "Now-time filters" for details.

The schedule is created using the request "create tasks schedule" to the API service, which specifies the contents of the task being created and the time interval for its launch. To specify the time interval, Cron expressions are used.

Cron expressions are used to determine the task execution schedule. They consist of five fields separated by spaces. Each field defines a specific time interval in which the task should be completed. The week number starts from Sunday.

For tasks that can only be performed using the Admin service (for example, the task of removing some objects using the GC task), you can assign a schedule only in the Admin service.

In response to the request, a "schedule_id" is issued, which can be used to get information about the status of the task, the time of the next task, etc. (requests "get tasks schedule and "get tasks schedules). The id and all additional information are stored in the "schedule" table of the Tasks database.

If necessary, you can create a delayed schedule, and then activate it using the "action" = "start" parameter of the "patch tasks schedule" request. Similarly, you can stop the scheduled task using "action" = "stop". To delete a schedule, you can use the "delete tasks schedule" request.

Permissions to work with schedules are specified in the token with the "task" permission. This means that if the user has permission to work with tasks, then he will also be able to use the schedule.

The possibility of schedule creation is also available for Lambda tasks.

Examples of Cron expressions#

This section describes various examples of Cron expressions.

1. Run the task every day at 3 a.m.:

0 3 * * *

1. Run the task every Friday at 18:30:

30 18 * * 5

1. Run the task every first day of the month at noon:

0 12 1 * *

1. Run the task every 15 minutes:

*/15 * * * *

1. Run the task every morning at 8:00, except weekends (Saturday and Sunday):

0 8 * * 1-5

1. Run the task at 9:00 am on the first and 15th day of each month, but only if it is Monday:

0 9 1,15 * 1

Send notification about task and subtask status changes#

If necessary, you can send notifications about changes in task and subtask status using the callback mechanism. Callbacks allow you to send data to a third-party system at a specified URL or to Telegram. To configure notifications, you need to configure the "notification_policy" in the request parameters of the corresponding task.

You can also configure sending notifications for tasks and subtask in the schedule settings.

If necessary, you can obtain information about the current state of the notification policy or change some policy data using "get task notification policy" and "replace task notification policy" requests.

Additional protection for passwords and tokens#

Passwords and tokens passed in Estimator task and in "notification_policy" can be additionally encrypted. To do this, pass custom values to the FERNET_PASSPHRASE and SALT environment variables when starting the Tasks service container.

FERNET_PASSPHRASE is the password or key used to encrypt data using the Fernet algorithm.

SALT is a random string added to the password before it is hashed.

Fernet is a symmetric encryption algorithm that provides authentication and data integrity as well as confidentiality. With this algorithm, the same key is used to encrypt and decrypt data.

Salt is added to make it more difficult to crack the password by brute force. Each time a password is hashed, a unique string is used, making identical passwords hashed differently. This increases the security of the system, especially if users have the same passwords.

Example of a container startup command passing environment variables:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=FERNET_PASSPHRASE=security_passphrase.
--env=SALT=salt_for_passwords_and_tokens.
...

Important: When the container is started with the above environment variables specified, the old passwords and tokens will no longer work. Additional migration steps must be performed (see section below).

Add encryption when updating#

In order to add additional protection for already existing passwords and tokens, you need to specify environment variables in the Tasks database migration command (see above). After that, you should start a new Tasks container with environment variables specified to enable the use of encryption when creating new objects.

Admin service#

The Admin service is used to perform general administrative routines:

• Manage user accounts.
• Receive information about objects belonging to different accounts.
• Create garbage collection tasks.
• Create tasks to extract descriptors with a new neural network version.
• Receive reports and errors on processed tasks.
• Cancel and delete existing tasks.

Admin service has access to all the data attached to different accounts.

Three types of accounts can be created in the Admin service — "user", "advanced_user" and "admin". The first two types are created using an account creation request to the API service, but the third type can only be created using the Admin service.

Using the "admin" account type, you can log in to the interface and perform the above tasks. An account with the "admin" type can be created either in the user interface (see above) or by requesting the "/4/accounts" resource of the Admin service. To create an account in the last way, you need to specify a username and password.

If you are creating an account for the first time, you must use the default login and password.

Example of CURL request to the "/4/accounts" resource of the Admin service:

curl --location --request POST 'http://127.0.0.1:5010/4/accounts' \
--header 'Authorization: Basic cm9vdEB2aXNpb25sYWJzLmFpOnJvb3Q=' \
--header 'Content-Type: application/json' \
--data '{
  "login": "mylogin@gmail.com",
  "password": "password",
  "account_type": "admin",
  "description": "description"
}' 

All the requests for Admin service are described in Admin service reference manual.

Admin user interface#

The user interface of the Admin service is designed to simplify the work with administrative tasks.

The interface can be opened in a browser by specifying the address and port of the Admin service: <Admin_server_address>:<Admin_server_port>.

The default Admin service port is 5010.

The default login and password to access the interface are root@visionlabs.ai/root. You can also use default login and password in Base64 format — cm9vdEB2aXNpb25sYWJzLmFpOnJvb3Q=.

You can change the default password for the Admin service using the "Change authorization" request.

There are three tabs on the page:

• Accounts. The tab is designed to provide information about all created accounts and to create new accounts.
• Tasks — The tab is designed for working with Garbage collection and Additional extraction tasks.
• Info — The tab contains information about the user interface and the LUNA PLATFORM license.

Accounts tab#

This tab displays all existing accounts.

You can manage existing accounts using the following buttons:

– View account information.

– Delete account.

Clicking the view info button opens a page containing general information about the account, lists created with that account, and faces.

When you click the "Create account" button, an account creation window opens, containing the standard account creation settings — login, password, account type, description and the desired "account_id".

See "Account" for details on accounts and their types.

Tasks tab#

This tab displays running/completed Garbage collection and Additional extraction tasks.

.

Tasks are displayed in a table whose columns can be sorted and also filtered by the date the tasks were completed.

When you press the "Start Garbage collection" and "Start Additional extraction" buttons, windows for creating the corresponding tasks open.

The "Garbage collection" window contains the following settings, similar to the parameters of the "garbage collecting task" request body to the Tasks service:

• Description — "description" parameter
• Target — "content > target" parameter
• Account ID — "content > filters > account_id" parameter
• Remove sample — "content > remove_samples" parameter
• Remove image origins — "content > remove_image_origins" parameter
• Delete data before — "content > create_time__lt" parameter

See "Garbage collection task" for details.

The "Additional extraction" window contains the following settings, similar to the parameters of the "additional extract task" request body to the Tasks service:

• Objects type — "content > filters > object_type" parameter
• Extraction type — "content > extraction_target" parameter
• Descriptor version — "content > options > descriptor_version" parameter
• Description — "description" parameter
• Account ID — "content > filters > account_id" parameter

See "Additional extraction task" for details.

After creating a task, its execution begins. The progress of the task is displayed by the icon . The task is considered completed when the "Parts done" value matches the "Parts total" value and the icon changes to . If necessary, you can stop the task execution using the icon .

The following buttons are available for each task:

– download the task result as a JSON file.

– go to the page with a detailed description of the task and errors received during its execution.

– delete task.

Tasks are executed by the Tasks service after receiving a request from the Admin service.

Schedules tab#

This tab is intended for working with task scheduling.

The tab displays all created task schedules and all relevant information (status, ID, Cron string, etc.).

When you click on the "Create schedule" button, the schedule creation window opens.

In the window you can specify schedule settings for the Garbage collection task. The parameters in this window correspond to the parameters of the "create tasks schedule" request.

After filling in the parameters and clicking the "Create schedule" button, the schedule will appear in the Schedules tab.

You can control delayed start using the following buttons:

– start the schedule.

– pause the schedule.

Using the button, you can edit the schedule. Using the button you can delete a schedule.

Info tab#

This tab displays complete license information and features that can be performed using the Admin UI.

.

See the detailed license description in the "License information" section.

By clicking on the "Download system info" button, you can also get the following technical information about the LP:

• LUNA PLATFORM version
• Versions of the services
• Number of descriptors
• Configuration files values
• License information
• Requests and estimations statistics (see section "Requests and estimations statistics gathering")

You can also get the above system information using the "get system info" request to the Admin service.

Configurator service#

The Configurator service simplifies the configuration of LP services.

The service stores all the required configurations for all the LP services in a single place. You can edit configurations through the user interface or special limitation files.

You can also store configurations for any third-party party software in Configurator.

The general workflow is as follows:

• User edits configurations in the UI.
• Configurator stores all changed configurations and other data in the database.
• LP services request Configurator service during startup and receive all required configurations. All the services should be configured to use the Configurator service.

During Configurator installation, you can also use your limitation file with all the required fields to create limitations and fill in the Configurator database. You can find more details about this process in the Configurator development manual documentation.

Settings used by several services are updated for each of the services. For example, if you edit the "LUNA_FACES_ADDRESS" section for the Handlers service in the Configurator user interface, the setting will be also updated for API, Admin and Python Matcher services.

Configurator UI#

Open the Configurator interface in your browser: <Configurator_server_address> :5070

This URL may differ. In this example, the Configurator service interface is opened on the Configurator service server.

LP includes the beta version of the Configurator UI. The UI was tested on Chrome and Yandex browser. The recommended screen resolution for working with the UI is 1920 x 1080.

The following tabs are available in the UI of Configurator:

• Settings. All the data in the Configurator service is stored on the Settings tab. The tab displays all the existing settings. It also allows to manage and filter them;
• Limitations. The tab is used to create new limitations for settings. The limitations are templates for JSON files that contain available data type and other rules for the definition of the parameters;
• Groups. The tab allows to group all the required settings. When you select a group on the Settings tab, only the settings corresponding to the group will be displayed. It is possible to get settings by filters and/or tags for a single specific service. For this purpose, the Groups tab is used.
• About. The tab includes information about the Configurator service interface.

Settings#

Each of the Configurator settings contain the following fields:

• "Name" — Name for the setting.
• "Description" — Setting description.
• "ID and Times" — Unique setting ID.
• "Create time" — Setting create time.
• "Last update time" — Setting last update time.
• "Value" — Body of the setting.
• "Schema" — Verification template for the schema body.
• "Tag" — Tags for the setting used to filter settings for the services.

The "Tags" field is not available for the default settings. You should press the Duplicate button and create a new setting on the basis of the existing one.

The following options for the settings are available:

• Create a new setting — press the Create new button, enter required values and press Create. You should also select an already existing limitation for the setting. The Configurator will try to check the value of a setting if the Check on save flag is enabled and there is a limitation selected for the setting;

• Duplicate existing setting — press the Duplicate button on the right side of the setting, change required values and press Create. The Configurator will try to check the setting value if the Check on save flag is enabled on the lower left side of the screen and there is such a possibility;

• Delete existing setting — press the Delete button on the right side of the setting.

• Update existing setting — change name, description, tags, value and press Save button on the right side of the setting.

• Filter existing settings by name, description, tags, service names, groups — use the filters on the left side of the screen and press Enter or click on Search button;

Show limitations. The flags are used to enable displaying of limitations for each of the settings.

JSON editors. The flag enables you to switch the mode of the value field representation. If the flag is disabled, the name of the parameter and a field for its value are displayed. If the flag is enabled, the Value field is displayed as a JSON.

The Filters section on the left side of the window enables you to display all the required settings according to the specified values. You may enter the required name manually or select it from the list:

• Setting. The filter enables you to display the setting with the specified name.
• Description. The filter enables you to display all settings with the specified description or part of description.
• Tags. The filter enables you to display all settings with the specified tag.
• Service filter. The filter enables you to display all settings that belong to the selected service.
• Group. The filter enables you to display all settings that belong to the specified group. For example, you can select to display all the services belonging to LP.

Use tagged settings#

Using tagged settings, you can run several identical services that will use different settings from the Configurator.

To do this, follow these steps:

1. Duplicate or create a new setting by specifying a tag for it. For example, you can duplicate the "LUNA_EVENTS_DB" setting and assign it the "EVENTS_DB_TAG" tag.

2. Pass the following arguments to the command "run.py" the corresponding container:

3. --luna-config — flag containing the address of the Configurator service

4. --<configuration_name> — flag containing the configuration and tag

See the "Service arguments" section of the installation guide for more information on the arguments.

For example, to configure "LUNA_EVENTS_DB" with the tag "EVENTS_DB_TAG", the container launch command will look like this:

docker run \
...
dockerhub.visionlabs.ru/luna/luna-events:v.4.12.9
python3 /srv/luna_events/run.py --luna-config http://127.0.0.1:5070/1 --LUNA_EVENTS_DB EVENTS_DB_TAG

Limitations#

Limitations are used as service settings validation schema.

Settings and limitations have the same names. A new setting is created upon limitation creation.

The limitations are set by default for each of the LP services. You cannot change them.

Each of the limitations includes the following fields:

• Name is the name of the limitation.
• Description is the description of the limitation.
• Service list is the list of services that can use settings of this limitation.
• Schema is the object with JSON schema to validate settings
• Default value is the default value created with the limitation.

The following actions are available for managing limitations:

• Create a new limitation — Press the Create new button, enter required values and press "Create". Also, the setting with default value will be created.
• Duplicate existing limitation — Press the Duplicate button on the right side of the limitation, change required values and press Create. Also, the setting with default value will be created.
• Update limitation values — Change name/description/service list/validation schema/default values and press the Save button on the right side of the limitation.
• Filter existing limitations by names, descriptions, and groups.
• Delete existing limitation — Press the Delete button on the right side of the limitation.

Groups#

Group has a name and a description.

It is possible to:

• Create a new group — Press the Create new button, enter the group name and optionally description and press Create.
• Filter existing groups by group names and/or limitation names — Use filters on the left side and press 'RETURN' or click on Search button.
• Update group description — Update the existing description and press the Save button on the right side of the group.
• Update linked limitation list — to unlink limitation, press "-" button on the right side of the limitation name, to link limitation, enter its name in the field at the bottom of the limitation list and press the "+" button. To accept changes, press the Save button.
• Delete group — Press the Delete button on the right side of the group.

Settings dump#

The dump file includes all the settings of all the LP services.

Receive settings dump#

You can fetch the existing service settings from the Configurator by creating a dump file. This may be useful for saving the current service settings.

To receive a dump file use the following options:

• wget: wget -O settings_dump.json 127.0.0.1:5070/1/dump
• curl: curl 127.0.0.1:5070/1/dump > settings_dump.json
• text editor

The current values, specified in the Configurator service, are received.

Apply settings dump#

To apply the dumped settings use the db_create.py script with the --dump-file command line argument (followed with the created dump file name): base_scripts/db_create.py --dump-file settings_dump.json.

You can apply full settings dump on an empty database only.

If the settings update is required, you should delete the whole "limitations" group from the dump file before applying it.

    "limitations":[
      ...
    ],

Follow these steps to apply the dump file:

1. Enter the Configurator container.

2. Run python3 base_scripts/db_create.py --dump-file settings_dump.json.

Limitations from the existing limitations files are replaced with limitations from the dump file, if limitations names are the same.

Limitations file#

Receive limitation file#

Limitation file includes limitations of the specified service. It does not include existing settings and their values.

To download a limitations file for one or more services, use the following commands:

1. Enter the Configurator container.

2. Create the output "base_scripts/results" directory: mkdir base_scripts/results

3. Run the "base_scripts/get_limitation.py" script: python3 base_scripts/get_limitation.py --service luna-image-store luna-handlers --output base_scripts/results/my_limitations.json.

Note the "base_scripts/get_limitation.py" script parameters:

• --service for specifying one or more service names (required)
• --output for specifying the directory or a file where to save the output. The default value: "current_dir/{timestamp}_limitation.json" (optional)

Licenses service#

General information#

The Licenses service stores information about the available licensed features and their limits.

There are three ways to get license information:

• Using the "get license" request to the Licenses service.
• Using the "get system info" request to the Admin service.
• Using the user interface of the "Admin service".

You can also use the "get platform features" request to the API service, in the response to which you can get information about the license status, the license functions enabled ("face_quality", "body_attributes" and "liveness") and the status of optional services (Image Store, Events, Tasks and Sender) from the "ADDITIONAL_SERVICES_USAGE" configuration of the Configurator service.

If you disable some license feature and try to use a request that requires this function, error 33002 will be returned with the description "License problem Failed to get value of License feature {value}".

License information#

LP license includes the following features:

• License expiration date.
• Maximum number of faces with linked descriptors or basic attributes.
• OneShotLiveness estimation availability.
• OneShotLiveness current balance.
• Deepfake estimation availability.
• Image check according to ISO/IEC 19794-5:2011 standard availability.
• Body parameters estimation availability.
• People count estimation availability.
• Using Lambda service availability.
• Possibility of using the Index Matcher service in the LUNA Index Module.
• Maximum number of streams created by the LUNA Streams service.

When ordering the license, you need to inform technical support about the need to use any of the above features.

The features "Possibility of using the Index Matcher service in the LUNA Index Module" and "Maximum number of streams created by the LUNA Streams service" are described in the LUNA Index Module and FaceStream documentation, respectively.

Notifications are available for some features when approaching the limit. Notifications work in the form of sending messages to the logs of the corresponding service. For example, when approaching the allowable number of created faces with descriptors, the following message will be displayed in the Faces service logs: "License limit exceeded: 8% of the available license limit is used. Please contact VisionLabs for license upgrade or delete redundant faces". Notifications work due to constant monitoring implemented using the Influx database. Monitoring data is stored in the corresponding fields of the Influx database.

See the detailed information in the section "Monitoring".

Expiration date#

When the license expires, you cannot use LUNA PLATFORM.

By default, the notification about the end of the license is sent two weeks before the expiration date.

When the license ends, the following message is returned "License has expired. Please contact VisionLabs for a license extension.".

The Licenses service writes data about the license expiration date to the logs and the Influx database in the "license_period_rest" field.

Faces limit#

The Faces service checks the number of faces left according to the maximum available number of faces received from the Licenses service. The faces with linked descriptors or basic attributes are counted only.

The percentage of the used limit for faces is written in the Faces log and displayed in the Admin GUI.

The Faces service writes data about the created faces to the logs and the Influx database in the "license_faces_limit_rate" field.

The created faces are written in the Faces log and displayed in the Admin GUI as a percentage of the database fullness. You should calculate the number of faces with descriptors left using the current percentage.

You start receiving notifications when there are 15% of available faces left. When you exceed the number of available faces, the message "License limit exceeded. Please contact VisionLabs for license upgrade or delete redundant faces" appears in logs. You cannot attach attributes to faces if the number of faces exceeds 110%.

Consequences of missing a feature

If this feature is disabled, it will be impossible to perform the following requests:

• "create face"
• "generate events" specifying a handler with "policies" > "storage_policy" > "face_policy" > "store_face"

OneShotLiveness#

An unlimited license or a license with a limited number of transactions is available to estimate Liveness using the OneShotLiveness estimator.

Each use of Liveness in requests reduces the transaction count. It is impossible to use the Liveness score in requests after the transaction limit is exhausted. Requests that do not use Liveness and requests where the Liveness estimation is disabled are not affected by the exhaustion of the limit. They continue to work as usual.

The Licenses service stores information about the liveness transactions left. The number of transactions left is returned in the response from the "/license" resource.

The Remote SDK service writes data on the number of available Liveness transactions to the logs and the Influx database in the "liveness_balance" field.

A warning about the exhaustion of the number of available transactions is sent to the monitoring and logs of the Remote SDK service when the remaining 2000 transactions of Liveness are reached (this threshold is set in the system).

See the "OneShotLiveness description" section for more information on how Liveness works.

Consequences of missing a feature

If this feature is disabled, it will be impossible to estimate Liveness (the "estimate_liveness" parameter) in the following requests:

• "sdk"
• "detect faces"
• "generate events" specifying a handler with "policies" > "detect_policy" > "estimate_liveness"
• "perform verification" specifying a handler with "policies" > "detect_policy" > "estimate_liveness"
• "estimator task" specifying a handler with "policies" > "detect_policy" > "estimate_liveness"
• "predict liveness"

Body parameters estimation#

This feature enables you to estimate body parameters. Two values can be set in the license — 0 or 1. Monitoring is not intended for this parameter.

Consequences of missing a feature

If this feature is disabled, it will be impossible to estimate the body parameters (parameters "estimate_upper_body", "estimate_lower_body", "estimate_body_basic_attributes", "estimate_accessories") in the following requests:

• "sdk"
• "generate events" specifying a handler with "policies" > "detect_policy" > "body_attributes"

People count estimation#

This feature enables you to estimate the number of people. Two values can be set in the license — 0 or 1. Monitoring is not intended for this parameter.

Consequences of missing a feature

If this feature is disabled, it will be impossible to estimate the number of people (the "estimate_people_count" parameter) in the following requests:

• "sdk"
• "generate events" specifying a handler with "policies" > "detect_policy" > "estimate_people_count"

Image check by ISO/IEC 19794-5:2011 standard#

This feature enables you to perform various image checks by ISO/IEC 19794-5:2011 standard. Two values can be set in the license — 0 or 1. Monitoring is not intended for this parameter.

Consequences of missing a feature

If this feature is disabled, it will be impossible to perform the following requests:

• "iso"
• "detect faces" with the parameter "estimate_face_quality"
• "generate events" specifying a handler with "policies" > "detect_policy" > "face_quality"
• "estimator task" specifying a handler with "policies" > "detect_policy" > "face_quality"
• "perform verification" specifying a handler with "policies" > "detect_policy" > "face_quality"

Lambda service#

The Lambda service is intended to work with user modules that mimic the functionality of a separate service. The service enables you to write and use your own handler or write an external service that will closely interact with the LUNA PLATFORM and immediately have several functions typical of LP services (such as logging, automatic configuration reload, etc.).

The Lambda service creates a Docker image and then runs it in a Kubernetes cluster. It is impossible to manage a custom module without Kubernetes. Full-fledged work with the Lambda service is possible when deploying LUNA PLATFORM services in Kubernetes. To use it, you must independently deploy LUNA PLATFORM services in Kubernetes or consult VisionLabs specialists. If necessary, you can use Minikube for local development and testing, thus providing a Kubernetes-like environment without the need to manage a full production Kubernetes cluster.

This functionality should not be confused with the plugin mechanism. Plugins are designed to implement narrow targeted functionality, while the Lambda service enables you to implement the functionality of full-fledged services.

It is strongly recommended to learn as much as possible about the objects and mechanisms of the LUNA PLATFORM (especially about handlers) before starting to work with this service.

A custom module running in a Kubernetes cluster is called lambda. Information about the created lambda is stored in the Lambda database.

The number of lambda created is unlimited. Each lambda has the option to add its own OpenAPI specification.

To work with the Lambda service, you need a special license feature. If the feature is not available, the corresponding error will be returned when requesting the creation of lambda.

Note: The description given below is intended for general acquaintance with the functionality of the Lambda service. See developer manual for more details. The "Quick start guide" section is available in the developer manual, which enables you to start working with the service.

Before start#

Before you start working with the Lambda service, you need to familiarize yourself with all the requirements and set the service settings correctly.

Code and archive requirements#

The module is written in Python and must be transferred to the Lambda service in a ZIP archive.

The code and archive must meet certain requirements, the main of which are listed below:

• Python version 3.11 or higher must be used.
• Development requires the "luna-lambda-tools" library, available in VisionLabs PyPI.
• Archive should not be password-protected.

Also, the files in the archive must have a certain structure. See the detailed information in the section "Requirements" of the developer manual.

Environment requirements#

To work with the Lambda service, the following environment requirements are required:

• Availability of running Licenses and Configurator services*.
• Availability of S3 bucket for storing archives.
• Availability of Docker registry for storing images.
• Availability of Kubernetes cluster.

* during its operation, lambda will additionally interact with some LUNA PLATFORM services. The list of services depends on the lambda type (see "Lambda types").

Write/read access to the S3 storage bucket must be provided and certain access rights in the Kubernetes cluster must be configured. You need to transfer the basic Lambda images to your Docker registry. The commands for transferring images are given in the LUNA PLATFORM installation manual.

When the Lambda service starts, the Docker registry and base images are checked.

For more information, see the "Requirements" section of the developer manuals.

Lambda service configuration#

In the Lambda service settings, you must specify the following data:

• Location of the Kubernetes cluster (see setting "CLUSTER_LOCATION"):

• "internal" — Lambda service works in a Kubernetes cluster and does not require other additional settings.

• "remote" — Lambda service works with a remote Kubernetes cluster and correctly defined settings "CLUSTER_CREDENTIALS" (host, token and certificate).
• "local" — Lambda service works in the same place where the Kubernetes cluster is running.

In the classic version of working with the Lambda service, it is assumed to use the "internal" parameter.

• S3 bucket settings (see setting "S3").

• Registry address for storing the Docker image (see setting "LAMBDA_REGISTRY").

• Addresses of insecure registries that may need access during lambda creation (see setting "LAMBDA_INSECURE_REGISTRIES").

For more information, see the "Configuration requirements" section of the developer manual.

Configuring lambda entities#

A specific set of settings is available for running lambda entities. They can be set in the Configurator by filtering the settings by the name "luna-lambda-unit".

The settings for the Lambda service and the settings for lambda entities are different.

The settings contain service addresses, connection timeouts, logging settings, etc., allowing lambda entities to effectively interact with LUNA PLATFORM services. See all available settings in the section "Lambda configuration".

The settings apply to all lambda at the same time.

Lambda types#

Lambda can be of three types:

• Handlers-lambda, intended to replace the functionality of the classic handler.
• Standalone-lambda, intended to implement independent functionality to perform close integration with the LUNA PLATFORM.
• Tasks-lambda, intended to implement additional custom long task types.

Each type has certain requirements for LUNA PLATFORM services, the actual settings of which will be automatically used to process requests. Before starting work, the user must decide which lambda he needs.

Handlers-lambda#

Examples of possible functionality:

• Performing verification with the possibility of saving an event.
• Matching of two images without performing the rest of the functionality of the classic handler.
• Adding your own filtering logic to the matching functionality;
• Circumventing certain limitations of the LUNA PLATFORM (for example, specify the maximum number of candidates greater than 100).
• Embedding the neural network SDK bypassing the LUNA PLATFORM.

During its operation, Handlers-lambda will interact with the following LUNA PLATFORM services:

• Configurator — To get the settings.
• Faces — For working with faces and lists.
• Remote SDK — For performing detections, estimations and extractions.
• Events* — For working with events.
• Python Matcher/Python Matcher Proxy** — For classical/cross-matching of faces or bodies.
• Image Store* — For storing samples and source images of faces or bodies.

The Lambda service will not check the connection to the disabled services and will give an error if the user tries to make a request to the disabled service.

To run the Lambda service, only the presence of the Configurator and Licenses services is required.

* the service can be disabled in the "ADDITIONAL_SERVICES_USAGE" setting.

** the service is disabled by default. To enable the service, see the setting "ADDITIONAL_SERVICES_USAGE".

The Lambda-handler can be used in two cases:

• As a custom handler that has its own response scheme, which may differ from the response of classic handlers and cannot be properly used in other LUNA PLATFORM services.

• As a custom handler that mimics the response of a classic handler. There are some requirements for such a case:

  • The response must match the response scheme of the event generation request.
  • The handler must process incoming data correctly so that other services can use it, otherwise there is no guarantee of compatibility with other services. That is, if such a handler implies face recognition, the module should return information about face recognition in response, if the handler implies body detection, the module should return body detection in response, etc.

  For example, if a Lambda handler satisfies the above conditions, then it can be used in Estimator task as a classic handler.

For more information and code examples for Handlers-lambda, see the "Handlers lambda development" section of the developer manual.

Standalone-lambda#

Examples of possible functionality:

• Filtering incoming images by format for subsequent sending to the Remote SDK service.
• Creation of a service for sending notifications by analogy with the Sender service.
• Creation of a service for recording a video stream and saving it as a video file to the Image Store service for subsequent processing by the FaceStream application.

During its operation, Standalone-lambda will interact at least with the Configurator service, which enables lambda to receive its settings (for example, logging settings).

To run the Lambda service, only the presence of the Configurator and Licenses services is required.

For more information and sample code for Standalone-lambda, see the "Standalone lambda development" section of the developer manual.

Tasks-lambda#

Examples of possible functionality:

• Unlink faces from lists if faces do not similar to the specified face.
• Remove duplicates from list.
• Recursively find events similar to the specified photo.

During its operation, Tasks-lambda will interact with the following LUNA PLATFORM services:

• Configurator
• Faces
• Python Matcher
• Remote SDK
• Tasks
• Events*
• Image Store*
• Handlers*

* the service can be disabled in the "ADDITIONAL_SERVICES_USAGE" setting

After creating lambda, the "lambda task" requests must be executed to create a Lambda task. The results of the task/subtask can be obtained using standard Tasks service requests.

The Lambda task is created according to the general task creation process, except that the Lambda service is used instead of the Tasks worker.

When writing code for Tasks-lambda, it is recommended to break each task into subtasks for the convenience of representing the process and parallelizing the execution of tasks.

You can create schedule for Tasks-lambda.

For more information and code examples for Tasks-lambda, see "Tasks lambda development" in the developer manual.

Create lambda#

To create a lambda, you need to do the following:

1. Write Python code in accordance with the type of future lambda and code requirements.
2. Move files to the archive in accordance with archive requirements.
3. Perform the "create lambda" request, specifying the following mandatory data:
4. "archive" — The address to the archive with the user module.
5. "credentials" > "lambda_name" — The name for the lambda being created.
6. "parameters" > "lambda_type" — The type of lambda being created ("handlers", "standalone" or "tasks").

Also optionally in the "deploy_parameters" section you can set the number of pods, allocate resources to pods, set namespaces, as well as enable GPU use.

It is also possible to specify labels for running lambda using specific Kubernetes nodes using the "deploy_parameters" > "selector" parameter. This enables for more granular control over lambda deployment, allowing administrators to specify the nodes on which lambda should be deployed based on their resource needs. By applying labels to Kubernetes nodes and using the "selector" parameter, administrators can manage resource allocation not only for individual lambdas, but also for lambda sections with similar resource requirements.

If necessary, you can specify a list of additional Docker commands to create a lambda container. See the "Lambda — Archive requirements" section of the developer manual.

In response to a successful request, the "lambda_id" will be issued.

The creation of lambda consists of several stages, namely:

• Creating a Docker image:
  • Getting the provided ZIP archive.
  • Addition of the archive with the necessary files.
  • Saving the archive in S3 storage.
  • Publishing the image to the registry.
• Creating a service in the Kubernetes cluster.

During lambda creation, you can run the following requests:

• "get lambda image creation status" to get the Docker image creation status ("in_progress", "error", "completed", "not_found")
• "get lambda image creation logs" to get Docker image creation logs
• "get lambda status" to get the lambda creation status ("running", "waiting", "terminated", "not_found")

See the lambda creation sequence diagram in "Lambda creation diagram".

See the detailed description of the lambda creation process in the "Creation pipeline" section of the developer manual.

Create handler for Handlers-lambda#

If lambda is supposed to be used as a custom handler simulating the response of a classic handler, then it is necessary to create a handler, specifying "handler_type" = "2" and the resulting "lambda_id".

During the creation of the handler, the Handlers service will perform a health check to the Kubernetes cluster.

The resulting "handler_id" can be used in requests "generate events" or "estimator task".

Use lambda#

The table below shows resources for working with lambda, depending on its type.

See the lambda processing sequence diagram in "Lambda processing diagram".

Each lambda has its own API, the description of which is also available using the "get lambda open api documentation" request.

Each lambda response will contain multiple headers, including:

• "Luna-Request-Id" — Classic external ID of the LUNA PLATFORM request.
• "Lambda-Version" — Contains the current lambda version.

Useful requests when working with lambda:

• "get lambda status" to get lambda creation status ("running", "waiting", "terminated", "not_found")
• "get lambda" to get complete information about the created lambda (creation time, name, status, etc.)
• "get lambda logs" to get lambda creation logs

Create GPU-enabled lambda#

It is possible to create a lambda that can leverage graphics processing unit (GPU) resources. To enable GPU usage, the "deploy_parameters" > "enable_gpu" parameter must be included in the lambda creation request.

Lambda supports only NVIDIA graphics processors. For additional information on GPU usage, refer to the Kubernetes documentation.

If there is only one GPU available but more are required, you can enable shared GPU access (refer to the official documentation).

The Lambda service does not manage cluster resources, including GPU allocation. Resource management is handled by the Kubernetes administrator.

Additionally, when deploying LUNA PLATFORM to Kubernetes, it is recommended to configure the manifest in a specific way for all containers that use the GPU (including the Lambda service container) to avoid problems with device visibility. For detailed information, refer to the "Configuring manifest GPU support" section in the installation manual.

Update lambda#

The base image for lambda containers is updated periodically. This image contains the necessary modules to interact with LUNA PLATFORM services. Applying updates requires that the lambda be recreated so that the container can be rebuilt based on the new image. After updating the base image and the "luna-lambda-tools" library, lambda functionality may be broken.

You can update the lambda using the request "update lambda" using the latest base image. It is recommended to have a backup copy of the archive on S3.

See the "Lambda updates" section of the developer manual for detailed information about the update mechanism, backup strategies, and restoring from a backup.

Backport 3#

The Backport 3 service is used to process the requests for LUNA PLATFORM 3 using LUNA PLATFORM 5.

Although most of the requests are performed in the same way as in LUNA PLATFORM 3, there are still some restrictions. See "Backport 3 features and restrictions" for details.

See Backport 3 OpenAPI specification for details about the Backport 3 API.

Backport 3 new resources#

Liveness estimation#

Backport 3 provides Liveness estimation in addition to the LUNA PLATFORM 3 features. See the "liveness > predict liveness" section in the Backport 3 OpenAPI specification.

Handlers#

The Backport 3 service provides several handlers: "extractor", "identify", "verify". The handlers enable to perform several actions in a single request:

• "handlers" > "face extractor" — Enables you to extract a descriptor from an image, create a person with this descriptor, attach the person to the predefined list.

• "handlers" > "identify face" — Enables you to extract a descriptor from an image and match the descriptor with the predefined list of candidates.

• "handlers" > "verify face" — Enables you to extract a descriptor from an image and match the descriptor with the person's descriptor.

The description of the handlers and all the parameters of the handlers can be found in the following sections:

• "handlers" > "patch extractor handler"
• "handlers" > "patch verify handler"
• "handlers" > "patch identify handler"

The requests are based on handlers and unlike the standard "descriptors" > "extract descriptors", "matching" > "identification", and "matching" > verification" requests the listed above request are more flexible.

You can patch the already existing handlers thus applying additional estimation to the requests. E. g. you can specify head angles thresholds or enable/disable basic attributes estimation.

The Handlers are created for every new account at the moment the account is created. The created handlers include default parameters.

Each of the handlers has the corresponding handler in the Handlers service. The parameters of the handlers are stored in the luna_backport3 database.

Each handler supports GET and PATCH requests thus it is possible to get and update parameters of each handler.

Each handler has its version. The version is incremented with every PATCH request. If the current handler is removed, the version will be reset to 1:

• For the requests with POST and GET methods:

  If the Handlers and/or Backport 3 service has no handler for the specified action, it will be created with default parameters.

• For requests with PATCH methods:

  If Handlers and/or Backport 3 service has no handler for the specified action, a new handler with a mix of default policies and policies from the request will be created.

Backport 3 architecture#

Backport 3 interacts with the API service and sends requests to LUNA PLATFORM 5 using it. In turn, the API service interacts with the Accounts service to check the authentication data.

Backport 3 has its own database (see "Backport 3 database"). Some of its tables are similar to the tables of the Faces database of LP 3. It enables you to create and use the same entities (persons, account tokens and accounts) as in LP 3.

The backport service uses Image Store to store portraits.

You can configure Backport 3 using the Configurator service.

Backport 3 features and restrictions#

The following features have core differences:

For the following resources on method POST default descriptor version to extract from image is 56:

• /storage/descriptors
• /handlers/extractor
• /handlers/verify
• /handlers/identify
• /matching/search

You can still upload the existing descriptors of versions 52, 54, 56. The older descriptor versions are no longer supported. - For resource /storage/descriptors on method POST, estimation of "saturation" property is no longer supported, and the value is always set to 1. - For resource /storage/descriptors on method POST, estimation of "eyeglasses" attribute is no longer supported. The attributes structure in the response will lack the "eyeglasses" member. - For resource /storage/descriptors on method POST, head position angle thresholds can still be sent as float values in range [0, 180], but they will be internally rounded to integer values. As before, thresholds outside the range [0, 180] are not taken into account.

Garbage collection (GC) module#

According to LUNA PLATFORM 3 logic, garbage is the descriptors that are linked neither to a person nor to a list.

For normal system operation, one needs to regularly delete garbage from databases. For this, run the system cleaning script remove_not_linked_descriptors.py from ./base_scripts/gc/ folder.

According to Backport 3 architecture, this script removes faces, which do not have links with any lists or persons from the Backport 3 database, from the Faces service.

Script execution pipeline#

The script execution pipeline consists of several stages:

1. A temporary table is created in the Faces database. See more info about temporary tables for oracle or postgres.
2. IDs of faces that are not linked to lists are obtained. The IDs are stored in the temporary table.

3. While the temporary table is not empty, the following operations are performed:

4. The batch of IDs from the temporary table is obtained. First 10k (or less) face ids are received.

5. Filtered IDs are obtained. Filtered IDs are ids that do not exist in the "person_face" table of the Backport 3 database.
6. Filtered IDs are removed from the Faces database. If some of the faces cannot be removed, the script stops.
7. Filtered IDs are removed from the Backport 3 database (foolcheck). A warning will be printed.
8. IDs are removed from the temporary table.

Script launching#

docker run --rm -t --network=host --entrypoint bash dockerhub.visionlabs.ru/luna/luna-backport-3:v.0.11.9 -c "python3 ./base_scripts/gc/remove_not_linked_descriptors.py"

The output will include information about the number of removed faces and the number of persons with faces.

Backport 4#

The Backport 4 service is used to process the requests for LUNA PLATFORM 4 using LUNA PLATFORM 5.

Although most of the requests are performed in the same way as in LUNA PLATFORM 4, there are still some restrictions. See "Backport 4 features and restrictions" for details.

See Backport 4 OpenAPI specification for details about the Backport 4 API.

Backport 4 architecture#

Backport 4 interacts with the API service and sends requests to LUNA PLATFORM 5 using it.

Backport 4 directly interacts with the Faces service to receive the number of existing attributes.

Backport 4 directly interacts with the Sender service. All the requests to Sender are sent using the Backport 4 service. See the "ws" > "ws handshake" request in the Backport 4 OpenAPI specification.

You can configure Backport 4 using the Configurator service.

Backport 4 features and restrictions#

The following features have core differences:

The current versions for LUNA PLATFORM services are returned on the request to the /version resource. For example, the versions of the following services are returned:

• "luna-faces"
• "luna-events"
• "luna-image-store"
• "luna-python-matcher" or "luna-matcher-proxy"
• "luna-tasks"
• "luna-handlers"
• "luna-api"
• "LUNA PLATFORM"
• "luna-backport4" — Current service

Resources changelog:

• Resource /attributes/count is available without any query parameters and does not support accounting. The resource works with temporary attributes.

• Resource /attributes on method GET: "attribute_ids" query parameter is allowed instead of "page", "page_size", "time__lt" and "time__gte" query parameters. Thus you can get attributes by their IDs not by filters. The resource works with temporary attributes.

• Resource /attributes/<attribute_id> on methods GET, HEAD, DELETE and resource /attributes/<attribute_id>/samples on method GET interact with temporary attributes and return attribute data if the attribute TTL has not expired. Otherwise, the "Not found" error is returned.

• If you already used the attribute to create a face, use the "face_id" to receive the attribute data. In this case, the "attribute_id" from the request is equal to "face_id".

• Resource /faces enables you to create more than one face with the same "attribute_id".

• Resource /faces/<face_id> on method DELETE enables you to remove face without removing its attribute.

• Resource /faces/<face_id> on method PATCH enables you to patch attribute of the face making the first request to patch "event_id", "external_id", "user_data", "avatar" (if required) and the second request to patch attribute (if required).

• If face attribute_id is to be changed, the service will try to patch it with temporary attribute data if the temporary attribute exists. Otherwise, the service tries to patch it with attribute data from the face with "face_id" = "attribute_id".

• The match policy of resource /handlers now has the default match limit that is configured using the "MATCH_LIMIT" setting from the Backport 4 "config.py" file.

• Resource /events/stats on method POST: "attribute_id" usage in "filters" object was prohibited as this field is no longer stored in the database. The response with the 403 status code will be returned.

• The "attribute_id" in events is not null and is equal to "face_id" for back compatibility. GC task is unavailable because all the attributes are temporary and will be removed automatically. Status code 400 is returned on a request to the /tasks/gc resource.

• The column "attribute_id" is not added to the report of the Reporter task and this column is ignored if specified in the request. Columns "top_similar_face_id", "top_similar_face_list", "top_similar_face_similarity" are replaced by the "top_match" column in the report if any of these columns is passed in the reporter task request.

• Linker task always creates new faces from events and ignores faces created during the event processing request.

• Resource /matcher does not check the presence of provided faces thus error "FacesNotFound" is never returned. If the user has specified a non-existent candidate of type "faces", no error will be reported, and no actual matching against that face will be made.

• Resource /matcher checks whether reference with type attribute has the ID of face attribute or the ID of temporary attribute and performs type substitution. Hence it provides sending references for matching in the way it was done in the previous version.

• Resource /matcher takes matching limits into account. By default, the maximum number of references or candidates is limited to 30. If you need to overcome these limits, configure "reference_limit" and "candidate_limit".

• Resource /ws has been added. There was no /ws resource in the LUNA PLATFORM 4 API as it was a separate resource of the Sender service. This added resource is similar to the Sender service resource, except that "attribute_id" of candidates faces is equal to "face_id".

• Resource /handlers returns the error "Invalid handler with id {handler_id}", if the handler was created in the LUNA PLATFORM 5 API and is not supported in LUNA Backport 4.

Backport 4 User Interface#

The User Interface service is used for the visual representation of LP features. It does not include all the functionality available in LP. User Interface enables you to:

• Download photos and create faces using them.
• Create lists.
• Match existing faces.
• Show existing events.
• Show existing handlers.

All the information in User Interface is displayed according to the account data, specified in the configuration file of the User Interface service ()./luna-ui/browser/env.js.

User Interface works with only one account at a time, which must be of "user" type.

You should open your browser and enter the User Interface address. The default value is: <server_url>:4200.

You can select a page on the left side of the window.

Lists/faces page#

The starting page of User Interface is Lists/Faces. It includes all the faces and lists created using the account specified in the configuration.

The left column of the workspace displays existing lists. You can create a new list by pressing the Add list button. In the appeared window you can specify the user data for the list.

The right column shows all the created faces with pagination.

Use the Add new faces button to create new faces.

On the first step, you should select photos to create faces from. You can select one or several images with one or several faces in them.

After you select images, all the found faces will be shown in a new dialog window.

All the correctly preprocessed images will be marked as "Done". If the image does not correspond to any of the requirements, an error will be displayed for it.

Press the Next step button.

On the next step, you should select the attributes to extract for the faces.

Press the Next step button.

On the next step, you can specify user data and external ID for each of the faces. You can also select lists to which each of the faces will be added. Press Add Faces to create faces.

You can change pages using arrow buttons.

You can change the display of faces and filter them using buttons in the top right corner.

Filer faces. You can filter faces by ID, external ID or list ID.

/

Change view of the existing faces.

Handlers page#

Handlers page displays all handlers created using the account specified in the configuration.

All the information about specified handler policies is displayed when you select a handler.

You can edit or delete a handler using edit

and delete

icons.

Events page#

The events page displays all the events created using the account specified in the configuration.

It also includes filters for displaying of events

.

Common information#

You can edit

or delete

an item (face, list or handler) using special icons. The icons appear when you hover the cursor on an item.

Matching dialog#

The Matching button in the bottom left corner of the window enables you to perform matching.

After pressing the button you can select the number of the received results for each of the references.

On the first step, you should select references for matching. You can select faces and/or events as references.

On the second step, you should select candidates for matching. You can select faces or lists as candidates.

On the last step, you should press the Start matching button to receive results.

Resource consumption by services#

Below is a table describing the most significant resource consumption by services. The Remote SDK and Python Matcher services perform the most resource-intensive operations. 

Additional Information#

OneShotLiveness description#

The Liveness technology enables to detect presentation attacks. A presentation attack is a situation when an imposter tries to use a video or photos of another person to circumvent the recognition system and gain access to the person's private data.

To estimate Liveness in the LUNA PLATFORM, the estimator LUNA SDK OneShotLiveness is used.

There are the following general types of presentation attacks:

• Printed Photo Attack. One or several photos of another person are used.
• Video Replay Attack. A video of another person is used.
• Printed Mask Attack. An imposter cuts out a face from a photo and covers his face with it.
• 3D Mask Attack. An imposer puts on a 3D mask depicting the face of another person.

Liveness has the ability to license the number of performed transactions. You can also choose between an unlimited license and a license with a limited number of transactions (a maximum of 16 777 215 transactions can be specified in the key). After the transaction limit is exhausted, it will be impossible to use the Liveness estimation in requests. Requests that do not use Liveness or with Liveness estimation disabled are not affected by the exhaustion of the limit, they continue to work as usual.

Liveness check results#

The Liveness algorithm uses a single image for processing and returns the following data:

• Liveness probability [0..1]. Here 1 means real person, 0 means spoof. The parameter shows the probability of a live person being present in the image, i.e. it is not a presentation attack. In general, the estimated probability must exceed the Liveness threshold.

• Image quality [0..1]. Here 1 means good quality, 0 means bad quality. The parameter describes the integral value of image, facial, and environmental characteristics. In general, the estimated quality must exceed the Quality threshold.

• Prediction. Based on the above data, LUNA PLATFORM issues the following prediction:

• 0 (spoof). Check revealed that the person is not real.

• 1 (real). Check revealed that the person is real.
• 2 (unknown). Result of the check is unknown. Such a prediction may return if the quality of the image being checked is below the Quality threshold, which determines the quality of the processed image.

Requests for estimating Liveness#

Liveness is used in the following resources:

• "/Liveness"
• "/sdk"
• "/handlers"
• "/verifiers"

You can filter events by Liveness in the "handlers/{handler_id}\/events" and "/verifiers/{verifier_id}\/verifications" resources, i.e. you can exclude "spoof", "real" or "unknown" results from image processing.

Filtering by Liveness is available for the following scenarios:

• When performing matching operations.
• When performing tasks.
• When sending data via web sockets.

You can also specify the Liveness estimation parameter when manually creating and saving events in the "handlers/{handler_id}\/events/raw" resource.

For multiple uploaded images, you can aggregate the Liveness results to obtain more accurate data.

Liveness requirements#

The requirements for the processed image and the face in the image are listed below.

See image quality thresholds in the "Image quality" section.

Liveness thresholds#

By default, two thresholds are used for checking Liveness — Quality threshold and Liveness threshold.

Quality threshold#

Quality threshold — the threshold of the processed image quality, lower which no check will be performed and the result "unknown" will be returned. This threshold is set in the "quality_threshold" setting from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service. The default threshold value is 0.5 (50%).

If the quality of the image being checked is lower than the Quality threshold value, then it is recommended to re-take a photo and send it to the system to perform the Liveness check. If this is not possible and you want to perform the Liveness check on the previously sent image, then you can set the Quality threshold to "0", or interpret the "unknown" value as "spoof".

When re-taking a photo, the average accuracy of the Liveness check increases. The table below shows the dependence of the FRN increase on the Quality threshold, where:

• FNR (False Negative Rate) — The percentage of real images rated as fake.
• Drop rate — The approximate percentage of images clipped by the Quality threshold (estimated based on a sample of 170,000 images).

Liveness threshold#

Liveness threshold — the threshold lower which the system will consider the result as a presentation attack ("spoof"). This threshold is set in the "real_threshold" setting from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service. The default threshold value is 0.5 (50%).

For images received from mobile devices, it is recommended to set a threshold 0.5 . For images obtained from a webcam, it is recommended to set a threshold 0.364 . At these threshold values, approximately equal results are obtained in terms of the accuracy of the algorithm.

If the quality of the image being checked is lower than the Quality threshold value, then setting the Liveness threshold will not make sense, since the result of the Liveness check will be unknown ("unknown").

Changing thresholds on "/handlers" and "/verifiers" resources#

In the "/handlers" and "/verifiers" there are two additional parameters:

• "quality_threshold" — Sets the Quality threshold.
• "liveness_threshold" — Sets the Liveness threshold.

Setting these parameters redefines the values of the "quality_threshold" and "real_threshold" settings from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service.

Changing thresholds on «/Liveness» and «/sdk» resources#

There are no additional parameters for overriding thresholds for the "/Liveness" and "/sdk" resources. Thresholds are set in the "quality_threshold" and "real_threshold" settings from the "LUNA_REMOTE_SDK_LIVENESS_ESTIMATOR_SETTINGS" section of the Configurator service.

Video analytics#

LUNA PLATFORM provides the ability to perform video analytics using the resource "videosdk".

Important: Currently, the resource is in beta testing status. Input and output schemes may be changed in future releases without backward compatibility support.

Video analytics is a set of functions that process frame by frame and assess valuable data.

To perform video analytics, an external link to a video file not exceeding 1 GB in size must be specified. If necessary, you can rotate the video before processing it at an angle of 90, 180 or 270 degrees (parameter "video" > "orientation" > "angle").

During video analytics, a certain number of events are generated according to some rules, where each event has a start and end. Events are recorded in the response body and contain specific information about the particular video analytics. The response body also contains basic metadata about the video (number of frames, frame rate, video duration). The obtained information is only available in the response body and is not saved anywhere.

Important: An event in video analytics is in no way related to events generated by handlers.

In the "LUNA_REMOTE_SDK_VIDEO_SETTINGS" section of the Remote SDK service, you can configure video processing settings, such as the number of decoder working processes, processor typem temporary directory for storing videos, etc.

People counting video analytics#

The event of people counting video analytics starts when, on the last frame of the specified consecutive frames (parameter "probe_count"), the specified number of people (parameter "people_count_threshold") appears. For example, if the minimum number of people is 10, and the number of consecutive frames is 3, the event will start if there are 10 people in 3 frames, starting from the 3rd frame. If there are 10 people in 2 frames, and 9 people in the third frame, the event will not start.

By default, every 10 frames are processed. If necessary, you can adjust the frame processing frequency using the "rate" parameter (for example, process every 3 seconds or every 3 frames).

In the request body, you can specify the "targets" parameter, which takes the following values:

• "coordinates" — Calculate coordinates of people.
• "overview" — Get an image with the maximum number of people.

You can set both values or none of them. If no values are set, a basic analysis will be performed, containing only information about the video recording, the number of people and the frames associated with them.

For such video analytics it is also possible to configure ROI area of interest on a frame (x, y coordinates, width, height, units - percent or coordinates). The principle of ROI is similar to FaceStream.

For this type of video analytics, you can also configure a rectangular region of interest (parameter "roi") on the frame (coordinates "x", "y", width, height, units - per cent or coordinates). The principle of ROI region of interest is similar to FaceStream.

Each event in people counting video analytics contains the following information:

• "event_id" — Event ID.
• "max_people_count" — Maximum number of people detected in the processed video segment.
• "video_segment" — Information about the video segment (start time of the event, end time of the event, first frame number, last frame number).
• "frames_estimations" — Array with information about estimates on each frame, containing the following information:
  • Frame number.
  • Time corresponding to the frame.
  • Number of people.

Human tracking video analytics#

For people tracking video analytics, the key concept is a track. A track is an object containing information about the position of one person in a sequence of frames. A track is created when the detection of a face, body or body with a face (depends on the detector type in the "detector_type" parameter) appears on the last frame of the specified number of consecutive frames ("probe_count" parameter). There can be as many tracks as the number of people detected on the frame. Along with a track, a video analytics event is started.

Note: Currently there can be only one event for one track. In future releases, it will be possible to have multiple events for a track, which can be sent without waiting for the track to end.

For example, if "probe_count" = "3" and there are detections on two consecutive frames and no detections on the third frame, the track and event will not start. If at any moment of tracking a person there were detections on two consecutive frames and no detections on the third frame, the track and event will end.

During tracking of a person it is possible to estimate attributes of faces, bodies, images (list of estimations is set in the "targets" parameter), and also to get the image with the best detection. The estimation can be performed by one best image from the track or by several (parameter "face_samples"/"body_samples" > "count"). If necessary, you can filter the best shots by "score", "head_pitch", "head_roll", "head_yaw" parameters. All scores set in the "targets" field depend on the type of detector used.

If necessary, you can enable Re-identification of the body (ReID). The ReID feature is used to improve tracking accuracy by combining different tracks of the same person.

For this type of video analytics, you can also configure a rectangular region of interest (parameter "roi") on the frame (coordinates "x", "y", width, height, units - per cent or coordinates). The principle of ROI region of interest is similar to FaceStream.

Each people count video analytics event contains the following information:

• "event_id" — Event ID.
• "track_id" — Track ID.
• "video_segment" — Information about the video segment (start time of the event, end time of the event, first frame number, last frame number).
• "frames_estimations" — Array with information about estimates on each frame, containing the following information:
  • Frame number.
  • Time corresponding to the frame.
  • Bbox coordinates and score.
• "aggregated_estimations" — Result of the score specified in the "targets" parameter in the request body.
• "track_count" - Number of tracks.

roi#

ROI (Region of Interest) specifies the region of interest in which the estimation is performed.

The specified rectangular area is cut out from the frame and LUNA PLATFORM performs further processing using this image.

Correct exploitation of the "roi" parameter significantly improves the performance of video analysis.

ROI on the source frame is specified by the "x", "y", "width", "height" and "mode" parameters, where:

• "x", "y" – Coordinates of the upper left point of the ROI area of interest.
• "width" and "height" — Width and height of the processed area of the frame.

• "mode" – Mode for specifying "x", "y", "width" and "height". Two modes are available:

  • "abs" — Parameters "x", "y", "width" and "height" are set in pixels.
  • "percent" — Parameters "x", "y", "width" and "height" are set as percentages of the current frame size.

  If the "mode" field is not specified in the request body, then the value "abs" will be used.

With width and height values of "0", the entire frame is considered the region of interest.

The coordinate system on the image is set similarly to the figure below.

When the values of width and height are set to "0", the entire frame will be the region of interest.

Below is an example of calculating ROI as a percentage:

droi#

Note: DROI is only available for people counting video analytics.

DROI specifies a region of interest relative to the source frame. DROI, unlike ROI, is not an estimation parameter, but works as a filter after estimation has been performed in accordance with the "targets" field. In other words, ROI is for estimation and optimization, DROI is for implementing business logic. You can use DROI together with ROI or separately. For example, if after execution the estimate taking into account the ROI of the number of people in the frame was equal to M, then after performing an additional filter on the DROI area, the number of people in the frame can be reduced to N.

The DROI on the source frame is specified by the following parameters:

• "area" — Geometry of the region of interest. The parameter is represented as an array of polygons. Each polygon is represented by an array of objects, where each object contains the x and y coordinates of the polygon's vertex. For example, you can define a triangular area of interest.
• "mode" – mode of specifying “x”, “y”. Two modes are available:
  • "abs" — Parameters "x", "y", "width" and "height" are set in pixels.
  • "percent" — Parameters "x", "y", "width" and "height" are set as percentages of the current frame size.
• "form" — Format of the region of interest. In the current implementation there is only one possible value, "common". Future implementations will support other formats in addition to polygons.

Filters#

The LUNA PLATFORM has the ability to filter objects according to certain rules. For example, you can match faces only from a certain list or get events for a certain period of time.

As a rule, filters are located in the "filters" field of various requests.

Filters by comparison operators#

For some filters, it is possible to specify comparison operators using the suffix "__operator". The operator is used both for time filters (from a certain date, for some time) and for ordinary filters (for some age of a person, for a list of IDs in lexicographic order).

__lt

The "less than" filter is used to search for values less than the specified one. For example, when using the filter "create_time__lt": "2022-08-11T09:11:41.674Z" in the "get events" request events with the creation time preceding the specified date and time will be returned, i.e. events created before August 11, 2022, 09:11:41.674 UTC.

__lte

The "less than or equal to" filter is used to search for values less than or equal to the specified one. For example, when using the filter "face_id__lte": "2046d39b-410d-481e-b9de-ede6a0c7367f" in the filters of the request "matching faces" for candidates, only those candidates whose IDs begin will be returned with "2046d39b-410d-481e-b9de-ede6a0c7367f" or less than this value in lexicographic order.

__gte

The "greater than or equal to" filter is used to search for values greater than or equal to the specified one. For example, when using the filter "similarity__gte": "0.9" in the "storage_policy" > "face_sample_policy" > "filters" > "match" of the "create handler" request, you can configure a rule for saving a face sample in the Faces database only if the "matching_policy" has determined that the similarity of the candidate with the reference is greater than or equal to "0.9".

Now-time filters#

For the time filters "create_time", "insert_time" and "end_time", the format for setting the time relative to the current time ("now-time" format) is available. For example, when creating a schedule for a task Garbage collection, you can specify the filter "create_time__lt": "now-30d", which will delete all objects except those created in the last 30 days.

It must be remembered that:

• When creating a schedule, the current time will be counted not from the creation of the schedule, but from the creation of the task by the schedule in accordance with the cron expression.
• When an event is generated by a handler that specifies the "now-time" filter (for example, in the "matching_policy" > "candidates" policy), the current time will be counted from the moment the event is generated.

To use the time relative to event generation, you can set policies directly in the request for event generation using the "multipart/form-data" scheme or using the Estimator task with the creation of a new handler.

In this format, the time is set according to the following pattern now-(\d+)[smhdwMy], where "[d+]" is a number, "[smhdwMy]" is the required period: m (minutes), h (hours), d (days), w (weeks), M (months), y (years).

User interfaces#

LUNA CLEMENTINE 2.0 Web Service#

The LUNA CLEMENTINE 2.0 web service can be used to perform basic LP requests to create objects and perform matching operations.

This service is not supplied with LUNA PLATFORM 5, it must be downloaded and configured separately.

Backport 3 and Backport 4 service interfaces#

Interfaces for interacting with LP 4 and LP 3 APIs are available in LUNA PLATFORM 5. The interfaces are intended only for those users who chose not to upgrade to the new version of the LUNA PLATFORM 5 API. They don't include most of the features available in LP 5 and will never include.

Other interfaces#

These interfaces are needed to simplify working with LUNA PLATFORM services.

Disableable services#

Some unused general services can be disabled. When a service is disabled, its main functions will not work. For example, if the Tasks service is disabled, attempting to execute a task will result in an error. Similarly, if the Sender service is disabled and an event handler with the "notification_policy" is used, it will also result in an error.

As a result, all resources that require a disabled service will return errors like <Service_name> service is disabled with a 403 response code.

If any service fails during the operation of the LUNA PLATFORM or it is manually stopped, the operation of dependent services will fail. For example, when the Licenses service is stopped, the following error will be displayed in the Faces service logs:

[0000001 2023-08-11 13:00:51.042000] ERROR: luna-faces 1690887528,00000000-0000-4000-a000-000000000001: Check connection to Luna Licenses : GET:http://127.0.0.1:5120/version:Cannot connect to host 127.0.0.1:5120 ssl:default [Connection refused]

The corresponding error will be returned when trying to make a request that depends on the problematic service.

Then the work of services dependent on the Faces service will also be terminated with a similar error.

You can check the dependence of services on each other using the interaction diagram or you can find a setting like "service_name_ADDRESS" in the settings of the service for which you want to define dependent services. For example, in the settings of the Faces service there is a setting "LUNA_LICENSES_ADDRESS" responsible for connecting to the Licenses service, and in the settings of the Admin service there is a setting "LUNA_FACES_ADDRESS" responsible for connecting to the Faces service.

Disabling services process#

• Open the Configurator user interface http://<configurator_server_ip>:5070.
• Enter the name of the setting "ADDITIONAL_SERVICE_USAGE" in the "Setting name" field and click "Apply Filters".
• Set the value for the required service to "false".
• Save the changes by clicking the "Save" button.

Image Store service disabling consequences#

When the Image Store service is disabled, there are some specific features to note:

• Objects of the type images, objects, samples, and sample save policies in handlers/verifiers will be unavailable.

• All tasks, except Garbage Collection, Linker, and Estimator, will become unavailable. However, there are some limitations for these tasks:

  • Garbage Collection, Estimator, Linker: task/subtask results will not be saved
  • Garbage Collection, Estimator, Linker: after the subtask completes, the task status will be updated to Done, and the task result ID will be None.
  • Garbage Collection: deleting samples will become unavailable.

  If the Image Store service is disabled after events with the image_origin_policy are generated, when using the Garbage Collection task and the remove_image_origins parameter, the Tasks service will still attempt to delete the source images with an external URL.

Image Store service disabling consequences for Backport 3#

In the Backport 3 service, the "BACKPORT3_ENABLE_PORTRAITS" setting is available, which enables you to disable the ability to use portraits, but leave the ability to use the rest of the functionality of the Image Store service. If the use of the Image Store service is disabled in the "ADDITIONAL_SERVICES_USAGE" setting, then the above setting must also be disabled.

When the Image Store service is disabled, samples and portraits, as well as the "get portrait" and "get portrait thumbnail" resources, become unavailable.

Handlers service disabling consequences#

When disabling the Handlers service:

• Launching the API service will lead to the unavailability of using the following requests:
  • "detect faces"
  • "extract attributes"
  • "estimator task"
  • All resource requests "/handlers"
  • All resource requests "/verifiers"
• Launching the Tasks service will lead to the unavailability of performing the tasks "Additional extraction" and "Estimator".
• Launching the Admin service will lead to the unavailability of performing the task "Additional extraction".

Events service disabling consequences#

When disabling the Events service:

• It will be possible to create a handler with an event saving policy, but the event itself will not be possible to generate.
• All requests to the "/events" resource will not work.
• Requests to resources "/handlers/{handler_id}/events", "/handlers/{handler_id}/events/raw" not will work.
• The "event_id" parameter in the "create face" request will not work.
• Matching by candidate events will not be available.
• Filters by events in tasks will not be available.
• Using the "ws handshake" request will not make sense.

Tasks service disabling consequences#

When disabling the Tasks service:

• All requests to the "/tasks" resource (sections "task processing", "task info", "task errors") will not work.
• The use of tasks in all services will be unavailable.

Tasks performed in the Admin service user interface are also unavailable when the Tasks service is disabled.

Sender service disabling consequences#

When disabling the Sender service:

• The "ws handshake" request will not work.
• The "notification_policy" policy in the handler will not work.

Disabling additional services#

Additional services can also be disabled if they were previously enabled.

If the Python Matcher Proxy service was previously used, then disabling it will make it impossible to use matching plugins or LIM module.

If the Lambda service was previously used, then disabling it will make it impossible to send requests to the "/lambdas" resources and work with the created lambda.

If Backport 3 or Backport 4 services were previously used, then disabling them will make it impossible to process LUNA PLATFORM 3/4 requests using LUNA PLATFORM 5.

Enabling Backport 3, Backport 4, User Interface 3 and User Interface 4 services is regulated only by launching the corresponding containers. There are no parameters in the "ADDITIONAL_SERVICE_USAGE" settings that enable these services.

Nuances of working with services#

When working with different services, it is necessary to take into account some nuances that will be described in this section.

Auto-orientation of rotated image#

It is not recommended to send rotated images to LP as they are not processed properly and should be rotated. There are two methods to auto-orient a rotated image — based on EXIF image data (query parameters) and using LP algorithms (Configurator setting). Both methods for automatic image orientation can be used together.

If auto-orientation is not used, the sample creation mechanism will rotate the image and produce an image with a random rotation angle.

Auto-orientation based on EXIF data#

This method of image orientation is performed in the query parameters using the "use_exif_info" parameter. This parameter can enable or disable auto-orientation of the image based on EXIF data.

This parameter is available and enabled by default in the following resources:

• "/detector"
• "/verifiers/{verifier_id}/verifications"
• "/sdk"
• "/handlers/{handler_id}/events"

The "use_exif_info" parameter cannot be used with samples. When the "warped_image" or "image_type" query parameter is set to the appropriate value, the parameter is ignored.

Auto-orientation based on Configurator setting#

This method of image orientation is performed in the Configurator using the "LUNA_REMOTE_SDK_USE_AUTO_ROTATION" setting. If this setting is enabled and the input image is rotated by 90, 180 or 270 degrees, then LP rotates the image to the correct angle. If this setting is enabled, but the input image is not rotated, then LP does not rotate the image.

Performing auto-orientation consumes a significant amount of server resources.

The "LUNA_HANDLERS_USE_AUTO_ROTATION" setting cannot be used with samples. If the "warped_image" or "image_type" query parameter is set to the appropriate value and the input image is a sample and rotated, then the "LUNA_HANDLERS_USE_AUTO_ROTATION" setting will be ignored.

Saving source images#

The URL to the source image can be saved in the "image_origin" field of the created events when processing the "/handlers/{handler_id}/events" request.

To do this, you should specify the "store_image" parameter in the "image_origin_policy" when creating handler.

Then you should set an image for processing in the "generate events" request.

If "use_external_references=0" and the URL to an external image was transferred in the "generate events" request, then this image will be saved to the Image Store storage, and the ID of the saved image will be added in the "image_origin" field of the generated event.

The "use_external_references" parameter enables you to save an external link instead of saving the image itself:

• If "use_external_references" = "1" and the URL to an external image was transferred in the "generate events" request, then that URL will be added in the "image_origin" field. The image itself will not be saved to the Image Store.
• If "use_external_references" = "1", the sample was provided in the "generate events" request and "face_sample_policy > store_sample" is enabled, the URL to the sample in the Image Store will be saved in the "image_origin" field. The duplication of the sample image will be avoided.

If an external URL is too long (more than 256 symbols), the service will store the image to the Image Store.

You can also provide the URL to the source image directly using the "/handlers/{handler_id}/events" resource. To do this, you should use the "application/json" or "multipart/form-data" body schema in the request. The URL should be specified in the "image_origin" field of the request.

If the "image_origin" is not empty, the provided URL will be used in the created event regardless of the "image_origin_policy" policy.

The image provided in the "image_origin" field will not be processed in the request. It is used as a source image only.

Neural networks#

Using neural networks, the parameters of faces or bodies are estimated and their descriptors are extracted.

There are two types of neural networks — for performing estimation and for extracting descriptors.

Neural networks for performing detection and estimation are located in the Remote SDK service container in the format <estimation_name>_<architecture>.plan, where <estimation_name> is the name of the neural network, <architecture> the architecture used (cpu-avx2 or gpu). Such neural networks are called detectors and estimators, respectively.

Neural networks for extracting descriptors are located in the Remote SDK service container in the format cnn<model>_<architecture>.plan, where <model> is the neural network model, <architecture> the architecture used (cpu-avx2 or gpu).

To work with a neural network, a configuration file of the cnndescriptor_<model>.conf format is used, where <model> is a neural network model. Configuration files for all neural networks are also located in the Remote SDK service container.

Next, information only for neural networks for extracting descriptors will be described.

It is possible to remove any neural network from the Remote SDK service container if use of some estimators or detectors is disabled.

The current build of LUNA PLATFORM supports neural network models for extracting descriptors:

The sizes of all neural network models for extracting face descriptors are given below:

The sizes of all neural network models for extracting body descriptors are given below:

See the detailed information about descriptor formats in the section "Descriptor formats".

Neural network 105 is newer version of neural network 102.

Descriptors received using different models of the neural networks are not comparable with each other. That is why you should re-extract all the descriptors from the existing samples if you are going to use a new NN model (see "Change neural network used").

You can store several descriptors for the same image with different NN linked to a single face.

See the "DEFAULT_FACE_DESCRIPTOR_VERSION" and "DEFAULT_HUMAN_DESCRIPTOR_VERSION" parameters in the Configurator service to check the current extraction neural network.

Before any actions with neural networks, you should consult with VisionLabs specialists.

Change neural network used#

Changing the used neural network model is required when it is necessary to improve the quality of face/body recognition or when old models are declared obsolete and removed from the Remote SDK service container.

When changing the neural network model used, one should:

• to preserve the possibility of using old descriptors: perform the long additional extraction task so the already existing descriptors can be extracted using the new neural network model (see "Launch Additional extraction task");
• set the new neural network model in LP configurations (see "Change neural network model in settings").

You should not change the default neural network, before finishing the additional extraction task.

You can both increase the model of the neural network, and lower it. To downgrade the neural network model, you need to perform similar steps to upgrading.

Launch Additional extraction task#

The additional extraction task can be launched using one of the following ways:

• Using the "/additional_extract" request to the Admin API.
• Using the Admin service user interface, by default located at http://<admin_server_ip>:5010.

Depending on the way, you need to specify the following information:

• Object type: faces or events.
• Extraction target: face descriptor, body descriptor or basic attributes.
• New neural network version (not applicable for basic attributes).

For more information, see "Additional extraction task" section.

Change neural network model in settings#

You should set the new neural network model in the configurations of services. Use the Configurator GUI for this purpose:

• Go to to the http://<configurator_server_ip>:5070.
• Set the required neural network in the "DEFAULT_FACE_DESCRIPTOR_VERSION" parameter (for faces) or the "DEFAULT_HUMAN_DESCRIPTOR_VERSION" (for bodies).
• Save changes using the "Save" button.
• Wait until the setting is applied to all the LP services.

Use non-delivery neural network model#

This section describes the process of moving a non-delivery neural network to the Remote SDK container. This is necessary if the user is using the old neural network from a previous version of LP and does not want to change it when upgrading to a new version of LP.

It is necessary to request an archive with neural network files from a VisionLabs representative. The archive contains the following files:

• Neural network file(s) cnn<model>_<architecture>.plan, where <model> — neural network model, <architecture> architecture used (cpu-avx2 and/or gpu).
• Configuration file cnndescriptor_<model>.conf, where <model> — neural network model.

After downloading the archive with the neural network and the archive with its configuration, you should perform the following steps:

• Unzip the archive.
• Assign rights to neural networks.
• Copy the neural networks and their configuration files to the launched Remote SDK container.
• Make sure that the required model of the neural network is used in the service configurations (see "Change neural network model in settings" section).

Below is an example of commands for transferring neural networks to the Remote SDK container.

Unzip neural networks#

Go to the directory with the archives and unzip them.

unzip <archive_name>.zip

Assign rights to neural networks#

chown -R 1001:0 <archive_name>/cnn<model>_<architecture>.plan

Copy neural network and configuration file to Remote SDK container#

Copy the neural network and its configuration file to the Remote SDK container using the following commands.

docker cp <archive_name>/cnn<model>_<architecture>.plan luna-remote-sdk:/srv/fsdk/data/

docker cp <archive_name>/cnndescriptor_<model>.conf luna-remote-sdk:/srv/fsdk/data/

luna-remote-sdk — Name of the launched Remote SDK container. This name may differ in your installation.

Check that the required model for the required device (CPU and/or GPU) was successfully loaded:

docker exec -t luna-remote-sdk ls /srv/fsdk/data/

Logging#

There are two ways to output logs in LUNA PLATFORM:

• Standard log output (stdout).
• Log output to a file.

Log output settings are set in the settings of each service in the <SERVICE_NAME>_LOGGER section.

By default, logs are output only to standard output.

You can view the service logs via standard output using the docker logs <service_name> command.

It is recommended to set up an external system for collecting and storing logs. This manual does not provide an example of configuring an external system to configure logs.

If necessary, you can use both methods of displaying logs.

When enabling logging to a file in a container, the default path is srv/logs for each container. Logs are saved in the format <service_name>_<type_of_logs>.txt, where:

• <service_name> — Name of the service, for example, luna-faces.
• <types_of_logs> — Type of output logs — "ERROR", "INFO", "WARNING", "DEBUG".

It is possible to create up to six files of each type. For each type of logs output in the Configurator service settings, you can set the maximum size (the "max_log_file_size" parameter in the <SERVICE_NAME>_LOGGER section). So, for example, six 1024 MB files can be created for the "INFO" type. For other types, the principle of operation is similar. Thus, if the maximum value of "max_log_file_size" is 1024 MB, then the total amount of memory occupied by logs in the container cannot exceed 6*4*1024=24576 MB. After the remaining space in the last file is used up, the first file will be overwritten. If it is necessary to reduce the amount of memory occupied by logs, then it is necessary to reduce the value of the parameter "max_log_file_size".

You can synchronize the service logs folder with the local folder on the server using the command -v /tmp/logs/<service_name>:/srv/logs\ when starting the container. To do this, you first need to create the corresponding directory on the server and assign it the necessary rights.

Without assigning rights to the mounted folder, the service will issue the corresponding error.

When you enable saving logs to a file, you should remember that logs occupy a certain place in the storage (see above), and the process of logging to a file negatively affects system performance.

Image check#

LUNA PLATFORM enables you to perform various front-type image checks. Check can be done either with thresholds conforming to ISO/IEC 19794-5:2011, and by manually entering thresholds and selecting the necessary checks.

The results of the checks are not stored in the database, they are returned only in the response.

ISO/IEC 19794-5:2011 checks are performed using the "/iso" resource (see detailed description in the "Image check according to ISO/IEC 19794-5:2011" section below).

Checks with manually specified thresholds using the "face_quality" group of checks of "detect_policy" policy of "/handlers" and "/verifiers" resources (see detailed description in the "Image check according to specified conditions" section below).

The possibility of performing such checks is regulated by a special parameter in the license file.

The result of all checks is determined by the "status" parameter, where:

• "0" — Any of the checks were not passed.
• "1" — All checks have been passed.

The result of each check is also displayed next to it (the "result" parameter).

You can enable checking for multiple faces in a photo using the "multiface_policy" parameter. Checks are performed for each face detection found in the photo. Check results are not aggregated.

For some checks, certain requirements should be met. For example, in order to get the correct results of the status of eyebrows, it is necessary that the head angles are in a certain range and the face width is at least 80 pixels. The requirements for the checks are listed in the section "Estimated data". If any requirements are not met when check a certain parameter, the results of checking this parameter may be incorrect.

For the following checks, it is not available to use a sample as a source image:

• Vertical/horizontal face position checks (parameters head_horizontal_center, head_vertical_center)
• Vertical/horizontal head sizes check (parameters head_width, head_height)
• Distance between eyes check (parameter eye_distance)
• Face width/height check (parameters indent_upper, intent_lower, intent_right, intent_left)
• Indents from image edges checks (parameters indent_upper, intent_lower, intent_right, intent_left)
• Illumination uniformity check according to ICAO standard (parameter illumination_uniformity)
• Dynamic range check (parameter dynamic_range)
• Background check (parameters background_lightness, background_uniformity)

The set of checks for "face_quality" and "/iso" resource is different (see the difference between checks in "Comparison table of available checks" section).

Image check according to ISO/IEC 19794-5:2011#

This check is performed using the "/iso" resource.

By default, images with one face present are checked. For each of the found faces, the estimates and coordinates of the found face will be returned. It should be noted that many ISO checks assume the presence of one face in the frame, so not all checks for multiple faces will be performed successfully.

The order of the returned responses after processing corresponds to the order of the transferred images.

You can additionally enable the extraction of EXIF data of the image in the request.

For each check, thresholds are set that comply with ISO requirements. The value of the thresholds for each check is given in the sample response to the "/iso" request in the OpenAPI documentation.

Some checks are united under one ISO requirement. For example, to successfully pass the eye status check, the statuses of the left and right eyes should take the value "open".

The following information is returned in the response body:

• Verdict on passing checks, which is 1 if all checks are successful.

• Results of each of the tests. This enables you to determine which particular test was not passed. The following values are returned:

  • The name of the check.
  • The value obtained after performing the check.
  • The default threshold. The thresholds are set in the system by the requirements of the ISO/IEC 19794-5:2011 standard and cannot be changed by the user.
  • The result of this check. When passing the thresholds, 1 is returned.

• The coordinates of the face.

If an error occurs for one of the processed images, for example, if the image is damaged, an error will be displayed in the response. Processing of the rest of the images will continue as usual.

In addition to the "/iso" resource, the ability to check for compliance with ISO/IEC 19794-5:2011 and ICAO standards is available in the "/detector" (parameter "estimate_face_quality") resource. The principle of performing checks is similar to the one described above, however, additional checks from the "face_quality" checks group are available in this resource.

Image check according to specified conditions#

The principle of operation is similar to check according the ISO standard, but the user has the right to decide for himself which checks need to be performed and which thresholds to set.

To enable checks, you should specify the value "1" in the "estimate" field for "face_quality". Image check is disabled by default. To disable a certain check, you need to set "0" in the "estimate" field for this check. By default, all checks are enabled and will be performed when "face_quality" is enabled.

Depending on the type of check, you can specify the minimum and maximum values of the threshold, or allowable values for this check. For this, the "threshold" field is used. If the minimum or maximum threshold is not set, the minimum or maximum allowable value will be automatically selected as the unset threshold. If the maximum value is unlimited (for example, ">=0"), then the value "null" will be returned in the "max" field in the event response body. If both thresholds are not specified, the check will be performed using the standard thresholds set in the "FACE_QUALITY_SETTINGS" section in the Configurator service or in the "face_quality" request body. If the threshold is specified in the "face_quality" request body, then this overrides the standard thresholds specified in the "FACE_QUALITY_SETTINGS" section.

Default thresholds are selected by VisionLabs specialists to obtain optimal results. These thresholds may vary depending on shooting conditions, equipment, etc.

When setting thresholds for the checks "Image quality" and "Head pose", it is recommended to take into account the standard thresholds preset in the system settings. For example, to check the image for blurriness (the "blurriness_quality" parameter), it is recommended to set the threshold in the range [0.57...0.65]. When setting a threshold outside this range, the results may be unpredictable. When choosing the angles of the head position, you need to pay attention to the recommended maximum thresholds for estimation in cooperative and non-cooperative modes. Information about these thresholds is provided in the relevant sections of the administrator manual.

It is recommended to consider the results of mouth state checks ("mouth_smiling", "mouth_occluded", "mouth_open", "smile_properties") together. So, for example, if the check revealed that the face is occluded with something, then the rest of the results of the mouth check will not be useful.

It is possible to enable filtering based on check results ("filter" parameter). If one of the "face_quality" checks for the detection fails, then the results and the reason for filtering will be returned. No further policies will be performed for this detection.

In addition, some checks are available in the "face_quality" group of checks, which are not available in the images check according the standard (see below).

Comparison table of available checks#

The following checks are available for "/iso" resource and "face_quality" group of checks of the "/handlers" and "/verifiers" resources:

* Several parameters can be specified for these checks.

Services health checks#

A "/healthcheck" resource is intended for service health checks.The resource can be used to actively check the health of a service, namely, whether the service can perform its functions in full or not. The ability to connect this service to the LP and DB services on which it depends is checked.

It is possible to set up a periodic resource check using HAProxy, NGINX or another system. This will enable you to determine that the service is unavailable and decide whether to disconnect the service from the contour or restart it.

Using the "include_luna_services" option, you can enable and disable health check for the LUNA PLATFORM services on which this service depends. If this option is enabled, additional requests are sent to the "/healthcheck" resources of these services. The "include_luna_services" option is disabled in order not to perform recursive checking of the same services. For example, when several services on which this service depends at once will send requests to the Faces service and thereby increase the load on it.

If the health check is successful, only the connection execution time in the "execution_time" field is returned.

If one or more services are unavailable, an error code 502 "Unhealthy" is returned. The response body lists the components, check statuses, and errors that have occurred. The error code 500 in the response body does not necessarily mean a problem with the service. A long request may fail due to exceeded timeouts, increased server load, network problems or other reasons.

When performing a request to the "/healthcheck" resource, it is recommended to set a timeout of several seconds. If the request does not have time to be processed, this is a sign that problems have arisen during the operation of the system.

To check the health of services, a "get health (redirect)" request is also available, which enables you to not specify the API version in the request.

Upload images from folder#

The "folder_uploader.py" script uploads images from the specified folder and processes uploaded images according to the preassigned parameters.

General information about the script#

The "folder_uploader.py" script can be utilized for downloading images using the API service only.

You cannot specify the Backport 4 address and port for utilizing this script. You can use the data downloaded to the LP 5 API in Backport 4 requests.

You cannot use the "folder_uploader.py" script to download data to Backport 3 service as the created objects for Backport 3 differs (e. g. "person" object is not created by the script).

Script usage#

Script pipeline:

1. Search images of the allowed type (formats: '.png', '.jpg', '.jpeg', '.bmp', '.ppm', '.tif', '.tiff', color model: RGB, CMYK) in the specified folder (source).
2. Start asynchronous image processing according to the specified parameters (see section "Script launching").

Image processing pipeline:

1. Detect faces and create samples.
2. Extract attributes.
3. Create faces and link them to a list.
4. Add record to the log file.

If an image was loaded successfully, the record is added to the {start_upload_time}_success_log.txt: success load logfile. The record has the following structure:

    {
    "image name": ..., 
    "face id": [...]
    }.

If errors occur at any step of the script processing, the image processing routine is terminated and a record is added to the error log file {start_upload_time}_error_log.txt: error. The record has the following structure:

    {
    "image name": ..., 
    "error": ...
    }

Install dependencies#

Before the script launching you must install all the required dependencies to launch the script.

It is strongly recommended to create a virtual environment for python dependencies installation.

Install Python packages (version 3.7 and higher) before launching installation. The packages are not provided in the distribution package and their installation is not described in this manual:

• python3.7
• python3.7-devel

Install gcc.

yum -y install gcc

Go to the directory with the script

cd /var/lib/luna/current/extras/utils/folder_uploader

Create a virtual environment

python3.7 -m venv venv

Activate the virtual environment

source venv/bin/activate

Install the tqdm library.

pip3.7 install tqdm 

Install luna3 libraries.

pip3.7 install ../luna3*.whl

Deactivate virtual environment

deactivate

GC script launching#

Use the command to run the script (the virtual environment must be activated):

python3.7 folder_uploader.py --account_id 6d071cca-fda5-4a03-84d5-5bea65904480 --source "Images/" --warped 0 --descriptor 1 --origin http://127.0.0.1:5000 --avatar 1  --list_id 0dde5158-e643-45a6-8a4d-ad42448a913b --name_as_userdata 1  

Make sure that the --descriptor parameter is set to 1 so descriptors are created.

The API version of the service is set to 6 by default in the script, and it cannot be changed using arguments.

--source "Images/" — "Images/" is the folder with images located near the "folder_uploader.py" script. Or you can specify the full path to the directory.

--list_id 0dde5158-e643-45a6-8a4d-ad42448a913b — Specify your existing list here.

--account_id 6d071cca-fda5-4a03-84d5-5bea65904480 — Specify the required account ID.

--origin http://127.0.0.1:5000 — Specify your current API service address and port here.

See help for more information about available script arguments:

python3.7 folder_uploader.py --help

Command line arguments:

• "account_id" — Account ID used in requests to LUNA PLATFORM (required).

• "source" — Directory with images to load (required).

• "warped" — Whether the images warped or not (0,1) (required).

• "descriptor" — Whether to extract descriptor (0,1). Default — 0.

• "origin" — Origin. Default — "http://127.0.0.1:5000".

• "avatar". Whether to set sample as avatar (0,1). Default — 0.

• "list_id" — List ID to link faces with (a new LUNA list will be created if list_id is not set and list_linked=1). Default — None.

• "list_linked" — Whether to link faces with list (0,1). Default — 1.

• "list_userdata" — User data for list to link faces with (for newly created list). Default — None.

• "pitch_threshold" — Maximum deviation pitch angle [0..180].

• "roll_threshold" — Maximum deviation roll angle [0..180].

• "yaw_threshold" — Maximum deviation yaw angle [0..180].

• "multi_face_allowed" — Whether to allow several face detection from single image (0,1). Default — 0.

• "get_major_detection" — Whether to choose major face detection by sample Manhattan distance from single image (0,1). Default — 0.

• "basic_attr" — Whether to extract basic attributes (0,1). Default — 1.

• "score_threshold" — Descriptor quality score threshold (0..1). Default — 0.

• "name_as_userdata" — Whether to use image name as user data (0,1). Default — 0.

• "concurrency" — Parallel processing image count. Default — 10.