Skip to content

Index services#

Index Manager service#

The service manages the process of creating indexes for lists containing face descriptors and performs the following tasks:

  • generates tasks for building the index;
  • sends tasks to the internal queue;
  • retrieves tasks from the internal queue and coordinates the process of sending tasks for indexing to the Indexer service.

It is recommended to run at least two manager instances for redundancy purposes. Since task management is carried out through the Redis, if one manager is down, the second one will be able to continue its work from the instant step.

Background routines#

The Index Manager service performs two types of background routines in parallel:

  • planning routine;
  • lookup routine.

In the planning routine, the Index Manager checks which sets of lists are to be indexed, then creates tasks under a unique "task_id" and places them in the internal queue. The planning procedure is executed with the period (sec) specified in the "planning_period" setting of the "LIM_MANAGER_INDEXING" of the Configurator service.

In the lookup routine, the Index Manager checks the status of all running Indexer instances. If any Indexer instance has finished building the index, the Index Manager updates the task information and submits the data for monitoring. If any Indexer instance is ready to accept the task, the Index Manager service retrieves the next task from the internal queue, sends the task to the free Indexer instance, and updates the task information. The lookup routine is performed with the period (sec) specified in the "lookup_period" setting of the "LIM_MANAGER_INDEXING" of the Configurator service.

Index Manager storage#

All information about the created tasks is stored in the Redis database. Also, using the Redis Redlock mechanism, work with multiple instances is regulated.

Work with multiple instances#

The multiple instance mode is supported by automatic selection of the master instance based on Redis Redlock.

See https://redis.io/docs/reference/patterns/distributed-locks/ for details on distributed locks done with Redis.

Only the master instance can perform planning and lookup background routines. The remaining instances can only accept requests for a one-time index creation, as well as issue responses to GET requests.

Requests to service#

Interaction with the Index Manager service is performed using HTTP requests. The main requests are listed below:

  • "get queue" - get list of tasks and their number from the queue

  • "get tasks" - get information on tasks:

    • task_id
    • status - "pending", "indexing", "success", "error"
    • create_time - index build create time in RFC 3339 format
    • start_time - index build start time in RFC 3339 format
    • end_time - index build end time in RFC 3339 format
    • indexer - address of the server where the Indexer instance that processes the specified task is running
    • error - error received during index build
    • content - processed "list_id"

    If necessary, you can filter the received tasks.

  • "create task" - create task to build the index once

  • "remove tasks" - delete tasks. If necessary, you can filter the tasks to be deleted.

  • "get indexes" - get the number of indexes, as well as the following information for each index:

    • index_id (equal to task_id)
    • index_type (list only)
    • label (processed "list_id")

  • "remove indexes" - delete the index from the repository by ID

  • "get most relevant indexes" - get information on the most relevant index, i.e. by the last built index for the list.

See the OpenAPI specification for more information about requests made to the Index Manager service and other requests.

Indexer service#

The Indexer service is intended to process tasks received by the Index Manager service and perform the indexes creation process.

Requests to the Indexer service are not intended for the user. All requests related to the LUNA Index Module must be made to the Index Manager service (see "Requests to Index Manager service").

The deployment of the Indexer service should be done on a separate server, because building an index takes a lot of resources for a long time. One Indexer instance can only build one index at a time, so it is recommended to run multiple indexer instances. The indexer must be also configured with storage, which must be large enough.

Indexed Matcher service#

The Indexed Matcher service loads the most relevant indexes from the index storage (file system) and processes matching requests.

On startup, the Indexed Matcher service caches all indexes of the latest version from the index storage and sets up Redis streams to accept match messages for all matching labels loaded into the index storage.

Requests to the Indexed Matcher service are not intended for the user. All requests related to the LUNA Index Module must be made to the Index Manager service (see "Requests to Index Manager service").

Indexed Matcher does not communicate with other LIM services. It only monitors the storage, and when indices appear it loads them into memory. Since matching requests processing is carried out through the Redis streams, any number of matcher instances could be run without any system config updates. The number of Indexed Matcher instances should be determined by performance requirements.

Index reloading#

In-memory indexes in the Indexed Matcher service are synchronized with the store by a periodic background process called index reloading.

If the index is removed from storage, the index is also removed from the Indexed Matcher service's memory.

If a new index with a new match label appears in the store, the Indexed Matcher service will attempt to load the new index into memory.

If a new index appears in the store with a newer version of the matching label than the index loaded into memory, the Indexed Matcher service will try to load the new index into memory instead of the old one.

Replacing index with outdated version of list (Index 1) with new one (Index 3)
Replacing index with outdated version of list (Index 1) with new one (Index 3)

To ensure that the given index can only be reloaded by one Indexed Matcher service at a time, the Redis Redlock mechanism is used. If a lock is set, the older version of the index is removed from the Indexed Matcher service's memory and the newer one is loaded.

If there is a problem loading the index, for example, lack of memory, an appropriate message is sent to the logs and monitoring.

When the index is reloaded, the Indexed Matcher service does not accept matching requests for the corresponding label. However, only one Indexed Matcher can reload the index for a particular label at a time. Therefore, it is recommended to run multiple instances of the Indexed Matcher in order to be able to match all labels at any time.