Skip to content

Index services#

Index Manager service#

The service manages the process of creating indexes for lists containing face descriptors and performs the following tasks:

  • generates tasks for building the index;
  • sends tasks to the internal queue;
  • retrieves tasks from the internal queue and coordinates the process of sending tasks for indexing to the Indexer service.

It is recommended to run at least two manager instances for redundancy purposes. Since task management is carried out through the Redis, if one manager is down, the second one will be able to continue its work from the instant step.

Background routines#

The Index Manager service performs two types of background routines in parallel:

  • planning routine;
  • lookup routine.

In the planning routine, the Index Manager checks which sets of lists are to be indexed, then creates tasks under a unique "task_id" and places them in the internal queue. The planning procedure is executed with the period (sec) specified in the "planning_period" setting of the "LIM_MANAGER_INDEXING" of the Configurator service.

In the lookup routine, the Index Manager checks the status of all running Indexer instances. If any Indexer instance has finished building the index, the Index Manager updates the task information and submits the data for monitoring. If any Indexer instance is ready to accept the task, the Index Manager service retrieves the next task from the internal queue, sends the task to the free Indexer instance, and updates the task information. The lookup routine is performed with the period (sec) specified in the "lookup_period" setting of the "LIM_MANAGER_INDEXING" of the Configurator service.

Index Manager storage#

All information about the created tasks is stored in the Redis database. Also, using the Redis Redlock mechanism, work with multiple instances is regulated.

Work with multiple instances#

The multiple instance mode is supported by automatic selection of the master instance based on Redis Redlock.

See https://redis.io/docs/reference/patterns/distributed-locks/ for details on distributed locks done with Redis.

Only the master instance can perform planning and lookup background routines. The remaining instances can only accept requests for a one-time index creation, as well as issue responses to GET requests.

Requests to service#

Interaction with the Index Manager service is performed using HTTP requests. The main requests are listed below:

  • "get queue" - get list of tasks and their number from the queue

  • "get tasks" - get information on tasks:

    • task_id
    • status - "pending", "indexing", "success", "error"
    • create_time - index build create time in RFC 3339 format
    • start_time - index build start time in RFC 3339 format
    • end_time - index build end time in RFC 3339 format
    • indexer - address of the server where the Indexer instance that processes the specified task is running
    • error - error received during index build
    • content - processed "list_id"

    If necessary, you can filter the received tasks.

  • "create task" - create task to build the index once

  • "remove tasks" - delete tasks. If necessary, you can filter the tasks to be deleted.

  • "get indexes" - get the number of indexes, as well as the following information for each index:

    • index_id (equal to task_id)
    • index_type (list only)
    • label (processed "list_id")

  • "remove indexes" - delete the index from the repository by ID

  • "get most relevant indexes" - get information on the most relevant index, i.e. by the last built index for the list.

See the OpenAPI specification for more information about requests made to the Index Manager service and other requests.

Indexer service#

The Indexer service is intended to process tasks received by the Index Manager service and perform the indexes creation process.

Requests to the Indexer service are not intended for the user. All requests related to the LUNA Index Module must be made to the Index Manager service (see "Requests to Index Manager service").

The deployment of the Indexer service should be done on a separate server, because building an index takes a lot of resources for a long time. One Indexer instance can only build one index at a time, so it is recommended to run multiple indexer instances. The indexer must be also configured with storage, which must be large enough.

Indexed Matcher service#

The Indexed Matcher service loads the most relevant indexes from the index storage (file system) and processes matching requests.

On startup, the Indexed Matcher service caches all indexes of the latest version from the index storage and sets up Redis streams to accept match messages for all matching labels loaded into the index storage.

Requests to the Indexed Matcher service are not intended for the user. All requests related to the LUNA Index Module must be made to the Index Manager service (see "Requests to Index Manager service").

Indexed Matcher does not communicate with other LIM services. It only monitors the storage, and when indices appear it loads them into memory. Since matching requests processing is carried out through the Redis streams, any number of matcher instances could be run without any system config updates. The number of Indexed Matcher instances should be determined by performance requirements.

Index reloading#

In-memory indexes in the Indexed Matcher service are synchronized with the store by a periodic background process called index reloading.

If the index is removed from storage, the index is also removed from the Indexed Matcher service's memory.

If a new index with a new match label appears in the store, the Indexed Matcher service will attempt to load the new index into memory.

If a new index appears in the store with a newer version of the matching label than the index loaded into memory, the Indexed Matcher service will try to load the new index into memory instead of the old one.

Replacing index with outdated version of list (Index 1) with new one (Index 3)
Replacing index with outdated version of list (Index 1) with new one (Index 3)

To ensure that the given index can only be reloaded by one Indexed Matcher service at a time, the Redis Redlock mechanism is used. If a lock is set, the older version of the index is removed from the Indexed Matcher service's memory and the newer one is loaded.

If there is a problem loading the index, for example, lack of memory, an appropriate message is sent to the logs and monitoring.

When the index is reloaded, the Indexed Matcher service does not accept matching requests for the corresponding label. However, only one Indexed Matcher can reload the index for a particular label at a time. Therefore, it is recommended to run multiple instances of the Indexed Matcher in order to be able to match all labels at any time.

Cached index refreshing#

By default, the Indexed Matcher service monitors lists with faces for changes. If new changes are made to the list, the Indexed Matcher service updates the corresponding indexes in its memory by gradually adding a small number of descriptors.

The use of this functionality is controlled by the "enabled" setting of the "LIM_MATCHER_REFRESH" section of the Configurator service.

This information is described for an index that is already loaded into the memory of the Indexed Matcher service. The index used and the index in the storage may differ.

When a cached index is updated, the Indexed Matcher service stops matching on that index, but continues to accept new match requests for that index. By adding a small number of descriptors to the cached index, the matching process is performed with minimal interruption. However, it should be taken into account that if elements are inserted into the list too often (dozens and hundreds of additions), this will affect a significant degradation in the speed of work, up to an almost complete stop of the matching process.

If this functionality is used, then it is not necessary and not recommended to perform frequent index rebuilds. Accordingly, it is recommended to increase the planning routine period ("planning_period" setting of the "LIM_MANAGER_INDEXING" section of the Configurator service). However, adding new faces to the cached index is slower than rebuilding the index, so it makes no sense to use this function if a very large number of faces have been added to the list. In this case, it is easier to rebuild the index again.

Unlinking faces from the list does not remove those faces from the cached index. In this case, the descriptors are marked as unsearchable, so the index retains the storage space allocated to them.