Task processing

To schedule index building LIM Manager performs two types of background jobs:

  1. planning routine to schedule background indexing tasks to build or refresh indexes (according to the app config)

  2. lookup routine to submit indices for build & check building process

Note

The multi-instance mode of LIM Managers is supported with master auto selection based on redlock. The background planning & indexer lookup is the sole responsibility of the master. Other managers can accept user requests for one-time index building, as well as provide some information about tasks & index storage.

Planning routine

By default, LIM Manager builds indexes at startup and creates a further indexing schedule based on the app config.

There are two ways to schedule indexing: you can either specify the LIM_MANAGER_INDEXING.PLANNING_PERIOD property (which is the default option) in the app config file. The PLANNING_PERIOD is specified in seconds and defines the period between indexing. Alternatively, you may use LIM_MANAGER_INDEXING.PLANNING_SCHEDULE. This parameter is a cron-like string (no letter symbols support at the moment).

Note

Cron schedule triggers when current time matches all specified time constraints.

Parameters:

  • minute (int|str) – minute (0-59)

  • hour (int|str) – hour (0-23)

  • day (int|str) – day of month (1-31)

  • month (int|str) – month (1-12)

  • day_of_week (int|str) – number or name of weekday starting from sunday (0-6)

Expression description:

  • * - Fire on every value

  • */a - Fire every a values, starting from the minimum

  • a-b - Fire on any value within the a-b range (a must be smaller than b)

  • a-b/c - Fire every c values within the a-b range

  • x,y,z - Fire on any matching expression; can combine any number of any of the above expressions

There is no preferred method to schedule indexing, but if both PLANNING_SCHEDULE and PLANNING_PERIOD are specified, LIM Manager prioritizes the use of PLANNING_SCHEDULE.

In both cases, LIM Manager considers existing indexes. Therefore, if there are two app instances running or if the app work was interrupted for some reason, it won’t rebuild a fresh index whose lifespan has not yet expired. It will also consider the last indexing time (but not app startup time) as the base for planning further indexing calculations.

Index refresh

There is also an option to refresh indexes instead of rebuilding them. In order to save time and resources, a manager might only consider newly added or removed descriptors. In this case, the indexer won’t rebuild indexes from scratch. To prioritize refreshing instead of rebuilding, you might set LIM_MANAGER_INDEXING.REBUILD_RULES.DEFAULT to 0, which means ‘do not rebuild’

Due to the indexing algorithm, deleting descriptors from a built index leads to index degradation. The more descriptors were removed, the worse quality the index possesses, resulting in less accurate matching performed on this index. It is highly recommended to rebuild indexes in case of a high number of faces being removed from the original dataset.

To specify the allowed number of descriptors deletion since the original index build, you might set LIM_MANAGER_INDEXING.REBUILD_RULES.MAX_REMOVAL_FOR_REBUILD. The default is 0, which means ‘never rebuild from scratch’

Lookup routine

The indexer lookup routine is carried out with the period specified by LIM_MANAGER_INDEXING.LOOKUP_PERIOD setting from app config. At this stage, the LIM Manager checks the status of all indexers. If some LIM Indexer has finished index building, the LIM Manager updates task information and sends data for monitoring. If some LIM Indexer is ready to accept the task, the LIM Manager pops next task from the pending queue, submits the task to the indexer and updates task information.