Task processing

To perform index building LIM Manager performs two types of background jobs:

1. planning routine to schedule background indexing tasks (according to the app config)

2. lookup routine to submit indices for build & check building process

Note

The multi-instance mode of LIM Managers is supported with master auto selection based on redlock. The background planning & indexer lookup is the sole responsibility of the master. Other managers can accept user requests for one-time index building, as well as provide some information about tasks & index storage.

Planning routine

The planning routine is carried out with the period specified by LIM_MANAGER_INDEXING.PLANNING_PERIOD setting from app config. At this stage, the LIM Manager checks which sets should be indexed, then creates tasks and puts them into the pending queue.

Lookup routine

The indexer lookup routine is carried out with the period specified by LIM_MANAGER_INDEXING.LOOKUP_PERIOD setting from app config. At this stage, the LIM Manager checks the status of all indexers. If some LIM Indexer has finished index building, the LIM Manager updates task information and sends data for monitoring. If some LIM Indexer is ready to accept the task, the LIM Manager pops next task from the pending queue, submits the task to the indexer and updates task information.