Indexing flow

Indexing consists of several steps.

1. At some point, LIM Manager decides that its time to index some set of descriptors (see indexing sets for details). When processing tasks, LIM Manager issues a request to LIM Indexer with indexing set content and index identifier specified to begin index building.

There is a POST /index_list request example for face list indexing:

{
    "list_id": "82d79f08-b152-40ad-8b4e-8f9b77f05d00",
    "index_id": "f9285806-371e-4c61-be00-0424029856e3"
}

2. Upon receiving indexing request, LIM Indexer spawns separate process for indexing. By using separate process we protect LIM Indexer from being killed by OOM. At this point LIM Indexer sets its state to “indexing”.

At this stage, GET request on /state resource should return the following response:

{
    "state": "indexing",
    "index_id": "f9285806-371e-4c61-be00-0424029856e3"
}

  1. At first, when the indexing process has started, it fetches descriptors from luna-faces service. Fetching is done in batches of 1000 items.

  2. After all descriptors have been fetched and loaded into memory, LIM Indexer begins building of the index with help of Luna SDK. A directed descriptor dependency graph is built, which will be used by the LIM Matcher.

  3. Next, when indexing has finished, the index itself is being saved using configured backend (either S3 or filesystem). See index_storage for details.

  4. In the end, after the index has been successfully stored, indexing process stops. At this point LIM Indexer sets its state to “success”.

After that, GET request on /state resource should return the following response:

{
    "state": "success",
    "index_id": "f9285806-371e-4c61-be00-0424029856e3"
}