Search plug-in

Integration & configuration

To integrate LVSM into Python Matcher Proxy plug-in system, follow the steps below:

1. Copy folder with LVSM plug-in (client/vsm_client) into Python Matcher Proxy plug-in folder (luna_python_matcher/plugins) under some name (e.g. vector_search_plugin):

   cp -r $LVSM_ROOT/client/vsm_client $LUNA_PYTHON_MATCHER_ROOT/luna_python_matcher/plugins/vector_search_plugin
   

   Note

   Skip this step if Python Matcher Proxy is running in docker. The vector search module plugin is integrated by default when running in container.

2. Change plug-in setting in Luna Configurator if needed.

   Vector database configuration setting example:

   LUNA_VECTOR_SEARCH_PLUGIN = {
       "DB": [{"type": "qdrant", "url": "http://127.0.0.1:6333", "token": "luna"}],
       "SEARCH_PARAMETERS": {"exact": true, "ef_search": 3200},
       "COLLECTION_OBSOLETE_PERIOD": null,
       "REQUEST_TIMEOUT": 60
   }
   

   Note

   Please note that the database must already be running with the parameters passed to LUNA_VECTOR_SEARCH_PLUGIN.DB settings

3. Extend LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting with new search plug-in:

   LUNA_MATCHER_PROXY_ACTIVE_PLUGINS = ["vector_search_plugin"]
   

   Note

   Please note that the plug-in name must be matching the directory name you have specified in step 1

Warning

Please note that plug-in settins are not Python Matcher Proxy settings, so changing them does not automatically restart Python Matcher Proxy and apply the new configuration. Therefore, changing plug-in settings after Python Matcher Proxy has started requires a manual restart of Python Matcher Proxy to apply the plug-in configuration.

Overview

Being integrated to Python Matcher Proxy plug-in system, LVSM plug-in performs an evaluation if the search request fits to any collection built. When considering the collection to search, the plug-in relies on the collection metadata stored in the database and performs the following checks:

1. Collection relevance (i.e. less than COLLECTION_OBSOLETE_PERIOD seconds has passed since the last update); if the COLLECTION_OBSOLETE_PERIOD setting is set to null, the synchronization status will not be checked.

2. Candidate origin in the search request matches the type of objects in the collection.

3. Candidate descriptor type and its version are present in the collection.

4. Candidate filters in the search request are a subset of the conditions for which the collection is built.

5. For candidate filters not completely covered by collection conditions, there are fields in the collection payload to perform the filtering by and select a candidate set from the collection.

Thus, for the collection with the following parameters:

{
    "object_type": "events",
    "descriptors": [{"descriptor_type": "face", "descriptor_version": 65}],
    "payload": ["event_id", "handler_id"],
    "counditions": {"account_id": "00000000-0000-4000-a000-000000311001"}
}

the following search request candidate filters will be acceptable:

{
    "origin": "events",
    "account_id": "00000000-0000-4000-a000-000000311001",
}

{
    "origin": "events",
    "account_id": "00000000-0000-4000-a000-000000311001",
    "event_id__gte": "5f8d954f-e76f-4381-be1a-7f4052013c63",
    "handler_ids": ["a9a98a13-94bf-4147-9b47-2041e0ee00b6"]
}

and the following ones will not:

{
    "origin": "events",
    "handler_ids": ["a9a98a13-94bf-4147-9b47-2041e0ee00b6"]
}

(filters are not a subset of the conditions for which the collection is built)

{
    "origin": "events",
    "account_id": "00000000-0000-4000-a000-000000311001",
    "handler_ids": ["a9a98a13-94bf-4147-9b47-2041e0ee00b6"],
    "age__gte": 42
}

(filters require some fields to be in the collection payload but they are missing)

After performing the search, applying the threshold and limit, LVSM plug-in returns the search result enriched with the payload to the Python Matcher Proxy.

Limitations

The LVSM plugin does not support filtering by geolocation (reach), by the event_id__lt and event_id__gte fields, or by like/nlike operators for text fields (e.g., user_data, composite fields in meta). If the values ​​of these filters differ from those specified in the collection, searching is impossible.