Thin event plugin¶

The thin event plugin is given as an example for fast matching faces descriptors from simplified events.

There are several things that make matching using plugin faster in comparison to the default way:

thin-event database contains fewer data fields

thin-event database store event_id and face descriptor in one table

thin-event database store age, gender and some other filter in one table

Additional information about simplified events structure is given further.

Plugin requirements¶

Luna Python Matcher Proxy configuration for the plugin usage:

parameter LUNA_EVENTS is set to 1 in ADDITIONAL_SERVICES_USAGE configuration section
section LUNA_EVENTS_DB is filled with postgres database credentials.
it is also required to initialize plugin before usage
plugin name thin_event should be added to the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting.

Initialization¶

The thin event plugin uses configuration from config.py from thin_event directory for initialization.

The thin event plugin requires its own table with events in the database to work with.

To create the table execute:

python luna_python_matcher/plugins/thin_event/scripts/create.py --table --no-triggers

There are several ways to synchronize events to a simplified events database, including

materialized views (for more information see postgres materialized views)

streaming replication (for more information see postgres streaming replication)

It is also possible to fill the database with events using triggers. A trigger will save only new events to the database, events that existed before the trigger creation will not be available.

To create triggers execute the following command:

python luna_python_matcher/plugins/thin_event/scripts/create.py --no-table --triggers

Plugin matching cost¶

The thin event plugin matching cost is a configurable value, the default value is 10 for any suitable request. To change matching cost, just modify config.py from thin_event directory.

Plugin database scheme¶

Thin event plugin uses its own database model:

Module contains thin event model

class luna_python_matcher.plugins.thin_event.db_models.ThinEvent(**kwargs)[source]¶

Thin event ORM class

Plugin filters¶

Each HTTP matching request can be processed using thin event matcher if request contains candidate filters with origin equals events and one or several of the following filters: event_ids, event_id_gte, event_id_lt, create_time__gte, create_time__lt, external_ids, user_data, sources, tags, emotions, masks, ethnic_groups, gender, age__gte, age__lt, end_time__gte, end_time__lt.

Plugin targets¶

Each HTTP matching request can be processed using thin event matcher if request contains one or several of the next candidate targets: event_id, create_time, external_id, source, gender, age, emotion, mask, ethnic_group, user_data, tag, similarity, end_time.

If any other target(s) are specified in the request, it will take some time for Luna Python Matcher Proxy to supplement data received from the thin event matcher before a response will be formed.

Plugin exceptions¶

If any error occurred during thin event plugin matching, it will raise LunaPluginException (inherited from Rematch exception).

Disable plugin¶

To disable thin event plugin, it is required to remove thin_event from the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS configuration setting.

Warning

After the thin event plugin was disabled, you can enable it again by editing the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting.

If triggers were used during initialization, it creates excess database load and it is required to drop tables and functions used by the thin event plugin.

To drop tables and functions used by the thin event plugin, it is required to execute the following command after configuring config.py from the thin_event directory

python luna_python_matcher/plugins/thin_event/scripts/clear.py

Cached Matcher plugin¶

The cached matcher plugin is given as an example for fast matching faces from large lists.

Matching faces by a list is a time-intensive process, so the following methods are implemented to improve performance using the plugin:

a separate service as a LUNA-CACHED-MATCHER is used to match candidates and references

data for the candidate (list_id, face_id, descriptor) is stored in the memory cache. It provides quick access to the data

data is horizontally partitioned by shards (the LUNA-CACHED-MATCHER service is used as a shard), which provides fast retrieval of matching results

Alternatively, you can use horizontal database sharding to quickly retrieve the candidate face descriptor (for more information see postgres logical replication).

Dependencies¶

One or more matching services (LUNA-CACHED-MATCHER) are required. This service load faces from the lists into the cache for quick access to their descriptors.

See the LUNA-CACHED-MATCHER service documentation for startup information.

Plugin name cached_list_plugin should be added to the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting.

Initialization¶

Warning

The service responsible for matching (LUNA-CACHED-MATCHER) must be running at the time of plugin initialization.

By default, the cached matcher plugin uses luna-configurator for shards pulling configuration, which means that you need to create a limitation with a special name. Additional information about creating settings is presented here.

Example limitation json:

{
  "limitation_name": "LUNA_CACHED_LIST_PLUGIN",
  "validation_schema": {
    "type": "object",
    "properties": {
      "cost": {
        "type": "number",
        "minimum": 0
      },
      "shards": {
        "type": "array",
        "items": {
          "type": "string",
          "format": "uri"
        },
        "minItems": 1
      }
    },
    "required": ["cost", "shards"]
  },
  "services": ["cached-matcher-plugin"],
  "default_value": {
    "cost": 10,
    "shards": [
      "http://127.0.0.1:5300",
      "http://127.0.0.1:5301"
    ]
  }
}

Note

Required fields:

cost - is a float numeric expression of matching request process complexity

shards - list of URLs of a specific shard

To get matcher configuration using a custom file, change the load options and set the file path in the CachedListPlugin class:

class CachedListPlugin(BaseMatcherPlugin):
    """Cached list plugin class."""

    class Meta:
        configSource = "file"
        configFile = "<path_to_dir_with_file>/cached_matcher_settings.json"

Example json file:

{"cost": 10, "shards": ["http://127.0.0.1:5300", "http://127.0.0.1:5301"]}

Plugin filters¶

Each match request must contain the following fields for candidate filters:

origin [required] (candidates type, only faces)

list_id [required] (list ID with linked faces)

account_id [optional] (user account id)

Plugin targets¶

Cached matcher plugin can process match requests with one or several of the following targets:

face_id, account_id, similarity.

If any other targets are specified in the request, it will take some time for Luna Python Matcher Proxy to supplement data received from cached matcher before a response will be formed.

Plugin exceptions¶

If the list is not found in the cache, it will raise the Rematch exception (which will cause the list to be searched in the database for further matching).

If any error occurs during the matching in the cached list plugin, it will raise MatcherException.

Disable plugin¶

To disable cached matcher plugin, it is required to remove cached_list_plugin from the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS configuration setting.