Thin event plugin

The thin event plugin is given as an example for fast matching faces descriptors from simplified events.

There are several things that make matching using plugin faster in comparison to the default way:

  • thin-event database contains fewer data fields

  • thin-event database store event_id and face descriptor in one table

  • thin-event database store age, gender and some other filter in one table

Additional information about simplified events structure is given further.

Plugin requirements

Luna Python Matcher Proxy configuration for the plugin usage:

  • parameter LUNA_EVENTS is set to 1 in ADDITIONAL_SERVICES_USAGE configuration section

  • section LUNA_EVENTS_DB is filled with postgres database credentials.

  • it is also required to initialize plugin before usage

  • plugin name thin_event should be added to the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting.

Initialization

The thin event plugin uses configuration from config.py from thin_event directory for initialization.

The thin event plugin requires its own table with events in the database to work with.

To create the table execute:

python luna_python_matcher/plugins/thin_event/scripts/create.py --table --no-triggers

There are several ways to synchronize events to a simplified events database, including

It is also possible to fill the database with events using triggers. A trigger will save only new events to the database, events that existed before the trigger creation will not be available.

To create triggers execute the following command:

python luna_python_matcher/plugins/thin_event/scripts/create.py --no-table --triggers

Plugin matching cost

The thin event plugin matching cost is a configurable value, the default value is 10 for any suitable request. To change matching cost, just modify config.py from thin_event directory.

Plugin database scheme

Thin event plugin uses its own database model:

Module contains thin event model

class luna_python_matcher.plugins.thin_event.db_models.ThinEvent(**kwargs)[source]

Thin event ORM class

Plugin filters

Each HTTP matching request can be processed using thin event matcher if request contains candidate filters with origin equals events and one or several of the following filters: event_ids, event_id_gte, event_id_lt, create_time__gte, create_time__lt, external_ids, user_data, sources, tags, emotions, masks, ethnic_groups, gender, age__gte, age__lt, end_time__gte, end_time__lt.

Plugin targets

Each HTTP matching request can be processed using thin event matcher if request contains one or several of the next candidate targets: event_id, create_time, external_id, source, gender, age, emotion, mask, ethnic_group, user_data, tag, similarity, end_time.

If any other target(s) are specified in the request, it will take some time for Luna Python Matcher Proxy to supplement data received from the thin event matcher before a response will be formed.

Plugin exceptions

If any error occurred during thin event plugin matching, it will raise LunaPluginException (inherited from Rematch exception).

Disable plugin

To disable thin event plugin, it is required to remove thin_event from the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS configuration setting.

Warning

After the thin event plugin was disabled, you can enable it again by editing the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting.

If triggers were used during initialization, it creates excess database load and it is required to drop tables and functions used by the thin event plugin.

To drop tables and functions used by the thin event plugin, it is required to execute the following command after configuring config.py from the thin_event directory

python luna_python_matcher/plugins/thin_event/scripts/clear.py

Cached Matcher plugin

The cached matcher plugin is given as an example for fast matching faces from large lists.

Matching faces by a list is a time-intensive process, so the following methods are implemented to improve performance using the plugin:

  • a separate service as a LUNA-CACHED-MATCHER is used to match candidates and references

  • data for the candidate (list_id, face_id, descriptor) is stored in the memory cache. It provides quick access to the data

  • data is horizontally partitioned by shards (the LUNA-CACHED-MATCHER service is used as a shard), which provides fast retrieval of matching results

Alternatively, you can use horizontal database sharding to quickly retrieve the candidate face descriptor (for more information see postgres logical replication).

Dependencies

One or more matching services (LUNA-CACHED-MATCHER) are required. This service load faces from the lists into the cache for quick access to their descriptors.

See the LUNA-CACHED-MATCHER service documentation for startup information.

Plugin name cached_list_plugin should be added to the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS setting.

Initialization

Warning

The service responsible for matching (LUNA-CACHED-MATCHER) must be running at the time of plugin initialization.

By default, the cached matcher plugin uses luna-configurator for shards pulling configuration, which means that you need to create a limitation with a special name. Additional information about creating settings is presented here.

Example limitation json:

{
  "limitation_name": "LUNA_CACHED_LIST_PLUGIN",
  "validation_schema": {
    "type": "object",
    "properties": {
      "cost": {
        "type": "number",
        "minimum": 0
      },
      "shards": {
        "type": "array",
        "items": {
          "type": "string",
          "format": "uri"
        },
        "minItems": 1
      }
    },
    "required": ["cost", "shards"]
  },
  "services": ["cached-matcher-plugin"],
  "default_value": {
    "cost": 10,
    "shards": [
      "http://127.0.0.1:5300",
      "http://127.0.0.1:5301"
    ]
  }
}

Note

Required fields:

  • cost - is a float numeric expression of matching request process complexity

  • shards - list of URLs of a specific shard

To get matcher configuration using a custom file, change the load options and set the file path in the CachedListPlugin class:

class CachedListPlugin(BaseMatcherPlugin):
    """Cached list plugin class."""

    class Meta:
        configSource = "file"
        configFile = "<path_to_dir_with_file>/cached_matcher_settings.json"

Example json file:

{"cost": 10, "shards": ["http://127.0.0.1:5300", "http://127.0.0.1:5301"]}

Plugin filters

Each match request must contain the following fields for candidate filters:

  • origin [required] (candidates type, only faces)

  • list_id [required] (list ID with linked faces)

  • account_id [optional] (user account id)

Plugin targets

Cached matcher plugin can process match requests with one or several of the following targets:

face_id, account_id, similarity.

If any other targets are specified in the request, it will take some time for Luna Python Matcher Proxy to supplement data received from cached matcher before a response will be formed.

Plugin exceptions

If the list is not found in the cache, it will raise the Rematch exception (which will cause the list to be searched in the database for further matching).

If any error occurs during the matching in the cached list plugin, it will raise MatcherException.

Disable plugin

To disable cached matcher plugin, it is required to remove cached_list_plugin from the LUNA_MATCHER_PROXY_ACTIVE_PLUGINS configuration setting.