Configuration¶

Options¶

The service offers two configuration options:

Using the Configurator service.

Utilizing the configuration file.

The service reads settings from the configuration file by default.

You can specify the luna-config option in run command for pulling settings from the Luna Configurator service.

The service supports an auto-reload of configurations. You can enable this feature by specifying a config-reload option in the command line. The service periodically checks new settings. If current settings and new pulled settings are the same the service will not perform any actions. You can specify the check period in the pulling-time command line argument (10 seconds by default).

If a configuration pulling has failed, the service will work without applying any changes to the existing configurations.

If check connections with new settings have failed, the service will retry pulling new configurations after 5 seconds. The service will shut down after 5 failed attempts.

Warning

Service can work incorrectly while the new settings are being applied. It is strongly recommended not to send requests to the service when you change important settings (LIM_MANAGER_DB, LUNA_FACES_ADDRESS and other).

Configuration files for administrator¶

# LIM Indexer configuration file

[LUNA_CONFIGURATOR]
#: [integer] load configs from LUNA Configurator
USE_CONFIGURATOR = 0
#: [string] LUNA Configurator protocol, IP address, and port
LUNA_CONFIGURATOR_ORIGIN = http://127.0.0.1:5070
#: [integer] API version of the LUNA Configurator
LUNA_CONFIGURATOR_API = 1


[LIM_INDEXER_LOGGER]

#: [string] folder, where logs are saved (relative path begins in directory with application)
FOLDER_WITH_LOGS = ./
#: [string] level of debug print, by priority: "ERROR", "WARNING", "INFO", "DEBUG"
LOG_LEVEL = INFO
#: [string] time format used in log records: "LOCAL" or "UTC"
LOG_TIME = LOCAL
#: [integer] maximum log file size in megabytes before performing rotation (0 - without rotation)
MAX_LOG_FILE_SIZE = 1024
#: [integer] send log to stdout
LOG_TO_STDOUT = 1
#: [integer] send log to file
LOG_TO_FILE = 0
#: [integer] allow multiline stack trace in logs
MULTILINE_STACK_TRACE = 1
#: log format
FORMAT=default


[LUNA_MONITORING]

#: [string] storage type
STORAGE_TYPE = influx
#: [integer] send data for monitoring to InfluxDB
SEND_DATA_FOR_MONITORING = 1
#: [string] InfluxDB 2.x workspace
ORGANIZATION = luna
#: [string] InfluxDB 2.x authentication token
TOKEN = kofqt4Pfqjn6o0RBtMDQqVoJLgHoxxDUmmhiAZ7JS6VmEnrqZXQhxDhad8AX9tmiJH6CjM7Y1U8p5eSEocGzIA==
#: [string] InfluxDB 2.x bucket name
BUCKET = luna_monitoring
#: [string] InfluxDB IP address
HOST = 127.0.0.1
#: [integer] InfluxDB port
PORT = 8086
#: [integer] use HTTPS to connect to InfluxDB
USE_SSL = 0
#: [integer] frequency of sending monitoring data to InfluxDB (in seconds)
FLUSHING_PERIOD = 1


[LUNA_FACES_ADDRESS]

#: [string] LUNA Faces protocol, IP address, and port
ORIGIN = http://127.0.0.1:5030
#: [integer] API version of the LUNA Faces
API_VERSION = 3


[LUNA_FACES_TIMEOUTS]

#: [integer] HTTP connection timeout
CONNECT = 30
#: [integer] HTTP request timeout (total)
REQUEST = 60

[LIM_INDEXER_HTTP_SETTINGS]

#: [integer] service request timeout (in seconds)
REQUEST_TIMEOUT = 60
#: [integer] service response timeout (in seconds)
RESPONSE_TIMEOUT = 600
#: [integer] max request size (in bytes), default 1gb
REQUEST_MAX_SIZE = 1073741824
#: [integer] service http keep-alive timeout (in seconds)
KEEP_ALIVE_TIMEOUT = 15


[LUNA_SERVICE_METRICS]

#: [int] whether to enable /metrics path and statistics collection
ENABLED = 0
# [string] statistics data format (prometheus is the only option yet)
METRICS_FORMAT = prometheus


[LUNA_SERVICE_METRICS.EXTRA_LABELS]


[OTHER]

#: [string] type of storage: "S3" or "LOCAL", where "LOCAL" is local filesystem
INDEX_STORAGE_TYPE = LOCAL
#: [string] folder, where images are stored (relative path begins in directory with application)
INDEX_STORAGE_LOCAL = ./local_storage
#: [string] default face descriptor version
DEFAULT_FACE_DESCRIPTOR_VERSION = 62
#: [list(string)] list of active plugins
LIM_INDEXER_ACTIVE_PLUGINS = []

# LIM Manager configuration file

[LUNA_CONFIGURATOR]
#: [integer] load configs from LUNA Configurator
USE_CONFIGURATOR = 0
#: [string] LUNA Configurator protocol, IP address, and port
LUNA_CONFIGURATOR_ORIGIN = http://127.0.0.1:5070
#: [integer] API version of the LUNA Configurator
LUNA_CONFIGURATOR_API = 1


[LIM_MANAGER_LOGGER]

#: [string] folder, where logs are saved (relative path begins in directory with application)
FOLDER_WITH_LOGS = ./
#: [string] level of debug print, by priority: "ERROR", "WARNING", "INFO", "DEBUG"
LOG_LEVEL = INFO
#: [string] time format used in log records: "LOCAL" or "UTC"
LOG_TIME = LOCAL
#: [integer] maximum log file size in megabytes before performing rotation (0 - without rotation)
MAX_LOG_FILE_SIZE = 1024
#: [integer] send log to stdout
LOG_TO_STDOUT = 1
#: [integer] send log to file
LOG_TO_FILE = 0
#: [integer] allow multiline stack trace in logs
MULTILINE_STACK_TRACE = 1
#: log format
FORMAT=default


[LIM_MANAGER_DB]

#: [string] database username
DB_USER =
#: [string] database password
DB_PASSWORD =
#: [string] database IP address
DB_HOST = 127.0.0.1
#: [integer] database listener port
DB_PORT = 6379


[LIM_MANAGER_DB.DB_SETTINGS]

#: [integer] database connection pool size
CONNECTION_POOL_SIZE = 100


[LIM_MANAGER_DB.SENTINEL]

#: [string] sentinel master name
MASTER_NAME = index_manager
#: [list(string)] list of Redis Sentinel listeners (example [127.0.0.1:26379, 10.0.4.1:26379])
SENTINELS = []
#: [string] sentinel username
USER =
#: [string] sentinel password
PASSWORD =


[LUNA_MONITORING]

#: [string] storage type
STORAGE_TYPE = influx
#: [integer] send data for monitoring to InfluxDB
SEND_DATA_FOR_MONITORING = 1
#: [string] InfluxDB 2.x workspace
ORGANIZATION = luna
#: [string] InfluxDB 2.x authentication token
TOKEN = kofqt4Pfqjn6o0RBtMDQqVoJLgHoxxDUmmhiAZ7JS6VmEnrqZXQhxDhad8AX9tmiJH6CjM7Y1U8p5eSEocGzIA==
#: [string] InfluxDB 2.x bucket name
BUCKET = luna_monitoring
#: [string] InfluxDB IP address
HOST = 127.0.0.1
#: [integer] InfluxDB port
PORT = 8086
#: [integer] use HTTPS to connect to InfluxDB
USE_SSL = 0
#: [integer] frequency of sending monitoring data to InfluxDB (in seconds)
FLUSHING_PERIOD = 1


[LUNA_FACES_ADDRESS]

#: [string] LUNA Faces protocol, IP address, and port
ORIGIN = http://127.0.0.1:5030
#: [integer] API version of the LUNA Faces
API_VERSION = 3


[LUNA_FACES_TIMEOUTS]

#: [integer] HTTP connection timeout
CONNECT = 30
#: [integer] HTTP request timeout (total)
REQUEST = 60


[LIM_MANAGER_INDEXING]

#: [list(string)] list of lim-indexer origins
INDEXER_ORIGINS = [http://127.0.0.1:5180]
#: [integer] LUNA Faces polling period, in seconds
PLANNING_PERIOD = 600
# [string] Cron-like string of faces polling period (prioritized over PLANNING_PERIOD if set)
PLANNING_SCHEDULE =
#: [integer] LIM Indexer lookup period, in seconds
LOOKUP_PERIOD = 5
#: [integer] index construction parameter
EF_CONSTRUCTION = 1600


[LIM_MANAGER_INDEXING.FACE_LISTS]

#: [integer] min face count in LUNA Faces list for indexing
MIN_INDEXING_LIST_SIZE = 50000
#: [list(string)] lists for indexation, for value [dynamic] all the lists with attribute count > MIN_INDEXING_LIST_SIZE will be indexed
INDEXING_LISTS = [dynamic]


[LIM_MANAGER_INDEXING.REBUILD_RULES]

#: [integer] default behavior for indexing, whether to perform full rebuild every time
DEFAULT = 1
#: [integer] the number of maximum descriptors deletion before full index rebuild.
#: 0 - always refresh instead of rebuilding (never rebuild from scratch).
MAX_REMOVAL_FOR_REBUILD = 0


[LIM_MANAGER_HTTP_SETTINGS]

#: [integer] service request timeout (in seconds)
REQUEST_TIMEOUT = 60
#: [integer] service response timeout (in seconds)
RESPONSE_TIMEOUT = 600
#: [integer] max request size (in bytes), default 1gb
REQUEST_MAX_SIZE = 1073741824
#: [integer] service http keep-alive timeout (in seconds)
KEEP_ALIVE_TIMEOUT = 15


[LUNA_SERVICE_METRICS]

#: [int] whether to enable /metrics path and statistics collection
ENABLED = 0
# [string] statistics data format (prometheus is the only option yet)
METRICS_FORMAT = prometheus


[LUNA_SERVICE_METRICS.EXTRA_LABELS]


[OTHER]

#: type of storage: "S3" or "LOCAL", where "LOCAL" is local filesystem
INDEX_STORAGE_TYPE = LOCAL
#: folder, where images are stored (relative path begins in directory with application)
INDEX_STORAGE_LOCAL = ./local_storage
#: [list(string)] list of active plugins
LIM_MANAGER_ACTIVE_PLUGINS = []
#: [string] time format used in DB entries: LOCAL or UTC
STORAGE_TIME = LOCAL
#: [int] default face descriptor version
DEFAULT_FACE_DESCRIPTOR_VERSION = 62

# LIM Matcher configuration file

[LUNA_CONFIGURATOR]
#: [integer] load configs from LUNA Configurator
USE_CONFIGURATOR = 0
#: [string] LUNA Configurator protocol, IP address, and port
LUNA_CONFIGURATOR_ORIGIN = http://127.0.0.1:5070
#: [integer] API version of the LUNA Configurator
LUNA_CONFIGURATOR_API = 1


[LIM_MATCHER_LOGGER]

#: [string] folder, where logs are saved (relative path begins in directory with application)
FOLDER_WITH_LOGS = ./
#: [string] level of debug print, by priority: "ERROR", "WARNING", "INFO", "DEBUG"
LOG_LEVEL = INFO
#: [string] time format used in log records: "LOCAL" or "UTC"
LOG_TIME = LOCAL
#: [integer] maximum log file size in megabytes before performing rotation (0 - without rotation)
MAX_LOG_FILE_SIZE = 1024
#: [integer] send log to stdout
LOG_TO_STDOUT = 1
#: [integer] send log to file
LOG_TO_FILE = 0
#: [integer] allow multiline stack trace in logs
MULTILINE_STACK_TRACE = 1
#: log format
FORMAT=default


[LIM_MATCHER_DB]

#: [string] database username
DB_USER =
#: [string] database password
DB_PASSWORD =
#: [string] database IP address
DB_HOST = 127.0.0.1
#: [integer] database listener port
DB_PORT = 6379


[LIM_MATCHER_DB.DB_SETTINGS]

#: [integer] database connection pool size
CONNECTION_POOL_SIZE = 100


[LIM_MATCHER_DB.SENTINEL]

#: [string] sentinel master name
MASTER_NAME = indexed_matcher
#: [list(string)] list of Redis Sentinel listeners (example [127.0.0.1:26379, 10.0.4.1:26379])
SENTINELS = []
#: [string] sentinel username
USER =
#: [string] sentinel password
PASSWORD =


[LUNA_MONITORING]

#: [string] storage type
STORAGE_TYPE = influx
#: [integer] send data for monitoring to InfluxDB
SEND_DATA_FOR_MONITORING = 1
#: [string] InfluxDB 2.x workspace
ORGANIZATION = luna
#: [string] InfluxDB 2.x authentication token
TOKEN = kofqt4Pfqjn6o0RBtMDQqVoJLgHoxxDUmmhiAZ7JS6VmEnrqZXQhxDhad8AX9tmiJH6CjM7Y1U8p5eSEocGzIA==
#: [string] InfluxDB 2.x bucket name
BUCKET = luna_monitoring
#: [string] InfluxDB IP address
HOST = 127.0.0.1
#: [integer] InfluxDB port
PORT = 8086
#: [integer] use HTTPS to connect to InfluxDB
USE_SSL = 0
#: [integer] frequency of sending monitoring data to InfluxDB (in seconds)
FLUSHING_PERIOD = 1


[LUNA_LICENSES_ADDRESS]

#: [string] LUNA Licenses protocol, IP address, and port
ORIGIN = http://127.0.0.1:5120
#: [integer] API version of the LUNA Licenses
API_VERSION = 1


[LUNA_FACES_ADDRESS]

#: [string] LUNA Faces protocol, IP address, and port
ORIGIN = http://127.0.0.1:5030
#: [integer] API version of the LUNA Faces
API_VERSION = 3


[LUNA_FACES_TIMEOUTS]

#: [integer] HTTP connection timeout
CONNECT = 30
#: [integer] HTTP request timeout (total)
REQUEST = 60


[LIM_MATCHING]

#: [integer] index search parameter
EF_SEARCH = 3200


[LIM_MANAGER_HTTP_SETTINGS]

#: [integer] service request timeout (in seconds)
REQUEST_TIMEOUT = 60
#: [integer] service response timeout (in seconds)
RESPONSE_TIMEOUT = 600
#: [integer] max request size (in bytes), default 1gb
REQUEST_MAX_SIZE = 1073741824
#: [integer] service http keep-alive timeout (in seconds)
KEEP_ALIVE_TIMEOUT = 15


[LUNA_SERVICE_METRICS]

#: [int] whether to enable /metrics path and statistics collection
ENABLED = 0
# [string] statistics data format (prometheus is the only option yet)
METRICS_FORMAT = prometheus


[LUNA_SERVICE_METRICS.EXTRA_LABELS]


[OTHER]

#: [string] type of storage: "S3" or "LOCAL", where "LOCAL" is local filesystem
INDEX_STORAGE_TYPE = LOCAL
#: [string] folder, where images are stored (relative path begins in directory with application)
INDEX_STORAGE_LOCAL = ./local_storage
#: [list(string)] list of active plugins
LIM_MATCHER_ACTIVE_PLUGINS = []
#: [string] path to the dir with cache (omit to disable)
LIM_MATCHER_CACHE =
#: [int] default face descriptor version
DEFAULT_FACE_DESCRIPTOR_VERSION = 62


[LIM_MATCHER_REFRESH]
#: [integer] whether to perform index refreshing
ENABLED = 1

Environment variables configuration¶

LIM_MANAGER_MASTER_LOCK - index manager redis master lock name, default “lim_manager_master”.
LIM_MATCHER_CONSUMER - redis consumer group for matching requests, default “lim_matcher”.
LIM_MATCHER_LOCK_PREFIX - matcher redis lock name prefix, default “lim_matcher”.

Extended administrator configuration¶

Application config with global variables.

manager.lim_manager.config.config.MASTER_KEY = 'lim_manager_master'¶: Index manager master lock name (redis)

Application config with global variables.

matcher.lim_matcher.config.config.CONSUMER_GROUP = 'lim_matcher'¶: Redis consumer group for matching requests

matcher.lim_matcher.config.config.LISTENER_COUNT = 2¶: Matcher listener count

matcher.lim_matcher.config.config.LOCK_PREFIX = 'lim_matcher'¶: Redis lock name prefix

matcher.lim_matcher.config.config.LOCK_TIMEOUT = 60¶: lock holding timeout, in seconds

matcher.lim_matcher.config.config.MEMORY_USAGE_LIMIT = 80¶: Memory utilization limit, in percent

matcher.lim_matcher.config.config.MESSAGE_LIMIT = 20¶: returned message number limit from Redis

matcher.lim_matcher.config.config.MESSAGE_TIMEOUT = 10¶: receiving message timeout from Redis stream (milliseconds)

class matcher.lim_matcher.config.config.MatcherConfig(efSearch, descriptorVersion, refreshIndexes=None, storagePath=None, cachePath=None, consumerGroup='lim_matcher', listenerCount=2)[source]¶: Indexed matcher configuration parameters.

matcher.lim_matcher.config.config.REFRESH_BATCH_SIZE = 10¶: Number of descriptors to add/remove per iteration when doing index refresh