Configuration¶

The service allowed two variant of configuration:

use the Configurator service

use the configuration file

The service reads settings from the configuration file by default.

You can specify the –luna-config option for pulling settings from the Configurator service, see configurator.

Auto reload configuration¶

The service supports an auto-reload of configurations. You can enable this feature by specifying a config-reload option in the command line. The service periodically checks new settings. If current settings and new pulled settings are the same the service will not perform any actions. You can specify the check period in the pulling-time command line argument (10 seconds by default).

If a configuration fetching has failed, the service will work without applying any changes to the existing configurations.

If check connections with the new configuration have failed, the service will retry pulling new configuration after pulling-time seconds. The service will shut down after 10 failed attempts.

Warning

Service can work incorrectly while the new settings are being applied. It is strongly recommended not to send requests to the service when you change important settings (s3 credentials, work plugins list, and others).

Configuration file for administrator¶

Warning

The Platform limits parameter enables you to configure various limits:

References and candidates limits in the request.
Limit of the candidates returned in the response for each reference.
Limit for the number of items for some filters.

# Luna-tasks configuration file


[LUNA_CONFIGURATOR]
#: load configs from LUNA Configurator
USE_CONFIGURATOR = 0
#: LUNA Configurator protocol, IP address, and port
LUNA_CONFIGURATOR_ORIGIN = http://127.0.0.1:5070
#: API version of the LUNA Configurator
LUNA_CONFIGURATOR_API  = 1


[LUNA_TASKS_DB]

#: type of database: "postgres" or "oracle"; "postgres" type is used by default
DB_TYPE = postgres
#: database username
DB_USER = luna
#: database password
DB_PASSWORD = luna
#: database name for "postgres" type, SID name for "oracle" type
DB_NAME = luna_tasks
#: database IP address
DB_HOST = 127.0.0.1
#: database listener port, 5432 - default for "postgres" type, 1521 - default for "oracle" type
DB_PORT = 5432


[LUNA_TASKS_DB.DB_SETTINGS]

#: [integer] database connection pool size
CONNECTION_POOL_SIZE = 5


[LUNA_TASKS_LOGGER]

#: folder, where logs are saved
FOLDER_WITH_LOGS = ./
#: level of debug print, by priority: "ERROR", "WARNING", "INFO", "DEBUG"
LOG_LEVEL = INFO
#: time format used in log records: "LOCAL" or "UTC"
LOG_TIME = LOCAL
#: maximum log file size in megabytes before performing rotation (0 - without rotation)
MAX_LOG_FILE_SIZE = 1024
#: send log to stdout
LOG_TO_STDOUT = 1
#: send log to file
LOG_TO_FILE = 0
#: allow multiline stack trace in logs
MULTILINE_STACK_TRACE = 1

[LUNA_MONITORING]

#: [string] storage type
STORAGE_TYPE = influx
#: [integer] send data for monitoring to InfluxDB
SEND_DATA_FOR_MONITORING = 1
#: [string] InfluxDB 2.x workspace
ORGANIZATION = luna
#: [string] InfluxDB 2.x authentication token
TOKEN = kofqt4Pfqjn6o0RBtMDQqVoJLgHoxxDUmmhiAZ7JS6VmEnrqZXQhxDhad8AX9tmiJH6CjM7Y1U8p5eSEocGzIA==
#: [string] InfluxDB 2.x bucket name
BUCKET = luna_monitoring
#: [string] InfluxDB IP address
HOST = 127.0.0.1
#: [integer] InfluxDB port
PORT = 8086
#: [integer] use HTTPS to connect to InfluxDB
USE_SSL = 0
#: [integer] frequency of sending monitoring data to InfluxDB (in seconds)
FLUSHING_PERIOD = 1


[LUNA_IMAGE_STORE_FACES_SAMPLES_ADDRESS]

#: LUNA Image Store protocol, IP address, and port (for samples)
ORIGIN = http://127.0.0.1:5020
#: API version of the LUNA Image Store (for samples)
API_VERSION = 1
#: *bucket* name, where samples will be stored
BUCKET = visionlabs-samples

[LUNA_IMAGE_STORE_BODIES_SAMPLES_ADDRESS]

#: LUNA Image Store protocol, IP address, and port (for samples)
ORIGIN = http://127.0.0.1:5020
#: API version of the LUNA Image Store (for samples)
API_VERSION = 1
#: *bucket* name, where samples will be stored
BUCKET = visionlabs-bodies-samples

[LUNA_IMAGE_STORE_IMAGES_ADDRESS]

#: LUNA Image Store protocol, IP address, and port (for samples)
ORIGIN = http://127.0.0.1:5020
#: API version of the LUNA Image Store (for samples)
API_VERSION = 1
#: *bucket* name, where samples will be stored
BUCKET = visionlabs-image-origin

[LUNA_IMAGE_STORE_FACES_SAMPLES_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 60


[LUNA_IMAGE_STORE_BODIES_SAMPLES_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 30
#: REQUEST TIMEOUT (total)
REQUEST =  60


[LUNA_IMAGE_STORE_IMAGES_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 30
#: REQUEST TIMEOUT (total)
REQUEST =  60


[LUNA_IMAGE_STORE_TASK_RESULT_ADDRESS]

#: LUNA Image Store protocol, IP address, and port (for task result)
ORIGIN = http://127.0.0.1:5020
#: API version of the LUNA Image Store (for task result)
API_VERSION = 1
#: *bucket* name, where samples will be stored
BUCKET = task-result


[LUNA_IMAGE_STORE_TASK_RESULT_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 60


[LUNA_TASKS_LOAD_EXTERNAL_ARCHIVE_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 100
#: SOCK READ TIMEOUT
SOCK_READ = 100


[LUNA_FACES_ADDRESS]

#: LUNA Faces protocol, IP address, and port
ORIGIN = http://127.0.0.1:5030
#: API version of the LUNA Faces
API_VERSION = 3


[LUNA_FACES_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 60


[LUNA_PYTHON_MATCHER_ADDRESS]

#: LUNA Python Matcher protocol, IP address, and port
ORIGIN = http://127.0.0.1:5100
#: API version of the LUNA Python Matcher
API_VERSION = 1


[LUNA_MATCHER_PROXY_ADDRESS]

#: LUNA Python Matcher Proxy protocol, IP address, and port
ORIGIN = http://127.0.0.1:5110
#: API version of the LUNA Python Matcher Proxy
API_VERSION = 1


[LUNA_TASKS_TO_MATCHER_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 30
#: REQUEST TIMEOUT (total)
REQUEST =  60


[ADDITIONAL_SERVICES_USAGE]

#: [integer] support LUNA Events service as a data source for tasks
LUNA_EVENTS = 1
#: [integer] use LUNA Python Matcher Proxy instead of LUNA Python Matcher
LUNA_MATCHER_PROXY = 1
#: [integer] use Luna Image store for images/samples storages
LUNA_IMAGE_STORE = 1
#: [integer] use Luna Handlers for estimator and re-extractor tasks
LUNA_HANDLERS = 1


[LUNA_EVENTS_ADDRESS]

#: LUNA Events protocol, IP address, and port
ORIGIN = http://127.0.0.1:5040
#: API version of the LUNA Events
API_VERSION = 2


[LUNA_EVENTS_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 60


[LUNA_HANDLERS_ADDRESS]

#: LUNA Handlers protocol, IP address, and port
ORIGIN = http://127.0.0.1:5090
#: API version of the LUNA Handlers
API_VERSION = 1


[LUNA_HANDLERS_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 60


[PLATFORM_LIMITS.CROSS_MATCH]

#: [integer] short array filter limit. The maximum number of items provided in filters. The limit is set for part of the filters only.
SHORT_ARRAY_FILTER_LIMIT = 1000
#: [integer] array filter limit. The maximum number of items provided in filters. The limit is set for part of the filters only.
ARRAY_FILTER_LIMIT = 20000
#: [integer] The maximum number of matching results for the candidates' list returned in the response.
RESULT_CANDIDATE_LIMIT = 20000
#: [integer] general limit
GENERAL_LIMIT = 100000


[EXTERNAL_LUNA_API_ADDRESS]

#: external luna-api origin
ORIGIN = http://127.0.0.1:5000
#: external luna-api origin
API_VERSION = 6

[LUNA_TASKS_HTTP_SETTINGS]

#: [integer] service request timeout (in seconds)
REQUEST_TIMEOUT = 60
#: [integer] service response timeout (in seconds)
RESPONSE_TIMEOUT = 600
#: [integer] max request size (in bytes), default 1gb
REQUEST_MAX_SIZE = 1073741824
#: [integer] service http keep-alive timeout (in seconds)
KEEP_ALIVE_TIMEOUT = 15


[TASKS_REDIS_DB_ADDRESS]

#: redis host for the service subscription
HOST = 127.0.0.1
#: redis port for the service subscription
PORT = 6379
#: [string] redis username for the service subscription
USER =
#: redis password for the service subscription
PASSWORD =
#: [integer] Redis database number (0-15)
NUMBER = 0


[TASKS_REDIS_DB_ADDRESS.SENTINEL]
#: [string] sentinel master name
MASTER_NAME = luna_tasks
#: [list(string)] list of Redis Sentinel listeners (example [127.0.0.1:26379, 10.0.4.1:26379]).
#: By setting this parameter, application will query Redis parameters (host and port) from sentinels
SENTINELS = []
#: [string] sentinel username
USER =
#: [string] sentinel password
PASSWORD =


[LUNA_SERVICE_METRICS]

#: [int] whether to enable /metrics path and statistics collection
ENABLED = 0
# [string] statistics data format (prometheus is the only option yet)
METRICS_FORMAT = prometheus


[LUNA_SERVICE_METRICS.EXTRA_LABELS]


[LUNA_LAMBDA_TASKS_UNIT_TIMEOUTS]

#: CONNECT TIMEOUT
CONNECT = 20
#: REQUEST TIMEOUT (total)
REQUEST = 60


[LUNA_IMAGE_STORE_OBJECTS_ADDRESS]

#: LUNA Image Store protocol, IP address, and port (for objects)
ORIGIN = http://127.0.0.1:5020
#: API version of the LUNA Image Store (for objects)
API_VERSION = 1
#: *bucket* name, where samples will be stored
BUCKET = visionlabs-objects


[OTHER]

#: storage time - "LOCAL" or "UTC"
STORAGE_TIME = LOCAL

#: list of active plugins
LUNA_TASKS_ACTIVE_PLUGINS = []
#: maximum errors count per task to store
MAX_ERROR_COUNT_PER_TASK = 100_000
#: Concurrency for hard requests to luna-faces
TASKS_TO_FACES_REQUESTS_CONCURRENCY = 5
#: Concurrency for requests to luna-image-store
TASKS_TO_IMAGE_STORE_REQUESTS_CONCURRENCY = 100
#: Concurrency for extract requests to luna-handlers
TASKS_TO_HANDLERS_CONCURRENCY = 8
#: Concurrency for requests to python-matcher
TASKS_TO_MATCHER_CONCURRENCY = 2

Extended administrator configuration¶

Module config with global variables.

configs.config.ARCHIVE_MAX_SIZE = 137438953472¶: Estimator archive max size (in bytes), 100GB by default

configs.config.CLUSTERIZER_MAX_RETRIES = 5¶: Retries count for match data

configs.config.COUNT_GC_TASK_PARTS = 16¶: Garbage Collecting Task garbage collector subtasks count per task

configs.config.DEFAULT_CLUSTERING_THRESHOLD = 0.81¶: Clustering Task default clustering threshold

configs.config.DEFAULT_CROSSMATCHING_LIMIT = 5¶: Cross matching Task default cross matching limit

configs.config.DEFAULT_CROSSMATCHING_THRESHOLD = 0.0¶: default cross matching threshold

configs.config.DEFAULT_GEO_POSITION_DELTA = 0.01¶: default filter longitude/latitude delta in degrees

configs.config.DEFAULT_PORT = 5050¶: Default ports

configs.config.EXPORTER_MAX_RETRIES = 3¶: Exporter Task Retries count for data downloading

configs.config.EXPORTER_TIME_BETWEEN_RETRY_ATTEMPTS = 1¶: Time between retry attempts

configs.config.EXTRACT_SUBTASK_DESCRIPTORS_COUNT = 1000¶: Additional Extract Task batch size of attributes to extract per subtask

configs.config.FTP_CONNECTION_TIMEOUT = 10¶: Estimator FTP server connection timeout

configs.config.MSGPACK_BUFFER_SIZE = 524288000¶: Size of the buffer for msgpack stream (in bytes), 500MB by default

configs.config.SAMBA_CONNECTION_TIMEOUT = 10¶: Estimator SAMBA server connection timeout

configs.config.SCHEDULE_CHECK_PERIOD = 0.5¶: schedule check period

configs.config.STREAM_CHUNK_SIZE = 104857600¶: Streaming chunk size (in bytes), 100MB by default

configs.config.TASK_SEPARATION_TIMEOUT = 60¶: waiting for task to subtasks separation timeout

configs.config.TOP_COUNT = 5¶: ROC Task Max top count to calculate getting top probability