Services launch#

This section gives examples for:

Databases tables creation
Buckets creation
Launching of containers

The commands are given in the order they must be performed.

It is recommended to launch containers one by one and wait for the container status to become "up" (use the docker ps command).

See the "Docker commands" section for details about working with containers.

General container launching parameters#

When launching a Docker container for a LUNA PLATFORM service you should specify additional parameters required for the service launching.

The parameters specific for a particular container are described in the section about this container launching.

All the parameters given in the service launching example are required for proper service launching and utilization.

When launching containers, you should specify a link to the image required for the container launching. This image will be downloaded from the VisionLabs registry. Before that, you should login to the registry.

Enter login .

docker login dockerhub.visionlabs.ru --username <username>

After running the command, you will be prompted for a password. Enter password.

The login and password are received from VisionLabs.

In the docker login command, you can enter the login and password at the same time, but this does not guarantee security because the password can be seen in the command history.

Description of launch parameters#

Common parameters for launching containers are described in this section.

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=<Port_of_the_launched_service> \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<service>:/srv/logs/ \
--name=<service_container_name> \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/<service-name>:<version>

docker run - the command for running the selected image as a new container.

dockerhub.visionlabs.ru/luna/<service-name>:<version> - the parameter specifies the image required for the container launching.

Links to download the container images you need are available in the description of the corresponding container launching.

--network=host - the parameter specifies that a network is not simulated and the server network is used. If you need to change the port for third-party party containers, you should change this string to -p 5440:5432. Where the first port 5440 is the local port and 5432 is the port used inside the container. The example is given for PostgreSQL.

--env= - the parameter specifies the environment variables required to run a container. The following general values are specified:

CONFIGURATOR_HOST=127.0.0.1 - the host where the Configurator service is running. The localhost is set in the case when the container is launched on the same server with the Configurator service.
CONFIGURATOR_PORT=5070 - the port where the Configurator service is listening. The 5070 port is used by default.
PORT=<Port_of_the_service> - the port where the service will listen.
WORKER_COUNT - specifies the number of worker processes for the service.
RELOAD_CONFIG enables auto-reload of configurations for the service when set to "1". See the "Automatic configurations reload" section in the LUNA PLATFORM 5 administrator manual.
RELOAD_CONFIG_INTERVAL specifies the configurations check period (10 seconds by default). See the "Automatic configurations reload" section in the LUNA PLATFORM 5 administrator manual.

--name=<service_container_name> - the parameter specifies the name of the launched container. The name must be unique. If there is a container with the same name, an error will occur.

--restart=always - the parameter specifies a restart policy. The daemon will always restart the container regardless of the exit status.

--detach=true - run the container in the background mode.

-v - the volume parameter enables you to mount the content of a server folder into a volume in the container. Thus their contents will synchronize. The following general data is mounted:

/etc/localtime:/etc/localtime:ro - sets the current time zone used by the system in the container.
/tmp/logs/<service>:/srv/logs/ - enables copying of the folder with service logs to your server /tmp/logs/<service> directory. You can change the directory where the logs will be saved according to your needs.

Description of database creation parameters#

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<service>:/srv/logs/ \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/<service-name>:<version> \
python3 ./base_scripts/db_create.py --luna-config http://localhost:5070/1

The following parameters are used when containers are launched for database migration and creation procedures.

--rm - the parameter specifies if the container is deleted after all the specified scripts finish processing

python3 ./base_scripts/db_create.py - the parameter specifies Python version and a script db_create.py launched in the container. The script is used for the database structure creation. --luna-config http://localhost:5070/1 - the parameter specifies where the launched script should receive configurations. By default, the service requests configurations from the Configurator service.

Stop LUNA PLATFORM 4 containers#

Use the following command to stop and delete the default LUNA PLATFORM 4.5.3 containers.

You should remove all the containers of LUNA PLATFORM services if you have started a different number of containers or they have other names.

docker container rm -f admin api sender tasks tasks-worker events luna-matcher luna-detector luna-extractor licenses faces_2 faces_1 image-store configurator influxdb

Note! It is also recommended to remove NGINX or edit its settings, as this instruction does not involve scaling using NGINX.

docker container rm -f nginx

Monitoring configuration#

InfluxDB OSS 2#

InfluxDB 2.0.8-alpine is required for LP monitoring purpose with data visualization using Grafana (for more information, see the "Monitoring" section in the administrator’s manual).

Note! If you already have InfluxDB 2.0.8-alpine installed, skip this step.

Run InfluxDB container#

Use the docker run command with these parameters:

docker run \
-e DOCKER_INFLUXDB_INIT_MODE=setup \
-e DOCKER_INFLUXDB_INIT_BUCKET=luna_monitoring \
-e DOCKER_INFLUXDB_INIT_USERNAME=luna \
-e DOCKER_INFLUXDB_INIT_PASSWORD=password \
-e DOCKER_INFLUXDB_INIT_ORG=luna \
-e DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=kofqt4Pfqjn6o0RBtMDQqVoJLgHoxxDU mmhiAZ7JS6VmEnrqZXQhxDhad8AX9tmiJH6CjM7Y1U8p5eSEocGzIA== \
-v /etc/localtime:/etc/localtime:ro \
-v /var/lib/luna/current/example-docker/influx:/var/lib/influxdb2 \
--restart=always \
--detach=true \
--network=host \
--name influxdb \
dockerhub.visionlabs.ru/luna/influxdb:2.0.8-alpine

Grafana#

Grafana is required to visualize LP monitoring. You can use it to create dashboards of the LP state (for more information, see the "LUNA Dashboards" section in the administrator's manual).

To work with Grafana you need to use InfluxDB version 2.

Note! If you already have Grafana installed or are not going to use it, skip this step.

Run Grafana container#

Use the docker run command with these parameters to run Grafana:

docker run \
--restart=always \
--detach=true \
--network=host \
--name=grafana \
-v /etc/localtime:/etc/localtime:ro \
dockerhub.visionlabs.ru/luna/luna-dashboards:v.0.0.4

Create dashboards inside the container using the following command:

docker exec -it grafana create_dashboards

Use "http://IP_ADDRESS:3000" to go to the Grafana web interface when the Grafana and Influx containers are running.

Monitoring parameters configuration#

Enable monitoring for services using Influx DB.

Configure access to Influx DB in configurations of Python services in the INFLUX_MONITORING section.

Monitoring for the services is enabled by default.

Service names in Configurator

Service	Service name in Configurator
API	luna-api
Faces	luna-faces
Image Store	luna-image-store
Accounts	luna-accounts
Tasks	luna-tasks
Events	luna-events
Sender	luna-sender
Admin	luna-admin
Licenses	luna-licenses
Python Matcher	luna-python-matcher
Handlers	luna-handlers
Backport 3	luna-backport3
Backport 4	luna-backport4

The Configurator service itself is configured in the configuration file only:

/var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf

The following parameters can be specified for InfluxDB:

SEND_DATA_FOR_MONITORING - enables monitoring for the service.

MONITORING_USE_SSL - enables HTTPS protocol usage for connection to InfluxDB (0 – do not use, 1 – use).

MONITORING_FLUSHING_PERIOD – the frequency of sending monitoring data to InfluxDB.

MONITORING_PORT - InfluxDB port.

MONITORING_HOST - InfluxDB IP address.

MONITORING_ORGANIZATION - Organization name.

MONITORING_TOKEN - Token received after registration.

MONITORING_BUCKET_NAME - Bucket name.

Configurator#

Optional services usage#

The listed below services are not mandatory for LP. You can disable them if their functionality is not required for your tasks.

Use the "ADDITIONAL_SERVICES_USAGE" section in the API service settings in the Configurator service to disable unnecessary services.

Optional services

Service	Dependent services
Events	API, Admin, Handlers, Tasks.
Tasks	API, Admin, Handlers
Sender	API, Admin, Handlers
Python Matcher Proxy (disabled by default)	API, Admin, Handlers, Tasks.

The API service does not depend on the state of the Admin, Backport3, Backport4, User Interface 3 and User Interface 4 services. These services do not need to be changed in the Configurator service, it is enough just not to launch them.

You can use the dump file provided in the distribution package to enable/disable services from the "Optional services" table before Configurator launch.

vi /var/lib/luna/current/extras/conf/platform_settings.json

You must restart a dependent service after changing its parameters if the service was already launched.

Migrate database data#

It is recommended to migrate database data from PostgreSQL 9.6 to PostgreSQL 12.

Data migration is not described in this manual.

It is not required to migrate Configurator database as it should be created from scratch and filled in manually.

Database recreation#

Use the following commands to delete and create the Configurator service database.

Delete the old database:

docker exec -it postgres psql -U luna -c "DROP DATABASE luna_configurator;"

Create the new database:

docker exec -it postgres psql -U luna -c "CREATE DATABASE luna_configurator;"

Grant privileges to the database user:

docker exec -it postgres psql -U luna -c "GRANT ALL PRIVILEGES ON DATABASE luna_configurator TO luna;"

Allow user to authorize in the DB:

docker exec -it postgres psql -U luna -c "ALTER ROLE luna WITH LOGIN;"

Configurator DB tables creation#

Use the docker run command with these parameters to create the Configurator database tables.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf:/srv/luna_configurator/configs/config.conf \
-v /var/lib/luna/current/extras/conf/platform_settings.json:/srv/luna_configurator/used_dumps/platform_settings.json \
--network=host \
-v /tmp/logs/configurator:/srv/logs \
--rm \
--entrypoint bash \
dockerhub.visionlabs.ru/luna/luna-configurator:v.2.0.74 \
-c "python3 ./base_scripts/db_create.py; cd /srv/luna_configurator/configs/configs/; python3 -m configs.migrate --config /srv/luna_configurator/configs/config.conf head; cd /srv; python3 ./base_scripts/db_create.py --dump-file /srv/luna_configurator/used_dumps/platform_settings.json"

/var/lib/luna/current/extras/conf/platform_settings.json - enables you to specify the path to the dump file with LP configurations.

./base_scripts/db_create.py; - creates database structure.

python3 -m configs.migrate head; - performs settings migrations in Configurator DB and sets revision for migration. The revision will be required during the upgrade to the new LP5 build.

--dump-file /srv/luna_configurator/used_dumps/platform_settings.json - updates settings in the Configurator DB with values from the provided file.

Run Configurator container#

Use the docker run command with these parameters to launch Configurator:

docker run \
--env=PORT=5070 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf:/srv/luna_configurator/configs/config.conf \
-v /tmp/logs/configurator:/srv/logs \
--name=luna-configurator \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-configurator:v.2.0.74

Configuring InfluxDB#

If you are going to use InfluxDB OSS 2, then you need to update the monitoring settings in Configurator service.

There are the following settings for InfluxDB OSS 2:

"send_data_for_monitoring": 1,
"use_ssl": 0,
"flushing_period": 1,
"host": "127.0.0.1",
"port": 8086,
"organization": "<ORGANIZATION_NAME>",
"token": "<TOKEN>",
"bucket": "<BUCKET_NAME>",
"version": <DB_VERSION>

You can update InfluxDB settings in the Configurator service by following these steps:

Open /var/lib/luna/current/extras/conf/influx2.json.
Set required data in "organization", "token", "bucket", "version" fields.
Save changes.
Copy the file to the influxDB container:

docker cp /var/lib/luna/current/extras/conf/influx2.json luna-configurator:/srv/

Update settings in the Configurator.

docker exec -it luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/influx2.json

You can also manually update settings in the Configurator service user interface.

Settings changing#

There are three general ways to change settings stored in the Configurator DB:

using Configurator GUI,
using Configurator API requests,
using dump file.

Settings can be changed after the Configurator service was launched.

The settings of Configurator itself are stored in the "luna_configurator_postgres.conf" file that is downloaded to the Configurator container during its launch.

Configurator Interface

You can enter the configurator GUI and change setting. By default, the following address on localhost are used: <Configurator_server_address>:5070.

Configurator API

You can use Configurator API to update settings. See "ConfiguratorReferenceManual.html".

Dump file

You can receive a dump file with all the LP services settings. Use one of the following commands:

wget -O /var/lib/luna/settings_dump.json 127.0.0.1:5070/1/dump

or

curl 127.0.0.1:5070/1/dump > /var/lib/luna/settings_dump.json

You should specify correct Configurator service address and port.

You should delete the "limitations" section from the file. You will not be able to apply your updated dump file if the section remains.

    "limitations":[
      ...
    ],

Edit parameters in the "settings" section.

    "settings":[
      ...
    ],

Copy the dump file to your Configurator container.

docker cp /var/lib/luna/settings_dump.json luna-configurator:/srv/

Apply the file:

docker exec luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/settings_dump.json

Enable logging to server directory#

If you want to save service logs in a single directory on your server, you should update services configurations.

Logs are not saved to the server directory by default.

To enable logs saving to the server you should:

create a directory for logs (see Create logs directory)
enable logs saving and change logs directory in the Configurator service

When logging to file is enabled and configured, service logs are written to the "/srv/logs" directory in the service container and to the specified service directory on the server (e. g., "/tmp/logs").

For more information about the LUNA PLATFORM logging system, see the "Logging" section in the administrator manual.

Change logs directory manually before Configurator launch#

The settings of the Configurator service are stored in file:

/var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf

You should change its logging parameters in this file before starting Configurator or restart Configurator after their change.

Change this logging parameter FOLDER_WITH_LOGS = ./ to FOLDER_WITH_LOGS = /srv/logs

Set the "log_to_file" parameter to "True" to enable logging to files.

Change logs directory manually after Configurator launch#

Specify "folder_with_logs" parameter value to "/srv/logs" for all the services where you want to write logs. The docker run commands are already configured for saving logs to your server to this directory. After launching the service container, the service will write logs to the "/srv/logs"directory.

Set the "log_to_file" parameter to "True" to enable logging to files.

Change logs directory using dump file#

You can use the dump file provided in the distribution package to update logging settings.

/var/lib/luna/current/extras/conf/logging.json

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /var/lib/luna/current/extras/conf/logging.json:/srv/luna_configurator/used_limitations/logging.json \
--network=host \
-v /tmp/logs/configurator:/srv/logs \
--rm \
--entrypoint=python3 \
dockerhub.visionlabs.ru/luna/luna-configurator:v.2.0.74 \
./base_scripts/db_create.py --dump-file /srv/luna_configurator/used_limitations/logging.json

Image Store#

Samples migrations#

To migrate from LUNA PLATFORM 4 you can assign an account to all the samples in your Image Store bucket. Otherwise all the objects created for the LUNA PLATFORM 4 can be accessed within any account, and also without any account.

You can skip the following script execution if the samples should not be linked to any account and can be accessed from any account.

Create a backup of all the samples buckets before launching the following script.

You should launch the "migrate_4_to_5.py" script to update account information for the stored samples.

Set user permissions:

chown -R 1001:0 /var/lib/luna/luna_v.4.5.4/example-docker-compose/image_store/

You should use existing buckets during the samples migration script execution. Set up actual storage configurations for the Image Store service in the Configurator service user interface.

docker run \
--rm -t \
-v /tmp/logs/image-store:/srv/logs \
-v /var/lib/luna/luna_v.4.5.4/example-docker-compose/image_store/:/srv/luna_image_store/local_storage/ \
--network=host \
--entrypoint bash dockerhub.visionlabs.ru/luna/luna-image-store:v.3.6.17 \
-c "python3 ./base_scripts/accounting/migrate_4_to_5.py --account_id=<account_id> --bucket=visionlabs-samples"

-v /var/lib/luna/luna_v.4.5.4/example-docker-compose/image_store/ - you should specify your current path to the local_storage of the Image Store samples.

<account_id> - you should specify the account ID which will have access to all the samples from the specified bucket (visionlabs-samples).

You should specify the current directory with samples buckets ("/var/lib/luna/luna_v.4.5.4/luna-image-store/luna_image_store/local_storage/" by default) instead of "/var/lib/luna/current/example-docker/image_store/" during the Image Store container launching: -v /var/lib/luna/current/example-docker/image_store/:/srv/local_storage/ See section Image Store container launch.

Image Store container launch#

Use the following command to launch the Image Store service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5020 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /var/lib/luna/current/example-docker/image_store/:/srv/local_storage/ \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/image-store:/srv/logs \
--name=luna-image-store \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-image-store:v.3.6.17

-v /var/lib/luna/current/example-docker/image_store/:/srv/local_storage/ - the data from the specified folder is added to the Docker container when it is launched. All the data from the specified Docker container folder is saved to this directory.

If you already have a directory with LP buckets you should specify it instead of /var/lib/luna/current/example-docker/image_store/.

Buckets creation#

Buckets are required to store data in Image Store. The Image Store service should be launched before the commands execution.

When upgrading from the previous version, it is recommended to launch the bucket creation commands one more time. Hence you make sure that all the required buckets were created.

If the "error_code":13006,"desc":"Unique constraint error","detail":"Bucket with name 'task-result' already exist"} error appears during launching of the listed above commands, the bucket is already created.

There are two ways to create buckets in LP.

You can run the listed above scripts to create buckets.

Run this script to create general buckets:

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/api:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-api:v.6.9.0 \
python3 ./base_scripts/lis_bucket_create.py -ii --luna-config http://localhost:5070/1

If you are going to use the Tasks service, use the following command to additionally create the "task-result" in the Image Store service:

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/tasks:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-tasks:v.3.13.0 \
python3 ./base_scripts/lis_bucket_create.py -ii --luna-config http://localhost:5070/1

If you are going to use the portraits, use the following command to additionally create the "portraits".

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/api:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-backport3:v.0.6.7 \
python3 ./base_scripts/lis_bucket_create.py -ii --luna-config http://localhost:5070/1

You can use direct requests to create required buckets.

The curl utility is required for the following requests.

The "visionlabs-samples" bucket is used for face samples storage. The bucket is required for LP utilization.

curl -X POST http://127.0.0.1:5020/1/buckets?bucket=visionlabs-samples

The "portraits" bucket is used for portraits storage. The bucket is required for Backport 3 utilization.

curl -X POST http://127.0.0.1:5020/1/buckets?bucket=portraits

The "visionlabs-bodies-samples" bucket is used for human bodies samples storage. The bucket is required for LP utilization.

curl -X POST http://127.0.0.1:5020/1/buckets?bucket=visionlabs-bodies-samples

The "visionlabs-image-origin" bucket is used for source images storage. The bucket is required for LP utilization.

curl -X POST http://127.0.0.1:5020/1/buckets?bucket=visionlabs-image-origin

The "visionlabs-objects" bucket is used for objects storage. The bucket is required for LP utilization.

curl -X POST http://127.0.0.1:5020/1/buckets?bucket=visionlabs-objects

The "task-result" bucket for the Tasks service. Do not use it if you are not going to use the Tasks service.

curl -X POST http://127.0.0.1:5020/1/buckets?bucket=task-result

Accounts#

Admin DB transformation#

Note. Follow the steps below only if you have used the Admin service before. Otherwise, skip this section.

By default, the Accounts service creates its own luna_accounts database, but if the Admin service was previously used, then the luna_admin should must be transformed to work with the Accounts service. To do this, you need to change the default luna_accounts database used by the Accounts service to luna_admin in the Configurator and run the data migration script.

Admin DB migration#

Run migration script to update the structure of the Admin database and transform it to work with the Accounts service.

Note that after the migration, the database will be named luna_admin, but will be used exclusively by the Accounts service.

It is recommended to create the back up of your database before applying any changes.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/accounts:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-accounts:v.0.0.17 \
alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

Accounts DB tables creation#

Note. Perform the steps below only if you are launching the Accounts service for the first time and have not used the Admin service before. If you used the Admin service before, skip this section and make sure you have completed the steps in the "Admin DB transformation" section.

Use the following command to create Accounts DB tables:

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/accounts:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-accounts:v.0.0.17 \
python3 ./base_scripts/db_create.py --luna-config http://localhost:5070/1

Accounts container launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5170 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/accounts:/srv/logs \
--name=luna-accounts \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-accounts:v.0.0.17

Licenses#

Licenses container launch#

Make sure that you have specified the license server address in the "hasp_30147.ini" file. See section "Specify license server address for LP".

Add the access right for the "luna" user to the "hasp_redirect" directory.

chown -R 1001:0 /var/lib/luna/current/example-docker/hasp_redirect/

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5120 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/licenses:/srv/logs \
-v /var/lib/luna/current/example-docker/hasp_redirect/hasp_30147.ini:/home/luna/.hasplm/hasp_30147.ini \
--name=luna-licenses \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-licenses:v.0.3.81

Faces#

Faces database migration#

You need to execute migration scripts to update your Faces database structure.

It is recommended to create the back up of your database before applying any changes.

Run the following command to perform the Faces DB migration.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/faces:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-faces:v.4.6.5 \
alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

Create VLMatch function for Faces DB#

The Faces service requires the VLMatch additional function to be added to the utilized database. LUNA PLATFORM cannot perform matching calculations without this function. The VLMatch function can be added to the PostgreSQL or Oracle database.

The VLMatch library is compiled for your particular database version.

Note! Do not use the library built for another version of DB. For example, the library build for the PostgreSQL of version 12 cannot be used for the PostgreSQL of version 9.6.

This section describes the function creation for PostgreSQL.

The instruction for the Oracle DB is given in the "VLMatch for Oracle" section.

Build VLMatch for PostgreSQL#

You can find all the required files for the VLMatch user-defined extension (UDx) compilation in the following directory:

/var/lib/luna/current/extras/VLMatch/postgres/

The following instruction describes installation for PostgreSQL 12.

For VLMatch UDx function compilation one needs to:

Make sure, that PostgreSQL of the required version is installed and launched.
Install the required PostgreSQL development environment. You can find more information on the official web site.
The llvm-toolset-7-clang is required for postgresql12-devel. Install it from the centos-release-scl-rh repository.

yum -y install centos-release-scl-rh

yum -y --enablerepo=centos-sclo-rh-testing install llvm-toolset-7-clang

Install epel-release for access to extended package repository

yum -y install epel-release

Install the development environment.

yum -y install postgresql12 postgresql12-server postgresql12-devel

Install the gcc-c++ package. The package version 4.8 or higher is required.

yum -y install gcc-c++.x86_64

Install CMAKE. The version 3.5 or higher is required.
Open the make.sh script using a text editor. It includes paths to the currently used PostgreSQL version. Change the following values (if necessary):

SDK_HOME specifies the path to PostgreSQL home directory. The default value is /usr/pgsql-12/include/server;

LIB_ROOT specifies the path to PostgreSQL library root directory. The default value is /usr/pgsql-12/lib.

Go to the make.sh script directory and run it:

cd /var/lib/luna/current/extras/VLMatch/postgres/

chmod +x make.sh

./make.sh

Add VLMatch function to Faces database#

The VLMatch function should be applied to the PostgreSQL DB.

Define the function inside the Faces database:

sudo -u postgres -h 127.0.0.1 -- psql -d luna_faces -c "CREATE FUNCTION VLMatch(bytea, bytea, int) RETURNS float8 AS 'VLMatchSource.so', 'VLMatch' LANGUAGE C PARALLEL SAFE;"

Test function by sending re following request to the service database:

sudo -u postgres -h 127.0.0.1 -- psql -d luna_faces -c "SELECT VLMatch('\x1234567890123456789012345678901234567890123456789012345678901234'::bytea, '\x0123456789012345678901234567890123456789012345678901234567890123'::bytea, 32);"

The result returned by the database must be "0.4765625".

Faces container launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5030 \
--env=WORKER_COUNT=2 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/faces:/srv/logs \
--name=luna-faces \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-faces:v.4.6.5

Events#

Create VLMatch function for Events DB#

The Events service requires the VLMatch additional function to be added to the utilized database. LUNA PLATFORM cannot perform matching calculations without this function. The VLMatch function can be added to the PostgreSQL.

The VLMatch library is compiled for your particular database version.

Note! Do not use the library built for another version of DB. For example, the library build for the PostgreSQL of version 12 cannot be used for the PostgreSQL of version 9.6.

This section describes the function creation for PostgreSQL. If you use the PostgreSQL database, you have already created and moved the created library during the Faces service launch. See section "Build VLMatch UDx".

Add VLMatch function to Events database#

The VLMatch function should be applied to the PostgreSQL DB.

Define the function inside the Events database:

sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "CREATE FUNCTION VLMatch(bytea, bytea, int) RETURNS float8 AS 'VLMatchSource.so', 'VLMatch' LANGUAGE C PARALLEL SAFE;"

Test function within call:

sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "SELECT VLMatch('\x1234567890123456789012345678901234567890123456789012345678901234'::bytea, '\x0123456789012345678901234567890123456789012345678901234567890123'::bytea, 32);"

The result returned by the database must be "0.4765625".

PostGIS installation for Events#

The Events service requires PostGIS for working with coordinates.

This instruction describes the PostGIS installation for PostgreSQL 12 database. The PostGIS version depends on the PostgreSQL version.

Install epel-release for access to extended package repository (necessary for RabbitMQ and other dependencies)

yum -y install epel-release

Install PostGIS:

yum -y install postgis25_12

Activate PostGIS in your database:

sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "CREATE EXTENSION postgis;";

See additional information about PostGIS on its website:

https://postgis.net/

Events database migration#

You need to execute migration scripts to update your Events database structure.

It is recommended to create the back up of your database before applying any changes.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/events:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-events:v.4.5.1 \
alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

Events container launch#

Note. If you are not going to use the Events service, do not launch this container and disable the service utilization in Configurator. See section "Optional services usage".

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5040 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/events:/srv/logs \
--name=luna-events \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-events:v.4.5.1

Matcher services#

For matching tasks, you can use either only the Python Matcher service, or additionally use the Python Matcher Proxy service, which redirects matching requests to either the Python Matcher service or matching plugins. This section describes how to use Python Matcher without Python Matcher Proxy.

Note. You need to use the Python Matcher Proxy service only if you are going to use matching plugins. Using Python Matcher Proxy and running the corresponding docker container are described in the "Use Python Matcher with Python Matcher Proxy" section.

See the description and usage of matching plugins in the administrator manual.

Use Python Matcher without Python Matcher Proxy#

The Python Matcher service with matching by the Faces DB is enabled by default after launching.

The Python Matcher service with matching by the Events is also enabled by default. You can disable it by specifying USE_LUNA_EVENTS = 0 in the ADDITIONAL_SERVICES_USAGE section. Thus, the Events service will not be used for LUNA PLATFORM.

The Python Matcher that matches using the matcher library is enabled when CACHE_ENABLED is set to "true" in the DESCRIPTORS_CACHE setting.

Note. A single image is downloaded for the Python Matcher service and the Python Matcher Proxy service.

Python Matcher container launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5100 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/python-matcher:/srv/logs \
--name=luna-python-matcher \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-python-matcher:v.1.2.6

Handlers#

Change the used database for Handlers#

The database used for the Handlers service in LUNA PLATFORM 5 is the same database that was used for the API service in LUNA PLATFORM 4.

You should change the database used for the Handlers service to the "luna_api" database in the Configurator service.

Go to the Configurator interface (http://<server_address>:5070 by default);
Specify the LUNA_HANDLERS_DB in the "Setting name" field of the filter and press [Apply filters] or select "luna-handlers" in the "Service name field" and find the setting there;
Set the "db_name": "luna_api" parameter in the value field.
Press [Save].

Changing the model of the neural network of bodies#

Starting from version 5.34.0, the model of the neural network for extracting body descriptors is changed by default from 104 to 107. In addition, the 104th and 106th neural network models are no longer contained in the Handlers container.

You must do one of the following operations before starting the Handlers container:

Operation 1. Manually change the setting value "DEFAULT_HUMAN_DESCRIPTOR_VERSION" from "104" or "106" to "107". For this:

go to the Configurator service interface http://<configurator_server_ip>:5070/
enter the value "DEFAULT_HUMAN_DESCRIPTOR_VERSION" in the "Setting name" field and click "Apply Filters"
go to the found field, specify the value "107" and click the "Save" button.

After changing the version of the neural network for extracting body descriptors, you must run the "Additional extraction" task after starting the Admin service (see the "Launch re-extraction task" section in the administrator manual. Otherwise, search and matching by old descriptors will not be available.

Operation 2. Disable the use of a neural network to extract a body descriptor using the following command when starting the Handlers container:

docker run \
...
--env=EXTEND_CMD="--enable-body-descriptor-estimator=0" \
...

This command does not prohibit the estimation of body parameters (upper/lower body, sleeve length, presence of a backpack, etc.). See "Enable/disable several estimators and detectors" section of the administrator manual for details.

Operation 3. Request the 104th model of the neural network to extract body descriptors from VisionLabs and transfer it to the Handlers container according to the instructions described in the section "Use non-delivery neural network model" of the administrator manual.

If one of the above operations is not performed, then starting the Handlers service will fail.

Handlers database migration#

You need to execute migration scripts to update your Handlers database structure.

It is recommended to create the back up of your database before applying any changes.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/handlers:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-handlers:v.2.13.0 \
alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

Handlers container launch#

You can run the Handlers service utilizing CPU (set by default) or GPU.

By default, the Handlers service is launched with all estimators and detectors enabled. If necessary, you can disable the use of some estimators or detectors when launching the Handlers container. Disabling unnecessary estimators enables you to save RAM or GPU memory, since when the Handlers service launches, the possibility of performing these estimates is checked and neural networks are loaded into memory. If you disable the estimator or detector, you can also remove its neural network from the Handlers container. See the "Enable/disable several estimators and detectors" section of the administrator manual for more information.

Run the Handlers service using one of the following commands according to the utilized processing unit.

Run Handlers utilizing CPU#

Use the following command to launch the Handlers service using CPU:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5090 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/handlers:/srv/logs \
--name=luna-handlers \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-handlers:v.2.13.0

Run Handlers utilizing GPU#

The Handlers service does not utilize GPU by default. If you are going to use the GPU, then you should enable its use for the Handlers service in the Configurator service.

If you need to use the GPU for all estimators and detectors at once, then you need to use the global_device_class parameter in the "LUNA_HANDLERS_RUNTIME_SETTINGS" section. All estimators and detectors will use the value of this parameter if the device_class parameter of their settings like "LUNA_HANDLERS__SETTINGS.runtime_settings" is set to global (by default for all estimators and detectors).

If you need to use the GPU for a specific estimator or detector, then you need to use the device_class parameter in sections like "LUNA_HANDLERS__SETTINGS.runtime_settings".

See section "Calculations using GPU" for additional requirements for GPU utilization.

Use the following command to launch the Handlers service using GPU:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5090 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--gpus device=0 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/handlers:/srv/logs \
--name=luna-handlers \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-handlers:v.2.13.0

--gpus device=0 - the parameter specifies the used GPU device and enables GPU utilization. A single GPU can be utilized per Handlers instance. Multiple GPU utilization per instance is not available.

Tasks#

Note. If you are not going to use the Tasks service, do not launch the Tasks container and the Tasks Worker container. Disable the service utilization in Configurator. See section "Optional services usage".

Tasks database migration#

You need to execute migration scripts to update your Tasks database structure.

It is recommended to create the back up of your database before applying any changes.

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/tasks:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-tasks:v.3.13.0 \
alembic -x luna-config=http://127.0.0.1:5070/1 upgrade head

Tasks and Tasks worker containers launch#

Luna Tasks Docker image includes the Tasks service and the Tasks worker. They both must be launched.

The "task-result" bucket should be created for the Tasks service before the service launch. The buckets creation is described in the "Buckets creation".

If it is necessary to use the Estimator task using a network disk, then you should first mount the directory with images from the network disk into special directories of Tasks and Tasks Worker containers. See the "Estimator task" section in the administrator manual for details.

Tasks Worker launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5051 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--env=SERVICE_TYPE="tasks_worker" \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/tasks-worker:/srv/logs \
--name=luna-tasks-worker \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-tasks:v.3.13.0

Tasks launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5050 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/tasks:/srv/logs \
--name=luna-tasks \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-tasks:v.3.13.0

Sender#

Sender container launch#

Note. If you are not going to use the Sender service, do not launch this container and disable the service utilization in Configurator. See section "Optional services usage".

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5080 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/sender:/srv/logs \
--name=luna-sender \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-sender:v.2.5.6

API#

API container launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5000 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--name=luna-api \
--restart=always \
--detach=true \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/api:/srv/logs \
--network=host \
dockerhub.visionlabs.ru/luna/luna-api:v.6.9.0

Account creation using API service#

The account is created using an HTTP request to the "create account" resource of the API service.

You can also create an account using the Admin service. This method requires an existing login and password (or the default login and password) and enables you to create an "admin" account. See the "Admin service" section of the administrator manual for details.

To create the account using a request to the API service, you need to provide the following mandatory data:

login — email address
password — password
account_type — account type ("user" or "advanced_user")

Create the account using your authentication details.

If you want to keep the ability to use the "account_id" that was used as the "Luna-Account-Id" header in previous LP versions (without creating an account in the Admin service), then you need to link the old "account_id" to the account being created.

Example of CURL-request to the "create account" resource:

curl --location --request POST 'http://127.0.0.1:5000/6/accounts' \
--header 'Content-Type: application/json' \
--header 'Luna-Account-Id: <your_old_account_id>' \
--data '{
  "login": "user@mail.com",
  "password": "password",
  "account_type": "user",
  "description": "description"
}'

It is necessary to replace the authentication data from the example with your own.

To work with tokens, you must have an account.

Admin#

Admin container launch#

Note. If you are not going to use the Admin service, do not launch this container.

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5010 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/admin:/srv/logs \
--name=luna-admin \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-admin:v.4.7.16

Monitoring data about the number of executed requests is saved in the luna-admin bucket of the InfluxDB. To enable data saving use the following command:

docker exec -it luna-admin python3 ./base_scripts/influx2_cli.py create_usage_task --luna-config http://127.0.0.1:5070/1

Backport 4#

The section describes launching of Backport 4 service.

Note. The service is not mandatory for utilizing LP5 and is required for emulation of LP 4 API only.

Backport 4 container launch#

Use the following command to launch the service:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5130 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--name=luna-backport4 \
--restart=always \
--detach=true \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/backport4:/srv/logs \
--network=host \
dockerhub.visionlabs.ru/luna/luna-backport4:v.1.2.69

User Interface 4#

The User Interface 4 is used with the Backport 4 service only.

User Interface 4 container launch#

Note. You should have the account with account type user before launching the User Interface 4 container. Its login and password in Base64 format will be used to work with the user interface.

Use the following command to launch the service:

docker run \
--env=PORT=4200 \
--env=LUNA_API_URL=http://<server_external_ip>:5130 \
--env=BASIC_AUTH=dXNlckBtYWlsLmNvbTpwYXNzd29yZA== \
--name=luna-ui-4 \
--restart=always \
--detach=true \
--network=host \
-v /etc/localtime:/etc/localtime:ro \
dockerhub.visionlabs.ru/luna/luna4-ui:v.0.1.0

--env=PORT - specifies the port for running User Interface 4.

--env=BASIC_AUTH - sets the Basic authorization for the account which data is displayed in the user interface. It is necessary to convert login:password created at the stage "Account creation using API service" to Base64 format. The account type should be set to user.

--env=LUNA_API_URL - specifies the URL of the Backport 4 service.

You should use the external IP of the service, not localhost.
You should specify the Backport 4 service port (5130 is set by default).

Services launch#

General container launching parameters#

Login to registry#

Description of launch parameters#

Description of database creation parameters#

Stop LUNA PLATFORM 4 containers#

Monitoring configuration#

InfluxDB OSS 2#

Run InfluxDB container#

Grafana#

Run Grafana container#

Monitoring parameters configuration#

Configurator#

Optional services usage#

Migrate database data#

Database recreation#

Configurator DB tables creation#

Run Configurator container#

Configuring InfluxDB#

Settings changing#

Enable logging to server directory#

Change logs directory manually before Configurator launch#

Change logs directory manually after Configurator launch#

Change logs directory using dump file#

Image Store#

Samples migrations#

Image Store container launch#

Buckets creation#

Accounts#

Admin DB transformation#

Admin DB migration#

Accounts DB tables creation#

Accounts container launch#

Licenses#

Licenses container launch#

Faces#

Faces database migration#

Create VLMatch function for Faces DB#

Build VLMatch for PostgreSQL#

Add VLMatch function to Faces database#

Faces container launch#

Events#

Create VLMatch function for Events DB#

Add VLMatch function to Events database#

PostGIS installation for Events#

Events database migration#

Events container launch#

Matcher services#

Use Python Matcher without Python Matcher Proxy#

Python Matcher container launch#

Handlers#

Change the used database for Handlers#

Changing the model of the neural network of bodies#

Handlers database migration#

Handlers container launch#

Run Handlers utilizing CPU#

Run Handlers utilizing GPU#

Tasks#

Tasks database migration#

Tasks and Tasks worker containers launch#

Tasks Worker launch#

Tasks launch#

Sender#

Sender container launch#

API#

API container launch#

Account creation using API service#

Admin#

Admin container launch#

Backport 4#

Backport 4 container launch#

User Interface 4#

User Interface 4 container launch#