Additional information#

This section provides the following additional information:

Monitoring and logs visualization using Grafana.
Useful commands for working with Docker.
Description of the parameters for launching LUNA PLATFORM services and creating databases.
Ways to change service settings.
Actions to enable saving LP service logs to files.
Configuring Docker log rotation.
Setting custom InfluxDB settings.
Using Python Matcher service with Python Matcher Proxy service.
System scaling.
Improve system performance.
Manual creation of databases when using an external PostgreSQL database (including creation of the VLMatch function for the Faces and Events database).
Compiling the VLMatch library for Oracle.

Monitoring and logs visualization using Grafana#

Monitoring visualization is performed by the LUNA Dashboards service, which contains the Grafana monitoring data visualization platform with configured LUNA PLATFORM dashboards.

If necessary, you can install customized dashboards for Grafana separately. See the "LUNA Dashboards" section in the administrator manual for more information.

Together with Grafana, you can use the Grafana Loki log aggregation system, which enables you to flexibly work with LUNA PLATFORM logs. The Promtail agent is used to deliver LUNA PLATFORM logs to Grafana Loki (for more information, see the "Grafana Loki" section in the administrator manual).

LUNA Dashboards#

Note: To work with Grafana you need to use InfluxDB version 2.

Run LUNA Dashboards container#

Use the docker run command with these parameters to run Grafana:

docker run \
--restart=always \
--detach=true \
--network=host \
--name=grafana \
-v /etc/localtime:/etc/localtime:ro \
dockerhub.visionlabs.ru/luna/luna-dashboards:v.0.0.9

Use "http://IP_ADDRESS:3000" to go to the Grafana web interface when the LUNA Dashboards and InfluxDB containers are running.

Grafana Loki#

Note: Grafana Loki requires LUNA Dashboards to be running.

Run Grafana Loki container#

Use the docker run command with these parameters to run Grafana Loki:

docker run \
--name=loki \
--restart=always \
--detach=true \
--network=host \
-v /etc/localtime:/etc/localtime:ro \
dockerhub.visionlabs.ru/luna/loki:2.7.1

Run Promtail container#

Use the docker run command with these parameters to run Promtail:

docker run \
-v /var/lib/luna/current/example-docker/logging/promtail.yml:/etc/promtail/luna.yml \
-v /var/lib/docker/containers:/var/lib/docker/containers \
-v /etc/localtime:/etc/localtime:ro \
--name=promtail \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/promtail:2.7.1 \
-config.file=/etc/promtail/luna.yml -client.url=http://127.0.0.1:3100/loki/api/v1/push -client.external-labels=job=containerlogs,pipeline_id=,job_id=,version=

Here:

-v /var/lib/luna/current/example-docker/logging/promtail.yml:/etc/promtail/luna.yml — Mounting the configuration file to the Promtail container.
-config.file=/etc/promtail/luna.yml — Flag with the address of the configuration file.
-client.url=http://127.0.0.1:3100/loki/api/v1/push — Flag with the address of deployed Grafana Loki.
-client.external-labels=job=containerlogs,pipeline_id=,job_id=,version= — Static labels to add to all logs sent to Grafana Loki.

Docker commands#

Show containers#

To show the list of launched Docker containers use the command:

docker ps

To show all the existing Docker containers use the command:

docker ps -a

Copy files to container#

You can transfer files into the container. Use the docker cp command to copy a file into the container.

docker cp <file_location> <container_name>:<folder_inside_container>

Enter container#

You can enter individual containers using the following command:

docker exec -it <container_name> bash

To exit the container, use the command:

exit

Images names#

You can see all the names of the images using the command:

docker images

Delete image#

If you need to delete an image:

Run the docker images command.
Find the required image, for example dockerhub.visionlabs.ru/luna/luna-image-store.
Copy the corresponding image ID from the IMAGE ID, for example, "61860d036d8c".
Specify it in the deletion command:

docker rmi -f 61860d036d8c

Delete all the existing images.

docker rmi -f $(docker images -q)

Stop container#

You can stop the container using the command:

docker stop <container_name>

Stop all the containers:

docker stop $(docker ps -a -q)

Delete container#

If you need to delete a container:

Run the "docker ps" command.
Stop the container (see Stop container).
Find the required image, for example dockerhub.visionlabs.ru/luna/luna-image-store.
Copy the corresponding container ID from the CONTAINER ID column, for example, "23f555be8f3a".
Specify it in the deletion command:

docker container rm -f 23f555be8f3a

Delete all the containers.

docker container rm -f $(docker container ls -aq)

Check service logs#

You can use the following command to show logs for the service:

docker logs <container_name>

Launching parameters description#

When launching a Docker container for a LUNA PLATFORM service you should specify additional parameters required for the service launching.

The parameters specific for a particular container are described in the section about this container launching.

All the parameters given in the service launching example are required for proper service launching and utilization.

Launching services parameters#

Example command of launching LP services containers:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=<Port_of_the_launched_service> \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<service>:/srv/logs/ \
--name=<service_container_name> \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/<service-name>:<version>

The following parameters are used when launching LP services containers:

docker run — Command for running the selected image as a new container.
dockerhub.visionlabs.ru/luna/<service-name>:<version> — Sets the image required for the container launching.

Links to download the container images you need are available in the description of the corresponding container launching.

--network=host — Sets that a network is not simulated and the server network is used. If you need to change the port for third-party party containers, you should change this string to -p 5440:5432. Where the first port 5440 is the local port and 5432 is the port used inside the container. The example is given for PostgreSQL.
--env= — Sets the environment variables required to run the container (see the "Service arguments" section).
--name=<service_container_name> — Sets the name of the launched container. The name must be unique. If there is a container with the same name, an error will occur.
--restart=always — Sets a restart policy. The daemon will always restart the container regardless of the exit status.
--detach=true — Run the container in the background mode.
-v — Enables you to mount the content of a server folder into a volume in the container. Thus their contents will synchronize. The following general data is mounted:
/etc/localtime:/etc/localtime:ro — Sets the current time zone used by the system in the container.
/tmp/logs/<service>:/srv/logs/ — Enables copying of the folder with service logs to your server /tmp/logs/<service> directory. You can change the directory where the logs will be saved according to your needs.

Service arguments#

Each service in LUNA PLATFORM has its own launch arguments. These arguments can be passed through:

Setting a flag for the launch script (run.py) of the corresponding service.
Setting environment variables (--env) on the Docker command line.

For example, using the --help flag you can get a list of all available arguments. An example of passing an argument to an API service:

docker run --rm dockerhub.visionlabs.ru/luna/luna-api:v.6.26.0 python3 /srv/luna_api/run.py --help

List of main arguments:

Launch flag	Environment variable	Description
`--port`	`PORT`	Port on which the service will listen for connections.
`--workers`	`WORKER_COUNT`	Number of workers for the service.
`--log_suffix` `--log_suffix`	`LOG_SUFFIX` `LOG_SUFFIX`	Suffix added to log file names (with the option to write logs to a file enabled).
`--config-reload`	`RELOAD_CONFIG`	Enable automatic configuration reload. See "Automatic configurations reload" in the LUNA PLATFORM 5 administrator manual.
`--pulling-time`	`RELOAD_CONFIG_INTERVAL`	Configuration checking period (default 10 seconds). See "Automatic configurations reload" in the LUNA PLATFORM 5 administrator manual.
`--luna-config` `--luna-config`	`CONFIGURATOR_HOST`, `CONFIGURATOR_PORT`	Address of the Configurator service for downloading settings. For `--luna-config` it is sent in the format `http://localhost:5070/1`. For environment variables, the host and port are set explicitly. If the argument is not given, the default configuration file will be used.
`--config`	None	Path to the file with service configurations.
`--<config_name>`	`--EXTEND_CMD=<config_name>`	Tag of the specified configuration in the Configurator. When setting this configuration, the value of the tagged configuration will be used. Example: `--INFLUX_MONITORING TAG_1` Note: You must pre-tag the appropriate configuration in. Configurator. Note: Only works with the `--luna-config` flag.
`--tls_cert`	None	Path to the SSL certificate for launching the service using the HTTPS protocol.
`--tls_key`	None	Path to the SSL private key for launching the service using the HTTPS protocol.
`--tls_key_pass`	None	Password for the SSL private key for launching the service using the HTTPS protocol.

The list of arguments may vary depending on the service.

It is also possible to override the settings of services at their start using environment variables.

The VL_SETTINGS prefix is used to redefine the settings. Examples:

--env=VL_SETTINGS.INFLUX_MONITORING.SEND_DATA_FOR_MONITORING=0. Using the environment variable from this example will set the "SEND_DATA_FOR_MONITORING" setting for the INFLUX_MONITORING section to "0".
--env=VL_SETTINGS.OTHER.STORAGE_TIME=LOCAL. For non-compound settings (settings that are located in the "OTHER" section in the configuration file), you must specify the "OTHER" prefix. Using the environment variable from this example will set the value of the "STORAGE_TIME" setting (if the service uses this setting) to "LOCAL".

Passing flags using environment variable

Flags for which an environment variable is not explicitly allocated can be passed using the environment variable EXTEND_CMD.

For example, you can pass the configurations tag in the following way:

--env=EXTEND_CMD="--INFLUX_MONITORING=TAG_1 --LUNA_EVENTS_DB=TAG_2"

Creating DB parameters#

Example command of launching containers for database migration or database creation:

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<service>:/srv/logs/ \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/<service-name>:<version> \
python3 ./base_scripts/db_create.py --luna-config http://localhost:5070/1

The following parameters are used when launching containers for database migration or database creation:

Here:

--rm — Sets if the container is deleted after all the specified scripts finish processing.
python3 ./base_scripts/db_create.py — Sets Python version and a script db_create.py launched in the container. The script is used for the database structure creation.
--luna-config http://localhost:5070/1 — Sets where the launched script should receive configurations. By default, the service requests configurations from the Configurator service.

Ways to change service settings#

There are three general ways to change settings stored in the Configurator DB:

Using Configurator GUI.
Using Configurator API requests.
Using dump file.

Settings can be changed after the Configurator service was launched.

The settings of Configurator itself are stored in the "luna_configurator_postgres.conf" file that is downloaded to the Configurator container during its launch.

Configurator Interface

You can enter the configurator GUI and change setting. By default, the following address on localhost are used: <Configurator_server_address>:5070.

Configurator API

You can use Configurator API to update settings. See the Configurator OpenAPI specification.

Dump file

You can receive a dump file with all the LP services settings. Use one of the following commands:

wget -O /var/lib/luna/settings_dump.json 127.0.0.1:5070/1/dump

or

curl 127.0.0.1:5070/1/dump > /var/lib/luna/settings_dump.json

You should specify correct Configurator service address and port.

You should delete the "limitations" section from the file. You will not be able to apply your updated dump file if the section remains.

    "limitations":[
      ...
    ],

Edit parameters in the "settings" section.

    "settings":[
      ...
    ],

Copy the dump file to your Configurator container.

docker cp /var/lib/luna/settings_dump.json luna-configurator:/srv/

Apply the file:

docker exec luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/settings_dump.json

Logging to server#

To enable saving logs to the server, you should:

Create directories for logs on the server.
Activate log recording and set the location of log storage inside LP service containers.
Configure synchronization of log directories in the container with logs on the server using the volume argument at the start of each container.

Create logs directory#

Below are examples of commands for creating directories for saving logs and assigning rights to them for all LUNA PLATFORM services.

mkdir -p /tmp/logs/configurator /tmp/logs/image-store /tmp/logs/accounts /tmp/logs/faces /tmp/logs/licenses /tmp/logs/events /tmp/logs/python-matcher /tmp/logs/handlers /tmp/logs/remote-sdk /tmp/logs/tasks /tmp/logs/tasks-worker /tmp/logs/sender /tmp/logs/api /tmp/logs/admin /tmp/logs/backport3 /tmp/logs/backport4 /tmp/logs/lambda

chown -R 1001:0 /tmp/logs/configurator /tmp/logs/image-store /tmp/logs/accounts /tmp/logs/faces /tmp/logs/licenses /tmp/logs/events /tmp/logs/python-matcher /tmp/logs/handlers /tmp/logs/remote-sdk /tmp/logs/tasks /tmp/logs/tasks-worker /tmp/logs/sender /tmp/logs/api /tmp/logs/admin /tmp/logs/backport3 /tmp/logs/backport4 /tmp/logs/lambda

If you need to use the Python Matcher Proxy service, then you need to additionally create the /tmp/logs/python-matcher-proxy directory and set its permissions.

Logging activation#

LP services logging activation#

To enable logging to file, you need to set the log_to_file and folder_with_logs settings in the <SERVICE_NAME>_LOGGER section of the settings for each service.

Automatic method (before/after starting Configurator)

To update logging settings, you can use the logging.json settings file provided with the distribution package.

Run the following command after starting the Configurator service:

docker cp /var/lib/luna/current/extras/conf/logging.json luna-configurator:/srv/luna_configurator/used_dumps/logging.json

Update your logging settings with the copied file.

docker exec -it luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/luna_configurator/used_dumps/logging.json

Manual method (after starting Configurator)

Go to the Configurator service interface (127.0.0.1:5070) and set the logs path in the container in the folder_with_logs parameter for all services whose logs need to be saved. For example, you can use the path /srv/logs.

Set the log_to_file option to true to enable logging to a file.

Configurator service logging activation (before/after Configurator start)#

The Configurator service settings are not located in the Configurator user interface, they are located in the following file:

/var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf

You should change the logging parameters in this file before starting the Configurator service or restart it after making changes.

Set the path to the logs location in the container in the FOLDER_WITH_LOGS = ./ parameter of the file. For example, FOLDER_WITH_LOGS = /srv/logs.

Set the log_to_file option to true to enable logging to a file.

Mounting directories with logs when starting services#

The log directory is mounted with the following argument when starting the container:

-v <server_logs_folder>:<container_logs_folder> \

where <server_logs_folder> is the directory created in the create logs directory step, and <container_logs_folder> is the directory created in the activate logging step.

Example of command to launch the API service with mounting a directory with logs:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5000 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--name=luna-api \
--restart=always \
--detach=true \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/api:/srv/logs \
--network=host \
dockerhub.visionlabs.ru/luna/luna-api:v.6.26.0

The example container launch commands in this documentation contain these arguments.

Docker log rotation#

To limit the size of logs generated by Docker, you can set up automatic log rotation. To do this, add the following data to the /etc/docker/daemon.json file:

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
        "max-file": "5"
    }
}

This will allow Docker to store up to 5 log files per container, with each file being limited to 100MB.

After changing the file, you need to restart Docker:

systemctl reload docker

The above changes are the default for any newly created container, they do not apply to already created containers.

Set custom InfluxDB settings#

If you are going to use InfluxDB OSS 2, then you need to update the monitoring settings in Configurator service.

There are the following settings for InfluxDB OSS 2:

"send_data_for_monitoring": 1,
"use_ssl": 0,
"flushing_period": 1,
"host": "127.0.0.1",
"port": 8086,
"organization": "<ORGANIZATION_NAME>",
"token": "<TOKEN>",
"bucket": "<BUCKET_NAME>",
"version": <DB_VERSION>

You can update InfluxDB settings in the Configurator service by following these steps:

Open the following file:

vi /var/lib/luna/current/extras/conf/influx2.json

Set required data.
Save changes.
Copy the file to the influxDB container:

docker cp /var/lib/luna/current/extras/conf/influx2.json luna-configurator:/srv/

Update settings in the Configurator.

docker exec -it luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/influx2.json

You can also manually update settings in the Configurator service user interface.

The Configurator service configurations are set separately.

Open the file with the Configurator configurations:

vi /var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf

Set required data.
Save changes.
Restart Configurator:

docker restart luna-configurator

Use Python Matcher with Python Matcher Proxy#

As mentioned earlier, along with the Python Matcher service, you can additionally use the Python Matcher Proxy service, which will redirect matching requests either to the Python Matcher service or to the matching plugins. Plugins may significantly improve matching processing performance. For example, it is possible to organize the storage of the data required for matching operations and additional objects fields in separate storage using plugins, which will speed up access to the data compared to the use of the standard LUNA PLATFORM database.

To use the Python Matcher service with Python Matcher Proxy, you should additionally launch the appropriate container, and then set a certain setting in the Configurator service. Follow the steps below only if you are going to use matching plugins.

See the description and usage of matching plugins in the administrator manual.

Python Matcher proxy container launch#

Use the following command to launch the service:

After starting the container, you need to set the "luna_matcher_proxy":true parameter in the "ADDITIONAL_SERVICES_USAGE" section in the Configurator service.

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5110 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--env=SERVICE_TYPE="proxy" \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/python-matcher-proxy:/srv/logs \
--name=luna-python-matcher-proxy \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-python-matcher:v.1.9.0

After launching the container, you need to set the following value in the Configurator service.

ADDITIONAL_SERVICES_USAGE = "luna_matcher_proxy":true

System scaling#

All LP services are linearly scalable and can be located on several services.

You can run additional containers with LP services to improve performance and fail-safety. The number of services and the characteristics of servers depend on your tasks.

To increase performance, you may either improve the performance of a single server or increase the number of servers used by distributing most resource-intensive components of the system.

Balancers are used for the distribution of requests among the launched service instances. This approach provides the necessary processing speed and the required fail-safety level for specific customer's tasks. In the case of a node failure, the system will not stop: requests will be redirected to another node.

The image below shows two instances of the Faces service balanced by Nginx. Nginx receives requests on port 5030 and routes them to Faces instances. The faces services are launched on ports 5031 and 5032.

It is strongly recommended to regularly back up databases to a separate server regardless of the fail-safety level of the system. It allows you not to lose data in case of unforeseen circumstances.

MQs, databases, and balancers used by LUNA PLATFORM are products of third-party developers. You should configure them according to the recommendations of the corresponding vendors.

The Remote SDK service and the Python Matcher service perform the most resource-intensive operations.

The Remote SDK service performs mathematical image transformations and descriptors extraction. The operations require significant computational resources. Both CPU and GPU can be used for computations.

GPU usage is preferable since it improves the processing of requests. However, not all types of video cards are supported.

The Python Matcher service performs matching with lists. Matching requires CPU resources, however, you also should allocate as much RAM as possible for each Python Matcher instance. The RAM is used to store descriptors received from a database. Thus matcher does not require to request each descriptor from the database.

When distributing instances on several servers, you should consider the performance of each server. For example, if a large task is executed by several Python Matcher instances, and one of the instances is on the server with low performance, this can slow down the execution of the entire task.

For each instance of the service, you can set the number of workers. The greater the number of workers, the more resources and memory are consumed by the service instance. See the detailed information in the "Worker processes" section of the LUNA PLATFORM administrator manual.

Nginx#

Nginx is required when you use several instances of LUNA PLATFORM services.

Nginx configuration#

The Nginx configuration file includes parameters for balancing of requests to API, Faces, Image Store and Events.

Check the "/var/lib/luna/current/extras/conf/nginx.conf" configuration file before running NGINX.

Nginx listens requests on the specified ports and sends the requests to the available instances of services. The following servers and ports are set in Nginx configuration file.

Service name	Port set in Nginx	Ports for the launched services
API	5000	5001-5004
Faces	5030	5031-5034
Image Store	5020	5021-5024
Events	5040	5041-5044

Run Nginx container#

Note: You should configure the file for using with your launched services.

Use the docker run command with these parameters to launch the Nginx container:

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /var/lib/luna/current/extras/conf/nginx.conf:/etc/nginx/nginx.conf \
--name=nginx \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/nginx:1.17.4-alpine

Here:

-v /var/lib/luna/current/extras/conf/nginx.conf:/etc/nginx/nginx.conf — Nginx configuration file used for Nginx launching.

See the example of Nginx configuration file for more services instances launch here: "/var/lib/luna/current/extras/conf/nginx.conf"

Launching several containers#

There are two steps required for launching several instances of the same LP service

Run several containers of the service.

You must launch the required number of service by using the corresponding command for the service.

For example, for the API service you must run the following command with updated parameters.

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=<port> \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<folder_name>:/srv/logs \
--name=<name> \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-api:v.6.26.0

When running several similar containers the following parameters of the containers must differ:

--env=PORT=<port> — Specified port for similar containers must differ. You must specify an available port for the instance. For example, "5001", "5002". The "5000" port will be specified for the Nginx balancer.
/tmp/logs/<folder_name>:/srv/logs — Specified folder name for logs must differ to distinguish logs for different service instances.
--name=<container_name> — Name of the launched container must differ as it is prohibited to launch two containers with the same name. For example, "api_1", "api_2".
--gpus device=0 — CORE services usually utilize different GPU devices. Thus you should specify different device numbers.
Configure your balancer (e.g., Nginx) for routing requests to the services.

For each scaled LP service, you must set a port where Nginx will listen to service requests and real ports of each service instance where Nginx will redirect the requests.

An example of Nginx configuration file can be found here:

“/var/lib/luna/current/extras/conf/nginx.conf”.

You can use another balancer, but its utilization is not described in this documentation.

Improve performance#

Follow these steps if you need to improve the performance of LUNA PLATFORM in Docker containers.

Docker security policies impose restrictions on the network interaction of LP services. If you perform the listed below actions and disable the policies, a new network connection of docker containers will be established approximately 20% faster.

Changing the settings may have a negative impact on security. Do not change the settings if are not an advanced Docker user.

Docker must be installed before performing these actions. See "Docker installation".

Add --network=host --security-opt seccomp=unconfined parameters to the launch of each LP service container. The launch commands are described in section "Services launch".

The example below is given for the Faces service.

docker run --env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5031 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
--name=luna-faces1 \
--restart=always \
--detach=true \
--network=host \
--security-opt seccomp=unconfined \
dockerhub.visionlabs.ru/luna/luna-faces:v.4.11.0

Add "userland-proxy": false to the Docker configuration.

Create configuration file for Docker.

echo 'DOCKER_OPTS="--config-file=/etc/docker/daemon.json"' > /etc/default/docker

Create the file.

vi /etc/docker/daemon.json

Add the following option to the opened file.

{
     "userland-proxy": false
}

Restart Docker.

systemctl restart docker

External DB#

This section describes commands required for configuring external PostgreSQL for working with LP. External means that you already have working DB and want to use it with LP.

You need to specify your external DB in the configurations of LP services.

Skip this section if you are using PostgreSQL from docker container!

After you perform all the commands from this sections go back to the "Configurator" section.

You database user and other parameters may differ. Consider it during the commands execution.

The sections below describes the databases creation. You should use the "db_create.py" script to create the tables structure, after the table is created. The script launch from the container is described for each service in sections above. E. g. see the "Faces DB tables creation".

PostgreSQL user creation#

Go to the directory.

cd /var/

Create a database user.

runuser -u postgres -- psql -c 'create role luna;'

Assign a password to the user.

runuser -u postgres -- psql -c "ALTER USER luna WITH PASSWORD 'luna';"

Configurator DB creation#

Create the database for the Configurator service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create the database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_configurator;'

Grant privileges to the database user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_configurator TO luna;'

Allow user to authorize in the DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

Accounts DB creation#

Create the database for the Accounts service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create the database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_accounts;'

Grant privileges to the database user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_accounts TO luna;'

Allow user to authorize in the DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

Handlers DB creation#

Create the database for the Handlers service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create a database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_handlers;'

Grant privileges to the database and the user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_handlers TO luna;'

Enable the user to login into DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

Backport 3 DB creation#

Create the database for the Backport 3 service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create a database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_backport3;'

Grant privileges to the database and the user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_backport3 TO luna;'

Enable the user to login into DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

Faces DB creation#

Create the database for the Faces service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create a database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_faces;'

Grant privileges to the database and the user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_faces TO luna;'

Enable the user to login into DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

Create VLMatch function for Faces DB#

The Faces service requires the VLMatch additional function to be added to the utilized database. LUNA PLATFORM cannot perform matching calculations without this function. The VLMatch function can be added to the PostgreSQL or Oracle database.

The VLMatch library is compiled for your particular database version.

Do not use the library built for another version of DB. For example, the library build for the PostgreSQL of version 12 cannot be used for the PostgreSQL of version 9.6.

This section describes the function creation for PostgreSQL.

The instruction for the Oracle DB is given in the "VLMatch for Oracle" section.

Build VLMatch#

Note: The following instruction describes installation for PostgreSQL 16.

You can find all the required files for the VLMatch user-defined extension (UDx) compilation in the following directory:

/var/lib/luna/current/extras/VLMatch/postgres/

The following instruction describes installation for PostgreSQL 16.

For VLMatch UDx function compilation one needs to:

Install RPM repository:

dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm

Install PostgreSQL:

dnf install postgresql16-server

Install the development environment:

dnf install postgresql16-devel

Install the gcc package:

dnf install gcc-c++

Install CMAKE. The version 3.5 or higher is required.
Open the make.sh script using a text editor. It includes paths to the currently used PostgreSQL version. Change the following values (if necessary):

SDK_HOME specifies the path to PostgreSQL home directory. Default value: /usr/include/postgresql/16/server;

LIB_ROOT specifies the path to PostgreSQL library root directory. Default value: /usr/lib/postgresql/16/lib.

Go to the make.sh script directory and run it:

cd /var/lib/luna/current/extras/VLMatch/postgres/

chmod +x make.sh

./make.sh

Add VLMatch function to Faces database#

The VLMatch function should be applied to the PostgreSQL DB.

Define the function inside the Faces database.

sudo -u postgres -h 127.0.0.1 -- psql -d luna_faces -c "CREATE FUNCTION VLMatch(bytea, bytea, int) RETURNS float8 AS 'VLMatchSource.so', 'VLMatch' LANGUAGE C PARALLEL SAFE;"

Test function by sending re following request to the service database.

sudo -u postgres -h 127.0.0.1 -- psql -d luna_faces -c "SELECT VLMatch('\x1234567890123456789012345678901234567890123456789012345678901234'::bytea, '\x0123456789012345678901234567890123456789012345678901234567890123'::bytea, 32);"

The result returned by the database must be "0.4765625".

Events DB creation#

Create the database for the Events service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_events;'

Grant privileges to database and the user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_events TO luna;'

Enable the user to login into DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

Create VLMatch function for Events DB#

Note: Do not use the library built for another version of DB. For example, the library build for the PostgreSQL of version 12 cannot be used for the PostgreSQL of version 9.6.

The Events service requires the VLMatch additional function to be added to the utilized database. LUNA PLATFORM cannot perform matching calculations without this function. The VLMatch function can be added to the PostgreSQL.

The VLMatch library is compiled for your particular database version.

This section describes the function creation for PostgreSQL. If you use the PostgreSQL database, you have already created and moved the created library during the Faces service launch. See section "Build VLMatch for PostgreSQL".

Add VLMatch function to Events database#

The VLMatch function should be applied to the PostgreSQL DB.

Define the function inside the Events database.

sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "CREATE FUNCTION VLMatch(bytea, bytea, int) RETURNS float8 AS 'VLMatchSource.so', 'VLMatch' LANGUAGE C PARALLEL SAFE;"

Test function within call.

sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "SELECT VLMatch('\x1234567890123456789012345678901234567890123456789012345678901234'::bytea, '\x0123456789012345678901234567890123456789012345678901234567890123'::bytea, 32);"

The result returned by the database must be "0.4765625".

PostGIS installation for Events#

The Events service requires PostGIS for working with coordinates.

This instruction describes the PostGIS installation for PostgreSQL 12 database. The PostGIS version depends on the PostgreSQL version.

Install epel-release for access to extended package repository (necessary for RabbitMQ and other dependencies).

yum -y install epel-release

Install PostGIS.

yum -y install postgis25_12

Activate PostGIS in your database.

sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "CREATE EXTENSION postgis;";

See additional information about PostGIS on its website: https://postgis.net/

Tasks DB creation#

Create the database for the Tasks service. It is assumed that the DB user is already created.

Go to the directory.

cd /var/

Create the database.

runuser -u postgres -- psql -c 'CREATE DATABASE luna_tasks;'

Grant privileges to the database user.

runuser -u postgres -- psql -c 'GRANT ALL PRIVILEGES ON DATABASE luna_tasks TO luna;'

Allow user to authorize in the DB.

runuser -u postgres -- psql -c 'ALTER ROLE luna WITH LOGIN;'

VLMatch library compilation for Oracle#

Note: The following instruction describes installation for Oracle 21c.

You can find all the required files for the VLMatch user-defined extension (UDx) compilation in the following directory:

/var/lib/luna/current/extras/VLMatch/oracle

For VLMatch UDx function compilation one needs to:

Install required environment, see requirements:

sudo yum install gcc g++

Change SDK_HOME variable — oracle sdk root (default is $ORACLE_HOME/bin, check $ORACLE_HOME environment variable is set) in the makefile:

vi /var/lib/luna/current/extras/VLMatch/oracle/make.sh

Open the directory and run the file "make.sh ".

cd /var/lib/luna/current/extras/VLMatch/oracle

chmod +x make.sh

./make.sh

Define the library and the function inside the database (from database console):

CREATE OR REPLACE LIBRARY VLMatchSource AS '$ORACLE_HOME/bin/VLMatchSource.so';
CREATE OR REPLACE FUNCTION VLMatch(descriptorFst IN RAW, descriptorSnd IN RAW, length IN BINARY_INTEGER)
   RETURN BINARY_FLOAT 
AS
   LANGUAGE C
   LIBRARY VLMatchSource
   NAME "VLMatch"
   PARAMETERS (descriptorFst BY REFERENCE, descriptorSnd BY REFERENCE, length UNSIGNED SHORT, RETURN FLOAT);

Test function within call (from database console):

SELECT VLMatch(HEXTORAW('1234567890123456789012345678901234567890123456789012345678901234'), HEXTORAW('0123456789012345678901234567890123456789012345678901234567890123'), 32) FROM DUAL;

The result returned by the database must be "0.4765625".