Additional information

This section provides the following additional information:

• Account creation.
• Creating a schedule for a garbage collection task.
• Manage descriptor encryption.
• Monitoring and logs visualization using Grafana.
• Useful commands for working with Docker.
• Description of the parameters for launching LUNA PLATFORM services and creating databases.
• Actions to enable saving LP service logs to files.
• Configuring Docker log rotation.
• Setting custom InfluxDB settings.
• Using Python Matcher service with Python Matcher Proxy service.
• Move old LUNA PLATFORM data to root directory

Account creation

The account is created using an HTTP request to the "create account" resource of the API service.

You can also create an account using the Admin service. This method requires an existing login and password (or the default login and password) and enables you to create an "admin" account. See the "Admin service" section of the administrator manual for details.

To create the account using a request to the API service, you need to provide the following mandatory data:

• "login" — Email address.
• "password" — Password.
• "account_type" — Account type ("user" or "advanced_user").

Create the account using your authentication details.

Example of CURL-request to the "create account" resource:

curl --location --request POST 'http://127.0.0.1:5000/6/accounts' \
--header 'Content-Type: application/json' \
--data '{
  "login": "user@mail.com",
  "password": "password",
  "account_type": "user",
  "description": "description"
}'

It is necessary to replace the authentication data from the example with your own.

To work with tokens, you must have an account.

GC task schedule creation

Before you start working with the LUNA PLATFORM, you can create a schedule for the Garbage collection task.

To do this, make a "create tasks schedule" request to the API service, specifying the necessary rules for the schedule.

An example of a schedule creation command for an account created in section "Account creation" is given below.

The example sets a schedule for the Garbage collection task for events older than 30 days with the removal of the samples and the source images. The task will be repeated once a day at 05:30 am.

curl --location --request POST 'http://127.0.0.1:5000/6/tasks/schedules' \
--header 'Authorization: Basic dXNlckBtYWlsLmNvbTpwYXNzd29yZA==' \
--header 'Content-Type: application/json' \
--data '{
    "task": {
        "task_type": 4,
        "content": {
            "target": "events",
            "filters": {
                "create_time__lt": "now-30d"
            },
            "remove_samples": true,
            "remove_image_origins": true
        }
    },
    "trigger": {"cron": "30 5 * * *", "cron_timezone": "utc"},
    "behaviour": {"start_immediately": false, "create_stopped": false}
}'

If necessary, you can create a schedule without automatically activating it. To do this, specify the parameter "create_stopped": "true". In this case, after creating the schedule, it must be activated manually using the "action" = "start" parameter of the "patch tasks schedule" request.

For more information, see the "Running scheduled tasks" section of the administrator manual.

Manage descriptor encryption

Note: See the "Descriptor encryption" section in the administrator manual for detailed information.

To update existing descriptors in the Faces/Attributes/Events database, run the descriptors_encryption.py script. This script offers three operational options:

• Encrypting original descriptors.
• Switching to a new encryption key.
• Decrypting encrypted descriptors.

The script requires setting three environment variables regardless of the chosen option:

• OLD_ENCRYPTION_KEY (can be empty)
• NEW_ENCRYPTION_KEY (can be empty)
• ENCRYPTION_ALGORITHM (mandatory and cannot be empty)

Encrypting original descriptors

This option allows encrypting all descriptors in the database if they were not previously encrypted. Already encrypted descriptors will be ignored, allowing the script to be run multiple times. For this option, fill the NEW_ENCRYPTION_KEY variable and leave OLD_ENCRYPTION_KEY empty.

Switching to a new encryption key

This option assumes encryption has already been performed, and you have the current encryption key that needs refreshing. Already encrypted descriptors will be ignored, allowing the script to be run multiple times. Specify both OLD_ENCRYPTION_KEY and NEW_ENCRYPTION_KEY.

Decrypting encrypted descriptors

This option restores descriptors to their original state. The script processes only encrypted descriptors, leaving original ones unchanged. To execute this option of the script, specify OLD_ENCRYPTION_KEY and leave NEW_ENCRYPTION_KEY empty.

Launch descriptor migration script

Important: The script must be executed when the service is stopped, after migrating the database and before starting the service again.

To perform the migration, execute the descriptors_encryption.py script with the appropriate arguments. Below is the list of available arguments:

• --config — Path to the configuration file.
• --luna-config — Address and API version of the Configurator service for fetching settings (e.g., "http://127.0.0.1:5070/1").
• --chunk-size — Batch size for updates.
• --LUNA_FACES_DB — Tag to set LUNA_FACES_DB for fetching settings from Configurator (for Faces DB migration only).
• --DATABASE_NUMBER — Tag to set DATABASE_NUMBER for fetching settings from Configurator.
• --LUNA_ATTRIBUTES_DB — Tag to set LUNA_ATTRIBUTES_DB for fetching settings from Configurator (for Faces DB migration only).
• --LUNA_EVENTS_DB — Tag to set LUNA_EVENTS_DB for fetching settings from Configurator (for Events DB migration only).
• -v, --verbose — Enable debug mode.

Ensure to pass the corresponding environment variables when executing the container command (as mentioned above).

Important: Run the script after the DB migration and before starting the service.

Example command to execute the descriptors migration in the Faces DB:

docker run \
--env=OLD_ENCRYPTION_KEY=<your_old_encryption_key> \
--env=NEW_ENCRYPTION_KEY=<your_new_encryption_key> \
--env=ENCRYPTION_ALGORITHM=aes256-gcm \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/faces:/srv/logs \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/luna-faces:v.4.12.21 \
python3 ./base_scripts/descriptors_encryption.py --luna-config=http://127.0.0.1:5070/1 

Monitoring and logs visualization using Grafana

Monitoring visualization is performed by the LUNA Dashboards service, which contains the Grafana monitoring data visualization platform with configured LUNA PLATFORM dashboards.

If necessary, you can install customized dashboards for Grafana separately. See the "LUNA Dashboards" section in the administrator manual for more information.

Together with Grafana, you can use the Grafana Loki log aggregation system, which enables you to flexibly work with LUNA PLATFORM logs. The Promtail agent is used to deliver LUNA PLATFORM logs to Grafana Loki (for more information, see the "Grafana Loki" section in the administrator manual).

LUNA Dashboards

Note: To work with Grafana you need to use InfluxDB version 2.

Note: Before updating, make sure that the old LUNA Dashboards container is deleted.

Run LUNA Dashboards container

Use the docker run command with these parameters to run Grafana:

docker run \
--restart=always \
--detach=true \
--network=host \
--name=grafana \
-v /etc/localtime:/etc/localtime:ro \
dockerhub.visionlabs.ru/luna/luna-dashboards:v.0.1.1

Use "http://IP_ADDRESS:3000" to go to the Grafana web interface when the LUNA Dashboards and InfluxDB containers are running.

Grafana Loki

Note: Grafana Loki requires LUNA Dashboards to be running.

Note: Before updating, make sure that the old Grafana Loki and Promtail containers are removed.

Run Grafana Loki container

Use the docker run command with these parameters to run Grafana Loki:

docker run \
--name=loki \
--restart=always \
--detach=true \
--network=host \
-v /etc/localtime:/etc/localtime:ro \
dockerhub.visionlabs.ru/luna/loki:2.7.1

Run Promtail container

Use the docker run command with these parameters to run Promtail:

docker run \
-v /var/lib/luna/current/example-docker/logging/promtail.yml:/etc/promtail/luna.yml \
-v /var/lib/docker/containers:/var/lib/docker/containers \
-v /etc/localtime:/etc/localtime:ro \
--name=promtail \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/promtail:2.7.1 \
-config.file=/etc/promtail/luna.yml -client.url=http://127.0.0.1:3100/loki/api/v1/push -client.external-labels=job=containerlogs,pipeline_id=,job_id=,version=

Here:

• -v /var/lib/luna/current/example-docker/logging/promtail.yml:/etc/promtail/luna.yml — Mounting the configuration file to the Promtail container.

• -config.file=/etc/promtail/luna.yml — Flag with the address of the configuration file.

• -client.url=http://127.0.0.1:3100/loki/api/v1/push — Flag with the address of deployed Grafana Loki.

• -client.external-labels=job=containerlogs,pipeline_id=,job_id=,version= — Static labels to add to all logs sent to Grafana Loki.

Docker commands

Show containers

To show the list of launched Docker containers use the command:

docker ps

To show all the existing Docker containers use the command:

docker ps -a 

Copy files to container

You can transfer files into the container. Use the docker cp command to copy a file into the container.

docker cp <file_location> <container_name>:<folder_inside_container>

Enter container

You can enter individual containers using the following command:

docker exec -it <container_name> bash

To exit the container, use the command:

exit

Images names

You can see all the names of the images using the command:

docker images

Delete image

If you need to delete an image:

• Run the docker images command.
• Find the required image, for example dockerhub.visionlabs.ru/luna/luna-image-store.
• Copy the corresponding image ID from the IMAGE ID, for example, "61860d036d8c".
• Specify it in the deletion command:

docker rmi -f 61860d036d8c

Delete all the existing images.

docker rmi -f $(docker images -q)

Stop container

You can stop the container using the command:

docker stop <container_name>

Stop all the containers:

docker stop $(docker ps -a -q)

Delete container

If you need to delete a container:

• Run the "docker ps" command.
• Stop the container (see Stop container).
• Find the required image, for example dockerhub.visionlabs.ru/luna/luna-image-store.
• Copy the corresponding container ID from the CONTAINER ID column, for example, "23f555be8f3a".
• Specify it in the deletion command:

docker container rm -f 23f555be8f3a

Delete all the containers.

docker container rm -f $(docker container ls -aq)

Check service logs

You can use the following command to show logs for the service:

docker logs <container_name>

Launching parameters description

When launching a Docker container for a LUNA PLATFORM service you should specify additional parameters required for the service launching.

The parameters specific for a particular container are described in the section about this container launching.

All the parameters given in the service launching example are required for proper service launching and utilization.

Launching services parameters

Example command of launching LP services containers:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=<Port_of_the_launched_service> \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<service>:/srv/logs/ \
--name=<service_container_name> \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/<service-name>:<version>

The following parameters are used when launching LP services containers:

• docker run — Command for running the selected image as a new container.

• dockerhub.visionlabs.ru/luna/<service-name>:<version> — Sets the image required for the container launching.

Links to download the container images you need are available in the description of the corresponding container launching.

• --network=host — Sets that a network is not simulated and the server network is used. If you need to change the port for third-party party containers, you should change this string to -p 5440:5432. Where the first port 5440 is the local port and 5432 is the port used inside the container. The example is given for PostgreSQL.

• --env= — Sets the environment variables required to run the container (see the "Service arguments" section).

• --name=<service_container_name> — Sets the name of the launched container. The name must be unique. If there is a container with the same name, an error will occur.

• --restart=always — Sets a restart policy. The daemon will always restart the container regardless of the exit status.

• --detach=true — Run the container in the background mode.

• -v — Enables you to mount the content of a server folder into a volume in the container. Thus their contents will synchronize. The following general data is mounted:

• /etc/localtime:/etc/localtime:ro — Sets the current time zone used by the system in the container.

• /tmp/logs/<service>:/srv/logs/ — Enables copying of the folder with service logs to your server /tmp/logs/<service> directory. You can change the directory where the logs will be saved according to your needs.

Service arguments

Each service in LUNA PLATFORM has its own launch arguments. These arguments can be passed through:

• Setting a flag for the launch script (run.py) of the corresponding service.
• Setting environment variables (--env) on the Docker command line.

For example, using the --help flag you can get a list of all available arguments. An example of passing an argument to an API service:

docker run --rm dockerhub.visionlabs.ru/luna/luna-api:v.6.40.0 python3 /srv/luna_api/run.py --help

List of main arguments:

Launch flag	Environment variable	Description
--port	PORT	Port on which the service will listen for connections.
--workers	WORKER_COUNT	Number of workers for the service.
--log_suffix --log_suffix	LOG_SUFFIX LOG_SUFFIX	Suffix added to log file names (with the option to write logs to a file enabled).
--config-reload	RELOAD_CONFIG	Enable automatic configuration reload. See "Automatic configurations reload" in the LUNA PLATFORM 5 administrator manual.
--pulling-time	RELOAD_CONFIG_INTERVAL	Configuration checking period (default 10 seconds). See "Automatic configurations reload" in the LUNA PLATFORM 5 administrator manual.
--luna-config --luna-config	CONFIGURATOR_HOST, CONFIGURATOR_PORT	Address of the Configurator service for downloading settings. For --luna-config it is sent in the format http://localhost:5070/1. For environment variables, the host and port are set explicitly. If the argument is not given, the default configuration file will be used.
--config	None	Path to the file with service configurations.
--<config_name>	--EXTEND_CMD=<config_name>	Tag of the specified configuration in the Configurator. When setting this configuration, the value of the tagged configuration will be used. Example: --LUNA_MONITORING TAG_1 | Note: You must pre-tag the appropriate configuration in. Configurator. Note: Only works with the --luna-config flag.
--tls_cert	None	Path to the SSL certificate for launching the service using the HTTPS protocol.
--tls_key	None	Path to the SSL private key for launching the service using the HTTPS protocol.
--tls_key_pass	None	Password for the SSL private key for launching the service using the HTTPS protocol.

The list of arguments may vary depending on the service.

It is also possible to override the settings of services at their start using environment variables.

The VL_SETTINGS prefix is used to redefine the settings. Examples:

• --env=VL_SETTINGS.LUNA_MONITORING.SEND_DATA_FOR_MONITORING=0. Using the environment variable from this example will set the "SEND_DATA_FOR_MONITORING" setting for the LUNA_MONITORING section to "0".
• --env=VL_SETTINGS.OTHER.STORAGE_TIME=LOCAL. For non-compound settings (settings that are located in the "OTHER" section in the configuration file), you must specify the "OTHER" prefix. Using the environment variable from this example will set the value of the "STORAGE_TIME" setting (if the service uses this setting) to "LOCAL".

Passing flags using environment variable

Flags for which an environment variable is not explicitly allocated can be passed using the environment variable EXTEND_CMD.

For example, you can pass the configurations tag in the following way:

--env=EXTEND_CMD="--LUNA_MONITORING=TAG_1 --LUNA_EVENTS_DB=TAG_2"

Creating DB parameters

Example command of launching containers for database migration or database creation:

docker run \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/<service>:/srv/logs/ \
--rm \
--network=host \
dockerhub.visionlabs.ru/luna/<service-name>:<version> \
python3 ./base_scripts/db_create.py --luna-config http://localhost:5070/1

The following parameters are used when launching containers for database migration or database creation:

Here:

• --rm — Sets if the container is deleted after all the specified scripts finish processing.

• python3 ./base_scripts/db_create.py — Sets Python version and a script db_create.py launched in the container. The script is used for the database structure creation.

• --luna-config http://localhost:5070/1 — Sets where the launched script should receive configurations. By default, the service requests configurations from the Configurator service.

Logging to server

To enable saving logs to the server, you should:

• Create directories for logs on the server.
• Activate log recording and set the location of log storage inside LP service containers.
• Configure synchronization of log directories in the container with logs on the server using the volume argument at the start of each container.

Create logs directory

Below are examples of commands for creating directories for saving logs and assigning rights to them for all LUNA PLATFORM services.

mkdir -p /tmp/logs/configurator /tmp/logs/image-store /tmp/logs/accounts /tmp/logs/faces /tmp/logs/licenses /tmp/logs/events /tmp/logs/python-matcher /tmp/logs/handlers /tmp/logs/remote-sdk /tmp/logs/tasks /tmp/logs/tasks-worker /tmp/logs/sender /tmp/logs/api /tmp/logs/admin /tmp/logs/backport3 /tmp/logs/backport4 /tmp/logs/luna-video-agent /tmp/logs/luna-video-manager

chown -R 1001:0 /tmp/logs/configurator /tmp/logs/image-store /tmp/logs/accounts /tmp/logs/faces /tmp/logs/licenses /tmp/logs/events /tmp/logs/python-matcher /tmp/logs/handlers /tmp/logs/remote-sdk /tmp/logs/tasks /tmp/logs/tasks-worker /tmp/logs/sender /tmp/logs/api /tmp/logs/admin /tmp/logs/backport3 /tmp/logs/backport4 /tmp/logs/luna-video-agent /tmp/logs/luna-video-manager

If you need to use the Python Matcher Proxy service, then you need to additionally create the /tmp/logs/python-matcher-proxy directory and set its permissions.

Logging activation

LP services logging activation

To enable logging to file, you need to set the log_to_file and folder_with_logs settings in the <SERVICE_NAME>_LOGGER section of the settings for each service.

Automatic method (before/after starting Configurator)

To update logging settings, you can use the logging.json settings file provided with the distribution package.

Run the following command after starting the Configurator service:

docker cp /var/lib/luna/current/extras/conf/logging.json luna-configurator:/srv/luna_configurator/used_dumps/logging.json

Update your logging settings with the copied file.

docker exec -it luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/luna_configurator/used_dumps/logging.json

Manual method (after starting Configurator)

Go to the Configurator service interface (127.0.0.1:5070) and set the logs path in the container in the folder_with_logs parameter for all services whose logs need to be saved. For example, you can use the path /srv/logs.

Set the log_to_file option to true to enable logging to a file.

Configurator service logging activation (before/after Configurator start)

The Configurator service settings are not located in the Configurator user interface, they are located in the following file:

/var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf

You should change the logging parameters in this file before starting the Configurator service or restart it after making changes.

Set the path to the logs location in the container in the FOLDER_WITH_LOGS = ./ parameter of the file. For example, FOLDER_WITH_LOGS = /srv/logs.

Set the log_to_file option to true to enable logging to a file.

Mounting directories with logs when starting services

The log directory is mounted with the following argument when starting the container:

-v <server_logs_folder>:<container_logs_folder> \

where <server_logs_folder> is the directory created in the create logs directory step, and <container_logs_folder> is the directory created in the activate logging step.

Example of command to launch the API service with mounting a directory with logs:

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5000 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--name=luna-api \
--restart=always \
--detach=true \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/api:/srv/logs \
--network=host \
dockerhub.visionlabs.ru/luna/luna-api:v.6.40.0

The example container launch commands in this documentation contain these arguments.

Docker log rotation

To limit the size of logs generated by Docker, you can set up automatic log rotation. To do this, add the following data to the /etc/docker/daemon.json file:

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
        "max-file": "5"
    }
}

This will allow Docker to store up to 5 log files per container, with each file being limited to 100MB.

After changing the file, you need to restart Docker:

systemctl reload docker

The above changes are the default for any newly created container, they do not apply to already created containers.

Set custom InfluxDB settings

If you are going to use InfluxDB OSS 2, then you need to update the monitoring settings in Configurator service.

There are the following settings for InfluxDB OSS 2:

"send_data_for_monitoring": 1,
"use_ssl": 0,
"flushing_period": 1,
"host": "127.0.0.1",
"port": 8086,
"organization": "<ORGANIZATION_NAME>",
"token": "<TOKEN>",
"bucket": "<BUCKET_NAME>",
"version": <DB_VERSION>

You can update InfluxDB settings in the Configurator service by following these steps:

• Open the following file:

vi /var/lib/luna/current/extras/conf/influx2.json

• Set required data.
• Save changes.
• Copy the file to the influxDB container:

docker cp /var/lib/luna/current/extras/conf/influx2.json luna-configurator:/srv/

• Update settings in the Configurator.

docker exec -it luna-configurator python3 ./base_scripts/db_create.py --dump-file /srv/influx2.json

You can also manually update settings in the Configurator service user interface.

The Configurator service configurations are set separately.

• Open the file with the Configurator configurations:

vi /var/lib/luna/current/example-docker/luna_configurator/configs/luna_configurator_postgres.conf

• Set required data.
• Save changes.
• Restart Configurator:

docker restart luna-configurator

Use Python Matcher with Python Matcher Proxy

As mentioned earlier, along with the Python Matcher service, you can additionally use the Python Matcher Proxy service, which will redirect matching requests either to the Python Matcher service or to the matching plugins. Plugins may significantly improve matching processing performance. For example, it is possible to organize the storage of the data required for matching operations and additional objects fields in separate storage using plugins, which will speed up access to the data compared to the use of the standard LUNA PLATFORM database.

To use the Python Matcher service with Python Matcher Proxy, you should additionally launch the appropriate container, and then set a certain setting in the Configurator service. Follow the steps below only if you are going to use matching plugins.

See the description and usage of matching plugins in the administrator manual.

Python Matcher proxy container launch

Use the following command to launch the service:

After starting the container, you need to set the "luna_matcher_proxy":true parameter in the "ADDITIONAL_SERVICES_USAGE" section in the Configurator service.

docker run \
--env=CONFIGURATOR_HOST=127.0.0.1 \
--env=CONFIGURATOR_PORT=5070 \
--env=PORT=5110 \
--env=WORKER_COUNT=1 \
--env=RELOAD_CONFIG=1 \
--env=RELOAD_CONFIG_INTERVAL=10 \
--env=SERVICE_TYPE="proxy" \
-v /etc/localtime:/etc/localtime:ro \
-v /tmp/logs/python-matcher-proxy:/srv/logs \
--name=luna-python-matcher-proxy \
--restart=always \
--detach=true \
--network=host \
dockerhub.visionlabs.ru/luna/luna-python-matcher:v.1.12.0

After launching the container, you need to set the following value in the Configurator service.

ADDITIONAL_SERVICES_USAGE = "luna_matcher_proxy":true

Move old data to root directory

The official installation manual for LUNA PLATFORM v.5.40.0 and higher provides example commands for creating directories for storing PostgreSQL, InfluxDB and Image Store data in the root directory /var/lib/luna/<db_or_bucket_folder>. In older versions of LUNA PLATFORM, example commands were given to create directories within a specific LUNA PLATFORM version directory /var/lib/luna/current/example-docker/<db_or_bucket_folder>.

Storing directories in the root directory allows you not to transfer data during the next update.

Example commands for launching PostgreSQL, InfluxDB and Image Store containers contain arguments for mounting the corresponding data directories from the root directory. If you are upgrading from version LUNA PLATFORM v.5.38.3 and lower and the previous version of LUNA PLATFORM was installed according to the official documentation, then in the container launch commands, specify the old directories with the mounted data or first transfer the old data according to the sections below.

The sections below provide example commands for version 5.38.3.

Important: Before transferring old data, stop the corresponding containers, because New data may appear during the transfer process.

Move Image Store buckets

The following step is required if you are storing Image Store buckets in the default directory.

Copy the "image_store" folder with all its buckets.

mv /var/lib/luna/luna_v.5.38.3/example-docker/image_store /var/lib/luna/

Set rights for the luna user.

chown -R 1001:0 /var/lib/luna/image_store

Move PostgreSQL data

The following step is required if you are using PostgreSQL in Docker container.

Copy the "data" folder.

mv /var/lib/luna/luna_v.5.38.3/example-docker/postgresql /var/lib/luna/

Move InfluxDB Data

The following step is required if you are using InfluxDB in Docker container.

Copy the "influx" folder with all its buckets.

mv /var/lib/luna/luna_v.5.38.3/example-docker/influx /var/lib/luna/