Migration from LP3¶
This section describes data transferring from the LUNA PLATFORM 3 (v.3.3.8) to LUNA PLATFORM 5 (v.5.0.0).
Note
You can configure all the listed settings in the Configurator service of LUNA PLATFORM 5 or configuration files of the corresponding services (if the Configurator service is not utilized).
Accounts, descriptors, and persons migration¶
This section describes accounts, descriptors, and persons migration from LUNA PLATFORM 3 databases to LUNA PLATFORM 5 databases.
This instruction describes migration from Aerospike and PostgreSQL (LUNA PLATFORM 3) databases to the PostgreSQL (LUNA PLATFORM 5) databases.
Warning
Create a backup for your databases before starting migration procedures.
Configuration file¶
You should edit the following configuration file before starting the migration:
> “./base_scripts/migrate_backport3/config/config.conf”.
Enter the following information:
API, Faces, Broker services general parameters (LUNA PLATFORM 3);
Backport 3, Accounts and Faces services general parameters (LUNA PLATFORM 5);
Luna-Image-Store general parameters;
Logging parameters.
Before migration¶
The server where the migrations script is launched should have a connection to all the specified databases and the Broker service.
Make sure that the Broker service (LUNA PLATFORM 3) is launched. It is used to get descriptors from the database.
It is not necessary to launch the LUNA PLATFORM 5 services to perform migrations.
The databases with all the required tables should be created for Backport3, Accounts and Faces services (LUNA PLATFORM 5) before running migrations. See the installation manuals of corresponding services.
The Faces (LUNA PLATFORM 3) and Faces (LUNA PLATFORM 5) database names are similar by default (luna_faces). You should fix it using one of the following ways:
You can use another PostgreSQL for creating a new database for the Faces service (LUNA PLATFORM 5) hence the “luna_faces” name will not be changed. In this case, you should specify the new address of the PostgreSQL database in the “LUNA_FACES_DB” section of the Faces configuration file/Configurator.
You can change the default name of the Faces DB (LUNA PLATFORM 5) in the “LUNA_FACES_DB” section of the Faces configuration file/Configurator and commands for the Faces database creation. For example, set the “luna_faces_5” name.
Warning
Make sure that the databases for the Backport 3 service (LUNA PLATFORM 5) and the Faces service (LUNA PLATFORM 5) are empty (there are no entries) before starting the migration.
Warning
Do not launch the creation of database tables for the Faces service before changing the database name/PostgreSQL address in the “LUNA_FACES_DB” section of the Faces service configuration file/Configurator. Otherwise, you can lose the data stored in the Faces database of LUNA PLATFORM 3.
Migration launch¶
Warning
The script was tested on the LUNA PLATFORM 3 of version 3.3.8. It was not tested on the other LUNA PLATFORM 3 versions.
All the migration procedures are performed using the “start_migration.py” script. See the “Migration script description” section below for additional information about the script.
Python of version 3.9 and its headers are required for the script launching.
The virtual environment for the Backport 3 service should be activated before the script launching. Read about the virtual environment activation and requirements installation in the install.html document.
Follow these steps to perform migrations:
Go to the Backport3 migration directory (“luna-backport3/base_scripts/migrate_backport3” by default).
cd luna-backport3/base_scripts/migrate_backport3
Create a virtual environment, activate it, install the dependencies:
python3.9 -m venv venv;
source venv/bin/activate;
pip install -r requirements.txt;
Run the script:
python ./start_migration.py
Migration script description¶
Data transfer will be performed in the following order:
database migration from API (LUNA PLATFORM 3) service to Backport 3 (LUNA PLATFORM 5)
fill up tokens in Backport 3 database (LUNA PLATFORM 5)
database migration from Faces (LUNA PLATFORM 3) to Backport 3 (LUNA PLATFORM 5)
database migration from Faces (LUNA PLATFORM 3) to Faces (LUNA PLATFORM 5)
updating descriptors in the Faces database (LUNA PLATFORM 5) from the CORE (LUNA PLATFORM 3) database
add accounts to samples and optionally portraits in the Luna-Image-Store storage
When the script is launched, the “luna-backport3_ERROR_migration.txt” and “luna-backport3_WARNING_migration.txt” files are created. The files include information about all the errors and warnings that occurred during migration.
Run the following script to get help:
python ./base_scripts/migrate_backport3/start_migration.py --help
To start individual migration steps, pass the “–migrate” command line argument.
The argument takes the following parameters:
stage_1 - migrate from API (LUNA PLATFORM 3) and Faces (LUNA PLATFORM 3) databases to Backport 3 (LUNA PLATFORM 5) database
stage_2 - fill up tokens in Backport 3 (LUNA PLATFORM 5) database
stage_3 - migrate from Faces (LUNA PLATFORM 3) database to Faces (LUNA PLATFORM 5) database
stage_4 - update descriptors in the Faces (LUNA PLATFORM 5) from the CORE (LUNA PLATFORM 3) database
stage_5 - add accounts for samples and optionally portraits (–migrate_portraits flag) in the Luna-Image-Store storage
all - perform all the above steps
Code example:
python ./base_scripts/migrate_backport3/start_migration.py --migrate stage_1
Note
If you are migrating from stage_4, check that the face and attribute tables in the Faces (LUNA PLATFORM 5) database have entries.
If something went wrong during stage_4, use the “–lower_boundary” argument to specify the last failed face ID to continue migration. The face ID can be found in migration logs.
For example:
python ./base_scripts/migrate_backport3/start_migration.py --migrate stage_4 --lower_boundary 02e7b0db-b3c3-4446-bbdd-0f0d9a566058
Samples migrations¶
Samples migration is required to add an account for each sample.
Implied that Luna-Image-Store storage from LUNA PLATFORM 3 will be used with LUNA PLATFORM 5. Storage transfer not provided during the migration.
If you are using local storage, you can specify the path to the storage to utilize it in the Image Store service of LUNA PLATFORM 5.
If you use Image Store in the docker container you should mount the directory with buckets during the Image Store container launch. See the Image Store launch manual.
If you use Image Store without docker container you should specify the path in the “LOCAL_STORAGE” parameter of the Image Store config.
You can also utilize the Image Store service provided with LUNA PLATFORM 3 but it is not recommended.
By default, the samples bucket in LUNA PLATFORM 3 was called “visionlabs-warps”. Configure its utilization for LUNA PLATFORM 5 using the “LUNA_IMAGE_STORE_SAMPLES_ADDRESS” setting. You should specify the “origin” of Image Store service and the “bucket” name.
Portraits migration¶
Backport uses samples as portraits by default. Thus it is not required to store portraits and samples simultaneously. If samples will not be used as portraits, its required to migrate portraits (run migration with –migrate_portraits flag).
If portraits are required, you should turn off the USE_SAMPLES_AS_PORTRAITS setting of Backport 3.
You should follow one of these steps if you are going to use portraits:
If you store portraits in the IMAGE STORE service (the SEND_TO_LUNA_IMAGE_STORE setting of the API service in LUNA PLATFORM 3 is enabled), you should set up the LUNA_IMAGE_STORE_PORTRAITS_ADDRESS setting. To do this, use the values of LUNA_IMAGE_STORE_ORIGIN and LUNA_IMAGE_STORE_BUCKET settings that can be found in the configuration file of the API service (LUNA PLATFORM 3).
If you use a plugin to store portraits, you should move the existing portraits to the Image Store service of LUNA PLATFORM 5. See the “./docs/ReferenceManuals/ImageStoreReferenceManual.html” document in the LUNA PLATFORM 5 distribution package for details about creating a bucket and saving images to the service.