Database

We support Oracle and Postgres databases.

DB scheme.

class luna_faces.db.faces_db_tools.models.faces_models.Attribute(**kwargs)

Database table model for attributes.

age

age

Type:

int

age_obtaining_method

how the gender was obtained

Type:

int

age_version

age version

Type:

int

create_time

date and time of creating attributes

Type:

DateTime

descriptor_samples_generation

descriptor samples generation

Type:

int

ethnicity

ethnicity, enum luna_faces.crutches_on_wheels.maps.vl_maps.ETHNIC_MAP

Type:

int

ethnicity_obtaining_method

how the ethnicity was obtained

Type:

int

ethnicity_version

ethnicity version

Type:

int

face_id

face id, uuid.

Type:

str

gender

gender. 0 - woman, 1- man

Type:

int

gender_obtaining_method

how the gender was obtained

Type:

int

gender_version

gender version

Type:

int

class luna_faces.db.faces_db_tools.models.faces_models.Column(*args, **kwargs)

Release some pretty methods for sqlalchemy.Column.

in_(other)

Split large “.in_” calls.

Parameters:

other (Any) – some values in list (or tuple) or select statement

Return type:

or_

Returns:

prepared or filter or just called in_

class luna_faces.db.faces_db_tools.models.faces_models.Descriptor(**kwargs)

Database table model for descriptors.

descriptor_generation

descriptor generation

Type:

int

descriptor_obtaining_method

how the descriptor was obtained

Type:

int

descriptor_version

descriptor version

Type:

int

face_id

face id, uuid.

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.Face(**kwargs)

Database table model for faces.

account_id

account uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”, to which this face belong

Type:

str

avatar

url to image, that represents the face

Type:

str

create_time

date and time of creating face

Type:

DateTime

external_id

external id of the face, if it has its own mapping in external system

Type:

str

face_id

face id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

last_update_time

date and time of last changed of the face

Type:

DateTime

user_data

client info about the face

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.List(**kwargs)

Database table model for lists.

Warning

trg_lists_deletion_log ~— after delete trigger for logging

account_id

account uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”, to which this list belong

Type:

str

create_time

date and time of creating list

Type:

DateTime

last_update_time

date and time of last changed of the list

Type:

DateTime

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

user_data

client info about the list

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.ListFace(**kwargs)

Database table model for links between faces and lists.

face_id

face id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

last_update_time

date and time of last attach/detach face to list

Type:

DateTime

link_key

number of link face to list

Type:

int

link_key_index = Index('list_id_link_key_index', Column('list_id', String(length=36), ForeignKey('list.list_id'), table=<list_face>, primary_key=True, nullable=False), Column('link_key', Integer(), table=<list_face>, nullable=False, default=Sequence('link_key', metadata=MetaData(bind=None))))

index link keys

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

matcher_load_delta_index = Index('matcher_load_delta_index', Column('link_key', Integer(), table=<list_face>, nullable=False, default=Sequence('link_key', metadata=MetaData(bind=None))))

index for load delta into matcher

class luna_faces.db.faces_db_tools.models.faces_models.ListsDeletionLog(**kwargs)

Database table model for lists deletions history

After delete trigger trg_lists_deletion_log` (table list) insert a data.

account_id

account uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”, to which this face belong

Type:

str

create_time

create time list

Type:

DateTime

deletion_id

deletion id

Type:

str

deletion_time

date list removig

Type:

DateTime

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.MV_LINK(**kwargs)

link_key

number of link face to list

Type:

int

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.MV_LINK_0(**kwargs)

link_key

number of link face to list

Type:

int

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.MV_LINK_1(**kwargs)

link_key

number of link face to list

Type:

int

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

class luna_faces.db.faces_db_tools.models.faces_models.MV_UNLINK(**kwargs)

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

unlink_key

number of link face to list

Type:

int

class luna_faces.db.faces_db_tools.models.faces_models.MV_UNLINK_0(**kwargs)

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

unlink_key

number of link face to list

Type:

int

class luna_faces.db.faces_db_tools.models.faces_models.MV_UNLINK_1(**kwargs)

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

unlink_key

number of link face to list

Type:

int

class luna_faces.db.faces_db_tools.models.faces_models.Sample(**kwargs)

Database model to store samples.

face_id

parent face id, uuid.

Type:

str

sample_id

id(uuid) of warp

Type:

str

type

enum for sample type, check ‘SampleType’ in luna_faces/utils/enums.py

Type:

int

class luna_faces.db.faces_db_tools.models.faces_models.UnlinkAttributesLog(**kwargs)

Database table model for history attach and detach attributes to lists.

face_id

face id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

link_key

number of link face to list

Type:

int

list_id

list id, uuid in format “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”

Type:

str

unlink_key

number of unlink face to list

Type:

int

unlink_key_index = Index('list_id_unlink_key_index', Column('list_id', String(length=36), ForeignKey('list.list_id'), table=<unlink_attributes_log>), Column('unlink_key', Integer(), table=<unlink_attributes_log>, primary_key=True, nullable=False, default=Sequence('unlink_key', metadata=MetaData(bind=None))))

index unlink keys

update_time

date and time of detach attributes from list

Type:

DateTime

DB matching installation

To allow matching in db, one needs to:

1. Compile a library with a VLMatch function using appropriate database development libraries.

2. Import the library in the running database and check it availability.

---

See base_scripts/database_matching/postgres/readme.md for Postgres Database:

Instruction for PostgreSQL database

You can find all the required files for the VLMatch user-defined extension (UDx) compilation in the following directory:

/var/lib/luna/current/luna-events/base_scripts/database_matching/postgres

The following instruction describes installation for PostgreSQL 12.

For VLMatch UDx function compilation one needs to:

1. Make sure, that PostgreSQL of the required version is installed and launched.

Install the required PostgreSQL development environment. You can find more information at postgres installation manual.

The llvm-toolset-7-clang is required for postgresql12-devel. Install it from the centos-release-scl-rh repository.

   yum -y install centos-release-scl-rh

   yum -y --enablerepo=centos-sclo-rh-testing install llvm-toolset-7-clang

Then install the development environment.

   sudo yum -y install postgresql12 postgresql12-server postgresql12-devel

2. Install the gcc-c++ package. The package version 4.8 or higher is required.

   yum -y install gcc-c++.x86_64
   

3. Install CMAKE. The version 3.5 or higher is required.

4. Open the make.sh script using a text editor. It includes paths to the currently used PostgreSQL version. Change the following values (if necessary):

   • SDK_HOME specifies the path to PostgreSQL home directory. The default value is /usr/pgsql-12/include/server;

   • LIB_ROOT specifies the path to PostgreSQL library root directory. The default value is /usr/pgsql-12/lib.

Go to the make.sh script directory and run it:

   cd /var/lib/luna/current/luna-events/base_scripts/database_matching/postgres/

   chmod +x make.sh

   ./make.sh

5. Define the function inside the service database:

   sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "CREATE FUNCTION VLMatch(bytea, bytea, int) RETURNS float8 AS 'VLMatchSource.so', 'VLMatch' LANGUAGE C PARALLEL SAFE;"
   

6. Test function by sending re following request to the service database:

   sudo -u postgres -h 127.0.0.1 -- psql -d luna_events -c "SELECT VLMatch('\x1234567890123456789012345678901234567890123456789012345678901234'::bytea, '\x0123456789012345678901234567890123456789012345678901234567890123'::bytea, 32);"
   

   The result returned by the database must be “0.4765625”.

7. Check that the USE_DB_MATCH_FUNCTION parameter is enabled in the service settings (/var/lib/luna/current/luna-events/luna_events/config/config.py).

---

See also base_scripts/database_matching/oracle/readme.md for Oracle Database:

For VLMatch UDx function compilation one needs to:

1. Install required environment, see requirements:

   • sudo yum install gcc g++ # needed gcc/g++ 4.8 or higher

2. Change SDK_HOME variable - oracle sdk root (default is $ORACLE_HOME/bin, check $ORACLE_HOME environment variable is set) in the makefile and run make from base_scripts/database_matching/oracle directory:

   • chmod +x make.sh

   • ./make.sh

3. Define the library and the function inside the database (from database console):

   CREATE OR REPLACE LIBRARY VLMatchSource AS '$ORACLE_HOME/bin/VLMatchSource.so';
   CREATE OR REPLACE FUNCTION VLMatch(descriptorFst IN RAW, descriptorSnd IN RAW, length IN BINARY_INTEGER)
     RETURN BINARY_FLOAT
   AS
     LANGUAGE C
     LIBRARY VLMatchSource
     NAME "VLMatch"
     PARAMETERS (descriptorFst BY REFERENCE, descriptorSnd BY REFERENCE, length UNSIGNED SHORT, RETURN FLOAT);
   

4. Test function within call (from database console):

   SELECT VLMatch(HEXTORAW('1234567890123456789012345678901234567890123456789012345678901234'), HEXTORAW('0123456789012345678901234567890123456789012345678901234567890123'), 32) FROM DUAL;
   

   • – result should be “0.4765625”