Database context

class luna_faces.db.context.DBContext(logger)

DB context

logger

request logger

async blockFaces(connection, filters, returnBlockedFaces=False)

Block some faces rows in the db, mechanics “select for update”. :rtype: tuple[int, Optional[tuple[str, …], None]] :returns: number of blocked faces and blocked faces or number of blocked faces and None, depends on the argument

async checkDbMatchFunctionExistence(saFuncCall, comparator, *, name)

Try call custom function that should exist in database. Assert expected result.

Parameters:

• saFuncCall (ColumnClause) – constructed sqlalchemy function object

• comparator (Callable[[Any], bool]) – function for checking saFuncCall result

• name (str) – function name for logging

Return type:

bool

Returns:

True - if request success execute otherwise False

async cleanListsDeletionLog(deletionTimeLt)

Clear lists deletion log :type deletionTimeLt: datetime :param deletionTimeLt: upper bound of list deletion time

Return type:

int

Returns:

count of removed rows

async cleanLog(updateTimeLt=None)

Remove notes from unlink tables.

Parameters:

updateTimeLt (Optional[datetime, None]) – lower bound of update time

Return type:

None

async classmethod closeDBContext()

Close all database contexts

async createFace(externalFaceId=None, listIds=None, attribute=None, **kwargs)

Create face.

Parameters:

• externalFaceId (Optional[str, None]) – external faceId

• listIds (Optional[set[str], None]) – luna lists

• attribute (Optional[TemporaryAttributes, None]) – temporary attribute container

Keyword Arguments:

• event_id – reference to event which created face

• user_data – face information

• account_id – id of account, required

• external_id – external id of the face, if it has its own mapping in external system

• avatar – image url that represents the face

Returns:

face id in uuid4 format

Return type:

faceId

async createList(account_id, listId, user_data='')

Create list for account.

Parameters:

• account_id (str) – account id

• user_data (Optional[str, None]) – user data

• listId (str) – list id

Return type:

None

property currentDBTimestamp: TextClause

Get current db timestamp function :rtype: TextClause :returns: function for computation current db timestamp

async deleteDescriptors(descriptorVersion, faceIdGte=None, faceIdLt=None, limit=1000)

Delete descriptors by version:

delete descriptors of preassigned version

Parameters:

• descriptorVersion (int) – descriptor version

• faceIdGte (Optional[str, None]) – lower face id including boundary

• faceIdLt (Optional[str, None]) – upper face id excluding boundary

• limit (int) – maximum attributes amount to delete

Return type:

list[dict[str, Union[str, int, list[str]]]]

Returns:

list of face ids id for deleted descriptors of preassigned version

async deleteFaceAttributes(faceId, accountId=None)

Delete face attributes.

Parameters:

• faceId (str) – face id to remove its attributes

• accountId (Optional[str, None]) – account id

Return type:

None

async deleteFaces(faces, accountId=None)

Remove faces.

Parameters:

• accountId (Optional[str, None]) – faces account id

• faces (list[str]) – faces ids

Return type:

int

Returns:

removed face count

async deleteLists(lists, accountId=None, withFaces=False)

Remove lists.

Parameters:

• accountId (Optional[str, None]) – lists account id

• lists (list[str]) – lists

• withFaces (bool) – remove lists with all faces which is contained in these lists

Return type:

int

Returns:

removed list count

async executeSearchFaces(filters, targets, page=1, pageSize=100)

Get faces searched by filters

Parameters:

• filters (SearchFacesFilters) – raw search request filters

• targets (Optional[list[str], None]) – target Face columns’ names to get info on

• page (Optional[int, None]) – page

• pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async executeSearchFacesFilteredByList(filters, targets, page=1, pageSize=100)

Get faces searched by filters associated with face-list model

Parameters:

• filters (SearchFacesFilters) – raw search request filters

• targets (list[str]) – target Face columns’ names to get info on

• page (Optional[int, None]) – page

• pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async getBasicAttrsCount(accountId=None)

Get count of basic attributes.

Parameters:

accountId (Optional[str, None]) – account id of the attributes

Return type:

int

Returns:

basic attributes count

async getDescriptorsBatchByFaceIds(facesIds, accountId=None, checkObjectExistence=True, descriptorVersion=None, receiveExternalId=None)

Get descriptors batch by faces Ids. ! Splitting into batches is used due to enormous queries (25Mb for 100k faces) that fails Oracle DB. (LUNA-3734)

Parameters:

• facesIds (set[str]) – faces ids the descriptors were attached to

• accountId (Optional[str, None]) – account id of the faces

• checkObjectExistence (bool) – check faces existence or not

• descriptorVersion (Optional[int, None]) – requested descriptor version

• receiveExternalId (Optional[bool, None]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] notFoundUuids: list[str] notExtractedUuids: list[str] externalIds: list[str]

Return type:

Dict

Raises:

VLException(Error.FacesNotFound) –

async getDescriptorsCount(accountId=None)

Get face descriptors count with filters.

Parameters:

accountId (Optional[str, None]) – account id

Return type:

list[dict[str, int]]

Returns:

list of dict. Keys of each dict are “descriptor_version” and “descriptor_count”

async getFaces(filters, targets=None, page=1, pageSize=100)

Get faces.

Parameters:

• filters (SearchFacesFilters) – raw search request filters

• targets (Optional[list[str], None]) – target Face columns’ names to get info on

• page (Optional[int, None]) – page

• pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

list of faces

async getFacesAttributeSamples(faceId, accountId)

Get all the attribute samples of the specified face

Parameters:

• faceId (list[str]) – face id

• accountId (Optional[str, None]) – account id

Return type:

list[str]

Returns:

face attribute samples list

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found –

async getFacesAttributes(faceIds, retrieveAttrs, descriptorVersion, accountId, getDescriptorAsBytes=False)

Retrieve attributes

Parameters:

• faceIds (list[str]) – face ids

• retrieveAttrs (Iterable[str]) – tuple of attributes what need retrieve

• descriptorVersion (int) – requested descriptor version

• accountId (Optional[str, None]) – account id

• getDescriptorAsBytes (bool) – whether to get descriptor as bytes otherwise as base64

Returns:

face_id: face id

attributes: target-specific face attributes dict

If some attribute data is not found it will be filled with ‘empty’ value:

None for attributes, [] for samples

Return type:

face attributes list with items with the following properties

async getFacesAttributesCount(accountId=None)

Return face attribute count

Parameters:

accountId (Optional[str, None]) – account id

Return type:

int

Returns:

face attribute count

async getFacesCount(filters)

Get count of faces.

Parameters:

filters (SearchFacesFilters) – raw search request filters

Return type:

int

Returns:

Number of faces

async getFacesCountByList(filters)

Get count of faces filtered by list id.

Parameters:

filters (SearchFacesFilters) – raw search request filters

Return type:

int

Returns:

Number of faces

async getListDeletions(deletionTimeLt=None, deletionTimeGte=None, page=1, pageSize=100)

Get lists deletion logs

Parameters:

• deletionTimeLt (Optional[datetime, None]) – upper bound of list deletion time

• deletionTimeGte (Optional[datetime, None]) – lower bound of list deletion time

• page (Optional[int, None]) – page

• pageSize (Optional[int, None]) – page size

Warning

trigger trg_lists_deletion_log inserts a data for table ListsDeletionLog

Returns:

list of deletions in the reverse order of deletion of lists

async getListFacesDescriptorsBatch(listId, linkKeyGte, limit, descriptorVersion=None, parity=None, receiveExternalId=None)

Get descriptors batch

Parameters:

• listId (str) – list id the descriptors were attached to

• linkKeyGte (int) – the lower including boundary

• limit (int) – descriptors count to return

• descriptorVersion (Optional[int, None]) – descriptor version

• parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

• receiveExternalId (Optional[bool, None]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] linkKeys: list[int] faceIds: list[str] externalIds: list[str]

Return type:

Dict

async getListMinusDelta(listId, unlinkKeyGte=None, unlinkKeyLt=None, limit=10000, parity=None)

Get history of detach attribute to lists

Parameters:

• listId – list id

• unlinkKeyLt (Optional[int, None]) – upper bound of unlink key value

• unlinkKeyGte (Optional[int, None]) – lower bound of unlink key value

• limit (int) – limit

• parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”, “unlink_key”

Return type:

List of dicts with following keys

async getListPlusDelta(listId, linkKeyGte=None, linkKeyLt=None, limit=10000, parity=None)

Get attach attributes to lists

Parameters:

• listId – list id

• linkKeyLt (Optional[int, None]) – upper bound of link key value

• linkKeyGte (Optional[int, None]) – lower bound of link key value

• limit (int) – limit

• parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”

Return type:

List of dicts with following keys

async getLists(filters, page=1, pageSize=100)

Get list

Parameters:

• filters (SearchListsFilters) – raw search request filters

• page (Optional[int, None]) – page

• pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

List with dicts

async getListsAndKeysFromMV(listIds, useParity)

Get lists with last link and unlink keys from materialized views :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys else max link/unlink keys

Return type:

list[dict[str, Any]]

Returns:

List with list ids and its link/unlink keys

async getListsCount(filters)

Count lists

Parameters:

filters (SearchListsFilters) – raw search request filters

Return type:

int

Returns:

Count of lists

async getListsWithKeys(listIds, useParity)

Get lists with last link and unlink keys from usual tables :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys, if 0 - max link/unlink keys

Return type:

list[dict[str, Any]]

Returns:

list with list ids and its link/unlink keys

async getMissingBasicAttrs(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)

Get missing basic attributes:

get attributes without basic attributes get samples

Parameters:

• accountId (Optional[str, None]) – account id of the attributes

• faceIds (Optional[list[str], None]) – list of attribute ids

• faceIdGte (Optional[str, None]) – lower face id including boundary

• faceIdLt (Optional[str, None]) – upper face id excluding boundary

• limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingBasicAttrsCount(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)

Get count of missing basic attributes.

Parameters:

• accountId (Optional[str, None]) – account id of the attributes

• faceIds (Optional[list[str], None]) – list of face ids

• faceIdGte (Optional[str, None]) – lower attribute id including boundary

• faceIdLt (Optional[str, None]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing basic attributes count

async getMissingDescriptors(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)

Get missing descriptors of ‘missingVersion’ version:

get attributes not having descriptor get samples

Parameters:

• missingVersion (int) – missing descriptor version

• accountId (Optional[str, None]) – account id of the attributes

• faceIds (Optional[list[str], None]) – list of attribute ids

• faceIdGte (Optional[str, None]) – lower face id including boundary

• faceIdLt (Optional[str, None]) – upper face id excluding boundary

• limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “generation”: <generation>, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingDescriptorsCount(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)

Get count of missing descriptors of ‘missingVersion’ version.

Parameters:

• missingVersion (int) – missing descriptor version

• accountId (Optional[str, None]) – account id of the attributes

• faceIds (Optional[list[str], None]) – list of face ids

• faceIdGte (Optional[str, None]) – lower attribute id including boundary

• faceIdLt (Optional[str, None]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing descriptors count

async getNonexistentFaceId(requiredFaceIds, accountId=None, dbConnection=None)

Get one of requiredFaceIds that not exists.

Parameters:

• requiredFaceIds (set[str]) – set of required face ids

• accountId (Optional[str, None]) – account id

• dbConnection (Optional[AbstractDBConnection, None]) – current connection. Needed not to get new connection from pool or to do some stuff in scope of the transaction.

Return type:

Optional[str, None]

Returns:

Non existing face id

async getNonexistentListId(requiredListIds, accountId=None)

Get one of requiredListIds that not exists.

Parameters:

• requiredListIds (set[str]) – set of required face ids

• accountId (Optional[str, None]) – account id

Return type:

Optional[str, None]

Returns:

Non existsing list id

async classmethod initDBContext(dbSettings, storageTime, defaultDescriptorVersion)

Initialize context :type dbSettings: DBSetting :param dbSettings: database settings :type storageTime: str :param storageTime: storage time :type defaultDescriptorVersion: int :param defaultDescriptorVersion: default descriptor version

Return type:

None

async insertFaceAttributeData(connection, faceId, attribute)

Insert face attribute data. :type connection: AbstractDBConnection :param connection: db connection :type faceId: str :param faceId: face id :type attribute: TemporaryAttributes :param attribute: temporary attribute container

Return type:

None

async isFacesExist(faceIds, accountId=None)

Checking to exist faces or not.

Parameters:

• accountId (Optional[str, None]) – account id

• faceIds (list[str]) – face ids

Return type:

bool

Returns:

True if all faces exist else false

async isListsExist(listIds, accountId=None)

Checking to exist lists or not.

Parameters:

• accountId (Optional[str, None]) – account id

• listIds (set[str]) – list ids

Return type:

bool

Returns:

True if all lists exist else false

async linkFacesToList(listId, faces)

Attach faces to list.

Parameters:

• listId (str) – list id

• faces (list[str]) – face ids

Raises:

• IntegrityError –

• Exception –

Return type:

tuple[list[str], list[str]]

Returns:

list of success link faces and failed link faces.

async ping(pingCount)

Ping database. Execute ‘SELECT 1’ from one to ‘pingCount’ times.

Parameters:

pingCount (int) – ping count

Return type:

bool

Returns:

True - if any request success execute otherwise False

static prepareListQueryFilters(filters)

Prepare search query filters on “List” models.

Parameters:

filters (SearchListsFilters) – query filters

Return type:

BooleanClauseList

Returns:

filters for List select query

static prepareSearchFacesFilters(filters, modelsFace=None)

Prepare search query filters on “Face” models.

Parameters:

• filters (SearchFacesFilters) – query filters

• modelsFace (Optional[AliasedClass, None]) – database table model for faces

Return type:

list[BooleanClauseList]

Returns:

filters for Face model

async putFaceAttributes(faceId, attribute, accountId)

Create attributes with exception wrap

Parameters:

• faceId (str) – face id

• attribute (TemporaryAttributes) – temporary attribute container

• accountId (Optional[str, None]) – account id

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found –

Return type:

None

static setOffsetAndOrderToQuery(query, order='desc', orderColumn=None, page=1, pageSize=100)

Apply OFFSET and ORDER BY criterion to the query.

Parameters:

• query (Query) – query object

• order (Optional[Literal[‘desc’, ‘asc’], None]) – result sort order (ask or desc)

• orderColumn (Optional[Column, None]) – result sort column

• page (Optional[int, None]) – pagination page value

• pageSize (Optional[int, None]) – pagination page size value, set -1 to get all faces

Return type:

Query

Returns:

updated query

async static setOracleIndexNames(logger)

Set index names for some requests with ‘use-force-index’

async unlinkFacesFromList(listId, faces, accountId=None)

Unlink faces from list.

Parameters:

• listId (str) – list id

• faces (list[str]) – face ids

• accountId (Optional[str, None]) – account id

async updateFace(faceId, accountId=None, **kwargs)

Update face.

Parameters:

• faceId (str) – face id

• accountId (Optional[str, None]) – account id

Keyword Arguments:

• user_data – face information

• event_id – reference to event which created face

• external_id – external id of the face, if it has its own mapping in external system

• avatar – image url that represents the face

Return type:

int

Returns:

updated face count

async updateFaceAttributes(faceId, attribute, accountId=None, forceUpdate=False)

Update attribute

Parameters:

• faceId (str) – updated attribute face id

• attribute (TemporaryAttributes) – temporary attributes container

• accountId (Optional[str, None]) – account id

• forceUpdate (Optional[bool, None]) – whether not to compare existing samples and new ones

Raises:

• VLException(Error.FaceSampleConflict.format(faceId), 400, False) if samples conflict –

• VLException(Error.AttributesForUpdateNotFound.format(faceId), 400, False) if attribute for update not found –

• VLException(Error.FaceNotFound.format(faceId), 404, False) if face not found –

Return type:

None

async updateListLastUpdateTime(listId)

Update last update time of lists.

Parameters:

listId (str) – list id

async updateListUserData(listId, userData, accountId=None)

Update user data of list

Parameters:

• listId (str) – list id

• userData (str) – user data

• accountId (Optional[str, None]) – account id

Return type:

int

Returns:

updated list count

class luna_faces.db.context.Reference(descriptorVersion, descriptor=None, attributeId=None, faceId=None)

Reference class.

luna_faces.db.context.getCompiledQuery(query, dbType)

Compile query to with literal_binds = True :type query: Union[Query, CompoundSelect] :param query: query made by sqlalchemy or CompoundSelect as result of union_all from n queries operation :type dbType: str :param dbType: database type

Return type:

str

Returns:

string with compiled query

luna_faces.db.context.makeOutputFaces(faces, storageTime)

Make result faces (from the database reply) proper for an user.

Parameters:

• faces (list[dict[str, Any]]) – face list

• storageTime (str) – storage time

Return type:

list[dict[str, Any]]

Returns:

faces with changed fields

luna_faces.db.context.makeOutputLists(lists, storageTime)

Make result of lists (from the database reply) proper for an user.

Parameters:

• lists (list[dict[str, Any]]) – list with lists

• storageTime (str) – storage time

Return type:

list[dict[str, Any]]

Returns:

lists with changed fields