Database context

class luna_faces.db.context.DBContext(logger)[source]

DB context

logger

request logger

async blockFaces(connection, filters, returnBlockedFaces=False)[source]

Block some faces rows in the db, mechanics “select for update”. :rtype: tuple[int, Optional[tuple[str, …], None]] :returns: number of blocked faces and blocked faces or number of blocked faces and None, depends on the argument

async checkDbMatchFunctionExistence(saFuncCall, comparator, *, name)[source]

Try call custom function that should exist in database. Assert expected result.

Parameters:
  • saFuncCall (ColumnClause) – constructed sqlalchemy function object

  • comparator (Callable[[Any], bool]) – function for checking saFuncCall result

  • name (str) – function name for logging

Return type:

bool

Returns:

True - if request success execute otherwise False

async cleanListsDeletionLog(deletionTimeLt)[source]

Clear lists deletion log :type deletionTimeLt: datetime :param deletionTimeLt: upper bound of list deletion time

Return type:

int

Returns:

count of removed rows

async cleanLog(updateTimeLt=None)[source]

Remove notes from unlink tables.

Parameters:

updateTimeLt (Optional[datetime, None]) – lower bound of update time

Return type:

None

async classmethod closeDBContext()[source]

Close all database contexts

async createFace(externalFaceId=None, listIds=None, attribute=None, **kwargs)[source]

Create face.

Parameters:
  • externalFaceId (Optional[str, None]) – external faceId

  • listIds (Optional[set[str], None]) – luna lists

  • attribute (Optional[TemporaryAttributes, None]) – temporary attribute container

Keyword Arguments:
  • event_id – reference to event which created face

  • user_data – face information

  • account_id – id of account, required

  • external_id – external id of the face, if it has its own mapping in external system

  • avatar – image url that represents the face

Returns:

face id in uuid4 format

Return type:

faceId

async createList(account_id, listId, user_data='')[source]

Create list for account.

Parameters:
  • account_id (str) – account id

  • user_data (Optional[str, None]) – user data

  • listId (str) – list id

Return type:

None

property currentDBTimestamp: TextClause

Get current db timestamp function :rtype: TextClause :returns: function for computation current db timestamp

async deleteDescriptors(descriptorVersion, faceIdGte=None, faceIdLt=None, limit=1000)[source]
Delete descriptors by version:

delete descriptors of preassigned version

Parameters:
  • descriptorVersion (int) – descriptor version

  • faceIdGte (Optional[str, None]) – lower face id including boundary

  • faceIdLt (Optional[str, None]) – upper face id excluding boundary

  • limit (int) – maximum attributes amount to delete

Return type:

list[dict[str, Union[str, int, list[str]]]]

Returns:

list of face ids id for deleted descriptors of preassigned version

async deleteFaceAttributes(faceId, accountId=None)[source]

Delete face attributes.

Parameters:
  • faceId (str) – face id to remove its attributes

  • accountId (Optional[str, None]) – account id

Return type:

None

async deleteFaces(faces, accountId=None)[source]

Remove faces.

Parameters:
  • accountId (Optional[str, None]) – faces account id

  • faces (list[str]) – faces ids

Return type:

int

Returns:

removed face count

async deleteLists(lists, accountId=None, withFaces=False)[source]

Remove lists.

Parameters:
  • accountId (Optional[str, None]) – lists account id

  • lists (list[str]) – lists

  • withFaces (bool) – remove lists with all faces which is contained in these lists

Return type:

int

Returns:

removed list count

async executeSearchFaces(filters, targets, page=1, pageSize=100)[source]

Get faces searched by filters

Parameters:
  • filters (SearchFacesFilters) – raw search request filters

  • targets (Optional[list[str], None]) – target Face columns’ names to get info on

  • page (Optional[int, None]) – page

  • pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async executeSearchFacesFilteredByList(filters, targets, page=1, pageSize=100)[source]

Get faces searched by filters associated with face-list model

Parameters:
  • filters (SearchFacesFilters) – raw search request filters

  • targets (list[str]) – target Face columns’ names to get info on

  • page (Optional[int, None]) – page

  • pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async getBasicAttrsCount(accountId=None)[source]

Get count of basic attributes.

Parameters:

accountId (Optional[str, None]) – account id of the attributes

Return type:

int

Returns:

basic attributes count

async getDescriptorsBatchByFaceIds(facesIds, accountId=None, checkObjectExistence=True, descriptorVersion=None, receiveExternalId=None)[source]

Get descriptors batch by faces Ids. ! Splitting into batches is used due to enormous queries (25Mb for 100k faces) that fails Oracle DB. (LUNA-3734)

Parameters:
  • facesIds (set[str]) – faces ids the descriptors were attached to

  • accountId (Optional[str, None]) – account id of the faces

  • checkObjectExistence (bool) – check faces existence or not

  • descriptorVersion (Optional[int, None]) – requested descriptor version

  • receiveExternalId (Optional[bool, None]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] notFoundUuids: list[str] notExtractedUuids: list[str] externalIds: list[str]

Return type:

Dict

Raises:

VLException(Error.FacesNotFound)

async getDescriptorsCount(accountId=None)[source]

Get face descriptors count with filters.

Parameters:

accountId (Optional[str, None]) – account id

Return type:

list[dict[str, int]]

Returns:

list of dict. Keys of each dict are “descriptor_version” and “descriptor_count”

async getFaces(filters, targets=None, page=1, pageSize=100)[source]

Get faces.

Parameters:
  • filters (SearchFacesFilters) – raw search request filters

  • targets (Optional[list[str], None]) – target Face columns’ names to get info on

  • page (Optional[int, None]) – page

  • pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

list of faces

async getFacesAttributeSamples(faceId, accountId)[source]

Get all the attribute samples of the specified face

Parameters:
  • faceId (list[str]) – face id

  • accountId (Optional[str, None]) – account id

Return type:

list[str]

Returns:

face attribute samples list

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found

async getFacesAttributes(faceIds, retrieveAttrs, descriptorVersion, accountId, getDescriptorAsBytes=False)[source]

Retrieve attributes

Parameters:
  • faceIds (list[str]) – face ids

  • retrieveAttrs (Iterable[str]) – tuple of attributes what need retrieve

  • descriptorVersion (int) – requested descriptor version

  • accountId (Optional[str, None]) – account id

  • getDescriptorAsBytes (bool) – whether to get descriptor as bytes otherwise as base64

Returns:

face_id: face id

attributes: target-specific face attributes dict

If some attribute data is not found it will be filled with ‘empty’ value:

None for attributes, [] for samples

Return type:

face attributes list with items with the following properties

async getFacesAttributesCount(accountId=None)[source]

Return face attribute count

Parameters:

accountId (Optional[str, None]) – account id

Return type:

int

Returns:

face attribute count

async getFacesCount(filters)[source]

Get count of faces.

Parameters:

filters (SearchFacesFilters) – raw search request filters

Return type:

int

Returns:

Number of faces

async getFacesCountByList(filters)[source]

Get count of faces filtered by list id.

Parameters:

filters (SearchFacesFilters) – raw search request filters

Return type:

int

Returns:

Number of faces

async getListDeletions(deletionTimeLt=None, deletionTimeGte=None, page=1, pageSize=100)[source]

Get lists deletion logs

Parameters:
  • deletionTimeLt (Optional[datetime, None]) – upper bound of list deletion time

  • deletionTimeGte (Optional[datetime, None]) – lower bound of list deletion time

  • page (Optional[int, None]) – page

  • pageSize (Optional[int, None]) – page size

Warning

trigger trg_lists_deletion_log inserts a data for table ListsDeletionLog

Returns:

list of deletions in the reverse order of deletion of lists

async getListFacesDescriptorsBatch(listId, linkKeyGte, limit, descriptorVersion=None, parity=None, receiveExternalId=None)[source]

Get descriptors batch

Parameters:
  • listId (str) – list id the descriptors were attached to

  • linkKeyGte (int) – the lower including boundary

  • limit (int) – descriptors count to return

  • descriptorVersion (Optional[int, None]) – descriptor version

  • parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

  • receiveExternalId (Optional[bool, None]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] linkKeys: list[int] faceIds: list[str] externalIds: list[str]

Return type:

Dict

async getListMinusDelta(listId, unlinkKeyGte=None, unlinkKeyLt=None, limit=10000, parity=None)[source]

Get history of detach attribute to lists

Parameters:
  • listId – list id

  • unlinkKeyLt (Optional[int, None]) – upper bound of unlink key value

  • unlinkKeyGte (Optional[int, None]) – lower bound of unlink key value

  • limit (int) – limit

  • parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”, “unlink_key”

Return type:

List of dicts with following keys

async getListPlusDelta(listId, linkKeyGte=None, linkKeyLt=None, limit=10000, parity=None)[source]

Get attach attributes to lists

Parameters:
  • listId – list id

  • linkKeyLt (Optional[int, None]) – upper bound of link key value

  • linkKeyGte (Optional[int, None]) – lower bound of link key value

  • limit (int) – limit

  • parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”

Return type:

List of dicts with following keys

async getLists(filters, page=1, pageSize=100)[source]

Get list

Parameters:
  • filters (SearchListsFilters) – raw search request filters

  • page (Optional[int, None]) – page

  • pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

List with dicts

async getListsAndKeysFromMV(listIds, useParity)[source]

Get lists with last link and unlink keys from materialized views :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys else max link/unlink keys

Return type:

list[dict[str, Any]]

Returns:

List with list ids and its link/unlink keys

async getListsCount(filters)[source]

Count lists

Parameters:

filters (SearchListsFilters) – raw search request filters

Return type:

int

Returns:

Count of lists

async getListsWithKeys(listIds, useParity)[source]

Get lists with last link and unlink keys from usual tables :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys, if 0 - max link/unlink keys

Return type:

list[dict[str, Any]]

Returns:

list with list ids and its link/unlink keys

async getMissingBasicAttrs(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)[source]
Get missing basic attributes:

get attributes without basic attributes get samples

Parameters:
  • accountId (Optional[str, None]) – account id of the attributes

  • faceIds (Optional[list[str], None]) – list of attribute ids

  • faceIdGte (Optional[str, None]) – lower face id including boundary

  • faceIdLt (Optional[str, None]) – upper face id excluding boundary

  • limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingBasicAttrsCount(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)[source]

Get count of missing basic attributes.

Parameters:
  • accountId (Optional[str, None]) – account id of the attributes

  • faceIds (Optional[list[str], None]) – list of face ids

  • faceIdGte (Optional[str, None]) – lower attribute id including boundary

  • faceIdLt (Optional[str, None]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing basic attributes count

async getMissingDescriptors(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)[source]
Get missing descriptors of ‘missingVersion’ version:

get attributes not having descriptor get samples

Parameters:
  • missingVersion (int) – missing descriptor version

  • accountId (Optional[str, None]) – account id of the attributes

  • faceIds (Optional[list[str], None]) – list of attribute ids

  • faceIdGte (Optional[str, None]) – lower face id including boundary

  • faceIdLt (Optional[str, None]) – upper face id excluding boundary

  • limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “generation”: <generation>, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingDescriptorsCount(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)[source]

Get count of missing descriptors of ‘missingVersion’ version.

Parameters:
  • missingVersion (int) – missing descriptor version

  • accountId (Optional[str, None]) – account id of the attributes

  • faceIds (Optional[list[str], None]) – list of face ids

  • faceIdGte (Optional[str, None]) – lower attribute id including boundary

  • faceIdLt (Optional[str, None]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing descriptors count

async getNonexistentFaceId(requiredFaceIds, accountId=None, dbConnection=None)[source]

Get one of requiredFaceIds that not exists.

Parameters:
  • requiredFaceIds (set[str]) – set of required face ids

  • accountId (Optional[str, None]) – account id

  • dbConnection (Optional[AbstractDBConnection, None]) – current connection. Needed not to get new connection from pool or to do some stuff in scope of the transaction.

Return type:

Optional[str, None]

Returns:

Non existing face id

async getNonexistentListId(requiredListIds, accountId=None)[source]

Get one of requiredListIds that not exists.

Parameters:
  • requiredListIds (set[str]) – set of required face ids

  • accountId (Optional[str, None]) – account id

Return type:

Optional[str, None]

Returns:

Non existsing list id

async classmethod initDBContext(dbSettings, storageTime, defaultDescriptorVersion)[source]

Initialize context :type dbSettings: DBSetting :param dbSettings: database settings :type storageTime: str :param storageTime: storage time :type defaultDescriptorVersion: int :param defaultDescriptorVersion: default descriptor version

Return type:

None

async insertFaceAttributeData(connection, faceId, attribute)[source]

Insert face attribute data. :type connection: AbstractDBConnection :param connection: db connection :type faceId: str :param faceId: face id :type attribute: TemporaryAttributes :param attribute: temporary attribute container

Return type:

None

async isFacesExist(faceIds, accountId=None)[source]

Checking to exist faces or not.

Parameters:
  • accountId (Optional[str, None]) – account id

  • faceIds (list[str]) – face ids

Return type:

bool

Returns:

True if all faces exist else false

async isListsExist(listIds, accountId=None)[source]

Checking to exist lists or not.

Parameters:
  • accountId (Optional[str, None]) – account id

  • listIds (set[str]) – list ids

Return type:

bool

Returns:

True if all lists exist else false

async linkFacesToList(listId, faces)[source]

Attach faces to list.

Parameters:
  • listId (str) – list id

  • faces (list[str]) – face ids

Raises:
  • IntegrityError

  • Exception

Return type:

tuple[list[str], list[str]]

Returns:

list of success link faces and failed link faces.

async ping(pingCount)[source]

Ping database. Execute ‘SELECT 1’ from one to ‘pingCount’ times.

Parameters:

pingCount (int) – ping count

Return type:

bool

Returns:

True - if any request success execute otherwise False

static prepareListQueryFilters(filters)[source]

Prepare search query filters on “List” models.

Parameters:

filters (SearchListsFilters) – query filters

Return type:

BooleanClauseList

Returns:

filters for List select query

static prepareSearchFacesFilters(filters, modelsFace=None)[source]

Prepare search query filters on “Face” models.

Parameters:
  • filters (SearchFacesFilters) – query filters

  • modelsFace (Optional[AliasedClass, None]) – database table model for faces

Return type:

list[BooleanClauseList]

Returns:

filters for Face model

async putFaceAttributes(faceId, attribute, accountId)[source]

Create attributes with exception wrap

Parameters:
  • faceId (str) – face id

  • attribute (TemporaryAttributes) – temporary attribute container

  • accountId (Optional[str, None]) – account id

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found

Return type:

None

static setOffsetAndOrderToQuery(query, order='desc', orderColumn=None, page=1, pageSize=100)[source]

Apply OFFSET and ORDER BY criterion to the query.

Parameters:
  • query (Query) – query object

  • order (Optional[Literal[‘desc’, ‘asc’], None]) – result sort order (ask or desc)

  • orderColumn (Optional[Column, None]) – result sort column

  • page (Optional[int, None]) – pagination page value

  • pageSize (Optional[int, None]) – pagination page size value, set -1 to get all faces

Return type:

Query

Returns:

updated query

async static setOracleIndexNames(logger)[source]

Set index names for some requests with ‘use-force-index’

async unlinkFacesFromList(listId, faces, accountId=None)[source]

Unlink faces from list.

Parameters:
  • listId (str) – list id

  • faces (list[str]) – face ids

  • accountId (Optional[str, None]) – account id

async updateFace(faceId, accountId=None, **kwargs)[source]

Update face.

Parameters:
  • faceId (str) – face id

  • accountId (Optional[str, None]) – account id

Keyword Arguments:
  • user_data – face information

  • event_id – reference to event which created face

  • external_id – external id of the face, if it has its own mapping in external system

  • avatar – image url that represents the face

Return type:

int

Returns:

updated face count

async updateFaceAttributes(faceId, attribute, accountId=None, forceUpdate=False)[source]

Update attribute

Parameters:
  • faceId (str) – updated attribute face id

  • attribute (TemporaryAttributes) – temporary attributes container

  • accountId (Optional[str, None]) – account id

  • forceUpdate (Optional[bool, None]) – whether not to compare existing samples and new ones

Raises:
  • VLException(Error.FaceSampleConflict.format(faceId), 400, False) if samples conflict

  • VLException(Error.AttributesForUpdateNotFound.format(faceId), 400, False) if attribute for update not found

  • VLException(Error.FaceNotFound.format(faceId), 404, False) if face not found

Return type:

None

async updateListLastUpdateTime(listId)[source]

Update last update time of lists.

Parameters:

listId (str) – list id

async updateListUserData(listId, userData, accountId=None)[source]

Update user data of list

Parameters:
  • listId (str) – list id

  • userData (str) – user data

  • accountId (Optional[str, None]) – account id

Return type:

int

Returns:

updated list count

class luna_faces.db.context.Reference(descriptorVersion, descriptor=None, attributeId=None, faceId=None)[source]

Reference class.

luna_faces.db.context.getCompiledQuery(query, dbType)[source]

Compile query to with literal_binds = True :type query: Union[Query, CompoundSelect] :param query: query made by sqlalchemy or CompoundSelect as result of union_all from n queries operation :type dbType: str :param dbType: database type

Return type:

str

Returns:

string with compiled query

luna_faces.db.context.makeOutputFaces(faces, storageTime)[source]

Make result faces (from the database reply) proper for an user.

Parameters:
  • faces (list[dict[str, Any]]) – face list

  • storageTime (str) – storage time

Return type:

list[dict[str, Any]]

Returns:

faces with changed fields

luna_faces.db.context.makeOutputLists(lists, storageTime)[source]

Make result of lists (from the database reply) proper for an user.

Parameters:
  • lists (list[dict[str, Any]]) – list with lists

  • storageTime (str) – storage time

Return type:

list[dict[str, Any]]

Returns:

lists with changed fields