Database context

class luna_faces.db.context.DBContext(dbSettings, storageTime, defaultDescriptorVersion, encryptionCtx, databaseNumber, useMaterialViews)

DB context

logger

request logger

async blockFaces(connection, filters, returnBlockedFaces=False)

Block some faces rows in the db, mechanics “select for update”. :rtype: tuple[int, Optional[tuple[str, ...]]] :returns: number of blocked faces and blocked faces or number of blocked faces and None, depends on the argument

async checkDbFunctionExistence(saFuncCall)

Try call custom function to check wheather it exists in database.

Parameters:

saFuncCall – constructed sqlalchemy function object

Return type:

bool

Returns:

True - if request success execute otherwise False

async cleanListsDeletionLog(deletionTimeLt)

Clear lists deletion log :type deletionTimeLt: datetime :param deletionTimeLt: upper bound of list deletion time

Return type:

int

Returns:

count of removed rows

async cleanLog(updateTimeLt=None)

Remove notes from unlink tables.

Parameters:

updateTimeLt (Optional[datetime]) – lower bound of update time

Return type:

None

async close()

Close all database contexts

async createFace(externalFaceId=None, listIds=None, attribute=None, **kwargs)

Create face.

Parameters:

• externalFaceId (Optional[str]) – external faceId

• listIds (Optional[set[str]]) – luna lists

• attribute (Optional[TemporaryAttributes]) – temporary attribute container

Keyword Arguments:

• event_id – reference to event which created face

• user_data – face information

• account_id – id of account, required

• external_id – external id of the face, if it has its own mapping in external system

• avatar – image url that represents the face

Returns:

face id in uuid4 format

Return type:

faceId

async createList(account_id, listId, user_data='')

Create list for account.

Parameters:

• account_id (str) – account id

• user_data (Optional[str]) – user data

• listId (str) – list id

Return type:

None

property currentDBTimestamp: TextClause

Get current db timestamp function :returns: function for computation current db timestamp

async deleteDescriptors(descriptorVersion, faceIdGte=None, faceIdLt=None, limit=1000)

Delete descriptors by version:

delete descriptors of preassigned version

Parameters:

• descriptorVersion (int) – descriptor version

• faceIdGte (Optional[str]) – lower face id including boundary

• faceIdLt (Optional[str]) – upper face id excluding boundary

• limit (int) – maximum attributes amount to delete

Return type:

list[dict[str, Union[str, int, list[str]]]]

Returns:

list of face ids id for deleted descriptors of preassigned version

async deleteFaceAttributes(faceId, accountId=None)

Delete face attributes.

Parameters:

• faceId (str) – face id to remove its attributes

• accountId (Optional[str]) – account id

Return type:

None

async deleteFaces(faces=None, accountId=None, createTimeGte=None, createTimeLt=None, userData=None, listId=None)

Remove faces.

Parameters:

• accountId (str | None) – faces account id

• faces (list[str] | None) – faces ids

• createTimeGte (str | None) – creation time greater

• createTimeLt (str | None) – create time lower

• userData (str | None) – user data

• listId (str | None) – list id where face exists

Return type:

int

Returns:

removed face count

async deleteFacesExtended(targets, accountId=None, createTimeGte=None, createTimeLt=None, userData=None, listId=None)

Remove faces.

Parameters:

• targets (list[str]) – deletion info targets

• accountId (str | None) – faces account id

• createTimeGte (str | None) – creation time greater

• createTimeLt (str | None) – create time lower

• userData (str | None) – user data

• listId (str | None) – list id where face exists

Return type:

list[dict] | None

Returns:

list with dicts for each deleted face.

async deleteLists(lists, accountId=None, withFaces=False)

Remove lists.

Parameters:

• accountId (Optional[str]) – lists account id

• lists (list[str]) – lists

• withFaces (bool) – remove lists with all faces which is contained in these lists

Return type:

int

Returns:

removed list count

async executeSearchFaces(filters, targets, page=1, pageSize=100)

Get faces searched by filters

Parameters:

• filters (SearchFacesFilters) – raw search request filters

• targets (Optional[list[str]]) – target Face columns’ names to get info on

• page (Optional[int]) – page

• pageSize (Optional[int]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async executeSearchFacesFilteredByList(filters, targets, page=1, pageSize=100)

Get faces searched by filters associated with face-list model

Parameters:

• filters (SearchFacesFilters) – raw search request filters

• targets (list[str]) – target Face columns’ names to get info on

• page (Optional[int]) – page

• pageSize (Optional[int]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async getBasicAttrsCount(accountId=None)

Get count of basic attributes.

Parameters:

accountId (Optional[str]) – account id of the attributes

Return type:

int

Returns:

basic attributes count

async getCachedFaceAttributesCount()

Get cached count of faces attributes.

Return type:

int

Returns:

Number of faces attributes

async getDescriptorsBatchByFaceIds(facesIds, accountId=None, checkObjectExistence=True, descriptorVersion=None, receiveExternalId=None)

Get descriptors batch by faces Ids. ! Splitting into batches is used due to enormous queries (25Mb for 100k faces) that fails Oracle DB. (LUNA-3734)

Parameters:

• facesIds (set[str]) – faces ids the descriptors were attached to

• accountId (Optional[str]) – account id of the faces

• checkObjectExistence (bool) – check faces existence or not

• descriptorVersion (Optional[int]) – requested descriptor version

• receiveExternalId (Optional[bool]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] notFoundUuids: list[str] notExtractedUuids: list[str] externalIds: list[str]

Return type:

Dict

Raises:

VLException(Error.FacesNotFound) –

async getDescriptorsCount(accountId=None)

Get face descriptors count with filters.

Parameters:

accountId (Optional[str]) – account id

Return type:

list[dict[str, int]]

Returns:

list of dict. Keys of each dict are “descriptor_version” and “descriptor_count”

async getFaces(filters, targets=None, page=1, pageSize=100)

Get faces.

Parameters:

• filters (SearchFacesFilters) – raw search request filters

• targets (Optional[list[str]]) – target Face columns’ names to get info on

• page (Optional[int]) – page

• pageSize (Optional[int]) – page size

Return type:

list[dict[str, str]]

Returns:

list of faces

async getFacesAttributeSamples(faceId, accountId)

Get all the attribute samples of the specified face

Parameters:

• faceId (list[str]) – face id

• accountId (Optional[str]) – account id

Return type:

list[str]

Returns:

face attribute samples list

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found –

async getFacesAttributes(faceIds, retrieveAttrs, descriptorVersion, accountId, getDescriptorAsBytes=False, descriptorFormat='raw')

Retrieve attributes

Parameters:

• faceIds (list[str]) – face ids

• retrieveAttrs (Iterable[str]) – tuple of attributes what need retrieve

• descriptorVersion (int) – requested descriptor version

• accountId (Optional[str]) – account id

• getDescriptorAsBytes (bool) – whether to get descriptor as bytes otherwise as base64

• descriptorFormat (Literal['raw', 'sdk']) – output descriptor format

Returns:

face_id: face id

attributes: target-specific face attributes dict

If some attribute data is not found it will be filled with ‘empty’ value:

None for attributes, [] for samples

Return type:

face attributes list with items with the following properties

async getFacesAttributesCount(accountId=None)

Return face attribute count

Parameters:

accountId (Optional[str]) – account id

Return type:

int

Returns:

face attribute count

async getFacesCount(filters)

Get count of faces.

Parameters:

filters (SearchFacesFilters) – raw search request filters

Return type:

int

Returns:

Number of faces

async getFacesCountByList(filters)

Get count of faces filtered by list id.

Parameters:

filters (SearchFacesFilters) – raw search request filters

Return type:

int

Returns:

Number of faces

async getListDeletions(deletionTimeLt=None, deletionTimeGte=None, page=1, pageSize=100)

Get lists deletion logs

Parameters:

• deletionTimeLt (Optional[datetime]) – upper bound of list deletion time

• deletionTimeGte (Optional[datetime]) – lower bound of list deletion time

• page (Optional[int]) – page

• pageSize (Optional[int]) – page size

Warning

trigger trg_lists_deletion_log inserts a data for table ListsDeletionLog

Returns:

list of deletions in the reverse order of deletion of lists

async getListFacesDescriptorsBatch(listId, linkKeyGte, limit, descriptorVersion=None, parity=None, receiveExternalId=None)

Get descriptors batch

Parameters:

• listId (str) – list id the descriptors were attached to

• linkKeyGte (int) – the lower including boundary

• limit (int) – descriptors count to return

• descriptorVersion (Optional[int]) – descriptor version

• parity (Optional[int]) – 0 for odd or 1 for even link keys to search for

• receiveExternalId (Optional[bool]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] linkKeys: list[int] faceIds: list[str] externalIds: list[str]

Return type:

Dict

async getListMinusDelta(listId, unlinkKeyGte=None, unlinkKeyLt=None, limit=10000, parity=None)

Get history of detach attribute to lists

Parameters:

• listId – list id

• unlinkKeyLt (Optional[int]) – upper bound of unlink key value

• unlinkKeyGte (Optional[int]) – lower bound of unlink key value

• limit (int) – limit

• parity (Optional[int]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”, “unlink_key”

Return type:

List of dicts with following keys

async getListPlusDelta(listId, linkKeyGte=None, linkKeyLt=None, limit=10000, parity=None)

Get attach attributes to lists

Parameters:

• listId – list id

• linkKeyLt (Optional[int]) – upper bound of link key value

• linkKeyGte (Optional[int]) – lower bound of link key value

• limit (int) – limit

• parity (Optional[int]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”

Return type:

List of dicts with following keys

async getLists(filters, page=1, pageSize=100)

Get list

Parameters:

• filters (SearchListsFilters) – raw search request filters

• page (Optional[int]) – page

• pageSize (Optional[int]) – page size

Return type:

list[dict[str, str]]

Returns:

List with dicts

async getListsAndKeysFromMV(listIds, useParity)

Get lists with last link and unlink keys from materialized views :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys else max link/unlink keys

Return type:

list[dict[str, Any]]

Returns:

List with list ids and its link/unlink keys

async getListsCount(filters)

Count lists

Parameters:

filters (SearchListsFilters) – raw search request filters

Return type:

int

Returns:

Count of lists

async getListsWithKeys(listIds, useParity)

Get lists with last link and unlink keys from usual tables :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys, if 0 - max link/unlink keys

Return type:

list[dict[str, Any]]

Returns:

list with list ids and its link/unlink keys

async getMissingBasicAttrs(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)

Get missing basic attributes:

get attributes without basic attributes get samples

Parameters:

• accountId (Optional[str]) – account id of the attributes

• faceIds (Optional[list[str]]) – list of attribute ids

• faceIdGte (Optional[str]) – lower face id including boundary

• faceIdLt (Optional[str]) – upper face id excluding boundary

• limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingBasicAttrsCount(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)

Get count of missing basic attributes.

Parameters:

• accountId (Optional[str]) – account id of the attributes

• faceIds (Optional[list[str]]) – list of face ids

• faceIdGte (Optional[str]) – lower attribute id including boundary

• faceIdLt (Optional[str]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing basic attributes count

async getMissingDescriptors(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)

Get missing descriptors of ‘missingVersion’ version:

get attributes not having descriptor get samples

Parameters:

• missingVersion (int) – missing descriptor version

• accountId (Optional[str]) – account id of the attributes

• faceIds (Optional[list[str]]) – list of attribute ids

• faceIdGte (Optional[str]) – lower face id including boundary

• faceIdLt (Optional[str]) – upper face id excluding boundary

• limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “generation”: <generation>, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingDescriptorsCount(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)

Get count of missing descriptors of ‘missingVersion’ version.

Parameters:

• missingVersion (int) – missing descriptor version

• accountId (Optional[str]) – account id of the attributes

• faceIds (Optional[list[str]]) – list of face ids

• faceIdGte (Optional[str]) – lower attribute id including boundary

• faceIdLt (Optional[str]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing descriptors count

async getNonexistentFaceId(requiredFaceIds, accountId=None, dbConnection=None)

Get one of requiredFaceIds that not exists.

Parameters:

• requiredFaceIds (set[str]) – set of required face ids

• accountId (Optional[str]) – account id

• dbConnection (Optional[AbstractDBConnection]) – current connection. Needed not to get new connection from pool or to do some stuff in scope of the transaction.

Return type:

Optional[str]

Returns:

Non existing face id

async getNonexistentListId(requiredListIds, accountId=None)

Get one of requiredListIds that not exists.

Parameters:

• requiredListIds (set[str]) – set of required face ids

• accountId (Optional[str]) – account id

Return type:

Optional[str]

Returns:

Non existsing list id

getQuery4Binary(column)

Generate query to select binary field

getRuntimeChecks(_)

Checks for healthcheck.

Return type:

list[tuple[str, Awaitable]]

async initialize()

Async __init__.

async insertFaceAttributeData(connection, faceId, attribute)

Insert face attribute data. :type connection: AbstractDBConnection :param connection: db connection :type faceId: str :param faceId: face id :type attribute: TemporaryAttributes :param attribute: temporary attribute container

Return type:

None

async isFacesExist(faceIds, accountId=None)

Checking to exist faces or not.

Parameters:

• accountId (Optional[str]) – account id

• faceIds (list[str]) – face ids

Return type:

bool

Returns:

True if all faces exist else false

async isListsExist(listIds, accountId=None)

Checking to exist lists or not.

Parameters:

• accountId (Optional[str]) – account id

• listIds (set[str]) – list ids

Return type:

bool

Returns:

True if all lists exist else false

async linkFacesToList(listId, faces)

Attach faces to list.

Parameters:

• listId (str) – list id

• faces (list[str]) – face ids

Raises:

• IntegrityError –

• Exception –

Return type:

tuple[list[str], list[str]]

Returns:

list of success link faces and failed link faces.

makeOutputFaces(faceRows, columns)

Make result faces (from the database reply) proper for an user.

Parameters:

• faceRows (list[Record | dict]) – face list from db

• columns (Iterable[str]) – selected columns

Return type:

list[dict[str, Any]]

Returns:

faces with changed fields

makeOutputLists(lists)

Make result of lists (from the database reply) proper for an user.

Parameters:

lists (list[Record | dict]) – list with lists

Return type:

list[dict[str, Any]]

Returns:

lists with changed fields

async ping(pingCount)

Ping database. Execute ‘SELECT 1’ from one to ‘pingCount’ times.

Parameters:

pingCount (int) – ping count

Return type:

bool

Returns:

True - if any request success execute otherwise False

static prepareListQueryFilters(filters)

Prepare search query filters on “List” models.

Parameters:

filters (SearchListsFilters) – query filters

Return type:

BooleanClauseList

Returns:

filters for List select query

static prepareSearchFacesFilters(filters, modelsFace=None, modelsListFace=None)

Prepare search query filters on “Face” models.

Parameters:

• filters (SearchFacesFilters) – query filters

• modelsFace (Optional[AliasedClass]) – database table model for faces

• modelsListFace (Optional[AliasedClass]) – database table model for list face

Return type:

list[BooleanClauseList]

Returns:

filters for Face model

async probe()

Ensure provided config is valid. Create new connection. Can be used without initialization

Return type:

bool

async putFaceAttributes(faceId, attribute, accountId)

Create attributes with exception wrap

Parameters:

• faceId (str) – face id

• attribute (TemporaryAttributes) – temporary attribute container

• accountId (Optional[str]) – account id

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found –

Return type:

None

static setOffsetAndOrderToQuery(query, order='desc', orderColumn=None, page=1, pageSize=100)

Apply OFFSET and ORDER BY criterion to the query.

Parameters:

• query (Query) – query object

• order (Optional[Literal['desc', 'asc']]) – result sort order (ask or desc)

• orderColumn (Optional[Column]) – result sort column

• page (Optional[int]) – pagination page value

• pageSize (Optional[int]) – pagination page size value, set -1 to get all faces

Return type:

Query

Returns:

updated query

async setOracleIndexNames()

Set index names for some requests with ‘use-force-index’

async unlinkFacesFromList(listId, faces, accountId=None)

Unlink faces from list.

Parameters:

• listId (str) – list id

• faces (list[str]) – face ids

• accountId (Optional[str]) – account id

async updateFace(faceId, accountId=None, **kwargs)

Update face.

Parameters:

• faceId (str) – face id

• accountId (Optional[str]) – account id

Keyword Arguments:

• user_data – face information

• event_id – reference to event which created face

• external_id – external id of the face, if it has its own mapping in external system

• avatar – image url that represents the face

Return type:

int

Returns:

updated face count

async updateFaceAttributes(faceId, attribute, accountId=None, forceUpdate=False)

Update attribute

Parameters:

• faceId (str) – updated attribute face id

• attribute (TemporaryAttributes) – temporary attributes container

• accountId (Optional[str]) – account id

• forceUpdate (Optional[bool]) – whether not to compare existing samples and new ones

Raises:

• VLException(Error.FaceSampleConflict.format(faceId), 400, False) if samples conflict –

• VLException(Error.AttributesForUpdateNotFound.format(faceId), 400, False) if attribute for update not found –

• VLException(Error.FaceNotFound.format(faceId), 404, False) if face not found –

Return type:

None

async updateListLastUpdateTime(listId)

Update last update time of lists.

Parameters:

listId (str) – list id

async updateListUserData(listId, userData, accountId=None)

Update user data of list

Parameters:

• listId (str) – list id

• userData (str) – user data

• accountId (Optional[str]) – account id

Return type:

int

Returns:

updated list count

class luna_faces.db.context.Reference(descriptorVersion, descriptor=None, attributeId=None, faceId=None)

Reference class.

luna_faces.db.context.getCompiledQuery(query, dbType)

Compile query to with literal_binds = True :type query: Union[Query, CompoundSelect] :param query: query made by sqlalchemy or CompoundSelect as result of union_all from n queries operation :type dbType: str :param dbType: database type

Return type:

str

Returns:

string with compiled query