Database context¶

class luna_faces.db.context.DBContext(logger)[source]¶

DB context

logger¶: request logger

async blockFaces(connection, filters, returnBlockedFaces=False)[source]¶: Block some faces rows in the db, mechanics “select for update”. :rtype: tuple[int, Optional[tuple[str, …], None]] :returns: number of blocked faces and blocked faces or number of blocked faces and None, depends on the argument

async checkDbMatchFunctionExistence(saFuncCall, comparator, *, name)[source]¶

Try call custom function that should exist in database. Assert expected result.

Parameters:

saFuncCall (ColumnClause) – constructed sqlalchemy function object
comparator (Callable[[Any], bool]) – function for checking saFuncCall result
name (str) – function name for logging

Return type:

bool

Returns:

True - if request success execute otherwise False

async cleanListsDeletionLog(deletionTimeLt)[source]¶

Clear lists deletion log :type deletionTimeLt: datetime :param deletionTimeLt: upper bound of list deletion time

Return type:: int
Returns:: count of removed rows

async cleanLog(updateTimeLt=None)[source]¶

Remove notes from unlink tables.

Parameters:: updateTimeLt (Optional[datetime, None]) – lower bound of update time
Return type:: None

async classmethod closeDBContext()[source]¶: Close all database contexts

async createFace(externalFaceId=None, listIds=None, attribute=None, **kwargs)[source]¶

Create face.

Parameters:

externalFaceId (Optional[str, None]) – external faceId
listIds (Optional[set[str], None]) – luna lists
attribute (Optional[TemporaryAttributes, None]) – temporary attribute container

Keyword Arguments:

event_id – reference to event which created face
user_data – face information
account_id – id of account, required
external_id – external id of the face, if it has its own mapping in external system
avatar – image url that represents the face

Returns:

face id in uuid4 format

Return type:

faceId

async createList(account_id, listId, user_data='')[source]¶

Create list for account.

Parameters:

account_id (str) – account id
user_data (Optional[str, None]) – user data
listId (str) – list id

Return type:

None

property currentDBTimestamp: TextClause¶: Get current db timestamp function :rtype: TextClause :returns: function for computation current db timestamp

async deleteDescriptors(descriptorVersion, faceIdGte=None, faceIdLt=None, limit=1000)[source]¶

Delete descriptors by version:: delete descriptors of preassigned version

Parameters:

descriptorVersion (int) – descriptor version
faceIdGte (Optional[str, None]) – lower face id including boundary
faceIdLt (Optional[str, None]) – upper face id excluding boundary
limit (int) – maximum attributes amount to delete

Return type:

list[dict[str, Union[str, int, list[str]]]]

Returns:

list of face ids id for deleted descriptors of preassigned version

async deleteFaceAttributes(faceId, accountId=None)[source]¶

Delete face attributes.

Parameters:

faceId (str) – face id to remove its attributes
accountId (Optional[str, None]) – account id

Return type:

None

async deleteFaces(faces=None, accountId=None, createTimeGte=None, createTimeLt=None, userData=None, listId=None)[source]¶

Remove faces.

Parameters:

accountId (UnionType[str, None]) – faces account id
faces (UnionType[list[str], None]) – faces ids
createTimeGte (UnionType[str, None]) – creation time greater
createTimeLt (UnionType[str, None]) – create time lower
userData (UnionType[str, None]) – user data
listId (UnionType[str, None]) – list id where face exists

Return type:

int

Returns:

removed face count

async deleteFacesExtended(targets, accountId=None, createTimeGte=None, createTimeLt=None, userData=None, listId=None)[source]¶

Remove faces.

Parameters:

targets (list[str]) – deletion info targets
accountId (UnionType[str, None]) – faces account id
createTimeGte (UnionType[str, None]) – creation time greater
createTimeLt (UnionType[str, None]) – create time lower
userData (UnionType[str, None]) – user data
listId (UnionType[str, None]) – list id where face exists

Return type:

UnionType[list[dict], None]

Returns:

list with dicts for each deleted face.

async deleteLists(lists, accountId=None, withFaces=False)[source]¶

Remove lists.

Parameters:

accountId (Optional[str, None]) – lists account id
lists (list[str]) – lists
withFaces (bool) – remove lists with all faces which is contained in these lists

Return type:

int

Returns:

removed list count

async executeSearchFaces(filters, targets, page=1, pageSize=100)[source]¶

Get faces searched by filters

Parameters:

filters (SearchFacesFilters) – raw search request filters
targets (Optional[list[str], None]) – target Face columns’ names to get info on
page (Optional[int, None]) – page
pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async executeSearchFacesFilteredByList(filters, targets, page=1, pageSize=100)[source]¶

Get faces searched by filters associated with face-list model

Parameters:

filters (SearchFacesFilters) – raw search request filters
targets (list[str]) – target Face columns’ names to get info on
page (Optional[int, None]) – page
pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

faces list

async getBasicAttrsCount(accountId=None)[source]¶

Get count of basic attributes.

Parameters:: accountId (Optional[str, None]) – account id of the attributes
Return type:: int
Returns:: basic attributes count

async getCachedFaceAttributesCount()[source]¶

Get cached count of faces attributes.

Return type:: int
Returns:: Number of faces attributes

async getDescriptorsBatchByFaceIds(facesIds, accountId=None, checkObjectExistence=True, descriptorVersion=None, receiveExternalId=None)[source]¶

Get descriptors batch by faces Ids. ! Splitting into batches is used due to enormous queries (25Mb for 100k faces) that fails Oracle DB. (LUNA-3734)

Parameters:

facesIds (set[str]) – faces ids the descriptors were attached to
accountId (Optional[str, None]) – account id of the faces
checkObjectExistence (bool) – check faces existence or not
descriptorVersion (Optional[int, None]) – requested descriptor version
receiveExternalId (Optional[bool, None]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] notFoundUuids: list[str] notExtractedUuids: list[str] externalIds: list[str]

Return type:

Dict

Raises:

VLException(Error.FacesNotFound) –

async getDescriptorsCount(accountId=None)[source]¶

Get face descriptors count with filters.

Parameters:: accountId (Optional[str, None]) – account id
Return type:: list[dict[str, int]]
Returns:: list of dict. Keys of each dict are “descriptor_version” and “descriptor_count”

async getFaces(filters, targets=None, page=1, pageSize=100)[source]¶

Get faces.

Parameters:

filters (SearchFacesFilters) – raw search request filters
targets (Optional[list[str], None]) – target Face columns’ names to get info on
page (Optional[int, None]) – page
pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

list of faces

async getFacesAttributeSamples(faceId, accountId)[source]¶

Get all the attribute samples of the specified face

Parameters:

faceId (list[str]) – face id
accountId (Optional[str, None]) – account id

Return type:

list[str]

Returns:

face attribute samples list

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found –

async getFacesAttributes(faceIds, retrieveAttrs, descriptorVersion, accountId, getDescriptorAsBytes=False)[source]¶

Retrieve attributes

Parameters:

faceIds (list[str]) – face ids
retrieveAttrs (Iterable[str]) – tuple of attributes what need retrieve
descriptorVersion (int) – requested descriptor version
accountId (Optional[str, None]) – account id
getDescriptorAsBytes (bool) – whether to get descriptor as bytes otherwise as base64

Returns:

face_id: face id: attributes: target-specific face attributes dict
If some attribute data is not found it will be filled with ‘empty’ value:: None for attributes, [] for samples

Return type:

face attributes list with items with the following properties

async getFacesAttributesCount(accountId=None)[source]¶

Return face attribute count

Parameters:: accountId (Optional[str, None]) – account id
Return type:: int
Returns:: face attribute count

async getFacesCount(filters)[source]¶

Get count of faces.

Parameters:: filters (SearchFacesFilters) – raw search request filters
Return type:: int
Returns:: Number of faces

async getFacesCountByList(filters)[source]¶

Get count of faces filtered by list id.

Parameters:: filters (SearchFacesFilters) – raw search request filters
Return type:: int
Returns:: Number of faces

async getListDeletions(deletionTimeLt=None, deletionTimeGte=None, page=1, pageSize=100)[source]¶

Get lists deletion logs

Parameters:

deletionTimeLt (Optional[datetime, None]) – upper bound of list deletion time
deletionTimeGte (Optional[datetime, None]) – lower bound of list deletion time
page (Optional[int, None]) – page
pageSize (Optional[int, None]) – page size

Warning

trigger trg_lists_deletion_log inserts a data for table ListsDeletionLog

Returns:: list of deletions in the reverse order of deletion of lists

async getListFacesDescriptorsBatch(listId, linkKeyGte, limit, descriptorVersion=None, parity=None, receiveExternalId=None)[source]¶

Get descriptors batch

Parameters:

listId (str) – list id the descriptors were attached to
linkKeyGte (int) – the lower including boundary
limit (int) – descriptors count to return
descriptorVersion (Optional[int, None]) – descriptor version
parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for
receiveExternalId (Optional[bool, None]) – receive or not external ids

Returns:

descriptors: list[bytes] descriptorVersion: int faceUuids: list[str] linkKeys: list[int] faceIds: list[str] externalIds: list[str]

Return type:

Dict

async getListMinusDelta(listId, unlinkKeyGte=None, unlinkKeyLt=None, limit=10000, parity=None)[source]¶

Get history of detach attribute to lists

Parameters:

listId – list id
unlinkKeyLt (Optional[int, None]) – upper bound of unlink key value
unlinkKeyGte (Optional[int, None]) – lower bound of unlink key value
limit (int) – limit
parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”, “unlink_key”

Return type:

List of dicts with following keys

async getListPlusDelta(listId, linkKeyGte=None, linkKeyLt=None, limit=10000, parity=None)[source]¶

Get attach attributes to lists

Parameters:

listId – list id
linkKeyLt (Optional[int, None]) – upper bound of link key value
linkKeyGte (Optional[int, None]) – lower bound of link key value
limit (int) – limit
parity (Optional[int, None]) – 0 for odd or 1 for even link keys to search for

Returns:

“face_id”, “link_key”

Return type:

List of dicts with following keys

async getLists(filters, page=1, pageSize=100)[source]¶

Get list

Parameters:

filters (SearchListsFilters) – raw search request filters
page (Optional[int, None]) – page
pageSize (Optional[int, None]) – page size

Return type:

list[dict[str, str]]

Returns:

List with dicts

async getListsAndKeysFromMV(listIds, useParity)[source]¶

Get lists with last link and unlink keys from materialized views :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys else max link/unlink keys

Return type:: list[dict[str, Any]]
Returns:: List with list ids and its link/unlink keys

async getListsCount(filters)[source]¶

Count lists

Parameters:: filters (SearchListsFilters) – raw search request filters
Return type:: int
Returns:: Count of lists

async getListsWithKeys(listIds, useParity)[source]¶

Get lists with last link and unlink keys from usual tables :type listIds: set[str] :param listIds: list ids :type useParity: int :param useParity: if 1 - get even and odd max link/unlink keys, if 0 - max link/unlink keys

Return type:: list[dict[str, Any]]
Returns:: list with list ids and its link/unlink keys

async getMissingBasicAttrs(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)[source]¶

Get missing basic attributes:: get attributes without basic attributes get samples

Parameters:

accountId (Optional[str, None]) – account id of the attributes
faceIds (Optional[list[str], None]) – list of attribute ids
faceIdGte (Optional[str, None]) – lower face id including boundary
faceIdLt (Optional[str, None]) – upper face id excluding boundary
limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingBasicAttrsCount(accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)[source]¶

Get count of missing basic attributes.

Parameters:

accountId (Optional[str, None]) – account id of the attributes
faceIds (Optional[list[str], None]) – list of face ids
faceIdGte (Optional[str, None]) – lower attribute id including boundary
faceIdLt (Optional[str, None]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing basic attributes count

async getMissingDescriptors(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None, limit=1000)[source]¶

Get missing descriptors of ‘missingVersion’ version:: get attributes not having descriptor get samples

Parameters:

missingVersion (int) – missing descriptor version
accountId (Optional[str, None]) – account id of the attributes
faceIds (Optional[list[str], None]) – list of attribute ids
faceIdGte (Optional[str, None]) – lower face id including boundary
faceIdLt (Optional[str, None]) – upper face id excluding boundary
limit (int) – maximum attributes amount to return

Returns:

{“face_id”: “<face_id>”, “generation”: <generation>, “samples”: [“<sample_id>”]}

Return type:

data array in the format

async getMissingDescriptorsCount(missingVersion, accountId=None, faceIds=None, faceIdGte=None, faceIdLt=None)[source]¶

Get count of missing descriptors of ‘missingVersion’ version.

Parameters:

missingVersion (int) – missing descriptor version
accountId (Optional[str, None]) – account id of the attributes
faceIds (Optional[list[str], None]) – list of face ids
faceIdGte (Optional[str, None]) – lower attribute id including boundary
faceIdLt (Optional[str, None]) – upper attribute id excluding boundary

Return type:

int

Returns:

missing descriptors count

async getNonexistentFaceId(requiredFaceIds, accountId=None, dbConnection=None)[source]¶

Get one of requiredFaceIds that not exists.

Parameters:

requiredFaceIds (set[str]) – set of required face ids
accountId (Optional[str, None]) – account id
dbConnection (Optional[AbstractDBConnection, None]) – current connection. Needed not to get new connection from pool or to do some stuff in scope of the transaction.

Return type:

Optional[str, None]

Returns:

Non existing face id

async getNonexistentListId(requiredListIds, accountId=None)[source]¶

Get one of requiredListIds that not exists.

Parameters:

requiredListIds (set[str]) – set of required face ids
accountId (Optional[str, None]) – account id

Return type:

Optional[str, None]

Returns:

Non existsing list id

async classmethod initDBContext(dbSettings, storageTime, defaultDescriptorVersion)[source]¶

Initialize context :type dbSettings: DBSetting :param dbSettings: database settings :type storageTime: str :param storageTime: storage time :type defaultDescriptorVersion: int :param defaultDescriptorVersion: default descriptor version

Return type:: None

async insertFaceAttributeData(connection, faceId, attribute)[source]¶

Insert face attribute data. :type connection: AbstractDBConnection :param connection: db connection :type faceId: str :param faceId: face id :type attribute: TemporaryAttributes :param attribute: temporary attribute container

Return type:: None

async isFacesExist(faceIds, accountId=None)[source]¶

Checking to exist faces or not.

Parameters:

accountId (Optional[str, None]) – account id
faceIds (list[str]) – face ids

Return type:

bool

Returns:

True if all faces exist else false

async isListsExist(listIds, accountId=None)[source]¶

Checking to exist lists or not.

Parameters:

accountId (Optional[str, None]) – account id
listIds (set[str]) – list ids

Return type:

bool

Returns:

True if all lists exist else false

async linkFacesToList(listId, faces)[source]¶

Attach faces to list.

Parameters:

listId (str) – list id
faces (list[str]) – face ids

Raises:

IntegrityError –
Exception –

Return type:

tuple[list[str], list[str]]

Returns:

list of success link faces and failed link faces.

classmethod makeOutputFaces(faceRows, columns)[source]¶

Make result faces (from the database reply) proper for an user.

Parameters:

faceRows (list[UnionType[Record, dict]]) – face list from db
columns (Iterable[str]) – selected columns

Return type:

list[dict[str, Any]]

Returns:

faces with changed fields

classmethod makeOutputLists(lists)[source]¶

Make result of lists (from the database reply) proper for an user.

Parameters:: lists (list[UnionType[Record, dict]]) – list with lists
Return type:: list[dict[str, Any]]
Returns:: lists with changed fields

async ping(pingCount)[source]¶

Ping database. Execute ‘SELECT 1’ from one to ‘pingCount’ times.

Parameters:: pingCount (int) – ping count
Return type:: bool
Returns:: True - if any request success execute otherwise False

static prepareListQueryFilters(filters)[source]¶

Prepare search query filters on “List” models.

Parameters:: filters (SearchListsFilters) – query filters
Return type:: BooleanClauseList
Returns:: filters for List select query

static prepareSearchFacesFilters(filters, modelsFace=None, modelsListFace=None)[source]¶

Prepare search query filters on “Face” models.

Parameters:

filters (SearchFacesFilters) – query filters
modelsFace (Optional[AliasedClass, None]) – database table model for faces
modelsListFace (Optional[AliasedClass, None]) – database table model for list face

Return type:

list[BooleanClauseList]

Returns:

filters for Face model

async putFaceAttributes(faceId, attribute, accountId)[source]¶

Create attributes with exception wrap

Parameters:

faceId (str) – face id
attribute (TemporaryAttributes) – temporary attribute container
accountId (Optional[str, None]) – account id

Raises:

VLException(Error.FaceNotFound.format(faceId), 404, isCriticalError=False) if face not found –

Return type:

None

static setOffsetAndOrderToQuery(query, order='desc', orderColumn=None, page=1, pageSize=100)[source]¶

Apply OFFSET and ORDER BY criterion to the query.

Parameters:

query (Query) – query object
order (Optional[Literal[‘desc’, ‘asc’], None]) – result sort order (ask or desc)
orderColumn (Optional[Column, None]) – result sort column
page (Optional[int, None]) – pagination page value
pageSize (Optional[int, None]) – pagination page size value, set -1 to get all faces

Return type:

Query

Returns:

updated query

async static setOracleIndexNames(logger)[source]¶: Set index names for some requests with ‘use-force-index’

async unlinkFacesFromList(listId, faces, accountId=None)[source]¶

Unlink faces from list.

Parameters:

listId (str) – list id
faces (list[str]) – face ids
accountId (Optional[str, None]) – account id

async updateFace(faceId, accountId=None, **kwargs)[source]¶

Update face.

Parameters:

faceId (str) – face id
accountId (Optional[str, None]) – account id

Keyword Arguments:

user_data – face information
event_id – reference to event which created face
external_id – external id of the face, if it has its own mapping in external system
avatar – image url that represents the face

Return type:

int

Returns:

updated face count

async updateFaceAttributes(faceId, attribute, accountId=None, forceUpdate=False)[source]¶

Update attribute

Parameters:

faceId (str) – updated attribute face id
attribute (TemporaryAttributes) – temporary attributes container
accountId (Optional[str, None]) – account id
forceUpdate (Optional[bool, None]) – whether not to compare existing samples and new ones

Raises:

VLException(Error.FaceSampleConflict.format(faceId), 400, False) if samples conflict –
VLException(Error.AttributesForUpdateNotFound.format(faceId), 400, False) if attribute for update not found –
VLException(Error.FaceNotFound.format(faceId), 404, False) if face not found –

Return type:

None

async updateListLastUpdateTime(listId)[source]¶

Update last update time of lists.

Parameters:: listId (str) – list id

async updateListUserData(listId, userData, accountId=None)[source]¶

Update user data of list

Parameters:

listId (str) – list id
userData (str) – user data
accountId (Optional[str, None]) – account id

Return type:

int

Returns:

updated list count

class luna_faces.db.context.Reference(descriptorVersion, descriptor=None, attributeId=None, faceId=None)[source]¶: Reference class.

luna_faces.db.context.getCompiledQuery(query, dbType)[source]¶

Compile query to with literal_binds = True :type query: Union[Query, CompoundSelect] :param query: query made by sqlalchemy or CompoundSelect as result of union_all from n queries operation :type dbType: str :param dbType: database type

Return type:: str
Returns:: string with compiled query