Core Concepts#

SDK workflow#

Object lifetime#

Most of the SDK features are exposed via interfaces (C++ virtual classes) whose implementations must be obtained by calling factory functions. Some of the factories are C-functions, such as createFaceEngine(...). The latter one produces a root object IFaceEngine, which in turn exposes many other factories of the IFaceEngine::createXYZ(...) form. A typical workflow consists of obtaining IFaceEngine, then calling its factories and using the produced child objects.

You do not destroy SDK objects directly, but instead deal with fsdk::Ref<T>, reference-counted smart pointers (see section "Automatic reference counting") to SDK interfaces. You only need to release all shared references, at which point fsdk::Ref<T> destroys the underlying object.

In terms of lifetime, IFaceEngine should outlast all its child objects.

Holding fsdk::Ref<T> objects in global variables is error-prone. If the variables are in different translation units, their construction order is undefined, which means the destruction order is out of control, too. Viable approaches include gathering all fsdk::Ref<T> objects in a single class or using an explicit stack to store them, as well as storing all fsdk::Ref<T> as local variables on the call stack in simple projects. In the case when it is necessary to store fsdk::Ref<T> objects as global or static variables, the correct order of releases should be guaranteed explicitly before the program ends:

//warning: a correct, but not a good example due to these global variables
fsdk::IFaceEnginePtr faceEngine = fsdk::createFaceEngine("./data");
fsdk::IDetectorPtr detector = faceEngine->createDetector();
fsdk::IBestShotQualityEstimator bestShotQualityEstimator = faceEngine->createBestShotQualityEstimator();

int main() {
    // application code here

    bestShotQualityEstimator.reset();
    detector.reset();
    faceEngine.reset();
    return 0;
}

Threading#

The part of the SDK that instantiates and destroys objects is not thread-safe. The SDK requires using one thread (let's call it init-thread) for calling all factory functions, as well as releasing the references to the produced objects. The SDK internally uses thread-local objects attached to init-thread, which makes init-thread special: as long as the SDK is alive, init-thread must be alive too. Therefore, there is a requirement that init-thread must outlast IFaceEngine.

Once SDK objects (such as detectors and estimators, but not IFaceEngine) have been created, they are thread-safe and can be used concurrently and on arbitrary threads. Before using an object concurrently on many threads, consider using asynchronous APIs of the SDK instead. For example, IDetector along with a synchronous detect(...) function also provides asynchronous detectAsync(...).

It is required that an object cannot be destroyed while it has at least one incomplete call, synchronous or asynchronous, on any thread.

Detailed constraints#

Here is a more detailed list of lifetime and threading constraints:

There should be at most one IFaceEngine object per process simultaneously. You can create a new IFaceEngine object after destroying the previous one, just avoid holding multiple IFaceEngine objects at the same time.
There should be at most one ITrackEngine object per process simultaneously. You can create a new ITrackEngine object after destroying the previous one, just avoid holding multiple ITrackEngine objects at the same time.
All factory functions should be called on init-thread (the thread that calls createFaceEngine()). This also implies that factory code is not thread-safe and all factory calls should be serialized in time. Factory functions include:
- C-style functions of the form createXYZ(...) such as createFaceEngine(...), createTrackEngine(...)
- member functions such as IFaceEngine::createXYZ(...), ITrackEngine::createXYZ(...)
activateLicense(...) is not thread-safe. There should be at most one invocation of activateLicense(...) per process simultaneously.
init-thread should live no shorter than IFaceEngine.
IFaceEngine should live no shorter than ITrackEngine.
IFaceEngine should live no shorter than its child objects (algorithms/estimators/detectors). I.e., IFaceEngine should be the last destroyed SDK object.
IFaceEngine should be destroyed on init-thread.
Algorithms/estimators/detectors should be destroyed on init-thread.
Algorithms/estimators/detectors can be destroyed when there are no pending or unfinished invocations of member functions of those objects, synchronous or asynchronous, on any threads.
Track Engine requirements: all Track Engine streams should be stopped, then destroyed, then ITrackEngine itself should be stopped, then destroyed.
ITrackEngine and all its streams should be destroyed on init-thread.

The only part of the SDK that allows multithreading is using member functions of already instantiated algorithms/estimators/detectors, such as IDetector:detect(...) and IAttributeEstimator::estimate(...). The member functions can be called on arbitrary threads and in parallel. Before resorting to this multithreaded scenario, please consider using asynchronous versions that accompany many synchronous functions of the SDK.

Common Interfaces and Types#

Reference Counted Interface#

Everything in FaceEngine object system starts from here. The IRefCounted interface provides methods for reference counter access, increment, and decrement. All reference counted objects imply a custom memory management model. This way they support automated destruction when reference count drops to zero as well as more sophisticated strategies of partial destruction and weak referencing required for FaceEngine internal needs. The bare minimum of such functions is exposed to a user allowing:

To notify the object that it is required by a client via retaining a reference to it.
To notify the object that it is no longer required by releasing a reference to it.
To get actual reference counter value.

Reference counted objects expect some special treatment as well. Be sure never to call delete on any pointer to object derived from IRefCounted! Doing so leads to heap corruption. Simply calling release notifies the system when the object should be destroyed and it does this properly for you.

However, we do not recommend that you interact with the reference counting mechanism manually as doing so may be error-prone. Instead, we recommend that you use smart pointers that are specially designed to handle such objects and provided by FaceEngine. See section "Automatic reference counting" for details.

Automatic reference counting#

For your convenience, a special smart pointer class Ref is provided. It is capable of automatic reference counter incrementing upon its creation and automatic decrementing upon its destruction. It also does an assertion of the inner raw pointer being non-null, thus preventing errors.

Two ways of working with Ref are possible:

Referencing - without acquiring ownership of object lifetime#

ISomeObject* createSomeObject();
{
/* Here createSomeObject returns an object with initial reference count of 1 (otherwise, it would be dead). Then Ref adds another one for itself making a total reference count of 2!
*/
Ref<ISomeObject> objref = make_ref(createSomeObject());
/* Here we use the object in any way we want expecting it to be properly destroyed when control will leave this scope.
*/

}
/* Here we have left the scope and Ref was automatically destroyed like any other object created on the stack. At the same time, it decreased reference count of its internal object by 1 making it 1 again.
*/

However, the object is not destroyed automatically! For this to happen, it should have precisely 0 references. Moreover, in this example, the raw pointer to the object is lost, so it is impossible to fix it in any way; thus a memory leak is introduced.

Acquiring - own object lifetime#

So keeping that in mind we introduce a concept of ownership acquiring. By acquiring an object, you mean that its raw pointer is not going to be used and only a valid Ref to it is required. To acquire ownership, use a special ::acquire() function. The fixed version of the above example would look like this:

ISomeObject* createSomeObject();
{
/* Here createSomeObject returns an object with initial reference count of 1 (otherwise, it would be dead). Then we acquire it leaving a total reference count of 1.
*/
Ref<ISomeObject> objref = acquire(createSomeObject());
/* Here we use the object in any way we want.
*/
}

/* Here we have left the scope and Ref was automatically destroyed like any other object created on the stack. At the same time, it decreased reference count of its internal object by 1 making it 0. The object is destroyed properly by the object system.
*/

Do not store or use raw pointers to the object when using the ::acquire() function, as ownership acquiring invalidates them.

Acquiring way of working with Ref is pretty like standard library shared_ptr own lifetime of the object after it returned by std::make_shared().

You can statically cast object type during acquiring or referencing. To achieve this, use special versions of the ::make_ref_as() and ::acquire_as() functions. It is your responsibility to ensure that such a cast is possible.

Please refer to FaceEngine Reference Manual for more details on available convenience methods and functions.

As a side note, be informed that typedefs for Ref's to all reference counted types are declared. All of them match the following naming convention: InterfaceNamePtr. So, for example, Ref<IDetector> is equivalent to IDetectorPtr.

Serializable object interface#

This interface represents an object. Object's contents may be serialized to some data stream and then read back. Think of this as loading and saving.

To interact with the aforementioned data stream, the serializable object needs a user-provided adapter. Such adapter is called the archive. See a detailed explanation of it in section "Archive interface" in chapter "Core facility".

Serializable interfaces: IDescriptor, IDescriptorBatch.

Auxiliary types#

Image type#

Since FaceEngine is a computer vision library, it is natural for it to implement some image concept. Therefore, an Image class exists. It is designed as a reference counted container for raw pixel color data. Reference counting allows a single image to be shared by several objects. However, one should understand, that each Image object is holding a reference to some data, so if the data is modified in any way, this affects all other objects holding the same reference. To make a deep copy of an Image, one should use the clone() method, since assignment operators just make a reference. It is also possible to clip a part of an image into a new image by means of extract() method.

Pixel data may be characterized by color channel layout, i.e., a number of color channels and their order. The engine defines a Format structure for that. The Format determines:

Number of color channels (e.g., RGB or grayscale);
Order of color channel (e.g., RGB vs. BGR).

FaceEngine assumes 8 bits (i.e., 1 byte) per color channel and implements 8 BPP grayscale, 24 BPP RGB/BGR and padded 32 BPP formats. Format conversion functions are also provided for convenience; see the convert() function family.

The Image class supports data range mapping. It is possible to map a subset of bytes in a rectangular area for reading or writing. The mapped pixels are represented by the SubImage structure. In contrast to Image, SubImage is just a data view and is not reference counted. You are not supposed to store SubImages longer that it is necessary to complete data modification. See the documentation of the map() function family for details.

The supports IO roitines to read/write OOM, JPEG, PNG and TIFF formats via FreeImage library.

The absence of image IO is dictated by the fact that FaceEngine focuses on being lightweight and with the minimum possible number of external dependencies. It is not designed solely with image processing purpose in mind. I.e., one may treat video frames as Images and process them one by one. In this case, an external (possibly proprietary) video codec is required.

Beta Mode#

Some features in LUNA SDK are available just in Beta mode. This is experimental features which may be unstable. If you want use them, you have to activate betaMode param in config (faceengine.conf).