Monitoring

Data for monitoring

We are currently processing several types of events for monitoring:

1. request (any http request)

2. error (failed http request)

---

Comparison of data formats for Clickhouse and InfluxDB:

InfluxDB: Each event is presented as a “point” in a time series. The structure of a point includes:

• series name

• start event time

• tags, indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

• fields, non indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

Clickhouse: In Clickhouse, the data structure resembles that of a traditional SQL table. Each event is represented as a record, where:

• The `time` field contains the record’s creation timestamp;

• The `data` field contains a JSON object with all the information that would otherwise be distributed across tags and fields in InfluxDB.

Important: In Clickhouse, there is no differentiation between “tags” and “fields”—all data is consolidated into a single JSON object within the data field.

Monitoring series

The structure and the meaning of each monitoring series remain consistent. However, for Clickhouse, data from tags and fields are merged into a single JSON object under the data field. Below are examples for each series:

• Requests series.

  Triggered on every request. Each point contains a data about corresponding request (execution time and etc).

  InfluxDB:

  Requests series tags

  tag name	description
  service	always “luna-handlers”
  route	concatenation of a request method and a request resource (POST:/extractor)
  status_code	http status code of response

  Requests series fields

  fields	description
  request_id	request id
  execution_time	request execution time

  Requests series additional tags

  tag name	resource	description
  handler_id	/handlers/{handlerId}/events	handler ID
  verifier_id	/verifiers/{verifierId}/verifications	verifier ID

  Requests series additional fields

  fields	resource	description
  download_images_time	/detector	time taken to download the image from Image Store
  save_warps_time	/detector	time taken to save warps to Image Store
  save_attributes_time	/extractor	time taken to save samples to Image Store
  load_face_samples_time	/extractor	time taken to download the face sample from Image Store
  load_face_samples_time	/extractor/upgrade	time taken to download the face sample from Image Store
  load_body_samples_time	/extractor/upgrade	time taken to download the body sample from Image Store
  save_face_attributes_time	/extractor/upgrade	time taken to save face attributes to Luna Faces
  save_event_attributes_time	/extractor/upgrade	time taken to save event attributes to Luna Faces
  load_images_for_processing_time	/handlers/{handlerId}/events	time taken to load image for processing from request
  estimation_policies_time	/handlers/{handlerId}/events	time taken to execute estimation policies
  estimation_policies_time	verifiers/{verifierId}/verifications	time taken to execute estimation policies
  face_sample_storage_policy_time	/handlers/{handlerId}/events	time taken to save face samples to Image Store
  face_sample_storage_policy_time	/verifiers/{verifierId}/verifications	time taken to save face samples to Image Store
  body_sample_storage_policy_time	/handlers/{handlerId}/events	time taken to save body samples to Image Store
  image_origin_storage_policy_time	/handlers/{handlerId}/events	time taken to save image origins to Image Store
  face_attribute_storage_policy_time	/handlers/{handlerId}/events	time taken to save face attributes to Image Store
  face_attribute_storage_policy_time	/verifiers/{verifierId}/verifications	time taken to save face attributes to Image Store
  face_storage_policy_time	/handlers/{handlerId}/events	time taken to save face with avatar to Luna Faces
  event_storage_policy_time	/handlers/{handlerId}/events	time taken to save event to Luna Events
  notification_storage_policy_time	/handlers/{handlerId}/events	time taken to send notification to Luna Sender
  match_policy_time	/handlers/{handlerId}/events	time taken for matching
  match_policy_time	/verifiers/{verifierId}/verifications	time taken for matching

  ClickHouse JSON `data` field Example:

  {
      "service": "luna-handlers",
      "route": "POST:/detector",
      "status_code": 200,
      "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
      "execution_time": 1.620,
      "download_images_time": 1.3
  }
  

• Errors series.

  Triggered on failed request. Each point contains error_code of luna error.

  InfluxDB:

  Errors series tags

  tag name	description
  service	always “luna-handlers”
  route	concatenation of a request method and a request resource (POST:/extractor)
  status_code	http status code of response
  error_code	Luna Platform error code

  Errors series fields

  fields	description
  request_id	request id

  Errors series additional tags

  tag name	resource	description
  handler_id	/handlers/{handlerId}/events	handler ID
  verifier_id	/verifiers/{verifierId}/verifications	verifier ID

  ClickHouse JSON `data` field Example:

  {
      "service": "luna-handlers",
      "route": "POST:/handlers/8b8b5937-2e9c-4e8b-a7a7-5caf86621b5a/events",
      "status_code": 400,
      "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
      "error_code": 13003,
      "handler_id": "8b8b5937-2e9c-4e8b-a7a7-5caf86621b5a"
  }
  

Database

You can refer to documentation for influx database and clickhouse database to compare the databases and choose what benefit your needs more. Note that clickhouse might be the better choice for aggregation You can setup your database credentials in configuration file in section “monitoring”.

Plugins

You can realize your own plugin for sending monitoring data. See plugins

Module request monitoring plugin example

class luna_handlers.crutches_on_wheels.cow.plugins.plugin_examples.request_monitoring_plugin_example.BaseRequestMonitoringPlugin(app)

Base class for requests monitoring.

abstractmethod async flushPointToMonitoring(point, logger)

All plugins must realize this method.

This function call after end of request

Return type:

None

Parameters:

• point – point for monitoring

• logger – logger

async handleEvent(points, logger)

Handle event

Parameters:

• *args – positional arg for event handler function

• **kwargs – named arg for event handler function

class luna_handlers.crutches_on_wheels.cow.plugins.plugin_examples.request_monitoring_plugin_example.RequestMonitoringPlugin(app)

Example plugin sends a request data for monitoring to third-party source. Only one instance of this class exist during the program execution.

async close()

Stop plugin.

Close all open connections and ect

async flushPointToMonitoring(point, logger)

Callback for sending a request monitoring data.

Return type:

None

Parameters:

• point – point for monitoring

• logger – logger

async initialize()

Initialize plugin.

Close all open connections and ect