Monitoring

Data for monitoring

We support two database options for collecting monitoring data: Clickhouse and InfluxDB. Depending on the database chosen, the structure and methodology for storing data vary.

Types of processed events

Our monitoring system processes the following event types:

1. request (any http request)

2. error (failed http request)

---

Comparison of data formats for Clickhouse and InfluxDB:

InfluxDB: Each event is presented as a “point” in a time series. The structure of a point includes:

• series name

• start event time

• tags, indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

• fields, non indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

Clickhouse: In Clickhouse, the data structure resembles that of a traditional SQL table. Each event is represented as a record, where:

• The `time` field contains the record’s creation timestamp;

• The `data` field contains a JSON object with all the information that would otherwise be distributed across tags and fields in InfluxDB.

Important: In Clickhouse, there is no differentiation between “tags” and “fields”—all data is consolidated into a single JSON object within the data field.

Monitoring series

‘Requests’ series.

Triggered on every HTTP request. Each point contains data about the corresponding request (execution time and etc).

InfluxDB:

Requests series tags

tag name	description
service	“luna-vector-search-module”
route	concatenation of a request method and a request resource (GET:/version)
status_code	http status code of response

Requests series fields

fields	description
request_id	request id
execution_time	request execution time

ClickHouse JSON `data` field Example:

{
    "service": "luna-events",
    "route": "POST:/events",
    "status_code": 204,
    "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
    "execution_time": 1.234
}

‘Errors’ series.

Triggered on failed request. Each point contains error_code of luna error.

InfluxDB:

Errors series tags

tag name	description
service	“luna-vector-search-module”
route	concatenation of a request method and a request resource (GET:/version)
status_code	http status code of response
error_code	Luna Platform error code

Errors series fields

fields	description
request_id	request id

ClickHouse JSON `data` field Example:

{
    "service": "luna-events",
    "route": "POST:/events",
    "status_code": 400,
    "error_code": 13037,
    "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f"
}

Database

You can refer to documentation for influx database and clickhouse database to compare the databases and choose what benefit your needs more. Note that clickhouse might be the better choice for aggregation You can setup your database credentials in configuration file in section “monitoring”.