Lambda metrics

Introduction

Lambdas can collect metrics the same way as any service, see service metrics. This metrics can be collected by Prometheus via /lambda_metrics path.

To enable metrics collection:

Prometheus service must be launched and configured to collect metrics from /lambda_metrics

LUNA_LAMBDA_METRICS_SETTINGS.origin must be set to actual Prometheus address.

LUNA_SERVICE_METRICS settings must be enabled

Lambda metrics can be received via luna-lambda service via /lambdas/<lambdaId>/metrics resource. Response format follows Prometheus HTTP Api.

User metrics

Lambda can be provided with custom set of metrics, that later can be collected by Prometheus.

To add metrics to lambda they must be instanced im module scope of lambda_main.py file:

from luna_lambda_tools.public.metrics.user_metrics import Counter, Gauge, Histogram, Summary

USER_SUM = Summary("user_sum", description="sum", labelNames=["custom_label"])
USER_HISTOGRAM = Histogram("user_histo", description="histogram", labelNames=["custom_label"])
USER_COUNTER = Counter("user_counter", description="counter")
USER_GAUGE = Gauge("user_gauge", description="gauge")

USER_METRICS = [
    USER_GAUGE,
    USER_SUM,
    USER_HISTOGRAM,
    USER_COUNTER,
]

Later this metrics can be used to collect data:

...
USER_SUM.observe(0.5, labels={"custom_label": "value"})
USER_HISTOGRAM.observe(0.6, labels={"custom_label": "value1"})
USER_COUNTER.inc(1.0)
USER_GAUGE.set(2.3)

Available metrics:

Counter - Used for monotonically increasing values — things that only go up.

USER_COUNTER.inc()  # Increment by 1
USER_COUNTER.inc(3) # Increment by 3

Gauge - Used for values that go up and down.

USER_GAUGE.set(100)  # Set to 100
USER_GAUGE.inc()  # Increment by 1
USER_GAUGE.inc(3) # Increment by 3
USER_GAUGE.dec(2) # Decrement by 2

Summary - Used to track event durations or sizes.
USER_SUM.observe(0.5) # Observes 0.5

Histogram - Used to measure distributions.

USER_HISTOGRAM.observe(0.5)  # Observes 0.5

After collection by prometheus each metric will be stored as subset of metrics following this pattern:

Metrics exposed in prometheus
Metric Type	Example Metric Name	Prometheus-Exposed Metrics
Counter	`lambda_requests_total`	`lambda_requests_total`
Gauge	`lambda_memory_usage_bytes`	`lambda_memory_usage_bytes`
Summary	`lambda_request_seconds`	`lambda_request_seconds_count`, `lambda_request_seconds_sum`
Histogram	`lambda_latency_seconds`	`lambda_latency_seconds_bucket`, `lambda_latency_seconds_count`, `lambda_latency_seconds_sum`

This metrics can be collected via /lambdas/<lambdaId>/metrics resource.

Note: LUNA_LAMBDA_METRICS_SETTINGS.enabled must be set to true, otherwise metrics data won`t be collected.

Here is an example of standalone lambda that mesures execution time on /main route and periodically tracks lambda cpu load:

requirements.txt

psutil

lambda_main.py

import asyncio
import time

import psutil
from luna_lambda_tools import StandaloneLambdaRequest, logger
from luna_lambda_tools.public.metrics.user_metrics import Gauge, Summary

LAMBDA_REQUEST_DURATION = Summary("lambda_request_duration", description="Request duration")
CPU_STATS = Gauge("cpu_stats", description="Lambda cpu load")

USER_METRICS = [CPU_STATS, LAMBDA_REQUEST_DURATION]


class UserCtx:

    def __init__(self):
        self.cpuStatsCollectorTask = None

    async def onStart(self):
        self.cpuStatsCollectorTask = asyncio.create_task(self._collectCpuStats())

    async def onShutdown(self):
        self.cpuStatsCollectorTask.cancel()

    @staticmethod
    async def _collectCpuStats():
        psutil.cpu_percent(interval=None)
        while True:
            await asyncio.sleep(0.1)
            CPU_STATS.set(psutil.cpu_percent(interval=None))


async def main(request: StandaloneLambdaRequest) -> dict:
    """Main route"""
    logger.info("Request processing starts")

    value = request.json.get("value")
    start = time.time()
    await asyncio.sleep(value)
    LAMBDA_REQUEST_DURATION.observe(time.time() - start)

    return {"result": "Metrics collected"}

request example

from luna3.luna_lambda.luna_lambda import LambdaApi

SERVER_ORIGIN = "http://lambda_address:lambda_port"  # Replace by your values before start
SERVER_API_VERSION = 1
lambdaApi = LambdaApi(origin=SERVER_ORIGIN, api=SERVER_API_VERSION)
lambdaId, accountId = "your_lambda_id", "your_account_id"  # Replace by your values before start


def makeRequest():
    reply = lambdaApi.proxyLambdaPost(lambdaId=lambdaId, path="main", accountId=accountId, body={"value": 0.1})
    return reply


if __name__ == "__main__":
    response = makeRequest()
    print(response.json)

Example of metrics that will be collected by prometheus after /main request:

...
# HELP cpu_stats Lambda cpu load
# TYPE cpu_stats gauge
cpu_stats{lambda_id="d95a072a-3b8a-461d-ad23-5df95ddd300d"} 17.4
# HELP lambda_request_duration Request duration
# TYPE lambda_request_duration summary
lambda_request_duration_count{lambda_id="d95a072a-3b8a-461d-ad23-5df95ddd300d"} 1.0
lambda_request_duration_sum{lambda_id="d95a072a-3b8a-461d-ad23-5df95ddd300d"} 0.10052204132080078