Lambda metrics
==============

Introduction
------------

Lambdas can collect metrics the same way as any service, see `service metrics <./metrics.html>`_.
This metrics can be collected by Prometheus via */lambda_metrics* path.

To enable metrics collection:

    - Prometheus service must be launched and configured to collect metrics from */lambda_metrics*
    - `LUNA_LAMBDA_METRICS_SETTINGS.origin` must be set to actual Prometheus address.
    - `LUNA_SERVICE_METRICS` settings must be enabled

Lambda metrics can be received via `luna-lambda` service via `/lambdas/<lambdaId>/metrics <./_static/api.html#tag/lambda-metrics/operation/getLambdaMetrics>`_ resource.
Response format follows `Prometheus HTTP Api <https://prometheus.io/docs/prometheus/latest/querying/api/#format-overview>`_.


User metrics
------------

Lambda can be provided with custom set of metrics, that later can be collected by Prometheus.

To add metrics to lambda they must be instanced im module scope of `lambda_main.py` file:

.. code-block:: python

    from luna_lambda_tools.public.metrics.user_metrics import Counter, Gauge, Histogram, Summary

    USER_SUM = Summary("user_sum", description="sum", labelNames=["custom_label"])
    USER_HISTOGRAM = Histogram("user_histo", description="histogram", labelNames=["custom_label"])
    USER_COUNTER = Counter("user_counter", description="counter")
    USER_GAUGE = Gauge("user_gauge", description="gauge")

    USER_METRICS = [
        USER_GAUGE,
        USER_SUM,
        USER_HISTOGRAM,
        USER_COUNTER,
    ]


Later this metrics can be used to collect data:

.. code-block:: python

    ...
    USER_SUM.observe(0.5, labels={"custom_label": "value"})
    USER_HISTOGRAM.observe(0.6, labels={"custom_label": "value1"})
    USER_COUNTER.inc(1.0)
    USER_GAUGE.set(2.3)

Available metrics:
    - Counter - Used for monotonically increasing values — things that only go up.

        .. code-block:: python

            USER_COUNTER.inc()  # Increment by 1
            USER_COUNTER.inc(3) # Increment by 3
    - Gauge - Used for values that go up and down.

        .. code-block:: python

            USER_GAUGE.set(100)  # Set to 100
            USER_GAUGE.inc()  # Increment by 1
            USER_GAUGE.inc(3) # Increment by 3
            USER_GAUGE.dec(2) # Decrement by 2
    - Summary - Used to track event durations or sizes.

        .. code-block:: python

            USER_SUM.observe(0.5)  # Observes 0.5
    - Histogram - Used to measure distributions.

        .. code-block:: python

            USER_HISTOGRAM.observe(0.5)  # Observes 0.5


After collection by prometheus each metric will be stored as subset of metrics following this pattern:

.. list-table:: Metrics exposed in prometheus
   :header-rows: 1
   :widths: 20 30 50

   * - Metric Type
     - Example Metric Name
     - Prometheus-Exposed Metrics
   * - Counter
     - ``lambda_requests_total``
     - ``lambda_requests_total``
   * - Gauge
     - ``lambda_memory_usage_bytes``
     - ``lambda_memory_usage_bytes``
   * - Summary
     - ``lambda_request_seconds``
     - ``lambda_request_seconds_count``, ``lambda_request_seconds_sum``
   * - Histogram
     - ``lambda_latency_seconds``
     - ``lambda_latency_seconds_bucket``, ``lambda_latency_seconds_count``, ``lambda_latency_seconds_sum``

This metrics can be collected via `/lambdas/<lambdaId>/metrics <./_static/api.html#tag/lambda-metrics/operation/getLambdaMetrics>`_ resource.

**Note**: `LUNA_LAMBDA_METRICS_SETTINGS.enabled` **must be set to true, otherwise metrics data won`t be collected**.

Here is an example of standalone lambda that mesures execution time on `/main` route and periodically tracks lambda cpu load:

  .. literalinclude:: examples/201/standalone_lambda_user_metrics/lambda/requirements.txt
       :caption: requirements.txt
       :language: text

  .. literalinclude:: examples/201/standalone_lambda_user_metrics/lambda/lambda_main.py
       :caption: lambda_main.py
       :language: python

  .. literalinclude:: examples/201/standalone_lambda_user_metrics/make_request.py
       :caption: request example
       :language: python


Example of metrics that will be collected by prometheus after `/main` request:

.. code-block:: none

    ...
    # HELP cpu_stats Lambda cpu load
    # TYPE cpu_stats gauge
    cpu_stats{lambda_id="d95a072a-3b8a-461d-ad23-5df95ddd300d"} 17.4
    # HELP lambda_request_duration Request duration
    # TYPE lambda_request_duration summary
    lambda_request_duration_count{lambda_id="d95a072a-3b8a-461d-ad23-5df95ddd300d"} 1.0
    lambda_request_duration_sum{lambda_id="d95a072a-3b8a-461d-ad23-5df95ddd300d"} 0.10052204132080078


Last activity metric
--------------------

Each lambda created with the `eternal` flag disabled will publish the `last_activity` metric on some processing depending on
lambda type.

    - any lambda type will publish this metric on `/main` (if it exists) or any of the additional routes requests.
    - The `tasks` lambdas publish this metric each time lambda receives task or subtask for processing.
    - The `agent` lambdas publish this metric during stream processing.

For cases when lambda starts some custom background task, this task can be wrapped with the `monitorLastActivity` decorator.
Each call of the wrapped function will then update the `last_activity` metric.

.. code-block:: python

    ...
    from luna_lambda_tools.public.metrics.user_metrics import monitorLastActivity

    @monitorLastActivity
    async def someTask():
        await asyncio.sleep(1)

    class UserCtx:
        def __init__(self):
            self.backgroundTask = None

        @staticmethod
        async def periodicTask():
            while True:
                await someTask()
                await asyncio.sleep(10)

        async def onStart(self):
            logger.info("start lambda")
            self.backgroundTask = asyncio.create_task(self.periodicTask())

        async def onShutdown(self):
            logger.info("shutdown lambda")
            if self.backgroundTask:
                self.backgroundTask.cancel()
    ...

The `last_activity` metric is periodically requested from the running lambdas and can be received via `get lambda <./_static/api.html#operation/getLambda>`_ and `get lambdas <./_static/api.html#operation/getLambdas>`_ requests.