Agent interaction ================= Foreword -------- The video-agent represents handler that can decode and execute analytics on stream provided by Luna-Video-Manager. The interaction between the Luna-Video-Manager and the video agent is facilitated via HTTP. Full functionality requires the video-agent to meet a number of requirements, which are detailed in this chapter by provided primitive example including a description of its implementation and limitations compared to a fully-fledged solution. This primitive example is available as zip-archive, for link, preparation and execution description, see `this paragraph <#anchor-example-download>`_. .. _anchor-analytics-registration: Analytics registration ---------------------- A video agent is intended to process streams via analytics. To read more about what analytics is it and why is it needed, see `analytics chapter <./analytics.html>`_. The Luna-Video-Manager should be aware which agent with which analytics can works, but Luna-Video-Manager doesn't know about any analytics in general, so each video-agent must register its analytics before start work as video-agent. To register analytics it is need to make `analytics registration request <_static/api.html#operation/createAnalytic>`_ with actual parameters. Note that only *analytics_name* parameter is required, but all other parameters are not useless: for example, *validation_schema* is used by Luna-Video-Manager to validate analytics parameters from user request before it gets to the video-agent, *documentation* can be viewed to user could correctly specify analytics parameters, etc. .. dropdown:: What if analytics already registered using external request? :animate: fade-in-slide-down It is possible that analytics registered by external request, but to make sure that agent analytics and analytics registered in Luna-Video-Manager are equal and not just that they have the same names, it is highly recommended to make video-agent register analytics itself. .. dropdown:: What if analytics already exists :animate: fade-in-slide-down It is possible when video-agent restarts or more than one video-agent register the same analytics to receive response with 409 status code and `already exists` text from Luna-Video-Manager. It is recommended to replace such analytics during the video-agent startup using `analytics replace request <_static/api.html#operation/putAnalytic>`_ to make sure the latest version of analytics with actual parameters is registered in Luna-Video-Manager. .. dropdown:: Additional notes :animate: fade-in-slide-down The Luna-Video-Manager allows to: - get analytics using `analytics get request <_static/api.html#operation/getAnalytic>`_, - check analytics existence using `check analytics request <_static/api.html#operation/checkAnalytic>`_, - get analytics documentation using `get analytics documentation request <_static/api.html#operation/getAnalyticDoc>`_ - delete analytics using `delete analytics request <_static/api.html#operation/removeAnalytic>`_ what makes possible to make custom algorythm of analytics registration in different cases. The provided primitive agent example contains analytics registration example `here <#anchor-agent-py>`_. .. _anchor-agent-registration: Agent registration ------------------ A video-agent is intended for work with Luna-Video-Manager. To make Luna-Video-Manager knows how many and what video-agents exists, which analytics can they process and so on, each video-agent must be registered in Luna-Video-Manager. .. note:: The agent registration is the only one required step which must be done before stream processing. To register agent it is need to make `agent registration request <_static/api.html#operation/registerAgent>`_ with actual parameters. Note that not all parameters in this request are required, but optional parameters are not useless: for example, *description* is used by user to get some additional information about video-agent and *agent_groups* parameter is allow to make Luna-Video-Manager specify only stream linked to some groups to agent (more description `here <./streams_distribution.html#streams-groups>`_), etc. If agent parameters such as list of available analytic names or maximum number of streams changed during agent existence, the agent must stop all streams processing, remove its registration (using `delete agent request <_static/api.html#operation/removeAgent>`_) and make registration request one more time. Agent port and api version will be used by `Luna-Video-Manager` for making websocket subscription to receive data from agent. The parameters from Luna-Video-Manager reply are also valuable and must be taken into account by video-agent .. dropdown:: *agent_id* :animate: fade-in-slide-down An id of agent generated by Luna-Video-Manager which must be used by video-agent in future requests to Luna-Video-Manager .. dropdown:: *refresh_period* :animate: fade-in-slide-down A video agent must repeat the `get streams for processing request <_static/api.html#operation/postAgentStreams>`_ at least this period, otherwise the video-agent will be excluded from list of active video-agents and `Luna-Video-Manager` will not distribute streams to this agent (see `agents downgrade <./downgrade.html#agents-downgrade>`_ for details) .. dropdown:: *alive_period* :animate: fade-in-slide-down The period which agent must works autonomously without connection to Luna-Video-Manager. It does not mean that video-agent might work without agent registration. The provided primitive agent example contains agent registration example `here <#anchor-agent-py>`_. .. _anchor-get-streams: Get streams for processing -------------------------- The video-agent must decode and execute analytics on stream provided by Luna-Video-Manager. To get streams which must be processed by video-agent it must periodically makes `get streams for processing request <_static/api.html#operation/postAgentStreams>`_ (see `agent registration <#anchor-agent-registration>`_ and `agents downgrade <./downgrade.html#agents-downgrade>`_ for related information). Is it required to correctly specify the *agent_id* received during `agent registration <#anchor-agent-registration>`_ in request. The reply can be empty what will mentions that there is no new streams for processing are appointed by Luna-Video-Manager to the video-agent. When reply is not empty, it will contains list of streams with much data, including stream reference, type, parameters for video-decoder and many other parameters, each of them must be proceed in the right place. The once agent receives stream for processing it must start stream processing and start `feedback sending <#anchor-feedback>`_ without any delay. Otherwise, it is not possible to `Luna-Video-Manager` for properly distribute streams to agents. It is not expected that stream with the same id and version appears in one request. See `streams distribution chapter `_, `streams downgrade chapter `_ and `agents downgrade chapter `_ for related information. .. dropdown:: Streams parameters and its' appointments :animate: fade-in-slide-down Parameters *stream_id*, *version* and *analytic_index* (from *analytics*) must be used in `feedback request <_static/api.html#operation/postStreamsFeedback>`_ to Luna-Video-Manager. Parameter *feedback_frequency* specified the maximum delay of feedback for stream must be sent. See `feedback sending paragraph <#anchor-feedback>`_ for related information. Parameter *version* must be used by agent to replace stream with the same id which is in process, cause *version* increase means that stream has been replaced by user and can contains all new parameters instead of *stream_id*. Parameters from *data* section must be used by video-decoder (and optionally, if video downloads). Parameters from *analytics* section contains list of analytics and its' parameters which must process the stream. Parameters from *notification_policy* are intended to specify when and what callbacks must be triggered by video-agent when stream change it's processing status. Other parameters such as *source*, *location* and other must be used in analytics callbacks and websocket notifications. The provided primitive agent example contains streams getting example `here <#anchor-agent-py>`_. The provided primitive agent example contains streams processing example `here <#anchor-stream-processing-py>`_. .. _anchor-feedback: Feedback sending ---------------- The video-agent must periodically (for details, see parameters description in `streams getting paragraph <#anchor-get-streams>`_) send feedback for all streams which are processed by video-agent to make Luna-Video-Manager knows actual statuses of stream processing. If Luna-Video-Manager not receive feedback in time streams processing will not happens as expected, see `streams downgrade chapter `_ for details. To send feedback, the video-agent must use `send feedback request <_static/api.html#operation/postStreamsFeedback>`_ including: - *stream_id*, *status* - stream id and actual stream processing status - *version* - actual stream version - *time* - feedback generation time in format RFC 3339 - *error* - text with error or message denoting that no errors occurred during stream processing - *analytics indexed* - the same indexes as *idx* from *analytics* in `get streams <_static/api.html#operation/postAgentStreams>`_ reply - *analytics_intervals* - actual analytics working intervals versions The reply on this requests from Luna-Video-Manager are also contains useful information which must by taken into account by video-agent (for each stream): - *stream_id* and *version* - for comparison with current stream id and it's version: if they are not the same, it means that stream version was updated and currently processing stream must be stopped - *action* - action which video-agent must apply to stream - continue it's processing or stop - optional *new_analytics_intervals* which contains new analytics working intervals and its' versions: if this field appears in feedback request reply, the new analytics working intervals must be immediately applied For more information about analytics working intervals see `this paragraph <#anchor-intervals>`_. .. dropdown:: Other situations when feedback must be sent independently of feedback sending period :animate: fade-in-slide-down - Agent receive new stream for processing, validate parameters and concludes that stream processing is not possible (for example, specified stream reference url is not available). In this case `Luna-Video-Manager` assumes that agent send feedback with stream `failed` status and agent will not process such stream. - Agent receive stream which processing must be stopped (for example due to user stop stream processing by making `stream patch request <_static/api.html#operation/patchStream>`_). In this case `Luna-Video-Manager` assumes that agent send feedback with stream `stopped` status and stop stream processing. - During stream processing fatal (fatal means that stream processing is not longer possible) exception occurred (for example, stream suddenly became unavailable). In this case `Luna-Video-Manager` assumes that agent send feedback with `failed` stream status and stop stream processing. - During stream processing or callback sending an non-fatal unexpected exception occurred (for example, callback url is unavailable). In this case `Luna-Video-Manager` assumes that agent send feedback with stream `in_progress` status and error contains text with exception description and stream processing will continue. - Analytics processing is not currently running as scheduled. - Stream ends (more applicable for video file, but rtsp-stream also can ends). In this case `Luna-Video-Manager` assumes that agent send feedback with `done` status and stop stream processing. - All other significant information also can be send to `Luna-Video-Manager` for better understanding of stream processing progress. The provided primitive agent example contains streams processing example `here <#anchor-stream-processing-py>`_. The provided primitive agent example contains feedback processing example `here <#anchor-feedback-py>`_. .. _anchor-intervals: Analytics working intervals --------------------------- There is a possibility that user specify some intervals when each analytics must work and when must not works. Within framework of Luna-Video-Manager and video-agent interaction such intervals named as analytics working intervals and analytics which must now work at the current moment is *paused*. If all stream analytics must not work at the moment so stream status is *paused*. The video-agent receives analytics working intervals during `getting streams for processing <#anchor-get-streams>`_ and it is possible that these intervals can be updated, removed or firstly appears in reply during `feedback processing <#anchor-feedback>`_. Independently of when and how analytics working intervals received, the video-agent must execute analytics according to actual intervals. .. dropdown:: Case when no analytics must work according to working intervals :animate: fade-in-slide-down It is possible to continue video decoding in this case, but it is not recommended. It is recommended to stop video decoding if all analytics are paused and continue only when at least one analytics must works. .. dropdown:: Feedback on paused analytics :animate: fade-in-slide-down When stream processing is paused (no one analytics must works according to working intervals) or at least one of analytics is paused, video-agent must send feedback with corresponding status to make Luna-Video-Manager knows that analytics processing works as expected and will continue in expected moment. See `feedback processing <#anchor-feedback>`_ for related information. The provided primitive agent example contains streams processing example `here <#anchor-stream-processing-py>`_. The provided primitive agent example contains working intervals processing example `here <#anchor-intervals-solver-py>`_. .. _anchor-ws: Websocket subscription ---------------------- During stream processing it is possible the need to get information about each analytics processing results arise. For this case video-agent must provide an opportunity to get all this information using websocket subscription: - it must `//ws` url available for ws subscription (where `api_version` is positive integer number which means api version of video-agent) - it must support required query parameters: *stream_id* and *analytic_index* where *stream_id* is id of stream of interest and the *analytic_index* is the same index as the one from *analytics* in `get streams <_static/api.html#operation/postAgentStreams>`_ reply - it must support optional query parameter *account_id* which must works as filter and not allow to receive data for stream with another *account_id* The provided primitive agent example contains websocket handler example `here <#anchor-ws-handler-py>`_. The provided primitive agent example contains websocket data provider example `here <#anchor-ws-provider-py>`_. The provided primitive agent example contains script which make ws subscription example `here <#anchor-ws-subscribe-py>`_. .. _anchor-shutdown: Graceful shutdown ----------------- It is not required, but highly recommended to graceful shutdown of video-agent when it stops to make data in Luna-Video-Manager consistent and works faster. It is recommended to make 2 actions before video-agent stops: - stop all stream processing and send corresponding feedback with corresponding messages (see `feedback paragraph <#anchor-feedback>`_ for details) .. dropdown:: What will happen if this is not done? :animate: fade-in-slide-down The `streams downgrade mechanisms <./downgrade.html#streams-downgrade>`_ will be executed and it will waste time that could have been saved. - remove itself from Luna-Video-Manager by using `delete agent request <_static/api.html#operation/removeAgent>`_ .. dropdown:: What will happen if this is not done? :animate: fade-in-slide-down The `agents downgrade mechanisms <./downgrade.html#streams-downgrade>`_ and `agents auto deletion mechanisms <./agent_auto_deletion.html>`_ will be executed and it will waste time that could have been saved. The provided primitive agent example contains agent graceful shutdown example `here <#anchor-agent-py>`_. .. _anchor-example-explanation: Primitive agent example explanation ----------------------------------- In this paragraph presented primitive agent example structure with explanation (for each module) of differences between the example and fully-fledged video-agent solution. This examples demonstrates how video-agent works: - register analytics and agent itself - get streams for processing - process streams using decoder and fake analytics - process analytics working intervals - imitates feedback and callbacks sending - send websocket events This primitive example is available as zip-archive, for link, preparation and execution description, see `this paragraph <#anchor-example-download>`_. .. warning:: It is critically not recommended to use this or similar solutions in production or development, this solution is highly suboptimal. This example is only intended for demonstration and research of Luna-Video-Manager and video-agent interaction rules. .. code-block:: :caption: Agent example file structure ├──agent.py ├──analytics.py ├──cli_args.py ├──decoder.py ├──feedback.py ├──intervals_resolver.py ├──run.py ├──stream_processing.py ├──structs_n_consts.py ├──ws_handler.py ├──ws_provider.py └──ws_subscribe.py .. _anchor-agent-py: .. dropdown:: (agent.py) Module which contains agent-example high-level, initialize and shutdown methods. :animate: fade-in-slide-down The fully-fledged video-agent may be similar to this part of example except for one case when stream must be replaced with stream which has the same id but higher version (triggered by `stream replace request <_static/api.html#operation/putStream>`_). .. dropdown:: This module is related to the following processes described above: :animate: fade-in-slide-down - `Analytics registration <#anchor-analytics-registration>`_. - `Agent registration <#anchor-agent-registration>`_. - `Agent requests to get stream for processing <#anchor-get-streams>`_. - `Agent graceful shutdown <#anchor-shutdown>`_. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/agent.py .. _anchor-analytics-py: .. dropdown:: (analytics.py) Module which contains agent-example fake analytics. :animate: fade-in-slide-down The fully-fledged video-agent cannot be similar to this part of example. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/analytics.py .. _anchor-cli-args-py: .. dropdown:: (cli_args.py) Module which contains agent-example cli arguments parser. :animate: fade-in-slide-down The fully-fledged video-agent scarcely cannot be similar to this part of example cause fully-fledged video-agent is a complex solution that required a lot of settings. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/cli_args.py .. _anchor-decoder-py: .. dropdown:: (decoder.py) Module which contains agent-example decoder. :animate: fade-in-slide-down The fully-fledged video-agent cannot be similar to this part of example cause it does not process any decoding parameters specified in *data* section (see `get streams for processing <#get-streams-for-processing>`_ for related information). .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/decoder.py .. _anchor-feedback-py: .. dropdown:: (feedback.py) Module which contains agent-example feedback processing functions. :animate: fade-in-slide-down The fully-fledged video-agent cannot be similar to this part of example due to several reasons, but provides basic functionality which allows this example to works as real video-agent: - *error* field does not describe real error - *last_processed_frame_time* is fake - *analytic_indexes* hardcoded and not assumes any other values .. dropdown:: This module is related to the following processes described above: :animate: fade-in-slide-down - `Stream processing and agent feedback requests <#anchor-feedback>`_. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/feedback.py .. _anchor-intervals-solver-py: .. dropdown:: (intervals_solver.py) Module which contains agent-example working intervals resolver. :animate: fade-in-slide-down The fully-fledged video-agent cannot be similar to this part of example due to several reasons, but provides basic functionality which allows this example to works as real video-agent: - it has no possibilities to process working intervals for all stream analytics - it is suboptimal - it has no possibilities to wait and execute next calculation only in time when next analytics state must switch .. note:: The algorythm of calculation is not the most suboptimal algorythm in this example and can be used in fully-fledged solution with/without minimalistic improvements. .. dropdown:: This module is related to the following processes described above: :animate: fade-in-slide-down - `Analytics working intervals <#anchor-intervals>`_. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/intervals_solver.py .. _anchor-run-py: .. dropdown:: (run.py) Module which contains agent-example execution functions. :animate: fade-in-slide-down The fully-fledged video-agent might be similar to this part of example except for it must provide a lot of other features. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/run.py .. _anchor-stream-processing-py: .. dropdown:: (stream_processing.py) Module which contains agent-example stream processing functions. :animate: fade-in-slide-down The fully-fledged video-agent cannot be similar to this part of example due to several reasons, but provides basic functionality which allows this example to works as real video-agent: - it does not provide and processing of *notification_policy*, but it might be similar to logs in some cases (for example - stream processing start, finish) - it does not provide possibility of analytics callbacks execution - there is no processing of stream replacement (triggered by `stream replace request <_static/api.html#operation/putStream>`_) - there is not processing of case when all analytics are paused so decoder works always .. dropdown:: This module is related to the following processes described above: :animate: fade-in-slide-down - `Stream processing and agent feedback requests <#anchor-feedback>`_. - `Analytics working intervals <#anchor-intervals>`_. - `Websocket interface <#anchor-ws>`_. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/stream_processing.py .. _anchor-structs-n-consts: .. dropdown:: (structs_n_consts.py) Module which contains agent-example auxiliary structs and constants. :animate: fade-in-slide-down .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/structs_n_consts.py .. _anchor-ws-handler-py: .. dropdown:: (ws_handler.py) Module contains agent-example websocket handler. :animate: fade-in-slide-down The fully-fledged video-agent cannot be similar to this part of example due to several reasons, but provides basic functionality which allows this example to works as real video-agent: - it does not validate query parameters - there is no error processing which can reply to with a user specific error .. dropdown:: This module is related to the following processes described above: :animate: fade-in-slide-down - `Websocket interface <#anchor-ws>`_. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/ws_handler.py .. _anchor-ws-provider-py: .. dropdown:: (ws_provider.py) Module which contains agent-example websocket provider. :animate: fade-in-slide-down The fully-fledged video-agent might be similar to this part of example. .. dropdown:: This module is related to the following processes described above: :animate: fade-in-slide-down - `Websocket interface <#anchor-ws>`_. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/ws_provider.py .. _anchor-ws-subscribe-py: .. dropdown:: (ws_subscribe.py) Module which contains simple websocket subscription script example. :animate: fade-in-slide-down This module is not part of video-agent example, but just shows how can websocket subscription can be implemented. .. dropdown:: Code source. :animate: fade-in-slide-down .. literalinclude:: ./agent_example/ws_subscribe.py .. _anchor-example-download: Primitive agent example execution --------------------------------- .. warning:: It is critically not recommended to use this or similar solutions in production or development, this solution is highly suboptimal. This example is only intended for demonstration and research of Luna-Video-Manager and video-agent interaction rules. .. card:: The primitive agent example .. only:: builder_html :download:`Source code is available for download here <./agent-example.zip>` .. dropdown:: Preparation and execution. :animate: fade-in-slide-down To prepare this example, it needs Luna-Video-Manager is available and some requirements to be installed, for example, this way: .. code-block:: bash unzip agent-example.zip cd agent_example python3.13 -m venv venv source venv/bin/activate pip install -r requirements.txt To run this example .. code-block:: bash python run.py --port=5555 Where 5555 is default port and it is allow to not specify it or use another port.