Skip to content

Overview#

LUNA Vinder Module is a module designed to enable flexible and high-performance event matching.

When working with millions/billions of descriptors, directly using a single LUNA PLATFORM database for comparison tasks creates a number of limitations:

  • It's difficult to efficiently work with different subsets of data — if you only need to compare events of a certain type or source, the database will still scan the entire table;
  • Parallel mass comparison queries create a load on the database, which can negatively impact its core functionality;
  • It's impossible to control which attributes to store and index for different comparison scenarios.

LUNA Vinder Module solves these problems by creating separate, specialized copies of the data — projections. The key advantage of this approach is that data can be grouped by specific characteristics into separate projections and the attributes stored within them can be selected. The data source remains the main database, and projections are "working copies" optimized for specific comparison scenarios.

Comparison with LUNA Vector Search Module

LUNA PLATFORM offers two modules for high-performance descriptors matching: LUNA Vinder Module and LUNA Vector Search Module. They use different approaches and are designed for different use cases:

  • LUNA Vinder Module performs the comparison procedure in RAM, ensuring maximum performance with properly configured indexes. It allows the creation of multiple indexes per projection for different search scenarios. It is limited by available RAM.
  • LUNA Vector Search Module uses the Qdrant vector database for matching, which is not limited by RAM (data can be stored on disk or distributed across multiple servers) and supports flexible filtering by time intervals and other attributes with a large number of unique values.

The basic principle of LUNA Vinder Module operation

A projection is created using the Projector service based on the data source, filters defining which events to include in the projection, and target fields. At this stage, Projector creates an "empty" projection—this is a definition of what data should be in this projection, but the data itself has not yet been loaded. For the created projection, one or more indexes—tree structures that define which fields can be filtered during comparison—are configured in the Configurator service. After the index is configured, the Matcher service loads the projection data into RAM, builds the indexes, and is ready to process requests.

When a user sends a match request through the API, it is redirected to the Python Matcher Proxy service. Its built-in comparison plugin determines whether the matching should be performed in the Python Matcher service or in the Matcher service of the LUNA Vinder Module module. After the matching is completed, a response is returned to the user.

This document contains the following main sections:

  • General concepts. The section contains:

    • Description of projections and index
    • Description of the matching

    It is recommended to start your acquaintance with LVM from this section.

  • Matching plugin for Python Matcher Proxy. This section describes the operation of the matching plugin built into the Python Matcher Proxy service, which is required to perform the matching.

  • Monitoring. This section describes the monitoring process for LVM services.
  • API errors. This section provides an extended description of the errors returned by LVM services.

  • Configuration parameters of services. This section describes the parameters for all LVM services.

  • Sequence diagrams. В разделе приведены диаграммы последовательности для основных операций LVM.