Indexes¶

Indexes are tree-based data structures in Matcher that organize projection data for fast search operations. An index defines which fields can be used to filter data during search and how the data is structured in memory. Properly configured indexes are critical for achieving high performance in Luna-Vinder.

Index Concept¶

After creating a projection, you need to define how Matcher will organize that data for efficient searching. This is where indexes come in. An index specifies a list of fields called composite fields that determine two important things: which attributes from the projection can be used as filters in search requests, and how the data will be hierarchically organized in Matcher’s memory for quick access.

When Matcher loads an index, it builds a tree structure where data is progressively divided according to the values of the specified composite fields. This tree allows Matcher to quickly navigate to the relevant subset of descriptors without scanning the entire dataset. For example, if your index has composite fields for gender and age, Matcher can instantly locate all descriptors for males aged 30 without examining descriptors for females or other age groups.

The key principle is that only fields included in the index’s composite fields can be used for filtering during search. If you want to search by a particular attribute, that attribute must be in the composite fields list. If a search request includes filters for fields not in the index, Matcher cannot process that request.

Note

The leaf nodes of the index tree store lists of descriptors belonging to each leaf. When a matching request with filters is received, Matcher traverses the tree to find all relevant descriptor lists, groups them into a single chunk, and performs the matching operation on this chunk.

Index Components¶

Projection Reference¶

Each index is built on a specific projection, referenced by its projection_id. The index can only work with data and attributes that are defined in that projection. This means the projection’s targets must include all attributes you want to use in the index’s composite fields.

Important

The projection must exist before you create an index on it. The projection defines what data is available, while the index defines how to organize that data for search.

Composite Fields¶

Composite fields are the heart of the index configuration. This is an ordered list of attribute names that determines the structure of the search tree and which filters can be applied during search.

The order of fields in composite fields is critically important and directly impacts search performance. Fields must be arranged in ascending order of cardinality - from fields with the fewest unique values to fields with the most unique values. Cardinality refers to the number of distinct values a field can take.

For example, a gender field typically has only 2-3 unique values (male, female, undefined) - this is low cardinality. An age field might have dozens of unique values - medium cardinality. A end_time timestamp field can have millions of unique values - high cardinality. The correct order would be: gender, then age, then end_time.

Why does order matter? Matcher builds a tree where each level corresponds to one composite field, and data is split by the values of that field. If the first field has low cardinality, the tree starts with a small number of branches - for example, 3 branches for gender. Each of those branches then splits further based on the next field. This creates a balanced, efficient tree structure.

If you place a high-cardinality field first, such as a timestamp with millions of unique values, the tree becomes extremely wide at the first level with millions of branches. This creates an inefficient structure that’s slow to navigate and consumes excessive memory. Always start with low-cardinality fields and progress to higher-cardinality fields.

Warning

Incorrect field ordering can severely degrade search performance. Always analyze the cardinality of your fields before configuring an index.

Empty composite fields (an empty list) are also valid and create an index with no filtering capability. Such an index allows searching across all data in the projection without applying any attribute filters. This can be useful for simple similarity searches where you want to compare against all available descriptors.

Note

The create_time field is always present inside every index, so filtering by this field is always available. There is dedicated logic and low-level optimizations for working with create_time to ensure efficient filtering and search performance.

Note

Currently, the index allows filtering fields using the following filter types: eq, neq, in, nin, gt, gte, lt, lte, like. The values referenced by index fields must be compatible with these filter types if you intend to use them for filtering.

Descriptor Version and Type¶

Indexes are specific to particular descriptor versions and types. When Matcher loads an index, it only includes descriptors that match the configured version and type. This allows you to maintain multiple indexes for different descriptor formats or algorithms, each optimized for its specific use case.

Create Time Ordering and Filtering¶

Index data is loaded sorted by the create_time field during initial loading. This ordering is critical for optimal filtering performance on time-based queries. The system expects that new data added after initial loading will have monotonically increasing create_time values - each new record should have a create_time greater than or equal to previously added records.

When data is added in the expected ascending create_time order, time-based filtering operates with maximum efficiency. The sorted structure allows Matcher to quickly locate relevant time ranges without scanning unrelated data. However, if records are added with create_time values that break this ordering - for example, adding older records after newer ones - filtering by create_time will still work correctly but with significantly degraded performance. The system must perform additional scanning to ensure all matching records are found.

A small degree of out-of-order insertion is tolerated. The system maintains a configurable buffer (determined by the minimum synchronization chunk size) that accommodates minor timing variations in record creation. This buffer allows for some records to arrive slightly out of order without performance impact. However, the general trend must be toward increasing create_time values. If your data ingestion pattern consistently produces records with decreasing or highly random create_time values, you should reconsider your data pipeline to ensure proper temporal ordering before data reaches the index.

Warning

While filtering by create_time remains functionally correct regardless of insertion order, maintaining ascending create_time order is essential for performance. Design your data ingestion process to preserve temporal ordering whenever possible.

Creating Indexes¶

Indexes are configured through Luna Configurator by adding entries to the LUNA_VINDER_INDEXES setting. Each index configuration specifies the projection to use and the ordered list of composite fields.

The configuration format is a list of index definitions, where each definition contains:

projection_id - UUID of the projection this index is built on
index_composite_fields - ordered list of field names for the index

Example configuration:

LUNA_VINDER_INDEXES = [
    {
        "projection_id": "49ddbfb3-4b19-4c00-a165-df2a9fc7f321",
        "index_composite_fields": ["gender", "age"]
    },
    {
        "projection_id": "a7b2c891-5d3e-4f12-8c9a-1e4f5b6d7c8e",
        "index_composite_fields": ["handler_id", "age", "emotion"]
    },
    {
        "projection_id": "3f8d9c2b-1a4e-4d5f-9b8c-7e6d5c4b3a2f",
        "index_composite_fields": []
    }
]

In this example, three indexes are defined: one allowing filtering by gender and age, another by handler, age, and emotion, and a third with no filtering fields for unrestricted similarity search.

After adding or modifying index configurations in Configurator, Matcher services automatically detect the changes and begin the loading process. Matcher contacts Projector to access the specified projection, retrieves all data according to the projection’s filters and targets, loads the descriptors into memory sorted by create_time, and builds the tree structure according to the composite fields specification.

This loading process can take considerable time, especially for projections containing millions of records. During loading, the index is not yet available for search. Once loading completes, Matcher marks the index as ready and begins accepting search requests that match the index’s capabilities.

Note

Remember that composite fields must be listed in ascending order of cardinality. In the first example above, gender has lower cardinality than age, so it appears first.

Index Selection and Matching¶

When a search request arrives through the matching plugin, the plugin analyzes the request filters and determines which indexes can handle it. For an index to be selected, several conditions must be met.

The origin specified in the search request must match the projection’s origin. The descriptor version and type in the request must match the index configuration. Most importantly, all filter fields specified in the search request must be present in the index’s composite fields. If the request includes even one filter for a field not in the composite fields, that index cannot be used.

Additionally, the projection’s filters must be compatible with the search request. The projection filters define a subset of data, and the search request can only further narrow that subset, not expand it. For example, if a projection has filter age__gte: 50 (age greater than or equal to 50), a search request with age__gte: 60 is compatible because it’s more restrictive. However, a request with age__gte: 30 would not be compatible because the projection doesn’t include ages 30-49.

This selection logic ensures that searches are routed to indexes that can actually process them, and results are accurate based on the available data.

Index Synchronization¶

Matcher continuously monitors projections for changes. When new events are added to a projection, when existing events are updated, or when events are removed, Matcher detects these changes and updates its in-memory indexes accordingly. This ensures that search results always reflect the current state of the data.

The synchronization happens automatically in the background without requiring manual intervention. The frequency of synchronization checks is configurable in Matcher settings, allowing you to balance data freshness requirements against system load.

During synchronization, new records are appended to the index. For optimal performance, ensure that your data ingestion process adds records with monotonically increasing create_time values. This maintains the temporal ordering established during initial loading and preserves efficient time-based filtering capabilities.

Performance Configuration¶

Thread Count¶

The THREAD_COUNT setting in the LUNA_VINDER_MATCHING configuration section determines how many parallel threads Matcher uses for search operations. This parameter has a significant impact on matching performance.

The default value of 16 threads is optimal for most modern server configurations and provides good parallel processing capabilities. However, the optimal value depends on your hardware configuration. If your system has fewer CPU cores than the configured thread count, performance gains plateau because threads compete for CPU resources. In such cases, set THREAD_COUNT equal to the number of available CPU cores for optimal performance.

The impact of thread count on performance can be substantial. For example, increasing from 8 to 16 threads can provide up to 2x speedup in matching operations on systems with sufficient CPU cores. The actual speedup depends on your specific workload, index size, and hardware configuration. Higher thread counts allow more search operations to execute concurrently, reducing overall response time when handling multiple simultaneous requests or processing large result sets.

When configuring THREAD_COUNT, consider your typical query patterns and concurrent load. If your service handles many simultaneous search requests, higher thread counts improve throughput. If queries are typically sequential or your hardware has limited cores, a lower thread count may be more appropriate.

Note

Monitor CPU utilization and query response times when adjusting THREAD_COUNT. If CPU usage is consistently below 100% and response times are high, increasing thread count may help. If CPU is saturated and context switching overhead is high, reducing thread count might improve performance.

Best Practices¶

When designing indexes, follow these principles to achieve optimal performance and functionality.

Analyze field cardinality carefully before defining composite fields. Use tools or queries to understand how many unique values each field contains in your dataset. List fields in strict ascending order of cardinality. Starting with the lowest-cardinality fields creates the most efficient tree structure. A common mistake is placing high-cardinality fields like timestamps or unique identifiers first, which creates inefficient indexes.

Include only fields you actually need for filtering. Each field in composite fields adds a level to the tree structure and increases memory usage. If you never filter by a particular attribute, don’t include it in composite fields even if it’s in the projection’s targets. Keep the index as simple as possible while meeting your filtering requirements.

Create multiple indexes on the same projection for different query patterns. If you have different search scenarios that require different filter combinations, create separate indexes optimized for each scenario. This is better than trying to create one complex index that serves all cases. Each index can be optimized for its specific use case with appropriate field ordering.

Consider your query patterns when designing indexes. Think about which filters are most commonly used together in your searches. Make sure those fields are included in your index composite fields. If certain filters are rarely used, consider whether they need to be in the index at all, or whether those queries can be served by a separate specialized index.

Maintain temporal ordering in your data pipeline. Design your data ingestion process to ensure records arrive with monotonically increasing create_time values. This preserves the sorted structure of the index and ensures optimal performance for time-based filtering. If your source data has inconsistent timestamps, consider adding a processing step to normalize or re-order records before they reach Luna-Vinder.

Monitor index size and loading times. Large indexes consume significant memory and take time to load. If an index becomes too large, consider whether the underlying projection can be more aggressively filtered to reduce its size, or whether the data should be split across multiple projections and indexes.

Document your index configurations. When managing multiple indexes, clear documentation helps the team understand what each index is for, which queries it supports, and why it was configured a particular way. This prevents confusion and duplicate configurations.

Troubleshooting¶

Several common issues can arise when working with indexes.

If searches are not being routed to your index, verify that all filter fields in your search requests are present in the index composite fields. Check that the projection’s filters are compatible with your search filters - the search cannot be broader than the projection. Ensure the descriptor version and type match between the search request and index configuration.

If index loading is slow or fails, check the size of the underlying projection - very large projections take longer to load. Verify that Projector is accessible and the projection exists. Review Matcher logs for specific error messages during loading. Consider whether the projection can be filtered more aggressively to reduce data volume.

If search performance is poor despite having an index, examine your composite fields ordering - incorrect cardinality ordering creates inefficient trees. Check if the index is too large and causing memory pressure. Consider whether multiple smaller indexes might be more efficient than one large index. Monitor whether searches are hitting the same index concurrently and causing contention.

If time-based filtering is slower than expected, verify that your data ingestion process maintains ascending create_time order. Check whether records are being added out of order, which forces the system to perform additional scanning. Review your synchronization logs to identify patterns of out-of-order insertions. If necessary, implement preprocessing to sort records by create_time before ingestion.