Qdrant

Storage

All data within one collection is divided into segments. Each segment has its independent vector and payload storage as well as indexes.

See storage configuration for details.

  • Vector storage

You can configure one of the data storage options depending on search speed requirements and the size of RAM used:

  1. In-memory storage

    Stores all vectors in RAM, has the highest speed since disk access is required only for persistence.

  2. Memmap storage

    Creates a virtual address space associated with the file on disk. Mmapped files are not directly loaded into RAM. Instead, they use page cache to access the contents of the file. This scheme allows flexible use of available memory. With sufficient RAM, it is almost as fast as in-memory storage.

  • Payload storage

You can configure one of two types of payload storages:

  1. InMemory

    InMemory payload storage is organized in the same way as in-memory vectors. This type of storage works quite fast, but it may require a lot of space to keep all the data in RAM.

  2. OnDisk

    In the case of large payload values, it might be better to use OnDisk payload storage. But if you need to query vectors with some payload-based conditions - checking values stored on disk might take too much time. In this scenario, it’s recommended creating a payload index for each field used in filtering conditions to avoid disk access. Once you create the field index, Qdrant will preserve all values of the indexed field in RAM regardless of the payload storage type.

  • Vector index

To speed up vector search, you can build a vector index.

A vector index is a data structure built on vectors through a specific mathematical model. For now, HNSW index for approximate search is only supported. Creating an index requires additional computational resources and memory, and can take a long time.

See vector index for details.

  • Payload index

To speed up filtering, you can build a payload indexes.

Payload index is built for a specific field and type, and is used for quick record requests by the corresponding filtering condition. Choosing fields to be indexed is essential, and is granted to the user.

See payload index for details.

Here are some examples of marking fields as indexable, according to field datatype (uuid, datetime, int, etc.). Please note, you can use dot notation to specify a nested field.

PUT /collections/{collection_name}/index
{
   "field_name": "handler_id",
   "field_schema": "uuid"
}

PUT /collections/{collection_name}/index
{
   "field_name": "create_time",
   "field_schema": "datetime"
}

PUT /collections/{collection_name}/index
{
   "field_name": "age",
   "field_schema": "int"
}

PUT /collections/{collection_name}/index
{
   "field_name": "upper_body.sleeve.length",
   "field_schema": "text"
}

Managing collections

You can use Web UI to manage collections, track indexing and configuration status, etc. See collection info for details.

Collection configuration parameters can be set up dymanically. See collection update configuration for details.

Here is an example of updating the above mentioned storage configuration for a collection:

PATCH /collections/{collection_name}
{
   "vectors_config": {
      "face_65: {
        "on_disk": false
     },
     "body_105: {
        "on_disk": false
     }
   },
   "hnsw_config": {
      "on_disk": false
   },
   "params": {
      "on_disk_payload": false
   }
}

Sharding

A collection can be made of one or more shards. A shard is an independent store of records which is able to perform all operations provided by collections. For automatic sharding, records are distributed among shards by using a consistent hashing algorithm, so that shards are managing non-intersecting subsets of records. Each node knows where all parts of the collection are stored through the consensus protocol, so when send a search request to one node, it automatically queries all other nodes to obtain the full search result.

See distributed deployment configuration for details.

Benchmark

250M collection with full-event payload

50M collection with minimal-event payload (compared to other platform-integrated tools)