VectorDBBench 2.0.0

VectorDBBench 2.0 is a major expansion of the project’s benchmark coverage. This release introduces three major capabilities:

1. Production-oriented cloud benchmark cases and a Cloud Leaderboard
2. First-class full-text search benchmarking
3. A substantially expanded public dataset catalog

1. ☁️ Cloud Benchmark Cases and Leaderboard

VectorDBBench previously focused primarily on insertion throughput, vector search performance, recall, and filtering. Version 2.0 adds benchmark cases designed around behavior commonly encountered in managed and serverless vector databases.

Cloud Insert

CloudInsertCase measures both write throughput and data readiness:

• Client-side insertion throughput
• Time until inserted vectors become searchable
• Time until the database reports the vectors as fully indexed

This helps distinguish accepting a write from actually being ready to serve production queries.

Payload Search

CloudPayloadSearchCase measures search performance with different response payloads:

• Vector IDs only
• Scalar metadata and labels
• Vector values

It supports unfiltered search, integer filtering, and label filtering. Results include QPS, latency, recall where applicable, and estimated response size.

This exposes the cost of returning production-shaped responses instead of measuring only minimal ID-only search requests.

Cold Latency

CloudColdLatencyCase measures first-query and cold-path latency separately from warm repeated-search latency.

This is particularly relevant to:

• Serverless databases
• Storage-tiered systems
• Intermittently active applications
• Workloads sensitive to the first request after an idle period

Multi-tenant Search

CloudMultiTenantSearchCase distributes records across deterministic tenants and routes queries to their corresponding tenant labels or namespaces.

This models SaaS workloads where tenant isolation, namespaces, or partition routing may behave differently from searches over a single global collection.

Cloud Leaderboard

These cases provide the foundation for a Cloud Leaderboard that compares databases across production-oriented dimensions including:

• Insert throughput
• Searchable and indexed readiness
• Warm and cold latency
• Payload-aware throughput
• Recall
• Multi-tenant performance

Contributor: @jamesgao-jpg.

2. 🔤 Full-Text Search

VectorDBBench 2.0 introduces BM25-style full-text retrieval as a first-class benchmark workload.

The full-text benchmark measures:

• Document ingestion
• Text index construction and optimization
• Serial retrieval quality
• Concurrent search throughput
• Search latency
• Response payload size

Quality is evaluated using semantic relevance judgments from ir_datasets. Results can include:

• Recall
• Mean Reciprocal Rank
• NDCG
• QPS
• Latency distributions

Filtered full-text search

Full-text workloads can also include deterministic scalar filters with configurable filter rates. This makes it possible to evaluate both ranking quality and performance when text retrieval is combined with structured filtering.

Payload-aware search

Full-text benchmarks support different response profiles, including:

• Document IDs
• Scalar metadata
• Document text

This makes the workload representative of applications that retrieve usable document content rather than only internal identifiers.

Contributors: @jamesgao-jpg and @Denise2004.

3. 🗃️ New Dataset Support

VectorDBBench 2.0 substantially expands its public dataset catalog for vector search benchmarks.

Hugging Face Download Support

Hugging Face has long been a destination for state-of-the-art public datasets. In 2.0, we introduced Hugging Face dataset support, making it easier for the community to contribute valuable datasets to VDBBench. Alongside this integration, we introduced the following two dataset families.

VDBBench multimodal datasets

The VDBBench team created and published three multimodal embedding datasets for vector database benchmarking. Each image-caption pair is encoded as one 4,096-dimensional, L2-normalized float32 vector using Qwen3-VL-Embedding-8B. Each dataset provides 10,000 queries and precomputed top-100 ground truth for inner-product search.

• multimodal-embedding-1m contains one million base vectors generated from Conceptual Captions 3M image-text pairs.
• multimodal-embedding-10m contains ten million base vectors generated from Conceptual Captions 12M image-text pairs.
• multimodal-embedding-100m contains one hundred million base vectors generated from LAION-400M-derived image-caption data.

Together, they provide a consistent modern multimodal workload at three scales, from one million to one hundred million vectors.

Dataset contributor: @zhuwenxing.

VIBE datasets

In addition, we integrated all 24 datasets introduced in paper VIBE: Vector Index Benchmark for Embeddings

The catalog includes:

• In-distribution and out-of-distribution queries
• L2, cosine, and inner-product distance metrics
• Text, image, and multimodal embeddings
• Multi-vector-derived embeddings
• Dimensions ranging from 200 to 5,120
• Dataset sizes ranging from hundreds of thousands to tens of millions of vectors

The VIBE datasets make it possible to compare databases across more varied embedding models, domains, dimensionalities, and query distributions. We thank the authors of VIBE paper for this wonderful contribution to the community. More details are available from the VIBE project.

VDBBench integration contributor: @jamesgao-jpg.

Additional capabilities

Large TopK Support

As demand for million-scale TopK search grows, VectorDBBench now provides LAION-100M Large-TopK support, with K values up to 1,000,000.

Contributors: @jamesgao-jpg and @frankleaf.

New database integrations and supports

As the vector database community continues to grow, VDBBench continues providing supports for vendors:

• Alibaba Cloud ADB-PG Nova, including Nova index modes and RaBitQ-related settings — @softhuafei
• VolcMySQL with native vectors, HNSW, and vector quantization settings — @FishMage
• Amazon OpenSearch Serverless with SigV4 authentication — @norrishuang
• LanceDB index, filtering, batch insertion, and remote-storage support — @XuQianJin-Stars
• AliSQL shard, quantization, and concurrent insertion support — @JoeJRW
• Lindorm index tuning — @Poisky
• Hologres loading and concurrent insertion — @zpatronus
• MongoDB exact-search configuration — @daviddallakyan2005

Benchmark controls

In 2.0 we shipped several features to further improve our user experiences. These include:

• Configurable insertion batch size — @jamesgao-jpg
• Rate-controlled streaming insertion — @jamesgao-jpg
• Configurable cooldown between concurrent and serial search — @zpatronus
• Experiment notes stored with results — @jamesgao-jpg
• Multiple corpus shards for custom Parquet datasets — @abner-ma
• Thread-safe concurrent insertion across database clients — @serhiizghama
• Milvus force-merge and index-readiness handling — @XuanYang-cn
• Compatibility with older filtered-search result files — @daviddallakyan2005
• Credential masking in published result artifacts — @hiimivantang
• Hologres loading and query efficiency — @zpatronus

Full changelog

Compare v1.0.22 with the current main branch