IVF vs HNSW vs Disk-Based Indexes: How to Choose the Right Vector Index

A practical comparison of the three index types that power vector search at scale

Introduction

When you evaluate a vector database, the most consequential technical decision hiding underneath the surface is the index type. The index determines how your vectors are organized for search, and that single choice cascades into everything that matters in production: query speed, result accuracy, memory cost, build time, and how gracefully the system handles growth.

In production, index choice is often the difference between a system that scales economically and one that becomes prohibitively expensive at 10x growth. Getting this right early saves significant pain later.

The three index families you will encounter in nearly every vector database are HNSW, IVF, and disk-based indexes (most commonly DiskANN). This article explains how each one works, where each one excels, and how to choose based on your actual workload.

How Each Index Works

HNSW (Hierarchical Navigable Small World)

HNSW builds a multi-layered graph where every vector is a node connected to its nearest neighbors. The top layers are sparse with long-range connections; the bottom layer has every vector with short-range connections. Search starts at the top and navigates down, narrowing the candidate set at each layer. A typical search over millions of vectors visits only a tiny fraction of nodes (often hundreds to a few thousand), which is why it is so fast.

IVF (Inverted File Index)

IVF partitions the vector space into clusters using k-means. At query time, the system compares the query to cluster centroids, identifies the closest clusters, and only scans vectors within those clusters. The number of clusters scanned (nprobe) controls the speed vs accuracy tradeoff.

Disk-Based Indexes (DiskANN)

DiskANN (from Microsoft Research) builds a graph similar to HNSW but stores it on SSD rather than RAM. It keeps compressed vector representations in memory for initial filtering and reads full-precision data from disk only when needed.

The Comparison That Matters

Numbers are representative ranges for typical configurations (HNSW with M=16, IVF with moderate nlist). Actual performance depends on data distribution, hardware, and implementation quality.

In practice, most teams start with HNSW, move to quantized variants as memory grows, and only adopt IVF or DiskANN when scale forces the tradeoff.

When to Choose HNSW

HNSW is the default choice for most production workloads. It consistently delivers the best recall-to-latency ratio of any index type.

You need the best possible search quality. At the same query latency, HNSW returns more correct results than IVF or DiskANN. For RAG, semantic search, or any use case where missing a relevant result has a real cost, HNSW's recall advantage matters.

Your dataset fits in memory (possibly with quantization). Ten million 768-dimensional vectors with HNSW require roughly 43 to 57 GB at full precision. Scalar quantization brings this to around 13 to 16 GB. HNSW combined with product quantization (HNSW+PQ) goes further, compressing vectors while keeping graph navigation intact. For most workloads under 50 million vectors, some form of quantized HNSW is feasible and hard to beat.

You have a steady stream of new data. HNSW handles incremental inserts well (O(log n) per insert) without requiring a full index rebuild.

What about scaling beyond RAM? Many modern databases implement memory-mapped I/O (mmap), which swaps portions of the HNSW graph to disk when RAM is full. This prevents crashes but degrades performance as more of the graph spills to disk. It is a safety net, not a replacement for having enough RAM. There are more deliberate approaches to this problem (covered at the end of this article).

The tradeoff you accept: High memory cost. This is the single biggest cost driver for HNSW-based systems.

HNSW and Metadata Filtering: A Known Weakness

HNSW struggles with highly selective metadata filters. If your query says "find similar vectors, but only among the 1% tagged as category X," the graph navigation breaks down. When 99% of nodes are filtered out, edges frequently lead to excluded nodes, creating dead ends. Recall drops significantly.

If your workload involves very selective filters, benchmark this scenario carefully. Some databases mitigate it with per-partition HNSW indexes, but that multiplies memory usage.

When to Choose IVF

IVF is the right choice when you need a functional index quickly, your dataset is large, and you can trade some recall for lower memory and faster builds.

Memory budget is tight. IVF does not store a graph, so overhead beyond raw vectors is small. For the same dataset, IVF uses 30 to 50% less memory than HNSW.

Your data distribution is relatively stable. You can add new vectors to existing clusters without rebuilding. However, as data distribution shifts, original centroids become stale. This is called centroid drift, and it gradually degrades recall. The fix is periodic re-training of centroids.

You need fast index builds, especially with GPU access. IVF's k-means training parallelizes extremely well on GPUs. With FAISS-GPU on an A100 or H100, you can train centroids for 100 million vectors in minutes rather than hours. HNSW has limited GPU acceleration because graph construction is inherently sequential.

A note on build time and nlist: Build time depends heavily on the number of clusters (nlist). A small nlist (1,000 to 4,000) trains quickly but gives coarser partitions. A large nlist (65,000+) can take hours on CPU for 100 million vectors.

IVF-PQ for extreme scale. Combining IVF with product quantization compresses vectors dramatically. Recall takes a hit (typically 85 to 95%), but searching a billion vectors on a single 32 GB machine is sometimes the only viable option.

IVF handles selective metadata filtering more naturally. Pre-filtering by metadata before scanning clusters avoids the graph connectivity problem that plagues HNSW under highly selective filters.

The tradeoff you accept: Lower recall than HNSW and susceptibility to centroid drift as data evolves.

When to Choose Disk-Based Indexes (DiskANN)

DiskANN exists for a specific scenario: your dataset is too large for RAM, but you still need graph-quality search, not the reduced recall of IVF-PQ.

Your dataset does not fit in memory, even with quantization. DiskANN stores the graph on SSD and keeps only compressed representations in RAM, cutting the memory requirement by 5x to 10x compared to HNSW.

You need better recall than IVF-PQ at large scale. At billion-scale, DiskANN typically delivers 90 to 97% recall, meaningfully higher than IVF-PQ at similar latency.

You can tolerate slightly higher latency. Expect 5 to 30 ms per query depending on SSD read patterns. Fine for RAG pipelines and backend search. Not suitable for live autocomplete with a 5 ms budget.

You want large-scale search on a single machine. A single server with 32 GB RAM and a fast NVMe SSD can serve DiskANN over a billion vectors.

The tradeoff you accept: Higher query latency than HNSW and dependency on SSD performance.

The Decision Framework

Start with your dataset size and memory budget. If your vectors fit in RAM (with quantization if needed), use HNSW. For most teams with datasets under 50 million vectors, this is the right path.

If memory is the binding constraint, compare IVF and DiskANN. IVF is simpler and builds faster (especially with GPU). DiskANN delivers better recall. If 85 to 95% recall works, IVF-PQ is cheaper. If you need 95%+, DiskANN is worth the complexity.

Factor in your write pattern. Static data favors IVF (centroid drift does not apply). Continuous ingestion favors HNSW.

Factor in your filter pattern. Highly selective metadata filters (less than 5% of vectors) can degrade HNSW recall. IVF or per-partition indexing may handle this better.

Test on your actual data. Benchmarks on synthetic datasets rarely match production data. Always evaluate on representative samples.

Composite Approaches

Production databases increasingly combine techniques. HNSW with scalar quantization gives HNSW's search quality with roughly 4x less vector storage. HNSW+PQ compresses further (16x to 32x) while retaining graph navigation. IVF-PQ with re-ranking uses compressed search for initial retrieval, then re-scores top candidates against full-precision vectors on disk. Tiered architectures keep hot data in memory and cold data on disk.

Splitting HNSW Across Storage Tiers

The biggest limitation of HNSW is memory. Standard implementations keep the entire graph and all vector data in RAM.

A more efficient approach is to split the graph across storage tiers based on access patterns. The upper layers, which guide long-range navigation and are accessed on every query, remain in memory. The bottom layer, which contains every vector in the dataset and accounts for the vast majority of the memory footprint, can be stored on disk and accessed only during the final stage of search.

This is fundamentally different from generic mmap, where the operating system blindly swaps arbitrary graph portions to disk. A layer-aware split exploits the structure of HNSW itself: the upper layers are tiny and latency-critical, so they belong in RAM. The bottom layer is massive but only touched during the beam search phase, so it can be served from fast disk with minimal impact on query performance.

Endee implements this architecture, reducing memory requirements by up to 10x while maintaining the recall and latency advantages of HNSW. This allows larger datasets to be served on smaller machines, without falling back to lower-recall index types like IVF-PQ.

Details and reproducible benchmarks (including datasets, configurations, and hardware) are available at endee.io.

Conclusion

There is no universally best vector index. HNSW gives you the best recall and lowest latency but costs the most memory. IVF gives you faster builds, lower memory, and better GPU utilization, but sacrifices recall. DiskANN lets you search beyond RAM limits but adds latency.

Start with HNSW. Move to quantized HNSW when memory gets tight. Consider IVF or DiskANN only when your workload genuinely demands it. And if the memory cost of HNSW is your primary concern, a layer-aware storage split may offer a way to keep HNSW's advantages without its biggest cost.