How Vector Databases Actually Work Internally

Vector databases have become a core piece of AI infrastructure. Every RAG pipeline, every semantic search system, and every recommendation engine that runs on embeddings needs one. But most explanations stop at "it stores vectors and finds similar ones." That is like saying a car engine "burns fuel and makes wheels turn." Technically true, not very useful.

This article explains what actually happens inside a vector database, step by step: how vectors get stored, how indexes are built, how queries are executed, and what engineering decisions separate a system that works in production from one that falls apart under real load.

To make this concrete, we will follow a real scenario throughout: a SaaS company building an AI-powered support assistant that searches across 2 million help articles and support tickets to answer customer questions.

What a Vector Database Actually Does

A traditional database indexes rows by exact values. You look up a user by their ID or filter orders by date. A vector database indexes by similarity. You search by meaning, not by matching a field.

The database must ingest vectors (often millions or billions of them), build index structures that make search fast, execute similarity queries in milliseconds, filter results by metadata, handle inserts and deletes without degrading performance, and do all of this reliably under production traffic. Getting any one of these wrong creates real problems: slow responses frustrate users, bad recall surfaces irrelevant results, and poor write handling means your data goes stale.

Step 1: Ingestion and Storage

When you insert a record into a vector database, you typically provide three things:

The vector. A list of floating-point numbers, usually 384 to 1536 dimensions. In our support assistant example, each help article is passed through an embedding model (like OpenAI's text-embedding-3-small) and converted into a 768-dimensional vector. At 32-bit precision, that is 3,072 bytes per vector. Two million articles means roughly 6 GB of raw vector data.

An ID. A unique identifier that links the vector back to its source document so the system can return the actual article text, not just a vector.

Metadata. Structured attributes attached to the vector: the article's product category, language, last-updated timestamp, access tier (free vs premium), and anything else you will want to filter on later. Our support assistant might tag each article with product: "billing", language: "en", updated: "2025-11-03".

Under the hood, vectors and metadata are stored separately. Vectors are laid out in dense, contiguous memory blocks optimized for fast sequential reads during search. Metadata goes into a secondary index (similar to a traditional database index) that supports fast filtering. This separation matters because similarity search and metadata filtering have very different access patterns, and optimizing for both simultaneously requires keeping them apart.

Step 2: Building the Index

Raw vectors in storage are useless for fast search. Scanning every vector for every query (brute force) works at a few thousand records but collapses at scale. At 2 million vectors, brute force search takes 50 to 200 milliseconds per query. At 100 million, it takes seconds. For a customer-facing support assistant handling real-time conversations, that is unacceptable.

Indexes solve this by organizing vectors so that search can skip the vast majority and only compare against a small, relevant subset. The choice of index type is one of the most consequential infrastructure decisions you will make, because it directly determines your system's speed, accuracy, memory cost, and operational complexity.

HNSW (Hierarchical Navigable Small World)

The most widely used index in production today. HNSW builds a multi-layered graph where each vector is a node connected to its nearest neighbors. Searching starts at the top layer (sparse, long-range connections) and navigates down to the bottom layer (dense, short-range connections), narrowing the search at each level.

Why teams choose it: Consistently delivers 95 to 99%+ recall at 1 to 10 ms latency, even at tens of millions of vectors. The query speed vs accuracy tradeoff is tunable at runtime without rebuilding the index.

The cost: HNSW keeps the full graph in memory. For our 2 million vectors, that adds roughly 3 to 6 GB of graph overhead on top of the 6 GB of raw vectors. At 100 million vectors, the total memory footprint pushes past 45 GB. This is the primary driver of infrastructure cost for HNSW-based systems.

IVF (Inverted File Index)

IVF partitions vectors into clusters using k-means. At query time, the system identifies the most relevant clusters and only scans vectors within them.

Why teams choose it: Lower memory overhead than HNSW because there is no graph to store. Faster to build. Good for large, relatively static datasets where memory budget is tight.

The cost: Lower recall than HNSW at the same query speed, especially when data distributions are complex or clusters overlap. Less effective for workloads with frequent inserts.

Quantization: Scalar, Product, and Binary

Quantization is not a standalone index but a compression technique applied alongside other indexes to reduce memory usage.

Scalar quantization (SQ) converts each 32-bit float to an 8-bit integer, cutting memory by 4x with minimal accuracy loss. This is the simplest and most commonly used form.

Product quantization (PQ) breaks vectors into subvectors and replaces each with a codebook entry. A 3,072-byte vector can be compressed to 96 to 192 bytes, a 16x to 32x reduction. Accuracy loss is more noticeable, but it allows datasets that would not fit in memory at full precision.

Binary quantization converts each float to a single bit (positive or negative). Extreme compression (96x for 768d vectors), very fast to compare, but significant accuracy loss. Useful as a first-pass filter before re-ranking with full-precision vectors.

Why this matters practically: Quantization is how you control your infrastructure bill. A dataset that requires a 500 dollar per month instance at full precision might fit on a, $120/month instance with scalar quantization at 95%+ of the original accuracy.

Flat (Brute Force)

No index. Every query compares against every vector. Perfect recall, O(n) speed. The right choice when your dataset is small (under 50,000 vectors) and the simplicity outweighs the performance cost.

Step 3: Query Execution

When our support assistant receives the question "Why was I charged twice this month?", here is what happens inside the vector database, typically in 1 to 20 milliseconds:

Distance Computation

The database computes the distance between the query vector and candidate vectors. Three metrics cover the vast majority of use cases:

Cosine similarity measures the angle between two vectors, ignoring their magnitude. Two vectors pointing in the same direction score 1.0 regardless of their length. This is the default choice for text embeddings because embedding models typically produce normalized vectors where direction encodes meaning.

Euclidean distance measures the straight-line distance between two points. Common for image embeddings and other representations where magnitude carries information.

Dot product is mathematically equivalent to cosine similarity when vectors are normalized (which most text embedding models guarantee). It is slightly faster to compute, so some systems use it as an optimization.

How to choose: If you are working with text embeddings, use cosine similarity or dot product. If you are working with image or multimodal embeddings, test both cosine and Euclidean on your data. The wrong metric will systematically rank results incorrectly, so this is worth getting right early.

Metadata Filtering

In almost every production use case, you do not want all similar vectors. You want similar vectors that also meet specific conditions. Our support assistant should only return English-language articles about billing, not Spanish articles about onboarding.

This is metadata filtering, and how a database handles it has a direct impact on both speed and result quality.

Pre-filtering applies metadata conditions before the vector search. The index only operates on vectors that pass the filter. This is fast when the filter is selective (removes most vectors) but can hurt recall if the filtered set is too small for the index to work effectively.

Post-filtering runs the full vector search first, then removes results that do not match. This preserves recall but wastes computation on results that will be discarded. Worse, if many top results are filtered out, you might return fewer than the requested k results.

Hybrid filtering dynamically chooses the strategy based on how selective the filter is. If the filter eliminates 90% of vectors, pre-filter. If it eliminates 10%, post-filter. Most production-grade vector databases implement this, but the quality of the implementation varies significantly and is worth benchmarking on your actual filter patterns.

Step 4: Keeping the Index Current

A static index that never changes is easy. A production index that handles continuous writes while serving reads is where the real engineering lives.

Inserts. Adding a new vector to HNSW means navigating the graph to find the right position and creating edges to nearby nodes. This costs O(log n) per insert and works well for steady ingestion. But inserting a large batch of vectors from a very different distribution than the original data (say, a new product category your support articles never covered before) can degrade graph quality over time.

Deletes. Removing a node from an HNSW graph risks breaking connectivity. If the deleted node was a bridge between two graph regions, nearby vectors become harder to reach. Most databases handle this with soft deletes: the vector is marked as deleted and skipped during search, but its edges remain. Accumulated soft deletes gradually degrade performance until the index is rebuilt.

Updates. Changing a vector's values is effectively a delete followed by an insert. Metadata-only updates are cheaper since only the metadata index changes.

Background maintenance. Production databases run background processes that compact storage, rebuild degraded index segments, merge new vectors into the main index, and reclaim space from soft deletes. This is invisible to users but essential for sustained performance. How well a database handles this background work determines whether performance stays consistent over months or slowly degrades.

Multi-Tenancy: Serving Multiple Teams or Customers

If you are building a SaaS product, your vector database needs to serve multiple customers from the same infrastructure, each with their own data that must be isolated from others.

There are two common approaches. Namespace isolation stores all vectors in a single index but tags each with a tenant ID, using metadata filtering to enforce isolation at query time. This is simpler to operate and more efficient on resources, but requires trusting the filtering layer completely. Physical isolation gives each tenant a separate index or collection. This provides stronger guarantees but increases operational complexity and cost, especially as tenant count grows.

Most teams start with namespace isolation and move to physical isolation for customers with strict compliance or performance requirements.

Durability: What Happens When Things Fail

A vector database that loses data on crash is useless in production. Durability guarantees determine what you can count on.

Write-ahead logging (WAL) is the standard technique. Every write is recorded to a durable log before being applied to the index. If the system crashes mid-operation, it replays the log on startup to recover. This is the same approach used by PostgreSQL, MySQL, and other mature databases.

Replication copies data across multiple nodes so that a single machine failure does not cause data loss or downtime. The database continues serving from replicas while the failed node recovers.

Snapshots capture the full state of the index at a point in time for backup and disaster recovery.

The level of durability you need depends on your use case. A search index that can be rebuilt from source data in minutes needs less durability than one containing data that would take days to reprocess.

Library vs Database: Why FAISS Is Not Enough

FAISS (from Meta) and Annoy (from Spotify) are excellent vector search libraries. They handle indexing and similarity computation well. But they are not databases.

What they do not provide: persistence across restarts, metadata filtering alongside vector search, safe concurrent reads and writes, replication and durability guarantees, and operational tooling like monitoring and backups.

A library is an engine. A database is the complete vehicle: engine, transmission, brakes, fuel system, and dashboard. For a quick prototype, the engine is enough. For a production system serving real users, you need the vehicle.

Scaling: What Happens at 100 Million Vectors and Beyond

At small scale, most vector databases behave similarly. Differences emerge when data and traffic grow.

Memory. At 100 million 768-dimensional vectors with HNSW, you need 30+ GB for vectors and 15 to 30 GB for the graph. This exceeds single-machine RAM. Solutions include sharding across machines, quantization to compress vectors, and tiered storage that keeps frequently accessed vectors in memory and the rest on disk.

Throughput. A single HNSW query takes 1 to 10 ms, but serving 10,000 queries per second requires careful parallelization, connection pooling, and resource isolation so that one heavy query does not slow down others.

Index build time. Building HNSW over 100 million vectors takes hours. In serverless architectures like Endee, this is managed transparently so you do not provision machines or babysit indexing jobs.

Cost. At scale, the largest cost driver is memory. Every decision in this article (index type, quantization, tiered storage, filtering strategy) ultimately affects how much infrastructure you need and what that costs per month. Understanding the internals is not just a technical exercise. It directly shapes your infrastructure budget.

Conclusion

A vector database is not a simple key-value store with a similarity function bolted on. It is a system that coordinates storage layout, index construction, query execution, metadata filtering, write handling, background maintenance, multi-tenancy, durability, and scaling to deliver fast, accurate similarity search under real production conditions.

The index (HNSW, IVF, quantization) determines speed, accuracy, and memory tradeoffs. The storage engine determines durability and write performance. The filtering layer determines whether you can combine vector search with structured conditions. And the operational layer (replication, sharding, background maintenance) determines whether the system holds up over months of use at growing scale.

Understanding these internals helps you make better decisions about which database to choose, how to configure it, and what to expect as your data and traffic grow.