Why filters are essential in production
A pure similarity search asks: what items are most similar to this query? But real applications almost always need to add constraints. A shopper searching for running shoes might want only items in stock, under $100, and in size 10. A financial analyst searching for similar investment reports needs only reports from the past two years. A multi-tenant application needs users to see only their own organization's documents, never another's.
These are called metadata filters: hard rules based on attributes (like price, date, category, or user ID) that results must satisfy in addition to being semantically similar. Without metadata filtering, similarity search alone cannot enforce these requirements.
Why naive filtering approaches fail
Two obvious but flawed approaches exist. The first is to search for the top 1,000 similar items and then discard those that fail the filter. This works when many items match the filter (if half the catalog is in your size, you get plenty of results after filtering). But when very few items match (if only 1% of items are in your size), you would need to retrieve 1,000 items just to get 10 results after filtering, which destroys performance.
The second approach is to build a separate index for each possible filter combination. This creates an unmanageable number of indexes as the number of filter attributes grows.
Filtered ANN: applying constraints during the search itself
The right approach is to apply the metadata filter during the search process itself, not before or after. As the algorithm navigates the index structure looking for similar items, it checks each candidate against the filter and skips items that do not qualify. The algorithm keeps searching until it has found enough qualifying results.
This sounds simple, but it requires careful engineering. At very low filter selectivity (when less than 1% of items qualify), the standard graph structure can lose connectivity, making it hard to find enough qualifying results without visiting most of the index. Endee's progressive filtering approach handles even the most selective filters while maintaining fast response times and high recall.