Building a Face Recognition System with ArcFace and Endee Vector Database

Face recognition is semantic search over faces. You encode a face into a vector, store those vectors in a searchable index, and at query time find the nearest stored vector.

The Core Idea

A face recognition system is three steps:

Encode a face into a fixed-size vector (the embedding)
Store those vectors in a searchable index
At query time, encode the new face and find the nearest stored vector

This is the same pattern as semantic search over documents - the only difference is the embedding model. Everything else (the vector store, cosine search, top-k ranking, metadata) transfers directly.

Traditional Approaches vs. Vector Database Search

Before vector databases became practical, face recognition systems used a different set of techniques. It is worth knowing them - they explain the design choices in modern systems and the tradeoffs involved.

Classical methods (pre-deep learning)

Eigenfaces (PCA) : projects faces into a lower-dimensional "face space" using principal component analysis. Fast and simple, but brittle: lighting, expression, and pose changes break it badly.
Fisherfaces (LDA) : improves on Eigenfaces by maximising inter-class separation, but still struggles with the same variations.
Local Binary Patterns (LBP) : describes local texture around each pixel; robust to illumination changes, very fast. Used widely in embedded systems. Accuracy caps out well below deep-learning methods.
Haar cascade classifiers : the classic OpenCV face detector. Works for frontal faces under good lighting; falls apart with occlusion, side profiles, or low resolution.

Deep learning without a vector database

Siamese networks : two identical CNN branches that take two images and output a similarity score directly. Good for one-shot verification, but you have to run the network for every stored face at query time. Does not scale to large databases.
Direct classifier : train a softmax classifier with one output node per person. Simple, but adding a new person requires retraining the model. Impractical for open-set scenarios.
FaceNet-style systems (with flat file storage) : embed with a deep network, then store embeddings in numpy arrays or SQLite. Works fine up to tens of thousands of faces; becomes a bottleneck beyond that (linear scan, no indexing, no metadata query).

Why a vector database changes this

Storing embeddings in Endee (or any dedicated vector database) gives you approximate nearest-neighbour search at scale, native cosine/L2 distance metrics, metadata stored alongside vectors (returned in search results without a second lookup), and upsert semantics so updating a registered face is a single write. The embedding model stays the same - you are just replacing the flat-file scan with an indexed search.

System Architecture

code

---------------------------------------------------------------
│                        INGESTION                            │
│                                                             │
│  Image → RetinaFace detection → ArcFace embedding (512-d)   │
│                                       │                     │
│                                       ▼                     │
│                               Endee collection              │
│                         (cosine, float32, 512-d)            │
---------------------------------------------------------------

---------------------------------------------------------------
│                         QUERY                               │
│                                                             │
│  Query image → same pipeline → 512-d vector                 │
│                                    │                        │
│                                    ▼                        │
│                    collection.search(top-k, cosine)         │
│                                    │                        │
│                                    ▼                        │
│              Ranked hits with similarity scores + metadata  │
---------------------------------------------------------------

Model Selection : Why RetinaFace and ArcFace

Face Detection: the options

Detector	Strengths	Weaknesses
Haar Cascade (OpenCV)	Fast, no GPU needed, built into OpenCV	Frontal-only, poor with occlusion/pose, many false positives
MTCNN	Detects faces + 5 landmarks, lightweight	Less accurate than RetinaFace on difficult images
RetinaFace	State-of-the-art accuracy, handles pose/scale/occlusion, returns landmarks for alignment	Heavier than MTCNN; slower on CPU
MediaPipe Face Detection	Very fast, runs on mobile/edge	Less robust on small or partially occluded faces
YOLO-based detectors	Excellent for multi-face scenes, real-time	General object detector repurposed for faces; overkill for single-face use

Why RetinaFace here: It returns a bounding box, 5 facial landmarks (eyes, nose, mouth corners), and a confidence score in a single pass. The landmarks are used for geometric alignment - rotating and cropping the face so the eyes are at fixed positions. ArcFace was trained on aligned faces, and skipping alignment measurably degrades similarity scores. No other lightweight detector gives you alignment landmarks as reliably.

Face Embedding: the options

Model	Dimensions	Backbone	Notes
Eigenfaces	~150	PCA	Classical baseline; poor accuracy
DeepFace (Facebook)	4096	9-layer CNN + 3D alignment	Early deep learning approach; uses explicit 3D face frontalization before embedding; high-dimensional, dated
OpenFace	128	SqueezeNet/Inception	Fast, open-source, lower accuracy than ArcFace
FaceNet (Google)	128	Inception-ResNet	Strong baseline; trained with triplet loss
VGGFace / VGGFace2	512	VGG-16 / ResNet-50	Good accuracy; large model size
ArcFace	512	ResNet-100	Top accuracy on standard benchmarks (LFW 99.83%), margin-based loss makes embeddings well-separated
AdaFace	512	ResNet-100	Newer; improves on ArcFace for low-quality images
ElasticFace	512	ResNet-100	Flexible margin loss; marginal gains over ArcFace

Why ArcFace here: ArcFace uses an additive angular margin loss (ArcLoss) during training that pushes same-identity embeddings together and different-identity embeddings apart in a more principled way than softmax or triplet loss. The result is embeddings that cluster tightly by identity and separate cleanly across identities - exactly what you need for cosine similarity search. It scores 99.83% on the LFW benchmark, is well-supported in DeepFace, and produces 512-d vectors that are compact enough to store efficiently at scale.

AdaFace is worth considering if your dataset has many low-resolution or motion-blurred images. For clean, well-lit face photos, ArcFace and AdaFace perform similarly.

The Stack

Layer	Tool	Why
Face detection & alignment	RetinaFace (via DeepFace)	Accurate, handles pose variation, returns alignment landmarks
Face embedding	ArcFace (512-d)	Best-in-class accuracy, tight identity clustering
Vector storage & search	Endee	Native dense vector support, cosine similarity, metadata co-stored
Normalisation	L2 norm	Makes cosine similarity interpretable (1.0 = same face)
Vector precision	int16	Quantised storage — smaller footprint than float32 with negligible accuracy loss

Step 1 : Installation

Endee needs numpy>=2.2.4 and the PyPI release of DeepFace breaks on numpy 2.x. The fix is to install DeepFace from source:

code

# Install Endee and tf-keras
!pip install tf-keras --quiet

# Install DeepFace dependencies manually
!pip install mtcnn retina-face fire gdown tqdm requests --quiet

# Install DeepFace from source (numpy 2.x compatible)
!pip install git+https://github.com/serengil/deepface.git --quiet

# !pip install endee==0.1.41b1
!pip install endee

# Keep numpy at 2.x — both Endee and Colab want it
!pip install "numpy>=2.2.4" --quiet

After install, restart your Colab runtime before running any imports.

Step 2 : Configuration

code

# Endee connection
TOKEN      = "your_token_here"       # NDD_AUTH_TOKEN from your server
DB_NAME    = "new"                   # Database name
COLLECTION = "faces"                 # Collection name

# Face model settings
MODEL_NAME  = "ArcFace"              # Embedding model (512-d)
DETECTOR    = "retinaface"           # Face detector
EMBED_DIM   = 512                    # ArcFace output dimension
TOP_K       = 3                      # Results to return per query
SIM_THRESH  = 0.5                    # Cosine similarity threshold for a match

Step 3 : Imports

code

import numpy as np
import cv2
import base64
import os
import warnings
import matplotlib.pyplot as plt
import matplotlib.patches as patches
from io import BytesIO
from PIL import Image as PILImage
warnings.filterwarnings('ignore')
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

from google.colab import files
from deepface import DeepFace
from endee import Endee

Step 4 : Connect to Endee and Create the Collection

code

# Connect with your token
client = Endee(token=TOKEN)

# Drop the collection if it already exists (clean slate)
try:
    client.delete_collection(COLLECTION)
except Exception:
    pass

# Create collection with a single 512-d cosine vector field
# precision="int16" quantises the stored vectors — a smaller footprint than
# float32 with negligible loss in similarity accuracy for ArcFace embeddings.
client.create_collection(
    name=COLLECTION,
    fields=[
        {
            "name": "embedding",
            "type": "vector",
            "params": {
                "dimension": 512,
                "space_type": "cosine",
                "precision": "int16",
            },
        }
    ],
)

collection = client.get_collection(COLLECTION)
print(f"Collection '{COLLECTION}' created")
print(f"Fields: {[f['name'] for f in collection.fields]}")

Output:

code

Collection 'faces' created
Fields: ['embedding']
Dimension: 512  |  Space: cosine  |  Precision: int16

Step 5 : Helper Functions

These helpers handle image encoding, face embedding, and visualisation.

code

def image_to_base64(image_path: str) -> str:
    """Encode image file to base64 string for storage in Endee meta."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


def base64_to_pil(b64_str: str) -> PILImage.Image:
    """Decode base64 string back to PIL image for display."""
    return PILImage.open(BytesIO(base64.b64decode(b64_str)))


def get_face_embedding(image_path: str) -> dict:
    """
    Detect face, align it, and return ArcFace embedding.
    Returns dict with:
        embedding : list[float]  - 512-d L2-normalised vector
        face_crop : np.ndarray   - aligned face crop (for display)
        bbox      : dict         - {x, y, w, h}
        confidence: float
    Raises ValueError if no face detected.
    """
    # Single detection pass - represent() returns facial_area and face_confidence
    # alongside the embedding. enforce_detection=True raises if no face is found
    # (no silent whole-image fallback).
    result = DeepFace.represent(
        img_path=image_path,
        model_name=MODEL_NAME,
        detector_backend=DETECTOR,
        enforce_detection=True,
        align=True,
    )
    if not result:
        raise ValueError(f"No face detected in {image_path}")

    # Pick the highest-confidence face; embedding, bbox, and crop all come
    # from this same entry : no mismatch possible.
    best = max(result, key=lambda r: r.get("face_confidence", 0))
    bbox = best["facial_area"]

    # Crop directly from the source image using the bbox - no second detection.
    img = cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)
    x, y, w, h = bbox["x"], bbox["y"], bbox["w"], bbox["h"]
    face_crop = img[y:y + h, x:x + w]

    raw = np.array(best["embedding"], dtype=np.float32)
    normalised = (raw / np.linalg.norm(raw)).tolist()  # L2-normalise

    return {
        "embedding":  normalised,
        "face_crop":  face_crop,
        "bbox":       bbox,
        "confidence": best.get("face_confidence", 0.0),
    }


def show_query_results(query_path: str, query_crop: np.ndarray,
                       hits: list, threshold: float):
    """Display query face alongside top-k retrieved results with similarity scores."""
    n = len(hits)
    fig, axes = plt.subplots(1, n + 1, figsize=(3.5 * (n + 1), 4))
    if not isinstance(axes, np.ndarray):
        axes = [axes]

    axes[0].imshow(query_crop)
    axes[0].set_title("Query face", fontsize=10, fontweight='bold')
    axes[0].axis('off')

    for i, hit in enumerate(hits):
        ax = axes[i + 1]
        sim = hit['similarity']
        meta = hit.get('meta', {})
        is_match = sim >= threshold

        if meta.get('image_b64'):
            stored_img = base64_to_pil(meta['image_b64'])
            ax.imshow(stored_img)

        color = '#1D9E75' if is_match else '#E24B4A'
        verdict = 'MATCH' if is_match else 'different'
        ax.set_title(
            f"Rank {i+1}: {meta.get('person_name', hit['id'])}\n"
            f"similarity={sim:.4f}  [{verdict}]",
            fontsize=9, color=color
        )
        for spine in ax.spines.values():
            spine.set_edgecolor(color)
            spine.set_linewidth(2.5)
        ax.axis('off')

    plt.suptitle(f"Top-{n} results  (threshold={threshold})", fontsize=11, y=1.02)
    plt.tight_layout()
    plt.show()

Embedding Pipeline

Every face goes through two stages before it touches the database.

Stage 1 : Detection and alignment

RetinaFace detects the face in the image, draws a bounding box, and aligns it (rotating/cropping so the eyes are level). This alignment step matters more than most people expect - ArcFace was trained on aligned faces, and a tilted input degrades similarity scores noticeably.

code

faces = DeepFace.extract_faces(
    img_path=image_path,
    detector_backend="retinaface",
    enforce_detection=False,
    align=True,
)
best = max(faces, key=lambda f: f.get("confidence", 0))

Stage 2 : ArcFace embedding

ArcFace produces a 512-dimensional vector representing the face's identity. The key detail: vectors are L2-normalised before storage. When every vector has unit length, cosine similarity reduces to a dot product, and the scores become interpretable - 1.0 is the same face, 0.0 is orthogonal, negative values mean dissimilar.

code

result = DeepFace.represent(
    img_path=image_path,
    model_name="ArcFace",
    detector_backend="retinaface",
    enforce_detection=False,
    align=True,
)
raw = np.array(result[0]["embedding"], dtype=np.float32)
normalised = (raw / np.linalg.norm(raw)).tolist()

Ingestion : Registering a Face

Run this flow once per person to register them in the database. Each run uploads an image, detects the face, embeds it, and upserts the vector alongside a full metadata record.

code

import uuid

# Upload image (Colab) : e.g. obama-2.jpeg
print("Upload a face image to register:")
uploaded = files.upload()
img_path = list(uploaded.keys())[0]

# Set identity
person_name = input("Enter person's name (e.g. 'obama'): ").strip()
object_id   = uuid.uuid4()

# Detect + embed
print(f"\nProcessing '{img_path}'...")
face_data = get_face_embedding(img_path)

print(f"  Face detected  : confidence: {face_data['confidence']:.3f}")
print(f"  Bbox: {face_data['bbox']}")
print(f"  Embedding dim: {len(face_data['embedding'])}")
print(f"  L2 norm: {np.linalg.norm(face_data['embedding']):.4f}  (should be 1.0)")

# Upsert into Endee
# meta  : arbitrary key-value stored alongside the vector (returned in search results)
# fields: the actual vector field : must match the collection schema
result = collection.upsert([
    {
        "id": object_id,
        "meta": {
            "person_name": person_name,
            "source_file": img_path,
            "model":       MODEL_NAME,
            "detector":    DETECTOR,
            "confidence":  round(face_data['confidence'], 4),
            "image_b64":   image_to_base64(img_path),   # full image stored for display
        },
        "fields": {
            "embedding": face_data["embedding"],        # 512-d vector, stored at int16 precision
        },
    }
])

print(f"\nUpsert result: {result}")
print(f"  Stored: id='{object_id}'  name='{person_name}'")

Example output after uploading cillian_murphy.jpeg:

code

Upload a face image to register:
Saving cillian_murphy.jpeg to cillian_murphy.jpeg
Enter person's name (e.g. 'obama'): cillian-murphy

Processing 'cillian_murphy.jpeg'...
  Face detected  — confidence: 1.000
  Bbox: {'x': 247, 'y': 73, 'w': 189, 'h': 251, 'left_eye': (408, 167), 'right_eye': (333, 171), 'nose': (399, 211), 'mouth_left': (411, 258), 'mouth_right': (350, 261)}
  Embedding dim: 512
  L2 norm: 1.0000  (should be 1.0)

Upsert result: {'upserted': 1}
  Stored: id='77249cb6-36d1-4973-ab65-4b40580d9aee'  name='cillian-murphy'

Each upsert carries the vector in fields and arbitrary metadata in meta. The metadata is returned alongside search results, so we store the person's name, the source filename, the detector/model used, and the original image as base64 - everything needed to display a result without a second lookup.

Run this cell once per person. For the query example later in this post, three people were registered this way:

Person	Image
`obama`
`obama-2`
`thomas`

Note obama and obama-2 are two different photos of the same person, registered as separate records - this is what produces two matching hits for a single query later.

Querying : Finding a Face

The query path is identical to ingestion: detect, align, embed, normalise. Then call collection.search() against the embedding field.

code

# Upload query image (Colab) : e.g. obama-3.jpeg
print("Upload a query face image:")
uploaded = files.upload()
query_path = list(uploaded.keys())[0]

# Detect + embed query face
print(f"\nEmbedding query face from '{query_path}'...")
query_data = get_face_embedding(query_path)

print(f"  Confidence : {query_data['confidence']:.3f}")
print(f"  Embedding  : {EMBED_DIM}-d  |  norm={np.linalg.norm(query_data['embedding']):.4f}")

# Similarity search on Endee
# search() returns the top-k nearest neighbours by cosine similarity
# Results include: id, similarity, meta
print(f"\nSearching Endee (top-{TOP_K}, cosine similarity)...")
response = collection.search(
    fields={"embedding": {"query": query_data["embedding"], "limit": TOP_K}}
)
hits = response["results"]["embedding"]

# Print results table
print("\n" + "═" * 55)
print(f"  TOP-{TOP_K} RESULTS  (threshold={SIM_THRESH})")
print("═" * 55)
for rank, h in enumerate(hits, 1):
    sim  = h['similarity']
    name = h.get('meta', {}).get('person_name', '-')
    tag  = "MATCH" if sim >= SIM_THRESH else "no match"
    print(f"  Rank {rank}: id={h['id']:15}  name={name:15}  "
          f"similarity={sim:.4f}  {tag}")

best_sim = hits[0]['similarity'] if hits else 0
if best_sim >= SIM_THRESH:
    best_name = hits[0].get('meta', {}).get('person_name', hits[0]['id'])
    print(f"\n  IDENTIFIED AS: {best_name}  (score={best_sim:.4f})")
else:
    print(f"\nUNKNOWN FACE  (best score={best_sim:.4f} < threshold={SIM_THRESH})")

# Visual results
show_query_results(query_path, query_data['face_crop'], hits, SIM_THRESH)

Each hit comes back with an id, a similarity score, and the full meta dict. A threshold (0.5 in this system) separates matches from non-matches:

Example output querying with obama.webp against a database containing obama, obama-2, and thomas:

code

Upload a query face image:
Saving obama.webp to obama.webp

Embedding query face from 'obama.webp'...
  Confidence : 1.000
  Embedding  : 512-d  |  norm=1.0000

Searching Endee (top-3, cosine similarity)...

═══════════════════════════════════════════════════════
  TOP-3 RESULTS  (threshold=0.5)
═══════════════════════════════════════════════════════
  Rank 1: id=4ac76ebc-f667-4429-b5c5-80a1f08c86cf  name=obama            similarity=0.6895  MATCH
  Rank 2: id=a2f45807-f526-491c-9c9d-51bbb1b31f7c  name=obama-2          similarity=0.5756  MATCH
  Rank 3: id=61532d97-18ff-4af3-8231-6709b6cfc340  name=thomas           similarity=0.2581  no match

  IDENTIFIED AS: obama  (score=0.6895)

The gap between rank 1 (0.69) and rank 3 (0.26) is the typical shape of a correct identification - the right person scores substantially higher than everyone else. The two obama entries at ranks 1 and 2 reflect that the same person was registered more than once (from different photos), both matching above the 0.5 threshold.

show_query_results() renders this visually - the query face on the left, followed by each hit's stored image, similarity score, and a green/red border marking match vs. non-match:

Top-3 face search results: query face matched against obama, obama-2, and thomas

Extending the System

A few directions worth exploring from this base:

Multi-face images : the current pipeline picks the highest-confidence face per image. For group photos you'd loop over all detected faces and upsert each one separately, tagging the metadata with position or index.

Incremental updates : Endee's upsert is idempotent on id, so re-registering a person with a better photo just overwrites the old vector. No need to delete first.

Threshold calibration : 0.5 is a reasonable default for cosine similarity with L2-normalised ArcFace vectors, but the right threshold depends on your use case. For high-security access control you'd push it higher (0.65+); for a soft "similar faces" search you'd lower it.

Swap the embedding model : replacing MODEL_NAME = "ArcFace" with "AdaFace" or "FaceNet" in the config is the only code change needed to benchmark a different model. Everything else stays the same.

Summary

Step	What happens
Install	numpy 2.x + DeepFace from source + Endee
Configure	Token, collection name, model, threshold
Ingest	Upload image → RetinaFace → ArcFace → L2 norm → upsert
Query	Same pipeline → cosine search → ranked hits
Display	Similarity scores + stored images from metadata

Built with DeepFace, ArcFace, RetinaFace, and Endee.