Vector databases transformed how we build AI applications. Before they existed, implementing semantic search meant wrestling with Elasticsearch’s dense vector fields or rolling custom FAISS indices. Now we have purpose-built systems that make similarity search as straightforward as SQL queries.

I first used vector databases building a document search system that needed to understand meaning, not just match keywords. A user searching for “how to fix a broken pipe” should find documents about “plumbing repairs” even if they never use those exact words. Traditional full-text search falls flat here, but embeddings capture semantic meaning.

The core insight: represent text (or images, or audio) as high-dimensional vectors where similar items are geometrically close. Then use specialized indexes to search billions of vectors in milliseconds. This unlocks RAG (Retrieval-Augmented Generation), semantic search, recommendation systems, and more.

How Vector Databases Work

At their core, vector databases solve one problem: find the k most similar vectors to a query vector—fast. “Similar” typically means cosine similarity or L2 distance in high-dimensional space (often 384 to 1536 dimensions).

The naive approach—calculating distance to every vector—doesn’t scale. For 10 million documents, that’s 10 million distance calculations per query. Vector databases use approximate nearest neighbor (ANN) algorithms that trade tiny amounts of accuracy for massive speedups.

Popular vector databases:

  • Pinecone - Fully managed, dead simple to use, great for startups
  • Weaviate - Open source, rich filtering, good hybrid search
  • Qdrant - Rust-based, excellent performance, Docker-friendly
  • Milvus - Battle-tested at scale, Zilliz backing
  • pgvector - Postgres extension, great for small-medium datasets

I typically use Pinecone for quick projects (amazing DX) and self-host Qdrant for production systems where I need full control.

Embeddings: Turning Content into Vectors

Embeddings are the bridge between human content and vector databases. An embedding model takes text (or images, audio, etc.) and produces a fixed-length vector—typically 384, 768, or 1536 dimensions.

Text Embeddings with OpenAI

OpenAI’s text-embedding-3-small is my go-to for production:

from openai import OpenAI

client = OpenAI(api_key="your-api-key")

# Generate embedding
response = client.embeddings.create(
    model="text-embedding-3-small",  # 1536 dimensions
    input="The quick brown fox jumps over the lazy dog"
)

embedding = response.data[0].embedding
# Returns: array of 1536 floats, e.g. [0.0234, -0.1567, 0.0892, ...]

print(f"Dimension: {len(embedding)}")  # 1536

Model choices:

  • text-embedding-3-small: 1536 dims, $0.02/1M tokens, fast and accurate
  • text-embedding-3-large: 3072 dims, $0.13/1M tokens, highest quality
  • text-embedding-ada-002: 1536 dims, legacy but still good

For self-hosted embeddings, sentence-transformers offers excellent models:

from sentence_transformers import SentenceTransformer

# all-MiniLM-L6-v2: 384 dims, fast, good quality
model = SentenceTransformer('all-MiniLM-L6-v2')

embeddings = model.encode([
    "This is a sentence",
    "This is another sentence"
])

print(embeddings.shape)  # (2, 384)

Key insight: Documents with similar meanings produce vectors that are close in embedding space. “dog” and “puppy” have high cosine similarity, while “dog” and “car” are far apart.

The core operation: given a query vector, find the k most similar vectors. We measure similarity using cosine similarity or Euclidean distance (L2).

Cosine Similarity

Cosine similarity measures the angle between vectors, ranging from -1 (opposite) to 1 (identical):

import numpy as np

def cosine_similarity(vec1, vec2):
    """
    Compute cosine similarity between two vectors.
    Returns value between -1 and 1, where 1 means identical direction.
    """
    dot_product = np.dot(vec1, vec2)
    norm1 = np.linalg.norm(vec1)
    norm2 = np.linalg.norm(vec2)
    return dot_product / (norm1 * norm2)

# Example: search documents
query_embedding = get_embedding("What is machine learning?")
document_embeddings = [
    get_embedding("Machine learning is a subset of AI"),
    get_embedding("Pizza is a type of food"),
    get_embedding("Neural networks learn from data")
]

similarities = [
    cosine_similarity(query_embedding, doc_emb)
    for doc_emb in document_embeddings
]

# Results: [0.89, 0.12, 0.84] - first and third documents are similar

In practice, vector databases handle this for you at massive scale. Here’s the same operation with Pinecone:

import pinecone
from openai import OpenAI

# Initialize
pc = pinecone.Pinecone(api_key="your-key")
index = pc.Index("your-index")

# Query
query = "What is machine learning?"
query_embedding = OpenAI().embeddings.create(
    model="text-embedding-3-small",
    input=query
).data[0].embedding

# Search returns top-k similar vectors
results = index.query(
    vector=query_embedding,
    top_k=5,
    include_metadata=True
)

for match in results.matches:
    print(f"Score: {match.score:.3f} - {match.metadata['text']}")

Distance metrics:

  • Cosine: Best for text embeddings (direction matters, not magnitude)
  • L2 (Euclidean): When magnitude matters (image embeddings)
  • Dot product: Fast when vectors are normalized

Most text applications use cosine similarity. Learn more in the Pinecone learning center.

Indexing Algorithms

Searching through millions of vectors naively (brute force) is too slow. Approximate Nearest Neighbor (ANN) algorithms trade tiny accuracy loss for massive speed gains.

HNSW (Hierarchical Navigable Small World)

HNSW is the gold standard for vector search—fast, accurate, and memory-efficient. It builds a multi-layer graph where each layer has progressively fewer nodes. Search starts at the top (sparse) layer and navigates down to the bottom (dense) layer.

Performance characteristics:

  • Query time: O(log n) with high probability
  • Build time: O(n log n)
  • Memory: Higher than alternatives (~50-100 bytes per vector overhead)
  • Recall: 95%+ with proper tuning

HNSW powers Qdrant, Weaviate, and many production systems. It’s particularly good for high-dimensional spaces (768-1536 dimensions).

Tuning parameters:

  • M: Connections per node (16-64, higher = better recall but more memory)
  • ef_construction: Search depth during build (100-200, higher = better index quality)
  • ef_search: Search depth during query (50-500, higher = better recall but slower)

IVF (Inverted File Index)

IVF partitions the vector space into clusters (Voronoi cells) using k-means. Search only examines vectors in the nearest clusters.

Performance characteristics:

  • Query time: O(k + d) where k is clusters searched, d is vectors per cluster
  • Build time: O(n * k * iterations)
  • Memory: Lower than HNSW
  • Recall: Lower than HNSW, but tunable

IVF works well for billion-scale datasets where memory is constrained. Used in FAISS and Milvus.

Product Quantization (PQ)

PQ compresses vectors to reduce memory, often combined with IVF (IVF-PQ). Instead of storing 1536 floats (6KB), store compressed version (~64-256 bytes).

Trade-off: 10-20x memory reduction, slight recall drop, great for massive scales.

Facebook’s FAISS library implements all these algorithms with excellent performance. See their guidelines for choosing an index.

Best Practices for Production

After building multiple RAG systems and semantic search applications:

  1. Choose embeddings wisely - OpenAI’s text-embedding-3-small ($0.02/1M tokens) offers the best price/performance. For self-hosted, all-MiniLM-L6-v2 is excellent.

  2. Normalize vectors - Most databases expect unit vectors. Normalize after embedding: 
    import numpy as np
embedding = embedding / np.linalg.norm(embedding)

  3. Index appropriately - HNSW for <10M vectors, IVF-PQ for billions. Start simple, optimize when needed.

  4. Monitor performance - Track query latency (aim for <50ms p95), recall rates (>90%), and index build times.

  5. Handle updates carefully - Vector databases vary in update performance. Pinecone handles realtime well; FAISS requires periodic rebuilds.

  6. Chunk documents intelligently - Don’t embed entire documents. Chunk into 200-500 tokens with overlap. Use LangChain’s text splitters.

  7. Test recall - Build a test set of known similar pairs and verify your system finds them. Aim for 90%+ recall@10.

  8. Use hybrid search - Combine vector search with keyword search (BM25) for best results. Weaviate and Qdrant support this natively.

  9. Metadata filtering - Most queries need filters (“find similar documents from 2024”). Choose databases with efficient filtered search.

  10. Mind costs - At scale, embedding generation and storage dominate costs. Cache embeddings, deduplicate content, and consider compression.

Example RAG Pipeline

Here’s a production-ready RAG implementation with Pinecone and OpenAI:

from openai import OpenAI
import pinecone
from typing import List

client = OpenAI()
pc = pinecone.Pinecone(api_key="your-key")
index = pc.Index("knowledge-base")

def chunk_document(text: str, chunk_size: int = 500) -> List[str]:
    """Split document into overlapping chunks."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - 50):  # 50 word overlap
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

def embed_and_store(document_id: str, text: str):
    """Chunk, embed, and store document."""
    chunks = chunk_document(text)
    
    # Batch embed for efficiency
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=chunks
    )
    
    # Store in Pinecone
    vectors = [
        {
            "id": f"{document_id}_chunk_{i}",
            "values": emb.embedding,
            "metadata": {"text": chunk, "doc_id": document_id}
        }
        for i, (chunk, emb) in enumerate(zip(chunks, response.data))
    ]
    index.upsert(vectors)

def search(query: str, top_k: int = 5) -> List[str]:
    """Semantic search with context."""
    # Embed query
    query_emb = client.embeddings.create(
        model="text-embedding-3-small",
        input=query
    ).data[0].embedding
    
    # Search
    results = index.query(
        vector=query_emb,
        top_k=top_k,
        include_metadata=True
    )
    
    return [match.metadata['text'] for match in results.matches]

# Usage
embed_and_store("doc1", "Long document text...")
context = search("How do I configure the database?")

Read more about RAG patterns in the Pinecone docs.

Conclusion

Vector databases are the infrastructure layer that makes modern AI applications possible. They bridge the gap between unstructured human content and LLM systems that need relevant context.

The ecosystem has matured rapidly. Two years ago, building semantic search meant gluing together FAISS, maintaining your own infrastructure, and praying nothing broke. Today, managed services like Pinecone “just work,” and open-source options like Qdrant and Weaviate offer production-grade reliability.

For new projects, I reach for Pinecone (managed simplicity) or Qdrant (self-hosted control). Both handle the hard parts—HNSW indexing, horizontal scaling, realtime updates—so you can focus on your application.

The future is multimodal. Current vector databases handle text embeddings, but the same infrastructure will power image search, audio similarity, code search, and hybrid retrieval across modalities. We’re just scratching the surface.

Further Reading:

Vector databases from March 2024, covering embeddings, similarity search, and indexing.