You Probably Don't Need a Vector Database for RAG: Simpler Alternatives That Work (2026)

Every new RAG project I see starts the same way: spin up a Pinecone index, configure a Weaviate cluster, or deploy a Qdrant instance. It’s become the default move — like reaching for React before considering vanilla HTML. But after building and maintaining several production RAG systems over the last two years, I’ve found that vector databases are often the wrong first choice.

The benchmark data backs this up. On the SQuAD dataset, BM25 keyword search achieves 88% recall@10 against 91.7% for OpenAI embeddings — a 3.7% gap that disappears in practice once you add reranking. Meanwhile, that vector database is eating 40-50% of your monthly RAG bill. If you’re running 50 queries per day in production, that’s roughly $1,000-$1,200/month just for the vector infrastructure.

This isn’t an argument against vector search. It’s an argument for being honest about what you actually need.

Why Vector Databases Became the Default for RAG

The pitch is seductive: embeddings capture semantic meaning, so “find documents about X” works even when the query uses different words than the document. That’s genuinely powerful. But somewhere along the way, the industry decided that semantic search was the only valid retrieval strategy, and that you needed a purpose-built database to do it.

The reality is that most RAG applications fall into one of two categories:

1. Domain-specific knowledge bases — internal docs, product manuals, legal contracts, codebases. The terminology is precise and consistent. A user searching for “connection timeout error” probably uses the same words the documentation does.
2. General-purpose Q&A — customer support, FAQ bots, content summarization. Here, semantic matching matters more, but the retrieval quality ceiling is still determined by your chunking strategy and reranking, not by whether you use cosine similarity over BM25.

For category 1 — which covers a huge percentage of real-world RAG deployments — keyword search matches or exceeds vector search in practice, with none of the infrastructure overhead.

The Benchmark Truth — BM25 Is Almost as Good as Embeddings

The RagIRBench benchmark from xetdata ran a controlled comparison on SQuAD. The numbers are worth looking at:

That 3.7% gap between BM25 and pure embeddings is real, but it’s also misleading. In production, you’re almost never doing a single-stage retrieval. You’re fetching candidates and reranking them. Once you add a reranking step — even a cheap one using the same embedding model — the hybrid approach actually outperforms pure vector search at 93.2% recall@10.

I’ve replicated this pattern across three different production systems. The hybrid pipeline (BM25 for first-pass retrieval, embedding reranking on the top 50 results) consistently beats pure vector search while using 10x less infrastructure. The vector database becomes a reranking cache, not the primary retrieval engine.

Alternative 1 — SQLite FTS5 (Zero-Infrastructure RAG)

SQLite’s FTS5 extension gives you production-grade full-text search with zero additional infrastructure. No server to deploy, no index to manage, no API keys to rotate. It’s a single file on disk.

The Roaming RAG project demonstrated this pattern in practice: they replaced a full vector database pipeline with SQLite FTS5 and eliminated their entire embedding pipeline cost. For domain-specific content where terminology is precise — think internal knowledge bases, code documentation, or compliance manuals — FTS5 handles retrieval with comparable accuracy to vector search.

Here’s how simple the setup is:

-- Create the FTS5 virtual table
CREATE VIRTUAL TABLE docs_fts USING fts5(
  title, content, content=docs, content_rowid=id
);

-- Insert your documents normally
INSERT INTO docs (id, title, content) VALUES (1, '...', '...');

-- Query with BM25 ranking
SELECT title, snippet(docs_fts, 1, '<b>', '</b>', '...', 32)
FROM docs_fts
WHERE docs_fts MATCH 'connection timeout'
ORDER BY rank;

That’s it. No Docker Compose, no cloud bill, no vector dimension math. For a team of 5-20 people using an internal RAG tool, this is often the right answer.

The tradeoff: FTS5 doesn’t do semantic matching. If your users search for “can’t log in” and your docs say “authentication failure”, you’ll miss it. But for many internal tools, that’s an acceptable limitation — and one you can fix with a synonym table or a lightweight embedding layer on top.

Alternative 2 — PostgreSQL Full-Text Search (Production-Grade)

If you’re already running PostgreSQL — and most teams are — you have a production-grade search engine sitting in your existing database. PostgreSQL’s tsvector and tsquery types, combined with GIN indexes, handle full-text search at scale.

Omni, an open-source workplace search tool, uses PostgreSQL full-text search as its primary retrieval engine. They process millions of documents across thousands of workspaces without a dedicated vector database. The approach works because workplace search queries are overwhelmingly keyword-driven: people search for file names, project codes, email subjects, and team names.

-- Add a tsvector column for full-text search
ALTER TABLE documents ADD COLUMN search_vector tsvector
  GENERATED ALWAYS AS (
    setweight(to_tsvector('english', title), 'A') ||
    setweight(to_tsvector('english', content), 'B')
  ) STORED;

-- Create a GIN index
CREATE INDEX documents_search_idx ON documents USING GIN(search_vector);

-- Query with ranking
SELECT title, ts_rank(search_vector, query) AS rank
FROM documents, plainto_tsquery('english', 'connection timeout') AS query
WHERE search_vector @@ query
ORDER BY rank DESC
LIMIT 10;

Postgres handles this at scale. I’ve seen it serve sub-50ms queries on tables with 10M+ rows. The GIN index is efficient, ts_rank gives you relevance scoring, and you can weight title matches higher than body matches using setweight.

The real advantage: you eliminate an entire service from your stack. No vector DB to monitor, no sync pipeline to maintain, no separate backup strategy. Your documents live in the same database as everything else.

Alternative 3 — Hybrid BM25 + Embedding Reranking (Best of Both Worlds)

This is the pattern I’ve settled on for most production systems. It’s the pragmatic middle ground: use cheap keyword search for the expensive first-pass retrieval, then apply embedding similarity only on the top candidates.

The pipeline looks like this:

1. First pass: BM25 (via SQLite FTS5, Postgres tsvector, or Elasticsearch) retrieves the top 50-100 candidates. Cost: essentially free.
2. Rerank: Compute embeddings for those 50 candidates and the query, then rerank by cosine similarity. Cost: 50 embedding API calls per query.
3. Assemble: Take the top 3-8 results and feed them to your LLM.

The production cost analysis from Hacker News showed that hybrid reranking (70% semantic + 30% keyword weighting) reduces retrieval from top-20 to top-8 while maintaining quality. Combined with token-aware context assembly — keeping context to ~3,200 tokens instead of 12,000 — you save 35% on LLM costs with identical accuracy.

import sqlite3
from sentence_transformers import SentenceTransformer
import numpy as np

# Lightweight reranker — runs locally, no API calls
model = SentenceTransformer('all-MiniLM-L6-v2')

def hybrid_retrieve(query: str, top_k: int = 8):
    # Step 1: BM25 first pass via FTS5
    conn = sqlite3.connect('docs.db')
    cursor = conn.execute(
        "SELECT id, title, content FROM docs_fts "
        "WHERE docs_fts MATCH ? ORDER BY rank LIMIT 50",
        (query,)
    )
    candidates = cursor.fetchall()
    
    # Step 2: Embedding rerank on candidates only
    query_emb = model.encode(query)
    candidate_embs = model.encode([c[2] for c in candidates])
    scores = np.dot(candidate_embs, query_emb)
    
    # Step 3: Return top-k
    top_indices = np.argsort(scores)[-top_k:][::-1]
    return [candidates[i] for i in top_indices]

This runs on a single $5/month VPS. No vector database required.

Alternative 4 — Reasoning-Based RAG with Tree Search

The most interesting alternative I’ve seen this year is reasoning-based RAG, exemplified by PageIndex. Instead of embedding documents into a vector space, PageIndex builds a hierarchical tree index where each node corresponds to a document section. Retrieval becomes a tree search — similar to how AlphaGo searches a game tree — rather than a nearest-neighbor lookup.

This approach achieves better retrieval accuracy than vector-based systems for professional and financial documents, where document structure matters more than semantic similarity. The key insight: nodes align with actual document sections (headings, paragraphs, clauses), preserving context without arbitrary chunking. A contract’s “Indemnification” section stays intact as a single node, rather than being split across three chunks that lose the section boundary.

For legal, financial, or compliance RAG — where document structure is part of the meaning — this approach outperforms vector similarity every time.

The Cost Argument — Vector DBs Are 40-50% of Your RAG Bill

Let’s talk real numbers. A production RAG system handling 50 queries per day costs roughly $2,400/month. Here’s the breakdown from actual deployments:

The vector database is the single largest line item. And for what? A 3.7% recall improvement that disappears once you add reranking.

Embedding caching alone — which achieves 45-60% intra-day hit rate — saves 20% on API costs. Token-aware context assembly (3.2K tokens vs 12K) saves another 35% on LLM costs. These optimizations are free. The vector database cost is not.

When You Actually Need a Vector Database

I don’t want to sound like vector databases are never the right call. They are, in specific scenarios:

• Multi-lingual search: If your corpus spans 5+ languages, keyword search breaks down. Embeddings handle cross-lingual semantic matching naturally.
• Image or audio retrieval: You can’t BM25 your way through image embeddings. Vector databases are the right tool for multimodal search.
• Scale beyond 100M documents: At massive scale, purpose-built vector databases with HNSW or IVF indexes outperform bolting search onto a general-purpose database.
• Real-time indexing with low latency: If you need sub-50ms p99 on 1B+ vectors, Pinecone or Weaviate are your only options.

But these are edge cases. Most RAG deployments handle 10,000 to 1,000,000 documents with a single language. For that range, the simpler alternatives work.

How to Choose the Right Retrieval Strategy

Here’s the decision tree I use:

1. Is your content domain-specific with consistent terminology? → Start with SQLite FTS5 or Postgres tsvector. Add embedding reranking only if recall is insufficient.
2. Do you already run Postgres? → Use tsvector. It’s already there, already backed up, already monitored.
3. Do you need semantic matching (synonyms, paraphrasing)? → Use hybrid BM25 + embedding reranking. Skip the vector database — just compute embeddings on the fly from your existing store.
4. Is document structure critical (legal, financial, compliance)? → Consider reasoning-based RAG with tree search (PageIndex approach).
5. Are you serving 100M+ documents in 10+ languages? → Now you need a vector database.

Migration Guide — Moving from Vector DB to Simpler Alternatives

If you’re already running a vector database and want to simplify, the migration is straightforward:

1. Export your documents with their original text and metadata. You don’t need the embeddings — you’ll recompute them on retrieval.
2. Set up FTS5 or tsvector on the text content. Index your documents.
3. Add a lightweight embedding model (all-MiniLM-L6-v2 is 80MB and runs on CPU) for reranking.
4. Route queries through the hybrid pipeline — BM25 first pass, embedding rerank on top 50.
5. Run both systems in parallel for a week. Compare recall on real user queries.
6. Decommission the vector database when you’re confident.

Every team I’ve walked through this migration has kept the simpler setup. The vector database never went back in.

Start Simple, Scale Smart

The RAG ecosystem has a tooling bias problem. Every tutorial, every starter template, every SaaS demo starts with a vector database because that’s what the vendors sell. But the data doesn’t support it as the default choice.

BM25 gets you 88% of the way there. Hybrid reranking gets you past 93%. SQLite FTS5 and PostgreSQL tsvector are production-ready, zero-infrastructure alternatives that eliminate your biggest cost center. And for domain-specific content, they often outperform vector search because they match the terminology your users actually use.

Next time you start a RAG project, try building it without a vector database first. Add one only when you can prove you need it. You’ll save money, reduce complexity, and — in most cases — end up with a system that works just as well.

Already running a vector database and thinking about simplifying? Check out our Vector Database Comparison 2026 for a detailed breakdown of when each option makes sense, and Bigger Context Windows Didn’t Make Our RAG Smarter for more on what actually drives retrieval quality in production.