library

Modern RAG (Retrieval Augmented Generation) Systems

ai · rag · vector-search|2025-12-15 · 8 min read

Executive Summary

Retrieval Augmented Generation (RAG) has evolved from a 2020 research concept into a foundational component of enterprise AI architecture. RAG addresses a critical limitation of Large Language Models: their reliance on static training data that can become outdated and their tendency to hallucinate when asked about unfamiliar topics.

The core principle is elegantly simple—instead of relying solely on what an LLM "knows" from training, RAG systems retrieve relevant information from external knowledge bases at query time and feed this context to the model. This grounds responses in verifiable, current information while dramatically reducing hallucinations.

Modern RAG systems comprise several interconnected components: document preprocessing and chunking, embedding models that convert text to semantic vectors, vector databases for efficient similarity search, retrievers that find relevant content, and the LLM that synthesizes retrieved context into coherent responses. Each component offers significant tuning opportunities that can dramatically impact system performance.

Key Findings

The RAG Architecture Pipeline

A RAG system operates through a well-defined pipeline:

  1. Ingestion Phase: Documents are loaded, cleaned, and split into manageable chunks
  2. Embedding Phase: Each chunk is converted to a dense vector representation using an embedding model
  3. Indexing Phase: Vectors are stored in a vector database with metadata
  4. Query Phase: User questions are embedded and similar chunks are retrieved
  5. Augmentation Phase: Retrieved context is combined with the query into a prompt
  6. Generation Phase: The LLM produces a grounded response using the augmented prompt

This architecture enables LLMs to access updated knowledge, provides transparency through source citations, reduces hallucinations by grounding responses in retrieved content, and eliminates the need for expensive model retraining when knowledge changes.

Vector Databases: The Backbone of Retrieval

Vector databases have become essential infrastructure for RAG, with the market projected to grow from $1.73 billion in 2024 to $10.6 billion by 2032. These specialized databases store embeddings and enable fast similarity search across millions of vectors.

Leading Options:

| Database | Best For | Key Strengths | |----------|----------|---------------| | Pinecone | Enterprise production with minimal ops | Fully managed, guaranteed SLAs, scales to billions of vectors | | Qdrant | High-performance open-source deployments | Rust-based speed, excellent metadata filtering, cost-effective self-hosting | | Weaviate | Semantic search with data relationships | GraphQL interface, hybrid search (dense + BM25), modular architecture | | Chroma | Prototyping and smaller applications | Zero-config setup, developer-friendly, lightweight | | Milvus | Large-scale open-source deployments | 35K+ GitHub stars, proven at scale, distributed architecture |

Key Technical Consideration—Filtering Strategy: Pre-filtering applies filters before vector search (faster but can reduce recall), while post-filtering searches first then removes non-matching results (maintains recall but scans more vectors). Understanding this trade-off is crucial for production systems.

Pricing Reality: Pinecone offers simplicity at premium cost ($50-500+/month minimum). Self-hosted solutions like Qdrant or Weaviate require upfront infrastructure investment but offer better long-term economics for high-volume use cases.

Embedding Models: Converting Text to Meaning

Embedding models convert text into numerical vectors that capture semantic meaning, enabling similarity-based retrieval. The choice of embedding model significantly impacts retrieval quality.

Top Embedding Models (2024-2025):

OpenAI text-embedding-3-large

  • 3072-dimensional vectors (can be reduced to save storage)
  • Best overall for production use according to head-to-head benchmarks
  • Innovative variable dimensions feature—shortened 256-dimension embeddings still outperform older ada-002 full embeddings
  • Good multilingual support

Cohere Embed v4

  • 1536-dimensional multimodal vectors
  • Supports text and images
  • 128K token context window
  • Excellent for 100+ languages

BGE-M3 (Open Source)

  • 1024-dimensional vectors
  • Apache 2.0 license—completely free
  • Multi-functionality: dense, multi-vector, and sparse retrieval in one model
  • Strong performance on English and Chinese text
  • Can be fine-tuned on private data

Benchmark Performance (MTEB Retrieval Scores):

  • NV-Embed-v2: 62.7%
  • SFR-Embedding-Mistral: 59.0%
  • e5-mistral-7b-instruct: 56.9%
  • OpenAI text-embedding-3-large: 55.4%
  • Cohere English v3: 55.0%

Selection Guidance:

  • Production quality focus: Cohere embed-v4 or OpenAI text-embedding-3-large
  • Budget-conscious: BGE-large self-hosted
  • Multilingual: Cohere embed-v4 or BGE-M3
  • Privacy/control: Open-source models (BGE, E5, Mistral Embed) for self-hosting

Chunking Strategies: The Foundation of Quality Retrieval

Chunking—how documents are split into pieces for embedding—is perhaps the most underappreciated yet impactful component of RAG systems. Poor chunking leads to poor retrieval, regardless of how good your embedding model or vector database is.

Core Strategies:

Recursive Character Splitting The most popular starting point. Splits text hierarchically using an ordered list of separators (paragraph breaks, sentence breaks, etc.). Recommended starting configuration: 400-512 tokens with 10-20% overlap.

How it works: Starts with highest priority separator, moves to next if chunks exceed target size. Overlap ensures context continuity between chunks.

Semantic Chunking Splits based on meaning rather than fixed sizes. Analyzes sentence embeddings and splits when detecting significant semantic shifts. More expensive (requires embedding computation during chunking) but preserves coherent meaning units.

NVIDIA's 2024 benchmarks showed page-level semantic chunking achieved 0.648 accuracy—the highest of tested strategies.

Document-Structure-Aware Chunking Respects document structure: headers, sections, code blocks, tables. Essential for technical documentation, legal documents, and structured content.

Strategy Selection Guide:

| Content Type | Recommended Strategy | |-------------|---------------------| | General text | Recursive character splitting (400-512 tokens, 10-20% overlap) | | Technical documentation | Document-structure-aware | | Code | Language-specific splitting | | Academic papers | Semantic chunking | | Mixed/unstructured | AI-driven or context-enriched chunking |

The Fundamental Trade-off: Every chunking strategy trades context preservation against retrieval precision. Smaller chunks match queries more precisely but lose surrounding context. Larger chunks preserve relationships between ideas but dilute relevance in embeddings. Experimentation is essential.

Key Metrics:

  • Context Precision: How precisely do chunks contain relevant info without noise?
  • Context Recall: How fully do chunks capture all critical info for a query?

Advanced RAG Techniques

Basic RAG provides a foundation, but production systems often benefit from advanced techniques that improve retrieval quality and response accuracy.

Reranking A second-pass refinement after initial retrieval. The similarity search retrieves a large candidate set (e.g., top 100 chunks), then a cross-encoder or LLM re-scores candidates based on query relevance.

Options include Cohere Rerank, mxbai-rerank (open-source), and LLM-based reranking. Research shows LLM reranking significantly outperforms naive RAG baselines.

Query Expansion Enriches the user's query to increase relevant document retrieval. Techniques include:

  • Synonym expansion
  • Conceptual expansion
  • Generating multiple query variations and retrieving for each

Reduces dependence on users' ability to phrase questions optimally.

HyDE (Hypothetical Document Embedding) Instead of embedding the query directly, first ask an LLM to generate a hypothetical answer, then embed that answer for retrieval. Bridges the gap between abstract queries and concrete documents. Particularly effective for niche or underspecified domains.

Hybrid Search Combines dense vector search with sparse keyword search (BM25). Uses Reciprocal Rank Fusion (RRF) to merge results. Catches both semantic matches and exact token matches.

RAG-Fusion Combines results from multiple reformulated queries through reciprocal rank fusion, improving recall.

Implementation Best Practice: Start simple, then layer complexity:

  1. Stabilize basic retrieval with good embeddings and sensible chunking
  2. Add a reranker to strengthen top-k results
  3. Add hybrid search combining BM25 with vectors
  4. Measure precision@k, recall@k, and groundedness to track improvements

RAG Evaluation with RAGAS

RAGAS (Retrieval Augmented Generation Assessment) provides standardized metrics for evaluating RAG pipelines, enabling objective measurement without extensive human annotation.

Core Metrics:

Context Precision Measures the signal-to-noise ratio of retrieved context. Are relevant chunks ranked higher? Do retrieved contexts contain useful information for answering the question?

Context Recall Measures whether all relevant information required to answer the question was retrieved. Requires ground-truth annotations for comparison.

Faithfulness Measures factual accuracy of the generated answer. Are statements in the response actually supported by the retrieved context?

Answer Relevancy Measures how pertinent the generated answer is to the question. Penalizes incomplete or redundant responses.

The RAGAS Score A composite metric—the mean of Faithfulness, Answer Relevancy, Context Recall, and Context Precision—providing a single measure of RAG system quality.

Why Not BLEU/ROUGE? Traditional text generation metrics don't capture what matters for RAG: factual correctness, relevance, and grounding in retrieved documents. RAGAS metrics are specifically designed for these requirements.

Agentic RAG RAG is evolving beyond simple retrieve-and-generate to multi-step, agent-driven workflows. Azure AI Search now offers "agentic retrieval" with LLM-assisted query planning, multi-source access, and structured responses optimized for agent consumption.

Hybrid Architectures RAG combined with tools, structured databases, and function-calling agents. RAG provides unstructured grounding while structured data handles precise tasks.

Multimodal Embeddings Models that can embed text alongside images, code, and structured data. Cohere embed-v4 already supports text and image embedding.

Context Length Expansion Embedding models supporting longer inputs (128K+ tokens) enable more holistic document understanding without aggressive chunking.

Variable Dimension Embeddings OpenAI's innovation allowing dimension reduction while preserving core properties—trading some performance for lower storage costs.

Implications & Applications

For Developers Building RAG Systems:

  • Start with RecursiveCharacterTextSplitter at 400-512 tokens with 10-20% overlap
  • Choose embedding model based on your constraints: OpenAI for quality, BGE for budget/control
  • Prototype with Chroma, migrate to Qdrant or Pinecone for production
  • Implement RAGAS evaluation early to establish baselines
  • Layer complexity gradually: basic retrieval → reranking → hybrid search

For Organizations Adopting RAG:

  • RAG is now mature enough for production enterprise use
  • The build-vs-buy decision is real: managed services (Pinecone) vs self-hosted (Qdrant, Weaviate)
  • Plan for evaluation infrastructure—you can't improve what you can't measure
  • Consider data pipeline requirements: keeping knowledge bases current is ongoing work

For Understanding AI System Design:

  • RAG represents a shift from monolithic models to modular AI systems
  • The pattern—retrieval augmenting generation—will likely extend to more AI architectures
  • Grounding AI in verifiable sources addresses fundamental trust and accuracy concerns

Open Questions

  1. Optimal chunking remains empirical: Despite guidelines, the best chunking strategy varies by domain and requires experimentation. No universal solution exists.

  2. Evaluation at scale: RAGAS metrics require compute and sometimes ground-truth data. How do you continuously evaluate production RAG systems cost-effectively?

  3. Knowledge base maintenance: Most RAG documentation focuses on initial setup. Strategies for keeping knowledge bases current, handling updates, and managing staleness deserve more attention.

  4. Multimodal RAG maturity: Text-based RAG is well-established, but image/video/audio RAG is earlier in adoption. Best practices are still emerging.

  5. Cost optimization: Embedding and reranking costs can add up at scale. Understanding the cost-quality trade-off curve for your specific use case requires measurement.

Sources

  1. What is RAG? - AWS - Comprehensive overview of RAG fundamentals
  2. RAG Overview - Databricks - Technical explanation of RAG architecture
  3. RAG in Azure AI Search - Microsoft - Enterprise implementation perspective and agentic RAG
  4. Vector Database Comparison - LiquidMetal AI - 2025 comparison of Pinecone, Weaviate, Qdrant, Chroma, Milvus
  5. Top Vector Database for RAG - AI Multiple - Analysis of vector database selection criteria
  6. Best Vector Databases 2025 - Firecrawl - Pricing and performance comparison
  7. Best Embedding Models for RAG - ZenML - Embedding model benchmarks and recommendations
  8. Choosing Embedding Models - Pinecone - Practical guide to embedding selection
  9. Top Embedding Models 2025 - ArtSmart - OpenAI, Cohere, BGE comparison
  10. Chunking Strategies - Databricks - Comprehensive chunking guide
  11. Best Chunking Strategies 2025 - Firecrawl - Current best practices
  12. Breaking Up: Chunking in RAG - Stack Overflow - Practical chunking insights
  13. Advanced RAG Techniques - Neo4j - Reranking and query expansion
  14. Re-ranking & Query Transformation - APXML - Implementation guidance
  15. Advanced RAG Techniques - Unstructured - HyDE and advanced retrieval
  16. RAGAS Documentation - Official metrics reference
  17. RAG Evaluation Guide - Qdrant - Best practices for RAG evaluation
  18. Evaluating RAG with RAGAS - Towards Data Science - Practical RAGAS implementation