RAG Systems (Retrieval-Augmented Generation)

Retrieval-Augmented Generation (RAG) is the technique that lets AI models access information beyond their training data. Instead of relying solely on what the model learned during training, RAG fetches relevant information from external sources and includes it in the prompt.

What is RAG?

Think of RAG as giving an AI model a library card. The model itself is like a knowledgeable person, but even knowledgeable people need to look things up. RAG lets the model search through your documentation, databases, or knowledge bases to find relevant information before generating a response.

A RAG system has three core steps:

1. Retrieve: Search for relevant information from your knowledge base
2. Augment: Add that information to the prompt as context
3. Generate: The model creates a response using both its training and the retrieved information

Why RAG Matters for Product Builders

Up-to-date information: LLMs are trained on data with a cutoff date. RAG lets you provide current information, like recent product updates or today's stock prices.

Proprietary knowledge: Your product documentation, customer data, and internal processes aren't in any LLM's training data. RAG gives the model access to this information.

Reduced hallucination: By grounding responses in retrieved documents, RAG dramatically reduces the model's tendency to make things up.

Cost effective: Fine-tuning a model on your data is expensive and complex. RAG is simpler and works with any model via the API.

Dynamic updates: When your documentation changes, RAG reflects those changes immediately. No retraining needed.

How RAG Works Under the Hood

Step 1: Indexing (One-Time Setup)

Before you can retrieve anything, you need to index your knowledge base:

1. Chunk your documents: Break long documents into smaller, digestible pieces (usually 200-500 words)
2. Generate embeddings: Convert each chunk into a vector (a list of numbers that represents its meaning)
3. Store in a vector database: Save these vectors where they can be quickly searched

Step 2: Retrieval (Every Query)

When a user asks a question:

1. Convert the question to a vector: Use the same embedding model you used for indexing
2. Search for similar vectors: Find the chunks whose vectors are closest to the question's vector
3. Retrieve the top matches: Usually the top 3-5 most relevant chunks

Step 3: Augmentation (Every Query)

Take the retrieved chunks and add them to your prompt:

Context from our documentation:
[Retrieved chunk 1]
[Retrieved chunk 2]
[Retrieved chunk 3]

User question: {user_question}

Answer the question based on the context provided above.

Step 4: Generation

The LLM generates a response using both its general knowledge and the specific context you provided.

Practical Implementation

Here's a simple RAG system in Python using LlamaIndex:

from llama_index import VectorStoreIndex, SimpleDirectoryReader

# 1. Load your documents
documents = SimpleDirectoryReader('docs').load_data()

# 2. Create an index (generates embeddings and stores them)
index = VectorStoreIndex.from_documents(documents)

# 3. Query the index
query_engine = index.as_query_engine()
response = query_engine.query("How do I reset my password?")

print(response)

That's it! LlamaIndex handles the chunking, embedding, storage, retrieval, and prompt augmentation.

Key Design Decisions

Chunk Size

Too small (< 100 words): You lose context and may need to retrieve many chunks

Too large (> 1000 words): You waste context window space and may include irrelevant information

Sweet spot: 200-500 words, with some overlap between chunks to maintain context

Number of Retrieved Results

More isn't always better. Each retrieved chunk uses tokens from your context window.

• 3-5 chunks: Good starting point for most use cases
• 10+ chunks: Useful for complex questions requiring broad context
• 1-2 chunks: When you need focused, specific answers

Embedding Models

Small models (OpenAI ada-002, Cohere embed-light): Fast and cheap, good for most use cases

Large models (OpenAI text-embedding-3-large): More accurate but slower and pricier

Domain-specific models: Consider fine-tuned embeddings for specialized domains (medical, legal, etc.)

Advanced RAG Techniques

Hybrid Search

Combine vector similarity with traditional keyword search. This catches exact matches (like product SKUs) that semantic search might miss.

Reranking

After retrieving 20 candidate chunks, use a reranker model to select the best 5. This two-stage approach improves accuracy.

Query Transformation

Before retrieving, rewrite the user's question to be clearer:

• Original: "it's not working"
• Transformed: "The user reports that the password reset feature is not functioning"

Metadata Filtering

Add metadata to your chunks (date, author, category) and filter before searching:

# Only search documentation updated in the last 6 months
query_engine.query(
  "How do I export data?",
  filters={"date": {"$gte": "2024-07-01"}}
)

Parent-Child Chunking

Retrieve small chunks for accuracy, but provide the surrounding paragraph for context:

1. Search using 100-word chunks
2. Return the full 500-word section containing that chunk

Common Pitfalls

Forgetting to update the index: Your RAG system is only as current as your last indexing run. Set up automated reindexing when documents change.

Not evaluating retrieval quality: Just because your RAG system returns results doesn't mean they're relevant. Manually review 50-100 queries to ensure good retrieval.

Overloading the context: Retrieving 20 chunks might seem safer, but it confuses the model and wastes tokens. Start small.

Ignoring chunk boundaries: Don't split chunks mid-sentence or mid-thought. Respect logical boundaries like paragraphs or sections.

When to Use RAG

RAG is perfect for:

• Customer support chatbots answering from documentation
• Internal tools searching company knowledge bases
• Research assistants citing specific sources
• Code assistants searching API documentation
• Legal or medical applications requiring source attribution

RAG isn't ideal for:

• Tasks requiring 100% of a large corpus (use fine-tuning instead)
• Real-time data that changes faster than you can index
• Highly creative tasks where grounding in sources limits output

Measuring RAG Performance

Track these metrics:

Retrieval metrics:

• Precision: What % of retrieved chunks are relevant?
• Recall: What % of relevant chunks did you retrieve?
• MRR (Mean Reciprocal Rank): How quickly do you find the right answer?

Generation metrics:

• Faithfulness: Does the answer stick to the retrieved context?
• Relevance: Does the answer actually address the question?
• User satisfaction: Are users clicking "helpful"?

Getting Started

1. Start with a small dataset: 10-20 documents you know well
2. Use a framework: LlamaIndex or LangChain handle complexity for you
3. Test retrieval separately: Before plugging into an LLM, verify you're retrieving the right chunks
4. Iterate on chunking: Try different sizes and strategies based on your content
5. Add metadata: Tag chunks with useful filters from day one

RAG is one of the most practical techniques for building production AI features. It's simpler than fine-tuning, more accurate than raw prompting, and essential for any AI product that needs to reference specific information.