Generative AI & LLM Engineering · Chapter 6 · ~20 min · Complete Pipeline
RAG: The Complete Pipeline Step by Step
Base LLMs are frozen in time and blind to your private business documents. Retrieval-Augmented Generation (RAG) gives the model an open-book exam: fetch relevant paragraphs in real time and inject them into the prompt with source citations.
- Compare Closed-Book LLM vs Open-Book RAG
- Trace the 5-stage ingestion & retrieval pipeline
- Construct grounded production prompts with guardrails
- Supercharge accuracy with Hybrid Search & Re-ranking
1When RAG or Fine-Tuning Fits
When teams want an LLM to use internal documentation, one option is fine-tuning the model on those documents. That can make sense for some goals, but it is often a poor fit for facts that change frequently or need source citations. Retrieval-Augmented Generation (RAG) is one common alternative: retrieve relevant material at answer time and provide it to the model.
Fine-Tuning (Model Weights)
- Expensive and slow to retrain every time a policy changes
- Can still produce inaccurate numbers and dates; factual recall is not guaranteed
- Does not inherently provide citations to source documents
- Does not replace application-level access controls
RAG (Open-Book Retrieval)
- Updates can be made without retraining; indexing and availability take time
- Uses retrieval infrastructure, which has its own operating costs
- Can attach source references when metadata is preserved
- Requires separate tenant, role, and permission checks
Compare a closed-book model with RAG, which can provide relevant private context and source references when retrieval and permissions are configured correctly.
"What is Acme Corp's maximum meal allowance on domestic client dinners?"
According to Section 4.2 of the Acme Corp Travel & Expense Policy (Jan 2026), domestic business meals with external clients are reimbursable up to **$85 per person**, with receipts required for anything over $25.
In this example, RAG retrieves this paragraph and adds it to the prompt. The answer can still omit or misstate details.
When should you use Fine-Tuning instead of RAG?
2The 5-Stage RAG Pipeline Step by Step
Every production RAG architecture follows five foundational steps — divided into an offline ingestion loop and an online query loop:
Click each stage to inspect the transformation from raw document to grounded AI answer.
Document Ingestion & Parsing
Extract clean text and extract rich structural metadata (author, created date, document category).
Raw files: PDFs, Markdown files, Word docs, Notion exports, Confluence pages.
Cleaned raw text stripped of headers, footers, tables, and markup noise.
// 1. Ingest raw document
const doc = await parsePdf("handbook.pdf");
const cleanText = sanitizeMarkdown(doc.text);Why is chunk overlap (e.g. 40 tokens) recommended when splitting documents?
3Anatomy of the Augmented Prompt
How does the model actually receive the context? We assemble an Augmented Prompt that sandwiches retrieved chunks between strict system rules and the user's question:
How production RAG apps assemble context, rules, and queries before calling the LLM.
You are a verified corporate intelligence assistant. Answer the user's question using ONLY the provided context documents below. If the answer is not explicitly stated in the context, respond strictly with: "I do not have enough information in the provided documentation to answer this." Do not extrapolate or guess. Cite the provided [Source ID] for supported factual claims; say when context does not support an answer.
"All team members must rotate their 1Password vault master keys every 90 days. Hardware security keys (YubiKey) are required for all GitHub and AWS logins."
"Employees working abroad for more than 30 consecutive days must obtain prior written tax approval from the HR Compliance desk."
User: "How often do we have to rotate our 1Password master keys, and what hardware key is required?"
What is the purpose of the 'Strict Fallback Rule' in a RAG system prompt?
4Production Upgrade: Hybrid Search & Re-ranking
Standard vector search is great for concepts, but struggles with exact acronyms, error codes, and serial numbers. Modern enterprise RAG uses Hybrid Search (Dense Vectors + BM25 Keywords) combined with a Cross-Encoder Re-ranker (such as Cohere Rerank or BGE-reranker):
Why production teams combine Dense Vectors + BM25 Keywords + Cross-Encoder Re-rankers.
Troubleshooting ERR_SSL_PROTOCOL_ERROR on Nginx gateway after rotating Cloudflare edge certificates.
Guide to updating root certificates on Alpine Linux Docker containers inside VPC subnets.
Configuring Kubernetes Ingress controller with cert-manager for automatic TLS renewal on AWS EKS.
Best practices for web server security, HTTPS headers, and HSTS preload submissions.
Notice how Pure Vector Search put general security at #1 because of broad semantic overlap! But when debugging error codes (like ERR_SSL_PROTOCOL_ERROR), Hybrid + Re-ranking ensures exact technical error matches take the top slot.
Why does adding BM25 Keyword Search help pure vector embeddings?
5Common RAG Failure Modes & Fixes
The vector DB returns irrelevant chunks because chunks were too big or query was too vague.
Fix: Smaller chunks (350 tokens) + Hybrid Search.
You stuffed 15 chunks into the prompt, and the model ignored the fact buried on chunk #8.
Fix: Re-rank down to top-3 to top-5 chunks max.
The model still hallucinates because pre-trained memory overrode the supplied context.
Fix: Set Temperature=0 and enforce strict refusal instructions.
6Chapter challenge
RAG Architecture Mastery
Answer all four questions correctly to complete Chapter 6.
1What does RAG stand for?
2What is the primary advantage of RAG over Fine-Tuning for company knowledge bases?
3In the RAG query pipeline, what happens immediately after a user enters a question?
4What does a Cross-Encoder Re-ranker do in an advanced RAG pipeline?
7Recap
- RAG vs Fine-Tuning: RAG supplies retrieved context and may expose source references; fine-tuning can adapt behavior or style. Both need evaluation.
- 5-Stage Pipeline: Ingest → Chunk → Embed & Store → Retrieve → Generate.
- Prompt Guardrails: Instruct the model to rely on provided chunks and state when they are insufficient; evaluate for remaining errors.
- Hybrid Search & Re-ranking: Combine dense vectors for concepts with BM25 for exact keywords, then re-rank.
- Stay Lean: Keep top-K focused (3–5 chunks) to avoid the "Lost in the Middle" attention penalty.
Solve the challenge above to complete the chapter, or