What Is RAG? Connecting Company Documents to an AI Assistant

What Is RAG and Why Is It Needed?

LLMs know their training data but not your corporate manual. RAG — Retrieval-Augmented Generation — first retrieves relevant passages from your company documents, then asks the LLM to "answer in light of these passages." You build AI that stays current without fine-tuning.

Typical Use Cases

• Internal help desk: HR policies, IT procedures
• Customer support: product manuals, FAQs, contracts
• Sales enablement: spec sheets, competitor comparisons, pricing
• Legal and regulation: statutes, case law search

Architecture: 6 Components

1. Document ingestion: Pulling from PDF, DOCX, Markdown, SharePoint, Notion, Google Drive
2. Chunking: Splitting documents into 200-800 token semantically coherent pieces
3. Embedding: Each chunk → vector (OpenAI text-embedding-3, Cohere, BGE-M3)
4. Vector DB: Postgres + pgvector, Qdrant, Weaviate
5. Retrieval: Embed the question, find N nearest chunks
6. Generation: Prompt the LLM to answer using retrieved chunks

Seven Rules for Quality RAG

1. Hybrid search: Combine semantic + keyword (BM25)
2. Re-ranking: Take top 20, re-rank to 5 with Cohere Rerank
3. Chunk context: Add title + preceding paragraph summary to each chunk
4. Metadata filtering: Pre-filter by department, date, ACL
5. Cite sources: Show which document backed the answer — critical for trust
6. Permission to say "I don''t know": Instruct the LLM not to fabricate when evidence is missing
7. Evaluation pipeline: 100 Q&A golden set, regression tested on every change

Hybrid Search: BM25 + Semantic

Pure embedding search fails for certain queries. If a user asks about "SKU-4782-A", semantic embedding won''t match; BM25 (keyword) will. Hybrid approach:

1. BM25 → top 50
2. Embedding → top 50
3. RRF (Reciprocal Rank Fusion) to combine
4. Re-rank with Cohere or cross-encoder to top 5

Chunk Strategies

Practical Stack (SMB Scale)

• Embedding: OpenAI text-embedding-3-small (cheap, strong multilingual)
• Vector DB: PostgreSQL + pgvector (no separate service)
• LLM: Claude Haiku or GPT-4o-mini
• Framework: LangChain or LlamaIndex — or a 200-line custom implementation
• Frontend: Next.js with streaming responses

Security and Permissions

Biggest RAG mistake: every user sees all documents. Solution: attach acl metadata to each chunk; filter at retrieval by role. Managers see salary policies, others don''t.

Evaluation Set

Most skipped step. A good RAG system needs:

• 100 golden Q&A pairs (human-labeled)
• Automated metrics: retrieval precision@5, recall@10
• LLM-as-judge: answer quality 1-5
• Regression: same set runs on every model/prompt change
• Feedback loop: hard queries from production logs added weekly

Cost

For 10,000 pages of corporate documents, initial indexing ~$10. Monthly 5,000 queries: LLM + embeddings total ~$50-100. Staff time savings usually 20-50x that.

Frequently Asked Questions

pgvector vs dedicated vector DB?

Under 10M chunks: pgvector is sufficient with no additional ops burden. Over 10M or high QPS: Qdrant/Weaviate recommended.

How to do multilingual RAG?

Multilingual embedding (Cohere embed-multilingual-v3 or BGE-M3) + language-specific BM25 tokenizer.

Can RAG replace fine-tuning?

For knowledge-based use cases, yes. For style/tone adaptation, fine-tuning still helps. Often used together.

Next Step

Set up RAG for your organization — book a technical call.