Large language models are brilliant writers with a poor memory for your business. Ask ChatGPT about your refund policy, your product specs or last quarter's pricing and it will either admit it doesn't know or, worse, confidently invent an answer. Retrieval-augmented generation (RAG) fixes that by giving the model the right information at the moment it answers.
RAG in one sentence
RAG is a pattern where your application first retrieves the most relevant passages from your own documents, then asks the language model to generate an answer using only those passages, usually with citations back to the source.
How a RAG system works, step by step
- Ingest: your documents (PDFs, help-centre articles, contracts, spreadsheets, database rows) are cleaned and split into small chunks.
- Embed: each chunk is turned into a vector, a list of numbers that captures its meaning, and stored in a vector database such as pgvector, Pinecone or Weaviate.
- Retrieve: when a user asks a question, it is embedded the same way and the closest chunks are fetched, often combined with keyword search for precision.
- Generate: the question and the retrieved chunks go to the LLM with an instruction like “answer only from these sources”.
- Cite and check: the answer is returned with links to its sources, and guardrails catch answers that aren't supported by the documents.
Where RAG pays off for businesses
- Customer support chatbots that answer from your help centre and policies, and hand off to a human when unsure.
- Internal knowledge assistants so staff can search SOPs, HR policies and technical docs in plain English.
- Sales enablement, pulling accurate specs, pricing and case studies into proposals.
- Legal and compliance search across contracts and regulations, with every answer traceable to a clause.
What makes a RAG system good (or bad)
Most disappointing AI chatbots fail at retrieval, not generation. If the wrong passages are fetched, even the best model will give a poor answer. The details that make the difference:
- Chunking strategy: splitting documents by meaning (headings, sections) rather than fixed character counts.
- Hybrid search: combining semantic vector search with classic keyword search so product codes and names aren't missed.
- Re-ranking: a second pass that re-orders results by true relevance before they reach the model.
- Evaluation: a test set of real questions with expected answers, run automatically on every change.
- Access control: users should only retrieve documents they're allowed to see.
How much does a RAG chatbot cost?
A focused RAG assistant over a few hundred documents typically takes 4–8 weeks to build properly, including evaluation and guardrails. Running costs depend on traffic, but for most small and mid-sized businesses the LLM and vector database bill is measured in tens to low hundreds of dollars a month. Use our cost estimator for a quick range, or see how our RAG development services work.
Is RAG always the right choice?
RAG is ideal when answers must come from specific, changing information. If you need the model to adopt a particular style or learn a narrow task format, fine-tuning may help instead. We compare the two in RAG vs fine-tuning.
