What Is RAG in Plain Words: How a RAG System Works and How to Make AI Answer from Your Documents

An AI model knows a lot, but nothing about your price list, policies, contracts or customers. RAG is a way to give it that knowledge without retraining: before answering, the system finds the relevant pieces of your documents and hands them to the model. The model answers from them, not from memory.

Below: RAG in plain words, how a RAG system is built, what embeddings and vector databases are, how RAG compares with fine-tuning and long context, where it breaks and how to launch it without overspending.

RAG in plain words

RAG stands for Retrieval Augmented Generation. Think of an open-book exam: the student does not remember everything but knows where to look. They find the right page first, then answer in their own words based on it.

Without RAG, a model answers only from what it learned in training. Your internal data is not there, so asked about your plan, it will either honestly say "I don't know" or, worse, invent a plausible answer. With RAG, the right page of your documentation is in front of it.

How RAG works

1. 01Documentspolicies, price lists, articles
2. 02Splittingchunks by meaning
3. 03Indexsearch by meaning, not keywords
4. 04Questionfind the right chunks
5. 05Answerthe model answers from them, with a source

Step by step:

1. Preparation. Documents are tidied up and split into chunks by meaning: a section, a clause, an answer to a question.
2. Index. Each chunk becomes an embedding, a numeric "fingerprint of meaning", stored in a vector database.
3. Retrieval. The question is also turned into an embedding, and the closest chunks by meaning are found.
4. Answer. The model gets the question, the chunks and a rule: "answer only from these".
5. Source. A good system shows which document and clause the answer came from.

Embeddings and vector databases in plain words

An embedding is a list of numbers describing the meaning of a text. Texts with similar meaning get similar numbers even when the words differ. "How do I return an item" and "processing a refund" end up close together despite sharing almost no words.

A vector database stores these embeddings and quickly finds those closest to a question. It can run on your own server: for example PostgreSQL with the pgvector extension, or a dedicated vector database such as Qdrant. For small volumes, the built-in features of automation builders such as n8n are enough.

Retrieval: by meaning and by keywords

Meaning-only search sometimes misses exact things: SKUs, contract numbers, surnames. Production systems often combine search by meaning with keyword search; this is called hybrid search. The found chunks are then sometimes re-ranked so the model gets the most relevant ones.

Chunking: half the quality depends on it

• Split on meaning boundaries: section, clause, Q&A, not blindly every N characters.
• Keep the section heading in every chunk, or "14-day term" loses what it refers to.
• Never split tables in the middle.
• Keep a document link and date on each chunk to show sources and filter out stale content.

RAG agents: when one search is not enough

A simple RAG system does one search per question. A RAG agent decides whether to search, what to search for and whether to search again. Asked "how does plan A differ from plan B", it finds both plan descriptions separately and only then compares them. Agents help with complex questions but are slower and more expensive, so start with the simple scheme.

A prompt for answering from documents

"You are the assistant of [company]. Answer the question only from the document fragments below. If the answer is not in them, say you found no information and suggest contacting a specialist. Do not assume. At the end, name the document the answer came from. Fragments: [retrieved chunks]. Question: [question]."

The key is the "not in the fragments, say you don't know" rule. Without it, RAG stops being reliable.

Where RAG helps a business

• Customer support. An assistant answers from the knowledge base, prices and terms, and honestly hands hard questions to a person.
• Internal helper. Staff ask about a policy, leave or a contract and get an answer with a link to the clause.
• Sales. Quick answers about the catalogue, specs and delivery terms.
• Legal and procurement. Finding the right clauses across many contracts.
• Technical documentation. An engineer asks about equipment parameters and the system finds the right manual page. The digital equipment passport in EnergyAI is built on the same principle.

RAG, fine-tuning or long context

	RAG	Fine-tuning	Everything in the context window
Adding data	drop a document into the base	train again	include in every request
Freshness	right after upload	after retraining	immediate
Answer source	can be shown	usually not	possible but harder
Cost per request	low, only relevant chunks	low	high at large volumes
Best for	answers from documents and facts	style, format, narrow skills	small volumes, one-off tasks

For most "answer from our documents" tasks, RAG is the right choice. Long context works when there are few documents; fine-tuning is for teaching manner or format, not facts.

Where RAG breaks

• Poor chunking. A chunk cuts off mid-table and the answer is incomplete.
• Stale documents. An old price list is in the base and the system quotes it confidently.
• Wrong match. Documents similar in wording but different in meaning get in each other's way.
• No "I don't know" rule. If the answer is not in the documents, the model fills gaps.
• Contradictory documents. Two versions of a policy, and the system picks one at random.
• No testing. The system was not tested on real questions before launch.

How to measure RAG quality

Collect 30-50 real questions with correct answers. Run them through the system and count three things: how many answers are right, how many cite the right source, and how often the system honestly says "I don't know" when the answer is not in the documents. The last one matters as much as the first: a confident invention does more harm than an honest "I'll check".

Launching RAG without overspending

1. Collect 30-50 real questions it needs to answer.
2. Prepare the documents that answer them and remove outdated ones.
3. Build a simple chain: retrieval, model, the "documents only" rule.
4. Test on those questions and see where it fails.
5. Connect it to a channel: website, Telegram, an internal messenger.
6. Assign someone to keep the base current.

Costs come from preparing documents, storing the base and model requests. Model spend is usually small: only the retrieved chunks go into each request, not the whole archive. Tokens and budgeting have their own article.

FAQ

What is RAG in simple terms?

A method where the AI model first finds the relevant pieces of your documents and answers from them, not from its own memory.

What is a RAG system?

A combination of a document base, meaning-based search and a language model with a rule to answer only from what was found.

What is RAG for LLMs?

A way to supplement a large language model (LLM) with external knowledge at answer time, without changing the model itself.

Do I need to retrain the model for RAG?

No. The model stays the same; only the documents handed to it change.

What are embeddings?

Numeric "fingerprints of meaning" of text. The system uses them to find fragments close in meaning to the question, even when the words differ.

Can RAG make mistakes?

Yes, if documents are outdated or retrieval finds the wrong thing. That is why you need testing on real questions, source links and a "not in the documents, say you don't know" rule.

How much does RAG cost to implement?

It depends on document volume, channels and quality requirements. A knowledge-base assistant can start small; a system with agents and integrations costs more.