Almost every organisation exploring generative AI reaches the same question: how do we make a large language model useful with our own information, such as policies, product documentation, contracts, tickets and knowledge bases? Two approaches dominate the conversation: retrieval-augmented generation (RAG) and fine-tuning.
In the RAG vs fine-tuning debate, the two approaches actually solve different problems, and choosing the wrong one wastes months. This guide explains how each works, when to use which, a reference architecture for enterprise RAG, how to secure and evaluate it and the mistakes we see most often.
The options at a glance
- Prompt engineering: writing better instructions and examples in the prompt. Always the first step, and sometimes enough.
- Retrieval-augmented generation (RAG): finding relevant passages from your data at query time and giving them to the model to answer from.
- Fine-tuning: training an existing model further on your own examples so its behaviour changes permanently.
- Tools and agents: letting the model call APIs or databases to fetch data or take actions, often combined with RAG.
How does retrieval-augmented generation work?
- Ingest: connect to sources such as SharePoint, Confluence, file stores, ticketing systems or databases.
- Chunk: split documents into passages sized to preserve meaning, often keeping headings and metadata.
- Embed: convert each chunk into a vector with an embedding model.
- Index: store vectors, text and metadata, including permissions, in a vector-capable search index.
- Retrieve: for each question, run a search, ideally hybrid keyword and semantic search, then re-rank the best candidates.
- Augment and generate: pass the top passages to the model with instructions to answer only from them and cite sources.
- Evaluate and monitor: measure answer quality, track failures and improve retrieval over time.
How does fine-tuning work?
- Define the behaviour you want: a format, tone, classification or specialised task.
- Curate training examples: hundreds to thousands of high-quality input and output pairs, reviewed by experts.
- Train: either full fine-tuning or parameter-efficient methods such as LoRA that adjust a small set of weights.
- Evaluate against a held-out test set and check for regressions in general capability.
- Deploy and maintain: retrain when requirements change or better base models appear.
RAG vs fine-tuning: detailed comparison
| Factor | RAG | Fine-tuning |
|---|---|---|
| Best at | Answering from specific, current knowledge | Consistent behaviour, style, format or narrow tasks |
| Keeping information up to date | Re-index changed documents, often automatically | Requires retraining |
| Citations and traceability | Natural: answers link to source passages | Difficult: knowledge is baked into weights |
| Respecting user permissions | Yes, by filtering retrieval | No practical way to hide trained knowledge per user |
| Hallucination control | Strong when instructed to answer only from sources | Model can still invent facts confidently |
| Data needed | Your existing documents and data | Curated, labelled examples |
| Upfront effort | Moderate: pipelines, index, evaluation | Moderate to high: data curation and training |
| Ongoing cost drivers | Retrieval infrastructure, longer prompts | Retraining, hosting a custom model |
| Typical use cases | Policy assistants, support knowledge, contract Q&A, sales enablement | Classification, extraction formats, domain phrasing, smaller specialised models |
When should you use RAG?
- Answers must reflect documents that change weekly or daily.
- Users need to see where an answer came from.
- Different users are allowed to see different information.
- You want to launch quickly with existing content.
- Regulators, auditors or customers may ask how an answer was produced.
When should you fine-tune?
- The model repeatedly fails to follow a required output structure even with good prompts.
- You need a specialised skill, such as classifying tickets into your own taxonomy, at high volume and low cost.
- You want a smaller, cheaper model to match a larger model on one narrow task.
- Your domain language is unusual enough that a general model misunderstands it.
When should you combine them?
Many production systems use RAG for facts and a fine-tuned or carefully prompted model for behaviour. For example, a support assistant retrieves the latest product documentation, while a fine-tuned model writes replies in the company’s required structure and tone. Start with prompting and RAG; add fine-tuning only when evaluation shows a behavioural gap that prompting cannot close.
A simple decision flow
- Can better prompts and examples solve it? If yes, stop there.
- Does the answer depend on your specific or changing information? If yes, use RAG.
- Do users need citations or permission-based access? If yes, RAG is required.
- Is the remaining problem about format, tone or a narrow task at scale? If yes, consider fine-tuning.
- Does the application need to take actions in other systems? Add tools or an agent framework with strict controls.
Reference architecture for an enterprise RAG application
| Layer | Purpose | Common choices |
|---|---|---|
| Connectors and ingestion | Pull content and permissions from source systems on a schedule or event | Custom connectors, integration platforms, cloud data pipelines |
| Processing | Clean, chunk, enrich with metadata, redact sensitive data | Python services, serverless functions, document parsers |
| Embeddings | Convert text to vectors | Embedding models from major cloud AI platforms or open-source models |
| Search index | Hybrid keyword and vector retrieval with permission filters | PostgreSQL with pgvector, OpenSearch, Azure AI Search, managed vector databases |
| Language model | Generate grounded answers | Managed model services on AWS (Amazon Bedrock), Microsoft Azure or Google Cloud (Vertex AI), or self-hosted open models |
| Orchestration | Query rewriting, retrieval, re-ranking, prompting, tool calls | Application code or an orchestration framework |
| Guardrails | Input and output checks, prompt-injection defence, topic limits | Platform safety features plus custom rules |
| Evaluation and observability | Quality metrics, tracing, cost and latency monitoring, user feedback | Test harnesses, logging and monitoring platforms |
If your knowledge and customer data already live in Salesforce, Salesforce’s own agent platform may be a faster route. Our Agentforce guide explains that option.
Securing an LLM application
- Permission-aware retrieval: filter by the user’s access rights before passages reach the model.
- Data minimisation: redact or exclude personal and sensitive data that the use case does not need.
- Prompt injection defences: treat retrieved content and user input as untrusted; never let documents override system instructions or trigger actions without checks.
- Vendor terms: confirm how model providers handle prompts and outputs, including retention and training use.
- Audit logging: record questions, retrieved sources and answers for review, with appropriate retention.
- Regulation: assess obligations under frameworks such as the EU AI Act, whose requirements are being phased in, and sector-specific rules.
How to evaluate quality
Create a "golden" test set of 100–300 real questions with approved answers and source documents, then measure:
- Retrieval quality: are the right passages in the top results?
- Faithfulness: is every claim in the answer supported by retrieved sources?
- Answer relevance and completeness
- Correct refusals when the information is not available
- Latency and cost per query
- User feedback collected in the interface
Run the suite after every change to chunking, prompts, models or data sources. Most quality gains come from better retrieval, not bigger models.
Common mistakes
- Fine-tuning to "teach the model our documents", then discovering it cannot cite or update them
- Indexing everything without cleaning duplicates, outdated versions and irrelevant files
- Ignoring permissions until a security review blocks launch
- Judging quality by a few demos instead of a measured test set
- No owner for content quality after launch
- Letting the model take actions without approval steps for high-impact operations
Where to start
Pick one knowledge-heavy workflow with clear value, such as answering internal policy questions or helping support agents find product answers, and build a measured proof of concept on a curated document set. Our overview of generative AI use cases with real ROI can help you choose, and our guide to building versus buying software covers whether to buy an off-the-shelf assistant instead.
Build your LLM application with Groviya
Groviya’s engineers design and build RAG applications, fine-tuning pipelines and AI agents on AWS, Azure, Google Cloud and Salesforce, with security, evaluation and monitoring built in from day one. Explore our application development services or talk to our AI team about a proof of concept.
