RAG or fine-tuning: diagnose the failure first

An assistant quotes an outdated policy. Before commissioning a training run, find out how that policy reached the answer. It might have come from the search index, a cached response or an example in the prompt. Training the model would leave some of those causes untouched.

The useful distinction is between an information problem and a behaviour problem. Retrieval supplies evidence for a particular request. Fine-tuning changes model behaviour through training. Either investment should have a failure it is expected to fix and a test that can demonstrate the improvement.

Inspect the evidence the model received

Consider a hypothetical returns assistant. The business changes its policy, yet the assistant still gives the previous return window. Preserve the question, the test user's access and the document version that should apply. Inspect the actual retrieved passages.

A missing current policy points to ingestion or retrieval. An obsolete passage suggests a versioning, deletion or cache problem. If the correct passage was supplied, investigate how the model used it. Manually providing the right evidence is a useful diagnostic experiment, followed by a broader set of representative questions.

This separates work that belongs in the search system from work that belongs in instructions or generation. It also gives a buyer something more concrete to review than a preference for one architecture.

When correct information is missing, inspect RAG sources, the index, permissions and cached answers. When the information is present, test instructions, examples and output validation. A variant that passes moves to the final comparison. A persistent behaviour failure in a stable task can justify a fine-tuning experiment. Compare quality, time and cost using the same queries.

Test retrieval when the facts must be looked up

Changing documentation, source inspection and user-specific access are reasons to evaluate RAG. Retrieval can use exact matching, semantic search or a combination chosen for the questions being asked. A vector database is one implementation choice.

Make document lifecycle part of the acceptance test. Upload a new policy, identify the indexed version and check which version appears in answers. Repeat the exercise for deletion. Hosted retrieval has its own consistency rules: OpenAI's retrieval documentation describes asynchronous operations and a delay before removed files disappear from search.

Citation checking needs two separate questions. Does the passage support this particular assertion? Can this user access it? A working link is insufficient evidence of either. Keep the document identifier and version so a later investigation can distinguish a changed source from an incorrectly generated answer.

Consider training for a stable task

Fine-tuning is worth testing when the model has the necessary information but repeatedly applies an established task incorrectly. Internal classification rules or specialised extraction decisions can fit this description.

First test instructions and representative examples. If the problem is output structure, enforce a schema and validate the returned values. Structured Outputs documentation explicitly notes that schema-conforming responses can still contain mistakes. Training should not become an expensive substitute for ordinary validation.

Build training examples from decisions that domain reviewers can agree on. Reserve separate evaluation data and keep it out of the training process. Disagreement over labels or an unstable definition of a correct answer needs to be resolved before the model can usefully learn it.

OpenAI's model optimisation guide places evaluation alongside prompting and training. The technique and a provider's hosted offering are separate decisions. Check current model availability, commercial access and retirement policies before committing to a particular training service.

Compare changes under the same conditions

Start with a model given sufficient context and clear instructions. Keep the questions and acceptance criteria fixed. Introduce retrieval to address observed information failures, then consider training for the remaining behavioural failures. A combined solution earns its place when each part makes a useful contribution.

Your evaluation should distinguish retrieval success from answer quality. Track unsupported assertions and appropriate abstention as well as correct answers. Include questions whose answers are absent, documents that disagree and cases with access restrictions.

Measure total task time and the cost of an accepted result. Include retries, infrastructure and the work needed to review or correct an answer. Comparing inference prices alone can favour a cheaper system that creates more manual work.

A useful evaluation report preserves failed cases rather than hiding them inside an average score. Repeat uncertain cases and test for regressions in tasks that the original system handled correctly. The decision is whether the change improves the work sufficiently for its cost and risk.

Buy a testable intervention

A technical proposal should name the failure, the proposed intervention and the result that would justify continuing. Ask what would make the team reject its preferred approach. Separate data preparation from implementation and ongoing operation so the cost of maintaining the system remains visible.

RAG consulting is relevant when evidence retrieval and access are the main problem. An AI technical audit is a better entry point when several layers already exist and the source of failure is unclear.

Send an anonymised failing query and the expected answer. That provides a concrete starting point for scoping a review before committing to a rebuild or a training programme.