What a RAG and agent audit should deliver

Your team is preparing to give an assistant to more users, yet an incorrect answer remains unexplained. Switching models might improve that answer while leaving the underlying defect in place. An audit should identify the failure and provide a decision the team can act on.

Start by naming the business decision. Expanding document access, enabling a write tool and retaining the current architecture require different evidence. A successful retrieval review does not establish that a new external action is safe to automate.

Agree on the inspection boundary

The audit brief should identify the system version, affected users and proposed change. List the integrations included in the review and the ones excluded. Name the person who can authorise tests that affect external systems.

Use a test environment with synthetic or anonymised records where possible. When production observation is necessary, agree on readable data, retention and deletion of working copies. Broad administrator access should not be a prerequisite for understanding the architecture.

The available access determines what can be concluded. Screenshots support a review of visible behaviour. Diagnosing permissions or retry logic requires code, configuration and execution evidence. Document these limits before assigning confidence to the result.

Reconstruct a failing request

Preserve the input, expected outcome and test identity. Record model and application versions alongside document or operation references. This allows an investigator to follow the request without copying complete customer records into every diagnostic file.

For RAG, inspect retrieval before judging generation. Was the correct document indexed with its current permissions? Did the search return it? Did the answer use the passage accurately, and does the cited evidence support the claim?

Include document replacement and removal tests. OpenAI's retrieval guide describes asynchronous operations and eventual consistency for removal. Whatever provider the application uses, verify its actual behaviour through the whole pipeline, including cached responses.

Examine one external action

For an agent, trace a command that changes state. Establish who requested it, which permission authorised it and what the external service confirmed. Interrupt a test run at a controlled point and inspect recovery.

Losing the response to an accepted write is a valuable test. The application may repeat the command, query its status or leave it unresolved. AWS's idempotent API guidance explains the role of a persistent request identifier. The audit must test how the application actually uses that contract.

Also change a test user's permission between proposal and execution. An earlier approval should not silently authorise a different command or override a revoked permission. Record whether the server rejects the action and what the interface tells the user.

Use restricted identities in permission tests

An administrator session can conceal defects that affect ordinary users. Run the same request with an allowed identity and a denied identity. Cover tenant separation, source links and cached answers.

OWASP's vector and embedding risk guidance includes cross-context leakage and inadequate access controls. For PostgreSQL applications, inspect the effective database role. Row security documentation identifies exceptions for table owners and roles that bypass RLS.

A denied access test should demonstrate enforcement. A restricted document failing to reach the top of a search result is not evidence that the requesting user could never retrieve it.

Make findings reproducible

Each finding needs a triggering condition, examined version and reproducible evidence. Describe its observed effect on the user or business. Then propose a correction and an acceptance test that can show whether the defect has been fixed.

Deliver a component map and a test record alongside the findings. Keep model errors distinct from integration failures and data defects. Cases that could not be tested should remain visible, with the reason and the decision they prevent.

Quality metrics need a defined denominator and an explanation of what passes. An overall answer score can conceal a severe permission failure affecting a small group. Break results down by task and access level. Repeat variable model evaluations and inspect a human-reviewed sample, while retaining deterministic tests for application permissions and contracts.

The output should be usable by the team maintaining the system. A fix list without reproduction steps creates another investigation; a failed case with a clear acceptance test gives the next engineer something to implement and verify.

End with a bounded recommendation

State what can remain, which defects need correction first and what blocks the proposed release. When a fix is included in the engagement, rerun its acceptance test and record the final version. Sampled results are limited evidence; they do not certify all future behaviour or replace specialist legal or regulatory review where needed.

AI engineering consulting can scope this review around the decision at hand. If missing knowledge appears to be the main issue, the RAG and fine-tuning decision guide helps narrow the investigation.

Send the failure you are concerned about and the decision you need to make. That is enough to begin defining the audit scope, its deliverables and the access needed before quoting the work.