Audit Sampling for AI Decisions: A Defensible Approach

Audit sampling for AI decisions works only if you can defend three choices: why you sampled instead of testing everything, how the sample was drawn so it reaches the cases most likely to fail, and what evidence each sampled decision carries. Classical sampling assumes a population of static records. An AI agent is a policy that produces thousands of individual decisions, each shaped by different inputs, so the sample has to be designed around that fact.

This guide is written for risk, compliance and internal audit leaders at regulated financial firms. It covers the sampling decision, a stratified design, the evidence a sampled item must contain, and the metadata index that makes any of it feasible.

TL;DR

• Sampling is a risk-control exercise, not a volume-reduction exercise. Document why a subset is acceptable before you draw it.
• Where a query can cover the whole population, full-population review is usually stronger than a sample. Where it cannot, stratify by risk instead of drawing at random.
• Random samples under-represent the margins of the input distribution, which is where probabilistic systems tend to fail. Add targeted and adversarial items.
• Every sampled decision should produce a complete evidence bundle: authority, rule, approval, execution and exception handling.
• None of this scales without an index by agent identity and decision type, built from metadata from the first pilot.

Table of Contents

• What audit sampling means when the decision-maker is an AI agent
• Decide first: sample, or test everything?
• A stratified design for AI decision samples
• Why a clean sample can coexist with a defective model
• What a sampled decision must contain
• The index that makes sampling feasible
• Documenting the justification for sampling
• Ownership and review cadence
• A worked example
• How AETHER Pulse supports this
• Perspective: sampling is an argument, not a statistic
• FAQ
• Sources

What audit sampling means when the decision-maker is an AI agent

The PCAOB's standard defines audit sampling as applying an audit procedure to less than 100 percent of the items in an account balance or class of transactions, in order to evaluate some characteristic of that balance or class. That definition still holds. What changes with AI is the nature of the population and the cost of reaching all of it.

Three differences matter in practice.

1. The unit of testing is the decision, not the agent. You cannot audit "the agent" as a single object. You test a population of decisions produced by one policy under many contexts.
2. Behavior is probabilistic. The same input class can yield different outputs, so a control that passed last month proves less than a deterministic rule would.
3. Full-population testing is often technically feasible. If decision logs are queryable, a firm can test every item against a rule. The auditor's question then shifts from "how large should the sample be?" to "why did you sample at all?"

The AI Assurance Council's guidance on testing a probabilistic control makes this point directly: sampling was built for records that sit still, and the most important artifact is frequently the written justification for choosing a sample over the full population. That framework maps to NIST AI RMF MEASURE 2.3, which asks that performance be demonstrated under conditions similar to the deployment setting.

Decide first: sample, or test everything?

Before choosing a sample size, answer a prior question in writing. The answer is the audit artifact reviewers will read first.

The combination in the last row is the practical default. Automated checks run against everything, and human review is sampled. The justification then records which tests covered 100 percent of decisions and which covered a subset.

A stratified design for AI decision samples

A single random draw from all decisions mostly returns routine traffic. That is the weakest evidence you can offer, because errors in probabilistic systems concentrate elsewhere. The following four-layer structure is an illustrative design, not a regulatory prescription. Set the numbers by your own risk assessment and document the reasoning.

Layer 1: High-value and above-threshold decisions

Review every decision above a materiality or exposure threshold your risk function has set. For a lending, payments or trading context, that threshold is typically tied to monetary exposure per decision or per customer. This layer is not sampled.

Layer 2: Exceptions, overrides and escalations

Review every item where the agent escalated, was overridden by a human, hit a policy exception, or failed a validation rule. These items show whether escalation behavior is correct, which matters most for ambiguous cases.

Layer 3: Boundary cases

Select items near decision thresholds, such as scores just either side of an approval cutoff, unusual input combinations, or low-confidence outputs. Draw these deliberately rather than randomly. This layer is where you test the margins of the distribution.

Layer 4: Routine decisions

Draw a statistical sample from everything else, using a method you can describe and reproduce: seed, selection logic, population definition and date range. Reproducibility is what allows an auditor to rerun your draw and get the same items.

Why a clean sample can coexist with a defective model

The common misreading is that a sample with no exceptions proves the system works. In a probabilistic system, errors cluster at the edges of the input distribution, which random sampling naturally under-represents. The AI Assurance Council guidance warns that a clean sample and a defective model can coexist, and recommends adversarial challenge logs, cases designed to fail, as a required complement to representative sampling.

In practice, this means three additions to your test plan:

• Challenge cases. Maintain a set of synthetic or historical inputs built to trigger a wrong outcome, and record how the agent handled each.
• Edge-focused selection. Define the margins of your deployment population (rare product types, vulnerable-customer indicators, unusual amounts) and sample them on purpose.
• Drift comparison. Compare the distribution of sampled decisions across periods, not only the error rate within each period.

What a sampled decision must contain

A sample is only as good as the evidence behind each item. For any sampled agent action, a reviewer should be able to assemble a complete bundle within audit timelines. Aptly's guidance on audit-ready agent approvals describes such a bundle as the authority record, the rule applied, the approval if required, the execution log and exception handling. In a financial services setting, we would test each item against the following checklist:

1. Authorization chain. Who or what authorized the agent to take this class of action, and was that authority in force at the time of the action?
2. Approval gate compliance. Where human approval was required, did a person actually review it? Timestamps that show a genuine review matter more than a recorded click.
3. Reasoning trace completeness. Can the inputs, retrieved context and decision logic be reconstructed?
4. Outcome against intent. Did the result match what the policy or customer instruction intended?
5. Escalation behavior. For ambiguous cases, did the agent escalate as designed?

The same checklist appears in Sphere's framework for auditing agent decisions in banking, which also stresses that a reflexive approval is not evidence of oversight. If your approvers clear items in seconds, a sample will show that, and a regulator can read the same timestamps.

On the control side, our articles on decision pipeline evidence and segregation of duties for AI cover how to record intent before execution and how to keep the approver independent of the agent's operator.

The index that makes sampling feasible

Without an index by agent identity and decision type, sampling collapses into a manual search through logs. You cannot stratify what you cannot filter. The AI Assurance Council notes that metadata-level indexing is what makes both sampling and full-population review technically workable.

A minimal sampling index holds, per decision:

• agent identity and version
• decision type and business process
• timestamp and period
• exposure or value band
• approval status and approver identity
• exception or escalation flag
• pointer to the evidence bundle

Note that none of these fields requires customer data. Building the index from metadata keeps the audit layer outside the privacy perimeter of the decisions themselves. We cover that design in more depth in AI logging for compliance teams and AI identity graphs.

It also pays to build early. Retroactive evidence construction costs more as agent volume grows, and firms often wish they had kept versioned delegation records from the pilot phase, because auditors look for proof of authority at the moment of action.

Documenting the justification for sampling

When a subset is used instead of the whole population, write the justification down. A reviewer should find, in one page:

1. Population definition. Which agents, decision types and dates are in scope, and how many decisions exist.
2. Why full testing was not used. For example, the test requires human reading of a reasoning trace, or reviewer capacity is the binding constraint.
3. What was tested at 100 percent. List the automated checks that covered everything.
4. Selection method. Strata, seed, selection logic and sample sizes per stratum.
5. How the margins were covered. Boundary selection and challenge cases.
6. Result handling. How exceptions were evaluated, whether they were extrapolated, and who signed off.

This follows the logic of the PCAOB standard, which treats sampling as a planned and evaluated procedure. If you are in a US-registered audit context, note that the PCAOB states the amended AS 2315 takes effect on December 15, 2026, so check the current text when you plan your next cycle.

Ownership and review cadence

Sampling results need an owner who can act on them. Accountability for AI-driven decisions should map to a senior individual, not only to IT or data teams. The UK government's ethics, transparency and accountability framework for automated decision-making is a UK public-sector document, but its principle of answerable, auditable systems with regular formal review travels well. Quarterly review points are a sensible minimum for catching drift and unintended consequences.

To turn sampling results into oversight, track them over time. Approaches such as accountability scoring in governance decision making show how to compare stated commitments with measured delivery, which is the same gap a quarterly sampling review should expose between a policy's intent and an agent's actual decisions.

For teams building the surrounding structure, the AI agent governance framework and deterministic controls for AI describe how to make rule-based checks part of the control environment, so that more of the population can be tested automatically.

A worked example

Consider a hypothetical firm whose agents pre-screen customer requests and route some to a human for approval. Over a quarter, the agents produce 120,000 decisions. The figures are illustrative.

• Full-population automated checks: every decision is tested for a valid authority record, an approval where the policy requires one, and a complete log entry. Failures go to the exception queue.
• Layer 1: all decisions above the firm's exposure threshold are reviewed by a second-line analyst.
• Layer 2: every escalation and every human override is reviewed.
• Layer 3: reviewers pull decisions within a narrow band of the approval cutoff and cases involving vulnerable-customer markers.
• Layer 4: a seeded random draw from routine decisions, with the seed and query stored.
• Challenge set: 30 adversarial inputs are replayed against the current agent version, and the results are filed with the sample.

The sampling memo then states what was covered at 100 percent, what was sampled, and why. If an auditor asks to repeat the draw, the stored seed and query reproduce it.

How AETHER Pulse supports this

Sampling depends on the inventory and the evidence behind it. AETHER Pulse is a read-only governance layer that connects through OAuth metadata only, touches no customer data, and builds an inventory and identity graph of an organization's AI agents. It surfaces risk concentration, including financial blast-radius exposure, and generates deterministic, provenance-tracked evidence packs signed with HMAC-SHA256, so auditors can verify them offline.

For teams preparing for EU AI Act Article 26 obligations, we offer an Article 26 readiness audit, and design partner and production plans are described on our pricing page. You can also start from the AETHER Pulse home page.

Perspective: sampling is an argument, not a statistic

We see firms treat AI sampling as an arithmetic problem: pick a confidence level, compute a size, pull the items. That framing misses what auditors and regulators ask. They ask whether the firm understood where its agents could fail, and whether it can prove authority and oversight at the moment each decision was made.

Our view is that the sample should be the smallest layer of evidence, not the largest. The inventory, the identity graph and the metadata index should let a firm test far more of the population automatically, so human review is spent on high-exposure decisions, exceptions and edge cases. We also hold that evidence should be deterministic and traceable: the same inputs produce the same evidence pack, and every classification cites a public regulatory or industry source.

The firms that struggle are usually those that begin building the index after the audit request arrives. Build it from metadata at the start, and keep customer data out of it.

— Eleye

FAQ

What is the objective of audit sampling?

The objective is to reach a conclusion about a whole population by testing part of it. For AI decisions, the objective extends to showing that the subset was chosen deliberately, covers the riskiest cases, and is documented well enough for a reviewer to repeat it.

Is sampling still acceptable when AI can test every decision?

Yes, but you should be able to explain the choice. Where a query can test the full population against a rule, regulators and auditors increasingly expect that, and a documented justification is needed when a subset is used instead. Sampling remains reasonable for tests that need human judgment.

How large should an AI decision sample be?

There is no single number. Size depends on the stratum, the risk assessment, the tolerable error and how much of the population was covered by automated checks. Review high-value and exception items in full, and set sizes for the remaining layers in your sampling memo, with reasons.

What should I do if the sample finds no errors?

Do not treat it as proof. Errors in probabilistic systems cluster at the margins, which random samples under-represent. Add boundary cases and adversarial challenge inputs, and compare decision distributions across periods to check for drift.

Sources

• AS 2315: Audit Sampling, PCAOB
• Testing Controls on a Probabilistic AI System, AI Assurance Council
• Audit-Ready AI Agent Approvals: Evidence & Controls, Aptly Resources
• How to Audit AI Agent Decisions in Banking, Sphere Inc.
• Ethics, Transparency and Accountability Framework for Automated Decision-Making, GOV.UK
• Accountability Scoring in Governance Decision Making

Recommended

• Segregation of Duties for AI: Audit Ready ADEL for Compliance Teams
• Build Audit Ready Deterministic Controls for AI in Regulated Finance
• AI Logging for Compliance Teams: Audit Evidence Without Customer Data
• Decision Pipeline Evidence for Finance: Prove Intent Before Execution