An AI agent can leave a detailed record of everything it does and still operate without meaningful oversight.
That distinction matters in a quality system. A log can show which records an agent opened, what it changed, and where it sent the result. But the log itself cannot tell you whether the agent reviewed the right evidence, reached a sound conclusion, or overlooked something important.
Someone—or a deliberately designed set of controls—still has to check.
Quality professionals already know this problem from data integrity. Turning on audit trails was never enough. Organizations also had to decide who would review them, what those reviewers should look for, and how concerns would be investigated.
AI agents bring that same challenge to a much larger volume of work.
In The Hidden Debts of AI, I called this oversight debt: the growing gap between the work AI systems perform and our capacity to supervise, investigate, and challenge that work.
Inside a quality system, that gap can be easy to miss. The documentation looks complete. The drafts read well. Approvals keep moving.
The question is whether anyone can tell when the agent is wrong.
The lesson from data integrity
An audit trail supports oversight by preserving evidence. It does not provide oversight simply by existing.
FDA’s data integrity guidance connects audit trail review to the review of the underlying GMP records. Where regulations do not specify a review frequency, the approach should reflect process knowledge and risk.
The underlying principle is familiar: automated work still needs verification.
For drug manufacturing, 21 CFR 211.68(b) ties the degree and frequency of computer input and output verification to the system’s complexity and reliability.
That principle is particularly relevant to AI agents. An agent may perform one task reliably and struggle with another. It might summarize an operator’s note accurately, yet fail to recognize that a separate alarm record contradicts it.
We therefore need to ask a more specific question than, “Does this agent usually produce good results?”
We need to ask, “How do we verify that it performs this particular task reliably, and what happens when it does not?”
Why reviewing the finished draft is not enough
Consider a fictional manufacturing plant that uses an AI agent to prepare deviation reports.
The agent reads batch records, retrieves related documents, drafts a description, proposes a classification, and routes the report to Quality.
Suppose it handles 40 deviations each week and performs about 50 actions per deviation. That produces 2,000 logged actions from one agent.
At just 30 seconds per action, reviewing every entry would take nearly 17 hours a week. That does not include investigating concerns, checking source records, or reviewing work from other agents.
In practice, the QA manager reviews the finished report. She checks whether the description is clear, whether the classification seems reasonable, and whether the proposed next steps make sense.
That review has value. But it may not reveal missing evidence.
Imagine that the agent classifies a cold-room temperature excursion as minor. Its report relies on an operator’s note stating that the excursion lasted 20 minutes.
The alarm history, however, shows that it lasted six hours.
The agent never retrieved that history.
A detailed activity log might help an investigator discover the omission. But unless the review process checks whether the required evidence was consulted, the omission may pass unnoticed.
A polished report can make an incomplete investigation look complete.
The solution is not necessarily to read all 2,000 log entries. For this task, a better control might require the alarm history to be retrieved, compared with the operator’s account, and presented to the reviewer before a classification can be approved.
That is the difference between collecting evidence of activity and designing effective oversight.
Attention is a limited resource, too
Even if we could allocate enough review hours, we would still face a human limitation: people struggle to remain alert when they repeatedly review work that appears correct.
Lisanne Bainbridge explored this problem in her 1983 paper, Ironies of Automation. Automation can leave people responsible for detecting rare failures while giving them fewer opportunities to practice the work they may need to take over.
AI agents create a similar tension.
When an agent produces accurate, well-written reports day after day, a reviewer may gradually shift from investigating the evidence to confirming that the output looks reasonable.
The approval step remains. The depth of review can diminish.
This is why “a human approves everything” is not, by itself, a complete oversight strategy. We also need to know what that person checks, what information they receive, how much time they have, and whether they can recognize and challenge a plausible mistake.
Design oversight around the consequences
Before deploying an agent, decide which decisions require human judgment, which activities can be checked through sampling, and which boundaries need automated controls.
These approaches should work together.
1. Require human decisions where the consequences warrant them
For consequential decisions, the agent can prepare the analysis, but a qualified person should make the decision.
Depending on the process and applicable requirements, examples may include deviation classification, batch disposition, and decisions about whether further investigation is necessary.
The reviewer needs more than a recommendation. They need the evidence supporting it, any conflicting information, and any unresolved gaps.
For the cold-room excursion, that would include the operator’s account, the alarm history, the applicable limits, and an explanation of any disagreement between those sources.
A list of records the agent accessed can help. A check against the records it was required to access is more useful. The agent cannot be relied on to identify every omission in its own work.
2. Sample routine work against the source records
Some activities may be suitable for risk-based sampling, provided that sampling does not replace required reviews.
Examples might include routine summaries, document routing, and administrative field completion.
The review should compare the agent’s work with the underlying records. Checking whether a summary sounds reasonable is different from checking whether it accurately represents the evidence.
The sample should also include work the agent considered acceptable. If reviewers examine only the exceptions the agent flags, they may never discover the problems it failed to recognize.
Sampling should adapt to experience. A new task, a system change, or a pattern of errors may justify more intensive review.
3. Enforce boundaries, and alert people to exceptions
An agent should have a defined scope of authority: the records it may access, the changes it may make, and the actions that require approval.
Where possible, technical controls should prevent actions outside that scope. Monitoring should detect attempted violations and other unexpected behavior.
Examples include attempts to modify an approved record, access an unauthorized system, bypass an approval, or perform an unusual volume of changes.
Every important alert needs a named owner and a defined response. Otherwise, alerts can become another growing collection of unread records.
The monitoring also needs testing. A control that is supposed to detect unauthorized changes should be challenged to confirm that it actually does.
Keep people capable of intervening
Oversight depends on people retaining enough understanding to recognize a problem and respond.
A reviewer who routinely compares conclusions with source evidence continues to exercise judgment. A reviewer who mostly clicks “approve” has fewer opportunities to maintain that skill.
Training should therefore include more than instructions for using the agent. People should practice identifying missing evidence, challenging incorrect conclusions, and taking over when the agent cannot be trusted to continue.
Bainbridge’s warning remains relevant: systems that rarely need human intervention can still require substantial investment in human readiness.
The less often people intervene, the more deliberately we may need to preserve their ability to do so.
What meaningful oversight should be able to demonstrate
If an auditor asks how an AI agent is controlled, activity logs are a starting point.
The more useful evidence shows how the organization verifies the agent’s work, detects failures, and responds.
Before putting an agent into a quality workflow, I would want clear answers to four questions.
1. Who is responsible for oversight?
Name the responsible person or role. Define the required competence, allocate time for the work, and provide authority to challenge or stop the agent.
2. What does the review actually examine?
Specify which decisions receive individual review, which outputs are sampled, and which source records must be checked. Review records should show what was examined, what was found, and what happened next.
A long history of “no findings” does not automatically indicate weak oversight. But it should be supported by evidence that the review process can detect meaningful errors.
3. Can someone effectively override or stop the agent?
The ability should be tested, even if it has never been needed in routine operation.
An absence of overrides does not prove that the agent is perfect or that oversight has failed. It does leave an important question: have we demonstrated that people recognize when intervention is needed and know how to carry it out?
4. What can the agent do without approval?
Document its authority, approve that scope, and verify that the controls enforce it. Reassess those boundaries when the agent’s tasks, integrations, or capabilities change.
These questions turn “we have a human in the loop” into something concrete and testable.
Build the oversight before the workload grows
This is the work we do at Certify Consulting Group. We help regulated manufacturers design AI oversight that fits the risk, produces defensible evidence, and keeps people capable of intervening.
Our AI in the Quality System Review is a fixed-fee, two-week assessment of where AI is writing, checking, or making recommendations within your quality system. We examine the records it affects, the controls around its work, and the gaps that could create inspection concerns.
You can book a free 30-minute call through the service page or send a note through the contact page.
The lesson from data integrity still applies: recording activity is only part of the job. Effective oversight also requires defined responsibilities, meaningful checks, and action when something is wrong.
AI agents need those controls before their output exceeds our ability to evaluate it.
If your agent missed a critical piece of evidence tomorrow, what would catch the mistake before someone acted on its conclusion?
First published in my LinkedIn newsletter, AI in Regulated Industries.
Jared Clark
Principal Consultant, Certify Consulting
Jared Clark is the founder of Certify Consulting, helping organizations achieve and maintain compliance with international standards and regulatory requirements.