AI Wrote Their GMP Documents. It Never Mentioned Process Validation. FDA Did.

Last October, FDA investigators told a small Michigan drug maker that it had never validated its manufacturing process. The company said it hadn't known that was required, because the AI agent that it used to write its procedures (or more to the point, that it used to run its quality unit) had never told it so.

That is an easy answer to laugh at. FDA's recommended fix seems to be the obvious one: If AI helps write your GMP documents, then you need someone competent enough to review them and clear them. I agree with that in principle, but my research has shown that even this is not enough. Purolea's AI didn't hand them a wrong number. It left out a requirement. Anyone who has reviewed documents for a living knows it's far easier to catch what is wrong on the page than to notice what isn't there, and it gets harder still when a machine has handed you something that already looks finished.

Here's what's inside:

• What FDA actually cited. What trade press is calling FDA's first AI warning letter, and the one sentence in it that sets the rule.
• Why expertise doesn't solve it. 27 radiologists, a deliberately wrong AI, and what happened to the most experienced of them.
• Where the blame lands. Why the signature on an AI-drafted document absorbs the accountability, whoever caused the miss.
• Four questions to ask before you sign. What a real review of an AI-drafted specification, SOP or batch record looks like.

What FDA actually cited

Purolea Cosmetics Lab made homeopathic drug products in Livonia, Michigan. FDA inspected the plant from October 28 to 30, 2025, and most of what the investigators found will look familiar to anyone who has read a 483. But this warning letter had a category that nobody had seen before.

The new category was a section titled "Inappropriate Use of Artificial Intelligence in Pharmaceutical Manufacturing." The company had told investigators it used AI agents to create its drug product specifications, procedures and master production and control records. The company also admitted that it was not aware of a major regulatory requirement, process validation, "as the AI agent … never told [them] it was required." FDA cited the firm under 21 CFR 211.22(c), the regulation that makes the quality unit responsible for approving procedures and specifications, and the rule fits in one sentence from the letter: "If you use AI as an aid in document creation, you must review the AI generated documents to ensure they were accurate and actually compliant with CGMP."

If you run a Quality function, you have probably drawn the easy lesson already. In essence, Purolea didn't have a human reviewer at all, and your company does. So you're safe, right? I'm not so sure. A reviewer only protects you if they can resist the very human tendency to outsource the hard thinking to whoever (or whatever) will do it for them. Recent research shows that even the most seasoned professionals are prone to automation bias and complacency when they work alongside AI.

Expertise lowers automation bias. It doesn't remove it.

In 2023, researchers at the University of Cologne had 27 radiologists rate 50 mammograms with help from what they were told was an AI system. For 12 of those mammograms, the researchers made the AI suggest the wrong answer on purpose. When the AI was right, all three experience groups rated about 80 percent of the mammograms correctly. When it was wrong, the least experienced radiologists fell to about 20 percent. The most experienced held up best, and they still got fewer than half right.

That is automation bias, our tendency to over-rely on what an automated system tells us. Experience helped the radiologists a great deal, and it still didn't make them immune, even though they were specialists doing the work they had trained for years to do.

Now put an experienced Quality Manager in front of an AI-drafted finished-product specification. The tests are already chosen, the limits are already proposed, and every choice comes with a sensible explanation. The reviewer isn't working the problem anymore. They're grading an answer, and grading is good at catching a wrong limit and bad at noticing that a glycerin specification has no USP limit test for diethylene glycol, or that nobody ever mentioned process validation. A requirement the AI never raised isn't on the page, and a reviewer who was handed a finished page is the person least likely to go looking for it.

The signature is where the blame lands

Madeleine Clare Elish, a researcher who studies accountability in automated systems, calls this a moral crumple zone. A car's crumple zone absorbs the force of a crash to protect the driver. A moral crumple zone absorbs the blame when an automated system fails, and in her words it "protects the integrity of the technological system, at the expense of the nearest human operator."

In a GMP plant, the nearest human operator is whoever signed. When the investigator finds the missing test and asks who approved the specification, the answer isn't the AI. It's the Quality Manager.

I don't think FDA is wrong about that. Section 211.22 puts that responsibility on the quality unit, and it should. But if we roll AI out in a way that is built for speed, with polished documents arriving faster and reviewers expected to keep pace, then we have made independent review harder and made the reviewer answer for the result. That is the crumple zone, and we build it ourselves.

Four questions to ask before you sign

None of this is an argument for keeping AI out of Quality. Your team is almost certainly using it already, and the productivity gain is real. What has to change is the review. Before an AI-drafted specification, SOP or batch record gets a signature, the reviewer should be able to answer four questions.

1. What did I decide the document needed before I opened the AI's draft? A few minutes with the monograph and the product's own risks gives you a list of your own to compare against, before the AI's version anchors you.
2. Where did every requirement and limit come from? Each one should trace to a regulation, a monograph or your own data. Anything without a source gets flagged.
3. What did the AI leave out? Check the draft against the regulation and the monograph for what is missing, and not only for what is wrong.
4. What did I check, and what did I change? A signature records that someone approved the document. It doesn't record what they looked at, and after Purolea I expect investigators to start asking.

Those questions don't ban AI. They make sure someone in your building finds the gap before an investigator does.

This is also where AI governance and your quality system meet. ISO/IEC 42001 asks an organization to decide what an AI system is for and who answers for its output. A GMP quality system already asks that about every instrument and supplier that feeds a release decision. In my view the AI deserves the same treatment: qualified for its intended use, controlled when it changes, and checked on a schedule.

This is the work we do at Certify Consulting Group. We help regulated manufacturers design the AI in their quality systems so it holds up under FDA scrutiny, and so the person who signs isn't left as the crumple zone. If your team is already using AI on specifications, SOPs or batch records, our AI in the Quality System Review is a good place to start: a fixed-fee, two-week look at where AI is already writing, checking or deciding inside your quality system, and whether the records it touches would survive an inspection. You can book a free 30-minute call from that page, or send a note through the contact page.

Purolea's AI never told them about process validation. I don't think the lesson is that we need an AI that remembers to. The lesson is that someone has to know what belongs in the document before the AI starts writing it.

Would you let AI draft a specification, an SOP or a master batch record in your quality system? If you would, what would have to be true before you'd put your name on it?

First published in my LinkedIn newsletter, AI in Regulated Industries.