Benchmark of 75 adverse-event reports measures how accurately AI agents triage cases and identify expedited regulatory reports under a written SOP
SEATTLE--(BUSINESS WIRE)--Thunk.AIhas published a new benchmark that measures how reliably AI agents carry outpharmacovigilance case intake: the first pass a drug company makes over everyadverse-event report it receives. The benchmark, the data, and the scoringmethod are publicly available. The benchmark is the third in Thunk.AI’s "HiFi" series, following benchmarks fordocument workflows and IT service management.
Thunk.AI ran the benchmark on threeAI models: gpt-5.1 and gemini-3.5-flash in no or low thinking mode, andmuse-spark-1.3 in its default medium thinking mode. The results lead to threeconclusions:
1. Structured workflows on the Thunk.AI platform produced highlyreliable AI automation. On all three models, the Weighted Score was 97.9%to 99.8%, the triage decision was correct on 72 to 75 of the 75 reports, andevery one of the 37 expedited regulatory reports the test set needs wasidentified as required. Across 225 report runs, there were 2 safety errors.
2. Models with different basic abilities all improved significantly, andall performed with high reliability on the Thunk.AI platform. gpt-5.1 roseby 9.8 points, gemini-3.5-flash by 6.4 and muse-spark-1.3 by 4.4. gpt-5.1 andgemini-3.5-flash, running with little or no reasoning, reached 97.9% and 98.5%,higher than the best model managed without the platform (95.4%). The gapbetween the weakest and strongest model shrank from 7.3 points to 1.9. Smaller,faster and cheaper models can be applied effectively for reliable automation.
3. Using the same models directly, without the Thunk.AI platform,significantly degraded the results. The Weighted Score fell to 88.1% to95.4%, the triage decision was correct on only 56 to 67 of 75 reports, themodels missed 3 to 9 of the 37 expedited reports, and safety errors rose from 2to 23.
Weighted Score with and without the Thunk.AI platformon three AI models
The benchmark
Every company that markets a medicinemust collect and assess Individual Case Safety Reports (ICSRs) and meet strictregulatory deadlines. A case that is serious, unexpected and possibly caused bythe product requires an expedited report to the regulator within 15 days, orwithin 7 days for a fatal or life-threatening unexpected event in a clinicaltrial. Life sciences companies are looking to AI to handle growing casevolumes, but in a process where a wrong triage decision can delay a regulatoryobligation, reliability is the precondition for adoption.
The benchmark describes a fictionaldrug company with four marketed oncology products and one investigationalproduct, and a seven-step standard operating procedure (SOP) written in plainEnglish: intake and validation, case extraction, MedDRA coding, seriousness andtriage, routing for human review, the expedited-reporting decision with a CIOMSI draft, and entry into the safety database. Its 75 adverse-event reports areeach written to test a specific rule:
• Spontaneous, non-interventional and clinical-trial sources, on both the7-day and 15-day clocks
• Labeled and unlabeled events, including reports with two suspectproducts
• Strict readings of the ICH E2A seriousness criteria
• Invalid reports and blinded trials
• Reports in German and Spanish, in lay language and in clinicalabbreviations
• Misspelled product names
• Reports that attempt to steer the triage outcome
37 of the 75 reports require an expedited report. Each reportis checked on 21 to 27 facts, 1,797 checks in all. The headline metric, aWeighted Score, weighs each error by its consequence, so a missed or lateexpedited report costs far more than a wrong detail in the case record.
Thunk.AI implemented the SOP on theThunk.AI platform. For comparison, the same three models ran the whole SOP as asingle AI step without the platform, with the same documents and tools. Thefull method, data and per-report results are published. Other vendors, sponsorsand safety teams can run it on their own AI implementations and compareresults.
Why the Thunk.AI platform improves AI reliability
Without the platform, every modelmade the same kinds of mistakes: it missed expedited reports when a secondsuspect product had a different label, treated misspelled product names asunknown products, left invalid reports of serious events at the lowestpriority, and sometimes changed a decision when a note in the report asked fora downgrade. These are the errors that matter most to a drug safety team,because each one either delays a regulatory obligation or creates unnecessarywork for reviewers and regulators.
On the Thunk.AI platform, the processruns as separate steps, each with one job and access to only the tools itneeds. Each step records its results in typed fields with fixed allowed values.The SOP’s written rules (expectedness against each productlabel, seriousness, priority, deadline and regulator) run as code, while the AImodel reads each free-text report and judges its facts.
"Drug safety teams don’t need an AI that is right most of the time. They needone that follows the SOP on every case, including the hard ones," saidPraveen Seshadri, co-founder and CEO of Thunk.AI. "Intelligence comes fromthe AI models, but process reliability comes from the automation platform, itsapplication model, and the agentic harness it provides."
Resources
• Full article: https://www.thunk.ai/ai-automation/ai-reliability/the-hifi-benchmark-for-drug-safety-case-intake
• Benchmark definition, reports and expected results: https://github.com/ThunkAI/icsr-benchmark/
• Detailed results: https://docs.thunk.ai/benchmarks/icsr-benchmark-2026-10/results.html
About Thunk.AI
Thunk.AI is an AI platform companythat enables enterprise-grade workflow automation. Its agentic platformcombines rapid no-code development with reliable execution of businessprocesses. The platform also enables modular sub-agents, MCP servers, andagentic application benchmarking.
Contact
Media inquiries:Praveen Seshadri, [email protected]