AI can audit papers, but not judge alone
AI agents are finding database and paper errors at scale, yet the cited studies range from useful high-precision checks to very low benchmark recall; every finding still needs human verification.
The 60-second version
AI agents can find scientific errors and rerun claims at scale, but current results range from useful high-precision checks to very low recall and unstable benchmark performance.
Key points
- A chemistry anomaly became a confirmed database error only after the original literature was checked.
- One preprint reported 83.2% precision after experts confirmed 263 of 316 candidate objective mistakes.
- On SPOT's 83 papers and 91 known serious errors, no tested model exceeded 21.1% recall or 6.1% precision.
- Reproduction failure does not itself prove author error, and novelty or significance still requires human judgment.
Verdict. AI is already useful as a scientific audit assistant, but trustworthy use requires reproducible traces, expert adjudication and clear separation between a flag and a confirmed error.
AI tools can scan databases and papers at a scale unavailable to human reviewers, and some have uncovered real errors. The same evidence also shows low recall, false positives and run-to-run instability. The defensible role today is machine-assisted triage followed by human verification, not autonomous verdicts on papers.
A useful patternAI finds an anomaly, people establish the fact
Chemist Sebastian Pios found that AI-predicted boiling points disagreed with a 75-year-old reference database. He initially suspected the model. Checking the original literature instead identified errors in the reference chain, including a typo and values linked to century-old measurements. The verification came from source records, not the model's confidence.
Objective checksOne checker reported high precision
A GPT-5-based Paper Correctness Checker focused on verifiable issues in formulas, derivations, calculations, figures and tables. Human experts reviewed 316 candidate mistakes and confirmed 263, for a reported precision of 83.2% in that reviewed set. The preprint intentionally excluded novelty, importance and writing quality.
Those results are promising, but they depend on the sampled papers, the checker's definitions and an AI-assisted pipeline. They do not show that all AI papers contain the same number of mistakes or that publication quality can be reduced to one count.
Hard benchmarkKnown serious errors were often missed
The SPOT benchmark paired 83 published papers with 91 serious errors that had prompted errata or retractions. No tested model exceeded 21.1% recall or 6.1% precision, and models rarely rediscovered the same errors across eight runs. This is a direct warning about missed faults, false alarms and instability.
| Anomaly detection | Flags a conflict between a model or calculation and a reference value; the source record must decide. |
|---|---|
| Objective error checking | Tests formulas, calculations, figures or tables where a ground truth can be established. |
| Reproduction | Attempts to rerun claims under an available environment; failure requires diagnosis before blame. |
| Peer judgment | Assesses novelty, importance, interpretation and ethics; these are not reducible to an automatic check. |
ReproducibilityA failed rerun is a question, not a conviction
A company analysis used agents on 168 ICML 2026 oral papers. Of 92 papers with at least five assessable claims, agents reproduced at least two-fifths of claims in 34 and more than 80% in eight. These numbers describe what the agents could rerun, not how many papers were fraudulent or false.
AI can make scientific checking faster; scientific due process still requires a trace, an expert and a chance to respond.
PracticeWhat a responsible audit needs
- Record the model, prompt, tools, code, data, environment and exact claim being checked.
- Attach a reproducible calculation or execution trace to every alleged error.
- Use domain experts to adjudicate findings and let authors supply missing context or corrections.
- Repeat checks or use independent systems to measure stability.
- Separate suspected issue, confirmed error, formal correction and retraction.
TakeawayUse AI to widen review, not erase responsibility
Researchers and editors can use agents for reference consistency, calculations, code execution and triage. Decisions about correctness and significance need reviewable evidence and accountable humans. The practical goal is not an AI score for every paper; it is a faster, auditable route from suspicion to confirmation or dismissal.