AccScience Publishing / AIH / Online First / DOI: 10.36922/AIH026370137
Cite this article
10
Download
295
Views
Related Info Links
More by Authors Links
Journal Browser
Volume | Year
Issue
Search
News and Announcements
View All
ORIGINAL RESEARCH ARTICLE

Verification feedback and self-revision for omissions in clinical recommendations by large language models: A paired evaluation

Sanjay Basu1,2* ,  Sadiq Patel2,3 ,  Parth Sheth2,3 ,  John Morgan2,4 ,  Rajaie Batniji2
Show Less
1 Department of Medicine, University of California, San Francisco, California , United States of America
2 Waymark, San Francisco, California , United States of America
3 School of Social Policy and Practice, University of Pennsylvania, Philadelphia, Pennsylvania , United States of America
4 Department of Medicine, Virginia Commonwealth University, Richmond, Virginia , United States of America
Received: 12 September 2026 | Revised: 21 October 2026 | Accepted: 28 October 2026 | Published online: 17 November 2026
© 2026 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0/ )
Abstract

Large language models (LLMs) in clinical decision support can omit required actions (omission) or recommend wrong ones (commission). Verification layers check responses, and models can revise without feedback. Based on 176 test cases, we delivered the first responses of three LLMs (GPT-6 Astra, Gemini 3.1 Pro, Claude Opus 5.5) in six versions: no feedback, the full findings of a rule-and-agent verification framework, or alerts calibrated to a 10% false-alert budget, each appended or returned to the model for revision. Three LLM judges scored the share of high-grade (high-certainty) items on LLM-generated checklists that each text omitted; physicians reviewed samples. Revision without feedback (self-revision) lowered omission from 8.1% to 7.1% (−0.9 percentage points; 95% confidence interval [CI]: −1.7 to −0.1). In post hoc comparisons, revision with the full findings omitted 2.2 percentage points more than self-revision (95% CI: 1.3 to 3.2), or 0.8 (95% CI: −0.1 to 1.8) without the Gemini 3.1 Pro revisions that replaced content with status phrases. Revision with the alerts, which reached 56 of 528 first responses, did not differ from self-revision (0.5 percentage points; 95% CI: −0.3 to 1.3). Appended, the full findings changed judged omission by −0.6 percentage points (95% CI: −1.20 to −0.04). Based on 354 responses to physician-reviewed commission probes, both deliveries of the full findings lowered commission, mostly citations to nonexistent guidelines, from 6.8% to 3.4% (appended) and 4.2% (revised); with hedged commission counted, only revision lowered it (post hoc). Revision with verification feedback did not reduce omission beyond self-revision, which evaluations should include as a baseline.

Graphical abstract
Keywords
Clinical decision support
Large language models
Patient safety
Self-revision
Verification
Conformal risk control
Medical errors
Funding
None.
Conflict of interest
The authors are employees of Waymark, a public benefit organization that provides free social and medical services for patients receiving Medicaid. Waymark developed the verification framework evaluated in this study. The authors declare no other competing interests.
References
  1. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi: 10.1038/s41586-023-06291-2
  2. Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: A randomized clinical trial. JAMA Netw Open. 2024;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969
  3. Brodeur PG, Buckley TA, Kanjee Z, et al. Performance of a large language model on the reasoning tasks of a physician. Science. 2026;392(6797):524-527. doi: 10.1126/science.adz4433
  4. Fischer HD, Juurlink DN, Mamdani MM, Kopp A, Laupacis A. Hemorrhage during warfarin therapy associated with cotrimoxazole and other urinary tract anti-infective agents: A population-based study. Arch Intern Med. 2010;170(7):617-621. doi: 10.1001/archinternmed.2010.37
  5. Liu NF, Lin K, Hewitt J, et al. Lost in the middle: How language models use long contexts. Trans Assoc Comput Linguist. 2024;12:157-173. doi: 10.1162/tacl_a_00638
  6. Bhattacharyya M, Miller VM, Bhattacharyya D, Miller LE. High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus. 2023;15(5):e39238. doi: 10.7759/cureus.39238
  7. Wu K, Wu E, Wei K, et al. An automated framework for assessing how well LLMs cite relevant medical references. Nat Commun. 2025;16(1):3615. doi: 10.1038/s41467-025-58551-6
  8. Henry K, Xu S, Blotske K, et al. Large language models management of medications: three performance analyses. arXiv. Preprint posted online 2025. doi: 10.48550/arXiv.2509.22926
  9. Chen S, Kann BH, Foote MB, et al. Use of artificial intelligence chatbots for cancer treatment information. JAMA Oncol. 2023;9(10):1459-1462. doi: 10.1001/jamaoncol.2023.2954
  10. Sendak MP, Kim JY, Hasan A, et al. Empowering US healthcare delivery organizations: Cultivating a community of practice to harness AI and advance health equity. PLoS Digit Health. 2024;3(6):e0000513. doi: 10.1371/journal.pdig.0000513
  11. Shah NH, Halamka JD, Saria S, et al. A nationwide network of health AI assurance laboratories. JAMA. 2024;331(3):245-249. doi: 10.1001/jama.2023.26930
  12. Coalition for Health AI. Responsible AI Guidance. Coalition for Health AI; 2023. Accessed October 20, 2026. https://www.chai.org/workgroup/responsible-ai/blueprint-for-trustworthy-ai
  13. OpenAI. Usage Policies. OpenAI; 2025. Accessed October 19, 2026. https://openai.com/policies/usage-policies
  14. Anthropic. Usage Policy. Anthropic; 2025. Accessed October 19, 2026. https://www.anthropic.com/legal/archive/22742366-2ef0-4c7a-a833-6523f10d3944
  15. Google. Generative AI Prohibited Use Policy. Google; 2024. Accessed October 19, 2026. https://policies.google.com/terms/generative-ai/use-policy
  16. Banerjee S, Agarwal A, Singla S. LLMs will always hallucinate, and we need to live with this. In: Lecture Notes in Networks and Systems. Springer Nature Switzerland; 2025:624-648. doi: 10.1007/978-3-031-99965-9_39
  17. Xu Z, Jain S, Kankanhalli M. Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv. Preprint posted online 2024. doi: 10.48550/arXiv.2401.11817
  18. Karpowicz MP. On the Fundamental Impossibility of Hallucination Control in Large Language Models. arXiv. Preprint posted online 2025. doi: 10.48550/arXiv.2506.06382
  19. Kalai AT, Vempala SS. Calibrated language models must hallucinate. In: Proceedings of the 56th Annual ACM Symposium on Theory of Computing. ACM; 2024:160-171. doi: 10.1145/3618260.3649777
  20. Riegel R, Gray A, Luus F, et al. Logical Neural Networks. arXiv. Preprint posted online 2020. doi: 10.48550/arXiv.2006.13155
  21. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138-147. doi: 10.1197/jamia.M1809
  22. Qazi IA, Ali A, Khawaja AU, Akhtar MJ, Sheikh AZ, Alizai MH. Automation Bias in Large Language Model–Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy — A Randomized Clinical Trial. NEJM AI. 2026;3(5). doi: 10.1056/aioa2501001
  23. Madaan A, Tandon N, Gupta P, et al. Self-Refine: Iterative Refinement with Self-Feedback. In: Advances in Neural Information Processing Systems 36. Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2023:46534-46594. doi: 10.52202/075280-2019
  24. Huang J, Chen X, Mishra S, et al. Large Language Models Cannot Self-Correct Reasoning Yet. arXiv. Preprint posted online 2023. doi: 10.48550/ARXIV.2310.01798
  25. Kamoi R, Zhang Y, Zhang N, Han J, Zhang R. When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs. Trans Assoc Comput Linguist. 2024;12:1417-1440. doi: 10.1162/tacl_a_00713
  26. Ning J, Li X, Yu C. Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines. arXiv. Preprint posted online 2026. doi: 10.48550/ARXIV.2604.01029
  27. Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60-69. doi: 10.1038/s41591-024-03425-5
  28. Jin D, Pan E, Oufattole N, Weng WH, Fang H, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl Sci. 2021;11(14):6421. doi: 10.3390/app11146421
  29. Hendrycks D, Burns C, Basart S, et al. Measuring Massive Multitask Language Understanding. arXiv. Preprint posted online 2020. doi: 10.48550/arXiv.2009.03300
  30. Jin Q, Dhingra B, Liu Z, Cohen W, Lu X. PubMedQA: A dataset for biomedical research question answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics; 2019:2567-2577. doi: 10.18653/v1/d19-1259
  31. Baum A, Batniji R, Ratcliffe H, DeGosztonyi M, Basu S. Supporting rising-risk Medicaid patients through early intervention. NEJM Catal Innov Care Deliv. 2024;5(11). doi: 10.1056/CAT.24.0060
  32. Bedi S, Cui H, Fuentes M, et al. Holistic evaluation of large language models for medical tasks with MedHELM. Nat Med. 2026;32(3):943-951. doi: 10.1038/s41591-025-04151-2
  33. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. doi: 10.1136/bmj.39489.470347.AD
  34. Chen S, Gao M, Sasse K, et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digit Med. 2025;8(1):605. doi: 10.1038/s41746-025-02008-z
  35. Angelopoulos AN, Bates S, Fisch A, Lei L, Schuster T. Conformal Risk Control. arXiv. Published online 2022. doi: 10.48550/ARXIV.2208.02814
  36. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37-46. doi: 10.1177/001316446002000104
  37. Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Series B Stat Methodol. 1995;57(1):289-300. doi: 10.1111/j.2517-6161.1995.tb02031.x
  38. Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc. 1927;22(158):209-212. doi: 10.1080/01621459.1927.10502953
  39. Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10(1):101-129. doi: 10.2307/3001666
  40. McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947;12(2):153-157. doi: 10.1007/BF02295996
  41. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76(5):378-382. doi: 10.1037/h0031619
  42. Zhan Z, Cui M, Zhang R. Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study. arXiv. Preprint posted online 2026. doi: 10.48550/arXiv.2604.00261
  43. Stechly K, Valmeekam K, Kambhampati S. On the self-verification limitations of large language models on reasoning and planning tasks. In: The Thirteenth International Conference on Learning Representations (ICLR 2025); 2025:98190-98243.
  44. Basu S, Huynh B. Mitigating hallucinations in healthcare AI: a systematic review of evidence-based strategies. BMC Health Serv Res. 2026;26(1). doi: 10.1186/s12913-026-14851-1
  45. Liu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: A systematic review, meta-analysis, and clinical development guidelines. J Am Med Inform Assoc. 2025;32(4):605-615. doi: 10.1093/jamia/ocaf008
  46. Xiong G, Jin Q, Lu Z, Zhang A. Benchmarking retrieval-augmented generation for medicine. In: Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics; 2024:6233-6251. doi: 10.18653/v1/2024.findings-acl.372
  47. Kim H, Sohn J, Gilson A, et al. Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights. arXiv. Preprint posted online 2025. doi: 10.48550/arXiv.2511.06738
  48. Wu K, Wu E, Zou J. ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. In: Advances in Neural Information Processing Systems 37. Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2024:33402-33422. doi: 10.52202/079017-1053
  49. Barnett S, Kurniawan S, Thudumu S, Brannelly Z, Abdelrazek M. Seven failure points when engineering a retrieval augmented generation system. In: Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI. ACM; 2024:194-199. doi: 10.1145/3644815.3644945
  50. Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE. Hallucination-free? Assessing the reliability of leading AI legal research tools. J Empir Leg Stud. 2025;22(2):216-242. doi: 10.1111/jels.12413
  51. Asai A, Wu Z, Wang Y, Sil A, Hajishirzi H. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In: The Twelfth International Conference on Learning Representations (ICLR 2024); 2024:9112-9141.
  52. Su H, Wang R, Ji DY, Lee J. RC-RAG: A Reliability-Centered Retrieval-Augmented LLM Framework for PHM Question Answering. IEEE Trans Rel. 2026;75:2159-2173. doi: 10.1109/tr.2026.3702056
  53. Fox S, Markham L, Lail R, Karotsieris M. LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It. arXiv. Preprint posted online 2026. doi: 10.48550/ARXIV.2608.31016
Share
Back to top
Artificial Intelligence in Health, Electronic ISSN: 3029-2387 Print ISSN: 3041-0894, Published by AccScience Publishing