News and Announcements
ORIGINAL RESEARCH ARTICLE
Leveraging summary of radiology reports with transformers
Show Less
1 Quod Analytics, Niterói, Rio de Janeiro,
Brazil
2 Department of Computer Science, University of Texas Austin, Austin, Texas,
United States of America
Received: 4 June 2024 | Accepted: 5 August 2024 | Published online: 26 September 2024
© 2024 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0/ )
Abstract
Two fundamental problems in health-care stem from patient handoff and triage. Doctors are often required to perform complex findings summarization to facilitate efficient communication with specialists and decision-making on the urgency of each case. To address these challenges, we present a state-of-the-art radiology report summarization model utilizing adjusted bidirectional encoder representation from transformers BERT-to-BERT encoder–decoder architecture. Our approach includes a novel method for augmenting medical data and a comprehensive performance analysis. Our best-performing model achieved a recall-oriented understudy for gisting evaluation-L F1 score of 58.75/100, outperforming specialized checkpoints with more sophisticated attention mechanisms. We also provide a data processing pipeline for future models developed on the MIMIC-chest X-ray dataset. The model introduced in this paper demonstrates significantly improved capacity in radiology report summarization, highlighting the potential for ensuring better clinical workflows and enhanced patient care.
Keywords
Text summarization
Natural language processing
Deep learning
Artificial intelligence
Health care
Bidirectional encoder representations from transformers
MIMIC-chest X-ray
Funding
None.
Conflict of interest
The authors declare that they have no competing interests.
References
- Makary MS, Takacs N. Are we Prepared for a Looming Radiologist Shortage? Radiology Today. Dignostic Imaging. 2022. Available from: https://www.diagnosticimaging.com/ view/are-we-prepared-for-a-looming-radiologist-shortage- [Last accessed on 2024 Jun 03].
- Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. Preprint posted online 2018. doi: 10.48550/ARXIV.1810.04805
- Johnson AEW, Pollard TJ, Greenbaum NR, et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv. Preprint posted online 2019. doi: 10.48550/arXiv.1901.07042
- De Padua RS, Qureshi I. Colab Notebook with Fine- Tuned T5 Model for Radiology Summarization. Available from: https://colab.research.google.com/ drive/14A3j4bsTiC3hh3GdbLxwWGtwZoFiwciv [Last accessed on 2024 Jun 03].
- Chen Z, Gong Z, Zhuk A. Predicting Doctor’s Impression for Radiology Reports with Abstractive Text Summarization. CS224N: Natural Language Processing with Deep Learning. Stanford University; 2021. Available from: https://web. stanford.edu/class/archive/cs/cs224n/cs224n.1214/reports/ final_reports/report005.pdf [Last accessed on 2024 Jun 03].
- Alsentzer E, Murphy JR, Boag W, et al. Publicly Available Clinical BERT Embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop (ClinicalNLP); 2019:72-78. Available from: https://aclanthology.org/W19- 1909 [Last accessed on 2024 Jun 03].
- Lin CY. ROUGE: A Package for Automatic Evaluation of Summaries. In: Text Summarization Branches Out. Barcelona, Spain. Association for Computational Linguistics; 2004:74-81.
- Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2020;21(1):5485-5551.
- Li Y, Wehbe RM, Ahmad FS, Wang H, Luo Y. Clinical-longformer and clinical-bigbird: Transformers for long clinical sequences. arXiv. Preprint posted online 2022. doi: 10.48550/arXiv.2201.11838
- Yalunin A, Umerenkov D, Kokh V. Abstractive summarization of hospitalisation histories with transformer networks. arXiv. Preprint posted online 2022. doi: 10.48550/arXiv.2204.02208
- Kraljevic Z, Searle T, Shek A, et al. Multi-domain clinical natural language processing with MedCAT: The Medical Concept Annotation Toolkit. Artificial Intelligence in Medicine. 2021;117:102083. doi:10.1016/j.artmed.2021.102083
- Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: Evaluating Text Generation with BERT. arXiv. Preprint posted online 2019. doi: 10.48550/arXiv.1904.09675
- Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2020:7871-7880. doi: 10.18653/v1/2020.acl-main.703
- Lamb A, Goyal A, Zhang Y, Zhang S, Courville A, Bengio Y. Professor forcing: A new algorithm for training recurrent networks. arXiv. Preprint posted online 2016. doi: 10.48550/arXiv.1610.09038
- Wolf T, Debut L, Sanh V, et al. Transformers: State-of-the-Art Natural Language Processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics; 2020:38-45. doi: 10.18653/v1/2020.emnlp-demos.6
- Zaheer M, Guruganesh G, Dubey A, et al. Big bird: Transformers for longer sequences. arXiv. Preprint posted online 2020. doi: 10.48550/arXiv.2007.14062
- Dahal P. Classification and Loss Evaluation - Softmax and Cross Entropy Loss. Available from: https://deepnotes.io/ softmax-crossentropy [Last accessed on 2024 Jun 03].
- Wołk K, Marasek K. Enhanced bilingual evaluation understudy. arXiv. Preprint posted online 2015. doi: 10.48550/arXiv.1509.09088
- Tay Y, Dehghani M, Bahri D, Metzler D. Efficient Transformers: A Survey. arXiv. Preprint posted online 2020. doi: 10.48550/arXiv.2009.06732
- Kaplan J, McCandlish S, Henighan T, et al. Scaling Laws for Neural Language Models. arXiv. Preprint posted online 2020. doi: 10.48550/arXiv.2001.08361
- Vig J. A Multiscale Visualization of Attention in the Transformer Model. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics; 2019:37-42. doi: 10.18653/v1/p19-3007