AccScience Publishing / GHES / Online First / DOI: 10.36922/ghes.3680
Submit to GHES
Cite this article
6
Download
289
Views
Journal Browser
Volume | Year
Issue
Search
Forthcoming Issue
Current Issue
View All
News and Announcements
View All
BRIEF REPORT

AI chatbots and cybersecurity: What do they tell the public?

Gloria Wu1* ,  Kayla Draves-Hau2 ,  Milan Del Buono3 ,  Adrial Wong4 ,  Mary Nguyen5 ,  Hasini Namala1
Show Less
1 Department of Ophthalmology, University of California, San Francisco, California, United States of America
2 Department: Molecular, Cell, And Developmental Biology, University of California, Santa Cruz, California, United States of America
3 Department of Bioengineering, University of California, Berkeley, California, United States of America
4 Department of Biological Sciences, University of California, Davis, California, United States of America
5 Department of Biological Sciences, University of California, Irvine, California, United States of America
GHES, 3680 https://doi.org/10.36922/ghes.3680
Submitted: 16 May 2024 | Accepted: 17 July 2024 | Published: 24 October 2024
© 2024 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0/ )
Download PDF
Cite
XML
Abstract

This study evaluates the performance of five large language model chatbots – ChatGPT, Copilot, Gemini, Claude, and Cohere – on topics related to cybersecurity, healthcare, and the environment. The chatbots were evaluated by asking five specific questions, and their responses were analyzed to determine how well they aligned with the cybersecurity principles outlined in the Paris Call. To assess the semantic similarity of their responses, 384-dimensional sentence embeddings from Hugging Face (HuggingFace.com) were used to calculate cosine distances, offering a quantitative measure of their alignment with the Paris Call principles. Repeated measures of analysis of variance (ANOVA) revealed no significant differences in how frequently the chatbots applied the nine Paris Call principles to individual questions, with similar application rates across the chatbots. However, a separate ANOVA across all five questions identified significant differences (p = 0.011) in the average use of these principles, suggesting that the chatbots likely rely on different datasets and did not consistently apply the principles across all questions. The study also found errors of omission, where certain key principles were left out of some responses. For example, several chatbots failed to mention critical elements, e.g. protecting the integrity of supply chains or ensuring accountability in technology use, highlighting gaps in their cybersecurity coverage. As a result, users may need to query multiple chatbots to gain comprehensive insights on these topics.

Keywords
Large language models
Artificial intelligence
Paris call for trust and security in cyberspace
Funding
None.
Conflict of interest
The authors declare no conflicts of interest.
References
  1. Alanazi, A.T., & Alanazi, A. (2023). Clinicians’ perspectives on healthcare cybersecurity and cyber threats. Cureus, 15(10):e47026. https://doi.org/10.7759/cureus.47026

 

  1. AL-Hawamleh, A.M. (2023). Predictions of cybersecurity experts on future cyber-attacks and related cybersecurity measures. International Journal of Advanced Computer Science and Applications, 14(2). https://doi.org/10.14569/IJACSA.2023.0140292

 

  1. Alshahrani, H., Islam, N., Syed, D., Sulaiman, A., Al Reshan, M.S., Rajab, K., et al. (2023). Sustainability in blockchain: A systematic literature review on scalability and power consumption issues. Energies, 16(3):1510. https://doi.org/10.3390/en16031510

 

  1. Arsenault, A.C., Kreps, S.E., Snider, K.L., & Canetti, D. (2024). Cyber scares and prophylactic policies: Cross-national evidence on the effect of cyberattacks on public support for surveillance. Journal of Peace Research, 61(3):413-428. https://doi.org/10.1177/00223433241233960

 

  1. Badrouchi, F., Aymond, A., Haerinia, M., Badrouchi, S., Selvaraj, D.F., Tavakolian, K., et al. (2020). Cybersecurity vulnerabilities in biomedical devices: A hierarchical layered framework. Internet of Things Use Cases for the Healthcare Industry. Cham: Springer, p.157-184. https://doi.org/10.1007/978-3-030-37526-3_7

 

  1. Bell, N., & Mbaziira, A. (2023). Exploring a Diplomatic System of Cooperation in the Cyber Space through a Proposed Cyber Diplomacy Cooperation Framework. In: International Conference on Cybersecurity and Cybercrime. Vol. 10, p.7-11. https://doi.org/10.19107/CYBERCON.2023.01

 

  1. Biswas, S.S. (2023). Role of chat GPT in public health. Annals of Biomedical Engineering, 51(5):868-869. https://doi.org/10.1007/s10439-023-03172-7

 

  1. Cartwright, A.J. (2023). The elephant in the room: Cybersecurity in healthcare. Journal of Clinical Monitoring and Computing, 37(5):1123-1132. https://doi.org/10.1007/s10877-023-01013-5

 

  1. Choudhury, A., & Shamszare, H. (2023). Investigating the impact of user trust on the adoption and use of ChatGPT: Survey analysis. Journal of Medical Internet Research, 25:e47184. https://doi.org/10.2196/47184

 

  1. Chouhan, A.S., Qaseem, M.S., Basheer, Q.M.A., & Mehdia, A. (2023). Blockchain based EHR system architecture and the need of blockchain in healthcare. Materials Today: Proceedings, 80:2064-2070. https://doi.org/10.1016/j.matpr.2021.06.114

 

  1. Dameff, C., Tully, J., Chan, T.C., Castillo, E.M., Savage, S., Maysent, P., et al. (2023). Ransomware attack associated with disruptions at adjacent emergency departments in the US. JAMA Network Open, 6(5):e2312270. https://doi.org/10.1001/jamanetworkopen.2023.12270

 

  1. De Kok, T. (2024). ChatGPT for Textual Analysis? How to Use Generative LLMs in Accounting Research. SSRN. https://doi.org/10.2139/ssrn.4429658

 

  1. Derecho, K.C., Cafino, R., Aquino-Cafino, S.L., Isla, A. Jr., Esencia, J.A., Lactuan, N.J., et al. (2024). Technology adoption of electronic medical records in developing economies: A systematic review on physicians’ perspective. Digital Health, 10:20552076231224605. https://doi.org/10.1177/20552076231224605

 

  1. Dyer, O.(2024). US hospitals face collapse as cyberattack on UnitedHealth cuts revenue streams. BMJ, 384:q686. https://doi.org/10.1136/bmj.q686

 

  1. Eppler, M., Ganjavi, C., Ramacciotti, L.S., Piazza, P., Rodler, S., Checcucci, E., et al. (2024). Awareness and use of ChatGPT and large language models: A prospective cross-sectional global survey in urology. European Urology, 85(2):146-153. https://doi.org/10.1016/j.eururo.2023.10.014

 

  1. Fernando, Y., & Saravannan, R. (2021). Blockchain technology: Energy efficiency and ethical compliance. Journal of Governance and Integrity, 4(2):88-95. https://doi.org/10.15282/jgi.4.2.2021.5872

 

  1. Firat, M. (2023). How Chat GPT can Transform Autodidactic Experiences and Open Education? OSF Preprints. https://doi.org/10.31219/osf.io/9ge8m

 

  1. Fox, S., & Duggan, M. (2013). Health online 2013, p.1-55. Available from: https://www.pewresearch.org/internet/2013/01/15/ health-online-2013 [Last accessed on 2024 Jul 08].

 

  1. George, A.S., & George, A.H. (2023). The emergence of cybersecurity medicine: Protecting implanted devices from cyber threats. Partners Universal Innovative Research Publication, 1(2):93-111. https://doi.org/10.5281/zenodo.10206563

 

  1. Hugging Face. (n.d.). Available from: https://huggingface.co/ sentence-transformers/all-minilm-l6-v2 [Last accessed on 2024 Jul 08].

 

  1. ISO/IEC 27001:2022. ISO. (2022). https://www.iso.org/ standard/27001 [Last accessed on 2024 Jul 08].

 

  1. Jakubek, K. (2022). AMA Adopts New Policy Declaring Climate Change a Public Health Crisis. American Medical Association. Available from: https://www.ama-assn.org/ press-center/press-releases/ama-adopts-new-policy-declaring-climate-change-public-health-crisis [Last accessed on 2024 Jul 08].

 

  1. Javaid, M., Haleem, A., Singh, R.P., & Suman, R. (2023). Towards insighting cybersecurity for healthcare domains: A comprehensive review of recent practices and trends. Cyber Security and Applications, 1:100016. https://doi.org/10.1016/j.csa.2023.100016

 

  1. Kelly, B., Quinn, C., Lawlor, A., Ronan, K., & Burrell, J. (2023) Cybersecurity in healthcare. In: Trends of Artificial Intelligence and Big Data for E-Health. Germany: Springer Nature, p.213-231. https://doi.org/10.1007/978-3-031-11199-0

 

  1. Kormiltsyn, A., Udokwu, C., Dwivedi, V., Norta, A., & Nisar, S. (2023). Privacy-conflict resolution for integrating personal and electronic health records in blockchain-based systems. Blockchain in Healthcare Today, 6(2):6. https://doi.org/10.30953/bhty.v6.276

 

  1. Machal, M.L. (2023). An overview about connected medical devices and their risks. Studies in Health Technology and Informatics, 305:119-122. https://doi.org/10.3233/SHTI230438

 

  1. Minnaar, A., & Herbig, F.J.W. (2021). Cyberattacks and the cybercrime threat of ransomware to hospitals and healthcare services during the COVID-19 pandemic. Acta Criminologica: African Journal of Criminology and Victimology, 34(3). https://hdl.handle.net/10520/ejc-crim_v34_n3_a10

 

  1. Ortiz, A. (2024). AT and T Resets Millions of Passcodes after Customer Records are Leaked. The New York Times. Available from: https://www.nytimes.com/2024/03/30/ business/att-passcodes-reset-data-breach.html [Last accessed on 2024 Jul 08].

 

  1. Pascoe, C., Quinn S., & Scarfone K. (2024). The NIST Cybersecurity Framework (CSF) 2.0. NIST Cybersecurity White Papers (CSWP). https://doi.org/10.6028/NIST.CSWP.29

 

  1. Pesl, R.D., Stötzner, M., Georgievski, I., & Aiello, M., (2023). Uncovering LLMs for Service-composition: Challenges and Opportunities. In: International Conference on Service- Oriented Computing, p.39-48. https://doi.org/10.18419/darus-3767

 

  1. Ratasha, M.A.B., Ismail, M.S.B., Aziz, A.S.A., Williams, G., & Rees, E. (2024). Towards Net Zero-better Decision Making for Enterprise Level Green House gas (GHG) Emission Management and Accounting. In: International Petroleum Technology Conference, D021S052R005. https://doi.org/10.2523/IPTC-23574-MS

 

  1. Reddy, J., Elsayed, N., ElSayed, Z., & Ozer, M. (2023). A review on data breaches in healthcare security systems. International Journal of Computer Applications, 184(45):1-4. https://doi.org/10.5120/ijca2023922333

 

  1. Roumeliotis, K.I., & Tselikas, N.D. (2023). Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6):192. https://doi.org/10.3390/fi15060192

 

  1. Sallam, M, Salim, N.A., Barakat, M., & Al-Tammemi, A. (2023). ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J, 3(1). https://doi.org/10.52225/narra.v3i1.103

 

  1. Shahsavar, Y., & Choudhury, A. (2023). User intentions to use ChatGPT for self-diagnosis and health-related purposes: Cross-sectional survey study. JMIR Human Factors, 10(1):e47564. https://doi.org/10.2196/47564

 

  1. Shen, C., & Wang, Y. (2023). Concerned or apathetic? Exploring online public opinions on climate change from 2008 to 2019: A comparative study between China and other G20 countries. Journal of Environmental Management, 332:117376. https://doi.org/10.1016/j.jenvman.2023.117376

 

  1. Shi, X., Xiao, H., Liu, W., Lackner, K.S., & Stocker, T.F. (2023). Confronting the carbon-footprint challenge of blockchain. Environmental Science and Technology, 57(3):1403-1410. https://doi.org/10.1021/acs.est.2c05165

 

  1. Shull, A., Boysen, A., Holland, B., Hathaway, M., Fay, R., Desai, N., et al. (2019). Governing Cyberspace During a Crisis in Trust. Centre for International Governance Innovation. Available from: https://www.cigionline.org/publications/ governing-cyberspace-during-crisis-trust [Last accessed on 2024 Jul 08].

 

  1. Stergiopoulos, G., Kotzanikolaou, P., Konstantinou, C., & Tsoukalis, A. (2023). Process-aware attacks on medication control of type-I diabetics using infusion pumps. IEEE Systems Journal, 17(2):1831-1842. https://doi.org/10.1109/JSYST.2023.3236690

 

  1. Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) Using ChatGPT for clinical practice and medical education: Cross-sectional survey of medical students and physicians perceptions. JMIR Medical Education. 9(1):e50658. https://doi.org/10.2196/50658

 

  1. Temsah, M.H., Aljamaan, F., Malki, K.H., Alhasan, K., Altamimi, I., Aljarbou, R., et al. (2023). Chatgpt and the future of digital health: A study on healthcare workers perceptions and expectations. Healthcare, 11(13):1812. https://doi.org/10.3390/healthcare11131812

 

  1. Terán, M., & Noelia, H. (2023). Environmental sustainability and ICT: A look at impact factors. LUTPub. Available from: https://lutpub.lut.fi/handle/10024/165468 [Last accessed on 2024 Jul 08].

 

  1. The 9 Principles. (2018) Paris Call. Available from: https:// pariscall.international/en/principles [Last accessed on 2024 Jul 08].

 

  1. Triplett, W.J. (2024). Cybersecurity vulnerabilities in healthcare: A threat to patient security. Cybersecurity and Innovative Technology Journal, 2(1):15-25. https://doi.org/10.53889/citj.v2i1.333

 

  1. Van Boven, L.S., Kusters, R.W., Tin, D., Van Osch, F.H.M., De Cauwer, H., Ketelings, L., et al. (2024). Hacking acute care: A qualitative study on the health care impacts of ransomware attacks against hospitals. Annals of Emergency Medicine, 83(1):46-56. https://doi.org/10.1016/j.annemergmed.2023.04.025

 

  1. Wasserman, L., & Wasserman, Y. (2022). Hospital cybersecurity risks and gaps: Review (for the non-cyber professional). Frontiers of Digital Health, 4:862221. https://doi.org/10.3389/fdgth.2022.862221
Share
Back to top
Global Health Economics and Sustainability, Electronic ISSN: 2972-4570 Published by AccScience Publishing
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here
Accept