AccScience Publishing / JCI / Online First / DOI: 10.36922/JCI025050004
ORIGINAL RESEARCH ARTICLE

A mixed-methods evaluation of a province-wide electronic immunization information system

Marie-Pierre Gagnon1,2* Maxime Sasseville1,2 Aude Motulsky3 Claude Sicotte3 Robyn Tamblyn4,5 David L. Buckeridge5
Show Less
1 Faculty of Nursing Sciences, Université Laval, Quebec City, Quebec, Canada
2 VITAM-Research Center in Sustainable Health, Quebec City, Quebec, Canada
3 Department of Health Management, Evaluation and Policy, School of Public Health, Université de Montréal, Montreal, Quebec, Canada
4 Department of Medicine, Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada
5 Department of Epidemiology, Biostatistics and Occupational Health, Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada
Received: 28 January 2025 | Revised: 9 April 2025 | Accepted: 24 April 2025 | Published online: 7 May 2025
© 2025 by the Author(s). This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution -Noncommercial 4.0 International License (CC-by the license) ( https://creativecommons.org/licenses/by-nc/4.0/ )
Abstract

An electronic immunization information system (IIS) should support the administration of vaccinations in primary care, enable follow-up for immunization schedules, and facilitate efforts to increase vaccine coverage. This study aimed to evaluate the benefits associated with the implementation of a province-wide IIS in Quebec, Canada, using the Benefits Evaluation Framework. A mixed-methods approach method was employed, including individual interviews, a survey, and a time-and-motion study. Interviews were conducted with 20 key informants identified in collaboration with the Quebec Ministry of Health and Social Services (French acronym: MSSS). Following the implementation of IIS, the MSSS distributed an email invitation to all potential system users across Quebec to participate in the survey. In addition, a time-and-motion study was conducted with IIS users at eight community health centers. Most survey participants (n = 651) reported positive perceptions regarding service quality, system usability, and the completeness of information provided by the IIS. The system was found to improve user productivity, and users reported a high level of satisfaction. Satisfaction was associated with information, service, and system quality, while productivity was further associated with both satisfaction and frequency of weekly system use. However, the results did not support a significant association between user satisfaction and overall system use. The time-and-motion study revealed no significant differences between sites in the mean duration required to perform various system tasks. Interviews also highlighted the existence of barriers to successful IIS implementation, such as resistance to change and insufficient resources. Despite limited supporting resources, the implementation of the provincial IIS in Quebec was generally successful. User acceptance emerged as a critical factor for the widespread adoption of future health information systems.

Keywords
Information system
Immunization
Mixed methods
Benefits adoption framework
Evaluation
Funding
This study was funded by Canada Health Infoway (contract # 2444-002).
Conflict of interest
The authors declare that they have no competing interests.
References
  1. Muscoplat MH, Rajamani S. Immunization information system and informatics to promote immunizations: Perspective from Minnesota immunization information connection. Biomed Inform Insights. 2017;9:1-8. doi: 10.1177/1178222616688893

 

  1. Stockwell MS, Natarajan K, Ramakrishnan R, et al. Immunization data exchange with electronic health records. Pediatrics. 2016;137(6):e20154335. doi: 10.1542/peds.2015-4335

 

  1. Gianfredi V, Moretti M, Lopalco PL. Countering vaccine hesitancy through immunization information systems, a narrative review. Hum Vaccin Immunother. 2019;15:2508-2526. doi: 10.1080/21645515.2019.1599675

 

  1. Wangia V. Testing an extended theoretical framework to explain variance in use of a public health information system. Online J Public Health Inform. 2013;4(3):e61037. doi: 10.5210/ojphi.v4i3.4238

 

  1. Kempe A, Hurley LP, Cardemil CV, et al. Use of immunization information systems in primary care. Am J Prev Med. 2017;52(2):173-182. doi: 10.1016/j.amepre.2016.07.029

 

  1. Heymann DL, Shindo N. COVID-19: What is next for public health? Lancet. 2020;395(10224):542-545. doi: 10.1016/s0140-6736(20)30374-3

 

  1. Castillo EM, Chan TC, Tolia VM, et al. Effect of a computerized alert on emergency department hepatitis a vaccination in homeless patients during a large regional outbreak. J Emerge Med. 2018;55:764-768. doi: 10.1016/j.jemermed.2018.09.004

 

  1. Rosewell A, Makita L, Muscatello D, et al. Health information system strengthening and malaria elimination in Papua New Guinea. Malar J. 2017;16:278. doi: 10.1186/s12936-017-1910-0

 

  1. Oza S, Jazayeri D, Teich JM, et al. Development and deployment of the Openmrs-Ebola electronic health record system for an Ebola treatment center in sierra Leone. J Med Internet Res. 2017;19(8):e294. doi: 10.2196/jmir.7881

 

  1. Carlton CA, Potter RC, Vranesich PA, Swanson RG. Assessing the ability of an immunization information system to prevent overimmunization of pediatric influenza vaccines. J Public Health Manag Pract. 2015;21(3):282-287. doi: 10.1097/phh.0000000000000122

 

  1. Freeman VA, DeFriese GH. The challenge and potential of childhood immunization registries. Annu Rev Public Health. 2003;24:227-246. doi: 10.1146/annurev.publhealth.24.100901.140831

 

  1. CDC. Immunization Information Systems Annual Report; 2019. Available from: AQ4 https://www.cdc.gov/vaccines/ programs/iis/annual-report-iisar/index.html [Last accessed on 2025 Apr 26].

 

  1. WHO. European Vaccine Action Plan 2015-2020. Regional Office for Europe; 2014. Available from: AQ4 https://iris.who.int/ handle/10665/340400 [Last accessed on 2025 Apr 26].

 

  1. Heidebrecht CL, Foisy J, Pereira JA, et al. Perceptions of immunization information systems for collecting pandemic H1N1 immunization data within Canada’s public health community: A qualitative study. BMC Public Health. 2010;10:523. doi: 10.1186/1471-2458-10-523

 

  1. Gagnon MP, Lampron A, Buyl R. Implementation and adoption of an electronic information system for vaccine inventory management. In: 49th Hawaii International Conference on System Sciences. Maui, HA: HICSS; 2016. doi: 10.1109/HICSS.2016.398

 

  1. Facciola A, Visalli G, Orlando A, et al. Vaccine hesitancy: An overview on parents’ opinions about vaccination and possible reasons of vaccine refusal. J Public Health Res. 2019:8(1):1436. doi: 10.4081/jphr.2019.1436

 

  1. Saville AW, Albright K, Nowels C, et al. Getting under the hood: Exploring issues that affect provider-based recall using an immunization information system. Acad Pediatr. 2011;11(1):44-49. doi: 10.1016/j.acap.2010.12.009

 

  1. Lau F, Hagens S, Muttitt S. A proposed benefits evaluation framework for health information systems in Canada. Healthc Q. 2017;10(1):112-116,118.

 

  1. DeLone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manage Inform Syst. 2003;19(4):9-30. doi: 10.1080/07421222.2003.11045748

 

  1. Creswell JW, Plano-Clark VL. Designing and Conducting Mixed Methods Research. United States: Sage Publications; 2017.

 

  1. Brinkmann M. Ultimate Stopwatch and Timer for Android. Ghacks Net; 2013. Available from: https://www.ghacks. net/2013/01/04/ultimate-stopwatch-and-timer-for-android [Last accessed on 2025 Apr 26].

 

  1. Nunnally JC. Psychometric Theory. United States: McGraw- Hill; 1978.

 

  1. Alali H, Salim J. Virtual communities of practice success model to support knowledge sharing behaviour in healthcare sector. Procedia Technol. 2013;11:176-183. doi: 10.1016/j.protcy.2013.12.178

 

  1. Palm JM, Colombet I, Sicotte C, Degoulet P. Determinants of user satisfaction with a clinical information system. AMIA Annu Symp Proc. 2006;2006:614-618.

 

  1. Williams KS, Shah GH, Leider JP, Gupta A. Overcoming barriers to experience benefits: A qualitative analysis of electronic health records and health information exchange implementation in local health departments. EGEMS (Wash DC). 2017;5(1):18. doi: 10.5334/egems.216

 

  1. McGreevy S, Murray M, Montero L, et al. Assessing the immunization information system and electronic health record interface accuracy for COVID-19 vaccinations. JAMIA open. 2023;6(2):ooad026. doi: 10.1093/jamiaopen/ooad026

 

Share
Back to top
Journal of Clinical Informatics, Published by AccScience Publishing