Usability Evaluation of Computerized Maintenance Management System Replacement in British Columbia Biomedical Engineering

Abstract

Introduction: British Columbia Biomedical Engineering (BCBME) is comprised of four distinct Biomedical Engineering organizations that provide healthcare technology management services for seven regional health authorities across British Columbia. BCBME shares a provincial Computerized Maintenance Management System (CMMS) that has been implemented for approximately 10 years. In 2023, BCBME initiated the search for a replacement CMMS, aiming to leverage modern features such as mobile CMMS application access and increased integration with external applications and systems.
This report presents the findings and recommendations of the Usability Working Group within the BCBME CMMS Replacement Project. The working group evaluated three CMMS options over a two-month period to assess their usability using a human factors (HF) approach. The usability evaluation aimed to inform CMMS selection from a usability perspective, increase staff engagement throughout the procurement process, and proactively identify and address usability concerns prior to implementation.

Methodology: Heuristics evaluation [1] and user walkthroughs were employed to identify usability issues and gather end user feedback. These methods allowed the working group to proactively identify causes of inef-ficiency, frustration, and data quality issues (e.g., missing, inaccurate and/or incomplete data) while ensuring input and buy-in from end users. Heuristics evaluation was completed by three HF experts with clinical engi-neering backgrounds while 18 users from all seven regional health authorities participated in user walkthroughs. Participants varied between front-line technologists, supervisors, and managers or clinical engineers.

Results: Key findings include the identification of usability issues across all three CMMS options categorized by severity and violation theme. Strengths and weaknesses of each system were identified based on user feed-back and heuristic analysis. For instance, Vendor B emerged as the most preferred option among participants and had the fewest number of usability issues; however, some users found it difficult to navigate due to the display of large amounts of unorganized data. Vendor A was noted for its familiarity but had issues with limited searching capabilities and unclear error messages. Vendor C faced criticism for being unintuitive and difficult to navigate.

Discussion: Analysis is still in progress; however, the intention of this work is to guide the BCBME CMMS Replacement Project in selecting a new CMMS. The selection process will consider not only traditional criteria such as finance and utility but also consider usability as a critical factor in decision making. Once a CMMS option is selected, findings from this usability evaluation will inform implementation by proactively identifying and mitigating issues discovered prior to system go live. Overall, this structured HF approach provides valuable insights into how the recommended and selected CMMS can support BCBME’s goals of enhancing operational efficiency, user satisfaction, and data quality throughout the healthcare technology management lifecycle.