Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
Objective Mobile health (mHealth) systems are becoming more common for chronic disease management, but usability studies are still needed on patients’ perspectives and mHealth interaction performance. This deficiency is addressed by our quantitative usability study of a mHealth diabetes system evaluating patients’ task performance, satisfaction, and the relationship of these measures to user characteristics. Materials and Methods We used metrics in the International Organization for Standardization (ISO) 9241-11 standard. After standardized training, 10 patients performed representative tasks and were assessed on individual task success, errors, efficiency (time on task), satisfaction (System Usability Scale [SUS]) and user characteristics. Results Tasks of exporting and correcting values proved the most difficult, had the most errors, the lowest task success rates, and consumed the longest times on task. The average SUS satisfaction score was 80.5, indicating good but not excellent system usability. Data trends showed males were more successful in task completion, and younger participants had higher performance scores. Educational level did not influence performance, but a more recent diabetes diagnosis did. Patients with more experience in information technology (IT) also had higher performance rates. Discussion Difficult task performance indicated areas for redesign. Our methods can assist others in identifying areas in need of improvement. Data about user background and IT skills also showed how user characteristics influence performance and can provide future considerations for targeted mHealth designs. Conclusion Using the ISO 9241-11 usability standard, the SUS instrument for satisfaction and measuring user characteristics provided objective measures of patients’ experienced usability. These could serve as an exemplar for standardized, quantitative methods for usability studies on mHealth systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We used standardized methods for usability studies of mobile health systems to assess effectiveness, efficiency, and satisfaction from the International Organization for Standardization 9241-11.30,42”
From A chatbot for hypertension self-management support: user-centered design, development, and usability testing (2023) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1993) · Cited by 9,879
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961
Nielsen, J. (1994) · Cited by 1,897
(2013) · Journal of Usability Studies archive · Cited by 1,357
The Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190 · cited 3×
“The sample size of 10 subjects is appropriate for usability testing and is greater than sample sizes recommended by Nielsen and Landauer(25), and Virzi.(26) However, these users may not be representative of all mHealth users of diabetes systems.”
BackgroundMethodInfluential
Research works printed but not linked: 5
- International Telecommunications Union, 2014, The World in 2014: ICT Facts and Figures
- CTIA -The Wireless Association, 2014, Wireless Industry Survey 2013
- Nielsen, 2000, Why You Only Need to Test with 5 Users
- TechSmith Corporation, 2014, Morae Usability Testing and Market Research Software
- IBM Corporation, 2013, IBM SPSS Statistics for Windows
Cited by
Telehealth and patient satisfaction: a systematic review and narrative analysis
Kruse, C. S., Krowski, N., Rodriguez, B., Tran, L., Vela, J., & Brooks, M. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016242 · Cited by 1,350
Lattie, E. G., Adkins, E. C., Winquist, N., Stiles-Shields, C., Wafford, Q. E., & Graham, A. K. (2019) · Journal of Medical Internet Research · doi:10.2196/12869 · Cited by 843
“Furthermore, in recent years, increased focus has been on assessing the user experience (including the usability and acceptability) of such interventions to identify and rectify user experience failings that could limit one’s ability and desire to continue to use a program [23-25].”
O’Connor, S., Hanlon, P., O’Donnell, C. A., Garcia, S., Glanville, J., & Mair, F. S. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0359-3 · Cited by 636
Usability of Commercially Available Mobile Applications for Diverse Patients
Sarkar, U., Gourley, G. I., Lyles, C. R., Tieu, L., Clarity, C., Newmark, L., Singh, K., & Bates, D. W. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3771-6 · Cited by 257
Schnall, R., Cho, H., & Liu, J. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.8851 · Cited by 239
“At the same time, a standardized usability instrument would improve methodological consistency, making it possible to begin comparing findings across mHealth app evaluations [18].”
BackgroundGronier, G., & Baudet, A. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1898828 · Cited by 237
Patients' perspectives on digital health tools
Madanian, S., Nakarada-Kordic, I., Reay, S., & Chetty, T. (2023) · PEC Innovation · doi:10.1016/j.pecinn.2023.100171 · Cited by 233
“Many studies recommend a participatory approach in the design of future digital health technologies [14,21,22,25,26].”
BackgroundFu, H., McMahon, S. K., Gross, C. R., Adam, T. J., & Wyman, J. F. (2017) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2017.06.016 · Cited by 173
Assessing usability of eHealth technology: A comparison of usability benchmarking instruments
Broekhuis, M., van Velsen, L., & Hermens, H. J. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.05.001 · Cited by 137
The Iterative Convergent Design for Mobile Health Usability Testing: Mixed Methods Approach
Alwashmi, M. F., Hawboldt, J., Davis, E., & Fetters, M. D. (2019) · JMIR mhealth and uhealth · doi:10.2196/11656 · Cited by 131
“Many contemporary mHealth studies have been successful in combining both ISO and SUS to assess usability [37-39].”
Anderberg, P., Eivazzadeh, S., & Berglund, J. S. (2019) · Journal of Medical Internet Research · doi:10.2196/13951 · Cited by 75
Contreras-Somoza, L. M., Irazoki, E., Toribio-Guzmán, J. M., Díez, I. D. L. T., Diaz-Baquero, A. A., Parra−Vidales, E., Bartolomé, M. V. P., & Franco, M. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.636116 · Cited by 89
Pinar, U., Anract, J., Perrot, O., Tabourin, T., Chartier-Kastler, E. J., Parra, J., Vaessen, C., TAILLE, A. D. L., & Rouprêt, M. (2020) · World Journal of Urology · doi:10.1007/s00345-020-03432-4 · Cited by 86
Sanger, P. C., Hartzler, A., Lordon, R. J., Armstrong, C. A., Lober, W. B., Evans, H. L., & Pratt, W. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv183 · Cited by 68
Zanaboni, P., Ngangue, P., Mbemba, G. I. C., Schopf, T. R., Bergmo, T. S., & Gagnon, M.-P. (2018) · Journal of Medical Internet Research · doi:10.2196/10202 · Cited by 79
“Patients seek information from the internet to learn more about their symptoms, diagnoses, and treatments and use a range of digital health interventions to manage their illness at home and support independent living and self-care [11].”
BackgroundUsability of a Novel Mobile Health iPad App by Vulnerable Populations
Miller Jr, D. P., Weaver, K. E., Case, L. D., Babcock, D., Lawler, D., Denizard-Thompson, N., Pignone, M. P., & Spangler, J. G. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.7268 · Cited by 63
“In contrast to our mPATH program, many of these apps require users to have advanced literacy, numeracy, and computer skills [17,33-36].”
Yan, A., Zou, Y., & Mirchandani, D. A. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa064 · Cited by 73
Georgsson, M., Staggers, N., Årsand, E., & Kushniruk, A. W. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103110 · Cited by 70
Self-citationTele-nursing in patients with chronic illness: a systematic review
Navarrete, B. A., & Ambrosio, L. (2019) · Anales del Sistema Sanitario de Navarra · doi:10.23938/assn.0645 · Cited by 67
Sim, R., & Lee, S. W. H. (2021) · Patient Preference and Adherence · doi:10.2147/ppa.s271449 · Cited by 51
Martínez, W., Threatt, A. L., Rosenbloom, S. T., Wallston, K. A., Hickson, G. B., & Elasy, T. A. (2018) · JMIR Human Factors · doi:10.2196/humanfactors.9569 · Cited by 62
“Several prior studies have reported the design and usability of patient-facing health apps and Web-based interventions for patients with diabetes [50,53-58].”
Broekhuis, M., van Velsen, L., Peute, L., Halim, M., & Hermens, H. J. (2021) · JMIR Formative Research · doi:10.2196/18198 · Cited by 61
“Formative usability tests use qualitative methods, think aloud protocols [1,2], interviews [3], cognitive walkthrough [4], heuristic evaluation [5], or quantitative methods, such as user task performance [6].”
MethodJeddi, F. R., Nabovati, E., Bigham, R., & Khajouei, R. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104026 · Cited by 58
Wu, D. T. Y., Vennemeyer, S., Brown, K. M., Revalee, J., Murdock, P., Salomone, S., France, A., Clarke‐Myers, K., & Hanke, S. P. (2019) · Applied Clinical Informatics · doi:10.1055/s-0039-1698466 · Cited by 54
Larbi, D., Randine, P., Årsand, E., Antypas, K., Bradway, M., & Gabarron, E. (2020) · Journal of Medical Internet Research · doi:10.2196/18480 · Cited by 52
“[29-34, 36-39 36-39, 41,46-49,52,”
Interactive systems for patient-centered care to enhance patient engagement
Tang, C., Lorenzi, N., Harle, C. A., Zhou, X., & Chen, Y. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv198 · Cited by 45
Ali, S., Romero, J., Morrison, K., Hafeez, B., & Ancker, J. S. (2018) · Applied Clinical Informatics · doi:10.1055/s-0038-1632396 · Cited by 49
Griffin, A. C., Khairat, S., Bailey, S. C., & Chung, A. E. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad073 · Cited by 46
Gimbel, R. W., Rennert, L., Crawford, P. F., Little, J. R., Truong, K. D., Williams, J. E., Griffin, S. F., Shi, L., Chen, L., Zhang, L., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17968 · Cited by 46
“Researchers have evaluated usability by applying definitions and evaluation metrics guided by the International Organization for Standardization’s 9241-11 usability framework and mHealth usability research [44].”
MethodJannati, N., Nakhaee, N., Yazdi‐Feyzabadi, V., & Tjondronegoro, D. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104474 · Cited by 43
Moghaddasi, H., Rabiei, R., Asadi, F., & Ostvan, N. (2017) · Healthcare Informatics Research · doi:10.4258/hir.2017.23.2.101 · Cited by 40
“Georgsson and Staggers [18], in their usability evaluation of a system for diabetes, used measures from the International Organization for Standardization (ISO 9241-11), specifically the IsoMetric usability questionnaire.”
MethodFarrahi, R., Jeddi, F. R., Nabovati, E., Jabali, M. S., & Khajouei, R. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0893-x · Cited by 39
Dang, S., Karanam, C., Gómez-Orozco, C., & Gómez-Marín, O. (2017) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2016.0224 · Cited by 39
“We also did not use standardized tools for usability evaluation.(20) Future studies should use monitoring systems that are compatible across multiple devices and use participants’ existing mobile phones, thus making such interventions more scalable.”
Administering Cognitive Tests Through Touch Screen Tablet Devices: Potential Issues
Jenkins, A. E., Lindsay, S., Eslambolchilar, P., Thornton, I. M., & Tales, A. (2016) · Journal of Alzheimer s Disease · doi:10.3233/jad-160545 · Cited by 38
“Mobile healthcare technology (mHealth) has been applied to many different healthcare challenges to help individuals living with chronic conditions such as diabetes [3].”
Effect of Prior Health Knowledge on the Usability of Two Home Medical Devices: Usability Study
Chaniaud, N., Métayer, N., Mégalakaki, O., & Loup‐Escande, É. (2020) · JMIR mhealth and uhealth · doi:10.2196/17983 · Cited by 33
“In addition, age [3,25,29,32] and experience in IT [24,33] impact usability.”
InfluentialFu, H. N., Adam, T. J., Konstan, J. A., Wolfson, J. A., Clancy, T. R., & Wyman, J. F. (2019) · JMIR Diabetes · doi:10.2196/11462 · Cited by 32
“Patient characteristics are hypothesized to influence user experience, but there is limited evidence on how demographic and clinical characteristics affect app usability [9-11].”
Reusable Software Usability Specifications for mHealth Applications
Zapata, B. C., Fernández‐Alemán, J. L., Toval, A., & Idri, A. (2018) · Journal of Medical Systems · doi:10.1007/s10916-018-0902-0 · Cited by 32
“Using a standardized approach also facilitates comparing results across apps [24].”
ResultSingh, G., MacGillivray, M. K., Mills, P. B., Adams, J. R., Sawatzky, B., & Mortenson, W. B. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1487-y · Cited by 32
Ding, H., Fatehi, F., Russell, A. W., Karunanithi, M., Menon, A., Bird, D., & Gray, L. (2017) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2017.0190 · Cited by 31
“The study is limited to a small sample size, although other reports with a sample size of 10 participants 39 and even smaller sizes of 4–5 participants were considered sufficient to reflect 80% of usability issues.”
Fu, H. N. C., Rizvi, R. F., Wyman, J. F., & Adam, T. J. (2020) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000596 · Cited by 31
“Users did not like apps that required multiple steps and too much time to use.(11) Patient ratings on diabetes app usability range from poor to good; some users liked multiple functions, and others did not.”
BackgroundmHealth for diabetes support: a systematic review of apps available on the Italian market
Rossi, M. G., & Bigi, S. (2017) · mHealth · doi:10.21037/mhealth.2017.04.06 · Cited by 25
Khairat, S., Chourasia, P. K., Muellers, K., Andreadis, K., Lin, J. J., & Ancker, J. S. (2023) · Telemedicine Reports · doi:10.1089/tmr.2023.0002 · Cited by 30
Usability testing of an mHealth device for swallowing therapy in head and neck cancer survivors
Constantinescu, G., Kuffel, K., King, B., Hodgetts, W. E., & Rieger, J. (2018) · Health Informatics Journal · doi:10.1177/1460458218766574 · Cited by 30
A User-Centered Design for Redesigning E-Government Website in Public Health Sector
Puspitasari, I., Cahyani, D. I., & Taufik, T. (2018) · International Seminar on Application for Technology of Information and Communication (iSemantic) · doi:10.1109/isemantic.2018.8549726 · Cited by 30
“In mobile health applications, implementing adequate usability on its development is crucial to ensure the correct exchange of health messages [4].”
BackgroundKružíková, A., Mužík, M., Knapová, L., Dědková, L., Šmahel, D., & Matyáš, V. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103667 · Cited by 29
Rajan, J. V., Moura, J., Gourley, G., Kiso, K., Sizilio, A., Cortez, A. M., Riley, L. W., Veras, M. A., & Sarkar, U. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0385-1 · Cited by 28
“We focused on two primary usability outcomes: data entry accuracy and end-user satisfaction [24, 25].”
BackgroundBangash, H., Makkawy, A., Gundelach, J. H., Miller, A. A., Jacobson, K. A., & Kullo, I. J. (2021) · JMIR Human Factors · doi:10.2196/32568 · Cited by 28
“The sample sizes for genetic counselors and patients with familial hypercholesterolemia were modest; however, the numbers were in line with those of previous usability studies [22-25] and allowed for thematic saturation to be reached.”
BackgroundMethodResultChaniaud, N., Mégalakaki, O., Capo, S., & Loup‐Escande, É. (2021) · JMIR Human Factors · doi:10.2196/24846 · Cited by 27
“Several researchers have recently investigated the relationship between user characteristics and the usability of connected devices in health care [11,18-22].”
InfluentialPerformance of a web-based application measuring spot quality in dried blood spot sampling
Veenhof, H., Koster, R. A., Brinkman, R., Senturk, E., Bakker, S. J. L., Berger, S. P., Akkerman, O. W., Touw, D. J., & Alffenaar, J. C. (2019) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2019-0437 · Cited by 26
Ruyobeza, B., Grobbelaar, S. S. S., & Botha, A. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02033-z · Cited by 25
“Of the 30 articles reviewed, 22 articles [58], [20], [36], [37], [39], [40], [41], [42], [46], [52], [53], [54], [55], [56], [57], [59], [61], [62], [65], [66], [67] and [68] discussed interventions which targeted one, single disease.”
BackgroundGiunti, G., Rivera-Romero, O., Kool, J., Bansi, J., Sevillano, J. L., Granja-Domínguez, A., Izquierdo‐Ayuso, G., & Giunta, D. H. (2020) · JMIR Research Protocols · doi:10.2196/18196 · Cited by 25
“The System Usability Scale provides a “quick and dirty,” reliable tool for measuring a product’s usability [56].”
MethodCao, H., Zhang, Z., Evans, R. D., Dai, W., Bi, Q., Zhu, Z., Xu, J., & Shen, L. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3066564 · Cited by 24
“Obviously, the SUS could be further used to measure users’ satisfaction [23], [57].”
BackgroundMethodBylappa, B. K., Kamath, D. Y., Josephine, I. S., Shaikh, J. U., Kamath, A., Rioniz, P., Kulkarni, S., Varghese, K., & Xavier, D. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-056962 · Cited by 24
Asthma self-management app for Indonesian asthmatics: A patient-centered design
Ghozali, M., Satibi, S., Ikawati, Z., & Lazuardi, L. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106392 · Cited by 22
Marien, S., Legrand, D., Ramdoyal, R., Nsenga, J., Ospina, G., Ramon, V., Boland, B. B., & Spinewine, A. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy107 · Cited by 22
Safiee, L., Rough, D. J., George, P., & Mudenha, R. (2023) · JMIR Human Factors · doi:10.2196/51691 · Cited by 21
Zhang, L., Babu, S. V., Jindal, M., Williams, J. E., & Gimbel, R. W. (2019) · JMIR Research Protocols · doi:10.2196/13502 · Cited by 21
“Standardized, systematic mHealth usability assessments need to be performed, but even these have been insufficient in the past [21].”
Rahayu, S. R., Zainafree, I., Merzistya, A. N. A., Cahyati, W. H., Farida, E., Wandastuti, A. D., Isbandi, Wahidah, N., Saefurrohim, M. Z., Islam, M. A. N., et al. (2022) · Healthcare Informatics Research · doi:10.4258/hir.2022.28.4.297 · Cited by 20
“47), indicating good but not excellent usability [26].”
BackgroundEvaluation and recommendations for effective data visualization for seizure forecasting algorithms
Chiang, S., Moss, R., Black, A. P., Jackson, M. C., Moss, C., Bidwell, J., Meisel, C., & Loddenkemper, T. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab009 · Cited by 20
van der Nat, D. J., Huiskes, V. J. B., Taks, M., Pouls, B. P. H., van den Bemt, B. J. F., & van Onzenoort, H. A. W. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-07967-7 · Cited by 19
“Based on literature [15, 18, 27], patient, clinical, hospital, and ICT-related factors were collected (Table 1).”
BackgroundMartinez, V. I., Marquard, J. L., Saver, B. G., Garber, L., & Preusse, P. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2016.1278340 · Cited by 19
“…designers to ensure system usability (Wozney et al., 2016) of an intervention, it will require extensive usability testing (Alana et al., 2016; Georgsson & Staggers, 2016; Lyles, Sarkar, & Osborn, 2014; Nelson, Bethune, Lagotte, & Osborn, 2016; Peute, Spithoven, Bakker, & Jaspers, 2008), and…”
Rasmussen, C. D. N., Svendsen, M. J., Wood, K., Nicholl, B. I., Mair, F. S., Sandal, L. F., Mork, P. J., Søgaard, K., Bach, K., & Stochkendahl, M. J. (2020) · JMIR Research Protocols · doi:10.2196/20308 · Cited by 18
“With the increased integration of digital technologies in our daily lives, digital health interventions such as smartphone apps have been suggested as promising platforms for supporting self-management [3,4], and they are increasingly being used to help people manage their chronic conditions [5].”
BackgroundSimpleAR: Augmented Reality high-level content design framework using visual programming
Apaza, Y. M. O., Valderrama, A. P., & Corrales-Delgado, C. (2019) · International Conference of the Chilean Computer Science Society (SCCC) · doi:10.1109/sccc49216.2019.8966427 · Cited by 18
“widely used to evaluate web, desktop, and mobile software [12].”
MethodDeveloping and validating a new scale to measure the acceptability of health apps among adolescents
Chen, E. S., Moracco, K. E., Kainz, K., Muessig, K. E., & Tate, D. F. (2022) · Digital Health · doi:10.1177/20552076211067660 · Cited by 19
“Developed in the 1990s for desktop computer systems,(42) the SUS has been used by mHealth researchers to collect data for health apps and other mHealth interventions.(43,44) The SUS consists of 10 items and includes items such as “I thought the system was easy to use” and “I found the system very cumbersome to use.”
MethodEvaluation of the quality of health applications using the System Usability Scale
Afrizal, S., Prihatni, R., & Hastono, S. P. (2024) · Jurnal Manajemen Informasi Kesehatan (Health Information Management) · doi:10.51851/jmis.v9i1.492 · Cited by 18
Raeesi, A., Abbasi, R., & Khajouei, R. (2021) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462321000052 · Cited by 17
“Efficiency means the expenditure of a certain amount of effort and resources to fully and properly achieve a goal and includes items such as time, costs, and ease of performing tasks (15;16).”
BackgroundAdjei, T. K., Mohammed, A., Acheampong, P. R., Acquah-Gyan, E., Sylverken, A. A., Twumasi-Ankrah, S., Owusu, M., & Owusu‐Dabo, E. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248363 · Cited by 17
“While studies have revealed that the use of mHealth systems have the potential to improve the health system and disease outcomes [6], not much has been done in determining factors of mIVR use among its users [7].”
BackgroundResultChin, W. S. Y., Kurowski, A., Gore, R. J., Chen, G., & Punnett, L. (2021) · JMIR Formative Research · doi:10.2196/20739 · Cited by 17
“In one study focusing on a diabetes mHealth system, younger users performed app tasks faster and had fewer errors [11].”
BackgroundInfluentialGimbel, R. W., Shi, L., Williams, J. E., Dye, C. J., Chen, L., Crawford, P. F., Shry, E. A., Griffin, S. F., Jones, K. O., Sherrill, W. W., et al. (2017) · JMIR Research Protocols · doi:10.2196/resprot.6993 · Cited by 17
“Research staff will evaluate usability by applying definitions and usability evaluation metrics guided by the International Organization for Standardization’s 9241-11 usability framework and mHealth usability research [65].”
The My Diabetes Care Patient Portal Intervention: Usability and Pre-Post Assessment
Martínez, W., Hackstadt, A. J., Hickson, G. B., KNOERL, T., Rosenbloom, S. T., Wallston, K. A., & Elasy, T. A. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1730324 · Cited by 16
Telemedicine in cystic fibrosis
Desimone, M. E., Sherwood, J. S., Soltman, S., & Moran, A. (2021) · Journal of Clinical & Translational Endocrinology · doi:10.1016/j.jcte.2021.100270 · Cited by 16
Digital Diabetes Communication: There’s an App for That
Klonoff, D. C., & Kerr, D. E. (2016) · Journal of Diabetes Science and Technology · doi:10.1177/1932296816660210 · Cited by 16
Usability of a Disease Management Mobile Application as Perceived by Patients With Diabetes
Janatkhah, R., Tabari-Khomeiran, R., Louyeh, A. A., & Kazemnejad, E. (2019) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000532 · Cited by 15
“Hence, evaluating the usability of health-related mobile apps is advised prior to any realworld implementation.(23,30,34,35) This study examines the usability of a mobile app that has been designed to improve self-care and disease management abilities among patients with diabetes mellitus (DM) in Iran.”
BackgroundThe application of Usability Testing Method for Evaluating the New Student Acceptance (NSA) System
Uska, M. Z., Wirasasmita, R. H., & Fahrurrozi, M. (2020) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1539/1/012028 · Cited by 15
“Meanwhile, the questioner used to see user satisfaction with the software or product that has been applied using the System Usability Scale (SUS) [18].”
MethodUsability Evaluation for Mobile Health Application: Systematic Literature Review
Septiani, W., Rahmawati, N., Safitri, D. M., & Luis, M. (2024) · SINERGI · doi:10.22441/sinergi.2024.2.009 · Cited by 14
“…app [42] Perform usability evaluation [43] Rate the usability of the BEDside Mobility app [45] Evaluating PresRx OCR for user interface acceptance [46] Testing the usability of the m-Health. diabetes system [12] Evaluating usability issues [8] Describe the development and usability of mobile…”
BackgroundMethodGiunti, G., Yrttiaho, T., Guardado, S., Sachinopoulou, A., Mylonopoulou, V., Fält, J., Paloniemi, B., Ryytty, M., Krüger, J., & Isomursu, M. (2025) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2025.106379 · Cited by 14
“The System Usability Scale (SUS) was used for measuring usability (Georgsson and Staggers, 2016).”
Sabater, F. R. L., & Garcías, A. P. (2020) · Pixel-Bit Revista de Medios y Educación · doi:10.12795/pixelbit.76299 · Cited by 14
Evaluasi dan Perancangan Ulang Tampilan Antarmuka Aplikasi SehatQ Terhadap Lansia
Konda, V. W. D., Restyandito, R., & Nugraha, K. A. (2022) · AITI · doi:10.24246/aiti.v19i2.228-247 · Cited by 14
“Pada tahapan ini ditemukan referensi mengenai penelitian dengan 10 orang responden merupakan jumlah yang tepat untuk usability testing [9].”
BackgroundFontecha, J., Gónzalez, I., & Bravo, J. (2019) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-019-01369-0 · Cited by 13
Need Assessment for Smartphone-Based Cardiac Telerehabilitation
Kim, J. S., Yun, D., Kim, H. J., Ryu, H. Y., Oh, J., & Kang, S.-M. (2018) · Healthcare Informatics Research · doi:10.4258/hir.2018.24.4.283 · Cited by 12
“One of the difficulties in applying mHealth is the lack of scientific evidence of efficiency [23].”
BackgroundCaregiver development of activity-supporting services for smart homes
Belloum, R., Yaddaden, A., Lussier, M., Bier, N., & Consel, C. (2021) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-210616 · Cited by 12
Wulandari, R., Pangarsa, E. A., Andono, P. N., Rachmani, E., Sidiq, M., Setyowati, M., & Waluyo, D. E. (2022) · Asian Pacific Journal of Cancer Prevention · doi:10.31557/apjcp.2022.23.12.4169 · Cited by 12
User Experience Research and Usability of Health Information Technology
Campbell, J. L. (2023) · Auerbach Publications eBooks · doi:10.1201/9781003460886 · Cited by 13
Agyemang, E., Adu‐Gyamfi, A. B., Achampong, E. K., & Esia-Donkoh, K. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11883-3 · Cited by 11
Feroz, I., & Ahmad, N. (2024) · Blockchain in Healthcare Today · doi:10.30953/bhty.v7.357 · Cited by 11
Semjonova, G., Davidoviča, A., Kozlovskis, N., Okss, A., & Kataševs, A. (2022) · Sensors · doi:10.3390/s22134779 · Cited by 11
“This 10-item Likert-scale tool is usually given to the user immediately after the interaction with the system, after the intervention, allowing users to record their initial feelings and responses to the system [31].”
BackgroundMethodInfluentialWeirauch, V., Soehnchen, C., Burmann, A., & Meister, S. (2024) · Journal of Medical Internet Research · doi:10.2196/55714 · Cited by 11
“For example, in some evaluations, usability was defined and therefore measured by underlying indicators like effectiveness, efficiency, and satisfaction [46-49] or others like acceptance and feasibility [50,51].”
BackgroundKrahe, M. A., Conway, M., Howells, S. R., Roffey, K., & Reilly, S. (2021) · Internet Journal of Allied Health Sciences and Practice · doi:10.46743/1540-580x/2021.2041 · Cited by 11
Content Analysis: First-Time Patient User Challenges with Top-Rated Commercial Diabetes Apps
Fu, H. N. C., Jin, D., & Adam, T. J. (2021) · Telemedicine and e-Health · doi:10.1089/tmj.2020.0128 · Cited by 10
Kanter, R., Boza, S., & Acuña-Salazar, P. (2023) · Frontiers in Sustainable Food Systems · doi:10.3389/fsufs.2023.1244759 · Cited by 10
Vannelli, S., Visintin, F., & Gitto, S. (2025) · Journal of Medical Internet Research · doi:10.2196/60694 · Cited by 10
Waki, K., Tsurutani, Y., Waki, H., Enomoto, S., Kashiwabara, K., Fujiwara, A., Orime, K., Kinguchi, S., Yamauchi, T., Hirawa, N., et al. (2024) · JMIR Research Protocols · doi:10.2196/53514 · Cited by 10
“In contrast, results may differ with a young population [67], as it has been reported that young populations are able to complete tasks more successfully on mHealth apps [68] and have high satisfaction compared with their older counterparts [69].”
ResultIkwue, U., Eyo-Udo, N. L., Onunka, O., Ekwezia, A. V., Nwankwo, E. E., & Daraojimba, C. (2023) · Agriculture Extension in Developing Countries · doi:10.26480/aedc.02.2023.78.88 · Cited by 10
A Behavioral Strategy to Nudge Young Adults to Adopt In-Person Counseling: Gamification
Piao, S., & Joo, J. (2022) · Behavioral Sciences · doi:10.3390/bs12020040 · Cited by 10
“The SUS score measured satisfaction because it allows comparison across environments and systems and can objectively measure satisfaction [77].”
BackgroundResultThe efficiency and effectiveness of surgery information systems in Iran
Abbasi, F., Khajouei, R., & Mirzaee, M. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01236-5 · Cited by 9
Pulantara, I. W., Wang, Y., Burke, L. E., Sereika, S. M., Bizhanova, Z., Kariuki, J. K., Cheng, J. S.-C., Beatrice, B., Loar, I., Cedillo, M., et al. (2024) · JMIR mhealth and uhealth · doi:10.2196/50043 · Cited by 8
“Typical mHealth systems [25-31] consist of a smartphone app for data collection and intervention delivery, sensors integration, and occasionally clinical or administrative portals.”
BackgroundNatkunam, T., Tristani, L., Peers, D., Fraser-Thomas, J. L., Latimer‐Cheung, A. E., & Bassett-Gunter, R. L. (2020) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12775 · Cited by 8
VisExPreS: A Visual Interactive Toolkit for User-Driven Evaluations of Embeddings
Ghosh, A., Nashaat, M., Miller, J., & Quader, S. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3039106 · Cited by 8
Bavaresco, C. S., Bragança, S., Fries, F. D. P., Sória, G. S., de Moura, F. R. R., Rivaldo, E. G., D’Ávila, O. P., Umpierre, R. N., Harzheim, E., & Rodrigues, J. A. (2020) · Iranian Journal of Public Health · doi:10.18502/ijph.v48i8.2982 · Cited by 8
“However, despite the fact that some studies previously published in the literature, report high rates of student satisfaction with DL courses in the field of dentistry (7-10), many studies do not discuss user assessment as a necessary outcome.”
ResultStrybel, D., & Kumar, A. R. (2020) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2020.103059 · Cited by 8
Metadata
- Authors
- Georgsson, M., & Staggers, N.
- Affiliations
- University of Utah (US)
- University of Maryland, Baltimore (US)
- Corresponding author
- Georgsson, Mattias
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-09-16
- Volume
- 23
- Issue
- 1
- Pages
- 5-11
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC4713903
- OpenAlex
- W2177887065
- Semantic Scholar
- 424879d126d8ebf5423079e3e55d8c21114440e5
- Cited by
- 200
- References
- 43
- Citations by year
- 2026: 11 · 2025: 12 · 2024: 22 · 2023: 16 · 2022: 23 · 2021: 31 · 2020: 29 · 2019: 21 · 2018: 15 · 2017: 12 · 2016: 8
- FWCI
- 21.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- mobile health, diabetes management, usability evaluation, ISO 9241-11, System Usability Scale, user satisfaction, user characteristics, chronic disease management, time on task, task errors, computer experience
- MeSH terms
- Adult, Age Factors, Aged, Consumer Behavior, Diabetes Mellitus, Efficiency, Female, Humans, Male, Middle Aged, Self Care, User-Computer Interface, Patient Satisfaction, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.