Using Electronic Data Collection Platforms to Assess Complementary and Integrative Health Patient-Reported Outcomes: Feasibility Project
Open accessPublished version of a preprint
Abstract
Background: The Veteran Administration (VA) Office of Patient-Centered Care and Cultural Transformation is invested in improving veteran health through a whole-person approach while taking advantage of the electronic resources suite available through the VA. Currently, there is no standardized process to collect and integrate electronic patient-reported outcomes (ePROs) of complementary and integrative health (CIH) into clinical care using a web-based survey platform. This quality improvement project enrolled veterans attending CIH appointments within a VA facility and used web-based technologies to collect ePROs. Objective: This study aimed to (1) determine a practical process for collecting ePROs using patient email services and a web-based survey platform and (2) conduct analyses of survey data using repeated measures to estimate the effects of CIH on patient outcomes. Methods: In total, 100 veterans from one VA facility, comprising 11 cohorts, agreed to participate. The VA patient email services (Secure Messaging) were used to manually send links to a 16-item web-based survey stored on a secure web-based survey storage platform (Qualtrics). Each survey included questions about patient outcomes from CIH programs. Each cohort was sent survey links via Secure Messaging (SM) at 6 time points: weeks 1 through 4, week 8, and week 12. Process evaluation interviews were conducted with five primary care providers to assess barriers and facilitators to using the patient-reported outcome survey in usual care. Results: This quality improvement project demonstrated the usability of SM and Qualtrics for ePRO collection. However, SM for ePROs was labor intensive for providers. Descriptive statistics on health competence (2-item Perceived Health Competence Scale), physical and mental health (Patient-Reported Outcomes Measurement Information System Global-10), and stress (4-item Perceived Stress Scale) indicated that scores did not significantly change over time. Survey response rates varied (18/100, 18.0%-42/100, 42.0%) across each of the 12 weekly survey periods. In total, 74 of 100 participants provided ≥1 survey, and 90% (66/74) were female. The majority, 62% (33/53) of participants, who reported the use of any CIH modality, reported the use of two or more unique modalities. Primary care providers highlighted specific challenges with SM and offered solutions regarding staff involvement in survey implementation. Conclusions: This quality improvement project informs our understanding of the processes currently available for using SM and web-based data platforms to collect ePROs. The study results indicate that although it is possible to use SM and web-based survey platforms for ePROs, automating scheduled administration will be necessary to reduce provider burden. The lack of significant change in ePROs may be due to standard measures taking a biomedical approach to wellness. Future work should focus on identifying ideal ePRO processes that would include standardized, whole-person measures of wellness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Qualtrics has demonstrated validity for collection of PRO data within the VA system.
“Qualtrics has demonstrated validity for collection of PRO data within the VA system. 66”
From Remotely Delivered Psychologically Informed Mindful Movement Physical Therapy for Pain Care: A Framework for Operationalization (2023) · cites this paper as [66]
The Abstract (Methods and Results; page unavailable) reports that Qualtrics was used to collect patient-reported outcomes at a VA facility and that the project demonstrated its usability for that purpose. The paper assesses feasibility and usability, not the validity of PRO data collected through Qualtrics, so the validity claim is unsupported. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601 · see citing works
A Global Measure of Perceived Stress
Cohen, S., Kamarck, T., & Mermelstein, R. (1983) · Journal of Health and Social Behavior · doi:10.2307/2136404 · Cited by 32,807 · see citing works
(2014) · Cited by 15,747 · see citing works
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
How we design feasibility studies
Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., Bakken, S., Kaplan, C. P., Squiers, L., Fabrizio, C., et al. (2009) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2009.02.002 · Cited by 3,922 · see citing works
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., Ader, D., Fries, J. F., Bruce, B., & Rose, M. (2007) · Medical Care · doi:10.1097/01.mlr.0000258615.42478.55 · Cited by 3,118 · see citing works
Hays, R. D., Bjorner, J. B., Revicki, D. A., Spritzer, K. L., & Cella, D. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9496-9 · Cited by 1,957 · see citing works
E-mail Survey Response Rates: A Review
Sheehan, K. B. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2001.tb00117.x · Cited by 1,441 · see citing works
Buntin, M. B., Burke, M. F., Hoaglin, M. C., & Blumenthal, D. (2011) · Health Affairs · doi:10.1377/hlthaff.2011.0178 · Cited by 1,311 · see citing works
Comparing response rates in e-mail and paper surveys: A meta-analysis
Shih, T.-H., & Fan, X. (2008) · Educational Research Review · doi:10.1016/j.edurev.2008.01.003 · Cited by 481 · see citing works
Matthew‐Maich, N., Harris, L., Ploeg, J., Markle‐Reid, M., Valaitis, R. K., Ibrahim, S., Gafni, A., & Isaacs, S. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5127 · Cited by 384 · see citing works
The development and validation of the Perceived Health Competence Scale
Smith, M. S. A., Wallston, K. A., & Smith, C. A. (1995) · Health Education Research · doi:10.1093/her/10.1.51 · Cited by 364 · see citing works
From parallel practice to integrative health care: a conceptual framework
Boon, H. S., Verhoef, M. J., O’Hara, D. P., & Findlay, B. (2004) · BMC Health Services Research · doi:10.1186/1472-6963-4-15 · Cited by 326 · see citing works
Validation of the Perceived Stress Scale in a community sample of older adults
Ezzati, A., Jiang, J. M., Katz, M. J., Sliwinski, M. J., Zimmerman, M. E., & Lipton, R. B. (2013) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.4049 · Cited by 252 · see citing works
Barrett, B. P., Marchand, L. R., Scheder, J. C., Plane, M. B., Maberry, R., Appelbaum, D., Rakel, D. P., & Rabago, D. (2003) · The Journal of Alternative and Complementary Medicine · doi:10.1089/107555303771952271 · Cited by 197 · see citing works
Krejci, L., Carter, K., & Gaudet, T. (2014) · Medical Care · doi:10.1097/mlr.0000000000000226 · Cited by 142 · see citing works
Haun, J. N., Lind, J. D., Shimada, S. L., Martin, T. L., Gosline, R. M., Antinori, N., Stewart, M., & Simon, S. R. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2976 · Cited by 139 · see citing works
Self-citationParticipation and factors associated with late or non‐response to an online survey in primary care
Aerny‐Perreten, N., Domínguez‐Berjón, M. F., Esteban-Vasallo, M. D., & García‐Riolobos, C. (2015) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12367 · Cited by 119 · see citing works
Rotondi, A. J., Eack, S. M., Hanusa, B. H., Spring, M. B., & Haas, G. L. (2013) · Schizophrenia Bulletin · doi:10.1093/schbul/sbt194 · Cited by 106 · see citing works
Confirmatory Factor Analysis of the Perceived Stress Scale‐4 in a Community Sample
Ingram, P. B., Clarke, E. W., & Lichtenberg, J. W. (2014) · Stress and Health · doi:10.1002/smi.2592 · Cited by 92 · see citing works
Wintner, L. M., Giesinger, J. M., Zabernigg, A. F., Rumpold, G., Sztankay, M. J., Oberguggenberger, A., Gamper, E., & Holzner, B. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0230-y · Cited by 60 · see citing works
Chumbler, N. R., Haggstrom, D. A., & Saleem, J. J. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181d558f9 · Cited by 52 · see citing works
Using Mobile Health to Support the Chronic Care Model: Developing an Institutional Initiative
Nundy, S., Dick, J. J., Goddu, A. P., Hogan, P. C., Lu, C.-Y. E., Solomon, M. C., Bussie, A., Chin, M. H., & Peek, M. E. (2012) · International Journal of Telemedicine and Applications · doi:10.1155/2012/871925 · Cited by 49 · see citing works
Ritenbaugh, C., Nichter, M., Nichter, M., Kelly, K. L., Sims, C. M., Bell, I. R., Castañeda, H. M., Elder, C. R., Koithan, M. S., Sutherland, E. G., et al. (2011) · BMC Complementary and Alternative Medicine · doi:10.1186/1472-6882-11-135 · Cited by 41 · see citing works
Evans, E., Herman, P. M., Washington, D. L., Lorenz, K., Yuan, A. H., Upchurch, D. M., Marshall, N. J., Hamilton, A. B., & Taylor, S. L. (2018) · Women s Health Issues · doi:10.1016/j.whi.2018.07.003 · Cited by 39 · see citing works
Haun, J. N., Chavez, M., Nazi, K. M., Antinori, N., Melillo, C., Cotner, B. A., Hathaway, W. A., Cook, A., Wilck, N. R., & Noonan, A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8614 · Cited by 35 · see citing works
Self-citationWallwiener, M., Heindl, F., Brucker, S. Y., Taran, F., Hartkopf, A. D., Overkamp, F., Kolberg, H., Hadji, P., Tesch, H., Ettl, J., et al. (2017) · Geburtshilfe und Frauenheilkunde · doi:10.1055/s-0043-116223 · Cited by 30 · see citing works
Using Patient-facing Kiosks to Support Quality Improvement at Mental Health Clinics
Cohen, A. N., Chinman, M., Hamilton, A. B., Whelan, F., & Young, A. S. (2013) · Medical Care · doi:10.1097/mlr.0b013e31827da859 · Cited by 29 · see citing works
Howard, J. S., Toonstra, J., Meade, A. R., Conley, C., & Mattacola, C. G. (2016) · Digital Health · doi:10.1177/2055207616644844 · Cited by 25 · see citing works
Bae, W. K., Kwon, J., Lee, H. W., Lee, S., Song, E., Shim, H., Ryu, K. H., Song, J.-M., Seo, S., Yang, Y., et al. (2018) · Supportive Care in Cancer · doi:10.1007/s00520-018-4232-z · Cited by 22 · see citing works
The Journey to e-Health: VA Healthcare Network Upstate New York (VISN 2)
Nazi, K. M. (2003) · Journal of Medical Systems · doi:10.1023/a:1021005111996 · Cited by 20 · see citing works
Text Messaging to Improve Disease Management in Patients With Painful Diabetic Peripheral Neuropathy
Bauer, V., Goodman, N., Lapin, B. R., Cooley, C., Wang, E., Craig, T. L., Glosner, S. E., Juhn, M. S., Cappelleri, J. C., Sadosky, A., et al. (2018) · The Science of Diabetes Self-Management and Care · doi:10.1177/0145721718767400 · Cited by 15 · see citing works
Miedany, Y. M. E., Gaafary, M. M. E., Aroussy, N. E., Bahlas, S. M., Hegazi, M. O., Palmer, D., & Youssef, S. (2017) · Clinical Rheumatology · doi:10.1007/s10067-017-3675-9 · Cited by 14 · see citing works
Gelhorn, H. L., Skalicky, A. M., Balantac, Z., Eremenco, S. L., Cimms, T., Halling, K., Hollen, P. J., Gralla, R. J., Mahoney, M. C., & Sexton, C. (2018) · Supportive Care in Cancer · doi:10.1007/s00520-018-4061-0 · Cited by 9 · see citing works
Peterson, K. K., Anderson, J., Ferguson, L. D., & Mackey, K. (2011) · Europe PMC (PubMed Central) · Cited by 8 · see citing works
Pinney, S., Otobo, E. O., Freeman, R., Rogers, J., Fasihuddin, F., Ramireddy, K., Deorocki, A., Hassanzadeh, N., Goldstein, M., Zlatopolsky, R., et al. (2017) · Iproceedings · doi:10.2196/iproc.8459 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [36]Veterans Affairs 2019 2019-09-27 National Center for Veterans Analysis and Statistics https://www.va.gov/vetdata/Veteran_Population.asp 31851445
Cited by
Sorted by citations
Vranceanu, A., Bakhshaie, J., Reichman, M., Doorley, J. D., Mace, R. A., Jacobs, C. A., Harris, M. B., Archer, K. R., Ring, D., & Elwy, A. R. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00208-8 · Cited by 48 · see citing works
Self-citationLyu, J., Zhang, H., Wang, H., Liu, X., Jing, Y., Yin, L., & Wang, A. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00654-0 · Cited by 32 · see citing works
Cracchiolo, J. R., Arafat, W., Atreja, A., Bruckner, L. B., Emamekhoo, H., Heinrichs, T., Raldow, A. C., Smerage, J. B., Stetson, P. D., Sugalski, J., et al. (2023) · Cancer · doi:10.1002/cncr.34844 · Cited by 27 · see citing works
Leveraging the Power of Nondisruptive Technologies to Optimize Mental Health Treatment: Case Study
Sadeh‐Sharvit, S., & Hollon, S. D. (2020) · JMIR Mental Health · doi:10.2196/20646 · Cited by 17 · see citing works
Multilingual Framework for Risk Assessment and Symptom Tracking (MRAST)
Šafran, V., Lin, S. M., Nateqi, J., Martin, A. G., Smrke, U., Ariöz, U., Plohl, N., Rojc, M., Bēma, D., Chávez, M., et al. (2024) · Sensors · doi:10.3390/s24041101 · Cited by 13 · see citing works
Bakhshaie, J., Doorley, J. D., Reichman, M., Mace, R. A., Laverty, D. C., Matuszewski, P. E., Elwy, A. R., Fatehi, A., Bowers, L. C., Ly, T., et al. (2022) · Translational Behavioral Medicine · doi:10.1093/tbm/ibac004 · Cited by 13 · see citing works
Self-citationSullivan, M. B., Hill, K., Ballengee, L. A., Knoblach, D., Fowler, C., Haun, J., & Saenger, M. (2023) · Global Advances in Integrative Medicine and Health · doi:10.1177/27536130231209751 · Cited by 10 · see citing works
Self-citationHaun, J. N., Venkatachalam, H. H., Fowler, C. A., Alman, A. C., Ballistrea, L. M., Schneider, T., Benzinger, R. C., Melillo, C., Alexander, N. B., Klanchar, S. A., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/49678 · Cited by 7 · see citing works
Self-citationHaun, J. N., Panaite, V., Cotner, B. A., Melillo, C., Venkatachalam, H. H., Fowler, C. A., Zilka, B., & Messina, W. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08752-2 · Cited by 6 · see citing works
Self-citationBreneman, C. B., Reinhard, M. J., Allen, N. J., Belouali, A., Chun, T. J., Crock, L., Duncan, A. D., & Dutton, M. A. (2023) · Global Advances in Integrative Medicine and Health · doi:10.1177/27536130231171854 · Cited by 6 · see citing works
Rodgers‐Melnick, S. N., Block, S., Rivard, R. L., & Dusek, J. A. (2023) · JMIR Human Factors · doi:10.2196/46528 · Cited by 6 · see citing works
Nakase‐Richardson, R., Haun, J. N., Moore, M., Bogner, J. A., Hoffman, J. M., Dams-O’Connor, K., Coulter, J., Kretzmer, T. S., Martin, A. M., & Silva, M. A. (2026) · JMIR Research Protocols · doi:10.2196/79738 · Cited by 3 · see citing works
Self-citationSchmidt, S. K., Dexheimer, J. W., Zorc, J. J., Palmer, C. A., Casper, T. C., Stukus, K. S., Pickett, M. L., Mollen, C. J., Elsholz, C. L., Cruz, A. T., et al. (2024) · Applied Clinical Informatics · doi:10.1055/a-2480-4628 · Cited by 2 · see citing works
Haun, J. N., Paykel, J. M., Fowler, C. A., Lapcevic, W. A., Panaite, V., Alman, A. C., Melillo, C., Venkatachalam, H. H., & French, D. D. (2022) · Military Medicine · doi:10.1093/milmed/usab567 · Cited by 2 · see citing works
Self-citationBayes, J., Palencia, J. A., & Wardle, J. (2023) · Military Medicine · doi:10.1093/milmed/usad392 · Cited by 2 · see citing works
Haun, J. N., Fowler, C. A., Smith, B. M., Cao, L., Stroupe, K. T., Lapcevic, W. A., Saenger, M. S., Benzinger, R. C., & French, D. D. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-10816-4 · Cited by 2 · see citing works
“Qualtrics has demonstrated usability for ePRO data collection within the VA system [57].”
BackgroundSelf-citationHaun, J. N., Fowler, C. A., French, D. D., McHugh, M., Heuer, J. N., Ballistrea, L. M., Benzinger, R. C., Klanchar, S. A., Sandbrink, F., & Murphy, J. L. (2025) · JMIR Research Protocols · doi:10.2196/72091 · Cited by 1 · see citing works
Self-citationGuo, Y., Hu, D., Zhou, Y., Lyu, T., & Zheng, K. (2025) · medRxiv · doi:10.1101/2025.05.19.25327936 · Cited by 1 · see citing works
Nakase‐Richardson, R., Haun, J. N., Moore, M., Bogner, J. A., Hoffman, J. M., Dams-O’Connor, K., Coulter, J., Kretzmer, T. S., Martin, A. M., & Silva, M. A. (2025) · doi:10.2196/preprints.79738
Self-citationHaun, J. N., Fowler, C. A., Smith, B., Cao, L., Stroupe, K. T., Lapcevic, W. A., Saenger, M. S., Benzinger, R. C., & French, D. D. (2024) · Figshare · doi:10.6084/m9.figshare.26698976
Self-citationEmmanuel, G., Martí, A., Alexandra, S. T., Rosa, R., Cossio‐Gil, Y., & Watson, C. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02864-7
Guo, Y., Hu, D., Zhou, Y., Lyu, T., Yang, Z., Reynolds, T. L., & Zheng, K. (2026) · npj Health Systems · doi:10.1038/s44401-026-00091-2 · Cited by 1 · see citing works
- Authors
- Haun, J. N., Alman, A. C., Melillo, C., Standifer, M., McMahon-Grenz, J., Shin, M., Lapcevic, W. A., Patel, N., & Elwy, A. R.
- Affiliations
- University of South Florida (US)
- James A. Haley Veterans' Hospital (US)
- VA Boston Healthcare System (US)
- United States Department of Veterans Affairs (US)
- Brown University (US)
- Corresponding author
- Haun, Jolie N
- ORCID
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2020-06-26
- Volume
- 8
- Issue
- 6
- Pages
- e15609
- ISSN
- 2291-9694 (ISSN-L)
- PMC
- PMC7381258
- OpenAlex
- W3021779918
- Cited by
- 22
- References
- 36
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 5 · 2023: 6 · 2022: 3 · 2021: 1 · 2020: 1
- FWCI
- 3.68
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Informatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Veterans Affairs
- Office of Research and Development
- Quality Enhancement Research Initiative (#PEC 16-354)
- National Institute on Drug Abuse (R25DA035692)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Complementary and alternative medicine › Complementary and Alternative Medicine Studies
- Keywords
- electronic patient-reported outcomes, complementary and integrative health, veterans, secure messaging, Qualtrics, survey response rates, perceived health competence, perceived stress, physical and mental health, barriers and facilitators, primary care providers, feasibility, holistic care, patient-centered care, Veterans Affairs health care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
