Toward a science of learning systems: a research agenda for the high-functioning Learning Health System
Open accessAbstract
Objective The capability to share data, and harness its potential to generate knowledge rapidly and inform decisions, can have transformative effects that improve health. The infrastructure to achieve this goal at scale—marrying technology, process, and policy—is commonly referred to as the Learning Health System (LHS). Achieving an LHS raises numerous scientific challenges. Materials and methods The National Science Foundation convened an invitational workshop to identify the fundamental scientific and engineering research challenges to achieving a national-scale LHS. The workshop was planned by a 12-member committee and ultimately engaged 45 prominent researchers spanning multiple disciplines over 2 days in Washington, DC on 11–12 April 2013. Results The workshop participants collectively identified 106 research questions organized around four system-level requirements that a high-functioning LHS must satisfy. The workshop participants also identified a new cross-disciplinary integrative science of cyber-social ecosystems that will be required to address these challenges. Conclusions The intellectual merit and potential broad impacts of the innovations that will be driven by investments in an LHS are of great potential significance. The specific research questions that emerged from the workshop, alongside the potential for diverse communities to assemble to address them through a ‘new science of learning systems’, create an important agenda for informatics and related disciplines.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The learning health system model is well-suited for pediatric vaccine trials and implementation.
“The LHS model50 is well-suited for this challenge.”
From SARS-CoV-2 vaccine testing and trials in the pediatric population: biologic, ethical, research, and implementation challenges (2021) · cites this paper as [50]
Coalitions of community stakeholders, caregivers, parents, children, academic health centers, and data managers are core elements of successful learning health systems.
“Coalitions of community stakeholders (e.g., local healthcare agencies and school personnel), caregivers, parents, children, academic health centers, and data managers are the core elements of successful LHS.”
From SARS-CoV-2 vaccine testing and trials in the pediatric population: biologic, ethical, research, and implementation challenges (2021) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The quality of health care delivered to adults in the United States
McGlynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., & Kerr, E. A. (2003) · New England Journal of Medicine · doi:10.1056/nejmsa022615 · Cited by 5,079 · see citing works
Built to last: Successful habits of visionary companies
Collins, J. C., & Porras, J. I. (1996) · Long Range Planning · doi:10.1016/s0024-6301(96)90211-3 · Cited by 2,317 · see citing works
The Path to Personalized Medicine
Hamburg, M. A., & Collins, F. S. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006304 · Cited by 1,970 · see citing works
Eliminating Waste in US Health Care
Hackbarth, A. D. (2012) · JAMA · doi:10.1001/jama.2012.362 · Cited by 1,754 · see citing works
Best care at lower cost: the path to continuously learning health care in America
Smith, M. A., Saunders, R., Stuckhardt, L., & McGinnis, J. M. (2014) · Choice Reviews Online · doi:10.5860/choice.51-3277 · Cited by 1,356 · see citing works
(2007) · Cited by 1,133 · see citing works
Bulger, R. E. (1992) · Kennedy Institute of Ethics journal · doi:10.1353/ken.0.0102 · Cited by 993 · see citing works
The Learning Healthcare System: Workshop Summary
Olsen, L., Aisner, D. L., & McGinnis, J. M. (2013) · Cited by 588 · see citing works
OECD (2013) · Health at a glance · doi:10.1787/health_glance-2013-en · Cited by 520 · see citing works
A Rapid-Learning Health System
Etheredge, L. M. (2007) · Health Affairs · doi:10.1377/hlthaff.26.2.w107 · Cited by 345 · see citing works
Self-citationIntegration and Implementation Sciences: Building a New Specialization
Bammer, G. (2005) · Ecology and Society · doi:10.5751/es-01360-100206 · Cited by 259 · see citing works
National Healthcare Quality Report, 2012
(2013) · PsycEXTRA Dataset · doi:10.1037/e553742013-001 · Cited by 211 · see citing works
Großmann, C., Powers, B. W., & McGinnis, J. M. (2011) · Cited by 163 · see citing works
Tannen, R. L., Weiner, M. G., & Xie, D. (2009) · BMJ · doi:10.1136/bmj.b81 · Cited by 145 · see citing works
Curtis, L. H., Brown, J. S., & Platt, R. G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0121 · Cited by 121 · see citing works
Self-citationSuper crunchers: why thinking-by-numbers is the new way to be smart
(2008) · Choice Reviews Online · doi:10.5860/choice.45-3829 · Cited by 117 · see citing works
Rapid Learning: A Breakthrough Agenda
Etheredge, L. M. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0043 · Cited by 94 · see citing works
Self-citationTransforming Academic Health Centers for an Uncertain Future
Dzau, V. J., Cho, A., ElLaissi, W. F., et al. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1302374 · Cited by 71 · see citing works
Digital Infrastructure for the Learning Health System
Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/12912 · Cited by 64 · see citing works
Code Red and Blue — Safely Limiting Health Care's GDP Footprint
Milstein, A. S. (2012) · New England Journal of Medicine · doi:10.1056/nejmp1211374 · Cited by 37 · see citing works
ROUNDTABLE ON VALUE & SCIENCE-DRIVEN HEALTH CARE
Yong, P. L., Saunders, R. S., & Olsen, L. (2010) · Cited by 29 · see citing works
Where Are The Health Care Entrepreneurs? The Failure of Organizational Innovation in Health Care
Cutler, D. (2010) · National Bureau of Economic Research · doi:10.3386/w16030 · Cited by 17 · see citing works
Forecasting the Use of Electronic Health Records: An Expert Opinion Ap
Blavin, F. E., & Buntin, M. J. B. (2013) · Medicare & Medicaid Research Review · doi:10.5600/mmrr.003.02.a02 · Cited by 12 · see citing works
Self-citationAn ultra-large-scale systems approach to national-scale health information systems
Sullivan, K., Knaus, W. A., & Marks, R. (2010) · doi:10.1145/1882362.1882436 · Cited by 4 · see citing works
Self-citationUltra-Large-Scale Systems: Scale Changes Everything
Northrop, L. (2008) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [3]Tapscott D Williams A Wikinomics: how mass collaboration changes everything, expanded edition New York, NY: The Penguin Group, 2008: 1– 351.
- [24]Norton L Creating infrastructure for big data in the public and private sectors. Presented at ECRI Institute Annual Conference: Data BIG and Small: What Healthcare Decision Makers are Using Now; November 2013 https://www.ecri.org/Video/2013_TA_Conf/4-Session-1-Norton.mov
- [27]Health Affairs. Using big data to transform care. July 2014: Volume 33, Issue 7. http://content.healthaffairs.org/content/33/7.toc
- [38]Summit D at the M. 2012L. H. S. (2012). Core Values Underlying a National-Scale Person-Centered Continuous Learning Health System (LHS) (pp. 1–2). http://www.learninghealth.org/corevalues/
- [46]National Research Council. Toward precision medicine: building a knowledge network for biomedical research and a new taxonomy of disease. Washington, DC: The National Academies Press, 2011. 22536618
Cited by
Sorted by citations
Kuziemsky, C., Maeder, A. J., John, O., Gogia, S. B., Basu, A., Meher, S., & Ito, M. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677897 · Cited by 183 · see citing works
The Cosmos Collaborative: A Vendor-Facilitated Electronic Health Record Data Aggregation Platform
Tarabichi, Y., Frees, A., Honeywell, S., Huang, C., Naidech, A. M., Moore, J. H., & Kaelber, D. C. (2021) · ACI Open · doi:10.1055/s-0041-1731004 · Cited by 139 · see citing works
Hindorff, L. A., Bonham, V. L., & Ohno-Machado, L. (2018) · Personalized Medicine · doi:10.2217/pme-2018-0037 · Cited by 58 · see citing works
Cooper, D. M., Afghani, B., Byington, C. L., Cunningham, C. K., Golub, S., Lu, K. D., Radom-Aizik, S., Ross, L. F., Singh, J., Smoyer, W. E., et al. (2021) · Pediatric Research · doi:10.1038/s41390-021-01402-z · Cited by 55 · see citing works
Future Directions for Pain Management
Mackey, S. (2016) · Hand Clinics · doi:10.1016/j.hcl.2015.08.012 · Cited by 22 · see citing works
Pragmatic Comparative Effectiveness Trials and Learning Health Systems in Pain Medicine
Salmasi, V., Terkawi, A. S., & Mackey, S. C. (2023) · Anesthesiology Clinics · doi:10.1016/j.anclin.2023.03.010 · Cited by 16 · see citing works
Mungia, R., Funkhouser, E., Makhija, S. K., Reyes, S. C., Cohen, R. A., Cochran, D. L., Meyerowitz, C., Rindal, D. B., Gordan, V. V., Fellows, J. L., et al. (2020) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2020.05.190339 · Cited by 11 · see citing works
Oo, C. T. L., Wider, W., Pang, N. T. P., Koh, E. B. Y., Vasanthi, R. K., Thet, K. Z. Z., Ramalho, R., Özdemir, B. N., & Mahboob, K. (2026) · International Journal of Qualitative Studies on Health and Well-being · doi:10.1080/17482631.2026.2621802 · Cited by 5 · see citing works
“These insights align with frameworks in learning health systems and precision public health that leverage data-driven approaches to improve intervention design (Elyoseph et al., 2023; Friedman et al., 2015; Xian, 2024).”
Gilbert, G. H., Allen, G. J., Burton, V. A., Calnon, W. R., Cochran, D. L., Fellows, J. L., Gordan, V. V., Meyerowitz, C., & Rindal, D. B. (2021) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2021.02.008 · Cited by 4 · see citing works
- Authors
- Friedman, C., Rubin, J., Brown, J., Buntin, M., Corn, M., Etheredge, L., Gunter, C., Musen, M., Platt, R., Stead, W., Sullivan, K., & Van Houweling, D.
- Affiliations
- University of Michigan (US)
- Harvard Pilgrim Health Care (US)
- Vanderbilt University (US)
- United States National Library of Medicine (US)
- George Washington University (US)
- University of Illinois Urbana-Champaign (US)
- Stanford University (US)
- University of Virginia (US)
- University of Michigan, Ann Arbor, Michigan, USA
- Corresponding author
- Friedman, Charles
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-10-23
- Volume
- 22
- Issue
- 1
- Pages
- 43-50
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC4433378
- OpenAlex
- W2097974960
- Cited by
- 509
- References
- 25
- Citations by year
- 2026: 56 · 2025: 44 · 2024: 56 · 2023: 40 · 2022: 57 · 2021: 56 · 2020: 45 · 2019: 37 · 2018: 33 · 2017: 27 · 2016: 48 · 2015: 10
- FWCI
- 51.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (1249354)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- learning health systems, research agenda, health data sharing, health data infrastructure, health informatics, research challenges, health decision-making, system-level requirements, interdisciplinary research
- MeSH terms
- Information Systems, United States, Medical Records Systems, Computerized, Computer Security, Delivery of Health Care, Integrated, Databases as Topic, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
