CoChat
Sign Up
Cited by 496Advanced Causal Inference TechniquesElectronic Health Records SystemsStatistical Methods in Clinical Trials

Real-world data: A brief review of the methods, applications, challenges and opportunities

Open access
Liu, Fang, Panagiotakos, Demosthenes·2022·BMC Medical Research Methodology
DOI 10.1186/s12874-022-01768-6PMC PMC9636688
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The increased adoption of the internet, social media, wearable devices, e-health services, and other technology-driven services in medicine and healthcare has led to the rapid generation of various types of digital data, providing a valuable data source beyond the confines of traditional clinical trials, epidemiological studies, and lab-based experiments. Methods: We provide a brief overview on the type and sources of real-world data and the common models and approaches to utilize and analyze real-world data. We discuss the challenges and opportunities of using real-world data for evidence-based decision making This review does not aim to be comprehensive or cover all aspects of the intriguing topic on RWD (from both the research and practical perspectives) but serves as a primer and provides useful sources for readers who interested in this topic. Results and Conclusions: Real-world hold great potential for generating real-world evidence for designing and conducting confirmatory trials and answering questions that may not be addressed otherwise. The voluminosity and complexity of real-world data also call for development of more appropriate, sophisticated, and innovative data processing and analysis techniques while maintaining scientific rigor in research findings, and attentions to data ethics to harness the power of real-world data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-022-01768-6

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The use of real-world data to investigate drug repurposing signals has become an appealing approach.

    “The use of real-world data to investigate drug repurposing signals has become an appealing approach (Xu et al., 2015;Park, 2021;Liu and Panagiotakos, 2022;Zong et al., 2022)”

    From Drug repurposing for Alzheimer’s disease from 2012–2022—a 10-year literature review (2023) · cites this paper as Liu and Panagiotakos, 2022

    The Introduction says growing access to real-world data has spurred interest in using it for clinical research and discoveries. That supports the claim’s general description of an appealing approach, but the passages in our copy do not identify drug-repurposing signals as an application (searched: “drug repurposing signals,” “repurposing drugs,” and “repurpos”). (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 119 · Cited by 496

Research works printed but not linked: 113

  1. [3]Powell AA Power L Westrop S McOwat K Campbell H Simmons R Real-world data shows increased reactogenicity in adults after heterologous compared to homologous prime-boost COVID-19 vaccination, March- June 2021, England Eurosurveillance. 2021 26 28 2100634 10.2807/1560-7917.ES.2021.26.28.2100634 34269172 PMC8284043 · DOI
  2. [4]Hunter PR, Brainard JS. Estimating the effectiveness of the Pfizer COVID-19 BNT162b2 vaccine after a single dose. A reanalysis of a study of ’real-world’ vaccination outcomes from Israel. medRxiv. 2021.02.01.21250957. 10.1101/2021.02.01.21250957. · DOI
  3. [5]Henry DA Jones MA Stehlik P Glasziou PP Effectiveness of COVID-19 vaccines: findings from real world studies Med J Aust 2021 215 4 149 10.5694/mja2.51182 34296443 PMC8447391 · DOI
  4. [6]Firth JA Hellewell J Klepac P Kissler S Kucharski AJ Spurgin LG Using a real-world network to model localized COVID-19 control strategies Nat Med. 2020 26 10 1616 1622 10.1038/s41591-020-1036-8 32770169 · DOI
  5. [7]Shapiro A Marinsek N Clay I Bradshaw B Ramirez E Min J Characterizing COVID-19 and influenza illnesses in the real world via person-generated health data Patterns. 2021 2 1 100188 10.1016/j.patter.2020.100188 33506230 PMC7815963 · DOI
  6. [8]Ahrens KF Neumann RJ Kollmann B Plichta MM Lieb K Tüscher O Differential impact of COVID-related lockdown on mental health in Germany World Psychiatr. 2021 20 1 140 10.1002/wps.20830 PMC7801843 33432755 · DOI
  7. [9]Hernández MA Stolfo SJ Real-world data is dirty: Data cleansing and the merge/purge problem Data Min Knowl Disc. 1998 2 1 9 37 10.1023/A:1009761603038 · DOI
  8. [10]Corrigan-Curay J Sacks L Woodcock J Real-world evidence and real-world data for evaluating drug safety and effectiveness Jama. 2018 320 9 867 868 10.1001/jama.2018.10136 30105359 · DOI · Paper page
  9. [11]Makady A de Boer A Hillege H Klungel O Goettsch W What is real-world data? A review of definitions based on literature and stakeholder interviews Value Health. 2017 20 7 858 865 10.1016/j.jval.2017.03.008 28712614 · DOI
  10. [13]Miksad RA Abernethy AP Harnessing the power of real-world evidence (RWE): a checklist to ensure regulatory-grade data quality Clin Pharmacol Ther. 2018 103 2 202 205 10.1002/cpt.946 29214638 PMC5814721 · DOI
  11. [14]Curtis MD Griffith SD Tucker M Taylor MD Capra WB Carrigan G Development and validation of a high-quality composite real-world mortality endpoint Health Serv Res. 2018 53 6 4460 4476 10.1111/1475-6773.12872 29756355 PMC6232402 · DOI
  12. [15]Booth CM Karim S Mackillop WJ Real-world data: towards achieving the achievable in cancer care Nat Rev Clin Oncol. 2019 16 5 312 325 10.1038/s41571-019-0167-7 30700859 · DOI
  13. [16]Swift B Jain L White C Chandrasekaran V Bhandari A Hughes DA Innovation at the intersection of clinical trials and real-world data science to advance patient care Clin Transl Sci. 2018 11 5 450 460 10.1111/cts.12559 29768712 PMC6132367 · DOI
  14. [17]Sun W, Cai Z, Li Y, Liu F, Fang S, Wang G. Data processing and text mining technologies on electronic medical records: a review. J Healthc Eng. 2018;2018:4302425. 10.1155/2018/4302425. PMC5911323 29849998 · DOI
  15. [18]Wu J, Roy J, Stewart WF. Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches. Med Care. 2010;48(6 Suppl):S106-13. 10.1097/MLR.0b013e3181de9e17, https://pubmed.ncbi.nlm.nih.gov/20473190/. 20473190 · DOI
  16. [19]Botsis T Hartvigsen G Chen F Weng C Secondary use of EHR: data quality issues and informatics opportunities Summit Transl Bioinforma. 2010 2010 1 PMC3041534 21347133
  17. [20]Kawaler E, Cobian A, Peissig P, Cross D, Yale S, Craven M. Learning to predict post-hospitalization VTE risk from EHR data. In: AMIA annual symposium proceedings. vol. 2012. p. 436. American Medical Informatics Association Country United States. PMC3540493 23304314
  18. [21]Shickel B Tighe PJ Bihorac A Rashidi P Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis IEEE J Biomed Health Inform. 2017 22 5 1589 1604 10.1109/JBHI.2017.2767063 29989977 PMC6043423 · DOI · Paper page
  19. [22]Poirier C Hswen Y Bouzillé G Cuggia M Lavenu A Brownstein JS Influenza forecasting for French regions combining EHR, web and climatic data sources with a machine learning ensemble approach PloS ONE. 2021 16 5 e0250890 10.1371/journal.pone.0250890 34010293 PMC8133501 · DOI
  20. [23]Zheng T Xie W Xu L He X Zhang Y You M A machine learning-based framework to identify type 2 diabetes through electronic health records Int J Med Inform. 2017 97 120 127 10.1016/j.ijmedinf.2016.09.014 27919371 PMC5144921 · DOI
  21. [24]Pivovarov R Perotte AJ Grave E Angiolillo J Wiggins CH Elhadad N Learning probabilistic phenotypes from heterogeneous EHR data J Biomed Inform. 2015 58 156 165 10.1016/j.jbi.2015.10.001 26464024 PMC8025140 · DOI
  22. [25]Zhao D Weng C Combining PubMed knowledge and EHR data to develop a weighted bayesian network for pancreatic cancer prediction J Biomed Informa. 2011 44 5 859 868 10.1016/j.jbi.2011.05.004 PMC3174321 21642013 · DOI
  23. [26]Veturi Y Lucas A Bradford Y Hui D Dudek S Theusch E A unified framework identifies new links between plasma lipids and diseases from electronic medical records across large-scale cohorts Nat Genet. 2021 53 7 972 981 10.1038/s41588-021-00879-y 34140684 PMC8555954 · DOI
  24. [27]Kwon BC Choi MJ Kim JT Choi E Kim YB Kwon S Retainvis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records IEEE Trans Vis Comput Graph. 2018 25 1 299 309 10.1109/TVCG.2018.2865027 30136973 · DOI · Paper page
  25. [28]Mahmoudi E, Kamdar N, Kim N, Gonzales G, Singh K, Waljee AK. Use of electronic medical records in development and validation of risk prediction models of hospital readmission: systematic review. BMJ. 2020;369:m958. PMC7249246 32269037
  26. [29]Desai RJ Wang SV Vaduganathan M Evers T Schneeweiss S Comparison of machine learning methods with traditional models for use of administrative claims with electronic medical records to predict heart failure outcomes JAMA Netw Open. 2020 3 1 e1918962 10.1001/jamanetworkopen.2019.18962 31922560 PMC6991258 · DOI
  27. [31]Bartlett VL Dhruva SS Shah ND Ryan P Ross JS Feasibility of using real-world data to replicate clinical trial evidence JAMA Netw Open. 2019 2 10 e1912869 10.1001/jamanetworkopen.2019.12869 31596493 PMC6802419 · DOI
  28. [32]Dreyer NA Garner S Registries for robust evidence Jama. 2009 302 7 790 791 10.1001/jama.2009.1092 19690313 · DOI
  29. [33]Larsson S Lawyer P Garellick G Lindahl B Lundström M Use of 13 disease registries in 5 countries demonstrates the potential to use outcome data to improve health care’s value Health Affairs. 2012 31 1 220 227 10.1377/hlthaff.2011.0762 22155485 · DOI
  30. [34]McGettigan P Alonso Olmo C Plueschke K Castillon M Nogueras Zondag D Bahri P Patient registries: an underused resource for medicines evaluation Drug Saf. 2019 42 11 1343 1351 10.1007/s40264-019-00848-9 31302896 PMC6834729 · DOI
  31. [35]Izmirly PM Parton H Wang L McCune WJ Lim SS Drenkard C Prevalence of systemic lupus erythematosus in the United States: estimates from a meta-analysis of the Centers for Disease Control and Prevention National Lupus Registries Arthritis Rheumatol. 2021 73 6 991 996 10.1002/art.41632 33474834 PMC8169527 · DOI
  32. [36]Jansen-Van Der Weide MC Gaasterland CM Roes KC Pontes C Vives R Sancho A Rare disease registries: potential applications towards impact on development of new drug treatments Orphanet J Rare Dis 2018 13 1 1 11 30185208 PMC6126025
  33. [37]Lacaze P Millis N Fookes M Zurynski Y Jaffe A Bellgard M Rare disease registries: a call to action Intern Med J. 2017 47 9 1075 1079 10.1111/imj.13528 28891182 · DOI
  34. [38]Gliklich RE, Dreyer NA, Leavy MB, editors. Registries for Evaluating Patient Outcomes: A User's Guide. 3rd ed. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Apr. Report No.: 13(14)-EHC111. PMID: 24945055. 24945055
  35. [39]Svarstad BL Shireman TI Sweeney J Using drug claims data to assess the relationship of medication adherence with hospitalization and costs Psychiatr Serv. 2001 52 6 805 811 10.1176/appi.ps.52.6.805 11376229 · DOI · Paper page
  36. [40]Izurieta HS Wu X Lu Y Chillarige Y Wernecke M Lindaas A Zostavax vaccine effectiveness among US elderly using real-world evidence: Addressing unmeasured confounders by using multiple imputation after linking beneficiary surveys with Medicare claims Pharmacoepidemiol Drug Saf. 2019 28 7 993 1001 10.1002/pds.4801 31168897 · DOI
  37. [41]Allen AM Van Houten HK Sangaralingham LR Talwalkar JA McCoy RG Healthcare cost and utilization in nonalcoholic fatty liver disease: real-world data from a large US claims database Hepatology. 2018 68 6 2230 2238 10.1002/hep.30094 29774589 PMC6240503 · DOI
  38. [42]Sruamsiri R Iwasaki K Tang W Mahlich J Persistence rates and medical costs of biological therapies for psoriasis treatment in Japan: a real-world data study using a claims database BMC Dermatol. 2018 18 1 1 11 10.1186/s12895-018-0074-0 29996929 PMC6042444 · DOI
  39. [43]Quock TP Yan T Chang E Guthrie S Broder MS Epidemiology of AL amyloidosis: a real-world study using US claims data Blood Adv. 2018 2 10 1046 1053 10.1182/bloodadvances.2018016402 29748430 PMC5965052 · DOI
  40. [44]Herland M, Bauder RA, Khoshgoftaar TM. Medical provider specialty predictions for the detection of anomalous medicare insurance claims. In: 2017 IEEE international conference on information reuse and integration (IRI). New York City: IEEE; 2017. p. 579–88.
  41. [45]Momo K Kobayashi H Sugiura Y Yasu T Koinuma M Kuroda SI Prevalence of drug–drug interaction in atrial fibrillation patients based on a large claims data PLoS ONE 2019 14 12 e0225297 10.1371/journal.pone.0225297 31815956 PMC6901225 · DOI
  42. [46]Ghiani M Maywald U Wilke T Heeg B RW1 Bridging The Gap Between Clinical Trials And Real World Data: Evidence On Replicability Of Efficacy Results Using German Claims Data Value Health. 2020 23 S757 S758 10.1016/j.jval.2020.08.2081 35315280 · DOI
  43. [47]Silverman E Skinner J Medicare upcoding and hospital ownership J Health Econ. 2004 23 2 369 389 10.1016/j.jhealeco.2003.09.007 15019762 · DOI
  44. [48]Kirlidog M Asuk C A fraud detection approach with data mining in health insurance Procedia-Soc Behav Sci. 2012 62 989 994 10.1016/j.sbspro.2012.09.168 · DOI
  45. [49]Li J Huang KY Jin J Shi J A survey on statistical methods for health care fraud detection Health Care Manag Sci. 2008 11 3 275 287 10.1007/s10729-007-9045-4 18826005 · DOI
  46. [50]Viaene S Dedene G Derrig RA Auto claim fraud detection using Bayesian learning neural networks Expert Syst Appl. 2005 29 3 653 666 10.1016/j.eswa.2005.04.030 · DOI
  47. [51]Phua C, Lee V, Smith K, Gayler R. A comprehensive survey of data mining-based fraud detection research. arXiv preprint arXiv:1009.6119. 2010. · arXiv
  48. [52]Roche N Small M Broomfield S Higgins V Pollard R Real world COPD: association of morning symptoms with clinical and patient reported outcomes COPD J Chronic Obstructive Pulm Dis. 2013 10 6 679 686 10.3109/15412555.2013.844784 24127914 · DOI
  49. [53]Small M Anderson P Vickers A Kay S Fermer S Importance of inhaler-device satisfaction in asthma treatment: real-world observations of physician-observed compliance and clinical/patient-reported outcomes Adv Ther. 2011 28 3 202 212 10.1007/s12325-010-0108-4 21331556 · DOI
  50. [55]Touma Z, Hoskin B, Atkinson C, Bell D, Massey O, Lofland JH, Berry P, Karyekar CS, Costenbader KH. Systemic lupus erythematosus symptom clusters and their association with Patient‐Reported outcomes and treatment: analysis of Real‐World data. Arthritis Care & Research. 2022;74(7):1079-88. 33345456
  51. [56]Martinez GJ, Mattingly SM, Mirjafari S, Nepal SK, Campbell AT, Dey AK, et al. On the quality of real-world wearable data in a longitudinal study of information workers. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). New York City: IEEE; 2020. p. 1–6.
  52. [57]Christensen JH Saunders GH Porsbo M Pontoppidan NH The everyday acoustic environment and its association with human heart rate: evidence from real-world data logging with hearing aids and wearables Royal Soc Open Sci. 2021 8 2 201345 10.1098/rsos.201345 PMC8074664 33972852 · DOI
  53. [58]Johnson KT Picard RW Advancing neuroscience through wearable devices Neuron. 2020 108 1 8 12 10.1016/j.neuron.2020.09.030 33058768 · DOI
  54. [59]Pickham D Berte N Pihulic M Valdez A Mayer B Desai M Effect of a wearable patient sensor on care delivery for preventing pressure injuries in acutely ill adults: A pragmatic randomized clinical trial (LS-HAPI study) Int J Nurs Stud. 2018 80 12 19 10.1016/j.ijnurstu.2017.12.012 29331656 · DOI
  55. [60]Adams JL Dinesh K Snyder CW Xiong M Tarolli CG Sharma S A real-world study of wearable sensors in Parkinson’s disease NPJ Park Dis. 2021 7 1 1 8 PMC8629990 34845224
  56. [62]Murray EJ, Swanson SA, Hernán MA. Guidelines for estimating causal effects in pragmatic randomized trials. arXiv preprint arXiv:1911.06030. 2019. · arXiv
  57. [63]Hernandez AF, Fleurence RL, Rothman RL. The ADAPTABLE Trial and PCORnet: shining light on a new research paradigm. Ann Intern Med. 2015;163(8):635-6. 26301537
  58. [64]Baigent C. Pragmatic trials-need for ADAPTABLE design. N Engl J Med. 2021;384(21). 33999542
  59. [65]Hernán MA Robins JM Using big data to emulate a target trial when a randomized trial is not available Am J Epidemiol. 2016 183 8 758 764 10.1093/aje/kwv254 26994063 PMC4832051 · DOI · Paper page
  60. [66]Ioannou GN Locke ER O’Hare AM Bohnert AS Boyko EJ Hynes DM COVID-19 vaccination effectiveness against infection or death in a National US Health Care system: a target trial emulation study Ann Intern Med. 2022 175 3 352 361 10.7326/M21-3256 34928700 PMC8697485 · DOI
  61. [67]García-Albéniz X Hsu J Hernán MA The value of explicitly emulating a target trial when using real world evidence: an application to colorectal cancer screening Eur J Epidemiol. 2017 32 6 495 500 10.1007/s10654-017-0287-2 28748498 PMC5759953 · DOI
  62. [68]Takeuchi Y Kumamaru H Hagiwara Y Matsui H Yasunaga H Miyata H Sodium-glucose cotransporter-2 inhibitors and the risk of urinary tract infection among diabetic patients in Japan: Target trial emulation using a nationwide administrative claims database Diabetes Obes Metab. 2021 23 6 1379 1388 10.1111/dom.14353 33606891 · DOI
  63. [69]Jen EY Xu Q Schetter A Przepiorka D Shen YL Roscoe D FDA approval: blinatumomab for patients with B-cell precursor acute lymphoblastic leukemia in morphologic remission with minimal residual disease Clin Cancer Res. 2019 25 2 473 477 10.1158/1078-0432.CCR-18-2337 30254079 · DOI
  64. [70]Gross AM Using real world data to support regulatory approval of drugs in rare diseases: A review of opportunities, limitations & a case example Curr Probl Cancer. 2021 45 4 100769 10.1016/j.currproblcancer.2021.100769 34247834 · DOI
  65. [71]Wu J Wang C Toh S Pisa FE Bauer L Use of real-world evidence in regulatory decisions for rare diseases in the United States—Current status and future directions Pharmacoepidemiol Drug Saf. 2020 29 10 1213 1218 10.1002/pds.4962 32003065 · DOI
  66. [72]Hayeems RZ Michaels-Igbokwe C Venkataramanan V Hartley T Acker M Gillespie M The complexity of diagnosing rare disease: An organizing framework for outcomes research and health economics based on real-world evidence Genet Med. 2022 24 3 694 702 10.1016/j.gim.2021.11.005 34906497 · DOI
  67. [74]Ho M, van der Laan M, Lee H, Chen J, Lee K, Fang Y, et al. The current landscape in biostatistics of real-world data and evidence: Causal inference frameworks for study design and analysis. Stat Biopharm Res. 2021. https://www.tandfonline.com/doi/abs/10.1080/19466315.2021.1883475. · DOI
  68. [75]Crown WH Real-world evidence, causal inference, and machine learning Value Health. 2019 22 5 587 592 10.1016/j.jval.2019.03.001 31104739 · DOI
  69. [76]Cui P, Shen Z, Li S, Yao L, Li Y, Chu Z, et al. Causal inference meets machine learning. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: Association for Computing Machinery; 2020. p. 3527–3528.
  70. [77]Xiong HY, Alipanahi B, Lee LJ, Bretschneider H, Merico D, Yuen RK, Hua Y, Gueroussov S, Najafabadi HS, Hughes TR, Morris Q. The human splicing code reveals new insights into the genetic determinants of disease. Science. 2015;347(6218):1254806. PMC4362528 25525159
  71. [78]Quang D Chen Y Xie X DANN: a deep learning approach for annotating the pathogenicity of genetic variants Bioinformatics. 2015 31 5 761 763 10.1093/bioinformatics/btu703 25338716 PMC4341060 · DOI
  72. [79]Anthimopoulos M Christodoulidis S Ebner L Christe A Mougiakakou S Lung pattern classification for interstitial lung diseases using a deep convolutional neural network IEEE Trans Med Imaging. 2016 35 5 1207 1216 10.1109/TMI.2016.2535865 26955021 · DOI · Paper page
  73. [83]Coccia M Deep learning technology for improving cancer care in society: New directions in cancer imaging driven by artificial intelligence Technol Soc. 2020 60 101198 10.1016/j.techsoc.2019.101198 · DOI
  74. [85]Johansson FD Collins JE Yau V Guan H Kim SC Losina E Predicting response to tocilizumab monotherapy in rheumatoid arthritis: a real-world data analysis using machine learning J Rheumatol. 2021 48 9 1364 1370 10.3899/jrheum.201626 33934070 · DOI
  75. [86]Ravì D Wong C Deligianni F Berthelot M Andreu-Perez J Lo B Deep learning for health informatics IEEE J Biomed Health Informa. 2016 21 1 4 21 10.1109/JBHI.2016.2636665 28055930 · DOI · Paper page
  76. [87]Suzuki K Overview of deep learning in medical imaging Radiol Phys Technol. 2017 10 3 257 273 10.1007/s12194-017-0406-5 28689314 · DOI
  77. [88]Shen D Wu G Suk HI Deep learning in medical image analysis Annu Rev Biomed Eng. 2017 19 221 248 10.1146/annurev-bioeng-071516-044442 28301734 PMC5479722 · DOI · Paper page
  78. [89]Litjens G Kooi T Bejnordi BE Setio AAA Ciompi F Ghafoorian M A survey on deep learning in medical image analysis Med Image Anal. 2017 42 60 88 10.1016/j.media.2017.07.005 28778026 · DOI · Paper page
  79. [90]Lee JG Jun S Cho YW Lee H Kim GB Seo JB Deep learning in medical imaging: general overview Korean J Radiol. 2017 18 4 570 584 10.3348/kjr.2017.18.4.570 28670152 PMC5447633 · DOI
  80. [91]Amyar A Modzelewski R Li H Ruan S Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation Comput Biol Med. 2020 126 104037 10.1016/j.compbiomed.2020.104037 33065387 PMC7543793 · DOI
  81. [92]Oh Y Park S Ye JC Deep learning covid-19 features on cxr using limited training data sets IEEE Trans Med Imaging. 2020 39 8 2688 2700 10.1109/TMI.2020.2993291 32396075 · DOI
  82. [93]Hemdan EED, Shouman MA, Karar ME. Covidx-net: A framework of deep learning classifiers to diagnose covid-19 in x-ray images. arXiv preprint arXiv:2003.11055. 2020. · arXiv
  83. [94]Wang S, Zha Y, Li W, Wu Q, Li X, Niu M, et al. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis. Eur Respir J. 2020;56(2). PMC7243395 32444412
  84. [95]Ardakani AA Kanafi AR Acharya UR Khadem N Mohammadi A Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networks Comput Biol Med. 2020 121 103795 10.1016/j.compbiomed.2020.103795 32568676 PMC7190523 · DOI
  85. [100]Van der Laan MJ, Rose S. Targeted learning: causal inference for observational and experimental data. 2011. Springer-Verlag New York Inc., United States.
  86. [101]Van der Laan MJ, Rose S. Targeted learning in data science. Causal Inference for Complex Longitudinal Studies 2018. Cham: Springer.
  87. [102]van der Laan MJ Luedtke AR Targeted learning of the mean outcome under an optimal dynamic treatment rule J Causal Infer. 2015 3 1 61 95 10.1515/jci-2013-0022 PMC4517487 26236571 · DOI
  88. [104]Chakravarti P Wilson A Krikov S Shao N van der Laan M PIN68 Estimating Effects in Observational Real-World Data, From Target Trials to Targeted Learning: Example of Treating COVID-Hospitalized Patients Value Health. 2021 24 S118 10.1016/j.jval.2021.04.1256 · DOI
  89. [105]Eichler HG Koenig F Arlett P Enzmann H Humphreys A Pétavy F Are novel, nonrandomized analytic methods fit for decision making? The need for prospective, controlled, and transparent validation Clin Pharmacol Ther. 2020 107 4 773 779 10.1002/cpt.1638 31574163 PMC7158212 · DOI
  90. [106]Chakraborty S, Tomsett R, Raghavendra R, Harborne D, Alzantot M, Cerutti F, et al. Interpretability of deep learning models: A survey of results. In: 2017 IEEE smartworld, ubiquitous intelligence & computing, advanced & trusted computed, scalable computing & communications, cloud & big data computing, Internet of people and smart city innovation. New York City: IEEE; 2017. p. 1–6.
  91. [107]Zhang Q, Zhu SC. Visual interpretability for deep learning: a survey. arXiv preprint arXiv:1802.00614. 2018. · arXiv
  92. [108]Hohman F Park H Robinson C Chau DHP Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations IEEE Trans Vis Comput Graph. 2019 26 1 1096 1106 10.1109/TVCG.2019.2934659 31443005 · DOI
  93. [109]Ghoshal B, Tucker A. Estimating uncertainty and interpretability in deep learning for coronavirus (COVID-19) detection. arXiv preprint arXiv:2003.10769. 2020. · arXiv
  94. [110]Raghu M, Gilmer J, Yosinski J, Sohl-Dickstein J. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. 2017; 31st Conference on Neural Information Processing Systems (NIPS 2017). Long Beach: NEURAL INFO PROCESS SYS F, LA JOLLA; 2017. ISBN: 9781510860964.
  95. [112]Barba LA. Terminologies for reproducible research. arXiv preprint arXiv:1802.03311. 2018. · arXiv
  96. [113]Stupple A Singerman D Celi LA The reproducibility crisis in the age of digital medicine NPJ Digit Med. 2019 2 1 1 3 31304352 PMC6550262
  97. [114]Carter RE Attia ZI Lopez-Jimenez F Friedman PA Pragmatic considerations for fostering reproducible research in artificial intelligence NPJ Digit Med. 2019 2 1 1 3 10.1038/s41746-019-0120-2 31304388 PMC6550149 · DOI
  98. [115]Liu C, Gao C, Xia X, Lo D, Grundy J, Yang X. On the replicability and reproducibility of deep learning in software engineering. ACM Transactions on Software Engineering and Methodology. 2021;31(1):1–46.
  99. [116]Springate DA Kontopantelis E Ashcroft DM Olier I Parisi R Chamapiwa E ClinicalCodes: an online clinical codes repository to improve the validity and reproducibility of research using electronic medical records PloS ONE. 2014 9 6 e99825 10.1371/journal.pone.0099825 24941260 PMC4062485 · DOI
  100. [117]Wang SV Schneeweiss S Berger ML Brown J de Vries F Douglas I Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1. 0 Value health 2017 20 8 1009 1022 10.1016/j.jval.2017.08.3018 28964431 · DOI
  101. [118]Panagiotou OA Heller R Inferential challenges for real-world evidence in the era of routinely collected health data: many researchers, many more hypotheses, a single database JAMA Oncol. 2021 7 11 1605 1607 10.1001/jamaoncol.2021.3537 34499102 · DOI
  102. [119]Belbasis L Panagiotou OA Reproducibility of prediction models in health services research BMC Res Notes. 2022 15 1 1 5 10.1186/s13104-022-06082-4 35690767 PMC9188254 · DOI
  103. [120]Dwork C, McSherry F, Nissim K, Smith A. Calibrating noise to sensitivity in private data analysis. In: Theory of cryptography conference. Springer; 2006. p. 265–284.
  104. [121]Konečnỳ J, McMahan B, Ramage D. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575. 2015. · arXiv
  105. [122]Konečnỳ J, McMahan HB, Yu FX, Richtárik P, Suresh AT, Bacon D. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492. 2016. · arXiv
  106. [123]McCradden MD Joshi S Mazwi M Anderson JA Ethical limitations of algorithmic fairness solutions in health care machine learning Lancet Digit Health. 2020 2 5 e221 e223 10.1016/S2589-7500(20)30065-0 33328054 · DOI · Paper page
  107. [124]Mitchell S Potash E Barocas S D’Amour A Lum K Algorithmic fairness: Choices, assumptions, and definitions Ann Rev Stat Appl. 2021 8 141 163 10.1146/annurev-statistics-042720-125902 · DOI · Paper page
  108. [125]Mhasawade V Zhao Y Chunara R Machine learning and algorithmic fairness in public and population health Nat Mach Intell. 2021 3 8 659 666 10.1038/s42256-021-00373-4 · DOI
  109. [126]Wong PH Democratizing algorithmic fairness Philos Technol. 2020 33 2 225 244 10.1007/s13347-019-00355-w · DOI · Paper page
  110. [128]Orsini LS Berger M Crown W Daniel G Eichler HG Goettsch W Improving transparency to build trust in real-world secondary data studies for hypothesis testing—why, what, and how: recommendations and a road map from the real-world evidence transparency initiative Value Health. 2020 23 9 1128 1136 10.1016/j.jval.2020.04.002 32940229 · DOI
  111. [129]Patorno E Schneeweiss S Wang SV Transparency in real-world evidence (RWE) studies to build confidence for decision-making: reporting RWE research in diabetes Diabetes Obes Metab. 2020 22 45 59 10.1111/dom.13918 32250527 PMC7472869 · DOI
  112. [130]White R Building trust in real-world evidence and comparative effectiveness research: the need for transparency Future Med. 2017 6 1 5 7 27759422
  113. [131]Rodriguez-Villa E Torous J Regulating digital health technologies with transparency: the case for dynamic and multi-stakeholder evaluation BMC Med. 2019 17 1 1 5 10.1186/s12916-019-1447-x 31801532 PMC6894205 · DOI

Authors
Liu, F., & Panagiotakos, D.
Affiliations
  • University of Notre Dame (US)
  • Harokopio University of Athens (GR)
Corresponding author
Liu, Fang
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2022-11-05
Volume
22
Issue
1
Pages
287-287
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-022-01768-6
PMC
PMC9636688
OpenAlex
W4308292299
Cited by
496
References
119
Citations by year
2026: 133 · 2025: 183 · 2024: 118 · 2023: 62
FWCI
13.12
Type
journal article
Language
en
Update notices
correction (2023-05-02)
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
real-world data, real-world evidence, digital health data, wearable devices, social media data, e-health, data analysis methods, evidence-based decision making, data ethics, confirmatory trials
MeSH terms
Delivery of Health Care, Humans, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review