Strategies for obtaining unpublished drug trial data: a qualitative interview study
Open accessThe DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Background: Authors of systematic reviews have difficulty obtaining unpublished data for their reviews. This project aimed to provide an in-depth description of the experiences of authors in searching for and gaining access to unpublished data for their systematic reviews, and to give guidance on best practices for identifying, obtaining and using unpublished data. Methods: This is a qualitative study analyzing in-depth interviews with authors of systematic reviews who have published Cochrane reviews or published systematic reviews outside of The Cochrane Library. We included participants who 1) were the first or senior author of a published systematic review of a drug intervention, 2) had expertise in conducting systematic reviews, searching for data, and assessing methodological biases, and 3) were able to participate in an interview in English. We used non-random sampling techniques to identify potential participants. Eighteen Cochrane authors were contacted and 16 agreed to be interviewed (89% response rate). Twenty-four non-Cochrane authors were contacted and 16 were interviewed (67% response rate). Results: Respondents had different understandings of what was meant by unpublished data, including specific outcomes and methodological details. Contacting study authors was the most common method used to obtain unpublished data and the value of regulatory agencies as a data source was underappreciated. Using the data obtained was time consuming and labor intensive. Respondents described the collaboration with other colleagues and/or students required to organize, manage and use the data in their reviews, generally developing and using templates, spreadsheets and computer programs for data extraction and analysis. Respondents had a shared belief that data should be accessible but some had concerns about sharing their own data. Respondents believed that obtaining unpublished data for reviews has important public health implications. There was widespread support for government intervention to ensure open access to trial data. Conclusions: Respondents uniformly agreed that the benefit of identifying unpublished data was worth the effort and was necessary to identify the true harms and benefits of drugs. Recent actions by government, such as increased availability of trial data from the European Medicines Agency, may make it easier to acquire critical drug trial data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Clinical trial registries are a critical source of information for systematic reviewers who use these registries to augment bibliographic database searches when compiling relevant evidence from clinical trials [17–19].”
From A systematic review of the processes used to link clinical trial registrations to their published results (2017) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Handbook of Qualitative Research
Crozier, G., Denzin, N., & Lincoln, Y. (1994) · British Journal of Educational Studies · doi:10.2307/3121684 · Cited by 26,666
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · cited 4×
“The Cochrane Handbook for Systematic Reviews of Interventions [16] suggests identifying unpublished data by contacting experts, pharmaceutical companies, national and international trial registers (for example, clinicaltrials. gov), and other specific trial registers.”
BackgroundInfluentialHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808 · cited 1×
“Our sample size was determined by guidelines from the qualitative methodology literature that suggests an adequate sample size for qualitative research should be 20 to 40 people [17-19].”
Method(2010) · Sociology for Pharmacists · doi:10.4324/9780203381175_chapter_9 · Cited by 14,152
Qualitative Analysis for Social Scientists
Strauss, A. L. (1987) · doi:10.1017/cbo9780511557842 · Cited by 12,277 · cited 1×
“This allowed us to connect the themes to understand better the interrelationships among them [17,22].”
Background(2013) · Cited by 12,072
The SAGE handbook of qualitative research
(2005) · Choice Reviews Online · doi:10.5860/choice.43-1330 · Cited by 10,183
Introduction: The discipline and practice of qualitative research.
Denzin, N. K., & Lincoln, Y. S. (2005) · Cited by 7,744
The Sage Handbook of Qualitative Research[1]
(2006) · Qualitative Research in Organizations and Management: An International Journal · doi:10.1108/17465640610666642 · Cited by 5,842
Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675 · cited 2×
“Legal actions with drug companies have identified many unpublished trials or data [7-9].”
BackgroundSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · cited 3×
“Researchers with experience searching for unpublished drug trial data uniformly agreed that finding unpublished data has important public health consequences in terms of identifying the true harms and benefits of drugs and improving the reliability of systematic reviews [5,6,12,13].”
BackgroundChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · cited 1×
“Background Selective reporting of data from clinical trials, particularly drug trials, has been well documented [1-3].”
BackgroundSelf-citationFull publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997 · cited 1×
“Background Selective reporting of data from clinical trials, particularly drug trials, has been well documented [1-3].”
BackgroundPublication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · cited 1×
“Investigators have also identified unpublished trials and data by searching for trial protocols [10,11].”
BackgroundMelander, H., Ahlqvist-Rastad, J., Meijer, G., & Beermann, B. (2003) · BMJ · doi:10.1136/bmj.326.7400.1171 · Cited by 635 · cited 3×
“Researchers with experience searching for unpublished drug trial data uniformly agreed that finding unpublished data has important public health consequences in terms of identifying the true harms and benefits of drugs and improving the reliability of systematic reviews [5,6,12,13].”
BackgroundOutcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use
Vedula, S. S., Bero, L., Scherer, R. W., & Dickersin, K. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0906126 · Cited by 361
Self-citationNarrative Review: The Promotion of Gabapentin: An Analysis of Internal Industry Documents
Steinman, M. A., Bero, L. A., Chren, M.-M., & Landefeld, C. S. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-4-200608150-00008 · Cited by 336 · cited 1×
“Legal actions with drug companies have identified many unpublished trials or data [7-9].”
BackgroundSelf-citationRising, K., Bacchetti, P., & Bero, L. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050217 · Cited by 328 · cited 1×
“Researchers have found that the full reports of drug studies identified by searching regulatory agency databases have often not been published, or contain far more data than published papers [4-6].”
BackgroundSelf-citationEffect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255 · cited 2×
“Researchers with experience searching for unpublished drug trial data uniformly agreed that finding unpublished data has important public health consequences in terms of identifying the true harms and benefits of drugs and improving the reliability of systematic reviews [5,6,12,13].”
BackgroundSelf-citationBero, L., Oostvogel, F., Bacchetti, P., & Lee, K. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040184 · Cited by 249 · cited 1×
“Other drug agencies should follow suit, as the traditional secrecy is not in the patients’ interest and cannot be defended ethically [25].”
BackgroundSelf-citationComparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243 · cited 1×
“Investigators have also identified unpublished trials and data by searching for trial protocols [10,11].”
BackgroundWhy we need easy access to all data from all clinical trials and how to accomplish it
Gøtzsche, P. C. (2011) · Trials · doi:10.1186/1745-6215-12-249 · Cited by 137 · cited 1×
“Trial data that are maintained in accessible databases or study reports should be available long after authors are out of contact via institutional mechanisms for data sharing [23].”
BackgroundSelf-citationUshering in a New Era of Open Science Through Data Sharing: The Wall Must Come Down
Ross, J. S., & Krumholz, H. M. (2013) · JAMA · doi:10.1001/jama.2013.1299 · Cited by 118
Opening up data at the European Medicines Agency
Gotzsche, P. C., & Jorgensen, A. W. (2011) · BMJ · doi:10.1136/bmj.d2686 · Cited by 106 · cited 2×
“2Nordic Cochrane Centre, Rigshospitalet, Dept 7811, Blegdamsvej 9, DK-2100 Copenhagen, Denmark.”
BackgroundSelf-citationSearching for unpublished trials in Cochrane reviews may not be worth the effort
van Driel, M., Sutter, A. D., Maeseneer, J. D., & Christiaens, T. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.010 · Cited by 100 · cited 1×
“The Cochrane Collaboration is a major producer of rigorous systematic reviews of health care interventions, but only 12% of Cochrane reviews from 2000 to 2006 included unpublished trials [15].”
BackgroundClinical trials and drug promotion: Selective reporting of study 329
Jureidini, J. N., McHenry, L. B., & Mansfield, P. R. (2008) · International Journal of Risk and Safety in Medicine · doi:10.3233/jrs-2008-0426 · Cited by 98 · cited 2×
“Researchers with experience searching for unpublished drug trial data uniformly agreed that finding unpublished data has important public health consequences in terms of identifying the true harms and benefits of drugs and improving the reliability of systematic reviews [5,6,12,13].”
BackgroundAssessment of Recent HIV-1 Infection by a Line Immunoassay for HIV-1/2 Confirmation
Schüpbach, J., Gebhardt, M. D., Tomasik, Z., Niederhauser, C., Yerly, S., Bürgisser, P., Matter, L., Gorgievski, M., Dubs, R. W., Schultze, D., et al. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040343 · Cited by 86
Searching for unpublished data for Cochrane reviews: cross sectional study
Schroll, J. B., Bero, L., & Gotzsche, P. C. (2013) · BMJ · doi:10.1136/bmj.f2231 · Cited by 43 · cited 2×
“In a prior cross-sectional, on-line survey [20], 200 Cochrane authors agreed to be interviewed for this study and 32 met our inclusion criteria.”
BackgroundSelf-citationWorldmapper: The Human Anatomy of a Small Planet
Dorling, D. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040001 · Cited by 41
Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness
Norris, S. L., Holmer, H. K., Ogden, L. A., Fu, R. F., Abou-Setta, A. M., Viswanathan, M., & McPheeters, M. L. (2012) · Europe PMC (PubMed Central) · Cited by 25
Experience of lay member of IFR panel
Hart, T. (2011) · BMJ · doi:10.1136/bmj.d4914 · Cited by 2
Research works printed but not linked: 2
- Lefebvre C, Manheimer E, Glanville J: Chapter 6: Searching for studies. In Cochrane Handbook for Systematic Reviews of Interventions. Edited by Higgins J, Green S. The Cochrane Collaboration; Published by The Cochrane Collaboration and John Wiley and Sons Ltd 2011 England: Chichester
- EMA Access to clinical-trial data and transparency: Workshop report Agency moves towards proactive publication of clinical-trial data 2012 London: European Medicines Agency http://www.ema.europa.eu/docs/en_GB/document_library/Report/2012/12/WC500135841.pdf
Cited by
Ventresca, M., Schünemann, H. J., Macbeth, F. R., Clarke, M. J., Thabane, L., Griffiths, G. O., Noble, S., Garcia, D. A., Marcucci, M., Iorio, A., et al. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00964-6 · Cited by 86
Leeuwen, M. V., Slot, D. E., & van der Weijden, G. A. (2014) · International Journal of Dental Hygiene · doi:10.1111/idh.12069 · Cited by 64
Hodkinson, A., Dietz, K. C., Lefebvre, C., Golder, S., Jones, M., Doshi, P., Heneghan, C., Jefferson, T., Boutron, I., & Stewart, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0766-x · Cited by 49
Bashir, R., Bourgeois, F. T., & Dunn, A. G. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0518-3 · Cited by 47
Isojarvi, J., Wood, H., Lefebvre, C., & Glanville, J. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1294 · Cited by 38
“Two studies have highlighted the value of looking for data on the websites of regulatory agencies, but noted that, for example, the FDA website was not user‐ friendly.(33,34) Detailed assessments of these websites have not been identified, but our own anecdotal experiences of identifying evidence using the FDA website…”
BackgroundMethodInfluential
Metadata
- Authors
- Wolfe, N., Gøtzsche, P. C., & Bero, L.
- Affiliations
- University of California, San Francisco (US)
- Rigshospitalet (DK)
- Corresponding author
- Bero, Lisa
- Venue
- Systematic Reviews
- Published
- 2013-05-16
- Volume
- 2
- Issue
- 1
- Pages
- 31-31
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC3685609
- OpenAlex
- W2170339327
- Cited by
- 30
- References
- 31
- Citations by year
- 2026: 1 · 2024: 5 · 2023: 1 · 2021: 2 · 2020: 3 · 2019: 3 · 2018: 4 · 2017: 4 · 2016: 2 · 2015: 2 · 2014: 3
- FWCI
- 2.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- data access, data extraction, data sharing, European Medicines Agency, drug safety
- MeSH terms
- Attitude, Cooperative Behavior, Data Collection, Pharmaceutical Preparations, Humans, Interviews as Topic, Publishing, Review Literature as Topic, Access to Information, Comprehension, Information Dissemination, Qualitative Research, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
