CoChat
Sign Up
Cited by 14Academic Publishing and Open AccessResearch Data Management PracticesMeta-analysis and systematic reviews

Assessing Open Science practices in physical activity behaviour change intervention evaluations

Open access
Norris, Emma, Sulevani, Isra, Finnerty, Ailbhe N, Castro, Oscar·2022·BMJ Open Sport & Exercise Medicine
DOI 10.1136/bmjsem-2021-001282PMC PMC9174779

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Objectives: Concerns on the lack of reproducibility and transparency in science have led to a range of research practice reforms, broadly referred to as ‘Open Science’. The extent that physical activity interventions are embedding Open Science practices is currently unknown. In this study, we randomly sampled 100 reports of recent physical activity randomised controlled trial behaviour change interventions to estimate the prevalence of Open Science practices. Methods: One hundred reports of randomised controlled trial physical activity behaviour change interventions published between 2018 and 2021 were identified, as used within the Human Behaviour-Change Project. Open Science practices were coded in identified reports, including: study pre-registration, protocol sharing, data, materials and analysis scripts sharing, replication of a previous study, open access publication, funding sources and conflict of interest statements. Coding was performed by two independent researchers, with inter-rater reliability calculated using Krippendorff’s alpha. Results: 78 of the 100 reports provided details of study pre-registration and 41% provided evidence of a published protocol. 4% provided accessible open data, 8% provided open materials and 1% provided open analysis scripts. 73% of reports were published as open access and no studies were described as replication attempts. 93% of reports declared their sources of funding and 88% provided conflicts of interest statements. A Krippendorff’s alpha of 0.73 was obtained across all coding. Conclusion: Open data, materials, analysis and replication attempts are currently rare in physical activity behaviour change intervention reports, whereas funding source and conflict of interest declarations are common. Future physical activity research should increase the reproducibility of their methods and results by incorporating more Open Science practices.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjsem-2021-001282

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A higher frequency (78%) than observed in our study was evidenced among interventional studies on physical activity behaviour change.

    “In fact, a higher frequency (78%) than the one observed in our study was evidenced among interventional studies on physical activity behaviour change [33].”

    From Do randomised clinical trials on dental caries adopt Open Science practices? (2024) · cites this paper as [33]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. [1]Munafò MR, Nosek BA, Bishop DVM, et al. A manifesto for reproducible science. Nat Hum Behav 2017; 1: 1– 9. 10.1038/s41562-016-0021 PMC7610724 33954258 · DOI · Paper page
  2. [2]Head ML, Holman L, Lanfear R, et al. The extent and consequences of p-hacking in science. PLoS Biol 2015; 13: e1002106. 10.1371/journal.pbio.1002106 25768323 PMC4359000 · DOI · Paper page
  3. [3]Simmons JP, Nelson LD, Simonsohn U. False-Positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011; 22: 1359– 66. 10.1177/0956797611417632 22006061 · DOI · Paper page
  4. [4]Murphy KR, Aguinis H. HARKing: how badly can Cherry-Picking and question Trolling produce bias in published results? J Bus Psychol 2019; 34: 1– 17. 10.1007/s10869-017-9524-7 · DOI
  5. [5]Munafò MR, Hollands GJ, Marteau TM. Open science prevents mindless science. BMJ 2018; 363: k4309. 10.1136/bmj.k4309 30322848 PMC6193470 · DOI
  6. [6]Nosek BA, Spies JR, Motyl M. Scientific utopia: II. restructuring incentives and practices to promote truth over Publishability. Perspect Psychol Sci 2012; 7: 615– 31. 10.1177/1745691612459058 26168121 PMC10540222 · DOI
  7. [8]Kathawalla U-K, Silverstein P, Syed M. Easing into open science: a guide for graduate students and their advisors. Collabra Psychol 2021; 7. 10.1525/collabra.18684 · DOI
  8. [9]Norris E, O'Connor DB. Science as behaviour: using a behaviour change approach to increase uptake of open science. Psychol Health 2019; 34: 1397– 406. 10.1080/08870446.2019.1679373 31661325 · DOI
  9. [10]Field SM, Wagenmakers E-J, Kiers HAL, et al. The effect of preregistration on trust in empirical research findings: results of a registered report. R Soc Open Sci 2020; 7: 181351. 10.1098/rsos.181351 32431853 PMC7211853 · DOI · Paper page
  10. [11]Yamada Y. How to crack PRE-REGISTRATION: toward transparent and open science. Front Psychol 2018; 9: 1831. 10.3389/fpsyg.2018.01831 30319516 PMC6168681 · DOI
  11. [12]Sullivan I, DeHaven A, Mellor D. Open and reproducible research on open science framework. Curr Protoc Essent Lab Tech 2019; 18: e32. 10.1002/cpet.32 · DOI
  12. [13]Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015; 349: p. 7647: g7647. 10.1136/bmj.g7647 25555855 · DOI · Paper page
  13. [14]Bakker M, Veldkamp CLS, van Assen MALM, et al. Ensuring the quality and specificity of preregistrations. PLoS Biol 2020; 18: e3000937. 10.1371/journal.pbio.3000937 33296358 PMC7725296 · DOI · Paper page
  14. [17]Caldwell AR, Vigotsky AD, Tenan MS, et al. Moving sport and exercise science forward: a call for the adoption of more transparent research practices. Sports Med 2020; 50: 449– 59. 10.1007/s40279-019-01227-1 32020542 · DOI
  15. [18]Halperin I, Vigotsky AD, Foster C, et al. Strengthening the practice of exercise and Sport-Science research. Int J Sports Physiol Perform 2018; 13: 127– 34. 10.1123/ijspp.2017-0322 28787228 · DOI
  16. [19]Büttner F, Toomey E, McClean S, et al. Are questionable research practices facilitating new discoveries in sport and exercise medicine? the proportion of supported hypotheses is implausibly high. Br J Sports Med 2020; 54: 1365– 71. 10.1136/bjsports-2019-101863 32699001 · DOI
  17. [20]Hardwicke TE, Wallach JD, Kidwell MC, et al. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). R Soc Open Sci 2020; 7: 190806. 10.1098/rsos.190806 32257301 PMC7062098 · DOI · Paper page
  18. [21]Hardwicke TE, Thibault RT, Kosie JE. Estimating the prevalence of transparency and Reproducibility-Related research practices in psychology (2014–2017). Perspect Psychol Sci p 2021. 10.1177/1745691620979806 PMC8785283 33682488 · DOI · Paper page
  19. [22]Norris E, He Y, Loh R. Assessing markers of reproducibility and transparency in smoking behaviour change intervention evaluations. J Smok Cessat 2021; 2021: e6694386: 12. 10.1155/2021/6694386 34306236 PMC8279208 · DOI
  20. [24]Rowhani-Farid A, Barnett AG. Badges for sharing data and code at biostatistics: an observational study. F1000Res 2018; 7: 90. 10.12688/f1000research.13477.1 29862016 PMC5843843 · DOI
  21. [25]Norris E, Castro O. Assessing open science practices in physical activity behaviour change intervention evaluations. medRxiv 2021. 10.1101/2021.12.01.21267126 PMC9174779 35722044 · DOI
  22. [26]Michie S, Thomas J, Johnston M, et al. The human Behaviour-Change project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implement Sci 2017; 12: 121. 10.1186/s13012-017-0641-5 29047393 PMC5648456 · DOI
  23. [27]Michie S, Thomas J, Mac Aonghusa P, et al. The human Behaviour-Change project: an artificial intelligence system to answer questions about changing behaviour. Wellcome Open Res 2020; 5: 122. 10.12688/wellcomeopenres.15900.1 32566761 PMC7287511 · DOI
  24. [30]Loder E, Loder S, Cook S. Characteristics and publication fate of unregistered and retrospectively registered clinical trials submitted to The BMJ over 4 years. BMJ Open 2018; 8: e020037. 10.1136/bmjopen-2017-020037 PMC5829901 29453302 · DOI · Paper page
  25. [33]Larivière V, Sugimoto CR. The Journal Impact Factor: A Brief History, Critique, and Discussion of Adverse Effects. In: Glänzel W, Moed HF, Schmoch U, et al, eds. Springer Handbook of science and technology indicators. Cham: Springer International Publishing, 2019: 3– 24. 10.1007/978-3-030-02511-3_1 · DOI
  26. [34]Chua SK, Qureshi AM, Krishnan V, et al. The impact factor of an open access Journal does not contribute to an article's citations. F1000Res 2017; 6: 208. 10.12688/f1000research.10892.1 28649365 PMC5464220 · DOI
  27. [36]Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas 2007; 1: 77– 89. 10.1080/19312450709336664 · DOI · Paper page
  28. [38]Wright AJ et al. Ontologies relevant to behaviour change interventions: a method for their development [version 3; peer review: 2 approved, 1 approved with reservations]. Wellc Open Res 2020; 5. PMC7786424 33447665
  29. [39]Ek A, Alexandrou C, Söderström E, et al. Effectiveness of a 3-month mobile Phone-Based behavior change program on active transportation and physical activity in adults: randomized controlled trial. JMIR Mhealth Uhealth 2020; 8: e18531. 10.2196/18531 32510462 PMC7308910 · DOI
  30. [40]Tudor-Locke C, Schuna JM, Swift DL, et al. Evaluation of Step-Counting interventions differing on intensity messages. J Phys Act Health 2020; 17: 21– 8. 10.1123/jpah.2018-0439 31698336 · DOI
  31. [41]Kayser JW, Cossette S, Côté J, et al. A web-based tailored nursing intervention (TAVIE en m@rche) aimed at increasing walking after an acute coronary syndrome: Multicentre randomized trial. J Adv Nurs 2019; 75: 2727– 41. 10.1111/jan.14119 31225667 · DOI
  32. [42]Nooijen CFJ, Blom V, Ekblom Örjan, et al. The effectiveness of multi-component interventions targeting physical activity or sedentary behaviour amongst office workers: a three-arm cluster randomised controlled trial. BMC Public Health 2020; 20: 1329. 10.1186/s12889-020-09433-7 32873260 PMC7466462 · DOI
  33. [44]Chambers C. What's next for registered reports? Nature 2019; 573: 187– 9. 10.1038/d41586-019-02674-6 31506624 · DOI · Paper page
  34. [45]Scheel AM, Schijen MRMJ, Lakens D. An excess of positive results: comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci 2021; 4: 25152459211007468: 251524592110074. 10.1177/25152459211007467 · DOI
  35. [46]Basu AP, Pearse JE, Rapley T. Publishing protocols for trials of complex interventions before trial completion - potential pitfalls, solutions and the need for public debate. Trials 2017; 18: 5. 10.1186/s13063-016-1757-7 28069042 PMC5223296 · DOI
  36. [47]Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Soc Sci Med 2018; 210: 2– 21. 10.1016/j.socscimed.2017.12.005 29331519 PMC6019115 · DOI · Paper page
  37. [48]Tetzlaff JM, Chan A-W, Kitchen J, et al. Guidelines for randomized clinical trial protocol content: a systematic review. Syst Rev 2012; 1: 43. 10.1186/2046-4053-1-43 23006870 PMC3533811 · DOI
  38. [49]Severin A, Egger M, Eve MP. Discipline-specific open access publishing practices and barriers to change: an evidence-based review. F1000Res 2020. PMC7194335 32399178
  39. [50]Flanagin A, Fontanarosa PB, Bauchner H. Preprints involving medical Research-Do the benefits outweigh the challenges? JAMA 2020; 324: 1840– 3. 10.1001/jama.2020.20674 33170226 · DOI · Paper page
  40. [51]McVay MA, Conroy DE. Transparency and openness in behavioral medicine research. Transl Behav Med 2021; 11: 287– 90. 10.1093/tbm/ibz154 31675096 PMC7877287 · DOI
  41. [53]Willmott T, Rundle-Thiele S. Are we speaking the same language? call for action to improve theory application and reporting in behaviour change research. BMC Public Health 2021; 21: 479. 10.1186/s12889-021-10541-1 33691658 PMC7944470 · DOI
  42. [55]Osborne C, Norris E. Pre-registration as behaviour: developing an evidence-based intervention specification to increase pre-registration uptake by researchers using the behaviour change wheel. Cogent Psychol 2022; 9: 1. 10.1080/23311908.2022.2066304 · DOI
  43. [56]Orben A. A Journal Club to fix science. Nature 2019; 573: 465– 6. 10.1038/d41586-019-02842-8 31551562 · DOI
  44. [57]Munafò MR, Chambers CD, Collins AM, et al. Research culture and reproducibility. Trends Cogn Sci 2020; 24: 91– 3. 10.1016/j.tics.2019.12.002 31892459 · DOI
  45. [58]Kidwell MC, Lazarević LB, Baranski E, et al. Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLoS Biol 2016; 14: e1002456. 10.1371/journal.pbio.1002456 27171007 PMC4865119 · DOI · Paper page

Cited by

Sorted by citations

Authors
Norris, E., Sulevani, I., Finnerty, A. N., & Castro, O.
Affiliations
  • Brunel University of London (GB)
  • University College London (GB)
  • Centre for Behaviour Change, University College London, London, UK
Corresponding author
Norris, Emma
ORCID
Venue
BMJ Open Sport & Exercise Medicine
Publisher
BMJ
Published
2022-05
Volume
8
Issue
2
Pages
e001282
ISSN
2055-7647 (ISSN-L)
DOI
10.1136/bmjsem-2021-001282
PMC
PMC9174779
OpenAlex
W4281290893
Cited by
14
References
7
Citations by year
2026: 1 · 2025: 1 · 2024: 3 · 2023: 5 · 2022: 4
FWCI
2.71
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Open Sport & Exercise Medicine (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Brunel University Research Archive (BURA) (Brunel University London) (submittedVersion)
  • Brunel University Research Archive (BURA) (Brunel University London) (submittedVersion, cc-by)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
  • ETH Zürich Research Collection
Funders
  • Pfizer
  • National Institute for Health and Care Research
  • National Institutes of Health
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
open science, physical activity interventions, behaviour change interventions, reproducibility, preregistration, open data, open materials, replication, open access publishing, conflict of interest disclosure, funding disclosure, research transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review