Secular trends and evaluation of complex interventions: the rising tide phenomenon
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Evaluations of service delivery interventions with contemporaneous controls often yield null results, even when the intervention appeared promising in advance. There can be many reasons for null results. In this paper we introduce the concept of a ‘rising tide’ phenomenon being a possible explanation of null results. We note that evaluations of service delivery interventions often occur when awareness of the problems they intend to address is already heightened, and pressure to tackle them is mounting throughout a health system. An evaluation may therefore take place in a setting where the system as a whole is improving – where there is a pronounced temporal trend or a ‘rising tide causing all vessels to rise’. As a consequence, control sites in an intervention study will improve. This reduces the difference between intervention and control sites and predisposes the study to a null result, leading to the conclusion that the intervention has no effect. We discuss how a rising tide may be distinguished from other causes of improvement in both control and intervention groups, and give examples where the rising tide provides a convincing explanation of such a finding. We offer recommendations for interpretation of research findings where improvements in the intervention group are matched by improvements in the control group. Understanding the rising tide phenomenon is important for a more nuanced interpretation of null results arising in the context of system-wide improvement. Recognition that a rising tide may have predisposed to a null result in one health system cautions against generalising the result to another health system where strong secular trends are absent.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974
The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting
Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015) · BMJ · doi:10.1136/bmj.h391 · Cited by 1,410
Self-citationExplanatory and pragmatic attitudes in therapeutical trials
Schwartz, D., & Lellouch, J. (1967) · Journal of Chronic Diseases · doi:10.1016/0021-9681(67)90041-0 · Cited by 1,185
The stepped wedge trial design: a systematic review
Brown, C. A., & Lilford, R. J. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-54 · Cited by 907
Self-citationA systematic review of rapid diagnostic tests for the detection of tuberculosis infection
Dinnes, J., Deeks, J., Kunst, H., Gibson, A., Cummins, E., Waugh, N., Drobniewski, F., & Lalvani, A. (2007) · Health Technology Assessment · doi:10.3310/hta11030 · Cited by 666
Research designs for studies evaluating the effectiveness of change and improvement strategies
Eccles, M. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.1.47 · Cited by 586
A systematic review and economic evaluation of statins for the prevention of coronary events
Ward, S., Lloyd Jones, M., Pandor, A., Holmes, M., Ara, R., Ryan, A., Yeo, W., & Payne, N. (2007) · Health Technology Assessment · doi:10.3310/hta11140 · Cited by 556
Contamination in trials: is cluster randomisation the answer?
Torgerson, D. J. (2001) · BMJ · doi:10.1136/bmj.322.7282.355 · Cited by 508
Thompson, C., Kinmonth, A., Stevens, L., Pevele, R., Stevens, A., Ostler, K., Pickering, R., Baker, N., Henson, A., Preece, J., et al. (2000) · The Lancet · doi:10.1016/s0140-6736(99)03171-2 · Cited by 489
Self-citationMdege, N. D., Man, M.-S., Taylor (nee Brown), C. A., & Torgerson, D. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.003 · Cited by 425
Cited by
Improving the health and welfare of people who live in slums
Lilford, R. J., Oyebode, O., Satterthwaite, D., Melendez-Torres, G. J., Chen, Y.-F., Mberu, B., Watson, S. I., Sartori, J., Ndugwa, R., Caiaffa, W., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(16)31848-7 · Cited by 305
Self-citationEvaluating the impact of healthcare interventions using routine data
Clarke, G. M., Conti, S., Wolters, A. T., & Steventon, A. (2019) · BMJ · doi:10.1136/bmj.l2239 · Cited by 164
Hume, N. E., Zerfas, I., Wong, A., Klein-Fedyshin, M., Smithburger, P. L., Buckley, M. S., Devlin, J. W., & Kane-Gill, S. L. (2024) · Critical Care Medicine · doi:10.1097/ccm.0000000000006178 · Cited by 45
Promoting learning from null or negative results in prevention science trials
Axford, N., Berry, V., Lloyd, J., Hobbs, T., & Wyatt, K. (2022) · Prevention Science · doi:10.1007/s11121-020-01140-4 · Cited by 34
Metadata
- Authors
- Chen, Y.-F., Hemming, K., Stevens, A. J., & Lilford, R. J.
- Venue
- BMJ Quality & Safety
- Publisher
- BMJ
- Published
- 2015-10-06
- Volume
- 25
- Issue
- 5
- Pages
- 303-310
- ISSN
- 1475-3898 (ISSN-L)
- 2044-5415
- 2044-5423
- OpenAlex
- W2247171379
- Cited by
- 99
- References
- 12
- FWCI
- 16.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Secular trends and evaluation of complex interventions: the rising tide phenomenon. https://app.cochat.ai/papers/10.1136/bmjqs-2015-004372
- The paper
- Chen, Y.-F., Hemming, K., Stevens, A. J., & Lilford, R. J. (2016). Secular trends and evaluation of complex interventions: the rising tide phenomenon. BMJ Quality & Safety. https://doi.org/10.1136/bmjqs-2015-004372
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.