The DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
A "take with a grain of salt" label for Holiday Review articles
Open accessAbstract
[A CMAJ Deputy Editor responds:] We thank Christopher Naugler for bringing this to our attention. Normally, spoof science does not appear in PubMed searches. However, Naugler's findings show that the system we have been using is neither inclusive nor foolproof. In the future, we will include a
Overview
CoChat Summary
TL;DR
In a brief CMAJ commentary, Sibbald reports that a Google Scholar search found at least 4 instances where authors cited Holiday Review spoof articles as if they were real studies. The author suggests adding a disclaimer to evidence-free Holiday Review abstracts or titles if they remain indexed in MEDLINE.
Key findings
- A Google Scholar search for citations of CMAJ Holiday Review articles published 1999–2006 found at least 4 instances where citing authors appeared to mistake evidence-free articles for real studies.
- Examples included a citation of a case report whose patient was a cartoon character, a citation of the author’s own “completely evidence-free paper,” and 2 other citations of Holiday Review spoof studies.
- The author proposes that if Holiday Review articles continue to be indexed in MEDLINE, each evidence-free abstract should include a note stating the article is for entertainment purposes only and is not a real study.
- Additional information states that in future, CMAJ will include a disclaimer in the titles of its Holiday Review science articles.
Methods
- Design: brief commentary/letter describing an informal citation search.
- Data: Google Scholar citations of Holiday Review articles published in CMAJ from 1999–2006; these citations were then reviewed.
- Analysis: qualitative review of whether citing authors understood the articles as tongue-in-cheek versus mistakenly treating them as evidence-based studies; no formal statistical analysis is reported.
Limitations
- The paper states no formal limitations section.
- Evident from the text: the search is described only briefly, with no reproducible search strategy, search date, number of citations screened, or inclusion criteria; the finding is reported as “at least 4 instances,” not a systematic count.
- Evident from the text: the article is a short commentary/letter, so it does not provide a systematic assessment of MEDLINE indexing or citation practices.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 1 Not checked · 5 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in at least 4 instances, the citing authors appear to have mistakenly cited evidence-free articles as if they were real studies: a citation 1 of a case report in which the patient was a cartoon character, 2 a citation 3 of my own completely evidence-free paper 4 and 2 other citations 5,6 of Holiday Review spoof studies.7,8”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“confusion has sometimes arisen because these articles are indexed in MEDLINE as if they were real research articles.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some instances, the authors citing Holiday Review articles clearly understood that they were citing a tongue-in-cheek”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“confusion has sometimes arisen because these articles are indexed in MEDLINE as if they were real research articles.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Probably not a substantial claim. 1
- It may state two separate claims as one. 2
Citations
Cites
Effectiveness of Antipsychotic Drugs in Patients with Chronic Schizophrenia
Lieberman, J. A., Stroup, T. S., McEvoy, J. P., Swartz, M. S., Rosenheck, R. A., Perkins, D. O., Keefe, R. S. E., Davis, S. M., Davis, C. E., Lebowitz, B. D., et al. (2005) · New England Journal of Medicine · doi:10.1056/nejmoa051688 · Cited by 5,892
Lieberman, J. A. (2007) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.0207e04 · Cited by 520
Recent developments and current controversies in depression
Ebmeier, K. P., Donaghey, C., & Steele, J. D. (2006) · The Lancet · doi:10.1016/s0140-6736(06)67964-6 · Cited by 434
Barbui, C., Furukawa, T. A., & Cipriani, A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.070693 · Cited by 131
Michels, P. J., Probst, J. C., Godenick, M. T., & Palesch, Y. Y. (2003) · Academic Medicine · doi:10.1097/00001888-200301000-00013 · Cited by 88
Management of depression in adults
Timonen, M., & Liukkonen, T. (2008) · BMJ · doi:10.1136/bmj.39478.609097.be · Cited by 80
Hypopituitarism following traumatic brain injury
Popovic, V., Aimaretti, G., Casanueva, F. F., & Ghigo, E. (2005) · Growth Hormone & IGF Research · doi:10.1016/j.ghir.2005.02.003 · Cited by 47
Poluzzi, E., Motola, D., Silvani, M. C., Ponti, F. D., Vaccheri, A., & Montanaro, N. (2003) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-003-0692-1 · Cited by 44
Cyr, A. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041405 · Cited by 28
Hypopituitarism following Traumatic Brain Injury
POPOVIC, V. M., Aimaretti, G., Casanueva, F. F., & Ghigo, E. (2005) · KARGER eBooks · doi:10.1159/000088399 · Cited by 25
Emergency department assessment of abdominal pain: clinical indicator tests for detecting peritonism
Taylor, S., & Watt, M. (2005) · European Journal of Emergency Medicine · doi:10.1097/00063110-200512000-00005 · Cited by 17
Bubble bubble, abdominal trouble: a new test to chew on.
Worrall, G. J., Holmes, P., & Robbins, M. (1999) · Europe PMC (PubMed Central) · Cited by 7
The impact of hissy fits in primary care.
Osmun, W. E., & Naugler, C. T. (1998) · Europe PMC (PubMed Central) · Cited by 5
A "take with a grain of salt" label for Holiday Review articles
Naugler, C. T. (2008) · Canadian Medical Association Journal · doi:10.1503/cmaj.1080004 · Cited by 5
Effectiveness of selective serotonin reuptake inhibitors
Barbui, C., Furukawa, T. A., & Cipriani, A. (2008) · Canadian Medical Association Journal · doi:10.1503/cmaj.1080028 · Cited by 5
Research works printed but not linked: 1
- Mangion K. The stethoscope. Malta Med J 2007;19: 41-4.
Cited by
Mohan, S., & Campbell, N. R. C. (2009) · Clinical Science · doi:10.1042/cs20080207 · Cited by 110
Metadata
- Authors
- Sibbald, B.
- Affiliations
- Dai Nippon Printing (Japan) (JP)
- Corresponding author
- Sibbald, B.
- Venue
- Canadian Medical Association Journal
- Publisher
- CMA Joule Inc.
- Published
- 2008-04-22
- Volume
- 178
- Issue
- 9
- Pages
- 1186-1186
- ISSN
- 0820-3946 (ISSN-L)
- 1488-2329
- PMC
- PMC2292784
- OpenAlex
- W1968308010
- Cited by
- 1
- References
- 14
- FWCI
- 1.25
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Canadian Medical Association Journal (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- Funders
- Pfizer
- Ministry of Education, Science and Technology
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Keywords
- medical journal publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.