Accuracy of References in Five Biomedical Informatics Journals
Open accessAbstract
OBJECTIVE: To determine the rate and type of errors in biomedical informatics journal article references. METHODS: References in articles from the first 2004 issues of five biomedical informatics journals, Journal of the American Medical Informatics Association, Journal of Biomedical Informatics, International Journal of Medical Informatics, Methods of Information in Medicine, and Artificial Intelligence in Medicine were compared with MEDLINE for journal, authors, title, year, volume, and page number accuracy. If discrepancies were identified, the reference was compared with the original publication. Two reviewers independently evaluated each reference. RESULTS: The five journal issues contained 37 articles. Among the 656 eligible references, 225 (34.3%) included at least one error. Among the 225 references, 311 errors were identified. One or more errors were found in the bibliography of 31 (84%) of the 37 articles. The reference error rates by journal ranged from 22.1% to 40.7%. Most errors (39.0%) occurred in the author element, followed by the journal (31.2%), title (17.7%), page (7.4%), year (3.5%), and volume (1.3%) information. CONCLUSION: The study identified a considerable error rate in the references of five biomedical informatics journals. Authors are responsible for the accuracy of references and should more carefully check them, possibly using informatics-based assistance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 10 cited sources checked · 0 supported · 0 flagged
- Findingown data
An error in the first author's name was found 39 times (eight errors in the last name and 31 in the first or middle initial).
“An error in the first author's name was found 39 times (eight errors in the last name and 31 in the first or middle initial).”
Quote in the paperNot a substantial claim - Findingown data
Inaccurate journal titles accounted for 97 (31.2%) errors.
“Inaccurate journal titles accounted for 97 (31.2%) errors.”
Quote in the paperNot a substantial claim - Findingown data
In 18 references, an inaccurate page listing accounted for 23 (7.4%) errors (four in the first page, nine in the last page, and five in both pages).
“In 18 references, an inaccurate page listing accounted for 23 (7.4%) errors (four in the first page, nine in the last page, and five in both pages).”
Quote in the paperNot a substantial claim - Findingown data
At the time of this study in summer 2004, a MEDLINE query for ''Int J Med Inform [jour]'' returned no results.
“For example, at the time of this study in summer 2004, a MEDLINE query for ''Int J Med Inform [jour]'' (International Journal of Medical Informatics) returned no results.”
Quote in the paperNot a substantial claim - Findingown data
Such an error is present in reference 20 of the present article, in which MEDLINE lists the third author's name as ''Kee WD,'' whereas the original article lists the correct name as ''Ngan Kee WD''.
“Such an error is present in reference 20 of the present article, in which MEDLINE lists the third author's name as ''Kee WD,'' whereas the original article lists the correct name of the author as ''Ngan Kee WD'' (same author as reference 26).”
Quote in the paperNot a substantial claim
- Findingown data
An error in the list of coauthors was found 52 times (16 errors in the last name and 36 in the first or middle initial).
“An error in the list of coauthors was found 52 times (16 errors in the last name and 36 in the first or middle initial).”
Quote in the paperNot a substantial claim - Findingown data
In this study of five biomedical informatics journals, error rates were between 22.1% and 40.7%.
“In this study of five biomedical informatics journals, we found error rates between 22.1% and 40.7%.”
Quote in the paperSubstantial claim - Findingown data
Six (16%) of the 37 articles had no errors as defined in the study.
“Six (16%) of the 37 articles had no errors as defined in the study.”
Quote in the paperNot a substantial claim - Findingown data
A missing or additional author or an incorrect order of authors was found 30 times.
“A missing or additional author or an incorrect order of authors was found 30 times.”
Quote in the paperNot a substantial claim - Findingown data
225 (34.3%) references contained a total of 311 errors in the journal, authors, title, year, volume, or page number element.
“We identified 225 (34.3%) references that contained a total of 311 errors in the journal, authors, title, year, volume, or page number element of a reference.”
Quote in the paperSubstantial claim - Premisecitation
High error rates (4.1%-40.3%) were even found among leading biomedical journals that check references prior to publication.
“High error rates (4.1%-40.3%)were even found among leading biomedical journals that check references prior to publication.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [23] 23. Sutherland AG, Craig N, Maffulli N, Brooksbank A, Moir JS. Accuracy of references in the orthopaedic literature. J Bone Joint Surg Br. 2000;82:9-10.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data
Many references in biomedical informatics journals include errors.
“Many references in biomedical informatics journals include errors.”
Quote in the paperSubstantial claim - Premisecitation
A previous study that examined the error rate in the library literature did not identify a lower overall reference error rate between biomedical and library publications.
“However, a previous study that examined the error rate in the library literature, in which lower error rates could also be expected due to author familiarity with bibliographies and references, did not identify a lower overall reference error rate between biomedical and library publications.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [5] 5. Celayir AC, Sander S, Celayir S. Accuracy of references in the pediatric surgery journals. J Pediatr Surg. 2003;38:653-4.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
If errors exist in references, successful linking does not occur in online publications.
“If errors exist in references, successful linking does not occur in online publications.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [22] 22. Siebers R, Holt S. Accuracy of references in five leading medical journals. Lancet. 2000;356:1445.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [23] 23. Sutherland AG, Craig N, Maffulli N, Brooksbank A, Moir JS. Accuracy of references in the orthopaedic literature. J Bone Joint Surg Br. 2000;82:9-10.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Study design and conductreasoning
We would have missed errors if authors incorporated references from MEDLINE that were incorrectly indexed.
“We would have missed errors if authors incorporated references from MEDLINE that were incorrectly indexed.”
Quote in the paperNot a substantial claim
- May combine two claims. 12
- Classified as probably not a substantial claim. 18
May combine two claims.
Findingown dataThe author element was the most frequent error category and contained 121 (39.0%) errors in 99 references.
“The author element was the most frequent error category and contained 121 (39.0%) errors in 99 references.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataAt that time, a look up in MEDLINE's journal database returned the applicable MEDLINE abbreviation Int J Med Inf.
“At that time, a look up in MEDLINE's journal database returned the applicable MEDLINE abbreviation Int J Med Inf.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataA single error occurred in 170 (75.6%) and multiple errors in 55 (24.4%) references (two errors in 51 and three errors in four references).
“A single error occurred in 170 (75.6%) and multiple errors in 55 (24.4%) references (two errors in 51 and three errors in four references).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe title element included 55 (17.7%) errors, the year element 11 (3.5%), and the volume element four (1.3%).
“The title element included 55 (17.7%) errors, the year element 11 (3.5%), and the volume element four (1.3%).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe five journal issues included 37 publications with 1,356 references, of which 656 (48.4%) met the inclusion criteria.
“The five journal issues included 37 publications with 1,356 references, of which 656 (48.4%) met the inclusion criteria.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data50 punctuation errors in any part of the 656 study references and one duplicate reference were found; neither error type was included in the overall error rate.
“We found 50 punctuation errors in any part of the 656 study references and one duplicate reference; neither error type was included in the overall error rate.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAn incorrect journal abbreviation was listed 93 times, and an incorrect journal name appeared four times.
“An incorrect journal abbreviation was listed 93 times, and an incorrect journal name appeared four times.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationMost errors affect the names of authors, followed by errors in the title element of a reference.
“Most errors affect the names of authors, followed by errors in the title element of a reference.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites 5,6,[20][21][22][23]25
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenPrevious investigators have reported on reference accuracy in various general and specialist biomedical journals and found error rates ranging from 3% to 60%.
“Previous investigators have reported on reference accuracy in various general and specialist biomedical journals and found error rates ranging from 3% to 60%.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningFor readers who are unfamiliar with the biomedical informatics field, it is more difficult to identify the correct journal.
“For readers who are unfamiliar with the biomedical informatics field, it is more difficult to identify the correct journal.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe Uniform Requirements for Manuscripts Submitted to Biomedical Journals recommends an abbreviated list of authors using et al. after the sixth author.
“The ''Uniform Requirements for Manuscripts Submitted to Biomedical Journals'' recommends an abbreviated list of authors using et al. after the sixth author.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [1] 1. Arrowsmith JE. The accuracy of references in journal articles. Anaesthesia. 1996;51:517-8.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [29]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenErrors in the author's name may only demonstrate a lack of attention to detail.
“Errors in the author's name may only demonstrate a lack of attention to detail.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningUnsuccessful linking increases the visibility of errors to the readers.
“Furthermore, unsuccessful linking increases the visibility of errors to the readers.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationThe names of authors are critical for retrieving articles and are relevant when author citations are used to measure research productivity.
“However, the names of authors are not only critical for retrieving articles but are also relevant when author citations are used to measure research productivity.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim. May combine two claims.
InterpretationreasoningMisspellings and variations in an author's name may prevent the program from retrieving all relevant citations and influence bibliometric measures and information retrieval approaches.
“Misspellings and variations in an author's name may prevent the program from retrieving all relevant citations and influence bibliometric measures and information retrieval approaches.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThese error rates are within the range of reference accuracy rates found in other biomedical journals.
“These error rates are within the range of reference accuracy rates found in other biomedical journals.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenOur study examined only one issue of each journal, and it is possible that other issues contain fewer errors in references.
“Our study examined only one issue of each journal, and it is possible that other issues contain fewer errors in references.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenDisagreements among the two reviewers were resolved by consensus involving the three participating reviewers.
“Disagreements among the two reviewers were resolved by consensus involving the three participating reviewers.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe Science Citation Index from Thomson ISI allows users to examine the citation frequency of an author's publications.
“For example, the Science Citation IndexÒ from Thomson ISI allows users to examine the citation frequency of an author's publications.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Study design and conductnone givenEach included reference was independently checked by two reviewers.
“Each included reference was independently checked by two reviewers.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductreasoningThis study included only references to journal articles that were indexed in MEDLINE; error rates for excluded references may differ from our results.
“This study included only references to journal articles that were indexed in MEDLINE; error rates for excluded references may differ from our results and need to be examined in a separate study.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenPunctuation errors and duplicate references were not included in the calculation of the overall error rates.
“Punctuation errors and duplicate references were not included in the calculation of the overall error rates.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenOur study did not examine MEDLINE's reference error rate.
“In addition, our study did not examine MEDLINE's reference error rate”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMEDLINE's reference error rate has been reported to be 1.5% in a study analyzing publications from 1976 to 1998.
“In addition, our study did not examine MEDLINE's reference error rate, which has been reported to be 1.5% in a study analyzing publications from 1976 to 1998.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
The paper relies on this without support.
The reference accuracy in biomedical informatics journals has not been previously examined.
“The reference accuracy in biomedical informatics journals, the aim of this study, has not been previously examined.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is well established that reference inaccuracies are highly prevalent in the peer-reviewed literature, ranging from 4% to 48% of citations.
“It is well established that reference inaccuracies are highly prevalent in the peer-reviewed literature, ranging from 4% to 48% of citations [8-11].”
From High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content (2023) · cites this paper as [8]
The Introduction reports substantial reference-error rates in biomedical journals, and the Results find errors in 34.3% of the references examined. But the paper gives prior rates of 3%–60% or 4.1%–40.3%, not the claimed 4%–48%; its 48.4% figure is the proportion of references meeting the study’s inclusion criteria, not an error rate. Page numbers are unavailable in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Included references exceed total references in two journals
MinorTable 1 lists 181 included references against 141 total references for the first journal, and 175 against 130 for the second. An included subset cannot exceed its corresponding total.
“| Total references | 141 | 130 | 223 | 131 | 75 |”
“| Included references | 181 | 175 | 145 | 109 | 46 |”
The cited reference 20 has no third author
MinorThe Discussion describes reference 20 as having a third author whose name is Ngan Kee WD, but the bibliography lists reference 20 as a single-author paper by Siebers.
“Such an error is present in reference 20 of the present article, in which MEDLINE lists the third author's name as ''Kee WD,'' whereas the original article lists the correct name of the author as ''Ngan Kee WD'' (same author as reference 26).”
“20. Siebers R. The accuracy of references of three allergy journals. J Allergy Clin Immunol. 2000;105:837-8.”
Reference 26 does not list Ngan Kee WD
MinorThe Discussion identifies Ngan Kee WD as an author of reference 26, but that bibliography entry lists Hecht F, Hecht BK, and Sandberg AA.
“''Ngan Kee WD'' (same author as reference 26).”
“26. Hecht F, Hecht BK, Sandberg AA. The journal ''impact factor'': a misnamed, misleading, misused measure. Cancer Genet Cytogenet. 1998;104:77-81.”
Error multiplicities account for fewer than 311 errors
MinorThe stated breakdown gives 170 single errors, 51 references with two errors, and four with three errors: 284 errors altogether, rather than 311.
Reported 311 · recomputed 284
“We identified 225 (34.3%) references that contained a total of 311 errors in the journal, authors, title, year, volume, or page number element of a reference.A single error occurred in 170 (75.6%) and multiple errors in 55 (24.4%) references (two errors in 51 and three errors in four references).”
Cites
Hecht, F., Hecht, B. K., & Sandberg, A. A. (1998) · Cancer Genetics and Cytogenetics · doi:10.1016/s0165-4608(97)00459-7 · Cited by 206 · see citing works
How accurate are quotations and references in medical journals?
de Lacey, G., Record, C., & Wade, J. (1985) · BMJ · doi:10.1136/bmj.291.6499.884 · Cited by 160 · see citing works
Quotational and Reference Accuracy in Surgical Journals: A Continuing Peer Review Problem
Evans, J. T. (1990) · JAMA · doi:10.1001/jama.1990.03440100059009 · Cited by 129 · cited 1× · see citing works
“Adapting the methodology of previous studies, 9,21–23, 25 we included We defined a reference to a publication as incorrect if one or more errors were identified in any of the following six elements: journal, authors, title, year, volume, and page numbers.”
MethodEichorn, P., & Yankauer, A. (1987) · American Journal of Public Health · doi:10.2105/ajph.77.8.1011 · Cited by 114 · see citing works
The accuracy of citation and quotation in otolaryngology/head and neck surgery journals
Fenton, J. E., Brazier, H., Souza, A., Hughes, J. P., & McShane, D. P. (2000) · Clinical Otolaryngology · doi:10.1046/j.1365-2273.2000.00322.x · Cited by 94 · see citing works
Accuracy of references in five leading medical journals
Siebers, R., & Holt, S. (2000) · The Lancet · doi:10.1016/s0140-6736(05)74090-3 · Cited by 71 · see citing works
Lifset, R., & Lindhqvist, T. (2001) · Journal of Industrial Ecology · doi:10.1162/10881980152830105 · Cited by 68 · see citing works
Trust, but verify. The accuracy of references in four anesthesia journals.
McLellan, M. F., Case, L. D., & Barnett, M. C. (1992) · PubMed · Cited by 61 · see citing works
Trust, but Verify: The Accuracy of References in Four Anesthesia Journals
McLellan, M. F., Case, L. D., & Barnett, M. C. (1992) · Anesthesiology · doi:10.1097/00000542-199207000-00024 · Cited by 60 · see citing works
Reference accuracy in the emergency medicine literature
Goldberg, R., Newton, E., Cameron, J., Jacobson, R., Chan, L., Bukata, W. R., & Rakab, A. (1993) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(05)81995-x · Cited by 49 · see citing works
Reference citations in radiology: accuracy and appropriateness of use in two major journals.
Hansen, M. E., & McIntire, D. D. (1994) · American Journal of Roentgenology · doi:10.2214/ajr.163.3.8079876 · Cited by 46 · see citing works
The quality of citations in major international obstetrics and gynecology journals
a, V. J. R., Lau, T. K., & Keeb, W. D. N. (1997) · American Journal of Obstetrics and Gynecology · doi:10.1016/s0002-9378(97)70303-x · Cited by 46 · see citing works
Reference accuracy in the dermatologic literature
George, P. M., & Robbins, K. (1994) · Journal of the American Academy of Dermatology · doi:10.1016/s0190-9622(94)70136-9 · Cited by 45 · see citing works
Improvement of the accuracy of references in theCanadian Journal of Anaesthesia
Asano, M., Mikawa, K., Nishina, K., Nobuhiro Maekawa, A., & Obara, H. (1995) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03015478 · Cited by 41 · see citing works
The accuracy of references in paediatric journals
Vargas-Origel, A. (2001) · Archives of Disease in Childhood · doi:10.1136/adc.85.6.497 · Cited by 37 · see citing works
The accuracy of references of three allergy journals
Siebers, R. W. L. (2000) · Journal of Allergy and Clinical Immunology · doi:10.1067/mai.2000.104935 · Cited by 34 · see citing works
A review of the accuracy of references in the journal Emergency Medicine
O’Connor, A. E. (2002) · Emergency Medicine Australasia · doi:10.1046/j.1442-2026.2002.00307.x · Cited by 26 · see citing works
The accuracy of references in Anaesthesia
Asano, M., Mikawa, K., Nishina, K., Nobuhiro Maekawa, A., & Obara, H. (1995) · Anaesthesia · doi:10.1111/j.1365-2044.1995.tb05956.x · Cited by 24 · see citing works
Incorrect citations: a comparison of library literature with medical literature.
Benning, S., & Speer, S. C. (1993) · Europe PMC (PubMed Central) · Cited by 23 · see citing works
HOW ACCURATE ARE REFERENCES IN THE AUSTRALIAN AND NEW ZEALAND JOURNAL OF SURGERY?
Kee, W. D. N., Roach, V. J., & Lau, T. K. (1997) · Australian and New Zealand Journal of Surgery · doi:10.1111/j.1445-2197.1997.tb02005.x · Cited by 19 · see citing works
Do authors check their references?
Orlin, W., Pehling, J., & Pogrel, M. A. (1996) · Journal of Oral and Maxillofacial Surgery · doi:10.1016/s0278-2391(96)90447-4 · Cited by 19 · see citing works
Reference Accuracy in The Journal of Hand Surgery
Jackson, K., Porrino, J. A., Tan, V., & Daluiski, A. (2003) · The Journal of Hand Surgery · doi:10.1053/jhsu.2003.50085 · Cited by 18 · see citing works
Accuracy of References in the Orthopaedic Literature
Sutherland, A. G., Craig, N., Maffulli, N., Brooksbank, A. J., & Moir, J. S. (2000) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.82b1.9035 · Cited by 16 · see citing works
ACCURACY OF REFERENCES IN THE ORTHOPAEDIC LITERATURE
Sutherland, A. G., Craig, N., Maffulli, N., Brooksbank, A., & Moir, J. S. (2000) · The Journal of Bone and Joint Surgery. British volume · doi:10.1302/0301-620x.82b1.0820009 · Cited by 12 · see citing works
How accurate are quotations and references in medical journals?
Leger, A. S. S. (1985) · BMJ · doi:10.1136/bmj.291.6506.1420 · Cited by 11 · see citing works
Celayir, A. C., Sander, S., & Celayir, S. (2003) · Journal of Pediatric Surgery · doi:10.1053/jpsu.2003.50154 · Cited by 11 · see citing works
How Accurate Are References in Clinical Chemistry?
Siebers, R. W. L. (2001) · Clinical Chemistry · doi:10.1093/clinchem/47.3.606 · Cited by 9 · see citing works
Is propofol infusion better than inhalational anaesthesia for paediatric MRI?
Macintyre, P. A., & Sury, M. R. J. (1996) · Anaesthesia · doi:10.1111/j.1365-2044.1996.tb07836.x · Cited by 9 · see citing works
Reference Accuracy in Critical Care Medicine
Nishina, K., Mikawa, K., Asano, M., Nobuhiro Maekawa, A., & Obara, H. (1995) · Critical Care Medicine · doi:10.1097/00003246-199509000-00029 · Cited by 7 · see citing works
Accuracy of References in Anesthesia & Analgesia Does Not Improve
Nishina, K., Asano, M., Mikawa, K., Nobuhiro Maekawa, A., & Obara, H. (1995) · Anesthesia & Analgesia · doi:10.1097/00000539-199503000-00047 · Cited by 7 · see citing works
Su, D., Jones, T. M., Brazier, H., et al. (2001) · Clinical Otolaryngology · doi:10.1046/j.1365-2273.2001.00414.x · Cited by 7 · see citing works
The accuracy of references in journal articles
Arrowsmith, J. E. (1996) · Anaesthesia · doi:10.1111/j.1365-2044.1996.tb07837.x · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 34 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content
Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023) · Cureus · doi:10.7759/cureus.39238 · Cited by 294 · see citing works
The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?
Wright, M., & Armstrong, J. S. (2008) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.1070.0317 · Cited by 83 · see citing works
“…in the 180 references examined (Schulmeister 1998), 40 percent in otolaryngology/head and neck surgery journals, with 12 percent major errors (Fenton et al. 2000), 36 percent in manual therapy journals (Gosling et al. 2004), and 34 percent in biomedical informatics journals (Aronsky et al. 2005).”
BackgroundTechnical editing of research reports in biomedical journals
Wager, E., & Middleton, P. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000002.pub3 · Cited by 68 · see citing works
Let not thy food be confused with thy medicine: The Hippocratic misquotation
Cárdenas, D. (2013) · e-SPEN Journal · doi:10.1016/j.clnme.2013.10.002 · Cited by 62 · see citing works
Aronsky, D., Madani, S., Carnevale, R. J., Duda, S. N., & Feyder, M. T. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2243 · Cited by 49 · see citing works
Self-citationAccuracy of references in burns journals
Al-Benna, S., Rajgarhia, P., Ahmed, S., & Sheikh, Z. (2009) · Burns · doi:10.1016/j.burns.2008.11.014 · Cited by 41 · see citing works
Accuracy of references in general surgical journals — An old problem revisited
Reddy, M. S., Srinivas, S., Sabanayagam, N., & Balasubramanian, S. P. (2008) · The Surgeon · doi:10.1016/s1479-666x(08)80067-4 · Cited by 41 · see citing works
Andrade, N. S., Flynn, J. P., & Bartanusz, V. (2013) · The Spine Journal · doi:10.1016/j.spinee.2013.06.071 · Cited by 38 · see citing works
“The citation process may be subject to technical error [83,84], but the type and magnitude of error reported do not seem sufficient to influence results of our analysis.”
ResultBiomedical journal editing: elements of success
Gasparyan, A. Y., Ayvazyan, L., & Kitas, G. D. (2011) · Croatian Medical Journal · doi:10.3325/cmj.2011.52.423 · Cited by 35 · see citing works
An Exploration of ebook Selection Behavior in Academic Library Collections
McKay, D., Hinze, A., Heese, R., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33290-6_2 · Cited by 25 · see citing works
“Buchanan and McKay show that the majority of those seeking books even in high end bookshops have a limited knowledge of the kinds of metadata (such as publisher) contained in front matter [3], and [1] and [26] demonstrate that both”
BackgroundComparison of Researcher’s Reference Management Software: Refworks, Mendeley, and EndNote
Basak, S. K. (2014) · Journal of Economics and Behavioral Studies · doi:10.22610/jebs.v6i7.517 · Cited by 23 · see citing works
“Reference management software reduces the workload burden for the researchers to edit, proofread, and avoid formatting errors (Aronsky et al., 2005).”
BackgroundNilashi, M., Dalvi, M. R., Ibrahim, O., Zamani, M., & Ramayah, T. (2016) · Journal of Librarianship and Information Science · doi:10.1177/0961000616668961 · Cited by 22 · see citing works
“RMS decreases researchers’ workload in terms of editing, proofreading and avoiding formatting errors (Aronsky et al., 2005).”
BackgroundReference accuracy in four rehabilitation journals
Unver, B., Senduran, M., Unver Kocak, F., Gunal, I., & Karatosun, V. (2009) · Clinical Rehabilitation · doi:10.1177/0269215508102968 · Cited by 21 · see citing works
Georgescu, C., & Wren, J. D. (2018) · Bioinformatics · doi:10.1093/bioinformatics/btx811 · Cited by 20 · see citing works
The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?Preprint
Wright, M., & Armstrong, J. S. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1941335 · Cited by 17 · see citing works
Bugyei, K. A., Kavi, R. K., & Obeng-Koranteng, G. (2019) · Journal of Information & Knowledge Management · doi:10.1142/s021964921950031x · Cited by 14 · see citing works
“RMS decreases researchers' workload in terms of editing, proofreading and avoiding the formatting errors (Aronsky et al., 2005). RMS not only appraises and codes search results but also organises and stores search results (King et al., 2011). According to Steele (2008) and Emanuel (2013), the primary reasons in consid…”
InfluentialReference accuracy in library and information science journals
Davies, K. S. (2012) · Aslib Journal of Information Management · doi:10.1108/00012531211244734 · Cited by 14 · see citing works
Yu, A. C., & Cimino, J. J. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.01.005 · Cited by 13 · see citing works
Do they practice what they preach? The presence of problematic citations in business ethics research
Serenko, A., Dumay, J., Hsiao, P.-C. K., & Choo, C. W. (2021) · Journal of Documentation · doi:10.1108/jd-01-2021-0018 · Cited by 11 · see citing works
Features Influencing Researchers’ Selection of Reference Management Software
Nilashi, M., Ibrahim, O., Sohaei, S., Ahmadi, H., & Almaee, A. (2016) · Journal of Information & Knowledge Management · doi:10.1142/s0219649216500325 · Cited by 10 · see citing works
Sandelowski, M. (2009) · Research in Nursing & Health · doi:10.1002/nur.20365 · Cited by 8 · see citing works
Accurate references add to the credibility
Mohta, A., & Mohta, M. (2016) · Indian Pediatrics · doi:10.1007/s13312-016-0976-z · Cited by 9 · see citing works
Cómo escribir y evaluar un artículo científico para la Revista Española de Geriatría y Gerontología
Formiga, F., Baztán, J. J., Montorio, I., et al. (2009) · Revista Española de Geriatría y Gerontología · doi:10.1016/j.regg.2009.03.004 · Cited by 9 · see citing works
Wicaksono, M. F., & Nurpratama, M. R. (2018) · Record and Library Journal · doi:10.20473/rlj.v3-i2.2017.209-219 · Cited by 7 · see citing works
McKay, D., & Buchanan, G. R. (2013) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/2541016.2541031 · Cited by 6 · see citing works
“…on questions asked in bookshops, is that users do not have a clear grasp of the metadata needed to construct more advanced queries (Buchanan et al. 2011), a finding reinforced by work that demonstrates students and academics alike struggle to create citations (Aronsky et al. 2005; Yu et al. 2006).”
BackgroundA study of citation accuracy in psychology theses submitted to the University of Mysore
Harinarayana, N. S., Chikkamanju, & Vasantha, R. N. (2011) · MyPrints@UOM (Mysore University Library) · Cited by 6 · see citing works
Accuracy of References in Eight Nursing Journals in Korea
Suk, M., Jang, H.-J., Park, J. S., Kim, H. W., Suh, Y. O., Shin, H., Yang, J.-H., Jung, M.-S., & Chung, M.-S. (2008) · Daehan ganho haghoeji · doi:10.4040/jkan.2008.38.1.180 · Cited by 6 · see citing works
Logan, S. W., Hussong-Christian, U., Case, L., & Noregaard, S. (2024) · Journal of Librarianship and Information Science · doi:10.1177/09610006231177715 · Cited by 6 · see citing works
Logan, S. W. (2021) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.2020.1853019 · Cited by 5 · see citing works
“Because reference accuracy is important, studies have investigated the issue across journals from different fields of study (Aronsky et al., 2005; Awrey et al., 2011; Boya et al., 2008; Gosling et al., 2004; Key & Roland, 1977; Mitchell-Williams et al., 2017; Mohta & Mohta, 2003; Narin et al.,…”
BackgroundResultInfluentialAccuracy of References in Journal Literature of Medical Sciences: A Review
Gupta, V. K., & Babel, P. (2018) · IRA-International Journal of Management & Social Sciences (ISSN 2455-2267) · doi:10.21013/jmss.v12.n3.p2 · Cited by 5 · see citing works
Errores en las referencias bibliográficas de las revistas médicas peruanas
Huamaní, C., & Romero, J. P. (2009) · Americanae (AECID Library) · doi:10.47892/rgp.2009.294.474 · Cited by 5 · see citing works
How accurate are cited references in Iranian peer-reviewed journals?
Ghane, M. R. (2016) · Learned Publishing · doi:10.1002/leap.1000 · Cited by 4 · see citing works
Accuracy of references: comparison between two premier Pakistani medical journals.
Samad, A. R., Khanzada, T. W., Kumar, B., & Malik, K. A. (2013) · PubMed · Cited by 4 · see citing works
Arandjelović, O. (2026) · Bioethics · doi:10.1111/bioe.70099 · Cited by 3 · see citing works
“The literature on reference accuracy shows that errors are common across biomedical fields [15] and the key question is whether journals treat them as understandable, tolerable errors or as breaches of scholarly trust.”
Efektivitas Mendeley sebagai Software Bantu untuk Manajemen Referensi
Nurwahyu, N., & Nafisah, S. (2021) · PUSTABIBLIA Journal of Library and Information Science · doi:10.18326/pustabiblia.v5i1.45-76 · Cited by 3 · see citing works
Reference Accuracy in Four Respiratory Medical Journals
Narin, S., Koçak, F. Ü., Özalevli, S., & Ilgin, D. (2010) · Respiration · doi:10.1159/000295808 · Cited by 3 · see citing works
Lawrence, D. W. (2008) · Figshare · Cited by 3 · see citing works
Messinger‐Rapport, B. J., Gammack, J. K., & Thomas, D. R. (2008) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2007.10.002 · Cited by 3 · see citing works
“A review of 5 biomedical journals found an error rate ranging from 22% to 41%, usually in the title or page number.(10) References should reflect the most current and highest quality research, avoiding citations of review articles, abstracts, unpublished data, or personal communications.”
BackgroundHersh, W. (2020) · Health informatics · doi:10.1007/978-3-030-47686-1_2 · Cited by 2 · see citing works
References in Australasian Medical Journal: Are they accurate?
Adhikari, P. (2010) · Australasian Medical Journal · doi:10.4066/amj.2010.556 · Cited by 2 · see citing works
“A good scientific journal must exhibit accuracy in the references cited, moreover it is the right of the reader to be facilitated by providing accurate references and hence it becomes the shared responsibility of the contributors and editorial board [3] .”
BackgroundA Method to Reduce Citation Errors While Compiling Bibliographies
Parashar, A., Makkar, S. S., Sharma, R. K., & Nanda, V. (2007) · Annals of Plastic Surgery · doi:10.1097/sap.0b013e318070da16 · Cited by 2 · see citing works
Leading Knowledge Producers Within the Biomedical Informatics Community
Househ, M. S. (2006) · Journal of Medical Systems · doi:10.1007/s10916-005-9002-z · Cited by 2 · see citing works
How Accurate are References in Nursing Journals?
Oermann, M. H., & Wilmes, N. A. (2005) · Nurse author & editor · doi:10.1111/j.1750-4910.2005.tb00557.x · Cited by 2 · see citing works
Basak, S. K. (2016) · International Conference on Electrical and Computer Engineering (ICECE) · doi:10.1109/icece.2016.7853867 · Cited by 1 · see citing works
“According to [5], if citations are inaccurate, then it becomes time-consuming, frustrating, or even impossible to retrieve for readers.”
BackgroundInfluentialHuamaní, C., & Romero, J. P. (2012) · Anales de la Facultad de Medicina · doi:10.15381/anales.v73i2.852 · Cited by 1 · see citing works
Mi derecho a réplica: errores en las referencias
Román, J. A. S. (2010) · Revista Española de Cardiología · doi:10.1016/s0300-8932(10)70053-1 · Cited by 1 · see citing works
A Framework for Verifying Citation Support Across Languages Using Multilingual Sentence Embeddings
Kubis, M., Dubec, J., Baťalík, P., et al. (2026) · doi:10.1109/iit.src67760.2026.11629011
“Although citation accuracy has long been recognized as problematic in scholarly writing [6] [7], systematic computational approaches to citation verification are only beginning to emerge.”
Nguyen, L. T., & Tuamsuk, K. (2025) · De Gruyter eBooks · doi:10.1515/9783111373485-024
EXPLORING REFERENCE ACCURACY ON UNDERGRADUATE THESIS OF ENGLISH LANGUAGE AND LITERATURE PROGRAM
(2023) · doi:10.32503/proficiency.v5i1.3286
“, with so many breaches found in this present study, it conforms Aronsky et.al (2005) that formatting a paper in APA style can be tricky.”
Sophisticated Embedding of Artificial Intelligence Techniques in Biomedical Engineering
Prasad, P. S., Sekhar, G. S., & Seelaboyina, R. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-6407-6_51
al., C. E. A. C. E., & TJPRC (2017) · International Journal of Library Science and Research · doi:10.24247/ijlsroct20173
Mohta, A., & Mohta, M. (2017) · doi:10.1007/978-981-10-4720-6_9
میزان بهرهگیری از نرمافزارهای مدیریت استناد در پایگاههای قابل دسترس دانشگاه علوم پزشکی تهران
امتی, ا., & آزاده, ف. (2016) · doi:10.22088/acadpub.bums.3.1.45
Aral, Y. Z., Turan, Y., Kozaci, D., Şavk, E., Karaman, G., & Yıldız, Y. (2015) · Meandros Medical And Dental Journal · doi:10.5152/adutfd.2015.1904
Basak, S. K. (2015) · IEEE International Conference on Cloud Computing Technology and Science
Aral, Y. Z., Turan, Y., Kozaci, D., Şavk, E., Karaman, G., & Yıldız, Y. (2014) · Meandros Medical And Dental Journal · doi:10.5152/adutfd.2014.141904
Collapsing Duplicates at Data Entry: Design Issues in Building Taiwan Citations Index
Chiang, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12823-8_31
Atıcı, E., Ercan, İ., Cangür, Ş., & Büyükuysal, Ç. (2013) · Turkiye Klinikleri Journal of Medical Sciences · doi:10.5336/medsci.2012-30089
Sørbye, L. W. (2011) · Sykepleien Forskning · doi:10.4220/sykepleienf.2011.0025
بررسي فاصله زماني انجام تحقيق تا انتشار مقالات در مجلات علمي پژوهشي دندانپزشكي كشور
Sheibaninia, A., Valaie, N., Ramezani, L., et al. (2010)
My Right to Reply: Inaccurate References
Román, J. A. S. (2010) · Revista Española de Cardiología (English Edition) · doi:10.1016/s1885-5857(10)70053-0
The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?Preprint
Armstrong, J. S. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1153123
- Authors
- Aronsky, D.
- Affiliations
- Meharry Medical College (US)
- Corresponding author
- Aronsky, D.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-11-23
- Volume
- 12
- Issue
- 2
- Pages
- 225-228
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC551554
- OpenAlex
- W2171590415
- Cited by
- 63
- References
- 34
- Citations by year
- 2026: 2 · 2025: 1 · 2023: 3 · 2022: 1 · 2021: 3 · 2020: 1 · 2019: 1 · 2018: 2 · 2017: 3 · 2016: 6 · 2015: 2 · 2014: 3 · 2013: 5 · 2012: 4
- FWCI
- 9.78
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- citation errors, MEDLINE, bibliographic references, author name errors
- MeSH terms
- Bibliographies as Topic, Medical Informatics, Periodicals as Topic, Bibliometrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
