The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index; 2 of the 34 sources its claims cite were checked for support.
Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review
Open accessAbstract
BACKGROUND: Economic evaluation can inform whether strategies designed to improve the quality of health care delivery and the uptake of evidence-based practices represent a cost-effective use of limited resources. We report a systematic review and critical appraisal of the application of health economic methods in improvement/implementation research. METHOD: A systematic literature search identified 1668 papers across the Agris, Embase, Global Health, HMIC, PsycINFO, Social Policy and Practice, MEDLINE and EconLit databases between 2004 and 2016. Abstracts were screened in Rayyan database, and key data extracted into Microsoft Excel. Evidence was critically appraised using the Quality of Health Economic Studies (QHES) framework. RESULTS: Thirty studies were included-all health economic studies that included implementation or improvement as a part of the evaluation. Studies were conducted mostly in Europe (62%) or North America (23%) and were largely hospital-based (70%). The field was split between improvement (N = 16) and implementation (N = 14) studies. The most common intervention evaluated (43%) was staffing reconfiguration, specifically changing from physician-led to nurse-led care delivery. Most studies (N = 19) were ex-post economic evaluations carried out empirically-of those, 17 were cost effectiveness analyses. We found four cost utility analyses that used economic modelling rather than empirical methods. Two cost-consequence analyses were also found. Specific implementation costs considered included costs associated with staff training in new care delivery pathways, the impacts of new processes on patient and carer costs and the costs of developing new care processes/pathways. Over half (55%) of the included studies were rated 'good' on QHES. Study quality was boosted through inclusion of appropriate comparators and reporting of incremental analysis (where relevant); and diminished through use of post-hoc subgroup analysis, limited reporting of the handling of uncertainty and justification for choice of discount rates. CONCLUSIONS: The quantity of published economic evaluations applied to the field of improvement and implementation research remains modest; however, quality is overall good. Implementation and improvement scientists should work closely with health economists to consider costs associated with improvement interventions and their associated implementation strategies. We offer a set of concrete recommendations to facilitate this endeavour.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic review identified 30 studies (2004–2016) applying health economic evaluation to implementation and improvement science, finding modest quantity but overall good quality. It recommends closer collaboration with health economists and better capture of implementation/hidden costs, plus greater use of ex-ante and hybrid evaluative designs.
Key findings
- From 1668 records, 30 studies were included: 16 improvement and 14 implementation studies. Most were European (62%) or North American (23%) and hospital-based (70%).
- The most common intervention was staffing reconfiguration in 13 studies (43%), specifically changing from physician-led to nurse-led care. Nineteen studies were ex-post empirical economic evaluations, including 17 cost-effectiveness analyses; four cost-utility analyses used economic modelling, and two were cost-consequence analyses.
- QHES appraisal covered 22 papers: average score was 76/100, 13 studies (62%) scored >75, and one scored 100. The abstract reports over half (55%) were rated “good.” Improvement studies performed better than implementation studies.
- Only six of 30 studies explicitly assessed implementation-related “hidden” costs such as training, development, and monitoring; three included rollout/scaling scenarios, and one considered funding sources.
Methods
- Systematic literature search across Agris, Embase, Global Health, HMIC, PsycINFO, Social Policy and Practice, MEDLINE, and EconLit, covering publications after 2003 through 16 March 2016; 1668 papers were screened in Rayyan and extracted into Microsoft Excel.
- Inclusion required English-language completed studies in public health/primary/secondary/tertiary care with implementation or improvement aspects, quantifiable patient/service outcomes, and specified economic methods; cost-only studies, protocols, methodological papers, and conference abstracts were excluded.
- Methodological quality was assessed with the Quality of Health Economic Studies (QHES) framework, scoring 16 criteria out of 100, with ≥75 considered “good.”
Limitations
- The authors state that strict inclusion criteria may have limited the number of studies identified and synthesised; they excluded cost-only studies and non-English papers, making criteria stricter than prior reviews.
- The authors state they cannot rule out whether early-phase economic analyses are not being done or are simply not reported in academic publications.
- Evident from the text: QHES appraisal excluded literature reviews, meta-analyses, and commentaries, so quality conclusions apply to 22 of 30 included papers; comparison with Hoomans et al. is limited because the two reviews used different quality appraisal frameworks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
39 Own result · 15 Not checked yet · 9 Own method · 8 Own interpretation · 5 No basis given · 1 Partly supported · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thirty studies were included in the final evidence review and synthesis.”
- Also stated in: § Results · plain
“Thirty studies were included—all health economic studies that included implementation or improvement as a part of the evaluation.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Studies included
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common intervention type, evaluated in 13 (43%) of the included studies, was staffing reconfiguration for service quality improvement, specifically changing from physician-led to nurse-led delivery of interventions to patients.”
- Also stated in: § Results · plain
“The most common intervention evaluated (43%) was staffing reconfiguration, specifically changing from physician-led to nurse-led care delivery.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More broadly, interventions involving general service reorganisation or changes to existing systems of care were the primary focus in ten (33%) of studies reviewed.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“six studies (23%) were based in either the USA or Canada”
- Also stated in: § Results · plain
“or North America (23%)”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of health care settings, 21 studies were hospital-based”
- Also stated in: § Results · plain
“and were largely hospital-based (70%)”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All these evaluations compared a new intervention against current practice.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common focus of the reviewed improvement studies was staff reconfigurations within a clinical area from medical to nursing staff”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for implementation studies, the most common focus was on implementation strategies of new care pathways or novel services.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The best performing QHES dimensions were the methodological dimensions.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Quality appraisal
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Comparison between economic studies identified in a previous review carried out by Hoomans et al. (covering the immediately preceding period 1998 to 2004) with those identified in this review (2004 to 2016) shows evidence of a general improvement in quality over the past two decades”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Quality of the evidence
What the sources say
- Not checked yetcites Hoomans et al.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 17
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Three of the included studies were literature reviews [11, 13, 36].”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
What the sources say
- Partly supportedcites [11] Umscheid CA, Williams K, Brennan PJ. Hospital-based comparative effectiveness centers: translating research into practice to improve the quality, safety and value of patient care. J Gen Intern Med. 2010;25(12):1352–5. sourceExperimental — accuracy not yet measured
The paper’s Abstract (pp. 1352–1355) synthesizes published evidence about hospital-based comparative-effectiveness centers, which supports describing it as an overview of the literature. It also presents the authors’ experience and recommendations, and our copy does not identify it as a literature review or describe a literature-review method. This source alone cannot establish the citing paper’s count of three. (LLM, conf=0.76)
- Not checked yetcites [13] Faulkner A, Mills N, Bainton D, Baxter K, Kinnersley P, Peters TJ. Sharp D. a systematic review of the effect of primary care-based service innovations on quality and patterns of referral to specialist secondary care. Br J Gen Pract. 2003;53(496):878–84.
- Supportedcites [36] Mdege ND, Chindove S, Ali S. The effectiveness and cost implications of task-shifting in the delivery of antiretroviral therapy to HIV-infected patients: a systematic review. Health Policy Plan. 2012;28(3):223–36. source
This is an identification citation: the cited paper is itself a literature review, as its title identifies it as a systematic review. Its Literature search section describes searching for and selecting studies. The sentence refers to studies included in the citing document, not studies included within this review. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“four from Australia and one each from Ethiopia, a subset of African countries (Uganda, Kenya and South Africa) and Malaysia.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seventeen of the studies (62%) were European-based (mostly from the UK—12 studies)”
- Also stated in: § Results · plain
“Studies were conducted mostly in Europe (62%)”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, the initial search strategy identified 1668 articles, of which 1566 were excluded, 1525 during the initial screen and 41 following full text screening.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Studies included
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sixteen of the included studies were identified as ‘improvement’ studies (see Table 3, panel 1a) and 14 were identified as ‘implementation’ studies (see Table 3, panel 1b).”
- Also stated in: § Results · plain
“The field was split between improvement (N = 16) and implementation (N = 14) studies.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plainn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Results · plain
None of the sources cited for this claim have been checked yet.
In the paper
“There were also four further CUAs that used economic modelling rather than empirical methods [8–10, 34], and two cost-consequence analyses [16, 35].”
- Also stated in: § Results · plain
“We found four cost utility analyses that used economic modelling rather than empirical methods. Two cost-consequence analyses were also found.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
What the sources say
- Not checked yetcites [8] Burr JM, Mowatt G, Hernández R, Siddiqui MA, Cook J, Lourenco T, Ramsay C, Vale L, Fraser C, Azuara-Blanco A, Deeks J. The clinical effectiveness and cost-effectiveness of screening for open angle glaucoma: a systematic review and economic evaluation. Health Technol Assess (Winch. Eng.). 2007;11(41):1–90.
- Not checked yetcites [9] Burr JM, Botello-Pinzon P, Takwoingi Y, Hernández R, Vazquez-Montes M, Elders A, Asaoka R, Banister K, van der Schoot J, Fraser C, King A, Lemij H, Sanders R, Vernon S, Tuulonen A, Kotecha A, Glasziou P, Garway-Heath D, Crabb D, Vale L, Azuara-Blanco A, Perera R, Ryan M, Deeks J, Cook J. Surveillance for ocular hypertension : an evidence synthesis and economic evaluation. Health Technol Assess. 2012;16(29):1–272. source
- Not checked yetcites [10] Robertson C, Arcot Ragupathy SK, Boachie C, Dixon JM, Fraser C, Hernández R, et al. The clinical effectiveness and cost-effectiveness of different surveillance mammography regimens after the treatment for primary breast cancer: systematic reviews registry database analyses and economic evaluation. Health Technol Assess (Winch. Eng.). 2011;15(34):v–322.
- Not checked yetcites [34] Hernández RA, Jenkinson D, Vale L, Cuthbertson BH. Economic evaluation of nurse-led intensive care follow-up programmes compared with standard care: the PRaCTICaL trial. Eur J Health Econ. 2014;15(3):243–52. source
- Not checked yetcites [16] Dawes HA, Docherty T, Traynor I, Gilmore DH, Jardine AG, Knill-Jones R. Specialist nurse supported discharge in gynaecology: a randomised comparison and economic evaluation. Eur J Obstet Gynecol Reprod Biol. 2007;130(2):262–70. source
- Not checked yetcites [35] Karnon J, Partington A, Horsfall M, Chew D. Variation in clinical practice: a priority setting approach to the staged funding of quality improvement. Appl Health Econ Health Policy. 2016;14(1):21–7. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reasons for exclusion were as follows: the study did not include implementation or quality improvement research aspects (575); it did not include economic aspects (447); was not within a health care/public health setting (437); it was in a language other than English (22); it was incomplete (19); or it was not a full refereed publication (e.g. conference abstracts, doctoral theses) (37).”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Studies included
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Nineteen studies were ex-post economic evaluations of which 17 were CEAs with one CUA [7, 12, 14, 15, 17] [18–30, 33].”
- Also stated in: § Results · plain
“Most studies (N = 19) were ex-post economic evaluations carried out empirically—of those, 17 were cost effectiveness analyses.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
What the sources say
- Not checked yetcites [7] Latour CHM, Bosmans JE, van Tulder MW, de Vos R, Huyse FJ, de Jonge P, et al. Cost-effectiveness of a nurse-led case management intervention in general medical outpatients compared with usual care: an economic evaluation alongside a randomized controlled trial. J Psychosom Res. 2007;62(3):363–70. source
- Not checked yetcites [12] Albers-Heitner CP, Joore MA, Winkens RA, Lagro-Janssen AL, Severens JL, Berghmans LC. Cost-effectiveness of involving nurse specialists for adult patients with urinary incontinence in primary care compared to care-as-usual: an economic evaluation alongside a pragmatic randomized controlled trial. Neurourol Urodyn. 2012;31(4):526–34. source
- Not checked yetcites [14] Bauer JC. Nurse practitioners as an underutilized resource for health reform: evidence-based demonstrations of cost-effectiveness. J Am Acad Nurse Pract. 2010;22(4):228–31. source
- Not checked yetcites [15] Brunenberg DE, van Steyn MJ, Sluimer JC, Bekebrede LL, Bulstra SK, Joore MA. Joint recovery programme versus usual care: an economic evaluation of a clinical pathway for joint replacement surgery. Med Care. 2005:1018–26. source
- Not checked yetcites [17] Furze G, Cox H, Morton V, Chuang LH, Lewin RJ, Nelson P, Carty R, Norris H, Patel N, Elton P. Randomized controlled trial of a lay-facilitated angina management programme. J Adv Nurs. 2012;68(10):2267–79. source
- Not checked yetcites [18] Judd WR, Stephens DM, Kennedy CA. Clinical and economic impact of a quality improvement initiative to enhance early recognition and treatment of sepsis. Ann Pharmacother. 2014;48(10):1269–75. source
- Not checked yetcites [19] Kifle YA, Nigatu TH. Cost-effectiveness analysis of clinical specialist outreach as compared to referral system in Ethiopia: an economic evaluation. Cost Eff Resource Allocation. 2010;8(1):13. source
- Not checked yetcites [20] Kilpatrick K, Kaasalainen S, Donald F, Reid K, Carter N, Bryant-Lukosius D, Martin-Misener R, Harbman P, Marshall DA, Charbonneau-Smith R, DiCenso A. The effectiveness and cost-effectiveness of clinical nurse specialists in outpatient roles: a systematic review. J Eval Clin Pract. 2014;20(6):1106–23. source
- Not checked yetcites [21] Maloney S, Haas R, Keating JL, Molloy E, Jolly B, Sims J, Morgan P, Haines T. Breakeven, cost benefit, cost effectiveness, and willingness to pay for web-based versus face-to-face education delivery for health professionals. J Med Internet Res. 2012;14(2):e47. source
- Not checked yetcites [22] Mortimer D, French SD, McKenzie JE, Denise AO, Green SE. Economic evaluation of active implementation versus guideline dissemination for evidence-based care of acute low-back pain in a general practice setting. PLoS One. 2013;8(10):e75647. source
plainn = 19- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“all of the studies included in our review were empirical and ex-post in nature.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Reflections on the evidence
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“diminished through use of post-hoc subgroup analysis, limited reporting of the handling of uncertainty and justification for choice of discount rates.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Abstract > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Taken together, these criteria are stricter than those applied to prior reviews, which were more inclusive of qualitative outcomes and costing studies.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Strengths and limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Scenario analysis for rollout or scaling up was included in three of the studies, and potential funding sources were considered by one study.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Justification for the outcome measures used was not reported in any of the studies included in Hoomans et al. but reported in 68% of studies included here.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Quality of the evidence
What the sources say
- Not checked yetcites Hoomans et al.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“only 42% of studies reviewed by Hoomans et al. included evaluation of costs and outcomes against ‘standard practice/status quo’ comparators, compared to 95% of studies in our review.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Quality of the evidence
What the sources say
- Not checked yetcites Hoomans et al.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These databases were chosen to attempt to capture the widest range of health improvement, social scientific and health economic studies.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Search strategy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The reference lists of the studies were checked for any related studies that were not picked up by the search.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Screening
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The QHES instrument was designed to more easily tell the difference between high-quality and low-quality studies [5].”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Quality appraisal
What the sources say
- Not checked yetcites [5] Ofman JJ, Sullivan SD, Neumann PJ, Chiou C-F, Henning JM, Wade SW, et al. Examining the value and quality of health economic analyses: implications of utilizing the QHES. J Manag Care Pharm. 2003;9(1):53–61. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Each paper’s methodological quality was assessed using the Quality of Health Economic Studies (QHES) standardised framework [4].”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Quality appraisal
What the sources say
- Not checked yetcites [4] Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Int J Technol Assess Health Care. 2005;21(1):149. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate for potential selection bias after screening, keyword searching was done in Rayyan for the main keywords within the excluded categories (primarily, those that were deemed to be topic-relevant but not containing economic methods). These were then re-screened by the first author.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Inclusion and exclusion criteria
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A systematic review methodology was undertaken.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Search strategy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These were cross-checked for accuracy during the data extraction stage.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Data extraction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“As per standard practice using this framework, the studies were deemed to be of good quality if they attained a score of 75/100 or higher [5].”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Quality appraisal
What the sources say
- Not checked yetcites [5] Ofman JJ, Sullivan SD, Neumann PJ, Chiou C-F, Henning JM, Wade SW, et al. Examining the value and quality of health economic analyses: implications of utilizing the QHES. J Manag Care Pharm. 2003;9(1):53–61. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also only considered papers reported in English.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Strengths and limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though there remains scope for continued improvement in key areas and for increased collaboration between health economics and implementation science.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The strict inclusion criteria mean that the reviewed evidence is cohesive.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Strengths and limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results of evaluating the 30 included papers paint a picture of an area of research that is still developing.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Reflections on the evidence
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The systematic appraisal we carried out also allows us a longitudinal critique of the quality of economic studies in this field.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Strengths and limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Existing implementation frameworks [40] make reference to the need to consider costs as part of an evaluative research strategy, but do not specify how this is to be done.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Implications for implementation and improvement research and future directions
What the sources say
- Not checked yetcites [40] Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38(2):65–76. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“we would argue that the QHES used here is based on Drummond’s guidelines (used in prior reviews) and is designed to cover the same topics, but offers a simpler, quantifiable format that is easier to apply. [32]”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Strengths and limitations
What the sources say
- Not checked yetcites [32] Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, Augustovski F, Briggs AH, Mauskopf J, Loder E. Consolidated health economic evaluation reporting standards (CHEERS) statement. Cost Eff Resource Allocation. 2013;11(1):6. source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“however, quality is overall good.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Abstract > Conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, the expanding application of health economic evaluations within the improvement and implementation sphere where high-quality study reporting has been a major recent focus has also plausibly contributed to improved reporting.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Quality of the evidence
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
The relationship between implementation outcomes, service outcomes and patient outcomes is central to understanding the benefits and costs and overall cost-effectiveness of an intervention.
“The relationship between implementation outcomes, service outcomes and patient outcomes is central to understanding the benefits and costs and overall cost-effectiveness of an intervention.”
The paper relies on this without support.
Economic evaluation can inform choices over whether and how resources should be allocated to improve services and for implementing evidence into health care practice.
“Economic evaluation can inform choices over whether and how resources should be allocated to improve services and for implementing evidence into health care practice.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Almost all implementation research involving financing strategies [24] or economic evaluations [2, 3] has focused on active implementation, yet all phases are relevant to financing decisions.”
From Translating economic evaluations into financing strategies for implementing evidence-based practices (2021) · cites this paper as [3]
Experimental — accuracy not yet measuredIn the citing paper
“Implementation researchers are increasingly using economic evaluation to explore the benefits produced by implementing evidence-based practices (EBPs) in healthcare settings.”
From Translating economic evaluations into financing strategies for implementing evidence-based practices (2021) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
“There were also four further CUAs that used economic modelling rather than empirical methods [8–10, 34], and two cost-consequence analyses [16, 35].”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Discussion > Reflections on the evidence
“we found that all of the studies included in our review were empirical and ex-post in nature.”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“In total, the initial search strategy identified 1668 articles, of which 1566 were excluded, 1525 during the initial screen and 41 following full text screening.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Studies included
Reported 30 · recomputed 102
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Reasons for exclusion were as follows: the study did not include implementation or quality improvement research aspects (575); it did not include economic aspects (447); was not within a health care/public health setting (437); it was in a language other than English (22); it was incomplete (19); or it was not a full refereed publication (e.g. conference abstracts, doctoral theses) (37).”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Studies included
Reported 1566 · recomputed 1537
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Seventeen of the studies (62%) were European-based”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
Reported 62 · recomputed 56.666667
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“six studies (23%) were based in either the USA or Canada”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
Reported 23 · recomputed 20
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Thirteen of the studies (62%) attained a ‘good’ score of over 75.”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Quality appraisal
Reported 62 · recomputed 59.090909
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
“| Faulkener et al | 2003 | UK | Improvement | Primary Care | Improved referral | N/A | Review | Review | N/A |”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Methods > Inclusion and exclusion criteria
“Were published after 2003 and before 16th March 2016”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Descriptive analysis of the evidence base
“Scenario analysis for rollout or scaling up was included in three of the studies”
§ Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review > Results > Quality appraisal
“| Maloney et al. 2012 | Training and set up costs | Roll out scenarios | Yes | | Mdege et al. 2012 | Training costs | Roll out scenarios | Yes | | Mortimer et al. 2013 | Development costs; amortisation; delivery costs; roll out costs | Roll out scenarios | No | | Purshouse et al. 2013 | None | Roll out scenarios | Yes, although sensitive to rollout costs |”
Citations
Cites
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015) · Implementation Science · doi:10.1186/s13012-015-0209-1 · Cited by 4,963
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J., Thomas, R., MacLennan, G., Fraser, C., Ramsay, C., Vale, L., Whitty, P., Eccles, M., Matowe, L., Shirran, L., et al. (2004) · Health Technology Assessment · doi:10.3310/hta8060 · Cited by 3,044
Guidelines for authors and peer reviewers of economic submissions to the BMJ
Drummond, M. F., & Jefferson, T. O. (1996) · BMJ · doi:10.1136/bmj.313.7052.275 · Cited by 1,827
What is "quality improvement" and how can it transform healthcare?
Batalden, P. B., & Davidoff, F. (2007) · BMJ Quality & Safety · doi:10.1136/qshc.2006.022046 · Cited by 1,104
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J. M., Thomas, R. E., MacLennan, G., Fraser, C., Ramsay, C. R., Vale, L., Whitty, P., Eccles, M. P., Matowe, L., Shirran, L., et al. (2005) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462305290190 · Cited by 864
Burr, J. M., Mowatt, G., Hernández, R. A., Siddiqui, M. A. R., Cook, J. A., Lourenço, T., Ramsay, C. R., Vale, L. D., Fraser, C., Azuara‐Blanco, A., et al. (2007) · Health Technology Assessment · doi:10.3310/hta11410 · Cited by 554
An Overview of Research and Evaluation Designs for Dissemination and Implementation
Brown, C. H., Curran, G., Palinkas, L. A., Aarons, G. A., Wells, K. B., Jones, L., Collins, L. M., Duan, N., Mittman, B. S., Wallace, A., et al. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044215 · Cited by 509
Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement
Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A. H., Mauskopf, J., Loder, E., et al. (2013) · Cost Effectiveness and Resource Allocation · doi:10.1186/1478-7547-11-6 · Cited by 381
Research works printed but not linked: 1
- Elmagarmid A, Fedorowicz Z, Hammady H, Ilyas I, Khabsa M, Ouzzani M. Rayyan: a systematic reviews web app for exploring and filtering searches for eligible studies for Cochrane reviews. In: Evidence-informed public health: opportunities and challenges. Abstracts of the 22nd Cochrane colloquium. Hyderabad: Wiley; 2014. p. 21–6.
Cited by
Sorted by citations
Beidas, R. S., Dorsey, S., Lewis, C. C., Lyon, A. R., Powell, B. J., Purtle, J., Saldana, L., Shelton, R. C., Stirman, S. W., & Lane-Fall, M. B. (2022) · Implementation Science · doi:10.1186/s13012-022-01226-3 · Cited by 230
A pragmatic method for costing implementation strategies using time-driven activity-based costing
Cidav, Z., Mandell, D., Pyne, J., Beidas, R., Curran, G., & Marcus, S. (2020) · Implementation Science · doi:10.1186/s13012-020-00993-1 · Cited by 174
Cost data in implementation science: categories and approaches to costing
Gold, H. T., McDermott, C., Hoomans, T., & Wagner, T. H. (2022) · Implementation Science · doi:10.1186/s13012-021-01172-6 · Cited by 152
Sohn, H., Tucker, A., Ferguson, O., Gomes, I., & Dowdy, D. W. (2020) · Implementation Science · doi:10.1186/s13012-020-01047-2 · Cited by 116
A Review of Interventions to Enhance HIV Medication Adherence
Whiteley, L., Olsen, E. M., Haubrick, K. K., Odoom, E., Tarantino, N., & Brown, L. K. (2021) · Current HIV/AIDS Reports · doi:10.1007/s11904-021-00568-9 · Cited by 95
de la Perrelle, L., Radisic, G., Cations, M., Kaambwa, B., Barbery, G., & Laver, K. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-4981-5 · Cited by 90
Aggarwal, N. K., Jarvis, G. E., Gómez-Carrillo, A., Kirmayer, L. J., & Lewis‐Fernández, R. (2020) · Transcultural Psychiatry · doi:10.1177/1363461520940481 · Cited by 68
Bowser, D. M., Henry, B. F., & McCollister, K. E. (2021) · Implementation Science · doi:10.1186/s13012-021-01094-3 · Cited by 61
The cost of oral cancer: A systematic review
Ribeiro‐Rotta, R. F., Rosa, E. A., Milani, V., Dias, N. R., Masterson, D., da Silva, E. N., & Zara, A. L. D. S. A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0266346 · Cited by 59
Extending analytic methods for economic evaluation in implementation science
O’Leary, M. C., Lich, K. H., Frerichs, L., Leeman, J., Reuland, D. S., & Wheeler, S. B. (2022) · Implementation Science · doi:10.1186/s13012-022-01192-w · Cited by 58
Wolcherink, M. O. O., Behr, C., Pouwels, X., Doggen, C., & Koffijberg, H. (2023) · PharmacoEconomics (Auckland) · doi:10.1007/s40273-023-01287-2 · Cited by 54
Translating economic evaluations into financing strategies for implementing evidence-based practices
Dopp, A. R., Kerns, S. E. U., Panattoni, L., Ringel, J. S., Eisenberg, D., Powell, B. J., Low, R., & Raghavan, R. (2021) · Implementation Science · doi:10.1186/s13012-021-01137-9 · Cited by 52
Wilson, P. M., & Kislov, R. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009237055 · Cited by 58
A guide to systems-level, participatory, theory-informed implementation research in global health
Seward, N., Hanlon, C., Hinrichs, S., Lund, C., Murdoch, J., Salisbury, T. T., Verhey, R., Shidhaye, R. R., Thornicroft, G., Araya, R., et al. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-005365 · Cited by 55
Self-citationXu, R. H., Keetharuth, A. D., Wang, L.-L., Cheung, A. W.-L., & Wong, E. L. (2021) · The European Journal of Health Economics · doi:10.1007/s10198-021-01359-0 · Cited by 51
Implementation costs of hospital-based computerised decision support systems: a systematic review
Donovan, T., Abell, B., Fernando, M., McPhail, S., & Carter, H. E. (2023) · Implementation Science · doi:10.1186/s13012-023-01261-8 · Cited by 44
Krebs, E., & Nosyk, B. (2021) · Current HIV/AIDS Reports · doi:10.1007/s11904-021-00550-5 · Cited by 39
Collaborating with health economists to advance implementation science: a qualitative study
Barnett, M. L., Dopp, A. R., Klein, C. C., Ettner, S. L., Powell, B. J., & Saldana, L. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00074-w · Cited by 38
Salvati, Z. M., Rahm, A. K., Williams, M. S., Ladd, I., Schlieder, V., Atondo, J., Schneider, J. L., Epstein, M. M., Lu, C. Y., Pawloski, P. A., et al. (2023) · Implementation Science Communications · doi:10.1186/s43058-023-00424-4 · Cited by 34
Malhotra, A., Thompson, R. R., Kagoya, F., Masiye, F., Mbewe, P., Mosepele, M., Phiri, J., Sambo, J., Barker, A. R., Cameron, D., et al. (2022) · Implementation Science · doi:10.1186/s13012-022-01248-x · Cited by 33
Yeung, K., Richards, J. A., Goemer, E., Lozano, P., Lapham, G. T., Williams, E. C., Glass, J. E., Lee, A., Achtmeyer, C. E., Caldeiro, R. M., et al. (2020) · Health Services Research · doi:10.1111/1475-6773.13592 · Cited by 32
Michaud, T. L., Pereira, E. L., Porter, G. C., Golden, C. A., Hill, J. L., Kim, J., Wang, H., Schmidt, C., & Estabrooks, P. A. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-060785 · Cited by 29
Rethinking How We Measure Costs in Implementation Research
Wagner, T. H. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-06104-6 · Cited by 23
Using decision analysis to support implementation planning in research and practice
Smith, N., Knocke, K., & Lich, K. H. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00330-1 · Cited by 27
Vaughn, A., Studts, C. R., Powell, B. J., Ammerman, A. S., Trogdon, J. G., Curran, G. M., Hales, D. P., Willis, E. A., & Ward, D. S. (2019) · Implementation Science · doi:10.1186/s13012-019-0949-4 · Cited by 21
Harden, S. M., Balis, L. E., Strayer, T. E., & Wilson, M. L. (2021) · Preventing Chronic Disease · doi:10.5888/pcd18.200513 · Cited by 21
“Cost‐effectiveness of ACL treatment is dependent on age and activity level: a systematic review”
Deviandri, R., van der Veen, H. C., Lubis, A. M. T., van den Akker‐Scheek, I., & Postma, M. J. (2022) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-022-07087-z · Cited by 20
Nguyen, M. X. B., Chu, A. V., Powell, B. J., Tran, H. V., Nguyen, L. H., Dao, A. T. M., Pham, M. D., Vo, S. H., Bui, N. H., Dowdy, D. W., et al. (2020) · Implementation Science · doi:10.1186/s13012-020-01020-z · Cited by 19
Huebschmann, A. G., Glasgow, R. E., Leavitt, I. M., Chapman, K., Rice, J. D., Lockhart, S., Stevens‐Lapsley, J. E., Reusch, J. E. B., Dunn, A. L., & Regensteiner, J. G. (2022) · Translational Behavioral Medicine · doi:10.1093/tbm/ibac014 · Cited by 18
Thusini, S., Milenova, M., Nahabedian, N., Grey, B., Soukup, T., & Henderson, C. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08171-3 · Cited by 17
Cidav, Z., Mandell, D. S., Ingersoll, B. R., & Pellecchia, M. (2023) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-022-01247-6 · Cited by 17
Scholl, I., Kobrin, S. C., & Elwyn, G. J. (2020) · Implementation Science · doi:10.1186/s13012-020-01042-7 · Cited by 17
Szewczyk, Z., Reeves, P., Kingsland, M., Doherty, E. L., Elliott, E. J., Wolfenden, L., Tsang, T. W., Dunlop, A. J., Searles, A., & Wiggers, J. H. (2022) · Implementation Science · doi:10.1186/s13012-021-01180-6 · Cited by 16
The economics of adaptations to evidence-based practices
Salloum, R. G., Wagner, T. H., Midboe, A. M., Daniels, S. I., Quanbeck, A., & Chambers, D. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00345-8 · Cited by 18
Hartzler, B. J., Hinde, J. M., Lang, S. G., Correia, N., Yermash, J., Yap, K., Murphy, C. M., Ruwala, R., Rash, C. J., Becker, S. J., et al. (2022) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-022-00283-1 · Cited by 18
Obesity prevention practices in early care and education settings: an adaptive implementation trial
Swindle, T., Rutledge, J. M., Selig, J. P., Painter, J. T., Zhang, D., Martin, J., Johnson, S. L., Whiteside-Mansell, L., Almirall, D., McElwee, T. M. B., et al. (2022) · Implementation Science · doi:10.1186/s13012-021-01185-1 · Cited by 15
Huebschmann, A. G., Trinkley, K. E., Gritz, M., & Glasgow, R. E. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00292-4 · Cited by 15
Belay, Y. B., Engel, L., Lee, Y. Y., Le, N., & Mihalopoulos, C. (2023) · PharmacoEconomics · doi:10.1007/s40273-023-01257-8 · Cited by 16
Yeshitila, Y. G., Gold, L. H., Abimanyi-Ochom, J., Riggs, E., Tolossa, T., & Le, H. (2024) · Social Science & Medicine · doi:10.1016/j.socscimed.2024.117250 · Cited by 14
Sáenz‐Ravello, G., Castillo‐Riquelme, M., Cuadrado, C., Gamonal, J. A., & Baeza, M. (2024) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.12999 · Cited by 14
Donovan, T., Abell, B., McPhail, S., & Carter, H. E. (2025) · Implementation Science · doi:10.1186/s13012-025-01423-w · Cited by 14
Bartáková, J., Zúñiga, F., Guerbaaï, R.-A., Basinska, K., Brunkert, T., Simon, M., Denhaerynck, K., Geest, S. D., Wellens, N. I. H., Serdaly, C., et al. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03182-5 · Cited by 14
Barnett, M. L., Stadnick, N. A., Proctor, E. K., Dopp, A. R., & Saldana, L. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00239-1 · Cited by 15
Karstens, S., Lang, S., & Saunders, B. (2020) · Health Services Insights · doi:10.1177/1178632920977894 · Cited by 13
Munishi, C. G., Mateshi, G., Mlunde, L. B., Njiro, B. J., Ngowi, J. E., Kengia, J. T., Kapologwe, N., Deng, L., Timbrell, A., Kitinya, W. J., et al. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001487 · Cited by 12
Ahern, S., Riordan, F., Murphy, A., Browne, J. P., Kearney, P. M., Smith, S. M., & McHugh, S. (2021) · Implementation Science · doi:10.1186/s13012-021-01085-4 · Cited by 12
Reporting standards, outcomes and costs of quality improvement studies in Ireland: a scoping review
McCarthy, S. E., Jabakhanji, S. B., Martin, J., Flynn, M. A., & Sørensen, J. R. (2021) · BMJ Open Quality · doi:10.1136/bmjoq-2020-001319 · Cited by 12
Nelson, R. E., Goto, M., Samore, M. H., Jones, M., Stevens, V. W., Evans, M. E., Schweizer, M. L., Perencevich, E. N., & Rubin, M. A. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa1591 · Cited by 12
Jorm, C. M., Iedema, R. A. M., Piper, D. A., Goodwin, N., & Searles, A. (2021) · Journal of Health Organization and Management · doi:10.1108/jhom-06-2020-0218 · Cited by 12
Rejon, C. S., Hollingworth, W., Opmeer, B. C., Luyt, K., & McLeod, H. (2025) · Applied Health Economics and Health Policy · doi:10.1007/s40258-025-00993-6 · Cited by 11
Lehman, V. E. R., Siegel, J. E., & Chiang, E. N. (2023) · Medical Care · doi:10.1097/mlr.0000000000001873 · Cited by 11
Kirchner, J. E., Waltz, T. J., Powell, B. J., Woodward, E. N., Smith, J. L., & Proctor, E. K. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197660690.003.0006 · Cited by 11
Burns, C. L., Wishart, L. R., Kularatna, S., & Ward, E. C. (2020) · Speech Language and Hearing · doi:10.1080/2050571x.2019.1693750 · Cited by 11
Economic Evaluation of Implementation Strategies
Severens, J. L., Hoomans, T., Adang, E., & Wensing, M. (2020) · Wiley eBooks · doi:10.1002/9781119488620.ch23 · Cited by 11
Hewage, S. A., Noviyani, R., Brain, D., Sharma, P., Parsonage, W. A., McPhail, S., Barnett, A. G., & Kularatna, S. (2023) · Cost Effectiveness and Resource Allocation · doi:10.1186/s12962-023-00486-0 · Cited by 10
Carrandi, A., Grove, A., Skouteris, H., Melder, A., Hu, Y., Dever, M., & Higgins, A. M. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00562-3 · Cited by 10
Brown, V., Tran, H. N. Q., Blake, M. R., Rachel, L., & Moodie, M. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-00633-6 · Cited by 10
Hinde, J. M., Garner, B. R., Watson, C. J., Ramanan, R., Ball, E. L., & Tueller, S. (2022) · Implementation Research and Practice · doi:10.1177/26334895221089266 · Cited by 10
Wang, Y., Guo, D., Xia, Y., Hu, M., Wang, M., Shi, Z., Guan, X., Zhu, D., & He, P. (2025) · Nature Communications · doi:10.1038/s41467-025-58120-x · Cited by 10
Heggie, R., Boyd, K., & Wu, O. (2021) · Health Research Policy and Systems · doi:10.1186/s12961-021-00766-2 · Cited by 9
Falkenbach, P., Raudasoja, A., Vernooij, R. W. M., Mustonen, J. M. J., Agarwal, A., Aoki, Y., Blanker, M. H., Cartwright, R., García‐Perdomo, H. A., Kilpeläinen, T. P., et al. (2023) · Implementation Science · doi:10.1186/s13012-023-01290-3 · Cited by 9
Heggie, R., Boyd, K., Kamaruzaman, H. F., & Wu, O. (2024) · Health Research Policy and Systems · doi:10.1186/s12961-024-01220-9 · Cited by 9
Uzoaru, F., Nwaozuru, U., Ong, J. J., Obi, F. A., Obiezu‐Umeh, C., Tucker, J. D., Shato, T., Mason, S., Carter, V., Manu, S., et al. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00177-y · Cited by 10
Gyllensten, H., Tistad, M., Fridberg, H., & Wallin, L. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-073829 · Cited by 8
Food consumption and mental health in children and adolescents: A systematic review protocol
Costa, M. B., Silva, R. M. F., Silva, K. V. C., Nogueira, S. M., Terra, L. F., Cordeiro, L. L., Villa‐González, E., Noll, P. R. E. S., & Noll, M. A. (2024) · MethodsX · doi:10.1016/j.mex.2024.103015 · Cited by 8
Donovan, T., Carter, H. E., McPhail, S., & Abell, B. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00648-y · Cited by 8
Garner, B. R., Patel, S. V., & Kirk, M. A. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01535-y · Cited by 8
Tozan, Y., Capasso, A., Namatovu, P., Kiyingi, J., Damulira, C., Nabayinda, J., Bahar, O. S., McKay, M. M., Hoagwood, K., & Ssewamala, F. M. (2022) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.21-0895 · Cited by 7
Michaud, T. L., Hill, J. L., Heelan, K. A., Bartee, R. T., Abbey, B. M., Malmkar, A., Masker, J., Golden, C. A., Porter, G. C., Glasgow, R. E., et al. (2022) · Implementation Science Communications · doi:10.1186/s43058-022-00287-1 · Cited by 7
Ling, V. B., Levi, E. E., Harrington, A. R., Zite, N. B., Rivas, S. D., Dalton, V. K., Smith, R. D., & Moniz, M. H. (2020) · BMJ Quality & Safety · doi:10.1136/bmjqs-2020-011111 · Cited by 7
Bankole, A. O., Conombo, B., Légaré, F., & Laberge, M. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13024-w · Cited by 6
Implementation science on the margins: How do we demonstrate the value of implementation strategies?
Hinde, J. M., Bray, J. W., & Cowell, A. J. (2020) · Families Systems & Health · doi:10.1037/fsh0000535 · Cited by 6
The Role of Economic Evaluation in Dissemination and Implementation Research
Dopp, A. R., Walker, S., & Raghavan, R. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197660690.003.0011 · Cited by 6
Who Conducts Health Economic Evaluations of Laboratory Tests? A Scoping Review
Giusepi, I., John, A. S., & Jülicher, P. (2020) · The Journal of Applied Laboratory Medicine · doi:10.1093/jalm/jfaa107 · Cited by 6
Chen, J., Wijesundara, J. G., Patterson, A. M., Cutrona, S. L., Aiello, S., McManus, D. D., McKee, M. D., Wang, B., & Houston, T. K. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-07031-w · Cited by 6
Economic evaluation methods used in home‐visiting interventions: A systematic search and review
Bailey, C., Skouteris, H., Morris, H., O’Donnell, R., Hill, B., & Ademi, Z. (2021) · Health & Social Care in the Community · doi:10.1111/hsc.13349 · Cited by 6
Jülicher, P., O’Kane, M., Price, C. P., Christenson, R. H., & John, A. S. (2021) · Annals of Clinical Biochemistry International Journal of Laboratory Medicine · doi:10.1177/00045632211013852 · Cited by 5
Brown, V., Tran, H. N. Q., Williams, J. W., Laws, R., & Moodie, M. (2022) · BMC Public Health · doi:10.1186/s12889-022-13754-0 · Cited by 5
Pilz, M. J., Seyringer, S., Singer, S., Ioannidis, G., Sykiotis, G. P., Arrarás, J. I., Husson, O., Iakovou, I. P., Fanetti, G., Führer, D., et al. (2024) · Thyroid · doi:10.1089/thy.2024.0396 · Cited by 5
Bavor, C., Brotherton, J. M. L., Smith, M. A., Prang, K.-H., McDermott, T. L., Rankin, N. M., Zammit, C. M., Jennett, C. J., Sultana, F., Machalek, D. A., et al. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-10040-6 · Cited by 4
The Costs of Implementing a Conversation Aid for Uterine Fibroids in Multiple Health Care Settings
Acquilano, S. C., Forcino, R. C., Schubbe, D. C., Engel, J., Tomaino, M., Johnson, L. C., Durand, M., & Elwyn, G. J. (2023) · Medical Care · doi:10.1097/mlr.0000000000001897 · Cited by 4
Mapping Organizational Theory With SCRIPTS
Trinidad, J. E. (2025) · Sociology Compass · doi:10.1111/soc4.70059 · Cited by 5
Carter, H. E., Donovan, T., White, N. M., Lee, X. J., Brown, C., Graves, N., McPhail, S., Cardona, M., White, B. P., Willmott, L., et al. (2025) · Age and Ageing · doi:10.1093/ageing/afaf280 · Cited by 3
Amritlal, S. T., Chandler, R., Ahari, A. M., Paterson, L., Avery, A. J., Ashcroft, D. M., Chuter, A., & Elliott, R. A. (2025) · PharmacoEconomics · doi:10.1007/s40273-025-01567-z · Cited by 3
Cost-effectiveness of an educational intervention to reduce potentially inappropriate medication
Rantsi, M., Pitkälä, K. H., Kautiainen, H., Hyttinen, V., & Kankaanpää, E. (2022) · Age and Ageing · doi:10.1093/ageing/afac112 · Cited by 3
Wynne, O., Szewczyk, Z., Hollis, J., Farragher, E., Doherty, E. L., Tully, B., Paolucci, F., Gillham, K. E., Reeves, P., Wiggers, J. H., et al. (2023) · Implementation Science Communications · doi:10.1186/s43058-023-00420-8 · Cited by 3
Early Rehabilitation Bundle in a Canadian PICU: Cost Analysis of Implementation in 2018–2020
Gertsman, S., Pavalagantharajah, S., Falk, L., Borhan, S., Kennedy, K., Thabane, L., Xie, F., Cupido, C. M., & Choong, K. L. M. (2025) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003806 · Cited by 3
Rejon, C. S., Hollingworth, W., McLeod, H., Opmeer, B. C., & Luyt, K. (2023) · Research Square · doi:10.21203/rs.3.rs-2678140/v1 · Cited by 3
Rejon, C. S., Hollingworth, W., McLeod, H., Opmeer, B. C., & Luyt, K. (2023) · Research Square · doi:10.21203/rs.3.rs-2678140/v2 · Cited by 3
Zomer, E., Tan, E. J., Kim, J., Turner, T., Cadilhac, D. A.-M., Hill, K. M., Zoungas, S., McGloughlin, S., Le, L. L., & Millard, T. N. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5382385 · Cited by 3
Economic Considerations in Oral Health: Bridging Gaps, Broadening Horizons
Benzian, H., & Schwendicke, F. (2024) · Journal of Dental Research · doi:10.1177/00220345241286462 · Cited by 2
Blackburn, N. A., Philbrick, S. M., Patel, S. V., Cance, J. D., Elek, E., Gibbons, B., Khan, S. S., Lambdin, B. H., & Graham, P. (2025) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2025.102610 · Cited by 2
Resource use and in-hospital costs after aneurysmal subarachnoid hemorrhage in the Netherlands
Mulder, F., van Dijck, J. T. J. M., Vreeburg, R. J. G., Engels, C. C., & Moojen, W. A. (2025) · Brain and Spine · doi:10.1016/j.bas.2025.104400 · Cited by 2
Szewczyk, Z., MACDONALD, H. M., Pinheiro, M. B., Nettlefold, L., Sims‐Gould, J., & McKay, H. A. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01826-4 · Cited by 2
Brundisini, F. K., Zomahoun, H. T. V., Légaré, F., Rhéault, N., Uwizeye, C. B., Massougbodji, J., Gogovor, A., Tchoubi, S., Assan, O. Q., & Laberge, M. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-050838 · Cited by 2
Keim‐Malpass, J., Heysell, S. K., Thomas, T. A., Lobo, J. M., Mpagama, S. G., Muzoora, C. K., & Moore, C. C. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20065041 · Cited by 2
Jabbar, S. A., Frödin, M., Wikström, E., Gillespie, B. M., Gyllensten, H., & Andersson, A. E. (2025) · Antimicrobial Resistance and Infection Control · doi:10.1186/s13756-025-01573-y · Cited by 2
Bartels, S. M., Phan, H. T. T., Hutton, H. E., Nhan, D. T., Sripaipan, T., Chen, J. S., Rossi, S. L., Ferguson, O., Nong, H. T. T., Nguyen, N. T. K., et al. (2024) · Implementation Science · doi:10.1186/s13012-024-01368-6 · Cited by 2
The Need to Conduct Studies on Economic Evaluation of Nursing Care in Iran
Nejatian, A. (2022) · Iran Journal of Nursing · doi:10.32598/ijn.35.139.3158 · Cited by 1
Michaud, T. L., Hill, J. L., Heelan, K. A., Fiechtner, L. G., Ruggiero, C. F., French, D. D., Smith, J. D., Salmon, J., Lindros, J., Jelalian, E., et al. (2025) · Childhood Obesity · doi:10.1177/21532176251389760 · Cited by 1
Metadata
- Authors
- Roberts, S. L. E., Healey, A., & Sevdalis, N.
- Affiliations
- King's College London (GB)
- Corresponding author
- Roberts, Sarah Louise Elin
- Venue
- Implementation Science
- Published
- 2019-07-15
- Volume
- 14
- Issue
- 1
- Pages
- 72-72
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC6631608
- OpenAlex
- W2959993899
- Semantic Scholar
- 7ec32b728ccd1fb347c955b4d846cc3dc610e90b
- Cited by
- 161
- References
- 41
- Citations by year
- 2026: 14 · 2025: 20 · 2024: 22 · 2023: 26 · 2022: 30 · 2021: 26 · 2020: 21 · 2019: 1
- FWCI
- 5.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Portal (King's College London) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Funders
- National Institute for Health Research
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Healthcare Quality and Management
- Keywords
- health economic evaluation, implementation research, improvement science, cost-effectiveness analysis, nurse-led care, implementation costs, cost-utility analysis, cost-consequence analysis, hospital care delivery
- MeSH terms
- Implementation Science, Cost-Benefit Analysis, Delivery of Health Care, Humans, Models, Economic, Evidence-Based Practice, Quality Improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
