CoChat

The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers

Open access

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 2 of the 26 sources its claims cite were checked for support.

Abstract

INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhance ments made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.

Overview

CoChat Summary

TL;DR

This paper presents version 2018 of the Mixed Methods Appraisal Tool (MMAT), a single critical appraisal tool for assessing the methodological quality of qualitative, quantitative, and mixed methods studies in mixed studies reviews. It describes the tool's 25 criteria across five study design categories, its development history, and guidance for use.

Key findings

The MMAT version 2018 includes a total of 25 criteria and 2 screening questions, covering five categories of study designs: (a) qualitative, (b) randomized controlled trial, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods, with five core criteria per category rated as yes, no, or can't tell. The tool was developed from a 2006 first version (15 criteria across four categories), revised in 2011, and further revised into the 2018 version based on usefulness testing with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Two interrater reliability studies used 32 and 261 papers respectively and showed the need to clarify criteria, particularly for nonrandomized and qualitative studies. The paper argues the MMAT offers a time-efficient alternative by limiting to core criteria and uniquely includes specific criteria for mixed methods studies, though its criteria are harder to judge because they target methodological rather than reporting quality.

Methods

This is a tool description paper (third in a three-part series on systematic reviews for information professionals) rather than an empirical study; it narrates the development of the MMAT and explains how to apply it. Development drew on a literature review on mixed studies reviews, pilot testing in workshops with four reviewers appraising six studies, two interrater reliability studies (32 and 261 papers), usefulness testing via interviews with 20 researchers, a literature review on critical appraisal tools, and a modified e-Delphi study with 73 methodological experts. Use involves three steps: two optional screening questions, selection of the appropriate design category (or categories), and rating each criterion as yes, no, or can't tell, with detailed presentation of ratings advised over an overall numerical score.

Limitations

The paper states that revision of the MMAT has so far focused on content validity and usefulness, and that further testing on its validity and reliability will be needed; it also notes that modifications may be necessary as evidence develops. It states that MMAT criteria are more difficult to judge than other appraisal tools because they focus on methodological quality rather than reporting quality. Evident from the text: the tool's development evidence base is described only in summary form, with reliability and validity results reported by reference to prior studies rather than presented here.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3233/efi-180221

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 0 Not valid · 18 Not checked · 11 Own results and interpretations

5 more claims

Create a free account to see all 31 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 7
  • Probably not a substantial claim. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

29 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 21 references

Cited by

Items 2,201–2,300 of 4,578

Metadata

Authors
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.
Affiliations
  • McGill University (CA)
  • Universitat Oberta de Catalunya (ES)
  • University of Warwick (GB)
  • University of Canberra (AU)
  • Université de Sherbrooke (CA)
  • Université Laval (CA)
  • University of Sheffield (GB)
  • Institut National de la Recherche Scientifique (CA)
Corresponding author
Pluye, Pierre
Venue
Education for Information
Published
2018-12-18
Volume
34
Issue
4
Pages
285-291
ISSN
  • 0167-8329 (ISSN-L)
  • 1875-8649
OpenAlex
W2901724277
Cited by
4,737
References
21
Citations by year
2026: 1312 · 2025: 974 · 2024: 800 · 2023: 610 · 2022: 501 · 2021: 370 · 2020: 143 · 2019: 22 · 2018: 3
FWCI
102.21
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Education for Information (publishedVersion)
  • O2 - Repositori Institucional (Universitat Oberta de Catalunya) (publishedVersion, cc-by-nc-nd)
  • RECERCAT (Consorci de Serveis Universitaris de Catalunya) (publishedVersion, cc-by-nc-nd)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • HAL (submittedVersion)
  • eScholarship@McGill (McGill) (submittedVersion)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
  • eScholarship@McGill (McGill) (submittedVersion, cc-by-nc-nd)
  • EspaceINRS (National Institute for Scientific Research (Canada)) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Social Sciences › Education › Qualitative Research Methods and Applications
Keywords
Mixed Methods Appraisal Tool, critical appraisal, study quality assessment, mixed methods, systematic reviews, qualitative research, quantitative research, randomized controlled trials, non-randomized studies, e-Delphi, inter-rater reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.