Questions and answers about conducting systematic reviews in sport and exercise psychology

May combine two claims.

Sometimes data are unavailable in primary studies, or are available with insufficient detail to be entered into meta-analysis.

“Alternatively, sometimes data are not available in primary studies, or are available with an insufficient level of detail to be entered into meta-analysis (e.g., means, standard deviations, and correlation tables are not provided).”

Probably not a substantial claim: may not pass S3 (load-bearing).

PRISMA-P was created to help researchers produce robust and replicable systematic review protocols.

“For example, the PRISMA-Protocol (PRISMA-P; Moher et al., 2015) was developed to help researchers create robust and replicable protocols for their systematic reviews.”

• cites Moher et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

PRISMA consists of evidence-based items for reporting reviews of quantitative primary studies.

“PRISMA is comprised of evidence-based items to be considered when reporting reviews of quantitative primary studies (Moher et al., 2009).”

• cites Moher et al., 2009Not checked yet

  Not checked in the free run.

May combine two claims.

Non-systematic reviews lack a standardized or reproducible methodology, tend to be broad, and discuss literature selectively.

“Non-systematic reviews do not follow a standardized or reproducible methodology, are often broad, and involve selectively discussing the literature on a particular topic (Ferrari, 2015).”

• cites Ferrari, 2015Not checked yet

  Not checked in the free run.

May combine two claims.

PRISMA is designed for intervention reviews but can be adapted for observational reviews.

“Although PRISMA is intended for reviews of intervention studies, it can be amended for reviews of observational studies.”

Classified as probably not a substantial claim.

Guidelines and reporting standards exist to outline best practices for robust and replicable systematic reviews.

“Guidelines and reporting standards have been developed to outline best practices for conducting robust and replicable systematic reviews.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Cochrane Qualitative and Implementation Methods group provides guidance for reviews summarizing qualitative studies.

“In conjunction with the Cochrane Handbook, guidance is available from the Cochrane Qualitative and Implementation Methods group (Cochrane, 2018) for conducting reviews to summarize qualitative studies.”

• cites Cochrane, 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Many Cochrane Handbook recommendations may need adaptation for sport and exercise psychology research.

“At this juncture it is important to note that many of the recommendations within the Cochrane Handbook might need to be adapted for sport and exercise psychology research.”

• AssumptionRests on an assumption: Having two independent reviewers extract data might not be feasible in sport and exercise psychology.

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple PRISMA checklists exist for different review types.

“Several different PRISMA checklists exist to cover a range of reviews.”

Classified as probably not a substantial claim.

Systematic reviews employ explicit, systematic, and reproducible methods intended to capture all studies meeting set inclusion and exclusion criteria.

“The methodology used in a systematic review is unlike that used in a non-systematic review because it is characterized by explicit, systematic, and reproducible methods that attempt to capture all studies that meet set inclusion and exclusion criteria (Chandler et al., 2019).”

• cites Chandler et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reviews fall into two categories: non-systematic and systematic.

“Reviews can be classified as either non-systematic or systematic (Ferrari, 2015).”

• cites Ferrari, 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

MOOSE is intended for systematic reviews of primary quantitative observational studies.

“Other examples of tools include the Meta-analysis Of Observational Studies in Epidemiology (MOOSE; Stroup et al., 2000) for systematic reviews that focus on primary quantitative studies that use observational designs (Stroup et al., 2000)”

• cites Stroup et al., 2000Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Cochrane Handbook offers detailed guidance for Cochrane Intervention reviews.

“The Cochrane Handbook for Systematic Reviews of Interventions (Higgins, Thomas, et al., 2019) provides in-depth guidance for researchers who are undertaking a Cochrane Intervention review (i.e., a systematic review that is prepared by a Cochrane Review Group).”

• cites Higgins, Thomas, et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Many sport and exercise psychology researchers use PRISMA checklist items and the PRISMA flow chart.

“Many researchers in sport and exercise psychology (e.g., Carson et al., 2016) implement the PRISMA checklist items and the PRISMA flow chart that is used to describe the flow of studies throughout screening (e.g., number of studies excluded after title screening, number of studies excluded with reasons after full-text screening; Moher et al., 2015).”

• cites Moher et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Systematic reviews involve many subjective decisions that, with poor reporting, could make them non-replicable or add misinformation.

“In particular, systematic reviews require numerous subjective decisions that, combined with poor reporting, could lead to systematic reviews that cannot be replicated, or at worst add misinformation to the literature (such as when systematic reviews have conflicting results) (Ioannidis, 2016).”

• cites Ioannidis, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Grey literature is typically not peer-reviewed and may be difficult to locate or access.

“Grey literature is typically not peer-reviewed and grey literature documents might be difficult to locate or access.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PICO was created to help focus research questions and identify relevant evidence for clinical or intervention-based quantitative research.

“The Population, Intervention, Comparison, and Outcome (PICO) framework was created to help focus research questions and identify relevant evidence for clinical or intervention-based quantitative research (Schardt, Adams, Owens, Keitz, & Fontelo, 2007).”

• cites Schardt et al., 2007Not checked yet

  Not checked in the free run.

May combine two claims.

There are trade-offs between review speed and scope, and between scope and generalizability.

“There are trade-offs between the speed at which a review can be conducted and its scope (e.g., a more narrow scope translates to a more rapid review) and between scope and generalizability (e.g., a more narrow scope translates to good generalizability to that specific context but poor generalizability more broadly).”

Classified as probably not a substantial claim. May combine two claims.

Open science describes strategies to increase transparency and replicability and has been recommended for sport and exercise psychology.

““Open science” describes strategies to increase research transparency and replicability (Nosek et al., 2015) and has been recommended for sport and exercise psychology researchers (Tamminen & Poucher, 2018).”

• cites Nosek et al., 2015Not checked yet

  Not checked in the free run.

• cites Tamminen & Poucher, 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Preregistration and protocols guide systematic reviews and should contain enough information for replication.

“Preregistration and protocols serve as guides for conducting systematic reviews and should contain sufficient information to allow for replication.”

May combine two claims.

AMSTAR and AMSTAR-2 are instruments for critically appraising systematic reviews that can also guide their conduct.

“Additionally, A MeaSurement Tool to Assess Systematic Reviews (AMSTAR and AMSTAR-2; Shea et al., 2007Shea et al., , 2017) ) is an instrument for critically appraising systematic reviews that can also be used to guide the conduct of systematic reviews.”

• cites Shea et al., 2007Not checked yet

  Not checked in the free run.

• cites Shea et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Developing and executing a search strategy can be linear or iterative.

“Developing and executing the search strategy can be linear or iterative.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Open science encourages preregistering studies, publishing protocols, sharing data and materials, and publishing open access.

“As part of the open science framework, researchers are encouraged to preregister studies, publish protocols, share data and materials, and publish in open access journals.”

May combine two claims.

Primary study findings are unlikely to be replicable, raising concerns about false positives or negatives, low statistical power, or insufficient transparency.

“In other words, findings from primary studies are unlikely to be replicable, raising concerns over false positives or false negatives, lack of statistical power, or insufficient transparency in research methods (Maxwell et al., 2015).”

• cites Maxwell et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Researchers can use the Open Science Framework website to register systematic reviews.

“Researchers can also use the Open Science Framework website (Open Science Framework, 2018; see Table 1) to register systematic reviews.”

• cites Open Science Framework, 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

SPIDER is a framework designed specifically for mixed-methods or qualitative studies.

“Another framework designed specifically for mixed-methods or qualitative studies is the Sample, Phenomenon of interest, Design, Evaluation, Research Type (SPIDER) tool (Cooke, Smith, & Booth, 2012).”

• cites Cooke et al., 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Each database has its own syntax and language requiring knowledge and expertise.

“This is particularly useful given that each database has its own syntax and language that requires knowledge and expertise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PROSPERO allows researchers to document all protocol aspects, including research questions, databases and search strategies, selection criteria, and quality assessment and data extraction procedures.

“Within PROSPERO, researchers can document all protocol aspects including research question(s), databases searched and search strategies, selection criteria, and procedures for quality assessment and data extraction.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some PROSPERO information may not apply to sport and exercise psychology research and may need adapted responses.

“Some of the information requested within PROSPERO might not be applicable to sport and exercise psychology research and may therefore require adapted responses.”

Classified as probably not a substantial claim.

It has been recommended that at least two reviewers screen each full-text article to prevent relevant studies from being erroneously excluded.

“As such, it has been recommended that at least two reviewers screen each full text article to ensure relevant studies are not erroneously excluded (Edwards et al., 2002).”

• cites Edwards et al., 2002Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Given that many studies are indexed in multiple databases, removing duplicates is important to reduce screening burden.

“Given that many studies are indexed in multiple databases, it is important that researchers remove duplicate studies to reduce the burden of screening articles.”

• AssumptionRests on an assumption: Many studies are indexed in multiple databases.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PROSPERO requests PICO information, and SPIDER may be a more suitable framework.

“For example, PROSPERO requests information about the study PICO and it is possible that SPIDER may be a more amenable framework (see descriptions of PICO and SPIDER in question 4).”

Probably not a substantial claim: may not pass S4 (based).

Databases overlap in the journals and publishers they index.

“Databases have some degree of overlap in terms of the journals and publishers that are indexed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ENTREQ encourages the PRISMA flowchart and includes items capturing sampling strategy, influenced by STARLITE.

“In particular, ENTREQ encourages the use of the PRISMA flowchart but is influenced by the Standards for Reporting Literature Searches (STARLITE; Booth, 2006) insofar as it includes items that capture the sampling strategy (Booth, 2016).”

• cites Booth, 2016Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Preregistration should occur before eligibility screening begins to prevent viewing articles from changing inclusion/exclusion criteria.

“Preregistration should occur before study screening for eligibility has begun, to negate the possibility that viewing articles could change inclusion/exclusion criteria (Stewart et al., 2012).”

• cites Stewart et al., 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Bramer and colleagues found that Embase, MEDLINE, Web of Science, and Google Scholar should be searched, with PsyINFO added for behavioral and health sciences reviews.

“Bramer and colleagues (2017) recently found that four databases (Embase, MEDLINE, Web of Science, and Google Scholar) should be searched, with PsyINFO being added for systematic reviews in the behavioral and health sciences.”

• cites Bramer et al., 2017Not checked yet

  Not checked in the free run.

May combine two claims.

Having more than one reviewer extract data minimizes bias and reduces errors.

“The Cochrane Handbook recommends more than one reviewer extract the data to minimize bias and reduce errors (Li et al., 2019).”

• cites Li et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

PRISMA guidelines advocate that journals and funding agencies require preregistration for publication or funding.

“Finally, PRISMA guidelines advocate for journals and funding agencies to require preregistration for publication or funding purposes (Moher et al., 2015).”

• cites Moher et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

Conference abstracts often contain interim or incomplete data, and abstracts lack sufficient information to judge confidence in the evidence.

“For instance, conference abstracts often contain interim data or incomplete data (e.g., only the results that were statistically significant were presented which may be subject to selective reporting or publication bias), and there is insufficient information in abstracts to determine the level of confidence in the evidence they contain.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PICOS adds study design to PICO, allowing specification of eligible quantitative and qualitative study types.

“For example, Methley et al. (2014) produced PICOS where the “s” refers to study design such that PICOS can be used to specify which types of quantitative (e.g., non-randomized trials, prospective cohorts, cross-sectional) and qualitative studies are eligible for inclusion.”

• cites Methley et al., 2014Not checked yet

  Not checked in the free run.

May combine two claims.

When a research question draws on a well-established body of literature, adding a few recent studies is unlikely to dramatically change the conclusions of a systematic review, so an update is less critical.

“unlikely that the addition of a handful of recent studies will dramatically alter the conclusions of a systematic review, rendering an update less critical.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Involving a librarian or research information specialist in search strategy development can help ensure relevant literature is captured across databases.

“Given that most researchers in sport and exercise psychology are unlikely to have library science training, involving a librarian or research information specialist in the development of a search strategy can help to ensure the search strategy will capture relevant literature across databases.”

• AssumptionRests on an assumption: Most researchers in sport and exercise psychology are unlikely to have library science training.

Classified as probably not a substantial claim. May combine two claims.

Indirect evidence is lower quality due to its separation from the comparison(s) of interest, and its quality should be clearly reported when included.

“Indirect evidence is “lower quality” by nature of its separation from the comparison(s) of interest; when included, the quality of the evidence should be clearly reported (see question 11).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

It is important to document protocol changes after a review begins; most preregistration platforms allow alterations and create an audit trail.

“It is important to document if changes to the protocol are required after the review has begun; most preregistration platforms allow researchers to alter their protocol and create an audit trail with information explaining the revisions for complete transparency.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ENTREQ was created to mirror PRISMA but for qualitative primary studies.

“ENTREQ was created to mirror the PRISMA guidelines but for application to qualitative primary studies.”

• cites Tong et al., 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Since October 2019, PROSPERO only accepts registrations where data extraction has not started.

“Effective October 2019, PROSPERO only accepts registrations where data extraction has not commenced.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When direct evidence is scarce, researchers may include indirect evidence.

“In some areas, when there is not a lot of direct evidence, researchers may choose to include indirect evidence.”

Classified as probably not a substantial claim.

Quality and risk of bias are distinct but related concepts.

““Quality” and “risk of bias” are often mistaken as interchangeable terms, when in fact they are distinct but related concepts (Higgins & Green, 2011).”

• cites Higgins & Green, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

It is usually not possible to know to what extent potential biases have affected the results of a particular study, so judgments about bias are best termed risks of bias.

“It is usually not possible to know to what extent potential biases have affected the results of a particular study (and results of a study may be unbiased despite methodological flaws), so judgments about bias are most appropriately termed 'risks' of bias (Boutron et al., 2019).”

• cites Boutron et al., 2019Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Making informed decisions a priori and reporting them explicitly and transparently will enhance the robustness of a systematic review.

“Making informed decisions a priori, and reporting them explicitly and transparently, will enhance the robustness of the systematic review.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The synthesis methods described are not exhaustive; other strategies for synthesizing quantitative and qualitative primary studies are available.

“The synthesis methods described above are not exhaustive; other strategies for synthesizing quantitative and qualitative primary studies are available (Barnett-Page & Thomas, 2009;Deeks et al., 2019;Holt et al., 2017).”

• cites Barnett-Page & Thomas, 2009Not checked yet

  Not checked in the free run.

• cites Deeks et al., 2019Not checked yet

  Not checked in the free run.

• cites Holt et al., 2017Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The Cochrane Handbook advises using GRADE to evaluate evidence quality across a set of quantitative studies for a specific outcome in a systematic review.

“The Cochrane Handbook (Higgins, Thomas, et al., 2019) recommends that the Grading of Recommendations Assessment, Development, and Evaluation (GRADE; Guyatt et al., 2008) framework be used to examine the quality of the evidence for the body of quantitative investigations (i.e., collection of studies) for a particular outcome of interest within a systematic review.”

• cites Higgins, Thomas, et al., 2019Not checked yet

  Not checked in the free run.

• cites Guyatt et al., 2008Not checked yet

  Not checked in the free run.

May combine two claims.

Vote counting provides no way to quantify effect size or its confidence limits; it only shows how many studies found an effect or the direction of the effect.

“Furthermore, with vote counting there is no way to quantify the effect size or its limits of confidence (Bushman, 1994); the only information provided is how many studies found an effect and/or the direction of the effect.”

• cites Bushman, 1994Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Under GRADE, the quality of a body of evidence is categorized as high, moderate, low, or very low.

“Using this framework, the body of evidence is classified as”

• cites Guyatt et al., 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Cochrane risk of bias tool is commonly used for reviews focusing on randomized controlled trials.

“The Cochrane risk of bias tool is commonly used for reviews focusing on randomized controlled trials (Higgins, Savović, Page, Elbers, & Sterne, 2019).”

• cites Higgins, Savović, Page, Elbers, & Sterne, 2019 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Under GRADE, evidence quality starts at high for randomized experiments and at low for observational studies.

“The quality of evidence begins with”

• cites Guyatt et al., 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Statistical heterogeneity can be assessed using statistical methods such as chi-square or I2.

“Statistical heterogeneity can be assessed using various statistical methods including chi-square or I 2 .”

Classified as probably not a substantial claim.

Precise methodological details are needed for a systematic review to be replicated or updated.

“In order for a review to be replicated or updated, the precise methodological details are needed.”

May combine two claims.

Under GRADE, factors such as major bias risk, indirect measures, inconsistent effects, and imprecise results lower evidence quality.

“Other limitations across studies lead to downgrading the quality of evidence, such as serious risk of bias, indirectness (e.g., the measures used were not direct measures of the variable of interest), inconsistency of effects (e.g., the effects observed were not consistent across studies), and imprecision (e.g., large confidence intervals for the results) (Guyatt et al., 2011).”

• cites Guyatt et al., 2011Not checked yet

  Not checked in the free run.