Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies

May combine two claims.

The sample comprised 140 systematic reviews, 124 academic and 16 practitioner.

“In total, 140 systematic reviews (124 academic and 16 practitioner) comprise the sample for investigation.”

May combine two claims.

Neither Martin and Assenov (2012) nor Bowen et al. (2010) identify the specific contribution of grey literature over and above academic literature, but each emphasizes that the added value of grey sources outweighs the negative effects of using a process that is not completely replicable.

“Neither Martin and Assenov (2012) nor Bowen et al. (2010) identify the specific contribution of grey literature in their reviews over and above that of the academic literature, but each article emphasizes that the added value of grey sources outweighs the negative effects of using a process that is not completely replicable.”

May combine two claims.

Garengo et al. (2005) found a lack of evidence on performance measurement systems for small and medium-sized enterprises in the white literature and turned to conference proceedings and internal reports, identifying new fields of inquiry.

“Garengo et al. (2005), for instance, found a lack of evidence on performance measurement systems for small and medium-sized enterprises in the white literature and so turned to conference proceedings and internal reports and, in this way, identified new fields of inquiry.”

May combine two claims.

de Menezes and Kelliher (2011) report that a rich understanding of their substantive issue was impaired by insufficient context in the academic literature, and they supplement their review with grey literature.

“de Menezes and Kelliher (2011) report that a rich understanding of their substantive issue of interest was impaired by insufficient information about context provided by the academic literature, and supplement their review with grey literature.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Bititci et al. (2012) explored general literature on global and business trends by drawing on management gurus and sources such as thinkers50.com, and argue this allowed emerging conclusions from one stream of literature to inform the other.

“To complement and contextualize their findings on performance measurement Bititci et al. (2012, p. 307) explore general literature on global and business trends by drawing on the works of 'management gurus', and sources such as http://www. thinkers50.com and 'Who are the gurus' gurus?'The merit of this approach, they argue, is that it allowed for emerging conclusions from one stream of literature”

Probably not a substantial claim: may not pass S3 (load-bearing).

Nitkin et al. (2009) can point to major adaptation initiatives in the insurance sector following a review of grey literature.

“Similarly, Nitkin et al. (2009) can point to major adaptation initiatives in the insurance sector following a review of grey literature.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Turner et al. (2013) include grey literature because many major articles have been aimed at practitioners, intending the study to be as inclusive as possible and to avoid eliminating potentially valuable contributions.

“Turner et al. (2013, p. 3) include grey literature for the reason that 'many of the major articles have been aimed at practitioners'; they intended the study to be as inclusive as possible and tried to avoid eliminating potentially valuable contributions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some authors explicitly claim that the resulting review does not become any less 'systematic', and they show how they adhere to the tenets of transparency and rigour, arguably more so in light of the grey literature's heterogeneity.

“Some authors explicitly claim that the resulting review does not become any less 'systematic', and they show how they adhere to the tenets of transparency and rigour, arguably more so in light of the grey literature's heterogeneity.”

May combine two claims.

Of the 20 environmental impact tools identified by Kaval (2011), 12 were identified in the white literature, one in the grey, and the remainder across both literatures.

“Of the 20 environmental impact tools identified by Kaval (2011), 12 were identified in the white literature, one in the grey and the remainder across both literatures.”

May combine two claims.

Adams et al. (2012), Albino et al. (2011) and Martin and Assenov (2012) all point to the contemporaneity or currency of grey literature, which can run ahead of academic research.

“Further, Adams et al. (2012), Albino et al. (2011) and Martin and Assenov (2012) all point to the contemporaneity or currency of grey literature, which can run ahead of academic research.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Perkmann and Walsh (2007) draw on complementary information from reports published by government agencies and other organizations (e.g. OECD) in their discussion of the UK's history of university-industry joint R&D establishments.

“Perkmann and Walsh (2007) draw on complementary information from reports published by government agencies and other organizations (e.g.OECD) in their discussion of the UK's history of university-industry joint R&D establishments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These sources were selected on the basis of their reputation, currency and authority as well as search functionality.

“These sources were selected on the basis of their reputation, currency and authority as well as search functionality.”

Probably not a substantial claim: may not pass S4 (based).

Many sites providing specialized knowledge are idiosyncratic in the way they present and archive knowledge content as well as in their search functionality.

“Many sites providing specialized knowledge are idiosyncratic in the way they present and archive knowledge content as well as in their search functionality.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Some sites do not adhere to the traditional bibliographic format to which scholarly reviewers have become accustomed.

“Some sites do not adhere to the traditional bibliographic format to which scholarly reviewers have become accustomed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Adams et al. (2012) explored a similar range of specialist sites as well as five blogs, using authority and reputation as their yardsticks for selection.

“Adams et al. (2012) explored a similar range of specialist sites as well as five blogs: authority and reputation their yardsticks for selection.”

May combine two claims.

Palomino et al. (2013) demonstrate the instability of search engine results, finding that results vary at an extremely fast pace, sometimes in only a matter of hours, with variable propensity of results to be higher or lower than in a previous search.

“Palomino et al. (2013) demonstrate the instability of search engine results.They found that, rather than repeated search generating an increasing number of results as new content is added over time, results vary at an extremely fast pace -sometimes in only a matter of hours, with variable propensity of results to be higher or lower than in a previous search.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Albino et al. (2011) screened 17 institutional databases, including those from consulting organizations, manufacturers associations, international research institutes and governmental organizations.

“Albino et al. (2011) screened 17 institutional databases, including those from consulting organizations, manufacturers associations, international research institutes and governmental organizations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Peloza and Yachnin (2008) searched Greenbiz.com, Socialfunds.com, UN Environment Programme Finance Initiative, World Business Council for Sustainable Development and other grey sources.

“Peloza and Yachnin (2008), for example, searched Greenbiz.com,Socialfunds.com,UN Environment Programme Finance Initiative, World Business Council for Sustainable Development and other grey sources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Such practices were echoed across several Category A reviews.

“Such practices were echoed across several Category A reviews (e.g.Avard et al. 2010;Stephan et al. 2013;Ton et al. 2013).”

• cites Avard et al. 2010Not checked yet

  Not checked in the free run.

• cites Stephan et al. 2013Not checked yet

  Not checked in the free run.

• cites Ton et al. 2013Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Such searches have necessitated developing new selection criteria in an endeavour to make the process as transparent as possible.

“Such searches have necessitated developing new selection criteria in an endeavour to make the process as transparent as possible.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Cataloguing the diversity of the search and specifying sources can be challenging.

“Cataloguing the diversity of the search and specifying sources can be challenging.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Kaval's (2011) Google search returned a nominal 462,000 results, but they found only 50 actual results visible and accessible.

“Kaval's (2011) Google search returned a nominal 462,000 results, but they found only 50 actual results visible and accessible.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In their Google searches, both Bowen et al. (2010) and Arvai et al. (2012) reviewed the first five pages of returns, and Bertels et al. (2010) the first 500 results.

“In their Google searches, both Bowen et al. (2010) and Arvai et al. (2012) reviewed the first five pages of returns, and Bertels et al. (2010) the first 500 results.”

Classified as probably not a substantial claim. May combine two claims.

Specialized knowledge sites are not equally easy to navigate, and often do not have sophisticated search facilities.

“They are not equally easy to navigate, and often do not have sophisticated search facilities.”

May combine two claims.

Scholars active in other disciplines report additional benefits such as addressing publication bias, of which there is some evidence in MOS but limited in our sample.

“Scholars active in other disciplines, with longer traditions of deploying systematic review, report additional benefits such as addressing publication bias (Hopewell et al. 2007), of which there is some evidence in MOS (Kepes et al. 2012), but limited in our sample.”

• cites Hopewell et al. 2007Not checked yet

  Not checked in the free run.

• cites Kepes et al. 2012Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Adams et al. (2012) considered this a useful generic approach, and made a similar request to scholars in the field of sustainability, but this strategy led to a disappointing response.

“Adams et al. (2012) considered this a useful generic approach, and made a similar request to scholars in the field of sustainability, but this strategy led to a disappointing response.”

May combine two claims.

Reviewers in the sample appear to take advantage of the flexibility of narrative summaries but may lack awareness of alternative approaches.

“Narrative summaries allow a highly flexible method, and reviewers appear to be taking advantage of this flexibility, though it appears they may lack awareness of alternative approaches.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Bertels et al. (2010) coded data at the level of sustainable practice and coded each practice in terms of level of empirical support.

“Bertels et al. (2010) coded data at the level of 'sustainable practice', coding each practice that they uncovered in terms of level of empirical sup-port.”

• cites Bertels et al. 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Bertels et al. (2010) were able to include practices where practitioner knowledge led extant theory and instances where academics proposed practices not directly used or tested in practice.

“In addition to reporting empirically supported practices, they were able to include practices where practitioner knowledge led extant theory as well as instances where academics had proposed practices that had not been directly used or tested in practice.”

• cites Bertels et al. 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Category B reviews draw only on scholarly literature and explicitly discount grey sources.

“Category B reviews draw only on the scholarly literature, and present a range of rationales for selection similar to those found in Category C reviews, but explicitly discount grey sources; a number explain why.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

McDermott et al. (2006) classified studies as higher, medium or lower quality on a range of criteria and reported findings against these qualifiers.

“McDermott et al. (2006) classified studies as of higher, medium or lower quality on a range of criteria, and reported findings against these qualifiers.”

• cites McDermott et al. 2006Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

The analysis of Category A reviews shows that scholars consistently adopt the reporting practice of descriptively summarizing salient characteristics of included studies.

“Our analysis of Category A reviews shows that scholars consistently adopt this reporting practice”

Probably not a substantial claim: may not pass S3 (load-bearing).

A number of Category A review authors explicitly state they include grey literature because they believe it provides better guidance for practice.

“A number of the authors of Category A reviews explicitly state that they include the grey literature because they believe that, by doing so, they are able to provide better guidance for practice (e.g.Adams et al. 2012;Bertels et al. 2010;McDermott et al. 2006).”

• cites Adams et al. 2012Not checked yet

  Not checked in the free run.

• cites Bertels et al. 2010Not checked yet

  Not checked in the free run.

• cites McDermott et al. 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

There is a relative failure in MOS reviews to keep findings, conclusions and recommendations relating to white and grey literature separate.

“Of particular concern for the current paper is the relative failure to keep findings, conclusions and recommendations relating to white and grey literature separate.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Claus and Briscoe (2009) point to inherent biases in their selection criteria and say additional insight could be gained from research outside their stated parameters.

“Claus and Briscoe (2009) more explicitly point to inherent biases in their selection criteria and say that additional insight could be gained from research outside their stated parameters.”

• cites Claus and Briscoe 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Pittaway and Cope (2007) conclude key aspects of their study were under-represented owing to inclusion criteria and argue future studies would benefit from examining grey literature.

“Pittaway and Cope (2007) conclude that key aspects of their study were somewhat under-represented owing to their inclusion criteria and argue that future studies would benefit from examining grey literature in more detail.”

• cites Pittaway and Cope 2007Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Previously published reviews in MOS have to a great extent been poor in methodological reporting.

“previously published reviews in MOS have to a great extent been poor in methodological reporting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Peer-review is used as a primary filter for inclusion as a proxy for quality in Category B articles.

“peer-review is used as a primary filter for inclusion as a proxy for quality in Category B articles.”

May combine two claims.

The majority of Category B authors are unapologetic about excluding grey literature: they recognize it exists but consider academic literature adequate for the immediate project.

“the majority of Category B authors are unapologetic about excluding grey literature: they recognize that it exists, but consider publications from the academic literature adequate, at least for the purposes of the immediate project (e.g.Schmeisser 2013;Tarí 2011;Vishanth et al. 2009).”

• cites Schmeisser 2013Not checked yet

  Not checked in the free run.

• cites Tarí 2011Not checked yet

  Not checked in the free run.

• cites Vishanth et al. 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Knoben and Oerlemans (2006) describe their chosen method of searching literature as disadvantageous, leading to omission of books and book chapters.

“Knoben and Oerlemans (2006) describe their chosen method of searching literature as disadvantageous, leading to the omission of books and book chapters.”

• cites Knoben and Oerlemans 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The extent of publication bias in MOS is unclear.

“The extent of publication bias in MOS is unclear”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

To our knowledge, no explicit and formal evaluations have been made of the contribution to practice of any Category A reviews.

“To our knowledge, no explicit and formal evaluations have been made of the contribution to practice of any Category A reviews.”

Classified as probably not a substantial claim.

This paper shows how MOS researchers incorporate grey literature in a fashion that reflects diverse use of evidence.

“This paper shows how MOS researchers incorporate the grey literature in a fashion that reflects this diverse use of evidence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One study included in the sample, a Category C review by Homberg and Bui (2013), reported that the published literature on the diversity performance link overestimates the strength of positive results.

“One study included in our sample, a Category C review by Homberg and Bui (2013), reported that the published literature on the diversity performance link overestimates the strength of positive results.”

• cites Homberg and Bui 2013Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

A number of authors in the reviewed sample reflected that their reviews might have been improved by including grey literature.

“A number of authors reflected that their reviews might have been improved by the inclusion of the grey literature.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In journals dealing with management issues in medicine, such as Journal of Nursing Management, quality assessments of individual articles appear routinely.

“Another exception is articles in journals that deal with management issues in medicine such as Journal of Nursing Management, where quality assessments of individual articles appear routinely.”

Classified as probably not a substantial claim.

Many authors in the reviewed sample were concerned with the time required for reviewing more extensive sources.

“Since many were concerned with the time required for review of more extensive sources, future researchers might consider running trials in which the same question is addressed, but by separate teams, one empowered to include the grey literature, the other not.”

Classified as probably not a substantial claim. May combine two claims.

The 12 guidelines are derived from a systematic review of reviews showing that grey material is already being considered and used by many MOS researchers, but then extended in light of trends, source availability and use in a digital age.

“The 12 guidelines that summarize this discussion are derived from a systematic review of reviews showing that grey material is already being considered and used by many MOS researchers, but then extended in light of trends, source availability and use in a digital age.”

Probably not a substantial claim: may not pass S4 (based).

There is an unruly and rapidly increasing amount of grey evidence available.

“There is an unruly and rapidly increasing amount of grey evidence available.”

Classified as probably not a substantial claim.

The extent of the impact on findings of including grey literature in Category B and C reviews is unclear.

“What is less clear is the extent, if any, of the impact on findings had the grey literature been included in Category B and C reviews.”

May combine two claims.

All tiers of the grey literature have the potential to define and contextualize phenomena of research knowledge because grey literature includes potentially relevant knowledge that is sometimes not reported adequately in academic articles.

“This evidence leads us to the general conclusion that all tiers of the grey literature have the potential to define and contextualize phenomena of research knowledge because it includes potentially relevant knowledge that is sometimes not reported adequately in academic articles.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Integrating user views into question formulation is particularly appropriate where the research topic is novel or requires contextualization.

“This is particularly appropriate where the research topic is novel or requires contextualization (Rutter et al. 2010).”

• cites Rutter et al. 2010Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Questions about impact and contribution in Category C reviews might be interpreted as benefiting from broader review, especially in dynamic or emergent fields where academic knowledge may lag.

“even though questions about impact and contribution in particular might be interpreted as benefiting from broader review, espe-cially in dynamic or emergent fields where academic knowledge may lag.”

May combine two claims.

Category A reviews deliberately reject some grey literatures, indicating that selection decisions are not made indiscriminately and active choices are pursued about inclusion and exclusion.

“The observation that Category A reviews deliberately reject some of the grey literatures indicates that selection decisions are not made indiscriminately, and active choices are pursued about inclusion and exclusion.”

Classified as probably not a substantial claim.

The decision to include grey literature requires conscious, explicit and different procedures, typically involving a review-specific strategy.

“The decision to include grey literature thus requires conscious, explicit and different procedures, typically involving a review-specific strategy.”

Probably not a substantial claim: may not pass S2 (checkable).

When extending searches beyond the electronic databases of published and unpublished studies, a multilayered and semistructured process is necessary to enable manageability, maintain control over a time-consuming process, and ensure transparency and replicability to the extent possible.

“When extending searches beyond the electronic databases of published and unpublished studies (e.g.conference papers, working papers), a multilayered and semistructured process is necessary to enable manageability, maintain control over a time-consuming process, and ensure transparency and replicability to the extent possible.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

It is important to explore electronic repositories, ideally with editorial independence and professional or institutional affiliations, to identify grey literature, creating site-specific search strings as necessary.

“In other words, it is important to explore electronic repositories (ideally with editorial independence and professional or institutional affilia-tions) to identify grey literature, creating site-specific search strings as necessary.”

Probably not a substantial claim: may not pass S2 (checkable).

Maintaining a documentary record of decisions and processes enables researchers to report credibly for journal submission.

“However, maintaining a documentary record of decisions and processes enables researchers to report credibly for journal submission.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such records, if made publicly available, are useful for meeting the increasingly common stipulation of funding bodies such as Research Councils (UK) and the National Science Foundation (USA).

“Such records, if made publicly available, are useful for meeting the increasingly common stipulation of funding bodies such as Research Councils (UK) and the National Science Foundation (USA).”

Probably not a substantial claim: may not pass S2 (checkable).

More flexible processes for search are needed to explore both generalist and specialist sites.

“More flexible processes for search are needed to explore both generalist and specialist sites.”

Classified as probably not a substantial claim.

It may be necessary to modify or replace search strings developed for academic databases when searching grey literature.

“Consequently, and owing to the broad range of document type and search options, it may be necessary to modify or replace search strings developed for academic databases (e.g.Albino et al. 2011).”

• cites Albino et al. 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Given that grey literature has rarely been through a peer review process and is generated in response to a much more diverse set of circumstances, there is a need to develop alternative and study-specific quality appraisal techniques and methods, where possible.

“Given that grey literature has rarely been through a peer review process and is generated in response to a much more diverse set of circumstances, there is a need to develop alternative and study-specific quality appraisal techniques and methods, where possible.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Varied outcomes may be an indication of the immaturity of a field of inquiry and so further indicate a need for deeper exploration of grey literature.

“Varied outcomes may be an indication of the immaturity of a field of inquiry and so further indicate a need for deeper exploration of grey literature.”

May combine two claims.

Additional and less clearly defined standards add time and expense to the process of systematic review, which has deterred some researchers from using them.

“Additional and less clearly defined standards add time and expense to the process of systematic review, which has deterred some researchers from using them (e.g.Conn et al. 2003)”

• cites Conn et al. 2003Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

There is strong justification for greater consideration of including the grey literature in future MOS systematic reviews.

“On the basis of these observations, we argue that there is strong justification for greater consideration of including the grey literature in future MOS systematic reviews.”

Classified as probably not a substantial claim.

Historically, access to many forms of grey literature has been a barrier to its inclusion in systematic review, since it has not been included in well-curated databases of academic disciplines.

“Historically, access to many forms of grey literature has been a barrier to its inclusion in systematic review, since it has not been included in well-curated databases of academic disciplines.”

May combine two claims.

Grey literatures typically have no abstract, and so relevance and other inclusion criteria often cannot be determined without reviewing the entire document.

“For example, grey literatures typically have no abstract, and so relevance and other inclusion criteria often cannot be determined without reviewing the entire document (Benzies et al. 2006).”

• cites Benzies et al. 2006Not checked yet

  Not checked in the free run.

May combine two claims.

Through digitization, the size and influence of grey literature has increased, and the need to include it in systematic review has become more evident.

“Through digitiza-tion, however, the size and influence of this type of literature has increased, and the need to include it in systematic review has become more evident, while better cataloguing and management has increasingly become a concern for librarians (Jeffery 2000).”

• cites Jeffery 2000Not checked yet

  Not checked in the free run.

May combine two claims.

Scholarly studies provide methodological descriptions that facilitate evaluating quality; such assurance is usually missing in grey publications, which tend to focus on conclusions rather than the process by which they were reached.

“Scholarly studies provide methodological descriptions that facilitate evaluating quality; such assurance is usually missing in grey publications, which tend to focus on conclusions rather than the process by which they were reached.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Experts queried may also be able to identify specialist websites with relative ease and professional confidence.

“Experts queried may also be able to identify specialist websites with relative ease and professional confidence.”

Probably not a substantial claim: may not pass S2 (checkable).

The analysis reported here more firmly connects systematic review to its original pragmatic and practice-oriented purpose in the evidence-based management paradigm.

“The analysis reported here more firmly connects systematic review to its original pragmatic and practice-oriented purpose (Tranfield et al. 2003) in the evidence-based management (EBMgt) paradigm (Briner et al. 2009;Huff et al. 2006;Rousseau 2012).”

• cites Tranfield et al. 2003Not checked yet

  Not checked in the free run.

• cites Briner et al. 2009Not checked yet

  Not checked in the free run.

• cites Huff et al. 2006Not checked yet

  Not checked in the free run.

May combine two claims.

Source evaluation in systematic reviews is not only a mechanism for excluding low-quality evidence; it also involves appraising and reporting included material so that the reliability of findings can be judged.

“An important and often overlooked point is that source evaluation is not simply a mechanism for excluding low-quality evidence, but is also about appraising and reporting what is included so that judgement can be made about the reliability of findings (Denyer and Tranfield 2009).”

• cites Denyer and Tranfield 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The grey literature can bring the disparate voices of experience into scholarly conversation to increase its relevance and impact.

“However, the grey literature can bring the disparate voices of experience into scholarly conversation to increase its relevance and impact.”

Probably not a substantial claim: may not pass S4 (based).

The heterogeneity of grey literature is a specific problem that makes it less amenable to traditional forms of archiving, retrieval, analysis, synthesis, bibliographic data capture, data extraction and integration.

“The heterogeneity of grey literature is a specific problem that makes it less amenable to traditional forms of archiving, retrieval, analysis, synthesis, bibliographic data capture, data extraction and integration.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Assessing grey literature requires time and complicated trade-offs whose appropriateness will vary according to study context.

“Assessing this material requires time and complicated trade-offs whose appropriateness will vary according to study context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the middle ground lie many government reports, news articles, company publications and so on that may be of interest even though source expertise and outlet control cannot be fully determined.

“In the middle ground lie the many government reports, news articles, company publications and so on that may be of interest even though source expertise and outlet control cannot be fully determined.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This categorization recognizes that experts generate a range of material that may be of scholarly interest.

“This categorization recognizes that experts generate a range of material that may be of scholarly interest.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The narrative mode of summary adopted in the sample resonates with Rousseau et al.'s synthesis by integration.

“The narrative mode of summary adopted in our sample resonates with Rousseau et al.'s (2008) 'synthesis by integration', involving the collection and comparison of evidence derived from two or more data collection methods in a search for patterns and connections.”

• cites Rousseau et al. 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Prominent outlets sometimes publish unreviewed material written by people with unknown training and experience.

“Similarly, prominent outlets sometimes publish unreviewed material written by people with unknown training and experience.”

May combine two claims.

The field of MOS has been characterized as ontologically and epistemologically diverse, making it difficult to develop a single agreed paradigm for evaluation.

“The field has been characterized as ontologically and epistemologically diverse (Tranfield and Starkey 1998), making it difficult to develop a single and agreed paradigm for evaluation, the production of which has been described as 'a forlorn hope' (Johnson et al. 2006, p. 146).”

• cites Tranfield and Starkey 1998Not checked yet

  Not checked in the free run.

• cites Johnson et al. 2006Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Given increasing likelihood that data collection methods go unreported through increasing grey tiers, the narrative mode can be understood as both a pragmatic and compensatory strategy.

“Given the increasing likelihood that data collection methods go unreported through increasing grey tiers of the literature (Figure 1), this can be understood as both a pragmatic and a compensatory strategy.”

Classified as probably not a substantial claim.

A widely accepted tenet of systematic review is that individual articles are assessed for quality.

“A widely accepted tenet of systematic review is that individual articles are assessed for quality (Tranfield et al. 2003).”

• cites Tranfield et al. 2003Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Assessing quality is even more difficult when grey literature must be evaluated.

“Assessing quality is even more difficult when grey literature must be evaluated.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The grey literature comes from a complex landscape of information artefacts generated in the course of real-life practices.

“In short, the grey literature comes from a complex landscape of information artefacts generated in the course of real-life practices.”

Classified as probably not a substantial claim.

Rutter et al. (2010) suggest that summaries of grey literature data should be reported separately from the synthesis of academic empirical studies so readers are clear what level of data informs which conclusions.

“Rutter et al.'s (2010) suggestion that, while grey literature contains important and relevant insight and knowledge, summaries of such data should be reported separately from the synthesis of academic empirical studies so that readers and users are clear what level of data is informing which conclusions.”

• cites Rutter et al. 2010Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. The quote only partly states this claim.

Current use suggests that findings and confidence levels of systematic reviews of white and grey literature should be reported separately unless the literatures are of similar status.

“current use suggests that findings and confidence levels of systematic reviews of white and grey literature should be reported separately.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Grey literature can point to the wisdom of practice, which may not be reflected in scientific evidence.

“supports the view that grey literature can point to the wisdom of practice, which may not be reflected in scientific evidence (Benzies et al. 2006).”

• cites Benzies et al. 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Pawson et al. argue there is a critical role for grey literature in realist or theory-led reviews aimed primarily at developing theory for subsequent investigation and testing.

“argue that there is a critical role for the grey literature in 'realist' or 'theory-led' reviews where the aim of the project is primarily to develop theory for subsequent investigation and testing.”

• cites Pawson et al. 2004Not checked yet

  Not checked in the free run.

• cites Pawson et al. 2005Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The grey literature can be used in interpretive synthesis to offer a richer set of accounts and supplementary narratives.

“in interpretive synthesis to offer a richer set of accounts and supplementary narratives”

Classified as probably not a substantial claim. May combine two claims.

Emergent fields are characterized by inconsistency and proliferation of constructs, a widely distributed or fragmented body of knowledge, lack of a universal language, and absence of coalescing and binding theories.

“Emergent fields are characterized by inconsistency and proliferation of constructs, a widely distributed or fragmented body of knowledge, lack of a universal language, and absence of coalescing and binding theories (Burgess et al. 2006).”

• cites Burgess et al. 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The presumption in Category C reviews is that scholarly journals and the double-blind peer-review process are a reliable proxy for quality.

“The (often tacit) presumption in Category C reviews is that scholarly journals and, in particular, the double-blind peerreview process, are a reliable proxy for quality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Peloza and Yachnin (2008) note that results in the academic literature tend to show a less positive relationship between sustainability and financial performance than do grey reports.

“They note that results in the academic literature tend to show a less positive relationship between sustainability and financial performance than do grey reports.”

• cites Peloza and Yachnin 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Report credibility for many audiences is likely to be enhanced through inclusion of comparative and descriptive analysis of grey literature findings, especially where findings supplement or challenge white literature.

“report credibility for many audiences is likely to be enhanced through the inclusion of comparative and descriptive analysis of grey literature findings, especially in instances where findings supplement or challenge those of the white literature.”

Probably not a substantial claim: may not pass S2 (checkable).

Findings from diverse bodies of evidence can be synthesized to deliver value in terms of their contribution to discourse and practice.

“we argue that the findings from diverse bodies of evidence can be 'synthesized' to deliver value in terms of their contribution to discourse and practice (Gough and Elbourne 2002).”

• cites Gough and Elbourne 2002Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Grey literature begins to lose its distinctive identity and contribution when treated and analysed as equivalent to white literature.

“grey literature begins to lose its distinctive identity and contribution when treated and analysed as an equivalent to the white literature.”

Classified as probably not a substantial claim.

Despite limited evidence in the sample of different approaches to synthesis being used, the grey literature can be used in explanatory synthesis to provide additional context specificity.

“we contend that the grey literature can be used in explanatory synthesis to provide additional context specificity”

Classified as probably not a substantial claim.

The grey literature can be used in integrative synthesis to explore opportunities for triangulation and contextualization.

“and in integrative synthesis to explore opportunities for triangulation and contextualization.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A range of evidence types is used in managerial decision-making.

“A range of evidence types is used in managerial decision-making (Briner et al. 2009;Kyratsis et al. 2014).”

• cites Briner et al. 2009Not checked yet

  Not checked in the free run.

• cites Kyratsis et al. 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

The inference that guidance is de facto better because of grey literature inclusion is at best a tenuous rationale.

“At best, this is a tenuous rationale.”

Probably not a substantial claim: may not pass S4 (based).

One reason for including grey literature in systematic reviews is to inform practice better.

“We have argued that one reason for including the grey literature in systematic reviews is to inform practice better.”

Probably not a substantial claim: may not pass S4 (based).

The same problems that afflict primary research, including confirmatory and publication bias, may also afflict systematic reviews.

“the same problems that afflict primary research, which include confirmatory and publication bias, may also afflict systematic reviews.”

Classified as probably not a substantial claim.

Systematic review's practical application in MOS has not reached maturity.

“its practical application in the field has not reached maturity.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Broader engagement of stakeholders is conducive to innovation in management practice because it increases the variety of interactions with evidence from past experience.

“Broader engagement of stakeholders is conducive to innovation in management practice because it increases the variety of interactions with evidence from past experi-ence.”

Classified as probably not a substantial claim.

Broad interrogation of evidence and its use is consistent with Pettigrew's call for holism and sensitivity to action, dynamics, context and complexity.

“Broad interrogation of evidence and its use is consistent with Pettigrew's call for 'a new concern for holism and sensitivity to action, dynamics, context and complexity'”

• cites Pettigrew 2011Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

A principle criterion for inclusion of grey literature is fitness for purpose, how it fits into ways findings are likely to be used.

“A principle criterion for inclusion of grey literature, then, becomes fitness for purpose, how it fits into the ways in which the findings are likely to be used (Boaz and Ashby 2003;Briner et al. 2009;Gough 2007;Nutley et al. 2013;Pawson 2006).”

• cites Boaz and Ashby 2003Not checked yet

  Not checked in the free run.

• cites Briner et al. 2009Not checked yet

  Not checked in the free run.

• cites Gough 2007Not checked yet

  Not checked in the free run.

May combine two claims.

The grey literature provides a perspective for contextualizing, critiquing and reflecting on published studies, serving scholars and practitioners with data, knowledge and experience for a more comprehensive and contextual view.

“It provides a perspective for contextualizing, critiquing and reflecting on published studies, and so serves scholars and practitioners alike with data, knowledge and experience that can offer a more comprehensive and contextual view of the topic”

Classified as probably not a substantial claim.

Current knowledge is constrained by the quality of reporting.

“current knowledge is constrained by the quality of reporting.”

Probably not a substantial claim: may not pass S2 (checkable).

Systematic reviews that include grey literature can provide an important source of information for research and practical projects, especially those concerning new fields of inquiry where knowledge from early experience is needed.

“systematic reviews that include grey literature can provide an important source of information for research and practical projects, especially those concerning new fields of inquiry where knowledge from early experience is needed.”

Probably not a substantial claim: may not pass S2 (checkable).

The grey literature can draw on practitioner expertise and judgement from local contexts in which decisions, actions and outcomes occur and the perspectives of those affected.

“the grey literature can draw on practitioner expertise and judgement from local contexts in which decisions, actions and outcomes occur and the perspectives of those people who might be affected by them.”

Classified as probably not a substantial claim.

Pawson et al. (2005) argue reviews based on research literature alone can fail to provide sufficiently rich, detailed and practical understanding of complex interventions.

“Our findings are consistent with Pawson et al. (2005), who argue that reviews based on the research literature alone can fail to provide a sufficiently rich, detailed and practical understanding of complex interventions.”

• cites Pawson et al. 2005Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Briner et al. (2009) suggest Evidence-based management is about making decisions through conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgment, evidence from local context, critical evaluation of best available research evidence, and perspectives of those affected.

“Briner et al. (2009, p. 19) specifically suggest that Evidence-based management is about making decisions through the conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgment, evidence from the local context, a critical evaluation of the best available research evidence, and the perspectives of those people who might be affected by the decision”

• cites Briner et al. 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

The opportunities offered by grey literature are congruent with the sentiments of the EBMgt practising framework.

“The opportunities offered by the grey literature thus are congruent with the sentiments of the EBMgt practising framework.”

Probably not a substantial claim: may not pass S2 (checkable).

Systematic review can be seen not just as a first step in research and practice projects, but also as an important contribution to what can be accomplished in those projects.

“Systematic review can thus be seen not just as a first step in research and practice projects, but also as an important contribution to what can be accomplished in those projects.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some scholars within MOS have argued that inclusion of grey literature in systematic reviews can counterbalance publication bias.

“Some scholars within MOS have argued that the inclusion of the grey literature in systematic reviews can counterbalance such bias (e.g.Briner et al. 2009), though no Category A review reports such intent.”

• cites Briner et al. 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both systematic review and EBMgt have to be further developed and tested.

“We acknowledge that, as all complex conceptualizations in their infancy, both systematic review and EBMgt have to be further developed and tested (Briner et al. 2009).”

• cites Briner et al. 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

In MOS, the most likely effect of publication bias will be either to exaggerate effect or under-inform context by dismissing replication studies.

“In MOS, the most likely effect of publication bias will be either to exaggerate effect or under-inform context by dismissing replication studies.”

Classified as probably not a substantial claim.

Studies in other disciplines have suggested that incorporating grey literature can help address publication bias.

“Studies in other disciplines have suggested that incorporating the grey literature can also help address publication bias, the 'file drawer' problem (Hopewell et al. 2007).”

• cites Hopewell et al. 2007Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

MOS is a relative latecomer to evidence-based practice.

“Management and organization studies is a relative latecomer to evidence-based practice (Madhavan and Niranjan 2015)”

• cites Madhavan and Niranjan 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

When narratives from grey literature are juxtaposed and synthesized with other sources of evidence, they can empower managers and researchers to create their own metanarratives regarding specific decisions or actions.

“When such narratives are juxtaposed and synthesized with other sources of evidence, they can empower managers and researchers to create their own metanarratives regarding specific decisions or actions.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Engaging with grey literature in the systematic review process is attractive because it captures heterogeneity and promotes inclusiveness of the evidence pool and lessens the divide between stakeholders.

“engaging with grey literature in the systematic review process is thus attractive because it captures heterogeneity and promotes inclusiveness of the evidence pool and lessens the divide between stakeholders.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Systematic reviews that include grey literature support dialectical and dialogical propensities and invite generative possibilities of difference in a more democratizing manner.

“In this way, systematic reviews that include grey literature support dialectical and dialogical propensities, and invite the generative possibilities of difference in a more democratizing manner”