The evidence regarding diversity's effect on firm performance

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The other article cited for this statement did not examine sex or race either.

“The other article cited for this statement did not examine sex or race either.52”

• cites [52]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

The beliefs presented in support of the California legislation, the Nasdaq regulation, and nonprofit advocacy are uniform in their assumed benefits of diversity and are framed as evidence-based.

“The beliefs presented in support of the California legislation, the Nasdaq regulation, and the nonprofit advocacy are uniform in their assumed benefits of diversity, and, in each case, the beliefs are framed as being evidence-based.”

Probably not a substantial claim.

Many existing literature reviews do not adequately differentiate between work that uses credible empirical strategies and work that does not.

“Many of the existing literature reviews do not adequately differentiate between work that uses credible empirical strategies and work that does not.”

It may state two separate claims as one.

Consultation of the studies cited by Cox and Blake indicates that the researchers did not explicitly examine demographic diversity (e.g., race or sex) as the descriptions in most cases do not mention the racial composition of the teams and in at least some cases the experimental subjects were all male.

“However, consultation of the cited studies indicates that the researchers did not explicitly examine demographic diversity (e.g., race or sex) as the descriptions in most cases do not mention the racial composition of the teams and in at least some cases the experimental subjects were all male.”

It may state two separate claims as one.

Although Cox and Blake imply that mixed gender groups performed better, the results of the experiment do not show that.

“Although Cox and Blake imply that the presence of mixed gender groups performed better, the results of the experiment do not show that.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Relatively less empirical academic work has focused on the link between diversity and performance in the nonprofit sector.

“This is perhaps understandable given that relatively less empirical academic work has focused on the link between diversity and performance in the nonprofit sector.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Cox and Blake largely failed to provide any basis for its claims about diversity's effects on organization performance.

“Cox and Blake, the first prominent literature review of the valueadded to organizational performance (indeed, the review that provided the evidentiary base upon which Robinson and Dechant based their business case for diversity, calling it”

• Also stated in: § Managing diversity enhances organizational flexibility. · plain

  “largely failed to provide any basis for its claims about diversity's effects on organization performance.”

• cites [53]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Most extant reviews of the diversity-performance literature do little beyond summarizing the underlying articles' results and often treat low-quality and high-quality studies equivalently.

“While extant reviews (such as those mentioned above) are a helpful starting point, most of them tend to do little beyond summarizing the underlying articles' results.They often treat low-quality and high-quality studies equivalently.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Making the business case for diversity continues to be a major part of the playbook in advocating for diversity policies.

“Nonetheless, making the business case for diversity continues to be a major part of the playbook in advocating for diversity policies.”

It may state two separate claims as one.

The diversity-performance literature often proceeds without methods to guard against omitted variable bias, or when it attempts such methods, implementation tends to be problematic.

“the literature on the effects of diversity on organizational performance often proceeds without them or, when the articles attempt to use these methods meant to isolate causality, implementation tends to be problematic.”

Probably not a substantial claim.

In the face of limited evidence that diversity affects performance on average, many researchers look for diversity's effects in particular situations or non-linear threshold effects.

“In the face of limited evidence that diversity affects performance on average, many researchers look for diversity's effects in particular situations (such as specialized industries) or look for non-linear threshold effects (sometimes called the”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

The basis for the claims made by nonprofit umbrella groups is unclear, since they do not provide much documentation of their claims.

“The basis for the claims made by these nonprofit umbrella groups is a little unclear, since they do not provide much by way of documentation of their claims.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The review cites an article that does not examine sex or race at all in its analysis.

“For this proposition, the review cites an article that does not examine sex or race at all in its analysis.51”

• cites [51]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

For the most part, the public and outside stakeholders are not given a full airing of the research record.

“For the most part, however, the public and outside stakeholders are not given a full airing of the research record.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

Because many literature reviews have been done by non-empiricists, there is no source that walks non-specialists through a critical assessment of the research methodologies used.

“Because many of the literature reviews have been done by non-empiricists, 19 there is no source that walks non-specialists through a critical assessment of the research methodologies used.”

Probably not a substantial claim. It may state two separate claims as one.

The author's prior review of studies cited in Nasdaq's proposed diversity rules concluded that the Nasdaq proposal cherry-picked supportive studies and largely ignored contrary studies.

“Given my conclusion in that review that the Nasdaq proposal cherry-picked supportive studies and largely ignored contrary studies, using such a filter is likely to provide an unrepresentative view of the literature.”

• cites [92]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Each of the effect averages for surface-level diversity worsening conflict, communication effectiveness, and social integration was statistically different from zero.

“and each of these effect averages was statistically different from zero.57”

• cites [57]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The hypothesis testing procedure allows the researcher to bound the likelihood that her estimates are driven by random noise.

“This hypothesis testing procedure allows the researcher to bound the likelihood that her estimates are driven by random noise.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

The efficacy of these approaches varies considerably.

“The efficacy of these approaches varies considerably.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

If the instruments are truly independent of the outcome variable except via the treatment variable, the predicted values of the treatment should not suffer from omitted variable bias, but otherwise track with the true treatment variable.

“If the instruments are truly independent of the outcome variable (except via the treatment variable), the predicted values of the treatment should not suffer from omitted 74 For a discussion of this problem, see Hansen, supra note 64, at 248-49.variable bias, but otherwise track with the true treatment variable.”

• cites [74]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A failure to account for other variables likely gives rise to omitted variable bias.

“A failure to account for these other variables likely will give rise to an omitted variable bias”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Meta-analyses offer an approach that is more disciplined than qualitative literature reviews.

“meta-analyses offer an approach that is more disciplined.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If behavioral change interacts with the treatment effect, it might be difficult to generalize results from the artificial setting to the policy-relevant real-world setting.

“If that behavioral change interacts with the effect of the treatment, it might be difficult to generalize results observed in the artificial setting to the policy-relevant real-world setting.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If the estimated regression coefficient is more than 1.96 standard errors away from zero, the chance of observing such a large estimate by random chance is less than 5%.

“if the estimated regression coefficient is more than 1.96 standard errors (i.e., the standard deviation divided by the square root of the sample size) away from zero, the chance that one would have observed such a large (in magnitude) estimate by random chance is less than 5%.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Stahl suggests that there is some evidence that cultural diversity affects creativity and innovation.

“Stahl does suggest that there is some evidence that cultural diversity affects creativity and innovation.”

• cites [56]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

It is important not to confuse statistical significance with importance.

“it is important to not confuse the concept of statistical significance with importance.75”

• cites [75]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

Some papers are better than others; some use more careful methods or more reliable data collection procedures, while others lack methodological rigor.

“Clearly, some papers are better than others.Some papers use more careful methods or more reliable data collection procedures, while others lack methodological rigor.”

Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

Even if particularized benefits of diversity are real, they call into question generic, one-size-fits-all diversity mandates.

“In any event, even if these more particularized benefits of diversity are real, they call into question generic, one size fits all diversity mandates.”

It may state two separate claims as one.

Stewart's 2006 meta-analysis, examining 93 studies, found a statistically significant negative average effect of team member heterogeneity in general and team performance, and the negative effect remained with demographic heterogeneity (sex and race) specifically.

“A 2006 meta-analysis by Stewart, examining 93 studies, found a statistically significant negative average effect of team member”

• Also stated in: § Managing diversity enhances organizational flexibility. · plain

  “heterogeneity in general and team performance, and the negative effect remained with demographic heterogeneity (i.e., sex and race) specifically.”

• cites [54]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

If the instruments are all good, they will all generate the same causal estimates of the treatment effect; if they do not satisfy the exclusion restriction, they likely will generate different biased estimates.

“The intuition is that if the instruments are all good in the sense described above, they will all generate the same causal estimates of the treatment effect.If, instead, the instruments do not satisfy the second condition above (the so-called exclusion restriction), they likely will generate different biased estimates.”

Probably not a substantial claim. It may state two separate claims as one.

Randomized controlled experiments can overcome the omitted variable bias problem because randomization ensures that the treatment is uncorrelated with other characteristics of the subject or group.

“randomized controlled experiments (which can overcome the omitted variable bias problem since randomization ensures that the treatment-such as diversity-is uncorrelated with any other characteristic of the subject or group)”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Just because a and b provide the best fit, it does not mean the resulting estimates can be interpreted causally.

“Now, just because a and b provide the best fit, it does not mean the resulting estimates can be interpreted causally.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

In randomized controlled experiments, the treatment is assigned at random.

“In such experiments, the treatment the researcher wishes to study is assigned at random”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

A consequence of random assignment is that the treatment designation will be uncorrelated with any characteristic of the subject and no omitted variable bias will arise.

“A consequence of this random assignment is that the treatment designation will then be uncorrelated with any characteristic of the subject and, therefore, no omitted variable bias will arise.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A common approach to aid in counterfactual analysis is to use regression analysis.

“A common approach to aid us in this counterfactual analysis is to use the tool of regression analysis.65”

• cites [65]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The central limit theorem allows inferences about the likelihood that an observed correlation is driven by random chance.

“A result known as the central limit theorem allows us to make some inferences about the likelihood that the correlation observed in our data is driven by random chance.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Multivariate regression has largely the same interpretation and estimation method as the simple single-variable case.

“This multivariate regression (with n explanatory variables) has largely the same interpretation (and estimation method) as the simple single-variable case.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Some meta-analyses provide conditional means which can be helpful to dig into more nuanced questions about the literature.

“some meta-analyses provide conditional means (e.g., the mean effect size yielded by a particular estimation approach or the mean effect size when moderating variables are included in the analysis) which can be helpful to dig into more nuanced questions about the literature.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Most phenomena involve more than one variable in determining their outcomes.

“Most phenomena will involve more than one variable in determining their outcomes.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

Meta-analyses provide a blunt averaging of estimated effects without allowing readers to differentially weight studies by their empirical credibility.

“Further, meta-analyses provide a blunt averaging of estimated effects without allowing readers to differentially weight studies by their empirical credibility.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

How forward-looking a firm is plausibly affects performance and is very likely to affect how diverse a firm is, yet it is not possible to rate firms with a numerical assessment in a principled and consistent manner.

“how forwardlooking a firm is plausibly affects performance and is very likely to affect how diverse a firm is, yet it is not possible to rate firms with a numerical assessment in a principled and consistent manner.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Wang et al.'s meta-analysis, in a sample of 44 studies, found that surface-level diversity had a negative association with metrics of creativity and innovation, though the average effect size was not statistically different from zero.

“In their sample of 44 studies, they found that surface-level diversity had a negative association with metrics of creativity and innovation, though the average effect size was not statistically different from zero.59”

• cites [59]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence.

Qualitative literature reviews necessarily rely on the selection choices and interpretation of the authors of the reviews.

“such reviews necessarily rely on the selection choices and interpretation of the authors of the reviews.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

In the absence of credible experiments, researchers have developed approaches to attempt to surmount omitted variable bias while achieving external validity.

“In the absence of credible experiments, researchers have developed a number of approaches to attempt to surmount the omitted variable bias problem while achieving external validity.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In both statistical analysis and legal analysis, we are not able to observe the counterfactual world.

“Unfortunately, in both statistical analysis and legal analysis, we are not able to observe the counterfactual world.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Stewart's results are similar to results reported in two other independent meta-analyses (Bowers et al., 2000; Webber & Donahue, 2001).

“This is similar to results reported in two other independent meta analyses (Bowers et al., 2000; Webber & Donahue, 2001).”

• cites Bowers et al., 2000Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Webber & Donahue, 2001Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

This approach to determining how unlikely it would be to observe an effect of a given size is susceptible to p-hacking concerns.

“This approach to determining how unlikely it would be to observe an effect of a given size is susceptible to the p-hacking concerns raised above.”

• cites [72]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A meta-analysis strips out many of the details of the individual studies.

“however, such an analysis strips out many of the details of the individual studies”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Including all relevant explanatory variables will never be possible in practice.

“Unfortunately, including all of the relevant explanatory variables will never be possible in practice.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If a researcher estimates 100 regressions on a random sample from a population where there is no correlation between outcome and predictor variables, she is likely to find five effect estimates that are statistically significant.

“if the researcher estimates 100 regressions even on a random sample from a population where there is no correlation between the outcome variables and the predictor variables, she is likely to find five effect estimates that are statistically significant.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Meta-analyses generally provide a protocol for how articles were identified and chosen.

“Meta-analyses generally provide a protocol for how articles were identified and chosen.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

If she is only matching on characteristics for which data exists, then the regression approach could have made the necessary adjustments already.

“Arguably, if she's only matching on characteristics for which data exists, then the regression approach could have made the necessary adjustments already.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If a researcher estimates a huge number of treatment effects, the probability she finds some extreme ones grows even if the results are driven by chance.

“If a researcher estimates a huge number of treatment effects, the probability she finds some extreme ones grows even if the results are driven by chance.”

Probably not a substantial claim.

From the resulting sample, quantitative estimates are extracted and treated as data points from which descriptive statistics can be calculated.

“From the resulting sample, the quantitative estimates are extracted and treated as data points from which descriptive statistics can be calculated, such as the mean effect size found in the literature, as well as more detailed information about the distribution of the effect sizes.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The generalized method of moments (GMM) estimation technique uses lagged explanatory variables as instruments.

“A fancier sounding but conceptually similar approach to instrumental variables is the generalized method of moments (GMM) estimation technique that uses lagged explanatory variables as the instruments.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

The researcher is unlikely to know all of the variables that should be in the model.

“First, the researcher is unlikely to know all of the variables that should be in the model.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

When estimating model coefficients, sometimes random correlations arise by mere chance.

“When estimating the model's coefficients (or really, even when calculating a simple correlation coefficient), sometimes random correlations arise by mere chance.”

It may state two separate claims as one.

Wang et al. found a positive relationship between deep-level cultural diversity and creativity, and the average effect is statistically different from zero, at least for teams that met face to face rather than via technological means.

“there was a positive relationship between deep-level cultural diversity (which the authors describe as”

• Also stated in: § Managing diversity enhances organizational flexibility. · plain

  “unobservable attributes, including personalities, values, and attitudes”

• Also stated in: § Managing diversity enhances organizational flexibility. · plain

  “and creativity and the average effect is statistically different from zero, at least for teams that met face to face rather than via technological means.61”

• cites [60]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [61]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

It is useful to have some method to distinguish between a relationship not driven by chance and those that are.

“It is useful to have some method to distinguish between a relationship between the X and Y variables that is not driven by chance and those that are.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Many believe that the best (and some believe the only) way to isolate causality is through randomized controlled experiments.

“Many believe that the best (and some believe that the only) way to isolate causality is through the use of randomized controlled experiments.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

The stakes involved in real world decisions are often much larger than those in experiments, which could substantially change the outcomes of injecting more or less diversity into an organization.

“Likewise, the stakes involved in real world decisions will often be much larger than those seen in experiments, which could substantially change the outcomes of injecting more or less diversity into an organization.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A p-value indicates how likely one would observe an estimate with at least as large a magnitude as the one calculated by chance alone if the population correlation were zero.

“a researcher often provides a”

• Also stated in: § 4 | CAUSAL INFERENCE PRIMER · plain

  “p value”

• Also stated in: § 4 | CAUSAL INFERENCE PRIMER · hedged

  “that indicates how likely one would observe an estimate with at least as large a magnitude as the one calculated by chance alone if the population correlation were zero.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

A meta-analysis allows one to summarize a literature in a systematic way.

“The benefit of a meta-analysis is that it allows one to summarize a literature in a systematic way”

Probably not a substantial claim. It may be too vague to check against evidence.

How far experimental results on diversity can be extrapolated is largely unknowable.

“How far experimental results can be extrapolated is largely unknowable.”

It may state two separate claims as one.

Unless one assumes that lagged variables are unrelated to each other, the GMM approach generates unreliable causal estimates, and the same critique of the diagnostic overidentification test applies in the GMM case.

“Unless one is willing to assume that these lagged variables are unrelated to each other (and few people have intuitions about, for instance, how the X variable lagged three periods relates to the X variable lagged four periods), this GMM approach generates unreliable causal estimates as well, and the same critique of the diagnostic overidentification test applies in the GMM case.79”

• cites [79]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Subsequent reviews provide more empirical grounding for the claimed benefits of diversity.

“Fortunately, subsequent reviews provide more empirical grounding for the claimed benefits of diversity.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Even when experiments can be run, the artificial context might lead individuals to behave differently than they would have outside an experiment.

“the artificial context of the experimental setting might lead individuals to behave differently than they would have had they not been part of an experiment.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If all variables affecting firm performance (or all correlated with the diversity treatment) are included in the regression, it would be accurate to interpret the b1 coefficient as the causal effect of diversity on firm performance.

“If the researcher were able to ensure that all of the variables affecting firm performance are included in the regression (or, at least, all of the variables affecting firm performance that are correlated with the diversity treatment), then it would be accurate to interpret the b 1 coefficient as the causal effect of diversity on firm performance.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The biggest problem with matching approaches is that a researcher can only match on those things for which she has data.

“the biggest problem is that a researcher can only match on those things for which she has data.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

The external validity or relevance problem might reasonably lead to doubts about the expected effects of the treatment outside the experimental setting.

“This external validity or relevance problem might reasonably lead to doubts about the expected effects of the treatment outside of the experimental setting.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The OLS method is general enough to accommodate additional variables.

“Luckily, the OLS method is general enough to accommodate additional variables.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Coincidental correlations happen with some frequency.

“That is, coincidental correlations happen with some frequency.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

If the primary concern was about variables for which the researcher had no data, matching continues to have that concern.

“if the primary concern was about variables for which the researcher had no data, matching continues to have that concern.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Matching approaches attempt to solve the comparator problem by finding entities in the dataset that are comparable along a host of dimensions, with one entity having received the treatment and one or more comparable entities not having received it.

“Matching approaches attempt to solve the comparator problem by finding entities in the dataset that are comparable along a host of dimensions and where one of the entities in the match has received the treatment the researcher wishes to study and one (or more) comparable entity that has not received the treatment.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The model can estimate as many parameters as there are data points in the sample being analyzed.

“Formally, the model can estimate as many parameters as there are data points in the sample being analyzed.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

The researcher is unlikely to have data for all of the variables she thinks should be in the model.

“Second, the researcher is unlikely to have data for all of the variables she thinks should be in the model.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Stahl et al.'s surface-level diversity indicator includes race among other indicators of diversity, but there is no presentation of how many of the studies actually examined race, as opposed to one of the other indicators of diversity.

“although Stahl et al.'s surface level diversity indicator includes race among other indicators of diversity, there is no presentation of how many of the studies actually examined race, as opposed to one of the other indicators of diversity”

• cites [56]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A crucial untestable requirement of the overidentification test is that each of the instruments is not suffering from the same bias.

“A crucial untestable requirement of this test, though, is that each of the instruments is not suffering from the same bias.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

While there is nominally a test for the second condition, it is not credible in the vast majority of cases.

“While there is nominally a test for the second condition, it is not credible in the vast majority of cases.”

Probably not a substantial claim. It may be too vague to check against evidence.

Separating systematic relationships from random coincidences in these situations is not straightforward.

“Separating out the systematic relationships from the random coincidences in these situations is not straightforward.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

A statistically significant result might be uninteresting or of trivial importance.

“A statistically significant result might be uninteresting or of trivial importance.”

It may state two separate claims as one.

Stewart concluded that the magnitude of relationships with heterogeneity is still rather small, suggesting that forming teams based on heterogeneity of member characteristics is not as important as choosing team members with high ability, expertise, and desirable personality traits.

“Nevertheless, the magnitude of relationships with heterogeneity is still rather small, suggesting that forming teams based on heterogeneity of member characteristics is not as important as choosing team members with high ability, expertise, and desirable personality traits.55”

• cites [55]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

Long-run effects of diversity may be very different from what is observed in short-term experiments.

“Long-run effects may be very different from what is observed in the necessarily short-term experiments studied by researchers.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The first condition for instrumental variables (strong relationship with treatment) is testable, while the second (uncorrelated with missing variables) is not.

“The first of these conditions is testable, while the second is not.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If the estimate is more than 2.58 standard errors away from zero, the likelihood of observing such an extreme value at random is less than 1%.

“If the estimate is more than 2.58 standard errors away from zero, the likelihood of observing such an extreme value at random is less than 1%.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Assessment of the empirical record can be important in justifying or invalidating statutory interventions.

“As seen in a recent case in California, assessment of the empirical record can be important in justifying or invalidating statutory interventions.16”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

For the instrumental variables approach to succeed, the instrument must exhibit a strong relationship with the treatment variable, and the instruments must really be uncorrelated with any of the model's missing variables.

“For this approach to succeed, two things need to be true.The instrument must exhibit a strong relationship with the treatment variable, and the instruments must really be uncorrelated with any of the model's missing variables.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The control group accounts for changes that would have occurred anyway, perhaps due to natural background trends.

“In essence, the control group accounts for changes that would have occurred anyway, perhaps due to natural background trends.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

For those familiar with empirical work, the uniformity of the presented results might raise suspicion because real world data do not conform so neatly to any given policy position.

“For those familiar with empirical work in general, the uniformity of the presented results might raise some suspicion as, most of the time, real world data do not conform so neatly to any given policy position.”

• AssumptionRests on an assumption: Real world data do not conform so neatly to any given policy position.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

OLS ensures that the estimated a and b parameters are the values that generate the best fit to the data among linear unbiased estimates.

“OLS ensures, via a calculus optimization problem, that the estimated a and b parameters are the values that generate the best fit to the data among linear unbiased estimates.67”

• cites [67]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Statistical significance does not imply a causal relationship.

“statistical significance does not imply a causal relationship.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Stahl et al.'s meta-analysis examines the effect of racial diversity (among other surface-level diversity attributes) across 108 empirical studies that analyzed 10,632 work teams.

“An extremely well-cited meta-analysis by Stahl and coauthors examines the effect of racial diversity (among other attributes they dub”

• Also stated in: § Managing diversity enhances organizational flexibility. · plain

  “surface-level”

• Also stated in: § Managing diversity enhances organizational flexibility. · plain

  “diversity) across 108 empirical studies that analyzed 10,632 work teams”

• cites [56]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

A statistically significant result could still suffer from omitted variable bias.

“A statistically significant result could still suffer from omitted variable bias”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Sometimes experiments are thought to be unethical or otherwise not feasible.

“Sometimes experiments are thought to be unethical or are otherwise not feasible.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Social desirability bias might lead researchers to abandon projects that do not support causes favored by themselves and their colleagues.

“Second, social desirability bias might lead researchers to abandon projects that do not support causes favored by themselves and their colleagues.”

It may state two separate claims as one.

Ahern and Dittmar's Table III shows that the quota announcement led to about a 4% reduction in stock returns for Norwegian firms that previously had no female directors, relative to those with at least one director and relative to same-industry U.S. firms, robust to many specifications.

“Table III shows that their estimated treatment effect of the quota announcement leading to about a 4% reduction (Norwegian dummy row) in the stock returns for Norwegian firms that previously had no female directors, relative to those with at least one director and relative to same-industry U.S. firms, is robust to many different specifications”

Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

A firm's headquarters location can have a direct effect on firm performance, so it is hard to argue it has no possible direct effect.

“It is hard to argue that a firm's headquarters location has no possible direct effect on firm performance.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If one examines enough correlations, eventually there will be statistically significant ones even if the outcome is entirely unaffected by the main variable of interest.

“As indicated above, if one examines enough correlations, eventually there will be a number of statistically significant ones even if an outcome is entirely unaffected by the main variable of interest.”

It may state two separate claims as one.

Yang and Konrad found mixed results in the main effects, no statistically significant two-way effects, and mixed results in the three-way effects.

“The authors find mixed results in the main effects, no statistically significant two-way effects, and mixed results in the three-way effects.106”

• cites [106]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The synthetic control approach developed by Abadie et al. is a relatively new approach in the quasi-random/natural experiment category.

“A relatively new approach in the quasi-random/natural experiment category is the synthetic control approach developed by Abadie et al. 82”

• cites [82]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper may give no reason why it holds.

If the synthetic comparator was a good match in the pre period, it presumably would have been a good match for the treated unit in the counterfactual world where treatment did not occur.

“The intuition behind the design is that if the synthetic comparator was a good match in the pre period, it presumably would have been a good match for the treated unit in the counterfactual world where treatment did not occur.”

It may state two separate claims as one.

Griffin et al.'s results are mostly correlational, but they attempt to leverage the Norwegian law change to isolate causal effects.

“Most of the authors' results are merely correlational with no examination of natural experiments or even plausible instruments; the article does, however, examine the Norwegian law change discussed above and attempts to leverage it to isolate the causal effect of gender diversity on the examined innovation measures.”

It may state two separate claims as one.

While there are examples of credible research designs in the business case for diversity literature, much of the research is not methodologically rigorous.

“As seen in the next section, while there are examples of credible research designs in the literature on the business case for diversity, much of the research is not methodologically rigorous.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

As long as the cut-off is truly arbitrary, confidence is strong in the causal interpretation of any estimated difference between the subset of the two groups.

“As long as the cut-off is truly arbitrary, confidence is strong in the causal interpretation of any estimated difference between this subset of the two groups.”

It may state two separate claims as one.

After showing firms met the quota's parity requirement, Bertrand et al. found no evidence of effects beyond this.

“After showing that firms were able to meet the quota's requirement that boards exhibit parity between men and women, they found no evidence of effects beyond this.”

• cites [122]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In this context, the natural experiment is referred to as an event study and comparator firms are sometimes used to adjust for background trends.

“In this context, the natural experiment is referred to as an event study and comparator firms are sometimes used to adjust for background trends.”

It may state two separate claims as one.

Bernile et al.'s instrumental variable estimates do not include firm fixed effects, so their correlations may include the effects of unobservable firm characteristics.

“Potentially concerning is the fact that their IV estimates do not include firm fixed effects and so there remains the possibility that their correlations include the effects of unobservable characteristics of the firm.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

A much broader range provides a larger sample size, but comparability is in greater doubt.

“A much broader range provides a larger sample size, but comparability is in greater doubt.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Such nuance creates empirical difficulties, including temptation to p-hack, especially when theoretical frameworks are hard to distinguish from ex post rationalization of absent main effects.

“As suggested above, although such nuance may be legitimate, it creates empirical difficulties in terms of the temptation to engage in p-hacking, especially when it is difficult to distinguish the theoretical frameworks from ex post rationalization of the absence of main effects.94”

• cites [94]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The Matsa and Miller result that the Norwegian mandate lowered short-term profits is an artifact of the 2008 financial crisis and did not endure.

“Matsa and Miller result is an artifact of the 2008 financial crisis and did not endure even though the mandate remained in place.120”

• cites [120]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The authors make no attempt to instrument for their diversity metric, so its main and interactive effects are subject to omitted variable bias criticism.

“In any event, the authors make no attempt to instrument for their diversity metric, meaning that its main effect and any of its interactive effects on performance will be subject to the omitted variable bias criticism.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The product recall paper does use firm-level fixed effects and an interesting instrumental variable analysis, but it appears not to account for dependence in standard errors, which could be a substantial problem given many significant effects are near the border.

“They appear, however, to not account for dependence in their standard errors, which could be a substantial problem given that many of their statistically significant effects are close to the border of not being significant.”

It may state two separate claims as one.

Griffin et al.'s instrument is indistinguishable from firm fixed effects and does not provide exogenous variation.

“Further, although the authors frame the analysis as using an instrument to eliminate omitted variable bias, their instrument is indistinguishable from their firm fixed effects (since the instrument, the director ratio in 2002, does not change for a given firm throughout time); they create the appearance of being able to estimate it separately from the firm fixed effect by interacting it with”

It may state two separate claims as one.

When Green and Hand attempted to replicate McKinsey's analyses, McKinsey was unwilling to share its data or identify its sample firms.

“In fact, when Green and Hand attempted to replicate the analyses presented in these reports, McKinsey was unwilling to share its data or to even identify which firms comprised its sample.99”

• cites [99]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Perhaps the most promising attempt to address omitted variable bias is the use of natural or quasi-experiments.

“Perhaps the most promising attempt to address the omitted variable bias is the use of natural or quasi-experiments.”

It may state two separate claims as one.

Nuber and Velte's own robustness results show that board size has a separate statistically significant effect on emissions even when female board membership variables are controlled for, suggesting board size cannot be a valid instrument.

“Beyond that, their own robustness results show that board size has a separate statistically significant effect on emissions even when female board membership variables are controlled for, 139 again suggesting that board size cannot be a valid instrument.”

• cites [139]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

There is no evidence of improvements for women working in firms most affected by the Norwegian reform, suggesting new female board members are not significantly changing opportunities for women within the firm.

“We see no evidence of improvements for women working in firms most affected by the reform, suggesting that the new female board members are not significantly changing the opportunities for women within the firm.”

• cites [123]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Regression discontinuity is available when the application of a rule or policy hinges on an effectively arbitrary cut-off point, with entities above the cut-off assigned to treatment and entities below left untreated.

“The technique is available in situations where the application of a rule or a policy hinges on an effectively arbitrary cut-off point in which entities above the cut-off are assigned to the treatment group while entities below the cut-off are left untreated.81”

• cites [81]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Entities are not randomly assigned diverse personnel, with some otherwise similar firms not getting those diverse workers.

“In the case of the effect of diversity on organizational performance, regardless of the empirical method employed, the concern is that entities are not randomly assigned diverse personnel, with some other otherwise similar firms not getting those diverse workers.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Natural experiments adopt the experimental framework but examine behavior in the real world so as not to run into external validity concerns.

“These approaches adopt the experimental framework (i.e., examine the change in outcomes before and after some treatment as compared to the contemporaneous change observed in entities not receiving the treatment) but examine behavior in the real world so as not to run into the external validity concerns raised above.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Green and Hand argue that McKinsey examines firm performance prior to the diversity metrics, which is more consistent with performance causing diversity rather than diversity affecting performance.

“Further, they raise the concern that McKinsey examines the relationship between firm performance in the period prior to the diversity metrics used as explanatory variables, which Green and Hand argue is more consistent with examining whether performance causes diversity rather than diversity affecting performance.103”

• cites [103]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The papers California relied on do little to ensure causality, making elementary mistakes like faulty instruments, standard errors not accounting for dependence, and overclaiming in the face of mixed results.

“They do little to ensure causality, making elementary mistakes like using faulty instruments (if they try anything at all to address omitted variable biases), using standard errors that do not account for dependence, and generally overclaiming in the face of mixed results.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Credible estimates of diversity's effect will seek to exploit serendipitous changes to race and sex composition as natural experiments.

“Credible estimates, instead, will seek to exploit serendipitous changes to the race and sex composition of these organizations as natural experiments.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

For some group of entities close to the cut-off, assignment was as good as random.

“The intuition is that for some group of entities close to the cut-off, assignment was as good as random.”

It may state two separate claims as one.

Nuber and Velte's suggestion that board size does not change often is actually a problem because it makes it more likely their estimates pick up effects of unaccounted constant variables that likely affect firm performance, given they do not control for firm fixed effects.

“when this actually makes it more likely that their estimates are picking up the effects of all sorts of unaccounted for constant variables that likely affect firm performance given that they do not control for firm fixed effects.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Standards in empirical social science have strengthened significantly over the last quarter century, leading to more rigorous research designs and greater credibility.

“Standards in empirical social science have been strengthened significantly over the last quarter century, leading to more rigorous research designs and, ultimately, greater credibility of the research.89”

• cites [89]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Ahern and Dittmar's Table VI shows additional women on the board were associated with increased firm leverage and increased acquisition activity.

“Performance-wise (Table VI), additional women on the board were associated with increased firm leverage and increased acquisition activity.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Griffin et al. find a statistically significant positive relationship between their diversity metric and patents, citations, and citation-weighted patents.

“Following this approach, they find a statistically significant positive relationship between their diversity metric and patents, citations, and citation-weighted patents.126”

• cites [126]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Since its introduction in 2010, the synthetic control method has been refined and used quite extensively.

“Since its introduction in 2010, the method has been refined and used quite extensively.85”

• cites [85]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

Pensions can affect firm performance directly.

“and there is a large literature on how pensions can affect firm performance directly.136”

• cites [136]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

California invoked a large empirical literature in its court defense of its diversity laws.

“Beyond the statutes themselves, California invoked a large empirical literature in its court defense of its diversity laws.”

It may state two separate claims as one.

Bernile et al. find that more diverse boards lead to less stock return volatility and improved financial metrics.

“Bernile et al., use an instrumental variables approach to examine the effect of board diversity on stock return volatility and a host of financial metrics, generally finding that more diverse boards lead to less volatility and improved financial metrics.”

• cites [129]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Focusing on most cited studies inevitably focuses on older papers that have had time to accrue citations and become influential.

“Also, such an approach would inevitably focus on older papers that have had time to accrue citations and become influential.”

It may state two separate claims as one.

Nuber and Velte examine European firms over 2009-2018 and find that total emissions decline as the number of female board members increases.

“Nuber and Velte examine European firms over the period 2009-2018 finding that total emissions decline as the number of female board members increases.”

• cites [134]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Special care must be taken with natural experiment models to account for statistical dependence across multiple observations of the same individual.

“Special care, however, must be taken with these models to account for the statistical dependence that is likely to exist across the multiple observations of each given individual.”

Probably not a substantial claim. The paper may give no reason why it holds.

The blunt averaging that takes place in meta-analyses does not provide any way to weight research results by methodological rigor.

“The blunt averaging that takes place in meta-analyses likewise does not provide any way to weight research results by methodological rigor.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Eckbo's paper draws into doubt the conclusion that the mandate worsened the value and performance of affected Norwegian firms.

“Although the Matsa and Miller work provided more detail supporting the Ahern and Dittmar market reaction story, Eckbo's paper draws the conclusion that the mandate worsened the value and performance of the affected Norwegian firms into doubt.121”

• cites [121]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The Yang and Konrad study is purely correlational and provides no plausible causal identification strategy.

“Putting aside the mixed results, the survey-based indicators, the small sample size (n = 182), and the highly conditional hypotheses tested with the various interacted effects, this study is purely correlational.It provides no plausible causal identification strategy.”

It may state two separate claims as one.

Nuber and Velte purport to address omitted variable bias through instrumental variables and GMM techniques, but it is not clear they do so successfully.

“Empirically, while the paper purports to address the omitted variable bias problem through instrumental variables and GMM techniques, it is not clear the paper does so successfully.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Nuber and Velte's paper does not wrestle with the possibility that reduced emissions, while socially beneficial, might not be in the best interests of the firm itself.

“First, conceptually, the paper does not really wrestle with the possibility that reduced emissions, while socially beneficial, might not be in the best interests of the firm itself.”

Probably not a substantial claim. It may state two separate claims as one.

The credibility of event studies depends on whether the event/shock was as good as random and whether comparator firms provide a compelling proxy for the treatment firm's counterfactual scenario.

“the important inquiry regarding the credibility of such studies is whether the event/shock was in fact as good as random and whether the comparator firms provide a compelling proxy for the treatment firm's counterfactual scenario.”

Probably not a substantial claim.

Reading the most influential papers shows researchers repeatedly struggle with the empirical finding that diversity's effects are decidedly mixed where any effect is detectable.

“That said, reading through these papers does provide context for how researchers focusing on organizational diversity repeatedly struggle with the empirical finding that the effects of diversity are decidedly mixed in those cases where any effect is detectable at all.”

It may state two separate claims as one.

Nuber and Velte provide GMM estimates without providing any intuition as to why lagged performance and firm characteristics are exogenous, or why we should be confident they are not endogenous.

“They also provide GMM estimates without providing any intuition as to why lagged performance and firm characteristics are exogenous, much less why we should be confident that they are not endogenous in similar ways since it is necessary to assume they are not for tests of overidentifying restrictions to generate any confidence.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Eckbo and coauthors present evidence that Ahern and Dittmar's earlier results weaken when other plausible timing scenarios are used as the shock.

“They present evidence that the earlier results weaken when other plausible timing scenarios are used as the shock.”

• cites [118]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

There is a well-known preference for research finding statistically significant effects, suggesting researchers may abandon projects where diversity does not affect outcomes or editors may be less likely to accept such papers.

“First, there is a well-known preference 90 for research that finds statistically significant effects, suggesting that researchers might abandon projects in which diversity does not affect outcomes or, if the authors persevere, journal editors may be less likely to accept papers that do not find statistically significant effects.”

• cites [90]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Griffin et al. (2021) found that greater female board representation is associated with more patents and better-cited patents.

“A 2021 paper by Griffin et al. specifically examines the relationship between board gender diversity and firm innovation in a sample of firms throughout the world as proxied by patent counts and citation-weighted patent counts, finding that greater female board representation is associated both with more patents and better-cited patents.125”

• cites [125]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Natural experiments focus on real-world shocks where it is easier to have credible intuitions about whether the treatment shock is otherwise related to the outcome variable.

“natural experiments focus on shocks in the real world where it is easier to have credible intuitions about whether the treatment shock is otherwise related to the outcome variable.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

California's primary expert in its defense of SB 826 and AB 979 was Alison Konrad.

“In its defense of both SB 826 and AB 979 in state court, California's primary expert was Alison Konrad, the inaugural Holder of the Corus Entertainment Chair in Women in Management at the Ivey Business School, Western University Canada.105”

• cites [105]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

Progressive firms often have different personnel, and their workforces change differentially than less progressive firms.

“Progressive firms will often have different personnel, and those workforces will change differentially than less progressive firms.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

A better research design does not guarantee cleaner or more compelling results; its value is that findings have more potential for credibility.

“As an interesting aside, just because researchers use a more compelling research design, it does not guarantee the researcher will find cleaner or more compelling results.The real world is messy.Rather, the value of the better research design is that any findings have more potential for credibility.122”

• cites [122]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

Board size can be related to firm performance directly.

“Conceptually, board size can surely be related to firm performance directly (either because smaller boards may be more cohesive or larger boards might draw in value-adding perspectives independent of female-specific effects)”

• cites [136]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Eckbo and coauthors note that Ahern and Dittmar fail to account for cross-sectional dependence in returns from unmodeled macroeconomic or marketwide phenomena; once accounted for, there was no statistically significant effect of the Norwegian quota on firm returns.

“Further, they also note that while Ahern and Dittmar account for dependence over time within firms, they fail to account for cross-sectional dependence in returns that arises because of unmodeled macroeconomic or other marketwide phenomena.Once this dependence is also accounted for, Eckbo and coauthors present evidence that there was no statistically significant effect of the Norwegian quota on”

• cites [118]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The paper is rife with methodological errors, including not accounting for dependence in panel data and using a random effects model instead of fixed effects.

“Beyond these concerns, the paper is rife with methodological errors.First, despite using repeated observations for each firm (i.e., panel or longitudinal data), they do not account for dependence in their data, implicitly assuming that observations for a firm are statistically independent.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Yadav and Lenka provide a comprehensive accounting of the most influential papers in this literature, covering 785 papers published between 1990 and 2019.

“Yadav and Lenka provide a comprehensive accounting of the most influential papers in this literature, covering 785 papers published between 1990 and 2019.93”

• cites [93]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.

Existing reviews do not sufficiently explain how various papers succeed or fail in isolating the causal effect of diversity on performance, leaving non-specialists unable to distinguish between credible and non-credible research designs.

“Perhaps the existing reviews do not sufficiently explain how various papers succeed or fail in isolating the causal effect of diversity on performance, leaving non-specialists unable to distinguish between credible and non-credible research designs.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

California offered a 2018 paper by Chelsea Liu that found firms with more women on their boards were sued less often for environmental infractions.

“For example, California offered a 2018 paper by Chelsea Liu that found that firms with more women on their boards were sued less often for environmental infractions.114”

• cites [114]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Liu does little to account for omitted variable bias; lagged diversity variables do not mitigate endogeneity because past unobservable firm characteristics may be similar to current ones.

“Additionally, Liu does little to account for omitted variable bias.She claims that her use of lagged diversity variables mitigates endogeneity, but this is not true as the unobservable characteristics of the firm in the past may (and likely are) quite similar to the current unobservable characteristics.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The extent of improvements in empirical social science standards depends on the academic discipline and, to some extent, the research topic.

“However, the extent of these improvements depends on the academic discipline and even, to some extent, on the research topic itself.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Natural experiment models are interrogated on two dimensions: whether a credible story exists that the shock is related to the outcome outside its effect on treatment, and whether the comparison group is a plausible counterfactual proxy.

“These models are interrogated on two dimensions: (1) is it possible to tell a credible story where the shock is related to the outcome variable outside of its effect on the treatment being studied (e.g., did the shock change other relevant variables in addition to changing the treatment status of some entities?);and (2) is the comparison group a plausible counterfactual proxy or is it possible to”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Clustering standard errors to account for dependence has become standard.

“clustering standard errors to account for this dependence has become standard.87”

• cites [87]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

There are problems with Griffin et al.'s identification approach, including that patents and citations come after indeterminate lags from R&D decisions, so a relationship at time t may not reflect current firm decisions.

“First, given that patents come after an indeterminant lag after the underlying research and development decisions are made, it is questionable whether a relationship between a firm characteristic at time t and a patent outcome at time t has anything to do with firm decisions at time t, as opposed to firm decisions that occurred well in the past.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Natural experiments combine the experimental structure with the intuition of the instrumental variable approach where the chosen instrument is intuitively random.

“In a sense, natural experiments combine the experimental structure with the intuition of the instrumental variable approach where the chosen instrument is intuitively random.”

It may state two separate claims as one.

Many papers develop theoretical frameworks explaining why diversity effects are highly contextual, leading empirical studies to examine interactive or moderating effects rather than general direct effects.

“Many of the papers end up developing theoretical frameworks to explain why effects of diversity are likely highly contextual and so empirical studies need to examine interactive or moderating effects as opposed to general direct effects of diversity on organizational performance.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

In the diversity literature, a natural experiment approach is to use market reactions in terms of stock market returns as the outcome variable and shocks to the diversity metric to identify the causal effect of diversity on firm performance, controlling for other influences on firm returns such as overall market or industry returns.

“one particular example of a natural experiment approach is to use market reactions in terms of the firm's stock market returns (as well as other indicators of performance such as accounting measures) as the outcome variable and shocks to the diversity metric (such as policy changes) to identify the causal effect of diversity on firm performance, controlling for other influences on the firm's”

It may state two separate claims as one.

The authors find some evidence that the share of women obtaining business-related graduate degrees might have improved after the Norwegian reform, though they suggest it may be unrelated to the reform or reflect mean reversion.

“The authors do find some evidence that the share of women obtaining graduate degrees relevant for business careers might have improved after the Norwegian reform, though they suggest this improvement may have been unrelated to the reform since it appears to also show up in the female share of graduate degrees that are less related to business.”

• cites [124]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In the extremely narrow case, confidence in comparability of treatment and control groups is intuitively higher, but one may be left with too few data points to make a valid statistical comparison.

“In the extremely narrow case, where there are just a few treatment units and a few control units, confidence in the comparability of the two groups is intuitively higher, but one may be left with too few data points to make a valid statistical comparison.”

It may state two separate claims as one.

Because firms rarely move headquarters, the authors have little within-firm variation in the instrument over their five-year sample, so the instrument will tend to be correlated with many unquantifiable firm characteristics that likely affect performance.

“Further, since firms tend not to move headquarters very frequently, the authors have very little within-firm variation in this instrument over their five-year sample period, meaning that the instrument will tend to be correlated with many unquantifiable characteristics of a firm that also are likely to affect performance.”

It may state two separate claims as one.

Ahern and Dittmar perform placebo analyses in other Scandinavian countries and the U.S., finding no statistically significant effect in the same time period.

“To rule out coincidental background effects driving this result, they perform placebo analyses (i.e., run the same natural experiment in places where the law is not changing) in other Scandinavian countries and in the U.S., not finding any statistically significant effect in the same time period.”

It may state two separate claims as one.

Propensity score matching only accounts for observable characteristics, not unobservable ones, so it does not mitigate omitted variable bias.

“As discussed above, matching only accounts for observable characteristics, not unobservable ones.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

If outcomes unrelated to the treatment being studied also differ at the cut-off, it is likely that the treatment and control groups differ in systematic ways beyond whether they received the treatment.

“All other things equal, if these outcomes unrelated to the treatment being studied also differ between the treatment and control groups, it is likely that the treatment and control groups differ in systematic ways beyond whether they received the treatment or not.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Regression discontinuity is a method within the natural experiment umbrella that attempts to solidify the intuition regarding the quasi-randomization process.

“One method within the natural experiment umbrella that attempts to solidify the intuition regarding the quasi-randomization process is the regression discontinuity technique.80”

• cites [80]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

In one of Nuber and Velte's two specifications, their estimates do not pass the test for overidentifying restrictions, providing some evidence that their instruments are not exogenous.

“Indeed, in one of their two specifications, their estimates do not pass the test for overidentifying restrictions, providing some evidence that their instruments are not exogenous.138”

• cites [138]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Natural experiment models can accommodate other control variables as well.

“These models can accommodate other control variables as well.”

It may state two separate claims as one.

Systematic meta-analyses find little evidence of a significant positive effect of diversity on firm performance, but they cannot distinguish or weight studies by methodological merits.

“while systematic meta-analyses find little evidence of a significant positive effect of diversity on firm performance, such studies, by virtue of their breadth, are not able to distinguish or weight studies based on methodological merits.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The synthetic controls model takes seriously the concern that matching on only observable characteristics leaves open the concern that the treatment group and comparator group differ in unmodeled or unobservable characteristics.

“the synthetic controls model takes seriously the concern that matching on only observable characteristics leaves open the concern that the treatment group and the comparator group differ in terms of their unmodeled or unobservable characteristics.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A 2011 article by Konrad and Yang examined the effect of the share of Canadian firms' workforces that were women and a measure of racial and ethnic diversity on organizational innovation.

“For example, a 2011 article by Konrad and Yang examined the effect of the share of Canadian firms' workforces that were women and a measure of racial and ethnic diversity on organizational innovation in the firm.”

• cites [106]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Ahern and Dittmar's Table IV examines a longer-run effect of the Norwegian quota by looking at industry-adjusted Tobin's Q for Norwegian firms before and after the quota became binding, controlling for firm and year fixed effects.

“Ahern and Dittmar's Table IV examines a longer run effect of the Norwegian quota by looking at industry adjusted Tobin's Q for Norwegian firms before and after the quota became binding, controlling for both firm and year fixed effects.”

It may state two separate claims as one.

Most consulting studies do not provide any way to determine statistical significance, and in the few instances where they do, they provide no discussion of technical issues such as how standard errors were calculated.

“Most of the consulting studies do not provide any way to determine statistical significance, and in the few instances where they do, they provide no discussion of technical issues such as how standard errors were calculated.”

Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.

The position that diversity mandates will be good for companies, investors, and the economy relies on a selective reading of the literature and a failure to apply even the most limited demand for rigor.

“This position relies on a selective reading of the literature and a failure to apply even the most limited demand for rigor.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

The Nasdaq stock exchange and the state of California both argued that diversity mandates for firms will be good for companies, their investors, and the economy more generally.

“The Nasdaq stock exchange and the state of California both argued that diversity mandates for firms will be good for companies, their investors, and the economy more generally.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The California state court decision invalidating SB 826 found that Defendant was unable to present specific evidence of actual, unlawful discrimination against any specific woman by any specific corporation subject to SB 826.

“During trial the Court found that Defendant was unable to present specific evidence of actual, unlawful discrimination against any specific woman by any specific corporation subject to S.B. 82.”

• cites [142]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

McKinsey's most recent report on diversity provides haphazard information on which findings are statistically significant, and there is no documentation of what statistical inference approach was used.

“McKinsey's most recent report on diversity provides haphazard information on which of the findings are statistically significant, and there is no documentation of what statistical inference approach was used in making these determinations”

Probably not a substantial claim. It may state two separate claims as one.

After admitting that the report provides no way to assess causality, the McKinsey authors assert that their results very likely demonstrate that diversity adds value to firms.

“after admitting that the report provides no way to assess causality, the authors assert that their results very likely demonstrate that diversity adds value to firms”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The Norway experience has allowed for the study of the subsidiary claim that mandates and other diversity policies might incentivize individuals and firms to develop more of a pipeline of individuals from underrepresented groups to fill firm positions.

“the Norway experience has allowed for the study of one of the subsidiary claims encountered in the diversity literature.Namely, regardless of the short-term effect of diversity on firm performance, mandates and other diversity policies might incentivize individuals and firms to develop more of a pipeline of individuals from underrepresented groups to fill firm positions.”

It may state two separate claims as one.

The Norway stock price impact analysis by Ahern and Dittmar is credible because the 2002 announcement was unanticipated and the estimated treatment effect allows adjustment for contemporaneous background trends via two sets of comparators.

“This stock price impact analysis is credible since the 2002 announcement was unanticipated (making it a powerful natural experiment) and the estimated treatment effect allows the researcher to adjust for contemporaneous background trends via two sets of comparators”

• cites [117]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.