A Randomized Controlled Study on the Effectiveness of Plain Language Summaries of Psychological Meta-Analyses

Classified as probably not a substantial claim.

Hendriks et al. (2015) differentiated three dimensions of epistemic trustworthiness: expertise, integrity, and benevolence.

“Hendriks et al. (2015) differentiated three dimensions of epistemic trustworthiness, i.e. expertise, integrity, and benevolence.”

• cites Hendriks et al., 2015 sourceSupported

Classified as probably not a substantial claim.

Iman (2016) found a negative relationship between publication bias awareness and evidence credibility ratings.

“Based on first empirical evidence of Iman (2016), who found a negative relationship between publication bias awareness and evidence credibility ratings (as well as further findings on trust and questionable research practices, e.g., Anvari & Lakens, 2018), we expected trust in research findings to decrease when providing information on publication bias.”

• cites Iman (2016)Supported

Probably not a substantial claim: may not pass S1 (asserted).

It has been proposed that putting the practical significance of research findings via PLSs into context may improve the public perception and impact of science.

“It has been proposed that putting the practical significance of research findings via PLSs into context may improve the public perception and impact of science (e.g., Kuehne & Olden, 2015).”

• cites Kuehne & Olden, 2015 sourceSupported

Classified as probably not a substantial claim.

Perceived trustworthiness decreases when scientists receive funding from industry collaborators.

“Furthermore, Besley et al. (2017) illustrated that perceived trustworthiness also decreases when scientists receive funding from industry collaborators.”

• cites Besley et al., 2017 sourceSupported

Classified as probably not a substantial claim.

Barbara et al. (2016) found that perceived personal relevance that caregivers and aging citizens attributed to research summaries was a key determinant of their decision to select research summaries for further inspection.

“For example, Barbara et al. (2016) found that the perceived personal relevance that caregivers and aging citizens attributed to research summaries was a key determinant of their decision to select research summaries for further inspection in a usability study on their Optimal Aging Portal.”

• cites Barbara et al. (2016) sourceSupported

May combine two claims.

The omnibus test on group differences was significant for integrity, F(2, 1187.886) = 3.488, p = .031, f = .072, but not for benevolence, F(2, 1189.573) = 1.722, p = .179, f = .050.

“Effect sizes were consistently very small, and the omnibus test on group differences was only significant for integrity, F(2, 1,187.886)= 3.488, p = .031,f = .072,but not for benevolence, F(2, 1,189.573)= 1.722, p = .179,f = .050,which is why these results should be interpreted cautiously.”

May combine two claims.

PLSs yielded a slightly higher proportion of correct responses to knowledge items (58.92%) than scientific abstracts (53.89%), with an odds ratio of 1.33 in the logistic mixed model.

“The proportion of correct responses to knowledge items was slightly higher for PLSs (58.92%) compared to scientific abstracts (53.89%)-with an odds ratio of 1.33 in the corresponding logistic mixed model (see Table E2 in ESM 1).”

May combine two claims.

User experience was higher for PLSs (M = 5.56, SD = 1.59) than for scientific abstracts (M = 4.88, SD = 1.84), and this difference was significant, F(1, 808.514) = 42.659, p < .001, f = .204.

“Additionally, we observed the expected difference between PLSs (M = 5.56, SD = 1.59) and scientific abstracts (M = 4.88, SD = 1.84) for user experience.Mixed model analyses revealed that these differences were significant, F(1, 808.514) = 42.659,p < .001,f = .204.”

May combine two claims.

All other hypothesized group differences for H3 were nonsignificant, and H3a, H3d, and H3e were rejected.

“All other hypothesized group differences were, however, nonsignificant.H3a, H3d, and H3e were therefore rejected.”

May combine two claims.

Credibility ratings of the presented evidence significantly decreased when PLSs indicated that COI existed (M = 5.77, SD = 1.54) compared to when PLSs indicated that COI were absent (M = 6.06, SD = 1.49, p = .002) or no information on COI was provided (M = 5.96, SD = 1.54, p = .044).

“Credibility ratings significantly decreased when PLSs indicated that COI existed (M = 5.77, SD = 1.54) compared to when PLSs indicated that COI were absent (M = 6.06,SD = 1.49, p = .002)or no information on COI was provided (M = 5.96, SD = 1.54, p = .044).”

May combine two claims.

There were no significant differences in perceived personal relevance (F(1, 802.749) = 2.205, p = .138, f = .045) or user experience (F(1, 794.861) = 2.567, p = .110, f = .051) between PLSs that provided a practical relevance statement and PLSs that provided no relevance statement.

“Mixed model results indicate, however, that these differences were not significant for personal relevance, F(1, 802.749) = 2.205, p = .138,f = .045,and user experience, F(1, 794.861) = 2.567, p = .110,f = .051(see Table E4 in ESM 1).”

May combine two claims.

Effects of the practical relevance statement tended to be-at least descriptively-beneficial and not detrimental.

“Still, effects of this statement tended to be-at least descriptively-beneficial and not detrimental and our results therefore remain overall inconclusive.”

May combine two claims.

Evidence on the effectiveness of PLSs of psychological meta-analyses, as well as guidance on how to write such PLSs, is lacking.

“Evidence on the effectiveness of PLSs of psychological meta-analyses, as well as guidance on how to write such PLSs are, however, lacking.”

• cites Stoll et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

H4a, H4b, and H4c were rejected.

“Consequently, H4a, H4b, and H4c were rejected.”

May combine two claims.

In a prior study, German university students rated PLSs to be more comprehensible than the corresponding scientific abstracts and also acquired more knowledge on the summarized studies when reading PLSs.

“In this study, German university students rated PLSs to be more comprehensible than the corresponding scientific abstracts and also acquired more knowledge on the summarized studies when reading PLSs.”

• cites Kerwer, Chasiotis, et al., 2021 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The target sample size was set to 2,268 participants.

“Thus, the target sample size was set to 2,268 participants.”

May combine two claims.

The current results support the unpublished prior findings, and a user experience index was computed as the mean of the three items.

“Consequently, the current results support these unpublished prior findings, and we computed a user experience index as the mean of the three abovementioned items.”

Probably not a substantial claim: may not pass S2 (checkable).

For laypersons, information that helps assessing the trustworthiness of scientific sources is especially important.

“For laypersons, information that helps assessing the trustworthiness of scientific sources (e.g., study authors) is especially important.”

May combine two claims.

H3c and H3f were confirmed.

“As a consequence, H3c and H3f were confirmed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The target quota size specified in the preregistration was not fully achieved for male participants younger than 45 years with Hauptschulabschluss (132 participants instead of 189).

“The target quota size specified in our preregistration was not fully achieved for male participants younger than 45 years with Hauptschulabschluss (132 participants instead of 189; see Discussion).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

2,451 participants responded to at least one outcome variable, with 2,256 completing the survey.

“2,451 participants responded to at least one outcome variable, with 2,256 completing the survey.”

Classified as probably not a substantial claim. May combine two claims.

In a recent study, Hendriks et al. (2016) showed that integrity and benevolence, but not expertise, were affected when ethical aspects were discussed within a research blog.

“In a recent study, their research group (Hendriks et al., 2016) showed that integrity and benevolence, but not expertise, were affected when ethical aspects were discussed within a research blog.”

• cites Hendriks et al., 2016 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Various sources of bias exist that can impact the trustworthiness of research outputs, such as publication bias.

“Various sources of bias exist that can impact the trustworthiness of research outputs, such as publication bias (Boutron et al., 2023).”

• cites Boutron et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The user experience index was only computed for cases that provided data on all three items, and McDonald's ω was .878.

“This user experience index was only computed for cases that provided data on all three items, and McDonald's ω was .878.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A prior study of the project suggested that English PLSs of individual psychological studies achieve their aim of fostering the public understanding of science.

“One exception is a prior study of our project which suggested that English PLSs of individual psychological studies achieve their aim of fostering the public understanding of science (Kerwer, Chasiotis, et al., 2021).”

• cites Kerwer, Chasiotis, et al., 2021 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

All study procedures were approved by the ethics committee of Trier University.

“All study procedures were approved by the ethics committee of Trier University.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study and its hypotheses were preregistered at PsychArchives prior to the start of data collection.

“The study and its hypotheses were preregistered at PsychArchives (https://doi.org/10.23668/psycharchives. 5181) prior to the start of data collection.”

Classified as probably not a substantial claim. May combine two claims.

Being affiliated to companies with a financial interest in the research subject can be construed as a potential persuasive intent, which might negatively affect integrity and benevolence judgements, but not judgements on expertise.

“Hendriks et al. (2016) argued that being affiliated to companies with a financial interest in the research subject at hand can be construed as a potential persuasive intent, which might negatively affect integrity and benevolence judgements, but not judgements on expertise.”

• cites Hendriks et al., 2016 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

This study was preregistered at PsychArchives.

“Open Data Preregistration: This study was preregistered at PsychArchives (https://doi.org/10.23668/psycharchives.5181).”

May combine two claims.

Mixed model and mixed model ANOVA results are comparable for simple designs with complete data, so results of the power analyses can be transferred (as a lower bound) to the mixed model approach.

“However, mixed model and mixed model ANOVA results are comparable for simple designs with complete data, which is why we argue that results of our power analyses can be transferred (as a lower bound) to this mixed model approach (see, for example, Murrar & Brauer, 2018, on the equivalence of both approaches in simple designs).”

• cites Murrar & Brauer, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The authors do not expect the recruitment shortfall to affect the internal validity of the findings, because participants were randomly allocated to experimental conditions and the distribution of quotas across conditions did not differ significantly.

“We do not expect this to affect the internal validity of our findings, since participants were randomly allocated to experimental conditions and the distribution of quotas across conditions did not differ significantly (see R-Output at PsychArchives (https://doi.org/10.23668/ psycharchives.14209)).”

May combine two claims.

Evaluation processes result in a discounting of evidence if a statement indicating a COI is reported, but not if no information on COI is provided.

“Exploratory findings on the credibility of the corresponding evidence (i.e., the meta-analytic findings that were presented by the PLSs) further suggest that these evaluation processes also result in a discounting of evidence if a statement indicating a COI is reported, but not if no information on COI is provided.”

May combine two claims.

The study faced problems in recruiting male participants younger than 45 years with Hauptschulabschluss, and the corresponding target quota size was not fully achieved.

“As mentioned above, a limitation of our study was that we faced problems in recruiting male participants younger than 45 years with Hauptschulabschluss, which is why the corresponding target quota size was not fully achieved.”

May combine two claims.

Content-related knowledge did not decrease with an increasing number of effects reported in the PLS, but user experience decreased when including additional effects.

“Content-related knowledge did not decrease with an increasing number of effects that were reported in the PLS, but user experience decreased when including additional effects.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Differences in user experience between scientific abstracts and PLSs might be even larger in reality, because the abstracts were written in German for the German-speaking sample and explicitly designed to contain the same information as the PLSs.

“However, differences in user experience between scientific abstracts and PLSs might be even larger in reality, since abstracts were written in German for our German-speaking sample and explicitly designed to contain the same information as the presented PLSs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The scientific abstracts analyzed were not actually published abstracts but were written by the study authors.

“With regard to our findings on Research Question 1, one might also object that the scientific abstracts we analyzed were not actually published abstracts but written by us, the study authors.”

Classified as probably not a substantial claim.

Mixed models are more flexible compared to repeated measures ANOVAs with between-level factors, especially when handling missing data.

“We analyzed our data by means of mixed models (i.e., with experimental conditions as factors), as they are more flexible compared to repeated measures ANOVAs with betweenlevel factors-especially when it comes to handling missing data (Brown, 2021).”

• cites Brown, 2021 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Research data and R-Output are available at PsychArchives.

“Research data and R-Output are available at PsychArchives (https://doi.org/10.23668/psycharchives.14209).”

May combine two claims.

PLSs educated readers on scientific matters such as meta-analyses as a methodological approach and publication bias as a problem in science.

“Moreover, we also demonstrated that PLSs educated readers on scientific matters such as meta-analyses as a methodological approach and publication bias as a problem in science.”

May combine two claims.

H6a and H6b were rejected.

“Therefore, H6a and H6b were rejected.”

May combine two claims.

Including COI statements supports laypersons in evaluating scientific evidence, while the evidence on further PLS characteristics was mixed.

“Investigating PLSs of psychological meta-analyses in a general population sample, we showed that including COI statements supports laypersons in evaluating scientific evidence, while the evidence on further PLS characteristics was mixed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The actual power of mixed models might be slightly higher in this specific case because incomplete cases can be included.

“The actual power of mixed models might be slightly higher in our specific case because incomplete cases can be included (i.e., if participants do not provide data on both research summaries).”