What is open peer review? A systematic review

“” (Suber, 2012) Page 4 of 32 Within just these two examples, there are already a multitude of factors at play, including the removal of anonymity, the publishing of review reports, interaction between participants, crowdsourcing of reviews, and making manuscripts public pre-review, amongst others.”

“Reviewers often fail to detect major methodological failings (Schroter et al., 2004), with eminent journals (whose higher rejection rates might suggest more stringent peer review processes) seeming to perform no better than others (Fang et al., 2012).”

“Nickerson et al. (2013) advise that the development of a taxonomy should begin by identifying the appropriate meta-characteristic – in this case distinct individual innovations to the traditional peer review system.”

“Published reports can enable readers to consider these criticisms themselves, and “have a chance to examine and appraise this process of ”creative disagreement” and form their own opinions” (Peters & Ceci, 1982).”

“Next, “confirmatory bias” is theorized to lead to conservatism, biasing reviewers against innovative methods or results contrary to dominant theoretical perspectives (Chubin & Hackett, 1990; Garcia et al., 2016; Mahoney, 1977).”

“, 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al.”

“Smith, an experienced editor, for example, reports reviewers stealing ideas and passing them off as their own, or intentional blocking or delaying publication of competitors’ ideas through harsh reviews (Smith, 2006).”

“, 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al.”

“…of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al., 1977).”

“A main reason for maintaining author anonymity is that it is assumed to tackle possible publication biases against authors with traditionally feminine names, from less prestigious institutions or non-English speaking regions (Budden et al., 2008; Ross et al., 2006).”

“Whatever the cause, recent sharp rises in the number of retracted scientific publications (Steen et al., 2013) testify that peer review sometimes fails in its role as the gatekeeper of science, allowing errors to enter the literature.”

“This system is perhaps more recent than one might expect, with its main formal elements only in general use since the mid-twentieth century in scientific publishing (Spier, 2002).”

“Moreover, factors such as close disciplinary communities and internet search capabilities, mean that author anonymity is only partially effective, with reviewers shown to be able to identify authors in between 26 and 46 percent of cases (Fisher et al., 1994; Godlee et al., 1998).”

“…to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006;…”

“Finally, factors like the pursuit of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies…”

“Reviewers often fail to detect major methodological failings (Schroter et al., 2004), with eminent journals (whose higher rejection rates might suggest more stringent peer review processes) seeming to perform no better than others (Fang et al., 2012).”

“• Lack of incentives: Traditional peer review provides little in the way of incentives for reviewers, whose work is almost exclusively unpaid and whose anonymous contributions cannot be recognised and hence rewarded (Armstrong, 1997; Ware, 2008).”

“To date, studies have failed to show any great effect in either direction (McNutt et al., 1990; van Rooyen et al., 1999; van Rooyen et al., 2010).”

“…of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al., 1977).”

“A main reason for maintaining author anonymity is that it is assumed to tackle possible publication biases against authors with traditionally feminine names, from less prestigious institutions or non-English speaking regions (Budden et al., 2008; Ross et al., 2006).”

“First is a preference for complexity over simplicity in methodology (even if inappropriate, c.f. Travis & Collins, 1991) and language (Armstrong, 1997).”

“…of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al., 1977).”

“Studies show reviewers’ views tend to show very weak levels of agreement (Kravitz et al., 2010; Mahoney, 1977), at levels only slightly better than chance (Herron, 2012; Smith, 2006).”

“Moreover, factors such as close disciplinary communities and internet search capabilities, mean that author anonymity is only partially effective, with reviewers shown to be able to identify authors in between 26 and 46 percent of cases (Fisher et al., 1994; Godlee et al., 1998).”

“Van Rooyen and her colleagues found that open reports correlate with higher refusal rates amongst potential reviewers, as well as an increase in time taken to write review but no concomitant effect on review quality (van Rooyen et al., 2010).”

“…disclosure of identities aims usually at increasing accountability and minimizing bias, c.f. “referees should be more highly motivated to do a competent and fair review if they may have to defend their views to the authors and if they will be identified with the published papers” (Armstrong, 1982).”

“• Social and publication biases: Although often idealized as impartial, objective assessors, in reality studies suggest that peer reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), inst…”

“Open identity peer review, also known as signed peer review (Ford, 2013; Nobarany & Booth, 2015) and “unblinded review” (Monsen & Horn, 2007), is review where authors and reviewers are aware of each other’s identities.”

“Open Science encompasses a variety of practices, usually including areas like open access to publications, open research data, open source software/tools, open workflows, citizen science, open educational resources, and alternative methods for research evaluation including open peer review (Pontika et al., 2015).”

“Reviewers can come from the wider research community, as well as those traditionally under-represented in scientific assessment, including representatives from industry or members of special-interest groups, for example patients in the case of medical journals (Ware, 2011).”

“Nature judged the trial to have been unsuccessful due to the small number of authors wishing to take part (just 5% of submitting authors), the small number of overall comments (almost half of articles received no comments) and the insubstantial nature of most of the comments that were received (Fitzpatrick, 2011).”

“Moreover, getting research out earlier increases its visibility, enables open participation in peer review (where commentary is open to all), and perhaps even, according to (Pöschl, 2012), increases the quality of initial manuscript submissions.”

“…objective assessors, in reality studies suggest that peer reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006;…”

“It could even be argued that this inability to define terms is to blame for the fact that, as Nicholas Kriegskorte has pointed out, “we have yet to develop a coherent shared vision for “open evaluation” (OE), and an OE movement comparable to the OA movement” (Kriegeskorte, 2012).”

“, 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al.”

“Many articles in top journals remain poorly cited, while many of the most highly-cited articles in their fields are published in lower-tier journals (Jubb, 2016).”

“For example, Herron (2012) produced a mathematical model of the peer review process which showed that “the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size”, although only if the numbers of…”

“…suggest that peer reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982),…”

“…al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006; Tregenza, 2002) and discipline (Travis & Collins, 1991).”

“These results suggest that in this era of increasing competition for publication and grants, cooperation is vital for accurate evaluation of scientific research” (Leek et al., 2011).”

“Summing up her argument elsewhere, she says: “my previous definition … broadly understands OPR as any scholarly review mechanism providing disclosure of author and referee identities to one another” (Ford, 2015).”

“…reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin,…”

“Open identity peer review, also known as signed peer review (Ford, 2013; Nobarany & Booth, 2015) and “unblinded review” (Monsen & Horn, 2007), is review where authors and reviewers are aware of each other’s identities.”

“Some journals enable pre-publication interaction between reviewers as standard (Hames, 2014).”

“At eLife, reviewers and editor engage in an “online consultation session” where they come to a mutual decision before the editor compiles a single peer review summary letter for the author to give them a single, non-contradictory roadmap for revisions (Schekman et al., 2013).”

“It aims to resolve possible conflicts associated with editorial selection of reviewers (e.g. biases, closednetworks, elitism) and possibly improve the reliability of peer review by increasing the number of reviewers (Bornmann et al., 2012).”

“The journal stopped publishing in 2002 (Sandewall, 2012).”

“Alternatively, services which overlay peer-review functionalities on repositories can produce functional publication platforms at reduced cost (Boldt, 2011; Perakakis et al., 2010).”

“Next, “confirmatory bias” is theorized to lead to conservatism, biasing reviewers against innovative methods or results contrary to dominant theoretical perspectives (Chubin & Hackett, 1990; Garcia et al., 2016; Mahoney, 1977).”

“, 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006; Tregenza, 2002) and discipline (Travis & Collins, 1991).”

“It may be that comments are open to anybody (anonymous or registered), or some credentials might first be required (e.g., Science Open requires an ORCID profile with at least five published articles).”

“…(Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006; Tregenza, 2002) and discipline (Travis &…”

“The mathematics journal Discrete Analysis, for example, is an overlay journal whose primary content is hosted on arXiv (Day, 2015).”

“Ross-Hellauer (2017) apresenta uma sistematização dessas modalidades que podem ser adotadas de forma isolada ou combinada: Identidades abertas - autores e revisores conhecem a identidade uns dos outros; Relatórios abertos - os pareceres são publicados junto ao artigo avaliado; Participação aberta -…”

“…common complaints about academia, including those focused on lack of reproducibility (Open Science Collaboration, 2015), problems with peer review (Ross-Hellauer, 2017; Smith, 2006; Tennant et al., 2017), and the lack of access to research, could conceivably be reduced by building mechanisms that…”

“Most notably, open peer review is being more widely introduced as an alternative paradigm of interaction between authors and reviewers (Ford, 2013; Ross-Hellauer, 2017).”