What is open peer review? A systematic review

“” (Suber, 2012) Page 4 of 32 Within just these two examples, there are already a multitude of factors at play, including the removal of anonymity, the publishing of review reports, interaction between participants, crowdsourcing of reviews, and making manuscripts public pre-review, amongst others.”

“Reviewers often fail to detect major methodological failings (Schroter et al., 2004), with eminent journals (whose higher rejection rates might suggest more stringent peer review processes) seeming to perform no better than others (Fang et al., 2012).”

“Nickerson et al. (2013) advise that the development of a taxonomy should begin by identifying the appropriate meta-characteristic – in this case distinct individual innovations to the traditional peer review system.”

“Published reports can enable readers to consider these criticisms themselves, and “have a chance to examine and appraise this process of ”creative disagreement” and form their own opinions” (Peters & Ceci, 1982).”

“Next, “confirmatory bias” is theorized to lead to conservatism, biasing reviewers against innovative methods or results contrary to dominant theoretical perspectives (Chubin & Hackett, 1990; Garcia et al., 2016; Mahoney, 1977).”

“, 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al.”

“Smith, an experienced editor, for example, reports reviewers stealing ideas and passing them off as their own, or intentional blocking or delaying publication of competitors’ ideas through harsh reviews (Smith, 2006).”

“, 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al.”

“…of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al., 1977).”

“A main reason for maintaining author anonymity is that it is assumed to tackle possible publication biases against authors with traditionally feminine names, from less prestigious institutions or non-English speaking regions (Budden et al., 2008; Ross et al., 2006).”

“Whatever the cause, recent sharp rises in the number of retracted scientific publications (Steen et al., 2013) testify that peer review sometimes fails in its role as the gatekeeper of science, allowing errors to enter the literature.”

“This system is perhaps more recent than one might expect, with its main formal elements only in general use since the mid-twentieth century in scientific publishing (Spier, 2002).”

“Moreover, factors such as close disciplinary communities and internet search capabilities, mean that author anonymity is only partially effective, with reviewers shown to be able to identify authors in between 26 and 46 percent of cases (Fisher et al., 1994; Godlee et al., 1998).”

“…to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006;…”

“Reviewers often fail to detect major methodological failings (Schroter et al., 2004), with eminent journals (whose higher rejection rates might suggest more stringent peer review processes) seeming to perform no better than others (Fang et al., 2012).”

“To date, studies have failed to show any great effect in either direction (McNutt et al., 1990; van Rooyen et al., 1999; van Rooyen et al., 2010).”

“…of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al., 1977).”

“A main reason for maintaining author anonymity is that it is assumed to tackle possible publication biases against authors with traditionally feminine names, from less prestigious institutions or non-English speaking regions (Budden et al., 2008; Ross et al., 2006).”

“First is a preference for complexity over simplicity in methodology (even if inappropriate, c.f. Travis & Collins, 1991) and language (Armstrong, 1997).”

“• Lack of incentives: Traditional peer review provides little in the way of incentives for reviewers, whose work is almost exclusively unpaid and whose anonymous contributions cannot be recognised and hence rewarded (Armstrong, 1997; Ware, 2008).”

“…of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al., 1977).”

“Studies show reviewers’ views tend to show very weak levels of agreement (Kravitz et al., 2010; Mahoney, 1977), at levels only slightly better than chance (Herron, 2012; Smith, 2006).”

“Moreover, factors such as close disciplinary communities and internet search capabilities, mean that author anonymity is only partially effective, with reviewers shown to be able to identify authors in between 26 and 46 percent of cases (Fisher et al., 1994; Godlee et al., 1998).”

“Van Rooyen and her colleagues found that open reports correlate with higher refusal rates amongst potential reviewers, as well as an increase in time taken to write review but no concomitant effect on review quality (van Rooyen et al., 2010).”

“…disclosure of identities aims usually at increasing accountability and minimizing bias, c.f. “referees should be more highly motivated to do a competent and fair review if they may have to defend their views to the authors and if they will be identified with the published papers” (Armstrong, 1982).”

“• Social and publication biases: Although often idealized as impartial, objective assessors, in reality studies suggest that peer reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), inst…”

“Open identity peer review, also known as signed peer review (Ford, 2013; Nobarany & Booth, 2015) and “unblinded review” (Monsen & Horn, 2007), is review where authors and reviewers are aware of each other’s identities.”

“Open Science encompasses a variety of practices, usually including areas like open access to publications, open research data, open source software/tools, open workflows, citizen science, open educational resources, and alternative methods for research evaluation including open peer review (Pontika et al., 2015).”

“Nature judged the trial to have been unsuccessful due to the small number of authors wishing to take part (just 5% of submitting authors), the small number of overall comments (almost half of articles received no comments) and the insubstantial nature of most of the comments that were received (Fitzpatrick, 2011).”

“Moreover, getting research out earlier increases its visibility, enables open participation in peer review (where commentary is open to all), and perhaps even, according to (Pöschl, 2012), increases the quality of initial manuscript submissions.”

“…objective assessors, in reality studies suggest that peer reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006;…”

“Reviewers can come from the wider research community, as well as those traditionally under-represented in scientific assessment, including representatives from industry or members of special-interest groups, for example patients in the case of medical journals (Ware, 2011).”

“It could even be argued that this inability to define terms is to blame for the fact that, as Nicholas Kriegskorte has pointed out, “we have yet to develop a coherent shared vision for “open evaluation” (OE), and an OE movement comparable to the OA movement” (Kriegeskorte, 2012).”

“Finally, factors like the pursuit of “impact” and “excellence” (Moore et al., 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies…”

“, 2017) mean that editors and reviewers seem primed to prefer positive results over negative or neutral ones (Bardy, 1998; Dickersin et al., 1992; Fanelli, 2010; Ioannidis, 1998), and to disfavour replication studies (Campanario, 1998; Kerr et al.”

“Many articles in top journals remain poorly cited, while many of the most highly-cited articles in their fields are published in lower-tier journals (Jubb, 2016).”

“For example, Herron (2012) produced a mathematical model of the peer review process which showed that “the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size”, although only if the numbers of…”

“…suggest that peer reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982),…”

“…al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006; Tregenza, 2002) and discipline (Travis & Collins, 1991).”

“These results suggest that in this era of increasing competition for publication and grants, cooperation is vital for accurate evaluation of scientific research” (Leek et al., 2011).”

“Summing up her argument elsewhere, she says: “my previous definition … broadly understands OPR as any scholarly review mechanism providing disclosure of author and referee identities to one another” (Ford, 2015).”

“…reviewers may be subject to social biases on the grounds of gender (Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin,…”

“Some journals enable pre-publication interaction between reviewers as standard (Hames, 2014).”

“At eLife, reviewers and editor engage in an “online consultation session” where they come to a mutual decision before the editor compiles a single peer review summary letter for the author to give them a single, non-contradictory roadmap for revisions (Schekman et al., 2013).”

“It aims to resolve possible conflicts associated with editorial selection of reviewers (e.g. biases, closednetworks, elitism) and possibly improve the reliability of peer review by increasing the number of reviewers (Bornmann et al., 2012).”

“Open identity peer review, also known as signed peer review (Ford, 2013; Nobarany & Booth, 2015) and “unblinded review” (Monsen & Horn, 2007), is review where authors and reviewers are aware of each other’s identities.”

“The journal stopped publishing in 2002 (Sandewall, 2012).”

“Alternatively, services which overlay peer-review functionalities on repositories can produce functional publication platforms at reduced cost (Boldt, 2011; Perakakis et al., 2010).”

“Next, “confirmatory bias” is theorized to lead to conservatism, biasing reviewers against innovative methods or results contrary to dominant theoretical perspectives (Chubin & Hackett, 1990; Garcia et al., 2016; Mahoney, 1977).”

“, 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006; Tregenza, 2002) and discipline (Travis & Collins, 1991).”

“It may be that comments are open to anybody (anonymous or registered), or some credentials might first be required (e.g., Science Open requires an ORCID profile with at least five published articles).”

“…(Budden et al., 2008; Lloyd, 1990; Tregenza, 2002), nationality (Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation (Dall’Aglio, 2006; Gillespie et al., 1985; Peters & Ceci, 1982), language (Cronin, 2009; Ross et al., 2006; Tregenza, 2002) and discipline (Travis &…”

“The mathematics journal Discrete Analysis, for example, is an overlay journal whose primary content is hosted on arXiv (Day, 2015).”

“Ross-Hellauer (2017) apresenta uma sistematização dessas modalidades que podem ser adotadas de forma isolada ou combinada: Identidades abertas - autores e revisores conhecem a identidade uns dos outros; Relatórios abertos - os pareceres são publicados junto ao artigo avaliado; Participação aberta -…”

“…common complaints about academia, including those focused on lack of reproducibility (Open Science Collaboration, 2015), problems with peer review (Ross-Hellauer, 2017; Smith, 2006; Tennant et al., 2017), and the lack of access to research, could conceivably be reduced by building mechanisms that…”

“Most notably, open peer review is being more widely introduced as an alternative paradigm of interaction between authors and reviewers (Ford, 2013; Ross-Hellauer, 2017).”

“This study did not include journals that implemented only one of the following OPR elements in Ross-Hellauer (2017): open participation, open interaction, open pre-review manuscripts, open final-version commenting and open platforms/decoupled review.”

“Following [16], we can summarise these proposed benefits and drawbacks thus: • Unreliability and inconsistency: Open identities and open reports are theorized to lead to better reviews, as the thought of having their name publicly connected to a work or seeing their review published encourages…”

“Greater transparency in processes of evaluation might eliminate bias in selection procedures [50,51].”

“This is sometimes referred to as “ open identities ” peer review, or “ no-blinding ” [30, 31].”

“Published and OA review reports are one of a few simple solutions to allow any reader or potential author to directly assess both quality and efficiency of the review system of any given journal, and the value for money of the requested APCs; thus evaluating whether a journal has ‘deceptive’ or predatory practices [42…”

“Equally long as the litany of debated issues is the list of proposed solutions, including alternatives to traditional peer review such as preprint repositories [11], double-blind peer review [12], and open peer review [13, 14, 15].”

“One of these is open peer review, for which there is, however, yet no commonly accepted and clear definition nor an agreed schema as elaborated in a secondary study by Ross-Hellauer (2017).”

“A further strand of innovation is described as ‘open peer review’, which encompasses different ways in which peer review can be opened (Ross-Hellauer, 2017).”

“Similar to the anonymity of the author, some discussions regarding peer review procedures have centred around the anonymity of the reviewer (Amsen 2014; Ross-Hellauer 2017).”

“Open peer review, in which reviewer identities and review reports are published, is a clear example (Ross-Hellauer, 2017).”

“Most notably, these models include diversity in the level of anonymity of both authors and reviewers (Pontille and Torny 2014), leading to systems ranging from doubleor even triple-blind review to radically open peer-review models (Ross-Hellauer 2017).”

“One of these is open peer review, for which there is, however, yet no commonly accepted and clear definition nor an agreed schema as elaborated in a secondary study by Ross-Hellauer (2017).”

“Open Peer-Review (OPR) mainly refers to ‘open identities’ of authors and reviewers (Ford 2013), but can also refer to ‘open reports’ (making peer-review reports publicly available), ‘open participation’ (crowdsourced peer-review) and other open review practices (Ross-Hellauer 2017).”

“For example, they could implement open peer review (pre- and post-publication), which tends to encourage more respectful and constructive feedback as well as clearer communication between reviewers, editors, and authors (Ross-Hellauer, 2017).”

“It might be argued that open and/or transparent peer review provides an alternative mechanism for reducing reviewer bias (Ross-Hellauer, 2017).”

“This variant addresses the problems many authors face when receiving contradictory reviews of their manuscripts and has already been implemented by the journal eLife (Schekman et al., 2013).”

“2017), the common characteristics of OPR are as follows (Ross-Hellauer 2017): open identities (both reviewers and authors are known to each other), open peer reports, open participation, open interaction, open prereview manuscripts, open final-version commenting and open platforms.”

“Ross-Hellauer [82] discusses a variety of problems with peer review, most of which affect the realms of quality and credibility fundamentally.”

“The space of possibilities for such reforms is large, and includes outsourcing peer-review to highly-trained experts [70], harnessing the wisdom of crowds via various forms of open peer-review [71], allowing scientists to evaluate each other’s peer-review quality [72], and supplementing peer review with "red teams" of…”

“, [6, 7]), we will use “open review” to refer to all reviews that are publicly available, whether anonymous or signed.”

“Lastly, to increase the transparency of the review process itself, journals and publishers are increasingly trialing “open peer review models,” including making reviewer and author identities known, publishing review reports alongside articles, and crowdsourcing participation in the peer review process (Ross-Hellauer,…”

“Of note, Ross-Hellauer [69], who conducted an assessment of definitions of OPR in the literature, found as many as 22 separate definitions, although ultimately, “optimal” OPR involved the following seven characteristics: (1) open identities in which “authors and reviewers are aware of each other’s…”

“8 Many definitions and understandings of the notion of open peer review have been offered (Ross-Hellauer 2017).”

“Indeed, as Ross-Hellauer (2017) notes, as reviewer comments often add context to the scholarly article, point to areas for future work, and even expose areas of tension in a theory or argument, they are considered potentially useful and re-usable scholarly information.”

“…accessible, interoperable and re-usable) sharing of data and code [5–8], transparency of processes and methods [9–12], transparent models of peer review [13,14] and engaging publics and others beyond academia (e.g. civil society, policymakers, industry) in participatory research processes [15–17].”

“Current reform initiatives concern open review (in which the reviews are publicly accessible; Ross-Hellauer, 2017), double-blind review (in which the reviewers do not know the identity of the authors; Snodgrass, 2006), or noncommercial peer review that is independent from journals.”

“The open peer review is based on the open participation of the wider scientific community (Ross-Hellauer, 2017).”

“In some cases, open peer-review is used to denote the practice of publishing peer-review reports alongside a publication (Ross-Hellauer 2017).”

“It is important to note that the adoption of the practice of publishing the peer-review reports is a positive development (Ross-Hellauer, 2017), leading to more openness and transparency, without which the detection of peer-review plagiarism reported here would not be possible.”

“open peer review (Ross-Hellauer, 2017), open grant writing and open evaluation.”

“In addition to biases, many other supposed, corroborated, or obvious phenomena can be found in the literature, such as overburdening (Kovanis, Porcher et al., 2016) or lack of transparency (Horbach, Hepkema, & Halffman, 2020; Ross-Hellauer, 2017).”

“Although the term ‘open peer review’ has no standardised definition, scholars use the term to indicate that the identities of the authors and reviewers are disclosed and that reviewer reports are openly available [13].”

“A number of journals have shifted to processes that unmask reviewers’ identity as either an optional or required part of peer review [19].”

“There is already a rich literature about this model, 67 and it has been tested experimentally by different initiatives (e.g. F1000Research, ScienceOpen, The Winnower).”

“The lack of accountability for reviewers (and the possibility of biased reviews) in the current peer-review system remains a major problem in academia [43–47] and taxonomy is no exception.”

“According to Ross-Hellauer (2017), OPR refers to the openness in one or several of the elements in the peer review process, including authors’ or reviewers’ identities, review content, participation of other actors, preview manuscripts, final-version commenting, the interaction between authors and…”

“…to accept requests to review for journals that have adopted open review practices, such as unblinded review, publication of reviews alongside the final article, or direct interaction between authors and reviewers during the review process (see Ross-Hellauer, 2017; Walker & Rocha da Silva, 2015).”

“…identities of reviewers and authors in the review of a scientific article, but it can also mean open sharing of reviewer comments and even allowing the public to participate in the review process (Ross-Hellauer, 2017; Ross-Hellauer & Görögh, 2019; Teixeira da Silva, 2018; Thelwall et al., 2021).”

“Open-identities peer review is not currently widespread [47], but is commonly promoted as a means to increase transparency and accountability in the peer-review process.”

“Besides a blind peer‐review policy (in which reviewers do not know the authors and their affiliation), journals are also experimenting with more open approaches to peer reviews, such as making the identity of peer reviewers and even the content of reviews openly available (Ross‐Hellauer, 2017).”

“Ross-Hellauer (2017) considered open peer review as an inclusive academic keyword.”

“…discrepancies and differences about what open peer review means and the different Quantitative Science Studies 939 Wikinformetrics models proposed (Ross-Hellauer, 2017), the three basic principles (open identities, reports, and participation) are clearly recognizable in Wikipedia (Table S2 in the…”

“Thus, the development of the different interpretations of what has been called ‘openness’ with varying levels of ‘transparency’ (Ross-Hellauer, 2017).”

“…today, approaches to author identification can run the gamut, from overtly emphasizing anonymity of authors (as in the “ Journal of Controversial Ideas ”) [70, 77],valuing the traditional double blind review process [64],to models of more radical transparency (such as Open Peer Review 12 ) [71].”

“Preprint servers facilitate more public discussion and criticism of works, and allow findings to become immediately available to other researchers (Ross-Hellauer 2017, Elmore 2018).”

“Supervisors can reinforce these principles by training graduate students and postdocs in practices such as contributorship (Holcombe, 2019) and encouraging publications in RRs or diamond/green OA journals with transparent peer review (Ross-Hellauer, 2017).”