“Let the community decide”? The vision and reality of soundness-only peer review in open-access mega-journals

Abstract

Purpose

The purpose of this paper is to better understand the theory and practice of peer review in open-access mega-journals (OAMJs). OAMJs typically operate a “soundness-only” review policy aiming to evaluate only the rigour of an article, not the novelty or significance of the research or its relevance to a particular community, with these elements being left for “the community to decide” post-publication.

Design/methodology/approach

The paper reports the results of interviews with 31 senior publishers and editors representing 16 different organisations, including 10 that publish an OAMJ. Thematic analysis was carried out on the data and an analytical model developed to explicate their significance.

Findings

Findings suggest that in reality criteria beyond technical or scientific soundness can and do influence editorial decisions. Deviations from the original OAMJ model are both publisher supported (in the form of requirements for an article to be “worthy” of publication) and practice driven (in the form of some reviewers and editors applying traditional peer review criteria to OAMJ submissions). Also publishers believe post-publication evaluation of novelty, significance and relevance remains problematic.

Originality/value

The study is based on unprecedented access to senior publishers and editors, allowing insight into their strategic and operational priorities. The paper is the first to report in-depth qualitative data relating specifically to soundness-only peer review for OAMJs, shedding new light on the OAMJ phenomenon and helping inform discussion on its future role in scholarly communication. The paper proposes a new model for understanding the OAMJ approach to quality assurance, and how it is different from traditional peer review.

Citation

Spezi, V., Wakeling, S., Pinfield, S., Fry, J., Creaser, C. and Willett, P. (2018), "“Let the community decide”? The vision and reality of soundness-only peer review in open-access mega-journals", Journal of Documentation, Vol. 74 No. 1, pp. 137-161. https://doi.org/10.1108/JD-06-2017-0092

1. Introduction

Peer review is a fundamental element of scholarly publishing. The process, which involves research papers submitted to journals being assessed by experts in the field in order to inform the decision to accept or reject papers and improve accepted ones, is seen as an essential form of quality assurance (Procter et al., 2010; Rowland, 2002). Organised by publishers and editors, and performed voluntarily by fellow researchers, it is an important instrument of self-regulation for the scholarly community in conducting academic research (Bohannon, 2013). In conventional journals, the peer review process takes place prior to publication and is based on a number of commonly accepted criteria applied in the selection of papers, including the rigour or soundness of the study, its novelty, its significance and its relevance to the journal’s readership. Journals vary in their selectiveness, which for the most part is a result of the thresholds applied to these criteria.

Since the web has been adopted as the primary means of disseminating scholarly content, new forms of journal publishing, often involving innovative approaches to peer review, have emerged. Notable amongst these are so-called “mega-journals” (OAMJs), which include PLOS ONE (the first journal of its kind), launched by the fully open-access (OA) publisher Public Library of Science in 2006, and Scientific Reports, launched by Nature in 2011. As journals published entirely online, these publications were freed from any restrictions imposed by page space, a constraint partly responsible for selectivity in conventional journals that originated in a print-based environment. Open-access mega-journals (OAMJs) have four main characteristics: large publishing volume, broad subject scope, an OA publishing model and a novel approach to peer review (Björk, 2015; Spezi et al., 2017). The last of these requires editors and reviewers to evaluate only a paper’s scientific or technical soundness, and not to take into account its novelty, significance or its relevance to a notional readership. Evaluation of these remaining factors is pushed downstream to be judged by the academic community after the article’s publication, with indicators such as article-level metrics serving as supporting tools for such post-publication assessment.

This decoupling of the judgement of novelty, significance and relevance from the pre-publication peer review stage has been a controversial and much debated development in scholarly publishing – a controversy fuelled by the launch of a large number of OAMJs in the wake of PLOS ONE’s success (Wakeling et al., 2016). Advocates claim that peer review for soundness only has the potential to alter radically the traditional filtering and gatekeeping role of journal editors, editorial boards and reviewers whose judgements on novelty and significance have traditionally been crucial in determining what is published. Judgements on novelty, significance and relevance are often criticised as being subjective, and so peer review focussed on soundness only is argued to be more objective. One apparent consequence of this is that editors, editorial boards and reviewers are no longer the sole arbiters of the boundaries of existing disciplinary paradigms. As such, OAMJs can be conceived of having democratising potential (Lăzăroiu, 2017). Advocates also argue that the OAMJ approach to peer review allows for greater efficiency in the publication process by eliminating the submission-rejection spiral undergone by many articles before they find a home in a selective journal (Cope and Phillips, 2014; Pattinson, 2014; Ware, 2011), and in allowing the publication of studies that might struggle to meet the significance requirements of traditional journals – the reporting of negative results and replication studies being often cited examples (Binfield, 2013). The approach can, therefore, be seen to support the broader goals of the Open Science movement. Summarising the views of many of its supporters, Gary Ward, Chair of the PLOS Board of Directors, was quoted in a 2011 interview as saying, “I love the concept of eliminating this huge waste of time by simply removing subjective evaluations of importance from the review process […] If the paper is well-written and the conclusions don’t overreach, then let the community decide the impact” (Adams, 2011). That mantra of, “let the community decide”, has become a central feature of the OAMJ rhetoric. On the other hand, soundness-only peer review has been criticised for lowering acceptance thresholds in the interests of journal scalability and a highly lucrative business model (Buriak, 2015).

This paper arises from one phase of a two-year empirical investigation of the OAMJ phenomenon, and reports a series of interviews with executives and senior editors and editorial board members at both OAMJ publishers and those who publish only conventional journals. While these interviews covered a range of topics, this paper focusses on the results relating to peer review in the context of OAMJs, perhaps the most prominent topic in the interviews and one that permeated many of the discussions. Discussion of strategic and general operational issues associated with OAMJs as reported by our participants can be found in two complementary papers (Wakeling, Spezi, Creaser, Fry, Pinfield and Willett, 2017; Wakeling, Spezi, Fry, Creaser, Pinfield and Willett, 2017).

Based on our findings, we present here a model designed to enable a better understanding of the relationship between the peer review processes of traditional and OAMJs, and more generally of how both systems address issues of quality assurance. The model considers soundness-only peer review in the context of the other elements of traditional peer review (novelty, significance and relevance). Analysing the evidence used to construct the model, the paper explores and highlights a series of tensions: tensions between what soundness-only peer review is often claimed to be, and what it is in practice; and also tensions between the vision and principles driving the development of OAMJ peer review, and the operational reality of managing a soundness-only peer review process. We conclude that those tensions are symptomatic of the difficulties associated with attempting to detach judgements of soundness from the traditional norms of peer review.

This study is based on unprecedented access to senior publishers and editors, enabling insight into their strategic and operational priorities. The paper is the first to report in-depth qualitative data relating specifically to soundness-only peer review for OAMJs, shedding new light on the OAMJ phenomenon and on the nature of soundness-only peer review and helping inform discussion on their future roles in scholarly communication. The paper proposes a novel model for understanding the OAMJ approach to quality assurance, and how it is different from traditional peer review, both in principle and reality.

2. Research context

2.1 Attitudes to peer review

A number of studies have investigated researcher perceptions of peer review, with sometimes contradictory results. Researchers have been found to be generally satisfied with the current system, and appear to highly value the practice (Mulligan et al., 2013; Nicholas et al., 2015; Thelwall et al., 2013; Ware, 2008a, b). In survey results reported by Ware (2008b), an overwhelming majority of respondents believed that peer review was both necessary (93 per cent) and helped scholarly communication (85 per cent). A more recent survey of researchers supported this, finding that peer review is overwhelmingly perceived to be the most important characteristic of a trust-worthy publication (Nicholas et al., 2015). Interestingly the same study also found that negative views of OA journals were often predicated on a belief that such journals do not employ a “proper” peer review process. Studies have commonly reported that researchers believe that peer review helps improve the quality of the papers published, with possible improvements including enhancement of a paper’s discussion, improvements in language and presentation and correction of scientific errors (Bornmann, 2011; Sense About Science, 2009; Taylor & Francis, 2015; Ware, 2008b).

However, many observers believe the peer review system as a whole to be under increasing pressure (Procter et al., 2010). Some go so far as to say it is “near breaking point” (Franklin, 2010; see also Fox and Petchey, 2010). Such tension is often attributed to two interrelated factors: the increasing volume of publications and the globalisation of science. The volume of research produced is growing at a steady but sustained pace, in the region of 3 per cent each year (Ware and Mabe, 2009). It is fuelled in no small part by a significant rise in the number of active researchers in developing countries, such as China and India (Ware, 2011). Evidence suggests, however, that researchers from these countries perform proportionally far fewer reviews than academics in the Global North (Warne, 2016). The result is a growing burden being placed on both reviewers, who are faced with an increasing volume of peer review requests, and publishers, who experience difficulties managing the process and recruiting reviewers.

There is also evidence that researchers are increasingly aware of the shortcomings of peer review (Ware and Mabe, 2015). Only 32 per cent of researchers think that peer review as currently practised is the best system that can be achieved (Sense About Science, 2009), reflecting authors’ awareness of the various limitations and problems identified in the literature, such as bias in the review process (based on a range of factors including gender, nationality and prestige) (Lee et al., 2013), inconsistency among reviewers (Blackburn and Hakel, 2006), the slow and expensive nature of the process (Research Information Network, 2015; Smith, 2006) and the questionable predictive validity of judgements of significance (Bornmann, 2011; Velterop, 2015). Researchers are aware that the importance placed by publishers on the journal impact factor (JIF), which is based on citation counts of articles published in the journal, may provide motivation for editors to publish only those articles that are likely to accrue high numbers of citations (Pulverer, 2013).

In recent years, some innovations have emerged (Tennant et al., 2017). These typically take advantage of technical advances and include open peer review (where reviewer reports and identities are published alongside the article), portable peer review (where the reviewer reports of rejected papers are made available to the editors of other journals), and even machine-based peer review (in which papers are automatically analysed using algorithms to check statistical data, plagiarism, paper structure and grammar). Post-publication peer review, where articles are published after an initial in-house check with more formal peer reviews added later (as, e.g., practised by F1000) represents another alternative model.

Whilst many of these innovations associated with peer review have experimental or early operational implementations, soundness-only peer review is to date perhaps the most widely operationalised. However, it is still little examined in the research literature. Björk and Catani’s (2016) quantitative study on the comparative outcomes of soundness-only peer review is an exception. They compared the citation distributions of OAMJ’s using soundness-only peer review compared with a group of traditional selective titles, finding surprisingly little difference between the two. From this they drew the tentative conclusion that since soundness-only peer review is likely to be quicker and less burdensome than conventional peer review, it may be preferable to the traditional system. Their study was, however, based on a sample of information science “elite” comparator titles, whereas selective journals in the biosciences (the disciplinary area of many OAMJ articles) may have somewhat different citation patterns. The OAMJ sample itself included journals with a range of coverage, including at least two covering the humanities and social sciences, disciplines with different citation conventions. Nevertheless, their study has illustrated the need for further research in this area. To date, however, there have been no published studies on how the soundness-only peer review process actually works in practice and what that means in terms of the vision put forward by proponents. This present study begins to address that gap.

The current study was, nevertheless, carried out in a context in which the conventional model of peer review has proved remarkably resilient. Despite some disquiet about its failings and the development of alternatives to conventional peer review, there has still not been any major challenge in practice to the primacy of the traditional model. Björk and Hedlund (2015) observe that not even the introduction of the web has had a profound impact on the way peer review is conducted for most journals. Even with its shortcomings, conventional peer review is commonly regarded as the best system available and any change to it is often regarded with caution and suspicion (Pinfield, 2015).

2.2 Elements of peer review

Our review of the literature relating to peer review, combined with an appraisal of “instructions for reviewers” available on journal websites, suggests that the traditional peer review process is constituted of four main elements (see Figure 1):

1. Novelty or originality: the way in which the paper makes an intellectual advance contributing in an innovative way to knowledge; this may include innovative methods, new empirical results, novel interpretations or original theoretical insights.

2. Significance or importance: the way the paper adds to the body of knowledge, making an impact by enhancing understanding or practice.

3. Scope or relevance: the anticipated interest of the article to its readership as defined by the journal’s stated subject coverage.

4. Soundness or rigour: sometimes referred to as “technical” or “scientific” soundness, this relates to issues such as methodological precision, coherence and integrity; it includes the quality of the argumentation, relating to the logic of research and the way data are interpreted.

Although conceptually distinct, these criteria may in practice be closely related. An output’s significance may, for example, derive in large part from its novelty, although it is possible to conceive of a novel paper which is not deemed significant or vice versa: a novel piece of research may be trivial in terms of implications and therefore not significant; a replication study or review article may be lacking in originality but nevertheless have significance. Other assessment criteria such as the UK’s Research Excellence Framework (REF) distinguish between novelty and significance, despite their often close relationship. Interestingly, the REF criteria also include rigour alongside originality and significance. Like the other criteria, it applies across different disciplines, and is defined as, “intellectual coherence, methodological precision and analytical power; accuracy and depth of scholarship; awareness of and appropriate engagement with other relevant work” (HEFCE, 2012).

These four elements of peer review are typically applied once a paper has undergone pre-peer review quality checks. These are likely to include plagiarism detection, compliance with file formatting requirements and inclusion of any required supplementary files, including data. The paper will also be initially checked for scope. In addition the content of the paper is likely to be assessed to ensure it meets basic standards of quality of expression and presentation. The latter requirement is in many respects a cross-cutting criterion that relates to all of the above main criteria. Peer reviewers may also comment on clarity of expression and presentation as part of the review process (as indicated on Figure 1 by the partial darker shading in this box). If a paper does not meet a basic threshold from this initial screening, it may be “desk rejected” by the editor of the journal before the main peer review criteria are applied. The extent to which the other peer review criteria are applied by the editor or editorial staff before the paper is sent to reviewers will vary by journal, with some highly selective journals undergoing more in-house scrutiny before circulation to external peer reviewers.

The criteria are all applied before publication in order to determine acceptance of the paper for publication. There are no post-publication components of the formal quality assurance process in the conventional system (they are therefore unshaded in Figure 1). Although judgements of article quality associated with citation count and JIF are inevitably made within the academic community, this is not directly associated with the publication process itself.

OAMJs have generally described themselves as journals for which peer review solely focusses on technical and scientific soundness. A relevant section of the instructions for reviewers of the largest OAMJ reads as follows:

To be published in Scientific Reports, a paper must be scientifically valid and technically sound in methodology and analysis. Manuscripts are not assessed based on their perceived importance, significance or impact; the research community makes such judgements after publication

(Scientific Reports, 2017).

Figure 2 applies these stated aims to our model. Here, and in the OAMJ discourse more generally, the only criterion for pre-publication peer review is technical or scientific soundness (with clarity of expression and presentation also assumed as an initial threshold). Novelty, significance and relevance are excluded from the peer review process, and left for the community to determine and communicate via post-publication quality indicators, such as article-level metrics and comments. Such post-publication indicators are in no way unique to OAMJs but have been viewed as essential to the vision articulated around the concept of soundness-only refereeing.

In the following sections, we analyse and discuss soundness-only peer review and its implications, based on the evidence gathered during the interviews with publishers and senior editors. After an account of our methods, we report definitions of soundness-only peer review suggested by our participants and we analyse arguments they use to justify (or criticise) such an approach. We then discuss some of the operational realities of soundness-only peer review. The elements presented in Figures 1 and 2 are discussed in the context of our findings.

3. Methodology

In order to gain a deep understanding of the running of OAMJs, from both a strategic and operational perspective, as well as the role they play in the journal landscape and more generally in scholarly communication, we conducted a series of interviews with publishers, editors and editorial board members. We identified all major OAMJs currently operating, and approached directly their senior executives and editors requesting interviews. We also approached some major publishers who are not currently operating an OAMJ. Additional individuals were also suggested by the publishers and editors owing to the role they played in the running of an OAMJ. A total of 31 semi-structured interviews were conducted with interviewees from 16 organisations, as summarised in Table I, with all of the publishers we approached taking part. The interviews were conducted either face-to-face or via Skype, and took place in Spring/Summer 2016. We had what we believe was unprecedented access to the publisher community at senior levels for this kind of research.

The interviews focussed on various aspects of OAMJ production and the broader scholarly communication context. For publishers the interview questions focussed on the strategic and operational aspects of OAMJs and their role in scholarly communication (Appendix 1). A similar schedule was used for editors, but with a greater emphasis on the peer review process and a reduced emphasis on strategic aspects (Appendix 2).

The interviews were audio-recorded, with participants’ permission, fully transcribed and analysed with the aid of qualitative analysis software (NVivo). The coding was done by two researchers following a robust coding process: inter-coder reliability was calculated on a sample of transcripts to ensure coding consistency between the two researchers, with a value of 0.72. This is generally considered as adequate and to indicate a reasonable level of agreement between the two coders (Lombard et al., 2002). However, to strengthen the quality of the coding, all transcripts were subsequently “proof coded”: after all transcripts had been coded, each one was reviewed by the researcher who had not done the original coding, and any discrepancies discussed and resolved. The 31 interviews yielded a data set of 30 hours of audio recording, with 2,834 coded units of text, of which 585 specifically dealt with peer review. Their thematic analysis (Braun and Clarke, 2006) forms the basis of this paper.

4. Findings: vision and principles

4.1 Defining “soundness-only” peer review

All participants were asked to explain what they understood by the term “soundness-only” peer review. Primarily, it was associated with reviewing the robustness of the approaches taken to assembling, analysing and interpreting the data underlying the study. There was a consensus that this meant evaluating the rigour and ethics of the research, with one interviewee summarising the process as “looking at the basics” (MJ editor 1). Interviewees generally suggested that the specifics of this include assessing whether the correct methodology has been used to address the research questions, that any statistical tests have been used appropriately, and that the research satisfies the necessary ethical requirements. Several participants also stated that the approach particularly scrutinises the conclusions of the paper – “editors and reviewers would judge the work based on whether the conclusions were supported by the data” (MJ publisher 1) – which also serves to identify sweeping claims or attempts to overstate the conclusions by exaggerating the importance and reach of the findings, a practice participants suggested was common in some very competitive disciplines.

It was notable that several participants described the soundness-only approach in more general terms. One suggested that it is “looking at […] [whether] the paper belongs in the published literature somewhere, and if it does, then it can be published in a mega-journal” (MJ publisher 2). Others used potentially more loaded and less clearly defined terms: “if it looks like good science then we are keen to publish it” (MJ editor 2); “there is a minimum acceptable quality standard” (MJ publisher 3). It was notable too that initial descriptions of soundness-only review from a number of interviewees also defined the approach in terms of what it does not do, highlighting the absence of any judgement of significance or importance: “we are not worried about priority or how sexy the science is” (MJ editor 2).

No agreed definition of soundness emerged in the context of the arts and humanities. Several interviewees thought soundness simply could not be transposed to these disciplines. As one publisher put it:

[…] [soundness] doesn’t translate into all fields. I have to say the humanities […] I’d be really hard pressed to really understand […] I mean it can’t translate terribly well

(Non-MJ publisher 1).

Others were more positive, with some acknowledging the important work done by Eve (2014) in attempting to define a set of criteria for soundness in the humanities, essentially emphasising the clarity of the argumentation and the proper use of the sources. This was expanded upon by one participant, who suggested that:

There are ways of pre-filtering on a much lower level for humanities work. For example, […] in humanities you can look for work that is written in an appropriate register that cites all the appropriate literature, that evinces an argument that is self-coherent, that tends to pre-empt criticisms that have been levelled at it. Essentially things that are more to do with the apparatus of a paper than the actual content

(Non-MJ publisher 2).

Mirroring what is found in the literature on OAMJs, and particularly the informal literature (see e.g. Taylor, 2015), it was notable that interviewees used a variety of qualifiers when describing soundness-only peer review or OAMJs. The two most commonly used terms were “objective” peer review and “peer review lite”. “Light touch”, “light weight”, “non-selective”, “impact neutral” peer review, as well as “low peer review barrier”, were also used by some interviewees. The rhetoric here is important. In most cases, it seemed to reflect an internalised value system: advocates of soundness-only peer review were more likely to use the “objective” qualifier, whereas critics tended to use the “light”/“lite” qualifier. Peer review “lite” was often associated with notions of lower-quality and broad acceptance; that such non-selective peer review conveyed a negative connotation. This perhaps explains the importance publishers placed on the choice of the more positive “objective” to describe the peer review process for their OAMJs.

The rhetoric may also reflect the degree of individual engagement with the concept itself. One interviewee (Non-MJ publisher 3), who was extremely vocal in criticising OAMJs, referred to the concept of “sound science” as used by OAMJs as “garbage” since in the interviewee’s view novelty and importance are “entirely part of sound science”.

4.2 Justifying the soundness-only approach

All participants acknowledged that the OAMJ peer review approach represents a significant departure from traditional editorial processes. This is reflected in the fact that several interviewees described in detail the difficulties they faced justifying the soundness-only approach to senior decision makers within their organisation. The removal of significance and impact as evaluation criteria in particular was apparently viewed by some as high risk, and even against the founding values of research excellence.

In response, most interviewees with an OAMJ background suggested that the weight placed on assessment of impact and significance was detrimental to the scholarly communication process and counter-productive to the progress of science, relying as it does on the free flow and exchange of ideas. Thus, the relaxation of the importance/significance constraint on academic publishing was often voiced as one of the main motivations for publishers and editors to engage with a OAMJ venture. They suggested that assessment of impact and significance was the most subjective and unreliable judgement that takes place in the traditional peer review process. One OAMJ editor likened an attempt to predict the impact of an article in five or ten years’ time to “reading a crystal ball”. A recurrent argument regarding the rationale for soundness-only peer review was that it removed this subjective element. Instead it was suggested that the OAMJ approach allowed the entire community to decide what is important. A key word used by some interviewees here was “democratisation”. They argued that the community is best placed to judge which papers are important and significant – more so than editors and reviewers. Such an approach was sometimes framed as a stance against elitism and the gatekeeping attitude that editors are perceived sometimes to exhibit: what an editor finds important may not be necessarily be what an individual researcher might find important. Some interviewees articulated a more extreme characterisation of editorial practices at some journals, suggesting that editors might be motivated to reject work out of self-interest:

I have certainly seen biases in place where a certain editorial viewpoint was being put forth and that contradictory research was basically being held back […] so removing that kind of nonsense from the process is noble

(MJ publisher 4).

This desire to move towards a system encouraging the productive and frictionless exchange of ideas emerged from the interviews. One participant summarised it by saying:

[…] the more journals that can operate this sort of model, the better, because science is not all about exciting, ground-breaking discoveries, a lot of it is about collecting data, verifying that data, providing the ground, the basis for other work

(MJ publisher 5).

Interviewees insisted that research findings which are not regarded as highly significant might still be very useful, providing “incremental steps” for other researchers to take forward. Soundness-only peer review facilitates the publication of just this research. Several participants also described how the approach aligns with the requirements of funders, who are increasingly vocal in their belief that all the results of publically funded research should be published.

A further argument in favour of OAMJ peer review put forward by participants related to the broader scholarly context. Soundness-only peer review was justified by many interviewees as an approach both divorced from and a challenge to prevalent modes of research evaluation. All interviewees recognised that the journal ranking system was a central cultural element of academic publishing, with one describing journal rank and JIF as a prestige marker for authors, an attitude reinforced by the use of such ranking and metrics in academic rewards systems and hiring processes. Many interviewees felt strongly that this system, while deeply ingrained, was fundamentally detrimental to the basic goals of academic research. Several participants noted that such measures are naturally irreconcilable with an OAMJ approach to peer review, since the publication of an article by an OAMJ is explicitly not an indicator of its potential importance. OAMJs, and by extension the articles they publish, cannot therefore be ranked against each other. Some participants felt that valid, sound research ended up not being published because of the filtering applied by traditional journals to improve or maintain their JIF. More specifically, several interviewees noted that within the traditional system little credit is given for the publication of replication or null-results studies, and that those studies usually do not help build a higher Impact Factor. Thus, authors tend not to write such studies, and if they do, publishers tend not to publish them. Soundness-only peer review was presented as a means of counteracting this effect.

It is important to note that not all interviewees were persuaded by the arguments in favour of soundness-only peer review. A key factor for critics of the model was the extent to which the approach simultaneously increased the volume of published articles, while removing a key filtering mechanism for readers. In this respect it was notable that one editor of a well-known OAMJ voiced concerns about the journal’s policy to ignore impact:

[…] they [OAMJ senior management] don’t concern themselves too much with impact, and I think that’s an important issue […] I think the problem then is just the sheer volume, and the fact that they may be watering down the seriousness of some of the content

(MJ editor 3).

This view was echoed and expanded upon by another interviewee, who felt that soundness-only review was failing readers:

I don’t think it is helping […] you know knowledge emerges from synthesised information and it is not doing that […] If you are really catering to readers, then you do care about novelty and quality and importance

(Non-MJ publisher 3).

Several participants also rejected the characterisation of editors as limiting factors in the spread of new ideas. As one put it: “editors are really good people and they want to develop their journals in ways that will help continue to lead the community” (Non-MJ publisher 4). A publisher also suggested that the OAMJ argument around democratisation failed to recognise the fact that those closest to the traditional publication process are themselves members of the academic community:

[…] there is a proletarian view of the community that [mega-journals] are peddling which says that editors and experts and peer reviewers and all the technical editors and the field editors and the manuscript editors who work 30 years in a field and probably know it better than most of the practitioners, that those people aren’t part of the community

(Non-MJ publisher 3).

Finally, the rhetoric of “objective” peer review was called into question by some of the interviewees – even amongst those most supportive of the OAMJ concept. They expounded two main arguments. First, that objectivity – and its corollary, consistency – is achievable only if the discipline encompasses a “well-defined community” with shared values and academic standards. If this is not the case, then it is extremely difficult to achieve consistency and objectivity (MJ publisher 6). Second, that since soundness-only review typically includes an assessment of the validity and reasonableness of the research questions, a certain amount of selectivity, and therefore subjectivity, is inevitable (Non-MJ publisher 4).

5. Findings: operational reality

5.1 Reviews, decision making and governance

Perhaps the most significant finding to emerge from the interviews was the acknowledgement by many OAMJ publishers and editors that their peer reviewers, despite instructions to the contrary, still regularly factor significance, novelty and impact into their evaluation of articles under review. As one interviewee put it, “the culture of selecting things on the basis of impact is very ingrained in scientists, and it’s very hard sometimes to get them to drop their old habits” (MJ publisher 5). It should be noted that there was an acknowledgement from representatives of the more established OAMJ titles that such reviews were becoming less prevalent, a trend explained by increasing general awareness among researchers of the soundness-only model, and the increasing likelihood that reviewers had reviewed previously for the OAMJ: “If it’s a new reviewer who’s never reviewed for [the mega-journal] before, we’re more likely to get somebody who judges against more standard criteria” (MJ editor 4). There was also a view that even within the strict soundness-only framework, reviewers were reluctant to relinquish the detailed contextual assessment associated with traditional peer review. This was summarised by one publisher: “what [reviewers] are not very good at is just looking at the science that is on the table and evaluating it to see whether it is publishable or not” (MJ publisher 7).

In response to this, interviewees involved in OAMJ production reported the need to provide reviewers with training and guidance materials about what soundness-only peer review is, and what is expected from them. The use of review templates limiting reviewers to scientific soundness and ethical assessment was a common tool deployed to make it easier for reviewers to understand what was expected from them, although it was reported that reviewers would often use comment boxes to circumvent these restrictions.

Given the challenge of changing reviewers’ habits, and the apparent reluctance of some to conduct reviews limited to soundness only, it became clear from the interviews that editors play a crucial role. Several participants described how in practice the model relied on editors to scrutinise referee’s reports and disregard elements relating to significance or novelty. As one editor put it:

[…] when I receive the reviews, I kind of mentally filter out those criticisms of “it’s not an advancement” or “it’s not exciting” and just purely look at the comments and criticisms against the publication criteria

(MJ editor 4).

It is not surprising then that senior OAMJ executives suggested that getting the right editors in place was crucial, as in practice it is they more than reviewers who often implement the soundness-only criterion. Recruiting editors who fully understand and embrace the soundness concept and the founding principles of OAMJs was seen as key to delivering a successful and consistent OAMJ. On this point it is striking to note that a small number of the OAMJ editors we interviewed actually showed some reservation about soundness-only peer review, and in two cases professed total ignorance that the journal they edited implemented such a policy!

The distributed editorial model generally adopted by OAMJs, with editors responsible for the review and decision processes for individual papers without any broader strategic role in the journal, was perceived to exacerbate governance challenges. In particular the structure of the OAMJ was reported to create risks around consistency of decision making: the flatter the structure of the OAMJ, the more control is devolved to the hundreds or even thousands of editors. Often supported by algorithmic matching of editors with new submissions, such a model was reported as essential to a journal’s ability to scale, but problematic from a quality assurance perspective. One interviewee – who is active in the scholarly communication debate and highly critical of OAMJs – was particularly vocal about the pitfalls of the OAMJ editorial structure. This participant suggested the system represented:

[…] a federated approach to editorial quality not a guided accountable approach […] there is nobody at the head who is going to get fired if something goes wrong

(Non-MJ publisher 3).

This argument was, however, countered by a OAMJ publisher, who reported that, “there is sacking, quite regular sacking of editors” (MJ publisher 1).

Most representatives of OAMJs we interviewed described quality control mechanisms in place to ensure editorial decisions were consistent, and in-line with the OAMJs editorial policies. It was notable that the nature of these mechanisms was closely linked to the size of the journal. Several interviewees described processes in place during the formative stages of their journal that were broadly similar to those found within traditional journals, with an editor-in-chief or section editor personally reviewing all accepted submissions prior to publication. Participants explained that such practices were, of course, unsustainable for journals publishing thousands of articles per year, and instead large OAMJs were found to rely on post hoc measures. Examples of auditing processes included the review of correspondence between editors and authors, and the calculation and monitoring of acceptance rates. One interviewee described a programme whereby editors with unusually high or low acceptance rates were informed of their figures, along with those of their peers, and subjected to increased levels of oversight. It is perhaps not surprising that some participants reported that such measures could lead to tensions between in-house staff and academic editors.

Some journals were said to have adopted a more innovative approach centred on the idea of consultation, often via discussion fora, in particular for editors to seek advice from peers and share best practice. It should be noted that not all OAMJs appear to offer such services, and some OAMJ editors felt that the training they received was primarily focussed on managing the software and technical elements of the review process rather than emphasising the selection criteria. Overall, despite OAMJ representatives’ faith in their operational processes to achieve consistency, there was some recognition that this was a particularly difficult task, and it was reported by several participants that acceptance rates could vary significantly between editors.

5.2 Objectivity, novelty and triviality: the OAMJ quandary

OAMJs were in part envisioned as a means of removing novelty and significance constraints (as they were seen) from academic publishing. Anecdotal evidence and careful examination of selection criteria published on OAMJ websites, however, suggested that this might not always be clear. There was a sense that the demarcation lines between sound science and new/significant science were blurred. Clarification on this issue was sought in the interviews with senior editors, board members and OAMJ publishers.

An inextricable and complex relationship between soundness, novelty and triviality emerged from the interviews. It was acknowledged that in the strictest interpretation, a OAMJ should publish anything deemed to be technically sound, regardless of any lack of novelty, and no matter how trivial. Some participants believed their OAMJs adhered to this interpretation:

[…] we do mean sound science as in you’re a scientist, you work somewhere, you’ve gone through the proper process – and that’s all it means […] you can’t say “sound science plus”. You’re then a different kind of journal, you’re then a selective journal

(MJ publisher 8).

The overall picture, however, was more complex. Most participants agreed that there was a general need for the publication of more replication and null studies – particularly in the medical and life sciences – and the OAMJ approach was designed in part to facilitate this. Early players in the OAMJ landscape had recognised that need and overtly welcomed such papers. However, as one interviewee explained:

[…] the mega-journal works when people are honest in their intentions and that the reason they are doing it is because want to share their discoveries and as soon as it becomes cynical and it becomes about getting a publication, then the model falls down I think

(MJ publisher 9).

Several participants noted that PLOS ONE originally operated with a strict policy to publish anything sound, until it became clear that authors were submitting unexpectedly large numbers of what were described as “replications of meta-analyses” – papers that essentially added nothing to the scholarly discourse, and were written only to benefit the publication record of the authors. The influx of such papers was seen to have damaged the journal, particularly in terms of its credibility with authors. It was in response to this phenomenon that PLOS introduced a requirement for papers to be “worthy of inclusion in the published scientific record”. As one interviewee noted:

[…] the philosophy of the founders of PLOS ONE, it really was publish everything including trivia, it really was about that and what happens is in practice it is very hard to implement

(MJ publisher 1).

The changing of the PLOS ONE policy, while apparently relatively minor, nonetheless represents an important concession that not all sound science should be published.

Numerous other interviewees provided hypothetical examples of sound but trivial science unworthy of publication: calculating the boiling point of water at sea level; or counting the number of corals encountered on a walk across a reef. It was clear though that many interviewees were somewhat uncomfortable with the implications of this, not least because many had previously highlighted the OAMJ philosophy that research does not need to be new or exciting to be published. The response of two interviewees, when asked directly about whether papers were required to be novel or non-trivial, illustrate the issues and apparent contradictions publishers face reconciling a soundness-only policy with the realities of academic publishing:

You could end up with a really trivial result that actually isn’t worth publishing, but technically you’re not able to reject it if you’re on objective peer review. So there are some things that we need to work on there

(MJ publisher 5).

[…] although there is not a test for novelty in that way, because everything that actually passes the peer review hurdle can be published, but we’re not necessarily […] but if we do have an editor in place, if he does think that the research is just truly useless, even if it’s passed the peer review hurdle but it’s reproducing something that we’ve done a thousand of times before, we still won’t accept it, because we do want to be […] there would need to be some kind of quality threshold

(MJ publisher 10).

It was recognised that “some kind of quality threshold” might easily be interpreted as a subjective judgement beyond just the soundness of the science. As one interviewee asked: “What is so completely trivial that it is not science? What is that?” (MJ publisher 1). A range of suggestions were offered by interviewees. One suggested the test was whether the work was “useful in some way to somebody” (MJ publisher 11), while another believed that the article should be “an original contribution […] adding to the core of scientific knowledge in any way” (MJ publisher 12). One participant set the bar higher, stating that replication and null result studies were rejected unless they “show that some major idea was wrong” (MJ editor 5), a significant departure from the stated aims of most OAMJs. In the case of non-STEM disciplines, novelty appeared to be an even more difficult issue: the body of knowledge cannot always be tested and often consists of paradigms for which soundness makes little sense.

5.3 Quality and the pressure of the JIF

The JIF and rankings based on it, have been one of the most important drivers in academic publishing over the last few decades, and we asked participants for their views on the apparent tension between OAMJs’ vision, and the pressure that JIF is putting on the publishing sector.

A majority of OAMJ publishers recognised this apparent tension, but were also quick in their disavowal of the JIF as a meaningful measure. In support of such claims, some mentioned that they were – or were about to become – signatories of the San Francisco Declaration on Research Assessment (DORA) (www.ascb.org/dora/) agreement. One of DORA’s aims is to combat the implicit association that is made between JIF and the quality of individual papers. Advocates of DORA believe this perceived correlation between JIF and quality is unfounded and may mislead the appraisal of scientific research, be it in hiring and reward mechanisms, funding selection or simply in research-quality performance audits. It was recognised that arguments like this underpin the initial development of the OAMJ concept.

Many participants noted that the OAMJ is not conducive to achieving and maintaining a high JIF. An obvious point made here was that by removing any assessment of likely impact from the selection of articles, OAMJs were likely to publish large numbers of articles accruing few citations: “almost by definition an objective peer review journal is never going to be better than mediocre” (Non-MJ editor 1). Several other factors were perceived to limit OAMJ Impact Factors. The OAMJ goal of publishing null-result and replication studies was mentioned, since they do not usually attract large numbers of citations. Similarly, certain subjects have different citation patterns, which can affect the JIF. With their broad subject scope, OAMJs are likely to publish papers in disciplines and sub-disciplines with low overall citation rates. This relates to particularly to OAMJs that publish research in the social sciences and humanities (which have traditionally much lower citation rates than the sciences) alongside articles in the science, technology and medical areas. Finally, it was noted that fast growing, large-volume journals tend to receive lower Impact Factors, since the denominator of the JIF formula rises disproportionally to citations.

While on the one hand OAMJs can be considered standard-bearers for a new scholarly communication paradigm in which the role of the JIF is marginalised, there was also a clear recognition from participants that OAMJs can and do benefit from achieving and maintaining a respectable Impact Factor. It was widely noted that several OAMJs experienced a surge in submissions following the award of a JIF, and several publishers reported a strong belief that JIF was a primary motivating factor for authors to submit to a journal. OAMJs are therefore subjected to what one participant termed “contradictory pressures” (MJ editor 4). This was articulated clearly by one publisher:

The impact factor is still the metric that is most commonly used […] And so we certainly would like the impact factor to go up and of course there is a little bit of tension between wanting an impact factor go up but claiming that we don’t care about the importance of the article, just the soundness

(MJ editor 5).

Several interviewees noted the relatively high Impact Factors of some OAMJs, and viewed these with suspicion. The implication was that such journals had expanded the criteria of their peer review beyond simply soundness. As one participant suggested, “there must still be an element of gatekeeping going on to enable these impact factors to be so high. I think there’s a lot more rejection than is made out” (MJ publisher 13). That this might represent a deliberate strategy on the part of the publisher was strongly disputed by representatives of the journals in question, with one stating that they were expecting to see their impact factor drop. Academic editors of the journals agreed that they had never been subjected to overt influence: “I’ve never been encouraged one way or another, when I’m looking at a paper, to think whether it’s going to help an impact factor” (MJ editor 4). However, others believed that the process might occur almost “accidentally”, with OAMJ staff previously unconcerned by their Impact Factor “cutting off the flow of some papers” in order to maintain an unexpectedly high JIF (Non-MJ publisher 4). It was also felt that academic editors could be influenced by a journal’s Impact Factor without any explicit instruction from the publisher to do so:

As the impact factor goes up, we will see instances in which the peer reviewers say everything is fine, and then the Editor makes a decision saying “I want to reject this because it’s not interesting enough”

(MJ publisher 12).

There was also a suggestion from one publisher that OAMJs in general, while perhaps not expanding the scope of the peer review, were instead filtering out submissions based on their type:

Most mega-journals don’t publish negative results, because I think it brings citations down. Remember it’s still a business model. I’m giving you the very honest, frank publisher view

(MJ publisher 8).

A recurring argument from OAMJ advocates was that the JIF was not a reliable indicator of the quality of individual articles. For some, quality was directly and clearly linked to certain levels of novelty and significance; for others, quality referred to a more fundamental characteristic of the research described in the article. There was recognition that there were some outstanding papers in OAMJs, and it was suggested that the vast increase in research outputs meant that these papers were no longer making it into traditional journals, owing to the fact that they were still constrained by the number of papers they can accept. Some interviewees commented that the negative views of OAMJs held by some publishers and authors, who were often felt to perceive them as lower-quality publications or “dumping grounds”, were directly related to the OAMJ peer review policy. According to some interviewees, this “dumping ground” perception of OAMJs seemed to be particularly prevalent in certain fields, for example, biomedical sciences, perhaps owing to the influx of papers of variable quality in these disciplines.

In response, interviewees often insisted that low selectivity does not necessarily mean low quality. A recurring argument was that the definition of quality had been distorted, and had become too closely linked to the JIF of the publishing journal. One interviewee argued that:

[…] the concept of a dumping ground is again I think predicated on the wrong definition of quality. An article is not poor quality simply because the findings are less exciting than some other article. I think we need to get away from this what I call almost celebrity scientist culture

(MJ publisher 5).

Most interviewees refuted the dumping ground claim, but did acknowledge the extreme variability of quality in papers published in OAMJs. Far from seeing this as an issue, it was seen as the result of OAMJs being conceived as much as repositories of papers as journals in the traditional sense. Given the discovery tools to which scholars have access, it was felt acceptable to have papers at both ends of the quality spectrum, from excellent to marginal quality, from highly cited to not cited at all, in one single place.

5.4 The limitations of post-publication indicators

As has been discussed, post-publication article level and altmetrics are often considered integral to the OAMJ concept. When asked how effective such indicators are in practice, however, our interviewees were circumspect. Some acknowledged that traditional post-publication measures such as citations or article downloads can provide useful feedback for readers regarding the potential importance or quality of an article, albeit with some limitations; citations, it was noted, take time to accrue, and therefore provide little in the way of immediate information. Several interviewees also acknowledged that commenting functionality is rarely used.

Much of the discussion of post-publication indicators centred on the use of altmetrics. There was general agreement that such measures could be useful for authors and journals as a means of gauging levels of social interest in an article in essentially real time. Views were much more mixed though on the extent to which altmetrics served as indicators of an article’s importance or significance. One participant suggested that it was too soon to determine how best altmetrics might be used:

I think altmetrics are really important, but I am not entirely sure we know what they mean. So it’s great to have a measure of how an article is received, but I’m not sure we know how to interpret those altmetrics scores properly yet

(MJ publisher 12).

This was echoed by other interviewees, who felt that there was not yet enough evidence to form firm opinions on the validity and usefulness of the measures. A number of others, however, were able to point to perceived issues with altmetrics. Typically these related to the potential for funny, quirky or controversial articles to achieve high altmetrics scores, a phenomenon participants linked to the measure’s lack of a qualitative element. As one interviewee put it:

You can get a massive altmetrics score if you’ve got a funny article. They’re indicators. I think there is a need for a qualitative assessment as well as quantitative how many views and downloads. Useful, but where do they come from, why is it happening?

(MJ publisher 13).

A number of participants felt that it was this factor that inhibited altmetrics’ potential usefulness as an indicator of an article’s importance, suggesting that “quantity is no predictor of real significance and impact” (Non-MJ publisher 4). Indeed some were relatively scathing about arguments for using altmetrics in such a way: “altmetrics are used in a very silly way which is that somehow magically readers are going to be able to psychically pick out good content from bad content” (MJ publisher 3).

Several interviewees felt that no metric would ever replace the need for researchers to read articles they were seeking to evaluate. Others felt that better tools to support this evaluation process were possible, and that the publishing industry could do more to make use of technologies being developed in other industries. One participant, while agreeing with this, noted that:

Building those sorts of tools that enable researchers to get connected to the right bits of content, published however, disseminated however is actually incredibly challenging and difficult and expensive to do

(MJ publisher 3).

6. Discussion

The views of soundness-only peer review offered by the senior publishers and editors reveal a rich picture of both the concept and reality of OAMJ publishing. Our findings suggest that there is often a tension between the rhetoric associated with OAMJs and the reality; the vision has not always been reflected in practice. There are multiple market, cultural and human factors influencing the shape of the practices adopted by OAMJs, meaning that deviations from the initial aims of publishers appear inevitable. These deviations fall into one of two basic categories: publisher supported, and practice driven.

Publisher-supported deviations represent the strategic and operational decisions made by publishers. Most notably, responses to the issue of triviality represent a potential separation of initial theory and current practice. While the pure OAMJ approach has no formal requirement for studies to be of particular interest or novelty, the realities of operating with such a policy appear to have been problematic. The example of PLOS ONE, who judged it necessary to amend their editorial policy to include a measure of “worthiness”, is instructive. Authors who are heavily incentivised to publish more articles were able essentially to game the system, resulting in the publication of papers with no recognisable value or new contribution. It was notable that the terms used to describe the practical thresholds applied by OAMJs were vague, and clearly subjective: “contribution”, “quality” and “useful”. Such approaches might reasonably be described as reviewing for novelty and significance by stealth. OAMJs claim they are looking only for soundness but in practice most OAMJs, if not all, converge towards the common idea that submitted papers need to add something to current knowledge, even if only incrementally.

As far as scope is concerned most OAMJs draw some sort of boundaries, even if very widely. Most publishers have clear pictures of the kinds of disciplinary areas their journals will cover, whether it is the breadth of a discipline (e.g. physics in the case of AIP Advances) or several related disciplines (biological and medical sciences in the case of PeerJ). Implicit in many of our interviews was the understanding that most OAMJs assume an STM publishing model and provide a venue for papers working within a positivist research paradigm. Although PLOS ONE has accepted papers in some social sciences areas, these papers tend to be quantitative in their approach or concerned with topics such as science policy. Only Heliyon publishes across all disciplines. Interestingly, SpringerPlus did publish across all disciplines but cited this approach as one of the reasons for its closure in 2016:

[…] we have found that researchers in the HSS (humanities and social sciences) and TEM (Technical Engineering and Mathematics) subjects prefer to publish in journals which cater specifically to them […] When you look at the wider publishing landscape, the megajournals which have had the most success (PLOS ONE, Scientific Reports) cater to a select range of disciplines

(Epstein, 2016).

It seems that even OAMJs have limits of scope and this was implicit in the views expressed by our participants.

Practice-driven deviations derive from the reality of the way editors and reviewers behave. There was a general recognition that OAMJ peer reviewers, whether through ignorance of OAMJ policies or belief in the value of the traditional model, frequently include judgements of novelty or significance in their reports and recommendations. This was agreed to place a significant burden on academic editors of OAMJs, who are required to disentangle novelty and significance from soundness in the reports they receive in order to inform the acceptance decisions they make. Evidence from the interviews, both in terms of the comments of active editors unaware of the soundness-only criterion, and the measures described by publishers to monitor editor performance, suggests that there is good reason to doubt whether all editors can be relied upon to conduct this disentanglement. One question arising from this, of course, is whether it is just a matter of time, and as the approach becomes more widely accepted, reviewers and editors can be relied upon to implement the model as intended. The answer to this is not yet clear. Additional suggestions that editors may be influenced in their work by the desire to improve or maintain an OAMJ’s Impact Factor further complicate matters. That there are clear potential benefits to a OAMJ of a more selective approach, most notably a higher JIF and associated reputational boost, has fuelled suspicion that some publications are not averse to more selective acceptance practices.

There is an important distinction to be made between the publisher-supported and practice-driven deviations described above, and how both relate to traditional peer review practices. The publisher-supported inclusion of a “worthiness” criterion, which in effect means a judgement of significance and novelty, is somewhat different to the evaluation of these factors undertaken during the traditional peer review process. The OAMJ peer review model described by our interviewees requires an essentially binary judgement: an article either has some potential novelty and significance, in which case it should be published, or it does not, and should therefore be rejected. In contrast, the traditional peer review model seeks a more graduated evaluation of novelty and significance; it asks whether the paper is significant or novel enough to merit publication in a particular journal. Practice-driven deviations from the OAMJ model – reviewers and editors utilising traditional peer review criteria in their OAMJ roles – naturally represent this more nuanced form of judgement. It can be argued, therefore, that these practice-driven deviations represent a more fundamental challenge to the core values of the OAMJ model.

A final de facto deviation from the OAMJ vision is associated with post-publication assessment. If the novelty or significance of a paper is to be decided by the community after publication, post-publication article-level metrics should form a crucial part of the evidence base of what the community has decided. Our participants acknowledged, however, that at present such metrics are at best nascent.

It is instructive to update the model of peer review presented earlier with these findings. As noted, the rhetoric of soundness-only peer review has stressed that judgements of the three other elements of traditional peer review (novelty/originality, significance/importance and scope/relevance) should be made by the community after publication. As Figure 3 illustrates, in practice, we find that such judgements to some extent inform pre-publication acceptance decisions. Publishers have often extended OAMJ pre-publication peer review either explicitly or implicitly to include a criterion of “worthiness” as a baseline threshold of both significance and novelty (indicated by the dark shaded areas on a small portion of the boxes novelty/originality and soundness/rigour in the model). They have for the most part also included notions of scope, although this varies in specificity, which seems to involve some kinds of assumptions of different types of scholarly literature they will publish. At the same time, there is evidence that reviewers and editors may sometimes apply traditional modes of article evaluation, despite guidance to the contrary (indicated by the dotted shading in each of the three boxes). Finally, post-publication measures, which ought to have some importance in the soundness-only peer review model by providing evidence of the way in which the “community has decided”, are underdeveloped as yet. At present, they provide at best only a partial view of novelty, significance or relevance (indicated by the dotted line to the article). Furthermore they cannot easily be used to distinguish the different criteria (e.g. between novelty and importance) in the way pre-publication peer review can (as indicated by the removal of the dividing lines between them in the Figure). At present, they can be claimed to be only a very minor part of the overall quality assessment process of OAMJs and have only a marginally more important in role in OAMJs than they do for conventional journals.

Perhaps more fundamental is the question of whether soundness-only peer review can ever remove subjectivity from the peer review process. Our participants recognised that the rhetoric of “objective” peer review may not always be matched by the reality. Even without additional criteria such as “worthiness”, the peer review process still has room for subjectivity and bias. If that is the case, some have argued that the way to address the possible biases is to create greater transparency in the system, and one way to achieve that is through opening up the peer review process itself. Some publishers have, therefore, shifted the model of peer review still further by moving judgements of soundness or rigour to a post-publication, rather than pre-publication stage (Figure 4). F1000 Research is perhaps the most prominent example of this. It also has a soundness-only peer review policy but one which takes place openly, including identification of reviewers and publication of their reviews alongside the article. The only pre-publication quality assurance process is the initial in-house screening. Whilst not typical of OAMJs, of course, it is an interesting variation on the soundness-only approach that attempts to address the perceived subjectivity problem in an even more radical way. An understanding of extent to which it and other publishing platforms like it are successful in achieving those aims and the ways in which peer reviewers actually carry out reviews in this environment would require further research.

It is important to emphasise that our analysis is not intended to debunk the fundamental tenets of the OAMJ model. The arguments presented by interviewees to justify OAMJ peer review policies are credible, and link closely to broader and widely supported movements to reduce barriers to the publication of, and access to, scientific research. The success of prominent OAMJ titles also suggests the approach is popular with authors. Rather, we hope that our findings serve to inform ongoing debates about the efficacy and practicalities of soundness-only peer review. It is ironic to note that many of the criticisms levelled at OAMJs, particularly those relating to quality assurance, might actually have been avoided had the original rhetoric better incorporated the notion of “worthiness” as a criterion for acceptance.

It should also be pointed out that whilst it may be possible to set a threshold for it, soundness is not a simple homogeneous concept. There are different dimensions of soundness, including quality of the research design, rigour of the data gathering or experimentation, accuracy of the data analysis, validity of the inferences drawn from the data and so on. These different dimensions may also be carried out with different levels of precision, verification and thoroughness. It is perhaps surprising that these issues were not discussed in any detail by our participants. They would certainly merit further investigation. Other traditional peer review criteria may also be multi-dimensional, of course – for example, a study may be novel in its design, subject, or theory development, or in other respects.

7. Conclusions

This paper is the first to examine in detail what soundness-only peer review means through the prism of an analytical model. By exploring the ways in which soundness-only peer review differs, or not, from traditional pre-publication peer review, it helps explain better its role in OAMJs. It also helps to bring to the fore the important tensions that exist between what OAMJs say they do (or aim to do) and what they can (and cannot) do in practice. This paper has aimed to present a realistic assessment of the vision OAMJs have vs the practicalities and constraints they face in the current publishing landscape; a recognition of the tensions between idealism and reality, intention and practicality.

Whilst the concept of soundness-only peer review undoubtedly has merit, implementing it creates challenges. Pre-existing attitudes and behaviours around peer review are deeply embedded in the publisher and academic communities and are difficult to change. We have identified a number of publisher-supported and practice-driven deviations from the rhetoric often surrounding soundness-only peer review. Prominent amongst the former is the requirement that articles be “worthy” of publication – a judgement which inevitably touches on criteria of novelty, significance and relevance. Our findings also show that in practice reviewers often produce referee reports in the same way as for conventional journals, consistency in editorial practice is difficult to achieve owing to both the flatter structure of OAMJs and the scale of operations, and significance and novelty can still be important factors informing rejection decisions. Understanding these publisher-supported and practice-driven deviations from the original vision of OAMJ peer review sheds new light on the OAMJ phenomenon. Whilst the aim to “let the community decide” is an apparently simple one, attempts to implement it fully have proven controversial and remain problematical.

Figures

Figure 1

Conventional pre-publication peer review model

Figure 2

Soundness-only peer review model

Figure 3

OAMJ quality assessment in practice

Figure 4

Post-publication quality assessment

Table I

Interviewees

	Publishing an OAMJ	Not publishing an OAMJ
Number of publishers	10	6
Commercial	6	2
Society	3	2
Not-for-profit	1	2
Interviewees	21	10
Publishers	12	7
Editors/Board members	9	3

Appendix 1. Interview schedule – publishers

The following set of questions is directed at strategic aspects of producing a mega-journal:

1. What is your understanding of the term, “mega-journal” (e.g. PLOS ONE, SAGE Open and Scientific Reports)? What are the key characteristics of a mega-journal? Is the concept of the mega-journal important in your thinking?

2. Does your organisation produce a mega-journal? When was it set up? What was the rationale for setting it up?

3. (No mega-journal) – was it a strategic decision not to develop a mega-journal? If so, why?

   • Is it something that (your organisation) is considering?

4. What do you see as the major challenges facing journal publishers now and in the near future?

5. (Yes mega-journal or if considering one) Do you see the production of a mega-journal as addressing these challenges to some extent?

6. What solutions (other than producing a mega-journal), if any, have you developed, or are in the process of developing to address these challenges?

7. What do you consider to be the opportunities/challenges of setting up a mega-journal?

   • This question may not be relevant if the answer to 3(a) is no.

8. (Yes mega-journal or if considering one) what is the strategy (of your organisation) with regard to mega-journals in relation to its other titles or publishing activities?

   • What were the major considerations?

     1. –

        e.g. speed of publication, scope, reaching a broader audience, impact, peer review, reputation of your organisation.

   • What aspects of the above considerations did you consider important from a marketing perspective?

   • What mechanisms are in place to assist readers in discovery articles?

   The following set of questions is directed at operational aspects of producing a mega-journal:

9. What is your contribution to/involvement in producing a mega-journal?

10. Do mega-journals offer economies of scale?

11. It has been suggested that high output mega-journals can be used to subsidise traditional journal titles – what are your thoughts on this? What is the strategic relationship between mega-journals and highly selective titles in your thinking?

12. What factors do you consider in setting APCs and how are they determined between mega-journals and traditional journal titles?

13. How does the mega-journal in which you are involved address the issue of quality control and what is the rationale for the approach taken? (If no mega-journal – what are their attitudes these issues?)

    • Concept of scientific “soundness” – relevance?

    • What steps have you taken to ensure consistency across the editorial and review process?

    • Post-publication measures of quality – use of metrics?

    • Mega-journals have been seen as lowering quality (“dumping ground”) how would you respond to this?

14. In terms of the future of scholarly communication what role do you think mega-journals will play?

15. Are there other innovations on the publishing industry’s roadmap that may augment/supplant mega-journals?

16. Do you feel that mega-journals are changing the dynamics between authors, readers and publishers?

    • Are they giving visibility to authors from geographic regions previously excluded from international academic journal publishing?

    • Are they providing greater access for readers?

    • What is the role of metrics?

    • Given the broad-scope and review for technical soundness are they changing the gatekeeping role of publishers?

17. Are mega-journals enhancing interdisciplinarity?

    • What are the indicators that this is happening?

    • Why do you think that interdisciplinarity is being enhanced?

18. Are mega-journals enhancing knowledge transfer beyond the academic research community?

19. Do you have anything else to add?

20. Could you recommend appropriate academic editors that we might interview?

Appendix 2. Interview schedule – academic editors/editorial board members

The following set of questions is directed at strategic aspects of producing a mega-journal:

1. What is your understanding of the term “mega-journal”? Key characteristics of a mega-journal?

   • Is the concept of the mega-journal important in your thinking?

   The following set of questions is directed at operational aspects of producing a mega-journal:

2. What is your contribution to/involvement in producing a mega-journal?

3. How do you see your role?

4. How does the mega-journal in which you are involved address the issue of quality control/peer review and what is the rationale for the approach taken?

   • Is it possible to maintain high-quality standards with mega-journals?

   • Is there a difference between the mega-journal you are involved in and your involvement in a traditional journal?

5. Can mega-journals create economies of scale compared with conventional titles?

6. In terms of the future of scholarly communication what role do you think mega-journals will play?

   • Are there other innovations on the publishing industry’s roadmap that may augment/supplant mega-journals?

7. Do you feel that mega-journals are changing the dynamics between authors, readers and publishers?

   • Are they changing the gatekeeping role of publishers?

   • What are the advantages/disadvantages of mega-journals from an author’s perspective?

   • What are the benefits/challenges of mega-journals from a publisher’s perspective?

   • What are the benefits/challenges from a reader’s perspective?

   The following set of questions is directed at knowledge exchange issues:

8. Are mega-journals enhancing interdisciplinarity?

9. Are mega-journals enhancing knowledge transfer beyond the academic research community?