The Need for Peer Review Reform. From Scientific Stagnation to Ideological Gatekeeping
Peer review could be responsible for scientific stagnation, paradigmatic monopolies and the ideological influence on certain disciplines such as economics. That is why we need peer review reform.
‘The ruling ideas of each age have ever been the ideas of its ruling class.’ (Marx & Engels)
Some academic disciplines are facing major challenges. These range from scientific stagnation to the monopolization of individual paradigms to ideological influence.
In what follows, I discuss these problems and argue that peer review, the most significant "quality control mechanism" within academic science, may be partly responsible for them.
Thus, I support here current calls for peer review reform, which seem to come very often from the field of medical research, where the lack of scientific progress may feel particularly painful for researchers.
Peer review reform has the purpose of liberating academic science from a host of problems and "biases" and aims at optimizing research. One may hope that it will help academic science return to its former glory.
Moreover, peer review reform may have the potential to re-establish pluralism in normative disciplines such as economics and free them from the neoliberal-neoclassical hegemony that has towered over them for decades. Peer review reform would thus have a significant sociopolitical impact.
Scientific Stagnation
Some academic disciplines face severe challenges. One, on which many can agree, is that academic science, in general, suffers from conservatism and status bias. That means, academic science overly engages with already established work and favors the work of so-called senior researchers from prestigious universities.
That has led in many disciplines to stagnation, both at the level of everyday applied research and big-picture paradigmatic theory development. That is of course concerning in itself.
Paradigmatic Monopolies
Another challenge is posed in at least some disciplines by the monopoly concentration of single paradigms. For instance, in philosophy, analytic philosophy. In economics, neoclassical economics in the neoliberal vein. It is, of course, an open question whether such monopoly positions are the result of scientific progress or the opposite thereof.
Yet, a cautionary glance at the history of ideas suggests that such monopoly positions are undesirable and might partially account for the stagnation described above. Another look at philosophy helps to clarify this. The unchallenged dominance of scholasticism in the Middle Ages is a negative point in case against monopoly positions.
Compared to that, the great epochs of philosophy, for instance, the Greek antique, early modern philosophy with the mutual inspiration between Rationalism and Empiricism, as well as 19th century and early 20th century Continental philosophy, were characterized by a bewildering number of competing paradigms.
Ideological Capture
Further, if we look at the challenges that I described above from the perspective of left-wing thought and politics, a bigger problem emerges. Roughly beginning with the neoliberal revolution in the 1970s and 1980s, we see in various disciplines the vanishing of classic left-wing ideas and their replacement by work that is either directly neoliberal or that does not challenge neoliberalism.
Leftist ideas have been replaced in ideologically relevant disciplines, such as economics, by paradigms that are either fully neoliberal or at least have nothing effective to show against neoliberal hegemony.
How this displacement process worked, in which neoliberal hegemony was achieved in academia, is, in my opinion, still not very well understood. There is good literature that points out the role of think tanks. In particular, Dieter Plehwe and Bernhard Walpen have done tremendous work in that respect.
They show, for instance, how members of the Mont Pèlerin Society—the neoliberal founding group per se—had a tremendous influence on the so-called “Nobel Prize in Economics”, which resulted in many members of the Mont Pèlerin Society and other neoliberals receiving the prestigious prize, which had a tremendous effect on academia.
Yet, that still leaves a lot of room to discuss other forms of influence-taking. In particular, how neoliberal academics were able to push other academic paradigms to the fringes while creating near unchallenged dominance for themselves. Accounting for this, is a central issue for the left.
Vital issue for the left: Neoliberal hegemony is also a hegemony of ideas
Neoliberalism today has an almost unchallenged hegemony even in public opinion. Intellectual alternatives to neoliberalism are not known or frowned upon by the public, which is obviously reflected in people's voting behavior.
This near hegemony on public thinking is classically explained by media and think tank influence. That is, the neoliberal revolution is considered a propagandistic revolution that the population is unaware of, but which has nonetheless permanently changed public opinion.
However, research on the role of universities in the dissemination of neoliberalism is still underdeveloped. Where early Critical Theory, Marcuse, Mills, Althusser, Bourdieu or even Chomsky had still critically examined universities in the best leftist tradition, it is increasingly hard to find this kind of thinking today.
This is surprising, because one would expect universities to have a massive impact on public thinking and thus on shaping our politics and society. Further, it is difficult to imagine that the neoliberal revolution could have succeeded without the universities. For instance, the technocratic expert rule of neoliberalism would not be possible if universities did not train the experts for it in the neoliberal vein, nor provide the "experts" who praise neoliberal policies in the mass media.
Furthermore, it is difficult to imagine that the neoliberal revolution could have succeeded without the universities. The historical revisionism that accompanies the neoliberal revolution could never have succeeded if even classically left-leaning disciplines had not been ideologically influenced or weakened in one way or another.
Considering further that the most prominent proponents of neoliberalism have been economics professors at prestigious universities, one should be open to the question of how far universities have contributed to the propagandistic propagation of neoliberalism.
This propagandistic dissemination, however, first presupposes the unjustified propagation of neoliberal ideas at universities themselves. If we assume that this dissemination is unjustified—for which there is much to be said, and not only from a leftist perspective—then we must try to identify the mechanisms by which neoliberals have been able to establish their hegemony.
Thus, I attempt here to offer such an explanatory approach. In this explanatory approach, I focus on peer review as a possible mechanism that partially explains the hegemony of neoliberalism within academia.
How does it all fit together?
At first glance, it may seem surprising what ideological hegemony has to do with research stagnation and research monopolies.
However, I discuss these things together here to first show that peer review is scientifically undesirable and structurally has the potential to lead to abuses of power and academic monopoly building. From there, it is easier to show the extent to which peer review can be used to establish ideological hegemony.
Peer Review
Peer review is widely recognized as the gold standard of scientific research. The official rationale behind peer review is that research findings must undergo a quality test before they can be made public or receive funding.
This quality test consists in researchers, who are deemed specialists in the field, reviewing and approving the research that other researchers want to publish or get funded; in that sense, research is reviewed by scientific peers, hence the name “peer review”.
In the following, I suggest that peer review might not be the quality control mechanism as which it is made out to be. First, peer review often seems to possess little validity and reliability, which means it does not qualify as a justifiable quality control mechanism.
Second, peer review has the potential to inhibit scientific creativity by making conservatism in research ("conservatism bias") and status bias ("status bias") more easily possible.
These biases partly explain scientific stagnation and point to peer review's potential for ideological gatekeeping, i.e., limiting access to academic jobs, funding, and publication opportunities because of ideological power that can thrive where room for status quo friendly biases lives.
Validity and reliability: Why peer review is not a good quality control mechanism
A growing body of literature on peer review strongly suggests that peer review reports disproportionately often exhibit low validity and reliability. For example, Conix and colleagues summarize a substantial portion of the research literature on peer review to demonstrate this circumstance. That is, peer review reports often do not predict well the success of the research they assess, nor do they often assess the quality and truth of that research particularly well either.
In this sense, peer review reports surprisingly often do not converge. Not only do they often not converge on the so-called "accept" and "reject" decisions by which research funding decisions as well as publication decisions are determined. Peer review reports also often do not converge in reliably identifying the same alleged strengths and weaknesses of research under investigation.
In addition, some researchers who examine peer review critically, complain that peer review has often little predictive value. This means that it is very common that peer review reports are not good at distinguishing between research that will later be successful and research that will turn out to be wrong.
The Peer Review Lottery
Sometimes it can seem like peer review is a lottery. Sometimes you get lucky with the review reports you receive, sometimes you get unlucky.
Not inappropriately, some researchers have argued that peer review should be replaced by lotteries when awarding grants, because the peer review process sometimes has as little validity and reliability as a lottery.
This may sound like a polemical proposal at first, but the underlying idea is to reduce the burden associated with peer review, i.e., to eliminate the excessive time commitment that researchers face either as reviewers or as authors who need to revise their research.
Working on peer review revisions or reviewing research takes a lot of time away from actual research. Since there is evidence that the validity and reliability of peer review reports are too often too low, researchers should invest their time in research rather than in peer review.
Conservatism bias and scientific stagnation
Peer review has the potential to inhibit creativity and scientific progress. Joshua Nicholson and John Ioannidis express this concern in Nature in an article aptly entitled, “Conform and be funded”. They point out that peer review is prone to conservatism bias.
It is easier to pass peer review if researchers write a paper that expresses little that is controversial and is well aligned with what is already published in peer review. Yet, of course, that is not a good recipe for scientific progress or quality for that.
The history of peer review is a surprisingly underdeveloped and unknown field. Most of us today believe that peer review is an intrinsic feature of science and research, yet it only emerged in the 17th century in England and France as a more benevolent version of publication control compared to the more usual theological and governmental censorship mechanisms. Yet, its wider application goes back only to the 1950s in the field of research funding, and not so much research publication, where it became more dominant even later.
From both a scientific and an intellectual point of view, the history of peer review suggests that the most influential work in scientific and social thought, from Newton to Darwin to Marx to Luxemburg to Curie to Einstein to Schrödinger, to name but a very few researchers, has emerged outside of peer review.
By contrast, most of us will find it difficult to name thinkers of similar caliber who have done research since the introduction of peer review, despite the fact that many more people are doing research today, with many more technical and financial resources at hand.
It seems that after the introduction of peer review, scientific progress came to a halt, pluralism disappeared and stagnation set in.
Status bias, prestige, and a first glance at intellectual hegemony
It is furthermore known that peer review is prone to biases, such as the status bias towards prestigious individuals and institutions. For instance, Huber et al. have shown that a paper written by a well-known researcher is three times more likely to be published than that of a little-known researcher; while also witnessing the even more disturbing phenomenon that an anonymized version of a submitted research paper was more likely to be accepted by reviewers than one that contained only the name of the lesser known researcher.
The latter expresses the discriminatory attitude towards junior researchers according to which reviewers are more willing to accept a paper of which they might hope that it is written by a prestigious researcher than by a junior researcher. As such, status bias is an expression of low validity and reliability, and of a bias towards what is already published—in this case by well-known researchers.
Ideological gatekeeping
The potential for gatekeeping looms particularly large in normative disciplines, such as economics. Heterodox economists like Ha-Joon Chang have complained about the monoculture that we have seen for a few decades in academic economics. As Chang states:
“Up to the 1970s, economics was populated by a diverse range of ‘schools’ containing different visions and research methods – classical, Marxist, neoclassical, Keynesian, developmentalist, Austrian, Schumpeterian, institutionalist, and behaviouralist, to name only the most significant.”
Today, neoclassical economics of the neoliberal variety has virtually displaced all these other paradigms and achieved a monopoly position. Even the figurehead of the neoliberal status quo in economics, Larry Summers, expressed a wish to see more intellectual competition in economics in this regard.
This lack of pluralism in economics puzzles many, but the gatekeeping power of peer review might partially explain it. If ideas not in accord with neoclassical economics are not published in prestigious peer-reviewed journals—of which there have to be only few (top 5)—over time because they are rejected in peer review on doubtful grounds, then over time only neoclassical economists will receive jobs, first at prestigious institutions and then everywhere else.
Massive concentration of power of top journals and top universities
Since there are tendencies towards status thinking in peer review that favor renowned elite scholars and experts, it is obvious that we can identify a hegemonic mechanism here, combined with the fact that, for example, in a country like the USA, only a few elite universities educate the majority of university professors.
If we further combine this line of thought with the fact that only a few elite journals determine some academic discourses, we might have identified here a massive potential for ideological hegemony through pure socio-economic power.
Even more, if we assume a little more critically that elite universities are "elite" not because of the quality of their research and teaching, but because their function is to maintain the elite status of their graduates and to produce elites who work as the managerial class, then it is even more plausible to conclude that the status thinking potentially inherent in peer review is well aligned with elite status and thus elite ideologies such as neoliberalism.
This concentration of power creates a kind of idea bottleneck that makes it possible to strategically orchestrate a non-epistemically motivated change of ideas throughout the university ecosystem.
The peer review revolution runs parallel to the neoliberal revolution
This hegemonic concentration of power is exactly what we saw in economics, partially parallel to the introduction of peer review in economics. If we then consider that neoclassical economics is most aligned with the political and economic status quo in society, the worry about gatekeeping deepens.
In that sense, I believe we can see this dominance of neoliberal ideas as an expression of Marx’s and Engel’s statement: ‘The ruling ideas of each age have ever been the ideas of its ruling class.’ This hegemony or ideological rule might be enabled by peer review because peer review is a powerful mechanism, since it controls the publication of ideas, academic employment and the funding of academic research.
The sociology of peer review
For academic outsiders, it might be difficult to grasp how peer review allows for gatekeeping. But a closer look at peer review can help gain insight. For one, peer review reports are usually written by anonymous referees who might or might not have professional, personal, or ideological conflicts of interest with the refereed research and the researchers they judge. Since peer review is anonymous and, in that sense, untransparent, there are no real safeguards in place to hinder intellectual or economic conflicts of interest.
Furthermore, although peer review reports can be de jure objected to in the sense that a dissatisfied researcher can directly contact the editors of a journal or address reviewers indirectly through a so-called “response letter”, the de facto likelihood that such an “appeal” is granted is highly unlikely.
Importantly, there are no official due diligence procedures that would guide such an appeal, in which the appealing researcher had a say or could have insight into. The chance of an appeal is completely dependent on the goodwill of the editors. Therefore, peer review in its current form is essentially unaccountable, unappealable, and untransparent.
Peer review is a systemic problem
I want to make it clear that my point here is not to blame individual researchers for the problems with peer review; let alone to claim that researchers usually act with bad intent in peer review. I have been a peer reviewer myself, and I know plenty of researchers who act with the best of intentions in peer review. Like any other social problem, the peer review problem is a systematic institutional one.
I like to compare peer review to the legal system in this regard. Peer review is like a legal system in which defendants (researchers submitting their research) have no rights and no defenders. The only actors with power are the prosecutors (reviewers) and judges (editors) (their roles are oddly mixed in peer review though).
The prosecutors are usually anonymous and their judgments are to be abided. The process is secret and one has no real right of appeal as a defendant. Such a system, characterized by non-transparency and non-appealability, results in a massive imbalance of power.
Such a system has the potential for serious institutional problems, such as scientific stagnation, monopoly positions, and ideological gatekeeping, even when the majority of actors act with the best of intentions. We would assume exactly the same in every other spehere of society. Therefore, peer review calls for systemic reform.
Peer review reform: The case of eLife
Such systemic reform is in our sight. eLife, a highly impactful journal in the medical and life sciences, has recently started to replace the classic peer review process, supported by an editorial that forwards a scathing critique of the classic model.
eLife replaces classic peer review with what we could call “public peer review”, which comes without, as the editors correctly call it, “gatekeeping”. While peer review was anonymous, not public, not appealable, and resulted in an “accept” or “reject” verdict, eLife gives up on these practices.
Public peer review is based on preprints. Preprints are usually research papers uploaded without gatekeeping restrictions to preprint repositories such as OSF preprints or SSRN where everyone can read and comment on the paper.
That concretely means that readers can judge the paper on their own and make suggestions on how to enhance the quality of the paper. However, these suggestions are not binding, cannot reject the paper and leave it open to the authors to implement them or not. Furthermore, the entire proceeding is visible to the authors and the public. These changes should make gatekeeping in its current form more difficult.
Making academic publishing and funding decisions publicly accountable
Peer review is a decision-making mechanism with significant consequences for science, science policy, and the sphere of the political. Peer review should therefore, in principle, be transparent, public, and democratic, and eLife’s decision might be exactly the step in the right direction. Yet, we could also go beyond that decision and do away with peer review completely.
If left-wingers are interested in breaking the dominance of neoliberalism, they should be at the forefront of either reforming or abolishing peer review. Because peer review—at the level of publication and research funding—has a high potential for hegemony creation that disfavors left-wing thought and intellectual alternatives to neoliberalism or any other paradigm that is supported by the ruling class.
This article is a translation from German. The original article can be found on my blog, Alexanderplatz.