Introduction. Peer review is fundamental to scientific progress. However, it remains a ‘black box’ due to the ambiguous and subjective decision-making rationale, hidden within reviewers’ perceptions and judgments.
Research gaps. The scarcity of peer review data and the limited analytical scope have hindered empirical studies from thoroughly exploring these underlying factors.
Method. We collected 114,006 review records from OpenReview. Developing deep learning models, we extracted reviewers’ perceptions of key factors from the review content.
Analysis. Our analysis focuses on how these perceptual factors influence reviewers’ decisions and outcomes of the papers.
Results. Originality is the element that reviewers value and recognize the most, followed by motivation. In contrast, chairs prioritize the overall validity of the work within the field, giving more weight to the substance and soundness of papers. The role of motivation is generally supportive but has limited distinguishing power on final decisions. Perceived clarity, replicability, and meaningful comparison have a relatively minor role in the review process.
Conclusion. Our study highlights the decision-making process based on reviewers’ perceptions and uncovers perceptual differences between reviewers and chairs. This contributes to greater transparency in the peer review and helps build trust within the scientific community.
Introduction
Peer review is considered the backbone of scientific construction (Alberts et al., 2008). By filtering out substandard work and providing expert feedback, it helps ensure the quality and credibility of scientific research, contributing to the evolution and enrichment of knowledge (Bailly, 2016; Jefferson et al., 2002; Riley & Jones, 2016). However, despite its importance and decisiveness, the decision-making process in peer review mechanisms often resembles a black box (Chong & Mason, 2021; Cutler et al., 2022), which may raise concerns within the scientific community. It arises not only from the non-transparency of the procedure (Hamilton et al., 2020) but also from the ambiguous and subjective rationale underlying the reviewers’ decision-making.
The review decision-making process involves complex cognitive processes (Brosch et al., 2013; Lindsay & Norman, 2013; Marietta & Barker, 2019), with perceptions and judgments formed based on the presented facts (Dror & Fraser-Mackenzie, 2008; Street & Ward, 2019). It is complex and subjective, influenced by reviewers’ cognitive biases and qualifications (Benda & Engels, 2011). Their decisions—whether to accept or reject a paper—are affected by factors they have perceived such as significance, originality, methodological rigor, and presentation (Blockeel et al., 2017).
The research framework of previous studies compared to our work. Prior empirical studies on peer review decision-making have focused on extracting the objective characteristics of papers to reveal the preliminary process of reviewers’ decisions. However, the reviewers’ perceptions and judgments remain unexplored.
none
Existing research empirically unveils reviewers’ decision-making rationale, biases, or preferences, typically employing a ‘fact-decision’ approach (Figure 1). For instance, they utilize external metrics, such as the disruption index, to objectively describe the characteristics of papers and then examine whether reviewers’ decisions show a preference for certain work (Liang et al., 2023; Teplitskiy et al., 2022). This approach overlooks the crucial intermediate step—the ‘perception and judgment’ phase, closely intertwined with the cognitive processes of reviewers (Lamont, 2009; Lee et al., 2013). Exploring this can offer valuable insights into the cognitive aspects of decision-making in peer review, enhance transparency, and build greater trust in the academic community.
Nevertheless, there are still some critical research gaps that may hinder further exploration: (a) The first challenge is about how to measure the reviewer’s perception. Generally, reviewers’ perceptions could be revealed in their comments like enthusiastic praise or critical criticism (Mavrogenis et al., 2020). However, most peer review processes are not publicly available and related data is extremely scarce (Balietti et al., 2016), making it very difficult to obtain reviewer comments in real-world scenarios. Furthermore, the review contents are usually unstructured data, which makes it harder to extract meaningful insights. Prior studies examined review content through single-dimensional features like sentiment (Matsui et al., 2021), tone (Nobarany & Booth, 2015) and length (Maddi & Miotti, 2024), but have struggled to capture the multifaceted nature of the perceptions. (b) Few empirical studies assess the relationship between reviewers’ perceptions and decision-making. This lack of evidence hampers theory development and highlights gaps in understanding the peer review process and perceptual factors of reviewers.
To address these research gaps, we propose the following research questions (RQs):
RQ1: how can we automate the extraction of reviewers’ perceptions and judgments of a paper?
RQ2: from the perspective of reviewers’ decisions, how do perceptual factors influence reviewers?
RQ3: from the perspective of papers’ decisions, how do reviewers’ perceptions affect the decision of a paper?
To answer the above research questions, we collected a substantial number of open review records from OpenReview and carefully filtered them to construct a valuable peer-review dataset, comprising 114,640 records from top-tier computer science conferences. Then, we develop deep learning models to automatically extract and measure reviewers’ multi-dimensional perceptual factors (RQ1). Based on these factors, we conceptualize the peer review decision-making process into two primary phases: individual reviewer decisions (RQ2) and paper decisions affected by multiple reviewers (RQ3) and examine the relationship between reviewers’ perceptions and decision-making in these phases.
Related work
Perception in peer review has been discussed by existing studies, particularly regarding how reviewers perceive the peer review process itself. Existing research has explored various aspects of reviewers’ perceptions, such as their motivations for reviewing, preferred roles, and preferences regarding review mechanisms (Curtin et al., 2018). Other studies have examined reviewers’ willingness to receive training, accept feedback, and sign their reviews (Manchikanti et al., 2015). Less work has empirically investigated reviewers’ perceptions of the paper. One such study uses surveys to quantify the gap between researchers and reviewers regarding the perceived importance of specific validity and reliability issues (Street & Ward, 2019).
The process from perception to decision in peer review is inherently subjective, and this subjectivity plays a crucial role in shaping the overall review process. Existing work argues that such subjectivity is not only unavoidable but also necessary, as it may help mitigate the herding effect and encourage diverse perspectives (Park et al., 2014). This subjectivity also contributes to the phenomenon of low inter-rater reliability among reviewers. While some researchers view this inconsistency as a natural and inevitable aspect of peer review (Roediger, 1991), others consider it problematic, suggesting that it undermines the reliability of the evaluation process (Marsh et al., 2007).
The process of review is not merely about evaluating the content of a paper; it is also shaped by ‘sense of self and their relative positioning’ of reviewers (Lamont, 2009). The connection between identity and evaluation raises concerns about the potential for cognitive biases, which are often cited as a threat to the progress and integrity of a field (Benda & Engels, 2011). Specifically, reviewers may show a preference for submissions from authors with similar ‘schools of thought’, a phenomenon known as ‘cognitive cronyism’ (Travis & Collins, 1991). Thus, while subjectivity in peer review can foster diverse viewpoints, it also presents challenges that must be addressed to ensure the fairness and accuracy of the evaluation process.
Dataset
To address the issue that the majority of journals and conferences’ reviews are not publicly available, open peer review offers a promising solution to enhance the transparency and fairness of the review process (Wolfram et al., 2020). OpenReview (Soergel et al., 2013) is one of the most famous open peer review websites to promote transparency in scientific communication. It provides submitted papers alongside their corresponding reviews and comments. After a thorough examination of each venue in OpenReview, we focused on the top-tier machine learning conferences—ICLR and NeurIPS for our study (ICLR makes reviews public regardless of acceptance, while NeurIPS allows authors of rejected papers to decide whether to make their reviews public). We collected all publicly available review data for these conferences. After data pre-processing and meticulous data filtering, we obtained review information for 30,440 papers from the year 2013 to 2024, comprising 114,640 records.
Measurement of variablesAutomatical extraction perceptual factors of reviewers (RQ1)
‘Perceptual factors’ can be defined as the reviewer’s subjective perception and judgment elements, which guide their review decision-making. The reviewer’s perception of one submission can be directly reflected in the review comments (Mavrogenis et al., 2020). According to previous studies (Yuan et al., 2022), we identify seven key perceptual factors(These dimensions follow the ACL review guidelines): motivation, originality, soundness, substance, replicability, meaningful comparison, and clarity.
To automatically and effectively measure these factors, we designed a pipeline that includes a discriminative model and a scoring model, each based on RoBERTa (Liu, 2019). Specifically, the discriminative model determines whether a sentence is perception related. If it does, the scoring model then assesses the multidimensional perceptual factors. To illustrate these indicators more clearly, we provide an implication and a simplified example in Figure 2. The detailed training process is outlined as follows (The hyperparameters for both models are set as follows: batch size of 16 and a learning rate of 5e-6. We employ the best model among 8 epochs with the lowest validation loss. The model is trained on an NVIDIA V100 32GB GPU):
We use AMPERE (Hua et al., 2019) and ASAP (Yuan et al., 2022) datasets to train our models. To train the discriminative model, we leverage AMPERE, which categorizes review comments into six types. We defined the task as a classification problem to identify whether a sentence belongs to the category ‘evaluation’. To train the scoring model, we use ASAP, which annotated seven dimensions of text spans with sentiment polarity. We converted it into a numeric labeling task. During training, the model takes the text as input and produces a 7-dimensional output, with each dimension representing a dimension. We use the Tanh function to scale each dimension. The loss is computed using Mean Squared Error (MSE). On the validation set (20% of total data), the model achieved a classification accuracy of 0.83 and an absolute deviation of 0.084 for each dimension.
The implication (A) and toy example (B) of the extracted perceptual factors. Each factor’s polarity is represented by a continuous value ranging from -1 to 1, where a value closer to 1 indicates higher confidence in a positive attitude. The first and last sentences in the example are categorized as ‘fact’ and ‘request’, respectively. Unlike the ‘evaluation’ category, these sentences exhibit less subjectivity and are thus excluded from measuring the reviewer’s perceptual factors. The three sentences in between cover various perceptual factors such as ‘motivation’, ‘originality’, ‘clarity’, and ‘soundness’, showcasing the reviewer’s diverse perceptions and judgments of the paper.
none
After measuring the perceptual factors of each sentence, we aggregate the scores as follows:
PFpi=∑PFseg,seg∈TpiPFp=1Np∑PFpi
Where PFpi represents the 7-dimensional vector of perceptual factors given by the i-th reviewer. seg represents the sentence of review content Tpi. PFp denotes the average judgment of all reviewers of the paper. The more frequently reviewers mention and emphasize a particular dimension in their comments, the higher the perceptual score for that dimension.
Reviewer decision
Reviewer decisions are a key criterion for chairs in determining paper acceptance, such as ‘8: accept, good paper’ or ‘3: reject, not good enough’. Because different conferences use varying scoring ranges from year to year, we standardize all scores to a range of 0 to 10 to ensure comparability. The formula for this calculation is as follows:
RDpi=rpi−minRmaxR−minR×10,r∈RRDp=1Np∑RDpi
Where p represents a paper reviewed at a particular conference (held in a specific year). R denotes all the ratings given at this conference. rpi refers to the original rating of the paper p given by the i- th reviewer, while RDpi represents the processed one. Np is the total number of reviews that the paper p received. RDp is the average rating of all reviewers.
ResultDescriptive statistics of reviewers’ perceptual factors
Description of the reviewers’ perceptual factors. (A) We calculated the Pearson correlation between different factors using data samples from individual reviews. It is shown that originality and motivation were the most correlated (and vice versa); soundness and clarity were the most correlated (and vice versa); substance and soundness were the most correlated; replicability and clarity were the most correlated; and meaningful comparison and substance were the most correlated. (B) We compared the average of reviewer decisions (RD) and perceptual factors for accepted and rejected papers. Unlike Figure 3 (A), which is based on single reviews, the data in Figure 3 (B) are based on individual papers, each with multiple reviews. We found that perceptual factors, whether relating to significance (Motivation), novelty (originality), or rigor (soundness, substance, e.g.), all have the potential to influence the decisions made about the papers. It shows the differences for each factor between the accepted and rejected categories, represented by the Cohen’s d.
none
Firstly, we observed the internal correlation of perceptual factors within reviewers’ comments. As shown in Figure 3 (A), we found that the internal correlations of these metrics were generally low (≤0.31), which means that there is a lower likelihood of serious multicollinearity issues in modeling.
Figure 3 (B) demonstrates that accepted papers receive higher evaluations from reviewers across all criteria compared to rejected ones. Rejected papers often get positive feedback on motivation but tend to receive negative or no feedback (-0.01) on originality. This indicates reviewers are more supportive of motivation regardless of their recommendation but more critical of originality in papers they dislike. Further, reviewers often give negative evaluations on rigor-related dimensions to prompt authors to improve.
We analyse individual reviewers to examine how their decisions are influenced by their perceptions and judgments. The model is specified as follows:
RDpi=β0+βPFpi+β'Xpi+αp+λp
RDpi represents the reviewer decision given by the i-th reviewer for paper p. The key variable PFpi denotes the vector of the reviewer’s perceptual factors. Xpi is a vector of control variables, including the request number and confidence of the reviewer. The request number represents the number of sentences related to the reviewers’ requests. Confidence is the assurance of reviewers, which is scored by themselves. Since the review process is double-blind, we do not control for author- related information in this model. αi and λi represent the fixed effects, capturing stable factors affecting the reviewer’s decision at the venue and time levels respectively. β0 represents the constant term. We conducted a VIF test (Table S1) and used OLS regression for our analysis (Table 1).
We identified the impact of various perceptual factors on reviewer decisions. Among all perceptual factors (Figure 4), reviewers prioritize originality, followed by motivation, as the primary basis for their decisions. Next in importance are soundness and substance, underscoring the paper’s overall persuasiveness and rigor. Lastly, reviewers focus on meaningful comparison, clarity, and replicability. They determine whether a paper is easy to follow and disseminate and they also form the basis for the paper’s credibility.
In Figure 5, we conducted a heterogeneity analysis of reviewers grouped by high and low confidence levels (the boundary is set at 5). The results showed that decisions made by high- confidence reviewers were more strongly associated with perceptual factors (except for replicability and clarity).
The regression result of reviewer decision analysis. The model has an R-squared value of 0.27, and the observation number is 107,272.
Dependent Variable: Reviewer decision
Coefficient
SE
P value
Confidence interval
Motivation
0.244
0.005
0.000
[0.235, 0.253]
Originality
0.372
0.005
0.000
[0.363, 0.381]
Soundness
0.214
0.003
0.000
[0.208, 0.220]
Substance
0.213
0.006
0.000
[0.201, 0.225]
Replicability
0.098
0.011
0.000
[0.076, 0.120]
Meaningful comparison
0.172
0.007
0.000
[0.158, 0.187]
Clarity
0.159
0.003
0.000
[0.154, 0.165]
Request number
0.020
0.001
0.000
[0.018, 0.021]
Confidence
-0.060
0.002
0.000
[-0.065, -0.056]
The coefficient comparison of reviewer decision analysis. Among all the perceptual factors, originality has the greatest impact on the reviewer’s decision, followed by the perception and judgment of motivation. ** represents p < 0.01, and * represents p < 0.05.
none
The impact of perceptions reflected in review content between high-confidence and low- confidence reviewers on their decisions (using the standardized median value, i.e., 5, as a cut-off, with the standardization process similar to RD). The middle line represents the coefficient for confidence at the median value of 5. The study found that, except for clarity and replicability, the perceptions of high- confidence reviewers are most closely associated with their decisions across various aspects. This reflects that decisions made by high-confidence reviewers are more interpretable and follow a logical progression from perception to decision-making.
nonePaper decision analysis (RQ3)
When deciding whether to accept a paper, the conference chairs take into account the reviewers’ scores and feedback to make the final decision on acceptance or rejection. We further analyse how the chairs’ decision is made to gain insights into the process of decision-making in peer review. Xp is the vector of control variables, including average RD, request number and the prestige of paper, where prestige refers to the average h-index of all authors, matched with S2AG (Version: May 21, 2024) (Wade, 2022). PaperDecisionp represents the final decision for paper p. It is a binary variable representing whether the paper is accepted. The model is specified as follows:
PaperDecisionp=β0+β1RDp+βPFp+β'Xpαp+λp
We conducted a VIF test (Table S2) and used logistic regression for our analysis (Table 2). We uncovered an interesting distinction (Figure 6): from the perspective of the chairs, the most critical perceptual factors are not originality or motivation, but rather substance and soundness. This contrasts with the focus of reviewers. The findings suggest that chairs place greater emphasis on ensuring the overall validity in the field. Moreover, factors such as motivation, replicability, clarity, and meaningful comparison do not have a significant impact on paper decisions. This could be due to the supportive nature (as shown in Figure 3) of factors like motivation, which may lack strong differentiating power from the perspective of papers’ decisions. It is shown in Figure 7 that in the high-RD group (RD below the median), positive judgments on validity-related dimensions, are more influential on papers’ decisions.
The regression results for paper decision analysis show that the pseudo-R-squared is 0.55, with a sample size of 17,620 observations. Since chairs’ decisions are heavily influenced by the average scores provided by reviewers, we excluded papers rated below the 1st percentile of accepted submissions (a total of 5,464 papers, with a 97% probability of rejection) and those rated above the 99th percentile of rejected submissions (a total of 2,268 papers, with a 97% probability of acceptance).
Dependent Variable: Paper decision
Coefficient
SE
P value
Odds multiplier
Confidence interval
Motivation
0.078
0.052
0.132
1.08
[-0.024, 0.180]
Originality
0.110
0.047
0.020
1.12
[0.017, 0.203]
Soundness
0.184
0.034
0.000
1.20
[0.118, 0.250]
Substance
0.192
0.064
0.003
1.21
[0.066, 0.318]
Replicability
-0.172
0.123
0.160
0.84
[-0.413, 0.068]
Meaningful comparison
-0.067
0.079
0.397
0.94
[-0.222, 0.088]
Clarity
0.059
0.032
0.070
1.06
[-0.005, 0.122]
Reviewer Decision
2.810
0.056
0.000
16.60
[2.700, 2.919]
Request number
-0.005
0.009
0.603
1.00
[-0.023, 0.013]
Prestige
0.005
0.002
0.020
1.01
[0.001, 0.010]
The coefficient comparison of paper decision analysis. We only present the main variables that are significant in the regression analysis. Among all the perceptual factors, substance and soundness have a significant positive correlation with the chair’s decision, followed by originality. The effects of other main variables on the paper’s decision are not significant. ** represents p < 0.01, and * represents p < 0.05.
none
We compared the high and low RD groups (divided by the median, with extremely high and low scores excluded in the same way as in Table 2) regarding reviewers’ perceptions of the papers. We found that decisions on papers in the high-RD group were more likely to be influenced by the reviewers’ perceptions. High scores already indicate the overall superiority of the paper’s quality. If reviewers’ evaluations of the validity-related factors are positive, it indicates that the chairs’ concerns have been addressed, thus increasing the likelihood that the chairs will accept the paper.
noneConclusion and discussion
Due to limited review content and narrow analytical dimensions, there’s little empirical research on the perception-to-decision process in peer review. Our findings show that originality is the most critical factor in reviewers’ decisions, followed by motivation. Chairs rely on reviewers’ decisions but prioritize assessments of soundness and substance, focusing on the research’s overall validity. Reviewers’ feedback on motivation is reassuring but has limited distinguishing ability for chairs. The Overall impact of perceived meaningful comparison, clarity, and replicability on decision-making in peer review is weak.
While many believe that novelty is always hampered in the publication process (Liang et al., 2023; Wang et al., 2017), our study empirically uncovers that reviewers place a substantial emphasis on their perception of originality and are more inclined to accept papers they deem novel. This highlights that originality is the most valued and recognized element by reviewers. However, the transition from perception to decision is complex and still largely unexplored. Specifically, reviewers’ perspectives can be influenced by preconceptions or past experiences, and there is often a gap between perception and its expression. Additionally, cognitive limitations may obscure underlying social biases. The intricate mechanisms and overall effectiveness of this system demand ongoing exploration and refinement. Within this context, reviewers’ cognition, perception, and decision-making are indispensable and critical components. While the peer review system benefits science, it also necessitates a continuous cycle of feedback to evolve and improve (Ward et al., 2015).
Limitation and future work
Due to challenges in accessing review content, our data has some constraints. We have also lightly addressed chairs’ perceptions, which we will explore in more depth. The requests from reviewers also contain important information, and there is potential for further analysis.
Acknowledgements
This work is supported by the National Natural Science Foundation of China (72104212).
ReferencesAlbertsB.HansonB.KelnerK. L.2008Reviewing Peer ReviewScience32158851515https://doi.org/10.1126/science.1162115BaillyB. A. F. L.2016Learning from peer reviewNature Nanotechnology112204https://doi.org/10.1038/nnano.2016.4BaliettiS.GoldstoneR. L.HelbingD.2016Peer review and competition in the Art Exhibition GameProceedings of the National Academy of Sciences1133084148419BendaW. G.EngelsT. C.2011The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in scienceInternational Journal of Forecasting271166182BlockeelC.DrakopoulosP.PolyzosN.TournayeH.García-VelascoJ.2017Review the ‘peer review’Reproductive Biomedicine Online356747749https://doi.org/10.1016/j.rbmo.2017.08.017BroschT.SchererK.GrandjeanD.SanderD.2013The impact of emotion on perception, attention, memory, and decision-makingSwiss Medical Weekly1431920w13786w13786ChongS. W.MasonS.2021Demystifying the process of scholarly peer-review: An autoethnographic investigation of feedback literacy of two award-winning peer reviewersHumanities and Social Sciences Communications81111CurtinP. A.RussialJ.TefertillerA. C.2018Reviewers’ Perceptions of the Peer Review Process in Journalism and Mass CommunicationJournalism & Mass Communication Quarterly95278299https://doi.org/10.1177/1077699017736031CutlerS.XiaY.BeddoesK.2022Peering into the Black Box of Peer Review: From the Perspective of Editors9th Research in Engineering Education Symposium (REES 2021) and 32nd Australasian Association for Engineering Education Conference (REES AAEE 2021)https://doi.org/10.52202/066488-0095DrorI. E.Fraser-MackenzieP. A.2008Cognitive biases in human perception, judgment, and decision making: Bridging theory and the real worldIn Criminal investigative failures7994RoutledgeGarcíaJ. A.Rodriguez-SánchezR.Fdez-ValdiviaJ.2015Bias and effort in peer reviewJournal of the Association for Information Science and Technology6610Article 10HamiltonD.FraserH.HoekstraR.FidlerF.2020Journal policies and editors’ opinions on peer revieweLife9https://doi.org/10.7554/eLife.62529HowardL.WilkinsonG.1998Peer review and editorial decision-makingThe British Journal of Psychiatry1732110113HuaX.NikolovM.BaduguN.WangL.2019Argument Mining for Understanding Peer ReviewsProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)21312137JeffersonT.WagerE.DavidoffF.2002Measuring the quality of editorial peer reviewJama2872127862790LamontM.2009How professors think: Inside the curious world of academic judgmentHarvard University PressLeeC. J.SugimotoC. R.ZhangG.CroninB.2013Bias in peer reviewJournal of the American Society for Information Science and Technology641217LiangZ.MaoJ.LiG.2023Bias against scientific novelty: A prepublication perspectiveJournal of the Association for Information Science and Technology74199114LindsayP. H.NormanD. A.2013Human information processing: An introduction to psychologyAcademic pressLiuY.2019Roberta: A robustly optimized bert pretraining approacharXiv Preprint arXiv:1907.11692MaddiA.MiottiL.2024On the peer review reports: Does size matter?Scientometrics121ManchikantiL.KayeA.BoswellM.HirschJ.2015Medical journal peer review: Process and biasPain Physician181E1E14MariettaM.BarkerD.2019The Psychology of Fact Perceptions IIOne Nation, Two Realitieshttps://doi.org/10.1093/OSO/9780190677176.003.0006MarshH. W.BondN. W.JayasingheU. W.2007Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalidAustralian Psychologist4213338MatsuiA.ChenE.WangY.FerraraE.2021The impact of peer review on the contribution potential of scientific papersPeerJ9e11999MavrogenisA. F.QuaileA.ScarlatM. M.2020The good, the bad and the rude peer-reviewIn International Orthopaedics44413415SpringerNobaranyS.BoothK. S.2015Use of politeness strategies in signed open peer reviewJournal of the Association for Information Science and Technology66510481064ParkI.-U.PeaceyM. W.MunafoM. R.2014Modelling the effects of subjective and objective decision making in scientific peer reviewNature50674869396PetersD. P.CeciS. J.1982Peer-review research: Objections and obligationsBehavioral and Brain Sciences52246255PhillipsJ.2011Expert bias in peer reviewCurrent Medical Research and Opinion2722292233https://doi.org/10.1185/03007995.2011.624090RileyB.JonesR. B.2016Peer review: Acknowledging its value and recognising the reviewersThe British Journal of General Practice: The Journal of the Royal College of General Practitioners66653629630https://doi.org/10.3399/BJGP16X688285RoedigerH. L.1991Is unreliability in peer review harmful?Behavioral and Brain Sciences141159160SoergelD.SaundersA.McCallumA.2013Open Scholarship and Peer Review: A Time for ExperimentationoStreetC. T.WardK.2019Cognitive Bias in the Peer Review Process: Understanding a Source of Friction between Reviewers and ResearchersData Base505270https://doi.org/10.1145/3371041.3371046TeplitskiyM.PengH.BlascoA.LakhaniK. R.2022Is novel research worth doing? Evidence from peer review at 49 journalsProceedings of the National Academy of Sciences11947e2118046119TomkinsA.ZhangM.HeavlinW. D.2017Reviewer bias in single-versus double-blind peer reviewProceedings of the National Academy of Sciences114481270812713TravisG. D. L.CollinsH. M.1991New light on old boys: Cognitive and institutional particularism in the peer review systemScience, Technology, & Human Values163322341TvinaA.SpellecyR.PalatnikA.2019Bias in the Peer Review Process: Can We Do Better?Obstetrics & Gynecologyhttps://doi.org/10.1097/A0G.0000000000003260WadeA. D.2022The semantic scholar academic graph (s2ag)Companion Proceedings of the Web Conference 2022739739WangJ.VeugelersR.StephanP.2017Bias against novelty in science: A cautionary tale for users of bibliometric indicatorsResearch Policy46814161436WardP.GraberK. C.Van Der MarsH.2015Writing quality peer reviews of research manuscriptsJournal of Teaching in Physical Education344700715WennerasC.WoldA.2001Nepotism and sexism in peer-reviewWolframD.WangP.HembreeA.ParkH.2020Open peer review: Promoting transparency in open scienceScientometrics125210331051YuanW.LiuP.NeubigG.2022Can we automate scientific reviewing?Journal of Artificial Intelligence Research75171212AppendixSupplementary materials