Demystifying perception and decision-making in peer review: a semantic perspective

Abstract

Introduction. Peer review is fundamental to scientific progress. However, it remains a ‘black box’ due to the ambiguous and subjective decision-making rationale, hidden within reviewers’ perceptions and judgments.

Research gaps. The scarcity of peer review data and the limited analytical scope have hindered empirical studies from thoroughly exploring these underlying factors.

Method. We collected 114,006 review records from OpenReview. Developing deep learning models, we extracted reviewers’ perceptions of key factors from the review content.

Analysis. Our analysis focuses on how these perceptual factors influence reviewers’ decisions and outcomes of the papers.

Results. Originality is the element that reviewers value and recognize the most, followed by motivation. In contrast, chairs prioritize the overall validity of the work within the field, giving more weight to the substance and soundness of papers. The role of motivation is generally supportive but has limited distinguishing power on final decisions. Perceived clarity, replicability, and meaningful comparison have a relatively minor role in the review process.

Conclusions. Our study highlights the decision-making process based on reviewers’ perceptions and uncovers perceptual differences between reviewers and chairs. This contributes to greater transparency in the peer review and helps build trust within the scientific community.