Paul Ralph /Evidence Standards Improve the Reliability of Scholarly Peer Review - CANCELED
Abstract
Background. Scholarly peer review is “the lynchpin about which the whole business of science is pivoted” (Ziman 1968). Most researchers believe peer review is effective (Ware 2008), but empirical research consistently shows that reviewers cannot reliably distinguish methodologically sound from fundamentally flawed studies (Cole 1981; Peters and Ceci 1982; Lock 1991; Rothwell and Martyn 2000; Price 2014; Ralph 2016). Consequently, Ralph et al. (2020) created comprehensive evidence standards and tools to improve peer review in software engineering and related fields.
Objective. The objective of this study is to investigate the impact of evidence standards on scholarly peer review.
Method. A randomized controlled experiment was conducted at an A-ranked software engineering conference. The program committee was randomly divided into two groups: one using a typical conference review process; the other using a standardized process based on the ACM SIGSOFT Empirical Standards for Software Engineering Research.
Results. Evidence standards significantly improve inter-reviewer reliability without harming authors’ or reviewers’ attitudes toward the review process.
Discussion. Asking reviewers to write qualitative comments about a paper and score it on a 6-point scale from strong reject to strong accept produces data statistically indistinguishable from random noise. This means that decisions are determined entirely by reviewer selection, not the merits of the research. Conventional review processes are therefore scientifically and morally indefensible. While evidence standards are not a silver bullet, standards-based review significantly improves reliability, and the data collected in this study facilitates further refinement of the standards and tooling toward still greater reliability. Adjacent research communities such as management information systems, management science, human-computer interaction and CS education should investigate adapting evidence standards to improve their peer review processes.
About the Speaker
Paul Ralph, PhD (British Columbia), is an award-winning scientist, author, consultant, and Professor of Software Engineering at Dalhousie University. Dr. Ralph’s research intersects software engineering, sustainable development, human-computer interaction, and project management. He has published more than 80 peer-reviewed articles in premier venues including IEEE Transactions on Software Engineering and the ACM/IEEE International Conference on Software Engineering. Dr. Ralph is editor-in-chief of the SIGSOFT Empirical Standards for Software Engineering Research.