naa a22 4500 528787233 20180924065503.0 cr unu---uuuuu 180924e20101214xx s 000 0 eng 10.3929/ethz-b-000026283 doi 10.1371/journal.pone.0014331 doi (ETHRESEARCH)oai:www.research-collection.ethz.ch:20.500.11850/26283 Bornmann Lutz A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants [Elektronische Daten] [Lutz Bornmann, Rüdiger Mutz, Hans-Dieter Daniel] PLoS ONE Open access ethresearch Background This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Methodology/Principal Findings Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen's Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed. Conclusions/Significance Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system). Creative Commons Attribution 3.0 Unported http://creativecommons.org/licenses/by/3.0 ethresearch Mutz Rüdiger joint author Daniel Hans-Dieter joint author PLoS ONE Lawrence, KS, USA : Public Library of Science 5 (12), p. e14331 1932-6203 http://hdl.handle.net/20.500.11850/26283 text/html WWW-Backlink auf das Repository (Open access) 1 Journal Article ethresearch ETHRESEARCH 856 40 http://hdl.handle.net/20.500.11850/26283 text/html WWW-Backlink auf das Repository (Open access) ETHRESEARCH 100 1- Bornmann Lutz ETHRESEARCH 700 1- Mutz Rüdiger joint author ETHRESEARCH 700 1- Daniel Hans-Dieter joint author ETHRESEARCH 773 0- PLoS ONE Lawrence, KS, USA : Public Library of Science 5 (12), p. e14331 1932-6203 BK010053 XK010053 XK010000 ETHRESEARCH ETHRESEARCH ETHRESEARCH Journal Article Open access