A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants
| LEADER | naa a22 4500 | ||
|---|---|---|---|
| 001 | 528787233 | ||
| 005 | 20180924065503.0 | ||
| 007 | cr unu---uuuuu | ||
| 008 | 180924e20101214xx s 000 0 eng | ||
| 024 | 7 | 0 | |a 10.3929/ethz-b-000026283 |2 doi |
| 024 | 7 | 0 | |a 10.1371/journal.pone.0014331 |2 doi |
| 035 | |a (ETHRESEARCH)oai:www.research-collecti.ethz.ch:20.500.11850/26283 | ||
| 100 | 1 | |a Bornmann |D Lutz | |
| 245 | 1 | 2 | |a A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants |h [Elektronische Daten] |c [Lutz Bornmann, Rüdiger Mutz, Hans-Dieter Daniel] |
| 246 | 0 | |a PLoS ONE | |
| 506 | |a Open access |2 ethresearch | ||
| 520 | 3 | |a Background This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Methodology/Principal Findings Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen's Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed. Conclusions/Significance Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system). | |
| 540 | |a Creative Commons Attribution 3.0 Unported |u http://creativecommons.org/licenses/by/3.0 |2 ethresearch | ||
| 700 | 1 | |a Mutz |D Rüdiger |e joint author | |
| 700 | 1 | |a Daniel |D Hans-Dieter |e joint author | |
| 773 | 0 | |t PLoS ONE |d Lawrence, KS, USA : Public Library of Science |g 5 (12), p. e14331 |x 1932-6203 | |
| 856 | 4 | 0 | |u http://hdl.handle.net/20.500.11850/26283 |q text/html |z WWW-Backlink auf das Repository (Open access) |
| 908 | |D 1 |a Journal Article |2 ethresearch | ||
| 950 | |B ETHRESEARCH |P 856 |E 40 |u http://hdl.handle.net/20.500.11850/26283 |q text/html |z WWW-Backlink auf das Repository (Open access) | ||
| 950 | |B ETHRESEARCH |P 100 |E 1- |a Bornmann |D Lutz | ||
| 950 | |B ETHRESEARCH |P 700 |E 1- |a Mutz |D Rüdiger |e joint author | ||
| 950 | |B ETHRESEARCH |P 700 |E 1- |a Daniel |D Hans-Dieter |e joint author | ||
| 950 | |B ETHRESEARCH |P 773 |E 0- |t PLoS ONE |d Lawrence, KS, USA : Public Library of Science |g 5 (12), p. e14331 |x 1932-6203 | ||
| 898 | |a BK010053 |b XK010053 |c XK010000 | ||
| 949 | |B ETHRESEARCH |F ETHRESEARCH |b ETHRESEARCH |j Journal Article |c Open access | ||