naa a22 4500 528784412 20180924065517.0 cr unu---uuuuu 180924e20121031xx s 000 0 eng 10.3929/ethz-b-000059384 doi 10.1371/journal.pone.0048509 doi (ETHRESEARCH)oai:www.research-collection.ethz.ch:20.500.11850/59384 Mutz Rüdiger Heterogeneity of Inter-Rater Reliabilities of Grant Peer Reviews and Its Determinants: A General Estimating Equations Approach [Elektronische Daten] [Rüdiger Mutz, Lutz Bornmann, Hans-Dieter Daniel] PLoS ONE Open access ethresearch Background One of the most important weaknesses of the peer review process is that different reviewers' ratings of the same grant proposal typically differ. Studies on the inter-rater reliability of peer reviews mostly report only average values across all submitted proposals. But inter-rater reliabilities can vary depending on the scientific discipline or the requested grant sum, for instance. Goal Taking the Austrian Science Fund (FWF) as an example, we aimed to investigate empirically the heterogeneity of inter-rater reliabilities (intraclass correlation) and its determinants. Methods The data consisted of N = 8,329 proposals with N = 23,414 overall ratings by reviewers, which were statistically analyzed using the generalized estimating equations approach (GEE). Results We found an overall intraclass correlation (ICC) of reviewer? ratings of ρ = .259 with a 95% confidence interval of [.249,.279]. In humanities the ICCs were statistically significantly higher than in all other research areas except technical sciences. The ICC in biosciences deviated statistically significantly from the average ICC. Other factors (besides the research areas), such as the grant sum requested, had negligible influence on the ICC. Conclusions Especially in biosciences, the number of reviewers of each proposal should be increased so as to increase the ICC. Creative Commons Attribution 3.0 Unported http://creativecommons.org/licenses/by/3.0 ethresearch Bornmann Lutz joint author Daniel Hans-Dieter joint author PLoS ONE Lawrence, KS, USA : Public Library of Science 7 (10), p. e48509 1932-6203 http://hdl.handle.net/20.500.11850/59384 text/html WWW-Backlink auf das Repository (Open access) 1 Journal Article ethresearch ETHRESEARCH 856 40 http://hdl.handle.net/20.500.11850/59384 text/html WWW-Backlink auf das Repository (Open access) ETHRESEARCH 100 1- Mutz Rüdiger ETHRESEARCH 700 1- Bornmann Lutz joint author ETHRESEARCH 700 1- Daniel Hans-Dieter joint author ETHRESEARCH 773 0- PLoS ONE Lawrence, KS, USA : Public Library of Science 7 (10), p. e48509 1932-6203 BK010053 XK010053 XK010000 ETHRESEARCH ETHRESEARCH ETHRESEARCH Journal Article Open access