caa a22 4500 605478244 CHVBK 20210128100404.0 cr unu---uuuuu 210128e20150901xx s 000 0 eng 10.1007/s10994-015-5486-z doi (NATIONALLICENCE)springer-10.1007/s10994-015-5486-z A Bayesian approach for comparing cross-validated algorithms on multiple data sets [Elektronische Daten] [Giorgio Corani, Alessio Benavoli] We present a Bayesian approach for making statistical inference about the accuracy (or any other score) of two competing algorithms which have been assessed via cross-validation on multiple data sets. The approach is constituted by two pieces. The first is a novel correlated Bayesian $$t$$ t test for the analysis of the cross-validation results on a single data set which accounts for the correlation due to the overlapping training sets. The second piece merges the posterior probabilities computed by the Bayesian correlated $$t$$ t test on the different data sets to make inference on multiple data sets. It does so by adopting a Poisson-binomial model. The inferences on multiple data sets account for the different uncertainty of the cross-validation results on the different data sets. It is the first test able to achieve this goal. It is generally more powerful than the signed-rank test if ten runs of cross-validation are performed, as it is anyway generally recommended. The Author(s), 2015 Bayesian hypothesis tests nationallicence Signed-rank test nationallicence Cross-validation nationallicence Poisson-binomial nationallicence Hypothesis test nationallicence Evaluation of classifiers nationallicence Corani Giorgio Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA), Scuola Universitaria Professionale della Svizzera Italiana (SUPSI), Università della Svizzera Italiana (USI), Manno, Switzerland aut Benavoli Alessio Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA), Scuola Universitaria Professionale della Svizzera Italiana (SUPSI), Università della Svizzera Italiana (USI), Manno, Switzerland aut Machine Learning Springer US; http://www.springer-ny.com 100/2-3(2015-09-01), 285-304 0885-6125 100:2-3<285 2015 100 10994 https://doi.org/10.1007/s10994-015-5486-z text/html Onlinezugriff via DOI BK010053 XK010053 XK010000 Metadata rights reserved Springer special CC-BY-NC licence nationallicence 1 research-article jats NATIONALLICENCE NATIONALLICENCE NL-springer NATIONALLICENCE 856 40 https://doi.org/10.1007/s10994-015-5486-z text/html Onlinezugriff via DOI NATIONALLICENCE 700 1- Corani Giorgio Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA), Scuola Universitaria Professionale della Svizzera Italiana (SUPSI), Università della Svizzera Italiana (USI), Manno, Switzerland aut NATIONALLICENCE 700 1- Benavoli Alessio Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA), Scuola Universitaria Professionale della Svizzera Italiana (SUPSI), Università della Svizzera Italiana (USI), Manno, Switzerland aut NATIONALLICENCE 773 0- Machine Learning Springer US; http://www.springer-ny.com 100/2-3(2015-09-01), 285-304 0885-6125 100:2-3<285 2015 100 10994