caa a22 4500 605468648 CHVBK 20210128100317.0 cr unu---uuuuu 210128e20150601xx s 000 0 eng 10.1007/s00500-014-1303-z doi (NATIONALLICENCE)springer-10.1007/s00500-014-1303-z How clumpy is my image? [Elektronische Daten] Scoring in crowdsourced annotation tasks [Hugo Hutt, Richard Everson, Murray Grant, John Love, George Littlejohn] The use of citizen science to obtain annotations from multiple annotators has been shown to be an effective method for annotating datasets in which computational methods alone are not feasible. The way in which the annotations are obtained is an important consideration which affects the quality of the resulting consensus annotation. In this paper, we examine three separate approaches to obtaining consensus scores for instances rather than merely binary classifications. To obtain a consensus score, annotators were asked to make annotations in one of three paradigms: classification, scoring and ranking. A web-based citizen science experiment is described which implements the three approaches as crowdsourced annotation tasks. The tasks are evaluated in relation to the accuracy and agreement among the participants using both simulated and real-world data from the experiment. The results show a clear difference in performance between the three tasks, with the ranking task obtaining the highest accuracy and agreement among the participants. We show how a simple evolutionary optimiser may be used to improve the performance by reweighting the importance of annotators. Springer-Verlag Berlin Heidelberg, 2014 Web-based citizen science nationallicence Classification nationallicence Consensus score nationallicence Crowdsourced annotation tasks nationallicence Evolutionary optimiser nationallicence Image clump nationallicence Ranking nationallicence Scoring nationallicence Internet nationallicence Evolutionary computation nationallicence Image classification nationallicence Pattern clustering nationallicence Microscopy nationallicence Correlation nationallicence Hutt Hugo Computer Science, The University of Exeter, Exeter, UK aut Everson Richard Computer Science, The University of Exeter, Exeter, UK aut Grant Murray Biosciences, The University of Exeter, Exeter, UK aut Love John Biosciences, The University of Exeter, Exeter, UK aut Littlejohn George Biosciences, The University of Exeter, Exeter, UK aut Soft Computing Springer Berlin Heidelberg 19/6(2015-06-01), 1541-1552 1432-7643 19:6<1541 2015 19 500 https://doi.org/10.1007/s00500-014-1303-z text/html Onlinezugriff via DOI BK010053 XK010053 XK010000 Metadata rights reserved Springer special CC-BY-NC licence nationallicence 1 research-article jats NATIONALLICENCE NATIONALLICENCE NL-springer NATIONALLICENCE 856 40 https://doi.org/10.1007/s00500-014-1303-z text/html Onlinezugriff via DOI NATIONALLICENCE 700 1- Hutt Hugo Computer Science, The University of Exeter, Exeter, UK aut NATIONALLICENCE 700 1- Everson Richard Computer Science, The University of Exeter, Exeter, UK aut NATIONALLICENCE 700 1- Grant Murray Biosciences, The University of Exeter, Exeter, UK aut NATIONALLICENCE 700 1- Love John Biosciences, The University of Exeter, Exeter, UK aut NATIONALLICENCE 700 1- Littlejohn George Biosciences, The University of Exeter, Exeter, UK aut NATIONALLICENCE 773 0- Soft Computing Springer Berlin Heidelberg 19/6(2015-06-01), 1541-1552 1432-7643 19:6<1541 2015 19 500