How clumpy is my image?
Scoring in crowdsourced annotation tasks
| LEADER | caa a22 4500 | ||
|---|---|---|---|
| 001 | 605468648 | ||
| 003 | CHVBK | ||
| 005 | 20210128100317.0 | ||
| 007 | cr unu---uuuuu | ||
| 008 | 210128e20150601xx s 000 0 eng | ||
| 024 | 7 | 0 | |a 10.1007/s00500-014-1303-z |2 doi |
| 035 | |a (NATIONALLICENCE)springer-10.1007/s00500-014-1303-z | ||
| 245 | 0 | 0 | |a How clumpy is my image? |h [Elektronische Daten] |b Scoring in crowdsourced annotation tasks |c [Hugo Hutt, Richard Everson, Murray Grant, John Love, George Littlejohn] |
| 520 | 3 | |a The use of citizen science to obtain annotations from multiple annotators has been shown to be an effective method for annotating datasets in which computational methods alone are not feasible. The way in which the annotations are obtained is an important consideration which affects the quality of the resulting consensus annotation. In this paper, we examine three separate approaches to obtaining consensus scores for instances rather than merely binary classifications. To obtain a consensus score, annotators were asked to make annotations in one of three paradigms: classification, scoring and ranking. A web-based citizen science experiment is described which implements the three approaches as crowdsourced annotation tasks. The tasks are evaluated in relation to the accuracy and agreement among the participants using both simulated and real-world data from the experiment. The results show a clear difference in performance between the three tasks, with the ranking task obtaining the highest accuracy and agreement among the participants. We show how a simple evolutionary optimiser may be used to improve the performance by reweighting the importance of annotators. | |
| 540 | |a Springer-Verlag Berlin Heidelberg, 2014 | ||
| 690 | 7 | |a Web-based citizen science |2 nationallicence | |
| 690 | 7 | |a Classification |2 nationallicence | |
| 690 | 7 | |a Consensus score |2 nationallicence | |
| 690 | 7 | |a Crowdsourced annotation tasks |2 nationallicence | |
| 690 | 7 | |a Evolutionary optimiser |2 nationallicence | |
| 690 | 7 | |a Image clump |2 nationallicence | |
| 690 | 7 | |a Ranking |2 nationallicence | |
| 690 | 7 | |a Scoring |2 nationallicence | |
| 690 | 7 | |a Internet |2 nationallicence | |
| 690 | 7 | |a Evolutionary computation |2 nationallicence | |
| 690 | 7 | |a Image classification |2 nationallicence | |
| 690 | 7 | |a Pattern clustering |2 nationallicence | |
| 690 | 7 | |a Microscopy |2 nationallicence | |
| 690 | 7 | |a Correlation |2 nationallicence | |
| 700 | 1 | |a Hutt |D Hugo |u Computer Science, The University of Exeter, Exeter, UK |4 aut | |
| 700 | 1 | |a Everson |D Richard |u Computer Science, The University of Exeter, Exeter, UK |4 aut | |
| 700 | 1 | |a Grant |D Murray |u Biosciences, The University of Exeter, Exeter, UK |4 aut | |
| 700 | 1 | |a Love |D John |u Biosciences, The University of Exeter, Exeter, UK |4 aut | |
| 700 | 1 | |a Littlejohn |D George |u Biosciences, The University of Exeter, Exeter, UK |4 aut | |
| 773 | 0 | |t Soft Computing |d Springer Berlin Heidelberg |g 19/6(2015-06-01), 1541-1552 |x 1432-7643 |q 19:6<1541 |1 2015 |2 19 |o 500 | |
| 856 | 4 | 0 | |u https://doi.org/10.1007/s00500-014-1303-z |q text/html |z Onlinezugriff via DOI |
| 898 | |a BK010053 |b XK010053 |c XK010000 | ||
| 900 | 7 | |a Metadata rights reserved |b Springer special CC-BY-NC licence |2 nationallicence | |
| 908 | |D 1 |a research-article |2 jats | ||
| 949 | |B NATIONALLICENCE |F NATIONALLICENCE |b NL-springer | ||
| 950 | |B NATIONALLICENCE |P 856 |E 40 |u https://doi.org/10.1007/s00500-014-1303-z |q text/html |z Onlinezugriff via DOI | ||
| 950 | |B NATIONALLICENCE |P 700 |E 1- |a Hutt |D Hugo |u Computer Science, The University of Exeter, Exeter, UK |4 aut | ||
| 950 | |B NATIONALLICENCE |P 700 |E 1- |a Everson |D Richard |u Computer Science, The University of Exeter, Exeter, UK |4 aut | ||
| 950 | |B NATIONALLICENCE |P 700 |E 1- |a Grant |D Murray |u Biosciences, The University of Exeter, Exeter, UK |4 aut | ||
| 950 | |B NATIONALLICENCE |P 700 |E 1- |a Love |D John |u Biosciences, The University of Exeter, Exeter, UK |4 aut | ||
| 950 | |B NATIONALLICENCE |P 700 |E 1- |a Littlejohn |D George |u Biosciences, The University of Exeter, Exeter, UK |4 aut | ||
| 950 | |B NATIONALLICENCE |P 773 |E 0- |t Soft Computing |d Springer Berlin Heidelberg |g 19/6(2015-06-01), 1541-1552 |x 1432-7643 |q 19:6<1541 |1 2015 |2 19 |o 500 | ||