caa a22 4500 445379685 CHVBK 20180317143013.0 cr unu---uuuuu 170323e20110601xx s 000 0 eng 10.1007/s10032-010-0133-5 doi (NATIONALLICENCE)springer-10.1007/s10032-010-0133-5 Reynaert Martin Tilburg Centre for Cognition and Communication, Tilburg University, Kamer D 342, P.O. Box 90153, 5000 LE, Tilburg, Netherlands aut Character confusion versus focus word-based correction of spelling and OCR variants in corpora [Elektronische Daten] [Martin Reynaert] We present a new approach based on anagram hashing to handle globally the lexical variation in large and noisy text collections. Lexical variation addressed by spelling correction systems is primarily typographical variation. This is typically handled in a local fashion: given one particular text string some system of retrieving near-neighbors is applied, where near-neighbors are other text strings that differ from the particular string by a given number of characters. The difference in characters between the original string and one of its retrieved near-neighbors constitutes a particular character confusion. We present a global way of performing this action: for all possible particular character confusions given a particular edit distance, we sequentially identify all the pairs of text strings in the text collection that display a particular confusion. We work on large digitized corpora, which contain lexical variation due to both the OCR process and typographical or typesetting error and show that all these types of variation can be handled equally well in the framework we present. The character confusion-based prototype of Text-Induced Corpus Clean-up (ticcl) is compared to its focus word-based counterpart and evaluated on 6years' worth of digitized Dutch Parliamentary documents. The character confusion approach is shown to gain an order of magnitude in speed on its word-based counterpart on large corpora. Insights gained about the useful contribution of global corpus variation statistics are shown to also benefit the more traditional word-based approach to spelling correction. Final tests on a held-out set comprising the 1918 edition of the Dutch daily newspaper ‘Het Volk' show that the system is not sensitive to domain variation. The Author(s), 2010 International Journal on Document Analysis and Recognition (IJDAR) Springer-Verlag 14/2(2011-06-01), 173-187 1433-2833 14:2<173 2011 14 10032 https://doi.org/10.1007/s10032-010-0133-5 text/html Onlinezugriff via DOI 1 research-article jats NATIONALLICENCE 856 40 https://doi.org/10.1007/s10032-010-0133-5 text/html Onlinezugriff via DOI NATIONALLICENCE 100 1- Reynaert Martin Tilburg Centre for Cognition and Communication, Tilburg University, Kamer D 342, P.O. Box 90153, 5000 LE, Tilburg, Netherlands aut NATIONALLICENCE 773 0- International Journal on Document Analysis and Recognition (IJDAR) Springer-Verlag 14/2(2011-06-01), 173-187 1433-2833 14:2<173 2011 14 10032 Metadata rights reserved Springer special CC-BY-NC licence nationallicence BK010053 XK010053 XK010000 NATIONALLICENCE NATIONALLICENCE NL-springer