caa a22 4500 445871520 CHVBK 20180317145513.0 cr unu---uuuuu 170323e20110101xx s 000 0 eng 10.1007/s10618-010-0180-z doi (NATIONALLICENCE)springer-10.1007/s10618-010-0180-z Web robot detection techniques: overview and limitations [Elektronische Daten] [Derek Doran, Swapna Gokhale] Most modern Web robots that crawl the Internet to support value-added services and technologies possess sophisticated data collection and analysis capabilities. Some of these robots, however, may be ill-behaved or malicious, and hence, may impose a significant strain on a Web server. It is thus necessary to detect Web robots in order to block undesirable ones from accessing the server. Such detection is also essential to ensure that the robot traffic is considered appropriately in the performance and capacity planning of Web servers. Despite a variety of Web robot detection techniques, there is no consensus regarding a single technique, or even a specific "type” of technique, that performs well in practice. Therefore, to aid in the development of a practically applicable robot detection technique, this survey presents a critical analysis and comparison of the prevalent detection approaches. We propose a framework to classify the existing detection techniques into four categories based on their underlying detection philosophy. We compare the different classes to gain insights into those characteristics that make up an effective robot detection scheme. Finally, we discuss why the contemporary techniques fail to offer a general solution to the robot detection problem and propose a set of key ingredients necessary for strong Web robot detection. The Author(s), 2010 Web Crawler nationallicence Web Robot nationallicence WWW nationallicence Web Robot Detection nationallicence Web User Classification nationallicence Doran Derek Department of Computer Science and Engineering, University of Connecticut, 06269, Storrs, CT, USA aut Gokhale Swapna Department of Computer Science and Engineering, University of Connecticut, 06269, Storrs, CT, USA aut Data Mining and Knowledge Discovery Springer US; http://www.springer-ny.com 22/1-2(2011-01-01), 183-210 1384-5810 22:1-2<183 2011 22 10618 https://doi.org/10.1007/s10618-010-0180-z text/html Onlinezugriff via DOI 1 research-article jats NATIONALLICENCE 856 40 https://doi.org/10.1007/s10618-010-0180-z text/html Onlinezugriff via DOI NATIONALLICENCE 700 1- Doran Derek Department of Computer Science and Engineering, University of Connecticut, 06269, Storrs, CT, USA aut NATIONALLICENCE 700 1- Gokhale Swapna Department of Computer Science and Engineering, University of Connecticut, 06269, Storrs, CT, USA aut NATIONALLICENCE 773 0- Data Mining and Knowledge Discovery Springer US; http://www.springer-ny.com 22/1-2(2011-01-01), 183-210 1384-5810 22:1-2<183 2011 22 10618 Metadata rights reserved Springer special CC-BY-NC licence nationallicence BK010053 XK010053 XK010000 NATIONALLICENCE NATIONALLICENCE NL-springer