An effective and efficient Web content extractor for optimizing the crawling process
Version of Record online: 27 MAR 2013
Copyright © 2013 John Wiley & Sons, Ltd.
Software: Practice and Experience
Volume 44, Issue 10, pages 1181–1199, October 2014
How to Cite
2014), An effective and efficient Web content extractor for optimizing the crawling process, Softw. Pract. Exper., 44, pages 1181–1199, doi: 10.1002/spe.2195, , , and (
- Issue online: 6 SEP 2014
- Version of Record online: 27 MAR 2013
- Manuscript Accepted: 1 MAR 2013
- Manuscript Revised: 23 FEB 2013
- Manuscript Received: 6 AUG 2012
- Web content extraction;
- Web crawling;
- intelligent systems
Classical Web crawlers make use of only hyperlink information in the crawling process. However, focused crawlers are intended to download only Web pages that are relevant to a given topic by utilizing word information before downloading the Web page. But, Web pages contain additional information that can be useful for the crawling process. We have developed a crawler, iCrawler (intelligent crawler), the backbone of which is a Web content extractor that automatically pulls content out of seven different blocks: menus, links, main texts, headlines, summaries, additional necessaries, and unnecessary texts from Web pages. The extraction process consists of two steps, which invoke each other to obtain information from the blocks. The first step learns which HTML tags refer to which blocks using the decision tree learning algorithm. Being guided by numerous sources of information, the crawler becomes considerably effective. It achieved a relatively high accuracy of 96.37% in our experiments of block extraction. In the second step, the crawler extracts content from the blocks using string matching functions. These functions along with the mapping between tags and blocks learned in the first step provide iCrawler with considerable time and storage efficiency. More specifically, iCrawler performs 14 times faster in the second step than in the first step. Furthermore, iCrawler significantly decreases storage costs by 57.10% when compared with the texts obtained through classical HTML stripping. Copyright © 2013 John Wiley & Sons, Ltd.