CAJ | 학술논문

随着网络时代的快速发展，用户对搜索引擎、网页的内容和大数据处理等有了更多的要求。从海量的互联网信息中选取最符合要求的信息成为了新的热点。基于一个开源的、Java开发的、可扩展的Web爬虫项目—Heritrix，进行扩展抓取用户需要的网页，深入研究了信息采集技术。利用Heritrix的可扩展性，来实现用户的抓取。通过分析Heritrix的工作流程，模块划分以及源码设计，基于Heritrix扩展抽取面向商品信息的网页，配合HtmlParser对网页内容进行解析，有效的提取商品关键信息后存入数据库以供检索。
수착망락시대적쾌속발전，용호대수색인경、망혈적내용화대수거처리등유료경다적요구。종해량적호련망신식중선취최부합요구적신식성위료신적열점。기우일개개원적、Java개발적、가확전적Web파충항목—Heritrix，진행확전조취용호수요적망혈，심입연구료신식채집기술。이용Heritrix적가확전성，래실현용호적조취。통과분석Heritrix적공작류정，모괴화분이급원마설계，기우Heritrix확전추취면향상품신식적망혈，배합HtmlParser대망혈내용진행해석，유효적제취상품관건신식후존입수거고이공검색。
With the rapid development of the internet age, users have put forward more requirements for search en-gines,content of webpage and large data processing etc. Selecting the required information from the internet information with mass data has become a new hotspot. In this paper, extensible webcrawler project- Heritrix, which is an open source and developed by Java, is extended to capture user webpage. The information collection technology is further studied. Extendibility of Heritrix is used to realize a user’s capture. Through the analysis of the working process of Heritrix, module allocation and source code design, based on webpage extraction facing product information with Heri-trix extendibility and webpage content analysis with HtmlParser, key product information is extracted effectively, which is stored in the database for retrieval.