CAJ | 학술논문

针对蒙古文主题爬虫主要面临的预测采集URL和发现隧道2个核心问题，提出一种基于主题团的站点聚类、排序和隧道发现的采集模型。通过站点的主题识别，将待爬行URL分为站点链接和非站点链接，使用文本相似度和超链图分析建立了预测URL优先级排序算法，基于站点粒度设计了站点自适应隧道发现算法，最后，构建了一个面向蒙古文主题的网络爬虫系统。实验结果表明：该算法在采准率、信息总量与采集速率上都得到了提高，明显优于基线算法。
침대몽고문주제파충주요면림적예측채집URL화발현수도2개핵심문제，제출일충기우주제단적참점취류、배서화수도발현적채집모형。통과참점적주제식별，장대파행URL분위참점련접화비참점련접，사용문본상사도화초련도분석건립료예측URL우선급배서산법，기우참점립도설계료참점자괄응수도발현산법，최후，구건료일개면향몽고문주제적망락파충계통。실험결과표명：해산법재채준솔、신식총량여채집속솔상도득도료제고，명현우우기선산법。
Forecast of collecting URL and tunnel discovery are two core issues in Focused crawler for Mongolian website. Therefore, a collecting model was proposed based on topic group of site clustering, ordering and tunnel discovery. First, through the topic identification text, to be crawling URL was divided into the site links and non site links. Second, a URL priority ordering algorithm was established by using the text similarity and the hyperlink graph analysis, and an adaptive tunnel discovery algorithm based on website was designed. Finally, the system of focused crawler for the Mongolian website was constructed. The experimental results show that the accurate rate of collecting, the amount of information and the collection rate have been improved significantly compared than the baseline algorithm.