CAJ | 학술논문

主题爬虫是垂直搜索引擎的核心组成部分，它为面向主题的用户查询准备数据资源；提出了一种基于HMM的主题爬虫方法，方法不仅分析网页内容，而且还考虑网页的上下文链接结构，首先将当前网页的聚类结果作为观察状态、将当前网页到目标网页的链接距离作为隐含状态，然后通过HMM模型学习用户的主题浏览模式并利用它采集更多的主题网页；实验结果表明：方法能采集大量与指定主题相关的高质量网页，主题爬行效率优于Best-First主题爬虫。
주제파충시수직수색인경적핵심조성부분，타위면향주제적용호사순준비수거자원；제출료일충기우HMM적주제파충방법，방법불부분석망혈내용，이차환고필망혈적상하문련접결구，수선장당전망혈적취류결과작위관찰상태、장당전망혈도목표망혈적련접거리작위은함상태，연후통과HMM모형학습용호적주제류람모식병이용타채집경다적주제망혈；실험결과표명：방법능채집대량여지정주제상관적고질량망혈，주제파행효솔우우Best-First주제파충。
Focused crawler is a core component of the vertical search engine, it collected data resources for the subject-oriented user＇s query. This paper proposes an approach for focused crawler based on HMM, it not only considers the web content, but also analyzes the context of web link structure. Firstly, the observation state represents the clustering of the current web page, the hidden state represents the link distance from current web page to target web page, then through the HMM model learning user browsing patterns, more topic webpages are downloaded by using the model. Experiments show that the focused crawler based on HMM can capture a large number of high quality web pages related to target topics, and its crawling oerforms better than Best-First crawler.