CAJ | 학술논문

随着信息技术的飞速发展，互联网中的网页急剧增长，在这海量、繁杂的网页中却呈现出一定比例的重复网页及近似网页。为了减少农业领域中近似及重复网页对农业垂直搜索引擎性能的影响，文中首先使用MD5算法去除网页集合中完全相同的网页，再利用向量空间模型(VSM)、基于知网的语义相似度模型及潜在语义分析(LSA)三种相似度判断方法对其余网页的相似度进行计算。实验结果显示，当相似度阈值r=60%、维数K=250时，潜在语义分析( LSA)的综合评价F1测度最高，且准确率达到了90.5%。
수착신식기술적비속발전，호련망중적망혈급극증장，재저해량、번잡적망혈중각정현출일정비례적중복망혈급근사망혈。위료감소농업영역중근사급중복망혈대농업수직수색인경성능적영향，문중수선사용MD5산법거제망혈집합중완전상동적망혈，재이용향량공간모형(VSM)、기우지망적어의상사도모형급잠재어의분석(LSA)삼충상사도판단방법대기여망혈적상사도진행계산。실험결과현시，당상사도역치r=60%、유수K=250시，잠재어의분석( LSA)적종합평개F1측도최고，차준학솔체도료90.5%。
With the rapid development of information technology,the Internet Web pages are growing sharply. In this massive,complex pages,preach a certain percentage of duplicate pages and similar pages. In order to reduce the influence of agricultural field approximation and repeated Web pages on agricultural vertical search engine performance,first use the MD5 algorithm to remove the same Web pages in the Web page set,then through three kinds of methods which include the Vector Space Model ( VSM) ,semantic similarity model based on HowNet and Latent Semantic Analysis ( LSA) ,calculate the similarity of the rest Web pages. The experimental results show that when the similarity threshold is 60% (r=60%),the dimension is 250 (K=250),the F1 comprehensive evaluation measure of LSA is highest, and the accuracy rate has reached 90. 5%.