CAJ | 학술논문

针对当前微博采集无精确去噪方法和微博无法无登陆采集现象，笔者提出了基于 Regex 网页去噪 Hash 对比的网络爬虫采集方案并利用插件采集实现了无登陆采集。该方法通过 Regex 构建 DFA 和 NFA 模型来去除网页噪声，通过 Hash 对比对确定采集页面，并通过插件权限提升实现无登陆技术。有效的避免了 Hash 值的变化与网页内容变化产生偏离的现象，解决了网络爬虫虚拟登录时多次对 URL 采集造成的身份认证问题。实验表明，该方法可以实时快速的获取微博信息，为舆情数据分析提供批量精准的数据。
침대당전미박채집무정학거조방법화미박무법무등륙채집현상，필자제출료기우 Regex 망혈거조 Hash 대비적망락파충채집방안병이용삽건채집실현료무등륙채집。해방법통과 Regex 구건 DFA 화 NFA 모형래거제망혈조성，통과 Hash 대비대학정채집혈면，병통과삽건권한제승실현무등륙기술。유효적피면료 Hash 치적변화여망혈내용변화산생편리적현상，해결료망락파충허의등록시다차대 URL 채집조성적신빈인증문제。실험표명，해방법가이실시쾌속적획취미박신식，위여정수거분석제공비량정준적수거。
In view of the current micro - blog acquisition without accurate denoising method and unable abundantly the non - debarkation gathering phenomenon,we present a web crawler acquisition scheme of Regex Webpage denoising Hash based on comparison and realize no landing collection by using plug - in acquisition. The method of Regex to construct DFA and NFA model to remove Webpage noise,comparing the Hash to determine the collection page,and the plug - in privilege without landing techniques are presented. Experiments show that,this method quickly gets micro - blog information in real time,and provides,accurate data for the mass public opinion data analysis.