CAJ | 학술논문

社交网络数据信息量大、主题性强，具有巨大的数据挖掘价值，是互联网大数据的重要组成部分。针对传统搜索引擎无法利用关键字检索技术直接索引社交网络平台信息的现状，基于众包模式，采用C/S架构，设计社交网络数据采集模型，包含服务端、客户端、存储系统与主题Deep Web爬虫系统4个模块。通过主题Deep Web爬虫的分布式机器节点自动向服务器请求爬虫任务并上传爬取数据，利用Hadoop分布式文件系统对爬取数据进行快速处理并存储结果数据。实验结果表明，主题Deep Web爬虫系统配置简单，支持功能扩展和目标信息直接获取，数据采集模型具有较快的数据获取速度及较高的信息检索效率。
사교망락수거신식량대、주제성강，구유거대적수거알굴개치，시호련망대수거적중요조성부분。침대전통수색인경무법이용관건자검색기술직접색인사교망락평태신식적현상，기우음포모식，채용C/S가구，설계사교망락수거채집모형，포함복무단、객호단、존저계통여주제Deep Web파충계통4개모괴。통과주제Deep Web파충적분포식궤기절점자동향복무기청구파충임무병상전파취수거，이용Hadoop분포식문건계통대파취수거진행쾌속처리병존저결과수거。실험결과표명，주제Deep Web파충계통배치간단，지지공능확전화목표신식직접획취，수거채집모형구유교쾌적수거획취속도급교고적신식검색효솔。
Social network data has the features of informative and strong topicality with significant value for data mining,and it is also a very important part of the Internet big data. However,traditional search engines can not use the keywords retrieve technology to index the information of social network platform directly,and under such circumstances, this paper designs and implements a data collection model based on crowdsourcing mode and C/S architecture. The model consists of four modules including server,client,storage sub-system and a Deep Web crawler system. The nodes run the topic Deep Web crawler system to request new tasks automatically and upload the acquired data,meanwhile the system uses the Hadoop Distributed File System( HDFS) to process data rapidly and store results. The topic Deep Web crawler system has the features of easy configuration,flexible scalability and direct data collection,and it also proves that data collection model is able to fulfill the tasks in a high success rate and collect data in an efficient way.