CAJ | 학술논문

分布式网络爬虫的广泛应用使得搜索引擎的数据规模呈几何式增长，面对数以 TB 甚至 PB 量级的数据，单机模式下的PageRank 算法由于 CPU、I/O 和内存的开销过大导致效率低下。为此，提出一种基于 MapReduce 框架的并行 PageRank 算法。在算法的一次迭代过程中，利用 Map 函数对网页拓扑信息文件进行解析，使用 Reduce 函数计算网页得分，从而并行化 PageRank 算法的中间迭代过程。通过计算全局网页得分控制迭代次数，得到较精确的网页排序结果。实验结果表明，该算法在保持原有单机PageRank 算法整体网页排序精度的基础上，具有较好的集群性能和较快的执行速度。
분포식망락파충적엄범응용사득수색인경적수거규모정궤하식증장，면대수이 TB 심지 PB 량급적수거，단궤모식하적PageRank 산법유우 CPU、I/O 화내존적개소과대도치효솔저하。위차，제출일충기우 MapReduce 광가적병행 PageRank 산법。재산법적일차질대과정중，이용 Map 함수대망혈탁복신식문건진행해석，사용 Reduce 함수계산망혈득분，종이병행화 PageRank 산법적중간질대과정。통과계산전국망혈득분공제질대차수，득도교정학적망혈배서결과。실험결과표명，해산법재보지원유단궤PageRank 산법정체망혈배서정도적기출상，구유교호적집군성능화교쾌적집행속도。
The emergence of distributed Web crawl largely expands the scale of related Web information. Since PageRank needs to process the topology of entire existed page set, the limitation of CPU, I/O and memory becomes the big issue when it confronts the data in TB or PB level. Aiming at these problems, this paper proposes a parallel PageRank algorithm based on MapReduce. In a certain iteration of algorithm, it processes the files containing the topology of Web page graph by Map function and calculates the pages’ scores by Reduce function. Using the global Web page score as convergence to control iterations and get more precise Web page sorting result. Experimental result shows that the improved algorithm has better clustering performance and faster execution speed on the basis of keeping the overall Web page sorting accuracy of single machine PageRank algorithm.