CAJ | 학술논문

时空复杂度较高以及物理机器内存不足，会导致传统聚类算法不能有效地分析处理大规模数据网络。针对该问题，在MapReduce分布式模型的基础上，提出一种网络数据分布式聚类算法。根据MRC理论设计有限MapReduce轮数，控制混洗过程所需时间，利用Map内合并技术对网络流量进行控制，在进行中间结果合并时仅对社团合并，而不考虑社团内部节点，以控制内存开销。使用模拟生成的数据在集群中进行实验，结果表明，当数据规模和集群规模增大时，该算法具有较好的加速比和扩展性。
시공복잡도교고이급물리궤기내존불족，회도치전통취류산법불능유효지분석처리대규모수거망락。침대해문제，재MapReduce분포식모형적기출상，제출일충망락수거분포식취류산법。근거MRC이론설계유한MapReduce륜수，공제혼세과정소수시간，이용Map내합병기술대망락류량진행공제，재진행중간결과합병시부대사단합병，이불고필사단내부절점，이공제내존개소。사용모의생성적수거재집군중진행실험，결과표명，당수거규모화집군규모증대시，해산법구유교호적가속비화확전성。
Due to the high time and space complexity and physical machines out of memory, traditional clustering algorithms usually can not effectively analyze and deal with large data network. To solve this problem, this paper proposes a distributed clustering algorithm for network data based on MapReduce model. It adopts the theory of MRC theory to design limited round number of MapReduce to control the time in shuffle stage, and utilizes the Map inner merging technology to control network flow. It proposes an idea that if merge the intermediate results, only merge clusters and do not consider the internal nodes, which can control memory overhead. It utilizes the data sets generated by simulation to do experiment. Experimental results show that when the data size and cluster scale increases, the CAMR algorithm has good speedup ratio and scalability.