CAJ | 학술논문

随着大规模数据的快速增长及高可靠性需求，将本地数据迁移到分布式数据库势在必行。针对这种情况，提出一种基于MapReduce的“快速并行导入”技术，充分利用集群的并行计算能力，直接向HBase底层存储文件HFile写入数据，既可避免上层数据导入时间的浪费，又节省资源开销。有效解决了从单机数据库向HBase分布式数据库导入数据功能低下、效率不高等问题。实验结果表明，在“快速并行导入”技术的基础上设计并实现的快速并行导入工具，支持多列族文本数据的快速导入。与传统使用API导入数据相比，速度提升一倍以上。
수착대규모수거적쾌속증장급고가고성수구，장본지수거천이도분포식수거고세재필행。침대저충정황，제출일충기우MapReduce적“쾌속병행도입”기술，충분이용집군적병행계산능력，직접향HBase저층존저문건HFile사입수거，기가피면상층수거도입시간적낭비，우절성자원개소。유효해결료종단궤수거고향HBase분포식수거고도입수거공능저하、효솔불고등문제。실험결과표명，재“쾌속병행도입”기술적기출상설계병실현적쾌속병행도입공구，지지다렬족문본수거적쾌속도입。여전통사용API도입수거상비，속도제승일배이상。
With the rapid growth of very-large data and its high reliability requirement,it is inevitable to transplant local data to distributed database.In light of this case,the paper presents a MapReduce-based “fast parallel importing”technology.It makes full use of parallel computational capability of the cluster to write data directly to underlying storage file HFile of HBase,which can either avoid time-wasters in upper data import and save resources overhead as well,thus effectively solves the problems of low performance and inefficiency when importing data from a single database to HBase distributed database.Experimental result demonstrates that the fast parallel import tool designed and implemented based on the“fast parallel importing”technology supports the fast import of multi-column text data.Compared with traditional way using API to import data,its speed heightens more than double.