CAJ | 학술논문

Hadoop中的HDFS是大数据存储处理的关键技术之一，HDFS有着存储超大数据集高效可靠等优点，HDFS存储小文件有着明显的缺陷。HBase是有着非常高效的数据查询能力，本文目标是结合HDFS和HBase各自的优点，设计一个复合式的大数据存储系统，将大于64MB文件存储在HDFS中；大于10M小于64MB文件存储在HDFS中，将文件目录存储在HBase中，提高检索速度；小于10M的文件直接存储在HBase中，较好的解决了大量小文件存储时NameNode内存瓶颈问题。实验证明这种设计能够提高存储效率。
Hadoop중적HDFS시대수거존저처리적관건기술지일，HDFS유착존저초대수거집고효가고등우점，HDFS존저소문건유착명현적결함。HBase시유착비상고효적수거사순능력，본문목표시결합HDFS화HBase각자적우점，설계일개복합식적대수거존저계통，장대우64MB문건존저재HDFS중；대우10M소우64MB문건존저재HDFS중，장문건목록존저재HBase중，제고검색속도；소우10M적문건직접존저재HBase중，교호적해결료대량소문건존저시NameNode내존병경문제。실험증명저충설계능구제고존저효솔。
HDFS in Hadoop is one of the key technologies of large data storage treatment. HDFS is efficient and reliable in large data storage, and it has obvious defects in small data storage. HBase has efficient data query ability. Combined with the advantages of HDFS and HBase, this paper designs a compound large data storage system. The file more than 64MB is stored in HDFS, the file more than 10M and less than 64MB is stored in HDFS, the file directory is stored in HBase to improve the retrieval speed. The file less than 10M is directly stored in HBase, it better solves the NameNode memory bottlenecks of the storage of a large number of small file. Experiment proves that this design can improve the efficiency of storage.