计算机工程与设计
計算機工程與設計
계산궤공정여설계
COMPUTER ENGINEERING AND DESIGN
2015年
2期
406-409
,共4页
Hadoop分布式文件系统%海量小文件%性能优化%职责分离%合并小文件
Hadoop分佈式文件繫統%海量小文件%性能優化%職責分離%閤併小文件
Hadoop분포식문건계통%해량소문건%성능우화%직책분리%합병소문건
Hadoop distributed file system%massive amount of small files%improving efficiency%segregation of duties%merge small files
为改善应用Hadoop分布式文件系统存储大量小文件时效率低下的问题,将NameNode职责分离,使用单独的NFS服务器同步存储元数据信息,以降低Client数据请求压力,提供大吞吐量数据访问并改善访问延迟;设计文件与数据块的对应模式,允许在同一块中存储多个小文件,并对系统加以实现,为海量小文件的存储提供了一个有效的解决方案。实验结果表明,该机制可以在数据迅速增长的背景下实现海量小文件的高效存取。
為改善應用Hadoop分佈式文件繫統存儲大量小文件時效率低下的問題,將NameNode職責分離,使用單獨的NFS服務器同步存儲元數據信息,以降低Client數據請求壓力,提供大吞吐量數據訪問併改善訪問延遲;設計文件與數據塊的對應模式,允許在同一塊中存儲多箇小文件,併對繫統加以實現,為海量小文件的存儲提供瞭一箇有效的解決方案。實驗結果錶明,該機製可以在數據迅速增長的揹景下實現海量小文件的高效存取。
위개선응용Hadoop분포식문건계통존저대량소문건시효솔저하적문제,장NameNode직책분리,사용단독적NFS복무기동보존저원수거신식,이강저Client수거청구압력,제공대탄토량수거방문병개선방문연지;설계문건여수거괴적대응모식,윤허재동일괴중존저다개소문건,병대계통가이실현,위해량소문건적존저제공료일개유효적해결방안。실험결과표명,해궤제가이재수거신속증장적배경하실현해량소문건적고효존취。
The HDFS is designed for the large file storage of the GB and the TB-level,which can not efficiently store large amounts of small files.By separating the NameNode duties and using a separate NFS server storing the metadata synchronization information,the data request pressure from the Client was reduced,the high throughput data access was provided and the access latency was improved.The corresponding modes for files and data blocks were designed that allowed multiple small files stored in the same block.The system was implemented,so as to provide an effective solution to the mass of small files storage.Experi-mental results show that this mechanism can realize the reliable massive small files access efficiently in a data rapidly growing background.