CAJ | 학술논문

针对 Hive在处理连接查询时所存在的执行时间长和带宽资源消耗大等问题，提出了一种基于数据预处理和双半连接的SDD-1改进算法。首先，引入预处理技术，在各分布节点对原始数据进行归并排序，以减少汇聚节点的数据映射次数，加快数据处理执行速度；其次，采用基于行和列的双半连接技术，进一步缩减在不同节点间的数据传输量，减少带宽资源消耗。仿真实验表明，相比原始的 Hive 连接算法，改进算法在元组数达到5000和8000时，可使查询速度提升10%，有效缩短查询的处理和响应时间，该改进算法可方便地应用到其他云计算平台上。
침대 Hive재처리련접사순시소존재적집행시간장화대관자원소모대등문제，제출료일충기우수거예처리화쌍반련접적SDD-1개진산법。수선，인입예처리기술，재각분포절점대원시수거진행귀병배서，이감소회취절점적수거영사차수，가쾌수거처리집행속도；기차，채용기우행화렬적쌍반련접기술，진일보축감재불동절점간적수거전수량，감소대관자원소모。방진실험표명，상비원시적 Hive 련접산법，개진산법재원조수체도5000화8000시，가사사순속도제승10%，유효축단사순적처리화향응시간，해개진산법가방편지응용도기타운계산평태상。
To solove the existence of the long execution time and bandwidth resource consumption and other issues when dealing with queries in Hive system,this paper presented based on data preprocessing and double half connected SDD-1 improved algorithm.Firstly,the introduction of pre-processing technology, the distribution of nodes in each merge sort the raw data in order to reduce the number of data aggregation node mapping,speed up data processing speed of execution;Secondly,the use of semi-connection technolo-gy based on double rows and columns,and further reduction in different data transfer between nodes,re-ducing bandwidth consumption.The simulation results show that,compared to the original Hive join algo-rithm,the improved algorithm in the number of tuples to 5 000 and 8 000,can make the query speed in-creased by 10%,shorten the processing and query response time,application of the improved algorithm can be convenient to other cloud computing platform.