CAJ | 학술논문

在大数据处理过程中，系统必须有非常高的数据处理效率。为了满足对大数据实时、高效、稳定处理的需求，提出了一种接收与处理分离的数据处理模型。该数据处理模型由数据接收单元、内存数据库、原始数据分发单元、数据处理单元、处理数据分发单元、数据归并单元组成。接收单元负责接收、整合结构化数据与非结构化数据，把每条完整的数据放入内存数据库中；分发单元从内存数据库中检测获取数据，按照海量数据负载均衡算法把数据分发到数据处理单元；数据处理单元处理数据，处理结果放入内存数据库；处理数据分发单元继续从内存数据库中获取处理后的数据，并按照海量数据负载均衡算法把数据分发给数据归并单元。实验证明，使用该模型方法，系统保持了非常高效的处理效率。
재대수거처리과정중，계통필수유비상고적수거처리효솔。위료만족대대수거실시、고효、은정처리적수구，제출료일충접수여처리분리적수거처리모형。해수거처리모형유수거접수단원、내존수거고、원시수거분발단원、수거처리단원、처리수거분발단원、수거귀병단원조성。접수단원부책접수、정합결구화수거여비결구화수거，파매조완정적수거방입내존수거고중；분발단원종내존수거고중검측획취수거，안조해량수거부재균형산법파수거분발도수거처리단원；수거처리단원처리수거，처리결과방입내존수거고；처리수거분발단원계속종내존수거고중획취처리후적수거，병안조해량수거부재균형산법파수거분발급수거귀병단원。실험증명，사용해모형방법，계통보지료비상고효적처리효솔。
The system must have high data processing efficiency during processing big data. This paper designs a data processing model based on receiving and processing separation. Using the method the system can real-time, effi-ciently, stably process data. The model includes the data receiving unit, the memory database, the original data distri-bution unit, the data processing unit, the processed data distribution unit and the data merging unit. The receiving unit receives and integrates structure and unstructured data to a whole structure datum and puts it into the memory database. The original data distribution unit checks data from the memory database and dispatches the data to the data processing unit using the big data load balancing method. The data processing unit processes data and puts the result data into the memory database again. The processed data distribution unit gets data from the memory database and dispatches the data to the data merging unit using the big data load balancing method. The experimental results show that the system keeps with the high efficiency using the model.