CAJ | 학술논문

高阶N-gram语言模型在OCR后处理方面有着广泛的应用,但也面临着因模型复杂度大导致的数据稀疏,以及耗费较多的时空资源等问题.该文针对印刷体汉字识别的后处理,提出了一种基于字节的语言模型的后处理算法.通过采用字节作为语言模型的基本表示单位,模型的复杂度大大降低,从而数据稀疏问题得到很大程度上缓解.实验证明,采用基于字节的语言模型的后处理系统能够以极少的时空开销获取很好的识别性能.在有部分分割错误的测试集上,正确率从88.67%提高到了98.32%,错误率下降了85.18%,运行速度较基于字以及基于词的系统有了大幅的提升,提高了后处理系统的综合性能;与目前常用的基于词的语言模型后处理系统相比,新系统能够节省95%的运行时间和98%的内存资源,但系统识别率仅降低了1.11%.
고계N-gram어언모형재OCR후처리방면유착엄범적응용,단야면림착인모형복잡도대도치적수거희소,이급모비교다적시공자원등문제.해문침대인쇄체한자식별적후처리,제출료일충기우자절적어언모형적후처리산법.통과채용자절작위어언모형적기본표시단위,모형적복잡도대대강저,종이수거희소문제득도흔대정도상완해.실험증명,채용기우자절적어언모형적후처리계통능구이겁소적시공개소획취흔호적식별성능.재유부분분할착오적측시집상,정학솔종88.67%제고도료98.32%,착오솔하강료85.18%,운행속도교기우자이급기우사적계통유료대폭적제승,제고료후처리계통적종합성능;여목전상용적기우사적어언모형후처리계통상비,신계통능구절성95%적운행시간화98%적내존자원,단계통식별솔부강저료1.11%.
In Chinese OCR post-processing, the high-order Chinese n-gram language models, such as word based tri-gram and four-gram is still a challenging issue because of the data sparseness issue and large memory cost led by big model size. In this paper, we focus on the post-processing of printed Chinese character recognition and propose a byte-based language model. By choosing byte as the representing unit of language model, we achieve a remarkable reduction of model size which overcomes the sparseness problem to a great extent. The experimental results show that the new language model based on byte works very well with higher performance and lowest time and space costs. For the test set with segmentation errors, the recognition accuracy increases from 88. 67% to 98. 32% , which means 85. 18% error reduction. Compared with the system using traditional word based tri-gram, the new system saves 95% time cost and nearly 98% memory cost at almost no cost in the accuracy performance.