CAJ | 학술논문

该文提出了一种无监督和有监督相结合的中文分词方法: 将邻接变化数(Accessor Variety,AV)引入基于条件随机场的中文分词系统中.针对邻接变化数在处理较少的训练数据时存在的缺陷,提出了一种归一化的改进方法,以减轻计算AV值时产生的波动.基于Bakeoff-4的中文分词实验表明,归一化的邻接变化数方法无论对于封闭测试,还是开放测试,都带来了性能的提升.
해문제출료일충무감독화유감독상결합적중문분사방법: 장린접변화수(Accessor Variety,AV)인입기우조건수궤장적중문분사계통중.침대린접변화수재처리교소적훈련수거시존재적결함,제출료일충귀일화적개진방법,이감경계산AV치시산생적파동.기우Bakeoff-4적중문분사실험표명,귀일화적린접변화수방법무론대우봉폐측시,환시개방측시,도대래료성능적제승.
This paper proposes a method combining supervised learning with unsupervised method to conduct Chinese word segmentation (CWS), which incorporates the Accessor Variety (AV) into the Conditional Random Fields (CRFs). To solve the flaw in Accessor Variety (AV) when dealing with limited training data, normalization is introduced to alleviate the fluctuation in the AV value in the phrase of unsupervised segmentation. Experiments on the Bakeoff-4 data indicate that normalized Accessor Variety is effective both for close and open tracks.