CAJ | 학술논문

汉语分词是中文信息处理的一项基础性工作。为避免人工阅读或机器处理时的分词歧义和未登录词难以识别的问题，有专家建议写作时在汉语词之间添加空格。文章从语言学本体研究、语言使用以及语言工程等不同角度对传统观念下的汉语分词存在的困难进行探讨，指出汉语分词在词的定义、群众语感以及分词规范、词表确定及工程应用等方面都存在不确定及不一致等因素。近年汉语自动分词处理不纠缠于词的确切定义，以字组词，针对标注语料和网络上带有丰富结构信息的海量文本，利用机器学习方法对汉语“切分单位”的标注取得了较好的进展。针对基础性的汉语分词规范，从语言规划的政策性、科学性及引导性角度提出建议，最后指出结合语言学指导和数据驱动的机器学习策略，可望为实现汉语自动分词的准确性和适应性提升服务。
한어분사시중문신식처리적일항기출성공작。위피면인공열독혹궤기처리시적분사기의화미등록사난이식별적문제，유전가건의사작시재한어사지간첨가공격。문장종어언학본체연구、어언사용이급어언공정등불동각도대전통관념하적한어분사존재적곤난진행탐토，지출한어분사재사적정의、군음어감이급분사규범、사표학정급공정응용등방면도존재불학정급불일치등인소。근년한어자동분사처리불규전우사적학절정의，이자조사，침대표주어료화망락상대유봉부결구신식적해량문본，이용궤기학습방법대한어“절분단위”적표주취득료교호적진전。침대기출성적한어분사규범，종어언규화적정책성、과학성급인도성각도제출건의，최후지출결합어언학지도화수거구동적궤기학습책략，가망위실현한어자동분사적준학성화괄응성제승복무。
Chinese word segmentation is fundamental for Chinese information processing.To a-void ambiguity and out-of-vocabulary word,there was a proposal for adding a manual space between Chinese words,which we disagree with it.This paper first elaborates difficulties in word segmenta-tion from the point of linguistic studies,language performance and language engineering,and then it discusses some uncertain factors in definition of the word,language awareness,word segmentation specification,construction of word list and its application in automatic text information processing. Not dwelling on exact definition of the word,the paper lists recent advances in character-based tag-ging with massive manually annotated recourses,which show an inspiring progress.At the end of the paper,we put forward a word segmentation guideline from a stance of language policy strategies. Guided by linguistic theory and data-driven machine learning algorithms,a practical word segmenta-tion system can achieve better success in precision and adaptability.