CAJ | 학술논문

编码识别是网页内容过滤的必要前提，多种中文编码共存给中文网页的内容过滤带来不便。针对上述问题，提出一种基于字频分布的中文网页编码识别算法。根据汉字的使用频率，选取使用频度较高的字符构成高频字符编码表，以高频字符编码作为关键字，使用改进的模式匹配算法查找待识别网页，并统计匹配次数。将编码的匹配结果作为分析的依据，最终判定待识别网页的真实码制。实验结果证明，与Unigram算法相比，该算法对目前通用的中文编码识别率较高，适合对未知编码的中文网页进行快速编码识别。
편마식별시망혈내용과려적필요전제，다충중문편마공존급중문망혈적내용과려대래불편。침대상술문제，제출일충기우자빈분포적중문망혈편마식별산법。근거한자적사용빈솔，선취사용빈도교고적자부구성고빈자부편마표，이고빈자부편마작위관건자，사용개진적모식필배산법사조대식별망혈，병통계필배차수。장편마적필배결과작위분석적의거，최종판정대식별망혈적진실마제。실험결과증명，여Unigram산법상비，해산법대목전통용적중문편마식별솔교고，괄합대미지편마적중문망혈진행쾌속편마식별。
Web coding identification is the premise of webpage content filtering,and coexistence of a variety of Chinese encoding makes Chinese webpage coded identification inconvenient. This paper presents a Chinese Web encoding identification algorithm———FKI ( Frequency Keyword Identification ) which is based on the frequency of Chinese character used. FKI selects the frequency of high character to construct high frequency character encoding tables, according to the frequency of the use of Chinese characters. Using high frequency character encoding as a keyword,FKI algorithm scans the Webpage by improved pattern matching algorithm, statistical matching number, and determines the real code of webpage based on the matching result. Experimental results show that, compared with the Unigram algorithm,this algorithm has a higher recognition rate. FKI algorithm is suitable for Chinese webpage which is unknown code to identify code quickly and accurately.