CAJ | 학술논문

为了使全文检索系统支持多种文件格式的检索，必须先对待检索的文件进行文本抽取以转化为便于建立索引的纯文本。针对多格式的文本抽取问题，文章设计了一种基于插件的支持多格式的文本抽取系统，该系统采用文件后缀名和魔数(magic number)结合的方式自动识别文件类型，以统一接口调用已存在的针对单一类型文件的抽取插件，对得到的纯文本进行编码转换以使得最终的输出编码统一，系统还针对目录输入设计了多进程并行优化以利用CPU多核优势，使用贪心算法优化任务分配以使总运行时间尽可能短。该系统易于扩展，编程接口简单。实验结果表明，该系统能正常抽取文本内容和元数据，且其抽取效率高于Apache的Tika等开源文本抽取系统。
위료사전문검색계통지지다충문건격식적검색，필수선대대검색적문건진행문본추취이전화위편우건립색인적순문본。침대다격식적문본추취문제，문장설계료일충기우삽건적지지다격식적문본추취계통，해계통채용문건후철명화마수(magic number)결합적방식자동식별문건류형，이통일접구조용이존재적침대단일류형문건적추취삽건，대득도적순문본진행편마전환이사득최종적수출편마통일，계통환침대목록수입설계료다진정병행우화이이용CPU다핵우세，사용탐심산법우화임무분배이사총운행시간진가능단。해계통역우확전，편정접구간단。실험결과표명，해계통능정상추취문본내용화원수거，차기추취효솔고우Apache적Tika등개원문본추취계통。
This paper designs a text extraction system that converts multi-format file sources to plain texts; such a system plays a key role in full-text retrieval tasks. The system is designed based on plugins and is able to support a variety of file formats. The system detects file types using the combination of file extensions and magic numbers, calls existing single-type-oriented plugins through a uniform interface, and unifies the encoding of output plain texts. Two novel features of the system include designing a greedy scheduling algorithm that minimizes the overall running time, as well as implementing the algorithm in a multi-process manner that takes full advantages of multiple cores. The system is easy to expand and has simple APIs. Experimental results show that the system can extract text contents and metadata of supported file formats, and outperform Apache's Tika, an existing open source system.