CAJ | 학술논문

Twitter已经成为微博中的代表性应用，但是通过分析发现twitter上的消息（推文）有很多完全一致或相似，这对后续对推文的分析和存储都带来很大的问题。为了处理这些内容完全一致或相似的消息（推文），针对推文特有的短文本的特点，基于规则处理完全一致的推文，采用simhash的方法来处理相似性的推文。实验采用实际抓取的240万条推文数据进行分析和处理，分别对中文和英文的推文重复情况进行了分析，实验结果发现重复的推文占总推文的10%左右。
Twitter이경성위미박중적대표성응용，단시통과분석발현twitter상적소식（추문）유흔다완전일치혹상사，저대후속대추문적분석화존저도대래흔대적문제。위료처리저사내용완전일치혹상사적소식（추문），침대추문특유적단문본적특점，기우규칙처리완전일치적추문，채용simhash적방법래처리상사성적추문。실험채용실제조취적240만조추문수거진행분석화처리，분별대중문화영문적추문중복정황진행료분석，실험결과발현중복적추문점총추문적10%좌우。
Twitter has become the representative applications of the micro-blog. By analysis on twitter a lot of messages (tweets)are the same or similar. Those messages bring up a trouble on the analysis and message storage, so it is needed to remove those messages which are the same or similar. According to the characteristics of short text on tweets, this paper proposes the following approach. It processes the same tweets based on the specific format, then uses the simhash to process the similar tweets. It uses 240 million tweets crawled on the Internet to experiment. In the experiment it only processes Chinese and English tweets. The repetition messages(tweets)is 10 percent of all the Chinese and English tweets.