StandardTokenizer.jj的代码

游客

标题:StandardTokenizer.jj的代码
正文:
下边为StandardTokenizer.jj的代码,省略了前面的注释!使用Javacc的语法写成的,学过 javacc的语法之后,在看这些代码就比较容易了,而且整个Standard包下其他很多代码都是由此文件生成的.根据此文件可以看 StandardTokenizer用来实现区分token,从而StandardAnalyzer使用的分词方法就是单字切分. 同徽B2B电子商务研究中心options { STATIC = false;//IGNORE_CASE = true;//BUILD_PARSER = false; UNICODE_INPUT = true; USER_CHAR_STREAM = true; OPTIMIZE_TOKEN_MANAGER = true;//DEBUG_TOKEN_MANAGER = true;}PARSER_BEGIN(StandardTokenizer)数据挖掘商道package org.apache.lucene.analysis.standard; 同徽公司/** A grammar-based tokenizer constructed with JavaCC. * * &lt;p&gt; This should be a good tokenizer for most European-language documents. * * &lt;p&gt;Many applications have specific tokenizer needs. If this tokenizer does * not suit your application, please consider copying this source code * directory to your project and maintaining your own grammar-based tokenizer. *基于语法的tokenizer */public class StandardTokenizer extends org.apache.lucene.analysis.Tokenizer {数据挖掘研究院
[<<][[1]][2][3][>>]

查看评论(0)
发表评论

首页