Class CustomJapaneseTokenizer

  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.Tokenizer
              extended by

public final class CustomJapaneseTokenizer
extends org.apache.lucene.analysis.Tokenizer

Nasty fork from Lucene to accomodate searching half width / full width Japanese characters. Remove when is addressed.

Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
Field Summary
Fields inherited from class org.apache.lucene.analysis.Tokenizer
Constructor Summary
CustomJapaneseTokenizer(Reader in)
          Construct a token stream processing the given input.
Method Summary
 org.apache.lucene.analysis.Token next()
          Returns the next token in the stream, or null at EOS.
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset, reset
Methods inherited from class org.apache.lucene.analysis.TokenStream
end, getOnlyUseNewAPI, incrementToken, next, reset, setOnlyUseNewAPI
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail


public CustomJapaneseTokenizer(Reader in)
Construct a token stream processing the given input.

in - I/O reader
Method Detail


public final org.apache.lucene.analysis.Token next()
                                            throws IOException
Returns the next token in the stream, or null at EOS. See for detail.

next in class org.apache.lucene.analysis.TokenStream
IOException - - throw IOException when read error
hanppened in the InputStream

Copyright © 2003-2013 Atlassian. All Rights Reserved.