Class ClassicTokenizerDescriptor

  • All Implemented Interfaces:
    TokenizerDescriptor

    public class ClassicTokenizerDescriptor
    extends Object
    implements TokenizerDescriptor
    This tokenizer has heuristics for special treatment of acronyms, company names, email addresses, and internet host names. However, these rules don’t always work, and the tokenizer doesn’t work well for most languages other than English.
    Since:
    7.0
    • Constructor Detail

      • ClassicTokenizerDescriptor

        public ClassicTokenizerDescriptor()