An Analyzer builds TokenStreams, which analyze text. It thus represents a
policy for extracting index terms from text.
Typical implementations first build a Tokenizer, which breaks the stream of
characters from the Reader into raw Tokens. One or more TokenFilters may
then be applied to the output of the Tokenizer.
WARNING: You must override one of the methods defined by this class in your
subclass or the Analyzer will enter an infinite loop.
Typical implementations first build a Tokenizer, which breaks the stream of characters from the Reader into raw Tokens. One or more TokenFilters may then be applied to the output of the Tokenizer.
WARNING: You must override one of the methods defined by this class in your subclass or the Analyzer will enter an infinite loop.