Inheritance diagram for ITokenizer:
Public Member Functions | |
bool | next (IBuffer buffer, Token token) |
Definition at line 134 of file Tokenizer.d.
|
Reimplemented in SimpleTokenizer, SpaceTokenizer, PunctTokenizer, and LineTokenizer. Referenced by LineTokenizerTemplate(), BoundToken::next(), TextReader::nextToken(), PunctTokenizerTemplate(), ReaderToken::read(), SimpleTokenizerTemplate(), and SpaceTokenizerTemplate(). |