Inheritance diagram for ITokenizer:
Public Member Functions | |
bool | next (IBuffer buffer, Token token) |
Definition at line 134 of file Tokenizer.d.
|
Reimplemented in SimpleTokenizer, SpaceTokenizer, PunctTokenizer, and LineTokenizer. Referenced by BoundToken::next(), TextReader::nextToken(), and ReaderToken::read(). |