Inheritance diagram for ITokenizer:
Public Member Functions | |
bool | next (IBuffer buffer, Token token) |
Definition at line 128 of file Tokenizer.d.
|
Reimplemented in SimpleTokenizer, SpaceTokenizer, PunctTokenizer, LineTokenizer, and RegexTokenizer. Referenced by BoundToken::next(), and TextReader::next(). |