Inheritance diagram for Scanner:
Public Member Functions | |
bool | next (IBuffer buffer, int(*scan)(char[])) |
int | notFound (Token token, char[] content) |
Definition at line 54 of file Tokenizer.d.
|
Scan the given IBuffer for another token, and place the results in the provided token. Note that this should be completely thread-safe so one can instantiate singleton tokenizers without issue. Each Token is expected to be stripped of the delimiter. An end-of-file condition causes trailing content to be placed into the token. Requests made beyond Eof result in empty tokens (length == zero). Returns true if a token was isolated, false otherwise. Definition at line 72 of file Tokenizer.d. References IBuffer::compress(), IBuffer::getConduit(), IBuffer::getPosition(), IConduit::read(), IBuffer::read(), IBuffer::readable(), IBuffer::skip(), and IBuffer::writable(). |
|
Clean up after we fail to find a token. Trailing content is placed into the token, and the scanner is told to try and load some more content (where available). Definition at line 114 of file Tokenizer.d. References Token::set(). Referenced by RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), and SimpleTokenizer::next(). |