Inheritance diagram for SimpleTokenizer:
Public Member Functions | |
this (char delimiter) | |
bool | next (IBuffer buffer, Token token) |
Private Member Functions | |
bool | next (IBuffer buffer, int(*scan)(char[])) |
int | notFound (Token token, char[] content) |
Private Attributes | |
char | delimiter |
Definition at line 145 of file Tokenizer.d.
|
Construct a SimpleTokenizer with the given delimiter char. More sophisticated delimiters can be constructed by using a RegexTokenizer instead. Definition at line 157 of file Tokenizer.d. |
|
Locate the next token from the provided buffer, and map a buffer reference into token. Returns true if a token was located, false otherwise. Note that the buffer content is not duplicated. Instead, a slice of the buffer is referenced by the token. You can use Token.clone() or Token.toString().dup() to copy content per your application needs. Note also that there may still be one token left in a buffer that was not terminated correctly (as in eof conditions). In such cases, tokens are mapped onto remaining content and the buffer will have no more readable content. Reimplemented from ITokenizer. Definition at line 181 of file Tokenizer.d. References Scanner::notFound(), and Token::set(). |
|
Scan the given IBuffer for another token, and place the results in the provided token. Note that this should be completely thread-safe so one can instantiate singleton tokenizers without issue. Each Token is expected to be stripped of the delimiter. An end-of-file condition causes trailing content to be placed into the token. Requests made beyond Eof result in empty tokens (length == zero). Returns true if a token was isolated, false otherwise. Definition at line 72 of file Tokenizer.d. References IBuffer::compress(), IBuffer::getConduit(), IBuffer::getPosition(), IConduit::read(), IBuffer::read(), IBuffer::readable(), IBuffer::skip(), and IBuffer::writable(). |
|
Clean up after we fail to find a token. Trailing content is placed into the token, and the scanner is told to try and load some more content (where available). Definition at line 114 of file Tokenizer.d. References Token::set(). Referenced by RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), and next(). |
|
Definition at line 147 of file Tokenizer.d. |