Inheritance diagram for LineScanner:
Public Member Functions | |
void | read (IReader r) |
bool | next (IBuffer buffer) |
Private Member Functions | |
bool | next (IBuffer buffer, int(*scan)(char[])) |
int | notFound (Token token, char[] content) |
Definition at line 352 of file Tokenizer.d.
|
IReadable interface to support Reader.get() Reimplemented from IReadable. Definition at line 360 of file Tokenizer.d. References IReader::getBuffer(), and next(). |
|
Eat all content until we see a ' Definition at line 372 of file Tokenizer.d. References IConduit. Referenced by read(). |
|
Scan the given IBuffer for another token, and place the results in the provided token. Note that this should be completely thread-safe so one can instantiate singleton tokenizers without issue. Each Token is expected to be stripped of the delimiter. An end-of-file condition causes trailing content to be placed into the token. Requests made beyond Eof result in empty tokens (length == zero). Returns true if a token was isolated, false otherwise. Definition at line 78 of file Tokenizer.d. References IBuffer::compress(), IConduit::Eof, IBuffer::getConduit(), IBuffer::getPosition(), IConduit, IConduit::read(), IBuffer::read(), IBuffer::readable(), IBuffer::skip(), and IBuffer::writable(). |
|
Clean up after we fail to find a token. Trailing content is placed into the token, and the scanner is told to try and load some more content (where available). Definition at line 120 of file Tokenizer.d. References IConduit, and Token::set(). Referenced by LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), and SimpleTokenizer::next(). |