Inheritance diagram for CompositeToken:
Public Member Functions | |
this (ITokenizer tk, IBuffer buffer) | |
this (ITokenizer tk, IConduit conduit) | |
IBuffer | getBuffer () |
bool | next () |
bool | get () |
Private Member Functions | |
this (ITokenizer tk) | |
ITokenizer | getTokenizer () |
bool | next (IBuffer buf) |
Private Attributes | |
IBuffer | buffer |
Compare this to usage of a basic Token:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a Token and bind it to both the file and a line-tokenizer CompositeToken line = new CompositeToken (Tokenizers.line, fc); // read file a line at a time. Method get() returns false when no more // tokens are found. while (line.get) Stdout (line) (CR);
You might also consider a HybridToken for further processing of token content.
Definition at line 428 of file Token.d.
|
Set this token to use the provided Tokenizer, and bind it to the given buffer. Reimplemented in HybridToken. Definition at line 439 of file Token.d. References buffer. |
|
Set this token to use the provided Tokenizer, and bind it to the buffer associated with the given conduit. Reimplemented in HybridToken. |
|
Return the associated buffer Definition at line 463 of file Token.d. References buffer. |
|
Extract the next token. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an Eof condition is seen. Check token.getLength() when this method returns false. For example:
while (token.next || token.getLength) // do something Reimplemented in HybridToken. Definition at line 489 of file Token.d. References buffer. Referenced by get(), HttpClient::open(), and HttpRequest::readHeaders(). |
|
Extract the next token, taking Eof into consideration. If next() returns false, then this function will still return true as long as there's some content available. For example:
while (token.get) // do something Definition at line 510 of file Token.d. References next(). Referenced by Properties::load(), testFile1(), and testHttpClient(). |
|
Reimplemented in ReaderToken. Definition at line 298 of file Token.d. References BoundToken::tk. |
|
Return the associated tokenizer Definition at line 309 of file Token.d. References BoundToken::tk. |
|
Extract the next token from the provided buffer. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false. For example:
while (token.next() || token.getLength()) // do something Definition at line 335 of file Token.d. References ITokenizer::next(), and BoundToken::tk. Referenced by StartLine::VersionUseTokenizer::parse(), HttpParams::parse(), HttpHeaders::parse(), ResponseLine::parse(), testFile2(), and testToken1(). |
|
Definition at line 430 of file Token.d. Referenced by getBuffer(), next(), and this(). |