Inheritance diagram for CompositeToken:
Public Member Functions | |
this (ITokenizer tk, IBuffer buffer) | |
this (ITokenizer tk, IConduit conduit) | |
IBuffer | getBuffer () |
bool | next () |
bool | get () |
Private Member Functions | |
this (ITokenizer tk) | |
ITokenizer | getTokenizer () |
bool | next (IBuffer buf) |
Static Private Member Functions | |
this () | |
Private Attributes | |
IBuffer | buffer |
Compare this to usage of a basic Token:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a Token and bind it to both the file and a line-tokenizer CompositeToken line = new CompositeToken (Tokenizers.line, fc); // read file a line at a time. Method get() returns false when no more // tokens are found. while (line.get) Stdout.put(line).cr();
You might also consider a HybridToken for further processing of token content.
Definition at line 211 of file TokenEx.d.
|
Set this token to use the provided Tokenizer, and bind it to the given buffer. Reimplemented in HybridToken. |
|
Set this token to use the provided Tokenizer, and bind it to the buffer associated with the given conduit. Reimplemented in HybridToken. |
|
Return the associated buffer |
|
Extract the next token. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an Eof condition is seen. Check token.getLength() when this method returns false. For example:
Reimplemented in HybridToken. Definition at line 272 of file TokenEx.d. Referenced by get(), HttpClient::open(), HttpRequest::readHeaders(), and wordRegex(). |
|
Extract the next token, taking Eof into consideration. If next() returns false, then this function will still return true as long as there's some content available. For example:
Definition at line 293 of file TokenEx.d. References Token::getLength(), and next(). Referenced by Properties::load(), testFile1(), testHttpClient(), and testLineRegex(). |
|
Reimplemented in ReaderToken. Definition at line 81 of file TokenEx.d. References BoundToken::tk. |
|
Construct a static exception instance. Definition at line 141 of file Token.d. References Token::Error. |
|
Return the associated tokenizer Definition at line 92 of file TokenEx.d. References BoundToken::tk. |
|
Extract the next token from the provided buffer. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false. For example:
Definition at line 118 of file TokenEx.d. References ITokenizer::next(), and BoundToken::tk. Referenced by StartLine::VersionUseTokenizer::parse(), HttpParams::parse(), HttpHeaders::parse(), ResponseLine::parse(), testFile2(), and testToken1(). |
|
|