Inheritance diagram for BoundToken:
Public Member Functions | |
this (ITokenizer tk) | |
ITokenizer | getTokenizer () |
bool | next (IBuffer buf) |
Private Types | |
typedef void(* | Accumulator )(int digit) |
Private Member Functions | |
Token | set (char[] content) |
int | getLength () |
Token | setType (int type) |
int | getType () |
int | toInt () |
uint | toUInt () |
long | toLong () |
ulong | toULong () |
real | toReal () |
Token | clone () |
char[] | toString (bool slice=true) |
int | opEquals (Object o) |
int | opCmp (Object o) |
uint | toHash () |
void | write (IWriter w) |
Static Private Member Functions | |
this () | |
Private Attributes | |
ITokenizer | tk |
Contrast this example with that shown in the Token class:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer(); // bind a line-tokenizer to our input token BoundToken line = new BoundToken (Tokenizers.line); // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (line.next(buffer) || line.getLength()) Stdout.put(line).cr();
One might also consider a CompositeToken or HybridToken.
Definition at line 73 of file TokenEx.d.
|
Definition at line 103 of file Token.d. Referenced by Token::parseDigits(). |
|
Reimplemented in ReaderToken. Definition at line 81 of file TokenEx.d. References tk. |
|
Return the associated tokenizer Definition at line 92 of file TokenEx.d. References tk. |
|
Extract the next token from the provided buffer. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false. For example:
Definition at line 118 of file TokenEx.d. References ITokenizer::next(), and tk. Referenced by StartLine::VersionUseTokenizer::parse(), HttpParams::parse(), HttpHeaders::parse(), ResponseLine::parse(), testFile2(), and testToken1(). |
|
Construct a static exception instance. Definition at line 141 of file Token.d. References Token::Error. |
|
Set the content of this token. Definition at line 152 of file Token.d. References Token::content. Referenced by TokenStack::clone(), Token::clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push(). |
|
Return the length of this token. Definition at line 164 of file Token.d. Referenced by CompositeToken::get(), TextReader::next(), HttpParams::parse(), HttpHeaders::parse(), testFile2(), and testFile3(). |
|
Set the type of this token. Token types can be useful when one wishes to categorize input patterns. Definition at line 176 of file Token.d. References Token::type. |
|
Return the type associated with this token. See setType(). Definition at line 188 of file Token.d. References Token::type. |
|
Convert this token to an integer. Definition at line 199 of file Token.d. References Token::parseDigits(). Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8(). |
|
Convert this token to an unsigned integer. Definition at line 217 of file Token.d. References Token::Error, and Token::parseDigits(). |
|
Convert this token to a long integer. Definition at line 237 of file Token.d. References Token::parseDigits(). Referenced by TextReader::int64(). |
|
Convert this token to an unsigned long integer. Definition at line 255 of file Token.d. References Token::Error, and Token::parseDigits(). |
|
Convert this token to a real. Definition at line 275 of file Token.d. References Token::divisors, Token::Error, and Token::parseDigits(). Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80(). |
|
Clone this token, making a copy of the content also. Definition at line 309 of file Token.d. References Token::set(), and Token::type. |
|
Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead). Definition at line 326 of file Token.d. Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), HybridToken::next(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split(). |
|
Is this token equal to another? Definition at line 339 of file Token.d. References Token::content. |
|
Compare this token to another. Definition at line 355 of file Token.d. References Token::content. |
|
Hash this token |
|
Make the Token class compatible with IWriter instances. Reimplemented from IWritable. Definition at line 384 of file Token.d. References IWriter::put(). |
|
Definition at line 75 of file TokenEx.d. Referenced by getTokenizer(), next(), and this(). |