Inheritance diagram for Token:
Public Types | |
typedef void(* | Accumulator )(int digit) |
Public Member Functions | |
Token | set (char[] content) |
int | getLength () |
Token | setType (int type) |
int | getType () |
int | toInt () |
uint | toUInt () |
long | toLong () |
ulong | toULong () |
real | toReal () |
Token | clone () |
char[] | toString (bool slice=true) |
int | opEquals (Object o) |
int | opCmp (Object o) |
uint | toHash () |
void | write (IWriter w) |
Static Public Member Functions | |
this () | |
Private Types | |
enum | State { Begin, Number, Fraction } |
Private Member Functions | |
bool | parseDigits (Accumulator dg, Accumulator fraction) |
Private Attributes | |
int | type |
char[] | content |
Static Private Attributes | |
TokenException | Error |
real | divisors [] |
Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.
Here's a brief example of how to apply Token with Tokenizers:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer (); // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength()) Stdout.put(token).cr();
See also BoundToken, ReaderToken, CompositeToken and HybridToken.
Definition at line 94 of file Token.d.
|
Definition at line 103 of file Token.d. Referenced by parseDigits(). |
|
|
|
Construct a static exception instance. Definition at line 141 of file Token.d. References Error. |
|
Set the content of this token. Definition at line 152 of file Token.d. References content. Referenced by TokenStack::clone(), clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push(). |
|
Return the length of this token. Definition at line 164 of file Token.d. Referenced by CompositeToken::get(), TextReader::next(), HttpParams::parse(), HttpHeaders::parse(), testFile2(), and testFile3(). |
|
Set the type of this token. Token types can be useful when one wishes to categorize input patterns. Definition at line 176 of file Token.d. References type. |
|
Return the type associated with this token. See setType(). Definition at line 188 of file Token.d. References type. |
|
Convert this token to an integer. Definition at line 199 of file Token.d. References parseDigits(). Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8(). |
|
Convert this token to an unsigned integer. Definition at line 217 of file Token.d. References Error, and parseDigits(). |
|
Convert this token to a long integer. Definition at line 237 of file Token.d. References parseDigits(). Referenced by TextReader::int64(). |
|
Convert this token to an unsigned long integer. Definition at line 255 of file Token.d. References Error, and parseDigits(). |
|
Convert this token to a real. Definition at line 275 of file Token.d. References divisors, Error, and parseDigits(). Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80(). |
|
Clone this token, making a copy of the content also. |
|
Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead). Definition at line 326 of file Token.d. Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), HybridToken::next(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split(). |
|
Is this token equal to another? Definition at line 339 of file Token.d. References content. |
|
Compare this token to another. Definition at line 355 of file Token.d. References content. |
|
Hash this token |
|
Make the Token class compatible with IWriter instances. Reimplemented from IWritable. Definition at line 384 of file Token.d. References IWriter::put(). |
|
Parse a set of digits, optionally prefixed by a '+' or '-', and optionally with a '.' contained within. Delegates come in quite handy here, and add relatively little overhead. Definition at line 397 of file Token.d. References Accumulator, and Error. Referenced by toInt(), toLong(), toReal(), toUInt(), and toULong(). |
|
|
|
Definition at line 97 of file Token.d. Referenced by opCmp(), opEquals(), and set(). |
|
Definition at line 99 of file Token.d. Referenced by parseDigits(), this(), toReal(), toUInt(), and toULong(). |
|
Initial value: [ 0.1 0.01 0.001 0.0001 0.00001 0.000001 0.0000001 0.00000001 0.000000001 0.0000000001 0.00000000001 0.000000000001 0.0000000000001 0.00000000000001 0.000000000000001 0.0000000000000001 0.00000000000000001 0.000000000000000001 0.0000000000000000001 0.00000000000000000001 0.000000000000000000001 0.0000000000000000000001 ] Definition at line 109 of file Token.d. Referenced by toReal(). |