Inheritance diagram for Token:
Public Types | |
typedef void(* | Accumulator )(int digit) |
Public Member Functions | |
Token | set (char[] content) |
int | getLength () |
Token | setType (int type) |
int | getType () |
int | toInt () |
uint | toUInt () |
long | toLong () |
ulong | toULong () |
real | toReal () |
Token | clone () |
char[] | toString (bool slice=true) |
int | opEquals (Object o) |
int | opCmp (Object o) |
uint | toHash () |
void | write (IWriter w) |
Static Public Member Functions | |
this () | |
Private Types | |
enum | State { Begin, Number, Fraction } |
Private Member Functions | |
bool | parseDigits (Accumulator dg, Accumulator fraction) |
Private Attributes | |
int | type |
char[] | content |
Static Private Attributes | |
TokenException | Error |
real | divisors [] |
Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.
Here's a brief example of how to apply Token with Tokenizers:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer (); // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength()) Stdout.put(token).cr();
See also BoundToken, ReaderToken, CompositeToken and HybridToken.
Definition at line 97 of file Token.d.
|
Definition at line 106 of file Token.d. Referenced by parseDigits(). |
|
|
|
Construct a static exception instance. Definition at line 144 of file Token.d. References Error. |
|
Set the content of this token. Definition at line 155 of file Token.d. References content. Referenced by TokenStack::clone(), clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push(). |
|
Return the length of this token. Definition at line 167 of file Token.d. Referenced by CompositeToken::get(), TextReader::next(), HttpParams::parse(), HttpHeaders::parse(), testFile2(), and testFile3(). |
|
Set the type of this token. Token types can be useful when one wishes to categorize input patterns. Definition at line 179 of file Token.d. References type. |
|
Return the type associated with this token. See setType(). Definition at line 191 of file Token.d. References type. |
|
Convert this token to an integer. Definition at line 202 of file Token.d. References parseDigits(). Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8(). |
|
Convert this token to an unsigned integer. Definition at line 220 of file Token.d. References Error, and parseDigits(). |
|
Convert this token to a long integer. Definition at line 240 of file Token.d. References parseDigits(). Referenced by TextReader::int64(). |
|
Convert this token to an unsigned long integer. Definition at line 258 of file Token.d. References Error, and parseDigits(). |
|
Convert this token to a real. Definition at line 278 of file Token.d. References divisors, Error, and parseDigits(). Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80(). |
|
Clone this token, making a copy of the content also. |
|
Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead). Definition at line 329 of file Token.d. Referenced by StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), HybridToken::next(), TextReader::opShr(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split(). |
|
Is this token equal to another? Definition at line 342 of file Token.d. References content. |
|
Compare this token to another. Definition at line 358 of file Token.d. References content. |
|
Hash this token |
|
Make the Token class compatible with IWriter instances. Reimplemented from IWritable. Definition at line 387 of file Token.d. References IWriter::put. |
|
Parse a set of digits, optionally prefixed by a '+' or '-', and optionally with a '.' contained within. Delegates come in quite handy here, and add relatively little overhead. Definition at line 400 of file Token.d. References Accumulator, and Error. Referenced by toInt(), toLong(), toReal(), toUInt(), and toULong(). |
|
|
|
Definition at line 100 of file Token.d. Referenced by opCmp(), opEquals(), and set(). |
|
Definition at line 102 of file Token.d. Referenced by parseDigits(), this(), toReal(), toUInt(), and toULong(). |
|
Initial value: [ 0.1 0.01 0.001 0.0001 0.00001 0.000001 0.0000001 0.00000001 0.000000001 0.0000000001 0.00000000001 0.000000000001 0.0000000000001 0.00000000000001 0.000000000000001 0.0000000000000001 0.00000000000000001 0.000000000000000001 0.0000000000000000001 0.00000000000000000001 0.000000000000000000001 0.0000000000000000000001 ] Definition at line 112 of file Token.d. Referenced by toReal(). |