Inheritance diagram for Token:
Public Member Functions | |
Token | set (char[] content) |
int | getLength () |
Token | setType (int type) |
int | getType () |
int | toInt () |
long | toLong () |
real | toReal () |
Token | clone () |
char[] | toString (bool slice=true) |
int | opEquals (Object o) |
int | opCmp (Object o) |
uint | toHash () |
void | write (IWriter w) |
Private Attributes | |
int | type |
char[] | content |
Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.
Here's a brief example of how to apply Token with Tokenizers:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer; // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength) Stdout (token) (CR);
See also BoundToken, ReaderToken, CompositeToken and HybridToken.
Definition at line 92 of file Token.d.
|
Set the content of this token. Definition at line 103 of file Token.d. References content. Referenced by TokenStack::clone(), clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push(). |
|
Return the length of this token. Definition at line 115 of file Token.d. Referenced by TextReader::next(), HttpParams::parse(), and HttpHeaders::parse(). |
|
Set the type of this token. Token types can be useful when one wishes to categorize input patterns. Definition at line 127 of file Token.d. References type. |
|
Return the type associated with this token. See setType(). Definition at line 139 of file Token.d. References type. |
|
Convert this token to an integer. Definition at line 150 of file Token.d. Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8(). |
|
Convert this token to a long integer. Definition at line 161 of file Token.d. Referenced by TextReader::int64(). |
|
Convert this token to a real. Definition at line 172 of file Token.d. Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80(). |
|
Clone this token, making a copy of the content also. |
|
Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead). Definition at line 200 of file Token.d. Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split(). |
|
Is this token equal to another? Definition at line 213 of file Token.d. References content. |
|
Compare this token to another. Definition at line 228 of file Token.d. References content. |
|
Hash this token |
|
Make the Token class compatible with IWriter instances. Reimplemented from IWritable. Definition at line 255 of file Token.d. References IWriter::put(). |
|
|
|
Definition at line 95 of file Token.d. Referenced by opCmp(), opEquals(), and set(). |