Inheritance diagram for Token:
Public Member Functions | |
Token | set (char[] content) |
int | getLength () |
Token | setType (int type) |
int | getType () |
int | toInt () |
long | toLong () |
real | toReal () |
Token | clone () |
char[] | toString (bool slice=true) |
int | opEquals (Object o) |
int | opCmp (Object o) |
uint | toHash () |
void | write (IWriter w) |
Private Attributes | |
int | type |
char[] | content |
Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.
Here's a brief example of how to apply Token with Tokenizers:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = new Buffer (fc); // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength) Stdout (token) (CR);
See also BoundToken, ReaderToken, CompositeToken and HybridToken.
Definition at line 92 of file io/Token.d.
|
Set the content of this token. Definition at line 103 of file io/Token.d. References content. Referenced by TokenStack::clone(), clone(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), TokenStack::push(), and TokenTemplate(). |
|
Return the length of this token. Definition at line 115 of file io/Token.d. References content. Referenced by TextReader::next(), HttpParams::parse(), and HttpHeaders::parse(). |
|
Set the type of this token. Token types can be useful when one wishes to categorize input patterns. Definition at line 127 of file io/Token.d. |
|
Return the type associated with this token. See setType(). Definition at line 139 of file io/Token.d. References type. |
|
Convert this token to an integer. Definition at line 150 of file io/Token.d. Referenced by TextReader::read(). |
|
Convert this token to a long integer. Definition at line 161 of file io/Token.d. Referenced by TextReader::read(). |
|
Convert this token to a real. Definition at line 172 of file io/Token.d. References content, and Double. Referenced by TextReader::read(). |
|
Clone this token, making a copy of the content also. Definition at line 183 of file io/Token.d. References clone(), set(), toString(), and type. Referenced by clone(). |
|
Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead). Definition at line 200 of file io/Token.d. References content. Referenced by clone(), TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TokenStack::isMatch(), ResponseLine::parse(), TokenStack::push(), TextReader::read(), HttpTokens::split(), and HttpTokens::write(). |
|
Is this token equal to another? Definition at line 213 of file io/Token.d. References content. Referenced by TokenTemplate(). |
|
Compare this token to another. Definition at line 228 of file io/Token.d. References content. Referenced by TokenTemplate(). |
|
Hash this token Definition at line 244 of file io/Token.d. References content. |
|
Make the Token class compatible with IWriter instances. Reimplemented from IWritable. Definition at line 255 of file io/Token.d. References content, and IWriter::put(). |
|
Definition at line 94 of file io/Token.d. Referenced by clone(), getType(), setType(), and TokenTemplate(). |
|
Definition at line 95 of file io/Token.d. Referenced by getLength(), opCmp(), opEquals(), set(), toHash(), toInt(), TokenTemplate(), toLong(), toReal(), toString(), and write(). |