Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

Token Class Reference

Inheritance diagram for Token:

IWritable BoundToken CompositeToken ReaderToken HybridToken List of all members.

Public Member Functions

Token set (char[] content)
int getLength ()
Token setType (int type)
int getType ()
int toInt ()
long toLong ()
real toReal ()
Token clone ()
char[] toString (bool slice=true)
int opEquals (Object o)
int opCmp (Object o)
uint toHash ()
void write (IWriter w)

Private Attributes

int type
char[] content

Detailed Description

Tokens used by Tokenizer class. Tokens do not copy their content so they are quite useful for parsing quantites of data quickly. Conversely since each token is mapped into an external buffer, you should be aware that changes to said buffer will impact any tokens based upon it. You may sidestep this by using the clone() method, or toString().dup

Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.

Here's a brief example of how to apply Token with Tokenizers:

// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer; // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength) Stdout (token) (CR);

See also BoundToken, ReaderToken, CompositeToken and HybridToken.

Definition at line 92 of file Token.d.


Member Function Documentation

Token set char[]  content  )  [inline]
 

Set the content of this token.

Definition at line 103 of file Token.d.

References content.

Referenced by TokenStack::clone(), clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push().

int getLength  )  [inline]
 

Return the length of this token.

Definition at line 115 of file Token.d.

Referenced by TextReader::next(), HttpParams::parse(), and HttpHeaders::parse().

Token setType int  type  )  [inline]
 

Set the type of this token. Token types can be useful when one wishes to categorize input patterns.

Definition at line 127 of file Token.d.

References type.

int getType  )  [inline]
 

Return the type associated with this token. See setType().

Definition at line 139 of file Token.d.

References type.

int toInt  )  [inline]
 

Convert this token to an integer.

Definition at line 150 of file Token.d.

Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8().

long toLong  )  [inline]
 

Convert this token to a long integer.

Definition at line 161 of file Token.d.

Referenced by TextReader::int64().

real toReal  )  [inline]
 

Convert this token to a real.

Definition at line 172 of file Token.d.

Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80().

Token clone  )  [inline]
 

Clone this token, making a copy of the content also.

Definition at line 183 of file Token.d.

References set(), and type.

char [] toString bool  slice = true  )  [inline]
 

Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead).

Definition at line 200 of file Token.d.

Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split().

int opEquals Object  o  )  [inline]
 

Is this token equal to another?

Definition at line 213 of file Token.d.

References content.

int opCmp Object  o  )  [inline]
 

Compare this token to another.

Definition at line 228 of file Token.d.

References content.

uint toHash  )  [inline]
 

Hash this token

Definition at line 244 of file Token.d.

void write IWriter  w  )  [inline]
 

Make the Token class compatible with IWriter instances.

Reimplemented from IWritable.

Definition at line 255 of file Token.d.

References IWriter::put().


Member Data Documentation

int type [private]
 

Definition at line 94 of file Token.d.

Referenced by clone(), getType(), and setType().

char [] content [private]
 

Definition at line 95 of file Token.d.

Referenced by opCmp(), opEquals(), and set().


The documentation for this class was generated from the following file:
Generated on Sun Mar 6 00:31:15 2005 for Mango by doxygen 1.3.6