Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

Token Class Reference

Inheritance diagram for Token:

IWritable BoundToken CompositeToken ReaderToken HybridToken List of all members.

Public Member Functions

Token set (char[] content)
int getLength ()
Token setType (int type)
int getType ()
int toInt ()
long toLong ()
real toReal ()
Token clone ()
char[] toString (bool slice=true)
int opEquals (Object o)
int opCmp (Object o)
uint toHash ()
void write (IWriter w)

Private Attributes

int type
char[] content

Detailed Description

Tokens used by Tokenizer class. Tokens do not copy their content so they are quite useful for parsing quantites of data quickly. Conversely since each token is mapped into an external buffer, you should be aware that changes to said buffer will impact any tokens based upon it. You may sidestep this by using the clone() method, or toString().dup

Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.

Here's a brief example of how to apply Token with Tokenizers:

        // open a file for reading
        FileConduit fc = new FileConduit ("test.txt");

        // create a buffer for reading the file
        IBuffer buffer = fc.createBuffer;

        // create a token for receiving the line
        Token token = new Token;

        // read file a line at a time. Method next() returns false when no more 
        // delimiters are found. Note there may be an unterminated line at eof
        while (Tokenizers.line.next(buffer, token) || token.getLength)
               Stdout (token) (CR);

See also BoundToken, ReaderToken, CompositeToken and HybridToken.

Definition at line 92 of file Token.d.


Member Function Documentation

Token set char[]  content  )  [inline]
 

Set the content of this token.

Definition at line 103 of file Token.d.

References content.

Referenced by TokenStack::clone(), clone(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push().

int getLength  )  [inline]
 

Return the length of this token.

Definition at line 115 of file Token.d.

References content.

Referenced by TextReader::next(), HttpParams::parse(), HttpHeaders::parse(), testFile2(), and testFile3().

Token setType int  type  )  [inline]
 

Set the type of this token. Token types can be useful when one wishes to categorize input patterns.

Definition at line 127 of file Token.d.

References type.

int getType  )  [inline]
 

Return the type associated with this token. See setType().

Definition at line 139 of file Token.d.

References type.

int toInt  )  [inline]
 

Convert this token to an integer.

Definition at line 150 of file Token.d.

References content.

Referenced by TextReader::read().

long toLong  )  [inline]
 

Convert this token to a long integer.

Definition at line 161 of file Token.d.

References content.

Referenced by TextReader::read().

real toReal  )  [inline]
 

Convert this token to a real.

Definition at line 172 of file Token.d.

References content.

Referenced by TextReader::read().

Token clone  )  [inline]
 

Clone this token, making a copy of the content also.

Definition at line 183 of file Token.d.

References clone(), set(), toString(), and type.

Referenced by clone().

char [] toString bool  slice = true  )  [inline]
 

Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead).

Definition at line 200 of file Token.d.

References content.

Referenced by clone(), TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TokenStack::isMatch(), ResponseLine::parse(), TokenStack::push(), TextReader::read(), and HttpTokens::split().

int opEquals Object  o  )  [inline]
 

Is this token equal to another?

Definition at line 213 of file Token.d.

References content.

int opCmp Object  o  )  [inline]
 

Compare this token to another.

Definition at line 228 of file Token.d.

References content.

uint toHash  )  [inline]
 

Hash this token

Definition at line 244 of file Token.d.

References content.

void write IWriter  w  )  [inline]
 

Make the Token class compatible with IWriter instances.

Reimplemented from IWritable.

Definition at line 255 of file Token.d.

References content, and IWriter::put().


Member Data Documentation

int type [private]
 

Definition at line 94 of file Token.d.

Referenced by clone(), getType(), and setType().

char [] content [private]
 

Definition at line 95 of file Token.d.

Referenced by getLength(), opCmp(), opEquals(), set(), toHash(), toInt(), toLong(), toReal(), toString(), and write().


The documentation for this class was generated from the following file:
Generated on Mon Nov 14 11:00:03 2005 for Mango by  doxygen 1.4.0