Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

Token Class Reference

Inheritance diagram for Token:

IWritable BoundToken CompositeToken ReaderToken HybridToken List of all members.

Public Types

typedef void(* Accumulator )(int digit)

Public Member Functions

Token set (char[] content)
int getLength ()
Token setType (int type)
int getType ()
int toInt ()
uint toUInt ()
long toLong ()
ulong toULong ()
real toReal ()
Token clone ()
char[] toString (bool slice=true)
int opEquals (Object o)
int opCmp (Object o)
uint toHash ()
void write (IWriter w)

Static Public Member Functions

 this ()

Private Types

enum  State { Begin, Number, Fraction }

Private Member Functions

bool parseDigits (Accumulator dg, Accumulator fraction)

Private Attributes

int type
char[] content

Static Private Attributes

TokenException Error
real divisors []

Detailed Description

Tokens used by Tokenizer class. Tokens do not copy their content so they are quite useful for parsing quantites of data quickly. Conversely since each token is mapped into an external buffer, you should be aware that changes to said buffer will impact any tokens based upon it. You may sidestep this by using the clone() method, or toString().dup

Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.

Here's a brief example of how to apply Token with Tokenizers:

// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer (); // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength()) Stdout.put(token).cr();

See also BoundToken, ReaderToken, CompositeToken and HybridToken.

Definition at line 94 of file Token.d.


Member Typedef Documentation

typedef void(* Accumulator)(int digit)
 

Definition at line 103 of file Token.d.

Referenced by parseDigits().


Member Enumeration Documentation

enum State [private]
 

Enumeration values:
Begin 
Number 
Fraction 

Definition at line 101 of file Token.d.


Member Function Documentation

this  )  [inline, static]
 

Construct a static exception instance.

Definition at line 141 of file Token.d.

References Error.

Token set char[]  content  )  [inline]
 

Set the content of this token.

Definition at line 152 of file Token.d.

References content.

Referenced by TokenStack::clone(), clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push().

int getLength  )  [inline]
 

Return the length of this token.

Definition at line 164 of file Token.d.

Referenced by CompositeToken::get(), TextReader::next(), HttpParams::parse(), HttpHeaders::parse(), testFile2(), and testFile3().

Token setType int  type  )  [inline]
 

Set the type of this token. Token types can be useful when one wishes to categorize input patterns.

Definition at line 176 of file Token.d.

References type.

int getType  )  [inline]
 

Return the type associated with this token. See setType().

Definition at line 188 of file Token.d.

References type.

int toInt  )  [inline]
 

Convert this token to an integer.

Definition at line 199 of file Token.d.

References parseDigits().

Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8().

uint toUInt  )  [inline]
 

Convert this token to an unsigned integer.

Definition at line 217 of file Token.d.

References Error, and parseDigits().

long toLong  )  [inline]
 

Convert this token to a long integer.

Definition at line 237 of file Token.d.

References parseDigits().

Referenced by TextReader::int64().

ulong toULong  )  [inline]
 

Convert this token to an unsigned long integer.

Definition at line 255 of file Token.d.

References Error, and parseDigits().

real toReal  )  [inline]
 

Convert this token to a real.

Definition at line 275 of file Token.d.

References divisors, Error, and parseDigits().

Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80().

Token clone  )  [inline]
 

Clone this token, making a copy of the content also.

Definition at line 309 of file Token.d.

References set(), and type.

char [] toString bool  slice = true  )  [inline]
 

Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead).

Definition at line 326 of file Token.d.

Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), HybridToken::next(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split().

int opEquals Object  o  )  [inline]
 

Is this token equal to another?

Definition at line 339 of file Token.d.

References content.

int opCmp Object  o  )  [inline]
 

Compare this token to another.

Definition at line 355 of file Token.d.

References content.

uint toHash  )  [inline]
 

Hash this token

Definition at line 372 of file Token.d.

void write IWriter  w  )  [inline]
 

Make the Token class compatible with IWriter instances.

Reimplemented from IWritable.

Definition at line 384 of file Token.d.

References IWriter::put().

bool parseDigits Accumulator  dg,
Accumulator  fraction
[inline, private]
 

Parse a set of digits, optionally prefixed by a '+' or '-', and optionally with a '.' contained within. Delegates come in quite handy here, and add relatively little overhead.

Definition at line 397 of file Token.d.

References Accumulator, and Error.

Referenced by toInt(), toLong(), toReal(), toUInt(), and toULong().


Member Data Documentation

int type [private]
 

Definition at line 96 of file Token.d.

Referenced by clone(), getType(), and setType().

char [] content [private]
 

Definition at line 97 of file Token.d.

Referenced by opCmp(), opEquals(), and set().

TokenException Error [static, private]
 

Definition at line 99 of file Token.d.

Referenced by parseDigits(), this(), toReal(), toUInt(), and toULong().

real divisors[] [static, private]
 

Initial value:

 
                        [
                        0.1 
                        0.01 
                        0.001 
                        0.0001 
                        0.00001 
                        0.000001 
                        0.0000001 
                        0.00000001 
                        0.000000001 
                        0.0000000001 
                        0.00000000001 
                        0.000000000001 
                        0.0000000000001 
                        0.00000000000001 
                        0.000000000000001 
                        0.0000000000000001 
                        0.00000000000000001
                        0.000000000000000001
                        0.0000000000000000001
                        0.00000000000000000001
                        0.000000000000000000001
                        0.0000000000000000000001
                        ]

Definition at line 109 of file Token.d.

Referenced by toReal().


The documentation for this class was generated from the following file:
Generated on Sun Nov 7 19:07:12 2004 for Mango by doxygen 1.3.6