Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

BoundToken Class Reference

Inheritance diagram for BoundToken:

Token CompositeToken ReaderToken HybridToken List of all members.

Public Member Functions

 this (ITokenizer tk)
ITokenizer getTokenizer ()
bool next (IBuffer buf)

Private Types

typedef void(* Accumulator )(int digit)

Private Member Functions

Token set (char[] content)
int getLength ()
Token setType (int type)
int getType ()
int toInt ()
uint toUInt ()
long toLong ()
ulong toULong ()
real toReal ()
Token clone ()
char[] toString (bool slice=true)
int opEquals (Object o)
int opCmp (Object o)
uint toHash ()
void write (IWriter w)

Static Private Member Functions

 this ()

Private Attributes

ITokenizer tk

Detailed Description

A style of Token that's bound to a Tokenizer. This can be a handy means of cleaning up client code, and limiting the scope of how a token is used by recieving methods.

Contrast this example with that shown in the Token class:

// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer(); // bind a line-tokenizer to our input token BoundToken line = new BoundToken (Tokenizers.line); // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (line.next(buffer) || line.getLength()) Stdout.put(line).cr();

One might also consider a CompositeToken or HybridToken.

Definition at line 73 of file TokenEx.d.


Member Typedef Documentation

typedef void(* Accumulator)(int digit) [inherited]
 

Definition at line 103 of file Token.d.

Referenced by Token::parseDigits().


Member Function Documentation

this ITokenizer  tk  )  [inline]
 

Reimplemented in ReaderToken.

Definition at line 81 of file TokenEx.d.

References tk.

ITokenizer getTokenizer  )  [inline]
 

Return the associated tokenizer

Definition at line 92 of file TokenEx.d.

References tk.

bool next IBuffer  buf  )  [inline]
 

Extract the next token from the provided buffer.

Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false.

For example:

while (token.next() || token.getLength()) // do something

Definition at line 118 of file TokenEx.d.

References ITokenizer::next(), and tk.

Referenced by StartLine::VersionUseTokenizer::parse(), HttpParams::parse(), HttpHeaders::parse(), ResponseLine::parse(), testFile2(), and testToken1().

this  )  [inline, static, inherited]
 

Construct a static exception instance.

Definition at line 141 of file Token.d.

References Token::Error.

Token set char[]  content  )  [inline, inherited]
 

Set the content of this token.

Definition at line 152 of file Token.d.

References Token::content.

Referenced by TokenStack::clone(), Token::clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push().

int getLength  )  [inline, inherited]
 

Return the length of this token.

Definition at line 164 of file Token.d.

Referenced by CompositeToken::get(), TextReader::next(), HttpParams::parse(), HttpHeaders::parse(), testFile2(), and testFile3().

Token setType int  type  )  [inline, inherited]
 

Set the type of this token. Token types can be useful when one wishes to categorize input patterns.

Definition at line 176 of file Token.d.

References Token::type.

int getType  )  [inline, inherited]
 

Return the type associated with this token. See setType().

Definition at line 188 of file Token.d.

References Token::type.

int toInt  )  [inline, inherited]
 

Convert this token to an integer.

Definition at line 199 of file Token.d.

References Token::parseDigits().

Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8().

uint toUInt  )  [inline, inherited]
 

Convert this token to an unsigned integer.

Definition at line 217 of file Token.d.

References Token::Error, and Token::parseDigits().

long toLong  )  [inline, inherited]
 

Convert this token to a long integer.

Definition at line 237 of file Token.d.

References Token::parseDigits().

Referenced by TextReader::int64().

ulong toULong  )  [inline, inherited]
 

Convert this token to an unsigned long integer.

Definition at line 255 of file Token.d.

References Token::Error, and Token::parseDigits().

real toReal  )  [inline, inherited]
 

Convert this token to a real.

Definition at line 275 of file Token.d.

References Token::divisors, Token::Error, and Token::parseDigits().

Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80().

Token clone  )  [inline, inherited]
 

Clone this token, making a copy of the content also.

Definition at line 309 of file Token.d.

References Token::set(), and Token::type.

char [] toString bool  slice = true  )  [inline, inherited]
 

Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead).

Definition at line 326 of file Token.d.

Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), HybridToken::next(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split().

int opEquals Object  o  )  [inline, inherited]
 

Is this token equal to another?

Definition at line 339 of file Token.d.

References Token::content.

int opCmp Object  o  )  [inline, inherited]
 

Compare this token to another.

Definition at line 355 of file Token.d.

References Token::content.

uint toHash  )  [inline, inherited]
 

Hash this token

Definition at line 372 of file Token.d.

void write IWriter  w  )  [inline, inherited]
 

Make the Token class compatible with IWriter instances.

Reimplemented from IWritable.

Definition at line 384 of file Token.d.

References IWriter::put().


Member Data Documentation

ITokenizer tk [private]
 

Definition at line 75 of file TokenEx.d.

Referenced by getTokenizer(), next(), and this().


The documentation for this class was generated from the following file:
Generated on Sun Nov 7 19:06:59 2004 for Mango by doxygen 1.3.6