Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

BoundToken Class Reference

Inheritance diagram for BoundToken:

Token CompositeToken ReaderToken HybridToken List of all members.

Public Member Functions

 this (ITokenizer tk)
ITokenizer getTokenizer ()
bool next (IBuffer buf)

Private Member Functions

Token set (char[] content)
int getLength ()
Token setType (int type)
int getType ()
int toInt ()
long toLong ()
real toReal ()
Token clone ()
char[] toString (bool slice=true)
int opEquals (Object o)
int opCmp (Object o)
uint toHash ()
void write (IWriter w)

Private Attributes

ITokenizer tk

Detailed Description

A style of Token that's bound to a Tokenizer. This can be a handy means of cleaning up client code, and limiting the scope of how a token is used by recieving methods.

Contrast this example with that shown in the Token class:

// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = fc.createBuffer; // bind a line-tokenizer to our input token BoundToken line = new BoundToken (Tokenizers.line); // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (line.next(buffer) || line.getLength) Stdout (line) (CR);

One might also consider a CompositeToken or HybridToken.

Definition at line 290 of file Token.d.


Member Function Documentation

this ITokenizer  tk  )  [inline]
 

Reimplemented in ReaderToken.

Definition at line 298 of file Token.d.

References tk.

ITokenizer getTokenizer  )  [inline]
 

Return the associated tokenizer

Definition at line 309 of file Token.d.

References tk.

bool next IBuffer  buf  )  [inline]
 

Extract the next token from the provided buffer.

Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false.

For example:

while (token.next() || token.getLength()) // do something

Definition at line 335 of file Token.d.

References ITokenizer::next(), and tk.

Referenced by StartLine::VersionUseTokenizer::parse(), HttpParams::parse(), HttpHeaders::parse(), and ResponseLine::parse().

Token set char[]  content  )  [inline, inherited]
 

Set the content of this token.

Definition at line 103 of file Token.d.

References Token::content.

Referenced by TokenStack::clone(), Token::clone(), RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), SimpleTokenizer::next(), Scanner::notFound(), and TokenStack::push().

int getLength  )  [inline, inherited]
 

Return the length of this token.

Definition at line 115 of file Token.d.

Referenced by TextReader::next(), HttpParams::parse(), and HttpHeaders::parse().

Token setType int  type  )  [inline, inherited]
 

Set the type of this token. Token types can be useful when one wishes to categorize input patterns.

Definition at line 127 of file Token.d.

References Token::type.

int getType  )  [inline, inherited]
 

Return the type associated with this token. See setType().

Definition at line 139 of file Token.d.

References Token::type.

int toInt  )  [inline, inherited]
 

Convert this token to an integer.

Definition at line 150 of file Token.d.

Referenced by TextReader::int16(), TextReader::int32(), and TextReader::int8().

long toLong  )  [inline, inherited]
 

Convert this token to a long integer.

Definition at line 161 of file Token.d.

Referenced by TextReader::int64().

real toReal  )  [inline, inherited]
 

Convert this token to a real.

Definition at line 172 of file Token.d.

Referenced by TextReader::float32(), TextReader::float64(), and TextReader::float80().

Token clone  )  [inline, inherited]
 

Clone this token, making a copy of the content also.

Definition at line 183 of file Token.d.

References Token::set(), and Token::type.

char [] toString bool  slice = true  )  [inline, inherited]
 

Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead).

Definition at line 200 of file Token.d.

Referenced by TextReader::get(), StartLine::VersionUseTokenizer::getMethod(), StartLine::VersionUseTokenizer::getPath(), StartLine::VersionUseTokenizer::getProtocol(), TextReader::int1(), TokenStack::isMatch(), ResponseLine::parse(), TokenStack::push(), and HttpTokens::split().

int opEquals Object  o  )  [inline, inherited]
 

Is this token equal to another?

Definition at line 213 of file Token.d.

References Token::content.

int opCmp Object  o  )  [inline, inherited]
 

Compare this token to another.

Definition at line 228 of file Token.d.

References Token::content.

uint toHash  )  [inline, inherited]
 

Hash this token

Definition at line 244 of file Token.d.

void write IWriter  w  )  [inline, inherited]
 

Make the Token class compatible with IWriter instances.

Reimplemented from IWritable.

Definition at line 255 of file Token.d.

References IWriter::put().


Member Data Documentation

ITokenizer tk [private]
 

Definition at line 292 of file Token.d.

Referenced by getTokenizer(), next(), and this().


The documentation for this class was generated from the following file:
Generated on Sat Apr 9 20:11:35 2005 for Mango by doxygen 1.3.6