Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

Copy of Token.d File Reference

Go to the source code of this file.

Functions

class TokenTemplate (T)
typedef TokenTemplate (char) Token
class BoundTokenTemplate (T)
typedef BoundTokenTemplate (char) BoundToken
class ReaderTokenTemplate (T)
typedef ReaderTokenTemplate (char) ReaderToken
class CompositeTokenTemplate (T)
typedef CompositeTokenTemplate (char) CompositeToken
class HybridTokenTemplate (T)
typedef HybridTokenTemplate (char) HybridToken

Variables

module mango io Token
import mango io Buffer
import mango io mango io Tokenizer
import mango convert Atoi
import mango convert mango
convert 
Double
import mango io model IWriter
import mango io model mango
io model 
IReader
import mango io model mango
io model mango io model 
IConduit


Function Documentation

class TokenTemplate  ) 
 

Tokens used by Tokenizer class. Tokens do not copy their content so they are quite useful for parsing quantites of data quickly. Conversely since each token is mapped into an external buffer, you should be aware that changes to said buffer will impact any tokens based upon it. You may sidestep this by using the clone() method, or toString().dup

Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere.

Here's a brief example of how to apply Token with Tokenizers:

        // open a file for reading
        FileConduit fc = new FileConduit ("test.txt");

        // create a buffer for reading the file
        IBuffer buffer = new Buffer (fc);

        // create a token for receiving the line
        Token token = new Token;

        // read file a line at a time. Method next() returns false when no more 
        // delimiters are found. Note there may be an unterminated line at eof
        while (Tokenizers.line.next(buffer, token) || token.getLength)
               Stdout (token) (CR);

See also BoundToken, ReaderToken, CompositeToken and HybridToken.

Set the content of this token.

Return the length of this token.

Set the type of this token. Token types can be useful when one wishes to categorize input patterns.

Return the type associated with this token. See setType().

Convert this token to an integer.

Convert this token to a long integer.

Convert this token to a real.

Clone this token, making a copy of the content also.

Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead).

Is this token equal to another?

Compare this token to another.

Hash this token

Make the Token class compatible with IWriter instances.

Definition at line 92 of file Copy of Token.d.

References AtoiTemplate(), Token::content, DoubleTemplate(), Token::opCmp(), Token::opEquals(), IWriter::put(), Token::set(), TokenTemplate(), Token::type, type(), and IWritable::write().

typedef TokenTemplate char   ) 
 

class BoundTokenTemplate  ) 
 

A style of Token that's bound to a Tokenizer. This can be a handy means of cleaning up client code, and limiting the scope of how a token is used by recieving methods.

Contrast this example with that shown in the Token class:

        // open a file for reading
        FileConduit fc = new FileConduit ("test.txt");

        // create a buffer for reading the file
        IBuffer buffer = new Buffer(fc);

        // bind a line-tokenizer to our input token
        BoundToken line = new BoundToken (Tokenizers.line);

        // read file a line at a time. Method next() returns false when no more 
        // delimiters are found. Note there may be an unterminated line at eof
        while (line.next(buffer) || line.getLength)
               Stdout (line) (CR);

One might also consider a CompositeToken or HybridToken.

Return the associated tokenizer

Extract the next token from the provided buffer.

Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false.

For example:

                        while (token.next() || token.getLength())
                               // do something

Definition at line 292 of file Copy of Token.d.

References BoundTokenTemplate(), and TokenTemplate().

Referenced by BoundTokenTemplate(), CompositeTokenTemplate(), and ReaderTokenTemplate().

typedef BoundTokenTemplate char   ) 
 

class ReaderTokenTemplate  ) 
 

ReaderToken adapts a BoundToken such that it can be used directly with any IReader implementation. We just add the IReadable methods to the basic BoundToken.

Here's a contrived example of how to use ReaderToken:

        // create a small buffer on the heap
        Buffer buf = new Buffer (256);

        // write items with a comma between each
        TextWriter write = new TextWriter (buf, ",");

        // write some stuff to the buffer
        write ("now is the time for all good men") (3.14159);

        // bind a couple of tokens to a comma tokenizer
        ReaderToken text = new ReaderToken (Tokenizers.comma);
        ReaderToken number = new ReaderToken (Tokenizers.comma);
        
        // create any old reader since we only use it for handling tokens
        Reader read = new Reader (buf);

        // populate both tokens via reader 
        read (text) (number);

        // print them to the console
        Stdout (text) (':') (number) (CR);

Construct a ReaderToken using the provided Tokenizer.

Read the next delimited element into this token.

Definition at line 380 of file Copy of Token.d.

References BoundTokenTemplate(), IReader::getBuffer(), IReadable::read(), and ReaderTokenTemplate().

Referenced by ReaderTokenTemplate().

typedef ReaderTokenTemplate char   ) 
 

class CompositeTokenTemplate  ) 
 

Another subclass of BoundToken that combines both a Tokenizer and an input buffer. This is simply a convenience wrapper than takes care of details that would otherwise clutter the client code.

Compare this to usage of a basic Token:

        // open a file for reading
        FileConduit fc = new FileConduit ("test.txt");

        // create a Token and bind it to both the file and a line-tokenizer
        CompositeToken line = new CompositeToken (Tokenizers.line, fc);

        // read file a line at a time. Method get() returns false when no more 
        // tokens are found. 
        while (line.get)
               Stdout (line) (CR);

You might also consider a HybridToken for further processing of token content.

Set this token to use the provided Tokenizer, and bind it to the given buffer.

Set this token to use the provided Tokenizer, and bind it to the buffer associated with the given conduit.

Return the associated buffer

Extract the next token.

Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an Eof condition is seen. Check token.getLength() when this method returns false.

For example:

                        while (token.next || token.getLength)
                               // do something

Extract the next token, taking Eof into consideration. If next() returns false, then this function will still return true as long as there's some content available.

For example:

                        while (token.get)
                               // do something

Definition at line 434 of file Copy of Token.d.

References BoundTokenTemplate(), Buffer, RawCodec1::buffer, buffer, and CompositeTokenTemplate().

Referenced by CompositeTokenTemplate(), and HybridTokenTemplate().

typedef CompositeTokenTemplate char   ) 
 

class HybridTokenTemplate  ) 
 

A subclass of CompositeToken that combines a Tokenizer, an input buffer, and the means to bind its content to a subordinate Reader or Token. This is another convenience wrapper than takes care of details that would otherwise complicate client code.

Compare this to usage of a CompositeToken:

        // open a file for reading
        FileConduit fc = new FileConduit ("test.txt");

        // create a Token and bind it to both the file and a line-tokenizer
        HybridToken line = new HybridToken (Tokenizers.line, fc);

        // now create a reader upon the token
        Reader input = new Reader (line.getHost);

        // read file a line at a time. Method get() returns false when no more 
        // tokens are found. 
        while (line.get)
              {
              int x, y;
                
              // reader is now bound to the content of the current line
              input (x) (y);

              Stdout (x) (y) (CR);
              }

You can use the same mechanism to bind subordinate Tokens:

        // open a file for reading
        FileConduit fc = new FileConduit ("test.txt");

        // create a Token and bind it to both the file and a line-tokenizer
        HybridToken line = new HybridToken (Tokenizers.line, fc);

        // now create a subordinate Token that splits on whitespace
        CompositeToken word = new CompositeToken (Tokenizers.space, line.getHost);

        // read file a line at a time. Method get() returns false when no more 
        // tokens are found. 
        while (line.get)
               // extract space delimited tokens from each line
               while (word.get)
                      Stdout (word) (CR);

Set this token to use the provided Tokenizer, and bind it to the given buffer.

Set this token to use the provided Tokenizer, and bind it to the buffer associated with the given conduit.

Return the associated host buffer. The host should be used for purposes of binding a subordinate Token or Reader onto the content of this token. Each call to next() will update this content appropriately, which is also reflected within said host buffer.

That is, token.toString == token.getHost.toString.

Extract the next token.

Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an Eof condition is seen. Check token.getLength() when this method returns false.

For example:

                        while (token.next || token.getLength)
                               // do something

Definition at line 580 of file Copy of Token.d.

References Buffer, buffer, CompositeTokenTemplate(), and HybridTokenTemplate().

Referenced by HybridTokenTemplate().

typedef HybridTokenTemplate char   ) 
 


Variable Documentation

module mango io Token
 

Definition at line 42 of file Copy of Token.d.

import mango io Buffer
 

Definition at line 44 of file Copy of Token.d.

import mango io mango io Tokenizer
 

Definition at line 44 of file Copy of Token.d.

import mango convert Atoi
 

Definition at line 47 of file Copy of Token.d.

import mango convert mango convert Double
 

Definition at line 47 of file Copy of Token.d.

import mango io model IWriter
 

Definition at line 50 of file Copy of Token.d.

import mango io model mango io model IReader
 

Definition at line 50 of file Copy of Token.d.

import mango io model mango io model mango io model IConduit
 

Definition at line 50 of file Copy of Token.d.


Generated on Sat Dec 24 17:28:35 2005 for Mango by  doxygen 1.4.0