Go to the source code of this file.
Functions | |
class | TokenTemplate (T) |
typedef | TokenTemplate (char) Token |
class | BoundTokenTemplate (T) |
typedef | BoundTokenTemplate (char) BoundToken |
class | ReaderTokenTemplate (T) |
typedef | ReaderTokenTemplate (char) ReaderToken |
class | CompositeTokenTemplate (T) |
typedef | CompositeTokenTemplate (char) CompositeToken |
class | HybridTokenTemplate (T) |
typedef | HybridTokenTemplate (char) HybridToken |
Variables | |
module mango io | Token |
import mango io | Buffer |
import mango io mango io | Tokenizer |
import mango convert | Atoi |
import mango convert mango convert | Double |
import mango io model | IWriter |
import mango io model mango io model | IReader |
import mango io model mango io model mango io model | IConduit |
|
Tokens used by Tokenizer class. Tokens do not copy their content so they are quite useful for parsing quantites of data quickly. Conversely since each token is mapped into an external buffer, you should be aware that changes to said buffer will impact any tokens based upon it. You may sidestep this by using the clone() method, or toString().dup Tokens can convert from a variety of numeric format to ascii text. Formats currently include int, uint, long, ulong, and real. Each number may be preceded by whitespace, and an optional '+' or '-' specifier. Note that real-number format is simplistic in that it does not support exponential declarations. Note the conversion methods should probably be moved elsewhere. Here's a brief example of how to apply Token with Tokenizers:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = new Buffer (fc); // create a token for receiving the line Token token = new Token; // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (Tokenizers.line.next(buffer, token) || token.getLength) Stdout (token) (CR); See also BoundToken, ReaderToken, CompositeToken and HybridToken. Set the content of this token. Return the length of this token. Set the type of this token. Token types can be useful when one wishes to categorize input patterns. Return the type associated with this token. See setType(). Convert this token to an integer. Convert this token to a long integer. Convert this token to a real. Clone this token, making a copy of the content also. Return a reference to this tokens content. Duplicate it only if 'slice' is explicitly set to false (defaults to a slice instead). Is this token equal to another? Compare this token to another. Hash this token Make the Token class compatible with IWriter instances. Definition at line 92 of file Copy of Token.d. References AtoiTemplate(), Token::content, DoubleTemplate(), Token::opCmp(), Token::opEquals(), IWriter::put(), Token::set(), TokenTemplate(), Token::type, type(), and IWritable::write(). |
|
|
|
A style of Token that's bound to a Tokenizer. This can be a handy means of cleaning up client code, and limiting the scope of how a token is used by recieving methods. Contrast this example with that shown in the Token class:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a buffer for reading the file IBuffer buffer = new Buffer(fc); // bind a line-tokenizer to our input token BoundToken line = new BoundToken (Tokenizers.line); // read file a line at a time. Method next() returns false when no more // delimiters are found. Note there may be an unterminated line at eof while (line.next(buffer) || line.getLength) Stdout (line) (CR); One might also consider a CompositeToken or HybridToken. Return the associated tokenizer Extract the next token from the provided buffer. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an EOF condition is seen. Check token.getLength() when this method returns false. For example:
while (token.next() || token.getLength()) // do something Definition at line 292 of file Copy of Token.d. References BoundTokenTemplate(), and TokenTemplate(). Referenced by BoundTokenTemplate(), CompositeTokenTemplate(), and ReaderTokenTemplate(). |
|
|
|
ReaderToken adapts a BoundToken such that it can be used directly with any IReader implementation. We just add the IReadable methods to the basic BoundToken. Here's a contrived example of how to use ReaderToken:
// create a small buffer on the heap Buffer buf = new Buffer (256); // write items with a comma between each TextWriter write = new TextWriter (buf, ","); // write some stuff to the buffer write ("now is the time for all good men") (3.14159); // bind a couple of tokens to a comma tokenizer ReaderToken text = new ReaderToken (Tokenizers.comma); ReaderToken number = new ReaderToken (Tokenizers.comma); // create any old reader since we only use it for handling tokens Reader read = new Reader (buf); // populate both tokens via reader read (text) (number); // print them to the console Stdout (text) (':') (number) (CR); Construct a ReaderToken using the provided Tokenizer. Read the next delimited element into this token. Definition at line 380 of file Copy of Token.d. References BoundTokenTemplate(), IReader::getBuffer(), IReadable::read(), and ReaderTokenTemplate(). Referenced by ReaderTokenTemplate(). |
|
|
|
Another subclass of BoundToken that combines both a Tokenizer and an input buffer. This is simply a convenience wrapper than takes care of details that would otherwise clutter the client code. Compare this to usage of a basic Token:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a Token and bind it to both the file and a line-tokenizer CompositeToken line = new CompositeToken (Tokenizers.line, fc); // read file a line at a time. Method get() returns false when no more // tokens are found. while (line.get) Stdout (line) (CR); You might also consider a HybridToken for further processing of token content. Set this token to use the provided Tokenizer, and bind it to the given buffer. Set this token to use the provided Tokenizer, and bind it to the buffer associated with the given conduit. Return the associated buffer Extract the next token. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an Eof condition is seen. Check token.getLength() when this method returns false. For example:
while (token.next || token.getLength) // do something Extract the next token, taking Eof into consideration. If next() returns false, then this function will still return true as long as there's some content available. For example:
while (token.get) // do something Definition at line 434 of file Copy of Token.d. References BoundTokenTemplate(), Buffer, RawCodec1::buffer, buffer, and CompositeTokenTemplate(). Referenced by CompositeTokenTemplate(), and HybridTokenTemplate(). |
|
|
|
A subclass of CompositeToken that combines a Tokenizer, an input buffer, and the means to bind its content to a subordinate Reader or Token. This is another convenience wrapper than takes care of details that would otherwise complicate client code. Compare this to usage of a CompositeToken:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a Token and bind it to both the file and a line-tokenizer HybridToken line = new HybridToken (Tokenizers.line, fc); // now create a reader upon the token Reader input = new Reader (line.getHost); // read file a line at a time. Method get() returns false when no more // tokens are found. while (line.get) { int x, y; // reader is now bound to the content of the current line input (x) (y); Stdout (x) (y) (CR); } You can use the same mechanism to bind subordinate Tokens:
// open a file for reading FileConduit fc = new FileConduit ("test.txt"); // create a Token and bind it to both the file and a line-tokenizer HybridToken line = new HybridToken (Tokenizers.line, fc); // now create a subordinate Token that splits on whitespace CompositeToken word = new CompositeToken (Tokenizers.space, line.getHost); // read file a line at a time. Method get() returns false when no more // tokens are found. while (line.get) // extract space delimited tokens from each line while (word.get) Stdout (word) (CR); Set this token to use the provided Tokenizer, and bind it to the given buffer. Set this token to use the provided Tokenizer, and bind it to the buffer associated with the given conduit. Return the associated host buffer. The host should be used for purposes of binding a subordinate Token or Reader onto the content of this token. Each call to next() will update this content appropriately, which is also reflected within said host buffer. That is, token.toString == token.getHost.toString. Extract the next token. Returns true if a token was isolated, false if no more tokens were found. Note that one last token may still be present when this return false; this may happen if (for example) the last delimiter is missing before an Eof condition is seen. Check token.getLength() when this method returns false. For example:
while (token.next || token.getLength) // do something Definition at line 580 of file Copy of Token.d. References Buffer, buffer, CompositeTokenTemplate(), and HybridTokenTemplate(). Referenced by HybridTokenTemplate(). |
|
|
|
Definition at line 42 of file Copy of Token.d. |
|
Definition at line 44 of file Copy of Token.d. |
|
Definition at line 44 of file Copy of Token.d. |
|
Definition at line 47 of file Copy of Token.d. |
|
Definition at line 47 of file Copy of Token.d. |
|
Definition at line 50 of file Copy of Token.d. |
|
Definition at line 50 of file Copy of Token.d. |
|
Definition at line 50 of file Copy of Token.d. |