Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

text/Token.d File Reference

Go to the source code of this file.

Functions

class TokenTemplate (T)

Variables

module mango text Token
import mango text Text


Function Documentation

class TokenTemplate  ) 
 

The base class for a set of tokenizers.

There are two types of tokenizers supported ~ exclusive and inclusive. The former are the more common kind, where a token is delimited by elements that are considered foreign. Examples include space, comma, and end-of-line delineation. Inclusive tokens are just the opposite: they look for patterns in the text that should be part of the token itself ~ everything else is considered foreign. Currently the only inclusive token type is exposed by RegexToken; everything else is of the exclusive variety.

The content provided to Tokenizers is supposed to be entirely read-only. All current tokenizers abide by this rule, but it's possible a user could mutate the content through a token slice. To enforce the desired read-only aspect, the code would have to introduce redundant copying or the compiler would have to support read-only arrays.

See LineToken, CharToken, RegexToken, QuotedToken, and SetToken.

Locate the next token. If this Token is configured for "refills", more content will be requested as needed.

Returns true if a token is found; false otherwise.

Construct a token upon the given content. Automatic refills are disabled

Iterate over the set of tokens. This provides read-only access to the tokens

Enable automatic trimming of tokens?

A Refill delegate should use this method to push the tail of the current text. Doing so will cause the tail to be prepended to the next 'found' token, after the content is primed to reference fresh incoming data. Thus, a Refill should do this:

copy token.tail() into buffer

append fresh content into buffer ...

token.prime (buffer); return true;

Set the content, ready for next() to start

Return the current token as a slice of the content

Return the current content-tail. This is typically used by "refill" handlers, when they stream more content in.

Return how much content has been consumed

Return the index of the current token. This is different from eaten() in that the current token may not yet have been consumed. Thus index() will always be less than or equal to eaten()

Trim spaces from the left and right of the current token. Note that this is done in-place on the current slice. The content itself is not affected.

Internal method for subclasses to call when they locate a token

Internal method for subclasses to call when they run out of content to scan. This invokes the "refill" facilities to provide additional content.

Default "refill" handler, which indicates there's no more content to be had

Definition at line 69 of file text/Token.d.

References pragma(), TextTemplate(), and TokenTemplate().

Referenced by BoundTokenTemplate(), BufferTokenizerTemplate(), CharTokenTemplate(), ITokenizerTemplate(), LineTokenizerTemplate(), LineTokenTemplate(), PunctTokenizerTemplate(), QuotedTokenTemplate(), RegexTokenTemplate(), ScannerTemplate(), SetTokenTemplate(), SimpleTokenizerTemplate(), SpaceTokenizerTemplate(), StringTemplate(), TextReaderTemplate(), and TokenTemplate().


Variable Documentation

module mango text Token
 

Definition at line 40 of file text/Token.d.

import mango text Text
 

Definition at line 42 of file text/Token.d.


Generated on Sat Dec 24 17:28:36 2005 for Mango by  doxygen 1.4.0