Go to the source code of this file.
Functions | |
class | TokenTemplate (T) |
Variables | |
module mango text | Token |
import mango text | Text |
|
The base class for a set of tokenizers. There are two types of tokenizers supported ~ exclusive and inclusive. The former are the more common kind, where a token is delimited by elements that are considered foreign. Examples include space, comma, and end-of-line delineation. Inclusive tokens are just the opposite: they look for patterns in the text that should be part of the token itself ~ everything else is considered foreign. Currently the only inclusive token type is exposed by RegexToken; everything else is of the exclusive variety. The content provided to Tokenizers is supposed to be entirely read-only. All current tokenizers abide by this rule, but it's possible a user could mutate the content through a token slice. To enforce the desired read-only aspect, the code would have to introduce redundant copying or the compiler would have to support read-only arrays. See LineToken, CharToken, RegexToken, QuotedToken, and SetToken. Locate the next token. If this Token is configured for "refills", more content will be requested as needed. Returns true if a token is found; false otherwise. Construct a token upon the given content. Automatic refills are disabled Iterate over the set of tokens. This provides read-only access to the tokens Enable automatic trimming of tokens? A Refill delegate should use this method to push the tail of the current text. Doing so will cause the tail to be prepended to the next 'found' token, after the content is primed to reference fresh incoming data. Thus, a Refill should do this: copy token.tail() into buffer append fresh content into buffer ... token.prime (buffer); return true; Set the content, ready for next() to start Return the current token as a slice of the content Return the current content-tail. This is typically used by "refill" handlers, when they stream more content in. Return how much content has been consumed Return the index of the current token. This is different from eaten() in that the current token may not yet have been consumed. Thus index() will always be less than or equal to eaten() Trim spaces from the left and right of the current token. Note that this is done in-place on the current slice. The content itself is not affected. Internal method for subclasses to call when they locate a token Internal method for subclasses to call when they run out of content to scan. This invokes the "refill" facilities to provide additional content. Default "refill" handler, which indicates there's no more content to be had Definition at line 69 of file text/Token.d. References pragma(), TextTemplate(), and TokenTemplate(). Referenced by BoundTokenTemplate(), BufferTokenizerTemplate(), CharTokenTemplate(), ITokenizerTemplate(), LineTokenizerTemplate(), LineTokenTemplate(), PunctTokenizerTemplate(), QuotedTokenTemplate(), RegexTokenTemplate(), ScannerTemplate(), SetTokenTemplate(), SimpleTokenizerTemplate(), SpaceTokenizerTemplate(), StringTemplate(), TextReaderTemplate(), and TokenTemplate(). |
|
Definition at line 40 of file text/Token.d. |
|
Definition at line 42 of file text/Token.d. |