Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

Copy of Tokenizer.d File Reference

Go to the source code of this file.

Classes

class  VersionAres
struct  Tokenizers

Functions

class ScannerTemplate (T)
typedef ScannerTemplate (char) Scanner
struct ITokenizerTemplate (T)
class SimpleTokenizerTemplate (T)
typedef SimpleTokenizerTemplate (char) SimpleTokenizer
class SpaceTokenizerTemplate (T)
typedef SpaceTokenizerTemplate (char) SpaceTokenizer
class PunctTokenizerTemplate (T)
typedef PunctTokenizerTemplate (char) PunctTokenizer
class LineTokenizerTemplate (T)
typedef LineTokenizerTemplate (char) LineTokenizer
class LineScannerTemplate (T)
typedef LineScannerTemplate (char) LineScanner

Variables

module mango io Tokenizer
import std ctype
import mango io Token
import mango io mango io Exception
import mango io model IReader
import mango io model mango
io model 
IBuffer
import mango io model mango
io model mango io model 
IConduit


Function Documentation

class ScannerTemplate  ) 
 

Extract tokens from an IBuffer. This is the base-class for all Tokenizers, but can also be used outside of the ITokenizer model.

Scan the given IBuffer for another token, and place the results in the provided token. Note that this should be completely thread-safe so one can instantiate singleton tokenizers without issue.

Each Token is expected to be stripped of the delimiter. An end-of-file condition causes trailing content to be placed into the token. Requests made beyond Eof result in empty tokens (length == zero).

Returns true if a token was isolated, false otherwise.

Clean up after we fail to find a token. Trailing content is placed into the token, and the scanner is told to try and load some more content (where available).

Definition at line 60 of file Copy of Tokenizer.d.

References buffer, IBuffer, IConduit, and TokenTemplate().

Referenced by LineScannerTemplate(), LineTokenizerTemplate(), PunctTokenizerTemplate(), SimpleTokenizerTemplate(), and SpaceTokenizerTemplate().

typedef ScannerTemplate char   ) 
 

typedef ITokenizerTemplate  ) 
 

Interface to define how Tokenizers should expose their functionality.

Definition at line 136 of file Copy of Tokenizer.d.

References buffer, and TokenTemplate().

class SimpleTokenizerTemplate  ) 
 

A simple delimiting tokenizer. Use this to tokenize simple streams such as comma-seperated text.

Construct a SimpleTokenizer with the given delimiter char. More sophisticated delimiters can be constructed by using a RegexTokenizer instead.

Locate the next token from the provided buffer, and map a buffer reference into token. Returns true if a token was located, false otherwise.

Note that the buffer content is not duplicated. Instead, a slice of the buffer is referenced by the token. You can use Token.clone() or Token.toString().dup() to copy content per your application needs.

Note also that there may still be one token left in a buffer that was not terminated correctly (as in eof conditions). In such cases, tokens are mapped onto remaining content and the buffer will have no more readable content.

Definition at line 156 of file Copy of Tokenizer.d.

References buffer, ITokenizer::next(), ScannerTemplate(), SimpleTokenizerTemplate(), and TokenTemplate().

Referenced by SimpleTokenizerTemplate().

typedef SimpleTokenizerTemplate char   ) 
 

class SpaceTokenizerTemplate  ) 
 

A tokenizer that isolates content enclosed by whitespace.

Locate the next token from the provided buffer, and map a buffer reference into token. Returns true if a token was located, false otherwise.

Note that the buffer content is not duplicated. Instead, a slice of the buffer is referenced by the token. You can use Token.clone() or Token.toString().dup() to copy content per your application needs.

Note also that there may still be one token left in a buffer that was not terminated correctly (as in eof conditions). In such cases, tokens are mapped onto remaining content and the buffer will have no more readable content.

Definition at line 220 of file Copy of Tokenizer.d.

References buffer, ITokenizer::next(), ScannerTemplate(), SpaceTokenizerTemplate(), and TokenTemplate().

Referenced by SpaceTokenizerTemplate().

typedef SpaceTokenizerTemplate char   ) 
 

class PunctTokenizerTemplate  ) 
 

A tokenizer for handling both whitespace and punctuation delimiters.

Locate the next token from the provided buffer, and map a buffer reference into token. Returns true if a token was located, false otherwise.

Note that the buffer content is not duplicated. Instead, a slice of the buffer is referenced by the token. You can use Token.clone() or Token.toString().dup() to copy content per your application needs.

Note also that there may still be one token left in a buffer that was not terminated correctly (as in eof conditions). In such cases, tokens are mapped onto remaining content and the buffer will have no more readable content.

Definition at line 267 of file Copy of Tokenizer.d.

References buffer, ITokenizer::next(), PunctTokenizerTemplate(), ScannerTemplate(), and TokenTemplate().

Referenced by PunctTokenizerTemplate().

typedef PunctTokenizerTemplate char   ) 
 

class LineTokenizerTemplate  ) 
 

Tokenize an entire line delimited by a single '\n' character, or by a "\r\n" pair.

Locate the next token from the provided buffer, and map a buffer reference into token. Returns true if a token was located, false otherwise.

Note that the buffer content is not duplicated. Instead, a slice of the buffer is referenced by the token. You can use Token.clone() or Token.toString().dup() to copy content per your application needs.

Note also that there may still be one token left in a buffer that was not terminated correctly (as in eof conditions). In such cases, tokens are mapped onto remaining content and the buffer will have no more readable content.

Definition at line 316 of file Copy of Tokenizer.d.

References buffer, LineTokenizerTemplate(), ITokenizer::next(), ScannerTemplate(), and TokenTemplate().

Referenced by LineTokenizerTemplate().

typedef LineTokenizerTemplate char   ) 
 

class LineScannerTemplate  ) 
 

Eat everything until we reach a newline. Use this with a Reader, where you wish to discard everything else in the current line.

IReadable interface to support Reader.get()

Eat all content until we see a '
' character. The content is simply discarded.

Definition at line 367 of file Copy of Tokenizer.d.

References buffer, IReader::getBuffer(), LineScannerTemplate(), IReadable::read(), and ScannerTemplate().

Referenced by LineScannerTemplate().

typedef LineScannerTemplate char   ) 
 


Variable Documentation

module mango io Tokenizer
 

Definition at line 39 of file Copy of Tokenizer.d.

import std ctype
 

Definition at line 41 of file Copy of Tokenizer.d.

import mango io Token
 

Definition at line 46 of file Copy of Tokenizer.d.

import mango io mango io Exception
 

Definition at line 46 of file Copy of Tokenizer.d.

import mango io model IReader
 

Definition at line 49 of file Copy of Tokenizer.d.

import mango io model mango io model IBuffer
 

Definition at line 49 of file Copy of Tokenizer.d.

import mango io model mango io model mango io model IConduit
 

Definition at line 49 of file Copy of Tokenizer.d.


Generated on Sat Dec 24 17:28:35 2005 for Mango by  doxygen 1.4.0