Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

ITokenizer Struct Reference

Inheritance diagram for ITokenizer:

LineTokenizer PunctTokenizer SimpleTokenizer SpaceTokenizer List of all members.

Public Member Functions

bool next (IBuffer buffer, Token token)

Detailed Description

Interface to define how Tokenizers should expose their functionality.

Definition at line 134 of file Tokenizer.d.


Member Function Documentation

bool next IBuffer  buffer,
Token  token
 

Reimplemented in SimpleTokenizer, SpaceTokenizer, PunctTokenizer, and LineTokenizer.

Referenced by LineTokenizerTemplate(), BoundToken::next(), TextReader::nextToken(), PunctTokenizerTemplate(), ReaderToken::read(), SimpleTokenizerTemplate(), and SpaceTokenizerTemplate().


The documentation for this struct was generated from the following file:
Generated on Sat Dec 24 17:28:40 2005 for Mango by  doxygen 1.4.0