Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

ITokenizer Struct Reference

Inheritance diagram for ITokenizer:

LineTokenizer PunctTokenizer RegexTokenizer SimpleTokenizer SpaceTokenizer List of all members.

Public Member Functions

bool next (IBuffer buffer, Token token)

Detailed Description

Interface to define how Tokenizers should expose their functionality.

Definition at line 128 of file Tokenizer.d.


Member Function Documentation

bool next IBuffer  buffer,
Token  token
 

Reimplemented in SimpleTokenizer, SpaceTokenizer, PunctTokenizer, LineTokenizer, and RegexTokenizer.

Referenced by BoundToken::next(), and TextReader::next().


The documentation for this struct was generated from the following file:
Generated on Sun Nov 7 19:07:07 2004 for Mango by doxygen 1.3.6