Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

ITokenizer Struct Reference

Inheritance diagram for ITokenizer:

LineTokenizer PunctTokenizer SimpleTokenizer SpaceTokenizer List of all members.

Public Member Functions

bool next (IBuffer buffer, Token token)

Detailed Description

Interface to define how Tokenizers should expose their functionality.

Definition at line 134 of file Tokenizer.d.


Member Function Documentation

bool next IBuffer  buffer,
Token  token
 

Reimplemented in SimpleTokenizer, SpaceTokenizer, PunctTokenizer, and LineTokenizer.

Referenced by BoundToken::next(), TextReader::nextToken(), and ReaderToken::read().


The documentation for this struct was generated from the following file:
Generated on Fri May 27 18:12:02 2005 for Mango by  doxygen 1.4.0