Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

Scanner Class Reference

Inheritance diagram for Scanner:

CookieParser LineScanner LineTokenizer PunctTokenizer RegexTokenizer SimpleTokenizer SpaceTokenizer List of all members.

Public Member Functions

bool next (IBuffer buffer, int(*scan)(char[]))
int notFound (Token token, char[] content)

Detailed Description

Extract tokens from an IBuffer. This is the base-class for all Tokenizers, but can also be used outside of the ITokenizer model.

Definition at line 54 of file Tokenizer.d.


Member Function Documentation

bool next IBuffer  buffer,
int(*  scan)(char[])
[inline]
 

Scan the given IBuffer for another token, and place the results in the provided token. Note that this should be completely thread-safe so one can instantiate singleton tokenizers without issue.

Each Token is expected to be stripped of the delimiter. An end-of-file condition causes trailing content to be placed into the token. Requests made beyond Eof result in empty tokens (length == zero).

Returns true if a token was isolated, false otherwise.

Definition at line 72 of file Tokenizer.d.

References IBuffer::compress(), IBuffer::getConduit(), IBuffer::getPosition(), IConduit::read(), IBuffer::read(), IBuffer::readable(), IBuffer::skip(), and IBuffer::writable().

int notFound Token  token,
char[]  content
[inline]
 

Clean up after we fail to find a token. Trailing content is placed into the token, and the scanner is told to try and load some more content (where available).

Definition at line 114 of file Tokenizer.d.

References Token::set().

Referenced by RegexTokenizer::next(), LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), and SimpleTokenizer::next().


The documentation for this class was generated from the following file:
Generated on Sun Nov 7 19:07:09 2004 for Mango by doxygen 1.3.6