Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

Scanner Class Reference

Inheritance diagram for Scanner:

CookieParser LineScanner LineTokenizer PunctTokenizer SimpleTokenizer SpaceTokenizer List of all members.

Public Member Functions

bool next (IBuffer buffer, int(*scan)(char[]))
int notFound (Token token, char[] content)

Detailed Description

Extract tokens from an IBuffer. This is the base-class for all Tokenizers, but can also be used outside of the ITokenizer model.

Definition at line 60 of file Tokenizer.d.


Member Function Documentation

bool next IBuffer  buffer,
int(*)(char[])  scan
[inline]
 

Scan the given IBuffer for another token, and place the results in the provided token. Note that this should be completely thread-safe so one can instantiate singleton tokenizers without issue.

Each Token is expected to be stripped of the delimiter. An end-of-file condition causes trailing content to be placed into the token. Requests made beyond Eof result in empty tokens (length == zero).

Returns true if a token was isolated, false otherwise.

Definition at line 78 of file Tokenizer.d.

References IBuffer::compress(), IConduit::Eof, IBuffer::getConduit(), IBuffer::getPosition(), IConduit, IConduit::read(), IBuffer::read(), IBuffer::readable(), IBuffer::skip(), and IBuffer::writable().

int notFound Token  token,
char[]  content
[inline]
 

Clean up after we fail to find a token. Trailing content is placed into the token, and the scanner is told to try and load some more content (where available).

Definition at line 120 of file Tokenizer.d.

References IConduit, and Token::set().

Referenced by LineTokenizer::next(), PunctTokenizer::next(), SpaceTokenizer::next(), and SimpleTokenizer::next().


The documentation for this class was generated from the following file:
Generated on Fri May 27 18:12:03 2005 for Mango by  doxygen 1.4.0