Go to the source code of this file.
Functions | |
class | BufferTokenizerTemplate (T) |
typedef | BufferTokenizerTemplate (char) BufferTokenizer |
Variables | |
module mango io | BufferTokenizer |
import mango text | Token |
import mango io | Buffer |
private import mango io model | IBuffer |
private import mango io model mango io model | IConduit |
This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for damages of any kind arising from the use of this software.
Permission is hereby granted to anyone to use this software for any purpose, including commercial applications, and to alter it and/or redistribute it freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment within documentation of said product would be appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software.
3. This notice may not be removed or altered from any distribution of the source.
4. Derivative works are permitted, but they must carry this notice in full and credit the original source.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Definition in file BufferTokenizer.d.
|
Tokenize input from a buffer or conduit All input is tokenized from the associated buffer, and exposed as a freachable property via method opApply(). Empty token are passed to the caller ~ this may happen if, for example, an empty line is seen by a line-tokenizer. Construct a BufferTokenizer on the provided buffer, using the specified Tokenizer instead of the default one. Construct a BufferTokenizer upon the buffer associated with the given conduit. Iterate over the set of tokens Refill the token content from our buffer. Returns false upon reaching EOF Definition at line 60 of file BufferTokenizer.d. References Buffer, RawCodec1::buffer, buffer, BufferTokenizerTemplate(), and TokenTemplate(). Referenced by BufferTokenizerTemplate(). |
|
|
|
Definition at line 40 of file BufferTokenizer.d. |
|
Definition at line 42 of file BufferTokenizer.d. |
|
Definition at line 44 of file BufferTokenizer.d. |
|
Definition at line 46 of file BufferTokenizer.d. |
|
Definition at line 46 of file BufferTokenizer.d. |