00001 /******************************************************************************* 00002 00003 @file Buffer.d 00004 00005 Copyright (C) 2004 Kris Bell 00006 00007 This software is provided 'as-is', without any express or implied 00008 warranty. In no event will the authors be held liable for damages 00009 of any kind arising from the use of this software. 00010 00011 Permission is hereby granted to anyone to use this software for any 00012 purpose, including commercial applications, and to alter it and/or 00013 redistribute it freely, subject to the following restrictions: 00014 00015 1. The origin of this software must not be misrepresented; you must 00016 not claim that you wrote the original software. If you use this 00017 software in a product, an acknowledgment within documentation of 00018 said product would be appreciated but is not required. 00019 00020 2. Altered source versions must be plainly marked as such, and must 00021 not be misrepresented as being the original software. 00022 00023 3. This notice may not be removed or altered from any distribution 00024 of the source. 00025 00026 00027 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 00028 00029 00030 @version Initial version, March 2004 00031 @author Kris 00032 00033 00034 *******************************************************************************/ 00035 00036 module mango.io.Buffer; 00037 00038 private import mango.io.Exception; 00039 00040 private import mango.io.model.IConduit; 00041 00042 public import mango.io.model.IBuffer; 00043 00044 /****************************************************************************** 00045 00046 ******************************************************************************/ 00047 00048 extern (C) 00049 { 00050 void * memcpy (void *dst, void *src, uint); 00051 } 00052 00053 /******************************************************************************* 00054 00055 The basic premise behind this IO package is as follows: 00056 00057 @li the central concept is that of a buffer. The buffer acts 00058 as a queue (line) where items are removed from the front 00059 and new items are added to the back. Buffers are modeled 00060 by mango.io.model.IBuffer, and a concrete implementation is 00061 provided this class. 00062 00063 @li buffers can be written to directly, but a Reader and/or 00064 Writer are typically used to read & write formatted data. 00065 These readers & writers are bound to a specific buffer; 00066 often the same buffer. It's also perfectly legitimate to 00067 bind multiple writers to the same buffer; they will all 00068 behave serially as one would expect. The same applies to 00069 multiple readers on the same buffer. Readers and writers 00070 support two styles of IO: put/get, and the C++ style << 00071 and >> operators. All such operations can be chained. 00072 00073 @li Any class can be made compatable with the reader/writer 00074 framework by implementing the IReadable and/or IWritable 00075 interfaces. Each of these specify just a single method. 00076 00077 @li Buffers may also be tokenized. This is handy when one is 00078 dealing with text input, and/or the content suits a more 00079 fluid format than most typical readers & writers support. 00080 Tokens are mapped directly onto buffer content, so there 00081 is only minor overhead in using them. Tokens can be read 00082 and written by reader/writers also, using a more relaxed 00083 set of rules than those applied to integral IO. 00084 00085 @li buffers are sometimes memory-only, in which case there 00086 is nothing left to do when a reader (or tokenizer) hits 00087 end of buffer conditions. Other buffers are themselves 00088 bound to a Conduit. When this is the case, a reader will 00089 eventually cause the buffer to reload via its associated 00090 conduit. Previous buffer content will thus be lost. The 00091 same concept is applied to writers, whereby they flush 00092 the content of a full buffer to a bound conduit before 00093 continuing. 00094 00095 @li conduits provide virtualized access to external content, 00096 and represent things like files or Internet connections. 00097 They are just a different kind of stream. Conduits are 00098 modelled by mango.io.model.IConduit, and implemented via 00099 classes FileConduit and SocketConduit. Additional kinds 00100 of conduit are easy to construct: one either subclasses 00101 mango.io.Conduit, or implements mango.io.model.IConduit. 00102 A conduit reads and writes from/to a buffer in big chunks 00103 (typically the entire buffer). 00104 00105 00106 An example of how to append a buffer follows: 00107 00108 @code 00109 char[] foo = "to write some D"; 00110 00111 // create a small buffer 00112 Buffer buf = new Buffer (256); 00113 00114 // append some text directly to it 00115 buf.append("now is the time for all good men ").append(foo); 00116 00117 // output the combined string 00118 Stdout.put (buf.toString).cr(); 00119 @endcode 00120 00121 You might use a GrowableBuffer instead where you wish to 00122 append beyond a preset limit. One common usage of buffers 00123 is in conjunction with a conduit, such as FileConduit. Each 00124 Conduit exposes a factory for creating a buffer of the most 00125 appropriate size or flavour: 00126 00127 @code 00128 FileConduit fc = new FileConduit ("file.name"); 00129 Buffer buf = fc.createBuffer (); 00130 @endcode 00131 00132 However, this is typically hidden by higher level constructors 00133 such as those of Reader and Writer derivitives. For example: 00134 00135 @code 00136 FileConduit fc = new FileConduit ("file.name"); 00137 Reader reader = new Reader (fc); 00138 @endcode 00139 00140 There is indeed a buffer between the Reader and Conduit, but 00141 explicit construction is unecessary in many cases. See both 00142 Reader and Writer for examples of formatted IO. 00143 00144 *******************************************************************************/ 00145 00146 class Buffer : IBuffer 00147 { 00148 static IOException ovf; 00149 00150 private void[] data; 00151 private uint limit; 00152 private uint capacity; 00153 private uint position; 00154 private IConduit conduit; 00155 00156 /*********************************************************************** 00157 00158 Ensure the buffer remains valid between method calls 00159 00160 ***********************************************************************/ 00161 00162 invariant 00163 { 00164 assert (position <= limit); 00165 assert (limit <= capacity); 00166 } 00167 00168 /*********************************************************************** 00169 00170 Construct an overflow exception for later use. 00171 00172 ***********************************************************************/ 00173 00174 static this () 00175 { 00176 ovf = new IOException ("IO buffer overflow"); 00177 } 00178 00179 /*********************************************************************** 00180 00181 Construct a Buffer with the specified number of bytes. 00182 00183 ***********************************************************************/ 00184 00185 this (uint capacity = 0) 00186 { 00187 this (new ubyte[capacity]); 00188 } 00189 00190 /*********************************************************************** 00191 00192 Prime buffer with an application-supplied array. There is 00193 no readable data present, and writing begins at position 0. 00194 00195 ***********************************************************************/ 00196 00197 this (void[] data) 00198 { 00199 this (data, 0); 00200 } 00201 00202 /*********************************************************************** 00203 00204 Prime buffer with an application-supplied array, and 00205 indicate how much readable data is already there. A 00206 write operation will begin writing immediately after 00207 the existing content. 00208 00209 ***********************************************************************/ 00210 00211 this (void[] data, uint readable) 00212 { 00213 setContent (data, readable); 00214 } 00215 00216 /*********************************************************************** 00217 00218 Create an instance of an IBuffer. Use this when you 00219 don't know anything about the concrete implementation, 00220 and have only the IBuffer interface available 00221 00222 Returns a Buffer with no content. 00223 00224 ***********************************************************************/ 00225 00226 IBuffer create () 00227 { 00228 return new Buffer; 00229 } 00230 00231 /*********************************************************************** 00232 00233 Return the backing array 00234 00235 ***********************************************************************/ 00236 00237 void[] getContent () 00238 { 00239 return data; 00240 } 00241 00242 /*********************************************************************** 00243 00244 Set the backing array with all content readable. Writing 00245 to this will either flush it to an associated conduit, or 00246 raise an Eof condition. Use IBuffer.clear() to reset the 00247 content (make it all writable). 00248 00249 ***********************************************************************/ 00250 00251 IBuffer setValidContent (void[] data) 00252 { 00253 return setContent (data, data.length); 00254 } 00255 00256 /*********************************************************************** 00257 00258 Set the backing array with some content readable. Writing 00259 to this will either flush it to an associated conduit, or 00260 raise an Eof condition. Use IBuffer.clear() to reset the 00261 content (make it all writable). 00262 00263 ***********************************************************************/ 00264 00265 IBuffer setContent (void[] data, uint readable) 00266 { 00267 this.data = data; 00268 this.limit = readable; 00269 this.capacity = data.length; 00270 00271 // reset to start of input 00272 this.position = 0; 00273 00274 return this; 00275 } 00276 00277 /*********************************************************************** 00278 00279 Overridable method to grow the buffer size when it becomes 00280 full. Default is to not grow at all. 00281 00282 ***********************************************************************/ 00283 00284 protected bool grow (uint size) 00285 { 00286 return false; 00287 } 00288 00289 /*********************************************************************** 00290 00291 Bulk copy of data from 'src'. Limit is adjusted by 'size' 00292 bytes. 00293 00294 ***********************************************************************/ 00295 00296 protected void* copy (void *src, uint size) 00297 { 00298 void* ret = &data[limit]; 00299 data[limit..limit+size] = src[0..size]; 00300 limit += size; 00301 return ret; 00302 } 00303 00304 /*********************************************************************** 00305 00306 Write a chunk of data into this buffer, and flush to the 00307 conduit as necessary. Returns true if all data was written, 00308 false otherwise (indicating eof, or eob). 00309 00310 This is typically used for streaming formatted data. Note 00311 that this should ideally have package visibility only, as 00312 it should be restricted to Writers etc. 00313 00314 ***********************************************************************/ 00315 /+ 00316 bool put (void *src, uint size) 00317 { 00318 if (writable() < size) 00319 // try to grow buffer ... 00320 if (grow (size)) 00321 return put (src, size); 00322 else 00323 // can we write externally? 00324 if (conduit) 00325 { 00326 conduit.flush (this); 00327 00328 // check for pathological case 00329 if (size > capacity) 00330 { 00331 void[] tmp = data; 00332 setContent (src[0..size], size); 00333 00334 if (conduit.flush (this)) 00335 return setContent (tmp, 0), true; 00336 return false; 00337 } 00338 } 00339 else 00340 return false; 00341 00342 return copy (src, size); 00343 } 00344 +/ 00345 00346 /*********************************************************************** 00347 00348 Write a chunk of data into this buffer, and flush to the 00349 conduit as necessary. Returns true if all data was written, 00350 false otherwise (indicating eof, or eob). 00351 00352 This is typically used for streaming formatted data. Note 00353 that this should ideally have package visibility only, as 00354 it should be restricted to Writers etc. 00355 00356 ***********************************************************************/ 00357 00358 void* put (void *src, uint size) 00359 { 00360 return copy (src, size); 00361 } 00362 00363 /*********************************************************************** 00364 00365 Read a chunk of data from the buffer, loading from the 00366 conduit as necessary. The specified number of bytes is 00367 loaded into the buffer, and marked as having been read 00368 when the 'eat' parameter is set true. When 'eat' is set 00369 false, the read position is not adjusted. 00370 00371 Returns the corresponding buffer slice when successful, 00372 or null if there's not enough data available (Eof; Eob). 00373 00374 ***********************************************************************/ 00375 00376 void[] get (uint size, bool eat = true) 00377 { 00378 if (readable() < size) 00379 { 00380 if (conduit is null) 00381 return null; 00382 00383 // move readable content to buffer start 00384 compress (); 00385 00386 // populate tail of buffer with new content 00387 do { 00388 if (conduit.read (this) == IConduit.Eof) 00389 return null; 00390 } while (readable() < size); 00391 } 00392 00393 void[] ret = data [position..position+size]; 00394 if (eat) 00395 position += size; 00396 00397 return ret; 00398 } 00399 00400 /*********************************************************************** 00401 00402 Write an array of data into this buffer, and flush to the 00403 conduit as necessary. Returns a chaining reference if all 00404 data was written; throws an IOException indicating eof or 00405 eob if not. 00406 00407 This is often used in lieu of a Writer. 00408 00409 ***********************************************************************/ 00410 00411 IBuffer append (void[] content) 00412 { 00413 if (writable() < content.length) 00414 throw ovf; 00415 00416 put (content, content.length); 00417 return this; 00418 } 00419 00420 /*********************************************************************** 00421 00422 Return a char[] slice of the buffer up to the limit of 00423 valid content. 00424 00425 ***********************************************************************/ 00426 00427 override char[] toString () 00428 { 00429 return cast(char[]) data[position..limit]; 00430 } 00431 00432 /*********************************************************************** 00433 00434 Skip ahead by the specified number of bytes, streaming from 00435 the associated conduit as necessary. 00436 00437 Can also reverse the read position by 'size' bytes. This may 00438 be used to support lookahead-type operations. 00439 00440 Returns true if successful, false otherwise. 00441 00442 ***********************************************************************/ 00443 00444 bool skip (int size) 00445 { 00446 if (size < 0) 00447 { 00448 size = -size; 00449 if (position >= size) 00450 { 00451 position -= size; 00452 return true; 00453 } 00454 return false; 00455 } 00456 return get (size) !== null; 00457 } 00458 00459 /*********************************************************************** 00460 00461 Return count of readable bytes remaining in buffer. This is 00462 calculated simply as limit() - position() 00463 00464 ***********************************************************************/ 00465 00466 uint readable () 00467 { 00468 return limit - position; 00469 } 00470 00471 /*********************************************************************** 00472 00473 Return count of writable bytes available in buffer. This is 00474 calculated simply as capacity() - limit() 00475 00476 ***********************************************************************/ 00477 00478 uint writable () 00479 { 00480 return capacity - limit; 00481 } 00482 00483 /*********************************************************************** 00484 00485 Exposes the raw data buffer at the current write position, 00486 The delegate is provided with a void[] representing space 00487 available within the buffer at the current write position. 00488 00489 The delegate should return the approriate number of bytes 00490 if it writes valid content, or IConduit.Eof on error. 00491 00492 Returns whatever the delegate returns. 00493 00494 ***********************************************************************/ 00495 00496 int write (int delegate (void[]) dg) 00497 { 00498 int count = dg (data [limit..capacity]); 00499 00500 if (count != IConduit.Eof) 00501 { 00502 limit += count; 00503 assert (limit <= capacity); 00504 } 00505 return count; 00506 } 00507 00508 /*********************************************************************** 00509 00510 Exposes the raw data buffer at the current read position. The 00511 delegate is provided with a void[] representing the available 00512 data, and should return zero to leave the current read position 00513 intact. 00514 00515 If the delegate consumes data, it should return the number of 00516 bytes consumed; or IConduit.Eof to indicate an error. 00517 00518 Returns whatever the delegate returns. 00519 00520 ***********************************************************************/ 00521 00522 int read (int delegate (void[]) dg) 00523 { 00524 int count = dg (data [position..limit]); 00525 00526 if (count != IConduit.Eof) 00527 { 00528 position += count; 00529 assert (position <= limit); 00530 } 00531 return count; 00532 } 00533 00534 /*********************************************************************** 00535 00536 If we have some data left after an export, move it to 00537 front-of-buffer and set position to be just after the 00538 remains. This is for supporting certain conduits which 00539 choose to write just the initial portion of a request. 00540 00541 Limit is set to the amount of data remaining. Position 00542 is always reset to zero. 00543 00544 ***********************************************************************/ 00545 00546 IBuffer compress () 00547 { 00548 uint r = readable (); 00549 00550 if (position > 0 && r > 0) 00551 // content may overlap ... 00552 memcpy (&data[0], &data[position], r); 00553 00554 position = 0; 00555 limit = r; 00556 return this; 00557 } 00558 00559 /*********************************************************************** 00560 00561 flush the contents of this buffer to the related conduit. 00562 Throws an IOException on premature eof. 00563 00564 ***********************************************************************/ 00565 00566 void flush () 00567 { 00568 if (conduit) 00569 conduit.flush (this); 00570 else 00571 clear(); 00572 } 00573 00574 /*********************************************************************** 00575 00576 Reset 'position' and 'limit' to zero 00577 00578 ***********************************************************************/ 00579 00580 IBuffer clear () 00581 { 00582 position = limit = 0; 00583 return this; 00584 } 00585 00586 /*********************************************************************** 00587 00588 Returns the limit of readable content within this buffer 00589 00590 ***********************************************************************/ 00591 00592 uint getLimit () 00593 { 00594 return limit; 00595 } 00596 00597 /*********************************************************************** 00598 00599 Returns the total capacity of this buffer 00600 00601 ***********************************************************************/ 00602 00603 uint getCapacity () 00604 { 00605 return capacity; 00606 } 00607 00608 /*********************************************************************** 00609 00610 Returns the current read-position within this buffer 00611 00612 ***********************************************************************/ 00613 00614 uint getPosition () 00615 { 00616 return position; 00617 } 00618 00619 /*********************************************************************** 00620 00621 Returns the conduit associated with this buffer. Returns 00622 null if the buffer is purely memory based; that is, it's 00623 not backed by some external medium. 00624 00625 Buffers do not require an external conduit to operate, but 00626 it can be convenient to associate one. For example, methods 00627 read and write use it to import/export content as necessary. 00628 00629 ***********************************************************************/ 00630 00631 IConduit getConduit () 00632 { 00633 return conduit; 00634 } 00635 00636 /*********************************************************************** 00637 00638 Sets the external conduit associated with this buffer. 00639 00640 Buffers do not require an external conduit to operate, but 00641 it can be convenient to associate one. For example, methods 00642 read and write use it to import/export content as necessary. 00643 00644 ***********************************************************************/ 00645 00646 void setConduit (IConduit conduit) 00647 { 00648 this.conduit = conduit; 00649 } 00650 } 00651 00652 00653 /******************************************************************************* 00654 00655 Subclass to support automatic flushing. This can be used for 00656 Stdout, Stderr, and other related conduits. 00657 00658 *******************************************************************************/ 00659 00660 class FlushBuffer : Buffer 00661 { 00662 /*********************************************************************** 00663 00664 Construct a flushing buffer with the given size upon the 00665 provided IConduit 00666 00667 ***********************************************************************/ 00668 00669 this (int size, IConduit conduit) 00670 in { 00671 assert (conduit); 00672 } 00673 body 00674 { 00675 super (size); 00676 setConduit (conduit); 00677 } 00678 00679 /*********************************************************************** 00680 00681 Intercept buffer writes so we can flush the new content 00682 00683 ***********************************************************************/ 00684 00685 protected override void* copy (void *src, uint size) 00686 { 00687 void *ret = super.copy (src, size); 00688 getConduit().flush (this); 00689 return ret; 00690 } 00691 } 00692 00693 00694 /******************************************************************************* 00695 00696 Subclass to provide support for content growth. This is handy when 00697 you want to keep a buffer around as a scratchpad. 00698 00699 *******************************************************************************/ 00700 00701 class GrowableBuffer : Buffer 00702 { 00703 /*********************************************************************** 00704 00705 Create a GrowableBuffer with the specified initial size. 00706 00707 ***********************************************************************/ 00708 00709 this (uint size) 00710 { 00711 super (size); 00712 } 00713 00714 /*********************************************************************** 00715 00716 Overridable method to grow the buffer size when it becomes 00717 full. Default is to not grow at all. 00718 00719 ***********************************************************************/ 00720 00721 protected override bool grow (uint size) 00722 { 00723 if (data.length < uint.max / 2) 00724 { 00725 uint i = data.length * 2; 00726 if (i < size) 00727 i = size; 00728 data.length = i; 00729 capacity = i; 00730 return true; 00731 } 00732 return false; 00733 } 00734 } 00735 00736 00737 00738 /******************************************************************************* 00739 00740 Subclass to treat the buffer as a seekable entity, where all 00741 capacity is available for reading and/or writing. To achieve 00742 this we must effectively disable the 'limit' watermark, and 00743 locate write operations around 'position' instead. 00744 00745 *******************************************************************************/ 00746 00747 class MappedBuffer : Buffer 00748 { 00749 /*********************************************************************** 00750 00751 Construct an empty MappedBuffer 00752 00753 ***********************************************************************/ 00754 00755 this () 00756 { 00757 super (0); 00758 } 00759 00760 /*********************************************************************** 00761 00762 Close this mapped buffer 00763 00764 ***********************************************************************/ 00765 00766 abstract void close (); 00767 00768 /*********************************************************************** 00769 00770 Flush any dirty content out to the drive 00771 00772 ***********************************************************************/ 00773 00774 abstract void flush (); 00775 00776 /*********************************************************************** 00777 00778 Set the read/write position 00779 00780 ***********************************************************************/ 00781 00782 void setPosition (uint position) 00783 { 00784 this.position = position; 00785 } 00786 00787 /*********************************************************************** 00788 00789 Seek to the specified position within the buffer, and return 00790 the byte offset of the new location (relative to zero). 00791 00792 ***********************************************************************/ 00793 00794 long seek (long offset, ISeekable.SeekAnchor anchor) 00795 { 00796 long pos = capacity; 00797 00798 if (anchor == ISeekable.SeekAnchor.Begin) 00799 pos = offset; 00800 else 00801 if (anchor == ISeekable.SeekAnchor.End) 00802 pos -= offset; 00803 else 00804 pos = position + offset; 00805 00806 return position = pos; 00807 } 00808 00809 /*********************************************************************** 00810 00811 Return count of writable bytes available in buffer. This is 00812 calculated simply as capacity() - limit() 00813 00814 ***********************************************************************/ 00815 00816 override uint writable () 00817 { 00818 return capacity - position; 00819 } 00820 00821 /*********************************************************************** 00822 00823 Bulk copy of data from 'src'. Position is adjusted by 'size' 00824 bytes. 00825 00826 ***********************************************************************/ 00827 00828 override protected void* copy (void *src, uint size) 00829 { 00830 void* ret = &data[position]; 00831 data[position..position+size] = src[0..size]; 00832 position += size; 00833 return ret; 00834 } 00835 00836 /*********************************************************************** 00837 00838 Exposes the raw data buffer at the current write position, 00839 The delegate is provided with a void[] representing space 00840 available within the buffer at the current write position. 00841 00842 The delegate should return the approriate number of bytes 00843 if it writes valid content, or IConduit.Eof on error. 00844 00845 Returns whatever the delegate returns. 00846 00847 ***********************************************************************/ 00848 00849 override int write (int delegate (void[]) dg) 00850 { 00851 int count = dg (data [position..capacity]); 00852 00853 if (count != IConduit.Eof) 00854 { 00855 position += count; 00856 assert (position <= capacity); 00857 } 00858 return count; 00859 } 00860 00861 /*********************************************************************** 00862 00863 Prohibit compress() from doing anything at all. 00864 00865 ***********************************************************************/ 00866 00867 override IBuffer compress () 00868 { 00869 return this; 00870 } 00871 00872 /*********************************************************************** 00873 00874 Prohibit clear() from doing anything at all. 00875 00876 ***********************************************************************/ 00877 00878 override IBuffer clear () 00879 { 00880 return this; 00881 } 00882 00883 /*********************************************************************** 00884 00885 Prohibit the setting of another IConduit 00886 00887 ***********************************************************************/ 00888 00889 override void setConduit (IConduit conduit) 00890 { 00891 assert (0); 00892 } 00893 } 00894 00895 00896