00001 /******************************************************************************* 00002 00003 @file Buffer.d 00004 00005 Copyright (c) 2004 Kris Bell 00006 00007 This software is provided 'as-is', without any express or implied 00008 warranty. In no event will the authors be held liable for damages 00009 of any kind arising from the use of this software. 00010 00011 Permission is hereby granted to anyone to use this software for any 00012 purpose, including commercial applications, and to alter it and/or 00013 redistribute it freely, subject to the following restrictions: 00014 00015 1. The origin of this software must not be misrepresented; you must 00016 not claim that you wrote the original software. If you use this 00017 software in a product, an acknowledgment within documentation of 00018 said product would be appreciated but is not required. 00019 00020 2. Altered source versions must be plainly marked as such, and must 00021 not be misrepresented as being the original software. 00022 00023 3. This notice may not be removed or altered from any distribution 00024 of the source. 00025 00026 4. Derivative works are permitted, but they must carry this notice 00027 in full and credit the original source. 00028 00029 00030 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 00031 00032 00033 @version Initial version; March 2004 00034 00035 @author Kris 00036 00037 00038 *******************************************************************************/ 00039 00040 module mango.io.Buffer; 00041 00042 private import mango.io.Exception; 00043 00044 private import mango.io.model.IConduit; 00045 00046 public import mango.io.model.IBuffer; 00047 00048 /****************************************************************************** 00049 00050 ******************************************************************************/ 00051 00052 extern (C) 00053 { 00054 void * memcpy (void *dst, void *src, uint); 00055 } 00056 00057 /******************************************************************************* 00058 00059 The basic premise behind this IO package is as follows: 00060 00061 @li the central concept is that of a buffer. The buffer acts 00062 as a queue (line) where items are removed from the front 00063 and new items are added to the back. Buffers are modeled 00064 by mango.io.model.IBuffer, and a concrete implementation is 00065 provided this class. 00066 00067 @li buffers can be written to directly, but a Reader and/or 00068 Writer are typically used to read & write formatted data. 00069 These readers & writers are bound to a specific buffer; 00070 often the same buffer. It's also perfectly legitimate to 00071 bind multiple writers to the same buffer; they will all 00072 behave serially as one would expect. The same applies to 00073 multiple readers on the same buffer. Readers and writers 00074 support three styles of IO: put/get, the C++ style << & 00075 >> operators, and the () whisper style. All operations 00076 can be chained. 00077 00078 @li Any class can be made compatable with the reader/writer 00079 framework by implementing the IReadable and/or IWritable 00080 interfaces. Each of these specify just a single method. 00081 00082 @li Buffers may also be tokenized. This is handy when one is 00083 dealing with text input, and/or the content suits a more 00084 fluid format than most typical readers & writers support. 00085 Tokens are mapped directly onto buffer content, so there 00086 is only minor overhead in using them. Tokens can be read 00087 and written by reader/writers also, using a more relaxed 00088 set of rules than those applied to integral IO. 00089 00090 @li buffers are sometimes memory-only, in which case there 00091 is nothing left to do when a reader (or tokenizer) hits 00092 end of buffer conditions. Other buffers are themselves 00093 bound to a Conduit. When this is the case, a reader will 00094 eventually cause the buffer to reload via its associated 00095 conduit. Previous buffer content will thus be lost. The 00096 same concept is applied to writers, whereby they flush 00097 the content of a full buffer to a bound conduit before 00098 continuing. 00099 00100 @li conduits provide virtualized access to external content, 00101 and represent things like files or Internet connections. 00102 They are just a different kind of stream. Conduits are 00103 modelled by mango.io.model.IConduit, and implemented via 00104 classes FileConduit and SocketConduit. Additional kinds 00105 of conduit are easy to construct: one either subclasses 00106 mango.io.Conduit, or implements mango.io.model.IConduit. 00107 A conduit reads and writes from/to a buffer in big chunks 00108 (typically the entire buffer). 00109 00110 00111 An example of how to append a buffer follows: 00112 00113 @code 00114 char[] foo = "to write some D"; 00115 00116 // create a small buffer 00117 Buffer buf = new Buffer (256); 00118 00119 // append some text directly to it 00120 buf.append("now is the time for all good men ").append(foo); 00121 00122 // output the combined string 00123 Stdout (buf.toString) (CR); 00124 @endcode 00125 00126 You might use a GrowableBuffer instead where you wish to 00127 append beyond a preset limit. One common usage of buffers 00128 is in conjunction with a conduit, such as FileConduit. Each 00129 Conduit exposes a factory for creating a buffer of the most 00130 appropriate size or flavour: 00131 00132 @code 00133 FileConduit fc = new FileConduit ("file.name"); 00134 Buffer buf = fc.createBuffer (); 00135 @endcode 00136 00137 However, this is typically hidden by higher level constructors 00138 such as those of Reader and Writer derivitives. For example: 00139 00140 @code 00141 FileConduit fc = new FileConduit ("file.name"); 00142 Reader reader = new Reader (fc); 00143 @endcode 00144 00145 There is indeed a buffer between the Reader and Conduit, but 00146 explicit construction is unecessary in many cases. See both 00147 Reader and Writer for examples of formatted IO. 00148 00149 *******************************************************************************/ 00150 00151 class Buffer : IBuffer 00152 { 00153 private void[] data; 00154 private uint limit; 00155 private uint capacity; 00156 private uint position; 00157 private IConduit conduit; 00158 00159 // error messages 00160 protected static char[] overflow = "output buffer overflow"; 00161 protected static char[] underflow = "input buffer underflow"; 00162 protected static char[] eofRead = "end-of-file whilst reading"; 00163 protected static char[] eofWrite = "end-of-file whilst writing"; 00164 00165 00166 /*********************************************************************** 00167 00168 Ensure the buffer remains valid between method calls 00169 00170 ***********************************************************************/ 00171 00172 invariant 00173 { 00174 assert (position <= limit); 00175 assert (limit <= capacity); 00176 } 00177 00178 /*********************************************************************** 00179 00180 Construct a Buffer with the specified number of bytes. 00181 00182 ***********************************************************************/ 00183 00184 this (uint capacity = 0) 00185 { 00186 this (new ubyte[capacity]); 00187 } 00188 00189 /*********************************************************************** 00190 00191 Prime buffer with an application-supplied array. There is 00192 no readable data present, and writing begins at position 0. 00193 00194 ***********************************************************************/ 00195 00196 this (void[] data) 00197 { 00198 this (data, 0); 00199 } 00200 00201 /*********************************************************************** 00202 00203 Prime buffer with an application-supplied array, and 00204 indicate how much readable data is already there. A 00205 write operation will begin writing immediately after 00206 the existing content. 00207 00208 ***********************************************************************/ 00209 00210 this (void[] data, uint readable) 00211 { 00212 setContent (data, readable); 00213 } 00214 00215 /*********************************************************************** 00216 00217 Throw an exception with the provided message 00218 00219 ***********************************************************************/ 00220 00221 protected void error (char[] msg) 00222 { 00223 throw new IOException (msg); 00224 } 00225 00226 /*********************************************************************** 00227 00228 Create an instance of an IBuffer. Use this when you 00229 don't know anything about the concrete implementation, 00230 and have only the IBuffer interface available 00231 00232 Returns a Buffer with no content. 00233 00234 ***********************************************************************/ 00235 00236 IBuffer create () 00237 { 00238 return new Buffer; 00239 } 00240 00241 /*********************************************************************** 00242 00243 Return the backing array 00244 00245 ***********************************************************************/ 00246 00247 void[] getContent () 00248 { 00249 return data; 00250 } 00251 00252 /*********************************************************************** 00253 00254 Set the backing array with all content readable. Writing 00255 to this will either flush it to an associated conduit, or 00256 raise an Eof condition. Use IBuffer.clear() to reset the 00257 content (make it all writable). 00258 00259 ***********************************************************************/ 00260 00261 IBuffer setValidContent (void[] data) 00262 { 00263 return setContent (data, data.length); 00264 } 00265 00266 /*********************************************************************** 00267 00268 Set the backing array with some content readable. Writing 00269 to this will either flush it to an associated conduit, or 00270 raise an Eof condition. Use IBuffer.clear() to reset the 00271 content (make it all writable). 00272 00273 ***********************************************************************/ 00274 00275 IBuffer setContent (void[] data, uint readable) 00276 { 00277 this.data = data; 00278 this.limit = readable; 00279 this.capacity = data.length; 00280 00281 // reset to start of input 00282 this.position = 0; 00283 00284 return this; 00285 } 00286 00287 /*********************************************************************** 00288 00289 Overridable method to grow the buffer size when it becomes 00290 full. Default is to not grow at all. 00291 00292 ***********************************************************************/ 00293 00294 protected bool grow (uint size) 00295 { 00296 return false; 00297 } 00298 00299 /*********************************************************************** 00300 00301 Bulk copy of data from 'src'. Limit is adjusted by 'size' 00302 bytes. 00303 00304 ***********************************************************************/ 00305 00306 protected void copy (void *src, uint size) 00307 { 00308 data[limit..limit+size] = src[0..size]; 00309 limit += size; 00310 } 00311 00312 /*********************************************************************** 00313 00314 Read a chunk of data from the buffer, loading from the 00315 conduit as necessary. The specified number of bytes is 00316 loaded into the buffer, and marked as having been read 00317 when the 'eat' parameter is set true. When 'eat' is set 00318 false, the read position is not adjusted. 00319 00320 Returns the corresponding buffer slice when successful, 00321 or null if there's not enough data available (Eof; Eob). 00322 00323 ***********************************************************************/ 00324 00325 void[] get (uint size, bool eat = true) 00326 { 00327 if (size > readable) 00328 { 00329 if (conduit is null) 00330 error (underflow); 00331 00332 // make some space? This will try to leave as much content 00333 // in the buffer as possible, such that entire records may 00334 // be aliased directly from within. 00335 if (size > writable) 00336 { 00337 if (size > capacity) 00338 error (underflow); 00339 compress (); 00340 } 00341 00342 // populate tail of buffer with new content 00343 do { 00344 if (conduit.read (this) == IConduit.Eof) 00345 error (eofRead); 00346 } while (size > readable); 00347 } 00348 00349 uint i = position; 00350 if (eat) 00351 position += size; 00352 return data [i .. i + size]; 00353 } 00354 00355 /*********************************************************************** 00356 00357 Append an array of data to this buffer, and flush to the 00358 conduit as necessary. Returns a chaining reference if all 00359 data was written; throws an IOException indicating eof or 00360 eob if not. 00361 00362 This is often used in lieu of a Writer. 00363 00364 ***********************************************************************/ 00365 00366 IBuffer append (void[] src) 00367 { 00368 uint size = src.length; 00369 00370 if (size > writable) 00371 // can we write externally? 00372 if (conduit) 00373 { 00374 flush (); 00375 00376 // check for pathological case 00377 if (size > capacity) 00378 { 00379 // we could write directly to the conduit, but 00380 // then we'd lose support for partial writes ... 00381 void[] tmp = data; 00382 00383 setContent (src, size); 00384 flush (); 00385 setContent (tmp, 0); 00386 return this; 00387 } 00388 } 00389 else 00390 // try to grow buffer ... 00391 if (! grow (size)) 00392 error (overflow); 00393 00394 copy (src, size); 00395 return this; 00396 } 00397 00398 /*********************************************************************** 00399 00400 Return a char[] slice of the buffer up to the limit of 00401 valid content. 00402 00403 ***********************************************************************/ 00404 00405 override char[] toString () 00406 { 00407 return cast(char[]) data[position..limit]; 00408 } 00409 00410 /*********************************************************************** 00411 00412 Skip ahead by the specified number of bytes, streaming from 00413 the associated conduit as necessary. 00414 00415 Can also reverse the read position by 'size' bytes. This may 00416 be used to support lookahead-type operations. 00417 00418 Returns true if successful, false otherwise. 00419 00420 ***********************************************************************/ 00421 00422 bool skip (int size) 00423 { 00424 if (size < 0) 00425 { 00426 size = -size; 00427 if (position >= size) 00428 { 00429 position -= size; 00430 return true; 00431 } 00432 return false; 00433 } 00434 return get (size) !is null; 00435 } 00436 00437 /*********************************************************************** 00438 00439 Return count of readable bytes remaining in buffer. This is 00440 calculated simply as limit() - position() 00441 00442 ***********************************************************************/ 00443 00444 uint readable () 00445 { 00446 return limit - position; 00447 } 00448 00449 /*********************************************************************** 00450 00451 Return count of writable bytes available in buffer. This is 00452 calculated simply as capacity() - limit() 00453 00454 ***********************************************************************/ 00455 00456 uint writable () 00457 { 00458 return capacity - limit; 00459 } 00460 00461 /*********************************************************************** 00462 00463 Exposes the raw data buffer at the current write position, 00464 The delegate is provided with a void[] representing space 00465 available within the buffer at the current write position. 00466 00467 The delegate should return the appropriate number of bytes 00468 if it writes valid content, or IConduit.Eof on error. 00469 00470 Returns whatever the delegate returns. 00471 00472 ***********************************************************************/ 00473 00474 int write (int delegate (void[]) dg) 00475 { 00476 int count = dg (data [limit..capacity]); 00477 00478 if (count != IConduit.Eof) 00479 { 00480 limit += count; 00481 assert (limit <= capacity); 00482 } 00483 return count; 00484 } 00485 00486 /*********************************************************************** 00487 00488 Exposes the raw data buffer at the current read position. The 00489 delegate is provided with a void[] representing the available 00490 data, and should return zero to leave the current read position 00491 intact. 00492 00493 If the delegate consumes data, it should return the number of 00494 bytes consumed; or IConduit.Eof to indicate an error. 00495 00496 Returns whatever the delegate returns. 00497 00498 ***********************************************************************/ 00499 00500 int read (int delegate (void[]) dg) 00501 { 00502 00503 int count = dg (data [position..limit]); 00504 00505 if (count != IConduit.Eof) 00506 { 00507 position += count; 00508 assert (position <= limit); 00509 } 00510 return count; 00511 } 00512 00513 /*********************************************************************** 00514 00515 If we have some data left after an export, move it to 00516 front-of-buffer and set position to be just after the 00517 remains. This is for supporting certain conduits which 00518 choose to write just the initial portion of a request. 00519 00520 Limit is set to the amount of data remaining. Position 00521 is always reset to zero. 00522 00523 ***********************************************************************/ 00524 00525 IBuffer compress () 00526 { 00527 uint r = readable (); 00528 00529 if (position > 0 && r > 0) 00530 // content may overlap ... 00531 memcpy (&data[0], &data[position], r); 00532 00533 position = 0; 00534 limit = r; 00535 return this; 00536 } 00537 00538 /*********************************************************************** 00539 00540 flush the contents of this buffer to the related conduit. 00541 Throws an IOException on premature eof. 00542 00543 ***********************************************************************/ 00544 00545 void flush () 00546 { 00547 if (conduit) 00548 while (readable) 00549 if (conduit.write (this) == conduit.Eof) 00550 error (eofWrite); 00551 } 00552 00553 /*********************************************************************** 00554 00555 Reset 'position' and 'limit' to zero 00556 00557 ***********************************************************************/ 00558 00559 IBuffer clear () 00560 { 00561 position = limit = 0; 00562 return this; 00563 } 00564 00565 /*********************************************************************** 00566 00567 Returns the limit of readable content within this buffer 00568 00569 ***********************************************************************/ 00570 00571 uint getLimit () 00572 { 00573 return limit; 00574 } 00575 00576 /*********************************************************************** 00577 00578 Returns the total capacity of this buffer 00579 00580 ***********************************************************************/ 00581 00582 uint getCapacity () 00583 { 00584 return capacity; 00585 } 00586 00587 /*********************************************************************** 00588 00589 Returns the current read-position within this buffer 00590 00591 ***********************************************************************/ 00592 00593 uint getPosition () 00594 { 00595 return position; 00596 } 00597 00598 /*********************************************************************** 00599 00600 Returns the conduit associated with this buffer. Returns 00601 null if the buffer is purely memory based; that is, it's 00602 not backed by some external medium. 00603 00604 Buffers do not require an external conduit to operate, but 00605 it can be convenient to associate one. For example, methods 00606 read and write use it to import/export content as necessary. 00607 00608 ***********************************************************************/ 00609 00610 IConduit getConduit () 00611 { 00612 return conduit; 00613 } 00614 00615 /*********************************************************************** 00616 00617 Sets the external conduit associated with this buffer. 00618 00619 Buffers do not require an external conduit to operate, but 00620 it can be convenient to associate one. For example, methods 00621 read and write use it to import/export content as necessary. 00622 00623 ***********************************************************************/ 00624 00625 void setConduit (IConduit conduit) 00626 { 00627 this.conduit = conduit; 00628 } 00629 } 00630 00631 00632 /******************************************************************************* 00633 00634 Subclass to provide support for content growth. This is handy when 00635 you want to keep a buffer around as a scratchpad. 00636 00637 *******************************************************************************/ 00638 00639 class GrowableBuffer : Buffer 00640 { 00641 /*********************************************************************** 00642 00643 Create a GrowableBuffer with the specified initial size. 00644 00645 ***********************************************************************/ 00646 00647 this (uint size) 00648 { 00649 super (size); 00650 } 00651 00652 /*********************************************************************** 00653 00654 Overridable method to grow the buffer size when it becomes 00655 full. Default is to not grow at all. 00656 00657 ***********************************************************************/ 00658 00659 protected override bool grow (uint size) 00660 { 00661 if (data.length < uint.max / 2) 00662 { 00663 uint i = data.length * 2; 00664 if (i < size) 00665 i = size; 00666 data.length = i; 00667 capacity = i; 00668 return true; 00669 } 00670 return false; 00671 } 00672 } 00673 00674 00675 00676 /******************************************************************************* 00677 00678 Subclass to treat the buffer as a seekable entity, where all 00679 capacity is available for reading and/or writing. To achieve 00680 this we must effectively disable the 'limit' watermark, and 00681 locate write operations around 'position' instead. 00682 00683 *******************************************************************************/ 00684 00685 class MappedBuffer : Buffer 00686 { 00687 /*********************************************************************** 00688 00689 Construct an empty MappedBuffer 00690 00691 ***********************************************************************/ 00692 00693 this () 00694 { 00695 super (0); 00696 } 00697 00698 /*********************************************************************** 00699 00700 Close this mapped buffer 00701 00702 ***********************************************************************/ 00703 00704 abstract void close (); 00705 00706 /*********************************************************************** 00707 00708 Flush any dirty content out to the drive 00709 00710 ***********************************************************************/ 00711 00712 abstract void flush (); 00713 00714 /*********************************************************************** 00715 00716 Set the read/write position 00717 00718 ***********************************************************************/ 00719 00720 void setPosition (uint position) 00721 { 00722 this.position = position; 00723 } 00724 00725 /*********************************************************************** 00726 00727 Seek to the specified position within the buffer, and return 00728 the byte offset of the new location (relative to zero). 00729 00730 ***********************************************************************/ 00731 00732 long seek (long offset, ISeekable.SeekAnchor anchor) 00733 { 00734 long pos = capacity; 00735 00736 if (anchor == ISeekable.SeekAnchor.Begin) 00737 pos = offset; 00738 else 00739 if (anchor == ISeekable.SeekAnchor.End) 00740 pos -= offset; 00741 else 00742 pos = position + offset; 00743 00744 return position = pos; 00745 } 00746 00747 /*********************************************************************** 00748 00749 Return count of writable bytes available in buffer. This is 00750 calculated simply as capacity() - limit() 00751 00752 ***********************************************************************/ 00753 00754 override uint writable () 00755 { 00756 return capacity - position; 00757 } 00758 00759 /*********************************************************************** 00760 00761 Bulk copy of data from 'src'. Position is adjusted by 'size' 00762 bytes. 00763 00764 ***********************************************************************/ 00765 00766 override protected void copy (void *src, uint size) 00767 { 00768 data[position..position+size] = src[0..size]; 00769 position += size; 00770 } 00771 00772 /*********************************************************************** 00773 00774 Exposes the raw data buffer at the current write position, 00775 The delegate is provided with a void[] representing space 00776 available within the buffer at the current write position. 00777 00778 The delegate should return the appropriate number of bytes 00779 if it writes valid content, or IConduit.Eof on error. 00780 00781 Returns whatever the delegate returns. 00782 00783 ***********************************************************************/ 00784 00785 override int write (int delegate (void[]) dg) 00786 { 00787 int count = dg (data [position..capacity]); 00788 00789 if (count != IConduit.Eof) 00790 { 00791 position += count; 00792 assert (position <= capacity); 00793 } 00794 return count; 00795 } 00796 00797 /*********************************************************************** 00798 00799 Prohibit compress() from doing anything at all. 00800 00801 ***********************************************************************/ 00802 00803 override IBuffer compress () 00804 { 00805 return this; 00806 } 00807 00808 /*********************************************************************** 00809 00810 Prohibit clear() from doing anything at all. 00811 00812 ***********************************************************************/ 00813 00814 override IBuffer clear () 00815 { 00816 return this; 00817 } 00818 00819 /*********************************************************************** 00820 00821 Prohibit the setting of another IConduit 00822 00823 ***********************************************************************/ 00824 00825 override void setConduit (IConduit conduit) 00826 { 00827 assert (0); 00828 } 00829 }