00001 /******************************************************************************* 00002 00003 @file Buffer.d 00004 00005 Copyright (c) 2004 Kris Bell 00006 00007 This software is provided 'as-is', without any express or implied 00008 warranty. In no event will the authors be held liable for damages 00009 of any kind arising from the use of this software. 00010 00011 Permission is hereby granted to anyone to use this software for any 00012 purpose, including commercial applications, and to alter it and/or 00013 redistribute it freely, subject to the following restrictions: 00014 00015 1. The origin of this software must not be misrepresented; you must 00016 not claim that you wrote the original software. If you use this 00017 software in a product, an acknowledgment within documentation of 00018 said product would be appreciated but is not required. 00019 00020 2. Altered source versions must be plainly marked as such, and must 00021 not be misrepresented as being the original software. 00022 00023 3. This notice may not be removed or altered from any distribution 00024 of the source. 00025 00026 4. Derivative works are permitted, but they must carry this notice 00027 in full and credit the original source. 00028 00029 00030 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 00031 00032 00033 @version Initial version, March 2004 00034 @author Kris 00035 00036 00037 *******************************************************************************/ 00038 00039 module mango.io.Buffer; 00040 00041 private import mango.io.Exception; 00042 00043 private import mango.io.model.IConduit; 00044 00045 public import mango.io.model.IBuffer; 00046 00047 /****************************************************************************** 00048 00049 ******************************************************************************/ 00050 00051 extern (C) 00052 { 00053 void * memcpy (void *dst, void *src, uint); 00054 } 00055 00056 /******************************************************************************* 00057 00058 The basic premise behind this IO package is as follows: 00059 00060 @li the central concept is that of a buffer. The buffer acts 00061 as a queue (line) where items are removed from the front 00062 and new items are added to the back. Buffers are modeled 00063 by mango.io.model.IBuffer, and a concrete implementation is 00064 provided this class. 00065 00066 @li buffers can be written to directly, but a Reader and/or 00067 Writer are typically used to read & write formatted data. 00068 These readers & writers are bound to a specific buffer; 00069 often the same buffer. It's also perfectly legitimate to 00070 bind multiple writers to the same buffer; they will all 00071 behave serially as one would expect. The same applies to 00072 multiple readers on the same buffer. Readers and writers 00073 support two styles of IO: put/get, and the C++ style << 00074 and >> operators. All such operations can be chained. 00075 00076 @li Any class can be made compatable with the reader/writer 00077 framework by implementing the IReadable and/or IWritable 00078 interfaces. Each of these specify just a single method. 00079 00080 @li Buffers may also be tokenized. This is handy when one is 00081 dealing with text input, and/or the content suits a more 00082 fluid format than most typical readers & writers support. 00083 Tokens are mapped directly onto buffer content, so there 00084 is only minor overhead in using them. Tokens can be read 00085 and written by reader/writers also, using a more relaxed 00086 set of rules than those applied to integral IO. 00087 00088 @li buffers are sometimes memory-only, in which case there 00089 is nothing left to do when a reader (or tokenizer) hits 00090 end of buffer conditions. Other buffers are themselves 00091 bound to a Conduit. When this is the case, a reader will 00092 eventually cause the buffer to reload via its associated 00093 conduit. Previous buffer content will thus be lost. The 00094 same concept is applied to writers, whereby they flush 00095 the content of a full buffer to a bound conduit before 00096 continuing. 00097 00098 @li conduits provide virtualized access to external content, 00099 and represent things like files or Internet connections. 00100 They are just a different kind of stream. Conduits are 00101 modelled by mango.io.model.IConduit, and implemented via 00102 classes FileConduit and SocketConduit. Additional kinds 00103 of conduit are easy to construct: one either subclasses 00104 mango.io.Conduit, or implements mango.io.model.IConduit. 00105 A conduit reads and writes from/to a buffer in big chunks 00106 (typically the entire buffer). 00107 00108 00109 An example of how to append a buffer follows: 00110 00111 @code 00112 char[] foo = "to write some D"; 00113 00114 // create a small buffer 00115 Buffer buf = new Buffer (256); 00116 00117 // append some text directly to it 00118 buf.append("now is the time for all good men ").append(foo); 00119 00120 // output the combined string 00121 Stdout.put (buf.toString).cr(); 00122 @endcode 00123 00124 You might use a GrowableBuffer instead where you wish to 00125 append beyond a preset limit. One common usage of buffers 00126 is in conjunction with a conduit, such as FileConduit. Each 00127 Conduit exposes a factory for creating a buffer of the most 00128 appropriate size or flavour: 00129 00130 @code 00131 FileConduit fc = new FileConduit ("file.name"); 00132 Buffer buf = fc.createBuffer (); 00133 @endcode 00134 00135 However, this is typically hidden by higher level constructors 00136 such as those of Reader and Writer derivitives. For example: 00137 00138 @code 00139 FileConduit fc = new FileConduit ("file.name"); 00140 Reader reader = new Reader (fc); 00141 @endcode 00142 00143 There is indeed a buffer between the Reader and Conduit, but 00144 explicit construction is unecessary in many cases. See both 00145 Reader and Writer for examples of formatted IO. 00146 00147 *******************************************************************************/ 00148 00149 class Buffer : IBuffer 00150 { 00151 private void[] data; 00152 private uint limit; 00153 private uint capacity; 00154 private uint position; 00155 private IConduit conduit; 00156 00157 // some pre-constructed exceptions 00158 package static EofException eof; 00159 package static IOException udf; 00160 package static IOException ovf; 00161 00162 /*********************************************************************** 00163 00164 Ensure the buffer remains valid between method calls 00165 00166 ***********************************************************************/ 00167 00168 invariant 00169 { 00170 assert (position <= limit); 00171 assert (limit <= capacity); 00172 } 00173 00174 /*********************************************************************** 00175 00176 Construct an overflow exception for later use. 00177 00178 ***********************************************************************/ 00179 00180 static this () 00181 { 00182 ovf = new IOException ("output buffer overflow"); 00183 udf = new IOException ("input buffer underflow"); 00184 eof = new EofException ("premature end-of-file"); 00185 } 00186 00187 /*********************************************************************** 00188 00189 Construct a Buffer with the specified number of bytes. 00190 00191 ***********************************************************************/ 00192 00193 this (uint capacity = 0) 00194 { 00195 this (new ubyte[capacity]); 00196 } 00197 00198 /*********************************************************************** 00199 00200 Prime buffer with an application-supplied array. There is 00201 no readable data present, and writing begins at position 0. 00202 00203 ***********************************************************************/ 00204 00205 this (void[] data) 00206 { 00207 this (data, 0); 00208 } 00209 00210 /*********************************************************************** 00211 00212 Prime buffer with an application-supplied array, and 00213 indicate how much readable data is already there. A 00214 write operation will begin writing immediately after 00215 the existing content. 00216 00217 ***********************************************************************/ 00218 00219 this (void[] data, uint readable) 00220 { 00221 setContent (data, readable); 00222 } 00223 00224 /*********************************************************************** 00225 00226 Create an instance of an IBuffer. Use this when you 00227 don't know anything about the concrete implementation, 00228 and have only the IBuffer interface available 00229 00230 Returns a Buffer with no content. 00231 00232 ***********************************************************************/ 00233 00234 IBuffer create () 00235 { 00236 return new Buffer; 00237 } 00238 00239 /*********************************************************************** 00240 00241 Return the backing array 00242 00243 ***********************************************************************/ 00244 00245 void[] getContent () 00246 { 00247 return data; 00248 } 00249 00250 /*********************************************************************** 00251 00252 Set the backing array with all content readable. Writing 00253 to this will either flush it to an associated conduit, or 00254 raise an Eof condition. Use IBuffer.clear() to reset the 00255 content (make it all writable). 00256 00257 ***********************************************************************/ 00258 00259 IBuffer setValidContent (void[] data) 00260 { 00261 return setContent (data, data.length); 00262 } 00263 00264 /*********************************************************************** 00265 00266 Set the backing array with some content readable. Writing 00267 to this will either flush it to an associated conduit, or 00268 raise an Eof condition. Use IBuffer.clear() to reset the 00269 content (make it all writable). 00270 00271 ***********************************************************************/ 00272 00273 IBuffer setContent (void[] data, uint readable) 00274 { 00275 this.data = data; 00276 this.limit = readable; 00277 this.capacity = data.length; 00278 00279 // reset to start of input 00280 this.position = 0; 00281 00282 return this; 00283 } 00284 00285 /*********************************************************************** 00286 00287 Overridable method to grow the buffer size when it becomes 00288 full. Default is to not grow at all. 00289 00290 ***********************************************************************/ 00291 00292 protected bool grow (uint size) 00293 { 00294 return false; 00295 } 00296 00297 /*********************************************************************** 00298 00299 Bulk copy of data from 'src'. Limit is adjusted by 'size' 00300 bytes. 00301 00302 ***********************************************************************/ 00303 00304 protected void copy (void *src, uint size) 00305 { 00306 data[limit..limit+size] = src[0..size]; 00307 limit += size; 00308 } 00309 00310 /*********************************************************************** 00311 00312 Read a chunk of data from the buffer, loading from the 00313 conduit as necessary. The specified number of bytes is 00314 loaded into the buffer, and marked as having been read 00315 when the 'eat' parameter is set true. When 'eat' is set 00316 false, the read position is not adjusted. 00317 00318 Returns the corresponding buffer slice when successful, 00319 or null if there's not enough data available (Eof; Eob). 00320 00321 ***********************************************************************/ 00322 00323 void[] get (uint size, bool eat = true) 00324 { 00325 if (size > readable) 00326 { 00327 if (conduit is null) 00328 throw udf; 00329 00330 // make some space? This will try to leave as much content 00331 // in the buffer as possible, such that entire records may 00332 // be aliased directly from within. 00333 if (size > writable) 00334 { 00335 if (size > capacity) 00336 throw udf; 00337 compress (); 00338 } 00339 00340 // populate tail of buffer with new content 00341 do { 00342 if (conduit.read (this) == IConduit.Eof) 00343 throw eof; 00344 } while (size > readable); 00345 } 00346 00347 uint i = position; 00348 if (eat) 00349 position += size; 00350 return data [i .. i + size]; 00351 } 00352 00353 /*********************************************************************** 00354 00355 Append an array of data to this buffer, and flush to the 00356 conduit as necessary. Returns a chaining reference if all 00357 data was written; throws an IOException indicating eof or 00358 eob if not. 00359 00360 This is often used in lieu of a Writer. 00361 00362 ***********************************************************************/ 00363 00364 IBuffer append (void[] src) 00365 { 00366 uint size = src.length; 00367 00368 if (size > writable) 00369 // can we write externally? 00370 if (conduit) 00371 { 00372 conduit.flush (this); 00373 00374 // check for pathological case 00375 if (size > capacity) 00376 { 00377 void[] tmp = data; 00378 00379 setContent (src, size); 00380 conduit.flush (this); 00381 setContent (tmp, 0); 00382 return this; 00383 } 00384 } 00385 else 00386 // try to grow buffer ... 00387 if (! grow (size)) 00388 throw ovf; 00389 00390 copy (src, size); 00391 return this; 00392 } 00393 00394 /*********************************************************************** 00395 00396 Return a char[] slice of the buffer up to the limit of 00397 valid content. 00398 00399 ***********************************************************************/ 00400 00401 override char[] toString () 00402 { 00403 return cast(char[]) data[position..limit]; 00404 } 00405 00406 /*********************************************************************** 00407 00408 Skip ahead by the specified number of bytes, streaming from 00409 the associated conduit as necessary. 00410 00411 Can also reverse the read position by 'size' bytes. This may 00412 be used to support lookahead-type operations. 00413 00414 Returns true if successful, false otherwise. 00415 00416 ***********************************************************************/ 00417 00418 bool skip (int size) 00419 { 00420 if (size < 0) 00421 { 00422 size = -size; 00423 if (position >= size) 00424 { 00425 position -= size; 00426 return true; 00427 } 00428 return false; 00429 } 00430 return get (size) !== null; 00431 } 00432 00433 /*********************************************************************** 00434 00435 Return count of readable bytes remaining in buffer. This is 00436 calculated simply as limit() - position() 00437 00438 ***********************************************************************/ 00439 00440 uint readable () 00441 { 00442 return limit - position; 00443 } 00444 00445 /*********************************************************************** 00446 00447 Return count of writable bytes available in buffer. This is 00448 calculated simply as capacity() - limit() 00449 00450 ***********************************************************************/ 00451 00452 uint writable () 00453 { 00454 return capacity - limit; 00455 } 00456 00457 /*********************************************************************** 00458 00459 Exposes the raw data buffer at the current write position, 00460 The delegate is provided with a void[] representing space 00461 available within the buffer at the current write position. 00462 00463 The delegate should return the appropriate number of bytes 00464 if it writes valid content, or IConduit.Eof on error. 00465 00466 Returns whatever the delegate returns. 00467 00468 ***********************************************************************/ 00469 00470 int write (int delegate (void[]) dg) 00471 { 00472 int count = dg (data [limit..capacity]); 00473 00474 if (count != IConduit.Eof) 00475 { 00476 limit += count; 00477 assert (limit <= capacity); 00478 } 00479 return count; 00480 } 00481 00482 /*********************************************************************** 00483 00484 Exposes the raw data buffer at the current read position. The 00485 delegate is provided with a void[] representing the available 00486 data, and should return zero to leave the current read position 00487 intact. 00488 00489 If the delegate consumes data, it should return the number of 00490 bytes consumed; or IConduit.Eof to indicate an error. 00491 00492 Returns whatever the delegate returns. 00493 00494 ***********************************************************************/ 00495 00496 int read (int delegate (void[]) dg) 00497 { 00498 00499 int count = dg (data [position..limit]); 00500 00501 if (count != IConduit.Eof) 00502 { 00503 position += count; 00504 assert (position <= limit); 00505 } 00506 return count; 00507 } 00508 00509 /*********************************************************************** 00510 00511 If we have some data left after an export, move it to 00512 front-of-buffer and set position to be just after the 00513 remains. This is for supporting certain conduits which 00514 choose to write just the initial portion of a request. 00515 00516 Limit is set to the amount of data remaining. Position 00517 is always reset to zero. 00518 00519 ***********************************************************************/ 00520 00521 IBuffer compress () 00522 { 00523 uint r = readable (); 00524 00525 if (position > 0 && r > 0) 00526 // content may overlap ... 00527 memcpy (&data[0], &data[position], r); 00528 00529 position = 0; 00530 limit = r; 00531 return this; 00532 } 00533 00534 /*********************************************************************** 00535 00536 flush the contents of this buffer to the related conduit. 00537 Throws an IOException on premature eof. 00538 00539 ***********************************************************************/ 00540 00541 void flush () 00542 { 00543 if (conduit) 00544 conduit.flush (this); 00545 } 00546 00547 /*********************************************************************** 00548 00549 Reset 'position' and 'limit' to zero 00550 00551 ***********************************************************************/ 00552 00553 IBuffer clear () 00554 { 00555 position = limit = 0; 00556 return this; 00557 } 00558 00559 /*********************************************************************** 00560 00561 Returns the limit of readable content within this buffer 00562 00563 ***********************************************************************/ 00564 00565 uint getLimit () 00566 { 00567 return limit; 00568 } 00569 00570 /*********************************************************************** 00571 00572 Returns the total capacity of this buffer 00573 00574 ***********************************************************************/ 00575 00576 uint getCapacity () 00577 { 00578 return capacity; 00579 } 00580 00581 /*********************************************************************** 00582 00583 Returns the current read-position within this buffer 00584 00585 ***********************************************************************/ 00586 00587 uint getPosition () 00588 { 00589 return position; 00590 } 00591 00592 /*********************************************************************** 00593 00594 Returns the conduit associated with this buffer. Returns 00595 null if the buffer is purely memory based; that is, it's 00596 not backed by some external medium. 00597 00598 Buffers do not require an external conduit to operate, but 00599 it can be convenient to associate one. For example, methods 00600 read and write use it to import/export content as necessary. 00601 00602 ***********************************************************************/ 00603 00604 IConduit getConduit () 00605 { 00606 return conduit; 00607 } 00608 00609 /*********************************************************************** 00610 00611 Sets the external conduit associated with this buffer. 00612 00613 Buffers do not require an external conduit to operate, but 00614 it can be convenient to associate one. For example, methods 00615 read and write use it to import/export content as necessary. 00616 00617 ***********************************************************************/ 00618 00619 void setConduit (IConduit conduit) 00620 { 00621 this.conduit = conduit; 00622 } 00623 } 00624 00625 00626 /******************************************************************************* 00627 00628 Subclass to provide support for content growth. This is handy when 00629 you want to keep a buffer around as a scratchpad. 00630 00631 *******************************************************************************/ 00632 00633 class GrowableBuffer : Buffer 00634 { 00635 /*********************************************************************** 00636 00637 Create a GrowableBuffer with the specified initial size. 00638 00639 ***********************************************************************/ 00640 00641 this (uint size) 00642 { 00643 super (size); 00644 } 00645 00646 /*********************************************************************** 00647 00648 Overridable method to grow the buffer size when it becomes 00649 full. Default is to not grow at all. 00650 00651 ***********************************************************************/ 00652 00653 protected override bool grow (uint size) 00654 { 00655 if (data.length < uint.max / 2) 00656 { 00657 uint i = data.length * 2; 00658 if (i < size) 00659 i = size; 00660 data.length = i; 00661 capacity = i; 00662 return true; 00663 } 00664 return false; 00665 } 00666 } 00667 00668 00669 00670 /******************************************************************************* 00671 00672 Subclass to treat the buffer as a seekable entity, where all 00673 capacity is available for reading and/or writing. To achieve 00674 this we must effectively disable the 'limit' watermark, and 00675 locate write operations around 'position' instead. 00676 00677 *******************************************************************************/ 00678 00679 class MappedBuffer : Buffer 00680 { 00681 /*********************************************************************** 00682 00683 Construct an empty MappedBuffer 00684 00685 ***********************************************************************/ 00686 00687 this () 00688 { 00689 super (0); 00690 } 00691 00692 /*********************************************************************** 00693 00694 Close this mapped buffer 00695 00696 ***********************************************************************/ 00697 00698 abstract void close (); 00699 00700 /*********************************************************************** 00701 00702 Flush any dirty content out to the drive 00703 00704 ***********************************************************************/ 00705 00706 abstract void flush (); 00707 00708 /*********************************************************************** 00709 00710 Set the read/write position 00711 00712 ***********************************************************************/ 00713 00714 void setPosition (uint position) 00715 { 00716 this.position = position; 00717 } 00718 00719 /*********************************************************************** 00720 00721 Seek to the specified position within the buffer, and return 00722 the byte offset of the new location (relative to zero). 00723 00724 ***********************************************************************/ 00725 00726 long seek (long offset, ISeekable.SeekAnchor anchor) 00727 { 00728 long pos = capacity; 00729 00730 if (anchor == ISeekable.SeekAnchor.Begin) 00731 pos = offset; 00732 else 00733 if (anchor == ISeekable.SeekAnchor.End) 00734 pos -= offset; 00735 else 00736 pos = position + offset; 00737 00738 return position = pos; 00739 } 00740 00741 /*********************************************************************** 00742 00743 Return count of writable bytes available in buffer. This is 00744 calculated simply as capacity() - limit() 00745 00746 ***********************************************************************/ 00747 00748 override uint writable () 00749 { 00750 return capacity - position; 00751 } 00752 00753 /*********************************************************************** 00754 00755 Bulk copy of data from 'src'. Position is adjusted by 'size' 00756 bytes. 00757 00758 ***********************************************************************/ 00759 00760 override protected void copy (void *src, uint size) 00761 { 00762 data[position..position+size] = src[0..size]; 00763 position += size; 00764 } 00765 00766 /*********************************************************************** 00767 00768 Exposes the raw data buffer at the current write position, 00769 The delegate is provided with a void[] representing space 00770 available within the buffer at the current write position. 00771 00772 The delegate should return the appropriate number of bytes 00773 if it writes valid content, or IConduit.Eof on error. 00774 00775 Returns whatever the delegate returns. 00776 00777 ***********************************************************************/ 00778 00779 override int write (int delegate (void[]) dg) 00780 { 00781 int count = dg (data [position..capacity]); 00782 00783 if (count != IConduit.Eof) 00784 { 00785 position += count; 00786 assert (position <= capacity); 00787 } 00788 return count; 00789 } 00790 00791 /*********************************************************************** 00792 00793 Prohibit compress() from doing anything at all. 00794 00795 ***********************************************************************/ 00796 00797 override IBuffer compress () 00798 { 00799 return this; 00800 } 00801 00802 /*********************************************************************** 00803 00804 Prohibit clear() from doing anything at all. 00805 00806 ***********************************************************************/ 00807 00808 override IBuffer clear () 00809 { 00810 return this; 00811 } 00812 00813 /*********************************************************************** 00814 00815 Prohibit the setting of another IConduit 00816 00817 ***********************************************************************/ 00818 00819 override void setConduit (IConduit conduit) 00820 { 00821 assert (0); 00822 } 00823 }