00001 /******************************************************************************* 00002 00003 @file Buffer.d 00004 00005 Copyright (c) 2004 Kris Bell 00006 00007 This software is provided 'as-is', without any express or implied 00008 warranty. In no event will the authors be held liable for damages 00009 of any kind arising from the use of this software. 00010 00011 Permission is hereby granted to anyone to use this software for any 00012 purpose, including commercial applications, and to alter it and/or 00013 redistribute it freely, subject to the following restrictions: 00014 00015 1. The origin of this software must not be misrepresented; you must 00016 not claim that you wrote the original software. If you use this 00017 software in a product, an acknowledgment within documentation of 00018 said product would be appreciated but is not required. 00019 00020 2. Altered source versions must be plainly marked as such, and must 00021 not be misrepresented as being the original software. 00022 00023 3. This notice may not be removed or altered from any distribution 00024 of the source. 00025 00026 4. Derivative works are permitted, but they must carry this notice 00027 in full and credit the original source. 00028 00029 00030 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 00031 00032 00033 @version Initial version; March 2004 00034 00035 @author Kris 00036 00037 00038 *******************************************************************************/ 00039 00040 module mango.io.Buffer; 00041 00042 private import mango.io.Exception; 00043 00044 public import mango.io.model.IBuffer; 00045 00046 public import mango.io.model.IConduit; 00047 00048 00049 /****************************************************************************** 00050 00051 ******************************************************************************/ 00052 00053 extern (C) 00054 { 00055 void * memcpy (void *dst, void *src, uint); 00056 } 00057 00058 /******************************************************************************* 00059 00060 The basic premise behind this IO package is as follows: 00061 00062 @li The central concept is that of a buffer. The buffer acts 00063 as a queue (line) where items are removed from the front 00064 and new items are added to the back. Buffers are modeled 00065 by mango.io.model.IBuffer, and a concrete implementation is 00066 provided this class. 00067 00068 @li Buffers can be written to directly, but a Reader and/or 00069 Writer are typically used to read & write formatted data. 00070 These readers & writers are bound to a specific buffer; 00071 often the same buffer. It's also perfectly legitimate to 00072 bind multiple writers to the same buffer; they will all 00073 behave serially as one would expect. The same applies to 00074 multiple readers on the same buffer. Readers and writers 00075 support three styles of IO: put/get, the C++ style << & 00076 >> operators, and the () whisper style. All operations 00077 can be chained. 00078 00079 @li Any class can be made compatable with the reader/writer 00080 framework by implementing the IReadable and/or IWritable 00081 interfaces. Each of these specify just a single method. 00082 Once compatable, the class can simply be passed to the 00083 reader/writer as if it were native data. 00084 00085 @li Buffers may also be tokenized. This is handy when one is 00086 dealing with text input, and/or the content suits a more 00087 fluid format than most typical readers & writers support. 00088 Tokens are mapped directly onto buffer content, so there 00089 is only minor overhead in using them. Tokens can be read 00090 and written by reader/writers also, using a more relaxed 00091 set of rules than those applied to discrete I/O. 00092 00093 @li Buffers are sometimes memory-only, in which case there 00094 is nothing left to do when a reader (or tokenizer) hits 00095 end of buffer conditions. Other buffers are themselves 00096 bound to a Conduit. When this is the case, a reader will 00097 eventually cause the buffer to reload via its associated 00098 conduit. Previous buffer content will thus be lost. The 00099 same concept is applied to writers, whereby they flush 00100 the content of a full buffer to a bound conduit before 00101 continuing. 00102 00103 @li Conduits provide virtualized access to external content, 00104 and represent things like files or Internet connections. 00105 They are just a different kind of stream. Conduits are 00106 modelled by mango.io.model.IConduit, and implemented via 00107 classes FileConduit and SocketConduit. Additional kinds 00108 of conduit are easy to construct: one either subclasses 00109 mango.io.Conduit, or implements mango.io.model.IConduit. 00110 A conduit reads and writes from/to a buffer in big chunks 00111 (typically the entire buffer). 00112 00113 @li Conduits may have one or more filters attached. These 00114 will process content as it flows back and forth across 00115 the conduit. Filter examples include compression, utf 00116 transcoding, and endian transformation. These filters 00117 apply to the entire scope of the conduit, rather than 00118 being specific to one data-type or another. 00119 00120 @li Readers & writers may have a transcoder attached. The 00121 role of a transcoder is to aid in converting between each 00122 representation of text (utf8, utf16, utf32). They're used 00123 to normalize string I/O according to a standard text-type. 00124 By default there is no transcoder attached, and the type 00125 is therefore considered "raw". 00126 00127 00128 An example of how to append a buffer follows: 00129 00130 @code 00131 char[] foo = "to write some D"; 00132 00133 // create a small buffer 00134 auto buf = new Buffer (256); 00135 00136 // append some text directly to it 00137 buf.append("now is the time for all good men ").append(foo); 00138 00139 // output the combined string 00140 Cout (buf.toString); 00141 @endcode 00142 00143 Alternatively, one might use a Writer to append the buffer: 00144 00145 @code 00146 auto write = new Writer (new Buffer(256)); 00147 write ("now is the time for all good men "c) (foo); 00148 @endcode 00149 00150 Or, using printf-like formatting: 00151 00152 @code 00153 auto write = new DisplayWriter (new Buffer(256)); 00154 write.print ("now is the time for %d good men %s", 3, foo); 00155 @endcode 00156 00157 You might use a GrowBuffer instead where you wish to 00158 append beyond a preset limit. One common usage of buffers 00159 is in conjunction with a conduit, such as FileConduit. Each 00160 conduit exposes a preferred-size for its associated buffers, 00161 utilized during Buffer construction: 00162 00163 @code 00164 auto file = new FileConduit ("file.name"); 00165 auto buf = new Buffer (file); 00166 @endcode 00167 00168 However, this is typically hidden by higher level constructors 00169 such as those of Reader and Writer derivitives. For example: 00170 00171 @code 00172 auto file = new FileConduit ("file.name"); 00173 auto read = new Reader (file); 00174 @endcode 00175 00176 There is indeed a buffer between the Reader and Conduit, but 00177 explicit construction is unecessary in most cases. See both 00178 Reader and Writer for examples of formatted IO. 00179 00180 Stdout is a predefined DisplayWriter, attached to a conduit 00181 representing the console. Thus, all conduit operations are 00182 legitimate on Stdout and Stderr: 00183 00184 @code 00185 Stdout.conduit.copy (new FileConduit ("readme.txt")); 00186 @endcode 00187 00188 In addition to the standard writer facilities, Stdout also has 00189 support for formatted output: 00190 00191 @code 00192 Stdout.println ("now is the time for %d good men %s", 3, foo); 00193 @endcode 00194 00195 Buffers are useful for many purposes within Mango, but there 00196 are times when it is more straightforward to avoid them. For 00197 such cases, conduit derivatives (such as FileConduit) support 00198 direct I/O via a pair of read() and write() methods. These 00199 alternate methods will also invoke any attached filters. 00200 00201 00202 *******************************************************************************/ 00203 00204 class Buffer : IBuffer 00205 { 00206 protected void[] data; 00207 protected Style style; 00208 protected uint limit; 00209 protected uint capacity; 00210 protected uint position; 00211 protected IConduit conduit; 00212 00213 private static char[] overflow = "output buffer overflow"; 00214 private static char[] underflow = "input buffer underflow"; 00215 private static char[] eofRead = "end-of-file whilst reading"; 00216 private static char[] eofWrite = "end-of-file whilst writing"; 00217 00218 /*********************************************************************** 00219 00220 Ensure the buffer remains valid between method calls 00221 00222 ***********************************************************************/ 00223 00224 invariant 00225 { 00226 assert (position <= limit); 00227 assert (limit <= capacity); 00228 } 00229 00230 /*********************************************************************** 00231 00232 Construct a Buffer upon the provided conduit 00233 00234 ***********************************************************************/ 00235 00236 this (IConduit conduit) 00237 { 00238 this (conduit.bufferSize()); 00239 this.style = conduit.isTextual ? Text : Binary; 00240 setConduit (conduit); 00241 } 00242 00243 /*********************************************************************** 00244 00245 Construct a Buffer with the specified number of bytes. 00246 00247 ***********************************************************************/ 00248 00249 this (uint capacity=0) 00250 { 00251 this (new ubyte[capacity]); 00252 } 00253 00254 /*********************************************************************** 00255 00256 Prime buffer with an application-supplied array. There is 00257 no readable data present, and writing begins at position 0. 00258 00259 ***********************************************************************/ 00260 00261 this (void[] data) 00262 { 00263 this (data, 0); 00264 } 00265 00266 /*********************************************************************** 00267 00268 Prime buffer with an application-supplied array, and 00269 indicate how much readable data is already there. A 00270 write operation will begin writing immediately after 00271 the existing content. 00272 00273 ***********************************************************************/ 00274 00275 this (void[] data, uint readable) 00276 { 00277 setContent (data, readable); 00278 } 00279 00280 /*********************************************************************** 00281 00282 Throw an exception with the provided message 00283 00284 ***********************************************************************/ 00285 00286 final void error (char[] msg) 00287 { 00288 throw new IOException (msg); 00289 } 00290 00291 /*********************************************************************** 00292 00293 Return the backing array 00294 00295 ***********************************************************************/ 00296 00297 protected void[] getContent () 00298 { 00299 return data; 00300 } 00301 00302 /*********************************************************************** 00303 00304 Return style of buffer 00305 00306 ***********************************************************************/ 00307 00308 Style getStyle () 00309 { 00310 return style; 00311 } 00312 00313 /*********************************************************************** 00314 00315 Set the backing array with all content readable. Writing 00316 to this will either flush it to an associated conduit, or 00317 raise an Eof condition. Use IBuffer.clear() to reset the 00318 content (make it all writable). 00319 00320 ***********************************************************************/ 00321 00322 IBuffer setValidContent (void[] data) 00323 { 00324 return setContent (data, data.length); 00325 } 00326 00327 /*********************************************************************** 00328 00329 Set the backing array with some content readable. Writing 00330 to this will either flush it to an associated conduit, or 00331 raise an Eof condition. Use IBuffer.clear() to reset the 00332 content (make it all writable). 00333 00334 ***********************************************************************/ 00335 00336 IBuffer setContent (void[] data, uint readable=0) 00337 { 00338 this.data = data; 00339 this.limit = readable; 00340 this.capacity = data.length; 00341 00342 // reset to start of input 00343 this.position = 0; 00344 00345 return this; 00346 } 00347 00348 /*********************************************************************** 00349 00350 Overridable method to grow the buffer to the specified size 00351 when it becomes full. Default is to not grow at all. 00352 00353 ***********************************************************************/ 00354 00355 protected bool grow (uint size) 00356 { 00357 return false; 00358 } 00359 00360 /*********************************************************************** 00361 00362 Bulk copy of data from 'src'. Limit is adjusted by 'size' 00363 bytes. 00364 00365 ***********************************************************************/ 00366 00367 protected void copy (void *src, uint size) 00368 { 00369 data[limit..limit+size] = src[0..size]; 00370 limit += size; 00371 } 00372 00373 /*********************************************************************** 00374 00375 Read a chunk of data from the buffer, loading from the 00376 conduit as necessary. The specified number of bytes is 00377 loaded into the buffer, and marked as having been read 00378 when the 'eat' parameter is set true. When 'eat' is set 00379 false, the read position is not adjusted. 00380 00381 Returns the corresponding buffer slice when successful, 00382 or null if there's not enough data available (Eof; Eob). 00383 00384 ***********************************************************************/ 00385 00386 void[] get (uint size, bool eat = true) 00387 { 00388 if (size > readable) 00389 { 00390 if (conduit is null) 00391 error (underflow); 00392 00393 // make some space? This will try to leave as much content 00394 // in the buffer as possible, such that entire records may 00395 // be aliased directly from within. 00396 if (size > writable) 00397 { 00398 if (size > capacity) 00399 error (underflow); 00400 compress (); 00401 } 00402 00403 // populate tail of buffer with new content 00404 do { 00405 if (fill(conduit) == IConduit.Eof) 00406 error (eofRead); 00407 } while (size > readable); 00408 } 00409 00410 uint i = position; 00411 if (eat) 00412 position += size; 00413 return data [i .. i + size]; 00414 } 00415 00416 /*********************************************************************** 00417 00418 Append an array of data to this buffer, and flush to the 00419 conduit as necessary. Returns a chaining reference if all 00420 data was written; throws an IOException indicating eof or 00421 eob if not. 00422 00423 This is often used in lieu of a Writer. 00424 00425 ***********************************************************************/ 00426 00427 IBuffer append (void[] src) 00428 { 00429 uint size = src.length; 00430 00431 if (size > writable) 00432 // can we write externally? 00433 if (conduit) 00434 { 00435 flush (); 00436 00437 // check for pathological case 00438 if (size > capacity) 00439 { 00440 conduit.flush (src); 00441 return this; 00442 } 00443 } 00444 else 00445 // try to grow buffer ... 00446 if (! grow (size)) 00447 error (overflow); 00448 00449 copy (src, size); 00450 return this; 00451 } 00452 00453 /*********************************************************************** 00454 00455 Return a char[] slice of the buffer up to the limit of 00456 valid content. 00457 00458 ***********************************************************************/ 00459 00460 override char[] toString () 00461 { 00462 return cast(char[]) data[position..limit]; 00463 } 00464 00465 /*********************************************************************** 00466 00467 Skip ahead by the specified number of bytes, streaming from 00468 the associated conduit as necessary. 00469 00470 Can also reverse the read position by 'size' bytes. This may 00471 be used to support lookahead-type operations. 00472 00473 Returns true if successful, false otherwise. 00474 00475 ***********************************************************************/ 00476 00477 bool skip (int size) 00478 { 00479 if (size < 0) 00480 { 00481 size = -size; 00482 if (position >= size) 00483 { 00484 position -= size; 00485 return true; 00486 } 00487 return false; 00488 } 00489 return cast(bool) (get (size) !is null); 00490 } 00491 00492 /*********************************************************************** 00493 00494 Return count of readable bytes remaining in buffer. This is 00495 calculated simply as limit() - position() 00496 00497 ***********************************************************************/ 00498 00499 uint readable () 00500 { 00501 return limit - position; 00502 } 00503 00504 /*********************************************************************** 00505 00506 Return count of writable bytes available in buffer. This is 00507 calculated simply as capacity() - limit() 00508 00509 ***********************************************************************/ 00510 00511 uint writable () 00512 { 00513 return capacity - limit; 00514 } 00515 00516 /*********************************************************************** 00517 00518 Exposes the raw data buffer at the current write position, 00519 The delegate is provided with a void[] representing space 00520 available within the buffer at the current write position. 00521 00522 The delegate should return the appropriate number of bytes 00523 if it writes valid content, or IConduit.Eof on error. 00524 00525 Returns whatever the delegate returns. 00526 00527 ***********************************************************************/ 00528 00529 uint write (uint delegate (void[]) dg) 00530 { 00531 int count = dg (data [limit..capacity]); 00532 00533 if (count != IConduit.Eof) 00534 { 00535 limit += count; 00536 assert (limit <= capacity); 00537 } 00538 return count; 00539 } 00540 00541 /*********************************************************************** 00542 00543 Exposes the raw data buffer at the current read position. The 00544 delegate is provided with a void[] representing the available 00545 data, and should return zero to leave the current read position 00546 intact. 00547 00548 If the delegate consumes data, it should return the number of 00549 bytes consumed; or IConduit.Eof to indicate an error. 00550 00551 Returns whatever the delegate returns. 00552 00553 ***********************************************************************/ 00554 00555 uint read (uint delegate (void[]) dg) 00556 { 00557 00558 int count = dg (data [position..limit]); 00559 00560 if (count != IConduit.Eof) 00561 { 00562 position += count; 00563 assert (position <= limit); 00564 } 00565 return count; 00566 } 00567 00568 /*********************************************************************** 00569 00570 If we have some data left after an export, move it to 00571 front-of-buffer and set position to be just after the 00572 remains. This is for supporting certain conduits which 00573 choose to write just the initial portion of a request. 00574 00575 Limit is set to the amount of data remaining. Position 00576 is always reset to zero. 00577 00578 ***********************************************************************/ 00579 00580 IBuffer compress () 00581 { 00582 uint r = readable (); 00583 00584 if (position > 0 && r > 0) 00585 // content may overlap ... 00586 memcpy (&data[0], &data[position], r); 00587 00588 position = 0; 00589 limit = r; 00590 return this; 00591 } 00592 00593 /*********************************************************************** 00594 00595 Try to fill the available buffer with content from the 00596 specified conduit. In particular, we will never ask to 00597 read less than 32 bytes. This permits conduit-filters 00598 to operate within a known environment. 00599 00600 Returns the number of bytes read, or throws an underflow 00601 error if there nowhere to read from 00602 00603 ***********************************************************************/ 00604 00605 uint fill () 00606 { 00607 if (! conduit) 00608 error (underflow); 00609 return fill (conduit); 00610 } 00611 00612 /*********************************************************************** 00613 00614 Try to fill the available buffer with content from the 00615 specified conduit. In particular, we will never ask to 00616 read less than 32 bytes ~ this permits conduit-filters 00617 to operate within a known environment. We also try to 00618 read as much as possible by clearing the buffer when 00619 all current content has been eaten. 00620 00621 Returns the number of bytes read, or Conduit.Eof 00622 00623 ***********************************************************************/ 00624 00625 uint fill (IConduit conduit) 00626 { 00627 if (readable is 0) 00628 clear(); 00629 else 00630 if (writable < 32) 00631 if (compress().writable < 32) 00632 error ("input buffer is too small"); 00633 00634 return write (&conduit.read); 00635 } 00636 00637 /*********************************************************************** 00638 00639 make some room in the buffer 00640 00641 ***********************************************************************/ 00642 00643 void makeRoom (int space) 00644 { 00645 if (conduit) 00646 drain (); 00647 else 00648 if (! grow (space)) 00649 error (overflow); 00650 } 00651 00652 /*********************************************************************** 00653 00654 Write as much of the buffer that the associated conduit 00655 can consume. 00656 00657 Returns the number of bytes written, or Conduit.Eof 00658 00659 ***********************************************************************/ 00660 00661 uint drain () 00662 { 00663 uint ret = read (&conduit.write); 00664 if (ret == conduit.Eof) 00665 error (eofWrite); 00666 00667 compress (); 00668 return ret; 00669 } 00670 00671 /*********************************************************************** 00672 00673 flush the contents of this buffer to the related conduit. 00674 Throws an IOException on premature eof. 00675 00676 ***********************************************************************/ 00677 00678 void flush () 00679 { 00680 if (conduit) 00681 if (conduit.flush (data [position..limit])) 00682 clear(); 00683 else 00684 error (eofWrite); 00685 } 00686 00687 /*********************************************************************** 00688 00689 Reset 'position' and 'limit' to zero 00690 00691 ***********************************************************************/ 00692 00693 IBuffer clear () 00694 { 00695 position = limit = 0; 00696 return this; 00697 } 00698 00699 /*********************************************************************** 00700 00701 Returns the limit of readable content within this buffer 00702 00703 ***********************************************************************/ 00704 00705 uint getLimit () 00706 { 00707 return limit; 00708 } 00709 00710 /*********************************************************************** 00711 00712 Returns the total capacity of this buffer 00713 00714 ***********************************************************************/ 00715 00716 uint getCapacity () 00717 { 00718 return capacity; 00719 } 00720 00721 /*********************************************************************** 00722 00723 Returns the current read-position within this buffer 00724 00725 ***********************************************************************/ 00726 00727 uint getPosition () 00728 { 00729 return position; 00730 } 00731 00732 /*********************************************************************** 00733 00734 Returns the conduit associated with this buffer. Returns 00735 null if the buffer is purely memory based; that is, it's 00736 not backed by some external medium. 00737 00738 Buffers do not require an external conduit to operate, but 00739 it can be convenient to associate one. For example, methods 00740 fill() & drain() use it to import/export content as necessary. 00741 00742 ***********************************************************************/ 00743 00744 IConduit getConduit () 00745 { 00746 return conduit; 00747 } 00748 00749 /*********************************************************************** 00750 00751 Sets the external conduit associated with this buffer. 00752 00753 Buffers do not require an external conduit to operate, but 00754 it can be convenient to associate one. For example, methods 00755 fill() & drain() use it to import/export content as necessary. 00756 00757 ***********************************************************************/ 00758 00759 void setConduit (IConduit conduit) 00760 { 00761 this.conduit = conduit; 00762 } 00763 }