Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

Buffer.d

Go to the documentation of this file.
00001 /*******************************************************************************
00002 
00003         @file Buffer.d
00004 
00005         Copyright (c) 2004 Kris Bell
00006         
00007         This software is provided 'as-is', without any express or implied
00008         warranty. In no event will the authors be held liable for damages
00009         of any kind arising from the use of this software.
00010         
00011         Permission is hereby granted to anyone to use this software for any 
00012         purpose, including commercial applications, and to alter it and/or 
00013         redistribute it freely, subject to the following restrictions:
00014         
00015         1. The origin of this software must not be misrepresented; you must 
00016            not claim that you wrote the original software. If you use this 
00017            software in a product, an acknowledgment within documentation of 
00018            said product would be appreciated but is not required.
00019 
00020         2. Altered source versions must be plainly marked as such, and must 
00021            not be misrepresented as being the original software.
00022 
00023         3. This notice may not be removed or altered from any distribution
00024            of the source.
00025 
00026         4. Derivative works are permitted, but they must carry this notice
00027            in full and credit the original source.
00028 
00029 
00030                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
00031 
00032       
00033         @version        Initial version, March 2004
00034         @author         Kris
00035 
00036 
00037 *******************************************************************************/
00038 
00039 module mango.io.Buffer;
00040 
00041 private import  mango.io.Exception;
00042 
00043 private import  mango.io.model.IConduit;
00044 
00045 public  import  mango.io.model.IBuffer;
00046 
00047 /******************************************************************************
00048 
00049 ******************************************************************************/
00050 
00051 extern (C)
00052 {
00053         void * memcpy (void *dst, void *src, uint);
00054 }       
00055 
00056 /*******************************************************************************
00057 
00058         The basic premise behind this IO package is as follows:
00059 
00060         @li the central concept is that of a buffer. The buffer acts
00061            as a queue (line) where items are removed from the front
00062            and new items are added to the back. Buffers are modeled 
00063            by mango.io.model.IBuffer, and a concrete implementation is 
00064            provided this class.
00065 
00066         @li buffers can be written to directly, but a Reader and/or
00067            Writer are typically used to read & write formatted data.
00068            These readers & writers are bound to a specific buffer;
00069            often the same buffer. It's also perfectly legitimate to 
00070            bind multiple writers to the same buffer; they will all
00071            behave serially as one would expect. The same applies to
00072            multiple readers on the same buffer. Readers and writers
00073            support two styles of IO: put/get, and the C++ style << 
00074            and >> operators. All such operations can be chained.
00075 
00076         @li Any class can be made compatable with the reader/writer
00077            framework by implementing the IReadable and/or IWritable 
00078            interfaces. Each of these specify just a single method.
00079 
00080         @li Buffers may also be tokenized. This is handy when one is
00081            dealing with text input, and/or the content suits a more
00082            fluid format than most typical readers & writers support.
00083            Tokens are mapped directly onto buffer content, so there
00084            is only minor overhead in using them. Tokens can be read
00085            and written by reader/writers also, using a more relaxed
00086            set of rules than those applied to integral IO.
00087 
00088         @li buffers are sometimes memory-only, in which case there
00089            is nothing left to do when a reader (or tokenizer) hits
00090            end of buffer conditions. Other buffers are themselves 
00091            bound to a Conduit. When this is the case, a reader will 
00092            eventually cause the buffer to reload via its associated 
00093            conduit. Previous buffer content will thus be lost. The
00094            same concept is applied to writers, whereby they flush 
00095            the content of a full buffer to a bound conduit before 
00096            continuing. 
00097 
00098         @li conduits provide virtualized access to external content,
00099            and represent things like files or Internet connections.
00100            They are just a different kind of stream. Conduits are
00101            modelled by mango.io.model.IConduit, and implemented via
00102            classes FileConduit and SocketConduit. Additional kinds
00103            of conduit are easy to construct: one either subclasses
00104            mango.io.Conduit, or implements mango.io.model.IConduit. 
00105            A conduit reads and writes from/to a buffer in big chunks
00106            (typically the entire buffer).
00107 
00108 
00109         An example of how to append a buffer follows:
00110 
00111         @code
00112         char[] foo = "to write some D";
00113 
00114         // create a small buffer
00115         Buffer buf = new Buffer (256);
00116 
00117         // append some text directly to it
00118         buf.append("now is the time for all good men ").append(foo);
00119 
00120         // output the combined string
00121         Stdout.put (buf.toString).cr();
00122         @endcode
00123 
00124         You might use a GrowableBuffer instead where you wish to 
00125         append beyond a preset limit. One common usage of buffers 
00126         is in conjunction with a conduit, such as FileConduit. Each 
00127         Conduit exposes a factory for creating a buffer of the most
00128         appropriate size or flavour:
00129 
00130         @code
00131         FileConduit fc = new FileConduit ("file.name");
00132         Buffer buf = fc.createBuffer ();
00133         @endcode
00134 
00135         However, this is typically hidden by higher level constructors 
00136         such as those of Reader and Writer derivitives. For example:
00137 
00138         @code
00139         FileConduit fc = new FileConduit ("file.name");
00140         Reader reader = new Reader (fc);
00141         @endcode
00142 
00143         There is indeed a buffer between the Reader and Conduit, but 
00144         explicit construction is unecessary in many cases.  See both 
00145         Reader and Writer for examples of formatted IO.
00146 
00147 *******************************************************************************/
00148 
00149 class Buffer : IBuffer
00150 {
00151         private void[]          data;
00152         private uint            limit;
00153         private uint            capacity;
00154         private uint            position;
00155         private IConduit        conduit;
00156 
00157         // some pre-constructed exceptions 
00158         package static EofException     eof;
00159         package static IOException      udf;
00160         package static IOException      ovf;
00161 
00162         /***********************************************************************
00163         
00164                 Ensure the buffer remains valid between method calls
00165                  
00166         ***********************************************************************/
00167 
00168         invariant 
00169         {
00170                assert (position <= limit);
00171                assert (limit <= capacity);
00172         }
00173 
00174         /***********************************************************************
00175         
00176                 Construct an overflow exception for later use.
00177 
00178         ***********************************************************************/
00179 
00180         static this ()
00181         {
00182                 ovf = new IOException  ("output buffer overflow");
00183                 udf = new IOException  ("input buffer underflow");
00184                 eof = new EofException ("premature end-of-file");
00185         }
00186 
00187         /***********************************************************************
00188         
00189                 Construct a Buffer with the specified number of bytes.
00190 
00191         ***********************************************************************/
00192 
00193         this (uint capacity = 0)
00194         {
00195                 this (new ubyte[capacity]);
00196         }
00197 
00198         /***********************************************************************
00199         
00200                 Prime buffer with an application-supplied array. There is 
00201                 no readable data present, and writing begins at position 0.
00202 
00203         ***********************************************************************/
00204 
00205         this (void[] data)
00206         {
00207                 this (data, 0);
00208         }
00209 
00210         /***********************************************************************
00211         
00212                 Prime buffer with an application-supplied array, and 
00213                 indicate how much readable data is already there. A
00214                 write operation will begin writing immediately after
00215                 the existing content.
00216 
00217         ***********************************************************************/
00218 
00219         this (void[] data, uint readable)
00220         {
00221                 setContent (data, readable);
00222         }
00223 
00224         /***********************************************************************
00225                 
00226                 Create an instance of an IBuffer. Use this when you
00227                 don't know anything about the concrete implementation,
00228                 and have only the IBuffer interface available
00229 
00230                 Returns a Buffer with no content.
00231 
00232         ***********************************************************************/
00233 
00234         IBuffer create ()
00235         {
00236                 return new Buffer;
00237         }
00238 
00239         /***********************************************************************
00240                 
00241                 Return the backing array
00242 
00243         ***********************************************************************/
00244 
00245         void[] getContent ()
00246         {
00247                 return data;
00248         }
00249 
00250         /***********************************************************************
00251         
00252                 Set the backing array with all content readable. Writing
00253                 to this will either flush it to an associated conduit, or
00254                 raise an Eof condition. Use IBuffer.clear() to reset the
00255                 content (make it all writable).
00256 
00257         ***********************************************************************/
00258 
00259         IBuffer setValidContent (void[] data)
00260         {
00261                 return setContent (data, data.length);
00262         }
00263 
00264         /***********************************************************************
00265         
00266                 Set the backing array with some content readable. Writing
00267                 to this will either flush it to an associated conduit, or
00268                 raise an Eof condition. Use IBuffer.clear() to reset the
00269                 content (make it all writable).
00270 
00271         ***********************************************************************/
00272 
00273         IBuffer setContent (void[] data, uint readable)
00274         {
00275                 this.data = data;
00276                 this.limit = readable;
00277                 this.capacity = data.length;   
00278 
00279                 // reset to start of input
00280                 this.position = 0;
00281 
00282                 return this;            
00283         }
00284 
00285         /***********************************************************************
00286 
00287                 Overridable method to grow the buffer size when it becomes 
00288                 full. Default is to not grow at all.
00289 
00290         ***********************************************************************/
00291 
00292         protected bool grow (uint size)
00293         {
00294                 return false;
00295         }
00296 
00297         /***********************************************************************
00298         
00299                 Bulk copy of data from 'src'. Limit is adjusted by 'size'
00300                 bytes.
00301 
00302         ***********************************************************************/
00303 
00304         protected void copy (void *src, uint size)
00305         {
00306                 data[limit..limit+size] = src[0..size];
00307                 limit += size;
00308         }
00309 
00310         /***********************************************************************
00311         
00312                 Read a chunk of data from the buffer, loading from the
00313                 conduit as necessary. The specified number of bytes is
00314                 loaded into the buffer, and marked as having been read 
00315                 when the 'eat' parameter is set true. When 'eat' is set
00316                 false, the read position is not adjusted.
00317 
00318                 Returns the corresponding buffer slice when successful, 
00319                 or null if there's not enough data available (Eof; Eob).
00320 
00321         ***********************************************************************/
00322 
00323         void[] get (uint size, bool eat = true)
00324         {   
00325                 if (size > readable)
00326                    {
00327                    if (conduit is null)
00328                        throw udf;
00329 
00330                    // make some space? This will try to leave as much content
00331                    // in the buffer as possible, such that entire records may
00332                    // be aliased directly from within.
00333                    if (size > writable)
00334                       {
00335                       if (size > capacity)
00336                           throw udf;
00337                       compress ();
00338                       }
00339 
00340                    // populate tail of buffer with new content
00341                    do {
00342                       if (conduit.read (this) == IConduit.Eof)
00343                           throw eof;
00344                       } while (size > readable);
00345                    }
00346 
00347                 uint i = position;
00348                 if (eat)
00349                     position += size;
00350                 return data [i .. i + size];               
00351         }
00352 
00353         /***********************************************************************
00354         
00355                 Append an array of data to this buffer, and flush to the
00356                 conduit as necessary. Returns a chaining reference if all 
00357                 data was written; throws an IOException indicating eof or 
00358                 eob if not.
00359 
00360                 This is often used in lieu of a Writer.
00361 
00362         ***********************************************************************/
00363 
00364         IBuffer append (void[] src)        
00365         {               
00366                 uint size = src.length;
00367 
00368                 if (size > writable)
00369                     // can we write externally?
00370                     if (conduit)
00371                        {
00372                        conduit.flush (this);
00373 
00374                        // check for pathological case
00375                        if (size > capacity)
00376                           {
00377                           void[] tmp = data;
00378 
00379                           setContent (src, size);
00380                           conduit.flush (this);
00381                           setContent (tmp, 0);
00382                           return this;
00383                           }
00384                        }
00385                     else
00386                        // try to grow buffer ...
00387                        if (! grow (size))
00388                              throw ovf;
00389 
00390                 copy (src, size);
00391                 return this;
00392         }
00393 
00394         /***********************************************************************
00395         
00396                 Return a char[] slice of the buffer up to the limit of
00397                 valid content.
00398 
00399         ***********************************************************************/
00400 
00401         override char[] toString ()
00402         {       
00403                 return cast(char[]) data[position..limit];
00404         }
00405 
00406         /***********************************************************************
00407         
00408                 Skip ahead by the specified number of bytes, streaming from 
00409                 the associated conduit as necessary.
00410         
00411                 Can also reverse the read position by 'size' bytes. This may
00412                 be used to support lookahead-type operations.
00413 
00414                 Returns true if successful, false otherwise.
00415 
00416         ***********************************************************************/
00417 
00418         bool skip (int size)
00419         {
00420                 if (size < 0)
00421                    {
00422                    size = -size;
00423                    if (position >= size)
00424                       {
00425                       position -= size;
00426                       return true;
00427                       }
00428                    return false;
00429                    }
00430                 return get (size) !== null;
00431         }
00432 
00433         /***********************************************************************
00434         
00435                 Return count of readable bytes remaining in buffer. This is 
00436                 calculated simply as limit() - position()
00437 
00438         ***********************************************************************/
00439 
00440         uint readable ()
00441         {
00442                 return limit - position;
00443         }               
00444 
00445         /***********************************************************************
00446         
00447                 Return count of writable bytes available in buffer. This is 
00448                 calculated simply as capacity() - limit()
00449 
00450         ***********************************************************************/
00451 
00452         uint writable ()
00453         {
00454                 return capacity - limit;
00455         }               
00456 
00457         /***********************************************************************
00458 
00459                 Exposes the raw data buffer at the current write position, 
00460                 The delegate is provided with a void[] representing space
00461                 available within the buffer at the current write position.
00462 
00463                 The delegate should return the appropriate number of bytes 
00464                 if it writes valid content, or IConduit.Eof on error.
00465 
00466                 Returns whatever the delegate returns.
00467 
00468         ***********************************************************************/
00469 
00470         int write (int delegate (void[]) dg)
00471         {
00472                 int count = dg (data [limit..capacity]);
00473 
00474                 if (count != IConduit.Eof) 
00475                    {
00476                    limit += count;
00477                    assert (limit <= capacity);
00478                    }
00479                 return count;
00480         }               
00481 
00482         /***********************************************************************
00483 
00484                 Exposes the raw data buffer at the current read position. The
00485                 delegate is provided with a void[] representing the available
00486                 data, and should return zero to leave the current read position
00487                 intact. 
00488                 
00489                 If the delegate consumes data, it should return the number of 
00490                 bytes consumed; or IConduit.Eof to indicate an error.
00491 
00492                 Returns whatever the delegate returns.
00493 
00494         ***********************************************************************/
00495 
00496         int read (int delegate (void[]) dg)
00497         {
00498                 
00499                 int count = dg (data [position..limit]);
00500                 
00501                 if (count != IConduit.Eof)
00502                    {
00503                    position += count;
00504                    assert (position <= limit);
00505                    }
00506                 return count;
00507         }               
00508 
00509         /***********************************************************************
00510 
00511                 If we have some data left after an export, move it to 
00512                 front-of-buffer and set position to be just after the 
00513                 remains. This is for supporting certain conduits which 
00514                 choose to write just the initial portion of a request.
00515                 
00516                 Limit is set to the amount of data remaining. Position 
00517                 is always reset to zero.
00518 
00519         ***********************************************************************/
00520 
00521         IBuffer compress ()
00522         {
00523                 uint r = readable ();
00524 
00525                 if (position > 0 && r > 0)
00526                     // content may overlap ...
00527                     memcpy (&data[0], &data[position], r);
00528 
00529                 position = 0;
00530                 limit = r;
00531                 return this;
00532         }               
00533 
00534         /***********************************************************************
00535         
00536                 flush the contents of this buffer to the related conduit.
00537                 Throws an IOException on premature eof.
00538 
00539         ***********************************************************************/
00540 
00541         void flush ()
00542         {
00543                 if (conduit)
00544                     conduit.flush (this);
00545         }               
00546 
00547         /***********************************************************************
00548         
00549                 Reset 'position' and 'limit' to zero
00550 
00551         ***********************************************************************/
00552 
00553         IBuffer clear ()
00554         {
00555                 position = limit = 0;
00556                 return this;
00557         }               
00558 
00559         /***********************************************************************
00560         
00561                 Returns the limit of readable content within this buffer
00562 
00563         ***********************************************************************/
00564 
00565         uint getLimit ()
00566         {
00567                 return limit;
00568         }
00569 
00570         /***********************************************************************
00571         
00572                 Returns the total capacity of this buffer
00573 
00574         ***********************************************************************/
00575 
00576         uint getCapacity ()
00577         {
00578                 return capacity;
00579         }
00580 
00581         /***********************************************************************
00582         
00583                 Returns the current read-position within this buffer
00584 
00585         ***********************************************************************/
00586 
00587         uint getPosition ()
00588         {
00589                 return position;
00590         }
00591 
00592         /***********************************************************************
00593         
00594                 Returns the conduit associated with this buffer. Returns 
00595                 null if the buffer is purely memory based; that is, it's
00596                 not backed by some external medium.
00597 
00598                 Buffers do not require an external conduit to operate, but 
00599                 it can be convenient to associate one. For example, methods
00600                 read and write use it to import/export content as necessary.
00601 
00602         ***********************************************************************/
00603 
00604         IConduit getConduit ()
00605         {
00606                 return conduit;
00607         }
00608 
00609         /***********************************************************************
00610         
00611                 Sets the external conduit associated with this buffer.
00612 
00613                 Buffers do not require an external conduit to operate, but 
00614                 it can be convenient to associate one. For example, methods
00615                 read and write use it to import/export content as necessary.
00616 
00617         ***********************************************************************/
00618 
00619         void setConduit (IConduit conduit)
00620         {
00621                 this.conduit = conduit;
00622         }
00623 }
00624 
00625 
00626 /*******************************************************************************
00627 
00628         Subclass to provide support for content growth. This is handy when
00629         you want to keep a buffer around as a scratchpad.
00630 
00631 *******************************************************************************/
00632 
00633 class GrowableBuffer : Buffer
00634 {
00635         /***********************************************************************
00636         
00637                 Create a GrowableBuffer with the specified initial size.
00638 
00639         ***********************************************************************/
00640 
00641         this (uint size)
00642         {
00643                 super (size);
00644         }
00645 
00646         /***********************************************************************
00647 
00648                 Overridable method to grow the buffer size when it becomes 
00649                 full. Default is to not grow at all.
00650         
00651         ***********************************************************************/
00652 
00653         protected override bool grow (uint size)
00654         {
00655                 if (data.length < uint.max / 2)
00656                    {
00657                    uint i = data.length * 2;
00658                    if (i < size)
00659                        i = size;
00660                    data.length = i;
00661                    capacity = i;
00662                    return true;
00663                    }
00664                 return false;
00665         }
00666 }
00667 
00668 
00669 
00670 /*******************************************************************************
00671 
00672         Subclass to treat the buffer as a seekable entity, where all 
00673         capacity is available for reading and/or writing. To achieve 
00674         this we must effectively disable the 'limit' watermark, and 
00675         locate write operations around 'position' instead. 
00676 
00677 *******************************************************************************/
00678 
00679 class MappedBuffer : Buffer
00680 {
00681         /***********************************************************************
00682                 
00683                 Construct an empty MappedBuffer 
00684 
00685         ***********************************************************************/
00686 
00687         this ()
00688         {
00689                 super (0);
00690         }
00691 
00692         /***********************************************************************
00693                 
00694                 Close this mapped buffer
00695 
00696         ***********************************************************************/
00697 
00698         abstract void close ();
00699 
00700         /***********************************************************************
00701                 
00702                 Flush any dirty content out to the drive
00703 
00704         ***********************************************************************/
00705 
00706         abstract void flush ();
00707 
00708         /***********************************************************************
00709                 
00710                 Set the read/write position
00711 
00712         ***********************************************************************/
00713 
00714         void setPosition (uint position)
00715         {
00716                 this.position = position;
00717         }
00718 
00719         /***********************************************************************
00720         
00721                 Seek to the specified position within the buffer, and return
00722                 the byte offset of the new location (relative to zero).
00723 
00724         ***********************************************************************/
00725 
00726         long seek (long offset, ISeekable.SeekAnchor anchor)
00727         {
00728                 long pos = capacity;
00729 
00730                 if (anchor == ISeekable.SeekAnchor.Begin)
00731                     pos = offset;
00732                 else
00733                    if (anchor == ISeekable.SeekAnchor.End)
00734                        pos -= offset;
00735                    else
00736                       pos = position + offset;
00737 
00738                 return position = pos;
00739         }
00740 
00741         /***********************************************************************
00742         
00743                 Return count of writable bytes available in buffer. This is 
00744                 calculated simply as capacity() - limit()
00745 
00746         ***********************************************************************/
00747 
00748         override uint writable ()
00749         {
00750                 return capacity - position;
00751         }               
00752 
00753         /***********************************************************************
00754         
00755                 Bulk copy of data from 'src'. Position is adjusted by 'size'
00756                 bytes.
00757 
00758         ***********************************************************************/
00759 
00760         override protected void copy (void *src, uint size)
00761         {
00762                 data[position..position+size] = src[0..size];
00763                 position += size;
00764         }
00765 
00766         /***********************************************************************
00767 
00768                 Exposes the raw data buffer at the current write position, 
00769                 The delegate is provided with a void[] representing space
00770                 available within the buffer at the current write position.
00771 
00772                 The delegate should return the appropriate number of bytes 
00773                 if it writes valid content, or IConduit.Eof on error.
00774 
00775                 Returns whatever the delegate returns.
00776 
00777         ***********************************************************************/
00778 
00779         override int write (int delegate (void[]) dg)
00780         {
00781                 int count = dg (data [position..capacity]);
00782 
00783                 if (count != IConduit.Eof) 
00784                    {
00785                    position += count;
00786                    assert (position <= capacity);
00787                    }
00788                 return count;
00789         }               
00790 
00791         /***********************************************************************
00792 
00793                 Prohibit compress() from doing anything at all.
00794 
00795         ***********************************************************************/
00796 
00797         override IBuffer compress ()
00798         {
00799                 return this;
00800         }               
00801 
00802         /***********************************************************************
00803 
00804                 Prohibit clear() from doing anything at all.
00805 
00806         ***********************************************************************/
00807 
00808         override IBuffer clear ()
00809         {
00810                 return this;
00811         }               
00812 
00813         /***********************************************************************
00814         
00815                 Prohibit the setting of another IConduit
00816 
00817         ***********************************************************************/
00818 
00819         override void setConduit (IConduit conduit)
00820         {
00821                 assert (0);
00822         }
00823 }

Generated on Sun Mar 6 00:30:55 2005 for Mango by doxygen 1.3.6