Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

Buffer.d

Go to the documentation of this file.
00001 /*******************************************************************************
00002 
00003         @file Buffer.d
00004 
00005         Copyright (c) 2004 Kris Bell
00006         
00007         This software is provided 'as-is', without any express or implied
00008         warranty. In no event will the authors be held liable for damages
00009         of any kind arising from the use of this software.
00010         
00011         Permission is hereby granted to anyone to use this software for any 
00012         purpose, including commercial applications, and to alter it and/or 
00013         redistribute it freely, subject to the following restrictions:
00014         
00015         1. The origin of this software must not be misrepresented; you must 
00016            not claim that you wrote the original software. If you use this 
00017            software in a product, an acknowledgment within documentation of 
00018            said product would be appreciated but is not required.
00019 
00020         2. Altered source versions must be plainly marked as such, and must 
00021            not be misrepresented as being the original software.
00022 
00023         3. This notice may not be removed or altered from any distribution
00024            of the source.
00025 
00026         4. Derivative works are permitted, but they must carry this notice
00027            in full and credit the original source.
00028 
00029 
00030                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
00031 
00032       
00033         @version        Initial version; March 2004
00034 
00035         @author         Kris
00036 
00037 
00038 *******************************************************************************/
00039 
00040 module mango.io.Buffer;
00041 
00042 private import  mango.io.Exception;
00043 
00044 private import  mango.io.model.IConduit;
00045 
00046 public  import  mango.io.model.IBuffer;
00047 
00048 /******************************************************************************
00049 
00050 ******************************************************************************/
00051 
00052 extern (C)
00053 {
00054         void * memcpy (void *dst, void *src, uint);
00055 }       
00056 
00057 /*******************************************************************************
00058 
00059         The basic premise behind this IO package is as follows:
00060 
00061         @li the central concept is that of a buffer. The buffer acts
00062            as a queue (line) where items are removed from the front
00063            and new items are added to the back. Buffers are modeled 
00064            by mango.io.model.IBuffer, and a concrete implementation is 
00065            provided this class.
00066 
00067         @li buffers can be written to directly, but a Reader and/or
00068            Writer are typically used to read & write formatted data.
00069            These readers & writers are bound to a specific buffer;
00070            often the same buffer. It's also perfectly legitimate to 
00071            bind multiple writers to the same buffer; they will all
00072            behave serially as one would expect. The same applies to
00073            multiple readers on the same buffer. Readers and writers
00074            support three styles of IO: put/get, the C++ style << &
00075            >> operators, and the () whisper style. All operations 
00076            can be chained.
00077 
00078         @li Any class can be made compatable with the reader/writer
00079            framework by implementing the IReadable and/or IWritable 
00080            interfaces. Each of these specify just a single method.
00081 
00082         @li Buffers may also be tokenized. This is handy when one is
00083            dealing with text input, and/or the content suits a more
00084            fluid format than most typical readers & writers support.
00085            Tokens are mapped directly onto buffer content, so there
00086            is only minor overhead in using them. Tokens can be read
00087            and written by reader/writers also, using a more relaxed
00088            set of rules than those applied to integral IO.
00089 
00090         @li buffers are sometimes memory-only, in which case there
00091            is nothing left to do when a reader (or tokenizer) hits
00092            end of buffer conditions. Other buffers are themselves 
00093            bound to a Conduit. When this is the case, a reader will 
00094            eventually cause the buffer to reload via its associated 
00095            conduit. Previous buffer content will thus be lost. The
00096            same concept is applied to writers, whereby they flush 
00097            the content of a full buffer to a bound conduit before 
00098            continuing. 
00099 
00100         @li conduits provide virtualized access to external content,
00101            and represent things like files or Internet connections.
00102            They are just a different kind of stream. Conduits are
00103            modelled by mango.io.model.IConduit, and implemented via
00104            classes FileConduit and SocketConduit. Additional kinds
00105            of conduit are easy to construct: one either subclasses
00106            mango.io.Conduit, or implements mango.io.model.IConduit. 
00107            A conduit reads and writes from/to a buffer in big chunks
00108            (typically the entire buffer).
00109 
00110 
00111         An example of how to append a buffer follows:
00112 
00113         @code
00114         char[] foo = "to write some D";
00115 
00116         // create a small buffer
00117         Buffer buf = new Buffer (256);
00118 
00119         // append some text directly to it
00120         buf.append("now is the time for all good men ").append(foo);
00121 
00122         // output the combined string
00123         Stdout (buf.toString) (CR);
00124         @endcode
00125 
00126         You might use a GrowableBuffer instead where you wish to 
00127         append beyond a preset limit. One common usage of buffers 
00128         is in conjunction with a conduit, such as FileConduit. Each 
00129         Conduit exposes a factory for creating a buffer of the most
00130         appropriate size or flavour:
00131 
00132         @code
00133         FileConduit fc = new FileConduit ("file.name");
00134         Buffer buf = fc.createBuffer ();
00135         @endcode
00136 
00137         However, this is typically hidden by higher level constructors 
00138         such as those of Reader and Writer derivitives. For example:
00139 
00140         @code
00141         FileConduit fc = new FileConduit ("file.name");
00142         Reader reader = new Reader (fc);
00143         @endcode
00144 
00145         There is indeed a buffer between the Reader and Conduit, but 
00146         explicit construction is unecessary in many cases.  See both 
00147         Reader and Writer for examples of formatted IO.
00148 
00149 *******************************************************************************/
00150 
00151 class Buffer : IBuffer
00152 {
00153         private void[]          data;
00154         private uint            limit;
00155         private uint            capacity;
00156         private uint            position;
00157         private IConduit        conduit;
00158 
00159         // error messages
00160         protected static char[] overflow  = "output buffer overflow";
00161         protected static char[] underflow = "input buffer underflow";
00162         protected static char[] eofRead   = "end-of-file whilst reading";
00163         protected static char[] eofWrite  = "end-of-file whilst writing";
00164 
00165 
00166         /***********************************************************************
00167         
00168                 Ensure the buffer remains valid between method calls
00169                  
00170         ***********************************************************************/
00171 
00172         invariant 
00173         {
00174                assert (position <= limit);
00175                assert (limit <= capacity);
00176         }
00177 
00178         /***********************************************************************
00179         
00180                 Construct a Buffer with the specified number of bytes.
00181 
00182         ***********************************************************************/
00183 
00184         this (uint capacity = 0)
00185         {
00186                 this (new ubyte[capacity]);
00187         }
00188 
00189         /***********************************************************************
00190         
00191                 Prime buffer with an application-supplied array. There is 
00192                 no readable data present, and writing begins at position 0.
00193 
00194         ***********************************************************************/
00195 
00196         this (void[] data)
00197         {
00198                 this (data, 0);
00199         }
00200 
00201         /***********************************************************************
00202         
00203                 Prime buffer with an application-supplied array, and 
00204                 indicate how much readable data is already there. A
00205                 write operation will begin writing immediately after
00206                 the existing content.
00207 
00208         ***********************************************************************/
00209 
00210         this (void[] data, uint readable)
00211         {
00212                 setContent (data, readable);
00213         }
00214 
00215         /***********************************************************************
00216         
00217                 Throw an exception with the provided message
00218 
00219         ***********************************************************************/
00220 
00221         protected void error (char[] msg)
00222         {
00223                 throw new IOException (msg);
00224         }
00225 
00226         /***********************************************************************
00227                 
00228                 Create an instance of an IBuffer. Use this when you
00229                 don't know anything about the concrete implementation,
00230                 and have only the IBuffer interface available
00231 
00232                 Returns a Buffer with no content.
00233 
00234         ***********************************************************************/
00235 
00236         IBuffer create ()
00237         {
00238                 return new Buffer;
00239         }
00240 
00241         /***********************************************************************
00242                 
00243                 Return the backing array
00244 
00245         ***********************************************************************/
00246 
00247         void[] getContent ()
00248         {
00249                 return data;
00250         }
00251 
00252         /***********************************************************************
00253         
00254                 Set the backing array with all content readable. Writing
00255                 to this will either flush it to an associated conduit, or
00256                 raise an Eof condition. Use IBuffer.clear() to reset the
00257                 content (make it all writable).
00258 
00259         ***********************************************************************/
00260 
00261         IBuffer setValidContent (void[] data)
00262         {
00263                 return setContent (data, data.length);
00264         }
00265 
00266         /***********************************************************************
00267         
00268                 Set the backing array with some content readable. Writing
00269                 to this will either flush it to an associated conduit, or
00270                 raise an Eof condition. Use IBuffer.clear() to reset the
00271                 content (make it all writable).
00272 
00273         ***********************************************************************/
00274 
00275         IBuffer setContent (void[] data, uint readable)
00276         {
00277                 this.data = data;
00278                 this.limit = readable;
00279                 this.capacity = data.length;   
00280 
00281                 // reset to start of input
00282                 this.position = 0;
00283 
00284                 return this;            
00285         }
00286 
00287         /***********************************************************************
00288 
00289                 Overridable method to grow the buffer size when it becomes 
00290                 full. Default is to not grow at all.
00291 
00292         ***********************************************************************/
00293 
00294         protected bool grow (uint size)
00295         {
00296                 return false;
00297         }
00298 
00299         /***********************************************************************
00300         
00301                 Bulk copy of data from 'src'. Limit is adjusted by 'size'
00302                 bytes.
00303 
00304         ***********************************************************************/
00305 
00306         protected void copy (void *src, uint size)
00307         {
00308                 data[limit..limit+size] = src[0..size];
00309                 limit += size;
00310         }
00311 
00312         /***********************************************************************
00313         
00314                 Read a chunk of data from the buffer, loading from the
00315                 conduit as necessary. The specified number of bytes is
00316                 loaded into the buffer, and marked as having been read 
00317                 when the 'eat' parameter is set true. When 'eat' is set
00318                 false, the read position is not adjusted.
00319 
00320                 Returns the corresponding buffer slice when successful, 
00321                 or null if there's not enough data available (Eof; Eob).
00322 
00323         ***********************************************************************/
00324 
00325         void[] get (uint size, bool eat = true)
00326         {   
00327                 if (size > readable)
00328                    {
00329                    if (conduit is null)
00330                        error (underflow);
00331 
00332                    // make some space? This will try to leave as much content
00333                    // in the buffer as possible, such that entire records may
00334                    // be aliased directly from within.
00335                    if (size > writable)
00336                       {
00337                       if (size > capacity)
00338                           error (underflow);
00339                       compress ();
00340                       }
00341 
00342                    // populate tail of buffer with new content
00343                    do {
00344                       if (conduit.read (this) == IConduit.Eof)
00345                           error (eofRead);
00346                       } while (size > readable);
00347                    }
00348 
00349                 uint i = position;
00350                 if (eat)
00351                     position += size;
00352                 return data [i .. i + size];               
00353         }
00354 
00355         /***********************************************************************
00356         
00357                 Append an array of data to this buffer, and flush to the
00358                 conduit as necessary. Returns a chaining reference if all 
00359                 data was written; throws an IOException indicating eof or 
00360                 eob if not.
00361 
00362                 This is often used in lieu of a Writer.
00363 
00364         ***********************************************************************/
00365 
00366         IBuffer append (void[] src)        
00367         {               
00368                 uint size = src.length;
00369 
00370                 if (size > writable)
00371                     // can we write externally?
00372                     if (conduit)
00373                        {
00374                        flush ();
00375 
00376                        // check for pathological case
00377                        if (size > capacity)
00378                           {
00379                           // we could write directly to the conduit, but
00380                           // then we'd lose support for partial writes ...
00381                           void[] tmp = data;
00382 
00383                           setContent (src, size);
00384                           flush ();
00385                           setContent (tmp, 0);
00386                           return this;
00387                           }
00388                        }
00389                     else
00390                        // try to grow buffer ...
00391                        if (! grow (size))
00392                              error (overflow);
00393 
00394                 copy (src, size);
00395                 return this;
00396         }
00397 
00398         /***********************************************************************
00399         
00400                 Return a char[] slice of the buffer up to the limit of
00401                 valid content.
00402 
00403         ***********************************************************************/
00404 
00405         override char[] toString ()
00406         {       
00407                 return cast(char[]) data[position..limit];
00408         }
00409 
00410         /***********************************************************************
00411         
00412                 Skip ahead by the specified number of bytes, streaming from 
00413                 the associated conduit as necessary.
00414         
00415                 Can also reverse the read position by 'size' bytes. This may
00416                 be used to support lookahead-type operations.
00417 
00418                 Returns true if successful, false otherwise.
00419 
00420         ***********************************************************************/
00421 
00422         bool skip (int size)
00423         {
00424                 if (size < 0)
00425                    {
00426                    size = -size;
00427                    if (position >= size)
00428                       {
00429                       position -= size;
00430                       return true;
00431                       }
00432                    return false;
00433                    }
00434                 return get (size) !== null;
00435         }
00436 
00437         /***********************************************************************
00438         
00439                 Return count of readable bytes remaining in buffer. This is 
00440                 calculated simply as limit() - position()
00441 
00442         ***********************************************************************/
00443 
00444         uint readable ()
00445         {
00446                 return limit - position;
00447         }               
00448 
00449         /***********************************************************************
00450         
00451                 Return count of writable bytes available in buffer. This is 
00452                 calculated simply as capacity() - limit()
00453 
00454         ***********************************************************************/
00455 
00456         uint writable ()
00457         {
00458                 return capacity - limit;
00459         }               
00460 
00461         /***********************************************************************
00462 
00463                 Exposes the raw data buffer at the current write position, 
00464                 The delegate is provided with a void[] representing space
00465                 available within the buffer at the current write position.
00466 
00467                 The delegate should return the appropriate number of bytes 
00468                 if it writes valid content, or IConduit.Eof on error.
00469 
00470                 Returns whatever the delegate returns.
00471 
00472         ***********************************************************************/
00473 
00474         int write (int delegate (void[]) dg)
00475         {
00476                 int count = dg (data [limit..capacity]);
00477 
00478                 if (count != IConduit.Eof) 
00479                    {
00480                    limit += count;
00481                    assert (limit <= capacity);
00482                    }
00483                 return count;
00484         }               
00485 
00486         /***********************************************************************
00487 
00488                 Exposes the raw data buffer at the current read position. The
00489                 delegate is provided with a void[] representing the available
00490                 data, and should return zero to leave the current read position
00491                 intact. 
00492                 
00493                 If the delegate consumes data, it should return the number of 
00494                 bytes consumed; or IConduit.Eof to indicate an error.
00495 
00496                 Returns whatever the delegate returns.
00497 
00498         ***********************************************************************/
00499 
00500         int read (int delegate (void[]) dg)
00501         {
00502                 
00503                 int count = dg (data [position..limit]);
00504                 
00505                 if (count != IConduit.Eof)
00506                    {
00507                    position += count;
00508                    assert (position <= limit);
00509                    }
00510                 return count;
00511         }               
00512 
00513         /***********************************************************************
00514 
00515                 If we have some data left after an export, move it to 
00516                 front-of-buffer and set position to be just after the 
00517                 remains. This is for supporting certain conduits which 
00518                 choose to write just the initial portion of a request.
00519                 
00520                 Limit is set to the amount of data remaining. Position 
00521                 is always reset to zero.
00522 
00523         ***********************************************************************/
00524 
00525         IBuffer compress ()
00526         {
00527                 uint r = readable ();
00528 
00529                 if (position > 0 && r > 0)
00530                     // content may overlap ...
00531                     memcpy (&data[0], &data[position], r);
00532 
00533                 position = 0;
00534                 limit = r;
00535                 return this;
00536         }               
00537 
00538         /***********************************************************************
00539         
00540                 flush the contents of this buffer to the related conduit.
00541                 Throws an IOException on premature eof.
00542 
00543         ***********************************************************************/
00544 
00545         void flush ()
00546         {
00547                 if (conduit)
00548                     while (readable)
00549                            if (conduit.write (this) == conduit.Eof)
00550                                error (eofWrite);
00551         }               
00552 
00553         /***********************************************************************
00554         
00555                 Reset 'position' and 'limit' to zero
00556 
00557         ***********************************************************************/
00558 
00559         IBuffer clear ()
00560         {
00561                 position = limit = 0;
00562                 return this;
00563         }               
00564 
00565         /***********************************************************************
00566         
00567                 Returns the limit of readable content within this buffer
00568 
00569         ***********************************************************************/
00570 
00571         uint getLimit ()
00572         {
00573                 return limit;
00574         }
00575 
00576         /***********************************************************************
00577         
00578                 Returns the total capacity of this buffer
00579 
00580         ***********************************************************************/
00581 
00582         uint getCapacity ()
00583         {
00584                 return capacity;
00585         }
00586 
00587         /***********************************************************************
00588         
00589                 Returns the current read-position within this buffer
00590 
00591         ***********************************************************************/
00592 
00593         uint getPosition ()
00594         {
00595                 return position;
00596         }
00597 
00598         /***********************************************************************
00599         
00600                 Returns the conduit associated with this buffer. Returns 
00601                 null if the buffer is purely memory based; that is, it's
00602                 not backed by some external medium.
00603 
00604                 Buffers do not require an external conduit to operate, but 
00605                 it can be convenient to associate one. For example, methods
00606                 read and write use it to import/export content as necessary.
00607 
00608         ***********************************************************************/
00609 
00610         IConduit getConduit ()
00611         {
00612                 return conduit;
00613         }
00614 
00615         /***********************************************************************
00616         
00617                 Sets the external conduit associated with this buffer.
00618 
00619                 Buffers do not require an external conduit to operate, but 
00620                 it can be convenient to associate one. For example, methods
00621                 read and write use it to import/export content as necessary.
00622 
00623         ***********************************************************************/
00624 
00625         void setConduit (IConduit conduit)
00626         {
00627                 this.conduit = conduit;
00628         }
00629 }
00630 
00631 
00632 /*******************************************************************************
00633 
00634         Subclass to provide support for content growth. This is handy when
00635         you want to keep a buffer around as a scratchpad.
00636 
00637 *******************************************************************************/
00638 
00639 class GrowableBuffer : Buffer
00640 {
00641         /***********************************************************************
00642         
00643                 Create a GrowableBuffer with the specified initial size.
00644 
00645         ***********************************************************************/
00646 
00647         this (uint size)
00648         {
00649                 super (size);
00650         }
00651 
00652         /***********************************************************************
00653 
00654                 Overridable method to grow the buffer size when it becomes 
00655                 full. Default is to not grow at all.
00656         
00657         ***********************************************************************/
00658 
00659         protected override bool grow (uint size)
00660         {
00661                 if (data.length < uint.max / 2)
00662                    {
00663                    uint i = data.length * 2;
00664                    if (i < size)
00665                        i = size;
00666                    data.length = i;
00667                    capacity = i;
00668                    return true;
00669                    }
00670                 return false;
00671         }
00672 }
00673 
00674 
00675 
00676 /*******************************************************************************
00677 
00678         Subclass to treat the buffer as a seekable entity, where all 
00679         capacity is available for reading and/or writing. To achieve 
00680         this we must effectively disable the 'limit' watermark, and 
00681         locate write operations around 'position' instead. 
00682 
00683 *******************************************************************************/
00684 
00685 class MappedBuffer : Buffer
00686 {
00687         /***********************************************************************
00688                 
00689                 Construct an empty MappedBuffer 
00690 
00691         ***********************************************************************/
00692 
00693         this ()
00694         {
00695                 super (0);
00696         }
00697 
00698         /***********************************************************************
00699                 
00700                 Close this mapped buffer
00701 
00702         ***********************************************************************/
00703 
00704         abstract void close ();
00705 
00706         /***********************************************************************
00707                 
00708                 Flush any dirty content out to the drive
00709 
00710         ***********************************************************************/
00711 
00712         abstract void flush ();
00713 
00714         /***********************************************************************
00715                 
00716                 Set the read/write position
00717 
00718         ***********************************************************************/
00719 
00720         void setPosition (uint position)
00721         {
00722                 this.position = position;
00723         }
00724 
00725         /***********************************************************************
00726         
00727                 Seek to the specified position within the buffer, and return
00728                 the byte offset of the new location (relative to zero).
00729 
00730         ***********************************************************************/
00731 
00732         long seek (long offset, ISeekable.SeekAnchor anchor)
00733         {
00734                 long pos = capacity;
00735 
00736                 if (anchor == ISeekable.SeekAnchor.Begin)
00737                     pos = offset;
00738                 else
00739                    if (anchor == ISeekable.SeekAnchor.End)
00740                        pos -= offset;
00741                    else
00742                       pos = position + offset;
00743 
00744                 return position = pos;
00745         }
00746 
00747         /***********************************************************************
00748         
00749                 Return count of writable bytes available in buffer. This is 
00750                 calculated simply as capacity() - limit()
00751 
00752         ***********************************************************************/
00753 
00754         override uint writable ()
00755         {
00756                 return capacity - position;
00757         }               
00758 
00759         /***********************************************************************
00760         
00761                 Bulk copy of data from 'src'. Position is adjusted by 'size'
00762                 bytes.
00763 
00764         ***********************************************************************/
00765 
00766         override protected void copy (void *src, uint size)
00767         {
00768                 data[position..position+size] = src[0..size];
00769                 position += size;
00770         }
00771 
00772         /***********************************************************************
00773 
00774                 Exposes the raw data buffer at the current write position, 
00775                 The delegate is provided with a void[] representing space
00776                 available within the buffer at the current write position.
00777 
00778                 The delegate should return the appropriate number of bytes 
00779                 if it writes valid content, or IConduit.Eof on error.
00780 
00781                 Returns whatever the delegate returns.
00782 
00783         ***********************************************************************/
00784 
00785         override int write (int delegate (void[]) dg)
00786         {
00787                 int count = dg (data [position..capacity]);
00788 
00789                 if (count != IConduit.Eof) 
00790                    {
00791                    position += count;
00792                    assert (position <= capacity);
00793                    }
00794                 return count;
00795         }               
00796 
00797         /***********************************************************************
00798 
00799                 Prohibit compress() from doing anything at all.
00800 
00801         ***********************************************************************/
00802 
00803         override IBuffer compress ()
00804         {
00805                 return this;
00806         }               
00807 
00808         /***********************************************************************
00809 
00810                 Prohibit clear() from doing anything at all.
00811 
00812         ***********************************************************************/
00813 
00814         override IBuffer clear ()
00815         {
00816                 return this;
00817         }               
00818 
00819         /***********************************************************************
00820         
00821                 Prohibit the setting of another IConduit
00822 
00823         ***********************************************************************/
00824 
00825         override void setConduit (IConduit conduit)
00826         {
00827                 assert (0);
00828         }
00829 }

Generated on Fri May 27 18:11:55 2005 for Mango by  doxygen 1.4.0