Main Page | Class Hierarchy | Alphabetical List | Class List | File List | Class Members | File Members | Related Pages

Buffer.d

Go to the documentation of this file.
00001 /*******************************************************************************
00002 
00003         @file Buffer.d
00004 
00005         Copyright (C) 2004 Kris Bell
00006         
00007         This software is provided 'as-is', without any express or implied
00008         warranty. In no event will the authors be held liable for damages
00009         of any kind arising from the use of this software.
00010         
00011         Permission is hereby granted to anyone to use this software for any 
00012         purpose, including commercial applications, and to alter it and/or 
00013         redistribute it freely, subject to the following restrictions:
00014         
00015         1. The origin of this software must not be misrepresented; you must 
00016            not claim that you wrote the original software. If you use this 
00017            software in a product, an acknowledgment within documentation of 
00018            said product would be appreciated but is not required.
00019 
00020         2. Altered source versions must be plainly marked as such, and must 
00021            not be misrepresented as being the original software.
00022 
00023         3. This notice may not be removed or altered from any distribution
00024            of the source.
00025 
00026 
00027                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
00028 
00029       
00030         @version        Initial version, March 2004
00031         @author         Kris
00032 
00033 
00034 *******************************************************************************/
00035 
00036 module mango.io.Buffer;
00037 
00038 private import  mango.io.Exception;
00039 
00040 private import  mango.io.model.IConduit;
00041 
00042 public  import  mango.io.model.IBuffer;
00043 
00044 /******************************************************************************
00045 
00046 ******************************************************************************/
00047 
00048 extern (C)
00049 {
00050         void * memcpy (void *dst, void *src, uint);
00051 }       
00052 
00053 /*******************************************************************************
00054 
00055         The basic premise behind this IO package is as follows:
00056 
00057         @li the central concept is that of a buffer. The buffer acts
00058            as a queue (line) where items are removed from the front
00059            and new items are added to the back. Buffers are modeled 
00060            by mango.io.model.IBuffer, and a concrete implementation is 
00061            provided this class.
00062 
00063         @li buffers can be written to directly, but a Reader and/or
00064            Writer are typically used to read & write formatted data.
00065            These readers & writers are bound to a specific buffer;
00066            often the same buffer. It's also perfectly legitimate to 
00067            bind multiple writers to the same buffer; they will all
00068            behave serially as one would expect. The same applies to
00069            multiple readers on the same buffer. Readers and writers
00070            support two styles of IO: put/get, and the C++ style << 
00071            and >> operators. All such operations can be chained.
00072 
00073         @li Any class can be made compatable with the reader/writer
00074            framework by implementing the IReadable and/or IWritable 
00075            interfaces. Each of these specify just a single method.
00076 
00077         @li Buffers may also be tokenized. This is handy when one is
00078            dealing with text input, and/or the content suits a more
00079            fluid format than most typical readers & writers support.
00080            Tokens are mapped directly onto buffer content, so there
00081            is only minor overhead in using them. Tokens can be read
00082            and written by reader/writers also, using a more relaxed
00083            set of rules than those applied to integral IO.
00084 
00085         @li buffers are sometimes memory-only, in which case there
00086            is nothing left to do when a reader (or tokenizer) hits
00087            end of buffer conditions. Other buffers are themselves 
00088            bound to a Conduit. When this is the case, a reader will 
00089            eventually cause the buffer to reload via its associated 
00090            conduit. Previous buffer content will thus be lost. The
00091            same concept is applied to writers, whereby they flush 
00092            the content of a full buffer to a bound conduit before 
00093            continuing. 
00094 
00095         @li conduits provide virtualized access to external content,
00096            and represent things like files or Internet connections.
00097            They are just a different kind of stream. Conduits are
00098            modelled by mango.io.model.IConduit, and implemented via
00099            classes FileConduit and SocketConduit. Additional kinds
00100            of conduit are easy to construct: one either subclasses
00101            mango.io.Conduit, or implements mango.io.model.IConduit. 
00102            A conduit reads and writes from/to a buffer in big chunks
00103            (typically the entire buffer).
00104 
00105 
00106         An example of how to append a buffer follows:
00107 
00108         @code
00109         char[] foo = "to write some D";
00110 
00111         // create a small buffer
00112         Buffer buf = new Buffer (256);
00113 
00114         // append some text directly to it
00115         buf.append("now is the time for all good men ").append(foo);
00116 
00117         // output the combined string
00118         Stdout.put (buf.toString).cr();
00119         @endcode
00120 
00121         You might use a GrowableBuffer instead where you wish to 
00122         append beyond a preset limit. One common usage of buffers 
00123         is in conjunction with a conduit, such as FileConduit. Each 
00124         Conduit exposes a factory for creating a buffer of the most
00125         appropriate size or flavour:
00126 
00127         @code
00128         FileConduit fc = new FileConduit ("file.name");
00129         Buffer buf = fc.createBuffer ();
00130         @endcode
00131 
00132         However, this is typically hidden by higher level constructors 
00133         such as those of Reader and Writer derivitives. For example:
00134 
00135         @code
00136         FileConduit fc = new FileConduit ("file.name");
00137         Reader reader = new Reader (fc);
00138         @endcode
00139 
00140         There is indeed a buffer between the Reader and Conduit, but 
00141         explicit construction is unecessary in many cases.  See both 
00142         Reader and Writer for examples of formatted IO.
00143 
00144 *******************************************************************************/
00145 
00146 class Buffer : IBuffer
00147 {
00148         static IOException      ovf;
00149 
00150         private void[]          data;
00151         private uint            limit;
00152         private uint            capacity;
00153         private uint            position;
00154         private IConduit        conduit;
00155 
00156         /***********************************************************************
00157         
00158                 Ensure the buffer remains valid between method calls
00159                  
00160         ***********************************************************************/
00161 
00162         invariant 
00163         {
00164                assert (position <= limit);
00165                assert (limit <= capacity);
00166         }
00167 
00168         /***********************************************************************
00169         
00170                 Construct an overflow exception for later use.
00171 
00172         ***********************************************************************/
00173 
00174         static this ()
00175         {
00176                 ovf = new IOException ("IO buffer overflow");
00177         }
00178 
00179         /***********************************************************************
00180         
00181                 Construct a Buffer with the specified number of bytes.
00182 
00183         ***********************************************************************/
00184 
00185         this (uint capacity = 0)
00186         {
00187                 this (new ubyte[capacity]);
00188         }
00189 
00190         /***********************************************************************
00191         
00192                 Prime buffer with an application-supplied array. There is 
00193                 no readable data present, and writing begins at position 0.
00194 
00195         ***********************************************************************/
00196 
00197         this (void[] data)
00198         {
00199                 this (data, 0);
00200         }
00201 
00202         /***********************************************************************
00203         
00204                 Prime buffer with an application-supplied array, and 
00205                 indicate how much readable data is already there. A
00206                 write operation will begin writing immediately after
00207                 the existing content.
00208 
00209         ***********************************************************************/
00210 
00211         this (void[] data, uint readable)
00212         {
00213                 setContent (data, readable);
00214         }
00215 
00216         /***********************************************************************
00217                 
00218                 Create an instance of an IBuffer. Use this when you
00219                 don't know anything about the concrete implementation,
00220                 and have only the IBuffer interface available
00221 
00222                 Returns a Buffer with no content.
00223 
00224         ***********************************************************************/
00225 
00226         IBuffer create ()
00227         {
00228                 return new Buffer;
00229         }
00230 
00231         /***********************************************************************
00232                 
00233                 Return the backing array
00234 
00235         ***********************************************************************/
00236 
00237         void[] getContent ()
00238         {
00239                 return data;
00240         }
00241 
00242         /***********************************************************************
00243         
00244                 Set the backing array with all content readable. Writing
00245                 to this will either flush it to an associated conduit, or
00246                 raise an Eof condition. Use IBuffer.clear() to reset the
00247                 content (make it all writable).
00248 
00249         ***********************************************************************/
00250 
00251         IBuffer setValidContent (void[] data)
00252         {
00253                 return setContent (data, data.length);
00254         }
00255 
00256         /***********************************************************************
00257         
00258                 Set the backing array with some content readable. Writing
00259                 to this will either flush it to an associated conduit, or
00260                 raise an Eof condition. Use IBuffer.clear() to reset the
00261                 content (make it all writable).
00262 
00263         ***********************************************************************/
00264 
00265         IBuffer setContent (void[] data, uint readable)
00266         {
00267                 this.data = data;
00268                 this.limit = readable;
00269                 this.capacity = data.length;   
00270 
00271                 // reset to start of input
00272                 this.position = 0;
00273 
00274                 return this;            
00275         }
00276 
00277         /***********************************************************************
00278 
00279                 Overridable method to grow the buffer size when it becomes 
00280                 full. Default is to not grow at all.
00281 
00282         ***********************************************************************/
00283 
00284         protected bool grow (uint size)
00285         {
00286                 return false;
00287         }
00288 
00289         /***********************************************************************
00290         
00291                 Bulk copy of data from 'src'. Limit is adjusted by 'size'
00292                 bytes.
00293 
00294         ***********************************************************************/
00295 
00296         protected void* copy (void *src, uint size)
00297         {
00298                 void* ret = &data[limit];
00299                 data[limit..limit+size] = src[0..size];
00300                 limit += size;
00301                 return ret;
00302         }
00303 
00304         /***********************************************************************
00305         
00306                 Write a chunk of data into this buffer, and flush to the
00307                 conduit as necessary. Returns true if all data was written, 
00308                 false otherwise (indicating eof, or eob).
00309 
00310                 This is typically used for streaming formatted data. Note 
00311                 that this should ideally have package visibility only, as
00312                 it should be restricted to Writers etc.
00313 
00314         ***********************************************************************/
00315 /+
00316         bool put (void *src, uint size)        
00317         {       
00318                 if (writable() < size)
00319                     // try to grow buffer ...
00320                     if (grow (size))
00321                         return put (src, size);
00322                     else
00323                        // can we write externally?
00324                        if (conduit)
00325                           {
00326                           conduit.flush (this);
00327 
00328                           // check for pathological case
00329                           if (size > capacity)
00330                              {
00331                              void[] tmp = data;
00332                              setContent (src[0..size], size);
00333 
00334                              if (conduit.flush (this))
00335                                  return setContent (tmp, 0), true;
00336                              return false;
00337                              }
00338                           }
00339                        else
00340                           return false;
00341 
00342                 return copy (src, size);
00343         }
00344 +/
00345 
00346         /***********************************************************************
00347         
00348                 Write a chunk of data into this buffer, and flush to the
00349                 conduit as necessary. Returns true if all data was written, 
00350                 false otherwise (indicating eof, or eob).
00351 
00352                 This is typically used for streaming formatted data. Note 
00353                 that this should ideally have package visibility only, as
00354                 it should be restricted to Writers etc.
00355 
00356         ***********************************************************************/
00357 
00358         void* put (void *src, uint size)
00359         {     
00360                 return copy (src, size);
00361         }
00362           
00363         /***********************************************************************
00364         
00365                 Read a chunk of data from the buffer, loading from the
00366                 conduit as necessary. The specified number of bytes is
00367                 loaded into the buffer, and marked as having been read 
00368                 when the 'eat' parameter is set true. When 'eat' is set
00369                 false, the read position is not adjusted.
00370 
00371                 Returns the corresponding buffer slice when successful, 
00372                 or null if there's not enough data available (Eof; Eob).
00373 
00374         ***********************************************************************/
00375 
00376         void[] get (uint size, bool eat = true)
00377         {   
00378                 if (readable() < size)
00379                    {
00380                    if (conduit is null)
00381                        return null;
00382 
00383                    // move readable content to buffer start
00384                    compress ();
00385 
00386                    // populate tail of buffer with new content
00387                    do {
00388                       if (conduit.read (this) == IConduit.Eof)
00389                           return null;
00390                       } while (readable() < size);
00391                    }
00392 
00393                 void[] ret = data [position..position+size];               
00394                 if (eat)
00395                     position += size;
00396 
00397                 return ret;
00398         }
00399 
00400         /***********************************************************************
00401         
00402                 Write an array of data into this buffer, and flush to the
00403                 conduit as necessary. Returns a chaining reference if all 
00404                 data was written; throws an IOException indicating eof or 
00405                 eob if not.
00406 
00407                 This is often used in lieu of a Writer.
00408 
00409         ***********************************************************************/
00410 
00411         IBuffer append (void[] content)
00412         {       
00413                 if (writable() < content.length)
00414                     throw ovf;
00415 
00416                 put (content, content.length);
00417                 return this;
00418         }
00419 
00420         /***********************************************************************
00421         
00422                 Return a char[] slice of the buffer up to the limit of
00423                 valid content.
00424 
00425         ***********************************************************************/
00426 
00427         override char[] toString ()
00428         {       
00429                 return cast(char[]) data[position..limit];
00430         }
00431 
00432         /***********************************************************************
00433         
00434                 Skip ahead by the specified number of bytes, streaming from 
00435                 the associated conduit as necessary.
00436         
00437                 Can also reverse the read position by 'size' bytes. This may
00438                 be used to support lookahead-type operations.
00439 
00440                 Returns true if successful, false otherwise.
00441 
00442         ***********************************************************************/
00443 
00444         bool skip (int size)
00445         {
00446                 if (size < 0)
00447                    {
00448                    size = -size;
00449                    if (position >= size)
00450                       {
00451                       position -= size;
00452                       return true;
00453                       }
00454                    return false;
00455                    }
00456                 return get (size) !== null;
00457         }
00458 
00459         /***********************************************************************
00460         
00461                 Return count of readable bytes remaining in buffer. This is 
00462                 calculated simply as limit() - position()
00463 
00464         ***********************************************************************/
00465 
00466         uint readable ()
00467         {
00468                 return limit - position;
00469         }               
00470 
00471         /***********************************************************************
00472         
00473                 Return count of writable bytes available in buffer. This is 
00474                 calculated simply as capacity() - limit()
00475 
00476         ***********************************************************************/
00477 
00478         uint writable ()
00479         {
00480                 return capacity - limit;
00481         }               
00482 
00483         /***********************************************************************
00484 
00485                 Exposes the raw data buffer at the current write position, 
00486                 The delegate is provided with a void[] representing space
00487                 available within the buffer at the current write position.
00488 
00489                 The delegate should return the approriate number of bytes 
00490                 if it writes valid content, or IConduit.Eof on error.
00491 
00492                 Returns whatever the delegate returns.
00493 
00494         ***********************************************************************/
00495 
00496         int write (int delegate (void[]) dg)
00497         {
00498                 int count = dg (data [limit..capacity]);
00499 
00500                 if (count != IConduit.Eof) 
00501                    {
00502                    limit += count;
00503                    assert (limit <= capacity);
00504                    }
00505                 return count;
00506         }               
00507 
00508         /***********************************************************************
00509 
00510                 Exposes the raw data buffer at the current read position. The
00511                 delegate is provided with a void[] representing the available
00512                 data, and should return zero to leave the current read position
00513                 intact. 
00514                 
00515                 If the delegate consumes data, it should return the number of 
00516                 bytes consumed; or IConduit.Eof to indicate an error.
00517 
00518                 Returns whatever the delegate returns.
00519 
00520         ***********************************************************************/
00521 
00522         int read (int delegate (void[]) dg)
00523         {
00524                 int count = dg (data [position..limit]);
00525                 
00526                 if (count != IConduit.Eof)
00527                    {
00528                    position += count;
00529                    assert (position <= limit);
00530                    }
00531                 return count;
00532         }               
00533 
00534         /***********************************************************************
00535 
00536                 If we have some data left after an export, move it to 
00537                 front-of-buffer and set position to be just after the 
00538                 remains. This is for supporting certain conduits which 
00539                 choose to write just the initial portion of a request.
00540                 
00541                 Limit is set to the amount of data remaining. Position 
00542                 is always reset to zero.
00543 
00544         ***********************************************************************/
00545 
00546         IBuffer compress ()
00547         {
00548                 uint r = readable ();
00549 
00550                 if (position > 0 && r > 0)
00551                     // content may overlap ...
00552                     memcpy (&data[0], &data[position], r);
00553 
00554                 position = 0;
00555                 limit = r;
00556                 return this;
00557         }               
00558 
00559         /***********************************************************************
00560         
00561                 flush the contents of this buffer to the related conduit.
00562                 Throws an IOException on premature eof.
00563 
00564         ***********************************************************************/
00565 
00566         void flush ()
00567         {
00568                 if (conduit)
00569                     conduit.flush (this);
00570                 else
00571                    clear();
00572         }               
00573 
00574         /***********************************************************************
00575         
00576                 Reset 'position' and 'limit' to zero
00577 
00578         ***********************************************************************/
00579 
00580         IBuffer clear ()
00581         {
00582                 position = limit = 0;
00583                 return this;
00584         }               
00585 
00586         /***********************************************************************
00587         
00588                 Returns the limit of readable content within this buffer
00589 
00590         ***********************************************************************/
00591 
00592         uint getLimit ()
00593         {
00594                 return limit;
00595         }
00596 
00597         /***********************************************************************
00598         
00599                 Returns the total capacity of this buffer
00600 
00601         ***********************************************************************/
00602 
00603         uint getCapacity ()
00604         {
00605                 return capacity;
00606         }
00607 
00608         /***********************************************************************
00609         
00610                 Returns the current read-position within this buffer
00611 
00612         ***********************************************************************/
00613 
00614         uint getPosition ()
00615         {
00616                 return position;
00617         }
00618 
00619         /***********************************************************************
00620         
00621                 Returns the conduit associated with this buffer. Returns 
00622                 null if the buffer is purely memory based; that is, it's
00623                 not backed by some external medium.
00624 
00625                 Buffers do not require an external conduit to operate, but 
00626                 it can be convenient to associate one. For example, methods
00627                 read and write use it to import/export content as necessary.
00628 
00629         ***********************************************************************/
00630 
00631         IConduit getConduit ()
00632         {
00633                 return conduit;
00634         }
00635 
00636         /***********************************************************************
00637         
00638                 Sets the external conduit associated with this buffer.
00639 
00640                 Buffers do not require an external conduit to operate, but 
00641                 it can be convenient to associate one. For example, methods
00642                 read and write use it to import/export content as necessary.
00643 
00644         ***********************************************************************/
00645 
00646         void setConduit (IConduit conduit)
00647         {
00648                 this.conduit = conduit;
00649         }
00650 }
00651 
00652 
00653 /*******************************************************************************
00654 
00655         Subclass to support automatic flushing. This can be used for 
00656         Stdout, Stderr, and other related conduits.
00657 
00658 *******************************************************************************/
00659 
00660 class FlushBuffer : Buffer
00661 {
00662         /***********************************************************************
00663         
00664                 Construct a flushing buffer with the given size upon the 
00665                 provided IConduit
00666 
00667         ***********************************************************************/
00668 
00669         this (int size, IConduit conduit)
00670         in {
00671            assert (conduit);
00672            }
00673         body
00674         {
00675                 super (size);
00676                 setConduit (conduit);
00677         }
00678 
00679         /***********************************************************************
00680         
00681                 Intercept buffer writes so we can flush the new content
00682 
00683         ***********************************************************************/
00684 
00685         protected override void* copy (void *src, uint size)
00686         {
00687                 void *ret = super.copy (src, size);
00688                 getConduit().flush (this);
00689                 return ret;
00690         }
00691 }
00692 
00693 
00694 /*******************************************************************************
00695 
00696         Subclass to provide support for content growth. This is handy when
00697         you want to keep a buffer around as a scratchpad.
00698 
00699 *******************************************************************************/
00700 
00701 class GrowableBuffer : Buffer
00702 {
00703         /***********************************************************************
00704         
00705                 Create a GrowableBuffer with the specified initial size.
00706 
00707         ***********************************************************************/
00708 
00709         this (uint size)
00710         {
00711                 super (size);
00712         }
00713 
00714         /***********************************************************************
00715 
00716                 Overridable method to grow the buffer size when it becomes 
00717                 full. Default is to not grow at all.
00718         
00719         ***********************************************************************/
00720 
00721         protected override bool grow (uint size)
00722         {
00723                 if (data.length < uint.max / 2)
00724                    {
00725                    uint i = data.length * 2;
00726                    if (i < size)
00727                        i = size;
00728                    data.length = i;
00729                    capacity = i;
00730                    return true;
00731                    }
00732                 return false;
00733         }
00734 }
00735 
00736 
00737 
00738 /*******************************************************************************
00739 
00740         Subclass to treat the buffer as a seekable entity, where all 
00741         capacity is available for reading and/or writing. To achieve 
00742         this we must effectively disable the 'limit' watermark, and 
00743         locate write operations around 'position' instead. 
00744 
00745 *******************************************************************************/
00746 
00747 class MappedBuffer : Buffer
00748 {
00749         /***********************************************************************
00750                 
00751                 Construct an empty MappedBuffer 
00752 
00753         ***********************************************************************/
00754 
00755         this ()
00756         {
00757                 super (0);
00758         }
00759 
00760         /***********************************************************************
00761                 
00762                 Close this mapped buffer
00763 
00764         ***********************************************************************/
00765 
00766         abstract void close ();
00767 
00768         /***********************************************************************
00769                 
00770                 Flush any dirty content out to the drive
00771 
00772         ***********************************************************************/
00773 
00774         abstract void flush ();
00775 
00776         /***********************************************************************
00777                 
00778                 Set the read/write position
00779 
00780         ***********************************************************************/
00781 
00782         void setPosition (uint position)
00783         {
00784                 this.position = position;
00785         }
00786 
00787         /***********************************************************************
00788         
00789                 Seek to the specified position within the buffer, and return
00790                 the byte offset of the new location (relative to zero).
00791 
00792         ***********************************************************************/
00793 
00794         long seek (long offset, ISeekable.SeekAnchor anchor)
00795         {
00796                 long pos = capacity;
00797 
00798                 if (anchor == ISeekable.SeekAnchor.Begin)
00799                     pos = offset;
00800                 else
00801                    if (anchor == ISeekable.SeekAnchor.End)
00802                        pos -= offset;
00803                    else
00804                       pos = position + offset;
00805 
00806                 return position = pos;
00807         }
00808 
00809         /***********************************************************************
00810         
00811                 Return count of writable bytes available in buffer. This is 
00812                 calculated simply as capacity() - limit()
00813 
00814         ***********************************************************************/
00815 
00816         override uint writable ()
00817         {
00818                 return capacity - position;
00819         }               
00820 
00821         /***********************************************************************
00822         
00823                 Bulk copy of data from 'src'. Position is adjusted by 'size'
00824                 bytes.
00825 
00826         ***********************************************************************/
00827 
00828         override protected void* copy (void *src, uint size)
00829         {
00830                 void* ret = &data[position];
00831                 data[position..position+size] = src[0..size];
00832                 position += size;
00833                 return ret;
00834         }
00835 
00836         /***********************************************************************
00837 
00838                 Exposes the raw data buffer at the current write position, 
00839                 The delegate is provided with a void[] representing space
00840                 available within the buffer at the current write position.
00841 
00842                 The delegate should return the approriate number of bytes 
00843                 if it writes valid content, or IConduit.Eof on error.
00844 
00845                 Returns whatever the delegate returns.
00846 
00847         ***********************************************************************/
00848 
00849         override int write (int delegate (void[]) dg)
00850         {
00851                 int count = dg (data [position..capacity]);
00852 
00853                 if (count != IConduit.Eof) 
00854                    {
00855                    position += count;
00856                    assert (position <= capacity);
00857                    }
00858                 return count;
00859         }               
00860 
00861         /***********************************************************************
00862 
00863                 Prohibit compress() from doing anything at all.
00864 
00865         ***********************************************************************/
00866 
00867         override IBuffer compress ()
00868         {
00869                 return this;
00870         }               
00871 
00872         /***********************************************************************
00873 
00874                 Prohibit clear() from doing anything at all.
00875 
00876         ***********************************************************************/
00877 
00878         override IBuffer clear ()
00879         {
00880                 return this;
00881         }               
00882 
00883         /***********************************************************************
00884         
00885                 Prohibit the setting of another IConduit
00886 
00887         ***********************************************************************/
00888 
00889         override void setConduit (IConduit conduit)
00890         {
00891                 assert (0);
00892         }
00893 }
00894 
00895 
00896 

Generated on Sun Nov 7 19:06:49 2004 for Mango by doxygen 1.3.6