Main Page | Class Hierarchy | Alphabetical List | Class List | Directories | File List | Class Members | File Members | Related Pages

Buffer.d

Go to the documentation of this file.
00001 /*******************************************************************************
00002 
00003         @file Buffer.d
00004 
00005         Copyright (c) 2004 Kris Bell
00006         
00007         This software is provided 'as-is', without any express or implied
00008         warranty. In no event will the authors be held liable for damages
00009         of any kind arising from the use of this software.
00010         
00011         Permission is hereby granted to anyone to use this software for any 
00012         purpose, including commercial applications, and to alter it and/or 
00013         redistribute it freely, subject to the following restrictions:
00014         
00015         1. The origin of this software must not be misrepresented; you must 
00016            not claim that you wrote the original software. If you use this 
00017            software in a product, an acknowledgment within documentation of 
00018            said product would be appreciated but is not required.
00019 
00020         2. Altered source versions must be plainly marked as such, and must 
00021            not be misrepresented as being the original software.
00022 
00023         3. This notice may not be removed or altered from any distribution
00024            of the source.
00025 
00026         4. Derivative works are permitted, but they must carry this notice
00027            in full and credit the original source.
00028 
00029 
00030                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
00031 
00032       
00033         @version        Initial version; March 2004
00034 
00035         @author         Kris
00036 
00037 
00038 *******************************************************************************/
00039 
00040 module mango.io.Buffer;
00041 
00042 private import  mango.io.Exception;
00043 
00044 public  import  mango.io.model.IBuffer;
00045 
00046 public  import  mango.io.model.IConduit;
00047 
00048 
00049 /******************************************************************************
00050 
00051 ******************************************************************************/
00052 
00053 extern (C)
00054 {
00055         void * memcpy (void *dst, void *src, uint);
00056 }       
00057 
00058 /*******************************************************************************
00059 
00060         The basic premise behind this IO package is as follows:
00061 
00062         @li The central concept is that of a buffer. The buffer acts
00063            as a queue (line) where items are removed from the front
00064            and new items are added to the back. Buffers are modeled 
00065            by mango.io.model.IBuffer, and a concrete implementation is 
00066            provided this class.
00067 
00068         @li Buffers can be written to directly, but a Reader and/or
00069            Writer are typically used to read & write formatted data.
00070            These readers & writers are bound to a specific buffer;
00071            often the same buffer. It's also perfectly legitimate to 
00072            bind multiple writers to the same buffer; they will all
00073            behave serially as one would expect. The same applies to
00074            multiple readers on the same buffer. Readers and writers
00075            support three styles of IO: put/get, the C++ style << &
00076            >> operators, and the () whisper style. All operations 
00077            can be chained.
00078 
00079         @li Any class can be made compatable with the reader/writer
00080            framework by implementing the IReadable and/or IWritable 
00081            interfaces. Each of these specify just a single method.
00082            Once compatable, the class can simply be passed to the 
00083            reader/writer as if it were native data.
00084 
00085         @li Buffers may also be tokenized. This is handy when one is
00086            dealing with text input, and/or the content suits a more
00087            fluid format than most typical readers & writers support.
00088            Tokens are mapped directly onto buffer content, so there
00089            is only minor overhead in using them. Tokens can be read
00090            and written by reader/writers also, using a more relaxed
00091            set of rules than those applied to discrete I/O.
00092 
00093         @li Buffers are sometimes memory-only, in which case there
00094            is nothing left to do when a reader (or tokenizer) hits
00095            end of buffer conditions. Other buffers are themselves 
00096            bound to a Conduit. When this is the case, a reader will 
00097            eventually cause the buffer to reload via its associated 
00098            conduit. Previous buffer content will thus be lost. The
00099            same concept is applied to writers, whereby they flush 
00100            the content of a full buffer to a bound conduit before 
00101            continuing. 
00102 
00103         @li Conduits provide virtualized access to external content,
00104            and represent things like files or Internet connections.
00105            They are just a different kind of stream. Conduits are
00106            modelled by mango.io.model.IConduit, and implemented via
00107            classes FileConduit and SocketConduit. Additional kinds
00108            of conduit are easy to construct: one either subclasses
00109            mango.io.Conduit, or implements mango.io.model.IConduit. 
00110            A conduit reads and writes from/to a buffer in big chunks
00111            (typically the entire buffer).
00112 
00113         @li Conduits may have one or more filters attached. These 
00114             will process content as it flows back and forth across
00115             the conduit. Filter examples include compression, utf
00116             transcoding, and endian transformation. These filters
00117             apply to the entire scope of the conduit, rather than
00118             being specific to one data-type or another.
00119 
00120         @li Readers & writers may have a transcoder attached. The
00121            role of a transcoder is to aid in converting between each
00122            representation of text (utf8, utf16, utf32). They're used
00123            to normalize string I/O according to a standard text-type.
00124            By default there is no transcoder attached, and the type
00125            is therefore considered "raw".           
00126 
00127 
00128         An example of how to append a buffer follows:
00129 
00130         @code
00131         char[] foo = "to write some D";
00132 
00133         // create a small buffer
00134         auto buf = new Buffer (256);
00135 
00136         // append some text directly to it
00137         buf.append("now is the time for all good men ").append(foo);
00138 
00139         // output the combined string
00140         Cout (buf.toString);
00141         @endcode
00142 
00143         Alternatively, one might use a Writer to append the buffer:
00144 
00145         @code
00146         auto write = new Writer (new Buffer(256));
00147         write ("now is the time for all good men "c) (foo);
00148         @endcode
00149 
00150         Or, using printf-like formatting:
00151 
00152         @code
00153         auto write = new DisplayWriter (new Buffer(256));
00154         write.print ("now is the time for %d good men %s", 3, foo);
00155         @endcode
00156 
00157         You might use a GrowBuffer instead where you wish to 
00158         append beyond a preset limit. One common usage of buffers 
00159         is in conjunction with a conduit, such as FileConduit. Each 
00160         conduit exposes a preferred-size for its associated buffers, 
00161         utilized during Buffer construction:
00162 
00163         @code
00164         auto file = new FileConduit ("file.name");
00165         auto buf = new Buffer (file);
00166         @endcode
00167 
00168         However, this is typically hidden by higher level constructors 
00169         such as those of Reader and Writer derivitives. For example:
00170 
00171         @code
00172         auto file = new FileConduit ("file.name");
00173         auto read = new Reader (file);
00174         @endcode
00175 
00176         There is indeed a buffer between the Reader and Conduit, but 
00177         explicit construction is unecessary in most cases.  See both 
00178         Reader and Writer for examples of formatted IO.
00179 
00180         Stdout is a predefined DisplayWriter, attached to a conduit
00181         representing the console. Thus, all conduit operations are
00182         legitimate on Stdout and Stderr:
00183 
00184         @code
00185         Stdout.conduit.copy (new FileConduit ("readme.txt"));
00186         @endcode
00187 
00188         In addition to the standard writer facilities, Stdout also has
00189         support for formatted output:
00190 
00191         @code 
00192         Stdout.println ("now is the time for %d good men %s", 3, foo);
00193         @endcode
00194 
00195         Buffers are useful for many purposes within Mango, but there
00196         are times when it is more straightforward to avoid them. For
00197         such cases, conduit derivatives (such as FileConduit) support
00198         direct I/O via a pair of read() and write() methods. These 
00199         alternate methods will also invoke any attached filters.
00200 
00201 
00202 *******************************************************************************/
00203 
00204 class Buffer : IBuffer
00205 {
00206         protected void[]        data;
00207         protected Style         style;
00208         protected uint          limit;
00209         protected uint          capacity;
00210         protected uint          position;
00211         protected IConduit      conduit;
00212 
00213         private static char[] overflow  = "output buffer overflow";
00214         private static char[] underflow = "input buffer underflow";
00215         private static char[] eofRead   = "end-of-file whilst reading";
00216         private static char[] eofWrite  = "end-of-file whilst writing";
00217 
00218         /***********************************************************************
00219         
00220                 Ensure the buffer remains valid between method calls
00221                  
00222         ***********************************************************************/
00223 
00224         invariant 
00225         {
00226                assert (position <= limit);
00227                assert (limit <= capacity);
00228         }
00229 
00230         /***********************************************************************
00231         
00232                 Construct a Buffer upon the provided conduit
00233 
00234         ***********************************************************************/
00235 
00236         this (IConduit conduit)
00237         {
00238                 this (conduit.bufferSize());
00239                 this.style = conduit.isTextual ? Text : Binary;
00240                 setConduit (conduit);
00241         }
00242 
00243         /***********************************************************************
00244         
00245                 Construct a Buffer with the specified number of bytes.
00246 
00247         ***********************************************************************/
00248 
00249         this (uint capacity=0)
00250         {
00251                 this (new ubyte[capacity]);              
00252         }
00253 
00254         /***********************************************************************
00255         
00256                 Prime buffer with an application-supplied array. There is 
00257                 no readable data present, and writing begins at position 0.
00258 
00259         ***********************************************************************/
00260 
00261         this (void[] data)
00262         {
00263                 this (data, 0);
00264         }
00265 
00266         /***********************************************************************
00267         
00268                 Prime buffer with an application-supplied array, and 
00269                 indicate how much readable data is already there. A
00270                 write operation will begin writing immediately after
00271                 the existing content.
00272 
00273         ***********************************************************************/
00274 
00275         this (void[] data, uint readable)
00276         {
00277                 setContent (data, readable);
00278         }
00279 
00280         /***********************************************************************
00281         
00282                 Throw an exception with the provided message
00283 
00284         ***********************************************************************/
00285 
00286         final void error (char[] msg)
00287         {
00288                 throw new IOException (msg);
00289         }
00290 
00291         /***********************************************************************
00292                 
00293                 Return the backing array
00294 
00295         ***********************************************************************/
00296 
00297         protected void[] getContent ()
00298         {
00299                 return data;
00300         }
00301 
00302         /***********************************************************************
00303                 
00304                 Return style of buffer
00305 
00306         ***********************************************************************/
00307 
00308         Style getStyle ()
00309         {
00310                 return style;
00311         }
00312 
00313         /***********************************************************************
00314         
00315                 Set the backing array with all content readable. Writing
00316                 to this will either flush it to an associated conduit, or
00317                 raise an Eof condition. Use IBuffer.clear() to reset the
00318                 content (make it all writable).
00319 
00320         ***********************************************************************/
00321 
00322         IBuffer setValidContent (void[] data)
00323         {
00324                 return setContent (data, data.length);
00325         }
00326 
00327         /***********************************************************************
00328         
00329                 Set the backing array with some content readable. Writing
00330                 to this will either flush it to an associated conduit, or
00331                 raise an Eof condition. Use IBuffer.clear() to reset the
00332                 content (make it all writable).
00333 
00334         ***********************************************************************/
00335 
00336         IBuffer setContent (void[] data, uint readable=0)
00337         {
00338                 this.data = data;
00339                 this.limit = readable;
00340                 this.capacity = data.length;   
00341 
00342                 // reset to start of input
00343                 this.position = 0;
00344 
00345                 return this;            
00346         }
00347 
00348         /***********************************************************************
00349 
00350                 Overridable method to grow the buffer to the specified size 
00351                 when it becomes full. Default is to not grow at all.
00352 
00353         ***********************************************************************/
00354 
00355         protected bool grow (uint size)
00356         {
00357                 return false;
00358         }
00359 
00360         /***********************************************************************
00361         
00362                 Bulk copy of data from 'src'. Limit is adjusted by 'size'
00363                 bytes.
00364 
00365         ***********************************************************************/
00366 
00367         protected void copy (void *src, uint size)
00368         {
00369                 data[limit..limit+size] = src[0..size];
00370                 limit += size;
00371         }
00372 
00373         /***********************************************************************
00374         
00375                 Read a chunk of data from the buffer, loading from the
00376                 conduit as necessary. The specified number of bytes is
00377                 loaded into the buffer, and marked as having been read 
00378                 when the 'eat' parameter is set true. When 'eat' is set
00379                 false, the read position is not adjusted.
00380 
00381                 Returns the corresponding buffer slice when successful, 
00382                 or null if there's not enough data available (Eof; Eob).
00383 
00384         ***********************************************************************/
00385 
00386         void[] get (uint size, bool eat = true)
00387         {   
00388                 if (size > readable)
00389                    {
00390                    if (conduit is null)
00391                        error (underflow);
00392 
00393                    // make some space? This will try to leave as much content
00394                    // in the buffer as possible, such that entire records may
00395                    // be aliased directly from within. 
00396                    if (size > writable)
00397                       {
00398                       if (size > capacity)
00399                           error (underflow);
00400                       compress ();
00401                       }
00402 
00403                    // populate tail of buffer with new content
00404                    do {
00405                       if (fill(conduit) == IConduit.Eof)
00406                           error (eofRead);
00407                       } while (size > readable);
00408                    }
00409 
00410                 uint i = position;
00411                 if (eat)
00412                     position += size;
00413                 return data [i .. i + size];               
00414         }
00415 
00416         /***********************************************************************
00417         
00418                 Append an array of data to this buffer, and flush to the
00419                 conduit as necessary. Returns a chaining reference if all 
00420                 data was written; throws an IOException indicating eof or 
00421                 eob if not.
00422 
00423                 This is often used in lieu of a Writer.
00424 
00425         ***********************************************************************/
00426 
00427         IBuffer append (void[] src)        
00428         {               
00429                 uint size = src.length;
00430 
00431                 if (size > writable)
00432                     // can we write externally?
00433                     if (conduit)
00434                        {
00435                        flush ();
00436 
00437                        // check for pathological case
00438                        if (size > capacity)
00439                           {
00440                           conduit.flush (src);
00441                           return this;
00442                           }
00443                        }
00444                     else
00445                        // try to grow buffer ...
00446                        if (! grow (size))
00447                              error (overflow);
00448 
00449                 copy (src, size);
00450                 return this;
00451         }
00452 
00453         /***********************************************************************
00454         
00455                 Return a char[] slice of the buffer up to the limit of
00456                 valid content.
00457 
00458         ***********************************************************************/
00459 
00460         override char[] toString ()
00461         {       
00462                 return cast(char[]) data[position..limit];
00463         }
00464 
00465         /***********************************************************************
00466         
00467                 Skip ahead by the specified number of bytes, streaming from 
00468                 the associated conduit as necessary.
00469         
00470                 Can also reverse the read position by 'size' bytes. This may
00471                 be used to support lookahead-type operations.
00472 
00473                 Returns true if successful, false otherwise.
00474 
00475         ***********************************************************************/
00476 
00477         bool skip (int size)
00478         {
00479                 if (size < 0)
00480                    {
00481                    size = -size;
00482                    if (position >= size)
00483                       {
00484                       position -= size;
00485                       return true;
00486                       }
00487                    return false;
00488                    }
00489                 return cast(bool) (get (size) !is null);
00490         }
00491 
00492         /***********************************************************************
00493         
00494                 Return count of readable bytes remaining in buffer. This is 
00495                 calculated simply as limit() - position()
00496 
00497         ***********************************************************************/
00498 
00499         uint readable ()
00500         {
00501                 return limit - position;
00502         }               
00503 
00504         /***********************************************************************
00505         
00506                 Return count of writable bytes available in buffer. This is 
00507                 calculated simply as capacity() - limit()
00508 
00509         ***********************************************************************/
00510 
00511         uint writable ()
00512         {
00513                 return capacity - limit;
00514         }               
00515 
00516         /***********************************************************************
00517 
00518                 Exposes the raw data buffer at the current write position, 
00519                 The delegate is provided with a void[] representing space
00520                 available within the buffer at the current write position.
00521 
00522                 The delegate should return the appropriate number of bytes 
00523                 if it writes valid content, or IConduit.Eof on error.
00524 
00525                 Returns whatever the delegate returns.
00526 
00527         ***********************************************************************/
00528 
00529         uint write (uint delegate (void[]) dg)
00530         {
00531                 int count = dg (data [limit..capacity]);
00532 
00533                 if (count != IConduit.Eof) 
00534                    {
00535                    limit += count;
00536                    assert (limit <= capacity);
00537                    }
00538                 return count;
00539         }               
00540 
00541         /***********************************************************************
00542 
00543                 Exposes the raw data buffer at the current read position. The
00544                 delegate is provided with a void[] representing the available
00545                 data, and should return zero to leave the current read position
00546                 intact. 
00547                 
00548                 If the delegate consumes data, it should return the number of 
00549                 bytes consumed; or IConduit.Eof to indicate an error.
00550 
00551                 Returns whatever the delegate returns.
00552 
00553         ***********************************************************************/
00554 
00555         uint read (uint delegate (void[]) dg)
00556         {
00557                 
00558                 int count = dg (data [position..limit]);
00559                 
00560                 if (count != IConduit.Eof)
00561                    {
00562                    position += count;
00563                    assert (position <= limit);
00564                    }
00565                 return count;
00566         }               
00567 
00568         /***********************************************************************
00569 
00570                 If we have some data left after an export, move it to 
00571                 front-of-buffer and set position to be just after the 
00572                 remains. This is for supporting certain conduits which 
00573                 choose to write just the initial portion of a request.
00574                 
00575                 Limit is set to the amount of data remaining. Position 
00576                 is always reset to zero.
00577 
00578         ***********************************************************************/
00579 
00580         IBuffer compress ()
00581         {
00582                 uint r = readable ();
00583 
00584                 if (position > 0 && r > 0)
00585                     // content may overlap ...
00586                     memcpy (&data[0], &data[position], r);
00587 
00588                 position = 0;
00589                 limit = r;
00590                 return this;
00591         }               
00592 
00593         /***********************************************************************
00594 
00595                 Try to fill the available buffer with content from the 
00596                 specified conduit. In particular, we will never ask to 
00597                 read less than 32 bytes. This permits conduit-filters 
00598                 to operate within a known environment.
00599 
00600                 Returns the number of bytes read, or throws an underflow
00601                 error if there nowhere to read from
00602         
00603         ***********************************************************************/
00604 
00605         uint fill ()
00606         {
00607                 if (! conduit)
00608                       error (underflow);
00609                 return fill (conduit);
00610         }
00611 
00612         /***********************************************************************
00613 
00614                 Try to fill the available buffer with content from the 
00615                 specified conduit. In particular, we will never ask to 
00616                 read less than 32 bytes ~ this permits conduit-filters 
00617                 to operate within a known environment. We also try to
00618                 read as much as possible by clearing the buffer when 
00619                 all current content has been eaten.
00620 
00621                 Returns the number of bytes read, or Conduit.Eof
00622         
00623         ***********************************************************************/
00624 
00625         uint fill (IConduit conduit)
00626         {
00627                 if (readable is 0)
00628                     clear();
00629                 else
00630                    if (writable < 32)
00631                        if (compress().writable < 32)
00632                            error ("input buffer is too small");
00633 
00634                 return write (&conduit.read);
00635         } 
00636 
00637         /***********************************************************************
00638 
00639                 make some room in the buffer
00640                         
00641         ***********************************************************************/
00642 
00643         void makeRoom (int space)
00644         {
00645                 if (conduit)
00646                     drain ();
00647                 else
00648                    if (! grow (space))
00649                          error (overflow);
00650         }
00651 
00652         /***********************************************************************
00653 
00654                 Write as much of the buffer that the associated conduit
00655                 can consume.
00656 
00657                 Returns the number of bytes written, or Conduit.Eof
00658         
00659         ***********************************************************************/
00660 
00661         uint drain ()
00662         {
00663                 uint ret = read (&conduit.write);
00664                 if (ret == conduit.Eof)
00665                     error (eofWrite);
00666 
00667                 compress ();
00668                 return ret;
00669         }
00670 
00671         /***********************************************************************
00672         
00673                 flush the contents of this buffer to the related conduit.
00674                 Throws an IOException on premature eof.
00675 
00676         ***********************************************************************/
00677 
00678         void flush ()
00679         {
00680                 if (conduit)
00681                     if (conduit.flush (data [position..limit]))
00682                         clear();
00683                     else
00684                        error (eofWrite);
00685         } 
00686 
00687         /***********************************************************************
00688         
00689                 Reset 'position' and 'limit' to zero
00690 
00691         ***********************************************************************/
00692 
00693         IBuffer clear ()
00694         {
00695                 position = limit = 0;
00696                 return this;
00697         }               
00698 
00699         /***********************************************************************
00700         
00701                 Returns the limit of readable content within this buffer
00702 
00703         ***********************************************************************/
00704 
00705         uint getLimit ()
00706         {
00707                 return limit;
00708         }
00709 
00710         /***********************************************************************
00711         
00712                 Returns the total capacity of this buffer
00713 
00714         ***********************************************************************/
00715 
00716         uint getCapacity ()
00717         {
00718                 return capacity;
00719         }
00720 
00721         /***********************************************************************
00722         
00723                 Returns the current read-position within this buffer
00724 
00725         ***********************************************************************/
00726 
00727         uint getPosition ()
00728         {
00729                 return position;
00730         }
00731 
00732         /***********************************************************************
00733         
00734                 Returns the conduit associated with this buffer. Returns 
00735                 null if the buffer is purely memory based; that is, it's
00736                 not backed by some external medium.
00737 
00738                 Buffers do not require an external conduit to operate, but 
00739                 it can be convenient to associate one. For example, methods
00740                 fill() & drain() use it to import/export content as necessary.
00741 
00742         ***********************************************************************/
00743 
00744         IConduit getConduit ()
00745         {
00746                 return conduit;
00747         }
00748 
00749         /***********************************************************************
00750         
00751                 Sets the external conduit associated with this buffer.
00752 
00753                 Buffers do not require an external conduit to operate, but 
00754                 it can be convenient to associate one. For example, methods
00755                 fill() & drain() use it to import/export content as necessary.
00756 
00757         ***********************************************************************/
00758 
00759         void setConduit (IConduit conduit)
00760         {
00761                 this.conduit = conduit;
00762         }
00763 }

Generated on Sat Dec 24 17:28:31 2005 for Mango by  doxygen 1.4.0