The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,737 other followers

.NET/C# – Issue with FileStream on network with transfers larger than 64 megabyte

Posted by jpluimers on 2009/04/27

Quite a while ago (2006!), I bumped into an issue when copying large chuncks of data to a network.

I posted it to Google, mentioned that breaking up the data in smaller blocks worked, but never had the time to post the solution.

So here it is :-)

First the problem:

The old code consistently fails when:

  • the FileStream is on a network
  • and the MemoryStream is 64 megabytes or larger

The old code succeeds when:

  • the MemoryStream is smaller than 64 megabytes
  • or the FileStream is not on a network

An example of the exception message you get upon failure:

System.IO.IOException: Insufficient system resources exist to complete the requested service.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
...

Then the old code:

        public static void WriteMemoryStreamToFile(string filename, MemoryStream memory)
        {
            using (Stream
                file = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.ReadWrite),
                fileBuffer = new BufferedStream(file)
            )
            {
                byte[] memoryBuffer = memory.GetBuffer();
                int memoryLength = (int)memory.Length;
                fileBuffer.Write(memoryBuffer, 0, memoryLength); //##jpl: drawback: works only up to 2 gigabyte!
                fileBuffer.Close();
                file.Close();
            }
        }

Note that the old code already has a limitation of 2 gigabyte, back then this was not an issue (in 2006 there were not that many people having more than 2 gigabytes of memory, now that is a different story).

The code below now only solves the issue of the 64 megabyte limit, but also allows memorystreams of more than 2 gigabyte size to be streamed to file:

using System;
using System.IO;

namespace bo.IO
{
public class MemoryStreamHelper
{
public static void WriteMemoryStreamToFile(string filename, MemoryStream memory)
{
using (System.IO.Stream file = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
using (Stream buffer = new System.IO.BufferedStream(file))
{
byte[] memoryBuffer = memory.GetBuffer();
long memoryLength = memory.Length;
int index = 0;
const int copyBufferSize = 65536; // 64k
byte[] copyBuffer = new byte[copyBufferSize];
while (memoryLength > 0)
{
int actualLength;
if (memoryLength > copyBufferSize)
actualLength = copyBufferSize;
else
actualLength = (int)memoryLength; //jpl: this cast is valid, as now memoryLength <= copyBufferSize Array.Copy(memoryBuffer, index, copyBuffer, 0, actualLength); buffer.Write(copyBuffer, 0, actualLength); memoryLength = memoryLength - actualLength; index = index + actualLength; } buffer.Flush(); buffer.Close(); } } } } } [/sourcecode] Another issue solved :-) --jeroen

6 Responses to “.NET/C# – Issue with FileStream on network with transfers larger than 64 megabyte”

  1. JMiller said

    It’s worth noting that the MemoryStream is a bit of a tricky devil — one of the obvious constructors to use would be the “new MemoryStream(byte[])” constructor which will throw an exception when your code goes for memory.GetBuffer() because, as apparently documented anywhere *but* in the framework itself, that constructor “does not expose the underlying stream. GetBuffer throws UnauthorizedAccessException.”

    I’d advise an overload, as follows, for the byte[] folks so they don’t get bit by that particular issue.

    public static void WriteMemoryStreamToFile(string filename, byte[] memoryToWrite)
    {
    MemoryStream memory = new MemoryStream(memoryToWrite, 0, memoryToWrite.Length, true, true);
    WriteMemoryStreamToFile(filename, memory);
    }

    HiH! (And other than that, this post looks like it’ll be useful to me, so thanks!)

  2. kiquenet said

    Hi mister, in forums not solution for me.

    I have this problem, I want to know if using BufferedStream can I get solution. Thanks

    OutOfMemoryException using BZip2 (SharpZipLib)

    I use Asp.net , .net 3.5, win2003, iis 6.0.

    I use Oracle for gathering files, saving file in SharpZipLib.BZip2 compressed format in field RAW in table Oracle.

    My application is Web, and I use WCF Service for get data (array of bytes) of a file. The aspx page send file to user (download file).

    My issue-problem:

    I read DATA from Oracle, (I call to WCF Service). I get array of bytes (byte[]),

    I try Uncompress file using SharpZipLib.BZip2

    using (MemoryStream inData = new MemoryStream(data))
    {
    using (MemoryStream outData = new MemoryStream())
    {
    BZip2.Decompress(inData, outData); // 500 MB) !!!

    compressed file: 4MB

    uncompressed file: > 500 MB

    I do test like this:

    BufferedStream bufin = new BufferedStream(instream);

    using (MemoryStream outData = new MemoryStream())
    {
    BZip2.Decompress(bufin, outData);

    return outData.ToArray();
    }

    But I get the same OutOfMemoryException

    Trace Stack of Exception

    en System.IO.MemoryStream.set_Capacity(Int32 value)
    en System.IO.MemoryStream.EnsureCapacity(Int32 value)
    en System.IO.MemoryStream.WriteByte(Byte value)
    en Reale.Frk.Compression.BZip2.BZip2.Decompress(Stream inStream, Stream outStream)

    Code of SharpZipLib.BZip2.Decompress

    public static void Decompress(Stream inStream, Stream outStream)

    {

    if ( inStream == null ) {

    throw new ArgumentNullException(“inStream”);

    }

    if ( outStream == null ) {

    throw new ArgumentNullException(“outStream”);

    }

    using ( outStream ) {

    using ( BZip2InputStream bzis = new BZip2InputStream(inStream) ) {

    int ch = bzis.ReadByte();

    while (ch != -1) {

    outStream.WriteByte((byte)ch);

    ch = bzis.ReadByte();

    }

    }

    }

    }

    any suggestions, comments, sample source code ?

  3. jpluimers said

    In this case, the file I wanted to write was already in memory, so filename is the destination, and memory is the source.

    It is quite easy to adapt this solution to any kind of source stream.

    Regards,

    –jeroen

  4. Mike Komnenous said

    Let me start with that I am new to C#. Here is my dilema. I am trying to write a file passed from a Sharepoint library to the network. I am not seeing in either of your examples where the information (1’s and 0’s) is coming from for the buffer.write. Am I supposed to put the file I am trying to copy in the MemoryStream? Thank you for your help.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: