.NET/C# – Issue with FileStream on network with transfers larger than 64 megabyte
Posted by jpluimers on 2009/04/27
Quite a while ago (2006!), I bumped into an issue when copying large chuncks of data to a network.
I posted it to Google, mentioned that breaking up the data in smaller blocks worked, but never had the time to post the solution.
So here it is :-)
First the problem:
The old code consistently fails when:
- the FileStream is on a network
- and the MemoryStream is 64 megabytes or larger
The old code succeeds when:
- the MemoryStream is smaller than 64 megabytes
- or the FileStream is not on a network
An example of the exception message you get upon failure:
System.IO.IOException: Insufficient system resources exist to complete the requested service. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count) at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count) at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count) ...
Then the old code:
public static void WriteMemoryStreamToFile(string filename, MemoryStream memory)
{
using (Stream
file = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.ReadWrite),
fileBuffer = new BufferedStream(file)
)
{
byte[] memoryBuffer = memory.GetBuffer();
int memoryLength = (int)memory.Length;
fileBuffer.Write(memoryBuffer, 0, memoryLength); //##jpl: drawback: works only up to 2 gigabyte!
fileBuffer.Close();
file.Close();
}
}
Note that the old code already has a limitation of 2 gigabyte, back then this was not an issue (in 2006 there were not that many people having more than 2 gigabytes of memory, now that is a different story).
The code below now only solves the issue of the 64 megabyte limit, but also allows memorystreams of more than 2 gigabyte size to be streamed to file:
using System;
using System.IO;
namespace bo.IO
{
public class MemoryStreamHelper
{
public static void WriteMemoryStreamToFile(string filename, MemoryStream memory)
{
using (System.IO.Stream file = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
using (Stream buffer = new System.IO.BufferedStream(file))
{
byte[] memoryBuffer = memory.GetBuffer();
long memoryLength = memory.Length;
int index = 0;
const int copyBufferSize = 65536; // 64k
byte[] copyBuffer = new byte[copyBufferSize];
while (memoryLength > 0)
{
int actualLength;
if (memoryLength > copyBufferSize)
actualLength = copyBufferSize;
else
actualLength = (int)memoryLength; //jpl: this cast is valid, as now memoryLength <= copyBufferSize
Array.Copy(memoryBuffer, index, copyBuffer, 0, actualLength);
buffer.Write(copyBuffer, 0, actualLength);
memoryLength = memoryLength - actualLength;
index = index + actualLength;
}
buffer.Flush();
buffer.Close();
}
}
}
}
}
[/sourcecode]
Another issue solved :-)
--jeroen
The Wiert Corner - irregular stream of stuff 
JMiller said
It’s worth noting that the MemoryStream is a bit of a tricky devil — one of the obvious constructors to use would be the “new MemoryStream(byte[])” constructor which will throw an exception when your code goes for memory.GetBuffer() because, as apparently documented anywhere *but* in the framework itself, that constructor “does not expose the underlying stream. GetBuffer throws UnauthorizedAccessException.”
I’d advise an overload, as follows, for the byte[] folks so they don’t get bit by that particular issue.
public static void WriteMemoryStreamToFile(string filename, byte[] memoryToWrite)
{
MemoryStream memory = new MemoryStream(memoryToWrite, 0, memoryToWrite.Length, true, true);
WriteMemoryStreamToFile(filename, memory);
}
HiH! (And other than that, this post looks like it’ll be useful to me, so thanks!)
jpluimers said
Thanks for the tip.
Glad you find the post useful!
–jeroen
kiquenet said
Hi mister, in forums not solution for me.
I have this problem, I want to know if using BufferedStream can I get solution. Thanks
OutOfMemoryException using BZip2 (SharpZipLib)
I use Asp.net , .net 3.5, win2003, iis 6.0.
I use Oracle for gathering files, saving file in SharpZipLib.BZip2 compressed format in field RAW in table Oracle.
My application is Web, and I use WCF Service for get data (array of bytes) of a file. The aspx page send file to user (download file).
My issue-problem:
I read DATA from Oracle, (I call to WCF Service). I get array of bytes (byte[]),
I try Uncompress file using SharpZipLib.BZip2
using (MemoryStream inData = new MemoryStream(data))
{
using (MemoryStream outData = new MemoryStream())
{
BZip2.Decompress(inData, outData); // 500 MB) !!!
compressed file: 4MB
uncompressed file: > 500 MB
I do test like this:
BufferedStream bufin = new BufferedStream(instream);
using (MemoryStream outData = new MemoryStream())
{
BZip2.Decompress(bufin, outData);
return outData.ToArray();
}
But I get the same OutOfMemoryException
Trace Stack of Exception
en System.IO.MemoryStream.set_Capacity(Int32 value)
en System.IO.MemoryStream.EnsureCapacity(Int32 value)
en System.IO.MemoryStream.WriteByte(Byte value)
en Reale.Frk.Compression.BZip2.BZip2.Decompress(Stream inStream, Stream outStream)
Code of SharpZipLib.BZip2.Decompress
public static void Decompress(Stream inStream, Stream outStream)
{
if ( inStream == null ) {
throw new ArgumentNullException(“inStream”);
}
if ( outStream == null ) {
throw new ArgumentNullException(“outStream”);
}
using ( outStream ) {
using ( BZip2InputStream bzis = new BZip2InputStream(inStream) ) {
int ch = bzis.ReadByte();
while (ch != -1) {
outStream.WriteByte((byte)ch);
ch = bzis.ReadByte();
}
}
}
}
any suggestions, comments, sample source code ?
jpluimers said
It is better if you ask these kinds of questions at StackOverflow.com: many more people will read it, so you have a much higher chance for getting an answer.
–jeroen
jpluimers said
In this case, the file I wanted to write was already in memory, so filename is the destination, and memory is the source.
It is quite easy to adapt this solution to any kind of source stream.
Regards,
–jeroen
Mike Komnenous said
Let me start with that I am new to C#. Here is my dilema. I am trying to write a file passed from a Sharepoint library to the network. I am not seeing in either of your examples where the information (1’s and 0’s) is coming from for the buffer.write. Am I supposed to put the file I am trying to copy in the MemoryStream? Thank you for your help.