I'm writing a C# app which downloads large files via http from a server. These files are often in excess of a gigabyte.

I am trying to figure out the best way to download these kinds of large files in .NET. While I've successefully downloaded test
files using WebClient, I don't think that's going to be a viable final solution because I need to be able process the data stream
before writing it to the local disk (basically, I need to strip off certain bytes). I haven't found a WebClient method which allows
me to "intercept" the downloaded data stream to process it.

I've played around with WebRequest and WebResponse, but am running into a situation where no more than the first 256K bytes of data
is readable. In other words, if I do something like this (pseudo code):

WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();

while( not enough bytes read yet )
{
    stream.Read(buffer...)
    localFileStream.Write(buffer...)
}

only the first 256K bytes are read from the stream. After that, the stream.Read() has a return value of 0.

So...what advice do folks have on how I should move forward?

- Mark

One correction: the "256K byte limit" problem was self-inflicted (some old logic I forgot to remove).

But I'm still interested in advice on downloading large files via http.

- Mark

Hi Mark,

Thanks for posting!

I'm glad to hear the issue has been resolved:

In addition, there are many samples from third party demonstrate how to
approach this. I suggest you follow two articles below:
"Cool C# File Downloader":
http://www.codeproject.com/csharp/CoolDownloader.asp

"Threaded WebDownload class with Progress Call-backs":
http://www.codeproject.com/csharp/webdownload.asp

They are appropriated at the current stage. Thanks for your understanding!

Regards,

Yuan Ren [MSFT]
Microsoft Online Support

Thus wrote Mark,

When you use WebClient.OpenStream(), you can process the reponse stream directly.


But for GB sized download, I'd rather think about something more advanced
than plain HTTP, like BITS for example.

Cheers,

ProcessStartInfo.WorkingDirectory and Mapped Network Drives
Clarification about Microsoft.mshtml component within .NET 2 frame
is win32 programming dead?
Help on "GetProcessesByName(String processName)"
What TCP-algorithm does dotnet use ?