Dowload File

microsoft.public.dotnet.framework

19 Sep 2006 2:00 PM

rony_16

Hi,
I have problem with downloading a file using WebRequest and
HttpWebResponse.
Inorder to download this file i need to call to url that tells to
prepare the file for download,
and then i call to another url that downloads the file.
I have a problem with that because:

1. i do not know when the server finished the preparetion.
2.i checked the HttpStatusCode but he doen't changes
(myresponse.StatusCode).

the problem causes the application to download only a part of the file
and that because the server hasn't prepared the file yet.
if i give him Thread.Sleep(100000) i do get the full file, but the file
can be 100Mb and 10sec won't help.

the code is:
//Preparing the file todownload...
//---------------------------------------------------------------------------------------------------------
   HttpWebRequest webRequest3 = WebRequest.Create(d1) as
HttpWebRequest;
   webRequest3.KeepAlive = true;
   webRequest3.Headers.Add("Keep-Alive", "300");
webRequest3.Headers.Add("Accept","text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
            webRequest3.Headers.Add("Accept-Language",
"en-us,en;q=0.5");
            webRequest3.Headers.Add("Accept-Encoding", "gzip,deflate");
            webRequest3.Headers.Add("Accept-Charset",
"ISO-8859-1,utf-8;q=0.7,*;q=0.7");

            webRequest3.CookieContainer = cookies;
            HttpWebResponse myResponse3 =
(HttpWebResponse)webRequest3.GetResponse();

              Thread.Sleep(10000);
            Stream ReceiveStream3 = myResponse3.GetResponseStream();

//-----------------------------------------------------------------------------------

//Downloading ....
//--------------------------------------------------------------------------------------
            Console.WriteLine("Connecting to download link...");
            HttpWebRequest webRequest2 = WebRequest.Create(re) as
HttpWebRequest;
            webRequest2.ContentType =
"application/x-www-form-urlencoded";
            webRequest2.KeepAlive = true;
            webRequest2.CookieContainer = cookies;
            HttpWebResponse myResponse2 =
(HttpWebResponse)webRequest2.GetResponse();
            Stream ReceiveStream2 = myResponse2.GetResponseStream();
            Console.WriteLine("Downloading...");

            Common.SaveStreamToFile(@"C:\scrapingtest\file.csv",
ReceiveStream2);
            Console.WriteLine("Done Downloading");

Thanks, Rony

19 Sep 2006 8:12 PM

Kevin Spencer

You're using the blocking "GetResponse" overload of the HttpWebRequest
method. This means that when the method call returns, the Response is there.
No need to Thread.Sleep.

--
HTH,

Kevin Spencer
Microsoft MVP
Digital Carpenter

A man, a plan, a canal,
a palindrome that has gone to s**t.

Show quote

"rony_16" <rony.vainb***@gmail.com> wrote in message
news:1158674411.285465.81510@h48g2000cwc.googlegroups.com...
> Hi,
> I have problem with downloading a file using WebRequest and
> HttpWebResponse.
> Inorder to download this file i need to call to url that tells to
> prepare the file for download,
> and then i call to another url that downloads the file.
> I have a problem with that because:
>
> 1. i do not know when the server finished the preparetion.
> 2.i checked the HttpStatusCode but he doen't changes
> (myresponse.StatusCode).
>
> the problem causes the application to download only a part of the file
> and that because the server hasn't prepared the file yet.
> if i give him Thread.Sleep(100000) i do get the full file, but the file
> can be 100Mb and 10sec won't help.
>
> the code is:
> //Preparing the file todownload...
> //---------------------------------------------------------------------------------------------------------
>   HttpWebRequest webRequest3 = WebRequest.Create(d1) as
> HttpWebRequest;
>   webRequest3.KeepAlive = true;
>   webRequest3.Headers.Add("Keep-Alive", "300");
> webRequest3.Headers.Add("Accept","text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
>            webRequest3.Headers.Add("Accept-Language",
> "en-us,en;q=0.5");
>            webRequest3.Headers.Add("Accept-Encoding", "gzip,deflate");
>            webRequest3.Headers.Add("Accept-Charset",
> "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
>
>            webRequest3.CookieContainer = cookies;
>            HttpWebResponse myResponse3 =
> (HttpWebResponse)webRequest3.GetResponse();
>
>              Thread.Sleep(10000);
>            Stream ReceiveStream3 = myResponse3.GetResponseStream();
>
> //-----------------------------------------------------------------------------------
>
> //Downloading ....
> //--------------------------------------------------------------------------------------
>            Console.WriteLine("Connecting to download link...");
>            HttpWebRequest webRequest2 = WebRequest.Create(re) as
> HttpWebRequest;
>            webRequest2.ContentType =
> "application/x-www-form-urlencoded";
>            webRequest2.KeepAlive = true;
>            webRequest2.CookieContainer = cookies;
>            HttpWebResponse myResponse2 =
> (HttpWebResponse)webRequest2.GetResponse();
>            Stream ReceiveStream2 = myResponse2.GetResponseStream();
>            Console.WriteLine("Downloading...");
>
>            Common.SaveStreamToFile(@"C:\scrapingtest\file.csv",
> ReceiveStream2);
>            Console.WriteLine("Done Downloading");
>
> Thanks, Rony
>

20 Sep 2006 6:23 AM

rony_16

I put the Thread.Sleep to check if i give a delay the file will be
full.
without it i get a file that is not fully downloaded.

How can i check if the link i was calling , fineshed prepering the the
file?
Kevin Spencer wrote:
Show quote

> You're using the blocking "GetResponse" overload of the HttpWebRequest
> method. This means that when the method call returns, the Response is there.
> No need to Thread.Sleep.
>
> --
> HTH,
>
> Kevin Spencer
> Microsoft MVP
> Digital Carpenter
>
> A man, a plan, a canal,
> a palindrome that has gone to s**t.
>
> "rony_16" <rony.vainb***@gmail.com> wrote in message
> news:1158674411.285465.81510@h48g2000cwc.googlegroups.com...
> > Hi,
> > I have problem with downloading a file using WebRequest and
> > HttpWebResponse.
> > Inorder to download this file i need to call to url that tells to
> > prepare the file for download,
> > and then i call to another url that downloads the file.
> > I have a problem with that because:
> >
> > 1. i do not know when the server finished the preparetion.
> > 2.i checked the HttpStatusCode but he doen't changes
> > (myresponse.StatusCode).
> >
> > the problem causes the application to download only a part of the file
> > and that because the server hasn't prepared the file yet.
> > if i give him Thread.Sleep(100000) i do get the full file, but the file
> > can be 100Mb and 10sec won't help.
> >
> > the code is:
> > //Preparing the file todownload...
> > //---------------------------------------------------------------------------------------------------------
> >   HttpWebRequest webRequest3 = WebRequest.Create(d1) as
> > HttpWebRequest;
> >   webRequest3.KeepAlive = true;
> >   webRequest3.Headers.Add("Keep-Alive", "300");
> > webRequest3.Headers.Add("Accept","text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
> >            webRequest3.Headers.Add("Accept-Language",
> > "en-us,en;q=0.5");
> >            webRequest3.Headers.Add("Accept-Encoding", "gzip,deflate");
> >            webRequest3.Headers.Add("Accept-Charset",
> > "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
> >
> >            webRequest3.CookieContainer = cookies;
> >            HttpWebResponse myResponse3 =
> > (HttpWebResponse)webRequest3.GetResponse();
> >
> >              Thread.Sleep(10000);
> >            Stream ReceiveStream3 = myResponse3.GetResponseStream();
> >
> > //-----------------------------------------------------------------------------------
> >
> > //Downloading ....
> > //--------------------------------------------------------------------------------------
> >            Console.WriteLine("Connecting to download link...");
> >            HttpWebRequest webRequest2 = WebRequest.Create(re) as
> > HttpWebRequest;
> >            webRequest2.ContentType =
> > "application/x-www-form-urlencoded";
> >            webRequest2.KeepAlive = true;
> >            webRequest2.CookieContainer = cookies;
> >            HttpWebResponse myResponse2 =
> > (HttpWebResponse)webRequest2.GetResponse();
> >            Stream ReceiveStream2 = myResponse2.GetResponseStream();
> >            Console.WriteLine("Downloading...");
> >
> >            Common.SaveStreamToFile(@"C:\scrapingtest\file.csv",
> > ReceiveStream2);
> >            Console.WriteLine("Done Downloading");
> >
> > Thanks, Rony
> >

22 Sep 2006 8:44 PM

Leon_Amirreza

Hi,
just want to share some a like problem (the file gets downloaded partially
and the downloader app fnishes downloading normally without any error, hint
.... ) downloading files from linux servers using windows tools (like IE,
DAP, ...) !?
I tried netscape on linux servers and never had the this problem?
It seems that there maybe some incompatibily or somthing with windows and
some linux file servers? (it maybe irrelevant, or i maybe wrong, ...)

just wanted to share the experience ;)

Show quote

"rony_16" <rony.vainb***@gmail.com> wrote in message
news:1158733411.846812.191600@i42g2000cwa.googlegroups.com...
>I put the Thread.Sleep to check if i give a delay the file will be
> full.
> without it i get a file that is not fully downloaded.
>
> How can i check if the link i was calling , fineshed prepering the the
> file?
> Kevin Spencer wrote:
>> You're using the blocking "GetResponse" overload of the HttpWebRequest
>> method. This means that when the method call returns, the Response is
>> there.
>> No need to Thread.Sleep.
>>
>> --
>> HTH,
>>
>> Kevin Spencer
>> Microsoft MVP
>> Digital Carpenter
>>
>> A man, a plan, a canal,
>> a palindrome that has gone to s**t.
>>
>> "rony_16" <rony.vainb***@gmail.com> wrote in message
>> news:1158674411.285465.81510@h48g2000cwc.googlegroups.com...
>> > Hi,
>> > I have problem with downloading a file using WebRequest and
>> > HttpWebResponse.
>> > Inorder to download this file i need to call to url that tells to
>> > prepare the file for download,
>> > and then i call to another url that downloads the file.
>> > I have a problem with that because:
>> >
>> > 1. i do not know when the server finished the preparetion.
>> > 2.i checked the HttpStatusCode but he doen't changes
>> > (myresponse.StatusCode).
>> >
>> > the problem causes the application to download only a part of the file
>> > and that because the server hasn't prepared the file yet.
>> > if i give him Thread.Sleep(100000) i do get the full file, but the file
>> > can be 100Mb and 10sec won't help.
>> >
>> > the code is:
>> > //Preparing the file todownload...
>> > //---------------------------------------------------------------------------------------------------------
>> >   HttpWebRequest webRequest3 = WebRequest.Create(d1) as
>> > HttpWebRequest;
>> >   webRequest3.KeepAlive = true;
>> >   webRequest3.Headers.Add("Keep-Alive", "300");
>> > webRequest3.Headers.Add("Accept","text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
>> >            webRequest3.Headers.Add("Accept-Language",
>> > "en-us,en;q=0.5");
>> >            webRequest3.Headers.Add("Accept-Encoding", "gzip,deflate");
>> >            webRequest3.Headers.Add("Accept-Charset",
>> > "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
>> >
>> >            webRequest3.CookieContainer = cookies;
>> >            HttpWebResponse myResponse3 =
>> > (HttpWebResponse)webRequest3.GetResponse();
>> >
>> >              Thread.Sleep(10000);
>> >            Stream ReceiveStream3 = myResponse3.GetResponseStream();
>> >
>> > //-----------------------------------------------------------------------------------
>> >
>> > //Downloading ....
>> > //--------------------------------------------------------------------------------------
>> >            Console.WriteLine("Connecting to download link...");
>> >            HttpWebRequest webRequest2 = WebRequest.Create(re) as
>> > HttpWebRequest;
>> >            webRequest2.ContentType =
>> > "application/x-www-form-urlencoded";
>> >            webRequest2.KeepAlive = true;
>> >            webRequest2.CookieContainer = cookies;
>> >            HttpWebResponse myResponse2 =
>> > (HttpWebResponse)webRequest2.GetResponse();
>> >            Stream ReceiveStream2 = myResponse2.GetResponseStream();
>> >            Console.WriteLine("Downloading...");
>> >
>> >            Common.SaveStreamToFile(@"C:\scrapingtest\file.csv",
>> > ReceiveStream2);
>> >            Console.WriteLine("Done Downloading");
>> >
>> > Thanks, Rony
>> >
>

22 Sep 2006 9:55 AM

Damien

rony_16 wrote:
Show quote

> Hi,
> I have problem with downloading a file using WebRequest and
> HttpWebResponse.
> Inorder to download this file i need to call to url that tells to
> prepare the file for download,
> and then i call to another url that downloads the file.
> I have a problem with that because:
>
> 1. i do not know when the server finished the preparetion.
> 2.i checked the HttpStatusCode but he doen't changes
> (myresponse.StatusCode).
>
> the problem causes the application to download only a part of the file
> and that because the server hasn't prepared the file yet.
> if i give him Thread.Sleep(100000) i do get the full file, but the file
> can be 100Mb and 10sec won't help.
>
> the code is:

>
> Thanks, Rony

Let me see if I can understand (and illustrate your problem to others).
You have a system where you perform a web request, say for instance to
http://site/prepareTheFile.aspx, and that produces a file (lets call it
TheFile.txt) which is accessible using a second url
http://site/PreparedFiles/TheFile.txt, and your problem is that you do
not know when the file is completely written (since prepareTheFile.aspx
continues to perform processing after it has returned to you).

If the above describes your situation approximately, then you're not
going to be able to fix this purely from the client end. Is the web
service one over which you have control? If not, then the best you can
do (if the file format is one in which you can tell whether or not it
is complete), is to repeatedly download the file and check whether it
is complete (I'd recommend starting at a wait of 1 second and then
doubling the amount of time you wait each time).

If you do have control over the server end, I can see two options.
Either make the prepareTheFile.aspx page wait until it's completed
production of the file before completing, or prepare the file in a
temporary location (on the same volume as it's eventual location), and
then Move it to the downloadable location once it's complete (in this
scenario, the client again has to poll for completion).

Damien