I have been working on a solution where we pull content using the
WebRequest/WebResponse request HTTP content and store it in a string.  For
some pages, I reliably get an IOException however the pages work perfectly
fine in web browsers and 3rd party utilities.

The following code isolates the problem:

String result = null;

WebResponse objResponse = null;

WebRequest objRequest = null;

objRequest =
System.Net.HttpWebRequest.Create("http://www.chemistry.org/portal/a/c/s/1/professionals.html");

objResponse = objRequest.GetResponse();

using (System.IO.StreamReader sr =

new System.IO.StreamReader(objResponse.GetResponseStream()))

{

result = sr.ReadToEnd();

}

Console.Write(result);

-------------------------------

Are there any recommendations for overcoming these issues?  I always get the
following exception...


System.IO.IOException was unhandled
  Message="Unable to read data from the transport connection: The connection
was closed."
  Source="System"
  StackTrace:
       at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32
size)
       at System.IO.StreamReader.ReadBuffer()
       at System.IO.StreamReader.ReadToEnd()
       at HTTPRequestTest.Program.Main(String[] args) in
C:\Projects\Test\HTTPRequestTest\HTTPRequestTest\Program.cs:line 24
       at System.AppDomain.nExecuteAssembly(Assembly assembly, String[]
args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence
assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()


Thanks

Rob

Hello Rob,

From your description, when you're using the HttpWebrequest component to
programmatically request some web pages, you get some IOException indicate
that can not read data. Also, those pages can be correctly requested
through standard webbrowser, correct?

As for the Exception, have you debugged into the code to see at which line
does the exception occur?  Since the error message said can not read data
...  I assume the exception occur after the " objRequest.GetResponse();"
call and raised at the "ReadToEnd" call, is this the case?  If so, it seems
the server-side has terminate the connection intermediately. 

Does this problem frequently occur when you use HttpWebRequest class to
request the pages? According to my research, there are several possible
issues for such error:

** the request's processing time has exceed the server-side application's
timeout setting and the server close the connection.

** The server-side terminate the current connection due to some reason(in
order to free the thread to process other task)

** the server instance encounter some exception and recycle the service
which cause the current connection be terminated.


Also, since you said that other browser and utility can correctly download
the page, I suggest you use some trace tools to capture the HTTP request
and response messages generated by both your application(that use
httpwebrequest) or webbrowser and compare them(mostly it is the http header
that may differ between the two cases).  You can consider some simple tools
like the "trace utility" in soap toolkit 3.0 or the tcptrace windows
version:

#SOAP Toolkit 3.0
http://www.microsoft.com/downloads/details.aspx?familyid=c943c0dd-ceec-4088-
9753-86f052ec8450&displaylang=en

#TcpTrace
http://www.pocketsoap.com/tcptrace/


In addition, here is a test program which can programmatically request the
page url you posted(I add some addiontal headers in the webrequest object).
You can refer to it and see whether it also works for you:

===================cs code=====================
class Program
    {
        static void Main(string[] args)
        {
            try
            {
                Run();
            }
            catch (Exception ex)
            {

                Console.WriteLine(ex.ToString());
            }
        }

        static void Run()
        {
            string url =
"http://www.chemistry.org/portal/a/c/s/1/professionals.html";

            string result = null;
            HttpWebRequest req = null;
            HttpWebResponse rep = null;



            req = WebRequest.Create(url) as HttpWebRequest;


            req.Method = "GET";
            req.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT
5.1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR
3.0.04506.30)";


            req.Accept = "image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, application/xaml+xml,
application/vnd.ms-xpsdocument, application/x-ms-xbap,
application/x-ms-application, */*";
            req.Headers[HttpRequestHeader.AcceptLanguage] =
"en-us,zh-CN;q=0.8,es-ES;q=0.5,fr-FR;q=0.3";




            rep = req.GetResponse() as HttpWebResponse;


            Console.WriteLine("status code: {0},contenttype:{1},
length:{2}", rep.StatusCode, rep.ContentType, rep.ContentLength);


            using (System.IO.StreamReader sr =

            new System.IO.StreamReader(rep.GetResponseStream()))
            {

                result = sr.ReadToEnd();

            }

            Console.Write(result);

        }
    }
============================

Hope this helps.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead



==================================================

Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.



Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.

==================================================



This posting is provided "AS IS" with no warranties, and confers no rights.

Hello Rob,

Have you got progress on this issue or does the suggestion and code in my
previous reply helps a little? If there is anything else we can help,
please feel free to post here.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


This posting is provided "AS IS" with no warranties, and confers no rights.

Thus wrote Rob,

Show quoteHide quoteRob, I've got no problem loading that page using a .NET based HTTP client.
You should try to include common HTTP headers such as Accept, Accept-Language,
and User-Agent in your request to make sure you don't confuse the web application
-- it may rely on these headers to work properly

Cheers,

Cyrillic characters in VS2005
VS@005 Extensions for .NET Framework 3.0
threads and GUI update
Exellent consultants available immediately with us
Strange server socket behaviour