C# / 2005 / Async Socket Buffer Corruption

23 Nov 2006 8:32 AM

EmeraldShield

I have been fighting a problem for about two weeks and I can't seem to find
any info on it. Any help is greatly appreciated.

We have an ASYNC I/O system for email receptions (in bound email).
This service has been working very well for years. I upgraded it to Dot Net
2 / VS 2005 and moved over to the Async I/O model for better performance.
It works great.

BUT, every now and then I will get a corrupt buffer. The first 4 bytes of a
connection will be XXXX instead of HELO or EHLO (Even though they sent one
of those two commands). It is ALWAYS XXXX.

This is the call inside the AsyncCallback for the read operation:
mystring = Encoding.UTF8.GetString(sp.ReadInBuffer, 0, sp.ReadInBufferSize);

I cannot get it to repeat in a pattern that I could actually sit in a
debugger and watch it.

So I added a special case in my code if it finds XXXX I treat it like hello,
but that is a hack.

Very odd.

23 Nov 2006 10:18 AM

Charles Wang[MSFT]

Hi,
Thanks for using Microsoft Online Managed Newsgroup.

From your description, I understand that:
After your upgrading your application to Visual Studio Studio .Net 2005,
you encountered that sometimes you received XXXX instead of the command
"HELO" or "EHLO" in the first 4 bytes. Your application used AsyncCallback
for the read operation.
If I have misunderstood, please let me know.

For further research, I would like to know:
1. Did the server also use UTF-8 encoding?
2. How did you judge the end of the receive operation?
3. What is kind of your server application?

Sincerely yours,
Charles Wang
Microsoft Online Community Support

======================================================
When responding to posts, please "Reply to Group" via your newsreader
so that others may learn and benefit from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

23 Nov 2006 4:17 PM

David Browne

Show quote

"EmeraldShield" <emeraldshield@noemail.noemail> wrote in message
news:u$N#gntDHHA.4992@TK2MSFTNGP03.phx.gbl...
>I have been fighting a problem for about two weeks and I can't seem to find
>any info on it. Any help is greatly appreciated.
>
> We have an ASYNC I/O system for email receptions (in bound email).
> This service has been working very well for years. I upgraded it to Dot
> Net 2 / VS 2005 and moved over to the Async I/O model for better
> performance. It works great.
>
> BUT, every now and then I will get a corrupt buffer. The first 4 bytes of
> a connection will be XXXX instead of HELO or EHLO (Even though they sent
> one of those two commands). It is ALWAYS XXXX.
>
> This is the call inside the AsyncCallback for the read operation:
> mystring = Encoding.UTF8.GetString(sp.ReadInBuffer, 0,
> sp.ReadInBufferSize);
>
> I cannot get it to repeat in a pattern that I could actually sit in a
> debugger and watch it.
>
> So I added a special case in my code if it finds XXXX I treat it like
> hello, but that is a hack.
>

Hard to say without a working reproduction, or at least some code, but the
most common and insidious TCP/IP programming error is assuming a read
returns the number of bytes requested. The number of bytes received may be
less, and it can work 99.9999% of the time.

David

23 Nov 2006 8:14 PM

William Stacey [C# MVP]

Agree. I "smells" like a typical async overlapped bug where another call
using same buffer is overwriting buf before 1st call processes the buffer.
Are you sure your not posting another read before handling buffer first?

--
William Stacey [C# MVP]

Show quote

"David Browne" <davidbaxterbrowne no potted m***@hotmail.com> wrote in
message news:uU$qsrxDHHA.4060@TK2MSFTNGP03.phx.gbl...
|
|
| "EmeraldShield" <emeraldshield@noemail.noemail> wrote in message
| news:u$N#gntDHHA.4992@TK2MSFTNGP03.phx.gbl...
| >I have been fighting a problem for about two weeks and I can't seem to
find
| >any info on it. Any help is greatly appreciated.
| >
| > We have an ASYNC I/O system for email receptions (in bound email).
| > This service has been working very well for years. I upgraded it to Dot
| > Net 2 / VS 2005 and moved over to the Async I/O model for better
| > performance. It works great.
| >
| > BUT, every now and then I will get a corrupt buffer. The first 4 bytes
of
| > a connection will be XXXX instead of HELO or EHLO (Even though they sent
| > one of those two commands). It is ALWAYS XXXX.
| >
| > This is the call inside the AsyncCallback for the read operation:
| > mystring = Encoding.UTF8.GetString(sp.ReadInBuffer, 0,
| > sp.ReadInBufferSize);
| >
| > I cannot get it to repeat in a pattern that I could actually sit in a
| > debugger and watch it.
| >
| > So I added a special case in my code if it finds XXXX I treat it like
| > hello, but that is a hack.
| >
|
| Hard to say without a working reproduction, or at least some code, but the
| most common and insidious TCP/IP programming error is assuming a read
| returns the number of bytes requested. The number of bytes received may
be
| less, and it can work 99.9999% of the time.
|
| David
|

25 Nov 2006 4:33 PM

Mike Blake-Knox

In article <uU$qsrxDHHA.4***@TK2MSFTNGP03.phx.gbl>, David Browne wrote:

> Hard to say without a working reproduction, or at least some code, but the
> most common and insidious TCP/IP programming error is assuming a read
> returns the number of bytes requested. The number of bytes received may be
> less, and it can work 99.9999% of the time.

I wonder if the OP has tried capturing the data actually sent by using
something like Ethereal to confirm whether or not packets are being fragmented
in an undexpected way (and to capture test data). For diagnostic purposes, the
OP could use Ethereal with a circular buffer, modify the code that detects the
XXX and have it send some kind of a pattern back to the client. Ethereal could
use this pattern as a Stop Condition.

Hope this helps

Mike

24 Nov 2006 8:35 PM

Chris Mullins

The .Net Async socket stuff is very, very, very solid. We transfer... well,
wel transfer alot of data under very high load conditions with extreme
reliabiilty and don't see the errors your describing.

There are some potential issues that you may be running into:
- reusing the buffer from another thread by mistake.
- calling read again with the same buffer prior to parsing it
- calling write somewhere using that same buffer prior to parsing it

I see also that you're expecting your data to be UTF-8 encoded. If you get a
partial read of data, and that data happens to end with an incomplete UTF-8
sequence then your "GetString" call will thow an exception. Keep in mind
here that while a UTF-8 character is typically just 1 byte, it can be as
many as 4.

> So I added a special case in my code if it finds XXXX I treat it like
> hello, but that is a hack.

Seeing as what you've described is 95%+ a thread bug, I think this hack is
likley to lead to signifigant trouble down the line.

Can you post the relevant code snippets? The pieces I would need to see are:
- the buffer allocation
- the initial call into BeginRead
- the BeginRead callback function and it's call to EndRead
- the subsiquest call into BeginRead (usually inside the callback, but not
always)
- the set of BeginWrite/Callback/Endwrite method. Especially their buffer
allocations.

--
Chris Mullins, MCSD.NET, MCPD:Enterprise
http://www.coversant.net/blogs/cmullins

Show quote

"EmeraldShield" <emeraldshield@noemail.noemail> wrote in message
news:u$N%23gntDHHA.4992@TK2MSFTNGP03.phx.gbl...
>I have been fighting a problem for about two weeks and I can't seem to find
>any info on it. Any help is greatly appreciated.
>
> We have an ASYNC I/O system for email receptions (in bound email).
> This service has been working very well for years. I upgraded it to Dot
> Net 2 / VS 2005 and moved over to the Async I/O model for better
> performance. It works great.
>
> BUT, every now and then I will get a corrupt buffer. The first 4 bytes of
> a connection will be XXXX instead of HELO or EHLO (Even though they sent
> one of those two commands). It is ALWAYS XXXX.
>
> This is the call inside the AsyncCallback for the read operation:
> mystring = Encoding.UTF8.GetString(sp.ReadInBuffer, 0,
> sp.ReadInBufferSize);
>
> I cannot get it to repeat in a pattern that I could actually sit in a
> debugger and watch it.
>
> So I added a special case in my code if it finds XXXX I treat it like
> hello, but that is a hack.
>
> Very odd.
>
>

29 Nov 2006 9:20 AM

Charles Wang[MSFT]

Hi,

I am interested in this issue. Would you mind letting me know the result of
the suggestions? If you need further assistance, feel free to let me know.
I will be more than happy to be of assistance.

Have a great day!

Charles Wang
Microsoft Online Community Support