See some strange results on Send and peer with a closed Receive.  Setup a
test harness.  Server accepts socket, then just does a Shutdown.Receive
right away and waits.   The client socket does both blocking Sends and
BeginSends to see the difference in behavior.

Results:
1) Using blocking Sends - The first send returns 10, which is the buffer
size.  Future Sends, return 0 and no exception.  This behavior seems not to
fly with the doco.  The send should block until 10 bytes are sent to kernel
mode or throw an exception, *not return 0.  Note the socket is in blocking
mode (default setting).

2) Using async BeginSend gives completly different results.  The first
EndSend returns 10 as above.  The second BeginSend returns a null
IAsyncResult and sets SocketError to "ConnectionReset".  I find this
behavior very strange.  BeginSend should never return a null IAsyncResult as
that FooBars the async pattern (it should always return IAsync or
exception).  It should save the exception for the EndSend call.  Second, why
does it set SocketError to ConnectionReset, when blocking Send does not seem
to care.  The two behaviors should be consistent.

3) Also, what is the point of SocketError being an out parm?  It should just
be included in the SocketException and thrown - no?

TIA

Should have included the test code:

            // Server.
            TcpListener l = new TcpListener(IPAddress.Any, 9001);
            l.Start();
            new Thread(
                delegate()
                {
                    using (NetPortNew srvClient = new
NetPortNew(l.AcceptSocket(), "srvClient"))
                    {
                        srvClient.Shutdown(SocketShutdown.Receive);
                        Thread.Sleep(2000);
                    }
                    l.Stop();
                }).Start();

            // Client.
            byte[] buf = new byte[10];
            using (Socket s = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp))
            {
                Console.WriteLine("Blocking mode:{0}", s.Blocking);
                s.Connect(IPAddress.Loopback, 9001);
                Thread.Sleep(2000);
                SocketError se = SocketError.Success;
                int read = 0;

                for (int i = 0; i < 3; i++)
                {
                    read = s.Send(buf, 0, buf.Length, SocketFlags.Partial,
out se);
                    Console.WriteLine("Bytes written to kernel:{0}", read);
                    Thread.Sleep(100);
                }
                // Note the different results with async send.
                IAsyncResult ar = s.BeginSend(buf, 0, buf.Length,
SocketFlags.None, out se, null, null);
                Console.WriteLine("BeginSend.");
                read = s.EndSend(ar);   // ar is null.
                Console.WriteLine("Bytes written to kernel:{0}", read);
            }

Show quoteHide quote

Hello, William!

WSM>             // Server.
WSM>             TcpListener l = new TcpListener(IPAddress.Any, 9001);
WSM>             l.Start();
WSM>             new Thread(
WSM>                 delegate()
WSM>                 {
WSM>                     using (NetPortNew srvClient = new
WSM> NetPortNew(l.AcceptSocket(), "srvClient"))
WSM>                     {
WSM>                         srvClient.Shutdown(SocketShutdown.Receive);
WSM>                         Thread.Sleep(2000);
WSM>                     }
WSM>                     l.Stop();
WSM>                 }).Start();

WSM>             // Client.
WSM>             byte[] buf = new byte[10];
WSM>             using (Socket s = new Socket(AddressFamily.InterNetwork,
WSM> SocketType.Stream, ProtocolType.Tcp))
WSM>             {
WSM>                 Console.WriteLine("Blocking mode:{0}", s.Blocking);
WSM>                 s.Connect(IPAddress.Loopback, 9001);
WSM>                 Thread.Sleep(2000);
WSM>                 SocketError se = SocketError.Success;
WSM>                 int read = 0;

WSM>                 for (int i = 0; i < 3; i++)
WSM>                 {
WSM>                 read = s.Send(buf, 0, buf.Length, SocketFlags.Partial,
WSM> out se);
WSM>                Console.WriteLine("Bytes written to kernel:{0}", read);
WSM>                     Thread.Sleep(100);
WSM>                 }
WSM>                 // Note the different results with async send.
WSM>                 IAsyncResult ar = s.BeginSend(buf, 0, buf.Length,
WSM> SocketFlags.None, out se, null, null);
WSM>                 Console.WriteLine("BeginSend.");
WSM>                 read = s.EndSend(ar);   // ar is null.
WSM>                 Console.WriteLine("Bytes written to kernel:{0}",
WSM> read);
WSM>             }

I have some comments on the code...

- after send returning 0 do you check returned SocketError? Is it SocketError.Success?
- why Send and BeginSend have different flags? ( Send has SocketFlags.Partial )

- Indeed, docs say nothing about BeginSend returning null. But, using Reflector, we can see
  such code, where "result1" is returned as IAsyncResult.
        if ((errorCode != SocketError.Success) && (errorCode != SocketError.IOPending)) {
            result1 = null;
      }
      else {
            result1.FinishPostingAsyncOp(ref this.Caches.SendClosureCache);
      }

- about those sent 10 bytes... Sender at first has no clue about what's going on the server.
Then it sends 10 bytes and with ack packet from the server it has to receive server's status ( via tcp flags ).
For e.g. if server closed connection then ConnectionReset can be set.  BeginSend seems to behave well (
first send - ok, second one - error ).  About Send(... ) do you check "se" returned?

   Yes, SocketError is ConnectionReset.

- why Send and BeginSend have different flags? ( Send has
SocketFlags.Partial )
   I was just trying stuff and did not change it back - result the same.
Here is the updated code, just put in in a button block to test.

   private void button6_Click(object sender, EventArgs e)
        {
            // Server.
            TcpListener l = new TcpListener(IPAddress.Any, 9001);
            l.Start();
            new Thread(
                delegate()
                {
                    using (NetPortNew srvClient = new
NetPortNew(l.AcceptSocket(), "srvClient"))
                    {
                        srvClient.Shutdown(SocketShutdown.Receive);
                        Thread.Sleep(2000);
                    }
                    l.Stop();
                }).Start();

            // Client.
            byte[] buf = new byte[10];
            using (Socket s = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp))
            {
                Console.WriteLine("Blocking mode:{0}", s.Blocking);
                s.Connect(IPAddress.Parse("192.168.1.1"), 9001);
                Thread.Sleep(2000);
                SocketError se = SocketError.Success;
                int read = 0;

                for (int i = 0; i < 3; i++)
                {
                    read = s.Send(buf, 0, buf.Length, SocketFlags.None, out
se);
                    Console.WriteLine("Bytes written to kernel:{0} SE:{1}",
read, se);
                    Thread.Sleep(100);
                }
                // Note the different results with async send.
                IAsyncResult ar = s.BeginSend(buf, 0, buf.Length,
SocketFlags.None, out se, null, null);
                Console.WriteLine("BeginSend.");
                read = s.EndSend(ar);   // ar is null.
                Console.WriteLine("Bytes written to kernel:{0}", read);
            }
        }

Show quoteHide quoteReflector, we can see
going on the server.
receive server's status ( via tcp flags ).
BeginSend seems to behave well (
Show quoteHide quote

Hello, William!

[skipped]

WSM> 3) Also, what is the point of SocketError being an out parm?

When you use method with SocketError, you do not receive an exception, instead you
have to check return value of SecketError param.
Actually you may receive exceptions, but it is unlikely to happen :8-).
Maybe this was done for perfomance reasons?

WSM>  It should just be included in the SocketException and thrown - no?

When getting socket exception there is ErrorCode that describes the error.
What the point of SocketError there?

Hi William,

Has nothing to do with blocking sends.  Reverse the order in which you call them, i.e. async before blocking, and you'll see the
same results.  Also, the docs say that when the send method succeeds it is not an indication that the data was actually received.
About Send returning zero, see #3.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcecomm5/html/wce50lrfsetsockoptWindowsSockets.asp

I agree about the null IAsyncResult being poorly designed, but you can simply check that it's not null before calling EndSend.  A
null IAsyncResult indicates that something went wrong and that you should check the SocketError.  Also, they are consistent.  In my
testing both async and blocking Send use the ConnectionReset constant to indicate the same failure.

Both blocking Send and Async Send will return SocketError.ConnectionReset after the connection has been reset.  In other words, it
doesn't matter which of the Sends, async or blocking, is executed first.  Since the server has shutdown receiving on that socket the
first call to Send (async or blocking) will reset the connection.  Subsequent Sends will fail, as you've witnessed.  As the docs
stated, the first succeeding call to Send (either async or blocking) does not indicate that the server actually received the data.
For this reason you should design software that acknowledges transmission on a higher level, such as in a custom communications
protocol.

You have the choice of not using a SocketError argument by using an overload of Send that doesn't accept the out parameter.  A
SocketException will be thrown for these overloads instead.  Use the SocketException.SocketErrorCode property to retrieve the
SocketError from the exception.


The bottom line is that you must perform some operation on the Socket so that it can determine the state of the connection.   The
state is not dynamic so that an action on one end point automatically affects the other.  By shutting down Receive on the server end
point you are basically setting up a wall.  Only by attempting to Send data will the socket update its internal state so that
subsequent calls will fail.  It would be nice, however, if the first send failed as well but I see that from the docs this "design"
was by choice.

Here's your code revised in a manner that will hopefully illustrate what I've written above:

private static readonly EventWaitHandle waitForAsyncSend = new EventWaitHandle(false, EventResetMode.AutoReset);

private static void SocketTest()
{
// Server.
TcpListener l = new TcpListener(IPAddress.Any, 9001);
l.Start();
new Thread(
   delegate()
   {
    using (Socket socket = l.AcceptSocket())
    {
     socket.Shutdown(SocketShutdown.Receive);

     WriteLine("Server shutdown receive.");

     waitForAsyncSend.WaitOne();

     // expecting 4 blocks of 10 bytes each
     WriteLine("Server about to poll for data");

     // examine first batch
     if (socket.Poll(8000000, SelectMode.SelectRead))
     {
      byte[] buffer = new byte[10];

      try
      {
       int read = socket.Receive(buffer);

       WriteLine("Server read bytes: " + read);
      }
      catch (SocketException ex)
      {
       if (ex.ErrorCode == 10053)
       {
        WriteLine("Server read error: " + ex.SocketErrorCode.ToString());
       }
       else
        throw ex;
      }
     }

     WriteLine("Closing client connection");
    }

    WriteLine("Server stopping");
    l.Stop();
   }).Start();

// Client.
byte[] buf = new byte[10];

using (Socket s = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp))
{
  WriteLine("Blocking mode:{0}", s.Blocking);
  s.Connect(IPAddress.Loopback, 9001);
  Thread.Sleep(2000);
  SocketError se = SocketError.Success;
  int read = 0;

  // Note the different results with async send.
  IAsyncResult ar = s.BeginSend(buf, 0, buf.Length, SocketFlags.None, out se, null, null);
  WriteLine("Non-blocking SocketError: " + se.ToString());

  if (ar != null)
   read = s.EndSend(ar);   // ar is null.

  WriteLine("Non-blocking bytes written to kernel:{0}", read);

  waitForAsyncSend.Set();

  Thread.Sleep(2000);

  for (int i = 0; i < 3; i++)
  {
   read = s.Send(buf, 0, buf.Length, SocketFlags.None, out se);
   WriteLine("Blocking bytes written to kernel:{0}\r\nSocketError:{1}", read, se);
   Thread.Sleep(500);
  }

  Console.WriteLine("Click 'Enter' to exit");
  Console.ReadLine();
}
}

private static readonly object sync = new object();

private static void WriteLine(string message)
{
lock (sync)
{
  Console.WriteLine(message);
  Console.WriteLine();
}
}

private static void WriteLine(string format, params object[] args)
{
lock (sync)
{
  Console.WriteLine(format, args);
  Console.WriteLine();
}
}

I see what they did with the two version of send now.  Send with the "out"
error overload should probably be changed to "bool TrySend(...)" to match
with current standards.  The null IAsyncResult thing still bugs me.   The
other thing we discover here via imperitive evidence is that
shutdown.receive will never be sent by server unless it can piggyback on an
ACK reply or an outgoing message.  Something I must remember.  Cheers.

Show quoteHide quotecall them, i.e. async before blocking, and you'll see the
not an indication that the data was actually received.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcecomm5/html/wce50lrfsetsockoptWindowsSockets.asp
simply check that it's not null before calling EndSend.  A
check the SocketError.  Also, they are consistent.  In my
indicate the same failure.
after the connection has been reset.  In other words, it
Since the server has shutdown receiving on that socket the
Subsequent Sends will fail, as you've witnessed.  As the docs
not indicate that the server actually received the data.
on a higher level, such as in a custom communications
overload of Send that doesn't accept the out parameter.  A
SocketException.SocketErrorCode property to retrieve the
that it can determine the state of the connection.   The
affects the other.  By shutting down Receive on the server end
data will the socket update its internal state so that
failed as well but I see that from the docs this "design"
I've written above:
EventWaitHandle(false, EventResetMode.AutoReset);
Show quoteHide quotese, null, null);
Show quoteHide quote

Hi William,

Agreed.

I haven't tried using an outgoing message.  Did you test that?

I guess the bit is not set in a send either - only the ACK.  So I guess that
means you must always do at least two sends (at the client) before you will
notice a shutdown receive at server.

Show quoteHide quote

Hello, William!

WSM> I guess the bit is not set in a send either - only the ACK.  So I
WSM> guess that means you must always do at least two sends (at the client)
WSM> before you will notice a shutdown receive at server.

There is an interesting API in the winsock - WSASendDisconnect/WSARecvDisconnect.
The result is virtually the same as with shutdown, but results in one call...

Interesting.  Is there a managed counterpart?

Show quoteHide quote

Hello, Dave!

DS> Interesting.  Is there a managed counterpart?

AFAIK there is no managed equivalent.

Show quote Hide quote What "bit"?  There is *no* flag for Receive_Closed!  There are two
relevant flags: the FIN flag which means Finished_Sending, and RST,
which means Reset: in general means 'I've no such connection, please
delete that connection from your list'.  The flags SYN, ACK, PSH, and
URG have no part to play here.

So when the server does "socket.Shutdown(SocketShutdown.Receive);", it
can't do anything based on that.  It just sits dumb waiting for another
action.  Then eventually a packet comes from the peer, and that will
contain data, so the server responds RST: 'I can't handle that'.  This
is I presume what Dave meant when he said "you are basically setting up
a wall": no signal, just that any data received will crash into the
wall.  However he wasn't quite correct in saying 'It would be nice,
however, if the first send failed as well but I see that from the docs
this "design" was by choice.'  The software writers have _no_ choice in
this, the TCP protocol doesn't allow such a behaviour.


So, remembering that the server has sent nothing after the local
application did SocketShutdown.Receive.  Lets look now at what happens
from the aspect of the client side.  Remember that TCP makes no
guarantee of maintaining send boundaries etc.  So the client could
buffer up multiple send before deciding to send a packet, that packet
then works its way across the network, meanwhile the client application
could be doing more sends, which are likely buffered up, and maybe sent
at some point, and again the client application could be doing more
sends, and even blocking or non-blocking :-,) on a receive.

And at some point the first send now reaches the server, who responds
with a packet with the RST flag set.  That then makes its way back
across the network.  Again the client application can be doing sends,
and maybe some packets are sent too.  Then the RST reaches the client
device, and works its way up through the stack.  Boom!  Now and only now
is ConnectionReset returned on every subsequent send or receive that the
client application makes.


So two things about TCP, firstly SocketShutdown.Receive cannot be
signalled.  That's a very rare and odd case that there's no need to
cover it: 'I've more to send, but I don't want to hear any error
responses from you'!  Odd, whereas the opposite is common: 'I've now
finished sending all I have, but I'm still listening for any feedback
you wish to send me.'

Secondly, that *errors* on the connection can only be signalled on later
send/receive calls.  Errors are only apparent after buffered data has
been sent and a error response has been received from the server across
the network, or if the network is down for instance that 'I've had to
retry sending data five (say) times and I've now given up'.


Note that SocketShutdown.Send, is signalled with flag FIN, which is
'data' so it is sent in a packet with or after the last byte in the send
buffer has been sent.  To get an error earlier some protocols send
regular packets to say 'are you alive?' (for instance HDLC, IrDA, SNA,
SPX (?my memory is fading)).  But due to the philosophy of TCP, that, to
try and ride over network failure do not send any packets unless there's
data to be sent, no such packets are sent.

I recommend that you put your server and client on different machines
and look at the traffic between then with a network analyser.  What's
happening below the APIs can then be seen.

Hi Alan,

That is what I meant.  I was just speaking from experience, so I didn't want to make any technical assumptions using terminology
that I didn't fully understand.

I understand that this limitation is indeed in the TCP protocol itself.  What I don't understand is why.  I still think it would be
nice if the first send would fail since the RST flag is in the stack anyway.  If you look at William's example with my revisions,
you'll see that each Send, including the call to BeginSend, is executed sequentially.  If RST is at the top of the stack before the
next call to Send then why can't the first call to Send (or BeginSend) just fail instead of returning to the caller?  And what kind
of "connected" protocol is TCP anyway if it acts "disconnected" in so many ways?  And if shutting down Send behaves in some manner
shouldn't shutting down Receive behave in a similar manner if just to preserve consistency in the protocol?

You've posted some interesting stuff.  I'm going to have to read about the protocol itself now to fully appreciate it.  Thanks a
lot!

I'm thinking about starting here:

TCP on Wikipedia.org
http://en.wikipedia.org/wiki/Transmission_Control_Protocol

Any suggestions on other reading material for this weekend?

Show quoteHide quote

What I don't understand is why.  I still think it would be
anyway.  If you look at William's example with my revisions,
sequentially.  If RST is at the top of the stack before the
just fail instead of returning to the caller?

AFAICT, RST is not "seen" by the client until the first ACK to client *after
a Shutdown.Receive by the server.  So I assume you may not even see the RST
until the 3rd send depending on speed, etc.  Thing I don't understand yet is
why RST is not set in the header of outgoing data if the server does a send
after Receive is closed?  Or why the server even lets you do a send after a
Shutdown.Receive if ultimately it forces both sides of the client down
anyway?

Note: I have no compelling reason I want to do this, other then experiments
and noting behavior and exceptions.  However it would seem like a use for
closing receive on server would be a stock ticker app.  The client sends a
single request message to server.  The server verifies and closes Receive -
as if to say "I do not expect any more "data" messages (i.e. not control
packets) from client and will be error.".  And then server just streams data
to client one-way and then closes send after requested number of stocks or
some N factor of stocks or time and closes socket.  The client reads until
0, then just closes socket.  Naturally, it is easy to refactor this to do
normal shutdown behavior.

--wjs

Hi William,

[To summarize:  we are discussing the mechanisms used by the TCP protocol when an end point shuts down receiving on the socket and
how that action affects its counterpart with regards to immediate changes in state and future changes in state]

Your explanation seems to be accurate with respect to my test results and previous experience with sockets, however Alan's
explanation led me to believe that a different mechanism was in place.  I am referring specifically to the following statement of
Alan's:

In your test case there was no concurrent sending, so I assumed that on a blocking send with no concurrent activity that the RST
flag would "make its way back across the network" immediately from Alan's statements.

Interesting that the RST made it back into the client stack but ConnectionReset is returned only on every subsequent request.  My
previous statements were referring to this particular statement of Alan's.  Why not just fail the initial send if the RST flag has
already been received, if in fact it is received?

Show quoteHide quoteI can see how that might be useful to reduce the amount of network traffic since the client doesn't expect any responses anyway.
Unfortunately, the server has to send something at least once to find out that it shouldn't send anything again.  Therefore, if you
design a protocol on top of TCP as such, your choices are then limited to either "all clients accept data" or "all clients do not
accept data" unless an error on the server is acceptable when trying to respond with data.

I was thinking that a forward-only connection could be useful to circumvent potential errors on a network topology that prevents
responses from being received by the counterpart.  By shutting down the receive functionality the end point is saying, "Don't even
try to send any data 'cause I know that it won't get to me".  Although I guess this won't work in TCP since the counterpart has to
Send data anyway in order to find out that little bit of information.  And the sender won't know whether the first Send actually
succeeded or was just sucked up by the RST mechanism without a higher-level protocol in place to respond with an acknowledgement.

Does anybody know what the intended purpose was of the shutdown-receive flag when it was designed?  i.e.,  What usage scenarios did
the designers have in mind?

I guess the usefulness of my participation in this discussion is moot without having read a substantial amount of the technical
specification for TCP.  I'm just trying to understand what I've already experienced by means of a more technical explanation.   If
I'm not making any sense please let me know.

Thanks for the link in your other response - I'll take a look.

previous experience with sockets, however Alan's
am referring specifically to the following statement of
blocking send with no concurrent activity that the RST
I think we are still talking about the same thing.  RST makes it way back to
the client on an ACK.  However, you may not see that ACK until after 1 or 2
sends.  I "think" tcp can send two packets without ACK, but then must wait
(correct me here if wrong) for ACK before sending more.  But eventually you
will get the ACK with the RST set and the next send/receive will fail.

ConnectionReset is returned only on every subsequent request.

I did not follow you here.

Why not just fail the initial send if the RST flag has
It will fail if it was received.  But in our case it not (and can not) be
received until after the first send, because it is set in the ACK of that
first send.  But I may already have did another Send before first ACK is
processed, so I may see it on second or third send.  If we always do the
"proper" shutdown both sides, then none of this matters and all should work
fine:
  1.. Finish sending data.
  2.. Call shutdown() with the how parameter set to 1.
  3.. Loop on recv() until it returns 0.
  4.. Call closesocket().
Cheers.
--wjs

Hi William,

Here are a few questions that I still cannot answer:

1. If shutting down receive on a socket blocks incoming data, does the socket respond with anything when data is actually sent from
the peer?
2. What actually is sent;  ACK or RST, ACK and RST, nothing, or something else?
3. Is the response immediate?
4. Does a blocking Send wait for a response before returning to the caller?
5. If Send does wait for a response, why doesn't the initial call to Send fail when RST is received (this question is derived from
how I've understood Alan's explanation thus far)?
6. Does BeginSend wait for a response before returning to the caller (I hope not)?  Does EndSend wait for a response?

I responded to your OP because I knew how that behavior could be explained in general terms, through simple experimentation.  Now
that we gone down a more technical road I'm intrigued, yet confused.

That's where I'm confused.  Alan's explanation does not mention ACK, except when he stated the following early on in his response:

<snip>
Yours and Alan's sound like competing explanations to me.  I'm trying to understand how this works, techincally speaking, but I'm
confused as to which explanation is correct.  Or are they both correct, but only partial answers?

I assumed that the peer was aware of the reset immediately after the first send, as per Alan's comments above.  If that is true, I'm
not sure why the first Send doesn't just fail with ConnectionReset and not just subsequent Sends.

I guess I'll have to do some reading on my own.  If ever I can explain this phenomena I'll post my findings.

Understood, but I don't see how that applies to the OP or this thread.  I thought we were trying to address problems that have been
identified with the unorthodox, but possible, scenario for using sockets whereby a peer attempts to shutdown receiving without a
higher-level protocol to notify its counterpart of such an occurrence.  A standardized shutdown sequence seems a bit off-topic to me
and does not address the problem at hand.

Sorry if I'm nitpicking ;)

Dave Sexton wrote:
Not when data is sent by peer, but when data from peer is received. Once
that RST segment reaches poor sender, all subsequent operation (both
read and send), should fail with "Connection reset".

server: Shutdown.Recieve
client: send ok
client: send ok
server: receives data -> send back RST immediatly
client: send ok
client: send ok
client: TCP stack receives RST
client: send failes with "Connection reset"
client: send failes with "Connection reset"
client: send failes with "Connection reset"
....

RST.

Sort of. It should RST when data is received that cannot be delivered to
  application. It may take a while to reach the other side and also that
RST packet could be lost.

Sort of. If OS has enaugh buffer space it will buffer and return
immediatly. If not, it will wait for ACK from other side that will free
some buffer space.

See 4.

BeginSend should not. EndSend I'm not sure...

....

Regards,
Goran

Hi Goran,

Thanks very much for your response.  As it turns out, I still have a lot of questions even though you've answered a few (but you've
created some more :).

I still have yet to answer the question, "Why doesn't the initial blocking Send fail?".  I believe now that there are many forces at
work here affecting the answer to my question and that the behavior of some of the socket functions are simply indeterministic due
to network latency, including the blocking Send itself.  What, exactly, does a blocking send do that requires it to block?

I'm gonna have to hit the books.

Something else that might be of interest to readers, which I found browsing around for more info on the subject of TCP/IP:

Nagle Algorithm on Wikipedia.org:
http://en.wikipedia.org/wiki/Nagel_algorithm

Show quoteHide quote

Eeeeh.  I hoped to clear things up in my posting; looks like I didn't
quite succeed. :-)


(Just to note firstly, doing Shutdown(Receive) alone/initially is very
very very rare, Close/Shutdown(Both), or Shutdown(Send) later followed
by Shutdown(Receive) is very much more common.  And if a particular
application/session layer protocol implied that one peer should do
Shutdown(Receive) then it would not be valid for it to then have the
other peer send data!)

Anyway, if the application on one device does Shutdown(Receive) then if
a packet containing data is received from the peer on that connection,
then that is an not a valid packet and a packet with the RST bit set is
sent clearing down the connection.

Firstly, just to be absolutely clear, there is no such thing as an ACK
packet, or a RST packet, or a SYN packet, etc.  In TCP there is simply
*one* type of packet, this is unlike HDLC (and its children), which
carries data in 'I' frames, has ACK frames which it calls 'RR' (Receiver
Ready), and lots more. :-(  Unfortunately its also often those such
terms that are used by our teachers...

Anyway, TCP's header is always the same format.  It contains some
numerical fields e.g one for the sequence number, has a set of flags:
URG, ACK, PSH, RST, SYN and FIN, and finally it optionally carries some
data.  So (to be very correct) we can have "a packet with the ACK bit
set", but no such thing as a "ACK packet".  Now often in reality we
generally call a packet with no data and the ACK bit set as an "ACK
packet" but it can confuse...

So in the case above, a packet with the RST bit will be sent.  I'm not
sure whether the ACK bit will be set, I'd have to go and read the
specification to be sure, and I'm not sure that it matters
particularly...

Err to what? :-)

The server 'immediately' sends a RST when it gets any packet that is not
valid.  And receiving a segment containing data is not valid where the
local application has done shutdown(receive)).  Of course there is time
for both of those packets to cross the network.

And as we note below, a send is _not_ immediate on the application
calling send (or it returning etc)...

No.  It just adds data to the buffer and returns.  (Ignoring here what
happens if the buffer becomes full...)  The TCP protocol layer then
decides when to take a segment's worth of data and send it in a packet.
In general we should consider the two as independent.

It doesn't.  This might be where the confusion lies. :-(

I got lost following the messages in the text below, let me know what if
anything that doesn't cover. :-)

Alan


Show quoteHide quote

Hi Alan,

Excellent response.  See inline:

Lol.  It's hard to explain all of TCP in a single post.

Fair enough.  I do understand the rarity of the operation under scrutiny.  I'd like a complete understing of TCP and the ability to
shutdown Receive only is part of the protocol (and, not to mention, the topic of this thread :)

Understood (from Goran's post as well).

I don't know anything about HDLC, and being an autodidactic anything that I don't undestand about TCP up until now is my own fault
Thanks for clearing that up.  Also, the wikipedia.org article (link in a previous thread of mine) contains a nice little chart of
the TCP header and the meaning of each section of bytes.

Understood now.  RST is sent;  ACK doesn't matter since it would be in the same header anyway.  What's important is the RST (and ACK
probably shouldn't be set, for that matter)

To the peer that sent the data!  I was leading in to my next question about whether Send blocked for that response because I assumed
the answer to this question was "yes"...

Understood now (from your response below as well).  There is an inherant asynchronicity in TCP due to network latency and,
therefore, RST is sent immediately but might not be received by the peer before another Send, even if the peer only calls Send in a
synchronous manner.

This leads me to believe that our test code in this thread does not accurately represent the behavior of a socket in all possible
circumstances.  The console output shows that the second Send will always raise an error since it received the RST, however, that
might not be the case in a real world application.  (Goran tried explaining this in his illustration but he failed to mention
whether his model was synchronous or asynchronous, and without this new knowledge of mine that Send does not block for a response, I
assumed he was talking about an asynchronous model only.)

Isn't it the Nagle algorithm, not the TCP protocol, that determines what a "segment" actually is and when it should be sent?

So it seems that Send blocks when it has to load data into an overflowing buffer [Also: Peter Duniho's response to William's
question in a branch of this thread] and not for a response from its peer.  This reallly confused me since I always assumed Send
waited for a response and BeginSend didn't.

This makes me change my outlook on TCP in general.  I always thought TCP provided "Control' over the flow of information on its own
connection, but now I think that "Control" in "Transmission Control Protocol" is referring to the fact that TCP is used commonly to
control the flow of information on a higher-level protocol that involves other connections, such as with a video game that uses UDP
to transfer data, or as with an FTP data connnection that is independant of the "Control" connection.  Is that correct or does
"Control" mean something else that I'm unaware of, such as an improvement on an ancestor protocol perhaps?

Yep, it was one of the sources of confusion.  It's not anymore.  This thread contains a few answers to that question, but none have
clearly defined whether or not a blocking Send blocks for the RST, which I needed to know first before I could undertstand anything
else.  Thank you for clearing that up.  I guess that those who already understand TCP on the protocol level felt it was too obvious
of an answer to mention ;)

Understood now from your answer to #5 as well.

You've answered all of the questions that I wrote so far.  Sorry if I don't explain myself thoroughly enough.  It's hard to ask
questions without fully understanding the terminology.  Thanks for your help.

So it seems that the behavior I witnessed when using TCP/IP Sockets, with regards to the example in the OP, was a side effect of the
actual implementation and not a valid indication of the real behavior of TCP/IP.  I can't say that I've used this mechanism in
production code but I have played around with it before and thought that I had the behavior pinned.

Again, I'm planning on reading more about TCP.  RTFM is an acceptible response to any of my inquiries, but I do appreciate the help

same header anyway.  What's important is the RST (and ACK
Back to the beginning I think..  RST is set, not sent independently AFAICT.
There can be no RST set unless it is set in an ACK packet of the client's
first send (in this case).   If the client never sends anything after server
did close, client would never see the RST - which now makes sense.
Example:

1) Server closes Receive.
2) Client does a send.  Does not block on either Send or BeginSend.
   2a) Client may do another send as RST has not been "seen" yet by the
client.
4) Client gets the ACK of first Send which has the RST set in the header.
5) Bang. Future Sends/Receive at client will error because server's RST has
been processed.

--wjs

Hi William,

Nice example.  It includes all of the points I was trying to understand.  This shows how the example in the OP does not acurrately
illustrate the behavior of this mechanism since it used synchronous Sends only, making it seem like the second Send will always
fail, when in fact it might very well be the thousandth send that will fail, given enough time.

Show quoteHide quote

This shows how the example in the OP does not acurrately
only, making it seem like the second Send will always
fail, given enough time.

TCP will only allow so many outstanding ACKs before it will not send
anymore - so we should not be able to send many before we get push-back.  I
thought I read somewhere that was 2 outstanding packets, but could be wrong.
So I guess that means Send could also block because of this (i.e. waiting to
post a buffer because the stack is waiting for outstanding ACK) until send
timeout.
--wjs

Hi William,

So "thousandth" is completely inaccurate :)

Are you suggesting that Send will block for ACK in some cases?  I have been led to believe that Send does not block for ACK in any
circumstance.

Show quoteHide quote

been led to believe that Send does not block for ACK in any
Ultimately, the send buffer will do the the push-back.  If TCP is waiting
for a pending ACK, it is not sending.  If it is not sending, the send buffer
is not poped and fills up.  Send will block (in non-blocking socket mode) if
it can not write the user buffer - so yes, it seems send could block for
upto sendtimeout.

Setting SendBufferSize to 0 seems to also make send block until write
complete (and it seems like I see ACK with RST right away also).

--wjs

Hi William,

Thanks, that clears it up a bit more.  I also see that you've mentioned SendBufferSize, which answers a related question I posed in
another branch of this thread.  I'm going to take a look at your response to that post now.

Show quoteHide quote

Also.  I am wondering about the usefullness of BeginSend?   Send actually
does not block, it just puts bytes on the kernel buffer(s) and returns - the
stack then takes it from there.  It seems as if BeginSend would actually be
more overhead because of the callback, context switch, and IAsync, etc.  So
is BeginSend actually usefull?  If so, why?  Does anyone have any perf data?
tia.

Assuming .NET Send is similar to Winsock send()/WSASend, it *can* block.
The kernel buffer is only so large, and if one sends a buffer larger than
can be buffered by the network driver, the call to send will block until all
of the data has been buffered (which for large sends implies that at least
some of the data has already been actually sent, though there's no way for
the sending application to know this).

It seems to me that if performance is an issue, then BeginSend is likely
just fine.  When performance is an issue, one is generally dealing with
large amounts of data, or large numbers of clients, or both.  A large amount
of data implies that even a regular Send is likely to block (and result in a
context switch).  A large number of clients implies that non-blocking
sockets are desirable as is not blocking the primary thread just to handle
one client.

I was very interested to learn, in some of the recent .NET socket threads,
that the Begin/End async versions of the Winsock calls use IOCP.  IOCP
addresses the above issues gracefully and with good performance.

There may well be more overhead with BeginSend, and it may well be that
overhead is unwarranted when dealing with short transmissions to few (or
just one) connections.  But in those cases, one is unlikely to notice the
overhead anyway (just as the overhead of managed code is unlikely to be
noticed in many situations).  And one very nice thing is that one can take
advantage of IOCP using sockets without having to actually write all the
relatively complex code that is normally required to support IOCP using
plain Win32 Winsock.

Pete

Fair enouph.  But where does "pinning" buffers come into play?  I thought
when you sent your buffer or buffers (with ArraySegment overload) *those
buffers get pinned and the driver uses them directly instead of making a
costly copy of user buffers to driver buffers.
I mean if it did just use the user buffers, then there is no buffers to
copy, just something to queue up.  Any light on this?

Show quoteHide quote

I don't know.  That's a .NET thing while my knowledge comes from experience
with Winsock itself.  I'm new to the whole .NET framework stuff.

That said, again borrowing from the underlying Winsock behavior...it is
common procedure when using IOCP to set the underlying network buffers to 0
length to force Winsock to send and receive from and to your own buffers,
avoiding an extra copy of the data.  So perhaps .NET is doing something
similar when one uses the Begin/End paradigm.

For a normal blocking Send, however, I would expect that *some* buffering
does occur.  I can't say this for sure, but an actual network send can take
a fairly long time and it would be kind of rude for .NET to cause your
thread to block until all of the data has been sent and acknowledged, at
least when the size of the sent data is small (an application shouldn't be
making large blocking sends unless it's prepared to sit and wait awhile).

For all I know, even in the buffered case, .NET has to pin your buffer to
ensure that it doesn't move while the underlying Winsock send occurs.  I
have no idea how .NET handles data or threading.  If it's got some kind of
maintenance thread (like, where does the garbage collector run?) that could
move memory around even while the thread using it is blocked on a system
API, I guess .NET would have to pin the buffer before calling the system
API.

Hopefully, these questions are academic and the .NET programmer using the
..NET sockets API doesn't need to worry too much about them.  It should be
safe to assume that when .NET tells you a socket operation has completed,
that the buffer is yours again.  For a blocking Send call, "completed"
should be as soon as control has been returned back to the calling program.
For a non-blocking method, such as BeginSend, I would expect to not be
permitted to touch the buffer until notification that the operation has
completed (either in the callback, or by blocking on the EndSend
method...the documentation isn't completely clear on this IMHO).

Pete

Hello, William!

WSM> Fair enouph.  But where does "pinning" buffers come into play?  I
WSM> thought when you sent your buffer or buffers (with ArraySegment
WSM> overload) *those buffers get pinned and the driver uses them directly
WSM> instead of making a costly copy of user buffers to driver buffers.
WSM> I mean if it did just use the user buffers, then there is no buffers
WSM> to copy, just something to queue up.  Any light on this?

"Pinning" also occurs when you're using Socket.Send(byte[] buffer .... ).
So, the Send process comes like this:
- Socket.Send(byte[] buffer) - buffer is pinned and marshaled to native send(....)
- Native send then passsed that buffer down to undelying WSPSend ( LSP related stuff )
- Finnaly buffer gets copied into kernel mode buffer, if kernel buffer size is less then buffer
specified by the user, then WSPSend is blocked, thus blocking all the upward call chain.

Hi Vadym,

So, I have to ask...

1. How often will a blocking Send block, and for how long?
2. I understand this depends on the size of the buffer, so how big is the kernel buffer?
3. Is the size of the buffer affected by the Nagle algorithm in any way?
4. Does the size of the buffer fluxuate, or can it be changed programmatically?
5. If a blocking Send isn't waiting for a response from the server why not just write the buffer directly into unmanaged memory (or
pin a copy) and return immediately to the caller? i.e., why block at all?  I take it that this is what BeginSend does?
6. The example in the OP attempts to send 10 bytes a few times, synchronously, and it seems that the second Send always failed after
RST in my testing.  Will increasing or decreasing the number of bytes sent in the first Send cause this behavior to change?  In
other words, if the first send no longer blocks (if it currently is blocking, depending on the size of the buffer and the number of
bytes sent), is it possible that the second Send will not always fail because the time it has taken to normally Send has decreased
even if the time it takes to receive the RST has remained the same?

I only ask the last question because it seems to me that this behavior is really unpredictable and that no real example can be
written that will function identically on each individual computer.  In other words, it's impossible to understand this behavior
only through testing.

(I just realized that the example in the OP blocked the thread for 100 milliseconds after each iteration.  In that case my question
(6) is still valid, but please ignore the context in which it was asked.)

RTFM is acceptible ;)  Just, where is the manual exactly?  I'll check out TCP I as William recommended, but if there is a genuine
manual that describes the protocol on the web somewhere I'd like to know.

Show quoteHide quote

    socket.SendTimeout (also paired ReceiveTimeout)

    socket.SendBufferSize (and ReceiveBufferSize)

     Don't think so, but not sure.

    See above.

just write the buffer directly into unmanaged memory (or
    It does not block if buffer space is available.  If space is available,
it copies the user buf and returns N.  Non-blocking socket mode gets a
little more complex.  It will copy upto the point it has space for and
return N or something < N, then your code needs to send the rest of the buf.

       BeginSend does not copy user buffer, but keeps it pinned and driver
uses user buffer directly.  Another reason why BeginSend can be more
efficient as no buffer copy overhead.  In a busy system, this can be a
drain.  Not sure if there is every a case where is does a copy and releases
the users buffer?

synchronously, and it seems that the second Send always failed after
in the first Send cause this behavior to change?  In
blocking, depending on the size of the buffer and the number of
because the time it has taken to normally Send has decreased
Interestingly, if you set SendBufferSize to 0 in the code we are talking
about, on my tests, the *first send does throw the error.  So it would seem,
it is blocking for the ACK because of this zero buffer.  Try it out and see
if you see the same.

really unpredictable and that no real example can be
other words, it's impossible to understand this behavior
But I think we are talking about an error in "our" protocol so not sure this
matters.  The connection is implicitly shutdown half-way by server.  Server
can send and client can receive - all good.  Client should not be sending
anyway sence it should "know" the state of the protocol - hence the error in
our protocol.  Client only knows explicitly, after it trys a send and gets
the ACK with the RST set.


TCP I as William recommended, but if there is a genuine
You can also read the RFCs (i.e. 793, 3168)
ftp://ftp.rfc-editor.org/in-notes/rfc793.txt

Hi William,

Thanks for your response.

That doesn't answer "How often will a blocking Send block?", although it does answer, "for how long?".

I was looking for an actual value but I found it in the docs for TcpClient.SendBufferSize on MSDN: "The default value is 8192
bytes".

You said in another branch of this thread (a few times) that Send will block for an ACK if it hasn't received one after a certain
number of packets have been sent without an ACK and that you believe that number to be 2.  Did I understand that correctly?

Given that the IP header + TCP header is 256 bytes [http://en.wikipedia.org/wiki/Transmission_Control_Protocol], + the 10 bytes of
data being sent by Send from the example in the OP means that the example was sending 266 bytes per packet, with one packet each
iteration.  Is that correct?  (Does the send buffer size include the size of the headers as well?)

Therefore, the first Send was obviously not filling up the 8192 byte send buffer.  The first Send returned im