|
dev
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
SSLStream broken due to heap fragmentationgiant Sockets server that provides TLS encryption at the channel leve. Before .Net 2.0, we used an open-source encryption channel from Mentalis, and have even looked at the Mono implementation for doing this. The problem comes from the SSLStream not doing any buffer management. None. Zero. In the "no buffer management" case, each SSLStream allocates bufferes whenevers it needs them, copies data into those buffers, and then calls Socket.BeingRead / Socket.BeginWrite. This causes pinning. Massive, long-lived, horrible pins. This leads to horrendus non-recoverable fragmentation. The killer here is that both of these socket methods immediatly pin the buffers, and pass the data off to unmanaged code. The BeginRead method may not return for 10 seconds, 10 minutes, or 10 hours, leaving a hole in the Managed Heap the entire time. BeginWrite causes the same problem, although for a smaller length of time. To get around this in the past, we've created a block of buffers ahead of time, and cycled through them. This keeps all the pinned memory together in a single spot. This technique is described in detail here: http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx http://www.coversant.net/dotnetnuke/Default.aspx?tabid=88&EntryID=9 In our last implementation (.Net 1.1), we put alot of work into managing these buffer pools and really trying hard to eliminate heap fragmentation. This worked great. With the .Net 2.0 SSLStream, the BeginRead method ends up allocating bufferes here: BeginRead->ProcessRead->EnsureInternalBufferSize That method allocates a buffer by: this._InternalBuffer = new byte[addSize + curOffset]; Notice there's no fregging pool here? No attempt to eliminate fragmentation. No attempt to be efficient. This buffer gets passed to the NetworkStream, and in turn to the Socket, which promptly pins it. The BeginWrite case is the same thing: BeginWrite->ProcessWrite->StartWriting->EncryptBuffers->EncryptData->Encrypt This method allocates it's buffer: buffer1 = new byte[(size + this.m_HeaderSize) + this.m_TrailerSize]; Again, no fregging attempt to use a buffer pool. No attempt to eliminate heap fragmentation. Because the SSLStream is retarded, it only works on NetworkStreams - and Network Streams only work on Sockets. This means there's nowhere in the whole chain I can insert code prior to the Pin that would use a pooled buffer. Ugh. Now, .Net 2.0 is improved at managing pinning in the heap: http://blogs.msdn.com/maoni/archive/2005/10/03/so-what-s-new-in-the-clr-2-0-gc.aspx .... but I have it on VERY good authority that their technique isn't nearly as effective as the BufferPool method. In fact, it's not even close. This has runied my whole day. Ugh. -- Chris Mullins, MCSD.NET, MCPD:Enterprise We did figure out a way around this, in case anyone's interested.
By deriving a class from NetworkStream and overring the relevant methods, we can swap out the buffers used by SSLStream with our own buffers. Because our buffers come from a bufferpool that's allocated in a way to minimize heap fragmentation, the heap stays unfragmented, and scalability is largely unaffected. We certainly end up doing number of buffer copies, but that's a much cheaper price to pay than heap fragmentation. Show quote "Chris Mullins" <cmull***@yahoo.com> wrote: > We've been using the SSLStream class found in System.Net.Security to build > a giant Sockets server that provides TLS encryption at the channel leve. > Before .Net 2.0, we used an open-source encryption channel from Mentalis, > and have even looked at the Mono implementation for doing this. > > The problem comes from the SSLStream not doing any buffer management. > None. Zero. In the "no buffer management" case, each SSLStream allocates > bufferes whenevers it needs them, copies data into those buffers, and then > calls Socket.BeingRead / Socket.BeginWrite. This causes pinning. Massive, > long-lived, horrible pins. This leads to horrendus non-recoverable > fragmentation. > > The killer here is that both of these socket methods immediatly pin the > buffers, and pass the data off to unmanaged code. The BeginRead method may > not return for 10 seconds, 10 minutes, or 10 hours, leaving a hole in the > Managed Heap the entire time. BeginWrite causes the same problem, although > for a smaller length of time. > > To get around this in the past, we've created a block of buffers ahead of > time, and cycled through them. This keeps all the pinned memory together > in a single spot. This technique is described in detail here: > http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx > http://www.coversant.net/dotnetnuke/Default.aspx?tabid=88&EntryID=9 > > In our last implementation (.Net 1.1), we put alot of work into managing > these buffer pools and really trying hard to eliminate heap fragmentation. > This worked great. > > > With the .Net 2.0 SSLStream, the BeginRead method ends up allocating > bufferes here: > BeginRead->ProcessRead->EnsureInternalBufferSize > That method allocates a buffer by: > this._InternalBuffer = new byte[addSize + curOffset]; > > Notice there's no fregging pool here? No attempt to eliminate > fragmentation. No attempt to be efficient. > > This buffer gets passed to the NetworkStream, and in turn to the Socket, > which promptly pins it. > > The BeginWrite case is the same thing: > BeginWrite->ProcessWrite->StartWriting->EncryptBuffers->EncryptData->Encrypt > This method allocates it's buffer: > buffer1 = new byte[(size + this.m_HeaderSize) + this.m_TrailerSize]; > > Again, no fregging attempt to use a buffer pool. No attempt to eliminate > heap fragmentation. > > Because the SSLStream is retarded, it only works on NetworkStreams - and > Network Streams only work on Sockets. This means there's nowhere in the > whole chain I can insert code prior to the Pin that would use a pooled > buffer. Ugh. > > Now, .Net 2.0 is improved at managing pinning in the heap: > http://blogs.msdn.com/maoni/archive/2005/10/03/so-what-s-new-in-the-clr-2-0-gc.aspx > > ... but I have it on VERY good authority that their technique isn't nearly > as effective as the BufferPool method. In fact, it's not even close. > > This has runied my whole day. Ugh. > > -- > Chris Mullins, MCSD.NET, MCPD:Enterprise >  |
|||||||||||||||||||||||
Other interesting topics