Geeks With Blogs
Paul's Petrov Whiteboard [BizTalk, Enterprise Application Integration, Business Process Automation, SOA, .NET]

When using WCF receive adapter with SQL binding in Polling mode please be aware of the following problem.

Problem:
At some regular but seemingly random intervals the application stops processing new requests, places a lock on the database and prevent other application from accessing it. Initially it looked like DTC issue, as it was distributed transaction that stalled most of the time.

Symptoms:
Orchestration instances in Dehydrated state, receive location not picking up new messages, exclusive locks on database tables, errors in DTC trace. You may see DTCXact transaction open when executing DBCC OPENTRAN. Other applications accessing the same tables may not be able to execute queries and return timeouts.

Reproduce:
Set up polling receive location in dedicated host instance. Set the polling interval to very small value to make effect noticeable (fro example 1 sec). Open performance monitor and add Process\ThreadCount$BTSNTSvc counters. Restart host instances and watch the counter for them. The one that hosts receive adapter will be adding one thread per second. On the screenshot below notice two host instances are adding threads very quickly comparing to other (on of themis highlighted white line). The steep break in line is host instance restart after reaching 100.

 

Cause:
Microsoft has confirmed that there is a bug in the WCF-SQL adapter that results in thread and memory leak. In the receive adapter binding configuration there's receiveTimeout property set to 10 minutes by default. If during this period data is not found in the table the adapter would start new thread and allocate more memory without releasing old resources. Thus if there's no new data in the table for a long time a new thread will be created in the host instance every 10 minutes until it reaches threshold (1000) and then there's no threads left for this host instance and it can't start/complete any tasks. Then this host instance won't be able to do anything. If other artifacts are hosted in the instance they will suffer consequences as well.

Solution:
- Set receiveTimeout to the maximum time 24.20:31:23.6470000.
- Place WCF-SQL receive locations in separate host to provide its own thread pool and eliminate impact on other processes
- Ensure WCF-SQL dedicated host instances are restarted at interval less or equal to receiveTimeout to flush threads and memory
- Monitor performance counters Process/Thread Count/BTSNTSvc{n} for thread count trend and respond to alert if it grows by restarting host instance

If you use WCF-SQL Adapter in the Notification mode then make sure to remove sqlAdapterInboundTransactionBehavior otherwise this location will exhibit the same issue. In this case though, setting receiveTimeout doesn't help and new thread will be created at default intervals (10 min) ignoring maximum setting.

Posted on Monday, May 17, 2010 3:31 PM BizTalk , EAI | Back to top


Comments on this post: Problem with WCF-SQL Adapter

# re: Problem with WCF-SQL Adapter
Requesting Gravatar...
Hmmmm,

They are messing stuff up at MS. We get SP1 that really broke things that worked before. Even the WCF adapter had it's share (import bindings failed with missing method exception). Now this.

Hope they will get a fix soon !
Left by Patrick Wellink on May 18, 2010 2:14 AM

# re: Problem with WCF-SQL Adapter
Requesting Gravatar...
Well done. I have been plagued with this problem in our dev machines if they are not being used awhile. An idea of the likelihood of a fix?
Left by Mark brimble on May 18, 2010 3:35 AM

# re: Problem with WCF-SQL Adapter
Requesting Gravatar...
Do you know if this is fixed by now ?
Left by Patrick Wellink on Sep 14, 2010 2:57 AM

# re: Problem with WCF-SQL Adapter
Requesting Gravatar...
Paul,

Looks like they released a hotfix for this in September, was chasing up an issue just like this for a client and found the hotfix: http://support.microsoft.com/kb/2300507

Bill
Left by Bill Chesnut on Nov 03, 2010 6:25 AM

# re: Problem with WCF-SQL Adapter
Requesting Gravatar...
Hi Paul,
Is there a similar problem with the WCF-Oracle adapter.
Thanks
Suruchi
Left by suruchi on Sep 08, 2011 2:32 PM

Your comment:
 (will show your gravatar)


Copyright © Paul Petrov | Powered by: GeeksWithBlogs.net