During stress test , with over 1000 files dropped at receive location at once, found 73 suspended instances, upon checking found the Thread starvation error
You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host -> Select the Host -> Settings) . You have to restart the Host Instance afterwards
3. Each host instance has its own set of resources such as memory, handles, and threads in the .NET thread pool.Thus it is important to allocate enough threads to the .NET thread pool associated with an instance of a BizTalk host to prevent thread starvation.
.NET CLR: Use this to update the number of Windows threads available in the .NET thread pool associated with an instance of a BizTalk host.You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host Instances -> Select the Host Instance -> Settings) . You have to restart the Host Instance afterwards-
Worker threads min/max values
By Default it is 5/25
Recommended 25/100
The values specified for the thread pools are per CPU. For example, setting MaxWorkerThreads to 100 has an effective value of 200 on a 2 CPU computer
Out of curiosity tried setting the number of threads for Max Worker/IO Threads to 5000 - but it didn't allow, limit is 500/1000 respectively
Worker threads are used to handle queued work items and I/O threads are dedicated callback threads associated with an I/O completion port to handle a completed asynchronous I/O request.
So if you see(point2 and 3) EPM(Messaging Engine) has its own thread pool which we can configure using attribute Maximum Engine thread(Maximum number of messaging engine threads per CPU) whereas Orchestration and send adapter share the worker threads - XLANG (Orchestration Engine) does most of its work, including accessing the BizTalk databases to send and receive messages, on .NET thread-pool worker threads and also the send adapters(SOAP/HTTP based).
4.Fourth thing to look would be maxconnection - Controls the maximum number of outgoing HTTP connections that you can initiate from a client to a specific IP address. This can be a bottleneck if BizTalk Server makes a large number of HTTP, SOAP, or WSE requests.
Increasing the maxconnection attribute results in increased thread pool and processor utilization. With the increase in the maxconnection value, a higher number of I/O threads will be available to make outbound concurrent calls to the Web service.
By Default it is 2
Recommended 12 * Number of CPUs
To increase the number of concurrent connections, you can add/modify the entry in the BizTalk Server configuration file, BTSNTSvc.exe.config or BTSNTSvc64.exe.config for 64-bit hosts, on each BizTalk server. You can increase/limit this for the specific servers being called by adding additional entry for it
or you can have one entry set for all, like below . Maximum value is 20.
Note: Although increasing maxconnection would help BizTalk to push more messages out, the capacity of the downstream system should be taken into account.
Every system has a breaking point, doing above steps will help to understand the limits of your server by playing around with various combination of loads and configuration.
Final step - If above steps are not sufficient, then its time to scale up/out your server.
Why it happened
It is clear from error that the number of threads available were not sufficient to cater the load of messages pumped in.
What to do
1. The first thought would be to create a separate Host for Receive/Orchestration/Send so that each process has sufficient threads/resources for processing - but here it doesn't seem that significant as this server has only one application deployed with simple orchestration calling web service and collecting response.
Note: It is best practice to have dedicated host for Receiving, Sending, Orchestration and Tracking.
2. So the next was to increase threads, you will find following in host settings
Maximum engine threads: it denotes the maximum number of messaging engine threads per CPU.
By Default it is set to 20, I set it to 30 - restarted host instance and tested again with over 1000 messages. No errors now.
You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host -> Select the Host -> Settings) . You have to restart the Host Instance afterwards
This option specifies the maximum number of threads that can be used by the End Point Manager (EPM) - the Messaging engine. The EPM doesn't use all threads instead it starts with the number of threads equivalent to 10% of this value and as load increases adds threads up to the specified value. As load is reduced, the number of threads allocated is reduced.
Out of curiosity tried setting the number of threads to 100 - but it didn't allow, the limit is 50
.NET CLR: Use this to update the number of Windows threads available in the .NET thread pool associated with an instance of a BizTalk host.You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host Instances -> Select the Host Instance -> Settings) . You have to restart the Host Instance afterwards-
By Default it is 5/25
Recommended 25/100
The values specified for the thread pools are per CPU. For example, setting MaxWorkerThreads to 100 has an effective value of 200 on a 2 CPU computer
Worker threads are used to handle queued work items and I/O threads are dedicated callback threads associated with an I/O completion port to handle a completed asynchronous I/O request.
So if you see(point2 and 3) EPM(Messaging Engine) has its own thread pool which we can configure using attribute Maximum Engine thread(Maximum number of messaging engine threads per CPU) whereas Orchestration and send adapter share the worker threads - XLANG (Orchestration Engine) does most of its work, including accessing the BizTalk databases to send and receive messages, on .NET thread-pool worker threads and also the send adapters(SOAP/HTTP based).
4.Fourth thing to look would be maxconnection - Controls the maximum number of outgoing HTTP connections that you can initiate from a client to a specific IP address. This can be a bottleneck if BizTalk Server makes a large number of HTTP, SOAP, or WSE requests.
Increasing the maxconnection attribute results in increased thread pool and processor utilization. With the increase in the maxconnection value, a higher number of I/O threads will be available to make outbound concurrent calls to the Web service.
Recommended 12 * Number of CPUs
To increase the number of concurrent connections, you can add/modify the entry in the BizTalk Server configuration file, BTSNTSvc.exe.config or BTSNTSvc64.exe.config for 64-bit hosts, on each BizTalk server. You can increase/limit this for the specific servers being called by adding additional entry for it
<system.net>
<connectionManagement>
<add address=" http://www.xyz.com" maxconnection="2" />
<add address="*" maxconnection="20" />
</connectionManagement>
</system.net>
Note: Although increasing maxconnection would help BizTalk to push more messages out, the capacity of the downstream system should be taken into account.
Every system has a breaking point, doing above steps will help to understand the limits of your server by playing around with various combination of loads and configuration.
Final step - If above steps are not sufficient, then its time to scale up/out your server.
If you have questions or suggestions, feel free to do in comments section below !!!
Do share if you find this helpful .......
Knowledge Sharing is Caring !!!!!!
great
ReplyDelete