As I have described in my previous post - you can track how long it takes to perform ADF BC activation in your system - Recording ADF BC View Object Instance Activation Time. Activation event can be potentially slow, as it involves re-quering and re-fetching of all active VO instances, initialized by current AM. In order to speed up ADF application runtime performance, we should tune it in a such way to avoid activation events as much as possible. Main goal of this post, is to prove with practical experiment, theory behind AM tuning for minimizing number activation events.
Firstly, you need to assign proper Referenced Pool Size value in AM properties configuration screen, to make sure it is equal to expected number of concurrent users. Don't forget to enable DB pooling - Stress Testing Oracle ADF BC Applications - Do Connection Pooling and TXN Disconnect Level, this will minimize open DB connections.
One of the key parts is to set longer time out value for inactive AM instance, you can set it to be equal to Web session timeout set in web.xml. This must be increased together with AM time to live setting to -1, as this will guarantee AM instance will be never removed forcefully. As AM time to live by default is 1 hour, ADF will remove AM instance, no matter for how long AM instance time out is set. This is based on recommendations posted by Duncan Mills - ADF Performance Presentation from UKOUG.
This is how sample application stresstest_v5.zip looks - there are two tabs, each using different AM. Navigation is involved, together with data update operation through ADF BC custom method - all this will tested by running JMeter stress load script:
JMeter script - ADFBC_Tuning.jmx, is created in a such way to simulate real user behaviour as much as possible. Script is starting 60 threads with 40 seconds delay, there are three breaks included - 3, 2 and 2 minutes. This allows to stretch stress test time and create more realistic load between requests:
AM from sample application is configured with 1 minute of Idle Instance Timeout and 1 minute of Pool Pooling Interval. By default these values are 10 minutes each, but I set both to be 1 minute, to fit it into JMeter script better - where breaks are by 3 and 2 minutes. There should be enough time for inactivity period, to mark AM instance as inactive and remove it. We are going to see how short inactivity time affects number of AM activations and runtime performance at the end. Updated settings for AM instance Idle Instance Timeout and Pool Pooling Interval. Of course, DB pooling is enabled as well, as per above recommendations:
Referenced Pool Size is set to 70, there should be enough available instances to handle 60 threads started from JMeter. Let's see how it works in practice.
Based on EM output - we can see that activations are happening for AM 1:
Similar picture for Activations is for AM 2:
This doesn't look logical at first, but really it is. Simply, when there is no activity - but user session is still active, inactive AM's are removed fast. When user is resuming his work - new AM instances are created again, this is when activation happens.
In order to avoid such effect, we need to increase Idle Instance Timeout, in this example I'm setting it to 3 hours (I would say this should be equal to web session timeout configured in web.xml):
Make sure to set AM time to live to be -1, otherwise AM instance will be forcefully removed after default 1 hour:
With these settings, we are getting much better results. For Referenced Pool Size = 70 and 60 concurrent JMeter threads, there are no activation events logged over longer time, there are no new AM instances, all active AM instances are reused:
The same is true for the second AM - no activation events are logged:
I can increase JMeter concurrent threads to 80, this will be higher than configured Referenced Pool Size = 70:
Naturally, this results in activation events - as Referenced Pool Size is too small to handle 80 concurrent threads in this situation - this is expected:
One final test - we need to verify if setting AM time to live to be -1 is really necessary. I'm going to set AM time to live to be 4 minutes, meaning after 4 minutes AM instance will be removed, no matter it is active or idle:
As it was presumed - we are starting to see AM activation events. This means it is not enough just to increase AM instance Idle Instance Timeout, but is really important to set AM instance Time to Live setting to be -1, or at least increase it to be longer than Idle Instance Timeout:
The same is for the second AM:
Summary - if you want to minimize number of AM activations in the system, make sure to set at least these three settings described in this post:
1. Enable DB pooling
2. Increase AM instance Idle Instance Timeout
3. Increase or set to -1, AM instance Time to Live
Firstly, you need to assign proper Referenced Pool Size value in AM properties configuration screen, to make sure it is equal to expected number of concurrent users. Don't forget to enable DB pooling - Stress Testing Oracle ADF BC Applications - Do Connection Pooling and TXN Disconnect Level, this will minimize open DB connections.
One of the key parts is to set longer time out value for inactive AM instance, you can set it to be equal to Web session timeout set in web.xml. This must be increased together with AM time to live setting to -1, as this will guarantee AM instance will be never removed forcefully. As AM time to live by default is 1 hour, ADF will remove AM instance, no matter for how long AM instance time out is set. This is based on recommendations posted by Duncan Mills - ADF Performance Presentation from UKOUG.
This is how sample application stresstest_v5.zip looks - there are two tabs, each using different AM. Navigation is involved, together with data update operation through ADF BC custom method - all this will tested by running JMeter stress load script:
JMeter script - ADFBC_Tuning.jmx, is created in a such way to simulate real user behaviour as much as possible. Script is starting 60 threads with 40 seconds delay, there are three breaks included - 3, 2 and 2 minutes. This allows to stretch stress test time and create more realistic load between requests:
AM from sample application is configured with 1 minute of Idle Instance Timeout and 1 minute of Pool Pooling Interval. By default these values are 10 minutes each, but I set both to be 1 minute, to fit it into JMeter script better - where breaks are by 3 and 2 minutes. There should be enough time for inactivity period, to mark AM instance as inactive and remove it. We are going to see how short inactivity time affects number of AM activations and runtime performance at the end. Updated settings for AM instance Idle Instance Timeout and Pool Pooling Interval. Of course, DB pooling is enabled as well, as per above recommendations:
Referenced Pool Size is set to 70, there should be enough available instances to handle 60 threads started from JMeter. Let's see how it works in practice.
Based on EM output - we can see that activations are happening for AM 1:
Similar picture for Activations is for AM 2:
This doesn't look logical at first, but really it is. Simply, when there is no activity - but user session is still active, inactive AM's are removed fast. When user is resuming his work - new AM instances are created again, this is when activation happens.
In order to avoid such effect, we need to increase Idle Instance Timeout, in this example I'm setting it to 3 hours (I would say this should be equal to web session timeout configured in web.xml):
Make sure to set AM time to live to be -1, otherwise AM instance will be forcefully removed after default 1 hour:
With these settings, we are getting much better results. For Referenced Pool Size = 70 and 60 concurrent JMeter threads, there are no activation events logged over longer time, there are no new AM instances, all active AM instances are reused:
The same is true for the second AM - no activation events are logged:
I can increase JMeter concurrent threads to 80, this will be higher than configured Referenced Pool Size = 70:
Naturally, this results in activation events - as Referenced Pool Size is too small to handle 80 concurrent threads in this situation - this is expected:
One final test - we need to verify if setting AM time to live to be -1 is really necessary. I'm going to set AM time to live to be 4 minutes, meaning after 4 minutes AM instance will be removed, no matter it is active or idle:
As it was presumed - we are starting to see AM activation events. This means it is not enough just to increase AM instance Idle Instance Timeout, but is really important to set AM instance Time to Live setting to be -1, or at least increase it to be longer than Idle Instance Timeout:
The same is for the second AM:
Summary - if you want to minimize number of AM activations in the system, make sure to set at least these three settings described in this post:
1. Enable DB pooling
2. Increase AM instance Idle Instance Timeout
3. Increase or set to -1, AM instance Time to Live
4 comments:
Hi Andrejus,
I have read various blogs of yours related to AM Fine tuning. Currently for our ADF 11.1.1.6.0 Version application we have lot of AM. We implemented Nested AM Concept as well. I have tuned AM Pooling parameters as below. Please suggust me whether i am ok or i need to do some thing better.
AM Time to Live= sessiontimeout in web.xml
Idle instance timeout=5 min
Pool Pooling Interval=5 min.
I have read some of your blog not to do doconnectionpooling = true due to activation/passivation cost. So i left that to default value. Is that ok.
Also we have cluster environment, should i enable doFailover=true, but it says it also consumes more activations/passivations. should i enable it. Please give u r best answers.
Hi,
On contrary, I recommend to set doconnectionpooling=true, check here: http://andrejusb.blogspot.com/2011/11/stress-testing-oracle-adf-bc_16.html
Idle Instance Timeout must be greater than Web session timeout.
If you want to support high availability, you should set doFailover=true.
In general, tuning depends and differs for every ADF application.
Regards,
Andrejus
Hi Andrejus,
Thanks for the useful information you share regarding ADF. It clears our concepts more and more.
Currently we are using ADF 11.1.1.6.0 Webcenter portal application, and we are facing a lot of issues due to AM connection pooling, frequently NPE in production.
We have two AM across our application.
Current settings of our AM is
Idle Instance Timeout = 600s
Pool Polling Interval = 600s
Initial Pool Size = 0
Max. Pool Size 4096
Referenced Pool Size = 10
Min. available size = 5
Max. Available Size = 25
Session Timeout in web.xml = 15 min
Timetolive=3600000
jbo.txn.disconnect_level = 1
jbo.doconnectionpooling = false
jbo.dofailover=false
As per your post, our issue could be because of following reasons:
The Session timeout > AM Idle Instance timeout
Time to live = 3600000
jbo.doconnectionpooling = false
These issues are observed in Production env. only. We do not have any Preprod env. Hence to simulate load conditions is quite difficult.
Is it recommended to make below changes directly in production or should it be attempted one by one
The Session timeout =15 min
AM Idle Instance timeout = 10 min
Time to live = -1
jbo.doconnectionpooling = true
jbo.dofailover=true
Please suggest the best possible way to minimize these errors and not to create new errors.
Regards,
Neha
As always, it is hard to advice production system settings. I can't recommend changing any tuning parameters, without checking the behaviour of your application.
Especially for production, what will be if it will break after my suggestions? :-)
Andrejus
Post a Comment