We had productive discussion with Chris Muir for my previous post - Stress Testing Oracle ADF BC Applications - Passivation and Activation (see post comments). Two questions were discussed:
1. Importance of AM Maximum Pool Size
2. Higher than expected DB connection usage during stress test
I did additional stress tests (download JMeter config file - AMTest.jmx) and was able to get more information from test results - I would like to share with you. Download test case application updated for this post - stresstest_v2.zip.
For stress test scenario and number of concurrent users, please refer to blog post mentioned above. I'm using the same AM pool tuning settings for today post:
First lets answer:
- Higher than expected DB connection usage during stress test
As we saw in the last post, when stress test with 80 concurrent users was applied (look into 3rd Pessimistic scenario), application was using more than 40 DB connections for 40 active AM instances:
We were thinking, why such behavior is happening. Finally we realized, this happens because of additional internal AM connections to support passivation/activation behavior. Why additional connections are not visible on normal load? It looks like when passivation/activation action happens really fast, additional internal connections are not reported. But on high load, when WebLogic is not able to process passivation/activation fast enough - usage of internal AM connections becomes visible.
In order to prove that, we have created second data source (jdbc/HrInternalDS) on WebLogic and declared it to be used for internal AM connection - jbo.server.internal_connection property:
After performing stress test with new data source configuration, we can confirm that main application data source reflects number of active AM instances, while dedicated data source for AM internal operations reports additional connections:
This means its a good practice to declare separate dedicated data source for AM internal operations, if you want to maintain predictable number of database connections for application data source.
- Importance of AM Maximum Pool Size
This time I was repeating stress test multiple times, after 4 stress test (refer to Pessimistic scenario) iterations WebLogic server became slower - request processing for 5th iteration slightly reduced.
We can notice that from number of completed passivation/activation operations per minute - reduced to 1000 (it was 1500 before). This means passivation/activation circle started to happen slower. As result, AM pool is not able anymore to serve all 80 concurrent users with 20 AM instances (as per Referenced Pool Size). In this situation, number of active AM instances starts to grow, while finally its reaching Maximum Pool Size (30):
If there are even more online concurrent users coming, they will give NullPointer exceptions - because application physically unable to handle so many passivation/activation circles. If you will experience such situation, this means you need to increase Maximum Pool Size, as well as adjust Referenced Pool Size.
If we continue stress test and move to second AM, because of high load on the server - second AM performs even less passivation/activation circles (just around 600, before it was 1500) for 80 concurrent users with Maximum Pool Size = 30:
Naturally in this situation, second AM will be able to serve even less concurrent users on high load.
1. Importance of AM Maximum Pool Size
2. Higher than expected DB connection usage during stress test
I did additional stress tests (download JMeter config file - AMTest.jmx) and was able to get more information from test results - I would like to share with you. Download test case application updated for this post - stresstest_v2.zip.
For stress test scenario and number of concurrent users, please refer to blog post mentioned above. I'm using the same AM pool tuning settings for today post:
First lets answer:
- Higher than expected DB connection usage during stress test
As we saw in the last post, when stress test with 80 concurrent users was applied (look into 3rd Pessimistic scenario), application was using more than 40 DB connections for 40 active AM instances:
We were thinking, why such behavior is happening. Finally we realized, this happens because of additional internal AM connections to support passivation/activation behavior. Why additional connections are not visible on normal load? It looks like when passivation/activation action happens really fast, additional internal connections are not reported. But on high load, when WebLogic is not able to process passivation/activation fast enough - usage of internal AM connections becomes visible.
In order to prove that, we have created second data source (jdbc/HrInternalDS) on WebLogic and declared it to be used for internal AM connection - jbo.server.internal_connection property:
After performing stress test with new data source configuration, we can confirm that main application data source reflects number of active AM instances, while dedicated data source for AM internal operations reports additional connections:
This means its a good practice to declare separate dedicated data source for AM internal operations, if you want to maintain predictable number of database connections for application data source.
- Importance of AM Maximum Pool Size
This time I was repeating stress test multiple times, after 4 stress test (refer to Pessimistic scenario) iterations WebLogic server became slower - request processing for 5th iteration slightly reduced.
We can notice that from number of completed passivation/activation operations per minute - reduced to 1000 (it was 1500 before). This means passivation/activation circle started to happen slower. As result, AM pool is not able anymore to serve all 80 concurrent users with 20 AM instances (as per Referenced Pool Size). In this situation, number of active AM instances starts to grow, while finally its reaching Maximum Pool Size (30):
If there are even more online concurrent users coming, they will give NullPointer exceptions - because application physically unable to handle so many passivation/activation circles. If you will experience such situation, this means you need to increase Maximum Pool Size, as well as adjust Referenced Pool Size.
If we continue stress test and move to second AM, because of high load on the server - second AM performs even less passivation/activation circles (just around 600, before it was 1500) for 80 concurrent users with Maximum Pool Size = 30:
Naturally in this situation, second AM will be able to serve even less concurrent users on high load.
Very informative post!
ReplyDelete