As a reference, please check Steve Muench article - Understanding Application Module Pooling Concepts and Configuration Parameters. This post will be based mostly on Application Pool Cleanup Parameters section. Download sample application with optimized data source connection pool usage - DatabaseConnectionLeak.zip.
Main topic I will talk today is how to control data source connection pool grow. Most of ADF 11g applications are developed using JSF Fragments. Usually separate JSF Fragments are based on separate Application Modules. This means, if main page contains 10 JSF Fragments, 10 Application Modules will be involved during main page rendering. If there will be 10 users, there will be 10 x 10 = 100 connections created in data source connection pool. I don't want to say its not good to have many Application Modules, its good - work load will be distributed across different Application Modules. What is not good - by default, data source connections are removed back to available pool after really long time. Systems were we have many concurrent users can run out of available data source connections.
First thing WebLogic administrator will do is to enable Inactive Connection Timeout, hoping this will return inactive data source connections back to available pool. I have set it to 60 seconds:
My system contains two Application Modules, both are used from the same main page:
When user opens main page, from data source connection monitoring, we can see following picture:
Connection Pool Size increases up to 2 connections - thats correct, because main page involves two Application Modules. Connections in Use pool contains two active connections. As mentioned above, Inactive Connection Timeout was set to 60 seconds, and after 60 seconds we can see both reserved connections were removed from Connections in Use pool. Thats correct, but same time Connection Leaks were reported. This means, we can't use WebLogic Inactive Connection Timeout option to manage data source connections. ADF application keeps data source connection and WebLogic is removing it by force - its why leaked connections are reported. You can set Inactive Connection Timeout to be triggered once per day, to ensure any really leaked connections are removed.
What we could do is to play with Application Pool parameters. Connection Pool section is disabled, because we are using JNDI Data Source defined on WebLogic:
I'm interested in 3 parameters from Application Pool:
- Idle Instance Timeout (10 minutes default)
- Pool Pooling Interval (10 minutes default)
- Maximum Instance Time to Live (available through Properties tab, 1 hour default)
For such systems, where we have many concurrent users with short requests, default values probably are not suitable. Let's change Idle Instance Timeout to 1 minute and Pool Pooling Interval to 0.5 minute (same as you can see on screenshot above). This will affect Application Pool, but not data source connection pool - reserved two connections will remain as Connections in Use, even when there will be no active users online:
Such behavior is not acceptable and will expand data source connection pool very fast, this will block your application.
There is Maximum Instance to Live property, it allows to control how long Application Module instance should be available in Application Pool:
Default value is 1 hour, while it is acceptable, in most of the systems it will be too long. While Application Module instance will be alive, it will keep opened database connection in use. Yes, new users will reuse Application Instances available in Application Module pool and this will prevent database connection grow. However, where we have many concurrent users and many Application Modules involved, 1 hour for Maximum Instance to Live can be too long.
You can test Maximum Instance to Live with 2 minutes time. I have enabled it only for one of my Application Modules. Now we see such picture - 2 database connections reserved and 1 removed back to available connections pool after 2 minutes:
After 2 minutes, user comes back to his screen and makes some action related to data refresh:
Second data source connection is reserved again and remains reserved in 2 minutes time after last user activity:
While Idle Instance Timeout and Pool Pooling Interval will help you to optimize Application Pool, Maximum Instance Time to Live property will help to optimize entire application workload.









59 comments:
Hi Andrejus
Thanks once again for the really useful article.
To clarify your last statement, what you're recommending is that we set Idle Instance Timeout, Pool Timeout and timeouttolive in combination, correct? ... not just 1 of these settings.
Also we note the ratios on your settings, Idle Instance = 60 sec, Pool Timeout = 30 sec, and timeouttolive = 120 sec. Is there a reason you set the values as you did in terms of timeouttolive > idle instance > pool timeout? In other words what's the relationship between the relative values, and what would be the effect if they were set the other way around?
Thanks again,
CM.
Hi Chris,
Yes, thats right - all three settings should be used in combination, not just independently.
Regarding ratios - Idle Instance Timeout (60 sec - how long application module instance will stay idle until it will be marked as inactive), Pool Pooling Interval (30 sec - interval to check for inactive application module instances and remove them from the pool. this means, if after 60 sec instance is marked as inactive, during next 30 sec it will be removed from the pool), Maximum Instance Time to Live (even after application module instance is removed from the pool, its still alive and keeps DB connection reserved - in order to clean it, maximum time to live is set. it definitely should be longer comparing to Idle Instance Timeout + Pool Pooling Interval).
Regards,
Andrejus
Thanks Andrejus, appreciate your follow up. See you soon!
CM.
See you next week !
Hi Andre,
I have tried setting up the weblogic server with the em. But the issue is my EM is displaying only one node that is admin server under the wc_domain. The other node to check the connection leakage is not visible to me. Can you please share is there any other ear we need to deploy in weblogic to see this detail
Thanks
Vipin
You mean its not displaying Managed Servers?
Andrejus
Yes, I guess these are managed server, which are displaying the information about the connections.
Actually I am trying to run the application in the example and its behavior in connection graphs, as given in this example.
--Thanks
Vipin
Hi,
AdminServer can display same information as well, should not be a problem.
Andrejus
I face the following issue in ADF 10g.
I have an ADF page with some input fields and Apply button which saves the input data.
Each row in the table is identified by a unique ID generated by a sequence.
When I leave the page idle for 5-10 mins and then select Apply button, the entered data gets updated in an already existing row in the table instead of creating a new row.
The page uses PPR on some of the fields.
What can be the possible reasons for this behaviour?
Hi,
We are using Jdev 11.1.1.4 to develop our application. We are facing some problems regarding the connection with DB.
We have a Dashboard page which when loaded uses nearly 300
connections / sessions (we found it using the below query).
SELECT s.machine, count(1)
FROM v$open_cursor oc, v$session s
WHERE oc.sid = s.sid
GROUP BY s.machine
HAVING COUNT(1) > 2
Order By Count(1) Desc
When we refresh a page, it again occupies another 300 connections and thereby exceeding 1000 connections easily and causing "MAXIMUM SESSION EXCEEDED" problem from DB. Can you please give us a solution for this.
Also we have another problem here:
The connections are not released even when the screen is idle. So we used the appmodule level configuration which you have mentioned in the blog.
We have set as
max Inactive Age = 60000
monitor sleep interval = 30000
time to live = 120000
After we have set them, we have the connections released after 2min
but most of the time it is not happening.
Is there any specific configuration to be done prior to this?
Please get back to us for any clarification on the issue.
Thanks,
Dinesh
Connection will be released, if no user is using it. Keep in mind, connections can be reused by other users.
Andrejus
Hi Andrejus,
Thanks for the reply.
Please clarify the below :
In our application, we have a read only view object which fetches around 1000 records and we display this view object as af:table in a jspx page.
When we run this page it takes nearly 60 connections.
Is the number of connections(i.e 60) taken up is normal or high?
Also, when we load the same page in new tab in browser, it again takes up another 60 connections.
Is this the way how ADF behaves?
Thanks,
Dinesh
Looks like your application design is invalid, something really not good. Its unrelated to ADF, but most probably related to wrong ADF usage.
Andrejus
It'd be interesting once you figure out why the app is using 60 connections to post your results. If it's simply just a table in a page based on one AM and VO, it should be 1 connection, so something is extremely screwy in your app.
By chance have the AM pooling options been changed from the default?
CM.
Hi Andrejus and Chris,
Thanks for the info.
In order to find out why huge connections are taken up, I tried to create a new project where I have only one appmodule.
I have a .jspx page which displays the contents from one view object.
It took 11 connections each time when the screen is loaded.
Chris said that it should take only one connection.
Could you please give me your suggestions.
Thats impossible to have 11 connections, if you have 1 AM instance.
Please send me your sample application with 11 connections, I will take a look.
Andrejus
Hi Andrejus,
You are right that the number of connections taken up is only one when there is only one AM and one View Object.
I was actually using the below query:
Select S.Machine, Count(1)
From V$open_Cursor Oc, V$session S
WHERE oc.sid = s.sid
GROUP BY s.machine
Having Count(1) > 2
Order By Count(1) Desc
and this showed the number of cursors(I misunderstood this as the number of connections) used up.
With one AM and one VO, the number of cursors taken up is 10.
In our application, it shows the error "maximum number of sessions exceeded" when the number of cursors reaches 1000.
Could you please give me a solution.
Thanks,
Dinesh
Hi,
Try to change VO rows fetching to FETCH_ALL, as per this blog - http://andrejusb.blogspot.com/2009/03/ora-01000-maximum-open-cursors-exceeded.html
How many cursors are opened now?
Andrejus
Hi Andrejus,
Thanks for the info.
I have used the FETCH_ALL configuration in the View object. I can see the changes in the number of rows fetched.
But I have used the below query
select sum(value) from (
select a.sid, a.value, b.name
from v$sesstat a, v$statname b Where A.Statistic# = B.Statistic#
And B.Name = 'opened cursors current' and a.value>0)
to check the number of opened cursors when the page is loaded.
Opened cursors = 3 (each time the page is loaded).
The opened cursors value is same for both FETCH_AS_NEEDED and FETCH_ALL configuration.
I repeatedly loaded the page and opened Cursors value kept increasing.
I couldn't identify whether cursors are closed or not.
Could you please give your suggestion.
Thanks,
Dinesh
You can send me you sample app? I will do same test on my side.
Regards,
Andrejus
Hi Andrejus,
I have sent our sample application to your mail.
Please check and give us your suggestion.
Thanks,
Dinesh
Ok, I will review it and will get back to you.
Andrejus
Hi,
I have tested, it consumes only 1 cursor on page load for every web session in your app. After timeout, this cursor is released.
Andrejus
Hi Andrejus,
Thanks for the reply.
We have found the number of opened cursors using the below query.
select sum(value) from (
select a.sid, a.value, b.name
from v$sesstat a, v$statname b Where A.Statistic# = B.Statistic#
And B.Name = 'opened cursors current' and a.value>0)
Please tell me if the query is wrong or can you share the query which you have used to check the number of cursors opened.
Did you do any other changes in the AM or in view object configuration because we always had a minimum of 3 cursors when the page is loaded.
Thanks,
Dinesh
I was using same query as yours. No, I just run your application, and on every new web session it adds only 1 cursor.
Andrejus
Hi Andrejus,
Can you please share your weblogic connection pool configuration.
Thanks,
Dinesh
I was testing through JDBC URL and using ADF BC Connection Pool. Its uses 1 cursor then.
With WebLogic default pool settings, its using 3 cursors - tested.
Andrejus
Hi Andrejus,
Thank you. Hope my setting are right.
Thanks,
Dinesh
Hi Andrejus,
I have checked the variations in the number of cursors and I need some clarifications here.
When will the cursors get closed?
1. Immediately after when the page is loaded.
or
2. The page has to be idle for some time?
Thanks,
Dinesh
After page will be idle for some time, when AM pool instance will be gone (after maximum instance time to live expired).
Andrejus
Hi Andrejus,
Do you mean the below settings in the appmodule.
jbo.ampool.timetolive
Thanks,
Dinesh
Yes, its 1 hour by default.
Andrejus
Hi Andrejus,
Thanks for the reply.
But the appmodule settings is for closing the connection.
Is there any way to close the cursor alone once the particular query is executed?
Thanks,
Dinesh
Good question :)
Andrejus
Hi Andrejus,
Could you please give me some solution for that.
Thanks,
Dinesh
Not now, I will need to research this.
Andrejus
Hi Andrejus,
Need a clarification for the below:
50% of my application uses getDBTransaction() for connecting to Database and I have never closed it.
Does the usage of getDBTransaction() method a better choice compared to the usage of a View Object?
If getDBTransaction() method is used, should that be closed explicitly?
Please give your suggestions.
Thanks,
Dinesh
I don't understand statement - "Does the usage of getDBTransaction() method a better choice compared to the usage of a View Object?"
Those two things can't be compared :)
Hi Andrejus,
I mean we can do some DB transactions with the help of view objects or by using getDBTransaction() method. Which one would be better?
Then, is it fine to use the getDBTransaction() method?
In getDBTransaction() method, Should we close the transaction explicitly?
Thanks,
Dinesh
With ADF BC, we never need to close DB connection explicitly (framework takes care for this). There is no difference of using VO or AM getDBTransaction() in respect to open cursors.
Andrejus
Hi Andrejus,
Thanks for the clarification.
Thanks,
Dinesh
Hi Andrejus,
Could you please inform, once you research gets completed on the below question.
Is there any way to close the cursor alone once the particular query is executed? (without closing the connection)
Thanks,
Dinesh
Its on my todo list, I will post a blog - once I will finish it.
Andrejus
Hi Andrejus,
Thank you. Let me know the link once it is completed.
Thanks,
Dinesh
It can be completed any time, may be after 1 week, may be after 3 months - I can't promise. It all depends on inspiration and time availability :) Anyway, you will see it on the blog.
Andrejus
Awsme.You certainly helped me.You clear my concern..thanks alot....
Hi Andrejus,
Hope you are good.
For the cursor issue which we have already discussed in the blog, I have made a change in the appmodule --> "Pooling and Scalability" --> Disconnect application module upon release.
By default, the check box is not selected. I selected it and the application is running fine now.
The cursors are getting released.
But I would like to know whether it causes any problem or affects the performance of the application in some other ways.
Please give me your suggestion.
Thanks,
Dinesh
Hi,
I strongly recommend not to set "Disconnect Upon Release", this will break ADF BC pooling for AM. Also, on each request ADF will passivate and activate AM instance. Also data might be lost during requests.
Its okej to have cursors reserved, they will be reused.
Andrejus
Hi Andrejus,
Thanks for the clarification.
But the problem in my application is when 3 or more users access our application then the cursors reach the maximum limit of 300 and application gets hanged.
Thanks,
Dinesh
I never faced such problem with cursors.
Andrejus
Hi Andrejus and dinesh,
I am also facing the exact issue where even after setting the 3 parameters I still find the open cursors issues.
When I was exploring on this fix I read that the statement cache might retain open cursors (by setting Statement Cache Size:10 by default) on weblogic side at times.
Is this open cursors due to this weblogic cache or some other caching mechanism in appmodule which holds the open cursors
I'm also facing the same problem.
Appmodules are setu up according to this blog post, and in test example of 3 concurrent users open cursor problem arrises.
Any ideas? We're using Jdev 11.1.2. and weblogic 10.3.5. with sherman patch(adf runtime 11.2.).
Regards,
Patrik
Hi,
This post doesnt describe anything related to cursors, it covers open db connections. I think its normal for Fusion apps to reserve cursors, its the reason while XA type data source is not suitable for Fusion apps (however I was not analyzing this deeply).
Andrejus
Hi,
later today I've discovered something strange/interesting. According to your blog post I was "playing" with weblogic data source parameters(Home-Services-Data sources - myAppDS). I've put max open connections to be 75 as 15 is not enough for our project, but I've discovered that max open connection that is allowed is still 15(default)and app was craching right after 15th opened conn.
By further exploring of weblogic console, I've discovered that data source properties under Home-Deployments - myApp - Modules- myAppDS is different than one I've set up under Data sources node, despite the fact that we're refering to same database conn.
After that, I've succesfully chnaged parameters of this one data source and everything is working as it should.
Point is that I wasn't aware of possibility to have different tuning of same data source for different project.
Maybe this "discovery" could help to Oceans and Dinesh too :)
Regards,
Patrik
Okej, may be you have set auto generate JDBC descriptors as by default. Then its possible to have internal JDBC connection.
See here: http://andrejusb.blogspot.com/2009/09/manual-oracle-adf-application.html
Andrejus
Yes, you're right(as allways :-)! I wasn't aware of that feature, although heard about it but never thought about it.
Thanx again.
Regards,
Patrik
Hi Partik and Andrew,
Thanks you for giving me few leads,I almost thought that this was the root cause ,but in my project environment the checkbox u mentioned in the link in unchecked,
And too I am not able to see any datasource related info from my deployment module nodes as partik said..... Is that the way it should be
Hi Andrejus,
Thanks for all your really usefull entrys.
Do you know if it is possible to tune the appModule properties you described directly in em o weblogic console, avoiding to redeploy de ear project?
Thanks,
Josue
Hi,
Some of them yes, you can do that from Enterprise Manager. Select ADF application and go to ADF Settings.
Andrejus
Post a Comment