JDeveloper 11g R2 IDE Application Performance Caching and Compilation Failures

If you are working with latest production release of JDeveloper IDE/ADF 11g R2 (11.1.2.1.0), you may face very strange issue related to ADF application compilation. JDeveloper may report that it can't find previously deleted files (for example ADF BC components). Why it still looks for un-existing (removed by developer) files and fails compilation process?

JDeveloper IDE 11g R2 provides application file caching feature, to speed up design time development and file load in the IDE. However, it looks like it fails to clean up file cache after files are removed from the application, still is trying to reference old files during compilation process and fails.

You will face file cache problem (this can be really frustrating), if log window reports something similar for deleted files (unable to copy to output directory):

It says - failed to find file, hey but this file was removed by me, just few minutes ago - JDeveloper, you should update your cache :)

Ah, Clean All and IDE restart will not help you this time - I did it multiple times with no luck. As you can see from the screenshot above, compiler complains about DepartmentsLovView object - this object was present before, but I removed it from the application - compilation should not fail (but it fails).

But there is always solution - open Application Properties wizard, there is IDE Performance Cache setting:

It specifies IDE cache location, application file cache is stored inside .data folder. If you go and remove .data folder - compilation completes successfully.

Oracle ADF 11g Table Insert with Empty Insert Row

Guess what ADF topic remains most popular over the years - CRUD. I have several blogs on this topic, for example - CRUD Operations in Oracle ADF 11g Table Without Auto PPR. Today I will describe one more scenario - how to maintain empty row, which can be used to insert new records into table.

Download sample application - TableInsertApp.zip. This example contains ADF table with empty row in the footer:

User is able to type data into empty row:

Press Insert button and populate new row into table rowset:

Good thing - empty doesn't care about validation, because all business rules will be checked ones user hits Save button (in this case, primary key is missing):

Fix validation error, now commit is successful:

How we implement empty row? Pretty easy, with SELECT ... FROM DUAL VO - JobsEntryView:

This VO should have as many attributes, as original VO:

SELECT ... FROM DUAL allows to query only 1 row into rowset. Make sure all attributes are set to be Updatable always, as well as with correct precision type, etc. (same as original VO, except we don't need validation rules):

On the UI part, each column footer is populated with corresponding attribute from our read-only VO:

Create operation is doing deep copy of attribute values from read-only VO into main VO (and clearing insert row):

CLIENT_STATE_MAX_TOKENS = 1 for Browser Back Button Control

There is documented approach which allows to control web browser back button. This approach works for ADF 11g applications implemented with pages and ADF Task Flows based on pages (doesn't work with ADF fragments).

The trick is to set org.apache.myfaces.trinidad.CLIENT_STATE_MAX_TOKENS = 1. This means, ADF will preserve only one client token in memory, once user will press browser back button - this token will be lost. On the next user click - application will reset to its initial state, this will ensure predictable behavior.

Download sample application - BackButtonExpire.zip. This sample implements sequence of ADF Task Flows with pages, first it defines Departments call:

Next is Employees call:

Employees Task Flow contains Return activity to navigate back to Departments:

As per documentation, we set org.apache.myfaces.trinidad.CLIENT_STATE_MAX_TOKENS = 1 in web.xml:

When we test this, lets open last Task Flow from the sequence - Employees. Press browser back button now:

Page from previous Departments Task Flow is opened - however client token will be already lost, as per our configuration. This is helpful if we want to avoid broken data, now if user performs any action - current page will be reset and reloaded:

Reload happens:

Page is reinitialized with fresh client token:

Custom Exception Registration for ADF BC EO Attribute

Sometimes customers prefer to implement business logic validation completely in Java, without using ADF BC declarative/Groovy validation rules. Thats fine, we can code business logic validation in ADF and implement different custom validation methods on VO/EO level. How to report validation message from custom method on VO/EO level? We can throw JboException, but it shows up on UI as popup. When custom validation is implemented for specific attribute, we would like to attach validation message directly to the attribute itself. Steve Muench helped us this week with EO Attribute Exception registration, I would like to share solution with you.

Download sample application - AttributeExceptionRegistration.zip. This sample implements custom validation method on VO Row Impl class, means custom logic can be applied per row:

If validation logic fails, we are calling our own validation failure method implemented inside EO Impl class. This method actually constructs AttrValException and attaches it to specific attribute using standard registerAttributeException(...) method. Validation message is retrieved from Message bundle. On the UI AttrValException will be rendered the same as declarative attribute level validation message declared on EO attribute. See detail code for attaching AttrValException here:

Validation message will be retrieved from Message bundle:

In this example, just for demo purposes, validation is triggered in UI from separate Action Listener method - just by calling custom validation logic from VO Row Impl:

Validation messages shows up nicely, directly attached to the attribute field:

Stress Testing Oracle ADF BC Applications - Do Connection Pooling and TXN Disconnect Level

Today I will describe how jbo.doconnectionpooling = true and jbo.txn.disconnect_level = 1 properties affect ADF application performance. Read more about these properties - ADF BC Tuning with Do Connection Pooling and TXN Disconnect Level. Previous posts related to ADF BC application stress testing - Stress Testing Oracle ADF BC Applications - Internal Connections.

We will see that with jbo.doconnectionpooling = true and jbo.txn.disconnect_level = 1, average request processing time is faster. Performance boost is achieved, because passivation/activation circle happens in memory (data remains in memory) - data is not being stored into PS_TXN table, this allows to save execution time. After each request, we are releasing DB connection, this allows to save server resources as well. However, please keep in mind - this tuning approach is suitable mostly for such use cases, where database connection doesn't keep temporary data in database (for example - post changes or long running locks). Because of frequent database connection switch, you may loose such data.

Download sample test case - stresstest_v3.zip.

Stress test is performed with 80 concurrent users, 16000 transactions in 10 minutes (see previous posts for more details on JMeter script).

jbo.doconnectionpooling = true

jbo.txn.disconnect_level = 1

Average request processing time is quite interesting - it is a bit slow for initial requests, but improves with time:

If we compare the same type of graph, when jbo.doconnectionpooling = false and jbo.txn.disconnect_level = 0, number of passivations/activations is reported to be the same. Average request processing time is slightly longer:

Average request processing time with jbo.doconnectionpooling = true and jbo.txn.disconnect_level = 1 is 43.12 (ms):

Average request processing time with jbo.doconnectionpooling = false and jbo.txn.disconnect_level = 0 is 75.23 (ms):

Database connection usage with jbo.doconnectionpooling = true and jbo.txn.disconnect_level = 1 is reduced significantly. There should not be PS_TXN passivation/activation happening, but it still reports some internal connections consumed by AM:

As expected - much more database connections are reserved with default jbo.doconnectionpooling = false and jbo.txn.disconnect_level = 0:

Stress Testing Oracle ADF BC Applications - Internal Connections

We had productive discussion with Chris Muir for my previous post - Stress Testing Oracle ADF BC Applications - Passivation and Activation (see post comments). Two questions were discussed:

1. Importance of AM Maximum Pool Size
2. Higher than expected DB connection usage during stress test

I did additional stress tests (download JMeter config file - AMTest.jmx) and was able to get more information from test results - I would like to share with you. Download test case application updated for this post - stresstest_v2.zip.

For stress test scenario and number of concurrent users, please refer to blog post mentioned above. I'm using the same AM pool tuning settings for today post:

First lets answer:

- Higher than expected DB connection usage during stress test

As we saw in the last post, when stress test with 80 concurrent users was applied (look into 3rd Pessimistic scenario), application was using more than 40 DB connections for 40 active AM instances:

We were thinking, why such behavior is happening. Finally we realized, this happens because of additional internal AM connections to support passivation/activation behavior. Why additional connections are not visible on normal load? It looks like when passivation/activation action happens really fast, additional internal connections are not reported. But on high load, when WebLogic is not able to process passivation/activation fast enough - usage of internal AM connections becomes visible.

In order to prove that, we have created second data source (jdbc/HrInternalDS) on WebLogic and declared it to be used for internal AM connection - jbo.server.internal_connection property:

After performing stress test with new data source configuration, we can confirm that main application data source reflects number of active AM instances, while dedicated data source for AM internal operations reports additional connections:

This means its a good practice to declare separate dedicated data source for AM internal operations, if you want to maintain predictable number of database connections for application data source.

- Importance of AM Maximum Pool Size

This time I was repeating stress test multiple times, after 4 stress test (refer to Pessimistic scenario) iterations WebLogic server became slower - request processing for 5th iteration slightly reduced.

We can notice that from number of completed passivation/activation operations per minute - reduced to 1000 (it was 1500 before). This means passivation/activation circle started to happen slower. As result, AM pool is not able anymore to serve all 80 concurrent users with 20 AM instances (as per Referenced Pool Size). In this situation, number of active AM instances starts to grow, while finally its reaching Maximum Pool Size (30):

If there are even more online concurrent users coming, they will give NullPointer exceptions - because application physically unable to handle so many passivation/activation circles. If you will experience such situation, this means you need to increase Maximum Pool Size, as well as adjust Referenced Pool Size.

If we continue stress test and move to second AM, because of high load on the server - second AM performs even less passivation/activation circles (just around 600, before it was 1500) for 80 concurrent users with Maximum Pool Size = 30:

Naturally in this situation, second AM will be able to serve even less concurrent users on high load.

Stress Testing Oracle ADF BC Applications - Passivation and Activation

When we are talking with customers about Oracle ADF performance, very often we can hear such question - "Hey, ADF works well when there are few concurrent users, how it behaves when there is more serious load?". In order to answer this question I will publish series of posts, where we will study different parameters for AM pool tuning and test AM pool configuration under stress loads. Today we will see how stable is AM pool passivation/activation mechanism.

Stress tests are implemented with JMeter. Check Chris Muir blog about JMeter configuration for ADF 11g. I'm using JMeter configuration file from Chris blog, however original file is updated with internal loops to repeat session requests. JMeter allows to run stress test with multiple parallel users, each users starts its own HTTP session. It is not enough for stress test, additionally I would like to repeat user action within the same HTTP session - its why internal loops are needed, these loops allow to iterate through the same requests multiple times.

We will run 3 test scenarios (hardware: 4 virtual processors, 4 GB RAM):

1. Optimistic (15 concurrent users, 300 transactions in 10 minutes)

2. Average (25 concurrent users, 5000 transactions in 10 minutes)

3. Pessimistic (80 concurrent users, 16000 transactions in 10 minutes)

Each scenario consists of these steps:

1. Session initialization

2. Open Departments page (local AM)

3. Perform data change and Commit operation. Repeat 100 times per user, with 3 seconds wait time

4. Open Employees page (region imported from ADF library)

5. Perform data change and Commit operation. Repeat 100 times per user, with 3 seconds wait time

JMeter configuration file - internal loop to repeat session requests:

Wait time in the loop - 3 seconds:

Download sample application for stress test - stresstest_v1.zip. Sample contains two applications - main and application library. Both applications implement two AM modules, pool settings for both AM's are configured to the same values. There are two pages inside main application, first page brings data from local AM, second from imported region and related AM:

Second page integrates ADF region from ADF library:

Stress test is designed to run such scenarion, where user clicks on Submit button from ADF UI, this button calls custom method from AM interface. Custom AM method is accessing VO row, by randomized value in range based on VO rowset size. Numeric attribute value is changed (incremented by 1) for every accessed row, transaction is committed or reverted back, depending on concurrent modification. Additionally we have one transient attribute, it is set to be passivated:

All 3 stress test scenarios (Optimistic, Average and Pessimistic) are executed with the same AM pool configuration (for both AM's) - based on rough estimate of approximately 20 concurrent users in the system:

- Initial Pool Size = 22 (number of AM instances created on first ever access, recommended to be 10% more than estimated number of concurrent users)

- Maximum Pool Size = 30 (number of maximum AM instances pool can create, recommended to be 20%-30% more than initial pool size)

- Referenced Pool Size = 20 (number of active AM instances, recommended to be the same as estimated number of concurrent users - to avoid frequent passivation/activation)

- Minimum Available Size = 0 (when set to 0, allows to release all AM instances after idle timeout, this helps to release reserved database connections as well. Is set to 0 for tests in stress environment, in your system you may set it to higher value, but less than referenced pool size)

- Maximum Available Size = 25 (maximum number of AM instances after pool clean-up)

- Idle Instance Timeout = 300 (AM instance is considered inactive after 5 minutes of inactivity, this if for stress test. In your system you would set it something close to Web session timeout - to prevent frequent passivation/activation)

- Pool Pooling Interval  = 120 (AM pool is cleaned every 2 minutes)

1. Optimistic (15 concurrent users, 300 transactions in 10 minutes)

This test is performed on first ever access. As you can see, 22 AM instances are created, as per initial pool size setting. However, soon (idle time 5 mins + pool cleaning interval 2 mins) 7 AM instances are removed - because we have only 15 online users. Passivation/activation never happens (because referenced pool is set to be for 20, and we have only 15).  MainModule statistics:

Same for HrModule module:

Very important, because Minimum Available Size = 0, after idle time entire pool is cleaned:

Same with DB connections, there is no need to wait for AM time to live, AM instances and DB connections can be released earlier - however this will trigger passivation. DB connection can be released before AM time to live, by tuning AM pool Minimum Available Size and AM Instance Timeout:

2. Average (25 concurrent users, 5000 transactions in 10 minutes)

This test with 25 online users is slightly above Referenced Pool Size = 20. We should experience passivation/activation behavior.

AM pool allocates 20 active instances for 25 users (as per Referenced Pool Size). Passivation/activation starts to take place, in order to support all 25 users (around 500 passivation/activation circles per minute):

Similar behavior for the second AM:

40 DB connections are used during peak time, when both AM's are active (20 + 20 active AM instances):

As you can see, even Maximum Pool Size = 30, active instance size is not exceeding Referenced Pool Size.

3. Pessimistic (80 concurrent users, 16000 transactions in 10 minutes)

With 80 users, AM pool starts to passivate/activate much more (as expected). There are 20 active AM's,  as per Referenced Pool Size and those 20 AM's need to serve all 80 users. There are lots of switch activity happening - around 1500 passivation/activation circles in 1 minute:

Runtime statistics for the second AM:

During strong stress test, we can see that we were using at some moments more than 40 (20 per each AM) database connections (as previous test). Means AM pool during heavy stress test may maintain more active AM instances than specified by Referenced Pool Size - but still, AM size will not exceed Maximum Pool Size setting (30 in our case):

Conclusion: AM Pool is strong enough to handle stress peaks with online users greatly exceeding number of AM Pool Size and and AM Referenced Pool Size. However, in your system you should try to maintain AM Pool sizing as close as possible to expected number of online users - this will help to avoid performance costly passivation/activation operations.