WebLogic Stuck Thread Case - Large Fetch Generated by Get Row ADF BC Method

This post is not about a bug, but rather about hidden underwater stone to avoid. Based on my previous use case for WebLogic Stuck Thread - Reproducing WebLogic Stuck Threads with ADF CreateInsert Operation and ORDER BY Clause, I will describe one more possible scenario for the same issue. This will be related to ADF BC API misuse, often it is unclear what side effect could produce at first friendly looking method. This method - getRow(key).

Please download complete sample application, if you are interested to reproduce it in your environment - LargeFetchApp_v3.zip.

This sample provides a method to generate dummy data for regions, around 10000 rows. View Object is using ORDER BY to display records in ascending order:

There is custom method created in AM implementation class. This method is calling ADF BC API getRow(key) method. Nothing dangerous at first, but here I'm using a key of the last record from the rowset - 10099. You may think, this is just a method to get a row by key.  Yes true, but what it does for you - before returning one row by key, it will fetch all rows until this row into memory. It travels through each row one by one, until it gets a row with defined key. This may consume a lot of memory, especially if rowset is large and row with defined key is somewhere at the end of the rowset. Example of such method:

Make sure ADF Logger config is enabled for RegionsViewImpl class:

Press Get Row button to invoke our custom method from above:

You will see - all rows are fetched and loaded into memory, before desired row is located:

If there will be concurrent users executing the same operation, sooner or later there will be Stuck Thread created in WebLogic and application will hang.

By the way, same applies for Last button use case in ADF, you should never use Last button - operation Last is going to fetch all rows in between, until it will get to the last row in the rowset.

Reusing and Extending ADF BC Entities from Common Model

This post is about ADF architecture and better application structuring with EO reuse from common model. I describe how to implement additional requirements to common model in extended ADF BC Entities. Great power of ADF framework - reusability. You should reuse as much as possible, this would simplify maintenance and future development of your application. I will be talking about ADF BC Entity Objects (EOs) reuse in this post. I would recommend to keep EOs in common model project and reuse them across the application. Fair requirement would be to have slightly different EO for specific use case - instead of creating new EO for the same DB table, we could extend original EO and implement specific changes. As for example, we may have different set of business rules, different doDML logic.

Sample application developed for this post includes Common Model library and Main application - eoreuse.zip. Common model library is based on Employees and Jobs EOs, associations and EO implementation generic class:

Employees EO from common model library implements business rule for Hire Date attribute - date must be today or in the past. The idea is to show that business rules from common EO will be inherited by extended EO as well:

Generic EO implementation class overrides doDML method, I do this to show the sequence of doDML calls from extended EO:

doDML method from Employees EO is overriden in the same way as in generic EO class:

We switch now to the main application - where common model library is imported successfully. Employees and Jobs EO's are available in the main application:

Here we can create new EO - extending Employees EO from common model:

In order to be able to change properties of specific attribute, we need to override it. Salary attribute is overriden and I have defined additional business rule - salary value check:

As we have extended main Employees EO, we need to define discriminator attribute. This is a key part, otherwise main Employees EO will stop working. As there is no really discriminator attribute in Employees, I'm going to create new transient attribute with default value 0:

Discriminator attribute is included into extended Employees EO, this attribute is set to be hidden - it will be never displayed on UI:

I have defined one additional association in main application - between Employees extended and Jobs EOs. We can assume, this would be required by specific use case:

Employees VO is created inside main application, calculated discriminator attribute is set to be populated from query expression - always returning 0:

This is how value is returned for discriminator attribute from SQL query:

The same value 0 is set for discriminator in VO based on extended Employees EO, it is calculated from SQL query expression:

Query for this VO is the same as for VO based on main Employees EO:

There are two View Links defined for Employees VO based on extended EO. First View Link is reusing Association from common model between employee and manager, this works perfectly for extended EO. Extended EO can reuse Association defined for parent EO:

Second View Link is using Association defined in main application - one between extended Employees EO and Jobs from common model:

Here you can see final ADF BC Data Model structure exposed. Master-Detail relationship between Jobs and Employees from common model, Master-Detail relationship between Employees VO based on extended EO and Jobs from common model, plus employee - manager relationship reusing Association from common model:

Sample application provides ADF UI implementation, where two tabs are given. First tab displays Employees VO structure based on main Employees EO. Second tab displays Employees VO structure based on extended Employees EO.

In the first tab - in the Employees list based on selected Job, business rule common model for Hire Date is working:

In the second tab, Hire Date business rule is working:

Plus additional business rule for Salary attribute, from extended EO is working as well:

Same rule is not reproduced for Employees data in VO based on main EO - as expected:

Make sure to enable ADF logger for application classes, we should check the sequence of doDML calls:

In the case of VO based on main Employees EO - firstly doDML from Employees EO is called and then from generic Entity Impl class:

Now, if you don't want to call doDML from main Employees EO, when calling doDML for extended Employees EO - make sure to change extending class:

As expected - doDML from extended EO is called and then doDML from generic Entity Impl class:

Reproducing WebLogic Stuck Threads with ADF CreateInsert Operation and ORDER BY Clause

This is a second post related to WebLogic Stuck Threads. You can read the first one - Evil Behind ChangeEventPolicy PPR in CRUD ADF 12c and WebLogic Stuck Threads. In the previous post, I was describing how to reproduce WebLogic Stuck Thread in ADF 12c, with ChangeEventPolicy = PPR. As per Steve Muench follow up comment, PPR works well, if you are using auto populated primary key. Today post is about something different, reproducible across all ADF versions - WebLogic Stuck Thread in relation to CreateInsert operation and ORDER BY clause usage for VO.

Here you can download test case application - LargeFetchApp_v2.zip. Make sure to set FINEST ADF Logger level for Regions VO Impl class, this will print fetched rows:

Make sure to disable AM Pooling, this will simulate stress test environment, passivation/activation will be happening on every request:

Region ID is set with default value, this is used to make sure new row gets primary key always set:

We set ORDER BY clause for REGION_ID and use DESC operator. This will bring rows with higher numbers first and display them in the top of ADF table. Make sure to use populateTable method (described in the previous post mentioned above) to insert 10 000 rows into REGIONS table:

Press Create button - new row with ID = 7 will be created, not inserted into DB yet:

Use Save button - to finally insert record into DB:

This is important step, new record will be inserted as last record in DB table, however UI displays it in first position. There is ORDER BY DESC set for this VO.

Try to navigate to other tab or do any other request, after record was saved to DB:

You will see in the log 10 000 rows fetched, until row with ID = 7 is found:

This large fetch is so unexpected and causes WebLogic Stuck Threads when multiple users are using application. The reason it fetches suddenly so many records, because new record with ID = 7 is displayed as the first row, but really this row is not retrieved with first range size of 25 top rows, it comes much later by ORDER BY DESC clause logic.

If we would insert new record with a key 10 100, large fetch would not be reproduced, as this row would be qualified to come with first range size of 25 rows.

We change to ASC order and test it again. Remove newly inserted record with ID = 7 from DB:

ORDER BY clause is updated with ASC order, instead of previous DESC - records with lower ID number will appear first:

Repeat the same steps as before, create new row - ID for the key attribute will be set automatically and equal 7:

Save and insert new row to the DB:

New row with ID = 7 is displayed on top, as it was just inserted, but still it will made into range size of the first 25 rows. We can see it from the log, it will 5th row fetched for the first range size:

As new row will be located early in the first range size, there will be no large fetch happening - application will continue to run fast:

I will describe in my next post, how to implement effective CRUD in ADF, taking into account large fetches. Generally this behaviour should be improved in ADF, instead of fetching entire collection of rows to locate needed row - it could do single fetch by key for a row not located in the first range size.

Evil Behind ChangeEventPolicy PPR in CRUD ADF 12c and WebLogic Stuck Threads

With this post I'm starting to prepare for our UKOUG'13 conference sessions. You can attend two of our sessions on UKOUG'13 Super Sunday, December 1st. These sessions are scheduled to run immediately one after another, so we are going to have two straight hours to discuss topics around ADF - ADF Anti-Patterns: Dangerous Tutorials, ADF Development Survival Kit. You should stop by to say hi, me and my colleague Florin Marcus, will be happy to answer any technical question about ADF.

Today post topic will be covered in the first session - ADF Anti-Patterns: Dangerous Tutorials. If you know how to implement CRUD functionality, you might be surprised - there are more things to know. I will describe one issue specifically related to ADF 12c, in the next post will present scenario reproduced in ADF 11g.

In your production application, you may experience WebLogic Stuck Threads. This is usually related to the large fetching, in most cases it happens unexpected and is not reproduced easily. In ADF 11g this is related to AM passivation/activation behaviour (will be described more in the next post), in ADF 12c I found another reason for unexpected large fetch in CRUD - ChangeEventPolicy = PPR setting usage for ADF iterator in Page Definition.

Here you can download fixed CRUD application for ADF 12c - LargeFetchApp.zip. This applications provides to methods, to insert 10000 rows into Regions table and remove the same rows. This is important to reproduce the error - it is easily reproduced when there are more records in the DB. Run populateTable method to insert 10000 rows into Regions table:

After running populateTable method, around 10000 rows should be available in the DB:

Regions VO implementation class contains overridden method for createRowFromResultSet. This method tracks every row fetched through this VO and reports it to the log:

Double check Regions iterator properties in Page Definition:

You will see ChangeEventPolicy = ppr, is set by default:

Run test application (make sure ChangeEventPolicy = ppr), press Create button - you will experience 15 to 30 seconds delay, before blank row will show up:

This is easily reproduced, if table is large enough (for example 10000 rows or more), as it was generated with populateTable method. To see ADF log output for Regions VO implementation class, make sure to set FINEST level in ADF logger config:

You will see lots of rows fetched during CreateInsert operation invocation, around 40000 rows fetched (means duplicates, as there are 10000 rows only in the table). This explains why Create is so slow, and this is by default in ADF 12c. It makes so many findByKey method calls:

In production system, when multiple users are doing same operation and even more data in the table - this generates Stuck Threads on WebLogic server and finally application stops. Simply because it consumes too much memory to create huge rowsets for all the data fetched from the DB.

You should change ChangeEventPolicy = none, this prevents unexpected large fetch on CreateInsert:

New row will be inserted instantly:

General recommendation - you should avoid using default setting for ChangeEventPolicy = ppr, it creates too many side effects usually. Seems like this functionality is not well tested yet.