Thursday, January 7, 2016

Issue with recurring Stuck Threads, Poor Performance of EBS 12.2.4

Check #1
A.      What is the version of OAConnectionPoolManagerImpl.java used ? 
If it is less then 120.7.12020000.3, then apply following :

Patch 21619282:R12.FWK.C FIX CONN.ISCLOSED() CHECK IN OACONNECTIONPOOLMANAGERIMPL.JAVA 

$ adident Header $OA_JAVA/oracle/apps/fnd/framework/server/OAConnectionPoolManagerImpl.class
From the Weblogic Console Set the Inactive Connection Timeout Value to 1800

B. Patch to Improve Results Caching Patch 19891697
Check the Result Cache Values
SQL> show parameter result_cache 
SQL> set serveroutput on
SQL> exec dbms_result_cache.memory_report;

SELECT dbms_result_cache.status() FROM dual;
 This is e-Business patch and documented in the README of 12.2.4 :
Oracle E-Business Suite Release 12.2.4 Readme ( Doc ID 1617458.1 )
Section 8: Apply Additional Critical Patches 
 set result_cache_max_size to be at least 300M . 


Check# 2
 WebLogic Version is 10.3.6 (FMW 11.1.1.6)or Higher
Re-evaluate the JVM arguments for managed servers on both Run and Patch file system is as follows to resize the -Xms and -Xmx values to -Xms2048m -Xmx2048m 
An Xmx of 4G had been the original recommendation from Development, but they have entirely backed off from that now in favor of 2G. Garbage Collection starts when heap memory gets low and when there is 4G to clean that takes a lot of time and that JVM cannot do any EBS work when garbage collection is going on so therefore more JVMs with smaller heaps are better than fewer JVMs with large heaps. 

Modify the following parameter tags for all the managed servers in all the apps nodes 
s_forms_jvm_start_options 
s_oacore_jvm_start_options 
s_oafm_jvm_start_options 
s_forms-c4ws_jvm_start_options 
Check# 3
Apply the missing patches, That are echoing from ETCC Code Checker on Both DB & APPS Nodes.
Check# 4
Navigate to Data Sources, and under the "Configuration" -> "Diagnostics" tab, check if any of the following profiling variables are enabled:

Profile Connection Usage
Profile Connection Reservation Wait
Profile Connection Leak
Profile Connection Reservation Failed
Profile Statement Cache Entry
Profile Statement Usage
Profile Connection Last Usage
Profile Connection Multithreaded Usage
Profile Harvest Frequency Seconds
Frequency Interceptor 
 If any of the above are enabled, please disable them.

Reference
OSB Alert Log Activities Generating Memory Leak on WebLogic Server ( Doc ID 1536484.1 ) 
Check# 5
 Verify the following setting:

a. Navigate as below
Services > Data Sources > EBSDataSource > Configuration > Connection Pool

b. Set "System Property" as below
v$session.program=weblogic.Name  

Check# 6
From Internal Research posted below on related logs, 

a. from 
Bug 19025537 ORACLE.IAS.CACHE.CACHEFULLEXCEPTION: J2EE JOC-017 THE CACHE IS FULL, the Java cache was configured low in the $EBS_DOMAIN_HOME/config/fmwconfig/servers/oacore_server1/javacache.xml file. 
1. Increase the following parameters in the $EBS_DOMAIN_HOME/config/fmwconfig/servers/oacore_server1/javacache.xml file and save: 

max-size="50" max-objects="100000"
b. Are you using NFS mount or have a Shared File System? 
If so, than follow 
Note 1375769.1 to have the following options: 

rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp

Check# 7
Revise and adjust the JDBC setting - 

FND_JDBC_MAX_CONNECTIONS: 200
FND_JDBC_BUFFER_MIN: 5
FND_JDBC_BUFFER_MAX: 50%
FND_JDBC_BUFFER_DECAY_INTERVAL: 60
FND_JDBC_BUFFER_DECAY_SIZE: 1
FND_JDBC_USABLE_CHECK: true
FND_JDBC_CONTEXT_CHECK: true
FND_JDBC_PLSQL_RESET: false