Srikanth Bolagani: November 2015

Tuesday, November 24, 2015

ADOP FS_CLONE Issues

Performing CLONE steps...
Log: /opt/oracle/CLOUD/fs_ne/EBSapps/log/adop/11/fs_clone_20150806_103139/CLOUD_ebsclone4
*******FATAL ERROR*******
PROGRAM : (/opt/oracle/*****/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Thu Aug 6 10:34:59 2015
FUNCTION: TXK::SQLPLUS::_doExecute [ Level 3 ]
MESSAGES:
SQLPLUS error: buffer=

SQL*Plus: Release 10.1.0.5.0 - Production on Thu Aug 6 10:34:59 2015

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> SQL> Connected.
SQL> SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' AND node_name = 'ebsclone4' AND adop_session_id =
*
ERROR at line 1:
ORA-00936: missing expression
:
:

A. For a manual workaround:

1. Apply Patch 19045166 to obtain the diagnostic information for the latest adop sessions.

2. From generated diagnostics check the adzdshowstatus.out.

For example:

=========================================================================
= Summary of the Last 10 ADOP Cycles
=========================================================================

Session ID Node Name Node Type Started Elapsed(Prepare) Elapsed(Cutover) Elapsed(Cleanup) Elapsed(Apply) Status Patches Applied Session Type
---------- ---------- --------- ---------- ---------------- ---------------- ---------------- --------------- ------------ -------------------- ---------------
13 ebsystem master 24-AUG-15 1:04:50 0:00:00 0:00:00 0:00:00 FAILED -- --

11 ebsystem master 04-AUG-15 0:33:39 0:00:00 0:01:56 0:00:00 FAILED CONFIG_CLONE CONFIG_CLONE

3. Take a back up for table ad_adop_session_patches.
4. Run the following sql:

update ad_adop_session_patches set
bug_number='CLONE',status='N',SESSION_TYPE='CLONE',CLONE_STATUS='NOT STARTED'
where bug_number='CONFIG_CLONE' and status <> 'Y' and adop_Session_id=<*> ;
commit;

Where <*> = the session_id related to the CONFIG_CLONE row, in above example eg. adop_Session_id= '11'.

5. Re-run adop phase=fs_clone to confirm the results.

B. For a patching solution:

1. Download and review the readme of Patch 21132723. This patch is part of 21132723:R12.AD.C 1-OFF Fixes on top of R12.AD.C.DELTA.6.

2. Apply the patch in a test instance after a backup and re-test.

3. Confirm the file version: $AD_TOP/perl/ADOP ClonePhase.pm 120.0.12020000.8

3. Apply the patch to other instances the same as needed.

12.2 E-Business Suite - Collecting Online Patching and fs_clone Log Files (Doc ID 1542162.1)

Friday, November 6, 2015

Statndard tables columns are missing, 12.2 Upgrade

Their is sync missing between original synonym and the Edition View.
Expected in Upgrade 12.2

Use Upgrade Procedure for a table to be in sync with Edition View
SQL> exec ad_zd_table.upgrade(table_owner, table_name);

How to Add a Column:

Altering a table – Adding Columns and Indexes
1. Alter table in custom schema the usual way.
alter table cust_table add cust_sex varchar2(1) NOT NULL;

Regenerate the editioning view using AD_ZD_TABLE.PATCH
SQL> exec ad_zd_table.patch(table_owner, table_name);

NB: Once the table is altered with the table name, “exec ad_zd_table.patch(‘XXSDS’, ‘XXSD_EMPLOYEE’)” needs to be applied every time

Then:
exec ad_zd_table.upgrade(table_owner, table_name);

Report# adzddbcc.lst

DOC> EBS Online Patching Database Compliance Checker

DOC> $Header: ADZDDBCC.sql 120.13.12020000.61 2015/03/03 13:10:21 seetsing ship $

DOC> **********************************************************************

Elapsed: 00:04:53.30

DOC> **********************************************************************

DOC> SECTION-20

DOC> **********************************************************************

DOC> "Table must have an Editioning View"

DOC>

DOC> - P2: These tables may not be patched using Online Patching

DOC> - Fix violations by calling the table upgrade procedure:

DOC> SQL> exec ad_zd_table.upgrade(table_owner, new_table_name)

DOC> - Note: Tables that end with "_A" are typically audit tables

DOC> by the Audit Trail feature, and can be ignored.

DOC>#

OWNER NAME

---------------- ------------------------------

APPLSYS TXK_TCC_RESULTS

Perform the below.

$ sqlplus system/****@$TWO_TASK

SQL> alter session set current_schema=APPS;
SQL> exec ad_zd_table.upgrade('APPLSYS', 'TXK_TCC_RESULTS');
SQL> commit;

2. Re-run Online Patching Readiness Report.

Unable to Login Self Service page or After Giving Login Credentials users getting Blank page 12.2

Monitore For Connection Leaks within the WebLogic Console

Oracle WebLogic Server provides monitoring options to system administrators for JDBC connections.

To set up and monitor these options, follow these steps.

Log in to the WebLogic Console, which is typically at machine.domain:port/console. If you are unsure of the port number, review the context variable s_wls_adminport.
Navigate to to Services (Tree Link) -> Data Sources (Tree Link) -> EBSDataSource (Page Link) -> Monitoring (Tab).
Click on Customize This Table. Move the options required from Available to Chosen.

Suggested values:

Active Connections
Current Count
Current Capacity
Leaked Connection Count
Number Available
Number Unavailable

Press Apply.

Note: Leaked Connection Count will need the Inactive Connection Timeout (ICT) parameter to be turned on. By default, it is turned off to cater for the dedicated connections that need to remain idle within an Oracle E-Business Suite managed server. Therefore, enabling the Inactive Connection Timeout parameter is only recommended during the debugging of suspected connection leaks.

Before enabling ICT, verify the version of OAConnectionPoolManagerImpl.class is 120.7.12020000.2 or higher.

For example:

$ adident Header $OA_JAVA/oracle/apps/fnd/framework/server/OAConnectionPoolManagerImpl.class

$Header OAConnectionPoolManagerImpl.java 120.7.12020000.2 2015/02/10 09:19:45 spunam ship $

If it is at a lower version than 120.7.12020000.2, apply Patch 21619282.

Inactive Connection Timeout

The number of inactive seconds on a reserved connection before WebLogic Server reclaims the connection and releases it back into the connection pool.

You can use the Inactive Connection Timeout feature to reclaim leaked connections - connections that were not explicitly closed by the application. Note that this feature is not intended to be used in place of properly closing connections.

When set to 0, the feature is disabled.

Minimum value: 0

Maximum value: 2147483647

To set the Inactive Connection Timeout parameter, follow these steps.

Login to the WebLogic Server console, which is typically at machine.domain.port:7001/console. The port number can be found by examining the s_wls_adminport variable in the context file.
Navigate to Services (Tree Link) -> Data Sources (Tree Link) -> EBSDataSource (Page Link)-> Connection Pool (tab) -> Advanced (Expand Arrow).
Click on the Lock and Edit Button in the top left hand corner of the screen.
Update the Inactive Connection Timeout to the desired value in seconds: for example, a setting of 3600 for one hour.
Monitor the system, and after two iterations review the following files from the run file system.

$EBS_DOMAIN_HOME/servers/servername/logs/managed_server#.log and $EBS_DOMAIN_HOME/servers/servername/logs/managed_server#.out files.

If multiple managed servers are deployed in different nodes, obtain the above files for each defined managed server.
A change to the DataSource will affect all target servers (in a cluster).

Within the log files, look for Oracle WebLogic Server message number BEA-001153.

For example, the following message and a stack trace specific to the connection are created when the connection is closed by Inactive Connection Timeout:

####<Oct 27, 2014 3:10:33 PM PDT> <Warning> <JDBC> <rws3220424> <oacore_server1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <55a558fff3133dee:65b5ca10:149539dd27e:-8000-000000000000006e> <1414447833111> <BEA-001153> <Forcibly releasing inactive/harvested connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@31e8" back into the data source connection pool "EBSDataSource", currently reserved by: java.lang.Exception

Example stack from a known issue:

at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:366)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:373)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImp @ l.java:339)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:454)
at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:538)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.apps.fnd.security.DBConnObjWLSDSPool.getConnection(DBConnObjWLSDSPool.java:373)
at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.dbConnect(AppsConnectionManagerWLSDS.java:223)
at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.localAppsConnect(AppsConnectionManagerWLSDS.java:193) at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.makeGuestConnection(AppsConnectionManagerWLSDS.java:68) at oracle.apps.fnd.security.DBConnObj.getLabelledConnection(DBConnObj.java:314)
at oracle.apps.fnd.security.DBConnObj.<init>(DBConnObj.java:233)
at oracle.apps.fnd.security.DBConnObjDS.<init>(DBConnObjDS.java:98)
at sun.reflect.GeneratedConstructorAccessor140.newInstance(Unknown @ Source)

You can run the following SQL to obtain additional information about connection usage.

set lines 132

set pages 500

column module heading "Module Name" format a48;

column machine heading "Machine Name" format a25;

column process heading "Process ID" format a10;

prompt

prompt Connection Usage Per process and module

prompt ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

select count(*), machine, process, module from gv$session where program like '%JDBC%' group by machine, process, module order by 3 asc;

Save this as a .sql file, for example ict.sql, then execute it:

Note: If you are on a lower codelevel than TXK Delta 6, do not run this SQL as the results will be unreliable. Instead, review My Oracle Support Knowledge Document 1958352.1, On E-Business Suite 12.2 V$SESSION.PROCESS incorrectly reports EBS Client Process ID as '1234'.

Example stack trace from a known issue:

The below patches are recommended for these connection leak and many other framework related issues.

For 12.2.x, with Delta 6

-Apply the following patches if not already applied:

19494816 - connection leak on ErrorStack.java, fix released -- AOL

19807163 - transactionscope - connection leak in 12.2.4 -- Oracle Applications Manager

19858552 - npe during error handling, fix released (12.2.4) – OAF

Might have to apply below patches as well and see if it can fix the issue

-Please use How To Check If a Patch is Applied in 12.2.x ( Doc ID 1963046.1 )

20034256

20043910

19891697

19494816

19807163

19858552

18886213

19782999

Wednesday, November 4, 2015

OAM or OID SSL: Weblogic Console URL in Browser Gives Error code: ssl_error_weak_server

After SSL enable we seen the below issue in Chrome & Firefox Browsers.

Accessing Weblogic Console URL in Browser Gives Error code: ssl_error_weak_server

Enterprise Manager (EM) 12c Cloud Control uses 10.3 weblogic server; accessing weblogic console URL (https://<OMS HOSTNAME>:7101/console) in browser gives the following error:

In Chrome Browser:
Server has a weak, ephemeral Diffie-Hellman public key
ERR_SSL_WEAK_SERVER_EPHEMERAL_DH_KEY

In Firefox Browser:
SSL received a weak ephemeral Diffie-Hellman key in Server Key Exchange handshake message.
(Error code: sl_error_weak_server_ephemeral_dh_key)

This is on HP-UX, Will be applicable to other Unix Flavors.

Find and confirm what possible ciphers been supported with your platform & JDK Version.

Possibly like below, Add them to config.xml under "SSL" tag.

<ssl>
<name>oam_server1</name>
<enabled>true</enabled>

#################

#################

And also check for enable or disable below parameters.

#################

<hostname-verification-ignored>true</hostname-verification-ignored>

<client-certificate-enforced>false</client-certificate-enforced>

#################

Once you are good with the parameters restart the Managed Server. And be sure that your certs loaded and Application url listening on desired Ports.

Good Luck!!

Srikanth

Monday, November 2, 2015

Set up SSL in OAM 11.1.2.3 (11gR2 PS3), When integrating with 12.2

1. Demo Certificates

The below messages in your Web logic Log indicates that your web logic domain holds a risk.

Whilst you do not want SSL communication for your web services or applications. The internal (administrative) processes in your Weblogic domain still relies on the default demotrust and with this everyone can access your domain

2. Generate the identity java keystore

keytool -genkey -keysize 2048 -keyalg RSA -alias hostname -keystore hostname_identity.jks

Subject: CN=host.domain.com,OU=OraFinHR,O=Becker,L=SF,ST=CA,C=US

3. Generate signing request

keytool -certreq -alias hostname -file certnew.csr -keystore hostname_identity.jks

4. Certificate Authority (CA)

Certificate Authority to sign your certificate requests (the .CSR files you generated).

You can create your own CA and self-sign them, Shall use an external Internet CA provider,

Since this becomes expensive and time prone when you need such an external provider to sign every SSL enabled server in your landscape.

The best situation for us would be if the current organization already has an internal CA provider, especially when the rootCA is trusted by the servers and machines in your landscape.

5. Convert .p7b file to Readable Format#

$ openssl pkcs7 -print_certs -in certnew.p7b -out cert_chain.cer

6. Importing the CA response

Import the certificates in your keystore, starting with the rootCA, then the intermediateCA, then the specific server alias

keytool -importcert -trustcacerts -alias rootca -file ca.crt -keystore hostname_identity.jks

keytool -importcert -alias intermediateca -file initca.crt -keystore hostname_identity.jks

keytool -importcert -alias server01 -file certnew.cer -keystore hostname_identity.jks

keytool -importcert -trustcacerts -alias rootca -file ca.crt -keystore hostname_trust.jks

keytool -importcert -alias intermediateca -file initca.crt -keystore hostname_trust.jks

7. Change Settings in Weblogic Server

NB: Make sure you take a back up of config.xml, before you proceed to Enable SSL.
Location: $MW_HOME/user_projects/domains/<DOMAIN>/config/config.xml

By Preserving this at any point you can roll back changes, and resume services with the old settings.

Go to Domain Structure > IDMDomain >>Environment >> Servers

Make sure SSL Listen Port should be checked.

Click on Keystores Tab & Click on Keystores::Change >> Select “Custom Identity and Custom Trust”

Custom Identity Keystore <Give full keystore path>

Custom Identity Keystore Type <JKS>

Custom Identity Keystore Passphrase <anything>

Same for Custom Trust Keystore: You can give keystore path or CACERTS path.

Click on SSL tab >

Private Key Alias > Give the alias name what you given while you generating keystore file and PassPhrase.

Click Save.

Goto Change Centre >> Activate Changes.

Restart the Admin Server. The Admin log should like below.
NB: Also make sure "JSSE" check box is checked, Under SSL> Advanced tab.

NB: Repeat these steps for intended Managed Servers

Verify your URL:
https://host.domain:sslport/console

8. Change the OAM protocol

NB: Make sure you take a back up of oam-config.xml, before you proceed to Enable Protocol.

Location: $MW_HOME/user_projects/domains/<DOMAIN>/config/fmwconfig/oam-config.xml

By Preserving this at any point you can roll back changes, and resume services with the old settings.

After Enable SSL, Change in (11gR2 PS3 OAM). Login into the "oamconsole" and change the Protocol to SSL

Goto Access Manager Settings > Change OAM Server Port to SSL Port & OAM Server Protocol to “https”

Restart the OAM Managed Server

Cheers!!

Srikanth