Thursday, August 25, 2016

How to Restore a Voting Disk 11gR2

RESTORE VOTE DISK


root@hrdevdb2 /oracrs/app/11.2.0.3/grid/bin =>crsctl query css votedisk

Start CRS in exclusive mode
./crsctl start crs -excl

Query for voting disk
./crsctl query css votedisk

If you did not store voting disks in Oracle ASM, then run the following command using the File Universal Identifier (FUID) obtained in the previous step:
$ crsctl delete css votedisk FUID
./crsctl delete css votedisk a4a849393fb14f4fbf92cbef0d2d215a
./crsctl delete css votedisk a8906deabdd3ff29ff0df2f382271ce0

Add a voting disk, as follows:
./crsctl add css votedisk /appvotingocrtie/vote/hhcvote1
./crsctl add css votedisk /apptmpctrltrcenv/vote/hhcvote2
./crsctl query css votedisk

root@hrdevdb2 /oracrs/app/11.2.0.3/grid/network/admin =>/oracrs/app/11.2.0.3/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   70165ff16d50df98ff135ecded8bd82e (/appvotingocr/vote/hhcvote0) []
 2. ONLINE   69e63cb962f84f1cbf3055292d79fe86 (/appvotingocrtie/vote/hhcvote1) []
 3. ONLINE   8a844b35ed364f85bfb3ee3ab8262215 (/apptmpctrltrcenv/vote/hhcvote2) []

root@eldevdb2 /apptmpctrltrcenv/vote =>/oracrs/app/11.2.0.3/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   70165ff16d50df98ff135ecded8bd82e (/appvotingocr/vote/hhcvote0) []
 2. ONLINE   69e63cb962f84f1cbf3055292d79fe86 (/appvotingocrtie/vote/hhcvote1) []
 3. ONLINE   8a844b35ed364f85bfb3ee3ab8262215 (/apptmpctrltrcenv/vote/hhcvote2) []
Located 3 voting disk(s).




How to Configure Huge Pages

What is HugePage?

HugePages is a feature of the Linux kernel  which allows larger pages to manage memory as the alternative to the small 4KB pagesize. For a detailed introduction, see Document 361323.1

Why Do You Need HugePages?

HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If your combined database SGAs is large (like more than 8GB, can even be important for smaller), you will need HugePages configured. Note that the size of the SGA matters. Advantages of HugePages are:
Larger Page Size and Less # of Pages: Default page size is 4K whereas the HugeTLB size is 2048K. That means the system would need to handle 512 times less pages.
Reduced Page Table Walking: Since a HugePage covers greater contiguous virtual address range than a regular sized page, a probability of getting a TLB hit per TLB entry with HugePages are higher than with regular pages. This reduces the number of times page tables are walked to obtain physical address from a virtual address.
Less Overhead for Memory Operations: On virtual memory systems (any modern OS) each memory operation is actually two abstract memory operations. With HugePages, since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided.
Less Memory Usage: From the Oracle Database perspective, with HugePages, the Linux kernel will use less memory to create pagetables to maintain virtual to physical mappings for SGA address range, in comparison to regular size pages. This makes more memory to be available for process-private computations or PGA usage.
No Swapping: We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead. HugePages are universally regarded as pinned.
No 'kswapd' Operations: kswapd will get very busy if there is a very large area to be paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are used, kswapd is not involved in managing them. See also Document 361670.1

How to Configure HugePages

The configuration steps below will guide you to do a persistent system configuration where you would need to do a complete reboot of the system. Please plan your operations accordingly:
Step 1: Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than installed RAM. e.g. If you have 64GB RAM installed, you may set:

*   soft   memlock    60397977
*   hard   memlock    60397977
There is no harm in setting this value large than your SGA requirements.
The parameters will be set by default on:

Oracle Linux with oracle-validated package (See Document 437743.1) installed.
Oracle Exadata DB compute nodes

Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the memlock limit
$ ulimit -l
60397977

Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic Memory Management (AMM) feature which is incompatible with HugePages. Disable AMM before proceeding. To disable, set the initialization parameters MEMORY_TARGET and MEMORY_MAX_TARGET to 0 (zero). Please see Document 749851.1 for further information in case you encounter the error below:

ORA-00845: MEMORY_TARGET not supported on this system

Step 4: Make sure that all your database instances are up (including ASM instances) as they would run on production. Use the script hugepages_settings.sh in Document 401749.1 to calculate the recommended value for the vm.nr_hugepages kernel parameter. e.g.:

$ ./hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 1496
Note: You can also calculate a proper value for the parameter yourself but that is not advised if you do not have extensive experience with HugePages and concepts.
Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:

...
vm.nr_hugepages = 1496
...
This will make the parameter to be set properly with each reboot.

Step 6: Stop all the database instances and reboot the server
Check and Validate the Configuration

After the system is rebooted, make sure that your database instances (including the ASM instances) are started. Automatic startup via OS configuration or CRS, or manual startup (whichever method you use) should have been performed. Check the HugePages state from /proc/meminfo. e.g.:

# grep HugePages /proc/meminfo
HugePages_Total:    1496
HugePages_Free:      485
HugePages_Rsvd:      446
HugePages_Surp:        0

Monday, August 1, 2016

Apps DN Password Expired


To provide an update on AppsDN password expiration in applications environments, and whether the AppsDN password can be set to never expire.


Applicable OID 11.1.1.7.0  & OAM 11.1.2.3.1 Versions


Error:

Unable to Call fnd_ldap_wrapper.create_user due to following reason.
ORA-20001: The LDAP APPS DN 
"orclApplicationCommonName=test,cn=EBusiness,cn=Products,cn=OracleContext,dc=com"
password has expired. Please contact the administrator to change the Password.
(USER_NAME=SCTEST101)


Check the original password in two different ways:


Using the query from DB:


SQL> select fnd_preference.eget('#INTERNAL', 'LDAP_SYNCH','EPWD','LDAP_PWD') Apps_Password from dual;

APPS_PASSWORD
--------------------------------------------------------------------------------

welcome123



Confirm whether it is working / expired.

Source the OID env

$ldapbind -h host -p <ldapPort> -D  "orclApplicationCommonName=test,cn=ebusiness,cn=products,cn=oraclecontext,dc=com" -w welcome123


bind successful



If the password expired and if you do not want to change password, you can follow below procedure to unlock the password.


Login into ODSM > Navigate to the Subtree till the Dn > "orclApplicationCommonName=test,cn=ebusiness,cn=products,cn=oraclecontext,dc=com" >
(The order is recursive here)




Click on Manage Attributes > Show all > Goto the Bottom 
You will see Update Password > Enter same password > "welcome123"



Retest the issue.

You can also disable to password expiration to the subtree or Extend the password lock time.

Login into ODSM > Navigate to Security > Password Policy > Check the Effective Subtree for the respective policy "cn=default"


Now Goto General, Change the Password Expire time to "0" to disable

    5184000 = 60 days (default)
    7776000 = 90 days 
    10368000 = 120 days 
    15552000 = 180 days 
    31536000 = 1 year



If you want to create separate policy for each user or for a separate sub tree please refer here:
How To Avoid the AppsDN Account Password From Expiring, and Steps to Create a Password Policy to Ensure AppsDN Account Password Never Expires (Doc ID 2148811.1)

Best Regards,

Srikanth

Tuesday, July 12, 2016

Capturing 12.2 or 12.1 Apache Opmn OHS Web Log files


Applicable EBS Version 12.2.x

12.2 Ebusiness Suite - Collecting Fusion Middleware Log Files (Doc ID 1362900.1)

Run the following from a Unix shell with the Apps environment sourced and upload the
resulting zip file under your  [ tmp ] directory.

a) OPMN and OHS log files
zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_OPMN_OHS.zip $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*log* $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*

b) WebLogic Server
zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_FMW.zip $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*out*
$EBS_DOMAIN_HOME/servers/oa*/logs/*log* \
$EBS_DOMAIN_HOME/servers/forms*/logs/*out* \
$EBS_DOMAIN_HOME/servers/forms*/logs/*log* \
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*out* \
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*log* \
$EBS_DOMAIN_HOME/sysman/log/*

Applicable 12.1 Version

R12 IAS / HTTP Server configuration and log file collection steps (Doc ID 1964851.1)

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_iAS1013.zip \
                $ORA_CONFIG_HOME/10.1.3/Apache/* \
                $ORA_CONFIG_HOME/10.1.3/j2ee/* \
                $ORA_CONFIG_HOME/10.1.3/opmn/* \
                $LOG_HOME/ora/10.1.3/Apache/* \
                $LOG_HOME/ora/10.1.3/opmn/* \
                $LOG_HOME/ora/10.1.3/j2ee/* \
                $CONTEXT_FILE

How To Gather Information To Diagnose A Java OutOfMemory Error In 11i, R12, or 12.1.1 (Doc ID 842857.1)

Tuesday, February 9, 2016

Cloning an Application Database 12.2






Prepare the Source System DB Tier for cloning.
$ cd <RDBMS ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>
$ perl adpreclone.pl dbTier

Prepare the Source Application Tier for Cloning
$ cd <INST_TOP>/admin/scripts
$ perl adpreclone.pl appsTier

As of Release 12.2, the adpreclone.pl process on the application tier creates a complete compressed archive of the Oracle Fusion Middleware and its components as follows:

A compressed archive of the Oracle WebLogic Server home, Oracle Web Tier Utilities home, Oracle Common Utilities home and the Oracle E-Business Suite home:
<COMMON_TOP>/clone/FMW/FMW_Home.jar

A compressed archive of the Oracle E-Business Suite WebLogic domain:
<COMMON_TOP>/clone/FMW/WLS/EBSdomain.jar

The Oracle E-Business Suite WebLogic domain's configuration template:
<COMMON_TOP>/clone/FMW/WLS/plan/moveplan.xml

A compressed archive of the Oracle Web Tier/Oracle HTTP Server configuration instance:
<COMMON_TOP>/clone/FMW/OHS/ohsarchive.jar

The Oracle HTTP Server configuration instance's configuration template:
<COMMON_TOP>/clone/FMW/OHS/moveplan.xml


The adpreclone log files are created in the <INST_TOP>/admin/log/clone directory.




Copy the database node file system

     1. Log on to the Source Application tier node as the applmgr user, and then:
    1. Shutdown the application tier processes.
  1. Log on to the Source System database node as the oracle user, and then: 
    1. Perform a normal shut down of the Source System database. 
    2. Copy the database (.dbf) files from the Source System to the Target System. 
    3. Copy the Source System database ORACLE_HOME to the Target System. 
    4. Start the Source applications system database and application tier processes. 
  2. Log on to the Target System database node as the oracle user, and then:
    1. Delete the oraInst.loc file in the Database ORACLE_HOME, if it exists.

Copy the application tier file system from the Source "Run Edition File System" to the Target "Run Edition File System"

Copy / rsync the “EBSapps” From Source Run FS to Target Run FS
(Use an identical File structure for source & target, i.e if source Run FS is FS1 use Target run FS as FS1)

$rsync –avz $RUN_BASE/fs1/EBSapps $RUN_BASE/fs1/EBSapps
$rsync –avz $RUN_BASE/fs2/EBSapps $RUN_BASE/fs2/EBSapps

FMW_Home & inst will be created by adcfgclone execution

NB: Preserve Old Env Files / Context Files / DBC File / .Profile / Inventory

De-register the Oracle Homes on both Run Edition and Patch Edition file system:

Verify if Oracle Inventory contains the following Oracle Home entries for the Run Edition or Patch Edition file system:
<FMW_HOME>/oracle_common
<FMW_HOME>/webtier
<FMW_HOME>/Oracle_EBS-app1

Ex: To de-register or detach that Oracle Home:
$ ./runInstaller -detachhome ORACLE_HOME=<Oracle Home Location> [-invPtrLoc <s_invPtrLoc>]

Issues:
Script Executed in 25420 milliseconds, returning status 255
Script failed, exit code 255

In: ../../fs1/EBSapps/comn/clone/FMW/logs the following error is seen:
ERROR - CLONE-20275   Insufficient space to create /tmp/CLONINGCLIENT048348542103845

Fix: Increase / Change the temp directory location by setting the T2P_JAVA_OPTIONS environment variable to point to a temporary location of your choice
Bug 12959323: T2P cloning tools require 10GB of free space in /tmp
          $ T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/home/applmgr/tmp"
          $ export T2P_JAVA_OPTIONS

$cd $COMMON_TOP/clone/bin
$ perl adcfgclone.pl appsTier dualfs (When you are above Delta6)

Since we are on Delta 6 I used

$ perl adcfgclone.pl appsTier

Prompts for Below inputs:

Enter the APPS password :
Enter the Weblogic AdminServer password :
Do you want to add a node (yes/no) [no] :

Target System File Edition type [run] :

Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [hostname] :

Target System Database SID : <SID>

Target System Database Server Node [hostname] :

Target System Database Domain Name [domain.com] :

Target System Base Directory : /../../applmgr/<SID>

Target System Base Directory set to /../../applmgr/<SID>

Target System Current File System Base set to /../../applmgr/<SID>/fs2

Target System Other File System Base set to /../../applmgr/<SID>/fs1

Target System Fusion Middleware Home set to /../../applmgr/<SID>/fs2/FMW_Home

Target System Web Oracle Home set to /../../applmgr/<SID>/fs2/FMW_Home/webtier

Target System Appl TOP set to /../../applmgr/<SID>/fs2/EBSapps/appl

Target System COMMON TOP set to /../../applmgr/<SID>/fs2/EBSapps/comn

Target System Instance Home Directory [/../../applmgr/<SID>] :

Target System Instance Top set to /../../applmgr/<SID>/fs2/inst/apps/<ContextName>

Do you want to preserve the Display [hostname:0.0] (y/n)  : n

Target System Display [hostname:0.0] :

Target System Root Service [enabled] :

Target System Web Administration [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :

Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

Target System Port Pool [0-99] : 10

Checking the port pool 10
done: Port Pool 10 is free
Report file located at /../../applmgr/<SID>/fs2/inst/apps/<ContextName>/admin/out/portpool.lst
Complete port information available at /../../applmgr/<SID>/fs2/inst/apps/<ContextName>/admin/out/portpool.lst

UTL_FILE_DIR on database tier consists of the following directories.

1. /../../applmgr/<SID>/fs1/inst/apps/<ContextName>/appltmp
2. /../../oracle/product/<SID>/db/tech_st/12.1.0/appsutil/outbound/<ContextName>
Choose a value which will be set as APPLPTMP value on the target node [1] :

Creating the new APPL_TOP Context file from :
  /../../applmgr/<SID>/fs2/EBSapps/comn/clone/context/apps/adxmlctx.tmp

The new APPL_TOP context file has been created :
  /../../applmgr/<SID>/fs2/inst/apps/<ContextName>/appl/admin/<ContextName>.xml

Log file located at /../../applmgr/<SID>/fs2/EBSapps/comn/clone/bin/CloneContext_0127155028.log




Remove Excessive Managed Servers:
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server2 -servicetype=oacore -logfile=$APPLRGF/TXK/delMS_OACORESrver2.log

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS  -formsc4ws=hostname.domain.com:7812

Run EBS Technology Codelevel Checker (ETCC) on the database tier.
On the database tier, run the EBS Technology Codelevel Checker (ETCC) described in My Oracle Support Knowledge Document 1594274.1Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes to confirm that all required database patches have been applied.

EBS Technology Codelevel Checker (ETCC) is available via Patch 17537119 and analyzes an Oracle Database Oracle Home, and warns of any missing database bug fixes required for Oracle E-Business Suite Release 12.2. It is run with the command checkDBpatch.sh (on UNIX) or checkDBpatch.cmd (on Windows). Further instructions are available in the patch readme. Ensure to install the latest version of ETCC into <RDBMS_ORACLE_HOME>/appsutil/etcc directory.

Repeat the steps for Patch FS


Remove Source Instances References:

As APPS user run

EXEC FND_CONC_CLONE.SETUP_CLEAN;

Please run below steps from DB Side to establish same patch level connectivity.

cd $ORACLE_HOME
Backup the appsutil directory
$ cp -pr appsutil appsutil.old
$ unzip -o $INST_TOP/admin/out/appsutil.zip

cd $ORACLE_HOME/appsutil/bin
Run Autoconfig on DB Node
./adconfig.sh 

Change APPS & SYSADMIN password using FNDCPASS.
$AFPASSWD –s applsys
$AFPASSWD –s sysadmin

Change the APPS password in WLS Data Source by running the the following script as shown:
$ perl $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl
When prompted, select 'updateDSPassword' to change the APPS password in the WLS Datasource.

Start all the application tier services using the $INST_TOP/admin/scripts/adstrtal.sh script.
Verify the WLS data source changes as follows:
Log in to the WLS Administration Console.

In the Domain Structure tree, expand Services, then select Data Sources.
On the Summary of JDBC Data Sources page, select EBSDataSource.
On the Settings for EBSDataSource page, select Monitoring > Testing.
Select "oacore_server1".
Click Test DataSource.
Look for the message "Test of EBSDataSource on server oacore_server1 was successful".

When Source is registered with SSO, De-register the References:

How to Remove OAM References for Clone Instances
===========================================

$FND_TOP/bin/txkrun.pl -script=SetSSOReg -removereferences=Yes -appspass=*******

$txkrun.pl -script=SetOAMReg -deregisteroam=yes -ebsProfileLevel=site \
-oamHost=http://host.domain.com:6017 -oamUserName=weblogic -skipConfirm=yes
After de-registering your Oracle E-Business Suite instance from Oracle Access Manager, you no longer need the Oracle E-Business Suite AccessGate deployment. Delete your Oracle E-Business Suite AccessGate using your WebLogic Administration Console, for example:
http://ebshost.example.com:7001/console
In the WebLogic Administration Console, navigate to EBS_domain_sid > Deployments, stop then delete the Oracle E-Business Suite AccessGate application named "accessgate". Ensure that you click 'Activate Changes' in the 'Change Center' pane, for the changes to take effect.
If you do not use the data source "OAEADatasource" for any other application, you may also delete the datasource. Navigate to EBS_domain_sid > Services > Data Sources, and delete data source "OAEADatasource". Ensure that you click 'Activate Changes' in the 'Change Center' pane, for the changes to take effect.
Delete the Managed Server on which accessgate was deployed:
1.    If the Managed Server oaea_server1 is currently running, shut it down as follows:
$ sh $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh stop oaea_server1
The script will prompt for the following passwords:
Enter the WebLogic Admin password.
Enter the required information when prompted.
2.    Run the command below on the application tier node where the oaea_server1 managed server resides. This will delete the managed server, and also update the respective context variables that contain references to the deleted managed server:
$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-delete-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=oaea_server1
The script will prompt for the following passwords:
Enter the APPS Schema password.
Enter the WebLogic AdminServer password.
Enter the required information when prompted.
The following confirmation message will be displayed: ManagedServer oaea_server1 deleted successfully.

3.    Remove the managed server and port from the mod_wl_ohs.conf configuration:
$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=removeMS -accessgate=<host>.<domain>:<port>
To determine the value of the Port that was used for the oaea_server1 managed server, search for 's_wls_oaeaport' in $CONTEXT_FILE.
Stop and restart the Oracle E-Business Suite Application Tier services.

As the APPS user, run the script to unlink all Oracle E-Business Suite users:

SQL>@$FND_TOP/patch/115/sql/fndssouu.sql %

Run Autoconfig on AppsTier

Source the Run FS
$ . /u*/app/applmgr/<SID>/EBSapps.env run
$adautocfg.sh

Source the Patch FS (Upload the context file to Database)
$ . /u*/app/applmgr/<SID>/EBSapps.env patch (No need to run autoconfig, Just upload the context file)
$$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=$CONTEXT_FILE logfile=/tmp/patchctxupload.log

ADOP: Run a Mock Cycle
adop phase=fs_clone
or
adop phase=prepare
adop phase=cutover


Menu’s not available Issue
Fix: Ask DBA’s to restore the below tables
FND_USER
WF_USER_ROLE_ASSIGNMENTS
WF_ALL_USER_ROLES
AK_WEB_USER_SEC_ATTR_VALUES
WF_LOCAL_USER_ROLES
WF_LOCAL_USERS


Post clone addendum (should  be included in the post clone steps)
For production clone:
 1.Change the profile option:
     Site name : instance name
     Java color scheme: Teal
2. Set the override address to One of the appsdba email address (before starting concurrent manager)
cd $FND_TOP/sql
$sqlplus apps/*****

SQL> @afsvcpup.sql

Issues: Prepare Phase is Failing with Below Issue
    [ERROR]     Error Message :
    [ERROR]     ORA-20010: ORA-01422: exact fetch returns more than requested number of rows
    [ERROR]     ORA-06512: at "APPS.AD_ZD_ADOP", line 2790
    [ERROR]     ORA-06512: at line 2 (DBD ERROR: OCIStmtExecute)
    [UNEXPECTED]Error occurred while adding new node information
    [UNEXPECTED]Unrecoverable error occured. Exiting the current session.

Fix: 
EXEC FND_CONC_CLONE.SETUP_CLEAN;

Cloning Oracle E-Business Suite Release 12.2 Environments integrated with Oracle Access Manager 11gR2 (11.1.2) and Oracle E-Business Suite AccessGate (Doc ID 1614793.1)