Text box

Please note that the problems that i face may not be the same that exist on your environment so please test before applying the same steps i followed to solve the problem .

Thursday 13 November 2014

Upgrade Grid infrastructure home from 11.2.0.3 to 11.2.0.4


Monday 4 August 2014

Useful Sqlplus tips

I want to share here the sqlplus tips i found in my daily life activities. I will keep this post open to share those tips and i will do my best to update this thread regularly.

Tips:

1-Replace the SQL> Prompt with the Current User Name

we can do this by using  set sqlprompt '_user> '


Example:

oracle@svtim02:[BSCSUAT] $ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 - Production on Mon Aug 4 14:54:31 2014

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining
and Real Application Testing options

SQL> set sqlprompt '_user> '
SYS>
SYS>

Thursday 5 June 2014

How to install OS watcher to monitor your OS and Submit the result to the support

I will try to explain today how to install and configure OS watcher that you must do on all your environment because it helps oracle support during the support for critical issues when you submit SR to them.
Below is the download link to the tool and in order to install it you must have acess to Oracle support website using your own username and password.
Download Link for the tool

after you download the tool you will have a folder which looks like oswebb730.tar.Upload this folder to your database machine.In case of RAC systems please upload it to all local nodes that are member of the cluster.
In my case i copied the file to /home/oracle/dbascripts/OSWATCHER.I then converted the tar file to a normal file using the below command:

oracle@svprtldb02:~/dbascripts/OSWATCHER$ tar -xvf oswbb730.tar
x oswbb, 0 bytes, 0 tape blocks
x oswbb/data, 0 bytes, 0 tape blocks
x oswbb/docs, 0 bytes, 0 tape blocks
x oswbb/docs/The_Analyzer, 0 bytes, 0 tape blocks
x oswbb/docs/The_Analyzer/OSWatcherAnalyzerOverview.pdf, 393822 bytes, 770 tape blocks
x oswbb/docs/The_Analyzer/oswbbaUserGuide.pdf, 2095591 bytes, 4093 tape blocks
x oswbb/docs/The_Analyzer/oswbba_README.txt, 29099 bytes, 57 tape blocks
x oswbb/docs/OSWatcher, 0 bytes, 0 tape blocks
x oswbb/docs/OSWatcher/oswbb_README.txt, 13481 bytes, 27 tape blocks
x oswbb/docs/OSWatcher/OSWatcherUserGuide.pdf, 3195426 bytes, 6242 tape blocks
x oswbb/Exampleprivate.net, 1864 bytes, 4 tape blocks
x oswbb/nfssub.sh, 740 bytes, 2 tape blocks
x oswbb/stopOSWbb.sh, 558 bytes, 2 tape blocks
x oswbb/call_du.sh, 67 bytes, 1 tape blocks
x oswbb/iosub.sh, 743 bytes, 2 tape blocks
x oswbb/profile, 0 bytes, 0 tape blocks
x oswbb/OSWatcherFM.sh, 4424 bytes, 9 tape blocks
x oswbb/ifconfigsub.sh, 772 bytes, 2 tape blocks
x oswbb/ltop.sh, 1486 bytes, 3 tape blocks
x oswbb/mpsub.sh, 542 bytes, 2 tape blocks
x oswbb/call_uptime.sh, 71 bytes, 1 tape blocks
x oswbb/gif, 0 bytes, 0 tape blocks
x oswbb/psmemsub.sh, 1445 bytes, 3 tape blocks
x oswbb/tar_up_partial_archive.sh, 4219 bytes, 9 tape blocks
x oswbb/oswnet.sh, 435 bytes, 1 tape blocks
x oswbb/vmsub.sh, 544 bytes, 2 tape blocks
x oswbb/call_sar.sh, 68 bytes, 1 tape blocks
x oswbb/oswib.sh, 414 bytes, 1 tape blocks
x oswbb/startOSWbb.sh, 2385 bytes, 5 tape blocks
x oswbb/Example_extras.txt, 626 bytes, 2 tape blocks
x oswbb/oswsub.sh, 524 bytes, 2 tape blocks
x oswbb/analysis, 0 bytes, 0 tape blocks
x oswbb/oswbba.jar, 235115 bytes, 460 tape blocks
x oswbb/locks, 0 bytes, 0 tape blocks
x oswbb/tmp, 0 bytes, 0 tape blocks
x oswbb/OSWatcher.sh, 34522 bytes, 68 tape blocks
x oswbb/topaix.sh, 537 bytes, 2 tape blocks
x oswbb/tarupfiles.sh, 746 bytes, 2 tape blocks
x oswbb/xtop.sh, 1486 bytes, 3 tape blocks
x oswbb/src, 0 bytes, 0 tape blocks
x oswbb/src/Thumbs.db, 10240 bytes, 20 tape blocks
x oswbb/src/OSW_profile.htm, 26288 bytes, 52 tape blocks
x oswbb/src/tombody.gif, 4175 bytes, 9 tape blocks
x oswbb/src/missing_graphic.gif, 8694 bytes, 17 tape blocks
x oswbb/src/coe_logo.gif, 2994 bytes, 6 tape blocks
x oswbb/src/watch.gif, 5621 bytes, 11 tape blocks
x oswbb/src/oswbba_input.txt, 135 bytes, 1 tape blocks
x oswbb/oswrds.sh, 825 bytes, 2 tape blocks


I then moved to the directory oswebb as per below:
/dbascripts/OSWATCHER/oswbb

Then run the below command to start the oswatcher:
 nohup ./startOSWbb.sh  30 168 gzip 2>/dev/null



The above command will keep the oswatcher running using nohup.The 30 is the interval per snapshot which means that every 30 seconds the os watcher will take snapshot.the maximum recommended time per oracle is 60 seconds between each snapshot.

The 168 will keep the data for 1 week.

the gzip will gzip all the files generated by the oswatcher.

Sunday 13 April 2014

Truncate table CASCADE is now avilable in 12c


Starting from 12c you can now truncate the table and all the tables referring to that table with referential integrity.
This new feature is only available in 12c. This new enhancement gets ride of the limitation existed in earlier releases.Starting from 12c you dont need to delete child tables before you delete the master table as you can now do it in one command as per below:


SQL> TRUNCATE TABLE mytable CASCADE; 

Sunday 23 March 2014

Cloning Grid control Oracle Agnet from running machine to a newly added clustered node

When you add a new node to the cluster then you need to install a new agent for the grid control on those new nodes.
These steps will help you clone already patched home and avoid the new installation procedures in few minutes.
To do so you have to follow the below steps:
On the Source machine:
tar cvf /{full path for the tar file to be created including the file name} {files or folders to be archived "Oracle Grid Home"}
After the tar file created then gzip it as below:

gzip agent11.tar
-bash-3.2$ ls
10.2                 agent10g.tar.gz      agent11g             stage
agent10g             agent11.tar.gz       oraInventory.tar.gz
-bash-3.2$ scp -rp agent11.tar.gz oracle@10.195.2.13:/app/portal/oracle/oracle
agent11.tar.gz       100% |************************************************************|   588 MB    00:51


On the destination machine:
set your desired time zone as below:
export TZ=Etc/GMT-2
Then install the home as below:
-bash-3.2$ ./runInstaller -clone -forceClone ORACLE_HOME=/app/portal/oracle/oracle/product/agent11g ORACLE_HOME_NAME=agent11g1 -silent b_chainedInstall=false oracle.sysman.top.agent:s_installType=AGENT OMS_HOST=csemp.bss.etisalat.eg OMS_PORT=4889 -local  CLUSTER_NODES={portaldb3}
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 29258 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-03-23_02-54-56PM. Please wait ...-bash-3.2$ Oracle Universal Installer, Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

You can find the log of this install session at:
 /app/portal/oracle/oracle/oraInventory/logs/cloneActions2014-03-23_02-54-56PM.log
.................................................................................................... 100% Done.



Installation in progress (Sunday, March 23, 2014 2:55:06 PM EET)
...............................................................Install successful

Linking in progress (Sunday, March 23, 2014 2:55:13 PM EET)
.                                                                64% Done.
Link successful

Setup in progress (Sunday, March 23, 2014 2:55:58 PM EET)
......................                                          100% Done.
Setup successful

End of install phases.(Sunday, March 23, 2014 2:56:19 PM EET)
Starting to execute configuration assistants
Configuration assistant "Oracle Configuration Manager Clone" succeeded
Configuration assistant "Agent Configuration Assistant" succeeded
WARNING:
The following configuration scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
/app/portal/oracle/oracle/product/agent11g/root.sh
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts

The cloning of agent11g1 was successful.
Please check '/app/portal/oracle/oracle/oraInventory/logs/cloneActions2014-03-23_02-54-56PM.log' for more details.
Configuring the agent:
-bash-3.2$ ./agentca -f -c "{portaldb3}"

Stopping the agent using /app/portal/oracle/oracle/product/agent11g/bin/emctl  stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
Running agentca using /app/portal/oracle/oracle/product/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/app/portal/oracle/oracle/product/agent11g ACTION=Configure MODE=Perform RESPONSE_FILE=/app/portal/oracle/oracle/product/agent11g/response_file RERUN=TRUE INV_PTR_LOC=/app/portal/oracle/oracle/product/agent11g/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform - mode is starting for action: Configure


Perform - mode finished for action: Configure

You can see the log file: /app/portal/oracle/oracle/product/agent11g/cfgtoollogs/oui/configActions2014-03-26_10-31-47-AM.log
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 29236 MB    Passed
The inventory pointer is located at /app/portal/oracle/oracle/product/agent11g/oraInst.loc
The inventory is located at /app/portal/oracle/oracle/oraInventory
'UpdateNodeList' failed.

Starting the agent using /app/portal/oracle/oracle/product/agent11g/bin/emctl  start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting agent ............................ started.

Stopping the agent using /app/portal/oracle/oracle/product/agent11g/bin/emctl  stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

Running Agent Addon Configuration using /app/portal/oracle/oracle/product/agent11g/perl/bin/perl /app/portal/oracle/oracle/product/agent11g/sysman/install/AddonConfig.pl
Arguments passed

Configuring Addon from xml : oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml

Running Command : /app/portal/oracle/oracle/product/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/app/portal/oracle/oracle/product/agent11g ACTION=configure MODE=perform RERUN=true  RESPONSE_FILE=/app/portal/oracle/oracle/product/agent11g/vt_responsefile COMPONENT_XML={oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml}
 Setting the invPtrLoc to /app/portal/oracle/oracle/product/agent11g/oraInst.loc

perform - mode is starting for action: configure


perform - mode finished for action: configure

You can see the log file: /app/portal/oracle/oracle/product/agent11g/cfgtoollogs/oui/configActions2014-03-26_10-33-20-AM.log

 Agent Addon Configuration done

Starting the agent using /app/portal/oracle/oracle/product/agent11g/bin/emctl  start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Agent is already running
Clustering the new agent nodes:
Then update the cluster node list by running the below command on all rac nodes to make the agent clustered:
 /app/portal/oracle/oracle/product/agent11g/oui/bin/runInstaller ORACLE_HOME=/app/portal/oracle/oracle/product/agent11g -updateNodeList "CLUSTER_NODES={portaldb1,portaldb2,portaldb3,portaldb4}" -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 29257 MB    Passed
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /app/portal/oracle/oracle/oraInventory
'UpdateNodeList' was successful.

In case you face issues and want to deattach and attach the home again:

To deattach Home:
runInstaller –silent -detachHome -invPrtLoc /app/portal/oracle/oracle/oraInventoryoraInst.loc \
ORACLE_HOME_NAME="agent11g1" \
ORACLE_HOME="/app/portal/oracle/oracle/product/agent11g"
To attach Home:
runInstaller –silent -attachHome -invPrtLoc /app/portal/oracle/oracle/oraInventoryoraInst.loc \
ORACLE_HOME_NAME="agent11g1" \
ORACLE_HOME="/app/portal/oracle/oracle/product/agent11g"
To Update the node list with clustered nodes:
/app/portal/oracle/oracle/product/agent11g/oui/bin/runInstaller ORACLE_HOME=/app/portal/oracle/oracle/product/agent11g -updateNodeList "CLUSTER_NODES={portaldb1,portaldb2,portaldb3,portaldb4}" -local

References:
http://docs.oracle.com/cd/E17559_01/install.111/e16847/clone_agent_usng_zip.htm



Monday 17 February 2014

Opatch 11.2.0.3 gives error when the opatch version higher than Grid Infrastructure 11.2.0.2 || applying patch set updates on GI_HOME

Hello,

While working to autopatch newly installed Standalone grid infrastructure the below error appeared.

Error:

root@svprtldbdev1:/app/11.2.0/grid/patchsetupdate# opatch auto .
Executing /app/grid/perl/bin/perl /app/grid/OPatch/crs/patch11203.pl -patchdir . -patchn . -paramfile /app/grid/crs/install/crsconfig_params
Bareword "ValidateCRSCTL" not allowed while "strict subs" in use at /app/grid/crs/install/crspatch.pm line 101.
Bareword "ValidateCRSCTL" not allowed while "strict subs" in use at /app/grid/crs/install/crspatch.pm line 112.
Compilation failed in require at /app/grid/OPatch/crs/patch11203.pl line 415.
BEGIN failed--compilation aborted at /app/grid/OPatch/crs/patch11203.pl line 415.

Note:
I was applying patch set update for 11.2.0.2 database and the patch number is p17272753_112020_SOLARIS64.zip.

workaround:

·       Grid Infrastructure Home:

$GRID_HOME/OPatch/opatch auto <UNZIPPED_PATCH_LOCATION> -oh <<Grid Home>> -olderver
·      The database Home:
 Please run below without olderver clause:

$GRID_HOME/OPatch/opatch auto <UNZIPPED_PATCH_LOCATION> -oh <<RDBMS Home>


The finding was that sometimes the higher version Opatch may cause the above error with higher version grid infrastructure.

Regards
Mohamed


Sunday 16 February 2014

Applying patch set update manually on Oracle Grid Infrastructure/Real Application cluster and Database 11.2.0.2.



The steps below helps you to manually apply patch set update for 11.2.0.2:
Patch Number:
Bug 17272753 - 11.2.0.2.12 (Oct 2013) Grid Infrastructure Patch Set Update (GI PSU) (Doc ID 17272753.8) - This is the latest PSU on top 11..2.0.2

Please download the patch and follow the below steps:

Make sure that you stopped the EM agent Processes running from the database home before patching the Oracle RAC or Oracle RAC one Node database or grid infrastructure home.
To Stop the EM agent processes  please execute:
<ORACLE_HOME>/bin/emctl stop dbconsole

Execute the following on each node of the cluster to apply the patch:
Stop the CRS managed resources running from DB homes.
As the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>
In case of Oracle Restart Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location>
1.    Run the pre root script.
In the   GI Home, as the root user execute:
# <GI_HOME>/crs/install/rootcrs.pl -unlock
In case of Oracle Restart Home, as the root user execute:
# <GI_HOME>/crs/install/roothas.pl -unlock
2.   Apply the CRS patch using.
As the GI home owner execute:
$ <GI_HOME>/OPatch/opatch napply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/16459322
As the GI home owner execute:
$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/17082367

4-Run the pre script for DB component of the patch.
As the database home owner execute:
$ <UNZIPPED_PATCH_LOCATION>/16459322/custom/server/16459322/custom/scripts/prepatch.sh -dbhome <ORACLE_HOME>
3-Apply the DB patch.
As the database home owner execute:
$ <ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/16459322/custom/server/16459322
$ <ORACLE_HOME>/OPatch/opatch apply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/17082367
4-Run the post script for DB component of the patch.
As the database home owner execute:
$ <UNZIPPED_PATCH_LOCATION>/16459322/custom/server/16459322/custom/scripts/postpatch.sh -dbhome <ORACLE_HOME>
5-Run the post script.
As the root user execute:
# <GI_HOME>/rdbms/install/rootadd_rdbms.sh
In  the GI Home, as the root user execute:
# <GI_HOME>/crs/install/rootcrs.pl -patch
In case of Oracle Restart Home, as the root user execute:
# <GI_HOME>/crs/install/roothas.pl -patch
6-Start the CRS managed resources that were earlier running from DB homes.
If this is a GI Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>
In case of Oracle Restart Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location>

Real Example:

1-The Pre Root Script.
root@svprtldb02:~# >/crs/install/rootcrs.pl -unlock
-bash: /crs/install/rootcrs.pl: No such file or directory
root@svprtldb02:~# cd /app/grid/
root@svprtldb02:/app/grid# cd crs/install/
root@svprtldb02:/app/grid/crs/install# ./rootcrs.pl -unlock
defined(@array) is deprecated at crsconfig_lib.pm line 2134.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2134.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2212.
(Maybe you should just omit the defined()?)
Using configuration parameter file: ./crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.diskmon' on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.gipcd' on 'svprtldb02'
CRS-2677: Stop of 'ora.cssdmonitor' on 'svprtldb02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'svprtldb02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'svprtldb02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'svprtldb02'
CRS-2677: Stop of 'ora.diskmon' on 'svprtldb02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'svprtldb02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'svprtldb02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /app/grid

2-Apply CRS Patch in order to patch the grid Infrastructure:
grid@svprtldb02:~/patchsetupdate$ /app/grid/OPatch/opatch napply -oh /app/grid -local /home/grid/patchsetupdate/16459322
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/grid
Central Inventory : /app/11.2.0/oraInventory
from : /app/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/grid/cfgtoollogs/opatch/opatch2014-02-09_19-47-10PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 16459322

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '16459322' to OH '/app/grid'
Patching component oracle.crs, 11.2.0.2.0...
Patching component oracle.usm, 11.2.0.2.0...
Verifying the update...
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
make: Warning: Too many rules defined for target libsrvm11.so
ld: warning: output object option (-o, --output) appears more than once, first setting taken
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver
Patch 16459322 successfully applied.
OPatch Session completed with warnings.
Log file location: /app/grid/cfgtoollogs/opatch/opatch2014-02-09_19-47-10PM_1.log
OPatch completed with warnings.
Based on the info given in the note Opatch Warning During Patch Apply : "ld: warning: option -YP appears more than once, first setting taken" (Doc ID 1571550.1) I will jump to next step.


grid@svprtldb02:~/patchsetupdate$ /app/grid/OPatch/opatch apply -oh /app/grid -local /home/grid/patchsetupdate/17082367
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/grid
Central Inventory : /app/11.2.0/oraInventory
from : /app/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/grid/cfgtoollogs/opatch/opatch2014-02-09_20-05-10PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 13696224 13923804 14275621 14727315 16056267 16619893 17082367
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '13696224' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] , [ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ] , [ oracle.network.cman, 11.2.0.2.0 ] , [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.ldap.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.sdo.locator, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '13923804' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.network.listener, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14275621' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.network.client, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.util, 11.2.0.2.0......
Patching component oracle.dbjava.jdbc, 11.2.0.2.0...
Patching component oracle.dbjava.ic, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14727315' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] , [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.util,

3-Make sure to run the pre database home patching step:
oracle@svprtldb02:~$ /home/oracle/patchsetupdate/16459322/custom/server/16459322/custom/scripts/prepatch.sh -dbhome /app/oracle/product/11.2.0/dbhome_1
Unable to determine value for ORACLE_BASE. Ignoring...
/home/oracle/patchsetupdate/16459322/custom/server/16459322/custom/scripts/prepatch.sh completed successfully.


4-Apply database patch:
oracle@svprtldb02:~$ /app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply -oh /app/oracle/product/11.2.0/dbhome_1 -local /home/grid/patchsetupdate/16459322/custom/server/16459322
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/oracle/product/11.2.0/dbhome_1
Central Inventory : /app/11.2.0/oraInventory
from : /app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-02-09_20-22-09PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 16459322
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/dbhome_1')
Is the local system ready for patching? [y|n]
Y
User Responded with: Y
Backing up files...
Applying interim patch '16459322' to OH '/app/oracle/product/11.2.0/dbhome_1'
Patching component oracle.rdbms, 11.2.0.2.0...
Verifying the update...
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
make: Warning: Too many rules defined for target libsrvm11.so
ld: warning: output object option (-o, --output) appears more than once, first setting taken
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver
Patch 16459322 successfully applied.
OPatch Session completed with warnings.
Log file location: /app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-02-09_20-22-09PM_1.log
OPatch completed with warnings.



grid@svprtldb02:~/patchsetupdate$ /app/grid/OPatch/opatch apply -oh /app/grid -local /home/grid/patchsetupdate/17082367
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/grid
Central Inventory : /app/11.2.0/oraInventory
from : /app/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/grid/cfgtoollogs/opatch/opatch2014-02-09_20-05-10PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 13696224 13923804 14275621 14727315 16056267 16619893 17082367
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '13696224' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] , [ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ] , [ oracle.network.cman, 11.2.0.2.0 ] , [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.ldap.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.sdo.locator, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '13923804' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.network.listener, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14275621' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.network.client, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.util, 11.2.0.2.0...
Patching component oracle.dbjava.jdbc, 11.2.0.2.0...
Patching component oracle.dbjava.ic, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14727315' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] , [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.util, 11.2.0.2.0...
Patching component oracle.sdo.locator, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '16056267' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.network.cman,


5-Make sure to apply post patch step:
/home/oracle/patchsetupdate/16459322/custom/server/16459322/custom/scripts/postpatch.sh -dbhome /app/oracle/product/11.2.0/dbhome_1
Reading /app/oracle/product/11.2.0/dbhome_1/install/params.ora..
Reading /app/oracle/product/11.2.0/dbhome_1/install/params.ora..
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/diskmon.bin
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/lsnodes
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/osdbagrp
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/rawutl
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/srvm/admin/ractrans
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/srvm/admin/getcrshome
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/gnsd
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/crsdiag.pl
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libhasgen11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libclsra11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libdbcfg11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libocr11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libocrb11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libocrutl11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libuini11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/librdjni11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libgns11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libgnsjni11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libagfw11.so
oracle@svprtldb02:~$

6-Run the post script.
In my case it was not progressing and hanged as per below and I got the below issue:
It is not progressing any more as below:
root@svprtldb02:/app/grid/crs/install# /usr/bin/perl rootcrs.pl -patch
defined(@array) is deprecated at crsconfig_lib.pm line 2194.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2194.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2272.
(Maybe you should just omit the defined()?)
Using configuration parameter file: ./crsconfig_params
Use of uninitialized value $u in concatenation (.) or string at /app/grid/lib/osds_acfslib.pm line 368.
/app/grid/bin/clsecho[270]: exec: /app/grid/bin/clsecho.bin: cannot execute [Permission denied]
/app/grid/bin/crsctl[270]: exec: /app/grid/bin/crsctl.bin: cannot execute [Permission denied]

I couldnot start the crs as per below:
root@svprtldb02:/app/grid/bin# ls -ltr crsctl.bin
-rw-r--r-- 1 grid oinstall 75558408 Apr 9 2013 crsctl.bin
root@svprtldb02:/app/grid/bin# chmod +x crsctl.bin
root@svprtldb02:/app/grid/bin# ls -ltr crsctl.bin
-rwxr-xr-x 1 grid oinstall 75558408 Apr 9 2013 crsctl.bin
root@svprtldb02:/app/grid/bin# ./crsctl stop crs -f
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
root@svprtldb02:/app/grid/bin# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.



[ohasd(1257)]CRS-5828:Could not start agent '/app/grid/bin/oraagent'. Details at (:CRSAGF00126:) {0:0:2} in /app/grid/log/svprtldb02/ohasd/ohasd.log.
2014-02-09 23:36:10.030
[ohasd(1257)]CRS-2758:Resource 'ora.gpnpd' is in an unknown state.
2014-02-09 23:45:18.905
[client(2986)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-09 23:45:36.249
[client(2986)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /app/grid/log/svprtldb02/client/ocrdump_2986.log.
2014-02-09 23:45:41.084
[client(3087)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
It was a permission issue and was fixed by doing the below:
ls -l GIHOME/bin

crsctl stop crs -f

cd GIHOME/bin
chmod 755

chmod 6751 oracle


Now start crs
root@svprtldb02:/app/grid/bin# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS 
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE 
ora.crf
1 ONLINE ONLINE svprtldb02 
ora.crsd
1 ONLINE OFFLINE 
ora.cssd
1 ONLINE OFFLINE STARTING 
ora.cssdmonitor
1 ONLINE ONLINE svprtldb02 
ora.ctssd
1 ONLINE OFFLINE 
ora.diskmon
1 ONLINE ONLINE svprtldb02 
ora.evmd
1 ONLINE OFFLINE 
ora.gipcd
1 ONLINE ONLINE svprtldb02 
ora.gpnpd
1 ONLINE ONLINE svprtldb02 
ora.mdnsd
1 ONLINE ONLINE svprtldb02

7-Stop and start the cluster :
Stopping and starting it should be working properly.Feel free to repeat the same steps on the other nodes.