Text box

Please note that the problems that i face may not be the same that exist on your environment so please test before applying the same steps i followed to solve the problem .

Monday 17 February 2014

Opatch 11.2.0.3 gives error when the opatch version higher than Grid Infrastructure 11.2.0.2 || applying patch set updates on GI_HOME

Hello,

While working to autopatch newly installed Standalone grid infrastructure the below error appeared.

Error:

root@svprtldbdev1:/app/11.2.0/grid/patchsetupdate# opatch auto .
Executing /app/grid/perl/bin/perl /app/grid/OPatch/crs/patch11203.pl -patchdir . -patchn . -paramfile /app/grid/crs/install/crsconfig_params
Bareword "ValidateCRSCTL" not allowed while "strict subs" in use at /app/grid/crs/install/crspatch.pm line 101.
Bareword "ValidateCRSCTL" not allowed while "strict subs" in use at /app/grid/crs/install/crspatch.pm line 112.
Compilation failed in require at /app/grid/OPatch/crs/patch11203.pl line 415.
BEGIN failed--compilation aborted at /app/grid/OPatch/crs/patch11203.pl line 415.

Note:
I was applying patch set update for 11.2.0.2 database and the patch number is p17272753_112020_SOLARIS64.zip.

workaround:

·       Grid Infrastructure Home:

$GRID_HOME/OPatch/opatch auto <UNZIPPED_PATCH_LOCATION> -oh <<Grid Home>> -olderver
·      The database Home:
 Please run below without olderver clause:

$GRID_HOME/OPatch/opatch auto <UNZIPPED_PATCH_LOCATION> -oh <<RDBMS Home>


The finding was that sometimes the higher version Opatch may cause the above error with higher version grid infrastructure.

Regards
Mohamed


Sunday 16 February 2014

Applying patch set update manually on Oracle Grid Infrastructure/Real Application cluster and Database 11.2.0.2.



The steps below helps you to manually apply patch set update for 11.2.0.2:
Patch Number:
Bug 17272753 - 11.2.0.2.12 (Oct 2013) Grid Infrastructure Patch Set Update (GI PSU) (Doc ID 17272753.8) - This is the latest PSU on top 11..2.0.2

Please download the patch and follow the below steps:

Make sure that you stopped the EM agent Processes running from the database home before patching the Oracle RAC or Oracle RAC one Node database or grid infrastructure home.
To Stop the EM agent processes  please execute:
<ORACLE_HOME>/bin/emctl stop dbconsole

Execute the following on each node of the cluster to apply the patch:
Stop the CRS managed resources running from DB homes.
As the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>
In case of Oracle Restart Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location>
1.    Run the pre root script.
In the   GI Home, as the root user execute:
# <GI_HOME>/crs/install/rootcrs.pl -unlock
In case of Oracle Restart Home, as the root user execute:
# <GI_HOME>/crs/install/roothas.pl -unlock
2.   Apply the CRS patch using.
As the GI home owner execute:
$ <GI_HOME>/OPatch/opatch napply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/16459322
As the GI home owner execute:
$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/17082367

4-Run the pre script for DB component of the patch.
As the database home owner execute:
$ <UNZIPPED_PATCH_LOCATION>/16459322/custom/server/16459322/custom/scripts/prepatch.sh -dbhome <ORACLE_HOME>
3-Apply the DB patch.
As the database home owner execute:
$ <ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/16459322/custom/server/16459322
$ <ORACLE_HOME>/OPatch/opatch apply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/17082367
4-Run the post script for DB component of the patch.
As the database home owner execute:
$ <UNZIPPED_PATCH_LOCATION>/16459322/custom/server/16459322/custom/scripts/postpatch.sh -dbhome <ORACLE_HOME>
5-Run the post script.
As the root user execute:
# <GI_HOME>/rdbms/install/rootadd_rdbms.sh
In  the GI Home, as the root user execute:
# <GI_HOME>/crs/install/rootcrs.pl -patch
In case of Oracle Restart Home, as the root user execute:
# <GI_HOME>/crs/install/roothas.pl -patch
6-Start the CRS managed resources that were earlier running from DB homes.
If this is a GI Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>
In case of Oracle Restart Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location>

Real Example:

1-The Pre Root Script.
root@svprtldb02:~# >/crs/install/rootcrs.pl -unlock
-bash: /crs/install/rootcrs.pl: No such file or directory
root@svprtldb02:~# cd /app/grid/
root@svprtldb02:/app/grid# cd crs/install/
root@svprtldb02:/app/grid/crs/install# ./rootcrs.pl -unlock
defined(@array) is deprecated at crsconfig_lib.pm line 2134.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2134.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2212.
(Maybe you should just omit the defined()?)
Using configuration parameter file: ./crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.diskmon' on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'svprtldb02'
CRS-2673: Attempting to stop 'ora.gipcd' on 'svprtldb02'
CRS-2677: Stop of 'ora.cssdmonitor' on 'svprtldb02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'svprtldb02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'svprtldb02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'svprtldb02'
CRS-2677: Stop of 'ora.diskmon' on 'svprtldb02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'svprtldb02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'svprtldb02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /app/grid

2-Apply CRS Patch in order to patch the grid Infrastructure:
grid@svprtldb02:~/patchsetupdate$ /app/grid/OPatch/opatch napply -oh /app/grid -local /home/grid/patchsetupdate/16459322
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/grid
Central Inventory : /app/11.2.0/oraInventory
from : /app/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/grid/cfgtoollogs/opatch/opatch2014-02-09_19-47-10PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 16459322

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '16459322' to OH '/app/grid'
Patching component oracle.crs, 11.2.0.2.0...
Patching component oracle.usm, 11.2.0.2.0...
Verifying the update...
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
make: Warning: Too many rules defined for target libsrvm11.so
ld: warning: output object option (-o, --output) appears more than once, first setting taken
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver
Patch 16459322 successfully applied.
OPatch Session completed with warnings.
Log file location: /app/grid/cfgtoollogs/opatch/opatch2014-02-09_19-47-10PM_1.log
OPatch completed with warnings.
Based on the info given in the note Opatch Warning During Patch Apply : "ld: warning: option -YP appears more than once, first setting taken" (Doc ID 1571550.1) I will jump to next step.


grid@svprtldb02:~/patchsetupdate$ /app/grid/OPatch/opatch apply -oh /app/grid -local /home/grid/patchsetupdate/17082367
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/grid
Central Inventory : /app/11.2.0/oraInventory
from : /app/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/grid/cfgtoollogs/opatch/opatch2014-02-09_20-05-10PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 13696224 13923804 14275621 14727315 16056267 16619893 17082367
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '13696224' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] , [ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ] , [ oracle.network.cman, 11.2.0.2.0 ] , [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.ldap.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.sdo.locator, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '13923804' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.network.listener, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14275621' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.network.client, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.util, 11.2.0.2.0......
Patching component oracle.dbjava.jdbc, 11.2.0.2.0...
Patching component oracle.dbjava.ic, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14727315' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] , [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.util,

3-Make sure to run the pre database home patching step:
oracle@svprtldb02:~$ /home/oracle/patchsetupdate/16459322/custom/server/16459322/custom/scripts/prepatch.sh -dbhome /app/oracle/product/11.2.0/dbhome_1
Unable to determine value for ORACLE_BASE. Ignoring...
/home/oracle/patchsetupdate/16459322/custom/server/16459322/custom/scripts/prepatch.sh completed successfully.


4-Apply database patch:
oracle@svprtldb02:~$ /app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply -oh /app/oracle/product/11.2.0/dbhome_1 -local /home/grid/patchsetupdate/16459322/custom/server/16459322
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/oracle/product/11.2.0/dbhome_1
Central Inventory : /app/11.2.0/oraInventory
from : /app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-02-09_20-22-09PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 16459322
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/oracle/product/11.2.0/dbhome_1')
Is the local system ready for patching? [y|n]
Y
User Responded with: Y
Backing up files...
Applying interim patch '16459322' to OH '/app/oracle/product/11.2.0/dbhome_1'
Patching component oracle.rdbms, 11.2.0.2.0...
Verifying the update...
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
make: Warning: Too many rules defined for target libsrvm11.so
ld: warning: output object option (-o, --output) appears more than once, first setting taken
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver
Patch 16459322 successfully applied.
OPatch Session completed with warnings.
Log file location: /app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-02-09_20-22-09PM_1.log
OPatch completed with warnings.



grid@svprtldb02:~/patchsetupdate$ /app/grid/OPatch/opatch apply -oh /app/grid -local /home/grid/patchsetupdate/17082367
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /app/grid
Central Inventory : /app/11.2.0/oraInventory
from : /app/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.2.0
Log file location : /app/grid/cfgtoollogs/opatch/opatch2014-02-09_20-05-10PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 13696224 13923804 14275621 14727315 16056267 16619893 17082367
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/app/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '13696224' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] , [ oracle.rdbms.dv, 11.2.0.2.0 ] , [ oracle.sysman.plugin.db.main.repository, 11.2.0.2.0 ] , [ oracle.network.cman, 11.2.0.2.0 ] , [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.ldap.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.sdo.locator, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '13923804' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.network.listener, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14275621' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.network.client, 11.2.0.2.0...
Patching component oracle.network.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.util, 11.2.0.2.0...
Patching component oracle.dbjava.jdbc, 11.2.0.2.0...
Patching component oracle.dbjava.ic, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '14727315' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.2.0 ] , [ oracle.precomp.lang, 11.2.0.2.0 ] , [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.3 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.rdbms, 11.2.0.2.0...
Patching component oracle.rdbms.rman, 11.2.0.2.0...
Patching component oracle.rdbms.rsf, 11.2.0.2.0...
Patching component oracle.rdbms.util, 11.2.0.2.0...
Patching component oracle.sdo.locator, 11.2.0.2.0...
Verifying the update...
Applying sub-patch '16056267' to OH '/app/grid'
ApplySession: Optional component(s) [ oracle.network.cman,


5-Make sure to apply post patch step:
/home/oracle/patchsetupdate/16459322/custom/server/16459322/custom/scripts/postpatch.sh -dbhome /app/oracle/product/11.2.0/dbhome_1
Reading /app/oracle/product/11.2.0/dbhome_1/install/params.ora..
Reading /app/oracle/product/11.2.0/dbhome_1/install/params.ora..
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Parsing file /app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Verifying file /app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/racgwrap
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/srvctl
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/srvconfig
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/diskmon.bin
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/lsnodes
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/osdbagrp
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/rawutl
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/srvm/admin/ractrans
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/srvm/admin/getcrshome
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/gnsd
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/bin/crsdiag.pl
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libhasgen11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libclsra11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libdbcfg11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libocr11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libocrb11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libocrutl11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libuini11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/librdjni11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libgns11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libgnsjni11.so
Reapplying file permissions on /app/oracle/product/11.2.0/dbhome_1/lib/libagfw11.so
oracle@svprtldb02:~$

6-Run the post script.
In my case it was not progressing and hanged as per below and I got the below issue:
It is not progressing any more as below:
root@svprtldb02:/app/grid/crs/install# /usr/bin/perl rootcrs.pl -patch
defined(@array) is deprecated at crsconfig_lib.pm line 2194.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2194.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at crsconfig_lib.pm line 2272.
(Maybe you should just omit the defined()?)
Using configuration parameter file: ./crsconfig_params
Use of uninitialized value $u in concatenation (.) or string at /app/grid/lib/osds_acfslib.pm line 368.
/app/grid/bin/clsecho[270]: exec: /app/grid/bin/clsecho.bin: cannot execute [Permission denied]
/app/grid/bin/crsctl[270]: exec: /app/grid/bin/crsctl.bin: cannot execute [Permission denied]

I couldnot start the crs as per below:
root@svprtldb02:/app/grid/bin# ls -ltr crsctl.bin
-rw-r--r-- 1 grid oinstall 75558408 Apr 9 2013 crsctl.bin
root@svprtldb02:/app/grid/bin# chmod +x crsctl.bin
root@svprtldb02:/app/grid/bin# ls -ltr crsctl.bin
-rwxr-xr-x 1 grid oinstall 75558408 Apr 9 2013 crsctl.bin
root@svprtldb02:/app/grid/bin# ./crsctl stop crs -f
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
root@svprtldb02:/app/grid/bin# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.



[ohasd(1257)]CRS-5828:Could not start agent '/app/grid/bin/oraagent'. Details at (:CRSAGF00126:) {0:0:2} in /app/grid/log/svprtldb02/ohasd/ohasd.log.
2014-02-09 23:36:10.030
[ohasd(1257)]CRS-2758:Resource 'ora.gpnpd' is in an unknown state.
2014-02-09 23:45:18.905
[client(2986)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-09 23:45:36.249
[client(2986)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /app/grid/log/svprtldb02/client/ocrdump_2986.log.
2014-02-09 23:45:41.084
[client(3087)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
It was a permission issue and was fixed by doing the below:
ls -l GIHOME/bin

crsctl stop crs -f

cd GIHOME/bin
chmod 755

chmod 6751 oracle


Now start crs
root@svprtldb02:/app/grid/bin# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS 
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE 
ora.crf
1 ONLINE ONLINE svprtldb02 
ora.crsd
1 ONLINE OFFLINE 
ora.cssd
1 ONLINE OFFLINE STARTING 
ora.cssdmonitor
1 ONLINE ONLINE svprtldb02 
ora.ctssd
1 ONLINE OFFLINE 
ora.diskmon
1 ONLINE ONLINE svprtldb02 
ora.evmd
1 ONLINE OFFLINE 
ora.gipcd
1 ONLINE ONLINE svprtldb02 
ora.gpnpd
1 ONLINE ONLINE svprtldb02 
ora.mdnsd
1 ONLINE ONLINE svprtldb02

7-Stop and start the cluster :
Stopping and starting it should be working properly.Feel free to repeat the same steps on the other nodes.

Wednesday 12 February 2014

Prerequisite for installaing 11g RAC/grid infrastructure Release 2 on solaries-64 bit operating system

Creating Job Role Separation Operating System Privileges Groups and Users:
Run the following commands:

/usr/sbin/groupadd -g 1020 asmadmin

/usr/sbin/groupadd -g 1022 asmoper

/usr/sbin/groupadd -g 1021 asmdba

/usr/sbin/groupadd -g 1032 oper

/usr/sbin/useradd -u 1100 -g oinstall -G dba grid

/usr/sbin/useradd -u 1101 -g oinstall -G dba,asmdba oracle
/usr/sbin/usermod -g oinstall -G dba,asmdba oracle
mkdir -p  /app/11.2.0/grid
mkdir -p /app/grid
mkdir -p /app/oracle 
chown grid:oinstall /app/11.2.0
chown grid:oinstall /app/11.2.0/grid
chown grid:oinstall /app/grid
chown oracle:oinstall /app/oracle

Configuring Shell Limits:
Recommended Value
Shell STACK
32768 as Minimum value
Shell NOFILES
4096 as Minimum value
DATA
1048576 as Minimum value
VMEMORY
16777216 can be increased
TIME
-1 (Unlimited)
FILE
-1 (Unlimited)










Software Requirements List for Solaris Operating System (SPARC 64-Bit) Platforms:

Item
Requirements
Operating System, Packages and patches for Oracle Solaris 11
Oracle Solaris 11 (11/2011 SPARC) or later, for Oracle Grid
Infrastructure release 11.2.0.3 or later.
pkg://solaris/developer/build/make
pkg://solaris/developer/assembler
No special kernel parameters or patches are required at this time.
Operating System, Packages and Patches for Oracle Solaris 10
Oracle Solaris 10 U6 (5.10-2008.10)
SUNWarc
SUNWbtool
SUNWcsl
SUNWhea
SUNWi1cs (ISO8859-1)
SUNWi15cs (ISO8859-15)
SUNWi1of
SUNWlibC
SUNWlibm
SUNWlibms
SUNWsprot
SUNWtoo
SUNWxwfnt
119963-14: Sun OS 5.10: Shared Library Patch for C++
120753-06: SunOS 5.10: Microtasking libraries (libmtsk)
patch
139574-03: SunOS 5.10
141414-02
141414-09 (11.2.0.2 or later)
146808-01 (for Solaris 10 U9 or earlier)
Database Smart Flash Cache (An Enterprise Edition only feature.)
The following patches are required for Oracle Solaris (SPARC
64-Bit) if you are using the flash cache feature:
125555-03
139555-08
140796-01
140899-01
141016-01
141414-10
141736-05
IPMI
The following patches are required only if you plan to configure Failure Isolation using IPMI on SPARC systems:
137585-05 or later (IPMItool patch)
137594-02 or later (BMC driver patch)
Oracle RAC
Oracle Clusterware is required; Oracle Solaris Cluster is supported
for use with Oracle RAC on SPARC. If you use Oracle Solaris Cluster 3.2, then you must install the following additional kernel
packages and patches:
SUNWscucm 3.2.0: 126106-40
VERSION=3.2.0,REV=2006.12.05.22.58 or later
125508-08
125514-05
125992-04
126047-11
126095-05
126106-33
Note: You do not require the additional packages if you are using
Oracle Clusterware only, without Oracle Solaris Cluster.
If you use a volume manager, then you may need to install additional kernel packages.
Packages and patches for Oracle Solaris Cluster
Note: You do not require Oracle Solaris Cluster to install Oracle
Clusterware.
For Oracle Solaris 11, Oracle Solaris Cluster 4.0 is the minimum supported Oracle Solaris Cluster version.
For Oracle Solaris 10, Oracle Solaris Cluster 3.3 or later
UDLM (optional):
ORCLudlm 64-Bit reentrant 3.3.4.10
CAUTION: If you install the ORCLudlm package, then it is detected
automatically and used. Install ORCLudlm only if you want to use the
UDLM interface for your Oracle RAC cluster. Oracle recommends
with Oracle Solaris Cluster 3.3 and later that you use the native
cluster membership functionality provided with Oracle Solaris
Cluster.
For more information, refer to Section 2.8, "Oracle Solaris Cluster
Configuration on SPARC Guidelines."
For Oracle Solaris Cluster on SPARC, install UDLM onto each node
in the cluster using the patch Oracle provides in the Grid_
home/clusterware/udlm directory before installing and configuring
Oracle RAC. Although you may have a functional version of the
UDLM from a previous Oracle Database release, you must install
the Oracle 11g release 2 (11.2) 3.3.4.10 UDLM.
Oracle Messaging Gateway
Oracle Messaging Gateway supports the integration of Oracle Streams Advanced Queuing (AQ) with the following software:
IBM MQSeries V6 (6.6.0), client and server Tibco Rendezvous 7.2
Pro*C/C++,
Oracle Call Interface,
Oracle C++ Call Interface,
Oracle XML Developer's Kit
(XDK)
Oracle Solaris Studio 12 (formerly Sun Studio) (C and C++ 5.9)
119963-14: SunOS 5.10: Shared library patch for C++
124863-12 C++ SunOS 5.10 Compiler Common patch for Sun C C++
(optional)

Oracle ODBC Driver


gcc 3.4.2
Open Database Connectivity (ODBC) packages are only needed if
you plan on using ODBC. If you do not plan to use ODBC, then you
do not need to install the ODBC RPMs for Oracle Clusterware,
Oracle ASM, or Oracle RAC.

Programming languages for
Oracle RAC database
 Pro*COBOL
Micro Focus Server Express 5.1
 Pro*FORTRAN
Oracle Solaris Studio 12 (Fortran 95)
Download at the following URL:
http://www.oracle.com/technetwork/server-storage/solarisst
udio/overview/index.html
Oracle JDBC/OCI Drivers
You can use the following optional JDK versions with the Oracle
JDBC/OCI drivers, however they are not required for the
installation:
 JDK 6 Update 20 (JDK6 - 1.6.20) or later
 JDK 5 (1.5.0_24) or later
Note: JDK 6 is the minimum level of JDK supported on Oracle
Solaris 11.
SSH
Oracle Clusterware requires SSH. The required SSH software is the default SSH shipped with your operating system.


Operating System Kernel Requirements:

Verifying UDP and TCP Kernel Parameters
/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000



/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500



/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000



/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500



Storage Requirements:
Item
Requirement
/tmp
To be increased to 3G
Swap
Available RAM Swap Space Required
If RAM Between 2.5 GB and 16 GB Then Equal to the size of RAM
If RAM More than 16 GB Then Equal to 16 GB
3 volume groups
5G for each
and run the following for each disk where dsk1ocr is an example for the disk:
chown grid:asmadmin dsk1ocr
chmod 660 dsk1ocr
Or clusterfile system can be used instead.
/app
To be increased to 150G


Checking Resource Limits for Solaris
On Solaris platforms, the /etc/pam.conf file controls and limits resources for users on the system. On login, control and limit resources should be set for users on the system so that users are unable to perform denial of service attacks.
By default, PAM resource limits are not set for Solaris operating systems. To ensure that resource limits are honored, add the following line to the login service section of /etc/pam/conf: login auth required pam_dial_auth.so.1
For example:
# login service (explicit because of pam_dial_auth)
#
login auth requisite pam_authtok_get.so.1
login auth required pam_dhkeys.so.1
login auth required pam_unix_cred.so.1
login auth required pam_unix_auth.so.1
login auth required pam_dial_auth.so.1



Network Time Protocol Setting
Oracle Clusterware requires the same time zone setting on all cluster nodes. During installation, the installation process picks up the time zone setting of the Grid installation owner on the node where OUI runs, and uses that on all nodes as the default TZ setting for all processes managed by Oracle Clusterware. This default is used for databases, Oracle ASM, and any other managed processes.You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service. Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server.On Oracle Solaris Cluster systems, Oracle Solaris Cluster software supplies a template file called ntp.cluster (see /etc/inet/ntp.cluster on an installed cluster host) that establishes a peer relationship between all cluster hosts. One host is designated as the preferred host. Hosts are identified by their private host names. Time synchronization occurs across the cluster interconnect. If Oracle Clusterware detects either that the Oracle Solaris Cluster NTP or an outside NTP server is set default NTP server in the system in the /etc/inet/ntp.conf or the /etc/inet/ntp.conf.cluster files, then CTSS is set to the observer mode.

Note: Before starting the installation of the Oracle Grid Infrastructure, Oracle recommends that you ensure the clocks on all nodes are set to the same time.If you have NTP daemons on your server but you cannot configure them to synchronize time with a time server, and you want to use Cluster Time
Synchronization Service to provide synchronization service in the cluster, then deactivate and deinstall the NTP.
To disable the NTP service, run the following command as the root user
# /usr/sbin/svcadm disable ntp
When the installer finds that the NTP protocol is not active, the Cluster Time Synchronization Service is installed in active mode and synchronizes the time across the nodes.
To confirm that ctssd is active after installation, enter the following command as the
Grid installation owner:
$ crsctl check ctss
If you are using NTP, and you prefer to continue using it instead of Cluster Time
Synchronization Service, then you need to modify the NTP initialization file to enable
slewing, which prevents time from being adjusted backward. Restart the network time
protocol daemon after you complete this task. To do this on Oracle Solaris without Oracle Sun Cluster, edit the /etc/inet/ntp.conf file to add "slewalways yes" and "disable pll" to the file. After you make these changes, restart ntpd (on Oracle Solaris 11) or xntpd (on Oracle Solaris 10) using the command /usr/sbin/svcadm restart ntp.
To do this on Oracle Solaris 11 with Oracle Solaris Sun Cluster 4.0, edit the /etc/inet/ntp.conf.sc file to add "slewaways yes" and "disablepll" to the file. After you make these changes, restart ntpd or xntpd using the command  /usr/sbin/svcadmn restart ntp. To do this on Oracle Solaris 10 with Oracle Sun
Cluster 3.2, edit the /etc/inet/ntp.conf.cluster file.
To enable NTP after it has been disabled, enter the following command:
# /usr/sbin/svcadm enable ntp


Automatic SSH Configuration During Installation
To install Oracle software, Secure Shell (SSH) connectivity should be set up between all cluster member nodes. OUI uses the ssh and scp commands during installation to run remote commands on and copy files to the other cluster nodes. You must configure SSH so that these commands do not prompt for a password.

Note: SSH is used by Oracle configuration assistants for configuration operations from local to remote nodes. It is also used by Oracle Enterprise Manager. You can configure SSH from the OUI interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.
To enable the script to run, you must remove stty commands from the profiles of any Oracle software installation owners, and remove other security measures that are triggered during a login, and that generate messages to the terminal. These messages, mail checks, and other displays prevent Oracle software installation owners from using the SSH configuration script that is built into the Oracle Universal Installer. If they are not disabled, then SSH must be configured manually before an installation
can be run.
By default, OUI searches for SSH public keys in the directory /usr/local/etc/, and ssh-keygen binaries in /usr/local/bin. However, on Oracle Solaris, SSH public keys typically are located in the path /etc/ssh, and ssh-keygen binaries are located in the path /usr/bin. To ensure that OUI can set up SSH, use the following command to
create soft links:
# ln -s /etc/ssh /usr/local/etc
# ln -s /usr/bin /usr/local/bin
In rare cases, Oracle Clusterware installation may fail during the "AttachHome" operation when the remote node closes the SSH connection. To avoid this problem, set the following parameter in the SSH daemon configuration file /etc/ssh/sshd_config on all cluster nodes to set the timeout wait to unlimited:
LoginGraceTime 0

Network Requirements:
1.      SCAN VIP IPs:
     a.      Need 3 IPs used for scan VIP with single name for them.
      b.      SCAN is a domain name registered to three IP addresses in the  
                domain name service (DNS). The SCAN name (a domain name)  
                 must be set up to round robin across 3 IP addresses. This
                  requires a SCAN name resolution via domain name service
                  (DNS).

          c.      Cluster Name: portalcluster
          d.      SCAN Name:  Portalcrsscan which will have the three IPs.
2.      One IP for single database.
3.      Private Interconnect:
        a.      Each interface on single machine has to be on different VLAN.
        b.      Interface Name should be identical on all nodes and within the
                  same VLAN.

Issues that you may face during 11.2.0.2  installation:
·      
Both search and domain entries are present in file "/etc/resolv.conf" with the same value will cause the cluster verify utility to fail during checking the prerequisites. You can guarantee everything is working properly by running cluster verify utility before the installation.
·      
Swap Space should be handled in a smart way and as per the oracle documentation requirements for example if you will assign 16G memory then swap space should be at least 16G.The cluster verify utility will calculate the needed swap and will let you know if you must change the swap space assigned in case you run it in verbose mode.

·       Please note that "project.max-shm-memory" represent the maximum shared memory available for a project, so the value for this parameter should be greater than sum of all SGA sizes for all databases in your environment. 

·       The last step of cluster verify utility failed with below error:

Errors &issues appeared during last step of installing Grid Infrastrcuture:

INFO: Liveness check failed for "xntpd"
INFO: Check failed on nodes:
INFO: svprtldb02,svprtldb01
INFO: PRVF-5494 : The NTP Daemon or Service was not alive on all nodes
INFO: PRVF-5415 : Check to see if NTP daemon or service is running failed
INFO: Clock synchronization check using Network Time Protocol(NTP) failed
INFO: PRVF-9652 : Cluster Time Synchronization Services check failed
INFO: Checking VIP configuration.
INFO: Checking VIP Subnet configuration.
INFO: Check for VIP Subnet configuration passed.
INFO: Checking VIP reachability
INFO: Check for VIP reachability passed.
INFO: Starting check for The SSH LoginGraceTime setting ...
INFO: WARNING:
INFO: PRVE-0038 : The SSH LoginGraceTime setting on node "svprtldb02" may result in users being disconnected before login is completed
INFO: PRVE-0038 : The SSH LoginGraceTime setting on node "svprtldb01" may result in users being disconnected before login is completed

The solution was to disable the NTP during the installation and then enable it back after installation because of the process CTSS  will be in observer mode.Please remove or remove /etc/ntp.conf or /etc/xntp.conf from all nodes and check no ntp daemon is running.

After disabling the NTDP process run the below:

grid@svprtldb01:/apps/grid/clusterverify/bin$ ./cluvfy comp clocksync -n svprtldb01,svprtldb02

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
CTSS resource check passed


Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP configuration file "/etc/inet/ntp.conf" existence check passed
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
Check of common NTP Time Server passed
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed


Oracle Cluster Time Synchronization Services check passed

Verification of Clock Synchronization across the cluster nodes was successful.



·        
I got the below errors as the disks were mounted to one node but refused to mount to the other node .The first question came to my mind if the disks permissions are set correctly on all nodes but disks permissions were ok.

Errors & issues:
======================================================
Sat Feb 08 23:23:52 2014
Starting background process RSMN
Sat Feb 08 23:23:52 2014
RSMN started with pid=34, OS id=12977
Sat Feb 08 23:23:52 2014
Sweep [inc][217]: completed
Sweep [inc][177]: completed
Sweep [inc2][217]: completed
Sweep [inc2][177]: completed
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Sat Feb 08 23:23:54 2014
ALTER DATABASE MOUNT
NOTE: Loaded library: System
ORA-15025: could not open disk "/dev/rdsk/c3d17s0"
ORA-27041: unable to open file
SVR4 Error: 13: Permission denied
Additional information: 9
SUCCESS: diskgroup SYSTEM_WEB was mounted
Errors in file /app/oracle/diag/rdbms/webprd/WEBPRD2/trace/WEBPRD2_ckpt_12943.trc (incident=
4177):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [
], [], [], []
Incident details in: /app/oracle/diag/rdbms/webprd/WEBPRD2/incident/incdir_4177/WEBPRD2_ckpt_12943_i4177.trc

To solve it follow the below Oracle  note:

Implement the solution provided in oracle note :ORA-00600 [kfioTranslateIO03] [17090] (Doc ID 1336846.1)

·        
After installation was successful i noticed that the process ora.crf status is not started  on both nodes. Seems that it was a bug in 11.2.0.2 and applying patchset update 12 solved that issue.

Errors & Issues:
================================================
root@svprtldb02:/app/grid/bin# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE svprtldb02 Started
ora.crf
1 ONLINE OFFLINE
ora.crsd
1 ONLINE ONLINE svprtldb02
ora.cssd
1 ONLINE ONLINE svprtldb02
ora.cssdmonitor
1 ONLINE ONLINE svprtldb02
ora.ctssd
1 ONLINE ONLINE svprtldb02 OBSERVER
ora.diskmon
1 ONLINE ONLINE svprtldb02
ora.evmd
1 ONLINE ONLINE svprtldb02
ora.gipcd
1 ONLINE ONLINE svprtldb02
ora.gpnpd
1 ONLINE ONLINE svprtldb02
ora.mdnsd
1 ONLINE ONLINE svprtldb02

After successful implementation of the patch set update the issue was fixed.

·       After successful installation of grid infrastructure and databases a node was rebooted for maintenance by System admin team and after startup the setup of the private interconnect was removed and required them to re-setup it again.
Also everything was ok and we started the node the issue was that there was a bug with the 11.2.0.2 and this bug was fixed in 11.2.0.3 patchset updates.

Symptoms:

GIPCD
-----
Line 16: 2014-02-10 05:02:04.504: [GIPCDMON][7] gipcdMonitorSaveInfMetrics: inf[ 0] ipmp0 - rank -1, avgms 30000000000.000000 [ 0 / 0 / 0 ]
...
Line 63636: 2014-02-10 12:42:25.549: [GIPCDMON][7] gipcdMonitorSaveInfMetrics: inf[ 0] ipmp0 - rank -1, avgms 30000000000.000000 [ 0 / 0 / 0 ]

Oracle Support notes regarding the issues:
Doc ID 1479380.1: 11gR2 GI Node May not Join the Cluster After Private Network is Functional After Eviction due to Private Network Problem 

For patches in 11.2.0.3 and above I kindly refer to Doc ID 1479380.1



A workaround for this bug follow the below:
a)   On surviving node, during non-peak time if possible, kill gipcd.bin process (kill -15 <gipcd.bin ospid>)
NOTE: In 11.2 will also lead to death of evmd.bin, crsd.bin and ctssd.bin processes. None of these processes are fatal. Clusterware will respawn all them automatically.

b)  Once gipcd.bin, evmd.bin, crsd.bin and ctssd.bin processes have been re-spawned on the surviving node, verify whether other nodes join cluster.

c)   Most of the time, GI will start, but in case it does not, re-start GI on the other nodes with crsctl command.

d)  Finally, if GI is still not starting, as a last resort, restart GI on the surviving node.







General Notes:
To check the RAC network issues, please upload the following:

1) Please upload results:

$GRID_HOME/bin/srvctl config network

$GRID_HOME/bin/srvctl config nodeapps -a

$GRID_HOME/bin/srvctl config scan

2) ++Kindly get the cluvfy utility output:

cluvfy stage -pre crsinst -n [nodelist] -verbose

References:
CTSSD Runs in Observer Mode Even Though No Time Sync Software is Running (Doc ID 1054006.1)
ORA-00600 [kfioTranslateIO03] [17090] (Doc ID 1336846.1).
ASM Metadata information using:Note:470211.1 "How To Gather/Backup ASM Metadata In A Formatted Manner?".
How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation (Doc ID 942166.1