How to Restore CRS after accidentally run localconfig on RAC system

For 10.2 and 11.1

Case I. “$ORACLE_HOME/bin/localconfig delete” has been done on some node(s) but not all of RAC system

Symptoms: no /etc/oracle/ocr.loc (or /var/opt/oracle/ocr.loc) exist on affected node(s)

The OCR has been removed in case it stands on a clustered file system. The OCRMIRROR still exist in case it has been defined. Scenario A. If both OCR and OCRMIRROR are on cluster file system

In this scenario, the OCRMIRROR can be used to recreate the OCR. As root user:

1. find the OCR location

grep “CRS_OCR_LOCATIONS=” $ORA_CRS_HOME/install/rootconfig |head -1

1. touch <OCR file> and set correct ownership and permission, this can be done on either node, for example:

touch /cfs/ocrdisk1 chown root:oinstall /cfs/ocrdisk1 chmod 640 /cfs/ocrdisk1

1. on one unaffected node, replace the ocr device: ocrconfig -replace ocr <OCR file name>

ocrconfig -replace ocr /cfs/ocrdisk1

1. on affected node(s) where “localconfig delete” has been run, as root user, rerun root.sh:

$ORA_CRS_HOME/root.sh

CRS will startup automatically after this.

Scenario B. If only OCR is used, no OCRMIRROR has been defined and it is on cluster file system

As the result of localconfig delete, crsd.bin will go down on the good node(s) as well, crsd.log reports:

[ OCRAPI][1225398592]procr_ctx_set_invalid_no_abort: ctx set to invalid [ OCRAPI][1225398592]procr_ctx_set_invalid: aborting…

The OCR need to be recreated from backup.

1. find OCR location

grep “CRS_OCR_LOCATIONS=” $ORA_CRS_HOME/install/rootconfig |head -1

1. touch <OCR file> and set correct ownership and permission

touch /cfs/ocrdisk1 chown root:oinstall /cfs/ocrdisk1 chmod 640 /cfs/ocrdisk1

1. look for latest OCR backup generated automatically

ocrconfig -showbackup

1. restore an OCR backup, eg:

ocrconfig -restore /u01/oracle/product/10.2.0/crs/cdata/crs/backup00.ocr

Once OCR restored, crsd.bin should start automatically on the good node(s). 5. on bad node(s), rerun $ORA_CRS_HOME/root.sh as root user.

$ORA_CRS_HOME/root.sh

CRS will startup automatically after this.

Scenario C. If OCR or OCRMIRROR is on raw device

localconfig delete will only remove the OCR device name (eg: /dev/raw/raw1), it will not remove the content of raw device. 1. restore the device name on Linux: as root, run:

service rawdevice restart

Please check with other OS vendor for restore the device name, or a reboot will restore the name too. 2. on bad node(s), rerun $ORA_CRS_HOME/root.sh as root user

$ORA_CRS_HOME/root.sh

CRS will startup automatically after this.

Case II. “$ORACLE_HOME/bin/localconfig reset” has been done on 1 or all nodes of RAC system Symptom: On the node(s) where “localconfig reset” has been run: 1. /etc/oracle/ocr.loc is like: ocrconfig_loc=<$ORACLE_HOME>/cdata/localhost/local.ocr local_only=TRUE

2. in /etc/inittab, following lines exist: h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null < CRS can not start on the affected node. Reboot will not help.

Solution: on all node(s) where “localconfig reset” has been done:

Caution! proceed with followings only when /etc/oracle/ocr.loc has not been manually modified to point back to real OCR. Eg, its content should still be:

ocrconfig_loc=<$ORACLE_HOME>/cdata/localhost/local.ocr local_only=TRUE

If you have modified ocr.loc to point to real OCR device, please restore the content back to above before continue with the followings. Otherwise the following commands will zero out the real OCR device. as root user: 1. run $ORA_CRS_HOME/install/rootdelete.sh, it will remove the init* scripts and place back the blank inittab

$ORA_CRS_HOME/install/rootdelete.sh

1. run  $ORA_CRS_HOME/install/rootdeinstall.sh, it will blank out the $ORACLE_HOME/cdata/localhost/local.ocr and remove the ocr.loc

$ORA_CRS_HOME/install/rootdeinstall.sh

1. run $ORA_CRS_HOME/root.sh, CRS should start automatically after this.

$ORA_CRS_HOME/root.sh

For all other cases, either restore the OCR from backup or follow Note.399482.1 How to recreate OCR/Voting disk accidentally deleted to reinitialize OCR and voting disks.

For 11.2+

case I “$ORACLE_HOME/bin/localconfig reset” has been done on some node(s) but not all of RAC system

Symptom: On the node(s) where “localconfig reset” has been run: 1. /etc/oracle/ocr.loc is like: ocrconfig_loc=<$GRID_HOME>/cdata/localhost/local.ocr local_only=TRUE

1. ps -ef |grep d.bin shows ohasd.bin is running, no other Grid Infrastructure processes are running.

2. Trying to start CRS on the affected node gets: # crsctl start crs CRS-4013: This command is not supported in a single-node configuration. CRS-4000: Command Start failed, or completed with errors.

Solution: on all node(s) where “localconfig reset” has been done:

As root user:

1. deconfig the existing GI configuration:

# $GRID_HOME/crs/install/rootcrs.pl -deconfig -force

1. rerun root.sh from GRID_HOME, GI stack should start up automatically at the end of root.sh

# $GRID_HOME/root.sh