SOLUTION
1. Start with both OCR copies in the correct state:
$ ls -l /ora10g/app/oracle/oradata/V10 lrwxrwxrwx 1 oracle dba 14 Jul 25 14:58 CrsVotingDisk -> /dev/raw/raw29 lrwxrwxrwx 1 oracle dba 14 Jul 25 14:58 OracleClusterRegistry -> /dev/raw/raw28 lrwxrwxrwx 1 oracle dba 14 Jul 25 14:58 OracleClusterRegistry2 -> /dev/raw/raw30
$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 200692 Used space (kbytes) : 3752 Available space (kbytes) : 196940 ID : 495185602 Device/File Name : /ora10g/app/oracle/oradata/V10/OracleClusterRegistry Device/File integrity check succeeded Device/File Name : /ora10g/app/oracle/oradata/V10/OracleClusterRegistry2 Device/File integrity check succeeded
Cluster registry integrity check succeeded
2. Corrupt the mirror copy for testing/demo purpose:
$ dd if=/dev/zero of=OracleClusterRegistry2 bs=1024k count=200
In reality, one might observe that the timestamp has not been modified for sometime compare with others:
root@ranode1:/root # ls -l /u0*/storage/* -rw-r—– 1 root dba 272756736 Apr 10 19:19 /u01/storage/ocr1 <<<<<<<<< -rw-r—– 1 root dba 272756736 May 14 15:14 /u02/storage/ocr2 -rw-r—– 1 root dba 272756736 May 14 15:14 /u03/storage/ocr3
3. Verify that ocrcheck reports the problem – mirror copy needs to be synchronized with the other device:
$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 200692 Used space (kbytes) : 3752 Available space (kbytes) : 196940 ID : 495185602 Device/File Name : /ora10g/app/oracle/oradata/V10/OracleClusterRegistry Device/File integrity check succeeded Device/File Name : /ora10g/app/oracle/oradata/V10/OracleClusterRegistry2 Device/File needs to be synchronized with the other device
Similar messages also reported for above example:
root@racnode1:/root # /grid/11.2.0.3/bin/ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3412 Available space (kbytes) : 258708 ID : 733779010 Device/File Name : /u01/storage/ocr1 Device/File needs to be synchronized with the other device <<<<<<<<<<< Device/File Name : /u02/storage/ocr2 Device/File integrity check succeeded Device/File Name : /u03/storage/ocr3 Device/File integrity check succeeded Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded
4. Correct the problem using ocrconfig with -replace ocrmirror option, as root user:
# ocrconfig -replace ocrmirror /ora10g/app/oracle/oradata/V10/OracleClusterRegistry2 PROT-22: Storage too small
This fails with “PROT-22: Storage too small” error. The problem is due to an Oracle bug where this operation requires the OCR mirror partition/file to be larger than the original by up to 128MB. The bug has been fixed in the 10.2.0.2 patchset.
To work around this error, increase the partition/filesize. If that is the partition (as it was in our case), resize it by 128MB (e.g. with fdisk), reboot all nodes in the cluster (so that all nodes recognize the new partition size) and then perform the replace. After the reboot, as root user issue:
# ocrconfig -replace ocrmirror /ora10g/app/oracle/oradata/V10/OracleClusterRegistry2
And check for the OCR status:
$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 200692 Used space (kbytes) : 3752 Available space (kbytes) : 196940 ID : 495185602 Device/File Name : /ora10g/app/oracle/oradata/V10/OracleClusterRegistry Device/File integrity check succeeded Device/File Name : /ora10g/app/oracle/oradata/V10/OracleClusterRegistry2 Device/File integrity check succeeded
Cluster registry integrity check succeeded
Now both OCR files in the correct state.
5. For 11.2 onwards, one can also use ocrconfig -delete and ocrconfig -add option to recreate the OCR mirror as root user, for example:
# ocrconfig -delete /u01/storage/ocr1 Add the ocr1 back using # ocrconfig -add /u01/storage/ocr1
Comments