Problem Statement:
Today, I encountered interesting but weird problem while successfully applying Patch 30501932 - GI Jan 2020 Release Update 12.2.0.1.200114 on node-1 when I tried to shutdown immediate for instance -2 for applying patch, it did not shutdown and session and was keep waiting for shutdown so I had to abnormally stop shutdown and issued shutdown abort, but post shutdown when I tried to start up instance-2, it throw below set of errors:
LRM-00109: could not open parameter file '/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/inittestdb2.ora'
SQL> exit
Created pfile from SPFILE on node-1 and copied to node-2 and tried starting instance 2 using newly created pfile:
SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/inittestdb2.ora'
SQL>
SQL> startup pfile='/tmp/inittestdb.ora';
ORACLE instance started.
Total System Global Area 5.5298E+10 bytes
Fixed Size 29964104 bytes
Variable Size 8724154552 bytes
Database Buffers 4.6439E+10 bytes
Redo Buffers 104251392 bytes
ORA-00205: error in identifying control file, check alert log for more info
Below is the list of Error reported in alert.log file:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: failed to register ASMB0 with ASM instance
2020-01-30T14:32:27.133429+03:00
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb2/trace/testdb2_asmb_31261082.trc:
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
WARNING: ASMB0 exiting with error
2020-01-30T14:32:27.137941+03:00
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/testdb/CONTROLFILE/current.261.1030897861'
ORA-17503: ksfdopn:2 Failed to open file +DATA/testdb/CONTROLFILE/current.261.1030897861
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/testdb/CONTROLFILE/current.262.1030897861'
ORA-17503: ksfdopn:2 Failed to open file +DATA/testdb/CONTROLFILE/current.262.1030897861
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
2020-01-30T14:32:27.145437+03:00
ORA-205 signalled during: ALTER DATABASE MOUNT...
2020-01-30T14:44:12.824562+03:00
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Issue Observation:
I observed that even though +ASM2 was up and all disks were mounted on node-2 still Instance-2 was not able to access diskgroups and hence upon DB startup it could not access SPFILE located on ASM diskgroup and hence oracle tried to start Instance with default PFILE but still it couldn't start as controlfile was not accessible due to diskgroup accessibility issue.
Further more Observed that after successfully applying GI Jan 2020 PATCH on node-1, even without applying patch on node-2, it had changed setuid permission for GI oracle binary file on node-2 and because of that Oracle was not able to access diskgroup upon instance startup, please find below evidence:
setuid permission for owner and group is changed from s to x and that's the issue.
On Node-2:
oracle@TESTHOST02:[/u01/app/12.2.0/grid/bin]$ls -rlta |grep oracle
-rw------- 1 grid oinstall 579571814 Jan 27 13:45 oracleO
-rwxr-x--x 1 grid oinstall 579571814 Jan 27 14:29 oracle
oracle@TESTHOST02:[/u01/app/12.2.0/grid/bin]$
Please observe setuid for GI Oracle binary on node-1, which is proper.
On Node-1:
===================
root@TESTHOST01:[/]#ls -lrta /u01/app/12.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 583710697 Jan 30 12:27 /u01/app/12.2.0/grid/bin/oracle
root@TESTHOST01:[/]#
Solution:
Perform below steps on node-2, (affected node), which should fix the issue and Instance on node-2 should come up just fine without any issue.
1.shutdown CRS as root
root@TESTHOST02:[/]#/u01/app/12.2.0/grid/bin/crsctl stop crs
2. Set setuid for Grid oracle binary.
root@TESTHOST02:[/u01/app/12.2.0/grid/bin]$ls -rlta |grep oracle
-rwxr-x--x 1 grid oinstall 579571814 Jan 27 14:29 oracle
root@TESTHOST02:[/]#chmod 6751 /u01/app/12.2.0/grid/bin/oracle
root@TESTHOST02:[/]#ls -lrta /u01/app/12.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 579571814 Jan 27 14:29 /u01/app/12.2.0/grid/bin/oracle
3. Start the CRS
root@TESTHOST02:[/]#/u01/app/12.2.0/grid/bin/crsctl start crs
Today, I encountered interesting but weird problem while successfully applying Patch 30501932 - GI Jan 2020 Release Update 12.2.0.1.200114 on node-1 when I tried to shutdown immediate for instance -2 for applying patch, it did not shutdown and session and was keep waiting for shutdown so I had to abnormally stop shutdown and issued shutdown abort, but post shutdown when I tried to start up instance-2, it throw below set of errors:
SQL> startup
ORA-01078: failure in processing system parametersLRM-00109: could not open parameter file '/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/inittestdb2.ora'
SQL> exit
Created pfile from SPFILE on node-1 and copied to node-2 and tried starting instance 2 using newly created pfile:
SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/inittestdb2.ora'
SQL>
SQL> startup pfile='/tmp/inittestdb.ora';
ORACLE instance started.
Total System Global Area 5.5298E+10 bytes
Fixed Size 29964104 bytes
Variable Size 8724154552 bytes
Database Buffers 4.6439E+10 bytes
Redo Buffers 104251392 bytes
ORA-00205: error in identifying control file, check alert log for more info
Below is the list of Error reported in alert.log file:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: failed to register ASMB0 with ASM instance
2020-01-30T14:32:27.133429+03:00
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb2/trace/testdb2_asmb_31261082.trc:
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
WARNING: ASMB0 exiting with error
2020-01-30T14:32:27.137941+03:00
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/testdb/CONTROLFILE/current.261.1030897861'
ORA-17503: ksfdopn:2 Failed to open file +DATA/testdb/CONTROLFILE/current.261.1030897861
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/testdb/CONTROLFILE/current.262.1030897861'
ORA-17503: ksfdopn:2 Failed to open file +DATA/testdb/CONTROLFILE/current.262.1030897861
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
2020-01-30T14:32:27.145437+03:00
ORA-205 signalled during: ALTER DATABASE MOUNT...
2020-01-30T14:44:12.824562+03:00
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Issue Observation:
I observed that even though +ASM2 was up and all disks were mounted on node-2 still Instance-2 was not able to access diskgroups and hence upon DB startup it could not access SPFILE located on ASM diskgroup and hence oracle tried to start Instance with default PFILE but still it couldn't start as controlfile was not accessible due to diskgroup accessibility issue.
Further more Observed that after successfully applying GI Jan 2020 PATCH on node-1, even without applying patch on node-2, it had changed setuid permission for GI oracle binary file on node-2 and because of that Oracle was not able to access diskgroup upon instance startup, please find below evidence:
setuid permission for owner and group is changed from s to x and that's the issue.
On Node-2:
oracle@TESTHOST02:[/u01/app/12.2.0/grid/bin]$ls -rlta |grep oracle
-rw------- 1 grid oinstall 579571814 Jan 27 13:45 oracleO
-rwxr-x--x 1 grid oinstall 579571814 Jan 27 14:29 oracle
oracle@TESTHOST02:[/u01/app/12.2.0/grid/bin]$
Please observe setuid for GI Oracle binary on node-1, which is proper.
On Node-1:
===================
root@TESTHOST01:[/]#ls -lrta /u01/app/12.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 583710697 Jan 30 12:27 /u01/app/12.2.0/grid/bin/oracle
root@TESTHOST01:[/]#
Solution:
Perform below steps on node-2, (affected node), which should fix the issue and Instance on node-2 should come up just fine without any issue.
1.shutdown CRS as root
root@TESTHOST02:[/]#/u01/app/12.2.0/grid/bin/crsctl stop crs
2. Set setuid for Grid oracle binary.
root@TESTHOST02:[/u01/app/12.2.0/grid/bin]$ls -rlta |grep oracle
-rwxr-x--x 1 grid oinstall 579571814 Jan 27 14:29 oracle
root@TESTHOST02:[/]#chmod 6751 /u01/app/12.2.0/grid/bin/oracle
root@TESTHOST02:[/]#ls -lrta /u01/app/12.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 579571814 Jan 27 14:29 /u01/app/12.2.0/grid/bin/oracle
3. Start the CRS
root@TESTHOST02:[/]#/u01/app/12.2.0/grid/bin/crsctl start crs
No comments:
Post a Comment