Tuesday, February 4, 2020

Steps to apply GI and RDBMS patch to Oracle Grid & RDBMS home and database- Oracle Database Patch 30501932 - GI Jan 2020 Release Update 12.2.0.1.200114

Scope of this Article:



  Mentioned steps to apply patch for GI and RDBMS homes and database can be used for any quarterly patches, however this particular article was prepared while applying Jan'20 PSU patch -Oracle Database Patch 30501932 - GI Jan 2020 Release Update 12.2.0.1.200114 on IBM AIX 64 bit Power systems.  Below mentioned steps will give you fair idea to apply PSU Patch, however I would highly encourage you to go through the patch readme thoroughly before proceeding with the patching.  It's always better to apply patch on lower environment first before applying it on production environment. 

First of all identify the patches required to be applied for your environment and download and keep zip files in specific location on the server.  for example I have downloaded and kept patches zip files at below location on my all nodes of the RAC cluster.

/backup/PATCH/Jan2020

Few Important points:

1. This document can be used to apply PSU patch for Unix/Linux and some of the portion can be used for Windows as well.
2. This method is used to apply patch for 12cR2, which is quite different than application of patch in 11g and prior.
3. Binary Patching activity will be performed as a root user.
4. Only  GI patch needs to be downloaded for RAC(which includes GI+RDBMS components), same patch needs to be applied on both Grid Infra home and RDBMS Home.
5. datapatch, which is post SQL Load/execution needs to performed only once on any single node , when binary patching completed on all/both the nodes.
6. It's recommended that patch related readme needs to be followed , there may be small changes here and there considering the patch version and OS version.
7. For RAC environment, Rolling forward method is the best as there will NOT be complete application downtime, however in flight transaction may impact while performing activity.


###1.1 Patch Information

The Grid Infrastructure patches are cumulative and include the Database Update Content and CPU program security content.


Configuration                                                     GI Version Database Versions Patch OPatch Command(1) Comments
----------------------------------------------------------------- ------------- -----------------   -------- ----------------- ---------------------------------------------------
GI Home in conjunction with RAC, RACOne, or Single Instance home     12.2.0.1   12.2.0.1            GI RU   opatchauto          GI Home and all the Database Homes will be patched


Table 1-2 Patch Numbers Getting Installed as Part of this Bundle Patch

Patch Number Description Applicable Homes
================== ===================                            ============================
30593149                Database Jan 2020 Release Update 12.2.0.1.200114       Only DB Home for non-Oracle RAC setup. 
                                                                               Both DB Homes and Grid Home for Oracle RAC setup.
30585969                OCW Jan 2020 Release Update 12.2.0.1.200114            Both DB Homes and Grid Home
30586063                ACFS Jan 2020 Release Update 12.2.0.1.200114Footnote 2 Only Grid Home
26839277                DBWLM Release Update 12.2.0.1.170913Footnote 2         Only Grid Home
30591794                Tomcat Release Update 12.2.0.1.0Footnote 2             Only Grid Home



###1.2 Patch Installation and Deinstallation  

==>1.2.1 Patch Installation Prerequisites

Note: Make sure that crontab jobs or any other scheduled jobs should be commented during patching.

-->1.2.1.1 OPatch Utility Information 
You must use the OPatch utility version 12.2.0.1.17 or later to apply this patch for all platforms. Oracle recommends that you use the latest released OPatch version for 12.1 releases, 
which is available for download from My Oracle Support patch 6880880 by selecting ARU link for the 12.2.0.1.0 release. It is recommended that you download the Opatch utility and 
the patch in a shared location to be able to access them from any node in the cluster for the patch application on each node.

When patching the GI Home, a shared location on ACFS only needs to be unmounted on the node where the GI Home is being patched.

The new opatch utility should be updated in all the Oracle RAC database homes and the GI home that are being patched. 


oracle@PRODHOSTDB01:[/home/oracle]$cd $ORACLE_HOME/OPatch
oracle@PRODHOSTDB01:[/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch]$opatch version
OPatch Version: 12.2.0.1.6 <--Which is less than 12.2.0.1.17 so we need to download latest 6880880 support patch.

OPatch succeeded.
oracle@PRODHOSTDB01:[/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch]$

Download the OPatch utility to a temporary directory.

For each Oracle RAC database home and the GI home that are being patched, run the following commands as the home owner to extract the OPatch utility.

Unzip the patch as grid home owner except for installations that do not have any grid homes
PATCH Location: /backup/PATCH/Jan2020
$ unzip <OPATCH-ZIP> -d <ORACLE_HOME>
$ <ORACLE_HOME>/OPatch/opatch version
The version output of the previous command should be 12.2.0.1.17 or later.

~~~>Make sure that you change below permission for directory from where you are applying patches otherwise you may face issue while patching due to permission issue. 
oracle@PRODHOSTDB01:[/backup/PATCH/Jan2020]$chmod -R 775 /backup/PATCH/Jan2020

UnZip the 6880880:

###Backing of the Binaries homes: make sure to backup binaries on both the nodes
~~~>Make sure that you change below permission otherwise you may face issue while patching due to permission issue.
oracle@PRODHOSTDB01:[/backup/PATCH/Jan2020]$chmod -R 775 /backup/PATCH/Jan2020

For GI Home: Run it as GRID user

-->To run backup of home in background
cd /backup/PATCH/Jan2020
nohup tar cvf - /u01/app/12.2.0/grid | gzip > /backup/PATCH/Jan2020/GRID_HOME_bkp1_30012020.tar.gz &
-->to run backup of home in foreground
tar cvf - /u01/app/12.2.0/grid | gzip > /backup/PATCH/Jan2020/Grid_home_bkp1_30012020.tar.gz 

OH home: RUN it as Oracle user
-->To run backup of home in background
cd /backup/PATCH/Jan2020
nohup tar cvf - /u01/app/oracle/product/12.2.0.1/dbhome_1 | gzip > ORACLE_HOME_bkp1_30012020.tar.gz &

OR

-->to run backup of home in foreground
cd /backup/PATCH/Jan2020
tar cvf - /u01/app/oracle/product/12.2.0.1/dbhome_1 | gzip >  /backup/PATCH/Jan2020/ORACLE_HOME_bkp1_08102019.tar.gz


-->1.2.1.2 Validation of Oracle Inventory

Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

Note:Perform this on both the nodes:

For Grid Home:
$ /u01/app/12.2.0/grid/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

/u01/app/12.2.0/grid/OPatch/opatch lsinventory -detail -oh /u01/app/12.2.0/grid |tee -a /backup/PATCH/Jan2020/grid_OPatch_lsinventory_B4.lst

/u01/app/12.2.0/grid/OPatch/opatch lsinventory -detail -oh /u01/app/12.2.0/grid |tee -a /backup/PATCH/Jan2020/grid_OPatch_lsinventory_detail_B4.lst

For RDBMS Home as oracle user:
If this command succeeds, it lists the Oracle components that are installed in the home. Save the output so you have the status prior to the patch apply.

/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatch lsinventory -oh /u01/app/oracle/product/12.2.0.1/dbhome_1 |tee -a /backup/PATCH/Jan2020/rdbms_OPatch_lsinventory_B4.lst
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatch lsinventory -detail -oh /u01/app/oracle/product/12.2.0.1/dbhome_1 |tee -a /backup/PATCH/Jan2020/rdbms_OPatch_lsinventory_detail_B4.lst

~~>>Execute below SQL to check the status of the patches from DB:

sqlplus / as sysdba
spool /backup/PATCH/Jan2020/datapatch_status_B4.lst

set line 500
 col patch_id for 9999999999
 col patch_uid for 9999999999
 col version for a10
 col action for a10
 col install_id  for 99
 col action_time for a30
 col description for a70
 col bundle_series for a20

 select patch_id, patch_uid, version, action, install_id, action_time, description, bundle_series  from dba_registry_sqlpatch;

spool off

~~>1.2.1.2 Validation of Oracle Inventory
Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

$ <ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

~~>1.2.1.3 Download and Unzip the Patch : I assume this step has been done


~~>1.2.1.4 Run OPatch Conflict Check

•For Grid Infrastructure Home, as home user: Change appropriate patch location
Login as grid

 $GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/30593149 |tee -a  /backup/PATCH/Jan2020/grid_ConflictCheck_30501932_30593149.txt
 $GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/30585969 |tee -a  /backup/PATCH/Jan2020/grid_ConflictCheck_30501932_30585969.txt
 $GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/30586063 |tee -a  /backup/PATCH/Jan2020/grid_ConflictCheck_30501932_30586063.txt
 $GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/26839277 |tee -a  /backup/PATCH/Jan2020/grid_ConflictCheck_30501932_26839277.txt
 $GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/30591794 |tee -a  /backup/PATCH/Jan2020/grid_ConflictCheck_30501932_30591794.txt


•For Database home, as home user:Change appropriate patch location
Login as oracle

 $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/30593149 |tee -a /backup/PATCH/Jan2020/oracle_ConflictCheck_30501932_30593149.txt
 $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/PATCH/Jan2020/30501932/30585969 |tee -a /backup/PATCH/Jan2020/oracle_ConflictCheck_30501932_30585969.txt

 The report will indicate the interim patches that conflict with the patch 30501932 and the interim patches for which patch 30501932 is a superset.

Note:

When OPatch starts, it validates the patch and ensures that there are no conflicts with the software already installed in the ORACLE_HOME.
OPatch categorizes conflicts into the following types:
Conflicts with a patch already applied to the ORACLE_HOME.
In this case, stop the patch installation and contact Oracle Support Services.
Conflicts with subset patch already applied to the ORACLE_HOME.
In this case, continue with the patch installation because as the new patch contains all the fixes from the existing patch in the ORACLE_HOME.
And, in any case, the subset patch will automatically be rolled back prior to the installation of the new patch.


-->1.2.1.5 Run OPatch SystemSpace Check

Check if enough free space is available on the ORACLE_HOME filesystem for the patches to be applied as given below:

⇒For Grid Infrastructure Home, as home user:grid
===============================
1.Create file /tmp/patch_list_gihome.txt with the following content:

su - grid

cd /backup/PATCH/Jan2020

echo "/backup/PATCH/Jan2020/30501932/30593149" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
echo "/backup/PATCH/Jan2020/30501932/30585969" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
echo "/backup/PATCH/Jan2020/30501932/30586063" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
echo "/backup/PATCH/Jan2020/30501932/26839277" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
echo "/backup/PATCH/Jan2020/30501932/30591794" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
cat  /backup/PATCH/Jan2020/patch_list_gihome.txt

2.Run the opatch command to check if enough free space is available in the Grid Infrastructure Home:

 $GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_gihome.txt

⇨For Database home, as home user:
===============================
su - oracle

1.Create file /backup/PATCH/Jan2020/patch_list_dbhome.txt with the following content:

echo "/backup/PATCH/Jan2020/30501932/30593149" >> /backup/PATCH/Jan2020/patch_list_dbhome.txt
echo "/backup/PATCH/Jan2020/30501932/30585969" >> /backup/PATCH/Jan2020/patch_list_dbhome.txt
cat /backup/PATCH/Jan2020/patch_list_dbhome.txt


2. Run opatch command to check if enough free space is available in the Database Home:
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_dbhome.txt

The command output reports pass and fail messages as per the system space availability:
If OPatch reports Prereq "checkSystemSpace" failed., then cleanup the system space as the required amount of space is not available.
If OPatch reports Prereq "checkSystemSpace" passed., then no action is needed. Proceed with patch installation.

~~~~~~~~~~~~~~~~~~~~~~output of checkSystemSpace execution~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
===>Node-1: checkSystemSpace
==================================

"/backup/PATCH/Jan2020/patch_list_gihome.txt" [New file] 7 lines, 202 characters
grid@PRODHOSTDB01:[/backup/PATCH/Jan2020]$cat "/backup/PATCH/Jan2020/patch_list_gihome.txt"

/backup/PATCH/Jan2020/30501932/30593149
/backup/PATCH/Jan2020/30501932/30585969
/backup/PATCH/Jan2020/30501932/30586063
/backup/PATCH/Jan2020/30501932/26839277
/backup/PATCH/Jan2020/30501932/30591794

grid@PRODHOSTDB01:[/backup/PATCH/Jan2020]$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_gihome.txt
ksh: GRID_HOME/OPatch/opatch:  not found.
grid@PRODHOSTDB01:[/backup/PATCH/Jan2020]$$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/12.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.2.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/12.2.0/grid/cfgtoollogs/opatch/opatch2020-01-30_11-23-25AM_1.log

Invoking prereq "checksystemspace"

Prereq "checkSystemSpace" passed.

OPatch succeeded.


~~>>For Oracle_home:RDBMS

oracle@PRODHOSTDB01:[/home/oracle]$
oracle@PRODHOSTDB01:[/home/oracle]$echo "/backup/PATCH/Jan2020/30501932/30593149" >> /backup/PATCH/Jan2020/patch_list_dbhome.txt
oracle@PRODHOSTDB01:[/home/oracle]$echo "/backup/PATCH/Jan2020/30501932/30585969" >> /backup/PATCH/Jan2020/patch_list_dbhome.txt
oracle@PRODHOSTDB01:[/home/oracle]$cat /backup/PATCH/Jan2020/patch_list_dbhome.txt
/backup/PATCH/Jan2020/30501932/30593149
/backup/PATCH/Jan2020/30501932/30585969
oracle@PRODHOSTDB01:[/home/oracle]$$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_dbhome.txt
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/oracle/product/12.2.0.1/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.2.0.1/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatch/opatch2020-01-30_11-25-59AM_1.log

Invoking prereq "checksystemspace"

Prereq "checkSystemSpace" passed.

OPatch succeeded.
oracle@PRODHOSTDB01:[/home/oracle]$

===>Node-2: checkSystemSpace
==================================
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#su - grid
grid@PRODHOSTDB02:[/home/grid]$
grid@PRODHOSTDB02:[/home/grid]$cd /backup/PATCH/Jan2020

grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$
echo "/backup/PATCH/Jan2020/30501932/30593149" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
echo "/backup/PATCH/Jan2020/30501932/30585969" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$echo "/backup/PATCH/Jan2020/30501932/30593149" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$echo "/backup/PATCH/Jan2020/30501932/30585969" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$echo "/backup/PATCH/Jan2020/30501932/30586063" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$echo "/backup/PATCH/Jan2020/30501932/26839277" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$echo "/backup/PATCH/Jan2020/30501932/30591794" >> /backup/PATCH/Jan2020/patch_list_gihome.txt
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$cat  /backup/PATCH/Jan2020/patch_list_gihome.txt
/backup/PATCH/Jan2020/30501932/30593149
/backup/PATCH/Jan2020/30501932/30585969
/backup/PATCH/Jan2020/30501932/30586063
/backup/PATCH/Jan2020/30501932/26839277
/backup/PATCH/Jan2020/30501932/30591794
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/12.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.2.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/12.2.0/grid/cfgtoollogs/opatch/opatch2020-01-30_11-44-12AM_1.log

Invoking prereq "checksystemspace"

Prereq "checkSystemSpace" passed.

OPatch succeeded.
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$exit
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#su - oracle
oracle@PRODHOSTDB02:[/home/oracle]$
oracle@PRODHOSTDB02:[/home/oracle]$echo "/backup/PATCH/Jan2020/30501932/30593149" >> /backup/PATCH/Jan2020/patch_list_dbhome.txt
oracle@PRODHOSTDB02:[/home/oracle]$echo "/backup/PATCH/Jan2020/30501932/30585969" >> /backup/PATCH/Jan2020/patch_list_dbhome.txt
oracle@PRODHOSTDB02:[/home/oracle]$cat /backup/PATCH/Jan2020/patch_list_dbhome.txt
/backup/PATCH/Jan2020/30501932/30593149
/backup/PATCH/Jan2020/30501932/30585969
oracle@PRODHOSTDB02:[/home/oracle]$
oracle@PRODHOSTDB02:[/home/oracle]$
oracle@PRODHOSTDB02:[/home/oracle]$$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /backup/PATCH/Jan2020/patch_list_dbhome.txt
Oracle Interim Patch Installer version 12.2.0.1.19
Copyright (c) 2020, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/app/oracle/product/12.2.0.1/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.2.0.1/dbhome_1/oraInst.loc
OPatch version    : 12.2.0.1.19
OUI version       : 12.2.0.1.4
Log file location : /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatch/opatch2020-01-30_11-45-00AM_1.log

Invoking prereq "checksystemspace"

Prereq "checkSystemSpace" passed.

OPatch succeeded.
oracle@PRODHOSTDB02:[/home/oracle]$

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~End of Ouput ChecySystemSpace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


==>1.2.2 One-off Patch Conflict Detection and Resolution

Run as ROOT user:

export GRID_HOME=/u01/app/12.2.0/grid
$GRID_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $GRID_HOME

export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
$ORACLE_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $ORACLE_HOME



===You can use below command to analyze both the home togather.
export GRID_HOME=/u01/app/12.2.0/grid
$GRID_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze
===

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~output of execution of analyze~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
===>Node -1 for GRID home:
=============================
root@PRODHOSTDB01:[/]#export GRID_HOME=/u01/app/12.2.0/grid
root@PRODHOSTDB01:[/]#$GRID_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $GRID_HOME
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Thu Jan 30 11:34:02 2020

System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-01-30_11-34-11AM.log.

Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-01-30_11-34-53AM.log
The id for this session is ZE7B

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid
Patch applicability verified successfully on home /u01/app/12.2.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:PRODHOSTdb01
CRS Home:/u01/app/12.2.0/grid
Version:12.2.0.1.0


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-35-18AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30586063
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-35-18AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/26839277
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-35-18AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30591794
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-35-18AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-35-18AM_1.log



OPatchauto session completed at Thu Jan 30 11:35:30 2020
Time taken to complete the session 1 minute, 28 seconds
root@PRODHOSTDB01:[/]#

===>Node -1 for RDBMS home:
=============================
root@PRODHOSTDB01:[/]#export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
root@PRODHOSTDB01:[/]#$ORACLE_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $ORACLE_HOME
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Thu Jan 30 11:30:21 2020

System initialization log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-01-30_11-30-30AM.log.

Session log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-01-30_11-31-23AM.log
The id for this session is YCBP

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:PRODHOSTdb01
RAC Home:/u01/app/oracle/product/12.2.0.1/dbhome_1
Version:12.2.0.1.0


==Following patches were SKIPPED:

Patch: /backup/PATCH/Jan2020/30501932/30586063
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/26839277
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/30591794
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-31-45AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-31-45AM_1.log

OPatchauto session completed at Thu Jan 30 11:32:46 2020
Time taken to complete the session 2 minutes, 25 seconds
root@PRODHOSTDB01:[/]#


===>Node -2 for GRID home:
=============================
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#export GRID_HOME=/u01/app/12.2.0/grid
$GRID_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $GRID_HOME

root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
$GRID_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $GRID_HOME
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Thu Jan 30 11:47:24 2020

System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-01-30_11-47-33AM.log.

Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-01-30_11-48-10AM.log
The id for this session is W7XV

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid
Patch applicability verified successfully on home /u01/app/12.2.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:PRODHOSTdb02
CRS Home:/u01/app/12.2.0/grid
Version:12.2.0.1.0


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-48-36AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30586063
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-48-36AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/26839277
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-48-36AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30591794
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-48-36AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-48-36AM_1.log

OPatchauto session completed at Thu Jan 30 11:48:46 2020
Time taken to complete the session 1 minute, 23 seconds
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#


===>Node -2 for RDBMS home:
=============================
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#$ORACLE_HOME/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -analyze -oh $ORACLE_HOME
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Thu Jan 30 11:48:47 2020

System initialization log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-01-30_11-48-55AM.log.

Session log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-01-30_11-49-45AM.log
The id for this session is KAXM

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:PRODHOSTdb02
RAC Home:/u01/app/oracle/product/12.2.0.1/dbhome_1
Version:12.2.0.1.0


==Following patches were SKIPPED:

Patch: /backup/PATCH/Jan2020/30501932/30586063
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/26839277
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/30591794
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-50-07AM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_11-50-07AM_1.log

OPatchauto session completed at Thu Jan 30 11:51:07 2020
Time taken to complete the session 2 minutes, 21 seconds
root@PRODHOSTDB02:[/backup/PATCH/Jan2020]#

------------------End of O/p Analyze-----------------------------------------------------------------------------


==> Check all the CRS services are listed...

ps -ef|grep d.bin

$crsctl status res -t
Node-1:
====================
grid@PRODHOSTDB01:[/home/grid]$ps -ef|grep d.bin
    root 14156106        1   1   Jan 27      -  1:08 /u01/app/12.2.0/grid/bin/cssdagent
    grid 14549326        1   0   Jan 27      -  5:54 /u01/app/12.2.0/grid/bin/oraagent.bin
    root 14614814        1   0   Jan 27      -  2:35 /u01/app/12.2.0/grid/bin/octssd.bin reboot
    grid 14942694 16777666   0   Jan 27      -  0:25 /u01/app/12.2.0/grid/bin/evmlogger.bin -o /u01/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.log
    root 15073760        1   0   Jan 27      -  3:43 /u01/app/12.2.0/grid/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.WE8ISO8859P1
    root 15466990        1   0   Jan 27      -  0:00 /bin/sh /u01/app/12.2.0/grid/bin/ocssd
    root 15860074        1   0   Jan 27      -  1:07 /u01/app/12.2.0/grid/bin/cssdmonitor
    grid 16122176        1   0   Jan 27      -  0:31 /u01/app/12.2.0/grid/bin/mdnsd.bin
    root 16318772        1   0   Jan 27      -  4:11 /u01/app/12.2.0/grid/bin/orarootagent.bin
    grid 16515512        1   0   Jan 27      -  2:32 /u01/app/12.2.0/grid/bin/gipcd.bin
    grid 16646534 15466990   1   Jan 27      -  5:59 /u01/app/12.2.0/grid/bin/ocssd.bin
    grid 16777666        1   0   Jan 27      -  2:36 /u01/app/12.2.0/grid/bin/evmd.bin
    grid 17236460        1   1   Jan 27      -  0:54 /u01/app/12.2.0/grid/bin/scriptagent.bin
    grid 18219414        1   0   Jan 27      -  0:01 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit
    grid 18481628        1   0   Jan 27      -  0:01 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
    root 18547174        1   0   Jan 27      -  8:59 /u01/app/12.2.0/grid/bin/ologgerd -M
    root 18809210        1   0   Jan 27      -  7:54 /u01/app/12.2.0/grid/bin/osysmond.bin
    grid 19595702        1   0   Jan 27      -  0:37 /u01/app/12.2.0/grid/bin/gpnpd.bin
    root 19726724        1   0   Jan 27      - 75:05 /u01/app/12.2.0/grid/bin/orarootagent.bin
    grid 19792224        1   0   Jan 27      -  0:28 /u01/app/12.2.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
    grid 19988910        1   0   Jan 27      -  0:02 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
    grid 23331312        1   0   Jan 27      - 54:21 /u01/app/12.2.0/grid/bin/oraagent.bin
    grid 24773044        1   0   Jan 27      -  0:02 /u01/app/12.2.0/grid/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
    root 26935722        1   0   Jan 27      -  4:23 /u01/app/12.2.0/grid/bin/crsd.bin reboot
    grid 29688294 42008834   0 11:58:57  pts/2  0:00 grep d.bin
  oracle 30474534        1   0   Jan 28      -  0:47 /u01/app/12.2.0/grid/bin/oraagent.bin



grid@PRODHOSTDB01:[/home/grid]$crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.DATA.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.FRA.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               OFFLINE OFFLINE      PRODHOSTdb02              STABLE
ora.GIMR.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.INDX.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               OFFLINE OFFLINE      PRODHOSTdb02              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.OCR.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.REDO.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               OFFLINE OFFLINE      PRODHOSTdb02              STABLE
ora.net1.network
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.ons
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.proxy_advm
               OFFLINE OFFLINE      PRODHOSTdb01              STABLE
               OFFLINE OFFLINE      PRODHOSTdb02              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       PRODHOSTdb01              169.254.113.95 10.80
                                                             .0.2,STABLE
ora.asm
      1        ONLINE  ONLINE       PRODHOSTdb01              Started,STABLE
      2        ONLINE  ONLINE       PRODHOSTdb02              Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.PRODHOSTdb01.vip
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.PRODHOSTdb02.vip
      1        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.knetpgt.db
      1        ONLINE  ONLINE       PRODHOSTdb01              Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             .1/dbhome_1,STABLE
      2        ONLINE  ONLINE       PRODHOSTdb02              Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             .1/dbhome_1,STABLE
ora.knetpgt.knetpgt_svc.svc
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
      2        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       PRODHOSTdb01              Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
--------------------------------------------------------------------------------
grid@PRODHOSTDB01:[/home/grid]$


Node-2:
=====================

grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$ps -ef|grep d.bin
    root 13631752        1   2   Jan 27      - 48:45 /u01/app/12.2.0/grid/bin/orarootagent.bin
    root 13828582        1   0   Jan 27      -  0:00 /bin/sh /u01/app/12.2.0/grid/bin/ocssd
    root 13959532        1   0   Jan 27      -  5:09 /u01/app/12.2.0/grid/bin/crsd.bin reboot
    grid 14287182 16318832   0   Jan 27      -  0:25 /u01/app/12.2.0/grid/bin/evmlogger.bin -o /u01/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.log
    grid 14352682 13828582   0   Jan 27      -  5:51 /u01/app/12.2.0/grid/bin/ocssd.bin
    grid 14811500        1   0   Jan 27      -  0:29 /u01/app/12.2.0/grid/bin/mdnsd.bin
    root 14942564        1   0   Jan 27      -  6:51 /u01/app/12.2.0/grid/bin/osysmond.bin
    root 15008054        1   0   Jan 27      -  1:07 /u01/app/12.2.0/grid/bin/cssdagent
    root 15139242        1   1   Jan 27      -  2:32 /u01/app/12.2.0/grid/bin/octssd.bin reboot
    root 15663392        1   0   Jan 27      -  1:05 /u01/app/12.2.0/grid/bin/cssdmonitor
    grid 16187658        1   0   Jan 27      -  8:20 /u01/app/12.2.0/grid/bin/oraagent.bin
    grid 16253186        1   0   Jan 27      -  2:30 /u01/app/12.2.0/grid/bin/gipcd.bin
    grid 16318832        1   0   Jan 27      -  2:33 /u01/app/12.2.0/grid/bin/evmd.bin
    root 16384338        1   0   Jan 27      -  3:31 /u01/app/12.2.0/grid/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.WE8ISO8859P1
    grid 16515334        1   0   Jan 27      -  0:02 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
    root 16581012        1   0   Jan 27      -  2:48 /u01/app/12.2.0/grid/bin/orarootagent.bin
    grid 16712134        1   1   Jan 27      - 23:35 /u01/app/12.2.0/grid/bin/oraagent.bin
    grid 16843084        1   0   Jan 27      -  0:33 /u01/app/12.2.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
    grid 16908772        1   0   Jan 27      -  0:36 /u01/app/12.2.0/grid/bin/gpnpd.bin
    grid 18547146        1   0   Jan 27      -  0:01 /u01/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
  oracle 19530086        1   0   Jan 28      -  0:44 /u01/app/12.2.0/grid/bin/oraagent.bin
    grid 35389790 19595636   0 11:59:26  pts/0  0:00 grep d.bin
grid@PRODHOSTDB02:[/backup/PATCH/Jan2020]$

grid@PRODHOSTDB02:[/home/grid]$crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.DATA.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.FRA.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.GIMR.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.INDX.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.OCR.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.REDO.dg
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.net1.network
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.ons
               ONLINE  ONLINE       PRODHOSTdb01              STABLE
               ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.proxy_advm
               OFFLINE OFFLINE      PRODHOSTdb01              STABLE
               OFFLINE OFFLINE      PRODHOSTdb02              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       PRODHOSTdb01              169.254.113.95 10.80
                                                             .0.2,STABLE
ora.asm
      1        ONLINE  ONLINE       PRODHOSTdb01              Started,STABLE
      2        ONLINE  ONLINE       PRODHOSTdb02              Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.PRODHOSTdb01.vip
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.PRODHOSTdb02.vip
      1        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.knetpgt.db
      1        ONLINE  ONLINE       PRODHOSTdb01              Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             .1/dbhome_1,STABLE
      2        ONLINE  ONLINE       PRODHOSTdb02              Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             .1/dbhome_1,STABLE
ora.knetpgt.knetpgt_svc.svc
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
      2        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       PRODHOSTdb01              Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       PRODHOSTdb02              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       PRODHOSTdb01              STABLE
--------------------------------------------------------------------------------
grid@PRODHOSTDB02:[/home/grid]$

~~~~~~~~~~~~~~~~~~~~~~~end of output crsctl cluster resources status~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


==>1.2.4 Patch Installation
###########################################################################################

==>check status of database connection and distribution before starting patch or any other information like invalid objects etc , you can capture it at this point

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run your scripts if any to gather any details before proceeding with the patch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Now shutdown the instance on the node-1, either you can use srvctl stop instance command or you can manually shutdown like below. I always prefere to manually shutdown instance like below:

oracle@PRODHOSTDB01:[/home/oracle]$sqlplus '/ as sysdba'

SQL*Plus: Release 12.2.0.1.0 Production on Thu Jan 30 12:11:54 2020

Copyright (c) 1982, 2017, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

==>Patching Oracle RAC Database Homes and GI Separately 


•Case 1: Oracle RAC, where the GI Home and the Database Homes are not shared and ACFS file system is not configured.

root@PRODHOSTDB01:[/]#ps -ef|grep sql
    root 29360580 43123118   0 12:14:52  pts/2  0:00 grep sql
root@PRODHOSTDB01:[/]#


==>For Grid Home:
================
---------------------------------------------------------------------------
su - grid

export GRID_HOME=/u01/app/12.2.0/grid    <--Not Mandatory
which opatch
opatch version

==> Please make sure to change the full permission otherwise patching may fail, due to file permission issue.
root@IPYDRDB01:[/backup/Jan2020]#chmod 777 -r /backup/Jan2020/29699173

echo $PATH
export PATH=/u01/app/12.2.0/grid/OPatch:$PATH

opatch version

root@# /u01/app/12.2.0/grid/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/12.2.0/grid


==>Oracle - DB:
================
-------------------------------------------------------------------------
su - oracle

export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
export PATH=/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch:$PATH
echo $PATH
which opatch
opatch version

root@#  /u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/oracle/product/12.2.0.1/dbhome_1


You can either patch GRID and RDBMS Homes separately or in single shot using below command which will patch both the homes with single command on the node.

====>To patch both the nodes to gather run below command.
As root user, execute the following command on each node of the cluster:
# <GI_HOME>/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/29708720


~~~~~~~~~~~~~~~~~~~~~~Output patch apply~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
==>Node-1 : Grid Patching:
=============================

root@PRODHOSTDB01:[/]#export GRID_HOME=/u01/app/12.2.0/grid
root@PRODHOSTDB01:[/]#echo $PATH
/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB01:[/]#export PATH=/u01/app/12.2.0/grid/OPatch:$PATH
root@PRODHOSTDB01:[/]#echo $PATH
/u01/app/12.2.0/grid/OPatch:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB01:[/]#opatch version
OPatch Version: 12.2.0.1.19

OPatch succeeded.
root@PRODHOSTDB01:[/]#/u01/app/12.2.0/grid/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/12.2.0/grid
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /u01/app/12.2.0/grid/OPatch /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin/usr/java8_64/bin.

OPatchauto session is initiated at Thu Jan 30 12:17:43 2020

System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-01-30_12-17-51PM.log.

Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-01-30_12-18-19PM.log
The id for this session is JS77

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid
Patch applicability verified successfully on home /u01/app/12.2.0/grid


Bringing down CRS service on home /u01/app/12.2.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/PRODHOSTdb01/crsconfig/crspatch_PRODHOSTdb01_2020-01-30_12-18-58AM.log
CRS service brought down successfully on home /u01/app/12.2.0/grid


Start applying binary patch on home /u01/app/12.2.0/grid
Successfully executed command: /usr/sbin/slibclean


Binary patch applied successfully on home /u01/app/12.2.0/grid


Starting CRS service on home /u01/app/12.2.0/grid

Postpatch operation log file location: /u01/app/grid/crsdata/PRODHOSTdb01/crsconfig/crspatch_PRODHOSTdb01_2020-01-30_12-28-34AM.log
CRS service started successfully on home /u01/app/12.2.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:PRODHOSTdb01
CRS Home:/u01/app/12.2.0/grid
Version:12.2.0.1.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /backup/PATCH/Jan2020/30501932/26839277
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-21-35PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-21-35PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30586063
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-21-35PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30591794
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-21-35PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-21-35PM_1.log



OPatchauto session completed at Thu Jan 30 12:40:54 2020
Time taken to complete the session 23 minutes, 12 seconds
root@PRODHOSTDB01:[/]#


~~~~~~~~~~RDBMS/ORACLE_HOME patching output~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


root@PRODHOSTDB01:[/]#
root@PRODHOSTDB01:[/]#
root@PRODHOSTDB01:[/]#export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
root@PRODHOSTDB01:[/]#export PATH=/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch:$PATH
root@PRODHOSTDB01:[/]#echo $PATH
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch:/u01/app/12.2.0/grid/OPatch:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB01:[/]#which opatch
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatch
root@PRODHOSTDB01:[/]#opatch version
OPatch Version: 12.2.0.1.19

OPatch succeeded.
root@PRODHOSTDB01:[/]#/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/oracle/product/12.2.0.1/dbhome_1
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch /u01/app/12.2.0/grid/OPatch /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Thu Jan 30 12:48:26 2020

System initialization log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-01-30_12-48-35PM.log.

Session log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-01-30_12-49-28PM.log
The id for this session is 1172

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Preparing to bring down database service on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Successfully prepared home /u01/app/oracle/product/12.2.0.1/dbhome_1 to bring down database service


Bringing down database service on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Database service successfully brought down on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Performing prepatch operation on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Start applying binary patch on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Successfully executed command: /usr/sbin/slibclean

Binary patch applied successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Performing postpatch operation on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Starting database service on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Database service successfully started on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Preparing home /u01/app/oracle/product/12.2.0.1/dbhome_1 after database service restarted
No step execution required.........


Trying to apply SQL patch on home /u01/app/oracle/product/12.2.0.1/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:PRODHOSTdb01
RAC Home:/u01/app/oracle/product/12.2.0.1/dbhome_1
Version:12.2.0.1.0
Summary:

==Following patches were SKIPPED:

Patch: /backup/PATCH/Jan2020/30501932/30586063
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/26839277
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/30591794
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-50-06PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-30_12-50-06PM_1.log


Patching session reported following warning(s):
_________________________________________________

[WARNING] The database instance 'TESTDB1' from '/u01/app/oracle/product/12.2.0.1/dbhome_1', in host'PRODHOSTdb01' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.


OPatchauto session completed at Thu Jan 30 12:54:26 2020
Time taken to complete the session 6 minutes, 0 second
root@PRODHOSTDB01:[/]#


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

####>Start the Cluster service if not started automatically: 


>>as ROOT user and make sure GRID_HOME is set

crsctl start crs

root@TESTDB02:[/backup/PATCH]#echo $GRID_HOME
/u01/app/12.2.0.1/grid
root@TESTDB02:[/backup/PATCH]#$GRID_HOME/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
root@TESTDB02:[/backup/PATCH]#crsctl status res -t


>>To check status of Cluster services (all the CRS services are listed).
$crsctl status res -t
ps -ef|grep d.bin

sqlplus / as sysdba

>>>Take the lsinvntory details for DB and GRID homes. 
opatch lsinventory -detail> grid_OPatch_lsinventory_detail_After_DBGrid_patch.lst

/u01/app/oracle/product/12.2.0/dbhome_1/OPatch/opatch lsinventory -detail >rdbms_OPatch_lsinventory_detail_After_DB_Patch.lst


==>start the instance:

grid@PRODHOSTDB01:[/home/grid]$srvctl status database -d TESTDB
Instance TESTDB1 is not running on node PRODHOSTdb01
Instance TESTDB2 is running on node PRODHOSTdb02


srvctl start instance -d TESTDB -i TESTDB1

srvctl status database -d TESTDB

grid@PRODHOSTDB01:[/home/grid]$srvctl status database -d TESTDB
Instance TESTDB1 is running on node PRODHOSTdb01
Instance TESTDB2 is running on node PRODHOSTdb02
grid@PRODHOSTDB01:[/home/grid]$

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
grid@TESTDB01:[/home/grid]$crs_stat -t

grid@PRODHOSTDB01:[/home/grid]$crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM.lsnr ora....er.type ONLINE    ONLINE    PRODHOSTdb01
ora.DATA.dg    ora....up.type ONLINE    ONLINE    PRODHOSTdb01
ora.FRA.dg     ora....up.type ONLINE    ONLINE    PRODHOSTdb01
ora.GIMR.dg    ora....up.type ONLINE    ONLINE    PRODHOSTdb01
ora.INDX.dg    ora....up.type ONLINE    ONLINE    PRODHOSTdb01
ora....ER.lsnr ora....er.type ONLINE    ONLINE    PRODHOSTdb01
ora....AF.lsnr ora....er.type OFFLINE   OFFLINE
ora....N1.lsnr ora....er.type ONLINE    ONLINE    PRODHOSTdb01
ora....N2.lsnr ora....er.type ONLINE    ONLINE    PRODHOSTdb02
ora....N3.lsnr ora....er.type ONLINE    ONLINE    PRODHOSTdb02
ora.MGMTLSNR   ora....nr.type ONLINE    ONLINE    PRODHOSTdb01
ora.OCR.dg     ora....up.type ONLINE    ONLINE    PRODHOSTdb01
ora.REDO.dg    ora....up.type ONLINE    ONLINE    PRODHOSTdb01
ora.asm        ora.asm.type   ONLINE    ONLINE    PRODHOSTdb01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    PRODHOSTdb02
ora....01.lsnr application    ONLINE    ONLINE    PRODHOSTdb01
ora....b01.ons application    ONLINE    ONLINE    PRODHOSTdb01
ora....b01.vip ora....t1.type ONLINE    ONLINE    PRODHOSTdb01
ora....02.lsnr application    ONLINE    ONLINE    PRODHOSTdb02
ora....b02.ons application    ONLINE    ONLINE    PRODHOSTdb02
ora....b02.vip ora....t1.type ONLINE    ONLINE    PRODHOSTdb02
ora.TESTDB.db ora....se.type ONLINE    ONLINE    PRODHOSTdb01
ora....svc.svc ora....ce.type ONLINE    ONLINE    PRODHOSTdb01
ora.mgmtdb     ora....db.type ONLINE    ONLINE    PRODHOSTdb01
ora....network ora....rk.type ONLINE    ONLINE    PRODHOSTdb01
ora.ons        ora.ons.type   ONLINE    ONLINE    PRODHOSTdb01
ora.proxy_advm ora....vm.type OFFLINE   OFFLINE
ora.qosmserver ora....er.type ONLINE    ONLINE    PRODHOSTdb02
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    PRODHOSTdb01
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    PRODHOSTdb02
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    PRODHOSTdb02
grid@PRODHOSTDB01:[/home/grid]$


grid@TESTDB01:[/home/grid]$crsctl status res -t


grid@TESTDB01:[/home/grid]$

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
###>>Get the post Patch evidence from Node-1 After patch application:


Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

Note:Perform this on both the nodes:

For Grid:
============
$ /u01/app/12.2.0/grid/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

/u01/app/12.2.0/grid/OPatch/opatch lsinventory  -oh /u01/app/12.2.0/grid |tee -a /backup/PATCH/Jan2020/grid_OPatch_lsinventory_After.lst
/u01/app/12.2.0/grid/OPatch/opatch lsinventory -detail -oh /u01/app/12.2.0/grid |tee -a /backup/PATCH/Jan2020/grid_OPatch_lsinventory_detail_After.lst

For RDBMS as oracle user:
=============
If this command succeeds, it lists the Oracle components that are installed in the home. Save the output so you have the status prior to the patch apply.


/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatch lsinventory -oh /u01/app/oracle/product/12.2.0.1/dbhome_1 |tee -a /backup/PATCH/Jan2020/rdbms_OPatch_lsinventory_After.lst
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatch lsinventory -detail -oh /u01/app/oracle/product/12.2.0.1/dbhome_1 |tee -a /backup/PATCH/Jan2020/rdbms_OPatch_lsinventory_detail_After.lst

~~~~~~~~~~~~~~~~~~~~~~~~completion of node-1  patching activity~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

###====>Perform the similar steps on Node-2:


==>1.2.4 Patch Installation ON NODE-2
########################################################################################

sqlplus / as sysdba
01:28:49 SQL> shut immediate;
                                       
Note: Post shutdown of the instance one, observed that connections/txns where moved slowly to node-2, after 10-15 min all connection(total 3593) were moved to node-2. Below is the output from instance -2:

Patching Oracle RAC Database Homes and GI Separately

•Case 1: Oracle RAC, where the GI Home and the Database Homes are not shared and ACFS file system is not configured.

oracle@:[/home/oracle]$ps -ef| grep sqlplus
  oracle  9307060  7275390   0 03:13:07  pts/3  0:00 grep sqlplus
oracle@:[/home/oracle]$


==>Grid HOME:
==============
---------------------------------------------------------------------------

==> Please make sure to change the full permission otherwise patching may fail, due to file permission issue.

root@TESTDB02:[/backup/PATCH]#chmod -R 775 /backup/PATCH/Jan2020

-->Make sure that instance is down, This is optional but it's better to shutdown instance manually to make sure it is cleanly shutting down.
   Login as oracle and shutdown instance.  "Shutdown immediate;"

-->Now as a root User:

export GRID_HOME=/u01/app/12.2.0/grid    <--Not Mandatory

echo $PATH
export PATH=/u01/app/12.2.0/grid/OPatch:$PATH
which opatch
opatch version

root@# /u01/app/12.2.0/grid/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/12.2.0.1/grid


==>Oracle - DB:
==============
-------------------------------------------------------------------------
-->As a root user:

export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1   <-- Not mandatory

export PATH=/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch:$PATH
echo $PATH
which opatch
opatch version

root@# /u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/oracle/product/12.2.0.1/dbhome_1

==>start the instance as an oracle user:
su - oracle

srvctl start instance -d TESTDB -i TESTDB2

srvctl status database  -d TESTDB



~~~~~~~~~~~~output of patch application on node-2~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
oracle@PRODHOSTDB02:[/home/oracle]$sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sun Feb 2 14:19:07 2020

Copyright (c) 1982, 2017, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

-----------------
root@PRODHOSTDB02:[/]#echo $PATH
/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB02:[/]#export PATH=/u01/app/12.2.0/grid/OPatch:$PATH
root@PRODHOSTDB02:[/]#echo $PATH
/u01/app/12.2.0/grid/OPatch:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB02:[/]#opatch version
OPatch Version: 12.2.0.1.19

OPatch succeeded.
root@PRODHOSTDB02:[/]#which opatch
/u01/app/12.2.0/grid/OPatch/opatch
root@PRODHOSTDB02:[/]#/u01/app/12.2.0/grid/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/12.2.0/grid
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /u01/app/12.2.0/grid/OPatch /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Sun Feb  2 14:39:53 2020

System initialization log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-02-02_02-40-02PM.log.

Session log file is /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/opatchauto2020-02-02_02-40-41PM.log
The id for this session is XEQZ

Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.2.0/grid
Patch applicability verified successfully on home /u01/app/12.2.0/grid


Bringing down CRS service on home /u01/app/12.2.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/PRODHOSTdb02/crsconfig/crspatch_PRODHOSTdb02_2020-02-02_02-41-20PM.log
CRS service brought down successfully on home /u01/app/12.2.0/grid


Start applying binary patch on home /u01/app/12.2.0/grid
Successfully executed command: /usr/sbin/slibclean

Binary patch applied successfully on home /u01/app/12.2.0/grid


Starting CRS service on home /u01/app/12.2.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/PRODHOSTdb02/crsconfig/crspatch_PRODHOSTdb02_2020-02-02_02-49-56PM.log
CRS service started successfully on home /u01/app/12.2.0/grid

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:PRODHOSTdb02
CRS Home:/u01/app/12.2.0/grid
Version:12.2.0.1.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /backup/PATCH/Jan2020/30501932/26839277
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_14-42-55PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_14-42-55PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30586063
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_14-42-55PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30591794
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_14-42-55PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/12.2.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_14-42-55PM_1.log


OPatchauto session completed at Sun Feb  2 14:55:05 2020
Time taken to complete the session 15 minutes, 12 seconds
root@PRODHOSTDB02:[/]#
----------------------------

==>Output of patch apply for RDBMS Home on Node-2:
===================================================

root@PRODHOSTDB02:[/]#echo $PATH
/u01/app/12.2.0/grid/OPatch:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB02:[/]#export PATH=/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch:$PATH
root@PRODHOSTDB02:[/]#echo $PATH
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch:/u01/app/12.2.0/grid/OPatch:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
root@PRODHOSTDB02:[/]#which opatch
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatch
root@PRODHOSTDB02:[/]#opatch version
OPatch Version: 12.2.0.1.19

OPatch succeeded.
root@PRODHOSTDB02:[/]#
root@PRODHOSTDB02:[/]#/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/opatchauto apply /backup/PATCH/Jan2020/30501932 -oh /u01/app/oracle/product/12.2.0.1/dbhome_1
which: 0652-141 There is no sudo in /bin /usr/bin /usr/local/bin /u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch /u01/app/12.2.0/grid/OPatch /usr/bin /etc /usr/sbin /usr/ucb /usr/bin/X11 /sbin /usr/java8_64/jre/bin /usr/java8_64/bin.

OPatchauto session is initiated at Sun Feb  2 15:03:59 2020

System initialization log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-02-02_03-04-07PM.log.

Session log file is /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-02-02_03-04-47PM.log
The id for this session is F3V4

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/12.2.0.1/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Preparing to bring down database service on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Successfully prepared home /u01/app/oracle/product/12.2.0.1/dbhome_1 to bring down database service


Bringing down database service on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Database service successfully brought down on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Performing prepatch operation on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Start applying binary patch on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Successfully executed command: /usr/sbin/slibclean

Binary patch applied successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Performing postpatch operation on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Starting database service on home /u01/app/oracle/product/12.2.0.1/dbhome_1
Database service successfully started on home /u01/app/oracle/product/12.2.0.1/dbhome_1


Preparing home /u01/app/oracle/product/12.2.0.1/dbhome_1 after database service restarted
No step execution required.........


Trying to apply SQL patch on home /u01/app/oracle/product/12.2.0.1/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/12.2.0.1/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:PRODHOSTdb02
RAC Home:/u01/app/oracle/product/12.2.0.1/dbhome_1
Version:12.2.0.1.0
Summary:

==Following patches were SKIPPED:

Patch: /backup/PATCH/Jan2020/30501932/30586063
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/26839277
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /backup/PATCH/Jan2020/30501932/30591794
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /backup/PATCH/Jan2020/30501932/30585969
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_15-05-26PM_1.log

Patch: /backup/PATCH/Jan2020/30501932/30593149
Log: /u01/app/oracle/product/12.2.0.1/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-02_15-05-26PM_1.log


Patching session reported following warning(s):
_________________________________________________

[WARNING] The database instance 'TESTDB2' from '/u01/app/oracle/product/12.2.0.1/dbhome_1', in host'PRODHOSTdb02' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.

OPatchauto session completed at Sun Feb  2 15:09:45 2020
Time taken to complete the session 5 minutes, 47 seconds
root@PRODHOSTDB02:[/]#
root@PRODHOSTDB02:[/]#
root@PRODHOSTDB02:[/]#su - oracle
oracle@PRODHOSTDB02:[/home/oracle]$
oracle@PRODHOSTDB02:[/home/oracle]$
oracle@PRODHOSTDB02:[/home/oracle]$srvctl start instance -d TESTDB -i TESTDB2
oracle@PRODHOSTDB02:[/home/oracle]$
oracle@PRODHOSTDB02:[/home/oracle]$srvctl status database  -d TESTDB
Instance TESTDB1 is running on node PRODHOSTdb01
Instance TESTDB2 is running on node PRODHOSTdb02

oracle@PRODHOSTDB02:[/home/oracle]$
---------------------------------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~END of output of patch application on binary on node-2~~~~~~~~~~~~~~~~~~~~~~~~~~



####==>Datapatch application: only should be run from one node only.

==>Run as an oracle user


cd $ORACLE_HOME/OPatch
./datapatch -verbose

~~~~~~~~~~~~~~~~~~~~~~ ==>o/p of :Datapatch application: only should be run from one node only.~~~~~~~~~~~~~~~~~~~~~~
-------------
oracle@PRODHOSTDB02:[/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch]$./datapatch -verbose
SQL Patching tool version 12.2.0.1.0 Production on Sun Feb  2 15:21:24 2020
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_34800046_2020_02_02_15_21_24/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Bundle series DBRU:
  ID 200114 in the binary registry and not installed in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
  Nothing to roll back
  The following patches will be applied:
    30593149 (DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114)

Installing patches...

Patch installation complete.  Total patches installed: 1

Validating logfiles...
Patch 30593149 apply: WITH ERRORS (RETRYABLE)
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30593149/23339232/30593149_apply_TESTDB_2020Feb02_15_22_57.log (errors)
    Error at line 10007: ORA-04021: timeout occurred while waiting to lock object SYS.DBMS_AQ
    Error at line 10051: ORA-04021: timeout occurred while waiting to lock object SYS.DBMS_AQADM_VAR
    Error at line 10057: ORA-04021: timeout occurred while waiting to lock object SYS.DBMS_AQADM_SYS

Adding patches to retry installation queue...
Retry installation queue:
  Nothing to roll back
  The following patches will be applied:
    30593149 (DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114)

Installing patches...

Patch installation complete.  Total patches installed: 1

Validating logfiles...
Patch 30593149 apply: SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30593149/23339232/30593149_apply_TESTDB_2020Feb02_16_11_06.log (no errors)
SQL Patching tool complete on Sun Feb  2 16:21:59 2020
oracle@PRODHOSTDB02:[/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch]$
oracle@PRODHOSTDB02:[/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch]$

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~End of datapatch application~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-----While datapatch faced issue of below as AQ job were running: If you also encounter similar issue while apply datapatch and $AQ- Advanced queue jobs are running then refer this article for the solution. :datapatch execution takes time and fails with ORA-04021 as AQ jobs are running while PSU Patching on 12c


~~>Check the status of datapatch/modified SQL

set line 500
col patch_id for 9999999999
col patch_uid for 9999999999
col version for a10
col action for a10
col install_id  for 99
col action_time for a30
col description for a60
col bundle_series for a20
col status for a30

select patch_id, patch_uid, version, action, install_id, action_time, description, bundle_series,status  from dba_registry_sqlpatch;

SQL> set line 500
SQL> col patch_id for 9999999999
SQL> col patch_uid for 9999999999
SQL> col version for a10
SQL> col action for a10
SQL> col install_id  for 99
SQL> col action_time for a30
SQL> col description for a60
SQL> col bundle_series for a20
SQL> col status for a30
SQL>
SQL> select patch_id, patch_uid, version, action, install_id, action_time, description, bundle_series,status  from dba_registry_sqlpatch;

   PATCH_ID   PATCH_UID VERSION    ACTION     INSTALL_ID ACTION_TIME                    DESCRIPTION                                                  BUNDLE_SERIES        STATUS
----------- ----------- ---------- ---------- ---------- ------------------------------ ------------------------------------------------------------ -------------------- ------------------------------
   30593149    23339232 12.2.0.1   APPLY               1 02-FEB-20 04.11.06.311937 PM   DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114             DBRU                 WITH ERRORS (RETRYABLE)
   30593149    23339232 12.2.0.1   APPLY               1 02-FEB-20 04.21.59.239175 PM   DATABASE JAN 2020 RELEASE UPDATE 12.2.0.1.200114             DBRU                 SUCCESS

SQL>


Hope this helps.