Patching, Upgrading Operating Systems on Servers running Oracle Database

Introduction

Today, enterprises are mandated to adhere to strict security practices, which means applying patches and upgrading operating systems (OS) is more frequent activity. Applications such as databases running in user mode typically remain unaffected by these patching activities even if the OS patch include kernel changes. In fact, kernel modules running in privileged mode also remain unaffected as these modules utilize kABI (Kernel Application Binary Interface) which is a Linux functionality that allows kernel drivers (and kernel modules) to interact with the kernel even if the kernel version changes.

Lately this functionality was found broken on Redhat RHEL 8.7 (Kernel version 4.18.0-425.3.1.el8.x86_64) and reported to Redhat via RedHat bugzilla bugs #2144161 and #2152206. This affects all third-party kernel modules built on kernel version 4.18.0-425.3.1.el8 including Oracle Databases using kernel modules like Oracle Automatic Storage Management Filter Driver (ASMFD) or Oracle Automatic Cluster File System (ACFS). Due to these bugs any OS patching operation that changes the OS kernel results in Oracle database startup failures as the OS kernel load kernel drivers such as ASMFD or ACFS cannot start and therefore Oracle Automatic Storage Management (ASM) cannot mount the disk groups as it cannot access the storage devices.

Solution

Customers planning to upgrade or apply any OS fix can execute a precheck command to determine if the new OS version is compatible with the filter driver. This precheck command can be invoked by running the following cluvfy command and passing the target kernel version.

cluvfy stage -pre osupg -n mycluster1 -checkGIdriverSupport 5.4.17-213
  ACFS Driver Checks ...FAILED
  Checking if Oracle ACFS driver from CRS home "/scratch/u01/app/grid" is
  compatible with kernel "5.4.17-2136" ...FAILED
  stbmXXXXXX: PRVH-0573 : Oracle ACFS driver from CRS home
       "/scratch/u01/app/grid" is supported on the current kernel
        version of the operating system, but will not be supported on
         the specified target kernel "5.4.17-21366        

** The checkGIdriverSupport option is included with Oracle Grid Infrastructure (GI) 19.17 and higher releases.

If the check reports that the target operating system is 

• Supported with the target OS version: Then the Oracle GI and the database should be able to start post the patching activity without any further intervention
• Not Supported with the target OS version: Then refer to My Oracle Support Note 1369107.1 and apply the release update that includes the drivers supported on the target kernel. If the note does not have the target OS version, then customers using AFD (not ACFS) can choose to bypass the ASMFD. This means the customer needs to configure udev rules to ensure device persistency and set the ASM disk string to match the block devices. Customers using ACFS may need to open an SR with Oracle Support if Note 1369107.1 does not have a RU which contains the kernel modules for their target OS version.

  Special thanks to Pawan Tare and Allan Graves for helping with this feature