On Demand hot extend non-clustered Oracle Disks online without downtime

The first blog focused on how to Scale up CPU and Memory by hot adding CPU and Memory on the fly to an Oracle workload which can be found here.

This 2^nd blog in the on-demand Scale up series focused on how to Hot Add / Hot Remove vmdk’s to / from a VM running an Oracle workload without any application downtime which can be found here.

This blog focuses on the OS and Oracle database steps to be taken once non-clustered vmdk(s) are hot extended while an Oracle database using ASM is running. Current restriction of shared vmdk’s using the multi-writer attribute is that Hot extend virtual disk is disallowed as per KB 1034165 & KB 2121181

vmdk(s) – Non-Clustered and Clustered

Contents

1 vmdk(s) – Non-Clustered and Clustered
2 Oracle Automatic Storage Management (ASM) and ASM Online Rebalancing
3 Linux udev
4 RHEL 7 Hot Extend
5 Test Setup for Hot Extend of non-clustered vmdk(s)
6 Test Steps
7 Summary

VMFS is a clustered file system that disables (by default) multiple virtual machines from opening and writing to the same virtual disk (.vmdk file). This prevents more than one virtual machine from inadvertently accessing the same .vmdk file. The multi-writer option allows VMFS-backed disks to be shared by multiple virtual machines.

As we all are aware of, Oracle RAC requires shared disks to be accessed by all nodes of the RAC cluster. KB 1034165 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s. Requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is

has to set to Eager Zero Thick provisioned
need not be set to Independent persistent

KB 2121181 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s on a VMware vSAN environment.

Starting VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001), the requirement to have the shared disks as Eager Zero Thick (EZT) has been removed, so the RAC shared disks can now be thin provisioned.

This applies to other clustered applications as well running on vSAN which uses multi-writer disks for clustering purposes e.g. Oracle RAC, Redhat Clustering, Veritas Infoscale etc

More information on running Oracle RAC on vSAN 6.7 P01 with thin provisioned vmdk’s can be found here.

Supported and Unsupported Actions or Features with Multi-Writer Flag ( KB 1034165 & KB 2121181 )

Current restriction of shared vmdk’s using the multi-writer attribute is that Hot extend virtual disk is disallowed as per KB 1034165 & KB 2121181

Oracle Automatic Storage Management (ASM) and ASM Online Rebalancing

Oracle ASM is Oracle’s recommended storage management solution that provides an alternative to conventional volume managers, file systems, and raw devices. Oracle ASM is a volume manager and a file system for Oracle Database files that supports single-instance Oracle Database and Oracle Real Application Clusters (Oracle RAC) configurations. Oracle ASM uses disk groups to store data files.

More information on Oracle ASM can be found here.

You can add or drop ASM disks to / from an ASM diskgroup online without downtime. After you add new disks, the new disks gradually begin to accommodate their share of the workload as rebalancing progresses. Oracle ASM automatically rebalances disk groups when their configuration changes, including changes to file group. However, you might want to do a manual rebalance operation to control the speed of what would otherwise be an automatic rebalance operation.

More information on Oracle ASM rebalancing can be found here.

Linux udev

Udev is the mechanism used to create and name /dev device nodes corresponding to the devices that are present in the system. Udev uses matching information provided by sysfs with rules provided by the user to dynamically add the required device nodes. To preserve the device names across reboots in Linux, udev rules is used.

More information on Linux custom udev rules in RHEL7 can be found here,

The SCSI device’s unique ID needs to be extracted and assigned to a symbolic name which will persist across reboots. To get the SCSI device’s unique ID, Linux command scsi_id can be used.

The key-value pair “disk.EnableUUID = “TRUE”” parameter needs to be added to the .vmx file for the VM to present the SCSI ID of the device.

[root@sb_ol76_ora19c ~]# /usr/lib/udev/scsi_id -gud /dev/sdc
36000c298e708af163b3f024b2b9421df
[root@sb_ol76_ora19c ~]#

More information on this can be found in the RedHat article “How to check ‘disk.EnableUUID’ parameter from VM in vSphere“.

RHEL 7 Hot Extend

Please refer to the RedHat article “Does RHEL 7 support online resize of disk partitions?”

As to older style partitions, this feature has been added in RHEL 7 current release with a feature request (RFE has been filed to add support for online resizing of disk partitions to RHEL 7 in private RHBZ#853105. With this feature, it’s possible to resize the disk partitions online in RHEL 7”

Test Setup for Hot Extend of non-clustered vmdk(s)

VM ‘SB-OL-Ora19C-HotDisk’ was created on ESXI 7.0 platform with OS OEL 7.6 UEK with Oracle 19c Grid Infrastructure & RDBMS installed. Oracle ASM was the storage platform with Oracle ASMLIB. Oracle ASMFD can also be used instead of Oracle ASMLIB.

The rest of the steps, whether we use Oracle ASMLIB or Oracle ASMFD or Linux udev, are the same when extending the size of Oracle ASM disk(s) of an Oracle ASM disk group.

As mentioned before, Current restriction of shared vmdk’s using the multi-writer attribute is that Hot extend virtual disk is disallowed as per KB 1034165 & KB 2121181

Details of the VM ‘SB-OL-Ora19C-HotDisk’ :

The VM has 3 vmdks:

Hard Disk 1 of 60G is for the OS
Hard Disk 2 of 60G is for the Oracle 19c Grid Infrastructure & RDBMS binaries
Hard Disk 3 of 600G is for the Oracle database which is using Oracle ASM storage with Oracle ASMLIB and is at SCSI position SCSI 2:0

Details of Hard Disk 3 600GB Oracle ASM Disk:

OS Details of the Oracle ASM Disk of size 600GB:

[root@sb_ol76_ora19c ~]# fdisk -lu /dev/sdc

Disk /dev/sdc: 644.2 GB, 644245094400 bytes, 1258291200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc027aaea

Device Boot Start End Blocks Id System
/dev/sdc1 2048 1258291199 629144576 83 Linux
[root@sb_ol76_ora19c ~]#

Oracle ASM and Oracle Database are online:

oracle@sb_ol76_ora19c:ora19c:/home/oracle> smon
0 S grid 11289 1 0 80 0 – 388557 – 17:06 ? 00:00:00 asm_smon_+ASM
0 S oracle 12095 1 0 80 0 – 640217 – 17:16 ? 00:00:00 ora_smon_ora19c
oracle@sb_ol76_ora19c:ora19c:/home/oracle>

List of Oracle ASM disks :

grid@sb_ol76_ora19c:+ASM:/home/grid> asmcmd lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Site_Name Site_GUID Site_Status Failgroup_Type Library Label Failgroup_Label Site_Label UDID Product Redund Path
614396 597492 614399 DATA_DISK02 DATA_DISK02 00000000000000000000000000000000 REGULAR ASM Library – Generic Linux, version 2.0.12 (KABI_V2) DATA_DISK02 UNKNOWN ORCL:DATA_DISK02
grid@sb_ol76_ora19c:+ASM:/home/grid>

Test Steps

ASM diskgroup DATA_DG has an existing ASM disk ‘DATA_DISK02’ of size 600GB at SCSI 2:0 position
Extend the existing disk from size 600GB to size 700GB
Observe change of disk size at OS and Oracle ASM level and report any anomalies if any

NO Changes in the size :

[root@sb_ol76_ora19c ~]# fdisk -lu /dev/sdc

Device Boot Start End Blocks Id System
/dev/sdc1 2048 1258291199 629144576 83 Linux
[root@sb_ol76_ora19c ~]#

Rescan the OS to see the new added disk and check via fdisk command:

[root@sb_ol76_ora19c ~]# fdisk -lu /dev/sdc

Disk /dev/sdc: 751.6 GB, 751619276800 bytes, 1468006400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc027aaea

Device Boot Start End Blocks Id System
/dev/sdc1 2048 1258291199 629144576 83 Linux
[root@sb_ol76_ora19c ~]#

[root@sb_ol76_ora19c ~]# tail -100 /var/log/messages
…
Aug 14 12:49:09 sb_ol76_ora19c kernel: sd 3:0:0:0: [sdc] 1468006400 512-byte logical blocks: (751 GB/700 GiB)
Aug 14 12:49:09 sb_ol76_ora19c kernel: sdc: detected capacity change from 644245094400 to 751619276800
[root@sb_ol76_ora19c ~]# tail -1000 /var/log/messages

The OS sees the increased size , ASM does not

grid@sb_ol76_ora19c:+ASM:/home/grid> asmcmd lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Site_Name Site_GUID Site_Status Failgroup_Type Library Label Failgroup_Label Site_Label UDID Product Redund Path
614396 597492 614399 DATA_DISK02 DATA_DISK02 00000000000000000000000000000000 REGULAR ASM Library – Generic Linux, version 2.0.12 (KABI_V2) DATA_DISK02 UNKNOWN ORCL:DATA_DISK02
grid@sb_ol76_ora19c:+ASM:/home/grid>

ASM and Database are both online
oracle@sb_ol76_ora19c:ora19c:/home/oracle> smon
0 S grid 11309 1 0 80 0 – 388557 – Aug13 ? 00:00:00 asm_smon_+ASM
0 S oracle 12034 1 0 80 0 – 641036 – Aug13 ? 00:00:01 ora_smon_ora19c
oracle@sb_ol76_ora19c:ora19c:/home/oracle>

Delete and Recreate the partition table to see the new disk size

[root@sb_ol76_ora19c ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): d
Selected partition 1
Partition 1 is deleted

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-1468006399, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1468006399, default 1468006399):
Using default value 1468006399
Partition 1 of type Linux and of size 700 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@sb_ol76_ora19c ~]#

[root@sb_ol76_ora19c ~]# partx -u /dev/sdc

[root@sb_ol76_ora19c ~]# fdisk -lu /dev/sdc

Disk /dev/sdc: 751.6 GB, 751619276800 bytes, 1468006400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc027aaea

Device Boot Start End Blocks Id System
/dev/sdc1 2048 1468006399 734002176 83 Linux
[root@sb_ol76_ora19c ~]#

ASM still shows old disk size. Run the below command to resize the diskgroup. The new size is written to the Oracle ASM disk header. More information can be found here.

grid@sb_ol76_ora19c:+ASM:/home/grid> sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 – Production on Thu Aug 13 20:18:53 2020
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.3.0.0.0

SQL> alter diskgroup DATA_DG resize all;
Diskgroup altered.
SQL>
SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.3.0.0.0
grid@sb_ol76_ora19c:+ASM:/home/grid>

The database alert log file shows operation is successful

tail -f alert_ora19c.log
…
2020-08-14T12:54:48.283665-07:00
SUCCESS: disk DATA_DISK02 (1.4042325815) re-identified in diskgroup DATA_DG

ASM still shows new disk size

grid@sb_ol76_ora19c:+ASM:/home/grid> asmcmd lsdsk -k
Total_MB Free_MB OS_MB Name Failgroup Site_Name Site_GUID Site_Status Failgroup_Type Library Label Failgroup_Label Site_Label UDID Product Redund Path
716796 699892 716799 DATA_DISK02 DATA_DISK02 00000000000000000000000000000000 REGULAR ASM Library – Generic Linux, version 2.0.12 (KABI_V2) DATA_DISK02 UNKNOWN ORCL:DATA_DISK02
grid@sb_ol76_ora19c:+ASM:/home/grid>

Summary

We can Hot Extend a vmdk(s) of a VM running an Oracle workload without any application downtime.
Capability to resize the disk partitions online is on the OS and the version of the OS being used
If the ASM disks have partition headers—Repartition each disk in the ASM diskgroup. Note that the Linux parted tool will warn that the contents of the disk will be lost. With ASM this is not the case and the warning can be safely ignored

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html