On Demand hot extend clustered vmdk’s online without downtime – Hot Extend RAC clustered disks

This blog focuses on the complete list of steps to be taken to increase the size of the clustered vmdk(s) using Multi-writer flag of an Oracle RAC Cluster. This blog describes the various options using both Native Oracle tools and vSphere tools to attempt to increase the size of the Oracle RAC clustered vmdk(s).

Current restriction of shared vmdk’s using the multi-writer attribute is that Hot extend virtual disk is disallowed as per KB 1034165 & KB 2121181

 

 

 

vmdk(s) – Non-Clustered and Clustered

VMFS is a clustered file system that disables (by default) multiple virtual machines from opening and writing to the same virtual disk (.vmdk file). This prevents more than one virtual machine from inadvertently accessing the same .vmdk file. The multi-writer option allows VMFS-backed disks to be shared by multiple virtual machines.

As we all are aware of, Oracle RAC requires shared disks to be accessed by all nodes of the RAC cluster. KB 1034165 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s.

Requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is

• has to set to Eager Zero Thick provisioned
• need not be set to Independent persistent

KB 2121181 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s on a VMware vSAN environment.

Supported and Unsupported Actions or Features with Multi-Writer Flag ( KB 1034165 & KB 2121181 )

Current restriction of shared vmdk’s using the multi-writer attribute is that Hot extend virtual disk is disallowed as per KB 1034165 & KB 2121181

 

RHEL 7 Hot Extend

Please refer to the RedHat article “Does RHEL 7 support online resize of disk partitions?”

As to older style partitions, this feature has been added in RHEL 7 current release with a feature request (RFE has been filed to add support for online resizing of disk partitions to RHEL 7 in private RHBZ#853105. With this feature, it’s possible to resize the disk partitions online in RHEL 7”

Oracle Automatic Storage Management (ASM) and ASM Online Rebalancing

Oracle ASM is Oracle’s recommended storage management solution that provides an alternative to conventional volume managers, file systems, and raw devices. Oracle ASM is a volume manager and a file system for Oracle Database files that supports single-instance

Oracle Database and Oracle Real Application Clusters (Oracle RAC) configurations. Oracle ASM uses disk groups to store data files.

More information on Oracle ASM can be found here.

You can add or drop ASM disks to / from an ASM diskgroup online without downtime. After you add new disks, the new disks gradually begin to accommodate their share of the workload as rebalancing progresses. Oracle ASM automatically rebalances disk groups when their configuration changes, including changes to file group. However, you might want to do a manual rebalance operation to control the speed of what would otherwise be an automatic rebalance operation.

More information on Oracle ASM rebalancing can be found here.

Oracle RAC Setup

An 2 node Oracle RAC Cluster ‘rdmrac’ was created with 2 RAC VM’s, ‘rdmrac1’ and ‘rdmrac2’.

Both RAC VM’s ‘rdmrac1’ and ‘rdmrac2’. were created on ESXI 7.0 platform with OS OEL 7.9 UEK with Oracle 19c Grid Infrastructure & RDBMS installed. Oracle ASM was the storage platform with Oracle ASMLIB. Oracle ASMFD can also be used instead of Oracle ASMLIB.

The rest of the steps, whether we use Oracle ASMLIB or Oracle ASMFD or Linux udev, are the same when extending the size of Oracle ASM disk(s) of an Oracle ASM disk group.

 

Details of the RAC VM’s ‘rdmrac1’ and ‘rdmrac2’

Both RAC VM’s ‘rdmrac1’ and ‘rdmrac2’ have 3 vmdks:

• Non-clustered Hard Disk 1 of 80G is for the OS
• Non-clustered Hard Disk 2 of 80G is for the Oracle 19c Grid Infrastructure & RDBMS binaries
• Clustered Hard Disk 3 of 120G is for the Oracle Cluster which is using Oracle ASM storage with Oracle ASMLIB and is at SCSI position SCSI 1:0

Lets check the clustered vmdk disk specifications:

To determine if the VMDK is zeroedthick or eagerzeroedthick , see KB Determining if a VMDK is zeroedthick or eagerzeroedthick (1011170) – TBZ represents the number of blocks in the disk remaining To Be Zeroed.

The output of the command below shows ‘tbz 0’ ie the disk is Eager Zero Thick (EZT)

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -D /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk
Lock [type 10c00001 offset 7725056 v 1009, hb offset 3932160
gen 57, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1927428
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 0, 31>, gen 979, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 449, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536
affinityFD <4,0,18>, parentFD <4,0,18>, tbzGranularityShift 20, numLFB 0
lastSFBClusterNum 0, numPreAllocBlocks 0, numPointerBlocks 0
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

Test Cases

Using Oracle ASM Rebalance & Drop old vmdk with POWER clause enables new vmdk(s) with increased size ONLINE, WITHOUT ANY DOWNTIME. (without having to physically increase the size of the old vmdk(s))

The details can be found here “RAC” n “RAC” all night – Oracle RAC on vSphere 6.x – Add shared vmdk online without downtime for Oracle RAC ASM / OCFS2

Below are the different test cases , not using Native Oracle tools , to test hot extend / increase clustered vmdk(s) size

• Powered on RAC Cluster
  • Extend clustered vmdk(s) of a Powered on RAC Cluster using Web Client
  • Extend clustered vmdk(s) of a Powered on RAC Cluster using vmkfstools

• Powered off RAC Cluster
  • Extend clustered vmdk(s) of a Powered off RAC Cluster using Web Client

Test Case Summary

Summary of the results of the test cases to hot extend / increase clustered vmdk(s) size

• Powered on RAC Cluster
  • Extend clustered vmdk(s) of a Powered on RAC Cluster using Web Client OR vmkfstools
    • With the RAC Cluster online (RAC VM’s are powered up), trying to hot extend the size of the clustered vmdk(s), using either the Web Client OR the vmkfstools command results in an ERROR which is as expected as per KB 1034165

• Powered off RAC Cluster – the rest of the steps to see the new increased size on GOS & Oracle ASM is the same for either of the 2 use cases below.
  • Extend clustered vmdk(s) of a Powered off RAC Cluster using Web Client
    • Hot extending EZT disks using Web Client ends up I the vmdk being a LZT after the resize operation
    • We would have to convert the LZT to EZT using vmkfstools command
    • We would the have to reload the VM’s using the ‘vim-cmd vmsvc/reload’ command to see the clustered vmdk(s) as EZT with the new extended size
  • Extend clustered vmdk(s) of a Powered off RAC Cluster using vmkfstools
    • We can increase the size of the clustered vmdk(s) using vmkfstools and retain the disk specification as EZT disks

Test Case – Extend clustered vmdk(s) of a Powered on RAC Cluster using Web Client or vmkfstools

We are well aware that Current restriction of shared vmdk’s using the multi-writer attribute is that Hot extend virtual disk is disallowed as per KB 1034165 & KB 2121181

Trying to hot extend the size of the clustered vmdk from120G to 150G , while the RAC Cluster is online (RAC VM’s are powered up) , fails with the below error which is as expected.

Let’s see of we can try hot extending the size of the clustered vmdk from120G to 150G using vmkfstools while the RAC Cluster is online (RAC VM’s are powered up).

KB Powering on the virtual machine fails with the error: Thin/TBZ disks cannot be opened in multiwriter mode (1033570)

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -X 120G -d eagerzeroedthick /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk

Failed to open virtual disk (/vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk): Failed to lock the file (16392)

With the RAC Cluster online (RAC VM’s are powered up), trying to hot extend the size of the clustered vmdk(s), using either the Web Client OR the vmkfstools command results in an error which is as expected as per KB 1034165.

Test Case – Extend clustered vmdk(s) of a Powered off RAC Cluster using Web Client

Let’s see if we can use the Web Client to increase the size of the clustered vmdk(s) after shutting the RAC Cluster and powering off the RAC VM’s.

Shutdown RAC Cluster and power off RAC VM’s ‘rdmrac1’ and ‘rdmrac2; Using the Web Client increase the size of the clustered vmdk from120G to 150G and record the observations.

2 observations

• We get an error message ‘Incompatible device backing specified for device ‘0’. But an interesting observation is the vmdk size is increased from 120GB to 150GB!!!
• The clustered vmdk is now provisioned as ‘Lazy Zero Thick’ instead of ‘Eager Zero Thick’

Obviously, powering on the RAC VM ‘rdmrac1’ will result in a failure, as per RAC cluster. KB 1034165 , requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is has to set to “Eager Zero Thick” provisioned on VMFS datastores.

Running the vmkfstools -D option gives ‘tbz 0’

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -D /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk
Lock [type 10c00001 offset 7725056 v 995, hb offset 3866624
gen 73, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1218251
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 0, 31>, gen 983, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 449, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536
affinityFD <4,0,18>, parentFD <4,0,18>, tbzGranularityShift 20, numLFB 0
lastSFBClusterNum 0, numPreAllocBlocks 0, numPointerBlocks 0
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

However, running the command ‘vmkfstools -t0’ with -t0 option shows  that the vmdk is a Lazy Zero Thick disk’ as per KB Determining if a VMDK is zeroedthick or eagerzeroedthick (1011170)

The blocks that contain a Z have not yet been written to and are zeroed in the first attempt of the virtual machine to write data on that block , hence its is a Lazy Zero thick vmdk.

KB Attempts to extend the size of an EagerZeroedThick VMDK from the vSphere Client might result in a LazyZeroedThick VMDK (2054563) explains why when we attempt to extend the size of an EagerZeroedThick VMDK from the vSphere Client, the extended part of the disk becomes LazyZeroedThick.

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -t0 /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk
Mapping for file /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk (161061273600 bytes in size):
[ 0: 168820736] –> [VMFS — LVID:5faa0685-b26b9948-2ed5-e4434b2d2ca8/5faa0685-a3312f2c-bafc-e4434b2d2ca8/1:( 1832995782656 –> 1833164603392)]
[ 168820736: 99614720] –> [VMFS — LVID:5faa0685-b26b9948-2ed5-e4434b2d2ca8/5faa0685-a3312f2c-bafc-e4434b2d2ca8/1:( 1832896167936 –> 1832995782656)]
…..
….
[ 128849018880: 536870912] –> [VMFS Z– LVID:5faa0685-b26b9948-2ed5-e4434b2d2ca8/5faa0685-a3312f2c-bafc-e4434b2d2ca8/1:( 2098647269376 –> 2099184140288)]
[ 129385889792: 536870912] –> [VMFS Z– LVID:5faa0685-b26b9948-2ed5-e4434b2d2ca8/5faa0685-a3312f2c-bafc-e4434b2d2ca8/1:( 2099184140288 –> 2099721011200)]
[ 160524402688: 536870912] –> [VMFS Z– LVID:5faa0685-b26b9948-2ed5-e4434b2d2ca8/5faa0685-a3312f2c-bafc-e4434b2d2ca8/1:( 2174882938880 –> 2175419809792)]
…..
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

Checking RAC VM ‘rdmrac2’, we can see the vmdk still shows up as EZT with 120GB size.?

Reloading the RAC VM ‘rdmrac2’ .vmx file can help in refreshing the VM specifications without having to remove the VM from inventory and registering the VM again. Refer to KB article Reloading a vmx file without removing the virtual machine from inventory (1026043) shows how to reload the virtual machine’s .vmx configuration file.

[root@sc2esx12:~] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
42 rdmrac2 [OraTintri] rdmrac2/rdmrac2.vmx oracleLinux7_64Guest vmx-17
[root@sc2esx12:~]

[root@sc2esx12:~] vim-cmd vmsvc/reload 42

Now RAC VM ‘rdmrac2’ shows the clustered vmdk as LZT with 150GB size.

Now the task to convert the LZT vmdk to EZT vmdk – vmkfstools to our rescue. KB Determining if a VMDK is zeroedthick or eagerzeroedthick (1011170) recommends using ‘vmkfstools -k’ command.

This command allows you to convert a preallocated virtual disk to eagerzeroedthick and maintains any existing data. The virtual machine must be powered off to run this command.

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -k “/vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk”
vmfsDisk: 1, rdmDisk: 0, blockSize: 1048576
Eagerly zeroing: 100% done.
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

You would need to reload the RAC VMs ‘rdmrac1’ and ‘rdmrac2’ .vmx file which can help in refreshing the VM specifications.

KB article Reloading a vmx file without removing the virtual machine from inventory (1026043) shows how to reload the virtual machine’s .vmx configuration file.

We can now see that the RAC VMs ‘rdmrac1’ and ‘rdmrac2’ clustered vmdk is now EZT.

Now we can power on the RAC VM’s

 

Test Case – Extend clustered vmdk(s) of a Powered off RAC Cluster using vmkfstools

Shutdown RAC Cluster and power off RAC VM’s ‘rdmrac1’ and ‘rdmrac2;

Use the vmkfstools command to increase the size of the clustered vmdk from120G to 150G and record the observations. This command needs to be done only one time against the RAC VM clustered vmdk(s).

 

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -D /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk
Lock [type 10c00001 offset 7725056 v 997, hb offset 3866624
gen 73, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1224637
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 0, 31>, gen 989, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 449, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536
affinityFD <4,0,18>, parentFD <4,0,18>, tbzGranularityShift 20, numLFB 0
lastSFBClusterNum 0, numPreAllocBlocks 0, numPointerBlocks 0
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

 

Run the vmkfstools command to resize the clustered vmdk from 120G to 150G and keep the vmdk as EZT

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -X 150G -d eagerzeroedthick /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk
Grow: 100% done.All data on ‘/vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk’ will be overwritten with zeros from sector <251658240> onwards.
vmfsDisk: 1, rdmDisk: 0, blockSize: 1048576
Zeroing: 100% done.
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

Run the vmkfstools command to check if the disk is still EZT

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vmkfstools -D /vmfs/volumes/OraPure/rdmrac1/rdmrac1.vmdk
Lock [type 10c00001 offset 7725056 v 1005, hb offset 3932160
gen 57, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2226558
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 0, 31>, gen 989, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 449, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536
affinityFD <4,0,18>, parentFD <4,0,18>, tbzGranularityShift 20, numLFB 0
lastSFBClusterNum 0, numPreAllocBlocks 0, numPointerBlocks 0
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

  

We can see the new disk size is 150G (314572800 * 512bytes)

[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] cat rdmrac1.vmdk
# Disk DescriptorFile
version=1
encoding=”UTF-8″
CID=84e9d186
parentCID=ffffffff
createType=”vmfs”

# Extent description
RW 314572800 VMFS “rdmrac1-flat.vmdk”

# The Disk Data Base
#DDB

ddb.adapterType = “lsilogic”
ddb.geometry.cylinders = “19581”
ddb.geometry.heads = “255”
ddb.geometry.sectors = “63”
ddb.longContentID = “399a78939255abddb484895b84e9d186”
ddb.uuid = “60 00 C2 9d c2 ed 23 67-ef 80 dc 1a f2 69 cd f3”
ddb.virtualHWVersion = “14”
[root@sc2esx11:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1]

 

  

We still see the old vmdk size for the RAC VM’s on the Web Client. You would need to reload the RAC VMs ‘rdmrac1’ and ‘rdmrac2’ .vmx file which can help in refreshing the VM specifications.

KB article Reloading a vmx file without removing the virtual machine from inventory (1026043) shows how to reload the virtual machine’s .vmx configuration file.

[root@sc2esx11:~] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
..
38 rdmrac2 [OraTintri] rdmrac2/rdmrac2.vmx oracleLinux7_64Guest vmx-17
[root@sc2esx11:~] vim-cmd vmsvc/reload 38

[root@sc2esx12:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
…
44 rdmrac1 [OraTintri] rdmrac1/rdmrac1.vmx oracleLinux7_64Guest vmx-17
[root@sc2esx12:/vmfs/volumes/5faa0685-b4cf32fa-c4e4-e4434b2d2ca8/rdmrac1] vim-cmd vmsvc/reload 44

We can see the clustered vmdk with new size 150G and EZT

Steps to see the new increased size by the OS and Oracle database

 

The rest of the steps to see the new increased size by the OS and Oracle database is the same for both of the above use cases “Extend clustered vmdk(s) of a Powered off RAC Cluster using vmkfstools and Web client”

• Rescan the OS to see the increased disk size and check via fdisk command
• The OS sees the increased size at this point of time, Oracle ASM does not see the increased size
• Delete and Recreate the partition table to see the new disk size
• Run the ‘partx -u /dev/sdX’ command to update the partition table in memory
• ASM still shows old disk size. Run the ‘alter diskgroup XXXX resize all;’ command to
  to resize the diskgroup. The new size is written to the Oracle ASM disk header

The details of the OS rescan, OS repartition , ASM rebalance and ASM drop process can be found here On Demand hot extend non-clustered Oracle Disks online without downtime – Hot Extend Disks

Summary

• Powered on RAC Cluster
  • Extend clustered vmdk(s) of a Powered on RAC Cluster using Web Client OR vmkfstools
    • With the RAC Cluster online (RAC VM’s are powered up), trying to hot extend the size of the clustered vmdk(s), using either the Web Client OR the vmkfstools command results in an ERROR which is as expected as per KB 1034165

• Powered off RAC Cluster – the rest of the steps to see the new increased size on GOS & Oracle ASM is the same for either of the 2 use cases below.
  • Extend clustered vmdk(s) of a Powered off RAC Cluster using Web Client
    • Hot extending EZT disks using Web Client ends up I the vmdk being a LZT after the resize operation
    • We would have to convert the LZT to EZT using vmkfstools command
    • We would the have to reload the VM’s using the ‘vim-cmd vmsvc/reload’ command to see the clustered vmdk(s) as EZT with the new extended size
  • Extend clustered vmdk(s) of a Powered off RAC Cluster using vmkfstools
    • We can increase the size of the clustered vmdk(s) using vmkfstools and retain the disk specification as EZT disks

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html