Oracle RAC storage migration from non-vSAN to vSAN 6.7 P01 – Through Thick to Thin

Storage migration of an Oracle RAC from non-vSAN Storage to vSAN 6.7 P01 – No remediation required for shared disks anymore

This blog is a follow up of the previous blog post “Oracle RAC on vSAN 6.7 P01 – No more Eager Zero Thick requirement for shared vmdk’s”.

This blog describes the steps to storage vMotion an Oracle RAC cluster with shared EZT vmdk’s from a non-vSAN (FC / NAS / iSCSI ) storage to a vSAN 6.7 P01 storage and highlights the fact that apart from putting the multi-writer attribute back on the shared vmdk’s , there is no need for any further reconfiguration of the shared vmdk’s.

2 test cases were performed against the RAC cluster. The test cases are to Storage migrate a 2 node Oracle RAC from

• 1st vSphere Cluster connected to an All Flash FC Storage array with VMFS6 datastores TO a 2nd VMware vSAN 6.7 P01 Cluster
• 2nd VMware vSAN 6.7 P01 Cluster back to the 1st vSphere Cluster connected to an All Flash FC Storage array with VMFS6 datastores

Oracle RAC on VMware platform – Concepts

By default, in a VM, the simultaneous multi-writer “protection” is enabled for all. vmdk files ie all VM’s have exclusive access to their vmdk files. So, in order for all of the VM’s to access the shared vmdk’s simultaneously, the multi-writer protection needs to be disabled. The use case here is Oracle RAC.

More details can be found in KB 1034165 and KB 2121181.

Oracle RAC on VMware vSAN prior vSAN 6.7 P01

KB 2121181 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s on VMware vSAN .

2 requirements for sharing disks for an Oracle RAC cluster on VMware vSAN prior vSAN 6.7 P01

• the shared disk/s have to be Eager Zero Thick provisioned
• the shared disk/s need to have the multi-writer attribute set

In vSAN , the Eager Zero thick attribute can be controlled by a rule in the VM Storage Policy called ‘Object Space Reservation’ which need to be set to 100% which pre-allocates all object’s components on disk.

More details can be found in KB 2121181.

VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001)  – release date DEC 5, 2019

VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001) release date DEC 5, 2019 , addresses a key limitation when provisioning shared disks with multi-writer for Oracle RAC cluster.

The current limitation is Oracle RAC shared vmdk’s need to be Eager Zero Thick for the multi-writer attribute to be enabled for the clustering software to work.This patch resolves this issue and now one can use thin provisioned vmdk’s with multi-writer attribute for Oracle RAC cluster starting VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001).

More details can be found here. The text relevant to Oracle RAC cluster is shown as below.

PR 2407141: You cannot provision virtual disks to be shared in multi-writer mode as eager zeroed thick-provisioned by using the vSphere Client

A shared virtual disk on a vSAN datastore for use in multi-writer mode, such as for Oracle RAC, must be eager zeroed thick-provisioned. However, the vSphere Client does not allow you to provision the virtual disk as eager zeroed thick-provisioned.

This issue is resolved in this release. You can share any type of virtual disks on the vSAN datastore in multi-writer mode.

Now there is only 1 requirement for sharing disks for an Oracle RAC cluster on VMware vSAN vSAN 6.7 P01

• the shared disk/s need to have the multi-writer attribute set

What VMware vSAN 6.7 P01 means to customers is

• customers NO longer have to provision storage up-front when provisioning space for an Oracle RAC shared storage on VMware vSAN 6.7 P01 (ESXi 6.7 Patch Release ESXi670-201912001)
• The RAC shared vmdk’s can be thin provisioned to start with and can slowly consume space as and when needed.

Oracle RAC on non-vSAN storage (FC / NAS / iSCSI)

On a non-vSAN storage ( FC / NAS / iSCSI storage ) , when using the multi-writer mode, the virtual disk must be Eager Zeroed thick (EZT) ; it cannot be zeroed thick or thin provisioned.

2 requirements for sharing disks for an Oracle RAC cluster on non-vSAN storage (FC / NAS / iSCSI storage)

• the shared disk/s have to be Eager Zero Thick provisioned
• the shared disk/s need to have the multi-writer attribute set

This is described in detail in “Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag (1034165)”.

Key points to take away from this blog

Migrating Oracle RAC storage from non-vSAN storage (FC / NAS / iSCSI storage) to

• VMware vSAN prior vSAN 6.7 P01
  • Further reconfiguration of the shared vmdk’s needed ie remediation of shared disks from LZT to EZT
  • Add multi-writer attribute to the shared vmdk’s
• VMware vSAN vSAN 6.7 P01
  • No further reconfiguration of the shared vmdk’s needed
  • Only step is to Add multi-writer attribute to the shared vmdk’s

This blog focuses on validating a test case ie Storage migrate a 2 node Oracle RAC from All Flash FC Storage array with VMFS6 datastores to VMware vSAN vSAN 6.7 P01.

The reverse process ie Storage migrate a 2 node Oracle RAC from VMware vSAN vSAN 6.7 P01 back to an All Flash FC Storage array with VMFS6 datastores would be the same process as shown in the test case except the further reconfiguration of the shared vmdk’s is needed ie remediation of shared disks from LZT to EZT.

Setup – Oracle RAC on VMware 6.7, 15160138 version

Lab setup is shown as below.

We  have 2 separate VMware vSphere clusters

• 1st vSphere Cluster connected to a non-vSAN (FC / NAS / iSCSI ) storage
• 2nd vSphere Cluster connected to a vSAN 6.7 P01 storage

1st vSphere Cluster connected to a non-vSAN (FC / NAS / iSCSI ) storage

• 1st cluster is a 4 node VMware vSphere cluster VMware ESXi, 6.7.0, 15160138
• vSphere cluster was connected to an All Flash FC Storage array with VMFS6 datastores
• Oracle RAC storage was provisioned from the All Flash FC Storage array with VMFS6 datastores
• 2 Node 19c Oracle RAC cluster on Oracle Linux Server release 7.6
• Database Shared storage on Oracle ASM using Oracle ASMLIB

2nd vSphere Cluster connected to a vSAN 6.7 P01 storage

• 4 node VMware vSAN vSAN 6.7 P01 was also created on similar ESXi nodes
• Each node has 28 CPU’s, 384 GB memory
• Each node has 2 vSAN disk groups
• Each disk group has 1x750GB for cache and 1×1.75 TB for capacity, all drives NVMe
• Create a new VM Storage Policy – ‘Oracle RAC vSAN – OSR 0’

vSAN Storage policy “Oracle RAC vSAN – OSR 0” is as below

• Failures to tolerate – 1 failure – RAID-1 (Mirroring)
• Object space reservation – Thin provisioning

 

Current Oracle RAC Cluster

• Both RAC VM has 8 vCPU’s, 96GB RAM
• RAC 19c instances with Grid Infrastructure and RDBMS binaries with multi-tenancy option (1 PDB)
• 1 ASM disk group DATA_DG with 1 ASM Disk of 1TB, EZT with MWF

RAC VM ‘rac19c1’

RAC VM ‘rac19c2’

Details of RAC VM ‘rac19c1’ non-shared vmdk and shared vmdk with multi-writer attribute is shown as below.

• Hard Disk 1 is Operating System
• Hard Disk 2 is for Grid Infrastructure and RDBMS binaries
• Hard Disk 3 is the shared storage for RAC Cluster. Observe the shared vmdk at SCSI 2:0 of size 1TB

Details of RAC VM ‘rac19c2’  non-shared vmdk and shared vmdk with multi-writer attribute is shown as below.

• Hard Disk 1 is Operating System
• Hard Disk 2 is for Grid Infrastructure and RDBMS binaries
• Hard Disk 3 is the shared storage for RAC Cluster. Observe ‘rac19c2’ points to ‘rac19c1’ shared vmdk at SCSI 2:0 of size 1TB

All Cluster Services & Resource are up.

[root@rac19c1 ~]# /u01/app/19.0.0/grid/bin/crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.LISTENER.lsnr
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
ora.chad
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
ora.net1.network
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
ora.ons
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac19c1 STABLE
2 ONLINE ONLINE rac19c2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA_DG.dg(ora.asmgroup)
1 ONLINE ONLINE rac19c1 STABLE
2 ONLINE ONLINE rac19c2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac19c2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac19c1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac19c1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac19c1 169.254.2.146 192.16
8.140.154,STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac19c1 Started,STABLE
2 ONLINE ONLINE rac19c2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac19c1 STABLE
2 ONLINE ONLINE rac19c2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rac19c1 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac19c1 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE rac19c1 STABLE
ora.rac19c.db
1 ONLINE ONLINE rac19c1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE rac19c2 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
ora.rac19c1.vip
1 ONLINE ONLINE rac19c1 STABLE
ora.rac19c2.vip
1 ONLINE ONLINE rac19c2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac19c2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac19c1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac19c1 STABLE
——————————————————————————–
[root@rac19c1 ~]#

Test Suite Setup

2 test cases were performed against the RAC cluster. The test cases are to Storage migrate a 2 node Oracle RAC from

• 1st vSphere Cluster connected to an All Flash FC Storage array with VMFS6 datastores TO a 2nd VMware vSAN 6.7 P01 Cluster
• 2nd VMware vSAN 6.7 P01 Cluster back to the 1st vSphere Cluster connected to an All Flash FC Storage array with VMFS6 datastores

Test Case 1 :  Storage migrate a 2 node Oracle RAC from All Flash FC Storage array with VMFS6 datastores to VMware vSAN vSAN 6.7 P01

One of the limitations of shared vmdk’s with multi-writer attribute is Storage vMotion is disallowed. This has been documented in both KB 1024165 and KB 2121181.

Owing to this limitation the Oracle RAC storage migration would have to be an offline one

0) The initial step would be to make sure that the FC VMFS6 datastores are mounted on the vSAN Cluster , so that VM’s on the vSAN cluster can see both VMFS6 and vSAN datastores

1) Since this scenario uses the Storage vMotion tool ,  the migration is an offline migration

2) Stop all RAC Cluster services on RAC VM’s ‘rac19c1’ and ‘rac19c2’

3) Power off RAC VM’s ‘rac19c1’ and ‘rac19c2’

4) On RAC VM ‘rac19c2’ , delete the shared 1TB vmdk . DO NOT check ‘delete files from datastore’ . Click Ok

Now RAC VM ‘rac19c2’ has 2 non-shared vmdk’s .

5) Storage vMotion RAC VM ‘rac19c2’ to vSAN 6.7 P01 using Storage Policy ‘vSAN Default Storage Policy’

6) RAC VM ‘rac19c2’ is now on to vSAN 6.7 P01 using Storage Policy ‘vSAN Default Storage Policy’

7) On RAC VM ‘rac19c1’ , remove the multi-writer attribute from the shared 1TB vmdk . DO NOT delete OR check ‘delete files from datastore’ . Click Ok

8) Storage vMotion RAC VM ‘rac19c1’ to vSAN 6.7 P01 using Storage Policy

• ‘vSAN Default Storage Policy’ for Hard Disk 1 (OS) and Hard Disk 2 (Oracle binaries). You can also choose to use vSAN Storage policy ‘Oracle RAC vSAN – OSR 0’.
• ‘Oracle RAC vSAN – OSR 0’ for Hard Disk 3 ie the shared vmdk

Click Finish

9) RAC VM ‘rac19c1’ is now on to vSAN 6.7 P01 using Storage Policy

• ‘vSAN Default Storage Policy’ for Hard Disk 1 (OS) and Hard Disk 2 (Oracle binaries)
• ‘Oracle RAC vSAN – OSR 0’ for Hard Disk 3 ie the shared vmdk

The Storage Policy ‘Oracle RAC vSAN – OSR 0’ shows that RAC VM ‘rac19c1’ is compliant as well.

RAC VM ‘rac19c2’ is now on to vSAN 6.7 P01 using Storage Policy

• ‘vSAN Default Storage Policy’ for Hard Disk 1 (OS) and Hard Disk 2 (Oracle binaries)

10) On RAC VM ‘rac19c1’, reapply the multi-writer attribute to the shared vmdk

11) On RAC VM ‘rac19c2’, Add the same shared vmdk on the same SCSI X:Y position as in VM ‘rac19c1’ with the multi-writer attribute using the ‘add existing hard drive’ option.

Ensure that the Storage Policy for the shared vmdk is set to ‘Oracle RAC vSAN – OSR 0’.

RAC VM ‘rac19c2’ is now on to vSAN 6.7 P01 using Storage Policy

• ‘vSAN Default Storage Policy’ for Hard Disk 1 (OS) and Hard Disk 2 (Oracle binaries)
• ‘Oracle RAC vSAN – OSR 0’ for Hard Disk 3 ie the shared vmdk

12) Power RAC VM’s ‘rac19c1’ and ‘rac19c2’, Check for all cluster services. All cluster services are up.

[root@rac19c1 ~]# /u01/app/19.0.0/grid/bin/crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.LISTENER.lsnr
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
ora.chad
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
ora.net1.network
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
ora.ons
ONLINE ONLINE rac19c1 STABLE
ONLINE ONLINE rac19c2 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac19c1 STABLE
2 ONLINE ONLINE rac19c2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA_DG.dg(ora.asmgroup)
1 ONLINE ONLINE rac19c1 STABLE
2 ONLINE ONLINE rac19c2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac19c1 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac19c2 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac19c2 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac19c2 169.254.26.87 192.16
8.140.155,STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac19c1 Started,STABLE
2 ONLINE ONLINE rac19c2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac19c1 STABLE
2 ONLINE ONLINE rac19c2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rac19c2 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac19c2 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE rac19c2 STABLE
ora.rac19c.db
1 ONLINE ONLINE rac19c1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE rac19c2 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
ora.rac19c1.vip
1 ONLINE ONLINE rac19c1 STABLE
ora.rac19c2.vip
1 ONLINE ONLINE rac19c2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac19c1 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac19c2 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac19c2 STABLE
——————————————————————————–
[root@rac19c1 ~]#

Summary

This blog focuses on validating a test case ie Storage migrate a 2 node Oracle RAC from All Flash FC Storage array with VMFS6 datastores to VMware vSAN vSAN 6.7 P01.

The reverse process ie Storage migrate a 2 node Oracle RAC from VMware vSAN vSAN 6.7 P01 back to an All Flash FC Storage array with VMFS6 datastores would be the same process as shown in the test case except the further reconfiguration of the shared vmdk’s is needed ie remediation of shared disks from LZT to EZT.

Migrating Oracle RAC storage from non-vSAN storage (FC / NAS / iSCSI storage) to

• VMware vSAN prior vSAN 6.7 P01
  • Further reconfiguration of the shared vmdk’s needed ie remediation of shared disks from LZT to EZT
  • Add multi-writer attribute to the shared vmdk’s
• VMware vSAN vSAN 6.7 P01
  • No further reconfiguration of the shared vmdk’s needed
  • Only step is to Add multi-writer attribute to the shared vmdk’s

All Oracle on vSphere white papers including Oracle on VMware vSphere / VMware vSAN / VMware Cloud on AWS , Best practices, Deployment guides, Workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop

http://www.vmw.re/oracle-one-stop-shop