Much has been discussed and documented about Storage vMotion of Oracle RAC Clusters on VMware SDDC which can be found in the numerous blog posts available at Oracle on VMware Collateral – One Stop Shop.
Development, QA , Pre-Production RAC clusters form an important part of any RAC deployment environment and while they may not be as large as the production RAC clusters, they are deemed equally critical for the software life cycle process . Fortunately they are not constrained by the 100% uptime business SLA’s which means admins can choose to go another route of storage vMotioning Oracle RAC with minimal downtime in a very short time.
This blog focuses on storage vMotion of a 2 Node non-production Oracle RAC Cluster with storage on one datastore to a datastore on a different storage system ,within same vSphere Cluster, with minimal downtime and time.
Let’s us first review some key ideas, including the deployment model of Oracle on vSphere and the fundamentals of Storage vMotion.
Contents
Oracle layout on vSphere – Single Instance and RAC
There are 3 main Oracle database deployment models, all of which are supported and can be deployed on the VMware SDDC stack.
The primary difference between Oracle Single instance and Oracle RAC vmdk/s setup is the vmdk sharing mode.
There are 3 values of disk sharing mode
- Unspecified
- No sharing
- Multi-writer – used for Clustering
Both Single Instance VM and RAC VM/s will have 2 genres of vmdk/s:
- non-database vmdk/s
- Operating System vmdk which houses the root file system (/)
- Oracle binaries vmdk which houses the Oracle binaries (Grid and RDBMS binaries) (/u01)
- database vmdk/s
- non-shared vmdk/s in case of Single Instance
- shared vmdk/s in case of RAC Cluster
- non-shared vmdk/s in case of Single Instance
VMware Storage vMotion
With VMware Storage vMotion technology, we can migrate VM’s to relocate the configuration file of a virtual machine and virtual disks, while the virtual machine is powered on, from one datastore to another.
The vSphere documentation details Storage vMotion Requirements and Limitations
Caveats with Multi-writer disks and Storage vMotion
The multi-writer capability for sharing vmdk/s between VM’s play an important role in Oracle RAC deployments on VMware SDDC.
VMFS is a clustered file system that disables (by default) multiple VM’s from opening and writing to the same virtual disk (.vmdk file). This prevents more than one VM from inadvertently accessing the same .vmdk file..
The multi-writer option allows VMFS-backed disks to be shared by multiple VM’s. This option can be used to disable the protection for certain cluster-aware applications e.g Oracle Clusterware , where the applications ensure that writes originating from two or more different VM’s does not cause data loss and ensures data consistency and concurrency.
Among the unsupported actions for shared vmdk/s using multi-writer capability includes
- Storage vMotion
- Snapshots
- Changed Block Tracking (CBT)
Attempting a storage vMotion of an RAC VM will result in an error:
The supported and unsupported actions / features using multi-writer attribute can be found in the KB 1034165.
More information is available in the blog post Around the “Storage World” in no time – Storage vMotion and Oracle Workloads .
Storage vMotion of an Oracle RAC Cluster without ANY downtime
The blog details how we can storage vMotion an Oracle RAC Cluster from one storage to another , within same vSphere Cluster , without ANY downtime.
The takeaways of the above solution are
- Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk and will NOT migrate the Operating System or Oracle binaries vmdk.
- VMware storage vMotion will migrates ALL vmdk/s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but can ONLY be used for non-shared vmdk/s [KB 1034165]
- We can deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above
Storage vMotion of an Oracle RAC Cluster with minimal downtime
For Development, QA , Pre-Production RAC clusters which are not constrained by the 100% uptime business SLA’s, admins can choose to go another route of storage vMotioning Oracle RAC with minimal downtime in a very short time.
The high level steps for this approach is as below.
Assumption, we have a 2 node Oracle RAC Cluster VM1 and VM2
- Shutdown all the RAC VM’s (VM1 and VM2)
- Make a copy the .vmx file for all RAC VM’s
- append a subscript ‘ORIG’ at the end of all VM vmx files e.g cp VM2.vmx VM2.vmx.ORIG
- Run the command on any RAC VM and save the output to a file
- cat VM1.vmx | grep -I multi-writer > multi-writer.out
- the contents of the ‘multi-writer.out’ file will have entries containing the multi-writer setting e.g scsiX:Y.sharing = “multi-writer”
- Save both files .vmx.ORIG and multi-writer.out to a backup location
- On VM2, remove ALL the shared vmdk/s using one of the 2 methods below
- Web Client [ DO NOT DELETE THE VMDK/s from storage]
- Edit the. vmx file and remove ALL lines referring to and associated with the shared vmdk/s and save .vmx file
- Repeat Step 5 for all RAC VM’s except VM1
- Perform storage vMotion of all VM’s to the new storage array except VM1
- All VM’s with non-shared vmdk/s except VM1 are now on the new storage array
- Now VM1 has 2 genres of vmdk
- non-shared vmdk/s ( i.e. OS , Oracle binaries etc )
- shared vmdk/s with multi-writer settings
- On VM1, remove the multi-writer setting for all shared disks using one of the 2 methods below
- Web Client
- Edit the. vmx file and remove all lines which contain the words ‘multi-writer’ e.g scsiX:Y.sharing = “multi-writer”
- Perform storage vMotion of VM1 to the new storage array
- At this point VM1 vmdk/s is on the new storage
- original non-shared vmdks
- common vmdk/s without multi-writer setting
- On VM1, add multi-writer settings for all common vmdk/s back using one of the 2 methods
- Web Client
- Modify the VM .vmx file and add contents of the ‘multi-writer.out’ file to the end of VM1 .vmx file and save .vmx file
- VM1 now has
- non-shared vmdk/s
- shared vmdk/s with multi-writer setting
- On the rest of the VM’s , add ALL shared vmdk/s with the multi-writer settings using the ‘Existing Hard Disk’ option using one of the 2 methods
- Web Client
- Modify the .vmx.ORIG file of VM to change the storage location to point to the new vsanDatastore and over write the contents of the current .vmx file with the .vmx.ORIG contents
- Do Step 10 for all RAC VM’s except VM1
- At this point all VM/s vmdk/s are on the new storage
- original non-shared vmdks
- common vmdk/s with multi-writer setting
- Power on RAC VM
- Click OK for option ‘I copied it’ when prompted
- RAC Cluster has been successfully migrated over to the new cluster
Note : Starting from vSphere 6.0, editing the .vmx file directly and adding scsiX:Y.sharing = “multi-writer” is discouraged. VMware highly recommends using the Web Client to make the changes.
In vSphere 6.0, there is now a new sharing attribute on the Virtual Disk backing property which accepts one of two values: sharingMultiWriter or sharingNone for specifying the Multiwriter flag.
More can be read here.
Migration steps
We had a fully functional 2 Node Oracle RAC 12.2.0.1.0 Cluster running on OEL 7.4 with vmdk/s on NFS VAAI compliant datastore on VMware vSphere 6.5 . We would like to migrate this Oracle RAC cluster to a different storage to run some tests against the new storage.
The RAC VM’s are ‘pvrdmarac1’ and ‘pvrdmarac2’.
1. First , backup both RAC VM’s ‘pvrdmarac1’ and ‘pvrdmarac2’ .vmx files to a backup location.
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1] cp pvrdmarac1.vmx pvrdmarac1.vmx.ORIG
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac2] cp pvrdmarac2.vmx pvrdmarac2.vmx.ORIG
2. Extract and save the multi-writer lines, from any VM .vmx file to another file , say ‘multi-writer.out’
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1] grep -i multi pvrdmarac1.vmx
scsi3:0.sharing = “multi-writer”
scsi3:1.sharing = “multi-writer”
scsi3:2.sharing = “multi-writer”
scsi3:3.sharing = “multi-writer”
scsi3:4.sharing = “multi-writer”
scsi3:5.sharing = “multi-writer”
scsi3:6.sharing = “multi-writer”
scsi3:8.sharing = “multi-writer”
scsi1:2.sharing = “multi-writer”
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1]
[root@wdc-esx27:/vmfs/volumes/146ec950-0b52c16b/pvrdmarac1] grep -i multi pvrdmarac1.vmx > multi-writer.out
3. Shutdown Oracle RAC Cluster on all RAC VM’s , ‘pvrdmarac1’ and ‘pvrdmarac2’
[root@pvrdmarac1 ~]# /u01/app/12.2.0/grid/bin/crsctl stop cluster -all
4. Power ALL RAC VM’s , ‘pvrdmarac1’ and ‘pvrdmarac2’ , down
5. Edit VM ‘pvrdmarac2’ settings and remove ALL shared devices from VM ‘pvrdmarac2’. Do not delete ANY of the vmdk/s from storage.
6. Now storage vMotion VM ‘pvrdmarac2’ to the new storage. Remember, pvrdmarac2.vmx.ORIG file will NOT get copied over , so move it manually somewhere safe.
7. Remove multi-writer settings from all the shared disks on VM ‘pvrdmarac1’. This can be done in one of the two ways below.
a. Using the Web Client , Click ‘Edit Settings’ for VM ‘pvrdmarac1’ , navigate to the shared vmdk/s and choose ‘No sharing’ option from the drop-down box
Do the same step for ALL shared vmdk/s.
b. ssh to the ESXI server and navigate to the VM ‘pvrdmarac1’ folder. Using the vi editor , edit pvrdmarac1.vmx file and remove all lines containing the word ‘multi-writer ‘ from vmx file and save the .vmx file.
8. Storage vMotion VM ‘pvrdmarac1’ to new storage
9. Once the storage vMotion of VM ‘pvrdmarac1’ completes successfully , restore multi-writer setting for all common shared vmdk/s to VM ‘pvrdmarac1’. This can be done in one of the two ways below.
a. Using the Web Client , Click ‘Edit Settings’ for VM ‘pvrdmarac1’ , navigate to the shared vmdk/s and choose ‘Multi-writer’ option from the drop-down box
b. ssh to the ESXI server and navigate to the VM ‘pvrdmarac1’ folder. Append the contents of the saved ‘multi-writer.out’ file to the end of ‘pvrdmarac1.vmx’ file
[root@wdc-esx27:/vmfs/volumes/XXXXXXXX/pvrdmarac1] cat multi-writer.out >> pvrdmarac1.vmx
10 . Now add back the shared disks with the multi-writer settings to VM ‘pvrdmarac2’.
a. Using the Web Client , Click ‘Edit Settings’ for VM ‘pvrdmarac2’ , choose ‘Existing Hard Disk’, navigate to the VM ‘pvrdmarac1’ folder where the vmdk is located and add it to the same SCSI X:Y slot as is present in VM ‘pvrdmarac1’.
b. ssh to the ESXI server and navigate to the VM ‘pvrdmarac2’ folder and copy the contents of VM ‘pvrdmarac2.vmx.ORIG’ file to ‘pvrdmarac2.vmx’ file. Next , edit ‘pvrdmarac2.vmx’ file and change storage location from original NFS datastore location to the new datastore location. Then , append the contents of the saved ‘multi-writer.out’ file to the end of ‘pvrdmarac2.vmx’ file as done earlier for ‘pvrdmarac1.vmx’ file
11. Startup ALL the RAC VM’s , ‘pvrdmarac1’ and ‘pvrdmarac2’ , Click OK for option ‘I copied it’ when prompted. Bring up all RAC serices as needed.
Oracle RAC with 2 VM’s ‘pvrdmarac1’ and ‘pvrdmarac2’ has now been successfully moved to the new datastore location.
Summary
Storage Migration of an Oracle RAC Cluster on VMware SDDC is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database
Conclusion
Business Critical Applications especially IO intensive Oracle workloads have a need for storage migration, based on the ever-demanding workload profile, from one Tier to another Tier within a storage array or between storage arrays.
VMware storage vMotion technology can be used to migrate non-production Oracle RAC with minimal downtime in a very short time.
All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found at Oracle on VMware Collateral – One Stop Shop
