Around the “Storage World” in no time – Storage vMotion for Oracle Workloads (RAC & Non-RAC) within same vSphere Cluster

“Around the World in Eighty Days” – A classic adventure novel and one of Jules Verne’s most acclaimed works which describes how Phileas Fogg and his valet Passepartout attempt to circumnavigate the world in 80 days on a wager.

In the world of Business Critical Applications, especially IO intensive Oracle workloads, there is always a need for storage migration, based on the ever demanding workload profile.

For example –

• Migrate Oracle database storage from one Tier to another Tier within a storage array , accessed by the same vSphere Cluster
• Migrate Oracle database storage from one array to another array (within and between data centers) for the same datastore type [ VMFS . NFS, iSCSI , vVOL , vSAN] ,accessed by the same vSphere Cluster
• Migrate Oracle database storage from one array to another array (within and between data centers) across different datastore types [ VMFS . NFS, iSCSI , vVOL , vSAN] , accessed by the same vSphere Cluster

The storage migration is made seamless across the various permutations and combinations, simply by using vmdk’s , the common  denominator of VM storage across all above flavors of VMware datastores.

This blog  focused on how we can storage vMotion an Oracle Standalone and Oracle RAC Cluster , from one datastore to another datastore within a storage array OR between 2 storage arrays, accessed by the same vSphere Cluster , without ANY downtime.

Key Takeaways

• Storage Migration of an Oracle Single Instance is relatively easy using VMware storage vMotion
• Storage Migration of an Oracle RAC Cluster from one datastore to another datastore , within an storage array OR between 2 storage arrays, on a VMware SDDC , is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database and VMware storage vMotion alone cannot be used for reasons explained later in the blog.
• Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk BUT will NOT migrate the Operating System or Oracle binaries vmdk.
• VMware storage vMotion will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but will not storage vMotion clustered vmdk’s, it can ONLY be used for non-shared vmdk’s [KB 1034165]
• This blog  focused on how we can storage vMotion an Oracle Standalone and Oracle RAC Cluster , without ANY downtime.
• We deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above
  • Using Oracle ASM technology to migrate cluster database data – RAC database can been migrated from one datastore to new datastore on an existing array or between 2 arrays
  • Using Storage vMotion technology to migrate OS and Oracle non-clustered vmdk’s of RAC VM’s – The OS and Oracle binaries vmdk’s for the RAC VM’s are now migrated from one datastore to new datastore on an existing array or between 2 arrays
    

Before describing this technique, we first review some key ideas, including the deployment model of Oracle on vSphere and the fundamentals of Storage vMotion.

Oracle layout on vSphere – Single Instance and RAC

There are 3 main Oracle database deployment models, all of which are supported and can be deployed on the VMware SDDC stack.

The primary difference between Oracle Single instance and Oracle RAC vmdk/s setup is the vmdk sharing mode.

There are 3 values of disk sharing mode

• Unspecified
• No sharing
• Multi-writer – used for Clustering

Both Single Instance VM and RAC VM/s will have 2 genres of vmdk’s :

• non-database vmdk’s
  • Operating System disks which houses the root file system (/)
  • Oracle binaries vmdk which houses the Oracle binaries (Grid and RDBMS binaries) (/u01)

• database vmdk’s
  • Database vmdk/s
    • non-shared in case of Single Instance VM
    • shared in case of RAC VM/s

Example of Disk layout of Oracle Single Instance VM

‘SB_oracle-RHEL’ VM houses a single instance Oracle database 12.2 running on Red Hat 7.4 operating system. The vmdk’s for the VM is on ‘TSA_TNTR_Cap_10TB_01’ datastore on a NAS array.

The VM ‘SB_oracle-RHEL’  has 3 vmdk’s-

• Hard Disk 1 of 50 GB capacity which houses the OS (/)
• Hard Disk 2 of 50 GB capacity which houses the Oracle binaries (/u01)
• Hard Disk 1 of 50 GB capacity which houses the Oracle database (ASM disk) – No sharing mode

 

Example of Disk layout of Oracle RAC VM’s

 Below is a setup of a 4 node RAC Cluster. VM’s ‘rac1221’, ‘rac1222’, ‘rac1223’ and ‘rac1224’ form the 4 Node 12.2 Oracle RAC cluster (Flex Cluster and Flex ASM) on Oracle Enterprise Linux 7.4.

All 4 RAC VM’s are in the same datastore ‘TSA_TNTR_Cap_10TB_01’.

Looking at VM ‘rac1221’ disk layout, we can see, ‘rac1221’ has 11 hard disks

• Hard Disk 1 of 60 GB capacity which is the OS disk (/)
• Hard Disk 2 of 60 GB capacity which is the Oracle binaries (/u01) disk
• Hard Disk 3 – Hard Disk 11 which are the shared Oracle RAC vmdk’s (ASM disks)

Observations

• Sharing attribute of vmdk’s Hard Disk [ 3 – 11] are set to ‘Multi-writer’
  • shared vmdk’s are created on 1 RAC VM and all other RAC VM’s simply refer/point to it
• Disks with multi-writer flag attribute have to be Eager Zero thick
• They need not be independent persistent KB 1034165

Let’s look at VM ‘rac1222’ disk layout. We can see ‘rac1222’ has 11 hard disks

• Hard Disk 1 of 60 GB capacity which is the OS disk
• Hard Disk 2 of 60 GB capacity which is the Oracle binaries (/u01) disk
• Hard Disk 3 – Hard Disk 11 which are the shared Oracle RAC vmdk’s (ASM disks)

Observations

• Sharing attribute of Hard Disk [ 3 – 11] for are also set to ‘Multi-writer’
  • shared vmdk’s are created on 1 RAC VM and all other RAC VM’s simply refer/point to it
• Hard Disks [ 3 – 11] of ‘rac1222’ refer to / point to ‘rac1221’ vmdk’s

RAC VM’s ‘rac1223’ and ‘rac1224’ disk layout matches ‘rac1222’.

Process of allocating disks to Oracle RAC VM’s

The process for adding new vmdk/s to a RAC cluster using the Multi-writer flag can be found in the article KB 1034165 .

High level steps for adding new vmdk’s to a RAC cluster:

• new vmdk/s are added to one RAC VM with ‘New Hard Disk’ option to a SCSI x:y controller slot

• then the new vmdk/s are then added to the rest of the RAC VM’s using the ‘Existing Hard Disk’ option, they need to be added to the same SCSI x:y controller slot as allocated above

After allocating all disk, the vmdk layout for the RAC VM’s would look as below:

Oracle RAC VM’s .vmx file

Lets look at the .vmx files of RAC VM’s ‘rac1221’ , ‘rac1222’ , ‘rac1223’ and ‘rac1224’ VM’s and see how the shared vmdk’s are attached , for example, to SCSI 1:0 slot.

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1221] cat rac1221.vmx
….
scsi1:0.deviceType = “scsi-hardDisk”
scsi1:0.fileName = “rac1221_2.vmdk” <- rac1221 vmdk at 1:0
scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
….
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1221]

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1222] cat rac1222.vmx
….
scsi1:0.deviceType = “scsi-hardDisk”
scsi1:0.fileName = “/vmfs/volumes/12cdf977-2b1b0ec4/rac1221/rac1221_2.vmdk” <- <- points to rac1221 vmdk at 1:0

scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
…
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1222]

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1223] cat rac1223.vmx
……
scsi1:0.deviceType = “scsi-hardDisk”

scsi1:0.fileName = “/vmfs/volumes/12cdf977-2b1b0ec4/rac1221/rac1221_2.vmdk” <- points to rac1221 vmdk at 1:0

scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
……
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1223]

[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1224] cat rac1224.vmx
……
scsi1:0.deviceType = “scsi-hardDisk”

scsi1:0.fileName = “/vmfs/volumes/12cdf977-2b1b0ec4/rac1221/rac1221_2.vmdk” <- points to rac1221 vmdk at 1:0

scsi1:0.mode = “independent-persistent”
scsi1:0.sharing = “multi-writer”
sched.scsi1:0.shares = “normal”
sched.scsi1:0.throughputCap = “off”
scsi1:0.present = “TRUE”
….
[root@wdc-esx28:/vmfs/volumes/12cdf977-2b1b0ec4/rac1224]

VMware Storage vMotion

 With VMware Storage vMotion technology, we can migrate VM’s to relocate the configuration file of a virtual machine and virtual disks, while the virtual machine is powered on, from one datastore to another.

The vSphere documentation details Storage vMotion Requirements and Limitations

Caveats with Multi-writer disks and Storage vMotion

The multi-writer capability for sharing vmdk’s between VM’s play an important role in Oracle RAC deployments on VMware SDDC.

VMFS is a clustered file system that disables (by default) multiple VM’s from opening and writing to the same virtual disk (.vmdk file). This prevents more than one VM from inadvertently accessing the same .vmdk file..

The multi-writer option allows VMFS-backed disks to be shared by multiple VM’s. This option can be used to disable the protection for certain cluster-aware applications e.g Oracle Clusterware , where the applications ensure that writes originating from two or more different VM’s does not cause data loss and ensures data consistency and concurrency.

Among the unsupported actions for shared vmdk’s using multi-writer capability includes

• Storage vMotion
• Snapshots
• Changed Block Tracking (CBT)

Attempting a storage vMotion of an RAC VM will result in an error:

The supported and unsupported actions / features using multi-writer attribute can be found in the KB 1034165.

Oracle Storage migration methods

There are a couple of ways we can migrate Oracle workload storage either from one datastore to another, within in an existing array or between 2 separate arrays.

This blog  focused on how we can storage vMotion an Oracle Standalone and Oracle RAC Cluster , from one datastore to another datastore within a storage array OR between 2 storage arrays, accessed by the same vSphere Cluster , without ANY downtime.

1. Use Oracle Automatic Storage Management (ASM) technology to migrate oracle blocks from one vmdk on a datastore to a new vmdk on same / different datastore

The process is to add new disks to the existing ASM disk group and remove old disks from the same ASM disk group , while the database continues to access files from the disk group.  When you add or remove disks from a disk group, Oracle ASM automatically redistributes the file contents and eliminates the need for downtime when redistributing the content. When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group.

More information can be found at Dropping Disks from Disk Groups.

The power setting parameter ASM_POWER_LIMIT determines the speed with which rebalancing operations occur.  The range of values is 0 to 1024. The default value is 1. A value of 0 disables rebalancing. Higher numeric values enable the rebalancing operation to complete more quickly, but might result in higher I/O overhead and more rebalancing processes.  With this parameter, the DBA has the power to control the migration throughput hence avoiding causing potential performance during peak production hours

This migration method uses Oracle ASM technology to migrate oracle blocks between the ASM disks and

• • will ONLY migrate Oracle database ASM disk
  • will NOT migrate the Operating System vmdk or Oracle binaries vmdk

2. Use VMware storage vMotion method of moving vmdk’s from one datastore to new datastore

 This migration method

• • will migrates ALL vmdk’s including Oracle database ASM disk, Operating System vmdk and Oracle binaries vmdk
  • will not migrate clustered vmdk’s, it can be used ONLY for non-shared vmdk’s [KB 1034165]

Any storage based migration, be it storage vMotion or array based migration, is faster than Oracle ASM method of add, drop and rebalancing disks.  However, the speed of migration is not regulated or controlled. This may cause potential performance during peak production hours.

For migration from existing to a new array, datastores from both arrays must be mounted on the vSphere cluster

There are also other methods for migrating storage which will require time to setup and cut over as well and are out of scope for this blog

• Standing up a target Oracle RAC as Physical standby for the source Oracle RAC cluster
• Using Data Pump / RMAN / Golden Gate / Third party Replication products to move data between source and new created target RAC database

Storage Migration of an Oracle RAC Cluster

Question , How do we now migrate an RAC cluster from one datastore to another , accessed to the same vSphere Cluster , without any downtime?

Remember , Oracle RAC VM’s has shared vmdk/s for RAC database. As stated above

• Oracle Automatic Storage Management (ASM) technology
  • will ONLY migrate Oracle database ASM disk
  • will NOT migrate the Operating System vmdk or Oracle binaries vmdk

• VMware storage vMotion
  • will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk
  • can be used ONLY for non-shared vmdk’s [KB 1034165]

Do we have a solution?

Solution: A two-step process which combines both Oracle ASM and storage vMotion technology for a RAC Cluster.

Storage Migration of an Oracle Single Instance database

 This will involve a 1-step process

• Step is using Storage vMotion technology to migrate OS and Oracle vmdk of Single Instance VM’s from one datastore to another datastore
  

Using Oracle ASM technology

The steps for using Oracle ASM technology for storage migrating Oracle database storage from one datastore to new datastore , either in an existing array or between 2 arrays , the array(s) accessed to the same vSphere Cluster , are

• add new vmdk/s from new datastore (either existing or new array) to VM
• partition the new device with appropriate partitioning offset
  • needed if ASMLIB is used, not needed for Linux udev
• create ASM device on new disk/s
• add new ASM disks to existing ASM disk group
• drop the old ASM disk from existing ASM disk group
  • When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the dropped disk to other disks in the disk group
• delete the old vmdk/s from the VM

At the end of this step

• database vmdk’s are on the new datastore
• operating system and oracle binaries vmdk are still on the old datastore

As stated earlier, this migration method uses Oracle ASM technology to migrate oracle blocks between the ASM disks and

1. will ONLY migrate Oracle database ASM disk
2. will NOT migrate the Operating System or Oracle binaries vmdk

Using Storage vMotion technology

Process of storage vMotion of an Oracle single instance deployment on VMware SDDC is the same as storage vMotioning any other workload on VMware SDDC.

Lets storage vMotion the VM ‘SB_oracle-RHEL’ from ‘TSA_TNTR_Cap_10TB_01’ datastore ‘TSA_TNTR_Cap_10TB_02’ on the same array.

Right click on VM ‘SB_oracle-RHEL’ on web client and choose the storage migration option

Choose the destination datastore

We have a choice to pick the datastore we want to place the vmdk’s, in this case we picked the same datastore for all the VM vmdk’s.

Confirm operation

Storage vMotion operation starts and completes successfully

We have successfully storage vMotioned the VM ‘SB_oracle-RHEL’ from ‘TSA_TNTR_Cap_10TB_01’ datastore ‘TSA_TNTR_Cap_10TB_02’ on the same array.

At the end of this step

• database vmdk’s are on the new datastore
• operating system and oracle binaries vmdk are on the new datastore

Storage Migration of an Oracle RAC Cluster

 This will involve a 2-step process

• 1st Step is using Oracle ASM technology to migrate cluster database data
• 2nd Step is using Storage vMotion technology to migrate OS and Oracle vmdk of RAC VM’s

Using Oracle ASM technology to migrate cluster database data

 The steps for using Oracle ASM technology for storage migrating Oracle RAC database storage from one datastore to new datastore on either an existing array or between 2 arrays are

• pick a RAC VM, for example ‘rac1221’ , as the VM for disk operation
• add new vmdk/s from new datastore (either existing or new array) to ‘rac1221’ using ‘New Hard disk’ option
• add these vmdk/s to the other RAC VM’s [ ‘rac1222’, ‘rac1223’, ‘rac1224’ ] using ‘Existing Hard disk’ option
• partition the new vmdk/s on the Guest with appropriate partitioning offset
  • perform this operation on RAC VM ‘rac1221’ only
  • needed if ASMLIB is used, not needed for Linux udev
• scan the SCSI bus on the other RAC VM’s [‘rac1222’, ‘rac1223’, ‘rac1224’] to see the partitions
• create ASM device on new disk/s
  • perform operation on RAC VM ‘rac11221’
  • run ‘oracleasm scandisks’ command on the other RAC VM’s [ ‘rac1222’, ‘rac1223’, ‘rac1224’ ] to see the new ASM disks
• add new ASM disks to existing ASM disk group
  • perform operation on RAC VM ‘rac11221’
• drop the old ASM disk from existing ASM disk group
  • perform operation on RAC VM ‘rac11221’
• delete the old vmdk/s from all the other RAC VMs [‘rac1222’, ‘rac1223’, ‘rac1224’ ]
  • do not check “Delete files from datastore” option
• delete the old vmdk/s from the RAC VM ‘rac1221’
  • check “Delete files from datastore” option before confirming delete

At the end of this step

• RAC database has been migrated from one set of vmdk’s to a new set of vmdk’s [from one datastore to new datastore on either an existing array or between 2 arrays]
  
• old database vmdk’s have been deleted from the RAC VM’s
• The OS and Oracle binaries vmdk’s for the RAC VM’s [ ‘rac1221’, ‘rac1222’, ‘rac1223’, ‘rac1224’ ] are still on the old datastore
  • they need to be moved to the new datastore as well

Using Storage vMotion technology to migrate OS and Oracle vmdk of RAC VM’s

The steps for migrating the non-clustered OS and Oracle vmdk’s for RAC VM’s from one datastore to new datastore on either an existing array or between 2 arrays are

• relocate / drain out Oracle user connections from RAC instance ‘rac1224’ to the remaining 3 RAC instances [ ‘rac1221’, ‘rac1222’, ‘rac1223’ ]
• shutdown RAC instance ‘rac1224’
• power down RAC VM ‘rac1224’
• make a copy of RAC VM ‘rac1224’ .vmx file just in case
• remove all shared vmdk’s from RAC VM ‘rac1224’
  • do not check “Delete files from datastore” option
• storage vMotion RAC VM ‘rac1224’ OS and Oracle vmdk’s to the new datastore
• add the shared vmdk/s back to RAC VM ‘rac1224’
  • keep in mind the shared vmdk’s have to be added in the same SCSI x:y controller slot positions as before
• power on RAC VM ‘rac1224’ and start cluster services on RAC VM ‘rac1224’
• repeat above steps for other RAC VMs ‘rac1221’ , ‘rac1222’ and ‘rac1223’

At the end of this step

• OS and Oracle binaries vmdk’s for the RAC VM’s [ ‘rac1221’, ‘rac1222’, ‘rac1223’, ‘rac1224’ ] are now on the new datastore

So at the end of both the steps

• Using Oracle ASM technology to migrate cluster database data – RAC database has been migrated from one datastore to new datastore on either an existing array or between 2 arrays
• Using Storage vMotion technology to migrate OS and Oracle vmdk of RAC VM’s – The OS and Oracle binaries vmdk’s for the RAC VM’s [ ‘rac1221’, ‘rac1222’, ‘rac1223’, ‘rac1224’ ] are now on new storage array

Summary

• Storage Migration of an Oracle Single Instance is relatively easy using VMware storage vMotion
• Storage Migration of an Oracle RAC Cluster from one datastore to another datastore , within an storage array OR between 2 storage arrays, on a VMware SDDC , is challenging as Oracle RAC VM’s uses shared vmdk/s for RAC database and VMware storage vMotion alone cannot be used for reasons explained later in the blog.
• Oracle Automatic Storage Management (ASM) technology will ONLY migrate Oracle database ASM disk and will NOT migrate the Operating System or Oracle binaries vmdk.
• VMware storage vMotion will migrates ALL vmdk’s including Oracle database ASM disk, Operating System and Oracle binaries vmdk but can ONLY be used for non-shared vmdk’s [KB 1034165
• This blog  focused on how we can storage vMotion an Oracle Standalone and Oracle RAC Cluster , without ANY downtime. We can deploy a two-step process which combines both Oracle ASM and storage vMotion technology as detailed above
  • Using Oracle ASM technology to migrate cluster database data – RAC database has been migrated from one datastore to new datastore on either an existing array or between 2 arrays
  • Using Storage vMotion technology to migrate OS and Oracle vmdk of RAC VM’s – The OS and Oracle binaries vmdk’s for the RAC VM’s [ ‘rac1221’, ‘rac1222’, ‘rac1223’, ‘rac1224’ ] are now on new storage array

Conclusion

Business Critical Applications especially IO intensive Oracle workloads have a need for storage migration, based on the ever-demanding workload profile, from one Tier to another Tier within a storage array or between storage arrays.

By combining both technologies, Oracle ASM technology and VMware storage vMotion, we are able to successfully migrate storage of an n-node Oracle RAC on VMware SDDC without any downtime or breech of SLA which is the fastest way to migrate RAC cluster on VMware SDDC.

All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found in the url below

Oracle on VMware Collateral – One Stop Shop