“I wanna “RAC” and “RAC” all night and party every day” – mantra of an Oracle RAC DBA
Contents
Much has been written , spoken and probably beaten to senseless about the magical “Multi-writer” setting and how it helps multiple VM’s share vmdk’s simultaneously for Clustering and FT used cases.
I still get question from customers interested in running Oracle RAC on vSphere about if we have the ability to add shared vmdk’s to a RAC cluster online without any downtime ? Yes we do.
Are the steps of adding shared vmdk’s to an extended RAC cluster online without any downtime the same? Yes.
Introduction
By default, the simultaneous multi-writer “protection” is enabled for all. vmdk files ie all VM’s have exclusive access to their vmdk files. So in order for all of the VM’s to access the shared vmdk’s simultaneously, the multi-writer protection needs to be disabled.
The below table describes the various Virtual Machine Disk Modes:
As we all are aware of , Oracle RAC requires shared disks to be accessed by all nodes of the RAC cluster.
KB Article 1034165 provides more details on how to set the multi-writer option to allow VM’s to share vmdk’s. Requirement for shared disks with the multi-writer flag setting for a RAC environment is that the shared disk is
- has to set to Eager Zero Thick provisioned
- need not be set to Independent persistent
While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
Supported and Unsupported Actions or Features with Multi-Writer Flag:
**** Important ***
• SCSI bus sharing is left at default and not touched at all in case of using shared vmdk’s
• It’s only used for RAC’s with RDM (Raw Device Mappings) as shared disks
Facts about vmdk and multi-writer
Before version 6.0, we had the ability to add vmdk’s with multi-writer option to an Oracle RAC online , the only caveat was that this ability was not exposed in the vSphere Web/C# Client .We had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online.
Setting Multi Writer Flag for Oracle RAC on vSphere using Power Cli
http://www.virtuallyghetto.com/2015/10/new-method-of-enabling-multiwriter-vmdk-flag-in-vsphere-6-0-update-1.html
With vSphere 6.0 and onwards, we can add shared disks to an Oracle RAC Cluster online using the Web Client.
Key points to take away from this blog
• VMware recommends using shared VMDK (s) with Multi-writer setting for provisioning shared storage for ALL Oracle RAC environments (KB 1034165)
• vSphere 6.0 and onwards, we can add shared vmdk’s to an Oracle RAC Cluster online using the Web Client
• Prior to version 6.0, we had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online
Example of an Oracle RAC Setup
As per best practices, the 2 VM’s, ‘rac01-g6’ and ‘rac02-g6’ , part of the 2 node Oracle RAC setup was deployed from a template ‘Template-12crac’.
The template has 10 vCPUs with 64 GB RAM with OEL7.3 as the operating system.
The template has 2 vmdk’s, 50GB each on SCSI 0 controller (Paravirtual SCSI Controller type)
• Hard disk 1 is on SCSI0:0 and is for root volume (/)
• Hard disk 2 is on SCSI0:1 and is for oracle binaries (/u01 for Grid and RDBMS binaries)
Hard Disk 1 (OS drive) & Hard Disk 2 (Oracle /u01) vmdk’s are set to
• Thin Provisioning
• No Sharing i.e. exclusive to the VM
• Disk mode is set to ‘Dependent’
Template has 2 network adapters of type VMXNET3.
• Public adapter
• Private Interconnect
Public Adapter:
Private Interconnect:
Lets add a shared vmdk of size say 50GB to both the VM’s online without powering down the VM’s.
1. Adding shared disks can be done online without downtime.
2. Add a PVSCSI Controller (SCSI 1) to RAC VM ‘rac01-g6’. Right Click on ‘rac01-g6’ , ‘Edit Settings’ and add New Controller of Type ‘Paravirtual’
Leave the SCSI Bus Sharing to ‘None’ (default)
3. Next step is to add a 50 GB shared vmdk to VM ‘rac01-g6’ to SCSI1:0 bus slot ( you can add the new vmdk it to any slot on SCSI 1 you want to)
Right Click on VM ‘rac01-g6’ and Choose ‘Edit Settings’. Choose ‘New Hard Disk’ , set Sharing to ‘Multi-writer’ , leave Disk mode to ‘Dependent’ and click ‘Add’. Click ‘OK’ and monitor progress.
4. Repeat Step 2 to add new ‘Paravirtual’ Controller SCSI 1 to RAC VM ‘rac02-g6’
5. . The new vmdk (vmdk with multi-writer option) created on VM ‘rac01-g6’ at SCSI1:0 bus slot needs to be shared with ‘rac02-g6’ VM for clustering purpose
6. Right Click on VM ‘rac02-g6’, Choose ‘Edit Settings’. Choose ‘Existing Hard Disk’ and Click ‘Add’.
7. Navigate to your Datastore [Group06], expand the Datastore contents and click on ‘rac01-g6’ folder. Click on the shared vmdk ‘rac01-g6_2.vmdk’ which was created on ‘rac01-g6’. Click ‘OK’
8. Note that the Sharing attribute for this vmdk needs to be set to ‘Multi-Writer’ and the SCSI controller set to the same SCSI controller as we did for ‘rac01-g6’ i.e SCSI1:0. Click ‘OK’ when done.
9. Scan the bus on the OS of both VM’s to see the new disk added and list the devices
[root@rac01-g6 ~]# fdisk -lu
Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00098df2
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 104857599 51379200 8e Linux LVM
….
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@rac01-g6 ~]#
[root@rac02-g6 ~]# fdisk -lu
Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00098df2
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 104857599 51379200 8e Linux LVM
….
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@rac02-g6 ~]#
10. Partition Align the shared disk (/dev/sdc) on ‘rac01-g6’ (do this on one node only) using the fdisk / parted utility / tool of your choice) :
11. After partition alignment:
root@rac01-g6 ~]# fdisk -lu /dev/sdc
……
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4402e64c
Device Boot Start End Blocks Id System
/dev/sdc1 2048 104857599 52427776 83 Linux
[root@rac01-g6 ~]#
[root@rac02-g6 ~]# fdisk -lu /dev/sdc
…..
Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4402e64c
Device Boot Start End Blocks Id System
/dev/sdc1 2048 104857599 52427776 83 Linux
[root@rac02-g6 ~]#
12. Create ASM disks using ASMLIB
The process of creating the ASM disks can be found at Installing and Configuring Oracle ASMLIB Software
root@rac01-g6 ~]# /usr/sbin/oracleasm createdisk DATA_DISK01 /dev/sda1
Writing disk header: done
Instantiating disk: done
[root@rac01-g6 ~]#
[root@rac01-g6 ~]# /usr/sbin/oracleasm listdisks
DATA_DISK01
[root@rac01-g6 ~]#
[root@rac02-g6 ~]# /usr/sbin/oracleasm scandisks
[root@rac02-g6 ~]# /usr/sbin/oracleasm listdisks
DATA_DISK01
[root@rac02-g6 ~]#
As we can see, we have now added a shared vmdk of size 50 GB to both VM’s without any downtime online and created ASM disk on this shared disk to be used for Oracle RAC ASM disk group.
The rest of the steps to create the Oracle RAC is exactly the same as shown in the Oracle documentation.
Summary
- VMware recommends using shared VMDK (s) with Multi-writer setting for provisioning shared storage for ALL Oracle RAC environments (KB 1034165)
- vSphere 6.0 and onwards, we can add shared vmdk’s to an Oracle RAC Cluster online using the Web Client
- Prior to version 6.0, we had to rely on PowerCLI scripting to add shared disks to an Oracle RAC Cluster online
- Best Practices needs to be followed when configuring Oracle RAC environment which can be found in the “Oracle Databases on VMware – Best Practices Guide”
- All Oracle on vSphere white papers including Oracle licensing on vSphere/vSAN, Oracle best practices, RAC deployment guides, workload characterization guide can be found at the “Oracle on VMware Collateral – One Stop Shop“
