SAP HANA with Intel Optane Persistent Memory on VMware vSphere

Intel Optane persistent memory (Optane PMem), is a new technology that bridges the gap between volatile DRAM and persistent storage. As the need for increased data processing continues to grow, optimizing SAP HANA with Intel Optane persistent memory on VMware vSphere will deliver much higher memory capacity much more affordably than with conventional DRAM memory alone. Increases in memory density combined with the power of virtualization, will help enterprises improve their overall total cost of ownership. Because persistent memory is non-volatile, data stays in-memory when the system is powered off, which enables dramatically faster startup times of SAP HANA databases and increased system uptime.

VMware supports Intel Optane PMem with vSphere 6.7 and 7.0. VMware and Intel worked with SAP to complete the validation of SAP HANA with Optane PMem enabled VMs.

I am very pleased to announce that validation is complete and the deployment of up to 6 TB (DRAM plus Optane PMem) SAP HANA VMs on vSphere 6.7 (beginning with version 6.7 EP 14), as well with vSphere 7.0 (beginning with version 7.0 P01), is now supported. For details check out SAP Note 2913410.

Check also out the SAP NetWeaver and SAP HANA with VMware vSphere 7.0 blog for the latest SAP support status for this vSphere release.

In this blog I describe the current support status, provide a configuration overview and some information on how to configure an SAP HANA database on vSphere that leverages Optane PMem.

Additional considerations before I cover the details:

VMware strongly recommends that customers get specific guidance from SAP. There are two major considerations for customers when planning to run virtual SAP HANA on Optane PMem.

For customers planning to use VMware High Availability (HA), this is currently not supported for SAP HANA Optane PMem-enabled VMs. Customers will need to consider a third-party product for their HA needs, like SAP HANA System Replication. Details can be found in the “Supported VMware features” section.

A second consideration is on how to properly size and deploy an SAP HANA workload with Optane PMem. Contact your SAP HANA sizing and architecture specialists and review SAP Note 2786237 – Sizing SAP HANA with Persistent Memory for details.

Unlike in a physical deployment, it is possible to consolidate several SAP HANA Optane PMem-enabled systems on a single ESXi host. This allows lower TCO, but adds some planning burden, since the Optane PMem ratio check must get done individually per SAP HANA system. See sizing section of this blog.

Why use Intel Optane PMem for SAP HANA on vSphere?

As we mentioned at the start of this blog post, Optane PMem brings some unique benefits to running SAP HANA. Back at Sapphire 2018, SAP discussed the benefits of greatly reduced startup times for SAP HANA where the startup time for data loading was reduced by a factor of over 12x*. This is a great benefit to running SAP by cutting startup times from hours to minutes potentially since in-memory data does not need to be written out to disk and reloaded after restart. Bringing this advantage to the benefits of running SAP HANA in a VMware virtual infrastructure, brings faster time to value, lower TCO, and greater flexibility and opens up new possibilities.

Learn more about the benefits of Intel Optane Persistent Memory for SAP HANA from SAP.

Before we start some pre-requisites and general SAP support limitations for Intel Optane PMem.

Review information published in SAP Note 2913410 – SAP HANA on VMware vSphere with Persistent Memory.
Only server platforms which are SAP HANA supported and certified and listed on VMware Compatibility Guide with “Persistent Memory” feature are supported.
The CPU and memory prerequisites of SAP HANA on vSphere must be met as a prerequisite. For details refer to SAP Note 2393917 – SAP HANA on VMware vSphere 6.7 in production and SAP Note 2937606 – SAP HANA on VMware vSphere 7.0 in production.
General questions and boundary conditions about SAP HANA and persistent memory are addressed in SAP Note 2700084, 2786237 and VMware KB 78094.

What is supported?

SAP has granted support for SAP HANA 2 SPS 4 (or later) on vSphere versions 6.7 (beginning with version 6.7 EP14) and vSphere 7.0 (beginning with version 7.0 P01) for 2 and 4 socket servers based on 2^nd Gen Intel Xeon Scalable processors (formerly code named Cascade Lake). The maximum DRAM plus Optane PMem host memory configurations can be up to 15 TB (current memory limit when DRAM with DCPMM gets combined) and must follow the hardware vendors Optane PMem configuration guidelines.

The maximum VM size with vSphere 6.7 and vSphere 7.0 is limited to 256 vCPUs and 6 TB of memory. This results in SAP HANA VM sizes of 6 TB maximum for OLTP and 3 TB VM sizes for OLAP workloads (Class L). vSphere 7.0 supports OLAP workloads up to 6 TB (Class M).

Supported DRAM:PMem ratios are 2:1, 1:1, 1:2 and 1:4. Please refer to SAP note 2700084 for further details, use cases, and assistance in determining whether Optane PMem is applicable at all for your specific SAP HANA workload.

Supported Optane PMem Module sizes are 128, 256 and 512 GB. Supported maximal host memory DRAM-PMem configurations are listed below. Up to two SAP HANA VMs are supported per CPU socket.

Performance of SAP HANA VMware VMs with Intel Optane PMem

During our SAP HANA Optane PMem validation with vSphere 6.7 and vSphere 7.0, different tests were conducted. For this we used, SAP HANA related benchmarks / tools that measure OLTP/OLAP type workloads, measuring NUMA node latencies and we verified typical use cases like massive data load activities.

Overall, the impact of Optane PMem, when compared to a DRAM-only SAP HANA VM on the same hardware configuration for CPU and memory, is very little. We observed performance impact ≤7%.

For data load activities where terabytes of data get moved, the difference between an SAP HANA Optane PMem and DRAM VM can be from 12% (test configuration 1) up to 17% (test configuration 2), which is comparable with the findings in increased load times as documented in following Fujitsu white paper, table 2. The tables below provide an overview of our findings with a 4-socket Intel Xeon Platinum 8280L processor-based server system when running the same tests on a DRAM-only or Ratio 1:1 Optane PMem SAP HANA VM. Note, in the table below the basis of the comparison is the SAP HANA DRAM VM. An increased percentage of the time related test cases means therefore a negative impact.

*4-socket Intel Xeon Platinum 8280L processor based server system with NVMe datastore. Results are average results from internal test with SAP HANA related benchmarks or workloads. Used DRAM:PMem ratio was 1:1.

Contents

1 *4-socket Intel Xeon Platinum 8280L processor based server system with NVMe datastore. Results are average results from internal test with SAP HANA related benchmarks or workloads. Used DRAM:PMem ratio was 1:1.
2
3 *4-socket Intel Xeon Platinum 8280L processor based server system with FC datastore. Results are average results from internal or published SAP specific benchmarks with different DRAM:PMem ratios.
4
5 *4-socket Intel Xeon Platinum 8280L processor based server system with FC datastore. Results are average results from internal or published SAP specific benchmarks with different DRAM:PMem ratios.

Comparing the different Optane PMem ratios, our tests show that mostly the data load tests are impacted, the CPU and memory related tests cases are impacted very little. The SAP HANA Optane PMem implementation using App Direct mode is the reason for this minimal impact on these workload types. The table below shows the load time of a DRAM VM as the basis and then compared with it a DRAM:PMem ratio 1:1, ratio 1:2 and ratio 1:4 configuration. The difference of a ratio 1:4, compared to a 1:2 ratio system, is very little.

*4-socket Intel Xeon Platinum 8280L processor based server system with FC datastore. Results are average results from internal or published SAP specific benchmarks with different DRAM:PMem ratios.

This table shows the same configuration. This time only the impact of Optane PMem on the queries performance, when compared with the DRAM VM.

*4-socket Intel Xeon Platinum 8280L processor based server system with FC datastore. Results are average results from internal or published SAP specific benchmarks with different DRAM:PMem ratios.

Sizing of SAP HANA Optane PMem VMs:

The sizing of Optane PMem-enabled SAP HANA VMs is similar to bare metal SAP HANA systems with the limitation of a maximum size of 6 TB (mix of DRAM and Optane PMem) per VM.

Please refer SAP note 2700084 and SAP Note 2786237 – Sizing SAP HANA with Persistent Memory for details on compute and memory sizing for Optane PMem enabled SAP HANA systems.

We recommend that an SAP HANA VM use the same DRAM:PMem ratio as the physical host/server DRAM:PMem ratio. However, if you have a growth plan, you might consider a larger physical memory configuration, and upgrade the VMs and SAP HANA over the lifetime.

For example, you have a 1:4 PMem ratio host, configured with 15 TB of total RAM (3 TB DRAM and 12 TB Optane PMem). An optimized resource scenario is to create four SAP HANA VMs on this server, each with 3.75 TB RAM (0.75 TB DRAM + 3 TB Optane PMem). If you are creating 6 TB VMs on this same 15 TB host, then only two SAP HANA VMs can be created, which is a non-optimized resource configuration as you can only leverage 12 of the installed 15 TB memory. In this case, a 1:1 DRAM:PMem configuration, with in total 12 TB (6 TB DRAM + 6 TB Optane PMem) represents a resource-optimized configuration. Future vSphere versions will support VM sizes >6 TB and will be able to leverage higher memory configurations!

See below examples for more details. The last example shows a configuration with 4 SAP HANA VMs with a 1:2 DRAM to PMem ratio.

4-Socket Host Configuration:

Four 2^nd Gen Intel Xeon Platinum processors, 24 x 128GB DRAM + 24 x 512GB Optane PMem = 15 TB total host memory with a 1:4 DRAM to PMem RATIO

VM Configuration

2 x 6 TB SAP HANA VM with 1.5 TB DRAM and 4.5 TB Optane PMem RAM, with a 1:3 DRAM to PMem RATIO

Challenges:

DRAM:PMem Ratio may not be suited for SAP HANA workload
HW configuration does not fit and will lead to unusable PMem (RATIO mismatch)

4-Socket Host Configuration:

Four 2^nd Gen Intel Xeon Platinum processors, 24 x 256GB DRAM + 24 x 256GB Optane PMem = 12 TB total host memory with a 1:1 DRAM to PMem RATIO

VM Configuration

2 x 6 TB SAP HANA VM with 3 TB DRAM and 3 TB Optane PMem RAM, with a 1:1 DRAM to PMem RATIO

Challenges:

Higher memory costs due to DRAM module prices

4-Socket Host Configuration:

Four 2^nd Gen Intel Xeon Platinum processors, 24 x 128GB DRAM + 24 x 256GB Optane PMem = 9 TB total host memory

VM Configuration

4 x VM with 0.75 TB DRAM and 1.5 TB Optane PMem RAM, total RAM per SAP HANA VM 2.25 TB with a 1:2 DRAM to PMem RATIO

Challenges:

SAP HANA Sizing to verify if Optane PMem Ratio is applicable and if CPU resources are enough!

Note: Workload based sizings (WBS) are supported and allow OLAP workloads with Class M CPU requirements to leverage up to 6 TB of total memory (DRAM+PMem).

Which vSphere versions are required for SAP HANA with Optane PMEM?

vSphere Persistent Memory^TM requires vSphere Enterprise Plus. For details please review following document (page 6).

Which VMware vSphere 6.7 and 7.0 features are supported?

The following advanced vSphere features are supported with Optane PMem including VMware vMotion, VMware DRS.

Using SAP HANA on VMware vSphere allows HANA users to leverage the flexibility of vSphere capabilities such as vMotion, which allow workloads to be migrated between vSphere hosts on Intel Xeon platforms without first having to be shut down. In addition, VMware’s Dynamic Resource Scheduler (DRS) works with a cluster of ESXi hosts to provide resource management capabilities like load balancing and virtual machine (VM) placement to ensure a balanced environment for VM workloads.

VMware HA is yet not supported for SAP HANA VM Optane PMem use cases. A later vSphere 7.x version will support VMware HA for SAP HANA Optane PMem-enabled VM’s.

However, vSphere can be configured to restart SAP HANA Optane PMem VMs automatically on the same host in case of a VM failure. To configure this, – enable vSphere HA on the cluster and enable VM monitoring. Please note, if you enable vSphere HA on the cluster, it will not restart the SAP HANA VM with Optane PMem on a different host, if local host fails. For SAP HANA DRAM only VMs, vSphere HA can get used without restrictions or limitations.

A word on VMware HA, VMware vSphere HA allows automated restarts of VMs, when there is a server outage, or an operating system failure detected in the vSphere environment. VMware HA requires a file system accessible from all hosts on the vSphere cluster. Optane PMem in App Direct mode is not a shared resource, therefore, VM failover to another host in the vSphere cluster is not possible and is, as of today, not supported with Optane PMem-enabled VMs.

SAP HANA System Replication with Intel Optane PMem enabled SAP HANA VMs:

Depending on your SAP HANA SLAs, customers can define a strategy to achieve them. All SAP HANA build-in capabilities, like SAP HANA System Replication, are supported with Intel Optane PMem and can be leveraged instead of VMWare HA to be used to help recover from catastrophic host failures for HANA VMs.

Following table shows a summary of the supported features and what needs to get considered when using these features:

SAP HANA System Replication with Optane PMem enabled SAP HANA VMs:

SAP HANA System replication can get leveraged to protect, just like with physical deployed SAP HANA systems, an SAP HANA DRAM only or Optane PMem enabled VM.

Following example highlights how persistent memory can help to lowering the cost by leveraging a Optane PMem host as the replication target for two SAP HANA DRAM VMs. The failover of the DNS name and IP address gets handled by a cluster solution like Linux Pacemaker. The SAP HANA data gets replicated via HANA System Replication.

After a failover of a production SAP HANA VM it may be required to free up CPU resources on the failover host. This can get done by either shutdown the other disaster recovery (DR) VM or by moving it via vMotion to another DR host. Once the defective host got repaired and is back online, the SAP HANA production instance should get moved back and SAP HANA System Replication should resume its work.

This is just an example how SAP HSR and Optane PMem can get used in combination to lower the TCO of an DR environment. This example shows a host with a DRAM to Optane PMem ratio of 1:1, which is in this case the best choice, since it will allow the operation of the example DRAM-only SAP HANA VMs without further sizing.

SAP HANA VM Configuration details:

To configure an SAP HANA Optane PMem-enabled VM for optimal performance it is necessary to align the VM configuration to the underlying HW and especially NUMA configuration.

VMware KB 78094 provides information on how to configure the NVDIMMs (VMware’s representation of Optane PMem) correctly and align the NVDIMMs to the physical NUMA architecture of the physical server.

By default, Optane PMem allocation in vmkernel for VM NVDIMMs does not consider NUMA. This can result in the VM running on a certain NUMA node and Optane PMem allocated from a different NUMA node. This will cause NVDIMMs access in the VM to be remote, resulting in poor performance. To solve this, you must add the following settings to a VM configuration using vCenter.

Example for a 4-socket wide VM:

nvdimm0:0.nodeAffinity=0
nvdimm0:1.nodeAffinity=1
nvdimm0:2.nodeAffinity=2
nvdimm0:3.nodeAffinity=3

Parameter sched.pmem.prealloc=TRUE is an optional parameter equivalent to eager zero thick provisioning of VMDKs and improves initial writes to Optane PMem.

Beside these parameters, you may also configure the CPU NUMA node affinity or CPU affinities (pinning), as described in the SAP HANA best practices parameter guidelines listed here. For details on CPU affinities please review the 2^nd page of this list.

Can I use vSAN / SAP HANA HCI with Optane PMem?

VMware vSAN does not have support for “App-Direct” mode (source) as cache or as a capacity tier device of vSAN. However, vSAN will work with vSphere hosts equipped with Intel Optane PMem in App-Direct mode and SAP HANA VM’s can leverage PMEN according to SAP Note 2913410. Please note that especially the VMware HA restriction (as described in note 2913410) applies and needs to get considered.

Conclusion:

We are excited for the possibilities of adding Intel Optane PMem support to the other benefits of running SAP HANA on vSphere, and we look forward to hearing how you use these powerful technologies together in your environment.

References: