Application Workload Guidance and Design for Virtualized SAP S/4HANA® on vSphere (Part 2/4)

In part 1 we introduced the concept of SAP HANA Application Workload guidance and using example business requirements to come up with a workload and vSphere cluster design for the SAP environment. In the second part we will look at storage, network and security design for the proposed customer environment.Availability Design

The availability design depends on the single point of failure (SPOF) analysis of components. There are components in the SAP infrastructure that are one of a kind and are potential SPOFs; other components are capable of having multiple instances for load balancing and availability.

SAP S/4HANA Architecture

This section summarizes SAP S/4HANA architecture concepts and terminology used in this document. SAP uses the term system landscape, which contains all the SAP systems that have been installed. It can consist of several system groups for which SAP systems are linked by transport routes. Transport routes refer to the path of code migrations between SAP systems, for example from development (DEV) to quality assurance (QAS) to production (PRD) (https://help.sap.com/saphelp_nw74/helpdata/en/63/a30a4ac00811d2851c0000e8a57770/content.htm).

The architecture of a single SAP system is multitier and consists of the following components:

• Application servers (SAP Web application servers) – These are ABAP or Java (J2EE) based, depending on the specific SAP product or module. Two types exist:
• Primary application server (PAS) – An application server instance that is installed with SAP Central Services in newer NetWeaver releases and is part of the base installation.
• Additional application servers (AAS) – Application servers installed as required for horizontal scalability.
• SAP Message Service – The SAP Message Service is used to exchange and regulate messages between SAP instances in an SAP system. It manages functions such as determining which instance a user logs in to during client connect and scheduling batch jobs on instances configured for batch.
• SAP Enqueue Service – The SAP Enqueue Service manages the locking of business objects at the SAP transaction level. Locks are set in a lock table stored in the shared memory of the host on which the SAP Enqueue Service runs.
• Database server – SAP S/4HANA and SAP S/4HANA Suite support SAP HANA as the backend database for all applications. Each module has its own individual standalone SAP HANA database.
• The following SAP services are defined based on the Message Service and Enqueue Service:
  • SAP Central Services – In newer SAP versions, the Message Service and Enqueue Service have been grouped into a standalone service. Separate SAP Central Services exist for ABAP- and Java-based application servers. For ABAP variants, it is called ABAP SAP Central Services (ASCS); for J2EE variants, it is called SAP Central Services (SCS).
  • Replicated enqueue server – This component consists of the standalone enqueue server and an enqueue replication server. The replicated enqueue server runs on another host and contains a replica of the lock table (replication table). If the standalone enqueue server fails, it must be restarted on the host on which the enqueue replication server is running, because this host contains the replication table in a shared memory segment. The restarted enqueue server uses this shared memory segment to generate the new lock table, after which the shared memory segment is deleted. SAP Central Services and the database are both SPOFs and therefore require considerations for high availability.

Single Point of Failure Analysis

The following SPOFs exist in the SAP NetWeaver architecture:

• Database – Every application work process makes a private connection to the database at the start. If the connection is interrupted due to database instance failure, the work process attempts to set up a new connection and changes to “database reconnect” state until the database instance restarts. User sessions with database activity in process receive SQL error messages, but their logged-in sessions are preserved on the application server.
• SAP Message Service and Enqueue Service – Failure of these services has a considerable effect on the system because all transactions that contain locks must be rolled back and any SAP updates being processed fail.

The isolation of the Message Service and Enqueue Service from the central instance (CI) helps address the high-availability requirements of these SPOFs. The SAP Central Services component is “lighter” than the CI and is much quicker to start up after a failure.

SAP Application Tier Availability

Every SAP application should have a minimum of two servers. Because they are serving the same function, these servers must be separated from each other by using antiaffinity rules. An SAP application server and database server can be collocated on the same physical server by using affinity rules to optimize performance in certain situations. vSphere DRS affinity rules help protect SAP application and database components by providing appropriate separation between the primary and standby servers as well as between multiple application servers.

Figure 1. SAP Application Tier Availability

vSphere HA for Database and vSphere FT for SAP Central Services:

The ideal high-availability solution for SAP S/4HANA SPOF components is to leverage vSphere HA and other SAP-specific mechanisms for the database and vSphere FT for SAP Central Services. By having more than two application servers with antiaffinity rules, the application services can be protected.

Figure 2. vSphere HA and vSphere FT – The Ideal High-Availability Solution for SAP S/4HANA SPOF Components

Storage Design

VMware storage virtualization can be categorized into three layers of storage technology. The bottom layer is the storage array, consisting of physical disks presented as logical disks—that is, storage array volumes or LUNs—to the layer above, the virtual environment occupied by vSphere. Storage array LUNs are formatted as VMware vSphere VMFS—that is, virtual machine file system—volumes in which virtual disks can be created. VMs consist of virtual disks that are presented to the guest OS as disks that can be partitioned and used in file systems. Figure 10 shows these storage layers.

Figure 3. Storage Virtualization Layers

vSphere VMFS is a cluster file system that provides storage virtualization optimized for VMs. Each VM is encapsulated in a small set of files, and vSphere VMFS is the default storage system for these files on physical SCSI disks and partitions. VMware supports Fibre Channel and iSCSI protocols for vSphere VMFS.

VMware also supports raw device mapping (RDM). RDM enables a VM to directly access a volume on the physical storage subsystem. It can be used only with Fibre Channel or iSCSI. RDM provides a symbolic link from a vSphere VMFS volume to a raw volume. The mapping makes volumes appear as files in a vSphere VMFS volume. The mapping file, not the raw volume, is referenced in the VM configuration.

To access virtual disks, a VM uses virtual SCSI controllers. These virtual controllers include BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS, and VMware Paravirtual. From the standpoint of the VM, each virtual disk appears as if it were a SCSI drive connected to a SCSI controller.

For more background on storage virtualization, see the vSphere Storage Guide at https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-60-storage-guide.pdf.

Virtual Storage Design

This section applies the virtual storage concepts to a storage design for an SAP database on vSphere. The example here is a scale-up SAP HANA system based on the virtual SCSI controller assignments described in the Best Practices and Recommendations for Scale-up Deployments of SAP HANA on VMware vSphere white paper (http://www.vmware.com/files/pdf/SAP_HANA_on_vmware_vSphere_best_practices_guide.pdf).

Virtual SCSI Driver	Virtual Device	File System
LSI Logic/Paravirtual	0:0	/usr/sapX
Paravirtual	1:0	/hana/data/<SID>
Paravirtual	2:0	/hana/log/<SID>
Paravirtual	3:0	/hana/shared/<SID>

Table 9. SAP HANA Layout for Virtual SCSI Controller

The boot drive can be configured with the paravirtual driver, but configuration depends on the guest OS version. For details, see VMware knowledge base article 1010298, Configuring Disks to Use VMware Paravirtual SCSI (PVSCSI) Adapters (http://kb.vmware.com/kb/1010398).

Figure 4. Example Storage Layout for Scale-Up SAP HANA

Storage Sizing by Module Based on SAP Guidelines

Module	Memory GB	Data GB	Log GB	Shared GB	Backup GB
BW/4HA_SRV	650	780	325	650	390
CRM	250	300	125	250	150
MDG	250	300	125	250	150
S/4	800	960	400	800	480
SRM	250	300	125	250	150
TM	250	300	125	250	150

Table 10. Database Component Storage Sizing for SAP S/4 HANA

As a default, the backup volume is stored under the /hana/shared volume. To store backup data, creation of a dedicated VMDK volume or NFS mount point is recommended. 

TDI Compliant Pure Storage:

The solution was implemented initially with FC Storage on an all All Flash Array from Pure Storage (M50). Pure Storage M50 array is an SAP TDI compliant storage.

Figure 5. Pure Storage FlashArray//M50

The following are some of the advantages of running SAP HANA on Pure Storage:

• Data reduction – SAP HANA performs data compression in column stores on two levels: dictionary and advanced. Further data reduction results of around 1.9–2.3 have been experienced. SAP HANA does not compress row store tables, so the data reduction is even higher when there are row store tables. This use case is analytics, so there are no row store tables.
• Encryption off-loading – SAP HANA encrypts only data volumes; it does not encrypt log volumes. Switch off the encryption on SAP HANA and use the Pure Storage FlashArray encryption, which can preserve valuable SAP HANA CPU cycles. That in turn can be used more efficiently to process analytical queries.
• SAP homogeneous system copy and backup and recovery using storage snapshots – System copy, backup, and recovery using Pure Storage snapshots are extremely fast and efficient. The entire process can be automated very easily via a script and can be scheduled on a regular basis. There is no need to invest in backint third-party SAP-certified tools. This process enables customers to achieve very low RTO and RPO.

Fibre Channel SAN Design:

A Brocade Generation 6 redundant and resilient SAN Fabric was used for this deployment. When designing the SAN for an all-flash array, understanding the application workloads and the intended scalability, redundancy, and resiliency requirements is the main factor to consider.

When deploying a Fibre Channel storage array, the main design consideration is the adequate sizing of the ISLs between the edge switches where the servers are connected and the core switches where the storage arrays are connected. In implementations where the Fibre Channel storage arrays are deployed to serve specific latency- and I/O-intensive applications, such as OLTP database servers, connecting both servers and Fibre Channel storage array to the core backbone switches can be advantageous.

Figure 6. Redundant Fibre Channel Fabric

Virtual SAN All Flash Storage:

After Validation of the solution, all the storage for the SAP modules were migrated to an All Flash virtual SAN infrastructure and the validation tests were repeated. Virtual SAN is not yet supported for SAP S4/HANA production but it can be used for QA and Dev/Test environments.

VMware virtual SAN was used for the management cluster and as an alternate for the TDI storage for SAP HANA. All storage components used were certified for VMware virtual SAN and provided by Western Digital.

VMware virtual SAN features the following components:

• Two disk groups per server
• Each disk group contains
• One NVMe 1.5TB drive for caching
• Two 3.5TB SSD drives for capacity

The total capacity of all-flash VMware vSAN is 52TB.

Figure 7. VMware vSAN All-Flash Datastore

SAP S/4HANA Network Design Considerations:

The network components were designed based on the foundational VVD concepts. VMware NSX is available as part of this design, to provide networking and security for application workloads. Details on the VVD network design and components can be found in Architecture and Design in VMware Validated Designs Documentation. VMware NSX is the foundation for networking in the Software-Defined Data Center (SDDC), which is leveraged for SAP S/4HANA. Each node has 4X10GBPS network connections across a pair of Brocade enterprise class VDX in redundant configuration.

SAP HANA Client Zone VMware VMware NSX Design:

VMware NSX greatly facilitates the creation and deployment of network services. The actual abstraction of these services, however, is a collaborative effort. To create an optimized virtual architecture, at a minimum this involves the network operations team, the storage team, virtual infrastructure administrators, application owners, and database administrators. This should not be an isolated task.

From a desktop environment using VMware NSX, the SAP HANA Network Requirements Guide for tailored data center integration deployments can be decomposed and translated into a virtual network design. Regarding the client zone, the application server network and the network for other SAP HANA clients such as the SAP HANA studio, business intelligence clients, and so on, can be either on the same network or on separate networks. For scalability, management, and security purposes, using separate networks via microsegmentation is recommended. Microsegmentation enables customers to secure, isolate, and characterize network traffic from workload to workload. We also recommend a distributed routing scheme rather than a centralized routing scheme, to optimize for both north–south traffic and east–west traffic.

The east–west dynamic routing capabilities of the DLR are key here because of how SAP HANA load-balances its client connections. SAP uses techniques such as statement routing, connection selection, and command splitting to direct queries to the proper nodes. These routing techniques are based on the type of query and on which node or nodes the data lives on.

As the database grows, the data distribution and types of queries can change. DLRs enable the central management and optimization of east–west traffic by proactively reacting to database growth and traffic patterns.

For external communication with SAP HANA servers that are initiated by a Web browser or a mobile application, a VMware NSX edge services perimeter gateway manages and optimizes north–south network traffic and leverages its multifunction capabilities as a firewall, load balancer, and virtual private network (VPN) device.

And because these external connections can have vastly different security requirements, customers can use VMware NSX to associate firewall rules at the router or at the VM level to achieve greater granularity.

Figure 8. SAP HANA Client Zone VMware NSX Network Design

SAP S4/HANA Security

Standard vSphere security best practices and hardening practices must be applied to secure the infrastructure. Follow guidelines for access control and security policies provided in the VMware Validated Designs Documentation. We address only the application-specific security for SAP S/4HANA in this section.

Microsegmentation for Securing SAP S/4HANA Workloads

SAP microsegmentation enables flexible security policies that align to the multitier architecture of an individual SAP system—presentation, application and database tiers—and also to the landscape of the SAP environment, separating production from nonproduction. Figure 17 shows an SAP microsegmentation example based on the NetWeaver ABAP stack with a backend database. The following are the various tiers and components of the SAP architecture:

• Client tier – In this example, the SAP client “SAPGUI” accesses the application tier. Customer environments can include browser-based access, load balancers, and a Web tier.
• Application or Central Services tier – Application servers based on the NetWeaver ABAP stack. SAP Central Services handles SAP locking services, messaging between the application servers, and an NFS share required by all the application servers.
• Database tier – Services are database dependent.
• The components are isolated into their own respective VMware NSX security groups. Although other classifications are possible, a VMware NSX security group in this example is a definition in VMware NSX and corresponds to a logical grouping of VMs within which there is free communication flow. Communication flow in and out of a security group, and from and to another group, depends on the firewall rules.

Figure 9. Segmentation of SAP S4/HANA with VMware NSX

Security policies shown in Figure 9 provide the following controls:

• Controlled communication path limited to specific services and protocols between tiers
• External access permitted to the application tier via the SAP presentation service
• Access between application and database VMs only via specific database services
• SAP services ports that vary depending on the “instance number” assigned to the application servers and SAP Central Services. Some values are shown here.

More details from this blog series can be found in this comprehensive Whitepaper. In part 3 we will look at monitoring & management, backup/recovery and disaster recovery for SAP S4/HANA.