External storage solution for HP ProLiant Moonshot Servers via iSCSI - Technical white paper

Technical white paper

External storage solution for HP ProLiant Moonshot Servers via iSCSI Table of contents Introduction .................................................................................................................................................................................... 3 HP Moonshot System ................................................................................................................................................................... 3 HP ProLiant m700 Server Cartridge overview ........................................................................................................................ 3 HP ProLiant m710 Server Cartridge overview ........................................................................................................................ 4 HP ProLiant SL4500 Server series ............................................................................................................................................. 4 HP 3PAR StoreServ 7400 Storage ............................................................................................................................................. 5 Section 1: HP ProLiant m700 Server Cartridge ....................................................................................................................... 5 Purpose ....................................................................................................................................................................................... 5 Configuration ............................................................................................................................................................................. 5 Scope ........................................................................................................................................................................................... 6 FIO benchmark consideration ................................................................................................................................................ 6 Overview of iSCSI ........................................................................................................................................................................... 7 iSCSI client and initiator ........................................................................................................................................................... 7 iSCSI target ................................................................................................................................................................................. 7 Tested iSCSI configurations ........................................................................................................................................................ 8 Single-client iSCSI initiator configuration............................................................................................................................. 8 Multi-client iSCSI initiator configuration ............................................................................................................................. 10 Recommended tunable parameters ....................................................................................................................................... 15 Tunable parameter for ProLiant SL4540 Server ............................................................................................................. 15 Tunable parameter for Moonshot Server Cartridges ...................................................................................................... 16 ProLiant SL4540 Server native (attached) disks results .................................................................................................... 16 Summary (Section 1) .................................................................................................................................................................. 17 Section 2: HP ProLiant m710 Server Cartridge ..................................................................................................................... 17 Purpose ..................................................................................................................................................................................... 17 Cartridge configuration .......................................................................................................................................................... 17 Network configuration ........................................................................................................................................................... 18 Scope ......................................................................................................................................................................................... 19

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Tested iSCSI configurations ...................................................................................................................................................... 19 iSCSI configuration with SAS disks from ProLiant SL4540 Server ............................................................................... 19 iSCSI I/O trend for sequential operation ............................................................................................................................. 23 iSER configuration with ProLiant SL4540 Server ................................................................................................................. 26 Configuring RDMA on a Moonshot system ............................................................................................................................ 26 Overview of iSER ..................................................................................................................................................................... 26 How to configure tgt (enabled with iSER) transport for RHEL ...................................................................................... 27 How to configure tgt (enabled with iSER) transport for Ubuntu .................................................................................. 28 How to configure LIO (enabled with iSER) for Ubuntu 14.04 using Inbox driver ...................................................... 29 How to configure LIO (enabled with iSER) for RHEL 7 using Inbox driver ................................................................... 31 Single-client iSCSI initiator configuration........................................................................................................................... 33 Multi-nodes iSCSI initiator configuration ........................................................................................................................... 34 I/O trend for sequential operation ...................................................................................................................................... 37 iSCSI configuration with SSD (SATA) disks from HP ProLiant SL4540 Server ................................................................ 39 Single-client iSCSI initiator configuration........................................................................................................................... 39 Multi-nodes iSCSI initiator configuration ........................................................................................................................... 40 iSCSI I/O trend for sequential operation ............................................................................................................................. 43 iSCSI configuration with HP 3PAR StoreServ 7400 Storage .............................................................................................. 45 Single-client iSCSI initiator configuration........................................................................................................................... 45 Multi-nodes iSCSI initiator configuration ........................................................................................................................... 46 iSCSI I/O trend for sequential operation ............................................................................................................................. 49 Comparisons................................................................................................................................................................................. 51 Summary (Section 2) .................................................................................................................................................................. 59 References .................................................................................................................................................................................... 59

2

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Introduction This white paper outlines some key configurations to provide external storage solution for HP ProLiant Moonshot Server Cartridges over iSCSI. We have chosen HP ProLiant m700 Server Cartridges and HP ProLiant m710 Server Cartridges for demonstrating iSCSI performance. And, HP ProLiant SL4540 Server/HP 3PAR StoreServ 7400 Storage are the storage targets. The low-cost and high-compute power of the HP Moonshot Server Cartridges coupled with high-density storage make it a suitable external storage solution for customers who depend on large storage space and data reliability. The configurations outlined in this paper have been tested by HP for deployment and feasibility. To test the configurations described in this paper, we have used the Linux® Flexible I/O (FIO) benchmark to carry out performance tests. It also characterizes the ProLiant m700 and ProLiant m710 Server Cartridges’ behavior, along with ProLiant SL4540 Server and HP 3PAR StoreServ 7400 Storage. The ProLiant m710 Server Cartridge introduces a NIC with remote direct memory access (RDMA) capabilities. So, we can demonstrate the performance gains from RDMA-over-Converged-Ethernet (RoCE) when the ProLiant m710 Server Cartridge is used with ProLiant SL4540 Server (also RoCE capable). This paper also presents native I/O performance results of the ProLiant SL4540 Server reference configuration to understand how iSCSI performs with regard to native I/O. This helps the customers decide whether the iSCSI performance meets their requirements with their current workloads before they consider migrating to ProLiant Moonshot systems for better total cost of ownership that includes savings on power and space.

HP Moonshot System HP Moonshot System consists of an HP Moonshot 1500 Chassis and software-defined ProLiant m700 or ProLiant m710 Server Cartridges. The Moonshot 1500 Chassis is a 4.3U rackmount unit that holds up to 45 hot-pluggable server cartridges, two internal network switches, two external 10 or 40GbE uplink modules, and other supporting components. Each ProLiant m700 Server Cartridge is a high-density design that has four system on a chips (SoCs) in it, amounting to 180 servers in the 4.3U chassis. Each ProLiant m710 Server Cartridge has one high-performance SoC amounting to 45 servers in a 4.3U chassis. This shared, federated approach saves energy, space, and cost. Just as importantly, the approach enables extreme scale-out capacity without a corresponding increase in complexity and management overhead. The network fabric is integrated into the system and shared to reduce cabling. To increase efficiency, every component above the processor level—including power, network fabric, and HP Integrated Lights Out (iLO) management—is pulled into the chassis and shared among all the server cartridges, resulting in a significant power and cooling savings.

HP ProLiant m700 Server Cartridge overview The ProLiant m700 Server Cartridge has a 4-core AMD processor with 4 x 8 GB (32 GB) RAM and up to 64 GB per node of integrated solid state drive (iSSD). Each cartridge has two internal 1 Gbps network interface cards, and the chassis is connected to the external network with six 10 Gbps SFP+ uplinks (or 1 Gbps using the RJ-45 SFP transceiver). The ProLiant m700 Server Cartridge supports the following operating systems: • Windows Server® 2012 R2 • Microsoft® Windows® 7 64-bit • Red Hat® Enterprise Linux 6.5 (RHEL 6.5) • Canonical Ubuntu 13.10 • SUSE Enterprise Linux

3

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

HP ProLiant m710 Server Cartridge overview The ProLiant m710 Server Cartridge has a 4-core Intel® Xeon® Processor E3 family with 4 x 8 GB (32 GB) RAM and up to 480 GB per node of solid state storage (M.2 SSD). Each cartridge has two internal 10 Gbps network interface cards with RoCE capability, and the chassis is connected to the external network with four 40 Gbps QSFP+ uplinks (or 10 Gbps using the SFP+ adapter and transceiver). The ProLiant m710 Server Cartridge supports the following operating systems: • Windows Server 2012 and 2012 R2 • Red Hat Enterprise Linux 6.5 and 7.0 • Canonical Ubuntu 14.04 • SUSE Linux Enterprise 11 Service Pack 3

HP ProLiant SL4500 Server series The SL4500 Server series is an excellent choice for running iSCSI target as it provides up to 60 LFF SAS/SATA/SSD storage capacity. It is a dual-socket Generation 8 (Gen8) server, with a choice of five different Intel Xeon processors, up to 192 GB memory, and one PCIe slot for expansion per node. Also, every compute node has its own dedicated networking ports. The ProLiant SL4540 Gen8 Server offers various advancements and benefits of the Gen8 platform: • HP SmartMemory: Offers memory performance improvements and enhanced functionality as listed for Gen8 in the

HP SmartMemory QuickSpecs at: h18000.www1.hp.com/products/quickspecs/14225_div/14225_div.html. • HP Smart Array RAID controllers: Provides HP Smart Array P420i Controller, which is located in the I/O module behind

the ProLiant SL4540 Server enclosure. It also controls LFF/3.5" hard drives in the chassis. • HP Agentless Management: With HP iLO Management Engine in every HP ProLiant Gen8 server, the base hardware

monitoring and alerting capability is built into the system (running on the HP iLO chipset) and starts working the moment that a power cord and an Ethernet cable are connected to the server. • HP Intelligent Provisioning: Includes everything needed for system setup and deployment. Firmware, drivers, and tools

to get systems online are embedded on a NAND flash chip on the server motherboard (so, HP SmartStart CDs or firmware DVDs are not necessarily needed). For further information, refer h17007.www1.hp.com/us/en/whatsnew/proliantgen8/index.aspx and QuickSpecs. Figure 1. ProLiant SL4540 Server and its 1-node configuration with 60 disk drives

4

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

HP 3PAR StoreServ 7400 Storage HP 3PAR StoreServ 7400 Storage offers enterprise Tier-1 hardware-assisted thin storage capable of achieving “six-nines” availability. Using two or four controller nodes and with eight 10 Gbps iSCSI ports, StoreServ 7400 supports up to 2048 initiators. It can accommodate 24 HDD/SSDs in a 2U controller enclosure, and scales up to 18 enclosures providing a RAW storage capacity of up to 1200 TB. For further information, refer: www8.hp.com/us/en/products/data-storage/3parstoreserv.html. For the latest information on supported operating systems, refer single point of connectivity knowledge (SPOCK) for HP Storage products: hp.com/storage/spock. Figure 2. HP 3PAR StoreServ 7400 Storage with 24 disk drives

Section 1: HP ProLiant m700 Server Cartridge Purpose The purpose of this white paper is to present iSCSI I/O performance results from HP ProLiant m700 Server Cartridges. These server cartridges use storage logical volumes presented by HP ProLiant SL4540 Server over iSCSI. The combination is a well-balanced and scalable iSCSI external storage solution that runs on HP Moonshot System. This high-compute and high-density system (with up to 180 servers in a chassis) also consists of ProLiant SL4540 Server that comes with high-density storage (up to 60 LFF SAS/SATA/SSD [i.e., 180 TB]). The performance numbers are presented for various RAID configurations.

Configuration The ProLiant m700 Server Cartridge has following specifications: CPU

(4) 15 W soldered down AMD Kyoto, 4-core 1.5 GHz with integrated GPU

Memory

(4) 8 GB DDR3 SO-DIMM ECC 1600 MHz

Network

(2) 1GbE NICs per nodes (BCM5720 on the cartridge)

Storage

128 GB iSSD (32 GB/node)

Management

(1) Satellite controller w/interface to CM iLO

The ProLiant SL4540 Server has following specifications: CPU

Dual 8-core Intel Xeon E5-2470 2.3 GHz processors with hyper-threading

Memory

96 GB DDR3 memory

Network

2 x 10GbE Ethernet NICs

Storage

60 x 3 TB SAS 15k rpm drives

Controllers

2 x HP Smart Array P420i Controller

5

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 3. iSCSI configuration

Note One NIC port of 1 GB on ProLiant m700 and 10 GB on ProLiant SL4540 have been used to capture the performance data points with the above configuration. However, one can configure to utilize both the internal NICs to provide better throughput.

We have used HP 2920 Switch Series to connect HP ProLiant Moonshot Server Cartridges and ProLiant SL4540 Server for convenience. But, the user can make direct connection between the two avoiding a switch. The impact in latency for going through a switch would be negligible.

Scope We have included the results of iSCSI performance results for configurations that demonstrate the capabilities of Moonshot Server Cartridges with ProLiant SL4540 Server’s reference architecture. We also comment on other configurations and their impact on the performance under the section “considerations and observations” at the end of single-client and multi-client configurations. • Benchmark: FIO • Test different RAID levels (RAID0, RAID10, RAID5, RAID6) • Logical volumes distributed across multiple disks with striping • Single client vs. multiple clients tested for scaling • Test I/Os on the RAW disks with no file system mounted • Test with a single iSCSI target daemon and multiple iSCSI target daemons • Perform native I/O tests on ProLiant SL4540 Server to compare with iSCSI • Identify tunable parameters to improve the iSCSI performance

FIO benchmark consideration FIO is a benchmarking tool that spawns a number of threads or processes doing a particular type of I/O action as specified by the user. Native The FIO benchmark tool is executed on ProLiant SL4540 Server (RHEL) targeting the attached or native disk drives in consideration. A set of disks are selected to form a logical volume and FIO is used on that volume. Remote iSCSI The FIO benchmark tool is executed on Moonshot Server Cartridges (Windows Server 2012 R2 and Ubuntu) targeting the remote volumes of ProLiant SL4540 Server presented to Moonshot through iSCSI. The same logical volume is used for the native test.

6

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Overview of iSCSI Internet small computer system interface or iSCSI is a way of connecting storage devices over a network using TCP/IP. It is a protocol that uses TCP to transport SCSI commands for a storage network, enabling existing TCP/IP infrastructure to be used as a SAN. iSCSI presents logical volume storage from target server to the initiator or client. Unlike traditional Fibre Channel (FC), which requires special-purpose cabling, iSCSI can be run over long distances using existing network infrastructure.

iSCSI client and initiator iSCSI client or initiator is a system that attaches to an IP network and initiates requests and receives responses from an iSCSI target. Each iSCSI initiator is identified by a unique iSCSI Qualified Name (iQN), this is equivalent to a FC Worldwide Name (WWN). An iSCSI driver must be installed on the client to transport blocks (SCSI) commands over the IP network. In the current scenario, ProLiant m700 Server Cartridge is used as an iSCSI initiator. The Linux driver version used as an iSCSI initiator is 1.0.38.

iSCSI target An iSCSI target is a device that receives iSCSI command and processes clients or initiators. The target service can be executed on a single to multiple port through which the connection is established on the clients. One or more iSCSI sessions are created for communication between target and initiator. In the current context, a ProLiant SL4540 Server is being used as an iSCSI target server. The Linux tgtd service is installed on this server and multiple instances of this service has been configured for the iSCSI sessions. The target framework daemon, version 1.0.24 has been used on ProLiant SL4540 Server. For an iSCSI initiator to establish an iSCSI session with a target, the initiator needs an IP address, TCP port information, and iSCSI name information. It establishes a TCP connection and the communication occurs over one or more connections. The TCP connections carry control messages, SCSI commands, parameters, and data within iSCSI Protocol Data Units (iSCSI PDUs). For every iSCSI connection, a dedicated TCP connection is used. One such connection is called an iSCSI session. A session is a group of connections that link an initiator with a target. TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one and the same target. Figure 4. iSCSI session

Multipath I/O (MPIO) is a fault-tolerance and performance enhancement technique whereby there is more than one physical path between the CPU in a computer system and its mass storage devices through the buses, controllers, switches, and bridge devices connecting them. Multiple Connections per Session (MC/S) is a feature of iSCSI protocol, which combines several connections inside a single session for performance and failover purposes. MC/S is done on the iSCSI level, while MPIO is done on the higher level. So, all MPIO infrastructure is shared among all SCSI transports, including FC, SAS, and others.

7

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 5. Multipath and multiple connections per session

Note Using the two available NICs (instead of one NIC) is another iSCSI configuration that can be deployed for higher throughput. However, this configuration has not been tested in the current context of ProLiant m700 Server Cartridges.

Tested iSCSI configurations Single-client iSCSI initiator configuration In this configuration, a single logical volume has been created with a set of 10 SAS 15k rpm disks available on ProLiant SL4540 Server. Later, the same logical volume is presented to one iSCSI initiator running on a ProLiant m700 Server Cartridge. Figure 6. Single iSCSI client—Sequential read on single volume from ProLiant SL4540 Server

8

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 7. Single iSCSI client—Sequential write on single volume from ProLiant SL4540 Server

Figure 8. Single iSCSI client—Random read-write mix (70:30) result on a single volume from ProLiant SL4540 Server

Considerations and observations • All the RAID levels consume full 1 Gbps network speed supported by ProLiant m700 Server Cartridge. • Even as performance data across all RAID levels are comparable, RAID0 demonstrated the best results in terms of

IOPS, latency, and bandwidth. • All the configurations in figures 6, 7, and 8 are tested with single volume presented to an iSCSI client/initiator. • The best result for the combination of buffer size and iodepth (IOD) and latency is 4k, IOD = 8. Though the IOPS increased

to 10–15 percent higher when iodepth was 32, the latency also increased by 3–4 times while using single client. • For steady state performance, the read and write cache has been disabled on ProLiant SL4540 Servers and

HP Smart Array P420i Controller. The output is directly proportional to the percentage of the caches used at the controllers. The test is executed for a size of 5 GB for 10 minutes with an iodepth of 8 and buffer size of 4k. • All the logical volumes are presented as RAW device and no file system was mounted on them. • The random read-write mix graph depicts a consolidated result of IOPS, latency, and bandwidth for all types of tested

RAID levels with a 70 percent read and 30 percent write ratio.

9

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Multi-client iSCSI initiator configuration In this configuration, multiple logical volumes are created with multiple disk sets on ProLiant SL4540 Server. These logical volumes are then presented to ProLiant m700 Server Cartridges. One node of ProLiant m700 Server Cartridge is used as an iSCSI initiator or client and presented on one logical volume. Figure 9. Multiple clients (15)—Sequential read test results from 1 of 15 iSCSI clients with single iSCSI volume, created from a volume group mounted on three sets of logical volumes (30 SAS disks) from ProLiant SL4540 Server

Figure 10. Multiple clients (15)—Sequential write test results from 1 of 15 iSCSI clients with single iSCSI volume, created from a volume group mounted on three sets of logical volumes (30 SAS disks) from ProLiant SL4540 Server

10

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 11. Multiple clients (30)—Sequential read test results from 1 of 30 iSCSI clients with single iSCSI volume, created from a volume group mounted on six sets of logical volumes (60 SAS disks) from ProLiant SL4540 Server

Figure 12. Multiple Clients (30)—Sequential write test results from 1 of 30 iSCSI clients with single iSCSI volume, created from a volume group mounted on six sets of logical volumes (60 SAS disks) from ProLiant SL4540 Server

11

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 13. Multiple clients (45)—Sequential read test results from 1 of 45 iSCSI clients with single iSCSI volume, created from a volume group mounted on six sets of logical volumes (60 SAS disks) from ProLiant SL4540 Server

Figure 14. Multiple clients (45)—Sequential write test results from 1 of 45 iSCSI clients with single iSCSI volume, created from a volume group mounted on six sets of logical volumes (60 SAS disks) from ProLiant SL4540 Server

12

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 15. Multiple clients (15)—Random read-write mix (70:30) test results from 1 of 15 iSCSI clients with single iSCSI volume, created from a volume group mounted on three sets of logical volumes (30 SAS disks) from ProLiant SL4540 Server

Figure 16. Multiple clients (30)—Random read-write mix (70:30) results from 1 of 30 iSCSI clients with single iSCSI volume, created from a volume group mounted on six sets of logical volumes (60 SAS disks) from ProLiant SL4540 Server

13

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 17. Multiple clients (45)—Random read-write mix (70:30) results from 1 of 45 iSCSI clients with single iSCSI volume, created from a volume group mounted on six sets of logical volumes (60 SAS disks) from ProLiant SL4540 Server

Considerations and observations • In this configuration, 15, 30, and 45 ProLiant m700 Server Cartridge nodes are concurrently performing the I/O on the logical volumes from ProLiant SL4540 Server. • Every 5–6 iSCSI clients and initiators (three ProLiant m700 Server Cartridges) are served by one tgtd daemon (server

process) with at least 32 to 64 threads running on each. Mapping one tgtd to each client does not make any difference in the throughput. • A volume group is created by combining three logical volumes (10 SAS disks each) from the ProLiant SL4540 Server

physical disks. This is further striped into multiple logical volumes. In case of 30 and 45 nodes testing, 60 disks across both the controllers are used and logical volumes created on this. • Each node of ProLiant m700 Server Cartridge has one NIC port enabled and contributes up to 1 Gb bandwidth. On

ProLiant SL4540 Server, one NIC of 10 Gb is enabled and contributes up to 10 Gb bandwidth. • Each partitioned logical volume is presented to one of 15, 30, and 45 iSCSI clients running on different ProLiant m700

Server Cartridge nodes through which concurrent I/O is performed. • It is observed that the overall CPU usage on ProLiant SL4540 Server is less than 30 percent for all the configurations even

when 15, 30, and 45 iSCSI clients concurrently performed I/Os on ProLiant SL4540 Server disks. SL4540 is capable of supporting a full chassis of 180 ProLiant m700 Server Cartridge nodes.

14

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Recommended tunable parameters Some of these parameters boosted the overall I/O performances on the ProLiant SL4540 Server as well as on iSCSI clients.

Tunable parameter for ProLiant SL4540 Server Tunable parameter

Comments

FIO/Disk test parameters

Block size = 4k iodepth = 8 to 32 Num jobs = 1 to 10

Number of threads in tgtd target process

It is 16 by default for tgtd version 1.0.44. Increased to up to 32.

Number of target processes

Performance doubled with 3–5 tgtd instances each serving 3–5 clients. Though one-to-one mapping did not make any difference.

RAID levels

• RAID0 resulted in best performance for IOPS, bandwidth, and latency. • A volume group created on top of RAID, later striped and partitioned into logical volumes with equal stripe size. • Number of disks contributing to logical volumes increased the performance.

NIC mapping

A set of initiators were mapped to one NIC and another set to another NIC on the same target.

NIC bonding

Multiple Ethernet cards can be bonded logically to increase throughput and load sharing beyond what a single connection could achieve. NIC bonding can be done at both HP SL4540 Server and HP ProLiant Moonshot Server Cartridge level. But NIC bonding was done only at ProLiant SL4540 Server for this testing. Significant improvements of throughput might not be expected out of this.

Set readahead block size

This setting tunes the sequential read performance. Should be preferred while dealing with applications such as databases, file and Web servers, and others. For example, this can be achieved at ProLiant SL4540 Server with following command: “blockdev --setra 2048 /dev/sdc” This sets the readahead to 2048 x 512 bytes = 1024 Kb

Disk parameter on ProLiant SL4540 Server

Following disk parameters were tuned to sync with readahead and stripe size of the data: Tuned disk parameters. echo noop > /sys/block/sdc/queue/scheduler echo 1024 > /sys/block/sdc/queue/nr_requests echo 1024 > /sys/block/sdc/queue/max_sectors_kb

Enable write-back cache

With a write-back cache, the device does not copy cache data to non-volatile media until absolutely necessary. This operation improves the performance of write operations. With a write-through cache, the device writes data to the cache and the media in parallel. This does not improve write performance, but it makes subsequent read operations faster. For example, this can be achieved at ProLiant SL4540 Server with following command: hpssacli ctrl slot=1 modify drivewritecache=enable

15

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Tunable parameter for Moonshot Server Cartridges Tunable parameter

Comments

Change Ethernet adapter settings at HP ProLiant Moonshot Server Cartridge level

Both N/W ports at the cartridge were enabled to share the TCP load. • On Windows®, Ethernet Port Adapter Properties->Max No. of - - RSS Queues was set to 4 or 2 depending on the number of cores sharing the load. This forces routing of packets to separate cores. RSS distributes incoming network I/O packets among logical processors. • Ports were dedicated CPU cores to distribute the traffic equally. For example, Set-NetAdapterRss -name “Ethernet 2” –BaseProcessorNumber 0 MaxProcessorNumber 1 Set-NetAdapterRss -name Ethernet –Base Processor Number 2 MaxProcessorNumber 3 • The profile was set to “Closest Static” to achieve the best result on the TCP packets distribution over the CPU cores. For example, Set-NetAdapterRss -name “Ethernet 2” –Profile ClosestStatic Set-NetAdapterRss -name Ethernet –Profile ClosestStatic

Dual redundant configuration of NICs

The first port of the NICs, both in the initiator and target in an iSCSI configuration, are in one subnet and the second ports are in the other subnet. Configuring multipath with this results in higher throughput.

ProLiant SL4540 Server native (attached) disks results In this configuration, FIO benchmark tool is executed directly on the ProLiant SL4540 Server with same parameters which were used on ProLiant m700 Server Cartridge. Figure 18. Single volume—Attached disk test on ProLiant SL4540 Server

Considerations and observations • In this configuration, a single logical volume is created on ProLiant SL4540 Server from 10 attached SAS disks. • For steady performance data, the caching is disabled and results are captured on a RAW device. No file system is

mounted on any of the logical volume.

16

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Summary (Section 1) • The increase in number of clients pump more transactions and hence bandwidth increases. Also, latency would increase

as it takes more time to complete the entire operation. • The best results are achieved with 4k buffer size and iodepth of 32 across tests. However, the latency is excessive at

3–4 times higher than the average. The iodepth of 8 resulted in a better latency. • Other disk I/O tools are used to validate the FIO test tools’ results. • A striping over RAID0 showed best results for all types of read and write operations while on native (attached) and

iSCSI (remote) disks. • RAID6 and RAID5 are very close, but almost comparable on native and iSCSI disks test for read-intensive workloads. • The striping brings about 10–20 percent increase in overall performance for all types of RAID performances on the

IOPS, latency, and bandwidth. • At first, a single instance of tgtd service with at least 16 threads per initiator, on ProLiant SL4540 Server is executed.

Later when number of instances (or multiple instances) of tgtd services are executed, the IOPS and bandwidth almost doubled. Though the latency varied for some of random read and write operations. • In multi-core processor systems such as ProLiant SL4540 Server, multiple tgtd processes could run and execute in

parallel. This resulted in better throughput compared to a single tgtd process. • The ProLiant SL4540 Server is quite effective in response even with 45 clients running on the ProLiant m700 Server

Cartridge nodes generating multiple I/Os concurrently. Even at maximum loads, the highest peak attained for overall CPU is 30 percent only. • The slow response is observed for scenarios where concurrent iSCSI I/Os are executed from initiators (10+) with single

tgtd on the target (ProLiant SL4540 Server) and the logical volumes are not striped. However, the memory and CPU still remains comparable. • The data published here using FIO benchmark is for reference purpose only. The results may vary, however, based on the

real-world applications.

Section 2: HP ProLiant m710 Server Cartridge Purpose The purpose of this white paper is to present iSCSI I/O performance results from HP ProLiant m710 Server Cartridges. These server cartridges use storage logical volumes presented by HP ProLiant SL4540 Server and HP 3PAR StoreServ 7400 Storage over iSCSI. The combination is a well-balanced and scalable iSCSI external storage solution that runs on HP Moonshot System. This high-compute and high-density system (with up to 45 ProLiant m710 Server Cartridges in a chassis) also consists of ProLiant SL4540 Server that comes with high-density storage (up to 60 LFF SAS/SATA/SSD [i.e., 180 TB]), and HP 3PAR StoreServ 7400 that delivers reliability and scalability (up to 18 enclosures and 1200 TB). The performance numbers are presented for RAID0 on different configurations.

Cartridge configuration The ProLiant m710 Server Cartridge has the following specifications: CPU

(1) Intel Xeon E3-1284L v3 CPU @ 1.80 GHz, 45 W, CPU list: 0–7

Memory

(4) 1600 MHz DDR3 SO-DIMM ECC (32 GB total)

Network

(1) Integrated 2-port 10GbE Mellanox MT27500 Family (ConnectX-3) NICs

Storage

(1) SATA disk (120 GB SSD)

Management

(1) Satellite controller with interface to iLO CM

17

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 19. HP ProLiant m710 Server Cartridge

Network configuration The network between ProLiant SL4540 Server and ProLiant m710 Server Cartridge has been configured as a dual-redundant configuration. The first port of the NICs, both in the initiator (ProLiant m710 Server Cartridge) and target (ProLiant SL4540 Server), are in one subnet, and the second port is in the other subnet. This enables high availability and better performance. Figure 20. ProLiant m710 Server Cartridge and ProLiant SL4540 Server network configuration

The target service configuration file at ProLiant SL4540 Server is configured in such a way that one tgtd service runs on one of the port and serves all the client request from one subnet. And another process of tgtd runs on the other port and serves all the client request from different subnet as indicated in figure 21. Only one logical volume is exported through both the subnets via multipath. For example: /dev/vg1/Ld1 is a logical volume on ProLiant SL4540 Server, which has been exported to initiator IPs (192.168.1.10 & 192.168.2.10) on different subnets but from a single ProLiant m710 Server Cartridge node.

18

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 21. ProLiant SL4540 Server iSCSI target service configuration layout

Scope We have included the results of iSCSI performance results for configurations that demonstrate the capabilities of ProLiant Server m710 Server Cartridge with ProLiant SL4540 Server and HP 3PAR StoreServ 7400 Storage. We also comment on other configurations and their impact on the performance under the section “considerations and observations” at the end of each single-client and multi-client configurations. • Benchmark: FIO • Test RAID0 with different iSCSI configurations • Logical volumes distributed across multiple disks • Single client vs. multiple clients up to 15 nodes tested for scaling • Test I/Os on the RAW disks with no file system mounted

Tested iSCSI configurations iSCSI configuration with SAS disks from ProLiant SL4540 Server Single-client iSCSI initiator configuration In this configuration, a single logical volume has been created from the SAS disks available on ProLiant SL4540 Server. Later, this logical volume is presented to one of the ProLiant m710 Server Cartridge node, which has iSCSI initiator running on it. Figure 22. Single client—4 KiB I/O patterns on a logical volume through iSCSI from ProLiant SL4540 Server

19

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 23. Single client—8 KiB I/O patterns on a logical volume through iSCSI from ProLiant SL4540 Server

Multi-node iSCSI initiator configuration In this configuration, multiple iSCSI clients concurrently participate in read-write operations on the respective iSCSI logical volumes exported on them. These logical volumes are exported from ProLiant SL4540 Server to ProLiant m710 Server Cartridge nodes. An average result per node has been showed graphically in the following figures. Figure 24. 5 nodes average—4 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

20

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 25. 5 nodes average—8 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Figure 26. 10 nodes average—4 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

21

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 27. 10 nodes average—8 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Figure 28. 15 nodes average—4 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

22

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 29. 15 nodes average—8 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Considerations and observations • In all the configurations, the iodepth is 32, where direct I/O parameter was used. • The multiple clients and initiator service request is served by the multiple tgtd service running on the target at different

ports (In this context—ProLiant SL4540 Server). • The 4 KiB I/O pattern attains a high-performance value for all types of I/O pattern compared to 8 KiB. • The multi-node result indicates the ProLiant m710 Server Cartridge node average. • The lower value of random I/O operations in multi-node scenarios indicates that there is an obstacle due to lesser disks

used in the read-write operations. When large number of disks are used, more spindles participated in overall read-write operations resulting in a better performance. • The latency during random operations are almost double, compared to the latency during the sequential operations.

The X-axis scale can be referred for latency. • In multiple iSCSI client environment, larger number of spindles or disks play a significant role and using more can boost

the bandwidth from each iSCSI clients.

iSCSI I/O trend for sequential operation In this configuration, sequential read-write pattern has been demonstrated to understand the bottleneck of the system. The CPU trend on ProLiant SL4540 Server has also been captured for the same. Figure 30. Single client—Sequential read with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

23

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 31. Single client—Sequential write with varying block sizes on one logical volume through iSCSI from ProLiant SL4540 Server

Figure 32. 5 nodes average—Sequential read with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

Figure 33. 5 nodes average—Sequential write with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

Figure 34. 10 nodes average—Sequential read with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

24

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 35. 10 nodes average—Sequential write with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

Figure 36. 15 nodes average—Sequential read with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

Figure 37. 15 nodes average—Sequential read with varying block sizes and the corresponding CPU consumption on ProLiant SL4540 Server

Considerations and observations • In all the configurations, an iodepth of 32 with direct I/O parameter have been used. • The IOPS trend indicates that when number of concurrent I/Os increases with multiple clients, the IOPS and

bandwidth decreases while the latency increases. The reason is the number and type of disks participating in the overall read-write operation. • The single client is easily able to scale up to the NICs’ bandwidth limit, which is 10G. However, the bandwidth decreases as

the block size increases. • CPU trend on ProLiant SL4540 Server has been indicated against each sequential I/O trends on ProLiant m710

Server Cartridge. Even when 15 nodes were performing read concurrent read-write, just 15 percent of overall CPU on SL4540 Server was occupied by system and user processes, and the rest 85 percent was idle. • The latency for larger block sizes were observed to be higher.

25

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

iSER configuration with ProLiant SL4540 Server The iSCSI Extensions for RDMA (iSER) is a protocol that extends iSCSI to use the RDMA protocol. RDMA moves the data from the memory of one computer directly into the memory of another computer with minimal involvement from their CPUs. Additional information included in the RDMA protocol allows a system to place the communicated data directly into its final memory destination without any additional or interim data copies. This “zero copy” or “direct data placement” (DDP) capability provides an efficient network communication between systems. Figure 38. iSER data transfer

Configuring RDMA on a Moonshot system The ProLiant m710 Server Cartridge and ProLiant SL4540 Server both use a RDMA aware Mellanox ConnectX-3 NIC. In order to use iSER as a transport layer for iSCSI in this configuration, both these servers, HP Moonshot-45XGc Switch Module and intermediary switches (if any) must be configured for RoCE operation. The latest Mellanox OpenFabrics Enterprise Distribution (OFED) drivers must be installed on the servers and VLANs must be configured on all interfaces used for iSCSI. Ethernet is a best-effort protocol, but RDMA relies on lossless delivery of packets. So, Priority Flow Control (PFC) IEEE 802.1Qbb must also be configured to provide lossless transport for the specific traffic class between the client, server, as well as through any intermediary switches.

Overview of iSER iSER is an extension of the data transfer model of iSCSI, a storage networking standard for TCP/IP. It uses the iSCSI components while taking the advantage of the RDMA protocol suite. What are the iSER advantages? iSER uses the RDMA protocol suite to supply higher bandwidth for block storage transfers (zero-time copy behavior). To that fact, it eliminates the TCP/IP processing overhead while preserving the compatibility with iSCSI protocol. What block storage packages support iSER? 1. Linux SCSI target framework (tgt) 2. Linux I/O target (LIO) 3. Generic SCSI target subsystem for Linux (SCST)—not covered in this document Table 1. iSER supported packages and their space

Kernel/user space

26

tgt

LIO

SCST

User space

Kernel space (Inbox)

Kernel space (out-of-the-box)

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Basic setup • For a basic setup we can have two servers back to back, or via a switch in the middle • IP connectivity between client and target server • Consider enlarging your Ethernet Maximum Transmission Unit (MTU) sizes to 4200. For large I/Os refer to

MTU considerations for RoCE-based applications. • Make sure RoCE is running before running tgt (enabled with iSER). Refer to How to setup RoCE connection

using Inbox driver (RHEL, Ubuntu). Network setup The connectivity between the iSCSI client and target should be on lossless network required for RoCE. PFC (802.1Qbb) or global pause (802.3x) should be enabled across the network.

How to configure tgt (enabled with iSER) transport for RHEL 1. MLNX_OFED installations install all relevant modules by default to enable iSER. 2. For servers without MLNX_OFED, you need to install several group modules using the commands: # yum groupinstall “InfiniBand Support” 3. 4.

The only configuration change between regular iSCSI and iSER is a simple flag. More information on the commands can be found in the documentation of the respective OSs. Basic management utility is “iscsiadm”, which is part of iSCSI package on any Linux distribution, and is capable of configuring iSER.

iSER target (server) configuration Server configuration is generic and fairly simple. In the following link, you can find example for RHEL, Fedora, or CentOS. switchroot.com/how-to-configure-a-iscsi-target-iscsi-server-on-rhelfedora-or-centos Server configuration example: # yum install -y scsi-target-utils # tgtd # tgt-setup-lun -n tg0 -d /tmp/null -b null -t iser Creating new target (name=iqn.2001-04.com.my-server-tg0, tid=1) Adding a logical unit (/tmp/null) to target, tid=1 Setting backing store type: rdwr Accepting connections from all initiators iSER initiator (client) configuration Client configuration (initiator) is generic and fairly simple as well. In the following link, you can find example for RHEL, Fedora, or CentOS. switchroot.com/how-to-configure-a-iscsi-initiator-iscsi-client-on-rhelfedora-or-centos Client configuration example: # yum install -y lsscsi fio iscsi-initiator-utils # modprobe ib_iser #iscsiadm -m discovery --op=new --op=delete --type sendtargets --portal 11.11.10.1:3260 -I iser 11.11.10.1:3260,1 iqn.2001-04.com.reg-r-vrt-010-tg0 # iscsiadm -m node –l Logging in to [iface: iser, target: iqn.2001-04.com.reg-r-vrt-010-tg0, portal: 11.11.1.10,3260] (multiple) Login to [iface: iser, target: iqn.2001-04.com.reg-r-vrt-010-tg0, portal: 11.11.1.10,3260] successful. # fio --rw=randread --bs=64k --numjobs=4 --iodepth=8 --runtime=30 --time_based -loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 -randrepeat=1 --norandommap --exitall --name task1 --filename=/dev/sdb

27

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

sync_on_close=1 --randrepeat=1 --norandommap --exitall --name task1 -filename=/dev/sdb task1: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8 ... fio-2.1.7 Starting 4 processes Jobs: 4 (f=4): [rrrr] [100.0% done] [4304MB/0KB/0KB /s] [68.9K/0/0 iops] [eta 00m:00s

How to configure tgt (enabled with iSER) transport for Ubuntu 1. 2.

MLNX_OFED installations install by default all relevant modules to enable iSER. For servers without MLNX_OFED, you need to install several modules using the command: # apt-get install libmlx4-1 tgt

3.

For debug purposes install: # apt-get install infiniband-diags ibutils ibverbs-utils rdmacm-utils perftest

4.

Make sure the following modules are loaded in the kernel (use “lsmod” command): a. mlx4_core b. mlx4_ib c. mlx4_en d. ib_core e. ib_addr f. ib_sa g. ib_cm h. rdma_cm i. rdma_ucm j. ib_iser k. ib_isert

5. 6.

The basic management utility is “iscsiadm”, which is part of iSCSI package on any Linux distribution, and is capable of configuring iSER. The only configuration change between regular iSCSI and iSER is a simple flag. More information on the commands can be found in the documentation of the respective OSs.

iSER target (server) configuration Server configuration is generic and fairly simple. In the following link, you can find example for Ubuntu server-world.info/en/note?os=Ubuntu_12.04&p=iscsi Server configuration example: # apt-get install infiniband-diags ibutils ibverbs-utils rdmacm-utils perftest libmlx4-1 tgt # tgtd # tgt-setup-lun -n tgt-2 -d /tmp/null -b null -t iser To verify that the target exists run # tgtadm -m target -o show | grep -i target Target 1: iqn.2001-04.com.reg-r-vrt-003.tgtiSER initiator (client) configuration Client configuration (initiator) is generic and fairly simple as well. In the following links, you can find examples for Ubuntu 14.04 help.ubuntu.com/14.04/serverguide/iscsi-initiator.html server-world.info/en/note?os=Ubuntu_12.04&p=iscsi&f=2 28

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Client configuration example: # apt-get install infiniband-diags ibutils ibverbs-utils rdmacm-utils perftest libmlx4-1 open-iscsi-utils open-iscsi # modprobe ib_iser # iscsiadm -m discovery -t st -p 11.11.11.3:3260 11.11.11.3:3260,1 iqn.200104.com.reg-r-vrt-003-tgt-2 # iscsiadm -m node 11.11.11.3:3260,1 iqn.2001-04.com.reg-r-vrt-003-tgt-2 # iscsiadm -m node -T iqn.2001-04.com.reg-r-vrt-003-tgt-2 | grep transport iface.transport_name = tcp # iscsiadm -m node -T iqn.2001-04.com.reg-r-vrt-003-tgt-2 -o update -n iface.transport_name -v iser # iscsiadm -m node -T iqn.2001-04.com.reg-r-vrt-003-tgt-2 | grep transport iface.transport_name = iser # iscsiadm -m node –l Logging in to [iface: iser, target: iqn.2001-04.com.reg-r-vrt-003-tgt-2, portal: 11.11.11.3,3260] (multiple) Login to [iface: iser, target: iqn.2001-04.com.reg-r-vrt-003-tgt-2, portal: 11.11.11.3,3260] successful. To verify disk run the following command: # fdisk -l

How to configure LIO (enabled with iSER) for Ubuntu 14.04 using Inbox driver iSER target (server) configuration LIO server configuration example: 1. Run the “targetcli” command to enter the LIO CLI console: # targetcli targetcli shell version 2.1.fb34 Copyright 2011-2013 by Datera, Inc. and others. For help on commands, type “help” /> 2. Create RAM disk named test1: /> backstores/rd_mcp create name=test1 size=100M Generating a wwn serial Created rd_mcp ramdisk test1 with size 100M /> 3.

Create iSCSI target: /> iSCSI/ create wwn=iqn.2003-01.org.setup.lun.test Created target iqn.2003-01.org.setup.lun.test Selected TPG Tag 1 Successfully created TPG 1 /> 4. Create LUN: Make sure that the RAM disk is activated when the LUN is created /> /iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/luns create /backstores/rd_mcp/test1 Selected LUN 0 Successfully created LUN 0 /> 5. Create portal: Make sure to supply the IP address of the interface you wish to run iSER on top (RoCE enabled) /> iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/portals create 11.11.1.11 Using default IP port 3260 Successfully created network portal 11.11.1.11:3260 />

29

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

6. Enable iSER on this portal: /> iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/portals/11.11.1.11:3260 iser_enable iser operation has been enabled /> 7. Allow access from any server (change ACL): />iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 Parameter demo_mode_write_protect is now '0’ Parameter authentication is now '0' Parameter generate_node_acls is now '1' Parameter cache_dynamic_acls is now '1' /> 8. Save configuration and exit: /> saveconfig Last 10 configs saved in /etc/target/backup Configuration saved to /etc/target/saveconfig.conf /> exit Global pref auto_save_on_exit=true Last 10 configs saved in /etc/target/backup Configuration saved to /etc/target/saveconfig.conf iSER initiator (client) configuration Client configuration example: 1. Make sure that “ib_iser” module is enabled: # lsmod | grep iser ib_iser 47000 2 libiscsi 57278 1 ib_iser scsi_transport_iscsi 99754 3 ib_iser,libiscsi rdma_cm 43464 3 ib_iser,rdma_krping,rdma_ucm ib_core 87335 13 rdma_cm, ib_cm,ib_sa, iw_cm,mlx4_ib, ib_mad, ib_ucm ib_iser, ib_umad, rdma_krping, ib_uverbs, rdma_ucm,ib_ipoi To enable it (in case, it is not enabled) run: # modprobe ib_iser 2. Run “iscsiadm” to discover the target and change the transport mode to iSER # iscsiadm -m discovery -t st -p 11.11.1.11:3260 11.11.1.11:3260,l iqn.2003-01.org.setup.lun.test 11.11.1.11:3260,1 iqn.2003-01.org.setup.lun.test # iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -o update -n iface.transport_name -v iser 3. Perform login and try to read-write from the target (for example, use “dd” command) # iscsiadm -m node –l Logging in to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 11.11.1.11,3260] (multiple) Login to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 11.11.1.11,3260] successful. # lsscsi [0:0:0:0] disk ATA WDC WD1003FBYX-0 01.0 /dev/sda [7:0:0:0] disk LIO-ORG test1 4.0 /dev/sdb [root@reg-r-vrt-002 ~]# dd if=/dev/sdb of=/dev/null bs=64k iflag=direct 1600+0 records in 1600+0 records out 104857600 bytes (105 MB) copied, 0.0782739 s, 1.3 GB/s 4. Run “fio” command (or similar) to test the link bandwidth # fio --rw=read --bs=128k --numjobs=1 --iodepth=4 --size="200m" --loops=1 -ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 -norandommap --exitall --name "test1" --filename="/dev/sdb" -output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305 Jobs: 1 (f=1): [R] [1.3% done] [4356MB/0KB/0KB /s] [34.6K/0/0 iops] [eta 05m:02s

30

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

How to configure LIO (enabled with iSER) for RHEL 7 using Inbox driver iSER target (server) configuration LIO server configuration example: 1. Run the “targetcli” command to enter the LIO CLI console: # targetcli targetcli shell version 2.1.fb34 Copyright 2011-2013 by Datera, Inc. and others. For help on commands, type 'help'. /> 2.

Create RAM disk named test1: /> backstores/ramdisk/ create name=test1 nullio=true size=100M Created ramdisk test1 with size 100M. />

3.

Create iSCSI target: /> iscsi/ create wwn=iqn.2003-01.org.setup.lun.test Created target iqn.2003-01.org.setup.lun.test. Created TPG 1. />

4. Create LUN. Make sure that the RAM disk is activated when the LUN is created. /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/luns create /backstores/ramdisk/test1 Created LUN 0. /> 5.

Create portal. Make sure to supply the IP address of the (RoCE enabled) interface you wish to run iSER on. /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/portals create 12.12.12.1 Using default IP port 3260 Created network portal 12.12.12.1:3260. />

6.

Enable iSER on this portal: /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/portals/12.12.12.1:3260 enable_iser boolean=true iSER enable now: True />

7.

Allow access from any server (change ACL): /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 Parameter demo_mode_write_protect is now '0'. Parameter authentication is now '0'. Parameter generate_node_acls is now '1'. Parameter cache_dynamic_acls is now '1'. />

8.

Save configuration and exit: /> saveconfig Last 10 configs saved in /etc/target/backup. Configuration saved to /etc/target/saveconfig.json /> exit Global pref auto_save_on_exit=true Last 10 configs saved in /etc/target/backup. Configuration saved to /etc/target/saveconfig.json

31

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

iSER initiator (client) configuration Client configuration example: 1. Make sure that “ib_iser” module is enabled: # lsmod | grep iser ib_iser 47000 2 libiscsi 57278 1 ib_iser scsi_transport_iscsi 99754 3 ib_iser,libiscsi rdma_cm 43464 3 ib_iser,rdma_krping,rdma_ucm ib_core 87335 13 rdma_cm, ib_cm,ib_sa, iw_cm,mlx4_ib, ib_mad, ib_ucm, ib_iser, ib_umad, rdma_krping, ib_uverbs, rdma_ucm,ib_ipoib To enable it (in case it is not enabled) run: # modprobe ib_iser

32

2.

Run “iscsiadm” to discover the target and change the transport mode to iSER # iscsiadm -m discovery -t st -p 12.12.12.1:3260 12.12.12.1:3260,1 iqn.2003-01.org.setup.lun.test 12.12.12.1:3260,1 iqn.2003-01.org.setup.lun.test # iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -o update -n iface.transport_name -v iser

3.

Perform Login and try to read/write from the target (e.g. use “dd” command) # iscsiadm -m node –l Logging in to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 12.12.12.1,3260] (multiple) Login to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 12.12.12.1,3260] successful. # lsscsi [0:0:0:0] disk ATA WDC WD1003FBYX-0 01.0 /dev/sda [7:0:0:0] disk LIO-ORG test1 4.0 /dev/sdb [root@reg-r-vrt-002 ~]# dd if=/dev/sdb of=/dev/null bs=64k iflag=direct 1600+0 records in 1600+0 records out 104857600 bytes (105 MB) copied, 0.0782739 s, 1.3 GB/s

4.

Run “fio” command (or similar) to test the link bandwidth # fio --rw=read --bs=128k --numjobs=1 --iodepth=4 --size="200m" --loops=1 -ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 -norandommap --exitall --name "test1" --filename="/dev/sdb" -output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305 Jobs: 1 (f=1): [R] [1.3% done] [4356MB/0KB/0KB /s] [34.6K/0/0 iops] [eta 05m:02s]

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Single-client iSCSI initiator configuration In this configuration, a single logical volume has been created from the disks available on ProLiant SL4540 Server. This logical volume is presented to a ProLiant m710 Server Cartridge through iSCSI. This configuration is similar to the normal iSCSI configuration except transport layer specified during discovery is iSER. The transport protocol used is iSER. Figure 39. Single client—4 KiB I/O patterns on one logical volume through iSCSI from ProLiant SL4540 Server with iSER transport

Figure 40. Single client—8 KiB I/O patterns on one logical volume through iSCSI from ProLiant SL4540 Server with iSER transport

33

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Multi-nodes iSCSI initiator configuration In this configuration multiple iSCSI clients concurrently participate in the read-write operations on the iSCSI logical volumes exported from ProLiant SL4540 Server to ProLiant m710 Server Cartridge. An average result per node is shown graphically in the following figures. Figure 41. 5 nodes average—4 KiB I/O pattern on a logical volume presented to the ProLiant m710 Server Cartridge node through iSCSI from ProLiant SL4540 Server with iSER transport

Figure 42. 5 nodes average—8 KiB I/O pattern on a logical volume presented to the ProLiant m710 Server Cartridge node through iSCSI from ProLiant SL4540 Server with iSER transport

34

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 43. 10 nodes average—4 KiB I/O pattern on a logical volume presented to ProLiant m710 Server Cartridge node through iSCSI from ProLiant SL4540 Server with iSER transport

Figure 44. 10 nodes average—8 KiB I/O pattern on a logical volume presented to the ProLiant m710 Server Cartridge node through iSCSI from ProLiant SL4540 Server with iSER transport

35

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 45. 15 nodes average—4 KiB I/O pattern on a logical volume presented to the ProLiant m710 Server Cartridge node through iSCSI from ProLiant SL4540 Server with iSER transport

Figure 46. 15 nodes average—8 KiB I/O pattern on a logical volume presented to the ProLiant m710 Server Cartridge node through iSCSI from ProLiant SL4540 Server with iSER transport

Considerations and observations • The discovery process of the initiator is similar to iSCSI except a switch specified during the discovery to specify the transport layer. For example: iscsiadm -m discoverydb --op=new --op=delete --type sendtargets --portal 10.10.11.101:3260 --discover -I iser Where: -I iser specifies the network layer interface name and –portal is target IP and port. • As indicated in the graphs (Figures 39 and 40), all single-client sequential operation is easily able to saturate the

NIC bandwidth limit which is 10G. • Random operations results are still limited due to number of disks and spindles participating in read-write operation.

36

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

• iSER uses memory to transfer the data and avoids any intermediate unnecessary data copying on the target and initiator. • With reference to the results discussed earlier in this section, we can derive that the performance obtained with iSER

transport is 50 percent higher on average when compared to normal iSCSI results. However, this is not the limit. It could go higher when more number of disks and larger memory is used.

I/O trend for sequential operation In this configuration, sequential read-write pattern have been demonstrated to understand the bottleneck of the systems. Figure 47. Single client—Sequential read with varying block sizes with iSER transport on a logical volume through iSCSI from ProLiant SL4540 Server

Figure 48. Single client—Sequential write with varying block sizes with iSER transport on a logical volume through iSCSI from ProLiant SL4540 Server

37

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 49. Single client—Memory consumption trend on ProLiant SL4540 Server, when a single ProLiant m710 Server Cartridge node performing sequential read-write operation

Considerations and observations • Figures 49 indicates memory consumption on ProLiant SL4540 Server during sequential I/O of different block sizes from a single-node ProLiant m710 Server Cartridge using iSER transport. • iSER uses memory to transfer the data from initiator to the client. The memory buffers participate in the

I/O transfers and get filled up as soon as read-write is initiated. • All block sizes are able to saturate the ProLiant m710 Server Cartridge NIC bandwidth limit (10 Gbps) for any

read-write operation. • The memory consumption trend would vary based on the available memory buffers on the server and size of

the data used by the client for read-write operation. • On an average, the above results obtained with iSER transport layer is 50 percent higher compared to normal

iSCSI results. It can go further higher if number of disks and larger memory is used at initiator and target. • The buffer can be flushed or overwritten whenever needed using following command:

For example: echo 3| sudo tee /proc/sys/vm/drop_caches • In all the configurations, an iodepth = 32, where direct I/O parameter was used. The storage devices are

SAS disks with 15k rpm.

38

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

iSCSI configuration with SSD (SATA) disks from HP ProLiant SL4540 Server Single-client iSCSI initiator configuration In this configuration, a single 200G logical volume was created from the 16 SSD (SATA) disks available on ProLiant SL4540 Server. Later, this logical volume was presented to one of the ProLiant m710 Server Cartridge node on which the iSCSI initiator was running. Figure 50. Single client—4 KiB I/O patterns on a logical volume through iSCSI from ProLiant SL4540 Server

Figure 51. Single client—4 KiB I/O patterns on a logical volume through iSCSI from ProLiant SL4540 Server

39

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Multi-nodes iSCSI initiator configuration In this configuration multiple iSCSI clients concurrently participate in read-write operations on the respective iSCSI logical volumes exported on them. There are 15 x 200G logical volumes which are exported from ProLiant SL4540 Server to the ProLiant m710 Server Cartridge nodes. An average result per node has been shown graphically in the following figures. Figure 52. 5 nodes average—4 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Figure 53. 5 nodes average—8 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

40

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 54. 10 nodes average—4 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Figure 55. 10 nodes average—8 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

41

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 56. 15 nodes average—4 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Figure 57. 15 nodes average—8 KiB I/O pattern on a logical volume presented through iSCSI from ProLiant SL4540 Server

Considerations and observations • The IOPS trend indicates that when number of concurrent I/Os increases with multiple clients, the IOPS and bandwidth decreases while latency increases. The reason is the number and type of disks participating in the overall read-write operation. • The single client is easily able to scale up to the NICs’ bandwidth limit. However, it decreases as the block size

changes. Smaller block of data are able to achieve higher IOPS and bandwidth. • CPU trend also indicates against each of sequential I/O trends. Even when 15 nodes are performing read concurrent

read/write, just 15 percent of overall CPU is occupied by system and user processes, rest 85 percent is idle. • The latency for larger block sizes are observed higher with smaller IOPS and bandwidth.

42

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

iSCSI I/O trend for sequential operation In this configuration, sequential read-write pattern have been demonstrated to understand the bottleneck of the systems. Figure 58. Single client—Sequential read with varying block sizes on a logical volume (SSD) through iSCSI from ProLiant SL4540 Server

Figure 59. Single client—Sequential write with varying block sizes on a logical volume (SSD) through iSCSI from ProLiant SL4540 Server

43

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 60. Single client—CPU trend sequential read with varying block sizes on a logical volume through iSCSI from ProLiant SL4540 Server

Figure 61. Single client—CPU trend sequential write with varying block sizes on a logical volume through iSCSI from ProLiant SL4540 Server

Considerations and observations • The single client is easily able to scale up to the NICs’ bandwidth limit—10G. However, the bandwidth decreases as the block size increases. Smaller block of data are able to achieve higher IOPS and bandwidth. • The latency for larger block sizes are observed to be higher with smaller IOPS and bandwidth. • The CPU trend indicates, during 4 KiB read, system processes consumed around 56 percent of overall CPU on the

ProLiant m710 Server Cartridge node, although user process consumed 14 percent of overall CPU. • Some of the best results for IOPS and bandwidth are observed at 4k, with an iodepth of 32.

44

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

iSCSI configuration with HP 3PAR StoreServ 7400 Storage Single-client iSCSI initiator configuration In this configuration, a single logical volume was created from the disks available on HP 3PAR 7400 Storage. Later, the same logical volume was presented to one iSCSI initiator running on an HP ProLiant m710 Server Cartridge. The HP 3PAR StoreServ 7400 Storage in consideration has 16 x 185 GB SSD (SAS) physical disks. Figure 62. Single client—4 KiB I/O patterns on one logical volume through iSCSI from HP 3PAR StoreServ 7400 Storage

Figure 63. Single client—8 KiB I/O patterns on one logical volume through iSCSI from HP 3PAR StoreServ 7400 Storage

45

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Multi-nodes iSCSI initiator configuration In this configuration, multiple iSCSI clients concurrently participate in the read-write operations on the iSCSI logical volumes exported from HP 3PAR 7400 Storage to the ProLiant m710 Server Cartridge node. An average result per node has been shown graphically in the following figures. Figure 64. 5 nodes average—4 KiB I/O pattern on logical volume presented through iSCSI from HP 3PAR StoreServ 7400 Storage

Figure 65. 5 nodes average—8 KiB I/O pattern on logical volume presented through iSCSI from HP 3PAR StoreServ 7400 Storage

46

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 66. 10 nodes average—4 KiB I/O pattern on logical volume presented through iSCSI from HP 3PAR StoreServ 7400 Storage

Figure 67. 10 nodes average—8 KiB I/O pattern on logical volume presented through iSCSI from HP 3PAR StoreServ 7400 Storage

47

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 68. 15 nodes average—4 KiB I/O pattern on logical volume presented through iSCSI from HP 3PAR StoreServ 7400 Storage

Figure 69. 15 nodes average—8 KiB I/O pattern on logical volume presented through iSCSI from HP 3PAR StoreServ 7400 Storage

Considerations and observations • Though the overall size of the total SSD disks are not large, thinly provisioned logical volumes have been created and presented to the ProLiant m710 Server Cartridge nodes. • Most of the sequential operations performed on the single node are able to exhaust 10G NIC bandwidth on ProLiant

m710 Server Cartridge. • The smaller bandwidth obtained on multi-node scenarios is due to smaller number of disks used. If larger set of disks

participate in the read-write operation, the performance would be higher. • For all the results obtained, the sequential write operation on the HP 3PAR StoreServ 7400 Storage SSDs are just

50 percent of the read operations. • Some of the best results for IOPS and bandwidth has been observed at 4k, with an iodepth of 32. • The latency during any random read-write operations have been observed higher than sequential I/Os.

48

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

iSCSI I/O trend for sequential operation In this configuration, sequential read-write patterns have been demonstrated to understand the bottleneck of the systems. Figure 70. Single client—Sequential read with varying block sizes on a logical volume through iSCSI from HP 3PAR StoreServ 7400 Storage

Figure 71. Single client—Sequential write with varying block sizes on a logical volume through iSCSI from HP 3PAR StoreServ 7400 Storage

49

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Figure 72. Single client—CPU trend sequential read with varying block sizes on a logical volume through iSCSI from HP 3PAR StoreServ 7400 Storage

Figure 73. Single client—CPU trend sequential write with varying block sizes on a logical volume through iSCSI from HP 3PAR StoreServ 7400 Storage

Considerations and observations • Both sequential read-write with 4 KiB is able to saturate the 10G NIC bandwidth on ProLiant m710 Server Cartridge. • For similar load and pattern, any read performance is better than write performance. • The CPU trend indicates, during 4 KiB read, system processes consume around 55 percent of overall CPU on the

ProLiant m710 Server Cartridge node, although user process consume 24 percent of overall CPU only. • The latency for larger block sizes are observed higher with smaller IOPS and bandwidth. • Some of the best results for IOPS and bandwidth has been observed at 4k, with an iodepth of 32.

50

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Comparisons 1.

1–15 nodes I/O pattern with 4 KiB block size on SAS disks exported through iSCSI from HP ProLiant SL4540 Server.

Figure 74. Comparison with 4 KiB block size on SAS disks exported through iSCSI from ProLiant SL4540 Server

51

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

2.

1–15 nodes I/O pattern with 8 KiB block size on SAS disks exported through iSCSI from ProLiant SL4540 Server.

Figure 75. Comparison with 8 KiB block size on SAS disks exported through iSCSI from ProLiant SL4540 Server

52

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

3.

1–15 Nodes I/O pattern with 4 KiB block size on SAS disks exported through iSER from ProLiant SL4540 Server.

Figure 76. Comparison with 4 KiB block size on SAS disks exported through iSER from ProLiant SL4540 Server

53

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

4.

1–15 nodes I/O pattern with 8 KiB block size on SAS disks exported through iSER from ProLiant SL4540 Server.

Figure 77. Comparison with 8 KiB block size on SAS disks exported through iSER from ProLiant SL4540 Server

54

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

5.

1–15 Nodes I/O pattern with 4 KiB block size on SSDs exported through iSCSI from ProLiant SL4540 Server.

Figure 78. Comparison with 4 KiB block size on SSDs exported through iSCSI from ProLiant SL4540 Server

55

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

6.

1–15 nodes I/O pattern with 8 KiB block size on SSDs exported through iSCSI from ProLiant SL4540 Server.

Figure 79. Comparison with 8 KiB block size on SSDs exported through iSCSI from ProLiant SL4540 Server

56

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

7. 1–15 nodes I/O pattern with 4 KiB block size on SSDs exported through iSCSI from HP 3PAR StoreServ 7400 Storage Figure 80. Comparison with 4 KiB block size on SSDs exported through iSCSI from HP 3PAR StoreServ 7400 Storage

57

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

8. 1–15 nodes I/O pattern with 8 KiB block size on SSDs exported through iSCSI from HP 3PAR StoreServ 7400 Storage. Figure 81. Comparison with 8 KiB block size on SSDs exported through iSCSI from HP 3PAR StoreServ 7400 Storage

58

Technical white paper | External storage solution for HP ProLiant Moonshot Servers via iSCSI

Summary (Section 2) • The data captured in this document demonstrates iSCSI connectivity as an excellent storage solution for HP ProLiant

m700 Server Cartridges and HP ProLiant m710 Server Cartridges. • Workloads that need more storage than available on the Moonshot cartridge, can use iSCSI solution to meet their

storage requirement. iSCSI has been a proven external storage solution. • The performance data obtained with iSER has been better than traditional iSCSI with HP ProLiant SL4540 Server. • HP 3PAR StoreServ Storage (having SSDs) with HP ProLiant Moonshot Servers are a recommended combination and

solution for quick response workloads. • Workloads that expect maximum bandwidth with minimal CPU involvement during an I/O has been recommended,

an iSER storage solution from ProLiant m710 Server Cartridges with ProLiant SL4540 Server. • Workloads expecting high IOPS with only few disks for an I/O has been recommended SSD storage solution from

ProLiant m710 Server Cartridge with ProLiant SL4540 Server with SSD. • The various iSCSI/iSER storage configurations discussed in this document can be altered, configured and used subject to

different workloads and actual application needs.

References HP Moonshot System h17007.www1.hp.com/us/en/enterprise/servers/products/moonshot/index.aspx HP ProLiant m700 Server Cartridge www8.hp.com/in/en/products/proliant-servers/product-detail.html?oid=6488207

Learn more at hp.com/go/3PAR

Sign up for updates hp.com/go/getupdated

Share with colleagues

Rate this document

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. AMD is a trademark of Advanced Micro Devices, Inc. Intel Xeon is the trademark of Intel Corporation in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Microsoft, Windows, Windows 7, and Windows Server are trademarks of the Microsoft Group of companies. Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries. 4AA5-5656ENW, December 2014