SPEC SFS(R)2014_vda Result IBM Corporation : IBM Spectrum Scale 4.2.1 with Cisco UCS and IBM FlashSystem 900 SPEC SFS2014_vda = 1720 Streams (Overall Response Time = 2.92 msec) =============================================================================== Performance =========== Business Average Metric Latency Streams Streams (Streams) (msec) Ops/Sec MB/Sec ------------ ------------ ------------ ------------ 172 0.8 1720 794 344 1.1 3441 1589 516 1.4 5162 2382 688 1.6 6882 3178 860 1.9 8603 3976 1032 2.1 10324 4772 1204 2.5 12044 5553 1376 2.8 13765 6350 1548 3.3 15486 7141 1720 18.4 17206 7940 =============================================================================== Product and Test Information ============================ +---------------------------------------------------------------+ |IBM Spectrum Scale 4.2.1 with Cisco UCS and IBM FlashSystem 900| +---------------------------------------------------------------+ Tested by IBM Corporation Hardware Available February 2016 Software Available September 2016 Date Tested November 2016 License Number 11 Licensee Locations Raleigh, NC USA IBM Spectrum Scale helps solve the challenge of explosive growth of unstructured data against a flat IT budget. Spectrum Scale provides unified file and object software-defined storage for high performance, large scale workloads on-premises or in the cloud. Spectrum Scale includes the protocols, services and performance required by many industries, Technical Computing, Big Data, HDFS and business critical content repositories. IBM Spectrum Scale provides world-class storage management with extreme scalability, flash accelerated performance, and automatic policy-based storage tiering from flash through disk to tape, reducing storage costs up to 90% while improving security and management efficiency in cloud, big data & analytics environments. Cisco UCS is the first truly unified data center platform that combines industry- standard, x86-architecture servers with networking and storage access into a single system. The system is intelligent infrastructure that is automatically configured through integrated, model-based management to simplify and accelerate deployment of all kinds of applications. The system's x86-architecture rack and blade servers are powered exclusively by Intel(R) Xeon(R) processors and enhanced with Cisco innovations. These innovations include the capability to abstract and automatically configure the server state, built-in virtual interface cards (VICs), and leading memory capacity. Cisco's enterprise-class servers deliver world-record performance to power mission-critical workloads. , Cisco UCS is integrated with a standards-based, high-bandwidth, low-latency, virtualization-aware 10-Gbps unified fabric, with a new generation of Cisco UCS fabric enabling an update to 40 Gbps. Solution Under Test Bill of Materials ===================================== Item No Qty Type Vendor Model/Name Description ---- ---- ---------- ---------- ---------- ----------------------------------- 1 1 Blade Cisco UCS 5108 The Cisco UCS 5108 Blade Server Server Chassis features flexible bay Chassis configurations for blade servers. It can support up to eight half- width blades, up to four full-width blades, or up to two full-width double-height blades in a compact 6 -rack-unit (6RU) form factor. 2 4 Blade Cisco UCS B200 UCS B200 M4 Blade Servers, each Server, M4 with: 2X Intel Xeon processors Spectrum E5-2680 v3 (24 core per node) 256 Scale Node GB of memory 3 2 Fabric Int Cisco UCS Cisco UCS 6300 Series Fabric erconnect 6332-16UP Interconnects support line-rate, lossless 40 Gigabit Ethernet and FCoE connectivity. 4 2 Fabric Cisco UCS 2304 Cisco UCS 2300 Series Fabric Extender Extenders can support up to four 40-Gbps unified fabric uplinks per fabric extender connecting Fabric Interconnect. 5 4 Virtual Cisco UCS VIC The Cisco UCS Virtual Interface Interface 1340 Card (VIC) 1340 is a 2-port, 40 Card Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)-capable modular LAN on motherboard (mLOM) mezzanine adapter. 6 2 FlashSyste IBM 9840-AE2 Each FlashSystem was configured m with 12 2.9TB IBM MicroLatency modules (feature code AF24). Configuration Diagrams ====================== 1) sfs2014-20170130-00017.config1.png (see SPEC SFS2014 results webpage) Component Software ================== Item Name and No Component Type Version Description ---- ------------ ------------ ------------ ----------------------------------- 1 Spectrum Spectrum 4.2.1.1 The Spectrum Scale File System is a Scale Nodes Scale File distributed file system that runs System on the Cisco UCS B200 M4 servers to form a cluster. The cluster allows for the creation and management of single namespace file systems. 2 Spectrum Operating Red Hat The operating system on the Scale Nodes System Enterprise Spectrum Scale nodes was 64-bit Red Linux 7.2 Hat Enterprise Linux version 7.2. for x86_64 3 FlashSystem Storage 1.4.4.2 The FlashSystem software covers all 900 System aspects of administering, configuring, and monitoring the FlashSystem 900. Hardware Configuration and Tuning - Physical ============================================ +----------------------------------------------------------------------+ | Spectrum Scale Nodes | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- numaMemoryInter yes Enables memory interleaving on NUMA leave based systems. multipath queue-length 0 Determines which algorithm to use when device: selecting paths. With this value the path_selector path with the least amount of outstanding I/O is selected. multipath multibus Determines which grouping policy to use device: path_gr for a set of paths. The multibus value ouping_policy causes all paths to be placed in one priority group. Hardware Configuration and Tuning Notes --------------------------------------- The first configuration parameter was set using the "mmchconfig" command on one of the nodes in the cluster. The multipath device parameters were set in the multipath.conf file on each node. A template multipath.conf file for the FlashSystem can be found in the "Implementing IBM FlashSystem 900" Redbook, published by IBM. Software Configuration and Tuning - Physical ============================================ +----------------------------------------------------------------------+ | Spectrum Scale Nodes | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- ignorePrefetchL yes Specifies that only maxMBpS and not the UNCount number of LUNs should be used to dynamically allocate prefetch threads. maxblocksize 1M Specifies the maximum file system block size. maxMBpS 10000 Specifies an estimate of how many megabytes of data can be transferred per second into or out of a single node. maxStatCache 0 Specifies the number of inodes to keep in the stat cache. pagepoolMaxPhys 90 Percentage of physical memory that can MemPct be assigned to the page pool scatterBufferSi 256K Specifies the size of the scatter ze buffers. workerThreads 1024 Controls the maximum number of concurrent file operations at any one instant, as well as the degree of concurrency for flushing dirty data and metadata in the background and for prefetching data and metadata. maxFilesToCache 11M Specifies the number of inodes to cache for recently used files that have been closed. pagepool 96G Specifies the size of the cache on each node. nsdBufSpace 70 Sets the percentage of the pagepool that is used for NSD (Network Shared Disk) buffers. nsdMaxWorkerThr 3072 Sets the maximum number of threads to eads use for block level I/O on the NSDs. nsdMinWorkerThr 3072 Sets the minimum number of threads to eads use for block level I/O on the NSDs. nsdMultiQueue 64 Specifies the maximum number of queues to use for NSD I/O. nsdThreadsPerDi 3 Specifies the maximum number of threads sk to use per NSD. nsdThreadsPerQu 48 Specifies the maximum number of threads eue to use per NSD I/O queue. nsdSmallThreadR 1 Specifies the ratio of small thread atio queues to small thread queues. Software Configuration and Tuning Notes --------------------------------------- The configuration parameters were set using the "mmchconfig" command on one of the nodes in the cluster. Both the nodes used mostly default tuning parameters. A discussion of Spectrum Scale tuning can be found in the official documentation for the mmchconfig command and on the IBM developerWorks wiki. Service SLA Notes ----------------- There were no opaque services in use. Storage and Filesystems ======================= Item Stable No Description Data Protection Storage Qty ---- ------------------------------------- ------------------ -------- ----- 1 FlashSystem 900 volumes, 700 GiB RAID-5 Yes 64 each, used as Network Shared Drives for Spectrum Scale. 2 FlashSystem 900 volumes, 100 GiB RAID-5 Yes 4 each, used to store the operating system of each Spectrum Scale node. Number of Filesystems 1 Total Capacity 44800 GiB Filesystem Type Spectrum Scale File System Filesystem Creation Notes ------------------------- A single Spectrum Scale file system was created with a 1 MiB block size for data and metadata, 4 KiB inode size, and a 128 MiB log size. The file system was spread across all of the Network Shared Disks (NSDs). Each client node mounted the file system. The file system parameters reflect values that might be used in a typical streaming environment. The nodes each had an ext4 file system that hosted the operating system. Storage and Filesystem Notes ---------------------------- Each of the FlashSystem 900 systems had 12 2.9 TiB flash modules. On each system one module was used as a spare and the remaining 11 modules were configured into a RAID-5 array. At that point volumes were created and mapped to the hosts, which were the Spectrum Scale nodes. There were two sets of volumes used in the benchmark. On one of the FlashSystem 900 systems 4 100 GiB volumes were created, and each was mapped to a single host to be used for the node operating system. On each of the 2 FlashSystem 900 systems 32 700 GiB volumes were created, and each volume was mapped to all 4 hosts. These volumes were configured as NSDs by Spectrum Scale and used as storage for the Spectrum Scale file system. The cluster used a single-tier architecture. The Spectrum Scale nodes performed both file and block level operations. Each node had access to all of the NSDs, so any file operation on a node was translated to a block operation and serviced on the same node. Transport Configuration - Physical ================================== Item Number of No Transport Type Ports Used Notes ---- --------------- ---------- ----------------------------------------------- 1 40 GbE cluster 4 Each node connects to a 40 GbE administration network network with MTU=1500 2 16 Gbps SAN 24 There were 16 total connections from storage and 8 total connections from servers. Transport Configuration Notes ----------------------------- Each of the Cisco UCS B200 M4 blade servers comes with a Cisco UCS Virtual Interface Card 1340. The two port card supports 40 GbE and FCoE. To the operating system on the blade servers the card appears as a NIC for Ethernet and as an HBA for fibre channel connectivity. Physically the card connects to the UCS 2304 fabric extenders via internal chassis connections. The eight total ports from the fabric extenders connect to the UCS 6332-16UP fabric interconnects. The two fabric interconnects function as both 16 Gbps FC switches and as 40 Gbps Ethernet switches. Switches - Physical =================== Total Used Item Port Port No Switch Name Switch Type Count Count Notes ---- -------------------- --------------- ------ ----- ------------------------ 1 Cisco UCS 6332-16UP 40 GbE and 16 40 12 The default #1 Gbps FC configuration was used on the switch. 2 Cisco UCS 6332-16UP 40 GbE and 16 40 12 The default #2 Gbps FC configuration was used on the switch. Processing Elements - Physical ============================== Item No Qty Type Location Description Processing Function ---- ---- -------- -------------- ------------------------- ------------------- 1 8 CPU Spectrum Scale Intel Xeon CPU E5-2680 v3 Spectrum Scale client nodes @ 2.50GHz 12-core nodes, load generator, device drivers Processing Element Notes ------------------------ Each of the Spectrum Scale client nodes had 2 physical processors. Each processor had 12 cores with two threads per core. Memory - Physical ================= Size in Number of Description GiB Instances Nonvolatile Total GiB ------------------------- ---------- ---------- ------------ ------------ Spectrum Scale node 256 4 V 1024 system memory Grand Total Memory Gibibytes 1024 Memory Notes ------------ Spectrum Scale reserves a portion of the physical memory in each node for file data and metadata caching. A portion of the memory is also reserved for buffers used for node to node communication. Stable Storage ============== All of the storage used by the benchmark was non-volatile flash storage. Modified writes were not acknowledged as complete until the data was written to the FlashSystem 900s. Each FlashSystem has two battery modules that in the case of a power failure allow the system to remain powered long enough for all of the data in the system's write cache to be committed to the flash modules. Solution Under Test Configuration Notes ======================================= The solution under test was a Spectrum Scale cluster optimized for streaming environments. The Spectrum Scale nodes were also the load generators for the benchmark. The benchmark was executed from one of the nodes. Other Solution Notes ==================== The WARMUP_TIME for the benchmark was 600 seconds. Dataflow ======== The 4 Spectrum Scale nodes were the load generators for the benchmark. Each load generator had access to the single namespace Spectrum Scale file system. The benchmark accessed a single mount point on each load generator. In turn each of mount points corresponded to a single shared base directory in the file system. The nodes process the file operations, and the data requests to and from the backend storage were serviced locally on each node. Block access to each LUN on the nodes was controlled via Linux multipath. Other Notes =========== IBM, IBM Spectrum Scale, IBM FlashSystem, and MicroLatency are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Cisco UCS is a trademark of Cisco in the USA and certain other countries. Intel and Xeon are trademarks of the Intel Corporation in the U.S. and/or other countries. Other Report Notes ================== None =============================================================================== Generated on Wed Mar 13 16:50:35 2019 by SpecReport Copyright (C) 2016-2019 Standard Performance Evaluation Corporation