SPEC SFS®2014_vda Result

The first three configuration parameters were set using the "mmchconfig" command on one of the nodes in the cluster. The verbs settings in the table above allow for efficient use of the InfiniBand infrastructure. The settings determine when data are transferred over IP and when they are transferred using the verbs protocol. The InfiniBand traffic went through two switches, item 3 in the Bill of Materials. The block device parameters "scheduler" and "nr_requests" were set on the server nodes with echo commands for each DeepFlash device. The parameters can be found at "/sys/block/DEVICE/queue/{scheduler,nr_requests}", where DEVICE is the block device name. The last parameter disabled Hyper-Threading on the client and server nodes.

The configuration parameters were set using the "mmchconfig" command on one of the nodes in the cluster. The parameters listed in the table above reflect values that might be used in a typical streaming environment with Linux nodes.

There were no opaque services in use.

A single Spectrum Scale file system was created with a 1 MiB block size for data and metadata, a 4 KiB inode size, and a 32 MiB log size, 2 replicas for data and metadata, and "relatime". The file system was spread across all of the Network Shared Disks (NSDs).

The client nodes each had an ext4 file system that hosted the operating system.

Each DeepFlash presented 32 JBOF LUNs to one of the server nodes. An NSD was created from each LUN. All of the NSDs attached to the first server node were placed in a failure group. All of the NSDs attached to the second server node were placed in a second failure group. The file system was configured with 2 data and 2 metadata replicas. Therefore a copy of all data and metadata was present on each DeepFlash 150.

The cluster used a two-tier architecture. The client nodes perform file-level operations. The data requests are transmitted to the server nodes. The server nodes perform the block-level operations. In Spectrum Scale terminology the load generators are NSD clients and the server nodes are NSD servers. The NSDs were the storage devices specified when creating the Spectrum Scale file system.

The 1 GbE network was used for administrative purposes. All benchmark traffic flowed through the Mellanox SX6036 InfiniBand switches. Each client node had two active InfiniBand ports. Each server node had four active InfiniBand ports. Each client node InfiniBand port was on a separate FDR fabric for RDMA connections between nodes.

Each of the Spectrum Scale client nodes had 2 physical processors. Each processor had 6 cores with one thread per core.

In the client nodes Spectrum Scale reserves a portion of the physical memory for file data and metadata caching. In the server nodes a portion of the physical memory is reserved for NSD buffers. A portion of the memory is also reserved for buffers used for node to node communication.