SPEC releases new cloud benchmark

SPEC Cloud IaaS 2018 benchmark uses I/O- and CPU-intensive workloads to test provisioning and runtime aspects of cloud implementations

GAINESVILLE, Va., December 18, 2018 - The Standard Performance Evaluation Corp. (SPEC) has released SPEC Cloud IaaS 2018, new benchmark software that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations.

The SPEC Cloud IaaS 2018 benchmark measures replicated application instances, overall performance, relative scalability, and mean instance provisioning time of IaaS cloud implementations, both public and private. It replaces the 2016 version, which will be retired in three months.

Cloud providers, users of cloud services, vendors of cloud-related hardware and software, and researchers can use the benchmark to evaluate how different implementations affect cloud performance.

SPEC members participating in the cloud subcommittee include Dell, Digital Ocean, IBM, Intel, Oracle, Red Hat and VMware. Long-time SPEC benchmark developer Yun-seng Chao is a supporting contributor.

Flexible, representative workloads

Two multi-instance workloads are included within the SPEC Cloud IaaS framework:

The Yahoo! Cloud Serving Benchmark (YCSB) that uses the Cassandra NoSQL database to store and retrieve data in a manner representative of social media applications.
Big data analytics based on a K-means clustering workload using Hadoop MapReduce.

The workloads are designed to test compute, storage and network performance of an IaaS cloud subject to strict quality of service (QoS) metrics. Each workload runs as a distributed application comprising six or seven cloud instances that make up an application instance. Adding new application instances increases the stress on the cloud infrastructure's instance-provisioning capabilities and on its available resources, including CPU, memory, disk and network.

The benchmark supports multi-tenancy and provides the flexibility to use a physical machine, a virtual machine, or a container as an instance type.

"SPEC Cloud IaaS 2018 incorporates improved methodologies that help ensure fairness, reliability and real-world rigor," says Paula Smith, chair of the SPEC Cloud subcommittee. "It provides metrics that go beyond relative performance and capacity measurement to show how typical cloud application workloads behave as underlying resources approach full capacity."

What's new for 2018

SPEC Cloud IaaS 2018 builds on the 2016 release with a variety of updates, including:

Easier installation of CBTOOL and workload images.
Improved support for public clouds, OpenStack, and containers and container managers.
Revised internal methodologies for scoring, reporting and metric calculations.
New versions of Hadoop for K-means and Cassandra for YCSB.
An increased I/O workload that tests the limits of current storage technologies.

Due to the methodology changes, results for SPEC Cloud IaaS 2018 are not comparable to those from the 2016 version.

Available for immediate download

The SPEC Cloud IaaS 2018 benchmark is available immediately for download from the SPEC website (www.spec.org/order.html) for $2,000. A $500 license is available for eligible academic and non-profit organizations. SPEC Cloud IaaS 2016 licensees will be offered a free upgrade to the 2018 version.

More information — including a design overview, user guide, FAQs document, glossary, and run and reporting rules — is available at http://www.spec.org/cloud_iaas2018.

About SPEC

Entering its 30th year, SPEC is a non-profit organization that establishes, maintains and endorses standardized benchmarks and tools to evaluate performance and energy consumption for the newest generation of computing systems. Its membership comprises more than 120 leading computer hardware and software vendors, educational institutions, research organizations, and government agencies worldwide.

Media contact:

Bob Cramblitt
Cramblitt & Company
919-481-4599; info@cramco.com

Company, product and service names mentioned herein may be the trademarks of their respective owners.

Standard Performance Evaluation Corporation