Evolving Trends in Cloud Market Call for New Benchmarks

By Sundar Iyengar, SPEC Cloud Committee Chair, and Ramesh Illikkal, Committee Member

Over the last few years, the cloud market has grown in its depth and breadth of offerings. From its simple beginnings, when on-premises workloads and applications could be run on instances rented on the cloud, the market has moved to designing cloud-native applications that run on disaggregated hardware.

Software architecture is shifting from monolithic architecture to distributed microservices architecture, increasing workload complexity. The underlying cloud system architecture is becoming more heterogenous and disaggregated, comprising a mix of CPUs, infrastructure processing units and special-purpose accelerators. The confluence of these trends has led to new urgencies for innovations in the underlying infrastructure. Unfortunately, the benchmarks that are being widely used by the industry to drive architecture features and software/hardware co-optimizations have been left behind in this fast-paced transformation. Traditional single-node, CPU-centric benchmarks cannot represent these emerging cloud use cases and industry trends. With the rapid changes in the cloud market, it’s more important than ever to have a system benchmark that represents real-world use cases that exercises relevant architecture features for accessing true measure of goodness of the underlying system architecture.

Therefore, we are thrilled to announce the start of a new benchmark development effort in the SPEC Cloud Committee.

Benchmarking is all about the desire to know what we have today and improve it for the future. To quote Lord Kelvin, after whom the SI unit of temperature is named, "To measure is to know," and "If you cannot measure it, you cannot improve it." As we design products optimized for the new cloud environment, we need a new way to measure what we have so that we can improve customer experiences with better products. The effort kicked off in the SPEC Cloud committee is intended to create this new measure.

The environment is complex, and hence it will be challenging to design a benchmark that addresses all its needs while remaining simple enough to enable wide adoption. We welcome strong participation from SPEC members to help us navigate through the complexity and create this new cloud system benchmark that is representative of the modern cloud environment, is easy to use and allows us to improve future product offerings in this space.

If you have any questions or would like to help create the next SPEC benchmark for the cloud, please contact info@spec.org.

[Back to SPEC blog]