SPECvirt ^® Datacenter 2021

• SPECvirt ^® Datacenter 2021 Design Document
  • 1.0 Introduction
  • 2.0 Overview
    • 2.1 Component workloads
    • 2.2 VMs and tiles
      • 2.2.1 Tile composition
      • 2.2.2 Multiple tiles
      • 2.2.3 Fractional tiles
    • 2.3 Measurement phases
      • 2.3.1 Workload throughputs
    • 2.4 Primary metric
    • 2.5 Measurement Interval (MI) and phases
  • 3.0 Testbed preparation
  • 4.0 Benchmark workflow
  • 5.0 Workload details
    • 5.1 Mail server workload
    • 5.2 Web server workload
    • 5.3 Collaboration server workload
    • 5.4 HammerDB workload
    • 5.5 BigBench workload
      • 5.5.1 Modified query list
  • 6.0 System Under Test (SUT) environment
  • 7.0 Workload controller
    • 7.1 Benchmark harness
    • 7.2 CloudPerf harness design
    • 7.3 CloudPerf and workload polling
      • 7.3.1 Throughput and events (LiveView GUI)
      • 7.3.2 Polling and run time result reporting
  • 8.0 CloudPerf and workload agent interaction
    • 8.1 workload.xml
    • 8.2 deployment.xml
    • 8.3 run.xml
    • 8.4 Invocation of hypervisor management operations
    • 8.5 SUT networking
      • 8.5.1 MAC and IP address assignment for VMs
    • 8.6 The result (raw) file
      • 8.6.1 Editable data
      • 8.6.2 Benchmark configuration data
      • 8.6.3 Benchmark results data
      • 8.6.4 Benchmark workload polling data
      • 8.6.5 Encode data
    • 8.7 The reporter
      • 8.7.1 Raw file editing and regeneration
      • 8.7.2 Submission (sub) file generation
      • 8.7.3 Formatted (html) file generation
    • Appendix A - Workload Parameters (fixed settings)
      • Mailserver
      • Webserver
      • Collaboration Server
      • HammerDB
      • BigBench
    • Appendix B - Example run.xml benchmark test run/control definition (one tile test)
• SPECvirt ^® Datacenter 2021 User Guide
  • 1.0 Introduction
    • 1.1 Workloads
    • 1.2 Run and Reporting Rules
    • 1.3 OVA distribution
  • 2.0 Requirements
    • 2.1 SUT hardware requirements
      • 2.1.1 CPU, memory, and storage requirements
      • 2.1.2 Network requirements
    • 2.2 Client hardware requirements
    • 2.3 Management server requirements
    • 2.4 svdc-director VM requirements
  • 3.0 Prepare the infrastructure
    • 3.1 Configure the datacenter infrastructure
      • 3.1.1 Enable hypervisor SSH access
      • 3.1.2 Configure management server
    • 3.2 Deploy the template to create the svdc-director VM
      • 3.2.1 Download the template from SPEC
      • 3.2.2 Deploy/Import the template
      • 3.2.3 Download the CentOS distribution
      • 3.2.4 Create the svdc-director VM
      • 3.2.5 Configure the svdc-director VM and harness
      • 3.2.7 Set up the GNOME Desktop for LiveView (recommended)
      • 3.2.8 Configure time synchronization
    • 3.3 Create the client VMs
      • 3.3.1 Deploy a client using settings in Control.config
      • 3.3.2 Deploy a client to host and/or local host storage pool
    • 3.4 Provision tiles
      • 3.4.1 Assign storage pools
      • 3.4.2 Assign vCPUs to workload VMs
      • 3.4.3 Use workload VM deployment scripts
      • 3.4.4 Add tiles with the prepSUT-addTile.sh script
      • 3.4.5 Example tile
  • 4.0 Run the benchmark
    • 4.1 Edit the Testbed.config file
    • 4.2 Edit the Control.config file
      • 4.2.1 Set SUT host pre-measurement states
      • 4.2.2 Set number of tiles and partial tiles
      • 4.2.3 Set other configurable parameters (optional, compliant)
    • 4.3 Start a measurement
      • 4.3.1 Run runspecvirt.sh and launch the LiveView GUI
      • 4.3.2 Run a measurement manually with LiveView
      • 4.3.3 Monitor a measurement from the CLI
  • 5.0 View measurement results
    • 5.1 View results files
    • 5.2 Regenerate the HTML report (FDR)
    • 5.3 Regenerate the submission (.sub) file if needed
  • 6.0 Control.config experimental options
    • 6.1 Change measurement durations
    • 6.2 Disable/Enable workloads
    • 6.3 Disable/Enable support file collection
  • Appendix - Control.config
• SPECvirt ^® Datacenter 2021 Run and Reporting Rules
  • 1.0 Introduction
  • 2.0 Philosophy
    • 2.1 Fair Use of SPECvirt Datacenter 2021 Results
    • 2.2 Research and Academic Usage
    • 2.3 SPECvirt Datacenter 2021 Benchmark Configuration
    • 2.4 About SPEC
      • 2.4.1 SPEC May Adapt Suite
      • 2.4.2 Reference Links Disclaimer
      • 2.4.3 Notes on Mitigation of Significant Security Vulnerabilities
  • 3.0 Definitions
  • 4.0 Running the SPECvirt Datacenter 2021 Benchmark
    • 4.1 Environment
      • 4.1.1 Testbed Configuration
      • 4.1.2 System Under Test (SUT)
    • 4.2 Workload VMs
      • 4.2.1 Departmental Workload VMs (Mailserver, Webserver, Collaboration Server)
      • 4.2.2 HammerDB Workload VMs
      • 4.2.2 BigBench Workload VMs
    • 4.3 Client and svdc-director VMs
    • 4.4 Measurement
      • 4.4.1 Quality of Service
      • 4.4.2 Benchmark Parameters
      • 4.4.3 Deploying and Running SPECvirt Datacenter 2021 VMs
      • 4.4.4 SUT Tuning
      • 4.4.5 VM Resource Allocation for SPECvirt Datacenter 2021 Benchmark
      • 4.4.6 Modifications to the SPECvirt Datacenter 2021 Template
  • 5.0 Reporting Results
    • 5.1 Metrics And Reference Format
    • 5.2 Testbed Configuration Disclosure
      • 5.2.1 SUT Hardware
      • 5.2.2 SUT Software
      • 5.2.3 Network Configuration
      • 5.2.4 Clients
      • 5.2.5 General Availability Dates
      • 5.2.6 Test Sponsor
      • 5.2.7 Notes
  • 6.0 Submission Requirements for SPECvirt Datacenter 2021 Results
    • 6.1 SUT Configuration Collection
    • 6.2 Client Configuration Collection
    • 6.3 SUT workload Guest Configuration Collection
    • 6.4 Sub file syntax checker tool
  • 7.0 SPECvirt Datacenter 2021 Benchmark Kit
  • Appendix A. Submitting a New SPECvirt Datacenter 2021 Hypervisor Toolset
    • Table A.1. Primary Scripts and Input Parameters Required for all Toolsets
    • A.1 Format of startRun.config and endRun.Config Files
    • Table A.2 Required Fields for startRun.config and endRun.config Files
  • Appendix B. External Workload References
• SPECvirt ^® Datacenter 2021 FAQ
  • Benchmark overview
    • 1. What is the SPECvirt ^® Datacenter 2021 V1.0 benchmark?
    • 2. How does the benchmark compare to the SPEC VIRT_SC ^® 2013 V1.1 benchmark?
    • 3. What does the benchmark measure?
    • 4. What is the performance metric for the benchmark?
    • 5. Are the management operations included in the metric?
    • 6. Does the benchmark support a single server?
    • 7. How long does it take to run the benchmark?
    • 8. What types of virtualization platforms do the benchmark support?
  • Workloads
    • 9. What is a tile?
    • 10. What is a fractional tile?
    • 11. Can I run the benchmark in the cloud?
    • 12. What are the limitations of the benchmark?
    • 13. Can I use the benchmark to determine the size of the server I need?
    • 14. Can I run other workload levels?
  • Benchmark kit
    • 15. How can I obtain the benchmark?
    • 16. What is included with the benchmark?
    • 17. What hardware is required to run the benchmark?
    • 18. What if I have a problem configuring or running the benchmark?
    • 19. Where are the the run and reporting rules?
    • 20. What control mechanism is used to drive the workloads?
    • 21. What is an SDK?
    • 22. What is a toolset?
    • 23. What skills do I need to run the benchmark?
  • Results submission, review, and result announcement
    • 24. What is a compliant result of the benchmark?
    • 25. When and where are the benchmark results available?
    • 26. Can I report results for an open source hypervisor?
    • 27. Are the results independently audited?
    • 28. How can I submit benchmark results?
    • 29. Can I announce my results before they are reviewed by the SPEC subcommittee?
    • 30. Are results sensitive to components outside of the SUT – e.g. client driver machines?
    • 31. Does the benchmark have a power measurement component?
  • Relationship to other benchmarks
    • 32. Can I compare the results of the SPECvirt Datacenter 2021 workloads to the results of the benchmarks from which they were derived? For example, can I compare an open source HammerDB result to the result of the SPECvirt Datacenter 2021 HammerDB component?
    • 33. Can I compare the benchmark with other virtualization benchmarks?
    • 34. How can I use the benchmark to research performance related to a specific component of the benchmark such as the memory, storage, hypervisor, or the application server VM?
• SPECvirt ^® Datacenter 2021 Known Issues
  • Current Known Issues
• SPECvirt ^® Datacenter 2021 Technical Support
  • On the svdc-director VM
    • Setting the MAC address on vSphere
    • Submitting a measurement manually
    • Encountering a Java connection error
    • Setting up SSH and vSphere
    • Encountering issues with the reporter
  • On client VMs
  • Submissions
  • HammerDB
    • HammerDB creation takes a long time
    • Monitoring HammerDB transactions
  • BigBench
    • Accessing the BigBench console
    • Checking BigBench status
• SPECvirt ^® Datacenter 2021 Credits
  • Development Team
  • Development Support
  • Special Thanks
• Additional SDKs for SPECvirt® Datacenter 2021
• SPECvirt ^® Datacenter 2021 Patches