• A1: Is this a Benchmark? Is the SERT Suite replacing SPECpower_ssj2008?
  • The SERT Suite does not replace SPECpower_ssj2008. The SERT Suite is a standardized benchmark created for the EPA and other global energy efficiency governing bodies to utilize for their server power efficiency programs. Unlike SPECpower_ssj2008, the use of SERT Suite data for marketing purposes is specifically disallowed.
• A2: Can I use the SERT Suite to make numeric comparisons?
  • No. The use of SERT Suite data for numeric comparisons (marketing or otherwise) is not allowed except for research purposes.
• A3: Where can I find the documentation?
  • Documentation is provided with the SERT Suite release in the "Documentation" directory.
• A4: Are updates to documents available?
  • Yes, the most recent documents are on the SPEC website on the SERT Information (http://www.spec.org/sert) web page.
  • Look on the left side of the page in the navigation bar.
• A5: What is a "compliant" run of the SERT Suite?
  • A compliant run meets all the requirements of the SERT Suite run and reporting rules for a valid run.
  • In addition to the run and report rules, several validation and tolerance checks are built-in to the SERT Suite.
• A6: What is a "FDR"?
  • FDR is an acronym for "Full Disclosure Report".
  • The FDR is the document that represents a full run of the SERT Suite
• A7: Can I "turn off" compliance checking? My results are only non-compliant because I do not have a temperature sensor.
  • No. This is to prevent a Full Disclosure report being used that is or might be non-compliant.
• A8: What is a "graduated workload"?
  • A graduated workload operates at several levels, usually stepping up or down by some regular value.
• A9: Why did SPEC choose to implement a graduated workload?
  • Server power consumption has been shown to be significantly different at low load levels versus high load levels, and server loads vary widely from hour to hour and day to day.
  • Since no definitive surveys exist, and all data centers are different, SPEC has chosen to provide graduated loads across a set of load levels to portray performance and power consumption in several reasonable ranges.
• A10: Can I acquire and run the SERT Suite in my own facility?
  • Yes. See "Acquiring the SERT Suite" below.
• A11: Does the SERT Suite support measuring systems with DC-DC power supplies? (DC=Direct Current)
• No.
• DC powered servers can not be compared to AC (Alternating Current) powered servers because the AC to DC power conversion cost (energy loss) is not present in DC powered servers.
• Although physically possible, SPEC has neither developed nor tested methods for DC system measurement.
• A12: Is there a "Users Guide" that walks me through set up, run and results interpretation?
  • Yes, that document is the "SERT Users Guide". See below
• A13: Can I get documentation without purchasing the SERT Suite?
  • Yes, the User Guide and other documents are available on the SPEC web site on the SERT Information (http://www.spec.org/sert) web page.
• A14: What are the key documents I should read before I get started?
  1. SPEC License
  2. SPEC Power and Performance Methodology
  3. The SERT Suite Users Guide
  4. Measurement Setup Guide
  5. The SERT Suite Design Documents
  6. FAQ (this document)
  7. Run and Reporting Rules
• A15: Does my company qualify for reduced or non-profit pricing for the SERT Suite product?
  • Reduced and non-profit pricing is reserved for organizations that qualify under IRS guidelines.
• A16: Does the SERT Suite include the latest version of PTDaemon®?
  • Each SERT Suite release includes the latest PTDaemon version available at that time. Please check to see if a newer PTDaemon is available here: http://www.spec.org/power/docs/SPECpower-Device_List.html .
• A17: Are updates to the SERT Suite and PTDaemon tool provided free of charge?
• This has yet to be determined. SPEC expects to provide minor bug fix updates without charge.

• B1: What is the minimum hardware required?
  • At minimum, you will need two network connected systems and one accepted power analyzer and accepted temperature sensor for fully compliant runs.
  • The minimum SUT (Server Under Test) hardware requirements are descipted in the SERT User Guide

• C1: Where can I obtain software listed as a supported option for SERT Suite?
• Please contact the vendor or provider of the software for further assistance if you are unable to locate it yourself.

• D1: Is a power analyzer required to run the SERT Suite?
  • For compliant runs, yes.
  • For any other runs, the SPEC PTDaemon® (Power Temperature Daemon) has a device type "dummy" for these occasions.
• D2: Can the SERT support multiple power analyzers?
• Yes. Multiple power analyzers and multiple temperature sensors are supported. Nevertheless, currenly not though the GUI.
• D3: What power analyzers have been tested with the SERT Suite?
  • A list of SPEC Accepted (supported and compliant) power measuring devices are documented on the SPEC web site at: Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)
  • The PTDaemon program can be used to display a list of power and temperature measuring devices along with their device codes;
  • For SPEC Accepted devices: Run the PTDaemon with the -h parameter, or zero parameters.
  • Listed devices (except 0/1000. Dummy) are valid for compliant runs of the SERT Suite.
• D4: Will my power analyzer work with the SPECpower PTDaemon even though my device does not appear on the accepted devices list shown on the SPEC web site at Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)?
  • No, only devices on the SPEC Accepted Device List are expected to produce valid results.
  • If your device is not present on the devices list, it is unlikely that PTDaemon will successfully read data.
  • Communication protocols are quite different among the various manufacturers and models of power analyzers.
• D5: Will the SPECpower PTDaemon (Power Temperature Daemon) connect to and read my power analyzer even though my device is not on the tested list?
  • It is unlikely that PTDaemon will successfully read a meter not on the "tested" list.
  • See the responses to the FAQ just above.
• D6: How can I get my power analyzer included in the SPEC accepted devices list?
  • Basically, documentation must show that the device has sufficient accuracy, code must be written to enable SPEC's PTDaemon software to read measured values from the device, and then testing must be performed to show that the device does meet SPEC's requirements in actual usage.
• D7: Will a power analyzer calibration certificate for one device apply to other power analyzers of exactly the same model?
• No. The calibration certificate issued is valid only for the specific device (one serial number) calibrated.
• A recently calibrated power analyzer is required to produce and report compliant results. "Recent" means that the calibration certificate must be dated within one year of the test date.
• D8: Is the test certificate I received with my instrument sufficient to meet the calibration requirements of the power analyzer requirements section of the run rules?
• If the certificate is entitled "Certificate of Calibration" or "Certificate of Traceable Calibration" and includes a unique calibration number or sticker, lab accreditation to ISO or other standards, and test equipment certification with traceability to a national standards institute, it probably meets the requirements. National standards institutes include NIST (USA), PTB (Germany), and NML (ROC - Taiwan) among others.
• D9: No data cable came with my power analyzer (RS-232), what now?
• Each brand/model power analyzer may require a different cable. some experimentation might be required.
• Some power analyzers will work with a "null modem" cable.
• One combination that has worked with some devices is a standard "Modem" cable with the addition of a "NULL MODEM" adapter.
• The above combination can also be plugged into a RS-232 to USB converter.
• Some data cable information can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
• D10: My power analyzer supports only RS-232 for data logging; my control system has no serial ports; neither DB-9 nor DB-25. - What do I do?
  • There are devices that convert RS-232 to USB, that cost $10-14 (US$).
  • Several brands of serial to USB converters have been found to work well.
  • With Windows, install the software per manufacturer instructions.
  • Then, use the device manager to see what "COM" port number was assigned.
  • Edit the runpower.bat file; change the default DEVICE_PORT number.
  • Note that the COM port assignment on your system may change if you disconnect and re-connect a device or change USB connections.
• D11: The PTDaemon software can't connect to my power analyzer.
  - What is wrong?
  • First, assure the analyzer is communicating on the data cable with the host machine.
    • The best way to verify this is to first use the vendor supplied software to read the analyzer.
  • Most problems can be resolved with this method, problems such as cable not connected, wrong cable, incorrect device settings and so on.
  • If the vendor supplied software shows the correct readings, assure that the following are correct:
    • In the SERT Suite GUI PTDaemon Configuration, ensure the following options are set correctly:
    • Device Enabled checkbox is checked.
    • Listening port is 8888 (increment port for additional power or temperature devices.
    • Measurement Device is set to your supported Power Analyzer name
    • Interface is set correctly (Serial, GPIB, USB, or Ethernet).
    • Interface Address is set to the correct port for your configuration (COM1, COM2, etc.)
    • Finally, ensure you click the "Test Connection" button before proceeding.
    • Repeat these steps for your Temperature Sensor.
    • If not using the SERT Suite GUI, ensure the following are set correctly in runpower.bat/runtemp.bat in the PTDaemon subdirectory:
    • NETWORK_PORT set correctly: 8888 for power analyzer 1, 8889 for temp sensor.
    • DEVICE set to a value that reflects your connected Power Analyzer (run the ptd exectable for a device list).
    • DEVICE_PORT is set correctly (COM1, COM2, etc.)
    • More details on device setup can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
    • If your problems persist, please fill out a SPECpower support ticket at this link: http://www.spec.org/sert/feedback/issuereport.html.
• D12: Why does my power analyzer display unexpected watts readings?
  • This is likely to be traced back to how the analyzer is connected or the device settings.
  • More details on device setup can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
  • Instructions will also be found in the power analyzer documentation or on the vendor web site. You may also contact the device vendor for further technical support.
• D13: I am testing at the 115V specified in the EPA EnergyStar document, and when I specify 115 for the voltage in the LineStandard section of test-environment.xml I get messages that 115V is not a known standard. What do I do?
• Specify 120 for the voltage value. The LineStandard values refer to a country's standardized voltage under which the tests are being performed; 120V is the nominal standard voltage for the U.S.A. 115V is not a standardized voltage but rather a utilization voltage that the EPA has chosen. As long as the voltage measured at the SUT remains above 114V (95% of 120V) you will not get any errors or warnings.

• E1: How do I run without a temperature sensor?
  • The PTDaemon (Power Temperature Daemon) has a sensor type of "dummy".
    • this is the best temporary fix if you will add or use a sensor in other runs.
    • The device code for "dummy" temperature sensor is 1001.
  • You may also remove the temperature sensor data source from the ccs.props file.
  • comment the lines that define the temperature sensor (# as first character)
• E2: What temperature sensors have been tested with the SERT Suite?
• A list of successfully tested and compliant temperature measuring devices are listed on the SPEC web site on the Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html) page.
• E3: How do I find the "device code" for a temperature sensor?
• Run the PTDaemon program with zero command line parameters or -h.
• A list of temperature sensors and power analyzers will be displayed, along with the 'help' information.

• F1: What if the PTDaemon can't connect to the power analyzer?
  • See FAQ in the Power Analyzer section
  • Check the COM port and the analyzer type passed to the power daemon (ptd_xxxx.exe)
  • Assure the analyzer is connected via serial, GPIB or USB cable and the analyzer is powered on.
• F2: What if I don't see the operating environment combination (processor architecture, operating system, JVM) that I need in the SERT-JVM_Options.html file?
• First, check the most current file at http://www.spec.org/sert/SERT-JVM_Options.html. There is a link to this URL in the version of that file that is shipped with the product. It is possible that the desired environment has been included in the latest web version. Also, be aware that the options listed in the file are intended to be used for submissions to energy efficiency programs, such as the US EPA's ENERGY STAR program. If you are using the SERT to conduct research for purposes other than submission for such a program, you have more flexibility in the options that you choose to use.
• For submission of results using hardware, operating system and JVMs that are related to those that are listed in the options file, but are not specifically listed in the desired combination (For example, a processor family that is older than those listed in the options file), SPEC recommends using the H1 or J1 option sets. If this delivers unsatisfactory results, the user can submit a service request to ask for the development of a better set of JVM options.
• For submission of results using any of A) a hardware architecture that is not represented in the options file or B) an operating system that is not represented in the options file or C) a JVM that is not represented in the options file, the user is encouraged to contact the suppliers of the component(s) to request that they work with SPEC to complete the necessary testing to have their product be included in the list of accepted operating environments.
• F3: How do I know which specific Oracle Hotspot or IBM J9 JVM version to choose?
• The SERT Suite is designed to run with 64-bit Java JRE 1.7.0. Using the latest release from Oracle or IBM is strongly encouraged. Consult these vendors' websites for access to the latest versions.
• F4: I am having problems running the correct JVM on my SUT or controller. How do I ensure that the SERT Suite is using a specific JVM?
• By default, the serthost.bat/.sh (SUT) or director.bat/.sh (Controller) will expect to find the Java executable in the system PATH. If a suitable Java version is not found, the test will fail.
• Some Java versions in Microsoft Windows may not automatically insert themselves into the system PATH during installation.
• Linux systems normally require explicit pathing to the java executable in the serthost.sh file or in the system PATH environmental variable.
• Please check the serthost.bat/.sh or director.bat/.sh file on the problem system, and ensure that the set JAVA=java line is modified to point directly to the JVM executable you intend to run on the SUT.
• When switching between multiple JVM vendors or versions, modifying the set JAVA= line to point to the correct version is the preferred method.
• Example modification in serthost.bat on a Microsoft Windows system: set JAVA=c:\Hotspot-1.7.0_u15\bin\java.exe
• Example modification in serthost.sh on a Linux system: set JAVA=/usr/local/IBM-J9/jre/bin/java
• F5: I am encountering unexpected hangs and exits in the GUI after inserting non-ASCII characters (e.g. German Umlaut Characters) into the test-environment.xml file.
• The current release of the SERT Suite does not support non-ASCII characters in the test-environment.xml file.
• F6: What do I do if the SERT Suite run has completed in the GUI, I click "Next", and the "View" button doesn't work?
• The system is likely to be generating all of the html result files, which can sometimes take as long as 45 minutes to complete for very large SUT configurations. Wait for the "View" button to work, and you can view the results file at that point.
• F7: For SERT Suite configurations running more than 100 JVM clients on the SUT the following error messages will show up in "client.<#>.out" files for # greater than 99 and no "log.<#>.txt" files are generated:
• Can't get load log handler "java.util.logging.FileHandler"
• java.io.IOException: Couldn't get lock for log.%u.txt
• These error messages can be ignored. All client processes will proceed and finish their tasks correctly including those, which generate above error messages. For clients 100 and above you won't find the "log.<#>.txt" files but "client.<#>.out" files only.
• F8: Why do I only get a results.xml.partial file in the results directory after a full run?
• If the user attempts to view the results before final processing has been completed, this can occur. It could also be caused by a failure during post-processing of the results file. Please review the file director.log on the controller. The error messages often point to the location in the results file where the error occurred.
• F9: What is the "coefficient of variation between clients"?
• Coefficient of variation is a mathematical term that measures the variability of a set (defined as the standard deviation divided by the mean). Each Java instance that is run on the SUT is referred to as a client. SERT worklets have different limits on the maximum coefficient of variation between clients, as a way of ensuring that the test is run in a stable and repeatable way.
• F10: Why do I get WARNING messages about coefficient of variation between clients on storage worklets?
• A single client is run for each physical disk during storage worklets. If the results for the same worklet between clients show high variability, the warning is produced. If the disks are of different types this message is to be expected. If the disks are identical, this message usually indicates inconsistent configuration between disks.
• F11: My run terminates early on a SUT running Windows Server 2008 R2
• There are several common causes for this issue, all documented in the User Guide and JVM options list. You must install Service Pack 1 for Windows Server 2008 R2. You must enable "lock pages in memory". See the User Guide for more information.
• F12: During the installation process on recent Windows OSes, the SERT Suite installed can't create the SERT install directory off of my C:\ drive.
  • This is a known permission issue on some recent Windows Operating Systems. As a workaround, you can manually create the SERT Suite installation path (C:\SERT-1.0.2) prior to invoking the installer.
• F13: I am seeing java.io.IOException error messages when I run reporter.bat against a path that contains a space while trying to reformat a SERT report.
  • Change your path to results to be reformatted to not contain any spaces.
• F14: I am seeing "Unable to establish listener connetion on port 8897 Q:Is UIListener enabled in listeners.xml on local director?"
  • The most common cause of this message is insufficient heap memory to run the director on the controller. The director run script(director.bat on Windows, director.sh on Linux) requests up to 2GB of heap space. The controller system may not be able to allocate the full 2GB of memory.
  • To verify that heap memory is the issue, look at the file director.log in the SERT installation directory. The final lines should show error messages similar to the following: "Error occurred during initialization of VM Could not reserve enough space for 2097152KB object heap"
  • To resolve this issue, edit the director run script (director.bat/director.sh). Edit the following line: set JAVAOPTIONS_DIRECTOR=-Xms64m -Xmx2048m -Djava.util.logging.config.file=logging.properties
  • Change the maximum heap value (-Xmx) to a smaller value, such as 1024m: set JAVAOPTIONS_DIRECTOR=-Xms64m -Xmx1024m -Djava.util.logging.config.file=logging.properties
  • This amount of heap space should be sufficient for most systems being tested, although it may not be enough for a multi-node or blade system.
• F15: The processor/JVM/Operating System combination I need to generate valid results for, does not appear in the SERT Client Configurations list. How do I get it added?
  • The enablement and acceptance process is reasonably straightforward, and can be initiated by the processor vendor, or an authorized OEM/ODM that ships platforms based on the processor for which acceptance is needed. Virtually any platform/architecture combination can acquire SERT support. Platform acceptance for x86, ARM, Itanium, SPARC or POWER architectures running Windows or Linux is usually lightweight. All that is needed is completion of the tests described in ISO/IEC 21836:2020 9.3 using the, simple to follow, questionnaire-based spreadsheet provided by SPEC. Additional detials can be found in the SERT Platform Enablement and Acceptance Process documentation.

• G1: Can I submit an Invalid Result?
• No
• G2: Why am I getting invalid results with messages that relate to my power analyzer?
• INVALID: samples with unknown uncertainty exceeding the threshold
• INVALID: The voltage reported was above the maximum allowed voltage
• INVALID: The calibration date for the power analyzer is over 1 year before the test date
• Uncertainty and voltage errors relate to how the Power Analyzer ranges are set for the test. Please refer to the below links for instructions:
• Measurement Setup Guide: http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf
• Section 5 of the SERT User Guide: https://www.spec.org/sert2/SERT-userguide.pdf
• For Calibration date issues, please refer to section 3.3.5 and 6.1 of the SERT User Guide: https://www.spec.org/sert2/SERT-userguide.pdf
• It is expected behavior for the "Range Discovery" run to show many invalid messages in the results file. Range discovery runs use shortened run times and sets power analyzers to autorange, neither of which is allowed for a valid run. The primary objective of a range setting run is to determine good current range settings with your system under test. To perform a valid run, you must run the "Full Test Suite" after the range discovery run. Please see section 5.1 of the SERT User Guide for more information.
• G3: Why do I get INVALID messages about coefficient of variation between clients on memory worklets?
• This usually indicates an unbalanced memory configuration between nodes on a multi-processor system. If channels, speeds and capacities are not identical on each node, it is unlikely that memory worklets can run successfully.
• G4: I am seeing the Storage worklets get invalid messages with USB Hard Drives
  • USB Hard drives are not supported and will not work with the SERT Suite.
• G5: I am seeing more invalids show up in system configurations with smaller memory sizes.
  • It is strongly recommended that you reboot your SUT in between runs with smaller memory configurations (less than 1GB per HW thread).
• G6: I am seeing invalid results with multi-node or blade server configurations.
  • In multi-node and blade server configurations, if every node is not identically configured, the variation of performance between nodes will be too high. In this case, the run will be invalidated with an error such as "INVALID: The coefficient of variation among hosts was 17.6%, which is greater than the threshold of 10.0% in the 100% interval of worklet Compress". SERT requires that all nodes be identically configured. In virtually all cases we have seen, the issue was caused by configuration problems in one or more nodes. However, configuration mistakes and hardware issues can occur, and this error can be difficult to isolate. There are several steps that should be followed to isolate the problem node(s) and then test the fixes:
  • If using the GUI, you can preview discovery data using the preview button in the Discovery section early in the GUI panels. For multi-node configuration, any differences detected by the discovery process will be highlighted and should be resolved before continuing such as mismatched processors and memory between nodes.
  • If no discovery mismatches are found and a run still completes with errors, you can an XSL template provided with SERT to generate a detailed result file:
  • For example, if your results file is in results-0005, you can execute the following command from the SERT base directory: reporter.bat -s -c -x results\XSL-File\csv-subset-detail.xsl -o result-subset-detail.csv -r results\sert-0005\results.xml
  • This will produce a file results\sert-0005\csv-subset-detail.csv, which includes performance results for every Java instance for each SERT measurement interval. Each individual result is named "hostname:instance", where instance identifies a Java instance on that node. For a system with high variation between hosts, it is usually fairly easy to visually inspect the csv file and identify the node(s) whose values are inconsistent with the majority.
  • There are many, many potential causes of such variations: differing BIOS settings, thermal issues, memory errors, incorrect memory population, etc. To shorten the time required to test possible fixes, it is recommended that you pick a known "good" node and run only the worklet showing the highest variation, using those results as a baseline. Then test the inconsistent nodes in the same way until you come up with matching results. Once the problem nodes have performance matching the baseline node, you should be able to do a full run of all nodes without any further CV errors.

• H1: For systems with more than 64 logical processors running Windows 2008 R2, be advised that a Microsoft hotfix addresses an issue where logical processors are unevenly allocated to K-Groups. This can cause performance issues and high run to run variation with the SERT. The hotfix to address this issue is found here: http://support.microsoft.com/kb/2510206

• How much does the SERT Suite cost?
  • The cost of the SERT Suite and the order form are on the web at: http://www.spec.org/order.html
• How do I order the SERT Suite?
  • You will find an order form at: http://www.spec.org/order.html

Disclaimer

SPEC, the SPEC logo, and the name PTDaemon are registered trademarks of the Standard Performance Evaluation Corporation (SPEC) and SERT is a trademark of SPEC. Additional product and service names mentioned herein may be the trademarks of their respective owners. Copyright ® 1998-2019 Standard Performance Evaluation Corporation (SPEC). All rights reserved.