SPECmail2009 User Guide

Version 1.0
Last modified: Dec 2008

1     Introduction

1.1   Terminology

1.2   Overview

2     How to install SPECmail2009

2.1   Before you begin - a pre-installation checklist

2.1.1 Network Configuration

2.1.2 Directory Services

2.1.3 Mail Domain Names

2.1.4 Disk Space Allocation

2.1.5 Mailbox Creation

2.1.6 Client Configuration

2.1.7 Synchronizing the Clocks of the Servers and Clients

2.2   Setting Up the Server

2.2.1 Installing your Server

2.2.2 Testing your Server

2.2.3 Creating User Accounts on the Server

2.3   Setting Up the Clients

2.3.1 Installing Java

2.3.2 Installing the SPECmail2009 Software

2.4   RC Files

2.4.1 Configuration Parameters

2.4.1.1 Transport Security

2.4.2 System Under Test (SUT) Description

2.4.3 Trace Parameters

2.4.4 Fixed Parameters

3     How to Run SPECmail2009

3.1   Starting the Load Generators

3.1.1 Starting the Load Generators on UNIX

3.1.2 Starting the Load Generators on MS Windows

3.2   Starting the Mail Sink

3.3   Starting the Benchmark Manager

3.3.1 Options

4     Results

4.1   Generating a SPECmail2009 HTML Report

4.1.1 In UNIX

4.1.2 In MS Windows

4.2   The HTML Results File

4.2.1 Results Block

4.2.2 Summary Results

4.2.3 System Descriptions

4.2.4 Configuration Diagram

4.2.5 Detailed Results

4.2.6 Notes/Tuning Information

5     Tips and Troubleshooting

5.1   Using Java Version v1.5

5.2   Benchmark Manager Only Prints Version Number

5.3   Connecting to Remote Clients Failed: InvalidClassException

6     Submitting Results

Appendix A – Configuration Parameters

Appendix B – System Description Parameters

Appendix C – Trace Parameters

Appendix D – Fixed Parameters

1. Introduction

SPECmail2009 is a client server benchmark for measuring mail server performance.

Software on one or more client machines generates a benchmark load for a System Under Test (SUT) and measures the SUT response times. A SUT can be a mail server running on a single system or a cluster of systems.

A SPECmail2009 'run' simulates a 100% load level associated with the specific number of users, as defined in the configuration file. The mail server must maintain a specific Quality of Service (QoS) at the 100% load level to produce a valid benchmark result. If the mail server does maintain the specified QoS at the 100% load level, the performance of the mail server is reported as either SPECmail_Ent2009 IMAP sessions per hour or SPECmail_Ent2009Secure IMAP sessions per hour. These metrics reflect the unique workload combination for a SPEC IMAP4 user.

This document is a practical guide for setting up and running a SPECmail2009 benchmark. This user guide discusses some, but not all, of the rules and restrictions pertaining to SPECmail2009. Before continuing with the benchmark, we strongly recommend you read the complete SPECmail2009 Run and Reporting Rules contained in the kit. For an overview of the benchmark architecture, see the SPECmail2009 Architecture White Paper also contained in the kit.

1.1. Terminology

The System Under Test (SUT) is the hardware and software that makes up the mail server being tested by the benchmark. It includes the hardware system, disks, and network components used, as well as the OS and mail server software. The client machines (clients) running the load generators are not considered part of the SUT.

The Load Generator is the program that creates traffic (IMAP4 and SMTP) for the SUT. You can have as many clients running load generators as you wish, but you should have a minimum of two. One client will be the SMTP mail sink and one will be the benchmark manager.

The SMTP Mail Sink is a load generator that receives traffic from the SUT and that simulates a remote mail server. During the benchmark run, the load generators will generate mail for internal and external mail recipients. Mail intended for external mail recipients should be relayed by the SUT to the SMTP mail sink.

The Benchmark Manager is the program that reads in the RC files and drives the load generators. You can only have one benchmark manager. The benchmark manager is responsible for starting and stopping the load generator and for collecting results and creating the raw results file. The system that runs the benchmark manager is known as the Primary Client.

1.2. Overview

Below is a quick overview of what needs to be done before running the SPECmail2009 benchmark:

Install and setup your Mail Server according to your Mail Server instructions.
On your Mail Server, create (provision) the target number of user accounts.
Optionally, you may have to create target mailboxes associated with each user.
Configure the SUT such that it relays external e-mails to the designated mail sink host and port.
Configure the client machines to drive the SUT. At least two client machines are recommended.
Determine which clients will serve as the benchmark manager, mail sink, and load generators.
Install the SPECmail2009 benchmark manager and load generator software on your clients.
On the primary client, where the benchmark manager resides, modify the SPECimap_sysinfo.rc file to reflect client/SUT configuration information
On the primary client, modify the SPECimap_config.rc to reflect the SUT host names, port numbers, and number of target users.
Verify that mail can be sent and retrieved by the SUT in accordance to how the SUT is defined in the SPECimap_config.rc file.

Use the following steps to check that the configuration is set up correctly:

Setup and start a short run.
- Setup the SPECimap_config.rc file to define your SUT and a small number of users. For instance, set USER_START to 1 and USER_END to 10. This will generate a load for only 10 users.
- In the SPECimap_config.rc file, change the value of RUN_SECONDS to 600. The test will run for approximately 20 minutes (10 minutes for warm-up and 10 minutes for measurement).
- Start the load generators and the mail sink (the clients)
- Start the benchmark manager. Note, the clients must have started properly before you can start the benchmark manager.
Verify that the mail sink is setup properly, by manually connecting to the mail sink during the short test run. The mail sink should display a banner indicating that it is a SPECmail2009 program.
After the short test has completed, examine the mail logs on the Mail Server to ensure local/remote mail was delivered, and local mail was retrieved.
Run the reporter program on the results file, to verify that statistics and other benchmark related information is being recorded during the test run.
After running this short test, the mailboxes for the users involved in this short test should no longer be used in the compliant test user range. The alternative would be to “clean” these mailboxes.

2. How to install SPECmail2009

2.1. Before you begin – a pre-installation checklist

Here is a checklist of items to review before installing the SPECmail2009 benchmark software.

2.1.1. Network Configuration

It is strongly recommended that the benchmark be run in an isolated network, to avoid external network traffic impacting your measured results. 100base-T Ethernet or faster media and switching are recommended for connecting the clients and server. The connections between a load generating client and the SUT must not use a SUT value of TIME_WAIT less then 60 seconds (see the SPECmail2009 Run and Reporting Rules for more details).

2.1.2. Directory Services

Running directory services is optional, and it is your choice whether to run the directory service on the mail server or on a separate system. In either case, if you are running a directory service it must be reported as part of the SUT. Note that if you are using a directory service, it needs to be populated with the user information before the benchmark can be run (see Creating User Accounts on the Server below).

2.1.3. Mail Domain Names

You must choose two domain names, one for local delivery to the mail server and one for remote delivery to be relayed by the mail server. These domain names need to be entered into your internal domain name service, network information service, distributed name service, or other equivalent service for the mail server. Your mail server needs to be configured to relay remote mail to the SMTP mail sink. The domain names you choose should be unique and isolated from the real world.

2.1.4. Disk Space Allocation

Your mail server will need enough disk space to hold the mail store and mail queue based on the SPECmail2009 user load you intend to generate. You can scale appropriately by using the following formula:

Total storage required = 160MB * number of users + 10% overhead

Note that these disk space values are just estimates, the actual amount of disk space required may vary based on your SUT. The benchmark load will not maintain a steady state. Thus the number of messages and disk space used will increase during the benchmark run so, you may need additional space on your SUT. The percent overhead represents disk space used for file fragmentation, log files, and storage overhead.

2.1.5. Mailbox Creation

The mailboxes must be created prior to running the benchmark, using the naming convention specified in Creating User Accounts on the Server below. The mailbox contents can be initialized as part of the benchmark run or initialization can be done independently using the –init flag with the benchmark manager.

2.1.6. Client Configuration

The client systems running the benchmark manager and load generators must all use the same version of the Java Runtime Environment (JRE). The load generators require at a minimum JRE v1.5.0. The software has been tested using v1.6.0_1, v1.5.0_15 and v1.5.0_9 on several different platforms. The number of load generators needed depends on the number of SPECmail2009 users being simulated and the performance of the clients –. We recommend that you look at existing posted results for an indication of the number and size of clients you will need to run the benchmark. There is no requirement that all clients be the same type or size.

Note: Using multi-CPU/core hardware with more than four (4) cores (4x1, 2x2 or 1x4)
The best way to utilized the full computational capacity is to run multiple load generators instances on each box, not one load generator. Each load generator spins off one thread per IMAP connection. Java VM's do better when managing a smaller number of concurrent threads. The configuration files and executables support an optional command line parameter (-p portNumber) that causes it to listen on designated port (portNumber) instead of the default.

2.1.7. Synchronizing the Clocks of the Servers and Clients

Synchronization of the clocks on the client and server is not required for the SPECmail2009 benchmark, but it is recommended. Synchronized clocks will simplify evaluating the log files created by the benchmark on the clients.

2.2. Setting Up the Server

2.2.1. Installing your Server

Please see the installation instructions for your version of the mail server. Your installation must present the appearance and behavior of a single logical server for each protocol to all the mail clients (see the SPECmail2009 Run and Reporting Rules for more information).

You will need to set up local and remote domains as defined in the configuration file. The local domain is for messages stored by your mail server, the remote domain is for messages forwarded to the mail sink (a SPECmail2009 load generator acts as the mail sink during the benchmark run). Note that for some mail servers, both the local and remote domains must resolve (they must be recognized by your system). If the remote domain does not resolve, the server will not forward remote mail to the mail sink.

2.2.2. Testing your Server

Once you have installed and started your mail server, you should perform the following tests on all clients to ensure that the clients can communicate with the mail server. Note that you must set up your domains and at least one user account before attempting these tests:

For SMTP (change the words in italics according to your setup):

# telnet <your-server> 25

<the server should greet you with its server name and version number, etc>

helo domain.com

mail from: test001@domain.com

rcpt to: test002@domain.com

data

Subject: test message for SPECmail2009

This is a test for SPECmail2009

quit

Then go to your mail server queue repository to check if the mail is queued, or check the mailbox to see if it has been delivered. You can also use the following IMAP4 commands to check if the mail has been delivered:

# telnet <your-server> 143

<the server should greet you with its server name and version number, etc>

a login test002 password

b select Inbox

c fetch 1:* (BODY[])

d logout

2.2.3. Creating User Accounts on the Server

The user accounts you create on your mail server for the benchmark must be of the form <USERNAME_PREFIX><USER_ID>, for example test5 or test1000. The username prefix is defined in the configuration file (see Configuration Parameters below). Each user has a unique user ID, which is a number whose lower and upper bounds are defined in the configuration file.

You can create as many user accounts on the server as you wish; the benchmark will only use the user accounts that are dictated by the username prefix and user IDs specified in the configuration file. It is recommended you create your target maximum number of user accounts.

NOTE: The benchmark does not maintain the mailboxes contents in a steady state, so re-initialization will be necessary from run to run. An alternative is to initialize the SUT and then backup the SUT message store and the benchmark manager's working directory data for later restores to the initialized state.

2.3. Setting Up the Clients

2.3.1. Installing Java

The load generators are Java applications, so your clients will need a Java version v1.5 or later. You can obtain Java for your client platform from one of the following web sites:

IBM: http://www.ibm.com/java/jdk/download/index.html
Linux: http://www.blackdown.com/index.html
FreeBSD: http://www.freebsd.org/java/dists/11index.html
HP: http://www.unix.hp.com/java/index.html
MS Windows: http://java.sun.com/products/index.html
SGI: http://www.sgi.com/developers/devtools/languages/java.html
SUN: http://java.sun.com/products/index.html

Refer to the vendor web sites for installing and configuring Java on your clients. Java is not necessary on the mail server unless required by your mail server software.

Note that some versions of Java (such as freeBSD) do not contain a Just In Time Java bytecode compiler (JIT). A JIT compiler significantly reduces the CPU consumption of Java. As a result, depending on the speed of the CPU, without a JIT several clients will be required to generate a load for a relatively small number of users. Since a JIT can yield a 10 to 20-fold speedup over the Java interpreter, you may find that running without a JIT is not practical for producing a SPECmail2009 load.

2.3.2. Installing the SPECmail2009 Software

You need these five files to run the SPECmail2009 benchmark:

specimap.jar – this file contains the Java class files used to run the benchmark.
check.jar – this file contains the Java class file used to verify the benchmark.
SPECimap_sysconfig.rc – this file contains the hardware and software
SPECimap_config.rc – this file contains the configuration parameters that should be changed to reflect your environment.
SPECimap_fixed.rc – this file contains the fixed parameters for the benchmark, altering this file will results in a non-compliant run.

The specimap.jar and check.jar files must be present on all the client systems that will be running load generators. The RC files should be installed on the primary client (the system running the benchmark manager) along with the specimap.jar and check.jar files. The full installation includes additional files, including sample scripts (in the “sample_scripts” sub-directory) for running the benchmark manager and the load generators.

Note: Running multiple load generators on same host
You do not need to install multiple copies of the SPECmail2009 software on each load generator host. If you plan to run multiple instances of the SPECmail2009 load client. Just modify the enclosed scripts to start each instance with the command line parameter (-p portNumber). Make sure that each portNumber is unique for that host. The portNumber value does not have to be unique across hosts.

2.4. RC Files

2.4.1. Configuration Parameters

Parameters for identifying clients and the mail server are contained in the SPECimap_config.rc file. This is where your client host names and port numbers are listed for the benchmark manager to use when controlling the load generators. This is also where you identify the SMTP server, the IMAP4 server, and the mail sink server name and port number.

At a minimum, to run a test you will need to modify the following parameters:

CLIENTS
SMTP_SERVER
SMTP_PORT
IMAP_SERVER
IMAP_PORT
SINK_SERVER
SINK_PORT
LOCAL_DOMAIN
REMOTE_DOMAIN
USERNAME_PREFIX
USER_END

See Appendix A for a detailed description of the configuration parameters.

Note: Running multiple load generators on same host
The CLIENTS configuration key must list each load generator instance separately. If your host's name is zoog and it runs four (4) load generator instances starting on port 1200, then you should use zoog:1201,zoog:1202,zoog:1203,zoog:1204[others...] in the list of load generator hosts.

2.4.1.1. Transport Security

The benchmarker has the option of running with or without transport security (SSL/TLS) between the load generators and the SUT. Runs without transport security produce results branded with the SPECmail_Ent2009 metric. Runs with transport security produce results branded with the SPECmail_Ent2009Secure metric. The "master switches" for transport security are the IMAP_SECURE and SMTP_SECURE configuration parameters. Both must be enabled for a SPECmail_Ent2009Secure result.

When IMAP_SECURE and/or SMTP_SECURE are enabled, several other fixed and configurable parameters come into play. The fixed parameters select key characteristics of the transport security, including the mechanism (SSL/TLS vs. STARTTLS), the security protocol (e.g., SSLv3 or TLSv1) and the cipher suite. Like all fixed parameters, these must not be altered for a compliant run. The configurable parameters allow the benchmarker to use secure transport for the benchmark's support phases also, in case the SUT always requires secure connections.

2.4.2. System Under Test (SUT) Description

The SUT is defined in SPECimap_sysinfo.rc file. This information is used to create a section describing the SUT in the HTML results file. This information is not necessary for running the benchmark, but it is necessary if you intend to publish results. The SPECmail2009 Run and Reporting Rules specifies that a published result should contain enough information about the SUT to allow someone else to reproduce the results.

Since a SUT may consist of more than one system, most of the parameters use an array format to allow you to identify multiple systems. For example, the function definition for the first system in your configuration uses the parameter SYSTEM.FUNCTION[0]. Additional systems would use the parameters SYSTEM.FUNCTION[1], SYSTEM.FUNCTION[2], SYSTEM.FUNCTION[3], etc. The main exceptions are SYSTEM.TITLE, which is used to name the SUT, and SYSTEM.CONFIG_DIAG, which is used to indicate a file containing a diagram of the SUT.

A description of the clients running the load generators is not required, but is recommended. The clients must appear in the configuration diagram, so placing the hardware and software descriptions in the SUT description will help when evaluating the results.

See Appendix B for detailed descriptions of the SUT parameters.

2.4.3. Trace Parameters

The SPECimap_config.rc file contains a section with trace variables. These trace variables are commented out initially, if you uncomment one or more of them the clients will produce trace information while running the benchmark. This trace information is provided for testing and debugging purposes. You can use the traces when initially setting up your mail server or to track down problems if you are having difficulty with your configuration.

As an alternative, you can disable the trace variables by uncomment them and changing their value to zero. Then to enable them you simply change their value to one.

See Appendix C for a detailed description of the trace parameters.

These trace variables will have a performance impact on the clients, so they should be commented out when running the benchmark to produce publishable results.

2.4.4. Fixed Parameters

The SPECimap_fixed.rc file contains the parameters used by the benchmark to create the compliant workload generated for the mail server, establish run times, and measure the mail server Quality of Service (QoS) criteria. You should not edit this file. If you would like to alter one of these parameters while testing your mail server, you can override these values by putting them in the SPECimap_config.rc file. Any of these values found in the SPECimap_config.rc file will override the corresponding value in the SPECimap_fixed.rc file, and a note will be entered into the results indicating that a fixed parameter was overridden. You cannot produce publishable results if any of these parameters are altered.

See Appendix D for a detailed description of the fixed parameters.

When running the benchmark, the benchmark manager checks the SPECimap_fixed.rc, if it is altered in any way the run will be marked non-compliant.

3. How to Run SPECmail2009

3.1. Starting the Load Generators

The benchmark manager controls the benchmark load generators, but the user must start them manually. Scripts are provided for starting the load generators, but you will need to modify them to work in your environment.

3.1.1. Starting the Load Generators on UNIX

If you are using Java 1.5 or later and java is in your path, the simplest way to launch the client would be to use the command:

  # java -classpath specimap.jar:check.jar specimapclient [-p portNumber]

The default client port is 1099.

You can check which version of java your are using by issuing the following command:

  # java -version

As an alternative, you can put the jar files in your CLASSPATH environment variable, using one of the following commands:

In Bourne/Korn Shell:

  # CLASSPATH=<install dir>/specimap.jar:<install dir>/check.jar

  # export CLASSPATH

In C-Shell:

  # setenv CLASSPATH <install dir>/specimap.jar:<install dir>/check.jar

Then you can run the client using:

  # java specimapclient

Note that the specimap.jar file must be the first file in your classpath.

The default heap size for your Java VM may be insufficient; setting the memory heap size to 256Mb (via the –mx256m flag) or larger should provide enough memory for running the load generator. If your client aborts with memory exceptions, try increasing this value.

The memory heap size must be scaled to match the number of target users. The benchmark collects and holds runtime statistics in memory as the test progresses. Therefore, memory heap sizes that work with 200 users will be insufficient for 2000 users. Some experimentation will be required to tune this value for your environment (number of load generators and number of users).

The load generators have a single flag, -p <port>, which allows you to set the port number the specimapclient will listen on for the benchmark manager. The port number given on the command line must match the port number indicated for that client in the configuration parameters file. If the port is not provided, a default port number of 1099 is used. You will need to alter the port number if you are running multiple instances of the load generator on the same client machine or have a port number conflict with other software running on the system. Port numbers below 1024 are generally reserved and should not be used for specimapclient.

Thus, an extended version of the command used to start the load generator using a different port number would look like this, assuming your CLASSPATH variable has been set:

  # java –mx256m specimapclient -p 1555

Shell scripts for starting the client are provided with the benchmark; you can modify and use them as well.

3.1.2. Starting the Load Generators on MS Windows

To run the load generators, it is simplest if the specimap.jar and check.jar files are in your CLASSPATH environment variable. To add the jar files to your CLASSPATH in MS Windows, do the following (this example is for MS Windows 2000, your system might differ slightly):

Open the control panel
Select System
In the System Properties window, select the Advanced tab
In the Advanced tab panel, select Environment Variables
In the Environment Variables window, select 'New...' under the User Variables
In the New User Variable Name field, enter 'CLASSPATH'
In the New User Variable Value field, enter <path>\specimap.jar; <path>\check.jar

Note that the specimap.jar file must be the first file in your CLASSPATH.

Once CLASSPATH is set, open a command window and issue the command:

  C:\> java specimapclient

The default heap size for your Java VM may be insufficient; setting the memory heap size to 256Mb (via the –mx256m flag) should provide enough memory for running the load generator (assuming that your client has 512Mb of RAM). If your client aborts with memory exceptions, try increasing this value.

The load generators have a single flag, -p <port>, which allows you to set the port number the specimapclient will listen on for the benchmark manager. When starting on a client, the port number given on the command line must match the port number indicated for that client in the configuration parameters file. If the port is not provided, a default port number of 1099 is used. You will need to alter the port number if you are running multiple instances of the load generator on the same client machine or have a port number conflict with other software running on the system.

Thus, an extended version of the command used to start the load generator using a different port would look like this:

  C:\> java –mx256m specimapclient -p 1555

Another way to start the load generator on MS Windows is to use a batch script. Batch scripts for starting the client are provided with the benchmark; you can modify and use them.

3.2. Starting the Mail Sink

The mail sink is started the same way you start the load generators; please see the section above. The specimapclient is used in both cases. A mail sink can also be a load generator, depending on how the configuration file is set up. Note that at this time only one system can act as a mail sink.

You will not be able to verify that the mail sink id started correctly until the benchmark begins the warm-up period. Just prior to the warm-up period, the Benchmark Manager connects to and starts up the mail sink. After the warm-up period has started, you can verify the mail sink has started correctly by performing the following:

  telnet <mail sink hostname> <SINK_PORT>

The SINK_PORT is the port number the mail sink will listen on for receiving outbound messages from the benchmark. A specimapclient banner will be displayed after connecting to the mail sink. Typically, mail servers use port 25 to relay outbound messages. Therefore, if your mail server is configured to use port 25 as the sink port, ensure there are no other processes using port 25 (e.g. the sendmail process).

3.3. Starting the Benchmark Manager

Starting the benchmark manager is similar to starting the load generators; the main difference is that you use specimap rather than specimapclient in the java command and provide command line flags to control the behavior of the benchmark. Scripts are provided for running the benchmark manager in both UNIX and MS Windows; you can also refer to them for examples of how to run the benchmark manager in your environment.

3.3.1. Options

Several options are available when launching the benchmark manager.

*Supported Command Line Parameters*
Parameter	Benchmark Mode/Actions
-v[ersion]	Prints the benchmark version number and exits.
-u[sage]	Prints the list of command line options, usage information and exits.
-init	This command line flag instructs the benchmark manager to initialize the mailboxes on the mail server and then terminate. The manager first empties all the mailboxes in the user range and then adds messages to the mail boxes based on the benchmark’s defined steady state.
-clean	This command line flag instructs the benchmark manager to delete all messages and folders in the mailboxes.
-verify	This command line flag instructs the benchmark manager to verify the mailboxes are in the compliance with the expected folder structure, message quantity and message size distributions. When running the benchmark, prior to starting a measurement run, the mailboxes are checked to ensure that the message store has the right distributions. This command line flag runs the verify step, indicates the result and terminates so you can check the state of your server prior to performing a compliant run.
-loadtest	This command line flag instructs the benchmark manager to initiate the peak hour work load. Note: This is not the same as a compliant run, even though it generates the same work load.
-compliant	This command line flag instructs the benchmark manager to check the RC files to ensure that the ensuing run will be compliant (i.e. no fixed parameters have been overridden). If the run will not be compliant, the benchmark manager terminates with warnings. Without this flag, the manager will still run the benchmark with non-complaint settings and will mark the results as invalid.
-config <filename>	This command line flag instructs the benchmark manager to use the identified file for the configuration parameters instead of the default Specimap_config.rc file. This flag is provided so you can use different configuration files for testing and/or place the configuration file in a different directory.
-fixed <filename>	This command line flag instructs the benchmark manager to use the identified file for the fixed parameters instead of the default Specimap_fixed.rc file. This flag is provided so you can place the fixed parameter file in a different directory or use a different file when testing.

So to start from a known state and initialize a new mail server (after setting up the user accounts), when running the benchmark manager on UNIX, use the following command:

  # java specimap -clean -init

In MS Windows:

  C:\> java specimap -clean -init

To check and see if the new mail server has been properly initialized or can pass the various folder and message distribution verification, use the following command:

  # java specimap -verify

In MS Windows:

  C:\> java specimap -verify

To start a compliant run on UNIX, use the following command:

  # java specimap -compliant

In MS Windows:

  C:\> java specimap -complaint

To remove all messages and/or folders for all configured user mailboxes on the mail server, use the following command:

  # java specimap -clean

In MS Windows:

  C:\> java specimap -clean

4. Results

After completing a run, the benchmark manager will create a directory named result-yyyyMMddhhmm and generate an output.raw file containing the raw results from the benchmark run. The directory name “result- yyyyMMddhhmm” indicate the date and time the benchmark was started. SPECmail2009 includes a utility for generating an HTML results file called the reporter. The reporter is written in Java, so running it is similar to starting the load generators or benchmark manager.

4.1. Generating a SPECmail2009 HTML Report

4.1.1. In UNIX

To run the reporter, it is simplest if the specimap.jar file is in your classpath. To add the specimap.jar file in your CLASSPATH, use one of the following commands:

In Bourne/Korn Shell:

  # CLASSPATH=<install dir>/specimap.jar:<install dir>/check.jar

  # export CLASSPATH

In C-Shell:

  # setenv CLASSPATH <install dir>/specimap.jar

Then cd into the directory where the output.raw file is. You run the reporter using:

  # java specimapreporter <filename>.raw

Where <filename>.raw is the raw file generated by the benchmark. By default this file will be called output.raw. The reporter will generate an HTML file of the form <filename>.html. The reporter also supports a ‘-d’ flag for producing a more detailed report. This report contains additional information you can use when evaluating the performance of your mail server. To generate a detailed report for output.raw, use the command:

  # java specimapreporter -d output.raw

4.1.2. In MS Windows

To run the reporter, it is simplest if the specimap.jar file is in your classpath. To add the specimap.jar file in your CLASSPATH, do the following (this example is for MS Windows 2000, your system might differ slightly):

Open the control panel
Select System
In the System Properties window, select the Advanced Tab
Select Environment Variables
In the Environment Variables window, select ‘New…’ under the User Variables
In the New User Variable Name field, enter ‘CLASSPATH’
In the New User Variable Value field, enter <path>\specimap.jar:<path>\check.jar

Then, open a command (“DOS”) window and cd into the directory where the output.raw file is. You run the reporter using:

  C:\> java specimapreporter <filename>.raw

 C:\> java specimapreporter -d output.raw

4.2. The HTML Results File

The HTML file created by the reporter is designed to be consistent with other SPEC reporting pages. It contains the metric of the benchmark run, a summary of the key results, and a complete system description. The detailed report includes additional information about the benchmark at the three different load levels.

4.2.1. Results Block

The results block contains the key information for the benchmark results. It includes the system name and results. It also includes publishing information such as who produced the results and when the results were created. In the case of an invalid run, a description below the results block will contain a link to the Error section that contains detailed information about why the run did not produce a valid (i.e. publishable) result.

4.2.2. Summary Results

The summary results section contains the Quality of Service (QoS) performance of the mail server on the key functions. These functions are the primary items of concern when testing a mail server using the SPECmail benchmark, and thus the strengths and weaknesses of a mail server can be determined from this table. It includes the following items:

Key Function	Criteria
SMTP Connect	QoS measure for how well the mail server responds to a SMTP connection request
SMTP Data	QoS measure for how well the mail server responds to a SMTP data command (which contains the body of an outgoing mail message)
IMAP Connect	QoS measure for how well the mail server responds to an IMAP4 connection request
IMAP Select	QoS measure for how well the mail server responds to an IMAP4 folder select request, which is how a mail user requests information on new mail in his mailbox
IMAP List	QoS measure for how well the mail server responds to an IMAP4 folder list request.
IMAP Fetch Uid BodyAll	QoS measure for how well the mail server responds to an IMAP4 retrieve-by-Message_UID request, which is how a mail user extracts messages from the server
IMAP Fetch SeqNum Body	QoS measure for how well the mail server responds to an IMAP4 retrieve-by-Message_sequence_number request, which is how a mail user extracts messages from the server
IMAP Store Set Flags	QoS measure for how well the mail server responds to an IMAP4 message status flag change request
Error Rate	QoS measure for how well the mail server responds under load

4.2.3. System Descriptions

The system description section contains detailed information about the SUT. This information is extracted from the configuration file, which the benchmarker sets up prior to running the benchmark. It is intended to provide information for viewers to use when evaluating different systems and comparing results. The SPECmail2009 Run and Reporting Rules specify that enough information be provided so that a third party could set up a system and duplicate the results published.

4.2.4. Configuration Diagram

When submitting results to SPEC for publication, you must include an image file with a drawing of you mail server configuration. The HTML file created by the reporter will contain a link to this file using the file name specific in the configuration file. The file must at a minimum indicate the network connections between the mail server and the load generators. The image must be in GIF, JPEG, or PNG format.

4.2.5. Detailed Results

If the ‘-d’ flag is used when creating the result file, a detailed results section for each load is included in the HTML file. This detail section includes more information on the summary results (above) and additional performance details recorded by the load generators. The detail section includes:

The actual and attempted loads, expressed as number of users and message rate.
The attempted and passed counts for the different mail server requests
Session and error counters
Timing information (minimum, maximum, average) for the different mail functions
Sizing information (minimum, maximum, average) for outgoing (SMTP) and incoming (IMAP) mail messages.

4.2.6. Notes/Tuning Information

The Notes/Tuning Information section contains additional information on the SUT configuration and tuning parameters.

If the results failed validation for some reason, an Errors block will be added to the end of the Notes/Tuning Information section with a detailed description of what caused the failure.

5. Tips and Troubleshooting

Tips and Troubleshooting contains additional information on running the benchmark.

5.1. Using Java Version v1.5

The SPEC Mail Server subcommittee suggests you use Java version 1.5 or later.

5.2. Benchmark Manager Only Prints Version Number

The scripts provided to run the benchmark manager use the '-v' flag which prints out the version number of the code. If you would like to run the benchmark manager using a script, you should modify the provided script to use the appropriate options for your configuration. The first time you run the benchmark, you should use the command:

java specimap -clean -init

which initializes the mail servers mailboxes. This initialization phase takes a significant amount of time. By default the benchmark assumes the mailboxes have been initialized prior to starting. The benchmark causes the average mailbox to grow in size by the end of the run. You will need to either re-initialize the message store before each load test phase, or backup/restore the message store and benchmark manager's working directory before each load test. You can verify that the mailboxes were initialized correctly using the command:

java specimap -verify

When performing a benchmark run to generate results for publication, you should use the command:

java specimap -compliant

The -compliant flag will cause the benchmark to terminate if an error or warning is encountered during the run. Without the -compliant flag, warnings will be printed to the console but the benchmark will continue to run.

5.3. Connecting to Remote Clients Failed: InvalidClassException

When launching the benchmark manager and adding hosts, you may see an error containing text similar to:

Error: Name lookup for load generator alt100 exception error

unmarshalling return; nested exception is:

        java.io.InvalidClassException: specimapclient_Stub; Local

class not compatible: Stream classdesc serialVersionUID=xxx local

class serial VersionUID=yyy

...

This error indicates that you have non-compatible versions of the benchmark code on your clients. All clients (running the benchmark manager, load generators and/or mail sink) must have the same version of the benchmark software installed and running in order to run the benchmark.

6. Submitting Results

Once you have a successful run, you can submit the results to SPEC for review by placing the output.raw file and the configuration diagram into either a zip or compressed tar archive file. Attach this compressed archive file (.zip or .tar.gz) to a message and e-mail it to submail2009@spec.org. There should be only one result set per archive file, but multiple archive files (thus multiple result sets) can be attached to one mail message. Each message attachment will be processed as a separate submission.

Note: The raw file contains the configuration information from the SPECimap_sysinfo.rc file. Please edit the RC file with the correct SUT information prior to running the benchmark for submission.

Every submission goes through a two-week review process, starting on a scheduled Mailserver sub-committee conference call. During the review, committee members may ask for additional information or clarification of the submission. Once the result has been reviewed and accepted by the committee, it is displayed on the SPEC web site at http://www.spec.org/mail2009.

Please refer to both the SPECmail2009 Run and Reporting Rules and the SPEC Fair Use Rules for the rules governing results publication and usage.

Appendix A – Configuration Parameters

Parameter Name	Usage
CLIENTS	A list of clients that will be running the load generators for the benchmark. A client entry consists of the machine name and the port number the load generator the benchmark manager will use to connect to the client.
SMTP_SERVER SMTP_PORT	The host name and port number of the SMTP server for your mail server.
IMAP_SERVER IMAP_PORT	The host name and port number of the IMAP4 server for your mail server.
SINK_SERVER SINK_PORT	The host name and the port number of the system that will provide mail sink services during the benchmark. Note that mail sink services are provided by a Specimap load generator, thus this system must have a load generator running and using this port.
LOCAL_DOMAIN	The domain name that represents local mail on the mail server (i.e. mail intended for local recipients that must be stored).
REMOTE_DOMAIN	The domain name that represents remote mail on the mail server (i.e. mail not intended for local recipients that must be forwarded).
USERNAME_PREFIX	The prefix to use on the user account names for the mailboxes; user account names are of the form <USERNAME_PREFIX><USER_ID>.
USERNAME_AS_PASSWORD	A flag to indicate that the user account name should also be used as the password (0=no, 1=yes).
USER_PASSWORD	If the account name is not used as the account password, this value will be used as the password for all the user accounts.
USER_START USER_END	The starting and ending numbers to use for the user account names for the mailboxes; user account names are of the form <USERNAME_PREFIX><USER_ID>.
SMTP_SEND_DELAY_MILLISECONDS	When sending messages to the mailboxes during mailbox initialization, this value indicates how long to wait between messages.
SMTP_INIT_REST_MINUTES	After completing mailbox initialization, the benchmark will ‘rest’ for the specified amount of time to allow the mail server to de-queue all the new messages.
RSL_FILENAME	The RSL file is an ASCII file generated by the benchmark manager when the run is completed that summarizes the raw file data for the run.
Options for secure transport:
IMAPS_PORT	The port number on the SUT where the mail server listens for IMAPS (IMAP + SSL) connections.
SMTPS_PORT	The port number on the SUT where the mail server listens for SMTPS (SMTP + SSL) connections.
SMTP_USE_AUTH	Determines whether the benchmark authenticates the user to the SMTP server using AUTH PLAIN. Set this if your SMTP server requires authenticated sessions.
IMAP_SECURE	Determines whether the benchmark uses any transport security for IMAP connections during any benchmark phase (a "master switch" for all IMAP transport security).
SMTP_SECURE	Determines whether the benchmark uses any transport security for SMTP connections during any benchmark phase (a "master switch" for all SMTP transport security).
IMAP_SECURE_INIT IMAP_SECURE_VERIFY IMAP_SECURE_GATHER IMAP_SECURE_CLEAN	Secure transport to use for all IMAP connections during the benchmark's support phases (init, verify, context gather, and clean).

Appendix B – System Description Parameters

The following parameters are singletons, they should appear only once in the configuration file.

Parameter Name	Usage
PREPARED_BY	The name of the person that ran and submitted this result.
TESTED.BY	The name of the company that ran and submitted this result.
LICENSE.NUM	If the company is a SPEC member, this is the SPEC license number assigned to that company. If the company is not a SPEC member, this is the license number assigned when the SPECmail2009 license was purchased from SPEC.
TEST.DATE	Month and Year of the benchmark run. This field must be of the form Mon-YYYY (i.e. “Nov-2000”).
NOTES	General notes to appear in the result file. Multiple notes can be entered, so the parameter uses an array notation. For example: NOTES[0] = “This is the first notes line.” NOTES[1] = “” NOTES[2] = “You can even include blank lines.”
SYSTEM.TITLE	The overall name for the SUT. This is used as the title for your results page.
SYSTEM.CONFIG_DIAG	Name of the file containing your configuration diagram.

The following parameters are system description parameters; they are repeated for each system being described as part of the SUT. The files uses an array context for these parameters, see the provided Specimap_config.rc file for examples.

Parameter Name	Usage
SYSTEM.FUNCTION	The label that will appear at the very top of the system description box.
SYSTEM.NUM_SYSTEMS	The number of systems of this type used in the test.
SYSTEM.HW_MODEL	This is the hardware model of the system. Note: this field is a searchable field, so model naming should be consistent with other SPEC submissions when publishing results.
SYSTEM.HW_VENDOR	This is the hardware vendor of the system. Note: this field is a searchable field, so vendor naming should be consistent with other SPEC submissions when publishing results.
SYSTEM.HW_AVAIL	The date the hardware is/will be shipping and generally available to the public. This field must be of the form Mon-YYYY (i.e. “Nov-2000”).
SYSTEM.OS	Operating System (including version number)
SYSTEM.FILESYSTEM	The file system used.
SYSTEM.SW_LABEL	The label that appears in the Software section of the system description box. A special value of “JVM” entered into this field causes the SW_NAME to be ignored and the JVM and JIT fields to be used instead (for use with client systems running the load generators).
SYSTEM.SW_NAME	The name and version number of the software under test (this field is ignored on client systems).
SYSTEM.SW_AVAIL	The date the software is/will-be shipping and generally available to the public. This field must be of the form Mon-YYYY (i.e. “Nov-2000”).
SYSTEM.JVM	Java Virtual Machine description and version number. This field is only used if the SYSTEM.SW_LABEL parameter contains “JVM”.
SYSTEM.JIT	Just In Time Java compiler description and version number. This field is only used if the SYSTEM.SW_LABEL parameter contains “JVM”.
SYSTEM.CPU	The type of central processing unit(s) in the system.
SYSTEM.CPUMHZ	The speed (in MHz) of the CPUs.
SYSTEM.CPU_ENABLED	The number of CPUs in the system.
SYSTEM.MEMORY	The amount of physical memory (in Megabytes) in the system. This field should be an integer (do not use “Mb” or “Gb” in this field).
SYSTEM.L1CACHE	The amount of level 1 cache, for instruction (I) and data (D) on each CPU.
SYSTEM.L2CACHE	The amount of level 2 cache on each CPU
SYSTEM.L3CACHE	The amount of level 3 cache on each CPU. If caches higher than level 3 exist, this value should represent all level 3 cache and higher.
SYSTEM.DISK	Size and Type of disks used by the system. If a complex disk system is used, additional information on the disks should be included in the SYSTEM.NOTES parameters.
SYSTEM.NETWORK	Type of network interface(s) used by the system.
SYSTEM.HW_OTHER	Other hardware in the system that is performance-related.
SYSTEM.NOTES	Additional notes about the system.

Appendix C – Trace Parameters

The trace parameters allow you to have the load generators print information messages to standard out when running the benchmark. If you are going to be activating a trace, you should redirect the output from the load generators to a file when you start them on the clients.

These trace variables will have a performance impact on the clients, so they should be commented out when running the benchmark to produce publishable results.

Parameter Name	Usage
TRACE_EVENTS	Print message for all events in the load generators.
TRACE_FOLDERVRFY	Print message for every IMAP4 folder verify event – a check of all folders (mailboxes) in each user’s mail store complies with the expected sub-folder and count distributions.
TRACE_MESSAGEVRFY	Print message for every IMAP4 message verify event – a check of all messages in each user’s mailbox/folder to ensure that the expected size and count distributions are met.
TRACE_IMAPCLEAN	Print message for IMAP4 clean events. Clean events occur when the benchmark empties the mail boxes.
TRACE_FOLDERINIT TRACE_MESSAGEINIT	Print message for IMAP4 initialization events specific to creating the target folder structures and populating these folders. Initialization events occur when the mailboxes are pre-populated with the benchmark mail distribution.
TRACE_TP1 TRACE_TP2 TRACE_TP3 TRACE_CS6	Print message for each IMAP4 command associated with specific Traffic Patterns. Only TP1 through TP5 are used for a compliant run. However, CS6 is used as a traffic pattern that has no random parameters.
TRACE_LOADGENERATOR	Print message for communications with load generators and their responses.
TRACE_SECURE_TRANSPORT	Report on the use of transport security (SSL/TLS).

TRACE_SMTPQOS	Print message for SMTP QoS events. QoS events are generated when the load generators create, send and retrieve mail message to verify the mail server meets the delivery time criteria.
TRACE_SMTP	Print message for SMTP events.
TRACE_SMTPSINK	Print message for SMTP SINK events. SINK events occur when a load generator, acting as the mail sink, receives message relayed by the mail server that are addressed to remote users.
TRACE_MISC	Print message for miscellaneous events. This prints messages for activity that does not fit well into any other category.
TRACE_MDIM_KEYS	When tracing a vector array, this trace will print out the key values being searched.
TRACE_CLIENT_MSGS	When processing the client list from the configuration file, this trace will print out the clients being extracted from the list.
TRACE_SETMSGSIZES	When processing the message size list from the file, this trace will print out the message sizes being extracted from the list.
TRACE_SETMBOXSIZES	When processing the mailbox size distribution list from the file, this trace will print out the items being extracted from the list.
TRACE_CONFIGREAD	When processing items in the configuration file, this trace will print out the items as they are parsed.
TRACE_READFROM	When processing the parameter files, this trace prints a message when each section is complete.
TRACE_SETRECIP_MSGS	When processing the mail recipient distribution list from the file, this trace will print out the items being extracted form the list.
TRACE_MODEM_RATES	When processing the modem rates to use in the load generators, this trace will print out the rates being extracted form the list.
TRACE_CALCRESULT	When running the reporter to generate HTML results, this trace will print out the values being used and the calculations that are performed.
TRACE_RESULTS	Print message when the load generators record results.
TRACE_MSGSIZE	Print message when the load generators create a new message to send.
TRACE_WORKLOAD	Print message when adding or changing the workloads on the clients.
TRACE_VERIFY_MBOX_COUNT	When verifying the number of messages in the mailboxes matches the expected distribution, this trace will print out the actual and expected message count distribution in the mailboxes.
TRACE_VERIFY_MSG_SIZE	When verifying the size of the messages in the mailboxes matches the expected distribution, this trace will print out the actual and expected message size distribution in the mailboxes.

Appendix D – Fixed Parameters

The fixed parameters are used by the benchmark to setup the load generators. You should never edit this file. If you would like to alter one of these parameters while testing your mail server, you can override these values by putting them in the SPECimap_config.rc file. Any of these values found in the SPECimap_config.rc file will be used in place of the value located in the SPECimap_fixed.rc file, and a note will be entered into the results indicating that a fixed parameter was overridden.

Parameter Name

Usage

WARMUP_SECONDS

How long to run the benchmark at load before starting to gather statistics.

RUN_SECONDS

How long to run the benchmark at load while gathering statistics.

RAMPDOWN_SECONDS

How long to wait after stopping the gathering of statistics and terminating the load before starting a new test (a.k.a. cool down period)

LOAD_FACTORS

The load factors with which we scale the load in a set of consecutive test runs.

MSG_SIZE_DISTRIBUTION

The message size distribution for mail messages sent by the load generators.

MSG_RECP_DISTRIBUTION

The number of recipient’s distribution for mail messages sent by the load generators.

MIME_PART_SIZE_DISTRIBUTION

The probability distribution of the various MIME parts that make up a sing message.

MIME_TOP_PART_DISTRIBUTION

The probability that a MIME level has the sublevels indicated in the various distribution buckets.

MIME_PART_LEVEL

The probability that a message has the number of nested MIME levels indicated in the distribution buckets.

MIME_TOP_PART_COUNT

The probability distribution of the number of MIME parts at the top level

POP_RETRY_BOXES_PERCENT

The percentage of mailboxes set aside for POP retries.

DATA_RATE_DISTRIBUTION

Ignored for this benchmark

SLOW_MODEM_DATA_RATE
FAST_MODEM_DATA_RATE
CABLE_MODEM_DATA_RATE

Characters per second for 28.8Kbps, 56.6Kbps and cable modems. These are ignored in the IMAP benchmark.

PROTOCOL_TIMEOUT_ SECONDS

The number of seconds after which a connection is deemed to have timed out.

PROTOCOL_TIMEOUT_ PERCENT

The percentage of connections that should NOT time out.

DISCONNECT_PERCENT

The percentage of client connections that will unexpectedly disconnect (i.e. the load generators will not properly terminate the connection).

REMOTE_REFUSED_PERCENT

Percent of remote deliveries will will be refused immediatley, forcing local queueing.

REMOTE_DELAY_DISTRIBUTION

Distribution of response delays (in milliseconds) for accepted SMTP connections (versus REFUSED).

DELIVERY_TIME_CHECK_PERCENT

The percentage of local SMTP messages that should be check for delivery time Quality of Service (QoS).

SMTP_DELIVERY_QOS_SECONDS

The number of seconds within which a local SMTP message should be delivered to meet the delivery time QoS limit.

SMTP_DELIVERY_QOS_PERCENT

The percentage of local SMTP messages that need to meet the delivery time QoS limit at 100% load in a valid benchmark run.

SIMPLE_COMMAND_QOS_SECONDS

The number of seconds within which the mail server should respond to a simple mail command.

SIMPLE_COMMAND_QOS_PERCENT

The percentage of simple mail commands that need to pass the QoS requirement at 100% load in a valid benchmark run.

COMPLEX_COMMAND_QOS_SECONDS

The number of seconds within which the mail server should respond to a complex mail command.

COMPLEX_COMMAND_QOS_PERCENT

The percentage of complex mail commands that need to pass the QoS requirement at 100% load in a valid benchmark run.

MSG_DESTINATION_LOCAL_PERCENT

The percentage of the mail generated by local users that is destined for other local users. The rest of the locally originated mail is destined for remote users, and all mail originating remotely is destined for local users. See the White Paper for more information.

PEAK_PCT_USERS

Percent of provisioned users receiving messages during the peak hour (aka active users)

MSG_RECEIVED_PER_PEAK_HOUR

Number of messages received by active users during the peak hour

LOCAL_TO_LOCAL_PCT

Percent of total messages sent from local to remote users

REMOTE_TO_LOCAL_PCT

Percent of total messages sent from remote to local users

LOCAL_TO_REMOTE_PCT

Percent of total messages sent from local to remote users

MSG_SENT_PER_DAY

The number (on average) of messages sent by local users every day.

PEAK_LOAD_PERCENT

The percentage of the daily load occurring during the peak hour (this is the load simulated by specimap at 100% load factor).

IA_ADJUSTMENT

Initialization seed for Inter-arrival time computation

CS1_PRIMARY_PCT

Percent of CS1 sessions already logged into the mail server and should already be connected at the beginning of the load test period.

CS2_PRIMARY_PCT

Percent of CS3 sessions already logged into the mail server and should already be connected at the beginning of the load test period.

CLIENT_TYPE_DISTRIBUTION

Distribution of load generator threads that will implement the specific combination of command sequences associated with each client type found in the sample data

LEVELXFOLDER[arrayIndex]

Distribution defining probability of number of subfolders at each level. The arrayIndex value indicates the folder depth (nesting level).

LEVELXWITHSUB[arrayIndex]

Distribution defining probability that each folder has zero or more subfolders at that depth. The arrayIndex value indicates the actual folder depth (nesting level).

USE_PERCENT_FOR_VERIFICATION

Percent of active user base to use for folder and message distribution verification.

CS6A

CS6R

The number of messages Appended and messages Retrieved during CS6 fixed load test

IMAP_SECURE_NONE_PCT

IMAP_SECURE_SSL_PCT

IMAP_SECURE_STARTTLS_PCT

Percent of IMAP requests to use no transport security, percent to use IMAPS (IMAP+SSL), and percent to use STARTTLS. Must sum to 100%

SMTP_SECURE_NONE_PCT

SMTP_SECURE_SSL_PCT

SMTP_SECURE_STARTTLS_PCT

Percent of local-to-local and local-to-remote SMTP requests to use no transport security, percent to use SMTPS (SMTP+SSL), and percent to use STARTTLS. Must sum to 100%. (Remote-to-local connections never use transport security.)

IMAP_SSL_PROTOCOL

IMAP_STARTTLS_PROTOCOL

SMTP_SSL_PROTOCOL

SMTP_STARTTLS_PROTOCOL

The secure transport protocol to use with each kind of connection.

IMAP_SSL_CIPHER

IMAP_STARTTLS_CIPHER

SMTP_SSL_CIPHER

SMTP_STARTTLS_CIPHER

The secure transport cipher to use with each kind of connection.

Parameter Name	Usage
WARMUP_SECONDS	How long to run the benchmark at load before starting to gather statistics.
RUN_SECONDS	How long to run the benchmark at load while gathering statistics.
RAMPDOWN_SECONDS	How long to wait after stopping the gathering of statistics and terminating the load before starting a new test (a.k.a. cool down period)
LOAD_FACTORS	The load factors with which we scale the load in a set of consecutive test runs.
MSG_SIZE_DISTRIBUTION	The message size distribution for mail messages sent by the load generators.
MSG_RECP_DISTRIBUTION	The number of recipient’s distribution for mail messages sent by the load generators.
MIME_PART_SIZE_DISTRIBUTION	The probability distribution of the various MIME parts that make up a sing message.
MIME_TOP_PART_DISTRIBUTION	The probability that a MIME level has the sublevels indicated in the various distribution buckets.
MIME_PART_LEVEL	The probability that a message has the number of nested MIME levels indicated in the distribution buckets.
MIME_TOP_PART_COUNT	The probability distribution of the number of MIME parts at the top level
POP_RETRY_BOXES_PERCENT	The percentage of mailboxes set aside for POP retries.
DATA_RATE_DISTRIBUTION	Ignored for this benchmark
SLOW_MODEM_DATA_RATE FAST_MODEM_DATA_RATE CABLE_MODEM_DATA_RATE	Characters per second for 28.8Kbps, 56.6Kbps and cable modems. These are ignored in the IMAP benchmark.
PROTOCOL_TIMEOUT_ SECONDS	The number of seconds after which a connection is deemed to have timed out.
PROTOCOL_TIMEOUT_ PERCENT	The percentage of connections that should NOT time out.
DISCONNECT_PERCENT	The percentage of client connections that will unexpectedly disconnect (i.e. the load generators will not properly terminate the connection).
REMOTE_REFUSED_PERCENT	Percent of remote deliveries will will be refused immediatley, forcing local queueing.
REMOTE_DELAY_DISTRIBUTION	Distribution of response delays (in milliseconds) for accepted SMTP connections (versus REFUSED).
DELIVERY_TIME_CHECK_PERCENT	The percentage of local SMTP messages that should be check for delivery time Quality of Service (QoS).
SMTP_DELIVERY_QOS_SECONDS	The number of seconds within which a local SMTP message should be delivered to meet the delivery time QoS limit.
SMTP_DELIVERY_QOS_PERCENT	The percentage of local SMTP messages that need to meet the delivery time QoS limit at 100% load in a valid benchmark run.
SIMPLE_COMMAND_QOS_SECONDS	The number of seconds within which the mail server should respond to a simple mail command.
SIMPLE_COMMAND_QOS_PERCENT	The percentage of simple mail commands that need to pass the QoS requirement at 100% load in a valid benchmark run.
COMPLEX_COMMAND_QOS_SECONDS	The number of seconds within which the mail server should respond to a complex mail command.
COMPLEX_COMMAND_QOS_PERCENT	The percentage of complex mail commands that need to pass the QoS requirement at 100% load in a valid benchmark run.
MSG_DESTINATION_LOCAL_PERCENT	The percentage of the mail generated by local users that is destined for other local users. The rest of the locally originated mail is destined for remote users, and all mail originating remotely is destined for local users. See the White Paper for more information.
PEAK_PCT_USERS	Percent of provisioned users receiving messages during the peak hour (aka active users)
MSG_RECEIVED_PER_PEAK_HOUR	Number of messages received by active users during the peak hour
LOCAL_TO_LOCAL_PCT	Percent of total messages sent from local to remote users
REMOTE_TO_LOCAL_PCT	Percent of total messages sent from remote to local users
LOCAL_TO_REMOTE_PCT	Percent of total messages sent from local to remote users
MSG_SENT_PER_DAY	The number (on average) of messages sent by local users every day.
PEAK_LOAD_PERCENT	The percentage of the daily load occurring during the peak hour (this is the load simulated by specimap at 100% load factor).
IA_ADJUSTMENT	Initialization seed for Inter-arrival time computation
CS1_PRIMARY_PCT	Percent of CS1 sessions already logged into the mail server and should already be connected at the beginning of the load test period.
CS2_PRIMARY_PCT	Percent of CS3 sessions already logged into the mail server and should already be connected at the beginning of the load test period.
CLIENT_TYPE_DISTRIBUTION	Distribution of load generator threads that will implement the specific combination of command sequences associated with each client type found in the sample data
LEVELXFOLDER[arrayIndex]	Distribution defining probability of number of subfolders at each level. The arrayIndex value indicates the folder depth (nesting level).
LEVELXWITHSUB[arrayIndex]	Distribution defining probability that each folder has zero or more subfolders at that depth. The arrayIndex value indicates the actual folder depth (nesting level).
USE_PERCENT_FOR_VERIFICATION	Percent of active user base to use for folder and message distribution verification.
CS6A CS6R	The number of messages Appended and messages Retrieved during CS6 fixed load test
IMAP_SECURE_NONE_PCT IMAP_SECURE_SSL_PCT IMAP_SECURE_STARTTLS_PCT	Percent of IMAP requests to use no transport security, percent to use IMAPS (IMAP+SSL), and percent to use STARTTLS. Must sum to 100%
SMTP_SECURE_NONE_PCT SMTP_SECURE_SSL_PCT SMTP_SECURE_STARTTLS_PCT	Percent of local-to-local and local-to-remote SMTP requests to use no transport security, percent to use SMTPS (SMTP+SSL), and percent to use STARTTLS. Must sum to 100%. (Remote-to-local connections never use transport security.)
IMAP_SSL_PROTOCOL IMAP_STARTTLS_PROTOCOL SMTP_SSL_PROTOCOL SMTP_STARTTLS_PROTOCOL	The secure transport protocol to use with each kind of connection.
IMAP_SSL_CIPHER IMAP_STARTTLS_CIPHER SMTP_SSL_CIPHER SMTP_STARTTLS_CIPHER	The secure transport cipher to use with each kind of connection.