SPECweb2009 Release 1.20 Backend Simulator (BeSim) Design Document

Version 1.20, Last modified 2010-04-29

Overview

SPECweb2009 includes new workloads to represent web server performance for applications such as online banking and e-commerce.  In real life the web servers supporting these applications would need to communicate with backend application and database servers.  The current literature on these Internet applications show an architecture where the web server or web server farm interfaces with an application server or servers via web services protocols (e.g. SOAP over HTTP); the application server connects to the database server.   Since the WEB subcommittee's focus is on measuring the performance of the web server, the SPECweb2009 benchmark needs to include a lightweight and efficient Backend Simulator (BeSim) to stand in for the Application and Database server portion of the solution architecture.  The Backend Simulator must be able to behave like a perfectly sized backend system for the SPECweb2009 web server under test (SUT) and not become a bottleneck that would effect the measurement of the SUT.

The Backend Simulator runs on one or more systems that are accessible by the system under test (SUT) and by the prime client.   The workload specific web server API code interfaces with BeSim via HTTP requests to keep the simulation more true to life and keep the processing required to something familiar to people who've worked with web servers and SPECweb99.  The workload specific API code sends requests to BeSim to obtain data that is typically acquired from backend systems such as a user's bank balance or a registered user's shopping preferences.   The data in the BeSim response is then used by the API code to complete the page to be returned as the response to a client's request.
 

Basic Goals and Functions for BeSim

• BeSim must be portable to the same platforms used for client load generators.
• BeSim should use a common code base.
• BeSim should allow new workloads to be easily supported.
• BeSim functions and processing must be keep as simple as possible to ensure that its does not become a bottleneck for the web server and effect the measurement of the SUT.
  • The total time spent in the API code setting up the request to BeSim plus the time spent parsing the BeSim response and formatting the page to be returned to the client should be an order of magnitude larger than any time spent in BeSim processing a request.   So while there is additional load on the server due to traffic between the SUT and BeSim, traffic to BeSim must not become backed up and delay its response beyond that expected for a given request. 
• BeSim must be synchronized by the prime client.  The prime client supplies the required test parameters to BeSim during the setup phase so that BeSim can generate the required responses to match the specified workload.
• BeSim logs (via web server access logging) sufficient information on the requests it receives so that it can be determined whether the number of requests handled by BeSim was within the expected range for the test. 
• BeSim and the prime client may exchange some unique pieces of data (such as a small random number and a larger random number) that the SUT would not share.
  • BeSim could use this data to position and set some portion of the response to the API code.  The API code would include that somewhere in its response to back to the client.  The client's validation code could know what to look for and verify that the data had been obtained from BeSim. 
• BeSim listens on a specified port (preferably a non-standard port, i.e. not 80 ) for HTTP requests from the prime client or the SUT.
  • Typically, these requests would be a modified form of the original dynamic request sent by the client.  The API code may append additional parameters to the query string as needed.
• BeSim returns an HTTP response; the data being returned may be encoded in the entity body of the response. For example:

    - HTTP/1.1 200 OK

    Connection: close
    Content-Length: 481
    Content-Type: text/html
  
    <html>
    <head><title>SPECweb2009 BESIM</title></head>    
    <body>
    <p>SERVER_SOFTWARE = Apache/2.0.53    
    <p>REMOTE_ADDR = 192.168.1.4    
    <p>SCRIPT_NAME = /fcgi-bin/besim_fcgi.fcgi    
    <p>QUERY_STRING = 1&4&58&3    
    <pre>    
    0    
    0000002058&/www/bank/images/user0000000058/CIF000003&    
    /www/bank/images/user0000000058/CIB000003  
    </pre>   
    </body></html>  
    

• BeSim supports init commands sent by the prime client for each workload. For Example:
  
  • To initialize BeSim for the workload and set values for validation of response:

    GET besim?<workload#>&<Command#>[&Parameter_n=Value...]\
    [&Unique1=Value1&Unique2=Value2]
    
    Example: B
    GET besim?1&0&1097157010&1&2000&200&/www/bank/images&0
    
    

BeSim Implementation

The key requirements for BeSim's implementation are portability, easy of maintenance, and easy of extension to support new workloads.  To meet these requirements, BeSim makes use of existing HTTP software and is implemented as a dynamic API module.  This approach leverages already developed HTTP software that knows all about handling multiple HTTP requests and allows for rapid prototyping for new workloads.  

BeSim consists of a common set of core code that is used within one of several common web server API's.  If a faster BeSim is needed, users can either tune the web server software or select a faster web server.

How BeSim Works

The BeSim system is connected to the Prime Client and the SUT, when a SPECweb2009 test is started, the Prime Client sends an initialization message in the form of a dynamic HTTP request with a query string containing all the parameters the BeSim software needs for the test.

Besim initializes its data structures and writes out a small file (i.e. /tmp/besim_<workload>.globals) so if there are multiple copies of BeSim (such as multiple fast-cgi instances) they can synchronize. 

A value exchanged during initialization is used by BeSim to construct some of the data in the responses. The clients, which also have this value use it to validate the dynamic data returned by the SUT in its responses to the clients dynamic requests.

Most dynamic HTTP/HTTPS requests sent by the clients to the SUT require that the SUT send a request to BeSim to acquire one or more pieces of data.  The SUT uses this data to construct a response to the client's request.   The lightweight nature of BeSim allows it to respond quickly to these requests so that the overhead does not significantly impact the SUT. 

A standardized request and response format is used between the SUT and BeSim.  All requests sent to BeSim are in the form:

        GET /<api-dir>/<besim_api>?<Workload_Type>&<Command_Type>&<param=value>[&<param=value>]

Responses returned by BeSim are in the form:

        HTTP 200 OK
        Cache-control: no-store
        Content-type: text/html
        Content-Length: [length of all return text - excluding headers]

        <html>
        <head><title>SPECweb2009 BESIM</title></head>
        <body>
        <p>SERVER_SOFTWARE = [ServerSoftware]
        <p>REMOTE_ADDR = [RemoteAddr]
        <p>SCRIPT_NAME = [ScriptName]
        <p>QUERY_STRING = [QueryString]
        <pre>
        [BeSIM MessageText]
        </pre>
        </body></html>

The BeSim MessageText includes a status value of "0\n" for OK and "1\n" for ERROR.  If  BeSim returns an Error it also returns an Error Message to indicate why it failed.  If the request was OK, the rest of the Message Text contains the data for the response.  The workload specific BeSim request-response sequences are included in the appendixes below.

Building BeSim

To build BeSim, the tester must select and install web server software on the system to server as the BeSim Box.  They must also select a compatible BeSim API.  SPECweb2009 supplies Fast-CGI, ISAPI, and NSAPI implementations.  The directions for building each API are included with the BeSim software in the Make_Readme and in the User documentation.   The kit also includes a set of perl scripts to allow simple testing of BeSim operations.

Appendix A.  BeSim For Banking Request-Response Sequences

Appendix B.  BeSim For Ecommerce Request-Response Sequences

Appendix C.  BeSim For Support Request-Response Sequences

Copyright © 2005-2010 Standard Performance Evaluation Corporation.  All rights reserved.