 |
Standard Performance Evaluation Corporation |
   |
|
|
|
SPEC High Performance Group
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| SUSTAINING MEMBER | ASSOCIATE | SUPPORTING CONTRIBUTOR | |
| DUES | FULL | DISCOUNT | NO |
| GENERAL MEMBERSHIP VOTING | YES | NO | NO |
| PARTICIPATE IN FACE-TO-FACE MEETING, CONFERENCE CALLS, BENCHMARK DEVELOPMENT | YES | YES | YES |
| GET WORK IN PROGRESS | YES | YES | YES - if providing support for code; otherwise NO. |
| ELIGIBLE TO ELECT HPG OFFICERS | YES | YES | NO |
| ELIGIBLE TO VOTE FOR BOARD MEMBERS | YES | NO | NO |
| ELIGIBLE TO PARTICIPATE IN ANY SUBCOMMITTEE | ANY/ALL | ANY/ALL | ANY/ALL IF SUPPORT IS PROVIDED |
| COPIES OF HPG SUITES | ALL | ALL | ONLY WHERE SUPPORT PROVIDED |
| SPEC/HPG EMAIL ALIAS | ANY/ALL | ANY/ALL | 1 |
| ACCESS TO SPEC/HPG SERVERS | YES | YES | LIMITED - Controlled by Chair to selected pages only. |
| RESULT PUBLICATION ON WWW.SPEC.ORG | UNLIMITED | UNLIMITED | Dependent on new suite support; otherwise, non-member price set by HPG. |
| TERM LIMIT | Annual Renewal | Annual Renewal | ONE YR., RENEWABLE. HPG may terminate at any time. |
| RESULTS REVIEW | YES | YES | May participate in the results review session for which their results were submitted, but they have no vote. |
| ELIGIBLE FOR HPG OFFICE | YES | YES | YES |
The HPG officers include a Chairperson, Vice Chairperson, Secretary and Release Manager(s) elected from the membership. Elections for officers will be held at the beginning of each calendar year to be set by the Chair. The HPG officers serve for one year and are eligible for re-election. If any elected office becomes vacant for any reason, HPG shall duly elect a replacement to serve the remaining term.
Each HPG Sustaining Member shall have one vote in all matters. Groups of Five Associates may combine to cast one vote; the single vote cast by each group of Associates is equal to a single Sustaining Member vote. When there are fewer than five Associates available to form a combinatorial group, those Associates present will be deemed a sufficient combinatorial group to cast one vote. Supporting Contributors are not entitled to voting rights.
To conduct business there must be a quorum. The group's quorum is a simple majority of the voting membership. All HPG votes shall be open and public votes, except the election of HPG officers, which is by secret ballot. Exceptions to the secret ballot may be made if there is only one candidate per office. A voting member may vote 'yes', 'no', 'abstain', or 'pass' when the vote is first called. If the member votes 'pass', the chair will return to that member to finalize their vote as either 'yes', no', or 'abstain' after the remaining members have voted. The motion will pass if the number of 'yes' votes exceeds the number of 'no' votes. Votes of 'abstain', count only toward establishing quorum. It is the responsibility of the chair to make sure that member's reasons for voting 'no' or 'abstain' are recorded in the minutes. Absent Members will be marked as absent, unless proxies have been received.
A voting member not attending two consecutive duly noticed meetings, including face to face meetings and conference calls, will have their voting rights suspended. The suspension of voting rights will take place at the beginning of the next scheduled meeting. Voting rights are reinstated at the beginning of the second of two consecutive meetings that the member attends.
HPG's Face-to-Face meetings shall be held no less than twice a year. One meeting shall be held in conjunction with SPEC's annual meeting. One of the other meetings will accommodate HPG's membership. Teleconferences are usually held every week, for no more than one hour. The schedule for these meetings is posted on the private page.
Each benchmark suite produced by HPG includes a set of specific run and reporting rules that the licensee must follow to produce a publishable result. These basic tenets of these rules require:
Furthermore, SPEC expects that any public use of results from this benchmark suite shall be for systems and configurations that are appropriate for public consumption and comparison. Thus, it is also expected that:
SPEC reserves the right to adapt the benchmark codes, workloads, and rules of its benchmarks as deemed necessary to preserve the goal of fair benchmarking. SPEC will notify members and licensees whenever it makes changes to a benchmark or its run and reporting rules.
HPG's results review process is detailed on the HPG private page (reviewprocess.html).The standard review cycle is two weeks. In cases where the submitter has made source code changes the review cycle is a minimum of four weeks. At the beginning of the review cycle, a primary reviewer is assigned by the Chair to review the results. The primary reviewer is encouraged to offer his preliminary comments on the results within one week. At the end of the review cycle, the results are accepted or rejected by a majority vote of the committee. Once a result is accepted, the full disclosure report is posted on the SPEC web site, and once posted on the SPEC web site (http://www.spec.org/), the information contained in the result disclosure is considered public information.
If HPG votes that the submission is not compliant and should not be published on SPEC's web site, the submitter may acquiesce to the majority position and withdraw the submission. Alternately, the submitter may make an appeal to the SPEC Board of Directors. The submitter may withdraw, correct, or replace the questioned submission at any time during this process.
If minor corrections are made to a submission, e.g. documentation of tuning options or product names, which do not affect performance, then the corrected result remains on the same publication schedule as the original submission. If major corrections are made, e.g. a new run, different performance results, or different system configuration, then the corrected result starts review over as a new submission.
To assure fairness to the submitter, HPG has established the following guidelines regarding the proper use of review materials:
Members participating in the review of results submissions are expected to be familiar with the Guidelines for Handling SPEC Information, particularly Section A.7.0 dealing with Results Review (see Appendix A.)
HPG may also [vote to] undertake a re-review of a previously published result, if significant questions as to its adherence to the applicable run and reporting rules arise. HPG reserves the ability to denote a publish result as non compliant if it has been found in violation of the run and reporting rules.
The following process has been established to handle instances where run and reporting rule violations are discovered in results that have been published on SPEC's web site.
Once HPG identifies a result, published on the public web site is in violation of the benchmark's run and reporting rules, the result may re-enter a review process unless the submitter agrees with the finding of non-compliance.
If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules, the following should occur.
The results are modified:
The various SPEC benchmarks' Run and Reporting Rules have clauses that at a certain time after the first publication (on SPEC's web site or otherwise), all components of the System Under Test, (SUT), must be available for general customer shipment. The purpose of this requirement is:
For the case that this availability status cannot be maintained over a minimum time length, the following rules apply:
Either at the request of the test sponsor or as a result of a resolution, HPG can undertake a re-review of a previously published result if a performance relevant component ceases to be available for 30 days or more within 90 days after initial overall availability (i.e. initial availability of all components of the SUT). If the committee's review returns a finding that this non availability condition holds for a performance relevant component, the result is marked "Not Available" (NA) in the following way:
For results not published by SPEC, HPG can ask the publisher to withdraw the publication.
"Performance relevant" is defined in the various benchmarks' Run and Reporting Rules' reproducibility clauses.
Replacement by a new similar component with equivalent or better performance is considered acceptable and not a reason for an NA marking.
In cases where the Run Rules allow use of a beta product, this beta product or a later equivalent or better performing regular product (see clause above) must remain available.
Note: Effective for results published on or after July 1, 2002.
HPG encourages result submitters to run the benchmark on all systems. Submitters are responsible for the accuracy of their results. There may be cases where a submitter may want to submit the same results file for multiple submissions. This is acceptable if that performance is representative of those systems (e.g., just the power supply or chassis is different between the systems). A definition of what is acceptable as "electronically equivalent" systems is provided in the next section.
As part of HPG review process, the submitter should expect to be asked to justify why the systems should have the same performance. It would be within the purview of the HPG committee to ask for a rerun on the exact system, if the technical criteria are not satisfied. In cases where HPG accepts the submitter's claim of electronic equivalence, the submitter would have to add a line in the notes section of each of the result stating that the systems are electronically equivalent. For example, if a submitter submits the same result file for Model A and Model, the following note needs to be added to both the results:
Example: "Vendor's Model A and Vendor's Model B are electronically equivalent."
There are no noticeable differences in the behavior of the systems down to the level of electronic signals.
Examples of when a system is considered electronically equivalent include:
Examples of when a system is not considered electronically equivalent include:
The members of HPG are responsible for providing adequate resources to handle technical support questions regarding current benchmark suites. It is the goal of technical support to provide timely response to questions regarding the nature and usage of these benchmark suites to help facilitate testing by other SPEC benchmark licensees. The purpose of SPEC's technical support is NOT to provide OS, compiler or other vendor tool support or assistance. At the technical support person's discretion, the user may be referred to a third party, who may charge a fee for such support.
For each released benchmark, the SPEC administrator will maintain an email alias (i.e. omp2001support@spec.org). HPG may establish a quarterly rotation schedule among its active members (based on license number) to serve as technical support person. In special cases, a member may be assigned to provide support for specific subset of technical support questions for an extended period or to handle support for a specific geography. It is the Chair's responsibility is to see that questions are answered promptly and provide backup when the vendor on rotation becomes unavailable. The current support representative or chairs may assign a technical support call to another member, if they believe vendor specific knowledge may be required to resolve the call.
The SPEC administrator remains the initial point of contact for technical support questions. The administrator will make the determination as to which email alias the problem is reported to.
HPG is open to any organizations that have an interest in the use and development of standardized performance benchmarks for high performance computer systems. By joining the HPG, members may:
Members are expected to uphold the policies and principles endorsed by SPEC and the High Performance Group. This includes policies described in this document and amended by resolutions of HPG; SPEC's Articles of Incorporation and Bylaws; policies adopted by the SPEC Board of Directors; and any applicable set of benchmark run and reporting rules. Members are also expected to remain members in good standing by the prompt remittance of annual dues. Membership privileges will be suspended if dues have not been received by the SPEC administrator by March 31st. Membership privileges will be restored once dues have been paid.
Consistency and fairness are guiding principles for SPEC. To help assure that these principles are met, any organization or individual who makes public use of SPEC benchmark results must do so in accordance with the SPEC Fair Use Rule, as posted at http://www.spec.org/fairuse.html.
HPG can, after notification in writing to the Board of Directors, enlist the aid of advisors and consultants from the industry and academia. Certain organizational activities require board participation and approval. For example, the board must approve press releases, set prices for benchmark licenses, and allocate resources of SPEC administrative staff and Public Relations to assist with the non technical aspects of releasing a new benchmark suite.
This work may be facilitated by a joint committee of the board and the HPG (e.g. Planning Committee) or proposals may be brought directly to the board by the HPG Chair.
The process to approve the release of new benchmark suites consists of several steps. HPG must finalize the benchmark and prepare a beta version that can be recommended for general membership review. General membership refers to all the members of HPG and not just the members who actively worked on developing and testing the benchmark suite.
When this beta is ready, the committee must send out a final call for comments to the General Membership. A member has four weeks from that time to review and comment on the benchmark. The member should send their comments back to HPG.
At the conclusion of the final call for comments period, the voting membership of HPG will take part in a final vote to approve the benchmark for release.
SPEC is an industry consortium whose membership includes many competing for-profit corporations. Membership in SPEC is open to all persons who are directly and materially affected by SPEC's activities, and participation in SPEC activities is generally open to all members. However, in order to protect the interests of SPEC and its members, and to encourage free and open participation and discussion to further SPEC's objectives, public release of SPEC internal information may be prohibited without explicit approval from the SPEC organization concerned. Violators of this policy will be subject to the full force of the SPEC penalties process.
Since all SPEC meetings are to be conducted in compliance with all applicable laws, including anti-trust laws, the following policies shall be followed in the course of SPEC meetings, telephone meetings, and electronic exchanges: Any discussions that relate to the validity or cost of patent use should be avoided. Any discussions or any material relating to an ongoing litigation shall be avoided, except for the Board in executive session with advice of counsel to discuss matters affecting SPEC. Any discussions of pricing or issues that would violate US antitrust laws shall be avoided.
SPEC officers and the SPEC Administrator may explicitly designate certain information classified and mark it appropriately. Any member that wishes to be sure that its information is treated as SPEC Confidential should clearly label each page (electronic or hard copy) as such. It is the member's responsibility to assure that such information is appropriately labeled. Information not explicitly marked but which meets the descriptions in this policy must be taken as though it were so marked.
Classified information may become public through the authorized public release of the information by SPEC or by a member. If classified information is disclosed without authorization (leaked), that does not automatically remove its classification and relieve participants of the obligation to protect that information from further disclosure. However, if an unauthorized disclosure is widespread, then a SPEC officer may redesignate that information as public, stating the reason for so doing.
SPEC identifies two levels of internal information which must be protected by SPEC members and associates, employees, and contractors, in accordance with these policies. "SPEC Confidential" is the ordinary classification of information which should not be disclosed outside SPEC. "SPEC Confidential Need-to-Know" refers to the most sensitive information described below.
This is information that must be restricted to only those individuals who must have it in order to carry out their SPEC responsibilities. In meetings, this information is discussed only when the board adjourns to executive session. Only the author of such a document, or the SPEC President, can change the classification or the approved reader list. Examples of Confidential Need-to-Know information, includes:
Documents of this sensitivity class will always be clearly labeled "SPEC Confidential Need-to-Know" on each page. The documents may be numbered for control purposes.
SPEC Private Documents must have a cover page which serves to:
Such documents should always be carefully transported. Documents should be kept inside an envelope, clearly labeled with the approved reader's name, when not in active use. Only persons on the approved reader list are privy to the document contents.
Email should be used sparingly for communicating such information. Messages should be sent directly to the intended recipient(s) rather than to any mail alias. Messages should be retained no longer than absolutely necessary, and should be kept separate from other messages. Encryption should be employed where possible.
The following information is "SPEC Confidential":
Note that non-members of SPEC should not be on mailing aliases. Please note that SPEC offers various levels of membership to fill different needs for information access.
Confidential information is provided to SPEC members through their designated primary SPEC representatives, and those representatives of SPEC subcommittees designated by the member's primary SPEC representative. It is the responsibility of those representatives to determine who within a member company should have access to the information in order to carry out the member's SPEC activities; and it is their responsibility to ensure that anyone who does have access to SPEC Confidential information understands and agrees to abide by these guidelines. A member's SPEC representatives should use whatever internal controls are appropriate in their company to ensure that the member complies with SPEC's policies on confidentiality.
Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The Board is responsible to carry out this policy.
The annual meeting of the SPEC membership must be open to any interested party with a legitimate interest in the work of the Corporation, including the press. Other meetings of each SPEC group and committee must be open to representatives of every member of that group. It is expected that meetings of any SPEC group will also be open to members of any other SPEC group, to encourage wider participation and exchange of ideas. However, a SPEC group may set a more restrictive policy for some or all meetings, subject to Board review. Interested parties who are not SPEC members, but who have a legitimate interest in its work, may be invited to its meetings by the SPEC group holding the meeting.
Press representatives who are not SPEC members are generally excluded from SPEC meetings other than the annual meeting, unless the chair of the SPEC group or subcommittee, invites the press to attend. If the press is in attendance, the chair should announce press attendance. The press is encouraged to use such attendance opportunities to gain background information in technical areas of performance benchmarking. SPEC is primarily an engineering organization which depends on frank and open exchange and debate of views. Views expressed in SPEC meetings are not expected necessarily to reflect official positions of SPEC member companies and organizations. SPEC asks that the press show restraint in any coverage based on SPEC meetings, so as not to jeopardize our open exchange of views.
SPEC committees may apply restrictions on the use of audio recording, video recording, or photography equipment where they may impede free discussion, or where they are disruptive. Such restrictions should be clearly identified, in advance, to attendees.
Benchmark results submitted for SPEC review, as for publication on the SPEC web site, are considered SPEC Confidential. This includes the benchmark result, the configuration, availability dates, and tuning notes. Within a member company this information should be shared with only those people necessary to provide a thorough review of the results, per paragraph 5.0.
Information which the submitting company makes public is not considered confidential. E.g., a company may issue a press release or briefing listing performance of new systems and, around the same time, submit those benchmark results for SPEC publication. In this case, details included in the full SPEC disclosure under review which were not included in the company's public announcement, are not considered SPEC Confidential because the run rules of each benchmark require the full SPEC disclosure to be publicly available.
If results are later published, then the result is public. If results are withdrawn or not accepted during SPEC review and are not published, then they remain SPEC Confidential. E.g., if a company submits results that are not accepted for SPEC publication, you may not make public use of that fact.
New and unique optimization techniques that appear in a result disclosure should not be utilized by another member until the original submission has completed its review cycle. The subcommittee may decide to suspend a result in review should this guideline not be followed. Trying to surpass a result currently under review is considered common practice and is not prohibited. Published information found in press releases, public web sites, manuals, etc. can be used at any time.
More restrictive confidentiality rules may be applied by the SPEC groups on the occasion of results review for SPEC's announcement of a new benchmark suite, in addition to all of the preceding rules.
Press releases from SPEC must be approved by the SPEC group concerned, and by the Board.
SPEC members may occasionally be interviewed by the press, e.g. on the subject of newly released benchmarks. Normally SPEC's PR consultant will participate in such interviews in order to see that SPEC is treated fairly, and to assure all SPEC members that information is being represented in an unbiased manner. The Board Press Committee may on a case by case basis, waive this requirement, based on such factors as: degree of controversy of the subject, potential for appearance of conflict of interest by the spokesman, legal advice, advice from SPEC's PR consultant, recommendation of steering committee and technical committees and their chairs, experience of spokesman, experiences with the reporter, etc.
One of the principal tenets of SPEC benchmarks is that the implementation of the System under Test (SUT) must be generally available, documented and supported by the providing vendor. The run and reporting rules for each benchmark includes this statement. The available, documented, and supported trio is frequently referred to with the single term: general availability.
The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. It is also intended to ensure that the benchmarked system does not contain so-called benchmark specials which improve benchmark performance but fail one or more of the tests of general availability. These tests include:
1. Has the system/component shipped or has the vendor committed to shipping the finished product within a time specified by the specific benchmark's run rules, usually 3 to 6 months from the date the result was made public?
A finished product will have the vendor's endorsement that the product is suitable for production use in an environment comparable to the one represented by the benchmark. Typically this version will be designated as the final released version and not designated as a prototype or a preliminary release (i.e. alpha, beta, field test, etc.).
Shipping the product refers to the vendor's standard process for getting said finished product into the hands of the end-user base. This can include putting it in a box and physically transporting it to end-user sites or simply advising the customer base that the product can be obtained on demand, for example by having an announced URL where the product can be downloaded or purchase information can be found.
The commitment to ship a product within a specific time frame provided in a results disclosure must be taken as a public statement of intent. Any potential end-user should be able to confirm the vendor's intent to ship the finished product within the stated time frame and that information should not be considered secret once the result has been publicly disclosed.
Shipping a product generally requires that one or more instances of the final product be in end-user hands or enroute to end-users with delivery imminent and based on shipping method used.
2. Is the product documented such that there are written descriptions on the product, how to install it, how to configure it and how and when to use it?
The product documentation should be consistent with the deployment of the product in the benchmark such that there are no undocumented features employed solely for the benchmark. Availability of documentation should be concurrent with product availability.
3. Is the product supported such that the vendor has established clear expectations and procedures with its end-users with regards to problem resolution? Also, is the level of support reasonable for the type of product?
Levels of support can clearly have a wide range depending on the type of product. An end-user acquiring a supported product will have access to some method of problem resolution consistent with the product's importance within the system. An inadequately supported product would be one that most consumers of that type of product could not afford either due to excessive costs imposed by the vendor for support or due to the risk involved in using a product with insufficient recourse to problem resolution.
The requirement for support does not preclude the use of open source software products or other products where support is typically acquired separately.
The requirement for support is present to prevent vendors from deploying benchmark specials that may seemingly meet availability and documentation tests but which are subtly discouraged from practical and common use by imposing unusually high costs and/or lengthy or inadequate methods for problem resolution.
4. Does the product meet the vendor's own definition of general availability, in addition to all the other tests specified by SPEC?
Many vendors have their own internal processes for product development that set specific requirements for products to be considered generally available. If so, those requirements must be met for the product to be considered generally available by SPEC, in addition to the tests listed here. If the vendor has no such definition, then all other tests specified by SPEC must still be met.
A vendor is not required to discuss or disclose its internal processes to SPEC. However the SPEC representative is responsible to identify the internal definition most closely matching SPEC's intent of general availability, and applying that definition to any statements to SPEC regarding general availability. Note that the vendor may use different terms, e.g. full customer availability, open shipping status, phase 4 exit, etc.
5. Does the vendor represent the product to customers as being generally available?
This is not a clear cut test, as products may have several different distribution channels, and a product may not become generally available in all channels simultaneously. However, it is an indication that a product may not be generally available if the vendor specifically says in some venue that it is not. E.g.:
It is an indication that a product may be generally available if the vendor specifically says so in some venue. E.g.,
In case of conflicting information, SPEC will weigh the preponderance of the evidence with the standard that it show the product to be generally available in some channel.
The rules for each benchmark may have slightly different requirements related to general availability and may specify other types of benchmark specials to avoid. The overall goal of ensuring fair benchmarking for the vendors submitting results and the end-users reviewing those results remains the same.
Note: The term "benchmark special" refers to any optimization which only improves benchmark performance.
