SPEC

The Standards Performance Evaluation Corporation (SPEC) is a non-profit organization that aims to produce fair, impartial and meaningful benchmarks for computers. SPEC was founded in 1988 and is financed by its member organizations which include all leading computer and software manufacturers. SPEC benchmarks are widely used today in evaluating the performance of computer systems; the results are published on the SPEC web site.

The benchmarks aim to test "real-life" situations. SPEC-WEB, for example, tests web server performance by performing various types of parallel HTTP requests, and SPEC-CPU tests CPU performance by measuring the run time of several programs such as the compiler gcc and the chess program crafty. The various tasks are assigned weights based on their perceived importance; these weights are used to compute a single benchmark result in the end.

SPEC benchmarks are written in a platform neutral programming language (usually C or Fortran), and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks.

In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars.

The Benchmarks

Current SPEC Benchmarks

• SPEC CPU2000, combined performance of CPU, memory and compiler
  • CINT2000, testing integer arithmetic, with programs such as compilers, interpreters, word processors, chess programs etc.
  • CFP2000, testing floating point performance, with physical simulations, 3D graphics, image processing, computational chemistry etc.
• SPEC WEB99, web server performance, measured by setting up a network of client machines that stress the server with parallel requests
• SPEC WEB99_SSL, web server performance using the SSL cryptography protocol.
• SPECviewperf, performance of an OpenGL 3D graphics system, tested with various rendering tasks from real applications
• SPECapc, performance of several 3D-intensive popular applications on a given system
• SPEC HPC2002, testing high end parallel computing systems with applications such as weather prediction and computational chemistry
• SPEC OMP, testing high end shared memory multiprocessor systems with OpenMP applications similar to those used for SPEC CPU2000
• SPEC JVM98, performance of a java client system running a Java virtual machine
• SPECjAppServer2001, performance of a server-side Java system based on J2EE
• SPECjAppServer2002
• SPECjAppServer2004
• SPEC JBB2000, performance of a server-side Java system
• SPEC MAIL2001, performance of a mail server, testing SMTP and POP protocols
• SPEC SFS97_R1, NFS file server throughput and response time

Future SPEC Benchmarks

• SPECappPlatform, performance of web enterprise platforms (such as J2EE and .NET)
• SPECimap2003, performance of an enterprise level mail server, testing SMTP and IMAP4 protocols
• SPEC CPU2004
• SPEC WEB2004

Retired SPEC Benchmarks

• SPEC CPU95
• SPEC HPC96
• SPEC SFS97
• SPEC WEB96