Sun accused of cheating on Java benchmarks

Pendragon Software Corporation, developer of the widely-used and recognized CaffeineMark, said yesterday that a Sun Microsystems Java compiler purposefully identifies part of CaffeineMark and then produce a misleadingscore. Sun issued a press release in October using this bogus result toclaim that its Solaris operating system runs Java faster than any otherenvironment. Sun is denying the charges.

Pendragon Software analyzed the results and found that Solaris inexplicablyscored 50 times higher than any other operating system.

"There were three possible explanations for this result: either the compiler was performing some impressive optimizations, there was a bug in the compiler, or there was a bug in the benchmark," Pendragon said in apress release.

To test the results, the company introduced a small change in the benchmark and ran it on Solaris and other platforms. For all operatingsystems except Solaris, the differences were minimal. On Solaris, however,the benchmark ran 300 times slower. The company then determined that theJava Virtual Machine in Solaris was simply pattern matching for the CaffeineMark benchmark "and performing optimisations that do not work in the general case." Further inspection led Pendragon to a block of code inthe JVM that matches the CaffeineMark bytecode