HTC One (M8) removed from 3DMark benchmark rankings
April 1, 2014
The new HTC One (M8) has the potential to be one of fastest Android devices in our 3DMark benchmark performance rankings. Instead, it appears delisted and without a score, below 1000 other models at the bottom of our rankings.
Unlike other benchmark makers who carelessly list and leak scores from unreleased hardware without verification, our 3DMark Best Smartphones and Tablets list only includes publicly available models that have been tested and reviewed by the press. Models that fail to comply with our benchmark rules are delisted, ensuring that our rankings are fair and accurate.
We have delisted the new HTC One (M8) from our 3DMark benchmark rankings because it does not comply with our rules for manufacturers. Specifically,
"The platform may not detect the launch of the benchmark executable. The platform must not...modify the usual functioning of the platform based on the detection of the benchmark."
The problem is illustrated by this chart from AnandTech's review of the HTC One (M8):
Image credit: AnandTech
The chart shows how the HTC One (M8) behaves when running the public version of 3DMark in the Play Store compared with running a renamed, but otherwise identical version of 3DMark not available to the public or manufacturers.
As you can see, the average CPU frequency is about 15% higher when running the public version of 3DMark, despite the two tests having identical workloads. The blue line shows the HTC One's real performance without the benchmark detection.
Surprisingly, HTC openly admits the HTC One (M8) uses benchmark optimizations, saying,
"...our engineers optimize in certain scenarios to produce the best possible performance. If someone would like to get around this benchmarking optimization there are ways to do so, but we think most often this will not be the case."
In fact, though the optimization can be enabled and disabled manually by users for other apps, it appears to be automatically and unavoidably enabled when running 3DMark, which makes it impossible to measure the HTC One's real-world performance.
HTC defends its optimizations saying, "Benchmarking tests look to determine maximum performance of the CPU and GPU."
This is a common misconception. The purpose of 3DMark, as with other Futuremark benchmarks, is not to measure maximum performance but to measure real-world performance. To do that, a device must run the benchmark as if it were any other application.
When a device detects 3DMark, and modifies its behaviour as a result, the benchmark scores are no longer accurate and cannot be used to make fair comparisons between devices.
We recently commended Samsung for removing the benchmark boosting effect from the Galaxy S IV and the Galaxy Note III. We hope this delisting will help persuade HTC to join them and realize the benefits of being fair and honest with their customers.
3DMark is a free Android benchmark app available from Google Play:
3DMark Device Channel
Compare over 1000 smartphones and tablets running Android, iOS and Windows:
Announcing end of support for PCMark 8, 3DMark Sky Diver, and VRMark for Android benchmarks
December 16, 2020
Easy PC health monitoring for IT professionals
December 3, 2020
New benchmarks from UL help content creators choose a PC
November 30, 2020
Enhancing 3DMark benchmark results with game performance data
November 25, 2020
New 3DMark test measures pure ray-tracing performance
November 2, 2020
New cross-platform benchmark for Android, iOS and Windows
October 15, 2020
Benchmark the AI performance of Android devices
September 21, 2020
Benchmark new NVIDIA GeForce RTX 30-Series GPUs with 3DMark
September 17, 2020
Introducing Testdriver Cloud Edition—Easy benchmark automation
September 15, 2020
Benchmarks.ul.com is now available in German
July 9, 2020
UL Delists MediaTek-Powered Phones with Suspect Benchmark Scores
April 15, 2020
PCMark 10 Storage Benchmarks add new language support
February 20, 2020
PCMark 10 benchmark shows performance improvement with new Microsoft Edge browser
February 6, 2020
3DMark is now available in Traditional Chinese
January 21, 2020
Announcing end of support for 3DMark 11, PCMark 7, Powermark, 3DMark Cloud Gate, and 3DMark Ice Storm
December 19, 2019