-
Software
- Procyon benchmark suite
- AI Image Generation Benchmark
- AI Inference Benchmark for Android
- AI Computer Vision Benchmark
- Battery Life Benchmark
- One-Hour Battery Consumption Benchmark
- Office Productivity Benchmark
- Photo Editing Benchmark
- Video Editing Benchmark
- Testdriver
- 3DMark
- 3DMark for Android
- 3DMark for iOS
- PCMark 10
- PCMark for Android
- VRMark
- MORE...
-
Services
-
Support
-
Insights
-
Compare
-
More
UL delists Huawei phones with suspect benchmark scores
September 6, 2018
Over the last five years, more than a few smartphone manufacturers have been caught and shamed for trying to manipulate benchmark scores. That's why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software.
Earlier this week, AnandTech reported that several recent smartphone models from Huawei and its sub-brand, Honor, seem to be producing artificially high and misleading benchmark scores. Huawei is the second largest smartphone manufacturer in the world by volume according to IDC.
After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings.
Which models have been delisted?
Based on our own testing, we have delisted the following models:
- Huawei P20 Pro
- Huawei Nova 3
- Honor Play
Based on AnandTech's testing and reporting, we have also delisted:
- Huawei P20
Delisted devices appear unranked, and without scores, at the bottom of our popular list of the best smartphones. 3DMark scores from delisted devices should not be used to compare models.
Why have you delisted these models?
We tested each model with the public version of 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers.
We found that the scores from the public 3DMark app were up to 47% higher than the scores from the private app, even though the tests are identical.
With the public 3DMark app, these devices appear to use a hidden "Performance Mode" that overrides the devices' usual power profile—see the AnandTech article for more details.
The difference in scores tells us that the devices are simply recognizing the 3DMark app by name rather than adapting to the type of work in the test.
This kind of detection and optimization is forbidden by our rules for manufacturers. We contacted Huawei with our findings, and we are happy to say that they have pledged to implement a more transparent approach in a future update,
"Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to."
Optional performance modes that can be set by the user—already available on some other manufacturers' models—are allowed under our current rules as long as they are disabled by default. A device must run the benchmark as if it were any other application.
We're committed to creating benchmarks you can trust
- 3DMark is protected by rules for manufacturers.
- 3DMark has public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
Test your phone with 3DMark
3DMark is a free Android benchmark app available from Google Play. Benchmark your smartphone then compare its performance with the latest models. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.
Recent news
-
New DirectStorage test available in 3DMark
December 4, 2024
-
New Opacity Micromap test now in 3DMark for Android
October 9, 2024
-
NPUs now supported by Procyon AI Image Generation
September 6, 2024
-
Test the latest version of Intel XeSS in 3DMark
September 3, 2024
-
Introducing the Procyon Battery Consumption Benchmark
June 6, 2024
-
3DMark Steel Nomad is out now!
May 21, 2024
-
Procyon AI Inference now available on macOS
April 8, 2024
-
Procyon AI Image Generation Benchmark Now Available
March 25, 2024
-
Announcing the Procyon AI Image Generation Benchmark
March 21, 2024
-
3DMark Steel Nomad will be free for all 3DMark users.
December 20, 2023
-
Using 3DMark to measure sustained performance.
December 13, 2023
-
New NPUs supported by Procyon AI Benchmark
November 16, 2023
-
A preview of 3DMark Steel Nomad
November 10, 2023
-
Qualcomm Uses Procyon AI Benchmark at Snapdragon Summit
November 1, 2023
-
3DMark celebrates its 25th year!
October 26, 2023