UL Delists MediaTek-Powered Phones with Suspect Benchmark Scores
April 15, 2020
No one benefits when a company tries to manipulate benchmark scores to its advantage. That's why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software.
Unfortunately, benchmark boosting continues to be a problem in the smartphone industry, despite the past scandals when manufacturers have been caught.
Last week, AnandTech reported that several smartphone models powered by MediaTek processors were producing artificially high and misleading benchmark scores. MediaTek is one of the largest fabless semiconductor companies in the world.
After reviewing AnandTech's benchmark data, we have delisted over 50 smartphone models powered by MediaTek processors. Delisted devices appear unranked, and without PCMark for Android scores, at the bottom of our popular list of the best smartphones.
Benchmark scores from delisted devices should not be used to compare models.
Which models have been delisted?
Based on AnandTech's data, we have temporarily delisted the PCMark for Android scores for all devices that use the following MediaTek chipsets:
- MediaTek Helio G90
- MediaTek Helio G70
- MediaTek Helio P95
- MediaTek Helio P90
- MediaTek Helio P65
- MediaTek Helio P60
- MediaTek Helio P20
- MediaTek Helio A22
In total, over 50 devices from more than 25 different vendors have been delisted including several recent, high-performing models such as the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.
Why has UL delisted devices with these chipsets?
AnandTech tested the Oppo Reno3 Pro with MediaTek Helio P95 chipset with the public version of PCMark for Android, available from Google Play, and a private version of the app that is not available to the public or manufacturers.
AnandTech found a big difference in the scores from the public and private apps, even though the tests in each are identical. In AnandTech's words,
"...the two versions of the benchmark did differ in their scores – but I was still aghast at the magnitude of the score delta: a 30% difference in the overall score, with up to a 75% difference in important subtests such as the writing workload."
With a bit of digging, AnandTech found a benchmark whitelist in the device's firmware files. Further investigation found similar whitelist files on a host of devices powered by MediaTek chipsets,
"Inspect the file, there we find amongst what seems to be a list of popular applications with various power management tweaks applied to them, with lo and behold, also a list of various benchmarks. We find the APK ID for PCMark, and we see that there’s some power management hints being configured for it, one common one being called a 'Sports Mode.'"
We recommend reading the full article on AnandTech, but in summary, the device uses a hidden "Sports Mode" and other settings to artificially boost its performance while running the public PCMark for Android app. With the private app, the phone performs as expected for normal, everyday use.
The difference in scores tells us that the device is simply recognizing the PCMark app by name rather than adapting to the type of work in the test. This kind of detection and optimization is forbidden by our rules for manufacturers.
Optional performance modes that can be set by the user—already available on some other manufacturers' models—are allowed under our current rules as long as they are disabled by default. A device must run the benchmark as if it were any other application.
Benchmark rules and industry standards
We contacted MediaTek with our findings, but did not receive a response. Shortly after AnandTech published its article, however, MediaTek responded with a post on its website.
While the response is mostly a point of view, we would specifically call out the following misleading claim,
"MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets."
Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an "accepted industry standard." It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device's true performance in everyday use.
Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.
As it has with similar cases in the past, we hope this delisting will help persuade MediaTek to change its approach and join the rest of the industry in adopting benchmarking best practices.
We're committed to creating benchmarks you can trust
- UL benchmarks are protected by rules for manufacturers.
- Our benchmarks have public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
Test your phone with 3DMark and PCMark for Android
3DMark is a free Android benchmark app for testing the gaming performance of your phone. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.
PCMark for Android is a free Android benchmark app that measures phone performance and battery life with tests based on common, everyday tasks such as browsing the web, editing photos and videos, and working with documents and data.
New CPU benchmarks for gamers and overclockers
June 24, 2021
New security and reporting features in Testdriver Cloud Edition
May 25, 2021
PCMark for Android adds new Work 3.0 benchmark with 64-bit support
May 12, 2021
New 3DMark benchmark for Windows, macOS, iOS and Android
May 3, 2021
New 3DMark test measures mesh shader performance
February 11, 2021
Announcing end of support for PCMark 8, 3DMark Sky Diver, and VRMark for Android benchmarks
December 16, 2020
Easy PC health monitoring for IT professionals
December 3, 2020
New benchmarks from UL help content creators choose a PC
November 30, 2020
Enhancing 3DMark benchmark results with game performance data
November 25, 2020
New 3DMark test measures pure ray-tracing performance
November 2, 2020
New cross-platform benchmark for Android, iOS and Windows
October 15, 2020
Benchmark the AI performance of Android devices
September 21, 2020
Benchmark new NVIDIA GeForce RTX 30-Series GPUs with 3DMark
September 17, 2020
Introducing Testdriver Cloud Edition—Easy benchmark automation
September 15, 2020
Benchmarks.ul.com is now available in German
July 9, 2020