No one benefits when a company tries to manipulate benchmark scores to its advantage. That's why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software.
Unfortunately, benchmark boosting continues to be a problem in the smartphone industry, despite the past scandals when manufacturers have been caught.
Last week, AnandTech reported that several smartphone models powered by MediaTek processors were producing artificially high and misleading benchmark scores. MediaTek is one of the largest fabless semiconductor companies in the world.
After reviewing AnandTech's benchmark data, we have delisted over 50 smartphone models powered by MediaTek processors. Delisted devices appear unranked, and without PCMark for Android scores, at the bottom of our popular list of the best smartphones.
Benchmark scores from delisted devices should not be used to compare models.
Which models have been delisted?
Based on AnandTech's data, we have temporarily delisted the PCMark for Android scores for all devices that use the following MediaTek chipsets:
- MediaTek Helio G90
- MediaTek Helio G70
- MediaTek Helio P95
- MediaTek Helio P90
- MediaTek Helio P65
- MediaTek Helio P60
- MediaTek Helio P20
- MediaTek Helio A22
In total, over 50 devices from more than 25 different vendors have been delisted including several recent, high-performing models such as the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.
Why has UL delisted devices with these chipsets?
AnandTech tested the Oppo Reno3 Pro with MediaTek Helio P95 chipset with the public version of PCMark for Android, available from Google Play, and a private version of the app that is not available to the public or manufacturers.
AnandTech found a big difference in the scores from the public and private apps, even though the tests in each are identical. In AnandTech's words,
"...the two versions of the benchmark did differ in their scores – but I was still aghast at the magnitude of the score delta: a 30% difference in the overall score, with up to a 75% difference in important subtests such as the writing workload."
With a bit of digging, AnandTech found a benchmark whitelist in the device's firmware files. Further investigation found similar whitelist files on a host of devices powered by MediaTek chipsets,
"Inspect the file, there we find amongst what seems to be a list of popular applications with various power management tweaks applied to them, with lo and behold, also a list of various benchmarks. We find the APK ID for PCMark, and we see that there’s some power management hints being configured for it, one common one being called a 'Sports Mode.'"
We recommend reading the full article on AnandTech, but in summary, the device uses a hidden "Sports Mode" and other settings to artificially boost its performance while running the public PCMark for Android app. With the private app, the phone performs as expected for normal, everyday use.
The difference in scores tells us that the device is simply recognizing the PCMark app by name rather than adapting to the type of work in the test. This kind of detection and optimization is forbidden by our rules for manufacturers.
Optional performance modes that can be set by the user—already available on some other manufacturers' models—are allowed under our current rules as long as they are disabled by default. A device must run the benchmark as if it were any other application.
Benchmark rules and industry standards
We contacted MediaTek with our findings, but did not receive a response. Shortly after AnandTech published its article, however, MediaTek responded with a post on its website.
While the response is mostly a point of view, we would specifically call out the following misleading claim,
"MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets."
Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an "accepted industry standard." It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device's true performance in everyday use.
Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.
As it has with similar cases in the past, we hope this delisting will help persuade MediaTek to change its approach and join the rest of the industry in adopting benchmarking best practices.
We're committed to creating benchmarks you can trust
- UL benchmarks are protected by rules for manufacturers.
- Our benchmarks have public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
Test your phone with 3DMark and PCMark for Android
3DMark is a free Android benchmark app for testing the gaming performance of your phone. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.
PCMark for Android is a free Android benchmark app that measures phone performance and battery life with tests based on common, everyday tasks such as browsing the web, editing photos and videos, and working with documents and data.
