UL delists Oppo phones with suspect benchmark scores
October 18, 2018
After Huawei was caught unfairly boosting benchmark scores last month, we were contacted by reviewers from Tech2 who had similar concerns about Oppo smartphones.
Extensive testing by Tech2 found that the flagship Oppo Find X smartphone produced artificially high and misleading benchmark scores.
UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software. The purpose of these rules is to ensure that users get benchmark results that accurately reflect the true performance of the device for apps and games.
After reviewing Tech2's benchmark results, and conducting testing in our own lab, we have decided to delist the affected models and remove them from our performance rankings.
Which models have been delisted?
Based on Tech2's testing data and reporting, we have delisted the Oppo Find X. Based on our own testing, we have delisted the Oppo F7.
The Oppo Find X was ranked #4 in our list of the Best Smartphones for 3DMark Sling Shot Extreme performance. It now appears unranked, and without a score, at the bottom of our rankings. 3DMark scores from delisted devices should not be used to compare models.
Why have you delisted these models?
Each model was tested with the public version of 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers.
We found that the scores from the public 3DMark app were up to 41% higher than the scores from the private app, even though the tests are identical.
The difference in scores tells us that the devices are simply recognizing the 3DMark app by name rather than adapting to the type of work in the test.
This kind of detection and optimization is forbidden by our rules for manufacturers.
We contacted Oppo with our findings. Oppo admits that the devices are detecting the public version of the app,
"When we detect that the user is running applications like games or 3D Benchmarks that require high performance, we allow the SoC to run at full speed for the smoothest experience.
Oppo explained how its devices manage power and performance for other apps that are not recognised,
"For unknown applications, the system will adopt the default power optimization strategy...After the user has not actively operated for 5 to 10 seconds, the device limits the system performance to 70% to 80% of the maximum performance (according to different platforms)...When there is a user operation, it will immediately cancel the performance limit, to ensure that the user experience is not affected."
In practice, this means that it is possible to improve performance in most apps by continually tapping on the screen. However, this has no effect on the public 3DMark app. Nor do we consider such tapping an accurate way to benchmark a device due to the variability in performance that it introduces.
Fortunately, Oppo also said that it is looking to improve on this approach for the benefit of its users.
"At the same time, we are working on upgrading the system, and strive to distinguish between the requirements of undetected apps or the subjective needs of users."
Under our rules, optimizing performance by detecting heavy computation demands is allowed. Simply detecting the benchmark app by name is not. A device must run the benchmark as if it were any other application.
We're committed to creating benchmarks you can trust
- 3DMark is protected by rules for manufacturers.
- 3DMark has public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
Test your phone with 3DMark
3DMark is a free Android benchmark app available from Google Play. Benchmark your smartphone then compare its performance with the latest models. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.
3DMark adds new SSD benchmark for gamers
November 10, 2021
Benchmark PC performance with Microsoft Office apps
October 4, 2021
UL benchmarks are ready for Windows 11
September 29, 2021
3DMark update adds new Steam achievements, trading cards, and more!
September 27, 2021
Reduce the risk of data loss with Testdriver Storage Health Test
August 26, 2021
Experience DirectX 12 Ultimate with 3DMark
August 25, 2021
New CPU benchmarks for gamers and overclockers
June 24, 2021
New security and reporting features in Testdriver Cloud Edition
May 25, 2021
PCMark for Android adds new Work 3.0 benchmark with 64-bit support
May 12, 2021
New 3DMark benchmark for Windows, macOS, iOS and Android
May 3, 2021
New 3DMark test measures mesh shader performance
February 11, 2021
Announcing end of support for PCMark 8, 3DMark Sky Diver, and VRMark for Android benchmarks
December 16, 2020
Easy PC health monitoring for IT professionals
December 3, 2020
New benchmarks from UL help content creators choose a PC
November 30, 2020
Enhancing 3DMark benchmark results with game performance data
November 25, 2020