News from September 2018

Two new benchmarks coming to 3DMark

September 20, 2018


3DMark for Windows

Every year, we add new benchmark tests to 3DMark. Each test is designed for a specific class of hardware or a particular API.

Since 2013, we've added tests that use DirectX 12, tests that run at 4K, tests for notebooks, tablets, and smartphones, as well as new features like stress testing and an API Overhead test.

Today, we'd like to tell you about two new tests that will take 3DMark into new areas. One features real-time ray tracing. The other is for the new category of Always Connected PCs and other mobile computing platforms.

3DMark Night Raid Benchmark

3DMark Night Raid

Night Raid is a DirectX 12 benchmark for PCs with integrated graphics. It is an ideal test for laptops, notebooks, tablets, and other mobile computing devices. You will also be able to use it to benchmark the latest low-power, Always Connected PCs powered by Windows 10 on ARM.

Night Raid will come to 3DMark in early October.


3DMark ray tracing benchmark (working title)

Later this year, we will release a new 3DMark benchmark test that combines real-time ray tracing with existing techniques to enhance reflections and other effects. The benchmark will run on any system that supports the DirectX Ray Tracing API.

The new benchmark is being developed with input from our Benchmark Development Program partners including AMD, Intel, NVIDIA, and other leading technology companies. We are working especially closely with Microsoft to create a first-class implementation of the DirectX Ray Tracing API.

We expect to release this new test in Q4, 2018.

About 3DMark

3DMark is a benchmarking app for gamers, overclockers and system builders who want to get more out of their hardware. With its wide range of tests, 3DMark has everything you need to test the gaming performance of your PC. Find out more.



UL and Huawei issue joint statement about benchmark optimization

September 7, 2018


Yesterday, we delisted several Huawei smartphones from our performance rankings for breaking our rules governing benchmark detection and optimization.

UL is committed to working with industry partners, and we have had open conversations with Huawei about this issue both before and since making our announcement.

Today, we are happy to announce that we are issuing a joint statement with Huawei that adds detail to Huawei's proposed solution and outlines a commitment to working together to develop future benchmarks tests.

We strongly believe that working with a wide range of industry partners is the best way to create benchmarks that are accurate, relevant, and fair.

We would like to commend Huawei for its positive decision to address the issue, and we are looking forward to working with them to create our next generation of smartphone benchmark tests.

Joint statement from Huawei and UL

"Huawei and UL (creators of 3DMark) have held comprehensive discussions on benchmarking practices this week, and have reached a positive agreement on the next steps in working together.

"In the discussion, Huawei explained that its smartphones use an artificial intelligent resource scheduling mechanism. Because different scenarios have different resource needs, the latest Huawei handsets leverage innovative technologies such as artificial intelligence to optimize resource allocation in a way so that the hardware can demonstrate its capabilities to the fullest extent, while fulfilling user demands across all scenarios.

"UL understands the intent of Huawei’s approach but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require the device to run the benchmark as if it were any other application.

"Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device.

"Huawei and UL have also discussed current common benchmark testing methodologies in general.  UL and Huawei would like to participate in an industry movement to develop benchmarking standards that best serve the needs of manufacturers, press, and consumers.

"To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode."


###


UL delists Huawei phones with suspect benchmark scores

September 6, 2018


Over the last five years, more than a few smartphone manufacturers have been caught and shamed for trying to manipulate benchmark scores. That's why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software. 

Earlier this week, AnandTech reported that several recent smartphone models from Huawei and its sub-brand, Honor, seem to be producing artificially high and misleading benchmark scores. Huawei is the second largest smartphone manufacturer in the world by volume according to IDC

After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings. 

Which models have been delisted?

Based on our own testing, we have delisted the following models:

  • Huawei P20 Pro
  • Huawei Nova 3
  • Honor Play

Based on AnandTech's testing and reporting, we have also delisted:

  • Huawei P20

Delisted devices appear unranked, and without scores, at the bottom of our popular list of the best smartphones. 3DMark scores from delisted devices should not be used to compare models. 

Why have you delisted these models?

We tested each model with the public version of 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers. 

Chart comparing 3DMark Sling Shot Extreme benchmark scores

We found that the scores from the public 3DMark app were up to 47% higher than the scores from the private app, even though the tests are identical. 

With the public 3DMark app, these devices appear to use a hidden "Performance Mode" that overrides the devices' usual power profile—see the AnandTech article for more details.

The difference in scores tells us that the devices are simply recognizing the 3DMark app by name rather than adapting to the type of work in the test. 

This kind of detection and optimization is forbidden by our rules for manufacturers. We contacted Huawei with our findings, and we are happy to say that they have pledged to implement a more transparent approach in a future update,

"Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to."

Optional performance modes that can be set by the user—already available on some other manufacturers' models—are allowed under our current rules as long as they are disabled by default. A device must run the benchmark as if it were any other application.

We're committed to creating benchmarks you can trust

Test your phone with 3DMark

3DMark is a free Android benchmark app available from Google Play. Benchmark your smartphone then compare its performance with the latest models. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.

Download 3DMark Android benchmark from Google Play


 



Highlights

Archive

UL and the UL logo are trademarks of UL LLC © 2018 All Rights Reserved.  |   Privacy Policy  |  Cookie Policy  |  Terms and conditions