3DMark for Android: Performance Preview

As I mentioned in our coverage of GL/DXBenchmark 2.7, with the arrival of Windows RT/8 we'd finally see our first truly cross-platform benchmarks. Kishonti was first out of the gate, although Futuremark was first to announce its cross platform benchmark simply called 3DMark.

Currently available for x86 Windows 8 machines, Futuremark has Android, iOS and Windows RT versions of 3DMark nearing release. Today the embargo lifts on the Android version of 3DMark, with iOS and Windows RT to follow shortly.

Similar to the situation with GL/DXBenchmark, 3DMark not only spans OSes but APIs as well. The Windows RT/8 versions use DirectX, while the Android and iOS versions use OpenGL ES 2.0. Of the three major tests in the new 3DMark, only Ice Storm is truly cross platform. Ice Storm uses OpenGL ES 2.0 on Android/iOS and Direct3D feature level 9_1 on Windows RT/8.

The Android UI is very functional and retains a very 3DMark feel. There's an integrated results brower, history of results and some light device information as well:

There are two options for running Ice Storm: the default and extreme presets.

Both benchmarks are rendered to an offscreen buffer at 720p/1080p and then scale up to the native resolution of the device being tested. This is a very similar approach we've seen by game developers to avoid rendering at native resolution on some of the ultra high resolution tablets. The beauty of 3DMark's approach here is the fact that all results are comparable, regardless of a device's native resolution. The downside is we don't get a good idea of how some of the ultra high resolution tablets would behave with these workloads running at their native (> 1080p) resolutions.

Ice Storm is divided into two graphics tests and a physics test. The first graphics test is geometry heavy while the second test is more pixel shader intensive. The physics test, as you might guess, is CPU bound and multithreaded.

Before we get to the results, I should note that a number of devices wouldn't complete the tests. The Intel based Motorola RAZR i wouldn't run, the AT&T HTC One X (MSM8960) crashed before the final score was calculated so both of those devices were excluded. Thankfully we got the Galaxy S 3 to complete, giving us a good representative from the MSM8960/Adreno 225 camp. Thermal throttling is a concern when running 3DMark. You have to pay close attention to the thermal conditions of the device you're testing. This is becoming something we're having to pay an increasing amount of attention to in our reviews these days.

Graphics Test 1

Ice Storm Graphics test 1 stresses the hardware’s ability to process lots of vertices while keeping the pixel load relatively light. Hardware on this level may have dedicated capacity for separate vertex and pixel processing. Stressing both capacities individually reveals the hardware’s limitations in both aspects.

In an average frame, 530,000 vertices are processed leading to 180,000 triangles rasterized either to the shadow map or to the screen. At the same time, 4.7 million pixels are processed per frame.

Pixel load is kept low by excluding expensive post processing steps, and by not rendering particle effects.

Although the first graphics test is heavy on geometry, it features roughly 1/4 the number of vertices from GL/DXBenchmark 2.7's T-Rex HD test. In terms of vertex/triangle count, even Egypt HD is more stressful than 3DMark's first graphics test. That's not necessarily a bad thing however, as most Android titles are no where near as stressful as what T-Rex and Egypt HD simulate.

Among Android smartphones, Qualcomm rules the roost here. The Adreno 320 based Nexus 4 and HTC One both do very well, approaching 60 fps in the first graphics test. The Mali 400MP4, used in the Galaxy Note 2 and without a lot of vertex processing power, brings up the rear - being outperformed by even NVIDIA's Tegra 3. ARM's Mali-T604 isn't enough to pull ahead in this test either; the Nexus 10 remains squarely behind the top two Adreno 320 based devices.

Graphics Test 2

Graphics test 2 stresses the hardware’s ability to process lots of pixels. It tests the ability to read textures, do per pixel computations and write to render targets.

On average, 12.6 million pixels are processed per frame. The additional pixel processing compared to Graphics test 1 comes from including particles and post processing effects such as bloom, streaks and motion blur.

In each frame, an average 75,000 vertices are processed. This number is considerably lower than in Graphics test 1 because shadows are not drawn and the processed geometry has a lower number of polygons.

As you'd expect, shifting to a more pixel shader heavy workload shows the Galaxy Note 2 doing a lot better - effectively tying the Tegra 3 based HTC One X+ and outperforming the Nexus 7. The Mali-T604 continues to, at best, tie for third place here. Qualcomm's Adreno 320 just seems to deliver better performance in 3DMark for Android.

The overall score pretty much follows the trends we saw earlier. Qualcomm's Adreno 320 leads things (Nexus 4/HTC One), followed by ARM's Mali-T604 (Nexus 10), Adreno 225 (SGS3), Adreno 305 (One SV), Tegra 3 (One X+/Nexus 7) and finally Mali 400MP4 (Note 2). The only real surprise here is just how much better Adreno 320 does compared to Mali-T604.

Physics Test

The purpose of the Physics test is to benchmark the hardware’s ability to do gameplay physics simulations on CPU. The GPU load is kept as low as possible to ensure that only the CPU’s capabilities are stressed.

The test has four simulated worlds. Each world has two soft bodies and two rigid bodies colliding with each other. One thread per available logical CPU core is used to run simulations. All physics are computed on the CPU with soft body vertex data updated to the GPU each frame. The background is drawn as a static image for the least possible GPU load.

The Physics test uses the Bullet Open Source Physics Library.

The physics results give us an indication of just how heavily threaded this benchmark is. The quad-core devices are able to outperform the dual-core Cortex A15 based Nexus 10, despite the latter having far better single threaded performance. The Droid DNA/Optimus G vs. Nexus 4 results continue to be a bit odd, perhaps due to the newer drivers included in the Nexus 4's use of Android 4.2 vs. 4.1.2 for the other APQ8064 platforms.

The overall Ice Storm results don't paint a vastly different picture from what we've seen thus far:

We will likely have to revisit this as non-Nexus devices get updated to Android 4.2, especially given the strange delta between the Nexus 4 and Optimus G/Droid DNA. When the One gets updated to Android 4.2.x, it'll be interesting to see if it pulls ahead of the Nexus 4 in 3DMark.

It's also interesting to note that 3DMark seems to value the Adreno 320 more than ARM's Mali-T604 in the Nexus 10. I tossed results from the HTC One into the T-Rex HD benchmark data from GL/DXBenchmark 2.7 and found that the two benchmarks seem to agree on this point as well:

The magnitude of difference between the One (Adreno 320) and Nexus 10 (Mali-T604) isn't as large, but we are looking at two very different benchmarks here.

3DMark includes an optional demo run that will generate a score as well (although these results aren't included in the overall Ice Storm score):

Just for kicks I ran the extreme (1080p, high quality textures and post processing) presets on a couple of devices to see how the numbers changed:

Moving up to the Extreme presets definitely increases the workload on these devices. It's important to note that today's flagships are close to running 3DMark at extreme presets at 30 fps in some cases. On average, we're looking at low to mid 20 fps for the best of the best on Android today.

3DMark for Android seems to be another good data point to have. While the holy grail remains a good collection of actual games with integrated benchmarks representative of game play, at least having a good selection of simulated game tests is a lot better than where we were just a year ago.

The fact that all of 3DMark's benchmarks are comparable across devices regardless of display resolution is a nice touch, although I do appreciate GLBenchmark's ability to target the native resolution of any device (especially when dealing with very high res tablets). The inclusion of a dedicated physics test is very nice, however I'd like to see equal emphasis placed on single (or lightly) threaded performance as the real world tends to not be as thread happy as 3DMark's physics test implies.

Overall, the 3DMark Ice Storm workload is clearly less stressful than GL/DXBenchmark, which makes me wonder how it'll age over time. On the bright side, 3DMark will make a nice companion to GL/DXBenchmark - offering a glimpse at what GPU performance would look like in workloads closer to what is presently available, rather than forecasting what may be possible in the future. We plan on using both benchmarks in our reviews going forward.