They're (Almost) All Dirty: The State of Cheating in Android Benchmarks

Thanks to AndreiF7's excellent work on discovering it, we kicked off our investigations into Samsung’s CPU/GPU optimizations around the international Galaxy S 4 in July and came away with a couple of conclusions:

1) On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and

2) On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect.

The first point applied exclusively to the Exynos 5410 equipped version of the Galaxy S 4. We did a lot of digging to confirm that max GPU frequency (450MHz) was never exceeded on the Snapdragon 600 version. The second point however applied to many, many more platforms.

The table below is a subset of devices we've tested, the silicon inside, and whether or not they do a benchmark detect and respond with a max CPU frequency (and all cores plugged in) right away:

We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop. With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It's possible that older Motorola devices might've done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

Looking at the table above you’ll also notice weird inconsistencies about the devices/OEMs that choose to implement the cheat/hack/festivities. None of the Nexus do, which is understandable since the optimization isn't a part of AOSP. This also helps explain why the Nexus 4 performed so slowly when we reviewed it - this mess was going on back then and Google didn't partake. The GPe versions aren't clean either, which makes sense given that they run the OEM's software with stock Android on top.

LG’s G2 also includes some optimizations, just for a different set of benchmarks. It's interesting that LG's optimization list isn't as extensive as Samsung's - time to invest in more optimization engineers? LG originally indicated to us that its software needed some performance tuning, which helps explain away some of the G2 vs Note 3 performance gap we saw in our review.

The Exynos 5420's behavior is interesting here. Instead of switching over to the A15 cluster exclusively it seems to alternate between running max clocks on the A7 cluster and the A15 cluster.

Note that I’d also be careful about those living in glass houses throwing stones here. Even the CloverTrail+ based Galaxy Tab 3 10.1 does it. I know internally Intel is quite opposed to the practice (as I’m assuming Qualcomm is as well), making this an OEM level decision and not something advocated by the chip makers (although none of them publicly chastise partners for engaging in the activity, more on this in a moment).

The other funny thing is the list of optimized benchmarks changes over time. On the Galaxy S 4 (including the latest updates to the AT&T model), 3DMark and Geekbench 3 aren’t targets while on the Galaxy Note 3 both apps are. Due to the nature of the optimization, the benchmark whitelist has to be maintained (now you need your network operator to deliver updates quickly both for features and benchmark optimizations!).

There’s minimal overlap between the whitelisted CPU tests and what we actually run at AnandTech. The only culprits on the CPU side are AndEBench and Vellamo. AndEBench is an implementation of CoreMark, something we added in more as a way of looking at native vs. java core performance and not indicative of overall device performance. I’m unhappy that AndEBench is now a target for optimization, but it’s also not unexpected. So how much of a performance uplift does Samsung gain from this optimization? Luckily we've been testing AndEBench V2 for a while, which features a superset of the benchmarks used in AndEBench V1 and is of course built using a different package name. We ran the same native, multithreaded AndEBench V1 test using both apps on the Galaxy Note 3. The difference in scores is below:

There's a 4.4% increase in performance from the CPU optimization. Some of that gap is actually due to differences in compiler optimizations (V1 is tuned by the OEMs for performance, V2 is tuned for compatibility as it's still in beta). As expected, we're not talking about any tremendous gains here (at least as far as our test suite is concerned) because Samsung isn't actually offering a higher CPU frequency to the benchmark. All that's happening here is the equivalent of a higher P-state vs. letting the benchmark ramp to that voltage/frequency pair on its own. We've already started work on making sure that all future versions of benchmarks we get will come with unique package names.

I graphed the frequency curve of a Snapdragon 600 based Galaxy S 4 while running both versions of AndEBench to illustrate what's going on here:

We're looking at is a single loop of the core AndEBench MP test. The blue line indicates what happens naturally, while the red line shows what happens with the CPU governor optimization enabled. Note the more gradual frequency ramp up/down. In the case of this test, all you're getting is the added performance during that slow ramp time. For benchmarks that repeat many tiny loops, these differences could definitely add up. In situations where everyone is shipping the same exact hardware, sometimes that extra few percent is enough to give the folks in marketing a win, which is why any of this happens in the first place.

Even when the Snapdragon 600 based SGS4 recognizes AndEBench it doesn't seem to get in the way of thermal throttling. A few runs of the test and I saw clock speeds drop down to under 1.7GHz for a relatively long period of time before ramping back up. I should note that the power/thermal profiles do look different when you let the governor work its magic vs. overriding things, which obviously also contributes to any performance deltas.

Vellamo is interesting as all of the flagships seem to game this test, which sort of makes the point of the optimization moot:

Any wins the Galaxy Note 3 achieves in our browser tests are independent of the CPU frequency cheat/optimization discussed above. It’s also important to point out that this is why we treat our suite as a moving target. I introduced Kraken into the suite a little while ago because I was worried that SunSpider was becoming too much of a browser optimization target. The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it’s just more frustrating in mobile since there aren’t many good benchmarks to begin with. Note that pretty much every CPU test that’s been gamed at this point isn’t a good CPU test to begin with.

Don’t forget that we’re lucky to be able to so quickly catch these things. After our piece in July I figured one of two things would happen: 1) the optimizations would stop, or 2) they would become more difficult to figure out. At least in the near term, it seems to be the latter. The framework for controlling all of this has changed a bit, and I suspect it’ll grow even more obfuscated in the future. There’s no single solution here, but rather a multi-faceted approach to make sure we're ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed. Honestly this is the same list of things we do on the PC side, so we've been doing it in mobile as well.

The Relationship Between CPU Frequency and GPU Performance

On the GPU front is where things are trickier. GFXBench 2.7 (aka GLBenchmark) somehow avoids being an optimization target, at least for the CPU cheat we’re talking about here. There are always concerned about rendering accuracy, dropping frames, etc... but it looks like the next version of GFXBench will help make sure that no one is doing that quite yet. Kishonti (the makers of GFX/GLBench) work closely with all of the OEMs and tend to do a reasonable job at keeping everyone honest but they’re in a tricky spot as the OEMs also pay for the creation of the benchmark (via licensing fees to use it). Running a renamed version of GFXBench produced similar scores to what we already published on the Note 3, which ends up being a bit faster than what LG’s G2 was able to deliver. As Brian pointed out in his review however, there are driver version differences between the platforms as well as differences in VRAM sizes (thanks to the Note 3's 3GB total system memory):

Note 3: 04.03.00.125.077
Padfone: 04.02.02.050.116
G2: 4.02.02.050.141

Also keep in mind that both LG and Samsung will define their own governor behaviors on top of all of this. Even using the same silicon you can choose different operating temperatures you’re comfortable with. Of course this is another variable to game (e.g. increasing thermal headroom when you detect a benchmark), but as far as I can tell even in these benchmark modes thermal throttling does happen.

The two new targets are tests that we use: 3DMark and Basemark X. The latter tends to be quite GPU bound, so the impact of a higher CPU frequency is more marginalized, but with a renamed version we can tell for sure:

The onscreen differences make sense to me, it's the off screen results that are a bit puzzling. I'm worried about what's going on with the off-screen rendering buffer. That seems to be too little of a performance increase if the optimization was dropping frames (if you're going to do that, might as well go for the gold), but as to what is actually going on I'm not entirely sure. We'll keep digging on this one. The CPU optimization alone should net something around the 3% gain we see in the on screen test.

3DMark is a bigger concern. As we discovered in our Moto X review, 3DMark is a much more balanced CPU/GPU test. Driving CPU frequencies higher can and will impact the overall scores here.

ASUS thankfully doesn’t do any of this mess with their Padfone Infinity in the GPU tests. Note that there are still driver and video memory differences between the Padfone Infinity and the Galaxy Note 3, but we’re seeing roughly a 10% performance advantage in the overall 3DMark Extreme score (the Padfone also has a slightly lower clocked CPU - 2.2GHz vs. 2.3GHz). It's tough to say how much of this is due to the CPU optimization vs. how much is up to driver and video memory differences (we're working on a renamed version of 3DMark to quantify this exactly).

The Futuremark guys have a lot of experience with manufacturers trying to game their benchmarks so they actually call out this specific type of optimization in their public rules:

"With the exception of setting mandatory parameters specific to systems with multiple GPUs, such as AMD CrossFire or NVIDIA SLI, drivers may not detect the launch of the benchmark executable and alter, replace or override any parameters or parts of the test based on the detection. Period."

If I'm reading it correctly, both HTC and Samsung are violating this rule. What recourse Futuremark has against the companies is up in the air, but here we at least have a public example of a benchmark vendor not being ok with what's going on.

Note that GFXBench 2.7, where we don't see anyone run the CPU optimization, shows a 13% advantage for the Note 3 vs. Padfone Infinity. Just like the Exynos 5410 optimization there simply isn't a lot to be gained by doing this, making the fact that the practice is so widespread even more frustrating.

Final Words

As we mentioned back in July, all of this is wrong and really isn't worth the minimal effort the OEMs put into even playing these games. If I ran the software group at any of these companies running the cost/benefit analysis on chasing these optimizations vs. negativity in the press it’d be an easy decision (not to mention the whole morality argument). It's also worth pointing out that nearly almost all Android OEMs are complicit in creating this mess. We singled out Samsung for the initial investigation as they were doing something unique on the GPU front that didn't apply to everyone else, but the CPU story (as we mentioned back in July) is a widespread problem.

Ultimately the Galaxy Note 3 doesn’t change anything from what we originally reported. The GPU frequency optimizations that existed in the Exynos 5410 SGS4 don’t exist on any of the Snapdragon platforms (all applications are given equal access to the Note 3’s 450MHz max GPU frequency). The CPU frequency optimization that exists on the SGS4, LG G2, HTC One and other Android devices, still exists on the Galaxy Note 3. This is something that we’re going to be tracking and reporting more frequently, but it’s honestly no surprise that Samsung hasn’t changed its policies here.

The majority of our tests aren’t impacted by the optimization. Virtually all Android vendors appear to keep their own lists of applications that matter and need optimizing. The lists grow/change over time, and they don’t all overlap. With these types of situations it’s almost impossible to get any one vendor to be the first to stop. The only hope resides in those who don’t partake today, and of course with the rest of the ecosystem.

We’ve been working with all of the benchmark vendors to try and stay one step ahead of the optimizations as much as possible. Kishonti is working on some neat stuff internally, and we’ve always had a great relationship with all of the other vendors - many of whom are up in arms about this whole thing and have been working on ways to defeat it long before now. There’s also a tremendous amount of pressure the silicon vendors can put on their partners (although not quite as much as in the PC space, yet), not to mention Google could try to flex its muscle here as well. The best we can do is continue to keep our test suite a moving target, avoid using benchmarks that are very easily gamed and mostly meaningless, continue to work with the OEMs in trying to get them to stop (though tough for the international ones) and work with the benchmark vendors to defeat optimizations as they are discovered. We're presently doing all of these things and we have no plans to stop. Literally all of our benchmarks have either been renamed or are in the process of being renamed to non-public names in order to ensure simple app detects don't do anything going forward.

The unfortunate reality is this is all going to get a lot worse before it gets better. We wondered what would happen with the next platform release after our report in July, and the Note 3 told us everything we needed to know (you could argue that it was too soon to incite change, perhaps SGS5 next year is a better test). Going forward I expect all of this to become more heavily occluded from end user inspection. App detects alone are pretty simple, but what I expect to happen next are code/behavior detects and switching behavior based on that. There are thankfully ways of continuing to see and understand what’s going on inside these closed platforms, so I’m not too concerned about the future.

The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we’ve seen how this story ends. We’re very much in the mid-1990s PC era in terms of mobile benchmarks. What follows next are application based tests and suites. Then comes the fun part of course. Intel, Qualcomm and Samsung are all involved in their own benchmarking efforts, many of which will come to light over the coming years. The problem will then quickly shift from gaming simple micro benchmarks to which “real world” tests are unfairly optimized which architectures. This should all sound very familiar. To borrow from Brian’s Galaxy Gear review (and BSG): “all this has happened before, and all of it will happen again.”