The Black Shark 2 Review: A Gaming Phone's Existential Crisis
by Andrei Frumusanu on September 25, 2019 8:00 AM EST- Posted in
- Mobile
- Smartphones
- Xiaomi
- Snapdragon 855
- Black Shark 2
Machine Learning Inference Performance
AIMark 3
AIMark makes use of various vendor SDKs to implement the benchmarks. This means that the end-results really aren’t a proper apples-to-apples comparison, however it represents an approach that actually will be used by some vendors in their in-house applications or even some rare third-party app.
Unfortunately for the Black Shark 2, the devices lacked the proper drivers to properly run AIMark, and the benchmark repeatedly crashed upon starting the benchmarks. We had the same issue on the OnePlus 7 Pro, pointing out to some software incompatibility.
AIBenchmark 3
AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.
We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initialising the test to actually executing the computation.
AIBenchmark 3 - NNAPI CPU
We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.
In AI Benchmark’s CPU workloads, the Black Shark 2 ends up with a bit of a odd spread of scores. In the shorter running benchmarks the phone is getting relatively average inference times, while on the longer running tests for some reason the BS2 falls behind other S855 devices. In fact it looks like the BS2 is landing as amongst the worse off S855 devices in the latter listed tests.
AIBenchmark 3 - NNAPI INT8
AIBenchmark 3 - NNAPI FP16
AIBenchmark 3 - NNAPI FP32
In the INT8, FP16 and FP32 accelerated tests which make use of acceleration blocks such as the Hexagon DSP and the GPU, we see the Black Shark 2 perform very well and in line with other Snapdragon 855 devices.
Overall, the Black Shark 2 is a good performer in the machine learning inferencing benchmarks, but like other devices, it’s not quite the very best in every regard, pointing out that the vendor could have improved upon its performance by keeping the software stack more up to date with what Qualcomm is offering, a widespread issue that I expect to persist over the next years as the ecosystem quickly evolves.
Facebook
Twitter
RSS
63 Comments
View All Comments
Jon Tseng - Wednesday, September 25, 2019 - link
Thanks interesting. I guess that definitely answers the question as to whether the paid Qualcomm partnership had any impact on your ratings (not that it would, knowing AT, I know!).PaulHoule - Wednesday, September 25, 2019 - link
What I found strange about the article is that, unlike most reviews of gaming hardware, there were not any benchmarks for games.Maybe that's because nobody cares how many frames per second you can render Candy Crush which is why they pack it in with Windows so Intel can claim there is a "game" you can play with the integrated graphics.
If there aren't any games that need a gaming phone, why does a gaming phone have to exist? It's like the sound of one hand clapping. isn't it?
Death666Angel - Wednesday, September 25, 2019 - link
I mean, there area pretty intensive games for smartphones out there, think PUBG, Fortnite, racing games. I don't really know if they provide accurate/useful ingame benchmarks and I don't know if there are any FRAPS / FCAT like apps for Android. I don't think we get anything better than 3DMark and GFXBench right now. :) Everything else is non repeatable and therefore of little use.PeachNCream - Wednesday, September 25, 2019 - link
Pretty sure the decision to include Candy Crush is between Microsoft and King. It isn't a decision related to Intel. After all AMD based Windows systems ship with it preloaded as well.crimson117 - Wednesday, September 25, 2019 - link
I think benchmarks were omitted because "the firmware detects benchmarks and disables thermal throttling in order to get better scores." So benchmarks would be meaningless and misleading.s.yu - Wednesday, September 25, 2019 - link
Generally Anandtech's benchmarks should be immune to detection, which is stated in the article.Andrei Frumusanu - Thursday, September 26, 2019 - link
Yes, we bypassed detection for the published scores.raghusa - Wednesday, September 25, 2019 - link
AT was always one of my go to place to get in-depth analysis and reviews. Now that AT is going with PAID partnership route, my view of AT analysis is definitely going to change, will likely look for other reputable sources ....Andrei Frumusanu - Thursday, September 26, 2019 - link
The review content is still fully independently written, and the editorial team including myself have no contact with the sponsor in this regard. In any case this series will come to a closure on Monday.Wardrive86 - Thursday, September 26, 2019 - link
I wish that it was not coming to an end! I very much enjoyed this series.