A Timely Discovery: Examining Our AMD 2nd Gen Ryzen Results

Conclusion: It Changes Our Results

When we first published our Ryzen-2000 series review, with HPET forced as the timer in the operating system, our results were broadly showing that the new processors leading the pack. In light of the audit, especially with the way that the Intel gaming results have changed, paint a different picture.

At 1080p, the Core i7-8700K has a clear lead in most titles, although that lead does somewhat vanish moving to 4K, except with Civilization. Ultimately for any user pushing the pixel count, in our tests for the most part, the chips retain parity performance. AMD’s claims for the Ryzen-2000 launch were more focused on the 1440p gaming, however it is clear that there is still margin that benefits Intel at the most popular resolutions, such as 1080p.

Why This Matters, and How AnandTech is Set to Test in the Future

The interesting thing to come out of both Intel and AMD is that they seem to not worry if HPET is enabled or not. Regardless of the advice in the past, both companies seem to be satisfied when HPET is enabled in the BIOS and irreverent when HPET is forced in the OS. For AMD, the result change was slight but notable. For Intel we saw substantial drops in performance when HPET was the forced timer, and removing that lock improved performance.

It would be odd to hear if Intel are not seeing these results internally, and I would expect them to be including an explicit mention of ensuring the HPET settings in the operating system in every testing guide. Or Intel's thinking could be that because HPET not being forced is the default OS position, they might not see it as a setting they need to explicitly mention in their reviewer guides. Unfortunately, this opens up possibilities when it comes to overclocking software interfering with how the timers are being used.

As noted above, overclocking and monitoring tools like Ryzen Master request a restart when used for the first time in order to make sufficient changes to the system to run correctly. Some of this software will be forcing HPET in the BCD in order to enable what it needs to do, and the adjustment is unlikely to be explicitly mentioned in the request to restart. In a standard review, it is typically expected that each system will have a fresh OS and fresh software install, such that systems are tested as if it were new. For any user looking to tune the system, this is the point where any potential software issues could occur. Now should a reviewer decide to first analyze the software bundled with the system before testing or after testing could have significantly different results. It can create a conundrum, as has clearly been the case for us.

Moving forward, the immediate goal here at AnandTech is to ensure that our readers have the most up-to-date and correct results, particularly for our Ryzen 2000-series review. As a result, we are taking a few steps both immediately and in the future to correct our data, update our Ryzen 2000-series review, and to prevent this issue going forward.

First and foremost, we have decided that force-enabling HPET is not how we want to test systems, as this is non-default behavior. While it has an important role in extreme overclocking, to verify accurate timing, ultimately it was akin to taking a sledgehammer to cracking an egg for our testing - we need to be testing systems at stock. So all further CPU testing going forward will be using HPET's default behavior, and we have even put checks in our scripts to ensure this is now the case.

As a result we are retracting our existing results for all of the processors we used in the Ryzen 2000-series review. This goes for both the review and for Bench. All of these products will be updated with revised results using the default HPET behavior just as soon as the updated data is available over the course of the next week. In fact we're already the process of running this updated testing, which we've used for this article and uploaded to Bench.

The end goal here is to cover most of the popular processors from the previous few generations on our existing 2017 benchmark suite in order to fully update and republish our Ryzen 2000-series review. Meanwhile, because the results in that review are still being updated, the conclusion for that review is also being retracted. We don't anticipate updated results meaningfully changing that conclusion, but it is inappropriate to have a conclusion remain published until we have all of the data we need.

Longer-term, because this issue goes back further than just the Ryzen 2000-series review and we are already on the cusp of organizing our 2018 CPU benchmarking suite, we're also accelerating our rollout of that suite. After replacing the data for key hardware on that 2017 test suite, we will be rolling out the 2018 update in earnest. The 2018 CPU benchmarking suite will upgrade to the latest software, drivers, and a change-up on games (F1 2017, Shadow of War, Far Cry 5; also had requests for Deus Ex). Our 2018 suite will require that Spectre and Meltdown patches are in place for the systems we test, to ensure that everyone has access to the latest data.

(ed: It should be noted that this only affects Ian's CPU review data; Brett and Nate run different tools in their laptop and GPU reviews respectively)

Overall we expect to be done collecting data to finish and update the Ryzen 2000-series review next week. After that, it will take some time to roll out the 2018 CPU benchmarking suite data, but that should only be on the order of weeks assuming that there are no further surprises (ed: knock on wood).

We also would like to give all of our readers and colleagues a sincere thank you for assisting with this analysis. We continually strive to publish the best possible data, so your input is and always will be invaluable for finding patterns and oddities we may have missed.

Finally, while we're on the subject of timers, we'd like to throw out an open-ended question to everyone: given what we've found, should the use/requirement of HPET in software be made clearer? Or is there a risk that information being more confusing than helpful? One of the issues we grappled with in writing this article is that while HPET can have a performance impact, it's also not necessarily wrong to use it given its unique accuracy. So we're interested in hearing from all of you on how you think the use of HPET should be documented, so that users aren't caught off-guard by the potential performance impact..

Update: 04/26

HPET and Invariant TSC

Since publishing this follow up, several readers have reached out about their experiences with timers, as well as offering deeper explanations of some of the key points in this article. I will attempt to cover some of them here.

The main on-die CPU timer is the Time Stamp Counter (TSC), which was one of the main timers in single core systems. With the movement to multi-core, HPET became the new more accurate timer that as described can protect against clock drift. HPET was preferred to TSC, but can take 10-100x longer to be probed, due to its location on the chipset. The industry however is moving back towards TSC through an Invariant TSC (ITSC), which is a version of TSC that is stable through CPU frequency changes and C-state changes. The ITSC is accessed through the RDTSC instruction, which can be used simultaneously by both the kernel and user code if permitted (unlike HPET, which is a locked timer), and is sufficient for multi-core systems. And although this method still has the RTC bias issue, the lower latency is favoured by all, except overclockers adjusting the platform's 100 MHz base frequency.

TL;DR: HPET can take 1000s of cycles to read, and reading it with multiple cores compounds the issue. Invariant TSC, as a core instruction, is a potential solution with lower latency already in use.

“There is a HPET Bug, No Intel is Not Cheating” and TimerBench

Matthias from Overclockers.at reached out to me and linked me to his article on how they have previously encountered the issue. The article is a nice read, and well worth clicking through:

The HPET bug: What it is and what it isn't

Matthais explains how during their X299 testing, they were experiencing slowdown in their game benchmarks, and pin-pointing the problem with HPET. (We also had similar issues, and didn’t post results, but never got to the bottom of the issue.) As a result, the team over at Overclockers.at developed a tool called TimerBench in order to determine the effect of HPET. As noted, HPET has a much longer latency, but is more accurate.

In the results from overclockers.at one metric stood out: moving from Broadwell-E to Skylake-X meant that the number of theoretical peak HPET calls per second reduced from 1.4 million to 0.2 million – the latency to make a HPET call suddenly became 7x longer with Skylake-X. TimerBench, the tool developed, provides an Unreal 4.7.2 scene and measures timer calls between a system running a game, and one without.

For our results, we used TimerBench on each system with a GTX 1080 in 1920x1080 mode, running fullscreen.

With the HPET timer, the i7-8700K system went from 214k timer calls per second outside of a game down to 144k timer calls per second, which is about the same fraction as with the ITSC timer. The big difference however is the frame rate, decreasing from 289 FPS with ITSC to 238 FPS with HPET, as well as the average GPU load, down from 97.6% to 78.1%. This is shown in the maximum frame time as well.

* No Spectre/Meltdown Patches

When I correlate this data with the systems I have currently running, we see that the AMD Ryzen 7 1800X system is not particularly affected, but all of our Intel systems are: Skylake-S, Skylake-X, Coffee Lake, and even our mobile device. What is clear that the HPET timer is causing performance degredation by virtue of having a lower average GPU load. If the GPU is waiting on the same timing delays caused by HPET, this would lover the overall GPU load.

So this interesting correlation leads me to think that maybe this issue, aside from potential Spectre/Meltdown related points, is related to the chipset. HPET circuits are normally found on the chipset/southbridge, and in this case Intel has a wide HSIO chipset design in all the systems tested. As the chipset is, among other things, a PCIe switch, then it has various buffers to deal with the data coming in and out. The effect of these wide chipset and buffers might be part of the HPET issue. I need to go dig out an older system.