Atomic investigates NVIDIA’s long-ignored accusation: Is ATI using FP16 demotion to artificially inflate benchmark scores?

The graphics processing world is ever-turbulent. Arguments appear cyclical in nature from both ATI and NVIDIA, and bickering over issues like physics engines isn’t uncommon – bringing about phrases like: ‘They’ve been cheating with their so-and-so’, or ‘We offer the better solution without a drawback like theirs!’ from both sides.

It’s all part and parcel of competition. However, when Atomic received the recent NVIDIA GTS450 card, press release and included reviewer’s guide, we noticed that it came with a page that implied certain unfavourable things about their competition, ATI.

We discovered that this page had been included in every reviewer’s guide sent to hardware reviewers since the launch of the GF100 ‘Fermi’ family: the GTX480, GTX470, GTX465, and both GTX460 cards. It had been included with the GTS450 and five other cards, but it seems no-one has paid it much attention, dismissing it offhandedly – as we admittedly dismiss most guides that attempt to dictate our testing methodology.

(…)

ATI/AMD FP16 Demotion Response

AMD Australia Technical Manager Garrath Johnson

Thank you for contacting us about statements made by our competitor regarding disabling Catalyst AI for proper performance testing. It has been brought to our attention that NVIDIA has accused us of cheating, claiming that we have optimized some aspects of Catalyst to improve our benchmark scores. Given that these same claims at one point referenced a specific member of the media as the source of this information, and that same member of the media has denied ever suggesting that we admitted to optimizing drivers for the sake of benchmark performance, the NVIDIA claims would appear to lack any substance. Nonetheless, we’d like to confront this issue head-on and provide you with the facts for you to decide.

The alleged “optimization” is the selective use of the HDR format R11G11B10 at times when the memory cost of the FP16 HDR format would otherwise impact game play. Given that in their own documents, NVIDIA indicates that the R11G11B10 format “offers the same dynamic range as FP16 but at half the storage” it would appear to us that our competitor shares the conviction that R11G11B10 is an acceptable alternative. Additionally, we rigorously test image quality before implementing this format change and only apply it if the difference is imperceptible.

The R11G11B10 format is similar to the HDR format used in the XBOX 360, the game platform for which many titles mentioned in a list provided by NVIDIA were initially designed. In many cases, the use of the HDR format with the smaller storage requirement provided the performance headroom to implement other IQ features such as anti-aliasing, in fact increasing the overall appearance of the game.

With respect to the list of titles highlighted by our competitor as proof of our apparent “cheat”, the titles themselves and the time frame these titles span serve as a good indication that this was not done for “benchmarking purposes”. Also, as mentioned above, it is selectively used on a game by game basis. For example, this optimization was enabled for Elder Scrolls IV: Oblivion yet it was not something that was enabled in Fallout 3 which featured the same engine.

To conclude, I would encourage you to think twice before disabling Catalyst AI. Doing so will disable numerous other features which would make such a comparison meaningless. In this situation you would have to also disable all NVIDIA optimizations in their driver, something that is made even more difficult by the fact that they do not provide an option to do so in their control panel.

Below are a few questions you might be asking in light of what has been presented to you.

Atomic) Is FP16 format replacement with R11G11B10 format activated in every gaming title?
AMD) No. It is enabled in few specific DirectX 9 titles as indicated in NVIDIA’s list.

Atomic) What are the requirements for enabling the R11G11B10 format?
AMD) The requirements are that we verify with QE testing that no visible image quality changes occur when it is enabled. In general we do not apply this optimization to DirectX 10 or DirectX 11 titles since the R11G11B10 format is available to developers in that API and titles can therefore already take advantage of it where appropriate. This not exposed to developers in DirectX 9 and this is not an option for DirectX 9 developers, so we do the legwork for them. This is similar to the legwork we sometimes have to do to enable MSAA in game titles that don’t otherwise support it. In this case we also do the legwork for our end users to enable additional optimizations such as this one that may not be available to developers through the API.

In the end, if an end user does see any visual quality differences, they always have the option to disable the optimization using Catalyst AI. This is what that option in the control panel is there for. However, we don’t see why anyone would ever want or need to disable this optimization in the select number of titles in which it has been applied given the benefits they receive from its presence.

Atomic) Does this format replacement reduce graphical workloads in applications at the expense of image quality?
AMD) No. In the select cases where it has been enabled there is not apparent reduction in image quality for the end user. If there is sufficient precision in the R11G11B10 surface for the level of HDR that the game is using, then there is no benefit to anyone by using the higher precision surface. This is merely an unwarranted cost in terms of memory usage and bandwidth. Conversely, some games use much wider HDR gamuts, exceeding that available precision in the R11G11B10 format. In these cases the optimization is not applicable and is therefore not applied.

Atomic) Does AMD support the use of NVIDIA’s “AMDDemotionHack” in benchmark testing?
AMD) If this is something that will be made available to their end users then, yes. Otherwise this will be a pure benchmarking effort and no end user will benefit from information gathered from these tests. Also, confirming that there is no visible effect on image quality would be vital to confirming the validity of these results.

NVIDIA FP16 Demotion Response

NVIDIA Technical Marketer Jeffrey Yen

I think there’s a misunderstanding with how our profiles function. The complete quote in our guide should be “NVIDIA’s official driver optimization’s policy is never to introduce a performance optimization via .exe detection that alters the application’s image quality, however subtle the difference.”

That doesn’t mean that profiles don’t look for .exe files. Just that we’re unwilling to alter the application’s image quality.

I’m sure you’re familiar with many of the performance improvements across games and other applications that our drivers have enabled over the years.

In regards to questions 3 and 4, the answers are below:

Atomic) What does “FP16 demotion” do, and why has it taken so long for it to be included in NVIDIA drivers?
NVIDIA) FP16 demotion is implemented in AMD drivers and has been for some time. What it does is that it demotes FP16 render targets to R11G11B10 render targets which are half the size and less accurate. There’s a change in image quality, and it alters the workload for benchmarking purposes.

And as mentioned above, NVIDIA’s policy is never to introduce performance optimizations that alter the application’s image quality. This is also the policy of FutureMark and various other benchmarks regarding legitimate driver optimizations.

Atomic) Why is it considered incorrect to test with Catalyst AI enabled at the launch of the GTS450, considering it has been included in AMD drivers for over five years?
NVIDIA) Due to what FP16 demotion does, we ask to disable this function to get true, apples-to-apples comparisons.

• READ MORE (Source): ATI ‘cheating’ benchmarks and degrading game quality, says NVIDIA – PC Games – Game – Features – Atomic MPC