A FCAT Conclusion

So what's the verdict on FCAT?

Well let me divide the answers to that question in several chapters.

Isn't Everybody Overreacting?

Stuttering, measuring anomalies...  isn't everybody overreacting? Well yes and no. Small anomalies rendered and which you can experience on screen always have been a part of a graphics card and your overall game experience. For years now we have had that, for years now most of you have not been bothered by it. Aside from a small group enthusiast end-users and analysts, that is the primary context you need to keep in mind when it comes to FCAT measurements, really. 

Yeah, average FPS is still (in my opinion) the most important denominator in terms of determining how fast a game can be rendered. Now that doesn't mean I am disqualifying frame time or what I like to call frame experience measurements, contrary. Frame experience measurements in my mindset will help as an extra tool and data-set to show you the relation of render performance versus what you see on screen. Frametime measurements are a tool to detect anomalies that we never really measured. So it is more a question of what can we accept when analyzing anomalies and what not, because some people will totally freak out if they see a couple of latency spikes in a chart. Realistically you'll be hard-pressed to notice it, heck one big massive scary spike in a chart could even something as simple as a game scene change. Frame Time / Frame Experience measurements however are becoming a part of Guru3D test and benchmark methodology. It will sit next towards what we have always shown you, average FPS, as average FPS we still consider to be the best measurement we can fire off at you if you are asking the question "how fast is my graphics cards". But an extra data-set that can detect anomalies obviously is great to have and show.

Should I still use FRAPS?

Definitely yes, you can do a time recording and get an average FPS output that is perfectly in line as to what the game engine renders. But average FPS will not tell you anything about that stutter/glitch that you have seen on screen. That's pretty much what we are trying to evangelize today. 

In direct relation to that question; can you trust FRAPS for Frametime / Frame Experience  recordings then? It is difficult to answer, as we have been demonstrating today, FRAPS measures at a preliminary stage, the game render time. Basically the game engine. Therefore the latency results can miss out on a lot of other stuff. Therefore the measurement can not be considered in line as to what you see on screen. There is an exception though, a single GPU the stutters and anamolies are far less frequency and as such you can say, the recorded frame time results are more consistent and in line with what you may expect. With multi-GPU solutions stutters, anomalies and micro stutters are just way more apparent. But the truth be told that it was FRAPS that was able to uncover weird stuff you guys have noticed on multi-GPU solutions. So looking at it from that point of view, it is still a nice tool to get frametime indications from.

What Now?

Again I like to make the following statement strong, stutters always have been part of graphics cards. So I do plea... how important is it really? How much are you bothered by something you probably hardly even notice? I mean if you play a game and every now and then you see a small glitch, a stutter for a fraction of a second... does that ruin your game experience? We think not, and as such the stutter/glitches issues are a bit blown out of proportion with everybody jumping onto it. For multi-GPU gaming it is different though, if you pay attention to micro-stuttering, you can see it. But the question again remains the same, how many of you are actually bothered by it. Sure, if you raise enough awareness then people will revolt and address the issue continuously. But we do like to ask you to look at things in perspective.

FACT or FCAT - Can We Trust FCAT?

Considering FCAT primarily is an open project we feel it is a reliable benchmarking methodology. The fact that the ideas and initial software needed originates from NVIDIA doesn't mean it is a subjective methodology. Contrary, this can become one of the best benchmark methods we have ever tried, it is very time consuming though. From the get-go NVIDIA has been open and transparent about everything related to this testing method. Once the tagged frames arrive at the FCAT machine honestly, it can't see what graphics card is rendering the game, as such it creates an equal playing field for any brand or type of card you connect to it.

The one piece of software that could be tainted as such would always be on the game machine side, and that's the software overlay - hence here at Guru3D we started implementing our own coded FCAT overlay version into RTSS, the overlay statistics software for MSI AfterBurner and Precision to immediately kill of that suspicion. Honestly, NVIDIA would shoot itself in the foot to mess with stuff to gain results in their favor, as the media will detect it, that I guarantee you. But we'd love to hear about it from you guys. 

Of course NVIDIA had alternative motives to start a discussion about FCAT, it exposes the things we see with AMD's AFR sync multi-GPU solutions versus what need to be more consant frame time. If we reverse that, if NVIDIA would have had issues at hand, they'd never introduced this methodology towards the public. But, that doesn't mean the FCAT methodology is flawed. Now I took Hitman as a bit of a showcase in this article, as it has extreme (way more then other games) issues with showing latency spikes that can not be seen on the monitor. Realistically FCAT didn't show them on AMD single GPU solutions, so that works out pretty well and benificial for AMD don't you think ? So yeah, I believe it is one of the best tools we have had at hand in a long time. But I'm very curious what you guys think about FCAT and will follow and love to listen to you guys in our forums.

Moving Forward

This article is just a quick introduction to a new benchmarking method, it's a start .. the tip of the iceberg really, just to warm you guys up and to get you guys used to all the new stuff. In our future graphics card reviews, the reference articles in particular we'll follow the most valid benchmark methods possible. Hence we'll continue to show you the average FPS as it still is the best method of showing performance, it's also the method that most people can understand. But next to that we will start adding multi-page chapters on Frame Experience recordings. These two measurements will show you the brute rendering game performance power of the graphics card, yet with the new frame experience measure also show you the smaller on-screen anomalies and we feel that data gives you guys a wider view on what you may expect of your game experience. With that said I like to conclude this article. I do hope it wasn't too confusing and/or difficult to understand. The testing methodology used is a bit steep, we know. So if it was too confusing we do apologize. 

We like to thank AMD and NVIDIA for allowing us to pick their brains, a special thanks to OCZ who provided the RevoDrive 3 X2 to allow us to capture uncompressed rendered frames at 450 MB/sec and GSKILL who supplied the memory for the FCAT system.

Hilbert out :)