An introduction to FCAT benchmarking

#4569613

hallryu

2013-04-04 09:49

Excellent read. Explains the new benchmarking methodology rather nicely.

#4569614

Phuncz

2013-04-04 09:50

Excellent write-up, I'm very pleased you are going to incorporate FCAT testing into your reviews ! This needs to be known, so the manufacturers (AMD more than NVIDIA at this point) can make constant frame-times a priority and eliminate micro-stutter. One point I'd like to make is that it would benefit all if you would include the same scaling for all your graphs and include zoomed-in versions indented or offset enough that it's clear it isn't the same. Because some aren't really comparable when they use different scaling like these two: http://www.guru3d.com/index.php?ct=articles&action=file&id=3099 22-44ms scaling (22ms range) http://www.guru3d.com/index.php?ct=articles&action=file&id=3101 0-30ms scaling (30ms range) It would have been more helpful if the charts were 0-50ms, which is basically with a broad margin the lowest-highest we'd ever want to see anyway.

#4569619

k1net1cs

2013-04-04 09:59

Hilbert, please don't use the term 'quickie introduction'...just 'quick introduction' would be enough without having the readers being rendered into wild imaginings. :P

#4569621

Hilbert Hagedoorn

Administrator

2013-04-04 10:07

Excellent write-up, I'm very pleased you are going to incorporate FCAT testing into your reviews ! This needs to be known, so the manufacturers (AMD more than NVIDIA at this point) can make constant frame-times a priority and eliminate micro-stutter. One point I'd like to make is that it would benefit all if you would include the same scaling for all your graphs and include zoomed-in versions indented or offset enough that it's clear it isn't the same. Because some aren't really comparable when they use different scaling like these two:

Yeah I know and actually mention that a couple of times in the article. Over time you'll see charts with fixed scaling. Percentile charts will range from 0 to 100% and the plots we need to figure out some sort of fixed y-axis for (likely ~60ms).

#4569628

hallryu

2013-04-04 10:31

Yeah I know and actually mention that a couple of times in the article. Over time you'll see charts with fixed scaling. Percentile charts will range from 0 to 100% and the plots we need to figure out some sort of fixed y-axis for (likely ~60ms).

How much time will this add to your reviews? The cost itself is eye-watering!

#4569636

xafier

2013-04-04 10:49

Hilbert, I've read your description a couple of times now and I'm not really sure why you need to store all the frame data? If all you care about is the differing colours couldn't you just store the 1 pixel from bottom left? The only reason I can see for not doing that is that it the video capture device can only store the full frame and instead you'd require CPU to pull out just the 1 pixel? But I cannot imagine that would be at all taxing as extracting a value from a known location in a frame is very simple. Just throwing out my opinion as a software developer, that thats the road I would have gone down to avoid that storage problem. It would also most likely simplify the data analysis later too.

#4569638

mohiuddin

2013-04-04 10:55

Thanks. And explanation of every graph helped me (noob) a lot. Again thanks.

#4569640

Osamar

2013-04-04 10:56

Hilbert, I've read your description a couple of times now and I'm not really sure why you need to store all the frame data? If all you care about is the differing colours couldn't you just store the 1 pixel from bottom left? The only reason I can see for not doing that is that it the video capture device can only store the full frame and instead you'd require CPU to pull out just the 1 pixel? But I cannot imagine that would be at all taxing as extracting a value from a known location in a frame is very simple. Just throwing out my opinion as a software developer, that thats the road I would have gone down to avoid that storage problem. It would also most likely simplify the data analysis later too.

If I am not wrong, you will need at least one pixel thick bar, becuase you could have mixed frames (tearing ???), so multiple colors in the bar.

#4569651

xafier

2013-04-04 11:16

If I am not wrong, you will need at least one pixel thick bar, becuase you could have mixed frames (tearing ???), so multiple colors in the bar.

Good point! But that still reduces the amount of storage by a very large amount.

#4569652

Ven0m

2013-04-04 11:24

Hilbert, thank you for this article. I do hope such analysis will be added to the new card reviews, especially when compering the performance of multi-GPU solutions. However, the question arises - how reliable and jitter-free is the capture device? 😀

#4569691

Pete J

2013-04-04 12:53

Just wanted to say thank you for an excellent article! Nice to see microstutter examined in such detail rather than subjective 'by eye' comments as has seemed to be the case up until now.

#4569692

yasamoka

2013-04-04 12:53

Hilbert, thank you for this article. I do hope such analysis will be added to the new card reviews, especially when compering the performance of multi-GPU solutions. However, the question arises - how reliable and jitter-free is the capture device? 😀

That question can probably be answered by feeding the capture card a VSynced signal. If the results turn out flat, there you go, no jitter (oversimplifying it, but oh well). Thank you Hilbert for a very interesting read!

#4569694

BLEH!

2013-04-04 12:56

Ow, my head. Makes a bit more sense, but owwwww.

#4569701

rsnubje

2013-04-04 13:11

Hilbert, I've read your description a couple of times now and I'm not really sure why you need to store all the frame data? If all you care about is the differing colours couldn't you just store the 1 pixel from bottom left? The only reason I can see for not doing that is that it the video capture device can only store the full frame and instead you'd require CPU to pull out just the 1 pixel? But I cannot imagine that would be at all taxing as extracting a value from a known location in a frame is very simple. Just throwing out my opinion as a software developer, that thats the road I would have gone down to avoid that storage problem. It would also most likely simplify the data analysis later too.

Basicly, as it is an analysis tool, you also would like to see the frame that comes with the time. So if the graph says all the frames are for example 30ms and one or more are outstanding with let's say 60ms, you want to be able to look back at that specific frame to see what it looks like, is it or is it not a tearing or stuttering. This way you can tell if it's really giving you a bad gaming experience or not.

#4569727

Stukov

2013-04-04 13:51

And Vsync eliminates these issues, correct?

#4569758

Norvekh

2013-04-04 14:46

More data, when it is applicable to increasing understanding, is always a good thing. This FCAT testing most certainly provides that. That's a ton of work, and expense, but it will provide better and more accurate performance analysis and that is absolutely to be commended. Great job, Hilbert and team. You're showing exactly why this is one top notch site to be kept on the short list that I always check for reviews come new hardware. On a sidenote, I took the FCAT of an entirely different nature while I lived in Florida. I just thought that was somewhat amusing, even if irrelevent.

#4569771

tsunami231

2013-04-04 15:03

And so it begins the race for who can render things with the lowest and most stable frame latency, Who will win Nvidia or AMD/ATI. Obviously Nvidia is better at this atm, but who knows what will happen. Personal it dont mater to if it make all GPU better and smooth as a result its becomes win win situation for all of us

#4569772