Rage3D Smoothness Testing Explained, Catalyst 13.1 vs 13.2
Earlier this month we introduced our new method of evaluating performance, in our review of the HIS Radeon HD 7970 GHz Edition IceQ X2. This method moved beyond pure FPS to looking at quantifying stutter and smoothness, and we've had some great feedback from our readers on what they liked and didn't like. It's clear that smoothness and stutter are aspects of performance that need showcasing, and the clearer the better. To that end, we've updated the analysis a little and cleaned up the presentation, per requests by people like you, dear reader.
First of all, we're basing our analysis on log file from the popular FRAPS benchmarking tool. This provides us with the draw call to draw call time, known as frame time. Traditionally you've seen averaged performance numbers in performance testing, approximated over a period of time (typically 1000 milliseconds). These numbers, particularly average, hide some information about the performance the user experiences, as gaming is not about throughput but responsiveness; latency. Performance enthusiasts want the best feel, graphics enthusiasts want the best look, and those two types of people are rarely separate entities.
Below you can see the data we'll present in a performance evaluation. First up is the traditional minimum, average and maximum frame per second results. You've said you still find value in this metric, and we tend to agree it helps to paint a picture of the performance level on offer. While the minimum and maximum are the FRAPS fastest and slowest second results recorded in the '-minmaxavg' file, we're using our own perceived average figure. This figure is determined using a method similar to that found in examining turbulence intensity in the field of fluid dynamics, although our method is nowhere near as grand. A local variation index is derived by creating an array of local frame rate averages, short sample periods averaged as a rolling index parses the array of individual frame render times. These local averages, expressed as differences from the global average, are useful finding uneven frame render times. Occurrences of sustained uneven frame render times, especially with large deltas, will have a deleterious effect upon the final result, as all the local frame rate variations are averaged into a global index value used as a modifier on the global average. This is the perceived average frame rate, and the more stuttering that is observed in the test range, the lower the average value will be, in keeping with how the gameplay feel is impacted.
Next we're going to show our smoothness rating, a value that indicates how much time was spent outside of our smoothness zone. The smoothness zone is based on finding the most common frame render time, the mode average. This value is used to determine slow and fast limits, using an offset of 20%. Here is an example of a frametime plot with these +/-20% limits shown (using mode for average). You can see that simply plotting a graph of all frame render times is messy and hard to intrepet but you can see the game sitting in the 'smoothness zone' except for isolated but repeated bursts of slow and fast frame times (click to zoom wherever you see the magnifying glass in Rage3D articles).
So instead of this plot, we count how many frames were rendered inside this performance boundary, expressed as a percentage of total frames rendered. The higher the percentage, the more time in the smoothness zone. This method penalizes performance outside our quite strict boundaries, penalizing both slow and very fast frames at the same time. This discounts average altering fast frames from impacting the overall presentation of performance.
Finally, we break down the frame render time distribution, so you can see the spread of time to render frames. We've set our bands to common performance markers. Frames 5ms and faster to render are grouped, and frames 1000ms and slower and grouped, with bands for 8.33, 10, 13.33, 16.67, 22.22, 33.33, 41.67 and 100ms in between. These correspond to 200, 120, 100, 90, 60, 45, 30, 24, 10 and 1 frames per second averages, if you're more used to thinking in those units than milliseconds. As the values indicated are percentages of total test sample frames, the tested title should spend a clear majority of its frames in one or two bands to be smooth and consistent. Extreme outlier frame render times will be evident here, the more frames in the end bands the more of a problem with large frame render time variations.
We will continue to test for both pure performance (vertical sync, buffering and frame rate limited disabled) and with smoothness measures enabled. Our first preference for smoothness is to enable a frame limiter, at 5fps higher than the display refresh rate. If this is not an application option then we will use either application or driver control panel vertical synchronization controls/overrides, or attempt to use a third party frame rate limiting technology like RadeonPro. Now, you may have noticed above that we're showing Battlefield 3 performance results, you're looking at Catalyst 13.1 WHQL vs. the leaked Catalyst 13.2 driver, as used by TechReport.com recently. As you can see, there is a measured improvement in smoothness for us from 13.1 to 13.2, at a small performance penalty; a classic case of higher FPS not indicating everything you needed to know. Fingers crossed the full rewrite driver (unfortunately still a ways off, in the mean time game profiles are being adjusted to offer smoother performance) will bring even more consistency and smoothness.