1. It's not clear to me how the following anomalies should be assessed in relation to the "apply to 95% or more of the game" rule:
-A game which contains a total of 100 levels, 4 of which are completely unplayable in 3D.
-A game which contains a total of 100 models (or textures), 4 of which have glaring anomalies in 3D.
-A game which features 100 weapons, 4 of which are sniper rifles with scopes which can't be used at all in 3d.
-A game which contains 100 levels, 4 of which contain a minor visual anomaly at one particular point within the level.
If I understand the current scoring system correctly then all of the above examples might receive a 10/10 score for 3d. This is imho a very very bad thing. To score 10/10 a game should be perfect in 3d with all graphics options enabled and no anomalies whatsover, so I'd remove the "95%" bit entirely. (I'd also limit a 9/10 score to games which run in perfect 3d when some very minor compromises are made wrt graphics options, and 8/10 to games which run in perfect 3d after more significant reductions in graphics options. I don't think any game which has any unfixable anomalies in 3d should ever score higher than a 7).
There are two types of anomalies. Anomalies that are so bad, you can't play a game from beginning to end (the 95% rule), and the occasional anomaly which doesn't seriously hinder the game, but exists nonetheless.
Can you think of a way to differentiate the two? Remember that most people could rate a game without playing it from beginning to end. They may play 10 hours of it, and not realize there is a single problem on the last level.
How about:
1. Please tick all the following anomalies that consistently appear from one level to the next in an estimated 95% or more of this game when creating a COMBINED DEPTH AND POP-OUT experience. Do not select anomalies that are successfully...
OR:
Look at the critical anomalies list. These are problems that should not appear on one screen and disappear on another. The HUD will always split, the doubled image will always be there, the blur effect in one eye will be consistent. I know this from experience. Are there anomalies in this list that are more likely to be scene specific? Maybe the solution is to take certain problems off the list, and move them to the tier 2 or individually scored list.
2. I also think it would be a good idea to remove completely any subjective terms like "seriously" as in "depth of an object is seriously inconsistent to its actual location". Either an object renders correctly or it doesn't. If it doesn't, then score penalties should apply, whether it's off my a meter or a mile.
We can look at this. If we have image samples, that may explain this enough.
3. There are a couple of occasions when "2d" is used in a description where it would be less misleading to say "rendered at the wrong depth". For example, just about all games render fire effects or very distant terrain on a 2d sprite or surface. So long as this 2d effect is drawn at the correct depth in the game relative to other objects this is fine and shouldn't detract from the score. The problem is only when such effects are rendered at the wrong depth (generally at screen depth).
I agree with this. An image sample will help, but I agree with this.
4. "FPS performance is less than 30% of the 2D mode performance (whilst driver is resident)". This is far too forgiving. 50% should be the minimum acceptable frame rate before significant score penalties apply. And what is meant by "whilst the driver is resident"? The 3d performance should be compared to running the game in 2d with the driver totally disabled (and all other settings the same). Comparing 3D to 2D with the 3D driver enabled in the nVidia control panel but not currently turned on with CTRL+T (which I think is what is meant by the "resident" above) is pointless as the 3D driver lowers fps even when not currently running. The only valid comparison is to playing in 2D with the 3D driver completely disabled in the control panel.
Also, why not incorporate the actual percentage reduction in fps in the score instead of having an arbitrary cut-off point like 30%. All other things being equal, a game which runs at 70% of 2d framerate should get a better score than one which runs at 50% of 2d framerate. This isn't currently the case.
I see where you are coming from. Let me explain my logic:
When some stereoscopic 3D drivers are resident, some performance is lost even in 2D mode because memory is reserved for S-3D mode. So, I'm saying resident meaning S-3D capability is on standby with the tap of a button.
The 50% expectation you have shared is based on a zero overhead from the driver. Nothing for the extra work needed to calculate the left and right images let alone the display of them. Even NVIDIA with their deep GPU access has had performance drops to just 40% of the 2D equivalent.
Game to game performance is also very different.
I went with 30% because it's low enough to take into account all the possible handicaps, and high enough that it demonstrates the performance is hardware limited rather than programming limited. I don't want it to be necessary for people to do too much benchmarking because that could hold back participation if it is a requirement.
There are also a few questions which aren't currently in the utility but could be worth adding. For example:
1. For what percentage of the time you spent playing the game were no anomalies whatsoever visible in 3d?
We have something in mind for this, but it will be a little different. Remember, what if the game hasn't been completed when this data is entered?
2. Are there any points in the game where 3D must be turned off to progress past a certain point? (eg a vital hint message which can't be read when playing in 3d or a level which simply crashes unless run in 2d).
I have to think about this one. We don't want a hardware bug or an individual's specific computer setup to reflect on the game's rating when it could work properly on 100 other machines.
3. If some settings must be adjusted to play in 3D without anomalies, which of the following applies:
A. The game offer a "3d mode" option within its menus which makes all the necessary changes for you.
B. Appropriate settings for 3d can be set by adjusting a series of options in the game's menus.
C. Appropriate settings for 3d can only be set by editing configuration files with a text editor or typing commands into an in-game console.
I think this is more of a native S-3D support type scenario. We can think about this.
P.S. It would be useful to show the weighting that has been applied to every question (at least during the testing phase). Some of them curently don't tell you how they impact the score so we can't give feedback on them without running the utility dozens of times changing one answer each time.
We can implement this. Though, I think it indicates a score count at the top.
P.P.S. It might also be interesting to add a final question just before the overall score is revealed: "How would you rate the 3D in the game overall out of 10?" This could be useful in callibrating the utility.
Perhaps a subjective portion of the score. Have to think about this. The more fact based it is, the better.
Regards,
Neil