Peer Review: The Evil Checklist
The Evil Checklist

See the Not So Evil Checklist for a list of things that are not reasons to fail a game
See the
Legal FAQ for a list of legal reasons why you should and should not fail a game
See the
Language FAQ for a list of language and translation reasons to fail or not fail a game

The Evil Checklist contains the most common reasons games fail Peer Review, plus a list of test cases that are designed to flush out these problems. This list is meant both for developers, who want to test their own games before submitting them to review, and for peer-reviewers, as a suggestion of how they could test a game they're reviewing, and what cases to consider.

Most Common Review Failure Reasons

  • The game contains any kind of forbidden content, as defined by Microsoft in the review rules.
  • The game crashes anywhere; even crashes in obscure situations will result in a failure if found by a reviewer.
  • The game hangs or appears to hang (even hangs in obscure situations will result in a failure) or exhibits other game defects.
  • The game can't be played with any controller index. For example, the game forces you to play with controller #1, or it allows you to start the game with controller #2 and navigate the menus, but then doesn't react to input from that controller in the game, and similar stuff. A user should never have to switch controllers from the one they have in their hand for any part of the game, including purchasing screens.
  • The game uses fonts so small that important parts of the text are unreadable on some TVs. If using text, use a 14-point or higher font to insure the font can be read on standard-definition TVs.
  • The game doesn't observe the title safe area, which causes important information or important parts of the gameplay to be off-screen on some TVs.
  • Misrepresentation of the game or promotional materials that aren't appropriate for all ages, including screenshots, box art, etc.
  • The game doesn't have a trial mode that showcases the actual game experience, or the trial mode is misused to provide the game for free.
  • The game isn't a game, but only a wrapper for a video or several videos, which is a misuse of Xbox LIVE Indie Games (XBLIG) as a video hosting service.

Ratings/Classification Issues

  • 1080p: If your game sets the back buffer to anything other than 1920x1080, then your game does not support this. 1280x720 isn't 1080p. Note that if you do 1920x1080, then the Xbox 360 console will not scale to 480i for you. You need an extra code path for that scenario.
  • Online Players: If your game supports online data sharing (levels, high scores, etc.), that doesn't mean it's a 2-31 player game. This classification appears on the marketplace and gamers will expect a multi-player experience. This is a valid fail reason. See this thread/post for details.

Test Cases

The following test cases are meant to flush out the issues mentioned above. Note that these test cases are meant as a guideline and inspiration only, as a basis for your own testing procedure. They are a guide only, and aren't meant to be a complete set of tests. A game may very well fail review even if it passes all of the tests mentioned here, if the reviewer finds some other problem. Be creative when testing; don't rely solely on this list.

Basic Tests

Test Case 1: Multiple controller support. Start the game with the second (or third/fourth) controller. Verify that the game is playable in various scenarios. For example:

  1. Have controller 1 switched off with no profile signed in during the test and your Xbox LIVE Indie Games profile logged in on controller 2.
  2. Have controller 1 switched on, again with no profile signed in during the test and your Xbox LIVE Indie Games profile logged in on controller 2.
  3. Have controller 1 switched on, with your CCO profile logged in, and no profile logged in on controller 2. The game must not simply ignore controller 2 in this situation, but must still allow you to play with controller 2; however, the game may ask you to login a profile on #2 first.

    ...or other multi-controller scenarios...

    (Developers: Support for switching controllers mid-game isn't required. The player usually implicitly declares the active controller at the Press Start or Main Menu screen. See "Caveat #6" below for more controller issues you should be aware of.)

Multicontroller support in multiplayer games:

  1. Verify that the multiplayer game is playable with various controller combinations (and not simply assumes controllers 1+2 for players 1+2, or similar).
  2. If the multiplayer game supports a varying number of players, play the game with the minimum required number of players while having an additional controller connected (try both with and without a signed in profile on this controller). Verify that the game doesn't force this additional controller to partake in the game. (Developers: See "Caveat #6" below for more details.)

Test Case 2: Profile logout. Log in with an Xbox LIVE Indie Games Membership gamertag on the first controller. Log in with another gamertag on the second controller*. Play the game with the second controller. Then at various locations in the game (for example, in the main menu or in the middle of the gameplay), use the guide button on the second controller and sign out the second gamertag. Verify that the game handles this gamertag sign-out correctly. May be repeated to check sign-out behavior at different locations in the game.

*For easier testing of Test Case 2, the second gamertag should either be a local gamertag (not Xbox LIVE enabled) or a gamertag that also has an Xbox LIVE Indie Games membership. This is because if the second gamertag is a Live gamertag without an Xbox LIVE Indie Games membership, then you won't be able to start the game while this gamertag is signed in. If you want to use such a gamertag anyway, the process is a bit more difficult, because you need to do a late sign-in: Keep the second gamertag logged out at first, then start the game with the first controller. When the usual start screen that says "Press Start to Begin" appears, use the second controller to finally log in the second gamertag (when the game is already started). Then press Start on the second controller to play the game with it. Note again: This late sign-in is only necessary if the second gamertag is a Live gamertag without an Xbox LIVE Indie Games membership. For a local gamertag or an Xbox LIVE Indie Games membership gamertag, this late sign-in isn't necessary.

Test Case 3: Storage device support.

  1. Start game with both a hard disk and a memory unit attached as storage devices, and select one of them when asked. Does it start properly?
  2. If so, start the game again and this time close the storage selection dialog with "B", that is, don't select any storage device. Check that the game handles this properly. For example, it either tells you that you have to select a device and re-opens the selector dialog, or it continues without saving (but of course also without crashing).

Advanced Test: If you cancel the selector dialog and the game then shows a message box of some kind, verify that this message box reacts to your input even if you're not playing with controller 1.

Test Case 4: Removing the storage device at any time. If case 3 doesn't cause problems, select the memory unit as the storage device and then remove it. Test removal prior to and during load/save operation, as well as during device selection.

Test Case 5: Video modes, font sizes, and safe area. Start the game in various video modes (whatever modes your TV supports). Check if the game crashes or if anything is drawn in such a way that the game is unplayable (important stuff outside of titlesafe area, fonts too small to read, etc.).

Test Case 6: Marketplace. If the game attempts to show the marketplace page for purchasing the full version while only a local gamertag is logged in (or no gamertag is logged in at all), an exception is thrown. If this exception isn't handled correctly, it causes a crash.

To test if this is handled or not, log in with either a local gamertag or with no gamertag at all (if the game allows this) and run the game in simulated trial mode. Then check that any menu items that would normally allow you to purchase the full game are either hidden or, if selected, don't cause a crash.

Note: Don't use an Xbox LIVE gamertag that simply has restricted permissions (for example a child account) for this test. Such an account won't trigger the exception/crash that this test is attempting to cause.

Test Case 7: Guide open during startup/dialog. Start the game and immediately open the guide (select "Play Game" on the dashboard and immediately hit the guide button while the game is still starting up). The guide will flicker on/off a few times. Wait until the flickering is over and the startup screen in the backup is no longer visible; then close the guide. Does this cause a crash? It’s also possible to crash if you time the Guide when another dialog (keyboard/device selector) is about to appear. Checking Guide.IsVisible is not 100% reliable: Always wrap your dialog code with try/catch.

(If you're testing your own game and actually are getting a crash, see this thread for a discussion about how to avoid this.

Advanced/Special Tests

Test Case 8: SpriteFont issues. If the game has some sort of text input by the player (for example, to enter your player name for a highscore list), try entering a non-standard symbol and/or foreign character. This can cause a crash if the font chosen by the developer doesn't support this symbol/character. (Developers: Use SpriteFont.Characters.Contains(char) to check if your font supports all chars entered by the player.)

Test Case 9: Locale issues. Start/play the game while your Xbox dashboard is set to a non-English language, to check for locale-specific bugs:

  1. Set the Xbox to a language that is associated with a locale where a "," is used as the decimal separator in float values (instead of the "." used in English/US type locales). This allows you to check for the floating point parsing bug (see "Caveat 1" below). Games that are affected by this bug exhibit all kinds of strange behavior. (Not starting correctly, hanging at some point during data loading, drawing everything totally out of scale, etc.). Languages that can be used for this test include Spanish, French, and German.
  2. If the game outputs number values in formatted form by using thousands separators, then set the Xbox to a language that is associated with a locale where a non-breaking space is used for the thousands separator (instead of the "," used in English/US type locales), so that, for example, the number 1,234,567 would look like "1 234 567" instead. Then do something that causes the game to output a number >= 1000 (so that the thousands separator is required). Make sure that this doesn't crash the game (see "Caveat 4" below). Languages that seem to use the non-breaking space separator include Russian and Polish.

Test Case 10: Network/Multiplayer issues. Special case for LIVE multiplayer games: With an established multiplayer network session, try pausing the game (both with the game's own pausing functionality, if it has one, and by opening the guide). Does the game handle this correctly? If the game indeed pauses (and not simply continues in the background), then this should not interfere with the other players (at least not in an unexpected fashion) and it must also not lead to a network timeout if you leave the game in this paused state for a while (which happens if the developer forgot to update the network session in paused mode).

Test Case 11: Verify claimed capabilities. When a game is submitted, the creator checks a set of capabilities, for example how many players the game supports, if it supports co-op play, the supported resolutions, if it supports Custom SoundTracks, etc. Check if all of these claims are indeed true, as far as this is possible for you.

Tip: To check if Custom SoundTracks are supported, try playing some music through the Xbox from a CD or your music library while the game is running. The game must then automatically switch off its own music and only play sound effects.

Test Case 12: Screen resolutions. If the game follows the best practices and simply sets the screen to 1280x720 (720p) then there are no reported issues. However, if the game uses 1920x1080 (1080p), then the code must have a special code path to deal with 640x480, as the Xbox console can't scale 1080p down to 640x480. In addition, note that any user with a VGA cable can set many different resolutions, some with odd aspect ratios. If the developer has written code to change behavior based on screen size, then one of the VGA resolutions may not have been tested.

See http://forums.xbox.com/28011158/ShowPost.aspx for a case in the wild.
To test, set 640x480 and other odd resolutions in the dashboard.

See http://hardware.teamxbox.com/reviews/xbox-360/40/Xbox-360-VGA-HD-AV-Cable/p1/ for the resolutions and aspect ratios.

Test Case 13: 'Restricted' XBLA terms. Microsoft has asked that developers not use the words Achievement or Leaderboard for features with similar functionality. See http://blogs.msdn.com/xna/archive/2008/12/12/creators-club-communiqu-11.aspx and see for discussion and alternate names. Check that the game doesn't use any of these terms either in-game or in the description, screenshots, or video. If they do then fail the game for 'Inappropriate for Xbox LIVE'.

Test Case 14: Guide.SimulateTrialMode. Games that leave the Guide.SimulateTrialMode flag on during review should be failed even though it has no effect in the marketplace because reviewers can't follow all code paths to determine crashes/hangs or bad content. See this discussion for details.

Test Case 15: Test Media library APIs with a media server/fileshare. If your game uses the Media Library APIs to iterate through songs or albums, then ensure you test with your Xbox console connected to an external file store such as Media Server. While the APIs will eventually return, there are orders of magnitude differences in speed compared to your Xbox console's hard drive or USB sticks. It's likely that your game will appear to users to have hung or crashed.

Development Caveats

The following are common caveats that have been encountered by developers several times, and which have required bug-fix releases or various resubmit-cycles during review. The below caveats are for the benefit of developers who use the "evil" checklist to test their own games before release.

Remember to also have a look into the Best Practices for Xbox LIVE Indie Games.

Caveat 1: Floating point number parsing. If your game uses textual representations of floating point numbers in its save data (either in game data or in game saves), and then needs to parse this text back into a float value while loading, then you must be careful to avoid culture/locale-based parsing problems. (Test case 9 above is an attempt to find bugs caused by this.)

Background: Different countries use different characters as the decimal separator. For example in the US and UK, a dot "." is used (such as 1.5 or similar). But Spain and Germany use a comma "," (such as 1,5 or similar) for this purpose.

Whenever you convert a float into a string with "value.ToString()" or parse text into a float with "float.Parse(text)", the .NET framework will use the decimal separator that matches the currently selected locale. On the Xbox console, the locale in turn is defined by the language that the Xbox dashboard is set to. So if you have a value of 1/2 and do "value.ToString()" on it, then on an Xbox set to English, the resulting string will be "0.5", but on a Xbox console set to Spanish, the resulting string will be "0,5". Similarly, "float.Parse(text)" will also expect the separator that matches the currently language, so if the text "0.5" is parsed on a Spanish Xbox (which expects "," as the separator), the result will be incorrect.

This has a lot of potential for bugs: Assume that you deploy your game with data that uses the "." seperator and this game is played on a Spanish Xbox. Loading your data will fail, or at least give unexpected results. Or assume that someone plays your game on a Spanish Xbox and saves his game. In the game save, a "," will be used as the separator. Now the player changes his Xbox language to English and restarts the game. When is game save is loaded, the parser will expect a "." as the separator, which does not match the "," that is actually in the data, and will thus result in a loading error.

Therefore, you should under all circumstances avoid plain calls to value.ToString() and float.Parse(text).

Instead, use value.ToString(CultureInfo.InvariantCulture) whenever you need to convert a float to a text, and float.Parse(text, CultureInfo.InvariantCulture) whenever you need to parse this text back into a float.

This will have the effect that the data is always written and parsed with a dot "." as the separator.

Of course, you then also need to make sure that all data that you deploy with the game also uses the dot "." as the decimal separator.

Note: Simply "plussing" a float to a string is comparable to using the plain value.ToString(). So you should avoid doing stuff like "this is text" + floatValue + "more text" when saving data. Instead use "this is text" + floatValue.ToString(CultureInfo.InvariantCulture) + "more text".

And of course the above applies only to saving data. If you want to convert a float into a text for display on screen, you should continue to use the normal value.ToString(), so that each player sees the value with the correct decimal separator for his or her country.

Note that the XMLSerializer is not affected by this problem. It is only something that you need to look out for if you're writing/parsing float values manually.

Caveat 2: Corrupting saved data by overwriting an existing file with shorter data without truncating it. When you open a file for writing with FileMode.OpenOrCreate and the file already exists, then the file won't be truncated, but will simply be opened. If you then write data into this file, but the new data is shorter than the data that is already in the file, then at the end of the file some trailing bytes from the previous data will remain. Depending on your save-format, this may corrupt the saved data, if during a subsequent loading of the data, the trailing old bytes will cause a loading error. This is the case, for example, if the save data is serialized XML, as there then will be superfluous XML tags from the old data after the actual end of the new XML data. Unfortunately, the XML serialization example in the XNA Game Studio documentation actually tells you to use FileMode.OpenOrCreate, which is not good advice, because of the above problem. Instead, you should use FileMode.Create; this will create the file if it doesn't exist, or truncate it if it already exists, so only the new data remains in the file after you write it. This should be good enough when coding for Xbox and Zune. The "safe save" method recommended elsewhere isn't necessary then, but is still recommended when coding for Windows.

There's not really a good test case for finding bugs caused by this caveat. If you want to test for this, your best bet would be to force the game to save and load data repeatedly. For example, do something that requires saving, then restart the game to force loading, and then repeat a few times.

Caveat 3: Storage device handling. Incorrect handling of the storage device selection is one of the main reasons why games fail peer review. The developer should be aware of a few facts about storage device selection:

  1. The device selector dialog appears only on an Xbox that has more than one storage device (i.e. a hard disk and a MU). If the Xbox has only one storage device, no dialog ever appears, but the single device is selected automatically. Therefore, to test the various-selector specific cases, you really need a MU.
  2. The player is able to cancel the device selector dialog with "B", which means that the game proceeds without a selected device. The game must handle this or it will crash as soon as it attempts to load or save.
  3. If the player selects a MU, he is able to later disconnect it in the middle of the game. If the game doesn't handle this, it will crash as soon as it attempts to load from or save to this device.

Caveat 4: Number output with thousands separator. Your game may crash if you're trying to output formatted number values >= 1000 in a format that includes thousands separators, if your SpriteFont doesn't include the thousands separator character that is used by the locale that the Xbox is currently set to.

Background: This caveat is related to Caveat 1 in in that it's also caused by locale-specific differences in number formats. Depending on the locale that the Xbox is currently set to, the thousands separator may be a different character than the "," used by the English/US-locales. Some locales seem to be using a non-breaking space as the thousands separator. This looks like a normal space, but has an ASCII-value of 160 (0xa0), instead of 32 (0x20). So a number that looks like "1,234,567" in an English/US-locale may look like "1 234 567" in a different locale. Now, if your SpriteFont doesn't contain the non-breaking space (if, for example, because you only specified a character range up to 128 when you created the SpriteFont, so that the code 160 wasn't included), then your game will crash if it tries to output a formatted number that is >= 1000 on an Xbox with such a locale.

To avoid this problem, either make sure that your SpriteFont contains the necessary character (160 / 0xa0), and/or when formatting your numbers, make sure to use a format that only generates characters that are supported by the SpriteFont(s) that you want to use to output them.

Caveat 5: Using Guide.BeginShowMessageBox before PlayerIndex is known. For a call to Guide.BeginShowMessageBox, you must specify a PlayerIndex. This means, that only the controller for the given index will be able to control the dialog. Again, this means that you should never use this to open a dialog before you have actually detected the correct PlayerIndex. Such detection is usually done via a "Press Start to Play" screen, or something similar. So you should not use Guide.BeginShowMessageBox before this screen, because if you do, then you have no idea what PlayerIndex to supply. And if you supply the wrong one, then the player won't be able to control the dialog (which looks as if the game is no longer responding to input and is hanging).

A common case where this is forgotten is the following: First the storage selector is shown (player independent); if the player cancels it, the game then tries to show a message box (to inform the player that saving is disabled, or that he has to select a device, or to ask him if he wants to select a device or continue without saving, etc.). If all of this happens before the PlayerIndex is detected, then most likely the default PlayerIndex.One will be supplied when showing this message box, so for anyone not playing on controller #1, the message box will not react to input.

Caveat 6: Controller Caveats. The following items are some controller-related issues that have led to problems and review fails. Developer should be aware of these issues:

  1. Wired guitars and drum kits will often grab controller #1 and #2 and the auto logged in profile, so people often play on controller #3 or higher with their normal controller and may not be logged in.
  2. Thumbsticks on some controllers don't register [0,0] even when not actively moved, and guitars (or other controllers) will make the right thumbstick register != [0,0] due to whammy bar and tilt. Therefore, such controllers and guitars can cause trouble if you try to determine the active controller by polling button presses or by polling thumbsticks (see http://forums.xna.com/forums/t/40701.aspx). The general advice is therefore not to do this. Instead, make a start screen that checks for "Start" and "A" button presses to determine the active controller.
  3. Beware of the incorrect condition in "for (PlayerIndex p = PlayerIndex.One; p < PlayerIndex.Four; p++)" or similar, because it will miss controller #4. There have been several games that don't respond to #4 as a result. Test with a fully loaded console. (The correct code would use "<=" in the condition instead of "<").
  4. Regarding multiplayer games: Sometimes people just want to watch but not play, or a wired guitar or drum kit or similar may still be connected. So in a multiplayer game, you can't simply assume that every connected controller equals a player who wants to partake in the game, even if a profile is signed in on that controller. Therefore, you can't simply force every connected controller to be a player in your multiplayer game. Instead, to select the active controllers that want to partake in the game, implement an "[A] to join, [B] to leave, [Start] to play" UI to allow any controller to play or not. It's OK to assume that the controller that started the game wants to play, so feel free to already include it if you want.
  5. Don't use a profile that is logged in another controller to let a player do something that wouldn't be possible with the profile that is logged in on the controller the player is using. This circumvents security features and also can be very confusing for the player. Or more generally: To perform an action that requires a PlayerIndex, do only use the PlayerIndex of the controller that triggers/executes the action. If the action isn't possible with the profile logged in on that controller (if the profile doesn't have the necessary rights, or no profile is logged in at all), then handle this accordingly, in a way that is not confusing to the player. (For example disable the menu item, or show an info message, or bring up the sign-in dialog, etc.).

    This may seem self-evident, but there have been games that ignored this, for example to let a player into an online match even though he was playing on a controller without a Gold-profile (by using the PlayerIndex of a second controller for all network access, a second controller on which a Gold-profile happened to be logged in). Do >strong>not do this.

    Note that the eight-minute trial screen in XNA Game Studio Connect is an emulation for dev purposes. It will let anyone purchase the game without any checks and is outside of a developer's control. Do not fail for this. After release, the marketplace will check correctly.

You're encouraged to read the Not So Evil Checklist for a list of things that historically are not reasons to fail a game in Peer Review, but are important to consider/fix to increase your games' quality.

Questions can be asked in the forums, or sent to creators@microsoft.com.

var gDomain='m.webtrends.com'; var gDcsId='dcschd84w10000w4lw9hcqmsz_8n3x'; var gTrackEvents=1; var gFpc='WT_FPC'; /*<\/scr"+"ipt>");} /*]]>*/
DCSIMG