|XNA Team Spotlight - Peer Review with Betsy Aoki
Explain to us who you are and what you do.
My name is Betsy Aoki and I am the Program Manager on the Community Games team responsible for the peer review, abuse reporting, and moderation features. I work for Albert Ho, the XNA Community Group Program Manager.
For folks new to Community Games, what is the peer review process?
Peer review is the approval process by which a Community Game gets onto the Xbox LIVE Marketplace for sale. When creators upload a game for approval, Microsoft does not approve the game – the XNA Creators Club Community does. Community peer reviewers make sure the game does not have any prohibited content, runs well enough to review, and that the game’s rating for violence, sex, and mature content actually reflects the gameplay experience.
Can anyone write a peer review of a Community Game?
We have done a scientific survey and we have found Community Games' peer reviewers to be more attractive, better paid, and have more fun at parties than "regular" people… Ok, seriously……peer reviewers have to be premium (paid) members of the XNA Creators Club and they should be able to speak the language of the game binary. They do not have to be residents of the country where the game is to be sold to consumers.
So, what makes a great peer reviewer?
Peer reviewers should be adventuresome, interested in reviewing games that break the mold, and have a keen eye for discovering abuse and game defects. People who speak multiple languages have an advantage – they get to review more games and see more of what’s hot in the game development world before it goes live to customers. Even if some reviewers only have time to review Community Games occasionally, they are helping other Community members realize their dreams of writing a game for the Xbox that millions of people can see and play. Reviewers' comments can change the creator’s code for the better.
I play game imports all the time in Japanese and don’t care what the characters in the RPG are really saying. Why can’t I review a Community Game in a language I don’t speak?
You won’t catch the profanity, euphemisms, and inappropriate cultural idioms that are prohibited content in that language. You may not care what that RPG character is saying when you play the game, but someone who speaks that language will.
Why should I review a game?
Help your fellow developers improve their games and learn about game development. Support the indie games community. Free games! (Peer reviewers get to play the latest Community Games for free during the peer review phase.)
I also think game authors should become peer reviewers because it can only help their game-creation process. Reviewing a lot of games can give you insights into ways to make your own games more fun and more marketable. It also builds up good karma for the time when you need people to review your game for marketplace sale and you ask people on the forums to vouch for your work.
The more successful you are as a peer reviewer, the more weight the peer review system will start giving your reviews. Future plans include reputation features that highlight the Community’s most prolific peer reviewers as a thank you to the community. That’s still under development, however, so don’t hold me to a date yet.
I want my game to pass peer review. What should I make sure to do?
Send your game to playtest first and get the bugs out before sending it for peer review approval. Peer reviewers are not beta testers, so they expect your code to be bug-free and ready for sale. It does really help to peer review five to 10 games before submitting your own. Not only will you get familiar with the standards your game will be reviewed on, you will also be able to determine how good your game has to be to compete in the marketplace.
I’ve come across a game I think is a real stinker, although it’s playable and all the functions work. There aren’t any peer review options for me to communicate this. What do I do? I want to reject this game!
Peer review isn’t intended to address questions like, “is the game fun?” or “will people buy this game?” It helps the consumers on the Xbox LIVE marketplace decide which games to buy.
Peer reviewers should approve games based on whether the content is appropriate, whether the gameplay matches the game’s rating, whether the metadata and screenshots are suitable for all ages, and whether the game has any major game defects. It can be a game you hate, but still ought to pass peer review, if it doesn’t commit any of those sins.
If you really can’t stand a game, or you are a first-person-shooter fan and hate puzzle games, there is no requirement that you review the kinds of games you hate.
But what if the graphics are really horrible? What if the game developer just needs some basic coaching on how to make a game?
This type of feedback should be made in the playtesting phase. Because the feedback is given in a forum open to only Creators Club members, this is a safe and respectful place where you can talk about overall game quality, technical excellence, best practices, and techniques the creator might apply to make the graphics flow better. By the time a game is peer reviewed, the game should be as complete as the creator can make it, and playtesting should be concluded – either through the Creator’s Club, or by friends in exchange for beer and pizza.
Remember, Community Games was never intended to be a managed portfolio. Microsoft employees can’t write peer reviews of these games, and aren’t dictating what kinds of games are approved. The goal of the Creator’s Club is to create a place for games that break the boundaries of what games are. Remember, one person’s pixilated evil is another person’s early Nintendo game joy!
I want to get better at peer reviewing. What else can I do?
We are starting a peer reviewer group for folks who indicated during the beta that they’d like to help mentor other peer reviewers. If you want to help with peer reviews, send an email to firstname.lastname@example.org. For more information, please read our FAQ or post your questions on the forums.