Our First Playtest
Nov 24, 2015
On Friday, we had our first playtest for Thimbleweed Park. My friend Sarah came over and I watched her play the game for close to three hours. Next week Gary, David, and Robert will host playtest sessions with friends and family. Our goal is to do one playtest session every week until the end of the project in our elusive quest to "make a good game".
There are different flavors of testing, each with a different goal in mind.
Bug Testing: This done by testers who are focused on finding bugs in the game. While it is part of what they do, their main focus isn't on if the game is fun or not. They are playing the game over and over for weeks and months at a time looking for bugs and then taking the most elusive ones and figuring out how to reproduce them. They write up very detailed bug reports detailing what the bug is and how it happened. It is grueling work and can leave you hating the game (if not games in general). Good bug testers are very special and hard to find.
Focus Testing: I hate focus testing and will never ever do it. Focus testing is where you bring potential customers in and see if they like something, and in most cases, before it has been built. Focus testing fails because people know what they like, but often don't know what they want. Focus testing is primarily used by visionless marketing departments. I will concede that focus testing might have a place in some kinds of products, but for art - which games are - it has no place. Rant: OFF.
Playtesting: Playtesting usually happens one-on-one or with very small groups of players. You're often watching testers as they play, taking notes and generally being very quiet (if not behind one-way glass). Your main goal is to see how a player approaches the game, where they spend their time and where they are confused and get stuck. Back at Lucas, and I believe it started with Indy, we would have Pizza Orgies. After work, the whole Game's group, plus friends and family would get together, eat pizza and everyone would pay the game for a few hours, then we'd have a large group debrief discussion. These were great because everyone got to chime in and build off other comments.
Alpha/Beta Testing: This is where you're sending hundreds (or in some cases thousands) of copies of the game to players and you're looking for big picture issues. The games often contain analytics so the developers can see how far people are getting and what they are doing. These tests usually involve a lot of people, so you don't drill down for specifics on each player. You're looking for global trends. Alpha/Beta testing is also good for finding bugs that only a fresh one thousand new players can find.
When I do playtesting, I have a set of rules and guidelines I like to follow:
Before the Playtest
◼︎ Tell players you're just going to be watching and not talking to them. Don't engage in friendly chit-chat. Pretend you are behind a one-way mirror watching them. You need to make sure their impression of the game is playing the game, not talking to you. This is very important when players are friends and family.
◼︎ Ask them to talk out loud about what they are doing and trying, but not to ask you questions.
◼︎ Tell them there are parts of the game that are not finished and you might occasionally tell them not to do something so it doesn't break the game.
◼︎ Tell them this isn't a test of their ability, it's for looking at issues with the game. We expect them to be confused and have issues, that's what we're looking for.
◼︎ Try and be as invisible as possible during the playtest.
◼︎ Remember, we're not doing this as a validation that we're doing a good job (that will come later), we're doing this to see all the places we aren't doing a good job. Players not liking the game is a good thing because it's all stuff we can fix.
During the Playtest
◼︎ Don't make excuses for the game. If you see a bug or a glitch, don't say "That's isn't supposed to happen" or "we're going to fix that". Most players won't even have noticed it.
◼︎ If the player takes a break for more than a few seconds, pause the game so we get an accurate count of the amount of time they played.
◼︎ Take notes on what people tried, not just when stuck but also as they explored the game. Did they go off in directions we didn't expect? Did they want to do something fun that we don't support.
◼︎ It's OK to ask people "What are you thinking?" or "What are you trying to do?" if they are not verbalizing, just try not to make it sound judgmental.
◼︎ Let players be stuck for long long periods of time. Don't hint players until they have been beating their head against something for 30 minutes or more. It's agonizing, but let them struggle, there is good information in watching how they struggle and what they try.
◼︎ The goal of the playtest isn't for them to successfully finish the game, it's to watch all the problems they have.
◼︎ If players are obviously stuck and are getting to the point of giving up, don't give them direct hints, instead ask them questions like "What are you trying to do?" and they will often figure it out just by verbalizing their thoughts, this also helps us figure out why people are confused.
◼︎ If they don't solve it, give them subtle hints like "Look carefully at your inventory" or "Make sure you've fully explored the City Hall".
◼︎ Players have a tendency to say "Is this what I should be doing?" and "Is this right?" when being watched, it's best to respond with silence.
After the Playtest
◼︎ Be very careful when playtesting with friends and family. People don't want to hurt your feelings and will tell you nice things. Let them know it's OK to be very harsh and critical, that it helps us make the game better.
◼︎ Every time a player tells you something they don't like, the game gets a little better.
◼︎ You might have to draw the negative comments out of people. Try not to look pleased when players say nice things, this encourages them to just say nice things and not bad things.
◼︎ Don't ask leading questions. For example, don't ask if they liked the music, ask if they have any feedback on the music.
◼︎ Ask follow up questions to probe deeper.
◼︎ Ask them to describe the story.
◼︎ Ask them who Ray is and to describe her.
◼︎ Ask them who Reyes is and to describe him.
◼︎ Ask about the the other characters. We're trying to understand if the right information was conveyed. Don't respond to questions about the characters or story yet.
◼︎ Ask them if there was anything about the story that was confusing.
◼︎ Ask them if there was anything about the game that was confusing or they didn't like.
◼︎ Ask them where they were most frustrated.
◼︎ Ask them if they have any general feedback on the game.
◼︎ Ask these next questions last.
◼︎ Ask them what about the story did they liked.
◼︎ Ask them what about the game did they liked.
◼︎ At this point, you can answer questions and tell them what they might missed, etc.
The playtest with Sarah went very well. She got stumped on a couple of puzzles that need better focus or clarification on our part. She got fixated on solving one puzzle that was leading her in the wrong direction, so we need to make it clear about it not solving the problem it appears to solve. There was one large backwards puzzle we need to fix. We were pretty sure it was going to be a problem and this confirms it. She wanted to call all the phone numbers she saw and we need to make sure they have satisfying responses. Having just played Maniac Mansion, she found all the references funny.
After the first of the year, we'll post a call for playtesters who want to come in and play for a few hours and endure our icy stares, cold silence, and hip lab coats and clipboards.
- Ron
The game looks great. But please dont give away to many reference-spoilers, like the Doctor Fred one :-)
We all know they are there, but it is cooler when I find them my self alone at night :-)
Life is strange.
great read nontheless, i think your approach to playtesting is very good because it happens early enough that things can still be changes and corrected instead of just finding bugs and glitches.
also the art looks more staggering with each new room that is somewhat shown :).
oh btw: after so many visuals, would it be possible to get an audio-teaser as well? maybe one of the shorter music tracks or the title music or something? i know the soundtrack is probably far from finished yet, but id love to have first impressions because it will be really important for the whole mood and atmosphere in the game.
I've also played MI2 with these dark purple boxes behind the verbs back then. For all I know they added the GUI from MI 2 to MI 1 when it was re-released on CD-ROM with 256 colors, albeit 3 verbs had to be dropped then.
http://gamesthatrocked.com/2013/07/19/games-that-rocked-jonny-garretts-world-44-the-secret-of-monkey-island/
I just checked in scummvm my "monkey madness" cd-rom version I bought in 1995-ish and it also has those boxes. If you want I can take some more screenshots, but I'm lazy and won't unless you ask me.
The verbs are also centred, not left aligned. And the icons are icons , not text (I.e. vga version)
P.S. Autocorrect doesn't know VGA?!
http://nerdlypleasures.blogspot.com/2015_09_01_archive.html
http://scummbar.com/games/index.php?sub=info&todo=19
http://www.rogerdavies.com/wp-content/uploads/2009/09/monkey-island-2-first-screen.jpg
The first time I played it, on PC, there were filled boxes.
Here is a screenshot:
http://www.pianetacellulare.it/UserFiles/image/Android/Apps/Giochi/monkey_island_le_chuck_revenge.jpg
http://www.google.com
EGA: http://i.imgur.com/Rqu81Ww.png
VGA: http://i.imgur.com/k55BgjC.png
The only other option would to do:
Left column, aligned left
Center column, centered
Right column, aligned right
This proposed alignment would look a little less funky if you added margins, too.
But, like I said, I think it looks fine.
They will never enjoy the game when it's finished like us.
I believe they deserve a dedicated tomb in the Thimbleqweaefwtver Park graveyard :-)
Back in my C64 days, playtesting always used to be a process of showing the game to whatever friend happened to call round that day, and was often just a few nods, affirmations, and polite suggestions. Nowadays it's incredibly clinical (and almost scientific) in its approach.
Today bug testing is an aspect of software development that I think developers are unfamiliar with. It's good to see Ron & Co are not only serious about doing the playtesting and bug crushing, but actually have a good plan for it.
And that's why things like Arkham Knight happens.
I'd playtest the heck out of it for ya, Ron!
The screenshots are just looking too good.
Can't wait...
I leave the testing to you. I prefer to play the game once it's released :-)
Ciao!
Keep up the great work!
Seeing these screenshots are really great, I'm going to say it again; we need to get our hands on the rights to Monkey Island and then lock Ron and the rest up.. Only to be released when they made 15 or so new MI games...
I have to say that those two games taught me a lot of English words... stale, couch, hamster, slimy, wax... none of those words you can learn in (italian) schools!
2) Is there already music in the test version of the game?
3) What does 'backwards puzzle' mean?
However I was refering to *'peanut flips'*. Though their tastes differ depending on where you buy them. So I wasn't sure how to call them. Anyway I mean the salty ones, without cheese or sugar.
I called them 'Bamba', even though it's a private brand which I had never tried out, since there is an article on the English Wikipedia site everyone can look it up with.
But I must warn you... if you ever come to visit Italy, don't say anyone that you eat Bamba... trust me... :-D :-D
(it's a synonym for drugs!)
Though, I'd rather treat it like a twitch stream, using a front facing camera and post it on youtube as a private video giving you access to it.
Then once the game goes "live" change it to public and get views.
But, that puts alot of faith in your playtesters.
Back on topic: I won't be in the US anytime soon, so I can't come over, but sign me up as a beta tester, will you? :)
Best regards,
Frédéric.
I hope that I will finally be able to open the medical cabinet in the "skeleton room". Or was there a way back in Maniac Mansion?
Or maybe it changes depending on the main color of the character clothes (e.g. blue, since the active player wears blue jacket and pants) ?
Do you have an automated way to capture all of the tester's verb + object combos? If so do you later go and look at meta data that shows you where to perhaps spend some time creating additional jokes? (i.e. if many testers want to put the hamster in the microwave, do you capture that and go back later and create more dialog, and perhaps a new puzzle?)
Ah, how I wish I lived in the States - I'm such a fan of your work Ron and I would have loved to help out :D