As you might have read in previous flow posts, last week ended Automattic’s 8th annual Grand Meetup, where 600+ of us made our way up to Whistler, Canada for a week packed full of conversations, meetings, food, drinks, and an endless amount of opportunities to never stop learning.
Automatticians have the option to chose to work on a project, take a class, or help out in support for the week. This time around I had the pleasure of teaching a class on customer research with my colleague Mike Shelton. 3 days is not a lot of time to dig into such a deep topic, particularly knowing you could spend 3 days alone just learning usability testing. Rather than going into detail on everything we taught, I wanted to share some lessons we learned getting a group to conduct usability testing for the first time, in one day’s time, with Gutenberg.
Planning for a test
As with any research project, the first step is to define what it is you are looking to learn. I like to do this by brainstorming questions I have, involving stakeholders to learn what questions they have, and mapping out the landscape of the topic to be explored. In the interest of time we had to help skip the class forward a bit, seeing they’d never used Gutenberg themselves before, we ended up pointing them to some resources already available as others in the community have already contributed to user testing for the project.
Lesson learned: you need more than a few hours to effectively teach basic usability principals, structuring the research, composing and conducting the test. 2 full days on this alone would have been more ideal.
Once they got the lay of the land a bit more, it was time to define the tasks they wanted to test. Without being involved in Gutenberg development it was a little difficult for the class to know what it is they should be testing. At Automattic we have adopted the idea of defining Jobs To Be Done (JTBD) for development. So even though Gutenberg was new to them and has fundamental changes to how we compose content, the basic concept right now is the same as what we are all used to: composing content. Framing it as JTBD helps us keep the customer in the forefront, giving us a narrative for the task needing to be completed.
Sandy, a 42 year old blogger wants to upload a gallery of images from her latest recipe.
Aryan, a 22 year old musician wants to share his latest song and video from YouTube.
Lesson learned: There are definitely benefits to guiding a team through a new project they develop themselves, in order to have all the background knowledge needed for research. Though, getting the team involved in Gutenberg testing, something that can contribute directly back to the WordPress community, seemed like a bigger win. This was made especially evident at the end of class when our colleague Kellie Peterson mentioned:
“I was never exactly sure how I could contribute back to (WordPress) core until going thru these usability tests today. I feel like this is an area I’m interested in helping push forward and I have a better idea of how to do that now.”
If you’re interested in getting started on usability testing yourself, Susan Farrell from the NNGroup does a great job breaking down How to Conduct Usability Testing in 7 Simple Steps. And we have Gutenberg testing plans and next steps in development right now ready for you to get involved.
Preparation is Key
When you think of teaching someone how to do usability testing for the first time, there are so many things they need to know once they’ve defined the research to be done. How to recruit, schedule, write a script, what tools to use, tips for moderating and documenting a session, understanding and managing bias, the list goes on! Knowing time was limited and that we had tons of Automatticians nearby, we were able to skip recruiting and scheduling and instead introduced the teams to guerrilla testing, where you just walk up to someone and ask them if they’d be interested in testing it out.
This seemed to be the easy part. The more difficult part was giving the teams all they needed to know to set it up, then actually going thru the process. For example, since Gutenberg is a new feature plugin it’s not as likely to be installed on someone’s site yet. The goal was to research the experience of composing content with Gutenberg, not installing it. This meant the teams needed to setup their own test site with Gutenberg pre-installed as to not waste any test participants time. Ideally, with any usability test you have the person test on the device they use most often, but due to the interest of time we had to make an exception in this case.
Next, they had to make sure they had enough direction for the test participants and that content was easily available. Again, as to not waste the test participants time or distract from the test at hand by asking participants to come up with a post to compose off the top of their head. This meant our teams had to come up with their own post ideas and what they wanted to look out for.
Lesson learned: In the interest of time it could have been better to help define what it is that should be tested, with a test plan they could execute. Although, there was value in the class going thru this process on their own, giving them a test plan to follow would have allowed for more testing time and ideally iterating the test as they went. This would have been a greater tradeoff.
Another lesson learned: People are eager to help with these types of tasks once you give them some direction. I think the most successful community approach for testing Gutenberg would be to make clear test plans for others to follow.
Iteration FTW!
Anyone that has tried usability testing before can likely attest, once you do those first few sessions it feels exhilarating! You’re able to work thru the initial nerves getting setup, remembering all the things, talking with someone new for the first time, and leading them thru the test. Hearing feedback firsthand and seeing someone else interact with the product can produce so many ah ha! moments. Once you take yourself out of the testing equation and hear how other people think of things, building their own mental models, you start to expand yours.
Even with the most thorough preparation in the exploration phase, all the preparation in the world will never produce a perfect test result. So the key after doing 1-2 is usually iterating on that test. Whether it’s the script, how you structure or present tasks, or even the product itself being tested. This is why testing early prototypes before development can be so valuable, otherwise you run the risk of developing something completely off base. Just listen to what James Dyson thinks about Thomas Edison related to this very topic.
Lesson learned: We didn’t plan for enough time to allow groups to test 1-2 times and then iterate. Next time, I’d plan a full day for testing, iterating, and testing again.
Analysis Immediately After
Once you’ve conducted the research it’s time for all researchers to come together and share their results. It was exciting to see that there were some overlapping feedback among the two groups tests. The initial interaction going from post title to post content wasn’t as seemless as the participants wanted.
Unfortunately, one of the groups learned a tough lesson early on, they thought they recorded their sessions only to find the file didn’t save. We all watched the videos that did record and shared thoughts about what we saw and heard and that was all we could squeeze in by the end of day 3. In a world with endless time, we could have spent at least half a day reviewing, summarizing, and writing up our notes then and there.
Lesson Learned: When you have a limited amount of time to teach a topic as in-depth as usability testing, it’s equally if not more important to make sure you schedule in enough time for review and analysis, otherwise you run the risk of insights getting lost. Write up your notes as soon as possible and make sure to tailor your report to the audience reading the findings. Not everyone will need to know all the details of the test, highlighting key takeaways with short video clips can be more effective than a multi-page report for some.
One more lesson learned: One thing I didn’t realize going into the class was that we would be sharing the space with a group of developers working on a Gutenberg project. Knowing that and looking back, we should have invited the developers to take 30 minutes out of their day to review the results with us live so they too could learn first hand from the tests. Next time I’ll be sure to try and include any stakeholders possible to help review the work, which would help everyone learn more, students, designers, and developers alike.
Last lesson learned: We didn’t take enough photos. I only got the one. Whomp whomp.
In Summary
While we might have underestimated the time we needed to allot for some areas, and possibly could have cut some corners giving the class test plans and guiding them more through the process I’m glad we did it the way we did. Just like with customer research, the only way to know is to try it out and iterate as needed. We’re looking forward to continuing to refine the content we taught in this class, eventually making it a public resource for anyone in the WordPress community that is interested.
The more we can all improve our research skills, understanding and building empathy by talking to actual people using WordPress, the more WordPress and the community will benefit in the end. Will you join us?
Automattic Design 