Capturing Customer Insights at Events

Last month 350+ WooCommerce developers, designers, freelancers and agencies from all around the world gathered in Seattle, WA for the 3rd annual WooCommerce Developers Conference: WooConf. The schedule was packed with 2 days of talks from industry experts on design, development, user experience, scaling WooCommerce, and SEO, to name a few. This week Patrick Rauland posted about 3 things we hope you take away from WooConf, including a link to the recording of all sessions, so that’s not what this post is about. Today I want to walk you through the Product Research Lab we setup, with tips on how you could do this at your next event.

Working in a distributed fashion like we do at Automattic, it’s a rare treat to make our way out of our office, into the real world, talking to customers face to face. As colleague Jeff Golenski suggests, you can’t create successful products if you don’t leave your desk. There are a growing number of cities that have started a WooCommerce meetup group, but not all of our staff live near these communities. WordCamps can also be a great way to connect WordPress developers to actual users, but without a structured plan to capture insights, you may risk losing the opportunity to ask the right questions, or have the ability to properly capture the insights you gain thru conversations. As anyone that has been to an event before can probably attest, if you don’t jot a quick note down about the discussion you just had it very likely can get lost in the 100 conversations to follow.

What is a Product Research Lab?

To setup the Product Research Lab at WooConf we took inspiration from the Atlassian Design Lab, who proudly explains how they use their conferences to listen to customers. We setup 3 stations with 2 representatives from our product teams to research 3 separate areas of the WooCommerce experience. One person led the interview while the other helped take notes. All sessions were recorded on audio or video. One station researched how customers discover, try, and buy extensions, another researched how customers manage the extensions they have purchased, and the 3rd station researched the new setup process for the WooCommerce plugin.

One difference from how Atlassian runs their design lab, we decided to try and pre-schedule sessions, to reduce the risk of teams sitting around with no one to talk to. Since this was our first go at this, we weren’t sure what type of response we would receive. Much to our delight, within 48 hours signups going live, 100% of the sessions were filled.

Planning for the Worst

To get started our team helped brainstorm a list of “What if’s” on ways this effort could fail. Imagining all possible scenarios before they occur can help you be more prepared for ways to mitigate failure. Here are some we came up with:

• What if people don’t show up?
• What if people hang on past time their time allotment? Or what if we finish early?
• What if the noise level gets too high?
• What if we’re not prepared?
• What if we don’t properly document the sessions?
• What if we fail to record?
• What if gather information without analyzing with most useful lessons learned?
• What if we don’t ask the right questions?
• What if the participant doesn’t do the thing we’re testing?
• What if we don’t iterate on site?
• What if all research stations are full and someone wants to participate?

We planned the space as spread out as possible to avoid noise levels getting to high and used USB mics to help record the audio from the sessions. We used our typical tools for hosting remote user research including a timed interview guide and Zoom to record the screen + participant. We checked after each session to confirm it was recorded, and we reminded our teams to iterate on-site as needed. If a participant didn’t do the thing we were testing, we had backup questions ready so we could steer the conversation in a different direction. If people didn’t show up we would take to the halls to recruit.

We didn’t get everything right, but we did accomplish our main goal getting our teams more comfortable talking with customers, learning how to ask the right questions, and bringing those insights back to our larger product teams so we can help prioritize improvements based on the feedback we received. This was the first time for some of our team to participate in a live research session, so it was a great learning opportunity for all.

As designer @jaykoster put it:

“I really enjoyed the experience. It was my first time leading interviews and I was definitely out of my comfort zone to begin with. But I learned quickly and began to form a ‘script’ which felt organic and genuine. I learned so much from listening to our customers stories and I’m excited to do more user research!”

And lead developer Justin Shreve explained:

“I thought it went great! I feel like, with the setup wizard specifically, we found a lot of actionable issues to fix. For example, people had problems finding currency and country in the drop downs, and every single person ran into. We now know what we can do to fix that. It was also great being able to talk to customers and figure out their setups and various things they use to run their business, so I as a developer can learn more about the features our users actually need or use.”

Planning for Success

We spent 3 weeks preparing plans for the lab, breaking out deadlines one week at a time.

• Week 1 – Space planning, logistics, scheduling tools, and name for room finalized.
• Week 2 – Schedule for sessions and marketing plans for promotion/sign-up finalized.
• Week 3 – all research/testing plans finalized to be practiced and refined one week prior to the event. Marketing to begin promoting sign-ups.

Space Planning

We were fortunate to have a large spare room to dedicate for the lab. If you don’t have a dedicated space where you can reduce the chance for distraction or noise levels getting too high, you may consider adapting your research plans to the space available. An interactive wall vs. a structured interview or usability test for example.

Try to understand what the expected traffic flows will be at the event in relation to the space you have. For example, are all main sessions upstairs limiting the time attendees are most likely to be near the lab space? Or are you in a crowded, noisy, sponsor area with times that it’s less likely to be busy when people are attending a talk? With this scenario it might be best to plan scheduled research sessions, taking advantage of that downtime.

Our team thought it worked well to have pipe and drape dividing the space a bit to help with this. They also liked the setup of a highboy table and taller chairs to sit at. This allowed for an easier walk-up, relaxed, welcoming experience. You will also want to consider running power to the station if using a laptop for the session.

If you don’t try to pre-schedule sessions or would like to organically attract participants you’ll want to plan for a high-traffic area, being mindful of noise distraction.

Logistics

Similar to Atlassian or Apple, we had greeters standing at the entrance welcoming attendees, checking-in the ones that had a scheduled session on an iPad. If someone didn’t have a scheduled appointment or a station was still wrapping up a previous session, we invited participants to interact with a few self-led question stations that we setup. We used a Typeform to help sign up participants for future research opportunities. Greeters would confirm when the assigned station was ready, led the participant to the station, introducing them to our teammates. If greeters weren’t busy, they could lead conversations with attendees.

Scheduling

We used Calendly to allow attendees to sign up for a 20 minute session in 30 minute time intervals. We hosted sessions 1.5 hours before and after lunch, which also overlapped with the breakout sessions that were on the same floor in an attempt to maximize attendee discovery. These time scales worked well for us as we didn’t want to take too much time away from the event for the attendee and we didn’t want our teams getting too fatigued with too many sessions at a time. 3 per block was a good pace.

Marketing/Recruiting

One week prior to the event a newsletter was sent to all attendees informing them of important event info, but also asking them to sign up for a session. We planned to share via social media and in subsequent mailers to attract more signups but this wasn’t necessary based on the success of the initial mailer. We also had our MCs and a few speakers mention stopping by the lab to check it out. If you have speakers talking about the area you are researching, having them mention the research opportunity and where to go to participate can be really helpful.

Knowing your Audience

It’s very important to know who the event attendees are, what their role is (or isn’t) with the product of service you are researching. When attendees purchased a ticket we asked what their role with WooCommerce was to learn that:

• ~70% build stores for themselves or others
• ~40% say they build extensions
• ~32% say they manage a store themselves

This info allowed us to plan our research accordingly. If you don’t have access to this data from attendees, using a few screener questions can help. If you pre-schedule sessions you can ask these questions when they sign up, or you can have the greeter use screener questions to qualify participants based on certain qualifications.

In either case it’s best to be prepared for the likely situation that someone stops by that isn’t the “ideal participant” but is still interested in participating. Having a clear path for these attendees, such as unmoderated testing/feedback opportunities and a sign up for future research sessions can help give them ways to still contribute.

Preparing Test Plans

Now on to the most important question of them all: what are you wanting to learn?

Once you take into consideration the space you have available, what the expected traffic flow is like, and the amount of time you have to complete a session, you can begin to tailor your research plans accordingly. You’ll want to:

• Prepare how you document and record each session.
• Work with key stakeholders to determine top questions the team has.
• Create an interview guide with timed sections for each area.
• Prepare any prototypes, workflows, or setup you need for the research sessions. If you are doing prototype testing consider how the participant will need to interact with what you are testing.
• Have your teams practice with the complete setup at least 1-2 times before getting on-site, iterating as needed.
• And most importantly make sure your teams plan for plenty of time to analyze results and communicate these to the relevant teams afterwards.

At the end of day 1 our teams spent an hour talking about what we learned, recognizing if there were any areas we wanted to iterate for day 2. The sooner after the event you can get teams to do this, the more they will remember from each session.

Read more about @kellychoffman’s perspective as a moderator here, including specifics of how we stricter the session.