#35 On UX Research Participant Recruiting

Pants. I put them on every weekday

 

How do you actually get the people to talk to for your research studies? Ten years ago, the researchers I knew were recruiting on Craigslist for moderated sessions. Now there’s a proliferation of services, but as you’d expect from a relatively new and unconsolidated industry, there’s a confusing array of overlapping options. Here are the options we’ve gone through:

Doing It Yourself

This is a temptation, especially when budgets are tight. You think you can save some money by cutting out the middleman. I haven’t found it to be worth the effort. Someone has to advertise the study, filter and select participants, remind them to show up, and remind them again. It’s a lot of overhead to manage and I haven’t found it to be cost-effective to try to do it internally. A very large business might create a position that does exactly that, but at my scale it’s expensive to have designers and researchers doing it.

 

Maybe you think you could save a whole lot of money by testing with friends and family to save both recruiting fees and test subject compensation. This is a bad idea. Even if friends and family are miraculously unbiased, they’re pulled from a very artificially constrained pool. You are unlikely to get a representative spread of demographics unless your product is targeted exactly at people like you and your social circle. 

 

Using a Dedicated Service for Recruiting

In this case, you’re using a service to leverage their reach and efficiencies. It’s important to note that in this case, your recruiting service is different from your testing platform. I’ve done this two different ways for two different use cases.

 

Dedicated online UX Research recruiting services give you some fine-grained control over selection for your study. They’re generally good for finding common demographic-style groups. Location, age, income, etc. There’s a couple of big downsides. For onsite moderated, I’ve generally had a 30%-50% no-show rate (and recruited extra upfront to account for that). For remote moderated, tech support falls on you since you’re moving them from one system to another. So you have to account for the burden of writing tech instructions and shepherding people into whatever platform you’re using, which they probably won’t pre-install no matter how many times you remind them. In both cases, you’re still stuck with the overhead of managing the scheduling and communication. 

 

The other use case is for finding very niche user groups for onsite moderated studies. For example, I once needed people who owned a specific range of models from a specific Japanese tractor manufacturer. The online services simply aren’t going to find these people. In which case, I turn to local market research companies. In Austin, Lone Star Market Research has done a fantastic job of finding uncommon people. This route costs more than the online alternative, but they handle scheduling and communication and can find people other services can’t. It also has a much lower no-show rate, hovering around 10%. 

 

Using a Dedicated Service Bundled with a Platform

In this case, your testing platform offers a recruiting service as an add-on. It might be a higher level annual product price tier to access their panel, or it might simply be an additional cost. I’ve used these for recurring studies like doing RITE method validation on a project every week. The upside is that they handle recruiting, scheduling, and communication. And, maybe most importantly, they handle the tech support since you’re testing on their system. So it’s a much lighter burden on your staff. 

 

There’s a couple of downsides. One is panel size. With one major service, after about 8 weeks of ten users per week, they started trying to recycle test subjects on me because they were having trouble finding new ones. Considering my target users were Texas-based grocery shoppers, it wasn’t a reasonable limitation. I’ve also experienced fraud, with what I would call “professional test subjects” who had obviously lied about their location (and which was later verified by the service and we were refunded). One simple, obvious limitation is that you’re locked into using their system, which precludes also using them to recruit for onsite non-platform studies. 

 

Summary

We’re still at a point where there’s no single solution, at least not for our variety of needs. There have been some acquisitions in the last year (like UserZoom’s acquisition of Validately) that point to some impending consolidation. Perhaps in a year or three, there will be a one-stop-shop. For now, I use platform-paired recruiting for recurring usability studies, unbundled services for stand-alone studies, and local market research companies for hard-to-find users.