Roberta Dombrowski on the challenges of launching and scaling a UX research team

If you’re a company’s first-ever UX researcher, research is only one part of your job. You'll also spend a lot of time:

• Battling misconceptions — that research is unnecessary, or a cure-all. 

• Securing quick wins to add immediate value and validate your existence. 

• Ensuring the organization is mature enough to accept research, so it doesn’t reject you entirely.

“You’re coming into this organism that’s already existing, and you can either be accepted or rejected," said Roberta Dombrowski, research advisor at Personify, mindful leadership coach, and former VP of UX research at User Interviews.

“By the time a researcher gets into an organization, it's usually two years too late anyways," she said about most orgs.

Daunting, yes. But Roberta has scaled new research teams a couple times, at companies like User Interviews, edX, and Pluralsight. And we recently had her on our podcast, “What’s in it for them?”, to discuss how to overcome the hardest parts of growing a fledgling research team, how to build a research tech stack, and how to approach incentives.

You can listen to the full discussion above while you’re driving or walking your dog. Or, if you’re stuck at your desk for a few more hours, you can read Roberta’s best advice from our conversation here.

Here’s what we dove into:

• How to be effective as the first research hire at an organization

• The optimal tech stack for a small UX research team

• Justifying incentive budgets

• Which kind of incentives make sense for different kinds of research

• How to overcome the hardest parts of the incentivization process

• How to determine whether you should join an organization’s research team, or join as their first hire

A lot of Roberta’s advice prompts researchers to do what they’re best at: collecting data  and asking good questions. 

Whether you’re evaluating an organization to see how you can be most effective, building the best possible tech stack, or trying to determine whether you should join a new org at all, Roberta stresses the value of asking the right questions. And she suggests what the right questions might be. 

The following is an abridged version of our discussion.

You’re the first new hire in the door. What should you be thinking about?

Angie Peng: What’s the most important consideration to make when you're starting up a research team or specifically a UX research team? 

Roberta Dombrowski: Yeah, this is my – I've done it twice now. Which is crazy to say.

The biggest consideration is just the state of the business. Each business is so different, and so the way that you build teams, whether it's a research team or marketing or sales, whatever it might be, it's gonna be dependent on what the business needs. What do your teammates need? What do your customers need?

What does the organization need? And so usually when I go into a new company, a new team, I actually do an internal research project where I try to just understand, “what's the state of the business?” I ask people, what're your hopes and dreams and fears around research? What's keeping you up at night?

If I could make your life easier,what would research have an impact on? So all of that really goes into my strategy that I end up putting together and the approaches that I use. 

When I did it the first time when I was at edX, I started with usability testing because people were super apprehensive.They didn't trust research. They said it took a really long time. And then when I went to User Interviews, it was the exact opposite. It's a research company, so everyone knew it. They were already doing research. And so I could start with really strategic product discovery with the team. And so there's different entry points depending on what is going on in the business at the time.

The expectations you’ll face as a new research hire

Angie Peng: What are some common hopes that people tend to have and what are some common fears people have about research? 

Roberta Dombrowski: Yeah, fears definitely are: this is gonna take too long. We already know all of this. What are you gonna tell us that's different? We don't need to change anything. Everything's great.

It's like, really? Then how did my role get funded? Why am I here? 

A lot of people have strong hopes of: we're gonna make the product experience so much better.

Or the go-to-market strategy, or the sales motion. It's all gonna change because of research. 

And that's great to have excellent expectations for research, but one thing I end up telling the teams that I work with a lot of the time is that research and companies are systems, they're organizations.

And so research is also not a panacea where we're gonna come in and fix everything in the business. That's impossible. 

We are coming into this organism that's already existing and we can either be rejected by the body, or it can be accepted. And so it really depends on the acceptingness of the rest of the team.

Are you able to get wins? And really prove the value over time. 

Assessing the maturity level of your new org

Kate Monica: How do you discern whether a team is mature enough or at that maturity level to accept research?

Roberta Dombrowski: Yeah, it definitely depends, because people can say that they're ready for it and there's funding for a role and then you get inside and you're like, oh, they're not ready.

I look at how curious people are. We mentioned earlier research is about learning, and so are people just innately curious on the team? You ask them, “when's the last time you learned something new about your customers?” 

I joined an organization a few years ago. I asked them that – they said two years. “I haven't talked to a customer in two years.” 

Okay. That’s a great opportunity, but at the same time, it's also giving you a signal of how easy you might be rejected from the team. And so I ended up joining because it was a huge opportunity at the time and we changed it around really quickly.

How frequently do they learn? Do they do experimentation on the team when it comes to testing new features, releases? How do they make decisions, too? Is it mostly quant data driven or is it qualitative? Is it a mix of both? And looking at what's current state and what's aspirational.

A lot of the times we ask people questions during job interviews and you're asking the candidate. But as a researcher, I'm interviewing the company too, and seeing, “How do they measure up to what I'm looking for in a role?” And so I ask for concrete examples and you can usually tell.

How to build your tech stack

Kate Monica: What tools do you think – baseline – that researchers should have to best set them up for success? 

Roberta Dombrowski: It depends. I always say that when it comes to research, when it comes to UX. Before I go out and buy new tools, I wanna see what's the current state of things.

What tools does the team already have? There's likely already some type of product and design team, and they may be duplicating tools in their stack. They may not have anything, but I want to know what's going on. 

When I joined at User Interviews, we had too many tools. And so it was about aggregating. “We don't need this many licenses. Let's trim it down.” For us, we focused a lot on recruitment, because we were a recruitment product. 

What do we use for incentives? I also looked at usability testing tools. So we got Maze, got into Grain. 

I love Grain. They should pick me up as a partner. Shout out to Grain. I'm always fangirling. Absolutely wonderful for recording transcripts and sharing things. So they became one of my key tools. And so that's baseline. If you're doing incentives and informed consent, you need something for signatures and documents as well, so it really – it can branch off from there.

There's so many different tools, and if you look at the User Interviews research survey that they send out every year, there's over 200 that people use, so it can be all over the place depending upon what the business has, what they need."

Angie Peng: What are some of the different scenarios that you'd need certain tools versus others?

Roberta Dombrowski: So when I do my onboarding research, I understand the state of research on the backend. I'm rating the maturity level of research when it comes to the team.

Are they just getting started? Are they doing usability testing? Are they more strategic? If we're doing strategic research, then I'm going heavy into Grain. I'm going into transcription, I'm going into video recording. 

If it's on the other end where we're doing usability testing, then I need a really great tool – I need maybe UserTesting or UserZoom or Maze so that I can do prototype testing really quickly. Async, synchronous, are we doing surveys? Are we investing in quantitative, are we a mixed methods team? Then I need a survey tool. And then I also need things like analytics. Do we have an analytics team or do I actually need to get some type of tool there?

And so it can get really expensive really quickly when it comes to tooling. In a previous role, our budget was over a hundred thousand dollars in tools, which is crazy to think about. That was before incentives or anything for testing. 

Increasing an organization’s research maturity level yourself

Angie Peng: I'm assuming you've been a part of teams as they've started, right? Like more on the less mature side, and grown the team to be mature. And I'm sure there are growing pains, but how do you think about that shift? 

Roberta Dombrowski: Yeah, it's one of my favorite things to do because it's like chess.

What's my move? Where am I gonna start? Typically I start with experimentation and investing in one area. So I'll really go into, “what's a quick win that I can get that will show them the power of research really quickly?” 

And so when I got into edX, they did not want to do strategic research because they had a ton of fears.

And so I worked with just one product pod to do experimentation. We did a usability test. They were going to release a feature without testing it first and I was like, you know what? I can get feedback in 24 hours. Let's do it. And so I worked with the team, gave them feedback, and then they updated it before the release went out in a few days.

And that gave me the trust so that I could expand in some of the other teams and eventually get into strategic work. And so it's really about noticing the nuances when it comes to collaboration, paying attention to people's fears and resistance and trying to build trust. 

Let’s talk money: how to approach incentives

Angie Peng: How do you think about a budget for incentives? Obviously it's a big line item, right? And, you know, how do you think about when you need incentives and when you don't need incentives when doing research?

Roberta Dombrowski: So that's actually a mistake that I see in a lot of teams when they first get started is that they don't pay out customer incentives. I am of the mindset that you should always pay customers for their time because incentives actually encourage people to show up, and you're showing them that they're giving value.

Their time is worth something, and you're giving them money to compensate them for their value and their input. 

I actually, when I was at User Interviews, they have the incentive calculator, so I used that a lot, I did that on a study-by-study basis. I feel very guilty for saying this, but I never had to ask for more budget.

Usually I'd say, we want to do this study. Here's all the questions. When I was prioritizing things or looking at the roadmap, I never got a, “No, you can't do this”, or, “That's not important.” The teams always trusted me, and so I would always move forward with the incentives, the budgets that we had.

I know that other teams – especially now – can be very risk averse with budgets. And so I try to convey just, what's the bigger picture? What are we going to get for this? What's the ROI going to be on time? What are we paying for? To really make the meat of a proposal that you might be putting together for a project.

Angie Peng: When I get a survey, sometimes you get $20 just for doing the survey, and sometimes you'll be put into a raffle for maybe a hundred dollars. How do you think about the certainty versus the more raffle-style of compensating customers for giving their feedback?

Roberta Dombrowski: Typically the raffles, I'm a believer of it for surveys. So typically, it's based on sample size, survey size. You need larger sample sizes. Depending on your audience, you're likely not going to give $20 to a hundred thousand people or whatever the audience size might be.That'll bankrupt the business. And so that's why raffles can come in handy. 

Typically, I'll do more certain incentives when it's a usability test or a qualitative interview. If it's longer in time – like if it's over 20 minutes or it's an hour face-to-face – if somebody's going through a prototype, whatever it might be, they're going to be guaranteed. They’re definitely going to get the money for it. 

And also, the incentives are usually going to be higher in that case too. Like it might be a hundred dollars up for an hour of that person's time. 

Kate Monica: What is difficult about the research incentivization process when you first start out, when you're building a team?

Roberta Dombrowski: You're building the plane and flying it at the same time. So by the time a researcher usually gets into an organization, it's usually two years too late anyways. And you wish you had the process for recruitment and incentivizing already, and you're like, “Why don't we have this process?”

I did my first research study without User Interviews or Tremendous or anything two months ago. And it was a freaking nightmare. It was literally a spreadsheet, and I would send out emails, and then I'd go back to the spreadsheet and be like, okay, did I send a follow up?Did I send the Amazon gift card?

And it just took forever. It was too long. It was super manual, and it wasn't worth my time, honestly. You're dealing with different countries potentially too. People might be located in different countries, areas.

Some companies might not actually be able to take incentives, and so it's like, what's the plan there? A lot of enterprise companies can't take incentives. Then you have to think about, okay, am I making a donation somewhere in their name for something? And so there's just a lot of considerations to think about that nobody that's doing research wants to do.

I don't wake up in the morning and say, oh, I can't wait to send an incentive today and spend half of my day doing that. And so it's really great when you can start to automate the process. 

–

Kate Monica: Are there some things people can do to make it a little easier? 

Roberta Dombrowski: Use tools. Use automation. Use Tremendous. Even though, when I was at User Interviews, we used our own product for incentives, I still used Tremendous sometimes just because I didn't want to wait for a study or I wanted to send incentives quicker.

I didn't need to get approval from someone. I could just sign up, send it, and then just do that on the back end. And so it's signing up for tools, anything that can send it, it's usually super quick. And then you get the invoice you can send to your team afterwards. You don't need to worry about anything.

It's really great. So yeah, definitely using automations in any way because no one wants to deal with a spreadsheet. I've lived my life in the past where I needed to buy Amazon gift cards and go 10 to 15 people at a time, like batch invites. It's not fun. 

Angie Peng: How do you think about the actual amount and the ways that it could impact research?

Roberta Dombrowski: I've never encountered the ‘paying too much’ scenario. Usually it's paying too little. 

I've seen a lot of studies where it's like, oh, here's $20 take a five minute survey or whatever it might be, and people try to game the system and so the quality of the feedback is pretty low.

They're going in, just answering the questions. Maybe they're skipping questions, they're not answering it as well. And so typically, that comes up with async studies. So if people are doing it on their own, if it's a survey, if it's unmoderated, or if they need to type in an answer, what does this even mean? This is not a coherent sentence. And so it's on the researcher a lot of the time to go through and validate the data. 

And also, we would see people applying to studies that were higher in money too. But typically, we would do those live as well. So that would be better for quality. 

Finding the right org for you

Angie Peng: How do you think about interviewing for UXR positions?What do you look for, how do you approach the interview? 

Roberta Dombrowski:  How curious are they? Curiosity is important. Research is all about learning. Are they able to show me an example of a time when they've worked on a study, they've collected insights, and it's had impact on the wider business?

How do they collaborate with other people? How do they solve, challenges, issues? I talk a lot about how research is like an octopus. We're tentacles branching out to all these different teams. It's product, it's sales, it's marketing. And so how do they interact with those teams?Do they interact with those teams? 

In some orgs, research is in a bubble, and that's not how I run my team or how I operate. So those are all things that I think about. I look for mixed methods skills. Usually there's some type of portfolio case study. I just want to see how someone can communicate and show the impact of a project that they had before.

How do they respond to questions when I'm asking for follow up information: do they get defensive, or are they super nerding out about it and excited about the details? There's definitely analytical critical skills, but a lot of it is soft skills.

Your impact as a researcher is really dependent on your ability to communicate with other people on the team and build relationships.