Understanding and Preventing Participant Fraud in Qualitative Research

By Michael Mermelstein
Executive Vice President/Partner
Nichols Research
Washington, D.C.
mmermelstein@nicholsresearch.com

Contributing Author: Jon Last
President & Founder
Sports and Leisure Research
White Plains, New York
jlast@sportsandleisureresearch.com

Among the many critical success factors that are presently top-of-mind for professional researchers, data quality continues to be at or near the top of the list. While there are quality imperatives that must be addressed in the methodological design, one aspect that can be easily overlooked is the quality of the respondent base overall in data collection and interpretation. In quantitative research, respondent fit and qualification have been widespread issues for researchers when using online panels. But respondent authenticity has also become a bigger issue for qualitative research, as recruiters cast a broader network across social media and other digital sources to find eligible participants for qualitative research methodologies. In our experience, there has been a particularly acute need to fend off imposters and other fraudulent practices that can undermine the unique benefits offered by qualitative research. Our focus here is to look at the underlying issues that qualitative researchers need to understand and some practical advice to manage against fraud.

For better or worse, digital communications and the proliferation of online research have peeled the lid back on what, for some less ethical consumers, has become a lucrative way of generating incremental income through frequent participation in research studies. For “professional respondents” whose goal is to qualify for as many interviews as possible and for recruiters who are focused more on efficient production and maintaining profit margins, a former standard such as must not have previously participated in a study over the past six months is an anachronism.

Concurrently, potential respondents have become savvier about how to “rig the system” and answer standard screening questions, though falsely, in ways that can optimize their selection for a particular study. There are even people training potential participants in how to successfully get past screening questions and how to find as many focus group opportunities as possible.

To understand and to help mitigate fraud as best as possible, a qualitative working group of the Global Data Quality Initiative (GDQI) has worked to define fraud, understand what role different players in the research chain have in combating fraud, and list steps that qualitative researchers and recruiters can take to lessen the instances of fraudulent participants. GDQI is a collaboration among IA, MRS, QRCA, ESOMAR, CRIC, AQR, Samplecon, The Research Society, and VMO. The goal of this collaborative effort is to provide the entirety of the research community with practical tools to boost data quality, as well as help companies deal with fraudulent respondents across the various methodologies used by the research community.

Identifying Fraudulent Participants

The qualitative work group of the GDQI is working to define fraudulent participants so that everyone in qualitative research has a common understanding of fraud. To first define fraud, we have identified and systematized common practices we see in the fraud environment today. The following are some of the most widespread:

• We typically see that fraudulent participants have a higher level of familiarity with the research process than general members of the public or participants who participate less frequently.
• We have seen guides published online and offers to train people on how to earn money from research by falsely answering screening surveys. There are videos on YouTube and TikTok about how to earn money from research by answering screening surveys dishonestly, as well as fraudulent participants or personas working in groups to provide tip-offs to new projects.
• We usually see hints from the screening process. Fraudulent participants have a hard time with specific terms of the category or product. It is often the case that fraudulent participants will take much shorter or longer timeframes in answering the screener—a deviation in either direction from the time it should take to answer can be a sign of potential fraud.
• Also, when asking for proof that they are who they say they are, fraudsters will try to hide some of the identifying details, or you will see every excuse you can think of. From proclamations that “I lent that product to a friend, so I brought another one,” to respondents bringing in “borrowed products” that still had price tags on them or were in the bag, likely to be returned right after the discussion.
• Finally, we have seen participants using multiple email addresses and identities in trying to get through screening questions the “right way.”

Combating Participant Fraud

In the qualitative research process, the most important time to avoid fraud is during recruitment. This is where working closely with recruiters to build a scrutinizing approach can have the biggest payoff for researchers in terms of participant and data quality. While there are some aspects that depend entirely on the recruiter, you, as a researcher, have a large part to play in combating fraud, both directly in the screener design and indirectly in ensuring that safety measures are in place.

What you can do as a researcher
As a qual researcher, your biggest input will be in designing the screener strategically and leveraging project-specific information from the client. The most effective screener makes it as difficult as possible for respondents to guess or figure out what the “right” answer is. Take great effort in building screeners that follow these guidelines:

1. Obfuscate as much as possible with a blinded question format.

• If you are looking for people who are recent flyers, that can mean building a two-level/two-question screen, with subsequent questions offering multiple choices about recent travel habits followed by frequency. The key is to ask a question that sequentially builds on the previous question. These will be very project-specific questions. The answers would make sense (or not) cumulatively.
• Ask yourself the question, “Can I fake my way into this by answering yes to everything?” Consider writing some of your screener questions with a definitive “no” qualifying response.
• Offer multiple mutually exclusive choices regarding recent behaviors so a potential respondent can’t discern which end of the spectrum you are looking for.
• You can also ask contradictory questions to ensure that potential recruits are consistently answering them in a way that shows both attention and verification.

2. Insert knowledge-based “challenge” questions.

• Taking the above process further, don’t just take the respondent’s word for it. If they claim to be a frequent user of a product category, design a knowledge-based question that only a true frequent user can answer immediately without hesitation.
• If your screening question can be easily answered by a quick Google search, then rewrite the question in a way that cannot be answered correctly by googling.
• Ask the client to provide specific knowledge of the category that can be translated into questions that an engaged customer will know the answer to, but a “cheater” or casual user will not.
• This also applies to industry or category buzzwords. If the potential respondent can’t speak the language of the category, their veracity should be questioned. For instance, in sports-related research, if you are looking for avid fans or serious participants in a sport or activity, work hard to find the types of technical inquiries that can’t be guessed or easily looked up.

3. Require proof and play the role of the cynic.

• Think about what your target respondent would have in their immediate possession that validates their qualification for a study and make them send a photo or some other proof of it within 10 minutes of being recruited. Ask for a receipt, a product photo, or some other testimonial that can’t easily be faked.
• Again, think through what would be easy for a truly qualified respondent to do but challenging for someone who isn’t legitimate.
• Apply the following rule, “When in doubt, throw them out.” If it seems suspicious, it likely is.

What you should ask the recruiter to do
Even if you are not directly involved in the behind the scenes of the recruitment process, as a researcher, you can collaborate with the recruiter to implement other ways to avoid fraud. Work closely with the recruitment team to ensure that these guidelines are followed:

1. Monitor the time it takes to answer screeners.

• Work with the recruiter to determine how long it should take to answer the screener, usually done by testing the screener as if you were an ideal participant for the study. Monitor the time it takes participants to respond. Consider removing prospective participants where there is a deviation—in either direction—from the time expected to complete a screener.
• In some cases, it may be worth having a follow-up telephone call with participants who deviate from the time expectations, but their answers to the screening questions are what they should be. Asking a few additional questions without time to search for the right answers can help you decide whether to keep them or not.

2. Send unique, secure invitation links.

• Ask the recruiter to send unique invitation links to prevent multiple attempts of
  fraudulent participation.
• Consider blocking VPNs and proxies—servers used to hide the IP addresses of respondents—that may be used to mask someone’s actual location.
• Review participants’ past behaviors, possibly tied to the respondents’ IP addresses or the frequency of responding to survey screenings. (For more information on this, visit the secure end links guidance on the GDQI website: globaldataquality.org.)

3. Rescreen with passion.

• Last, but not least, be sure to organize with the recruiter a rescreening of respondents upon arrival and ensure consistency of their answers. Where there are discrepancies, additional challenge questions can be asked that will make it painfully obvious when someone is trying to fake their way into a discussion.
• In some product-related research, you can ask respondents to indicate what product they own in the screener, down to the specific SKU level detail, and then ask them to bring the product to the interview.

A Collective Effort

Combating fraud, especially as fraudsters capitalize on artificial intelligence (AI) to help overcome our efforts, requires the entirety of the research community to work together to ensure quality results. But, ultimately, it is the qualitative researcher who is responsible for the success of a project. That starts and ends with effective recruitment. All practitioners of qualitative research need to work together so clients are confident in research results. For better or worse, taking the scrutinizing approach outlined above is often the only way to ensure that you are getting what you and the client are paying for.

When qualitative researchers work on their studies, great care and attention are inevitably directed to designing a discussion guide that assures proper conversational flow, respondent candor, and examination of the most critical insight needs.

Good qualitative researchers will also exert great effort in considering and developing a healthy mix of projective exercises and concept evaluations that strive to uncover deep-rooted beliefs and perceptions that are not as readily surfaced through direct questioning. But all these efforts are for naught if we aren’t speaking to the right respondents.