Looking to hire a survey contractor? Here are 6 ways to pick a winner.

For public and private sector organizations looking to better understand the populations they serve, those tensions and opportunities can make it difficult to select the right survey research partner — and avoid those whose practices, design, and delivery will not meet your needs.

When it comes to understanding things like health outcomes, political attitudes, or socioeconomic disparities — just to name a few — most researchers want to avoid errors. Fortunately, there are a few key criteria stakeholders can use to help determine which partners they’re considering are most worth the investment. Here’s how to break it down.

1. They demonstrate expertise in your area, topic, or population of interest.

It may be obvious that you need a large and experienced survey company to conduct complex and technically challenging aspects of your research, but what about their content- and population-related expertise? Many clients bring their own subject matter expertise to a project; but even they can benefit from the survey team’s expertise in the specific research topic (such as health policy, education, energy infrastructure) or special populations (such as low income, minorities, chronically ill, students and veterans).

Whatever the breakdown, the strongest project teams ensure that technical and subject matter experts are working together to tailor the overall design, questionnaire, sampling, and data collection protocol.

Questions to ask:

• Does the survey research company have extensive experience studying the population(s) you plan to sample in your research?
• When you review LinkedIn and professional bios of employees on websites other than the company’s site, what do they indicate about each person’s subject matter expertise?

2. They adhere to the highest methodological standards — and push back on flawed approaches.

The most qualified companies, conversely, will assume research methods that meet stringent technical or scientific standards; these standards are often reflected in their price. A best-value company, furthermore, remains flexible and understands that all research is a trade-off between rigor, deadlines, and cost. These are the teams that will help you reconcile budget, timeline, and quality to guide you to the right product for your job.

Surveys that do not have the right methodological or topic expertise on the team usually disappoint. While less rigorous companies may be able to offer you a lower price, they will likely not meet the standards required of most government or academic research.

Other companies may have the resources and expertise to conduct sound surveys, but may too easily bend to the whims of the client, regardless of whether the approach is considered a survey or data science best practice—in some cases, that translates to research with obvious, major methodological flaws.

Questions to ask:

• Does the company speak intelligently about the differences in procedures and risks of error related to various types of surveys (e.g., household surveys versus establishment surveys, surveys of adults versus surveys of children, opinion surveys versus health behavior surveys)?
• Does the company understand the unique standards related to government- or academic-sponsored research?
• Does the company counsel clients on how to consider rigor, deadlines, and cost components to arrive at the best possible research solution?
• Where time is the primary concern, does the company develop protocols that optimize all three components?

3. They prioritize flexible survey design and real-time problem solving.

Survey design specialists with experience in a broad array of data collection modes can quickly and easily identify potential problems and solutions for your project. These specialists should have formal training in survey methodology and related statistical areas such as sampling and questionnaire design.

They should also be able to collect data in multiple modes, such as mail, telephone, web, and in-person—and be able to deftly combine modes to achieve data collection goals while considering the Total Survey Error impacts introduced by the survey modes under consideration.

Whether you are looking for soup-to-nuts survey support or only data collection, you need to know whether the company’s methodological staff is able to recognize design issues that may affect the survey administration, and possibly introduce survey error. Remember that capacity for data collection does not necessarily equate to methodological expertise, so scrutinize the company’s capabilities closely.

Questions to ask:

• Does the company have a wide range of survey methods expertise so that it can advise on a variety of design decisions?
• Does the company have a diverse set of experiences in case the method you plan to use has yet to be determined?
• Is the company able to offer you different survey research strategies based on the modes—or a combination of the modes?
• Does the company’s staff participate in survey conferences and are they published in survey journals? (This increases the likelihood that the company can deliver cutting-edge methods, while understanding the value of traditional methods).

4. Their corporate connections and partners add value to your project.

It is important to understand the full breadth of resources a survey company can bring to your study, including those beyond the company’s own walls. These connections can add valuable skills, resources, and experience to your project, but it’s important to clarify how these additional connections might interface with your projects and how they will be managed. Additionally, there may be contractual requirements for the participation of small or disadvantaged businesses. These subcontractors may provide specialized infrastructure, experience and expertise to the project.

Questions to ask:

• While researching a company’s reputation, what universities, agencies, and other companies have they worked with? Are projects they have worked together on current or in the distant past? Are these studies notable ones?
• At the contracting stage, are you clear on the role and contributions of other potential resources? Which tasks will be done in-house, by company-employed staff, and which will be done by subcontractors?
• Will your survey contractor take responsibility for the work of all partners or subcontractors? Does the contractor have a positive working history and clear management plan for working with the subcontractor(s)?

5. They’re transparent and accountable.

The acceptance of survey findings by government, consumers, media, or academic publication requires sufficient transparency of survey methods and outcomes, and communication of results in such a way that that their veracity can be evaluated by data users and stakeholders.

Survey contractors should be expected to include clear and detailed descriptions of the sampling process and overall method of collecting the data, releasing verbatim question text, and conveying response distributions so that response rate calculations can be independently evaluated. They also follow transparency and response rate calculation best practices that have been endorsed by the American Association of Public Research (AAPOR) and/or similar industry groups.

• Does the company’s practices coincide with transparency guidelines set forth by major accrediting bodies like AAPOR?
• Is the company willing to share details about their sampling process, question construction, and methodology?
• Can they point to past examples of work that has been verified by independent stakeholders, like the government, the media, or academic publications?

6. They document quality assurance — and they have the credentials to prove it.

As we’ve mentioned, quality is one of the most important considerations when collecting data, so most researchers require it and most companies promise it. However, survey data quality is not a simple concept. Response rate is often cited as one measure, or even the measure of survey quality, yet response rates are not essentially correlated with non-response bias. Seek out a survey company who understands the limitations of response rates, and quality more holistically. The Total Survey Error paradigm, for example, defines quality metrics for various aspects of a survey, including sampling, nonresponse, estimation, and measurement.

Quality should permeate all aspects of the project, and quality metrics should touch all facets of data collection—from the staff members chosen to work on your project, to the processes for checking the final products—and these should be documented and available for review.

Of course, if you don’t have a background in survey research, it can be difficult to confirm whether a company actually adheres to these standards or just says that they do. Further, some Total Survey Error quality metrics are more developed than others. That’s why, beyond documentation, your survey research team should have strong credentials from major accrediting bodies to back up their claims.

Questions to ask:

• Can the company assure quality in the data collection processes themselves, and in the estimates those data produce?
• Is the company certified by an independent auditing body? One example is the International Organization for Standardization (ISO), which provides an independent evaluation of process quality across a broad spectrum of survey operations.
• When reading the company’s website, do you see memberships in or recognition by major survey accreditation bodies such as American Association of Public Opinion Research (AAPOR) and the American Statistical Association (ASA)? Have they been involved with other organizations of which you’re a part?

Identifying a survey research partner is critical to meeting your survey research objectives—and the ultimate success of your project.  To find the best-fit survey research company for your project, look for someone who: 1) listens to your research needs and goals; 2) has the required experience, flexibility, and background to design a quality research plan; and 3) meets your research objectives and goals while balancing time and cost constraints.