Incidence rate, the hidden number behind every invite

The number the tile doesn’t show

Every survey invite shows you two numbers: roughly how long it’ll take and roughly what it pays. There’s a third number behind both of them that almost no panel shows, and it explains more about your experience as a respondent than either of the other two.

That number is the incidence rate. It’s the share of people who start a study’s screener and turn out to qualify. If a hundred people start the screener and twenty qualify, the incidence rate is twenty percent.

For the research buyer, incidence is one of the two main inputs into what the study costs. For the panel, it determines how many invites have to go out to fill the study. For you, it’s the main driver of whether you’ll be screened out, and it quietly shapes the reward you’re offered.

Once you understand incidence, a lot of things that seem random start to make sense. Why some surveys pay much more than others of the same length. Why some invites almost always end in a screen-out. Why a study that looked open this morning is closed by lunchtime. Why the same study can feel easy for one person and impossible for another.

This piece explains what incidence rate means to the people who buy research, how it flows through into rewards and screen-outs, how it changes during fieldwork, and how to read the clues about it that do show up in invites. It’s the core mechanic behind screen-outs. Related topics, like how quotas fill by age group or how routers move you between studies, have their own pieces. Any figures below are illustrations, not measurements of a specific panel or study.

What incidence actually measures

Incidence rate sounds simple, and mostly it is. But it’s worth being precise about what it counts, because the details matter.

The basic definition is the number of people who qualify for a study divided by the number who were screened. Qualifying means passing every eligibility question in the screener. Screened means starting the screener and answering enough of it to be judged.

Some definitions are narrower. People who drop out partway through the screener, without being judged either way, are often excluded from both sides. People who qualified but couldn’t get in because a quota had already filled are sometimes counted separately, because they did match the criteria. Different platforms and buyers use slightly different definitions, and the numbers can shift depending on which one is used.

Incidence is also a property of a study and a population together, not of the study alone. A study looking for dog owners has one incidence among the general population and a much higher one among a panel’s members who’ve said in their profiles that they own a dog. Which population you measure against changes the number.

That last point is the one that matters most for you. The incidence rate a buyer quotes for a study usually describes the whole target population, or the whole panel. The incidence you experience depends on how well the panel has pre-targeted its invites. A panel that sends the dog-owner study only to people with dogs in their profiles will see a much higher incidence among the people it invites than one that sends it to everybody.

Keep that distinction in mind. There’s the study’s incidence, and there’s your incidence, and the gap between them is what good targeting closes.

Why buyers care about it so much

From the research buyer’s side, incidence is one of the biggest factors in what a study costs.

The price of market research sample is usually quoted per complete. The two main inputs into that price are the length of the questionnaire, often called length of interview, and the incidence rate. Longer surveys cost more because respondents need more incentive to finish them. Lower incidence costs more because the panel has to screen more people to find each one who qualifies.

The second effect can be dramatic. At an incidence of fifty percent, the panel screens about two people for every complete. At five percent, it screens about twenty. At one percent, about a hundred. Each screen costs the panel something: invites, screener time, the goodwill of members who get screened out. So the price per complete rises steeply as incidence falls.

Before a study launches, the buyer and panel usually agree on an estimated incidence. That’s part of what’s called a feasibility check. The panel looks at the target audience, estimates how many of its members fit, and quotes a price and a timeline. If the buyer says the target is anyone who’s bought groceries in the past month, feasibility is easy. If the buyer wants people who’ve bought a particular type of appliance in the past three months and are considering replacing it, the panel has to think harder.

Buyers have an incentive to estimate incidence optimistically, because higher incidence means a lower price. Panels have an incentive to be realistic, because if actual incidence comes in lower than quoted, the panel has to screen far more people than it priced for. That tension shows up later in fieldwork, and it affects you.

How incidence becomes your reward

The reward you see on an invite is shaped by what the buyer is paying per complete, and that’s shaped by incidence.

When a study has low incidence, the buyer is paying a higher price per complete. The panel has more room in that price to pay a larger reward, and it often needs to, because it wants people to take the screener even though most of them will fail. A larger reward helps keep members willing to try.

When a study has high incidence, the price per complete is lower. The reward is smaller, but most people who try will qualify.

So, roughly, rewards and incidence move in opposite directions. The survey that pays unusually well for its length is often one that most people won’t qualify for. The survey that pays modestly is often one most people will.

That relationship isn’t exact. Length matters too. Some panels pass on more of the price to members than others. Buyers sometimes pay a premium for speed, which can raise rewards even on higher-incidence studies. And specialist audiences, like business decision-makers or people with particular health conditions, often come with higher rewards that reflect both low incidence and the difficulty of recruiting them in the first place.

But the general pattern is reliable enough to be useful. A reward that looks generous relative to the stated length is a hint that incidence is low. That doesn’t mean you shouldn’t try. It means you should expect the screener to be selective, and you should count the likely screen-out time when you decide whether it’s worth opening.

How incidence becomes your odds

Incidence is, quite literally, the probability that a randomly chosen person from the screened group qualifies. If incidence is twenty percent, a random person has about a one-in-five chance.

You aren’t a random person, though. Your chances depend on how you compare with the target audience.

If your profile matches what the study wants, your chances are much higher than the headline incidence. A panel that has targeted the study well will have sent it mostly to people like you, and most of you will qualify. If your profile only partly matches, or the panel sent the study broadly, your chances are closer to the headline figure, or lower.

This is why good targeting matters so much to your experience. A panel with detailed profiles that it actually uses can raise the effective incidence among the people it invites far above the study’s population incidence. A panel that sends everything to everyone leaves you facing the raw number, with the screen-outs that come with it.

It’s also why completing your profile honestly helps. Profile data is what lets the panel target. A thin profile means the panel can’t tell whether you’re likely to fit, so you get more invites to studies you’ll fail. A detailed, accurate profile means more invites to studies you’ll pass.

But there’s a limit. Targeting can only use what’s in your profile. Many screeners ask things profiles don’t cover, such as recent purchases, intentions or opinions. For those, even perfect targeting can’t raise incidence all the way to certainty.

And there’s a caution. Your odds depend on your actual situation, not on what you say it is. Changing answers to fit a study’s apparent target doesn’t raise your real odds of being useful to the buyer, and panels check for exactly that pattern. The honest profile is the one that produces the best long-term incidence.

When the estimate is wrong

Incidence estimates are made before a study launches, and they’re often wrong. When they’re wrong, it’s usually on the optimistic side, and that has consequences.

Say a buyer estimated incidence at twenty percent, and the panel priced and planned the study on that basis. After launch, the real incidence turns out to be eight percent. To reach the same number of completes, the panel now needs to screen two and a half times as many people as it expected.

Several things can happen next. The panel may send many more invites than planned, which means more members get screened out. It may widen the invite list to include people whose profiles only loosely match, which lowers the incidence further. Fieldwork slows down, sometimes a lot. The panel may go back to the buyer to renegotiate the price or relax the criteria. Sometimes the buyer agrees to pay more. Sometimes it agrees to accept fewer completes, or to loosen a requirement. Sometimes the study is pulled from that panel and moved elsewhere.

For respondents, a study with badly overestimated incidence is a frustrating one. You’re more likely to be invited despite a weak match, more likely to be screened out, and more likely to find the study behaving oddly as the panel adjusts it.

Panels that route studies through automated exchanges face this constantly. The exchange may track each study’s actual incidence during fieldwork and deprioritise studies that are converting badly, or raise the reward to attract more attempts. That’s one reason rewards on the same study can change during the day.

You can’t see the estimate or the actual number. But you can notice when a type of study consistently screens you out despite seeming to fit, which is often a sign of optimistic estimates at work.

How incidence changes during fieldwork

Even when the initial estimate is accurate, incidence doesn’t stay fixed. It changes as the study runs, and usually in the direction that makes qualifying harder.

The main reason is quotas. Most studies want a particular mix of respondents: so many from each age group, region or other category. At the start, every cell is open, so anyone who meets the core criteria qualifies. As cells fill, people who belong in those cells start getting screened out as over-quota, even though they meet the criteria. The effective incidence for new arrivals falls.

By the end of fieldwork, only the hardest cells may still be open. The study might be looking only for older men in a particular region, or only for people who use a competitor’s product. Anyone else is screened out. A study that had a reasonable incidence on day one might have a very low one on day three.

This is why arriving early matters. A study invite that arrives and is opened quickly has a better chance of finding open cells. The same invite opened a day later may find most of the study closed.

It also explains a common experience: getting screened out of a study you’re sure you fit. You may well fit the core criteria, but your cell has filled. From your side, the screener looks the same. From the study’s side, it’s no longer looking for you.

Some panels try to prevent this by closing invitations to groups whose cells have filled. Others don’t, or can’t react quickly enough. When you see a survey invite for a study that’s been in field for a while, the chances are that some cells are closed.

Quotas themselves have their own piece. The point here is that they’re one of the main reasons your personal incidence drops as a study runs.

Where the hard question sits in the screener

Incidence tells you how many people qualify. It doesn’t tell you how long it takes to find out, and that depends on how the screener is built.

A well-designed screener puts the most selective question first. If a study wants people who’ve bought a particular appliance in the past three months, the efficient screener asks that straight away. Most people fail it within seconds and lose almost nothing. The remaining questions, about age, region and so on, only get asked of the small group who passed the first one.

A poorly designed screener does the opposite. It asks about age, household size, employment and several general topics first, then asks the selective question near the end. Everyone spends a few minutes answering before most of them are disqualified. The incidence is the same, but the time cost of each screen-out is several times higher.

Some screeners are deliberately indirect. To stop people guessing what the study wants, they bury the key question in a list of similar ones, or ask about several categories before revealing which one matters. That’s a legitimate technique for protecting data quality, and it means low-incidence studies with careful screeners can take a while to fail.

Screener order also interacts with quotas. A quota check on age might happen before or after the selective question. If it comes after, a person who passes the hard question can still be turned away as over-quota, having spent the whole screener to get there.

For the hourly maths, this means the cost of a low-incidence study depends on two things together: how unlikely you are to qualify, and how long each failure takes. A one-in-ten study with a thirty-second screen-out costs you far less than a one-in-ten study with a four-minute one. Your own records are the only way to learn which panels and study types tend to put the hard question first.

Reading the clues in an invite

Panels rarely show incidence directly. But invites often carry clues, and learning to read them helps you decide which to open.

The reward-to-length ratio is the biggest clue. A survey paying noticeably more than usual for its stated length is often low-incidence. One paying less than usual for its length is often high-incidence, or very broad.

The topic is another. If the invite names a subject, ask how many people it plausibly applies to. A survey about “your recent smartphone purchase” is lower incidence than one about “your views on shopping”. A survey about a niche hobby is lower incidence than one about television.

Some panels show a success or conversion indicator on each survey tile, sometimes as a percentage, a star rating or a label. These usually reflect how often members like you have qualified recently. They’re effectively a personal incidence estimate, and when a panel provides one, it’s the most useful clue available. Treat it with some caution, since it’s based on recent data that can change as quotas fill.

The survey’s age is a clue if you can see it. A study that’s just launched is more likely to have open cells than one that’s been running for days.

Targeting language helps. An invite that says something like “we think you’d be a good fit for this survey” or mentions a profile attribute suggests the panel has targeted it to you. An invite that’s generic, or that you received alongside many similar invites, suggests broad sending.

And your own history matters. If you’ve been screened out of a certain type of study repeatedly, that type probably has low incidence for your profile. Adjust accordingly.

None of these clues is definitive. Together they give you a reasonable sense of whether a survey is likely to screen you out.

A worked comparison using incidence

The value of understanding incidence becomes clearer with an example. The figures below are invented.

Say two surveys are on offer. Survey A pays 3 dollars for 20 minutes. Survey B pays 1 dollar for 10 minutes. On the tile, A looks better: 9 dollars an hour against 6.

Now suppose you have a rough sense of your odds. For A, the clues suggest low incidence for your profile, say one in ten. Its screener takes about 3 minutes before it disqualifies you. For B, the clues suggest high incidence, say six in ten, with a 2-minute screener.

On average, how much time does each complete cost you? For A, you’d expect about nine screen-outs for every complete. That’s 27 minutes of screen-outs, plus 3 minutes of screener and 20 of survey on the attempt that works. About 50 minutes for 3 dollars, or roughly 3.60 an hour.

For B, you’d expect about two screen-outs for every three completes, a bit under one per complete. That’s under 2 minutes of screen-outs per complete, plus 2 minutes of screener and 10 of survey. About 13 or 14 minutes for 1 dollar, or roughly 4.50 an hour.

B wins, even though A looked much better on the tile. The difference is incidence.

This kind of calculation is approximate. You rarely know your odds precisely, and a single attempt is either a pass or a fail, not an average. But over many surveys, the averages are what determine your effective hourly rate. Thinking in terms of expected time per complete, rather than reward per survey, helps you choose better.

Why incidence differs from person to person

Two people can open the same survey and have very different chances of qualifying. That’s incidence at the individual level, and it depends on several things.

Profile fit is the biggest. If the study wants people who’ve bought a car in the last year and you have, your odds are high. If you haven’t, they’re zero. Everything in between depends on how many criteria you match.

Demographics matter for quotas. Common demographic groups tend to fill quotas fastest, so people in those groups are more likely to be screened out as over-quota as fieldwork continues. Rarer groups stay open longer.

Location matters. Studies often target specific countries, regions or cities. Within a country, some regions may be over-represented in a panel’s membership, so their quotas fill quickly.

Device can matter. Some studies are designed for desktop and screen out mobile users, or vice versa. Some need a camera or microphone.

Past participation matters. Many studies exclude people who’ve taken part in similar research recently. Active panel members hit these exclusions more often than occasional ones.

Occupation matters because of security screeners. People who work in market research, advertising, journalism or the industry being studied are often excluded.

Timing matters because of quota filling. Opening the invite early gives you a better chance than opening it late.

All of this means there’s no single incidence rate that applies to you across all surveys. There’s a different one for every study, and it changes during fieldwork. Your best proxy is your own history: which types of study you qualify for, and which you don’t.

Using incidence without gaming it

Understanding incidence is useful. Trying to exploit it by changing your answers is not.

The temptation is obvious. If you know a study wants recent car buyers, why not say you bought a car? If you know older age groups stay open longer, why not claim to be older? You’d qualify more often, and your personal incidence would rise.

It doesn’t work, for reasons covered in more detail elsewhere. Screeners check answers against your profile. Inconsistent answers across surveys are flagged. The survey itself often reveals people who don’t actually fit. Buyers clean data after the fact and reverse completes from respondents who look wrong. And the pattern of changing answers to qualify is exactly what panel quality systems look for. The result is usually a lower quality score, fewer invites and sometimes a closed account.

There’s also the point that incidence estimates exist to price real research. When people misreport to qualify, the buyer’s data gets worse, and the research is less useful. Over time, that damages the whole system that pays respondents.

The legitimate uses of incidence are about choosing, not changing. Knowing that a high reward often means low incidence helps you decide whether to spend three minutes on a screener. Knowing that quotas close as fieldwork runs helps you prioritise new invites. Knowing that profile data raises your effective incidence encourages you to complete your profile accurately. None of that requires answering anything other than truthfully.

That’s the right way to use it. Let incidence guide which surveys you open and when. Let your honest answers determine whether you qualify.

What panels could show you

Panels have most of the information needed to tell you about incidence. Few share much of it.

The most useful thing a panel could show is a personal qualification estimate for each survey: how likely someone with your profile is to qualify right now. Some panels do show something like this, as a success rate or rating. Where it exists, it’s the most helpful single number on the tile.

A panel could also show how long a study has been in field, which would help you judge how many quotas might have closed. And it could show the expected screener length separately from the survey length, so you’d know how much time a screen-out would cost.

Panels have reasons not to show all this. Showing low odds might discourage people from opening surveys the panel needs filled. Showing detailed incidence might encourage people to change answers. And some panels don’t have accurate real-time incidence data, especially for studies sourced from external exchanges.

Still, the panels that show more tend to treat members’ time with more respect. A success indicator on a survey tile is a sign that the panel is trying to help you avoid wasted screeners. It’s worth noticing when you choose which panels to prioritise.

If your panel doesn’t show anything like this, your own records are the substitute. Track which types of survey you pass and fail, and after a few weeks you’ll have your own rough incidence estimates by category.

Building your own incidence record

You can’t see a study’s incidence, but you can measure your own. A simple record does it.

For each survey you attempt, note the category or topic, the stated length, the reward and whether you qualified, were screened out or hit a full quota. If you can tell how long the screener took, note that too. If you keep session times already, this adds a few columns.

After a few weeks, group the attempts by category. For each category, work out what share you qualified for. That’s your personal incidence for that type of study on that panel.

You’ll probably notice patterns. Some categories you qualify for often. Others you almost never pass. Some panels target better than others, so the same category might have different personal incidence on different panels.

Use those numbers to choose. When a survey arrives in a category where your personal incidence is low, you’ll know to expect a screen-out, and you can decide whether the reward justifies the screener time. When one arrives in a category where your incidence is high, you’ll know it’s likely worth opening.

Your personal incidence also feeds straight into the effective hourly rate. The screen-outs you record are the unpaid minutes that pull the rate down. Seeing them by category shows you where the time goes.

Keep the record honest. Log screen-outs even when they’re annoying, and log quota-fulls separately if you can tell. The point is to see your actual experience, not a flattering version of it.

What to do this week

Start noticing the reward-to-length ratio on invites. When a survey pays unusually well for its length, expect a selective screener.

Complete your panel profiles accurately and thoroughly. Profile data is what lets panels target you, which raises your effective incidence.

Open new invites promptly when you can. Early arrival means more open quota cells.

If your panel shows a success rate or similar indicator, use it. It’s the closest thing to a personal incidence figure you’ll get.

Add a category column and an outcome column to your survey log. After a few weeks, work out your personal incidence by category and by panel.

Use expected time per complete, not reward per survey, when comparing options. A lower reward with high incidence often beats a higher reward with low incidence.

Never change your answers to fit a study’s apparent target. Let incidence guide what you open, not what you say.

And keep it in perspective. Understanding incidence won’t turn surveys into a full-time income. It will help you spend less of your limited time on screeners you were always going to fail.

For more on how panels set pay, screen people out and decide what your time is worth, head back to the Survey Farm home page.