eagleclosed

The Washington Post

Shambala Preserve, Acton, California

Stringer

Homegrown tomatoes

Contributor

Surfside Tragedy Makes Condo Buying Challenging Nationwide

For The Washington Post

staff

Winter Hill offers single-family homes near Falls Church downtown

Maansi Srivastava for The Washington Post

A layer of ice coats the leaves of a Japanese maple tree after an ice storm in Toronto

Reuters

Daron Taylor

Could gun ownership help explain the partisan divide in hearing loss?

Science

Blueberry Picking in Full Swing at Butler's Orchard Farm

Staff Photographer

HOGP

Old town Lahaina used to be crisscrossed by canals and spring-fed ponds. But sugar plantations diverted the water and the wetlands were buried under baseball fields and parking lots. 

Pop-up kitten lounch for adoption in Georgetown

for The Washington Post

President Donald J. Trump

YEARINPHOTOS2018

DATA DEPARTMENT DOD RELIGION July 1, 2024 Promo image.

Couple In Their Backyard

International student workers who have come to this country to work for the summer are preparing to leave for the season, in Ocean City, MD.

Respond to prefers-color-scheme dark in app web views

Whether leaderboard ad sticky behavior is disabled

Follow section front redirects in Spectrum

Enable NLP source in PCS for load testing

Switch off the News Alerts Survey Widget when we have enough responses

Poll to determine if new LUF updates are available

Allows the winning A/B test variant to be turned on without Spectrum deployment

Show expanded byline information on articles

Show infinite scrolling sections in Assembler MultiView

Show option for NLP article summaries in Action Bar

Show NLP summaries collection recirc on article pages

A general feature flag for Recipe Features that aren't yet released

Show the Balance of Power component on the new Election subnav component

Feature flag for media team features that aren't yet released

Enable new Elections subnav on Site pages

Use new Orca video player for preroll plugin in Spectrum

If true, will use a spoofed user agent to scrape and summarize content from paywalled sources

flip back to old full page reload when new updates come in

Show the snow tracker (instead of the heat tracker) on Spectrum

Editor’s note: This column was published before Tuesday’s election. It has been updated to reflect the results.

Before the election, we heard from reader after reader, all of you asking pretty much the same question. Here’s Greg Devenish in Houston:

“How accurate is political polling given that in today’s world landlines are the exception rather than a rule, the younger generation does not answer their cellphones, and it seems the only people that do answer voice calls are old fogies like myself?”

Department of Data Column Branding

We here at the Department of Data are dedicated to exploring the weird and wondrous power of the data that defines our world. <a href="https://washingtonpost.com/business/2022/06/29/dept-of-data/" target=_blank>Read more.</a>

Stellar question, Greg! In hindsight, you were right to ask questions. The Washington Post’s average of high-quality polls had Vice President <a href="https://www.washingtonpost.com/kamala-harris/" target="_blank" class="contextual_link">Kamala Harris</a> leading in the popular vote by two percentage points. But as we write this, it seems likely that Harris will lose it, perhaps by more than a point.

That’s not a catastrophic miss. It falls inside the typical margin of error. But it’s the third straight election in which pollsters have underestimated support for once and future president <a href="https://www.washingtonpost.com/donald-trump/" target="_blank" class="contextual_link">Donald Trump</a>.

Where did the forecast go wrong? Well, don’t blame us — not a single pollster called us this cycle. And maybe that’s part of the problem? Deb Buehler speaks for many of you when she writes: “I am a 71-year-old female from Minnesota, and I have NEVER been polled for my candidate choice. Why?”

We wondered the same thing. So we grabbed our favorite envelope, flipped it over and got started on some quick-and-dirty math.

During the <a href="https://www.washingtonpost.com/elections/election-2024/" target="_blank" class="contextual_link">2024 election</a> season, more than 175 national polls were conducted using methods that The Washington Post deems high-quality. Those surveys wrangled opinions from about 260,000 Americans (or fewer, since some will be duplicates or panels that repeatedly survey the same people). There are 262 million American adults. Penciling that out, you have about a 1 in 1,000 chance of being picked for a high-quality national poll as of late October.

Even if we include national presidential polls with lower standards — <a href="https://projects.fivethirtyeight.com/polls/">538</a> tracks more than 1,500 of them — you’ll still have about a 99 percent chance of not getting that fateful call.

To be sure, we’re not counting a roughly equal number of state presidential polls, about 60 percent of which are conducted in just seven swing states. Michigan and Pennsylvania together have seen as many polls as the 35 least-polled states combined. So if your home games feature the Detroit Lions or the Pittsburgh Steelers, your odds of being polled would rise a bit.

But then we put our pencils down and started dialing experts. And we discovered that even our back-of-the-envelope math vastly overstates your chances of being called.

Why? Because, as Greg surmised, nobody answers their phone anymore.

Pew Research Center’s Courtney Kennedy is one of the nation’s most-cited polling methods experts. When she started in this business two decades ago, you had to call about three random numbers to get one to answer, a low response rate that already was sending tremors through the polling industry.

By last year, Pew had to hit up about 140 random numbers to get that one response.

“Very few people answer the phone nowadays. … We literally stopped doing phone surveys because of that,” Kennedy said, her voice hoarse after fielding nonstop calls from desperate, poll-addled journalists in the final throes of a close election.

What caused the polling apocalypse? Kennedy — who wrote her <a href="https://deepblue.lib.umich.edu/bitstream/handle/2027.42/75977/ckkenned_1.pdf?sequen">dissertation</a> on that subject back when it was still on the cutting edge — said the transition from landlines to cellphones was a mere hiccup compared with what would follow: We stopped answering unknown numbers as phone spammers proliferated, everybody got caller ID and “huge swaths of the population” switched to text messaging as their preferred means of communication.

As recently as 2012, dialing random digits still drove most polls. By 2022, it was the weapon of choice for <a href="https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/">just 4 out of 69 pollsters</a> tracked by Kennedy and her Pew friends. Today, she can think of only one or two national outfits that still rely exclusively on the old arts.

To be sure, others — most notably the top-rated <a href="https://www.nytimes.com/article/times-siena-poll-methodology.html">New York Times-Siena College Poll</a> — still succeed with a modified version of this approach. They call registered voters. It only gives them a subset of the population, but it’s a subset that’s quite relevant to election horse-race polling.

But most horse-race pollsters have switched steeds. The most common public, national polls use online panels that recruit willing participants, often through ads, and then try to massage their demographics until they match the U.S. population. So, one reason so many of you haven’t been polled is that you haven’t answered any of these ads! But you also may not care about their results. With some exceptions, these polls aren’t included in Post averages.

Instead, many top pollsters adopt a “whatever works” approach. They pick thousands of American households, often by selecting random mailing addresses, then pull out all the stops to wring an answer from each via phone, mail, internet or — in some cases — in-person visits. Grinding out these reluctant responses is important because it ensures counting more than just the easiest-to-poll people, a group that differs from most Americans in key ways (more on that later). Kennedy estimates that this approach costs about 10 times as much as nonrandom online polling but is less likely to produce<a href="https://www.pewresearch.org/short-reads/2024/03/05/online-opt-in-polls-can-produce-misleading-results-especially-for-young-people-and-hispanic-adults/"> misleading results</a>.

“One of the lessons many of us have learned from 2016 and 2020 is that if you’re just giving people one way to participate — you’re just doing online, or you’re just doing phone — you’re not reaching everybody,” Kennedy said.

Like other elite-tier pollsters, Pew uses the slow-but-solid mail approach to build a panel that they can hit up more frequently. It takes a truly <a href="https://www.washingtonpost.com/entertainment/books/new-versions-of-hercule-poirot-keep-coming-and-thats-a-good-thing/2018/08/23/9d5b29ba-9f26-11e8-93e3-24d1703d2a7a_story.html">herculean</a> effort to track down enough random Americans to sample for their big national survey, but they eventually badger about a third of them into responding. Pew then recruits a representative subset of those folks into being regular (paid) respondents to its surveys.

So, can we blame recent polling misses on falling response rates? To answer that question, we need to figure out who’s least likely to respond to a poll. Of course, we can’t very well commission a poll of people who don’t answer polls. By definition, they are a public-opinion black hole. But as with literal black holes, we can guess where they are based on the effect they have on the world around them — that is to say, we can look at how people who do respond are weighted in survey data.

To ensure polls represent all U.S. adults, surveys give each respondent a different statistical weight. For example, if you call 1,000 random adults and get 50 responses from women and 100 responses from men, you’d use Census Bureau data to determine that the true gender balance is close to 50-50. Then you would adjust the weights so that each woman’s response counts twice as much as each man’s.

Pew’s widely cited industry benchmark, the <a href="https://www.pewresearch.org/the-national-public-opinion-reference-survey-npors/">National Public Opinion Reference Survey</a>, publishes such detailed data on weighting that you can estimate about how much of it reflects each person’s likeliness to respond. The clearest takeaway? In political polls, any answer you can wring from the tuned-out teens in the 18-20 age range counts for about three times that of an average American adult.

The easiest folks to poll, on the other hand, tend to be retirees. The moment you hit 65 and (hopefully, theoretically) enjoy hard-earned free time to pursue your dreams — whether it be circumnavigating the globe or filling in bubbles on opinion polls — your responses count for less. In Pew’s survey, retirement-age men see the sharpest drop; they weigh a little more than half as much as an average American.

It’s easier to reach women than men, and easier to reach White people than Black, Asian or Hispanic Americans — even when you account for the fact that White Americans tend to be older. Latinos are particularly valuable to pollsters, largely due to the difficulty of reaching Spanish speakers. They can be twice as valuable as anglophones, even for bilingual or multilingual pollsters.

Education may lurk behind some of those gaps as well. Folks who never made it beyond high school are worth more than twice as much as those who got a bachelor’s degree or higher. And the more of these factors you combine, the more they compound: Retirement-age college graduates, the pollster’s easiest targets, get the lowest weights in many surveys.

Kennedy and Jennifer Benz, who helps run the Associated Press-NORC polls at NORC at the University of Chicago, said all that weighting doesn’t really keep pollsters up at night — it’s a tried and true solution to falling response rates. The nightmare scenario would be people avoiding polls based on some non-demographic factor for which we couldn’t easily adjust.

And by “nightmare scenario,” we mean “that’s exactly what happened in 2016 and 2020.”

“The problem is not so much the response rates are low,” Kennedy said. “It’s that whether you take the survey these days is related to your political views in a way that wasn't true 20-plus years ago. That is what really complicated things for us.”

In 2016 and 2020, polls famously underestimated support for  Donald Trump, forcing many pollsters to add a trickier weight to their polls: your partisan affiliation (or in some cases, who you say you voted for in 2020). It changes more unpredictably than age or education — and is harder to define. And for political polls, it can have outsize influence because your weights are based on something strongly correlated to the opinions you’re hoping to measure.

Today, most rigorous pollsters use some variety of partisan weighting to correct for partisan poll antipathy, Kennedy said.

“It was unheard of, at the beginning of my career, that you would weight to something like trying to get the balance of Democrats and Republicans, or even that you would weight to something like civic engagement,” Benz said. “But that is becoming more necessary and more acceptable these days.”

All of these judgment calls make modern polling a fraught balancing act. Modeling choices can swing your results significantly. Before this Election Day, Vanderbilt political scientist Josh Clinton demonstrated that a single online poll could show Harris up anywhere from 0.9 to nine percentage points, depending on which perfectly reasonable weighting decisions you chose to run with. (He published <a href="https://goodauthority.org/news/election-poll-vote2024-data-pollster-choices-weighting/">his work</a> in <a href="https://goodauthority.org/">Good Authority</a>, the successor to The Post’s Monkey Cage.)

That doesn’t mean you can’t trust polls, Benz, Kennedy and Clinton said. It means polls might not be able to call a reasonably close horse race like the one we just lived through. But Pew, NORC and the Vanderbilt Poll specialize in deeper questions — say, societal attitudes toward same-sex marriage or affirmative action — and there, where  the fate of the nation doesn’t hinge upon a single percentage point, polls remain as valuable as ever.

So the existential threat to polling isn’t that folks aren’t picking up their phones. Pollsters solved that problem with money, effort and ingenuity. It’s that — as with everything from statuary to light beer — people’s reactions have become polarized.

So why did polls underestimate Trump again in 2024, despite heroic efforts not to make the same mistake a third time? The answer might lie in the charts above. The groups that Trump targeted this election — young men, less-educated people and Hispanic Americans — just happened to be the absolute hardest ones to poll.

Even if pollsters adjusted for all their past misses, it wouldn’t have been enough. Trump keeps increasing his potential polling error because he keeps turning out increasingly unlikely voters.

“In other words,” Kennedy told us after the election, “Trump won over even more people who have a low propensity to take surveys. That could certainly have contributed to the fact that polls systematically underestimated his support again.”

Hi! The Department of Data is on the lookout for quantifiable questions. What are you curious about: Does anybody actually like pickles? What actions reduce the partisan divide? Are refugees more likely to live near airports? <a href="https://thewashingtonpost.formstack.com/forms/department_of_data_questions">Just ask!</a>

If your question inspires a column, we’ll send you an official Department of Data button and ID card. This week, we’ll mail buttons to our Post polling friends Emily Guskin and Scott Clement, as well as Greg, Deb and many other folks who sent in probing, inspiring and hilarious questions on this topic including: Susannah Reilly in Austin; Bruce Smith in Murfreesboro, Tennessee; Michael Geglio of South Bend, Indiana; C. Smith in Falls Church, Virginia; Susan Swensen in Mountain View, California; Glen Grab in Crofton, Maryland; Carol Breimeier in Tucson; Mike Murphy of Dripping Springs, Texas; and Barrie Olsen in New York City.

University of Oklahoma

University of Missouri

Andrew Van Dam writes the Department of Data column each week for The Washington Post. He has covered economics and wrangled data and graphics for The Post and the Wall Street Journal. 

Andrew Van Dam

the-washington-post

Getty Images

Michael Ochs Archives

Economy

Economic Policy

Personal Finance

Work

Technology

Business of Climate

Business

washpost

prism://prism.query/site-articles-only,/business&limit=20

Get a curated selection of 10 of our best stories in your inbox every weekend.

The latest business news, analysis and columns, including on the economy, personal finance, work, technology and more.

Did the polls miss again because nobody answered their phones?