538 and Misleading Audiences With Numbers

Poll-watching site 538 presents a master class in carefully choosing your poll results to present a narrative.

by Evan Urquhart

For good and ill, most Americans just don’t care that much about transgender issues. When you ask people what issues should be a top priority for the president and congress, restricting transgender people’s participation in society doesn’t make the top 20, according to Pew Research. If you ask people to name their top issue, as Gallup does regularly, in March of 2023 only 2 percent mentioned morals, religion, and family decline, down from 4 percent February. Polling specifically on questions relating to trans rights finds similar results. For example, parents largely see culture warring as a distraction from more important matters.

In light of Americans’ widespread disinterest in this issue, polling results that ask about policies relating to trans rights need to be taken with a grain of salt. The exact wording of the question matters hugely, as does whatever’s been in the news recently.

This broader context was weirdly absent from a piece on 538 today, which collected some of the least encouraging poll results on trans issues and used them to argue that Republican-controlled states probably won’t see much backash over legislations targeting transgender people. In order to tell this story authors Mary Radcliffe and Kaleigh Rogers highlighted some mainstream polling, some polling from conservative outlets such as Deseret News, and at least one result that seems to be a push poll (a partisan poll where the questions are worded to attempt to drive opinion in the desired direction).

screenshot from 538

See that 65 percent result in the lower third of the paragraph? The link goes to a poll from a conservative think tank called the Foundation for Government Accountability. Looking closer, you’ll find this very same poll first defines Critical Race Theory as teaching that “the United States government and its laws were established with the express purpose of discriminating against minorities and that people are naturally racist” then asks respondents if they support or opposed it. These are questions designed to try and elicit the sorts of answers that a conservative think tank finds useful.

screenshot from Foundation for Government Accountability pdf

None of this is to say that the hypothesis put forward by the 538 authors is untrue. Republican legislators in conservative states are probably not going to see a major backlash from outraged defenders of transgender equality. They’re also correct that trans women’s participation in sports has been an area where people are most likely to take the anti-trans side, perhaps because it’s a genuinely nuanced area where the interests of inclusion and fairness can compete and the best policies are carefully tailored to the specific age group, sport, and level of competition.

Polls can be useful. They can show broadly how attitudes have changed over a long time period, and they have a lot of value for candidates seeking data on how their campaign is doing. Reporting on polling is arguably less valuable, but the worst sort of writing about polling functions as a sneaky way to invoke a higher authority. This seems to be the main purpose of the 538 article. The authors have collected as many polls as they can find that show a lack of support for trans issues, and the implication is it’s okay not to support trans issues, it’s normal, and everybody else is already doing it. To accomplish this they downplayed results that didn’t fit the narrative, used a push poll without highlighting it as such, and prominently featured the one issue, trans women in athletics, where public opinion is the least favorable.

The 538 piece is doing something that’s a lot like what that push poll they included is doing, and subtly suggesting to the audience that there’s a right and wrong answer on transgender acceptance. The reality is that bias against trans people remains widespread, but the Republican assault on trans rights isn’t particularly popular. It’s not unpopular because Republicans disagree with it, though, but because the GOP’s obsession with this issue is out of step with the priorities of actual people.