Michael Emerson and Laura Essenberg (both at Rice University) recently publicized a white paper on American attitudes toward marriage. The paper is based on a two-wave panel study that examined respondents in both 2006 and in 2012. These sorts of panel data are a gold-standard approach to examining change over time. Most surveys are simple cross-sectional analyses that take a proverbial ‘snapshot’ of popular opinion at any given moment. While these surveys are valuable when conducted well, they are, by definition, at the mercy of sample quality. Even when done well, a typical survey may get a bad, or non-representative, sample about 5% of the time. Beyond that, these snapshots are reflections of a moment and do not necessarily reveal trends. A panel study allows researchers to look for change within individuals themselves. Put differently, do we see evidence that particular people are changing their own minds about gay marriage?

The conventional wisdom suggests we are in the midst of an attitudinal transformation, so one might assume Emerson and Essenberg’s data reveal this transformation at the individual level. In this case, the conventional wisdom is not necessarily correct. It appears that gay marriage is a polarizing issue, where people are taking strong positions both for and against, so evidence of a wave in favor of gay marriage is non-existent. Conversely, their findings show there is no statistically significant difference in overall support for gay marriage between 2006 and 2012. This evidence stands against the dominant cultural narrative, which is one reason why it has gotten so little media attention.

Most fascinatingly, however, is their finding that while there are people who are changing their opinions on gay marriage, the net impact is that more people changed toward the traditional definition of marriage. This shows issue fluidity in an unexpected direction and suggests that as a political, cultural, legal, and social matter, marriage is still in flux and that traditional marriage advocates have the potential to persuade opponents.

As Mark Regnerus, Sociology Professor at the University of Texas (Austin), notes, Emerson and Essenberg’s findings conflict with recent cross-sectional analyses, especially by Gallup. Why the difference? Regnerus argues that polling methodology may play an important role. First, Gallup asks respondents about gay marriage attitudes AFTER asking a direct question about the propriety of making homosexual conduct illegal. This practice, known as ‘priming’ among public opinion students, has the potential to bias respondents by planting a particular association in their minds that can then affect attitudes on the later marriage question. Regnerus notes that such practices likely underestimate opposition to gay marriage, and overestimate support, by 6-7%.

Second, Gallup phrases the question so that people who support traditional marriage must take a negative position (“Do you think marriages between same-sex couples should or should not be recognized by the law as valid, with the same rights as traditional marriages?”), while the Emerson and Essenberg survey allows those who affirm traditional marriage to take a positive position (“The only legal marriage should be between one man and one woman”). There are a couple of lurking methodological issues. People prefer to say ‘yes’ as opposed to ‘no.’ Perhaps more critically, people have a tendency to affirm socially desirable positions when confronted by strangers. If respondents believe their true opinion will be looked down upon, even over the phone, they hedge. When a question does not give respondents much room to hedge, there is a temptation to affirm the subjectively perceived ‘popular’ view, especially when the respondents are conflicted about the issue.

What do we do with all of this? Always take public opinion polls with a shaker full of salt. Examine public opinion over time and through multiple methods and then take a step back to determine carefully the state of public opinion. Also, there is no strong evidence that gay marriage is fully settled. While we have seen significant changes in favor of homosexuality in general and gay marriage in particular, we do not know the point of equilibrium.