Measuring Movement Outcomes

By William Gamson

Gathering my thoughts on how we measure social movement failure is, for me, a deja vu experience. Imagine that it is 40 years ago and I am struggling with this question as I conduct the research reported in The Strategy of Social Protest (Dorsey: 1975). I write the following:

Success is an elusive idea. What of the group whose leaders are honored or rewarded while their supposed beneficiaries linger in the same cheerless state as before? Is such a group more or less successful than another challenger whose leaders are vilified and imprisoned even as their program is eagerly implemented by their oppressor? Is a group a failure if it collapses with no legacy save inspiration to a generation that will soon take up the same cause with more tangible results? And what do we conclude about a group that accomplishes exactly what it set out to achieve and then finds its victory empty of real meaning for its presumed beneficiaries? Finally, we must add to these questions the further complications of groups with multiple antagonists and multiple areas of concern. They may achieve some results with some targets and little or nothing with others. (p. 28).

If I were writing this today, I wouldn’t change anything. The same questions remain and while we have made considerable progress on some of them, others remain only partially and incompletely answered. It is striking to me that the often quite insightful entries in the social movement failure blog do not really take on the task of answering them. They address the issue of why some movements fail but they largely ignore the question of measurement. Since I was dealing with a random sample of social movement groups in Strategy, I was forced to face measurement issues in comparing groups that “succeeded” or “failed” in some meaningful sense of those terms.

I will return to some of the insights on the why question raised by the various essays, but first I want to review the state of measuring movement outcomes, suggesting solutions that I believe are useful and lasting ones and identifying questions that seem to me still unresolved.

Acceptance and New Advantages

It is generally accepted that we should think of success and failure as a set of outcomes and that a given movement may get different scores on different valid outcome measures. Strategy suggested two basic clusters: one concerned with the fate of the challenging group as an organization and one with the distribution of new advantages for the group’s presumed beneficiary. The cluster on acceptance is about a movement group being treated by its targets as a valid spokesperson for a legitimate set of interests. The cluster on new advantages is about whether the presumed beneficiaries gain new advantages during the active phase of the movement or its short-term aftermath.

The first edition of Strategy was published before the field of sociology took a cultural turn—a turn that I followed in my own work. I came increasingly to recognize the failure of the acceptance/new advantages measure to reflect cultural changes which are central for many social movements. I believe that I found an effective way of extending the original outcome measure to the cultural realm in “Social Movements and Cultural Change.”

The equivalent of “acceptance” can be measured by media standing. This means that journalists treat movement spokespersons as agents of some larger constituency whom they are representing. It is measured by who gets quoted in mainstream media. The equivalent of “new advantages” can be measured by the increasing prominence of the way that the movement frames the issues it is addressing. There are challenging issues of measurement—especially of the prominence of the movement’s preferred frame—but there has also been a considerable effort to develop computer-assisted programs for doing the kind of text analysis involved.

I would submit this acceptance/standing and new advantages/increasing frame prominence as the basic starting point for understanding and measuring movement failure and success. But it leaves many difficult questions unanswered or answered in a somewhat arbitrary manner. I will comment on some of them without really offering any solution.

Movements as a Field of Actors

The measures of outcomes suggested above focus on a social movement organization rather than on a movement as a whole. We now recognize that a movement is a field of actors and that the dynamic among actors in this field may be an important variable in determining outcomes. “When demonstrators are arrested at [the] Seabrook [nuclear power plant], phones ring at the Union of Concerned Scientists,” I wrote in arguing for a radical flank effect on producing standing for a more mainstream group. It seems hard to specify “standing” for the whole field of actors although we might want to measure failure on this variable by the fact that none of the actors achieve it.

Similarly, the movement as a whole might not have a single preferred frame, but instead consist of a coalition of actors with different frames. Some of the actors may have frames which gain in prominence as a result of movement activities while others find no increase whatsoever in their preferred frames. Again, it seems hard to specify an increase in frame prominence for the whole field of actors although we might want to measure failure on this variable by the fact that none of the preferred frames of movement actors achieve increased prominence.

Representing a Variable by a Fixed Outcome

Obviously, there are many gradations of success and failure. This is true not only because there are several different dimensions as I argued above, but also because movement groups often have multiple goals, some shorter run and others viewed as long-term objectives with many steps along the way. Even if the unit of analysis is a particular movement actor (rather than the whole field of actors), there is a degree of arbitrariness in drawing a line in the middle of a variable and declaring that the group has failed if the outcomes are not above it.

Consider, for example, a group whose goal is to abolish private property, but also participates in what it considers ameliorative campaigns such as raising the minimum wage. If the campaign succeeds, do we treat that as a success even though it hardly seems like even the tiniest step toward the goal of abolishing private property? In the research reported in Strategy, I treated participation in collective action campaigns aimed at achieving change as evidence of success if the campaign led even to some modest change in the status quo. But I was aware that one could make a plausible argument for drawing the line elsewhere. Again, my solution was to treat even the most minor goal achievement as success while failure meant no progress on any of the group’s efforts.

The problem is simply compounded if we want to measure the success of a movement as a whole. Assuming there is an achievement of some changes, these may be regarded as significant ones by some actors and dismissed as tokenism by others. If we want to define it as a “success,” we are implicitly taking the point of view of some of these actors and rejecting the view of others. Again, one can avoid this problem by defining failure as the absence of any member of the field of actors who would claim success on any of the multiple dimensions of outcome.

Observation on Blogger Essays

While the bloggers do not directly address the issues of measuring movement failure, some of their insightful discussion of why movements fail has measurement implications. Christian Davenport draws our attention to the dependency of movement outcomes on an interaction between the social change efforts of a movement and the social control strategies of authorities. “In a sense,” he writes, “social movement failure is the study of government ‘success’ and social movement success is the study of government failure.” Kevin Harris in his account of the green movement in Iran observes that “Initially, spectacular acts of state violence tended to spur on waves of crowd-building emotional energy.”

This calls our attention to the importance of social control errors—efforts of authorities to contain a social movement which, accidentally, have the opposite effect of mobilizing the movement, allowing it to draw in bystanders as allies and participants. From the standpoint of authorities, repression is a problematic strategy; if applied with sufficient skill, it may succeed in demobilizing a movement, but it is often applied in a clumsy fashion and provides movements with new opportunities. The #Occupy movement provided some excellent examples—such as police arresting reporters who were covering the movement for their media outlet or allowing witnesses to record compelling visual images of police roughing up peaceful demonstrators with no visible provocation. Hence, I strongly agree with Davenport that any explanation of movement failure must consider the interaction between movements and authorities and their respective strategies.

As a final comment, I would like to respond to Ed Amenta’s question: “How can we make sense of influential movements, organizations, and campaigns without an equal attention to those that lacked influence?” I have an idea: How about by taking a random sample?