This post was originally written as a review for The Journal of Things We Like Lots, but was not published there because reasons. I am posting it here to help an excellent work of scholarship receive the recognition and readership it deserves.
The Internet is using you. Every time you post and comment, every time you like and subscribe, yes you are using the Internet to express a bit of yourself, but it is also extracting that bit of you, and melting it down into something strange and new. That story you forwarded was misinformation, that video you loved was a publicity stunt, that quiz trained an AI model, and that tweet you just dunked on was from someone who’s going to have to delete their account to deal with the waves of hate.
Alice Marwick’s Morally Motivated Networked Harassment as Normative Reinforcement, Social Media + Society, Apr.-June 2021, at 1, is an illuminating analysis of how online mobs come together to attack a target. Building on interviews with harassment victims and trust-and-safety professionals, she describes the social dynamics of the outrage cycle with empathy and precision. The “networked” part of her framing, in particular, has important implications for content moderation and social-media regulation.
Marwick’s model of morally motivated networked harassment (or “MMNH”) is simple and powerful:
[A] member of a social network or online community accuses an individual (less commonly a brand or organization) of violating the network’s moral norms. Frequently, the accusation is amplified by a highly followed network node, triggering moral outrage throughout the networked audience. Members of the network send harassing messages to the individual, reinforcing their own adherence to the norm and signaling network membership, thus re-inscribing the norm and reinforcing the network’s values.
Three features of the MMNH model are particularly helpful in understanding patterns of online harassment: the users’ moral motivations, the catalyzing role of the amplifying node, and the networked nature of the harassment.
First, the moral motivations are central to the toxicity of online harassment. Debates about the blue-or-white dress went viral, but because color perception usually has no particular moral significance, the arguments were mostly civil. Similarly, the moral framing explains why particular incidents blow up in the first place, and the nature of the rhetoric used when they do. People get angry at what they perceive as violations of shared moral norms, and they feel justified in publicly expressing that anger. Indeed, because morality involves the interests and values of groups larger than the individual, participants can feel that getting righteously mad is an act of selfless solidarity; it affirms their membership in something larger than themselves.
Second, Marwick identifies the catalytic role played by “highly followed network node[s].” Whether they are right-wing YouTubers accusing liberals of racism or makeup influencers accusing each other of gaslighting, they help to focus a community’s diffuse attention on a specific target and to crystalize its condemnation around specific perceived moral violations. These nodes might be already-influential users, like a “big name fan” within a fan community. In other cases, they serve as focal points for collective identity formation through MMNH itself: they are famous and followed in part because they direct their followers to a steady stream of harassment targets.
Third and most significantly, the MMNH model highlights the genuinely networked character of this harassment. It is not just that it takes place on a much greater scale than an individual stalker can inflict. Instead, participants see themselves as part of a collective and act accordingly; each individual act of aggression helps to reinforce the shared norm that something has happened here that is worthy of aggression. Online mobs offer psychological and social benefits to their members, and Marwick’s theory helps make sense of this particular mob mentality.
The MMNH model has important implications for content moderation and platform regulation. One point is that the perception of being part of a community with shared values under urgent threat matters greatly. Even when platforms are unwilling to remove individual posts that do not cross the line into actionable harassment, they should pay attention to how their recommendation and sharing mechanisms facilitate the formation of morally motivated networks. Of course, morally motivating messages are often the ones that drive engagement – but this is just another reminder that engagement is the golden calf at which platforms worship.
Similarly, the outsize role that influencers, trolls, and other highly followed network nodes play in driving networked harassment is an uncomfortable fact for legal efforts to mitigate its harms. As Marwick and others have documented, it is often enough for these leaders to describe a target’s behavior as a moral transgression, leaving the rest unsaid. Their followers and their networks take the next step of carrying out the harassment. One possibility is that MMNH is such a consistent pattern that certain kinds of morally freighted callouts should be recognized as directed to inciting or producing imminent lawless action, even when they say nothing explicit about defamation, doxxing, and death threats. “[I]n the surrounding circumstances the likelihood was great that the message would be understood by those who viewed it,” we might say.
Morally Motivated Networked Harassment as Normative Reinforcement is a depressing read. Marwick describes people at their worst. The irony is that harassers’ spittle-flecked outrage arises because of their attempts to be moral people online, not in spite of it. We scare because we care.