Reading Faces, Inferring Intent

Do facial expressions convey the same emotions around the world? Los Angeles Times, April 17, 2012, is a useful way of getting acquainted with the debate over facial expressions and emotions.

You will see that the work of Paul Ekman is central to that debate.

Ekman is known for devising the Facial Action Coding System (FACS). It is a taxonomy of the musculature of the human face and of their activation when we communicate an emotion. From information provided by FACS, researchers infer what emotion the person is communicating. On this basis, another inference is made about what the person intends to do.

Ekman’s work is contentious and has been severely criticized. He provides no evidence to support those two inferences, some argue. It is all in the eye of the beholder.

But the security and intelligence industry has been won over. Ekman ‘consults’ for the Department of Homeland Security, the Central Intelligence Agency, the Department of Defence, and the Transportation Security Administration. He taught guards at Abu Ghraib on how to extract ‘information and truth’ from prisoners; and Special Forces in Afghanistan on how to determine enemy intent to kill.

FACS is used at airports to scan the faces of travellers for signs of guilty intent. On this see Intent to Deceive? Nature, 26 May, 2010. Ekman regrets not being able to have detected the facial expressions of the 9/11 terrorists so that they could have been stopped.

His home page is Cutting Edge Behavioral Science for Real World Applications.

You’re not worried, because you are not a terrorist? Well, one day you might be because the definition of ‘terrorist’ is broadening by the day. In the United States it is broad enough to cover most forms of dissent. It is as well to be interested in a system that claims to know your intent from your facial expression.

FACS is now being adapted for integration into software and soon a computer will be interacting with you on the basis of what it knows about your emotional state. Keep an eye on Facebook’s Facial Recognition.

As with all authors, it is best to read Ekman yourself.

Here I want to pay attention to an article that did much to popularize his ideas: Malcolm Gladwell. ‘The Naked Face: Can you read people’s thoughts just be looking at them?’ The New Yorker. August 5, 2002. The short answer, for Gladwell, seems to be, Yes.

The article begins and ends with a vignette about two police officers. One shot an assumed assailant, the other did not. Both acted on ‘hunches’. Gladwell connects these hunches to Ekman’s work on how to detect emotions from facial expressions. This is the opening vignette:

If you looked at it logically, I should have shot him. But logic had nothing to do with it. Something just didn’t feel right. It was a gut reaction not to shoot—a hunch that at that exact moment he was not an imminent threat to me’ (Yarbrough).

Something did not ‘feel’ right. It was a ‘gut’ reaction. Perhaps a ‘gut feeling’; a hunch. Now this sounds like what Antonio Damasio would call a ‘somatic marker’, i.e., an emotional memory of what happened in a similar situation, something we feel in the body. In other words, this perception was informed by experience. Hence, the police officer’s assessment that ‘the longer you’ve been working the stronger that instinctive reaction’.

Note that the officer and his potential assailant were in possession of all their senses; not just sight, but also hearing, touch, smell, taste. I make this point because Gladwell immediately ignores all this and focuses on the face and sight alone. Just as Ekman himself does.

Gladwell tells us that something Officer Yarbrough saw ‘held him back’. And yet Yarbrough tells us that what held him back was a ‘gut feeling’, a ‘hunch’. It was not just what Yarbrough saw that held him back, it was how he felt about what he saw.

Nevertheless, on this shaky foundation, Yarbrough is elevated to the ranks of those who can see what others cannot. ‘What do they see that we miss?’

Ekman, Gladwell tells us, ‘could look at a face [on video tape] and pick up a flicker of emotion that last no more than a fraction of a second’. That’s impressive—if there is any evidence of it. Like God, like U.S. security, Ekman sees and knows everything.

The problem here is that emotions are seldom, if ever, experienced in their ‘pure’ form. We usually experience an adulterated and constantly changing mixture of several, often conflicting, emotions. They are processes, qualities of social relationships, not static things that ‘belong’ to individuals.

Ekman sees the face as a screen featuring the emotion channel. His subjects are images of faces displayed on televisions, photographs or movies. He arrests the image, infers what the subject is feeling and, from that, what the subject is thinking. From there it is but a short leap to the subject’s presumed intent.

Emotions, however, are experienced throughout the body. The face is certainly important, but so too is the language of the body, tone of voice and the social context of the person. What we feel inside but do not express outside is as important as what we display to others.

Note how Gladwell’s presentation of Ekman’s argument slides from his ‘taxonomy of facial expressions’ to his ‘catalogue of the essential repertoire of human emotion’, without any intervening evidence.

The article concludes with another vignette. This features Officer Harms and his colleague, Scott. Harms too had a gut feeling, a hunch about a potential assailant.

Harms pulled out his gun and shot the man through the open window. “Scott looked at me and was, like, ‘What did you do?’ because he didn’t perceive any danger,” Harms said. “But I did”.

This is intended as a clinching example of the truth of Ekman’s understanding of the connection between facial expressions and emotions. Some people are just gifted this way and we’ve got to give them license to do what they’ve got to do.

But it’s evidence of something rather different, I think. The police officer shoots a man without any evidence that he’s breaking the law. It parallels Ekman’s own lack of evidence to sustain his theory of the connection between facial expressions and emotions. Ekman just knows. In both cases, the scientist and the police officer are omniscient and not to be doubted.

But what if the man who was shot had perceived, or had a gut feeling, that the officer was going to shoot him and shot him first? Would that have been OK too?

It is plain why Ekman’s model has been embraced by the security apparatus of the United States. It meshes with its panoptic surveillance and practice of preemption.

First there were body scanners, now there are face scanners. Both claim to be able to see what is beyond the surface—what dangerous weapons, feelings or thoughts we may be harbouring.

They will judge our intentions and act upon them. They’re looking for pre-crime. There is no right of appeal. The U.S. National Defense Authorization Act (NDAA) of 2012, allows for the indefinite detention of accused terrorists (including Americans) without charge or trial or evidence. [The Wikipedia account is here.] To repeat, ‘terrorism’ is broad enough to embrace defenders of civil liberties, anyone who dissents.

Ekman is a psychologist and psychology clearly has a lot to offer. But it does not have a monopoly on truth nor does it give the complete picture. This course aims for an interdisciplinary study of emotions. As this case suggest, sociology and politics also have something to say about emotions.

David Matsumoto and Paul Ekman. Facial expression analysis. Scholarpedia. 2008.

Beth Azar. What’s in a face? Do facial expressions reflect inner feelings? Or are they social devices for influencing others? APA Monitor on Psychology. January 2000.

Ian Leslie. Amanda Knox: What’s in a face? The Guardian, 8 October, 2011.

Kevin Randall. Human Lie Detector Paul Ekman Decodes The Faces Of Depression, Terrorism, And Joy. Fast Company. 12.14.2011.

One thought on “Reading Faces, Inferring Intent

  1. Pingback: Modelling the Smile | Open to the Future

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s