Emotion-Recognition Software Knows What Makes You Smile

A computer program that reads human expressions may bring an about-face in marketing.

Dutch researchers using the software recently for a consumer test project seconded what wise men have always known: Sweets are the surest way to make a woman smile.

Some 300 women in six European countries were filmed as they ate five foods: vanilla ice cream, chocolate, cereal bars, yogurt and apples. Not surprisingly, ice cream and chocolate produced the most happy expressions across the Old Continent.

Researchers chose women -- who tend to be more expressive than men -- at universities, shopping malls and city centers to test foods at face value. Cameras first recorded volunteers noshing, then participants provided a "posed" version of the expression they felt to give a more emphatic face for comparison.

Marketers increasingly use technology to determine what gives consumers bliss. Food and consumer goods giant Unilever, which used brain scans to demonstrate why we all scream for ice cream, hired software developers Theo Gevers and Nicu Sebe from the science department of the University of Amsterdam to run the European tests after reading about their experimental work deciphering the Mona Lisa's smile.

"We know ice cream is a real pleasure food; we turned to technology to back that up," said Mandy Mistlin, consumer scientist at Unilever UK. The software may eventually be used to test reduced-fat and -calorie ice creams to see if they maintain the "pleasure principle," she added.

The software, or others like it, may put a new face on market surveys. For professor Deborah Small of The Wharton School, who recently examined the effects of facial expressions in charity ad campaigns, excitement surrounding these technologies is considerable. The real test, she says, is whether they can become sophisticated enough to predict our responses.

But how does software analyze emotion?

When we smile, frown or grimace, thousands of tiny facial muscles are at work. Emotion-recognition software, or ERS, creates a 3-D face map, pinpointing 12 key trigger areas like eye and mouth corners.

Then a face-tracking algorithm matches the movements to six basic expression patterns, corresponding to anger, sadness, fear, surprise, disgust and happiness, or a mixture of them.

The ERS used in the taste test is a kissing cousin to programs created by MIT and Carnegie Mellon. Unlike those projects, the Dutch software, which works in real time and runs on a standard PC and webcam, is built with commercial applications in mind.

"I was happy when the testing was over," said Gevers. "Using the software on people eating was a challenge, something we would not have done in an academic lab. We didn't know precisely how well it would work, but it did." Gevers cited cultural differences (poker-faced Germans, stiff-lipped Brits) as another obstacle overcome during market research.

Not surprisingly, the software registered fewer smiley faces for healthy foods. Apples produced 87 percent neutral expressions, with Italians and Swedes registering disappointment when eating them; yogurt didn't fare much better, evoking "sad" expressions for 28 percent of Europeans.

"It's true to a certain extent that we are hard-wired to get pleasure from sweet foods," said psychologist Marcia Pelchat of the Monell Chemical Senses Center. "But you can learn to enjoy what's good for you, bearing in mind the distinction between liking and wanting."

Computers will probably not substitute for trained human observation in finding out what makes us tick. Although Pelchat has employed functional magnetic resonance imaging to study food cravings, she said that behavior study remains the "gold standard" of research.

"Technology helps when subjects don't have good conscious access to what's going on, or where people might want to conceal things," Pelchat said. "But it will never do the job alone."

It may already be time to mask your mien: Gevers and Sebe have a lot of bright ideas for the ERS, including a simplified version for consumers that will go on sale in August. Come fall, a site called Glad or Sad (which is not online yet) will analyze up to 1,000 user-provided photos daily.