Forget Police Sketches: Researchers Perfectly Reconstruct Faces by Reading Brainwaves

Singularity Hub – by Shelly Fan

Picture this: you’re sitting in a police interrogation room, struggling to describe the face of a criminal to a sketch artist. You pause, wrinkling your brow, trying to remember the distance between his eyes and the shape of his nose.

Suddenly, the detective offers you an easier way: would you like to have your brain scanned instead, so that machines can automatically reconstruct the face in your mind’s eye from reading your brain waves?  

Sound fantastical? It’s not. After decades of work, scientists at Caltech may have finally cracked our brain’s facial recognition code. Using brain scans and direct neuron recording from macaque monkeys, the team found specialized “face patches” that respond to specific combinations of facial features.

Like dials on a music mixer, each patch is fine-tuned to a particular set of visual information, which then channel together in different combinations to form a holistic representation of every distinctive face.

The values of each dial were so predictable that scientists were able to recreate a face the monkey saw simply by recording the electrical activity of roughly 200 brain cells. When placed together, the reconstruction and the actual photo were nearly indistinguishable.

“This was mind-blowing,” says lead author Dr. Doris Tsao.

Even more incredibly, the work “completely kills” the dominant theory of facial processing, potentially ushering in “a revolution in neuroscience,” says Dr. Rodrigo Quian Quiroga, a neuroscientist at the University of Leichester who was not involved in the work.

What’s in a face?

On average, humans are powerful face detectors, beating even the most sophisticated face-tagging algorithms.

Most of us are equipped with the uncanny ability to spot a familiar set of features from a crowd of eyes, noses and mouths. We can unconsciously process a new face in milliseconds, and—when exposed to that face over and over—often retain that memory for decades to come.

Under the hood, however, facial recognition is anything but simple. Why is it that we can detect a face under dim lighting, half obscured or at a weird angle, but machines can’t? What makes people’s faces distinctively their own?

When light reflected off a face hits your retina, the information passes through several layers of neurons before it reaches a highly specialized region of the visual cortex: the inferotemporal (IT) region, a small nugget of brain at the base of the brain. This region is home to “face cells”: groups of neurons that only respond to faces but not to objects such as houses or landscapes.

In the early 2000s, while recording from epilepsy patients with electrodes implanted into their brains, Quian Quiroga and colleagues found that face cells are particularly picky. So-called “Jennifer Aniston” cells, for example, would only fire in response to photos of her face and her face alone. The cells quietly ignored all other images, including those of her with Brad Pitt.

This led to a prevailing theory that still dominates the field: that the brain contains specialized “face neurons” that only respond to one or a few faces, but do so holistically.

But there’s a problem: the theory doesn’t explain how we process new faces, nor does it get into the nitty-gritty of how faces are actually encoded inside those neurons.

Mathematical faces

In a stroke of luck, Tsao and team blew open the “black box” of facial recognition while working on a different problem: how to describe a face mathematically, with a matrix of numbers.

Using a set of 200 faces from an online database, the team first identified “landmark” features and labeled them with dots. This created a large set of abstract dot-to-dot faces, similar to what filmmakers do during motion capture.

Then, using a statistical method called principle component analysis, the scientists extracted 25 measurements that best represented a given face. These measurements were mostly holistic: one “shape dimension,” for example, encodes for the changes in hairline, face width, and height of eyes.

mathematically-generated-faces-facial-recognition-algorithms-Tsao
This figure shows eight different real faces that were presented to a monkey, together with reconstructions made by analyzing electrical activity from 205 neurons recorded while the monkey was viewing the faces. Credit: Doris Tsao

By varying these “shape dimensions,” the authors generated a set of 2,000 black-and-white faces with slight differences in the distance between the brows, skin texture, and other facial features.

Division of labor

In macaque monkeys with electrodes implanted into their brains, the team recorded from three “face patches”—brain areas that respond especially to faces—while showing the monkeys the computer-generated faces.

As it turns out, each face neuron only cared about a single set of features. A neuron that only cares about hairline and skinny eyebrows, for example, would fire up when it detects variations in those features across faces. If two faces had similar hairlines but different mouths, those “hairline neurons” stayed silent.

What’s more, cells in different face patches processed complementary information. The anterior medial face patch, for example, mainly responded to distances between features (what the team dubs “appearance”). Other patches fired up to information about shapes, such as the curvature of the nose or length of the mouth.

In a way, these feature neurons are like compasses: they only activate when the measurement is off from a set point (magnetic north, for a compass). Scientists aren’t quite sure how each cell determines its set point. However, combining all the “set points” generates a “face space”—a sort of average face, or a face atlas.

From there, when presented with a new face, each neuron will measure the difference between a feature (distance between eyes, for example) and the face atlas. Combine all those differences, and voilá—you have a representation of a new face.

Reading faces

Once the team figured out this division of labor, they constructed a mathematical model to predict how the patches process new faces.

Here’s the cool part: the medley of features that best covered the entire shape and look of a face was fairly abstract, including the distance between the brows. Sound familiar? That’s because the brain’s preferred set of features were similar to the “landmarks” that the team first intuitively labeled to generate their face database.

“We thought we had picked it out of the blue,” says Tsao.

But it makes sense. “If you look at methods for modeling faces in computer vision, almost all of them…separate out the shape and appearance,” she explains. “The mathematical elegance of the system is amazing.”

The team showed the monkeys a series of new faces while recording from roughly 200 neurons in the face patches. Using their mathematical model, they then calculated what features each neuron encodes for and how they combine.

The result? A “stunning accurate” reconstruction of the faces the monkeys were seeing. So accurate, in fact, that the algorithm-generated faces were nearly indistinguishable from the original.

“It really speaks to how compact and efficient this feature-based neural code is,” says Tsao, referring to the fact that such a small set of neurons contained sufficient information for a full face.

Forward facing

Tsao’s work doesn’t paint the full picture. The team only recorded from two out of six face patches, suggesting that other types of information processing may be happening alongside Tsao’s model.

But the study breaks the “black box” norm that’s plagued the field for decades.

“Our results demonstrate that at least one subdivision of IT cortex can be explained by an explicit, simple model, and ‘black box’ explanations are unnecessary,” the authors conclude (pretty sassy for an academic paper!).

While there aren’t any immediate applications, the new findings could eventually guide the development of brain-machine interfaces that directly stimulate the visual cortex to give back sight to the blind. They could also help physicians understand why some people suffer from “face blindness,” and engineer prosthetics to help.

Singularity Hub

Jim

8 thoughts on “Forget Police Sketches: Researchers Perfectly Reconstruct Faces by Reading Brainwaves

  1. *** Suddenly, the detective offers you an easier way: would you like to have your brain scanned instead, so that machines can automatically reconstruct the face in your mind’s eye from reading your brain waves? ***

    I wouldn’t like that at all if it involved implanting electrodes in my brain, as was done with the monkeys in this study.

    *** While there aren’t any immediate applications, the new findings could eventually guide the development of brain-machine interfaces that directly stimulate the visual cortex to give back sight to the blind. They could also help physicians understand why some people suffer from “face blindness,” and engineer prosthetics to help. ***

    I’m sure organizations that hate our freedom, like DARPA and the various three-letter agencies, will be disappointed if they can’t benefit from this research in some way.

  2. This is a pretext to involuntary interrogation techniques, they just showed their hand.

    1. No doubt they’d like to be able to do something like that. But I think they still have a ways to go before that’s is technologically achievable.

      In a nutshell, here’s my understanding of what they’ve done with this research:

      (1) They recorded the signals and patterns in the monkeys’ brains that occurred while those monkeys were looking at faces, using electrodes implanted in their brains.

      (2) Those recorded signals were analyzed and decoded.

      (3) Finally, they used the “cracked code” to reconstruct the faces the monkeys had been looking at when their brain signals were being recorded.

      Granted, this is an impressive (if disturbing) achievement. But it’s important to note that the researchers didn’t read the monkeys’ MEMORIES to create those faces. So that bit of speculative fiction at the beginning of the article, with the detective offering to hook someone up to a machine to reconstruct a face, is a big leap from what was done here. To reconstruct a face you’d seen in the past, they’d have to implant electrodes in your brain (unless they can find a way to do this with electrodes on the scalp, like an EEG uses), somehow force you to remember the specific face they want (as opposed to any of the other faces you’ve seen in your life), and somehow ensure that you remember it correctly.

      I don’t think they’re even close to that sort of capability yet. To the best of my knowledge, even simple lie detection using very sophisticated, laborious, and invasive techniques such as fMRI still aren’t completely reliable.

        1. I hear ya. I used to think science and technology were so cool, but now it seems like we constantly have to worry about how the next big breakthrough will be used by governments for evil purposes. It’s easy to get discouraged.

          On the bright side, a lot of the government’s evil technology has come back to bite it in the ass. The Internet is a great example. It has some major downsides, but I can’t begin to think of all the important things I wouldn’t have learned about weapons, countermeasures, politics, etc., if not for the info shared on the Web.

  3. Good thing they don’t scan a lot of people’s brain waves with this new technology.

    Because all they would get is a 3D reconstruction of a big dumb…

    Asswhole.

    For a face…

    Frightening thought isn’t it.

    Gives a new meaning for that old saying.

    “Sh!t for Brains.”

    Oh.. I forgot. ..

    Bada …ching.

Join the Conversation

Your email address will not be published. Required fields are marked *


*