Our biometric data is freely available to anybody with an AI model and a camera. Facial recognition software is such a pervasive technology that we submit our data whenever we go through airport security or walk into a drug store. You start to wonder if it’s possible to hide our facial features or—on the extreme end—change our appearance to such an extent that it fools the AI algorithm.

Couldn’t you just wear an N95 mask, scarf, and sunglasses to dodge Big Brother? So far, the best way to avoid being picked up by facial recognition is to avoid cameras. But that task may soon become near impossible. Privacy experts warn that we may already be on the losing end of protecting our biometric data. Soon, the only real defense may be federal regulation.

Cynthia Rudin

Gilbert, Louis, and Edward Lehrman Distinguished Professor of Computer Science; Departments of Computer Science, Electrical and Computer Engineering, Statistical Science, Mathematics, and Biostatistics & Bioinformatics; Duke University

I think you could not realistically change your face to fool state-of-the-art facial recognition. I think during the pandemic they changed the systems to rely heavily on the shape of people’s eyes, because so many people were wearing masks over their noses and mouths. I don’t honestly know how people could realistically change the shape of their eyes to fool these systems. If you wore sunglasses and then did something to your face (maybe wear a mask or crazy dramatic makeup) then it would be harder to detect your face, but that’s cheating on the question—that’s not changing your face, that’s just hiding it!

But let’s say you did something dramatic to change your face—something really, really dramatic—so that a face recognition system wouldn’t recognize you. Perhaps it would be some kind of plastic surgery. Well, then what? As soon as your face ends up on the internet with your name (think of a friend tagging you on social media or you giving a lecture that appears online), then all the facial recognition systems that look for people on the internet will be able to identify you anyway.

And now your face won’t match your driver’s license or passport, so traveling will be really difficult for you. So, honestly, why bother? In any case, I’m glad you asked this question, because it shows how futile it is to avoid other people capturing our biometrics. Asking our governments to create laws to protect us is much easier than changing our faces dramatically all the time.

Walter Scheirer

Dennis O. Doughty Collegiate Professor of Engineering; Department of Computer Science & Engineering; University of Notre Dame

The answer to the question of how much one must alter their appearance to avoid facial recognition depends on the way the facial recognition algorithm is being used. In human biometrics, there are two common modes of matching identities: 1-to-1 and 1-to-many. In the 1-to-1 mode, a verification is made that the claimed identity of the person in front of the camera matches a previously enrolled photo of that identity in the system’s database. This scenario has been common for many years for high-security computer authentication and law enforcement investigations, but is now common in other consumer-facing contexts such as boarding an international flight at the airport. In the 1-to-many mode, a photo of an unknown subject is matched against a set of previously enrolled photos of identities of interest. This mode is frequently used in video-based surveillance settings, including law enforcement and government intelligence operations.

Evading the 1-to-1 mode in a controlled setting (e.g., in a booking room at the local jail) is very difficult. Major advances have been made in facial recognition algorithms through the use of sophisticated artificial neural networks, which achieve remarkably high matching accuracies across a wide range of appearances for a single individual. If the acquired photo has a frontal pose, with a neutral expression, good lighting, and a controlled background, basic evasion techniques such as cosmetics, adding/removing facial hair, changing hairstyle, etc., will not work. Recent research has examined the impact of plastic surgery on face recognition, and while unaesthetic drastic alterations to facial structure can work somewhat, more common cosmetic procedures don’t have as large of an impact as one might think.

Evading the 1-to-many mode in an uncontrolled surveillance setting is a bit easier—one need not resort to surgical measures. Even the best neural networks struggle with low-quality photos that lack information-rich pixels of the human face, especially when matching against a large list of potential identities. Thus the first step is to deny the algorithm those pixels by occluding the face. Cover the face in cases where that isn’t suspicious, e.g., wear a scarf in the wintertime, sun glasses on a bright day. Hats with wide brims are also a confound, as they can hide the forehead and hair, and cast a shadow on the face. Holding a hand over the face is also good for this. The second step is to look down while in motion so any camera in the vicinity will not capture a good frontal image of the face. Third, if one can move quickly, that might cause motion blur in the captured photo—consider jogging or riding a bike.

My best practical advice for evasion: know where facial recognition is being deployed and simply avoid those areas. How long this advice remains useful though depends on how widespread the technology becomes in the coming years.

Today’s algorithms are rather tolerant of subtle changes to facial appearance, both innocent (e.g., acne, mild swelling) or not (e.g., botox).

Xiaoming Liu

Anil K. & Nandita K. Jain Endowed Professor; Computer Science and Engineering (CSE), College of Engineering; Michigan State University

First of all, my definition of “avoid facial recognition” means that a Facial Recognition System (FRS) fails to recognize a subject’s face when the subject is captured by a camera.

There are a few ways to “proactively” fail a FRS:

1. Physical adversarial attacks. Most of AI models are vulnerable to adversarial attacks, i.e., a minor modification of the input data sample may completely fail an AI system. The same thing applies to FRS. The key here is learn a specific “minor modification” so that such modification is able to fail FRS. For example, CMU has one paper on designing special glasses that can fail a FRS. You could imagine that someone can follow similar idea to design a scarf, facial mask, or even mustache that can also fail FRS

2. You can also proactively change your facial appearance so FRS would recognize you as someone else. A common way is to apply makeup. However, it is tricky to answer the question, that is, where and how much amount of makeup I shall apply so that I can just fail FRS. The answer is very much subject dependent. The reason is that some individuals’ face appearance is more common and more similar to others, thus a relatively small makeup modification might be sufficient to misrecognize him as someone else. In contrast, if one individual’s face appearance is very unique, then a lot more makeup modification would be needed. One interesting application might be the following: an interactive smartphone app looks at my face via phone’s camera, tells me where I shall start to apply makeup, and iteratively gives me instructions on where and maybe what color of makeup so that I can be misrecognized by FRS with minimal makeup. Other than makeup, one can also use a high-cost facial mask, which may be more common in Hollywood movies though.

As you may tell, the probability of successfully failing FRS is somehow correlated with the amount of effort the subject is making, too. Approach 1 is easier for the users, but not too reliable, especially when one likes to design a “universal” adversarial attack, such as one glasses for everyone. Approach 2 is more personalized and works better, yet requires more effort.

Kevin W. Bowyer

Schubmehl-Prein Family Professor of Computer Science & Engineering; University of Notre Dame

The answer is: “it depends.” It depends (at least) on the face matching algorithm used, and the threshold used with that algorithm.

To understand better, start with the fact that face recognition is about comparing two images and deciding if the faces in the images are (a) similar enough that they must be the same person, or (b) dissimilar enough that they must have come from different people.

Each face recognition algorithm is a particular method of computing a “feature vector” (typically called an “embedding” these days) from an image of a face, and a method of comparing two feature vectors to give a value for how similar they are. A single face image might get reduced to a list of 512 numbers (the “feature vector” or “embedding.”) The feature vectors from two face images might be compared and give a similarity result between 0 and 100, or between -1 and +1. The 100 or the +1 would only result if you compared two copies of the same image; it would be an unusual result to see in practice.

Imagine we are using a state-of-the-art face recognition algorithm and using a similarity value that falls into the -1 to +1 range. The similarity values for comparisons between all sorts of pairs of images of different people might be centered around 0.0 or just slightly above that. The similarity values for comparisons between all sorts of pairs of images of the same person might be centered around 0.8 or just slightly above that. If the image acquisition for the application is well-controlled, perhaps like a driver’s license photo, then the average similarity value for two images of the same person will be higher. If the image acquisition is less well-controlled, perhaps like images taken from frames of video as people enter a store, then the average similarity value for two images of the same person will be lower.

Someone will decide on a threshold value to be used for recognition. If the value 0.7 is selected as the threshold, then when two images are compared and their similarity is below 0.7, the system says that they must be images of different persons. If the value is equal to or above 0.7, the system says that they must be images of the same person.

At this point, we can see that the original question, “How much do I need to change my appearance to avoid facial recognition?” can be reformulated to “What are the best things to do to lower the similarity value for my new image when it is compared to my old image?”

There are lots of things that you might do. You might put on dark sunglasses, and change your hairstyle and still look natural. You might make some exaggerated facial expression, but that probably won’t look natural. You might avoid looking directly at the camera, so that the new photo is off-angle. More drastically, you might gain or lose weight. Or you might apply cosmetics to “change your look.” None of these things can guarantee that you won’t match your old photo. You don’t necessarily know what old photo of you will be used to compare with your new photo, or what algorithm will be used, or what threshold will be used. If you knew all of those things, you could experiment with the most effective approach to take.