How Does Facial Recognition Work?

Executive Summary

1. What Is Facial Recognition? Facial recognition is a way of using software to determine the similarity between two face images in order to evaluate a claim. The technology is used for a variety of purposes, from signing a user into their phone to searching for a particular person in a database of photos.

2. What Is Facial Characterization? Facial characterization refers to the practice of using software to classify a single face according to its gender, age, emotion, or other characteristics. Facial classification is distinct from facial recognition, whose purpose is instead to compare two different faces. Facial characterization is often confused with facial recognition in popular reporting, but they are actually distinct technologies. Many claims about the dangers of facial recognition are actually talking about characterization.

3. How Does Facial Recognition Work? Facial recognition uses computer-generated filters to transform face images into numerical expressions that can be compared to determine their similarity. These filters are usually generated by using deep “learning,” which uses artificial neural networks to process data.

4. How Accurate Is Facial Recognition? Facial recognition is improving rapidly, but while algorithms can achieve very high performance in controlled settings, many systems have lower performance when deployed in the real world. Summarizing the accuracy of a facial recognition system is difficult, however, as there is no single measure that provides a complete picture of performance.

5. What Are Similarity Scores? Similarity scores provide feedback to human operators about how similar the algorithm believes two images are. These scores can be misunderstood and are often treated as providing more authoritative information than they really do because of something known as the “prosecutor’s fallacy.”

6. What Are Comparison Thresholds? Facial recognition systems face a trade-off between low false negative rates and low false positive rates. Comparison thresholds are a way of using the similarity scores calculated by facial recognition algorithms to tune a system’s sensitivity to these two types of errors. Thresholds are adjusted to account for trade-offs between accuracy and risk when returning results to human adjudicators.

7. Is Facial Recognition Biased? Demographic differences in facial recognition accuracy rates have been well documented, but the evidence suggests that this problem can be addressed if sufficient attention is paid to improving both the training process for algorithms and the quality of captured images.

8. What Does This Mean? Facial recognition is usually discussed only in the context of its most dystopic applications, but it is a multifaceted tool that can be applied to a range of different problems. Facial recognition is used to aid human decisionmaking rather than replace it. Human oversight helps to mitigate the risk of errors. Operators need to understand how system performance can be affected by deployment conditions in order to put in place the right safeguards to manage trade-offs between accuracy and risk. A better understanding of the issues covered in this report will help ensure this technology can be deployed safely in ways that let us capture its benefits while managing risks.

This report was funded in part by the Department of Homeland Security as part of its homeland security mission to defend the homeland while upholding our nation’s values.

This report does not constitute the official position of the Department of Homeland Security. Any questions can be directed to the Office of Public Affairs at the Office of Biometric Identity Management.


James Andrew Lewis
Senior Vice President; Pritzker Chair; and Director, Strategic Technologies Program

William Crumpler