Facial recognition system
A facial recognition system is a computer application capable of identifying or verifying a person from a digital image or a video frame from a video source. One of the ways to do this is by comparing selected facial features from the image and a facial database.
It is typically used in security systems and can be compared to other biometrics such as fingerprint or eye iris recognition systems. Recently, it has also become popular as a commercial identification and marketing tool.
Some facial recognition algorithms identify facial features by extracting landmarks, or features, from an image of the subject's face. For example, an algorithm may analyze the relative position, size, and/or shape of the eyes, nose, cheekbones, and jaw. These features are then used to search for other images with matching features. Other algorithms normalize a gallery of face images and then compress the face data, only saving the data in the image that is useful for face recognition. A probe image is then compared with the face data. One of the earliest successful systems is based on template matching techniques applied to a set of salient facial features, providing a sort of compressed face representation.
Recognition algorithms can be divided into two main approaches, geometric, which looks at distinguishing features, or photometric, which is a statistical approach that distills an image into values and compares the values with templates to eliminate variances.
Popular recognition algorithms include principal component analysis using eigenfaces, linear discriminant analysis, elastic bunch graph matching using the Fisherface algorithm, the hidden Markov model, the multilinear subspace learning using tensor representation, and the neuronal motivated dynamic link matching.
A newly emerging trend, claimed to achieve improved accuracies, is three-dimensional face recognition. This technique uses 3D sensors to capture information about the shape of a face. This information is then used to identify distinctive features on the surface of a face, such as the contour of the eye sockets, nose, and chin.
One advantage of 3D facial recognition is that it is not affected by changes in lighting like other techniques. It can also identify a face from a range of viewing angles, including a profile view. Three-dimensional data points from a face vastly improve the precision of facial recognition. 3D research is enhanced by the development of sophisticated sensors that do a better job of capturing 3D face imagery. The sensors work by projecting structured light onto the face. Up to a dozen or more of these image sensors can be placed on the same CMOS chip—each sensor captures a different part of the spectrum.
Even a perfect 3D matching technique could be sensitive to expressions. For that goal a group at the Technion applied tools from metric geometry to treat expressions as isometries A company called Vision Access created a firm solution for 3D facial recognition. The company was later acquired by the biometric access company Bioscrypt Inc. which developed a version known as 3D FastPass.
A new method is to introduce a way to capture a 3D picture by using three tracking cameras that point at different angles; one camera will be pointing at the front of the subject, second one to the side, and third one at an angle. All these cameras will work together so it can track a subject’s face in real time and be able to face detect and recognize.
Skin texture analysis
Another emerging trend uses the visual details of the skin, as captured in standard digital or scanned images. This technique, called skin texture analysis, turns the unique lines, patterns, and spots apparent in a person’s skin into a mathematical space.
A different form of taking input data for face recognition is by using thermal cameras, by this procedure the cameras will only detect the shape of the head and it will ignore the subject accessories such as glasses, hats, or make up. A problem with using thermal pictures for face recognition is that the databases for face recognition is limited. Diego Socolinsky, and Andrea Selinger (2004) research the use of thermal face recognition in real life, and operation sceneries, and at the same time build a new database of thermal facial images. The research uses low-sensitive, low-resolution ferro-electric electrics sensors that are capable of acquire long wave thermal infrared (LWIR). The results show that a fusion of LWIR and regular visual cameras has the greater results in outdoor probes. Indoor results show that visual has a 97.05% accuracy, while LWIR has 93.93%, and the Fusion has 98.40%, however on the outdoor proves visual has 67.06%, LWIR 83.03%, and fusion has 89.02%. The study used 240 subjects over the period of 10 weeks to create the new database. The data was collected on sunny, rainy, and cloudy days.
Notable software with face recognition ability include:
- digiKam (KDE)
- iPhoto (Apple)
- Lightroom (Adobe)
- OpenCV (Open Source)
- OpenFace (Open Source)
- Photos (Apple)
- Photoshop Elements (Adobe Systems)
- Picasa (Google)
- Picture Motion Browser (Sony)
- Windows Live Photo Gallery (Microsoft)
- DeepFace (Facebook)
- visage SDK (Visage Technologies)
- Ayonix face SDK (Ayonix 3D face technology)
- TrueKey (Intel)
Notable users and deployments
The Australian and New Zealand Customs Services have an automated border processing system called SmartGate that uses facial recognition. The system compares the face of the individual with the image in the e-passport microchip to verify that the holder of the passport is the rightful owner.
Law enforcement agencies in the United States, including the Los Angeles County Sheriff, use arrest mugshot databases in their forensic investigative work. Law enforcement has been rapidly building a database of photos in recent years.
The U.S. Department of State operates one of the largest face recognition systems in the world with 117 million American adults including mostly law abiding citizens in its database. The photos are typically drawn from driver's license photos. Although it is still far from completion, it is being put to use in certain cities to give clues as to who was in the photo. The FBI uses the photos as an investigative lead not for positive identification.
In recent years Maryland has used facial recognition by comparing people's faces to their driver's license photos. The system drew controversy when it was used in Baltimore to arrest unruly protesters after the death of Freddie Gray in police custody. Many other states are using or developing a similar system however some states have laws prohibiting it's use.
The FBI has also instituted its Next Generation Identification program to include facial recognition, as well as more traditional biometrics like fingerprints and iris scans, which can pull from both criminal and civil databases.
In addition to being used for security systems, authorities have found a number of other applications for facial recognition systems. While earlier post-9/11 deployments were well publicized trials, more recent deployments are rarely written about due to their covert nature.
At Super Bowl XXXV in January 2001, police in Tampa Bay, Florida used Viisage facial recognition software to search for potential criminals and terrorists in attendance at the event. 19 people with minor criminal records were potentially identified.
In the 2000 presidential election, the Mexican government employed facial recognition software to prevent voter fraud. Some individuals had been registering to vote under several different names, in an attempt to place multiple votes. By comparing new facial images to those already in the voter database, authorities were able to reduce duplicate registrations. Similar technologies are being used in the United States to prevent people from obtaining fake identification cards and driver’s licenses.
There are also a number of potential uses for facial recognition that are currently being developed. For example, the technology could be used as a security measure at ATMs. Instead of using a bank card or personal identification number, the ATM would capture an image of the customer's face, and compare it to the account holder's photo in the bank database to confirm the customer's identity.
Facial recognition systems are used to unlock software on mobile devices. An independently developed Android Marketplace app called Visidon Applock makes use of the phone's built-in camera to take a picture of the user. Facial recognition is used to ensure only this person can use certain apps which they choose to secure.
Face detection and facial recognition are integrated into the iPhoto application for Macintosh, to help users organize and caption their collections.
Also, in addition to biometric usages, modern digital cameras often incorporate a facial detection system that allows the camera to focus and measure exposure on the face of the subject, thus guaranteeing a focused portrait of the person being photographed. Some cameras, in addition, incorporate a smile shutter, or automatically take a second picture if someone blinks during exposure.
Because of certain limitations of fingerprint recognition systems, facial recognition systems could be used as an alternative way to confirm employee attendance at work for the claimed hours.
Another use could be a portable device to assist people with prosopagnosia in recognizing their acquaintances.
Advantages and disadvantages
Compared to other technologies
Among the different biometric techniques, facial recognition may not be most reliable and efficient. However, one key advantage is that it does not require the cooperation of the test subject to work. Properly designed systems installed in airports, multiplexes, and other public places can identify individuals among the crowd, without passers-by even being aware of the system. Other biometrics like fingerprints, iris scans, and speech recognition cannot perform this kind of mass identification. However, questions have been raised on the effectiveness of facial recognition software in cases of railway and airport security.
Face recognition is far from perfect and struggles to perform under certain conditions. Ralph Gross, a researcher at the Carnegie Mellon Robotics Institute, describes one obstacle related to the viewing angle of the face: "Face recognition has been getting pretty good at full frontal faces and 20 degrees off, but as soon as you go towards profile, there've been problems."
Current facial recognition still often misidentifies people which can sometimes led to controversy. Google was criticized for racism in it's system when a black couple were misidentified as gorillas. Facial recognition software generally doesn't do as well in identifying minorities when most of the subjects used in testing the technology were from the majority group.
Other conditions where face recognition does not work well include poor lighting, sunglasses, hats, scarves, beards, long hair, makeup or other objects partially covering the subject’s face, and low resolution images.
Another serious disadvantage is that many systems are less effective if facial expressions vary. Even a big smile can render the system less effective. For instance: Canada now allows only neutral facial expressions in passport photos.
On several different occasions, firms like Google, Flickr and Nikon have been criticized for their software's lack of ability to recognize faces with darker toned skin colors. Link
There is also inconstancy in the datasets used by researchers. Researchers may use anywhere from several subjects to scores of subjects, and a few hundred images to thousands of images. It is important for researchers to make available the datasets they used to each other, or have at least a standard dataset.
Critics of the technology complain that the London Borough of Newham scheme has, as of 2004, never recognized a single criminal, despite several criminals in the system's database living in the Borough and the system having been running for several years. "Not once, as far as the police know, has Newham's automatic facial recognition system spotted a live target." This information seems to conflict with claims that the system was credited with a 34% reduction in crime (hence why it was rolled out to Birmingham also). However it can be explained by the notion that when the public is regularly told that they are under constant video surveillance with advanced face recognition technology, this fear alone can reduce the crime rate, whether the face recognition system technically works or does not. This has been the basis for several other face recognition based security systems, where the technology itself does not work particularly well but the user's perception of the technology does.
A system at Boston's Logan Airport was shut down in 2003 after failing to make any matches during a two-year test period.
Civil rights right organizations and privacy campaigners such as the Electronic Frontier Foundation and the ACLU express concern that privacy is being compromised by the use of surveillance technologies. Some fear that it could lead to a “total surveillance society,” with the government and other authorities having the ability to know the whereabouts and activities of all citizens around the clock. This knowledge has been, is being, and could continue to be deployed to prevent the lawful exercise of rights of citizens to criticize those in office, specific government policies or corporate practices. Many centralized power structures with such surveillance capabilities have abused their privileged access to maintain control of the political and economic apparatus, and to curtail populist reforms.
Facial recognition can be used not just to identify an individual, but also to unearth other personal data associated with an individual – such as other photos featuring the individual, blog posts, social networking profiles, Internet behavior, travel patterns, etc. – all through facial features alone. Moreover, individuals have limited ability to avoid or thwart facial recognition tracking unless they hide their faces. This fundamentally changes the dynamic of day-to-day privacy by enabling any marketer, government agency, or random stranger to secretly collect the identities and associated personal information of any individual captured by the facial recognition system.
Social media web sites such as Facebook have very large numbers of photographs of people, annotated with names. This represents a database which may be abused by governments for facial recognition purposes. Facial recognition was used in Russia to harass women allegedly involved in online pornography. In Russia there is an app 'FindFace' which can identify faces with about 70% accuracy using the social media app called VK. This app would not be possible in other countries which do not use VK as their social media platform photos are not stored the same way as with VK.
In July 2012, a hearing was held before the Subcommittee on Privacy, Technology and the Law of the Committee on the Judiciary, United States Senate, to address issues surrounding what facial recognition technology means for privacy and civil liberties.
In 2014, the National Telecommunications and Information Association (NTIA) began a multi-stakeholder process to engage privacy advocates and industry representatives to establish guidelines regarding the use of facial recognition technology by private companies. In June 2015, privacy advocates left the bargaining table over what they felt was an impasse based on the industry representatives being unwilling to agree to consent requirements for the collection of facial recognition data. The NTIA and industry representatives continued without the privacy representatives, and draft rules are expected to be presented in the spring of 2016.
States have begun enacted legislation to protect citizen's biometric data privacy. Illinois enacted the Biometric Information Privacy Act in 2008. Facebook's DeepFace has become the subject of several class action lawsuits under the Biometric Information Privacy Act, with claims alleging that Facebook is collecting and storing facial recognition data of its users without obtaining informed consent, in direct violation of the Biometric Information Privacy Act. The most recent case was dismissed in January 2016 because the court lacked jurisdiction. Therefore, it is still unclear if the Biometric Information Privacy Act will be effective in protecting biometric data privacy rights.
Pioneers of automated facial recognition include Woody Bledsoe, Helen Chan Wolf, and Charles Bisson.
During 1964 and 1965, Bledsoe, along with Helen Chan and Charles Bisson, worked on using the computer to recognize human faces (Bledsoe 1966a, 1966b; Bledsoe and Chan 1965). He was proud of this work, but because the funding was provided by an unnamed intelligence agency that did not allow much publicity, little of the work was published. Given a large database of images (in effect, a book of mug shots) and a photograph, the problem was to select from the database a small set of records such that one of the image records matched the photograph. The success of the method could be measured in terms of the ratio of the answer list to the number of records in the database. Bledsoe (1966a) described the following difficulties:
|“||This recognition problem is made difficult by the great variability in head rotation and tilt, lighting intensity and angle, facial expression, aging, etc. Some other attempts at facial recognition by machine have allowed for little or no variability in these quantities. Yet the method of correlation (or pattern matching) of unprocessed optical data, which is often used by some researchers, is certain to fail in cases where the variability is great. In particular, the correlation is very low between two pictures of the same person with two different head rotations.||”|
|— Woody Bledsoe, 1966|
This project was labeled man-machine because the human extracted the coordinates of a set of features from the photographs, which were then used by the computer for recognition. Using a graphics tablet (GRAFACON or RAND TABLET), the operator would extract the coordinates of features such as the center of pupils, the inside corner of eyes, the outside corner of eyes, point of widows peak, and so on. From these coordinates, a list of 20 distances, such as width of mouth and width of eyes, pupil to pupil, were computed. These operators could process about 40 pictures an hour. When building the database, the name of the person in the photograph was associated with the list of computed distances and stored in the computer. In the recognition phase, the set of distances was compared with the corresponding distance for each photograph, yielding a distance between the photograph and the database record. The closest records are returned.
Because it is unlikely that any two pictures would match in head rotation, lean, tilt, and scale (distance from the camera), each set of distances is normalized to represent the face in a frontal orientation. To accomplish this normalization, the program first tries to determine the tilt, the lean, and the rotation. Then, using these angles, the computer undoes the effect of these transformations on the computed distances. To compute these angles, the computer must know the three-dimensional geometry of the head. Because the actual heads were unavailable, Bledsoe (1964) used a standard head derived from measurements on seven heads.
After Bledsoe left PRI in 1966, this work was continued at the Stanford Research Institute, primarily by Peter Hart. In experiments performed on a database of over 2000 photographs, the computer consistently outperformed humans when presented with the same recognition tasks (Bledsoe 1968). Peter Hart (1996) enthusiastically recalled the project with the exclamation, "It really worked!"
By about 1997, the system developed by Christoph von der Malsburg and graduate students of the University of Bochum in Germany and the University of Southern California in the United States outperformed most systems with those of Massachusetts Institute of Technology and the University of Maryland rated next. The Bochum system was developed through funding by the United States Army Research Laboratory. The software was sold as ZN-Face and used by customers such as Deutsche Bank and operators of airports and other busy locations. The software was "robust enough to make identifications from less-than-perfect face views. It can also often see through such impediments to identification as mustaches, beards, changed hair styles and glasses—even sunglasses".
In about January 2007, image searches were "based on the text surrounding a photo," for example, if text nearby mentions the image content. Polar Rose technology can guess from a photograph, in about 1.5 seconds, what any individual may look like in three dimensions, and claimed they "will ask users to input the names of people they recognize in photos online" to help build a database . Identix, a company out of Minnesota, has developed the software, FaceIt. FaceIt can pick out someone’s face in a crowd and compare it to databases worldwide to recognize and put a name to a face. The software is written to detect multiple features on the human face. It can detect the distance between the eyes, width of the nose, shape of cheekbones, length of jawlines and many more facial features. The software does this by putting the image of the face on a faceprint, a numerical code that represents the human face. Facial recognition software used to have to rely on a 2D image with the person almost directly facing the camera. Now, with FaceIt, a 3D image can be compared to a 2D image by choosing 3 specific points off of the 3D image and converting it into a 2D image using a special algorithm that can be scanned through almost all databases. In 2006, the performance of the latest face recognition algorithms were evaluated in the Face Recognition Grand Challenge (FRGC). High-resolution face images, 3-D face scans, and iris images were used in the tests. The results indicated that the new algorithms are 10 times more accurate than the face recognition algorithms of 2002 and 100 times more accurate than those of 1995. Some of the algorithms were able to outperform human participants in recognizing faces and could uniquely identify identical twins.
U.S. Government-sponsored evaluations and challenge problems have helped spur over two orders-of-magnitude in face-recognition system performance. Since 1993, the error rate of automatic face-recognition systems has decreased by a factor of 272. The reduction applies to systems that match people with face images captured in studio or mugshot environments. In Moore's law terms, the error rate decreased by one-half every two years.
Low-resolution images of faces can be enhanced using face hallucination. Further improvements in high resolution, megapixel cameras in the last few years have helped to resolve the issue of insufficient resolution.
- AI effect
- Applications of artificial intelligence
- Automatic number plate recognition
- Biometric technology in access control
- Coke Zero Facial Profiler
- Computer processing of body language
- Computer vision
- Face detection
- Face perception
- Glasgow Face Matching Test
- Iris recognition
- Artificial intelligence for video surveillance
- Multimedia information retrieval
- Multilinear subspace learning
- Pattern recognition, analogy and case-based reasoning
- Retinal scan
- Super recognisers
- Template matching
- Three-dimensional face recognition
- Vein matching
- "Facial Recognition Applications". Animetrics. Retrieved 2008-06-04.
- "Facial Recognition: Who's Tracking You in Public?". Consumer Reports. Retrieved 2016-04-05.
- "Airport Facial Recognition Passenger Flow Management". hrsid.com.
- Bonsor, K. "How Facial Recognition Systems Work". Retrieved 2008-06-02.
- Smith, Kelly. "Face Recognition" (PDF). Retrieved 2008-06-04.
- R. Brunelli and T. Poggio, "Face Recognition: Features versus Templates", IEEE Trans. on PAMI, 1993, (15)10:1042-1052
- R. Brunelli, Template Matching Techniques in Computer Vision: Theory and Practice, Wiley, ISBN 978-0-470-51706-2, 2009 ( TM book)
- Williams, Mark. "Better Face-Recognition Software". Retrieved 2008-06-02.
- Crawford, Mark. "Facial recognition progress report". SPIE Newsroom. Retrieved 2011-10-06.
- Kimmel, Ron. "Three-dimensional face recognition" (PDF). Retrieved 2005-01-01.
- Face Recognition with OpenCV
- "Next Generation Identification". FBI. Retrieved 2016-04-05.
- Vogel, Ben. "Panama puts names to more faces". IHS Jane's Airport Review. Archived from the original on 2014-10-12. Retrieved 2014-10-07.
Under the USD11 million contract, a cluster of sixty computers, a fibre optic network, and 150 surveillance cameras were installed in the terminal and at about 30 gates.
- Greene, Lisa (2001-02-15). "Face scans match few suspects" (SHTML). St. Petersburg Times. Archived from the original on 2014-11-30. Retrieved 2011-06-30.
By using Viisage software, police matched 19 people's faces to photos of people arrested in the past for minor pickpocketing, fraud and other charges. They weren't charged with any game-day misdeeds. THIS IS A FARCE
- Krause, Mike (2002-01-14). "Is face recognition just high-tech snake oil?". Enter Stage Right. ISSN 1488-1756. Archived from the original on 2002-01-24. Retrieved 2011-06-30.
- "Mexican Government Adopts FaceIt Face Recognition Technology to Eliminate Duplicate Voter Registrations in Upcoming Presidential Election". Business Wire. 2000-05-11. Retrieved 2008-06-02.
- House, David. "Facial recognition at DMV". Oregon Department of Transportation. Archived from the original on 2007-02-05. Retrieved 2007-09-17.
Oregon DMV is going to start using “facial recognition” software, a new tool in the prevention of fraud, required by a new state law. The law is designed to prevent someone from obtaining a driver license or ID card under a false name.
- Schultz, Zac. "Facial Recognition Technology Helps DMV Prevent Identity Theft". WMTV News, Gray Television. Retrieved 2007-09-17.
Madison: ...The Department of Motor Vehicles is using... facial recognition technology [to prevent ID theft]
- Visidon. "Visidon AppLock - Android Apps on Google Play". google.com.
- "iPhoto '09 & iPhoto '11: Improving face recognition results". apple.com.
- "Passport Canada - Photos". passportcanada.gc.ca.
- ForbesTech: Google Photos Tags Two African-Americans As Gorillas Through Facial Recognition Software
- Albiol,A., Albiol,A., Oliver,J., Mossi,J.M.(2012). Who is who at different cameras: people re- identification using depth cameras. Computer Vision, IET. Vol 6(5), 378-387.
- "These Goofy-Looking Glasses Could Make You Invisible to Facial Recognition Technology". Slate. 18 January 2013. Retrieved 22 January 2013.
- Meek, James (2002-06-13). "Robo cop". London: UK Guardian newspaper.
- "Birmingham City Centre CCTV Installs Visionics' FaceIt". Business Wire. 2008-06-02.
- Willing, Richard (2003-09-02). "Airport anti-terror systems flub tests; Face-recognition technology fails to flag 'suspects'" (Abstract). USA Today. Retrieved 2007-09-17.
- "EFF Sues FBI For Access to Facial-Recognition Records". Electronic Frontier Foundation.
- "Q&A On Face-Recognition". American Civil Liberties Union.
- "Civil Liberties & Facial Recognition Software". About.com, The New York Times Company. pp. pp. 2. Archived from the original on 2006-03-01. Retrieved 2007-09-17.
A few examples which have already arisen from surveillance video are: using license plates to blackmail gay married people, stalking women, tracking estranged spouses...
- Harley Geiger (2011-12-06). "Facial Recognition and Privacy". Center for Democracy & Technology. Retrieved 2012-01-10.
- Martin Koste (2013-10-28). "A Look Into Facebook's Potential To Recognize Anybody's Face". NPR. Archived from the original on 2013-11-01. Retrieved 2013-12-25.
- What Facial Recognition Technology Means for Privacy and Civil Liberties: Hearing before the Subcommittee on Privacy, Technology and the Law of the Committee on the Judiciary, United States Senate, One Hundred Twelfth Congress, Second Session, July 18, 2012
- "Privacy Multistakeholder Process: Facial Recognition Technology". National Telecommunications and Information Association. Retrieved April 5, 2016.
- McCabe, David. "Facial recognition talks break down as privacy advocates withdraw". TheHill. Retrieved 2016-04-05.
- Weaver, Dustin. "Business eyes facial recognition guidelines". TheHill. Retrieved 2016-04-05.
- "740 ILCS 14/ Biometric Information Privacy Act.". www.ilga.gov. Retrieved 2016-04-05.
- "Facebook Keeps Getting Sued Over Face-Recognition Software, And Privacy Groups Say We Should Be Paying More Attention". International Business Times. Retrieved 2016-04-05.
- Herra, Dana. "Judge tosses Illinois privacy law class action vs Facebook over photo tagging; California cases still pending". cookcountyrecord.com. Retrieved 2016-04-05.
- "Mugspot Can Find A Face In The Crowd -- Face-Recognition Software Prepares To Go To Work In The Streets". ScienceDaily. 12 November 1997. Retrieved 2007-11-06.
- R. Kimmel and G. Sapiro (2003-04-30). "The Mathematics of Face Recognition". SIAM News. Retrieved 2003-04-30.
- "Face Homepage". nist.gov.
- What are Biometrics? White Paper Published by Aware, Inc., January 2014
- Farokhi, Sajad; Shamsuddin, Siti Mariyam; Flusser, Jan; Sheikh, U.U; Khansari, Mohammad; Jafari-Khouzani, Kourosh (2014). "Near infrared face recognition by combining Zernike moments and undecimated discrete wavelet transform". Digital Signal Processing. 31 (1). doi:10.1016/j.dsp.2014.04.008.
- "The Face Detection Algorithm Set to Revolutionize Image Search" (Feb. 2015), MIT Technology Review
- Garvie, Clare; Bedoya, Alvaro; Frankle, Jonathan (October 18, 2016). Perpetual Line Up: Unregulated Police Face Recognition in America. Center on Privacy & Technology at Georgetown Law. Retrieved October 22, 2016.
- "Facial Recognition Software 'Sounds Like Science Fiction,' but May Affect Half of Americans". As It Happens. Canadian Broadcasting Corporation. October 20, 2016. Retrieved October 22, 2016. Interview with Alvaro Bedoya, executive director of the Center on Privacy & Technology at Georgetown Law and co-author of Perpetual Line Up: Unregulated Police Face Recognition in America.
- Media related to Facial recognition system at Wikimedia Commons
- A Photometric Stereo Approach to Face Recognition". University of the West of England. http://www1.uwe.ac.uk/et/mvl/projects/facerecognition.aspx