Facial recognition – fascinating and intriguing

But more about that later.

In this web dossier, you'll discover the seven face recognition facts and trends set to shape the landscape in 2020.

Top technologies and providers
AI impact - Getting better all the time
2019-2024 markets and dominant use-cases
Face recognition in China, India, United States, EU, and the UK, Brazil, Russia...
Privacy vs Security: laissez-faire or freeze, regulate or ban?
Latest hacks: can facial recognition be fooled?
Moving forward: towards hybridized solutions.
Let’s jump right in.

How facial recognition works
Facial recognition is the process of identifying or verifying the identity of a person using their face. It captures, analyzes, and compares patterns based on the person's facial details.

The face detection process is an essential step as it detects and locates human faces in images and videos.
The face capture process transforms analogue information (a face) into a set of digital information (data) based on the person's facial features.
The face match process verifies if two faces belong to the same person.

Today it's considered to be the most natural of all biometric measurements.

And for a good reason – we recognize ourselves not by looking at our fingerprints or irises, for example, but by looking at our faces.

Thales has specialized in biometric technologies for almost 30 years. The company has always collaborated with the best players when it comes to research, ethics, and biometric applications.

Face match

Before we go any further, let's quickly define two keywords: "identification" and "authentication".

Face recognition data to identify and verify
Biometrics are used to identify and authenticate a person using a set of recognizable and verifiable data unique and specific to that person.

For more on biometrics definition, visit our web dossier on biometrics.

Identification answers the question: "Who are you?"

Authentication answers the question: "Are you really who you say you are?"

Stay with us. Here are some examples :

In the case of facial biometrics, a 2D or 3D sensor "captures" a face. It then transforms it into digital data by applying an algorithm before comparing the image captured to those held in a database.

These automated systems can be used to identify or check the identity of individuals in just a few seconds based on their facial features: spacing of the eyes, bridge of the nose, the contour of the lips, ears, chin, etc.

They can even do this in the middle of a crowd and within dynamic and unstable environments. Proof of this can be seen in the performance achieved by Thales' Live Face Identification System (LFIS), an advanced solution resulting from our long-standing expertise in biometrics.

Owners of the iPhone X have already been introduced to facial recognition technology. However, the Face ID biometric solution developed by Apple was heavily criticized in China in late 2017 because of its inability to differentiate between individual Chinese faces.

Of course, other signatures via the human body also exist, such as fingerprints, iris scans, voice recognition, digitization of veins in the palm, and behavioural measurements.

Why facial recognition, then?

Facial biometrics continues to be the preferred biometric benchmark.

That's because it's easy to deploy and implement. There is no physical interaction required by the end-user.

Moreover, face detection and face match processes for verification/identification are speedy.

Best face recognition software

So, what is the best face recognition software?

#1 Top facial recognition technologies
In the race for biometric innovation, several projects are vying for the top spot.

Google, Apple, Facebook, Amazon, and Microsoft (GAFAM) are also very much in the mix.

All the software web giants now regularly publish their theoretical discoveries in the fields of artificial intelligence, image recognition, and face analysis in an attempt to further our understanding as rapidly as possible.

There's more.

The very latest results of tests conducted in March 2018 and published in May by the US Homeland Security Science and Technology Directorate, known as the Biometric Technology Rally, also provide an excellent indication of the best face recognition software available on the market.

But let’s take a closer look :

Academia
The GaussianFace algorithm developed in 2014 by researchers at The Chinese University of Hong Kong achieved facial identification scores of 98.52% compared with the 97.53% achieved by humans. An excellent rating, despite weaknesses regarding memory capacity required and calculation times.

Facebook and Google
Again in 2014, Facebook announced the launch of its DeepFace program, which can determine whether two photographed faces belong to the same person, with an accuracy rate of 97.25%. When taking the same test, humans answer correctly in 97.53% of cases, or just 0.28% better than the Facebook program.

In June 2015, Google went one better with FaceNet. On the widely used Labeled Faces in the Wild (LFW) dataset, FaceNet achieved a new record accuracy of 99.63% (0.9963 ± 0.0009).
Using an artificial neural network and a new algorithm, the company from Mountain View has managed to link a face to its owner with almost perfect results.

This technology is incorporated into Google Photos and used to sort pictures and automatically tag them based on the people recognized. Proving its importance in the biometrics landscape, it was quickly followed by the online release of an unofficial open-source version known as OpenFace.

Microsoft, IBM, and Megvii
A study done by MIT researchers in February 2018 found that Microsoft, IBM, and China-based Megvii (FACE++) tools had high error rates when identifying darker-skin women compared to lighter-skin men.

At the end of June 2018, Microsoft announced in a blog post that it had made substantial improvements to its biased facial recognition technology.

Amazon
In May 2018, Ars Technica reported that Amazon is already actively promoting its cloud-based face recognition service named Rekognition to law enforcement agencies. The solution could recognize as many as 100 people in a single image and can perform face match against databases containing tens of millions of faces.

In July, Newsweek reported that Amazon’s facial recognition technology falsely identified 28 members of US Congress as people arrested for crimes.

Key biometric matching technology providers
At the end of May 2018, the US Homeland Security Science and Technology Directorate published the results of sponsored tests at the Maryland Test Facility (MdTF) done in March. These real-life tests measured the performance of 12 face recognition systems in a corridor measuring 2 m by 2.5 m.

Thales' solution utilizing a Facial recognition software (LFIS) achieved excellent results with a face acquisition rate of 99.44% in less than 5 seconds (against an average of 68%), a Vendor True Identification Rate of 98% in less than 5 seconds compared with an average 66%, and an error rate of 1% compared with an average 32%.

Face tracking

March 2018 – The live testing done using more than 300 volunteers identified the best-performing facial recognition technologies.

More on performance benchmarks: The NIST (National Institute of Standards and Technology) report, published in November 2018, details recognition accuracy for 127 algorithms and associates performance with participant names.

The NIST Ongoing Face Recognition Vendor Test (FRVT) 3 performed at the end of 2019 provides additional results. See NIST report.

NIST also demonstrated that the best facial recognition algorithms have no racial nor sex bias, as reported in January 2020 by ITIF. Critics were wrong.

Mid-June 2020, IBM said it will no longer offer facial recognition technology and stop its research and development activities, and Microsoft pulled its face recognition solutions from law enforcement agencies in the United States.
In a blog post published on 10 June, Amazon is putting a moratorium of one year on the use of its technology by police. The e-commerce giant said it’s giving time for federal laws to be initiated and protect human rights and civil liberties in this domain.
Facial emotion detection and recognition
Emotion recognition (from real-time of static images) is the process of mapping facial expressions to identify emotions such as disgust, joy, anger, surprise, fear, or sadness on a human face with image processing software.

Its popularity comes from the vast areas of potential applications.

It's different from facial recognition which goal is to identify a person, not an emotion.

Face expression may be represented by geometric or appearance features, parameters extracted from transformed images such as eigenfaces, dynamic models, and 3D models.

Providers include Kairos (face and emotion recognition for brand marketing), Noldus, Affectiva, Sightcorp, Nviso, among others.

#2 Learning to learn through deep learning
The feature common to all these disruptive technologies is known as Artificial Intelligence (AI) and, more precisely, deep learning where a system is capable of learning from data.

Why is it important?

It's a central component of the latest-generation algorithms developed by Thales and other key players in the market. It holds the secret to face detection, face tracking, and face match as well as real-time translation of conversations.

The result?

Face recognition systems are getting better all the time.

According to a recent NIST report, massive gains in accuracy have been made in the last five years (2013- 2018) and exceed improvements achieved in the 2010-2013 period.

Most of the face recognition algorithms in 2018 outperform the most accurate algorithm from late 2013.

In its 2018 test, NIST found that 0.2% of searches, in a database of 26.6 million photos, failed to match the correct image, compared with a 4% failure rate in 2014.

Yes, you read that right.

It's a 20x improvement over four years.

Think about it this way:

Artificial neural network algorithms are helping face recognition algorithms to be more accurate.

#3 Facial recognition markets
Face recognition markets
A study published in June 2019, estimates that by 2024, the global facial recognition market would generate $7 billion of revenue, supported by a compound annual growth rate (CAGR) of 16% over the period 2019-2024.

For 2019, the market is estimated at $3.2 billion.

The two most significant drivers of this growth are surveillance in the public sector and numerous other applications in diverse market segments.

According to the study, the top facial recognition vendors include :

Accenture, Aware, BioID, Certibio, Fujitsu, Fulcrum Biometrics, Thales, HYPR, Idemia, Leidos, M2SYS, NEC, Nuance, Phonexia, and Smilepass.

The main facial recognition applications can be grouped into three principal categories.

What is facial recognition used for?
Here are the top three application categories where facial recognition is being used.

1. Security - law enforcement
This market is led by increased activity to combat crime and terrorism.

The benefits of facial recognition systems for policing are evident: detection and prevention of crime.

Facial recognition is used when issuing identity documents and, most often combined with other biometric technologies such as fingerprints (prevention of ID fraud and identity theft).

Face match is used at border checks to compare the portrait on a digitized biometric passport with the holder's face. In 2017, Thales was responsible for supplying the new automated control gates for the PARAFE system (Automated Fast Track Crossing at External Borders) at Roissy Charles de Gaulle airport in Paris. This solution has been devised to facilitate evolution from fingerprint recognition to facial recognition during 2018.
Face biometrics can also be employed in police checks, although its use is rigorously controlled in Europe. In 2016, the "man in the hat" responsible for the Brussels terror attacks was identified thanks to FBI facial recognition software. The South Wales Police implemented it at the UEFA Champions League Final in 2017.
In the United States, 26 states (and probably as many as 30) allow law enforcement to run searches against their databases of driver’s license and ID photos. The FBI has access to driver’s license photos of 18 states.
​Drones combined with aerial cameras offer an interesting combination for facial recognition applied to large areas during mass events, for example. According to the Keesing Journal of Documents and Identity of June 2018, some hovering drone systems can carry a 10-kilo camera lens that can identify a suspect from 800 meters from a height of 100 meters. As the drone can be connected to the ground via a power cable, it has an unlimited power supply. The communication to ground control can’t be intercepted as it also uses a cable.
Facial recognition CCTV systems can improve performance in carrying public security missions. Let's illustrate this with four examples:
Find missing children and disoriented adults
Identify and find exploited children
Identify and track criminals
Support and accelerate investigations
facial recognition cctv

1. Find Missing children and disoriented adults.
Face recognition CCTV systems can significantly accelerate operators’ efforts by enabling them to add a reference photo provided by the missing child’s parents and match it with past appearances of that face captured on video. Police can use face recognition to search video sequences (aka video analytics) of the estimated location and time the child has been declared missing.

Police officers can better figure out the child’s movements before going missing and locate where he/she was last seen. A real-time alert can trigger an alarm whenever there's a match. Police can then confirm its accuracy and do what's necessary to recover the missing children. The same process can be applied for disoriented missing adults (e.g. with dementia, amnesia, epilepsy, or Alzheimer’s disease).

2. Identify and find exploited children.
Isolating the appearances of specific individuals in a video sequence is critical. It can accelerate investigators’ jobs in child exploitation cases as well.

Video analytics can help build chronologies, track activity on a map, reveal details and discover non-obvious connections among the players in a case.

3. Identify and track criminals.
Face recognition CCTV can be used to enable police to track and identify past criminals suspected of perpetrating an additional infraction. Police can also take preventive actions. By using an image of a known criminal from a video or an external picture (or a database), operators can use to detect matches in live video and react before it’s too late.

4. Support and accelerate investigations.
Facial recognition CCTV systems can be used to support investigators searching for video evidence in the aftermath of an incident.

The ability to isolate the appearances of suspects and individuals is critical for accelerating investigators’ review of video evidence for relevant details. They can better understand how situations developed.

2. Health
Significant advances have been made in this area.

Thanks to deep learning and face analysis, it is already possible to:

track a patient's use of medication more accurately
detect genetic diseases such as DiGeorge syndrome with a success rate of 96.6%
support pain management procedures.
face analysis for health

3. Marketing and retail
This area is undoubtedly the one where the use of facial recognition was least expected. And yet quite possibly it promises the most. Know Your Customer (KYC) is sure to be a hot topic in 2020. This important trend is being combined with the latest marketing advances in customer experience.

By placing cameras in retail outlets, it is now possible to analyze the behavior of shoppers and improve the customer purchase process.

How exactly?

Like the system recently designed by Facebook, sales staff are provided with customer information taken from their social media profiles to produce expertly customized responses.

The American department store Saks Fifth Avenue is already using such a system. Amazon GO stores are reportedly using it.

How long before the selfie payment?

Since 2017, KFC, the American king of fried chicken, and Chinese retail and tech giant Alibaba have been testing a face recognition payment solution in Hangzhou, China.

#4 Mapping of new users
While the United States currently offers the largest market for face recognition opportunities, the Asia-Pacific region is seeing the fastest growth in the sector. China and India lead the field.

Face recognition in China
Face recognition technology is the new hot topic in China, from banks and airports to police.

Now authorities are expanding the facial recognition sunglasses program as police are beginning to use them in the outskirts of Beijing.

China is also setting up and perfecting a video surveillance network countrywide.

Over 200 million surveillance cameras were in use at the end of 2018, and 626 million are expected by 2020. The facial recognition towers in Chinese cities are emblematic of this move.

This is linked to the social credit system the Chinese government is developing.

In the TOP 10 cities with most street cameras per person, Chongqing, Shenzhen, Shanghai, Tianjin, and Ji’nan are leading the pack.

London is #6 and Atlanta #10, according to the Guardian of 2 December 2019.

There's more.

Chinese police are working with artificial intelligence companies such as Yitu, Megvii, SenseTime, and CloudWalk, according to The New York Times of 14 April 2019.

China's ambitions in AI (and facial recognition technology) are high. The country aims to become a world leader in AI by 2030.

Surprisingly, China provides strong biometric data protection against private entities AND increases government's access to personal information. This paradox is evidenced by privacy expert Emmanuel Pernot- Leplay in his report dated 27 March 2020.

Facial recognition in Asia

Facial recognition will be a significant topic for the 2020 Olympic Games in Tokyo (postponed to September 2021). This technology will be used to identify authorized persons and grant them access automatically, enhancing their experience and safety.

In Sydney, face recognition is undergoing trials at airports to help move people through security much faster and in a safer way.

In India, the Aadhaar project is the largest biometric database in the world. It already provides a unique digital identity number to 1.26 billion residents as of August 2020. UIDAI, the authority in charge, announced that facial authentication would be launched in a phased roll-out by September 2018. Face authentication will be available as an add-on service in fusion mode along with one more authentication factor like fingerprint, Iris, or OTP.

India could also roll-out the world's most extensive face recognition system in 2020.

The National Crime Records Bureau (NCRB) has issued an RFP inviting bids to develop a nationwide facial recognition system. According to the 160-page document, the system will be a centralized web application hosted at the NCRB Data Center in Delhi. It will be available for access to all the police stations.

It will automatically identify people from CCTV videos and images. The Bureau states that it will help police catch criminals, find missing people, and identify dead bodies.

Other large projects
In Brazil, the Superior Electoral Court (Tribunal Superior Eleitoral) is involved in a nationwide biometric data collection project. The aim is to create a biometric database and unique ID cards by 2020, recording the information of 140 million citizens.

In Africa, Gabon, Cameroon, and Burkina Faso have chosen Thales to meet the challenges of biometric identity to uniquely identify voters in particular.

Russia's Central Bank has been deploying a countrywide program since 2017 designed to collect faces, voices, iris scans, and fingerprints. But the process is progressing very slowly according to the Biometricupdate website of 13 March 2019.

The city of Moscow claims one of the world’s largest network of 160,000 surveillance cameras by the end of 2019 and are to be fitted with facial recognition technology for public safety.

The roll-out started in January 2020.

Russian law does not regulate non-consensual face detection and analysis.

Biometric information

#5 When face recognition strengthens the legal system
The ethical and societal challenge posed by data protection is radically affected by the use of facial recognition technologies.

Do these technological feats, worthy of science-fiction novels, genuinely threaten our freedom?

And with it, our anonymity?

EU and UK biometric data protection
In Europe and the UK, the General Data Protection Regulation (GDPR) provides a rigorous framework for these practices. Any investigations into a citizen's private life or business travel habits are out of the question, and any such invasions of privacy carry severe penalties.

Applicable from May 2018, the GDPR supports the principle of a harmonized European framework, in particular protecting the right to be forgotten and the giving of consent through clear affirmative action. This directive is bound to have international repercussions.

Yes, you read it well. There's now one law for 500 million people.

US biometric data protection landscape
In America, the State of Washington was the third US state (after Illinois and Texas) to formally protect biometric data through a new law introduced in June 2017.

California was the fourth state as of January 2020.

The California Consumer Privacy Act (CCPA) passed in June 2018 and effective as of 1 January 2020 will have a serious impact for privacy rights and consumer protection not only for residents of California but for the whole nation as the law is frequently presented as a model for a federal data privacy law.

In that sense, the CCPA has the potential to become as consequential as the GDPR.

In July 2018, Bradford L. Smith, Microsoft’s president, compared the face recognition technology to products like medicines that are highly regulated, and he urged Congress to study it and oversee its use.

In May 2019, US Rep. Alexandria Ocasio-Cortez voiced her "absolute" concerns in a recent Committee Hearing on facial recognition Technology (Impact on our Civil Rights and Liberties).

More recently, a New York State law called the Stop Hacks and Improve Electronic Data Security (SHIELD) became effective 21 March 2020. It requires the implementation of a cybersecurity program and protective measure fro NY State residents.

The act applies to businesses that collect the personal information of NY residents.

With the act, New York now stands beside California.

Facial recognition bans (San Francisco, Somerville, Oakland, San Diego, Boston...)
Privacy and civil rights concerns have escalated in the country as face recognition gains traction as a law enforcement tool and, on 6 May 2019, San Francisco voted to ban facial recognition.

It is the first ban of its kind on the use of face recognition.

The anti-surveillance ordinance signed by San Francisco's Board of Supervisors bars city agencies, including San Francisco PD, from using the technology as of June 2019.

Yes, this includes law enforcement.

There's more.

As reported by the Boston Globe of 27 June 2019, the Somerville City Council (Massachusetts) voted to ban the use of facial recognition, making the city the second community to take such a decision.

Lather, rinse, repeat.

On 16 July 2019, Oakland (California) took the same decision and became the third US city to ban the use of face recognition technology. It is interesting to note that the Oakland Police department is not using this technology and was not planning to use it.

San Diego took the same decision at the end of December 2019 in advance of the new Californian law.

This new law (Assembly Bill 215) about facial recognition and other biometric surveillance) specifically prohibits the use of police body cameras in California. The ban is in place for three years as of 1 January 2020.

Since the San Francisco, Sommerville, Oakland, and now San Diego rulings, the debate gets louder in many cities and not only in the U.S.

Portland (Oregon) is considering a ban for 2020. Early January, the vote has been put on hold until June, however. Portland could be the first city to extend it to private stores, airlines, and event venues.

On 24 June 2020, Boston voted to ban the use of face surveillance technology by police as reported by Boston Herald.

In Europe, at the end of August 2019, Sweden's Data Protection Authority decided to ban facial recognition technology in schools and fined a local high school (the first GDPR penalty in the country).

How to better regulate emerging technologies?
So...

Should other cities or countries follow this example?
Is the ban just a "pause button" to better assess risks?
Is this a step backwards for public safety?
Is there a policy vacuum? At which level?
Stay tuned for the outcome of all these discussions as the US Congress is getting pressure from activists to ban the technology and from providers (see box below) to regulate.

The EU Commission is planning to act on indiscriminate use of facial identifier technology. The new European Commission president Ursula von der Leyen wants a coordinated approach to the human and ethical implications of artificial intelligence. She has pledged to publish an AI legislation blueprint very soon. The very first draft of the European commission whitepaper is available online.

The document mentions “a time-limited ban on the use of facial recognition by private or public actors in public spaces.”

Again the questions of privacy, consent, and function creep (data collected for one purpose being used for another) are central to the debate.

Find more on biometric data protection laws (EU, UK and US perspective) in our biometric data dossier.

India and its national biometric identification scheme, Aadhaar
In India, thanks to the Puttaswamy judgment delivered on 27 August 2017, the Supreme Court has enshrined the right to privacy in the country's constitution. This decision has rebalanced the relationship between citizen and state and posed a new challenge to the expansion of the Aadhaar project.

The Indian government, however, approved the use of the country's biometric EID program by private entities on 28 February 2019.

Rebound effect: the legal system and its professions get even stronger.

As both ambassadors and guardians of data protection regulation, the post of data protection officer has become necessary for businesses and a much sought-after role.

can face recognition be fooled

#6 The rebels – facial recognition hackers
Despite this technical and legal arsenal designed to protect data, citizens, and their anonymity, critical voices have still been raised.

Some parties are concerned and alarmed by these developments. Some have taken actions.

But can facial recognition be fooled?

In Russia, Grigory Bakunov has invented a solution to escape the eyes permanently watching our movements and confuse face detection devices. He has developed an algorithm that creates special makeup to fool the software. However, he has chosen not to bring his product to market after realizing how easily criminals could use it.
In Germany, Berlin artist Adam Harvey has come up with a similar device known as CV Dazzle. He is now working on clothing featuring patterns to prevent detection. The hyperface camouflage includes patterns in fabric, such as eyes and mouths, to fool the face recognition system.
In late 2017, a Vietnamese company successfully used a mask to hack the Face ID face recognition function of Apple's iPhone X. However, the hack is too complicated to implement for large-scale exploitation.
Around the same time, researchers from a German company revealed a hack that allowed them to bypass the facial authentication of Windows 10 Hello by printing a facial image in infrared.
​Forbes announced in an article from May 2018 that researchers from the University of Toronto have developed an algorithm to disrupt facial recognition software (aka privacy filter).
In August 2020, the Verge detailed a "cloaking" app named Fawkes. The software imperceptibly distorts your selfies and other pics you may leave on social media. The tool is coming from the University of Chicago’s Sand Lab.
In short, a user could apply a filter that modifies​ specific pixels in an image before putting it on the web. These changes are imperceptible to the human eye but are very confusing for facial recognition algorithms.

The industry is working on anti-spoofing mechanisms, and two topics have been specifically identified by standardization groups :

Make sure the captured image has been done from a person and not from a photograph (2D), a video screen (2D) or a mask (3D), (liveness check or liveness detection)
Make sure that facial images (morphed portraits) of two or more individuals have not been joined into a reference document, such as a passport.
#7 Further together – towards hybridized solutions
The identification and authentication solutions of the future will borrow from all aspects of biometrics.

This will lead to "biometrix" or a biometric mix capable of guaranteeing total security and privacy for all stakeholders in the ecosystem.

It's very much the spirit of Thales Gemalto IdCloud Fraud Prevention, a risk assessment, and fraud detection software for payments.

In this solution, geolocation, IP-addresses (the device being used) and keying patterns can create a strong combination to authenticate users for on-line banking or egovernment services securely.

This seventh trend belongs to us.

It's our job to envisage it together and make it happen through high-added-value biometric projects.

Face recognition and you
Now it's your turn.

The months to come hold many changes in store. Indeed, we can't claim to predict all the essential topics that will emerge in the short term future.

Can you fill in some of the gaps?

If you've something to say on face recognition, tech or trends, a question to ask, or have simply found this article useful, please leave a comment in the box below.

We'd also welcome any suggestions on how it could be improved or proposals for future articles.

We look forward to hearing from you.

關於中文翻譯，可參考3S Market https://3smarket-info.blogspot.com/2020/09/blog-post_73.html