Wow! It turns out that most facial recognition actually is biassed.
𝗘𝘅𝗰𝗲𝗽𝘁 𝗳𝗼𝗿 𝗣𝗮𝗿𝗮𝘃𝗶𝘀𝗶𝗼𝗻.
It’s often said that, when it comes to #facialrecognition matching, as long as you're using a top 20-ranked NIST-tested vendor, it doesn’t really matter which: they’re all so close.
… and at first glance that may appear to be true.
But things often are not as they appear at first glance.
The National Institute of Standards and Technology (NIST) rankings are based on False Non-Match Rates (FNMR: the error rate at which two photos of the same person do not match) at a pegged False Match Rate (FMR: the error rate at which two photos of two different people do match).
(Think of somebody trying to get through an eGate at an airport using their own passport, and the photo doesn’t match and they get referred to secondary.)
… and indeed the top 20 vendors do all have close FNMRs.
𝗘𝘅𝗰𝗲𝗽𝘁 𝗵𝗲𝗿𝗲’𝘀 𝘁𝗵𝗲 𝘁𝗵𝗶𝗻𝗴:
In order to assess bias, #NIST compares vendors’ FMRs (not FNMRs) across different demographic groups, or the rate at which a photo of somebody within a demographic group falsely matches with a photo of a different person within that same group.
(Think of somebody successfully getting through an eGate at an airport using somebody else’s passport, or falsely being identified using facial recognition on a surveillance camera.)
NIST doesn't present a topline FMR ranking, and this level of detail is just not apparent when looking at the FNMR leaderboard.
When analysing FMR across biographic groups, the differences are startling:
🚀Almost every other vendor except Paravision struggles with all areas of origin outside of Eastern Europe.
🚀Paravision shows exceptional and consistent accuracy across all groups.
🚀Paravision performs better than every other vendor for people of Central American, Western African, Caribbean, Eastern African, South Asian, and East Asian origin, and on par with leading vendors for people of Eastern European origin.
NIST only undertakes demographic analysis on their 1:1 tests, but these issues are only compounded on 1:N uses of facial recognition.
And if a vendor does not submit to NIST’s 1:1 testing, there is no independent benchmark analysis of #bias and demographic inclusion for that vendor.
So, you may think that facial recognition matching is commoditised and there is no real difference between the top vendors.
That may be true, unless of course, you are concerned with:
✅Human and Civil rights.
✅Maximising demographic inclusion.
✅Minimising legal and reputational risk.
✅Remaining compliant with emerging regulations.
✅Increasing operational efficiency across all demographic groups.
✅Minimising security risks.
In which case, the choice is clear.
𝗗𝗼𝘄𝗻𝗹𝗼𝗮𝗱 𝘁𝗵𝗲 𝘄𝗵𝗶𝘁𝗲𝗽𝗮𝗽𝗲𝗿 𝗶𝗻 𝘁𝗵𝗲 𝗹𝗶𝗻𝗸 𝗯𝗲𝗹𝗼𝘄 𝗳𝗼𝗿 𝗳𝘂𝗿𝘁𝗵𝗲𝗿 𝗱𝗲𝘁𝗮𝗶𝗹.
Get in touch if you’d like to discuss running Paravision in Challenger Mode with your existing technology to compare the results.
New demographic insights from NIST FRTE 1:1 👇
We recently highlighted the importance of inclusion in face recognition technology, focusing on the critical metric of FMR Max. We're excited to share new insights and a more comprehensive white paper that delves deeper into this topic.
The graph below shows leading vendors’ error rates for each reported area of origin, pulled from their individual National Institute of Standards and Technology (NIST) FRTE 1:1 Report Cards. While most vendors struggle with all origins but Eastern Europe, Paravision singularly shows exceptional accuracy across all groups. Here are some additional highlights:
🥇 Best Performance Across Groups: Paravision performs better than every other vendor for people of Central American, Western African, Caribbean, Eastern African, South Asian, and East Asian origin, and on par with leading vendors for people of Eastern European origin.
📊 Consistency of Accuracy: The heat map in the comments shows the geographic-specific error rates of each leading vendor in NIST FRTE 1:1. Blue color means better, showing Paravision’s consistency across all groups.
Financial institutions, government services, and other sensitive sectors face significant risks if they fail to address poor demographic performance. Our updated white paper not only analyzes the NIST FRTE 1:1 results but also provides a full guide on how to achieve true inclusion in face recognition technology. Read the full guide via the link in comments: 🔗
-
Read the full guide here: https://www.paravision.ai/resource/inclusion-in-face-recognition/