NIST’s FRVT Isn’t Up to Task of Evaluating Video Surveillance Algos, says Oosto

While virtually every biometrics vendor now agrees on the importance of standards-based, third party testing, Oosto is seeking to clarify some of the shortcomings of one of the most respected testing programs in the business.

In a new blog post, Oosto’s Chief Marketing Officer, Dean Nicolls, respectfully lays out the five major shortcomings of the National Institute of Standards and Technology’s Face Recognition Vendor Test (NIST FRVT).

As Nicolls explains, the testing program is designed to measure the effectiveness of facial recognition algorithms with an eye toward government-focused application areas such as border control and law enforcement. And Nicolls readily acknowledges that “FRVT serves its purpose,” providing assessments relevant for the use cases concerned. The problem, from Oosto’s perspective, is that the NIST’s FRVT testing isn’t adequate for assessing and comparing facial recognition solutions based on video surveillance, for five key reasons:

1. FRVT doesn’t evaluate solutions using real-time videos. That means that it can’t adequately assess a solution like Oosto’s OnWatch, which uses AI and neural network technology that is “fundamentally different” from the kind of tech used in the basic 1:1 and 1:n matching algorithms that FRVT is meant to evaluate.
2. FRVT’s “in the wild” category for evaluating partially obscured faces doesn’t come close to the difficulty of conditions at which Oosto’s technology is adept at navigating. “NIST’s wild images may be blurry, or of low quality, but the images are still relatively clear, the camera is generally at face level, and the lighting is good,” explains Nicolls. “In the real world, perhaps within a casino or stadium, subjects are far less cooperative and do not look directly at the camera.”
3. FRVT was never designed to evaluate the performance of a facial recognition system operating against a crowd. Systems that perform well in the NIST’s testing program may end up with a number of false positives and false negatives when applied to surveillance feeds scanning large numbers of subjects at a time.
4. FRVT makes assumptions about the quality of cameras being used for face capture that may not be relevant for organizations deploying “smart facial recognition cameras” that can deliver dramatically improved performances even in challenging circumstances.
5. FRVT mounts cameras at face-level, artificially boosting the performance of algorithms that would otherwise struggle when processing images from an overhead surveillance camera. This is fine for highly controlled scenarios such as border checkpoints, but not for general video surveillance across a range of contexts.

To be clear, none of this is meant as a criticism of the NIST or even of its FRVT program in and of itself. Nicolls’ aim is rather to help to better inform organizations that are considering the use of face-based surveillance about what FRVT performance really means, and what it doesn’t mean.

That having been said, Oosto’s post concludes by calling on the NIST to create a new testing methodology that can compare video surveillance-based facial recognition solutions. It would be a welcome complement to the renowned testing programs that the NIST already has in place.

With its constructive criticism of the biometrics industry’s testing landscape, Oosto joins other face biometrics specialists in a wider call for evaluation programs that take into account new and recently evolved technologies. Earlier this year, FaceTec – a 3D face biometrics company that specializes in ID verification and user authentication – announced a contractual relationship with A.C.C.S. Ltd. intended to bring forward the world’s first 3D face matching assessment.

–

July 11, 2022 – by The FindBiometrics Editorial Team