Login

Benchmark area: Face Morphing Challenge

This benchmark area is aimed at analyzing the effects of image morphing on face recognition accuracy. The robustness against morphing alterations is evaluated by comparing morphed images against other images of the subjects used for the morphing. Algorithms submitted to these benchmarks are required to compare face images to determine whether they belong to the same subject or not (one-to-one comparisons).

In scenarios where the user template is created from printed photos rather than from images acquired live during enrollment (e.g., identity documents), digital image alterations could severely affect the recognition results of a face recognition system. In particular, with the widespread adoption of Automated Border Control systems (ABC) [1] [2], image morphing alteration (obtained by digitally mixing face images of two subjects) can cause an increment of the false acceptance rate and consequently of the possibility that a criminal succeeds to bypass border controls [3] [4].

Benchmarks

Currently, this benchmark area contains the following benchmarks:

  • FMC-TEST: A simple dataset useful to test algorithm compliancy with the testing protocol (results obtained on this benchmark are only visible in the participant private area and cannot be published).
  • FMC-1.0: A dataset containing high-resolution face images with neutral expressions and good illumination. Each morphed image is compared against both subjects used for morphing.
  • FMC-CRM-1.0: A dataset containing high-resolution face images with neutral expressions and good illumination. Each morphed image is compared against the less similar subject used for morphing (i.e., the criminal [4]).

The table below reports the main characteristics of each benchmark:

Benchmark Minimum Image Size Maximum Image Size Genuine Attempts Impostor Attempts Morphing Attempts
FMC-TEST 399x517 517x768 50 100 10
FMC-1.0 399x496 496x768 526 19944 160
FMC-CRM-1.0 399x496 496x768 526 19944 80

The following sections report the testing protocol and the performance indicators common to all benchmarks in this area.

Protocol

Each participant is required to submit, for each algorithm, an executable in the form of Win32 console application.

  • The executable (match.exe) will take the input from command-line arguments and will append the output to a text file. It compares two face images and produces a similarity score; the command-line syntax is:
    match.exe <faceimagefile1> <faceimagefile2> <outputfile>
    where:
    faceimagefile1 the first input  face image pathname. The supported image formats are: .BMP, .JPG, .PNG.
    faceimagefile2 the second input face image pathname. The supported image formats are: .BMP, .JPG, .PNG.
    outputfile the output text-file, where a log string (of the form faceimagefile1 faceimagefile2 result similarity) must be appended; result is "OK" if the comparison can be performed or "FAIL" if the comparison cannot be executed by the algorithm; similarity is a floating point value ranging from 0 to 1 which indicates the similarity between the two images: 0 means no similarity, 1 maximum similarity

  • The executable has to operate only on the explicitly-given inputs, without exploiting any learning technique or template consolidation/update based on previous comparisons.
  • C, C# and Matlab language skeletons for match.exe are available in the download page to reduce the participants implementation efforts.

Constraints

During test execution the following constraints will be enforced:

Benchmark Maximum time for each match Memory allocation limit for match process
FMC-TEST 10 seconds No limit
FMC-1.0 5 seconds No limit
FMC-CRM-1.0 5 seconds No limit

Each matching attempt that violates one of the above constraints results in a failure to match.

The following time breaks are enforced between two consecutive submissions to the same benchmark by the same participant.

Benchmark Minimum break
FMC-TEST 3 hour(s)
FMC-1.0 3 day(s)
FMC-CRM-1.0 3 day(s)

Performance Evaluation

For each algorithm, genuine (matching two face images of the same subject) and impostor (matching two face images of different subjects) attempts are performed to compute False Non Match Rate FNMR (also referred as False Rejection Rate - FRR) and False Match Rate FMR (also referred as False Acceptance Rate - FAR).
Moreover, for each algorithm, morph attempts (comparing a morphed image against an image of one of the subjects used for morphing) are performed to compute Morph Acceptance Rate MAR.

For each algorithm the following performance indicators are reported:

  • REJNGRA (Number of rejected faces during genuine matches)
  • REJNIRA (Number of rejected faces during impostor matches)
  • REJNMRA (Number of rejected faces during morph matches)
  • EER (equal-error-rate)
  • FMR100 (the lowest FNMR for FMR1%)
  • FMR1000 (the lowest FNMR for FMR0.1%)
  • FMR10000 (the lowest FNMR for FMR0.01%)
  • ZeroFMR (the lowest FNMR for FMR=0%)
  • ZeroFNMR (the lowest FMR for FNMR=0%)
  • Average matching time
  • Maximum amount of memory allocated
  • Impostor and Genuine score distributions
  • FMR(t)/FNMR(t) curves, where t is the acceptance threshold
  • DET(t) curve
  • MAR@FMR100 (the lowest MAR for FMR1%)
  • MAR@FMR1000 (the lowest MAR for FMR0.1%)
  • MAR@ZeroFMR (the lowest MAR for FMR=0%)
  • Graph of the trade-off between Morph Acceptance Rate (MAR) and False Match Rate (FMR)

Note that, according to the guidelines [1] provided by FRONTEX (the European Agency for the Management of Operational Cooperation at the External Borders of the Member States of the European Union) [5], ABC systems operating in verification mode have to ensure a security level in terms of the False Match Rate (FMR) of at least 0.1% with a False Non Match Rate (FNMR) of at most 5%. Then, simulating the operating conditions of a real ABC system, the percentage of morphed images that successfully would pass through the border control is denoted by MAR@FMR1000.

Terms and Conditions

All publications and works that cite FMC Benchmark Area must reference [4].

Bibliography

[1] FRONTEX - Research and Development Unit, "Best Practice Technical Guidelines for Automated Border Control (ABC) Systems" - v2.0, 2012.
[2] IATA. Airport with Automated Border Control Systems. Online.
[3] M. Ferrara, A. Franco and D. Maltoni, "The Magic Passport", in proceedings of the International Joint Conference on Biometrics (IJCB), Clearwater, Florida, USA, pp.1-7, October 2014.
[4] M. Ferrara, A. Franco and D. Maltoni, "On the Effects of Image Alterations on Face Recognition Accuracy", in Thirimachos Bourlai, Face Recognition Across the Electromagnetic Spectrum, Springer, 2016.
[5] FRONTEX. FRONTEX Web Site. Online.


For information or suggestions: fvcongoing@csr.unibo.it Copyright © 2017 Biometric System Laboratory