Benchmark area: Single-image Morph Attack Detection

This benchmark area contains face morphing detection benchmarks. Morphing detection consists in analyzing a face image to determine whether it is the result of a morphing process (mixing faces of two subjects) or not. Algorithms submitted to these benchmarks are required to analyze a suspected morph image and produce a score representing the probability of the image to be morphed.

Benchmarks

Currently, this benchmark area contains the following benchmarks:

• SMAD-TEST: A simple dataset useful to test algorithm compliancy with the testing protocol (results obtained on this benchmark are only visible in the participant private area and cannot be published).
• SMAD-BIOLAB-1.0: A dataset containing high-resolution face images with frontal pose, neutral expressions and good illumination. The morphed images have been generated from manually selected landmarks, and finally manually retouched. The dataset contains the morphed images used in [1] and the genuine images used to generate them (some images contribute to different morphings).
• SMAD-MORPHDB_D-1.0: A dataset of high-quality images, with frontal pose, natural expression and good illumination. The morphed images have been generated from automatically detected landmarks, and finally manually retouched. The dataset is described in [2].
• SMAD-MORPHDB_P&S-1.0: The dataset contains the same images of SMAD-MORPHDB_D-1.0, printed on high quality photographic paper by a professional photographer and scanned at 300 DPI. The dataset is described in [2].
• SMAD-SOTAMD_D-1.0: A dataset containing high-resolution digital face images with frontal pose, natural expression and good illumination collected in the SOTAMD project [3].
• SMAD-SOTAMD_P&S-1.0: A dataset containing high-resolution printed and scanned face images with frontal pose, natural expression and good illumination collected in the SOTAMD project [3].
• SMAD-SOTAMD_PM_D-1.0: A dataset containing a subset of the high-resolution digital face images with frontal pose, natural expression and good illumination collected in the SOTAMD project [3]. In particular, it contains only manual post-processed morphed images.
• SMAD-SOTAMD_UC_P&S-1.0: A dataset containing a subset of the high-resolution printed and scanned face images with frontal pose, natural expression and good illumination collected in the SOTAMD project [3]. In particular, it contains only uncompressed images.
• SMAD-IMARS-HQ_FULL-1.0: A dataset created in the iMARS project [5] containing high-resolution digital face images with frontal pose, natural expression and good illumination. The bona fide images have been collected in the SOTAMD project [3] while the morphed images have been generated using 16 different state-of-the-art morphing algorithms and multiple morphing factors.
• SMAD-IMARS-HQ_SMALL-1.0: A dataset containing a subset of the SMAD-IMARS-HQ_FULL-1.0 benchmark. The bona fide images are the same contained in the SMAD-IMARS-HQ_FULL-1.0 benchmark while the morphed images are the 20% of the morphed images in the SMAD-IMARS-HQ_FULL-1.0 benchmark, randomly selected.
• SMAD-IMARS-HQ_HARD-1.0: A dataset containing a subset of the SMAD-IMARS-HQ_FULL-1.0 benchmark. The bona fide images are the same contained in the SMAD-IMARS-HQ_FULL-1.0 benchmark while the morphed images are the subset of images presenting the highest attack potential, i.e. morphed images successfully verified by three commercial face recognition systems with ten probe images of both contributing subjects.
• SMAD-IMARS-MQ_FULL-1.0: A dataset containing high-resolution digital face images with frontal pose, natural expression and good illumination collected in the iMARS project [5]. The morphed images have been generated using 12 different state-of-the-art morphing algorithms and multiple morphing factors.
• SMAD-IMARS-MQ_SMALL-1.0: A dataset containing a subset of the SMAD-IMARS-MQ_FULL-1.0 benchmark. The bona fide images are the same contained in the SMAD-IMARS-MQ_FULL-1.0 benchmark while the morphed images are the 20% of the morphed images in the SMAD-IMARS-MQ_FULL-1.0 benchmark, randomly selected.
• SMAD-IMARS-MQ_HARD-1.0: A dataset containing a subset of the SMAD-IMARS-MQ_FULL-1.0 benchmark. The bona fide images are the same contained in the SMAD-IMARS-MQ_FULL-1.0 benchmark while the morphed images are the subset of images presenting the highest attack potential, i.e. morphed images successfully verified by three commercial face recognition systems with four probe images of both contributing subjects.

The table below reports the main characteristics of each benchmark:

The following sections report the testing protocol and the performance indicators common to all benchmarks in this area.

Protocol

Participants can submit their algorithms in one of the following three different forms:

• an executable in the form of a Win32 console application;
• a Python script;
• a Linux dynamically-linked library compliant to NIST FRVT MORPH specifications.

    Win32 console application

• The executable (detectMorph.exe) will take the input from command-line arguments and will append the output to a text file. It evaluates a single face image and produces a morph score; the command-line syntax is:
  detectMorph.exe <suspectedmorphfile> <label> <outputfile>
  where:

  suspectedmorphfile	the suspected morph face image pathname. The supported image formats are: .BMP, .JPG, .PNG., .JP2, .JPF
  label	an integer describing the format of the input image: - unknown photo format/origin (0), - non-scanned digital photo (1), - a photo that is printed, then scanned (2).
  outputfile	the output text-file, where a log string (of the form suspectedmorphfile result isMorph score) must be appended. - result is "OK" if the detection can be performed or "FAIL" if the detection cannot be executed by the algorithm; - isMorph is "TRUE" if the image contains a morph or "FALSE" otherwise; -score is a floating point value ranging from 0 to 1 representing how confident the algorithm is that the image contains a morph: 0 means certainty that image does not contain a morph and 1 represents certainty that image contains a morph.

• The executable has to operate only on the explicitly-given inputs, without exploiting any learning technique or template consolidation/update based on previous comparisons.
• C and C# language skeletons for detectMorph.exe are available in the download page to reduce the participants implementation efforts.
• During the upload process, please select Win32 executable as Algorithm type.

    Python script

• In the Python script (smad_detect.py) two functions must be implemented: initialize and detect_morph.
• The initialize function initializes all internal parameters and loads auxiliary resources (e.g., configuration files, weights of a pre-trained model, etc.) in preparation of the morphing detection task.
  

  Signature	def initialize(alg_extra_data_folder_path) -> ReturnCode
  Input parameters	- alg_extra_data_folder_path: a string with the absolute path to a read-only directory containing auxiliary resources that cannot be directly submitted during the upload process because they exceed the submission file size limit (see below for more information on how to upload large sized files). All auxiliary resources submitted with the Python script can be loaded from the current directory ignoring this parameter.
  Return values	- ReturnCode: an IntEnum representing the return value of the function (see the Python script sample in the download page for more details).

• The detect_morph takes one input image (suspected) and an associated image label describing the image format/origin and it outputs a binary decision on whether the suspected image is a morph and a score indicating how confident the algorithm thinks it is a morph.
  

  Signature	def detect_morph(suspected_morph_file_path, label) -> Tuple[ReturnCode, bool, float]
  Input parameters	- suspected_morph_file_path: a string containing the suspected morph face image file path. The supported image formats are: .BMP, .JPG, .PNG, .JP2, .JPF.; - label: an IntEnum describing the format of the suspected morph image: unknown photo format/origin (0), non-scanned digital photo (1) or a photo that is printed, then scanned (2). See the Python script sample in the download page for more details.
  Return values	- ReturnCode: an IntEnum representing the return value of the function (see the Python script sample in the download page for more details); - bool: it is True if the image contains a morph or False otherwise; - float: it is the morph score ranging from 0 to 1 representing how confident the algorithm is that the image contains a morph: 0 means certainty that image does not contain a morph and 1 represents certainty that image contains a morph.

• The script has to operate only on the explicitly-given inputs, without exploiting any learning technique or template consolidation/update based on previous comparisons.
• A Python script sample is available in the download page to reduce the participants implementation efforts.
• Auxiliary resources (e.g., trained models, configuration files, Python modules, etc.) can be submitted with the Python script during the upload process. If they exceed the submission file size limit, please contact the system administrator at fvcongoing@csr.unibo.it.
• During the upload process, please select:
  • Python script as Algorithm type.
  • the Docker image on which the submitted script will be executed (see the docker page for more details).

    Linux dynamically-linked library

To submit a Linux dynamically-linked library compliant to NIST FRVT MORPH specifications (v. 3.0 - May 19, 2022) proceed with the following steps:

1. Create a ZIP file with the following NIST FRVT MORPH folders:
   • lib - containing all participant-supplied software libraries;
   • config – containing all configuration and developer-defined data, e.g., trained models.
2. Send the ZIP file (using a file sender service or a direct link) from the e-mail address of your FVC-onGoing participant account to fvcongoing@csr.unibo.it.
3. Wait to receive an e-mail from the FVC-onGoing organizers with a ZIP file attached (containing a TXT file).
4. Submit the ZIP file (without changing it) to the desired benchmark using the Upload function. During the upload process, please select:
   • NIST dynamic library as Algorithm type.
   • the Docker image on which the submitted library will be executed.

The table below reports the docker images available for the evaluation of a NIST Linux dynamically-linked library on the SMAD benchmark area:

Constraints

During test execution the following constraints will be enforced:

Each detection attempt that violates one of the above constraints results in a failure to detect.

The following time breaks are enforced between two consecutive submissions to the same benchmark by the same participant.

Performance Evaluation

For each algorithm, bona fide (evaluating bona fide face images) and morph (evaluating morphed face images) attempts are performed to compute Bona fide Presentation Classification Error Rate (BPCER) and Attack Presentation Classification Error Rate (APCER). As defined in [4] the BPCER is the percentage of bona fide presentations falsely classified as morphing presentation attacks while the APCER is the proportion of morphing attack presentations falsely classified as bona fide presentations.

Although it is possible to reject images, this is strongly discouraged. In fact, rejection is fused with other error rates for measuring the performance indicators; in particular, each rejection will be considered as a detection of a morphed image (isMorph="TRUE" and score=1).

For each algorithm the following performance indicators are reported:

• EER (detection Equal-Error-Rate: the error rate for which both BPCER and APCER are identical)
• BPCER₁₀ (the lowest BPCER for APCER≤10%)
• BPCER₂₀ (the lowest BPCER for APCER≤5%)
• BPCER₁₀₀ (the lowest BPCER for APCER≤1%)
• REJ_NBFRA (Number of bona fide face images that cannot be processed)
• REJ_NMRA (Number of morphed face images that cannot be processed)
• Average detection time
• Maximum amount of memory allocated
• Bona fide and Morph detection score distributions
• APCER(t)/BPCER(t) curves, where t is the detection threshold
• DET(t) curve (the plot of BPCER against APCER)

Terms and Conditions

All publications and works that cite SMAD Benchmark Area must reference [3].

Bibliography

[1] M. Ferrara, A. Franco and D. Maltoni, "On the Effects of Image Alterations on Face Recognition Accuracy", in Thirimachos Bourlai, Face Recognition Across the Electromagnetic Spectrum, Springer, 2016.
[2] M. Ferrara, A. Franco and D. Maltoni, "Face Demorphing", IEEE Transactions on Information Forensics and Security, vol.13, no.4, pp.1008-1017, April 2018.
[3] K. Raja, M. Ferrara, A. Franco, L. Spreeuwers, I. Batskos, F. de Wit, M. Gomez-Barrero, U. Scherhag, D. Fischer, S. Venkatesh, J. M. Singh, G. Li, L. Bergeron, S. Isadskiy, R. Ramachandra, C. Rathgeb, D. Frings, U. Seidel, F. Knopjes, R. Veldhuis, D. Maltoni, C. Busch, "Morphing Attack Detection - Database, Evaluation Platform and Benchmarking", IEEE Transactions on Information Forensics and Security, November 2020.
[4] ISO/IEC JTC1 SC37 Biometrics, ISO/IEC IS 30107-3:2017, IT – Biometric presentation attack detection – Part 3: Testing and Reporting, 2017.
[5] iMARS Project Web Site. https://imars-project.eu
[6] M. Ferrara, A. Franco and D. Maltoni, "Decoupling texture blending and shape warping in face morphing", International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, September 2019.