Benchmark area: Face Image ISO Compliance Verification

This benchmark area contains face image ISO compliance verification benchmarks. Algorithms submitted to these benchmarks are required to check the compliance of face images to ISO standard.

Benchmarks

Currently, this benchmark area contains the following benchmarks:

• FICV-TEST: A simple dataset useful to test algorithm compliancy with the testing protocol (results obtained on this benchmark are only visible in the participant private area and cannot be published).
• FICV-1.0: A large dataset of high-resolution face images related to all the requirements specified in Table I. This benchmark is described in detail in [1].

The table below reports the main characteristics of each benchmark:

Information about reproducing the FICV-TEST dataset are available in the download page.

The following sections report the list of requirements, the testing protocol and the performance indicators common to all benchmarks in this area.

Requirements

Starting from the indications provided into the ISO standard [2], a set of 24 tests has been defined to evaluate the accuracy of automatic systems for compliance verification (see Table I). The tests can be organized into two categories:

• Feature extraction accuracy: this test evaluates the accuracy of eye centers detection¹;
• Photographic and pose-specific tests: the face must be clearly visible and recognizable; this requirement implies several constraints that generate most of the uncertainties in the interpretation of the ISO standard. A precise formalization is here proposed to limit as much as possible the ambiguity (see Fig. 1).

Moreover the possibility of transforming the image in Token Format (see section 9.2.3 in [2]) is evaluated on the basis of the eye positions¹. An image is tokenizable (without padding) if:

• the distance (E_Dist) between eyes is at least 60 pixels;
• the rectangular region of size W×H (with W=4·E_Dist and H=W·4/3), determined so that the eyes are horizontally aligned and their center is in position C_E=(W·1/2,W·3/5), is totally enclosed in the original image (see Fig. 2).

¹ The eye center points are defined to be the horizontal and vertical midpoints of the eye corners (see section 5.6.5 in [2]).
² ±5 degrees for rotations in pitch and yaw, and ±8 degrees for rotation in roll (see section 7.2.2 in [2]).

Protocol

Each participant is required to submit, for each algorithm, one executable named Check.exe in the form of Win32 console applications.

The executable will take the input from command-line arguments and will append the output to a text file.

• The command-line syntax is:

  Check.exe <faceimagefile> <outputfile>

  where:
  

  faceimagefile	is the input face image pathname (e.g., c:\test\db1\image1.png). The supported image formats will be: .BMP, .JPG, .PNG.
  outputfile	is the output text-file pathname (e.g., c:\test\db1\out.txt), where a log string (described in the following) must be appended. If the file does not exist, it must be created by the executable.

• Each line of the outputfile must contain the following fields separated by a blank space character:
  • ImageName is the image file name (e.g., image1.png).
  • RetVal is an integer value indicating if the input image can be processed or not by the executable³:
    • 1 if the image can be processed.
    • 0 if the image cannot be processed and no more information is available.
    • -1 if the image cannot be processed due to unsupported image size.
    • -2 if the image cannot be processed due to unsupported image format.
    • -3 if the image cannot be processed due to unuseful image content.
  • LE_x is an integer value indicating the X coordinate (in pixels) of the left eye center⁴.
  • LE_y is an integer value indicating the Y coordinate (in pixels) of the left eye center⁴.
  • RE_x is an integer value indicating the X coordinate (in pixels) of the right eye center⁴.
  • RE_y is an integer value indicating the Y coordinate (in pixels) of the right eye center⁴.
  • Test_2 is an integer value in the range [0;100] indicating the compliance degree of the input image with respect to the Compliance Test 2. 0 means no compliancy, 100 maximum compliancy. Moreover, the following special characters have been defined:
    • ‘-‘ if the executable is not able to evaluate this requirement.
    • ‘?’ if, usually, the executable is able to evaluate this requirement but it was not able to evaluate it on the current input image for an identified failure (e.g., the SDK is not able to evaluate the presence of red eyes since the eyes are closed).
    • ‘!’ if, usually, the executable is able to evaluate this requirement but it was not able to evaluate it on the current input image for an unknown failure.
  • Test_3 analogous to test 2 but related to test 3.
  • ...
  • Test_24 analogous to test 2 but related to test 24.
• C, C# and Matlab language skeletons for check.exe are available in the download page to reduce the participants implementation efforts.
• The executable has write permission only on the output file. All configuration files should be opened in read mode.

³ If the input image cannot be processed (i.e., RetVal≠1) the other fields must be empty.
⁴ The origin of the coordinate system shall be the upper left corner of the input image; the X and Y values start from 1 (and not from 0).

Constraints

During test execution the following constraints will be enforced:

Each check attempt that violates one of the above constraints results in a failure.

The following time breaks are enforced between two consecutive submissions to the same benchmark by the same participant.

Performance Evaluation

For each image, the compliance to the requirements listed in Table I is evaluated using the submitted algorithm. To evaluate the accuracy of eye localization the relative error measure based on the distances between the expected and the estimated eye positions introduced in [3] is calculated as:

where C_l/r  and Ĉ_l/r  are the ground truth and the positions returned by the algorithm, respectively. This measure is scale independent and therefore it permits to compare data sets characterized by different resolution.

For each algorithm the following general performance indicators are reported:

• Percentage of Checking Failures due to crash, processing failure, timeout or memory limit exceeded
• Average check time
• Maximum check time
• Maximum amount of memory allocated
• Eye Localization Accuracy Distribution
• Percentage of images correctly identified as Tokenizable or not

Moreover, for each Photographic and pose-specific test (see Table I) some accuracy indicators are calculated. In order to perform balanced tests, the dataset has been further divided into 23 subsets, each related to a specific requirement, containing the same number of compliant and non-compliant images. It is worth noting that each subset is related to a specific characteristic and that consequently only the algorithm response for that characteristic is considered for indicators calculation.

For each requirement, starting from the distribution of the degrees of compliance, the following performance indicators are reported:

• EER (equal-error-rate)
• FMR₁₀₀ (the lowest FNMR for FMR≤1%)
• Zero_FMR (the lowest FNMR for FMR=0%)
• Zero_FNMR (the lowest FMR for FNMR=0%)
• Rejection (percentage of images where the requirement was not evaluated)
• Impostor and Genuine score distributions
• FMR(t)/FNMR(t) curves, where t is the acceptance threshold
• DET(t) curve

Note that, according to the best practices, rejections are here implicitly included in the calculation of the performance indicators by assuming that a degree of compliance equal to 0 (for the given requirement) is returned in case of rejection. This choice is aimed at discouraging the algorithm to reject the most uncertain cases thus improving the performance over processed images.

Terms and Conditions

All publications and works that cite FICV Benchmark Area must reference [1].

Bibliography

[1] M. Ferrara, A. Franco, D. Maio and D. Maltoni, "Face Image Conformance to ISO/ICAO standards in Machine Readable Travel Documents", IEEE Transactions on Information Forensics and Security, vol.7, no.4, pp.1204-1213, August 2012.
[2] ISO/IEC 19794-5, Information technology - Biometric data interchange formats - Part 5: Face image data, 2011.
[3] O. Jesorsky, K. J. Kirchberg, and R. W. Frischholz, "Robust face detection using the hausdorff distance", Lecture Notes in Computer Science 2091, pp.90-95, 2001.