A Benchmark Data Set to Evaluate the Illumination Robustness of Image Processing Algorithms for Object Segmentation and Classification

Abstract

Developers of image processing routines rely on benchmark data sets to give qualitative comparisons of new image analysis algorithms and pipelines. Such data sets need to include artifacts in order to occlude and distort the required information to be extracted from an image. Robustness, the quality of an algorithm related to the amount of distortion is often important. However, using available benchmark data sets an evaluation of illumination robustness is difficult or even not possible due to missing ground truth data about object margins and classes and missing information about the distortion. We present a new framework for robustness evaluation. The key aspect is an image benchmark containing 9 object classes and the required ground truth for segmentation and classification. Varying levels of shading and background noise are integrated to distort the data set. To quantify the illumination robustness, we provide measures for image quality, segmentation and classification success and robustness. We set a high value on giving users easy access to the new benchmark, therefore, all routines are provided within a software package, but can as well easily be replaced to emphasize other aspects.

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Fig 1. Benchmark data sets.

Overview of images in R = 4 scenes corresponding to increasing artifact and noise levels

Fig 2. Representation of ground truth objects using color shading and numbers as labels.

Left: Brightfield image X of a benchmark data set with marked object edges, right: Ground truth image Xtruth, with object types in numbers. Gray scales denote the value of xij,truth (0: black (background), 5: white)

Fig 3. Exemplary pipeline for the segmentation of benchmark images.

Fig 4. Results of benchmark data set r = 1.

Total quality criterion Q(r, b, n) vs. increasing artifact level A(r, b, n) for image series with stepwise addition of both shading and noise for each successive image. The first row of images indicates original images from data set r = 1. The second row shows corresponding segmentation and classification results using Otsu’s method. The third row shows corresponding segmentation and classification results using edge detection method. Brown color represents correct classification of the segmented BLOB w.r.t corresponding ground truth BLOB and light green color shows an erroneous classification object. Robustness values for Otsu thresholding and Sobel edge detection are Rotsu = 0.54 and Redge = 0.47 respectively.