Exploring the limits of color accuracy in technical photography

Abstract

The growing importance of publishing art collections online has led to increasingly strict tolerances on digital photography of art objects. There are two internationally recognized sets of guidelines for creating high-quality digital images, Metamorfoze and FADGI. These guidelines require using sets of standardized color patches in museal photography. The X-Rite ColorChecker SG chart with 140 color patches is often used. Recent studies showed that even in standardized conditions it is often difficult to satisfy the strictest guidelines on color accuracy for camera profiling, with no indications for improvements. We report results of our investigation into the bottlenecks in achieving high color accuracy. We show that a large part of the color deviations originates from the 15 black color patches of the ColorChecker SG chart. These patches have a large impact on the average color deviation and the maximum color deviation that are the performance measures for color accuracy in the FADGI and Metamorfoze guidelines. We show that spectrophotometer measurements for the black patches produce color deviations dE(CIE 1976) ranging from 3.7 to 5.2 with respect to reference data, making it impossible to meet the strictest Metamorfoze guidelines. The black patches push the average color difference CIEDE2000 from 0.59 to 0.82 with respect to reference data already when using spectrophotometer data. Since the strictest FADGI guidelines prescribe an average CIEDE2000 = 2.0, this leaves little tolerance for errors due to lighting and camera profile. Our results indicate that the common practice of manually tweaking camera profiles until software suggests sufficient color accuracy is obtained with respect to suppliers’ reference data often does not improve color representation but makes it worse, especially for representing dark nuances. This is unfortunate for example for the digital photography of seventeenth century Dutch paintings, where dark passages occupy large areas of the art works. We show that the key step in achieving color accurate digital photography is to use customer reference data rather than commonly used generic reference data. We explain the results by investigating not only the color properties of the ColorChecker SG chart, but also its glossiness.

Introduction

Many major museums and art institutes run large multi-year projects to digitize their collections for decades now [1]. Since the aim of the digitization process is to obtain as close a digital representation of the original object as possible [2], the collected data needs to be of the highest accuracy and quality [3]. Digital photography and color management have been recognized as important components of this process. Therefore photograph studios of museums are continuously developing workflows and standardized imaging systems in which color management plays a major role.

Also in the technical analysis of cultural heritage the role of imaging science is growing. Several trends combine to lead to increasingly tighter color tolerances for technical photography. For example, photographs are increasingly used to help making decisions on conservation strategies, making it essential that conservators can trust the accuracy of the details they see in these images. Photographs are also used more than before to evaluate the progress and results of conservation campaigns. In this case, photographs captured before, during and after the conservation treatment need to be compared with each other, which requires being able to exactly reproduce the conditions and camera profiling after months or years. A third example is the growing awareness that for investigating art works in relation to pieces located in other museums it is essential to use highly reproduceable and repeatable images.

The development of processes for technical photography at color tolerance levels tighter than ever before is also an enabler for new approaches and technical analyses. A first striking example of this is the extremely high-resolution photography of historical paintings which is being developed over the past few years, which enables new technical analyses and conservation approaches [4, 5]. High resolution imaging requires composing increasingly larger numbers of partial photographs into one overall image. For example, for the photography of Rembrandt's Marten and Opjen paintings at 1250 ppi resolution, 242 partial photographs (also known as image tiles) had to be stitched and registered in 2016 [4], while for the most recent photography of Rembrandt’s Night Watch a total of 11,845 image tiles were captured and combined [6]. In order to ensure color consistency throughout the composite image, and to make automated stitching of image tiles feasible, very tight color tolerances are required within each image tile.

A second example of the benefits of using highly standardized processes for capturing photographs of artworks is that parts of this work can be outsourced to external institutes or companies. The Rijksmuseum is already working for years on digitizing their collection of one million art objects with high resolution photographs and making this publicly available [1]. For achieving this goal, it is essential to be able to outsource art of the work to external contractors that are able to work according to strict guidelines.

However, various investigations have shown that even when working with professional cameras and well-controlled lighting set up, it is often very difficult to satisfy the highest color quality level as defined by internationally recognized guidelines [7,8,9]. These investigations showed already that for obtaining good color accuracy it is required to create a custom camera profile, instead of using default ICC camera profiles [7, 8]. Even then, the user often needs a considerable amount of time for visual editing to obtain a higher inter- and intra-camera performance [7].

Finally, the earlier investigations showed that the color patches of universal targets (such as the ColorChecker SG chart mentioned below) are intended to suit a wide variety of colored objects, often presenting a much wider color gamut than what is needed for digitizing a particular artwork [8]. Since software that creates custom camera profiles puts equal weight to all color patches, standard targets have a problem with accurately reflecting colors in selected areas which are important for a given artwork [8, 9]. This aspect is certainly relevant when digitizing seventeenth century Dutch art, with its many shades of black that are hardly present in universal targets.

In this article we will further analyze factors that cause color inaccuracies. This study aims to identify the main contributions to color deviations in the technical photography process when using current guidelines. Over the past decades the science behind accurate color reproduction has resulted into consensus for most aspects of the process. An early summary was written by Berns [10]. Currently there are two internationally recognized sets of guidelines to standardize the workflow for creating high-quality digital images of cultural heritage. FADGI [11] was originally developed by the Library of Congress in the US, whereas Metamorfoze [12] was developed by the National Library in the Netherlands. We note that both guidelines are included in the ISO 19,264 standard [13]. These guidelines cover more aspects than only color, such as spatial resolution and misregistration. In the present work we will only discuss quality aspects related to color.

"Experimental" section summarizes the experimental part of our study. It describes the current process for digital photography, including camera profiling, when working according to internationally recognized guidelines. In the same section we also introduce the X-Rite ColorChecker SG chart, which is most often used for camera profiling.

In "Results and Discussion" section. we will show how the black color patches in color charts are a major contributor to color deviations in the camera profiling process, and we will analyze why this is the case. We will show how color quality improves by avoiding the use of generic ground truth data. Improved color quality can be obtained by introducing custom spectrophotometer measurements, and does not require using highly expensive instruments.

We summarize our conclusions in "Conclusions" section, and we identify the most promising routes for further improvements in color accuracy.

Experimental

Based on many years of experience as professional photographers for the Rijksmuseum, two of the present authors (CW and HB) confirm the conclusion from several other investigations [7,8,9]. It is very hard to ensure that the color quality and contrast of the image meet the highest quality level as defined by the internationally recognized guidelines. This turns out to be the case even when following the workflow of digital photography (summarized in "Process for Technical Photography" section), when using standardized color charts (introduced in "Color charts" section), and when working with high-end professional camera and lighting set-up mentioned in "Camera, lighting, spectrophotometers and software" section.

As specified in "Process for Technical Photography" section, image quality is quantified by calculating for each color patch the color difference between on the one hand CIE-Lab color coordinates calculated from camera RGB values using the camera (ICC) profile, and on the other hand reference CIE-Lab data published by X-Rite and created with a spectrophotometer [14].

We will investigate the main contributions to color inaccuracies, using spectrophotometers and software to process the measurement data as described in "Camera, lighting, spectrophotometers and software" section. Before describing how we investigated the contribution of measurement errors and variations in color charts to the final image quality in "Results and Discussion" section, we will introduce the main experimental aspects of this investigation.

Process for technical photography

The FADGI and Metamorfoze guidelines are very similar in many aspects, and both aim at satisfying the same set of ISO standards for measuring imaging performance. For the large majority of metrics, they agree on aim and tolerance levels [15]. For the performance measure for color tolerance Metamorfoze chooses dE (CIE 1976), whereas this is CIEDE2000 for FADGI (both measures defined by the International Commission on Illumination (CIE) [16]).

After pioneering work by Martínez et al. [17] to develop a procedure to optimize the reproduction process, different workflows have emerged for creating images according to these guidelines. Since there is no consensus on the best workflow, we limit ourselves here to sketching a procedure that includes the most essential steps as used by technical photographers in the Rijksmuseum, as well as described in recent publications [2, 18, 19].

• 1. Flat-field calibration. The camera aperture is varied to optimize sharpness. A uniform chart is captured, and the standard deviation in brightness across the image is minimized. This compensates for nonuniformity in lighting as well as deviations caused by the camera lens [20].

• 2. White calibration. This is to ensure that the white color is captured correctly. The camera exposure time and/or lighting power are optimized by comparing camera RGB values with the known CIE-Lab value of a white standard, consistent with the D50 white point.

• 3. Fine-tune brightness variation across the image. The X-Rite ColorChecker SG chart is captured with the camera. Using only the white color patches of this chart, a final verification is obtained to ensure that brightness variation is small across the image.

• 4. Calibration of the Tone Response Curve (TRC), also known as the Opto-Electronic Conversion Function (OECF). Using all color-neutral patches of the ColorChecker SG chart, the goal of this step is to make sure that all neutral colors between white and black are mapped to accurate CIE-L* values. The color deviations (or color encoding errors, as they are called in FADGI) need to be within the tolerances provided by the guidelines. This procedure accounts for the nonlinearity of the camera system. It ensures contrasts in the image correspond to visually perceived contrasts. By making sure that the a* and b* values that result for the achromatic patches are sufficiently close to zero, the white balance is checked across the image and a color cast is avoided.

• 5. Verify sampling efficiency. Evaluate what percentage of intended image resolution (in ppi) is obtained.

• 6. Camera characterization, also known as building the camera (ICC) profile. The full X-Rite ColorChecker SG chart is captured. Software detects all color patches in the image and calculates average RGB values. The camera (ICC) profile is built by optimizing model parameters in such a way that converting the RGB values of each patch leads to CIE-Lab values with a minimized color difference with respect to reference CIE-Lab data for the patch.

• 7. Determine image quality (see below).

We note that the camera (ICC) profile thus created is only valid for the camera under the conditions used when capturing the images with the color patches. It is assumed that the same ICC profile remains valid for the photography of other objects, as long as the lighting and camera conditions (such as lenses) remain unchanged. The custom ICC profile is assigned to the photographic image, replacing the standard color ICC profile color space [2].

In the last step of the process, the image quality is assessed. It would have been best if this would include a visual assessment that quantifies the quality of the image in terms of its accuracy in reproducing the original art object. However, there is no standard procedure for this and it would be highly subjective.

Below, we will identify the cause why even when following the workflow and using high-end professional equipment often a camera profile results that leads to color deviations that are too large to be acceptable when aiming for the strictest tolerances in international guidelines. In such cases, photographers may try to manually adapt the values of some of the parameters that characterize the camera profile, even though this results in values that are no longer the result of optimizations. By this manual tweaking of the camera profile, sometimes the color deviations can be reduced to acceptable levels, as measured by the software. However, this makes the whole process time consuming and badly reproduceable. Also, it is an attempt to correct for errors that are made elsewhere in the process of camera building, as we will see below.

We will not elaborate on this aspect further, other than by remarking that during such a subjective assessment the art object should be lighted with the same light that was assumed in the camera profiling (i.e., D50 spectrum), and the image would need to be visualized on a color calibrated monitor display (page 231 of [21]).

Instead of a visual assessment of image quality, the color accuracy of the process is expressed in the color deviations (or color encoding errors, as they are called in FADGI) that are tolerated in these guidelines. The quality of the image is evaluated by determining color differences for all 140 color patches. The color difference for each patch is calculated between on the one hand CIE-Lab values calculated with the camera profile using the camera RGB values as input, and on the other hand reference CIE-Lab data as measured by a spectrophotometer.

The strictest FADGI guidelines require the average to be smaller than CIEDE2000 ≤ 2 [22]. We note that the previous version of FADGI demanded a CIEDE2000 of 3.0 on average, and a maximum value lower than 6.0 [11].

For Metamorfoze, the strictest guidelines demand the maximum color difference to be dE(CIE 1976) ≤ 10.0 and the average to be dE(CIE 1976) ≤ 4.0 [12]. Apart from that, for neutral patches it is required that dL* ≤ 2, dC* ≤ 2 and dE(CIE 1976) ≤ 2.83 [12].

Color charts

"Process for technical photography" section shows that most steps in the process of camera profiling require charts with series of standardized color patches. Here we will introduce two color charts that we used in our investigation: the X-Rite Digital ColorChecker SG chart (Fig. 1) and the Munsell Linear Gray Scale chart (Fig. 2). We note other charts are available as well: the Golden Thread Object-level Target and Device-level Target [8, 23, 24], the Universal Test Target and the Next Generation Target [25,26,27].

Fig. 1

X-Rite ColorChecker SG chart

Fig. 2

Munsell Linear Gray Scale

Digital ColorChecker SG

The X-Rite Digital ColorChecker SG is a direct descendent of the original MacBeth ColorChecker Rendition Chart [28] introduced in 1976 [16][29]. That chart included 24 colors representing frequently occuring colors in common photography [30]. This chart proved to be useful by including it as a color reference in high-end photography of objects [31]. When digital cameras became popular it was found that the MacBeth ColorChecker was less suitable for creating camera profiles, as its colors cover only a relatively small color gamut [31].

The Gretag-Macbeth ColorChecker DC was launched in 2000 specifically for profiling Digital Cameras (DC). The number of colors had increased to 237 and also the color gamut covered by them was greatly expanded [28]. Eight of these patches were high-gloss, probably to cover a wider color gamut with higher color saturation, and all other patches were matte [32]. In practical use the high-gloss patches often picked up reflections, creating out-of-gamut colors in the image that ruined the resulting profile [31]. Also, the camera profiles created with this chart were often not smooth enough, possibly due to the very large number of color patches in this chart [33]. For these reasons this chart was soon replaced by the ColorChecker SG.

In 2005, GretagMacBeth/X-Rite launched the Digital ColorChecker SG [34]. The new chart contains an extended color palette of 140 patches, all of them being Semi-Gloss (SG). Since the patches are more glossy than those on the earlier ColorChecker charts, more saturated colors are reached, and also darker colors are reached by the Digital ColorChecker SG chart. Compared to the original MacBeth ColorChecker Chart it provides a wider variety of skin tones, and a series of achromatic colors to determine the camera white balance under different light sources [16]. Just like the simpler MacBeth ColorChecker Chart, the Digital ColorChecker SG chart is widely used by professional photographers. For the most demanding applications, such as the photographic documentation in the conservation of cultural heritage, the ColorChecker SG replaced the MacBeth ColorChecker Chart as the common color chart [35]. In order to enable this group of highly professional photographers to build camera profiles and to do camera characterization (cf. step 6 in "Color charts" section), X-Rite produced reference data obtained with a spectrophotometer for each of the 140 color patches, and made this data publicly available [14].

In 2015 X-Rite announced that “for regulatory and compliance reasons”, the color formulations of all color patches in the X-Rite ColorChecker Classic and the X-Rite Digital ColorChecker SG had to be changed. This meant that for both charts there are different versions of the colorimetric ground-truth data, suitable for charts produced before or after November 2014 [14].

In this investigation, we will use several copies of the ColorChecker SG chart, which were purchased at different occasions between 2014 and 2020. We will provide more details on these charts below.

Munsell grayscale

For assessing image quality, it would have been best to use a different set of standardized color patches than the set used for building the camera profile. However, by lack of materials in practice often the X-Rite ColorChecker SG chart is used for both steps in the process. In order to create some independent data at least for the neutral colors sometimes the Munsell Grayscale is used.

The Munsell Linear Gray Scale [36] contains 18 semi-gloss neutral grey color patches, with colors equally spaced from almost white (L* = 95) to almost black (L* = 10). It also contains two high-gloss black patches with L* = 5, making it darker than any patch on the X-Rite ColorChecker SG chart.

Camera, lighting, spectrophotometers and software

In this investigation we use a high-end professional lighting set-up from Broncolor, in combination with a high-end professional camera from Hasselblad. It produces 14-bit raw images, which are processed by Hasselblad’s Phocus software with a 16-bit imaging pipeline.

We use an i1Pro spectrophotometer from X-Rite to measure reflectance values in a 45°/0° measurement geometry with ring illumination, as well as a spectro2guide spectrophotometer from BYK-Gardner.

We use software package iQ-Analyzer [37] to analyze image quality. We also use the calculational tool RM that was developed by the Rijksmuseum (co-author HB), which can analyze Metamorfoze, FADGI, X-Rite standards and customer look up tables.

Samples

For this investigation we collected a range of samples with neutral colors, ranging from dark to pitch black, as well as some neutral samples with brighter neutral colors. The samples also varied in gloss level as indicated in the following list:

• 7 samples with presumably the same black color S-9000 N, while varying in gloss level from very matte up to high gloss. These samples are the blackest samples of the NCS Gloss Scale (supplier: Natural Color System) [38].

• 21 color patches from the Munsell Linear Gray Scale [36]. This set contains two high-gloss very black patches, and 18 semigloss neutral grey color patches ranging from very dark to very light.

• 7 car refinishes paint samples with the same black basecoat, covered by clearcoats (Mix & Matte, supplier AkzoNobel) that vary from matte to very high gloss.

Results and discussion

The nature of the color deviations and their magnitude

Our analyses indicate that the largest deviations are found for the darkest color patches of the X-Rite ColorChecker SG chart. In order to further analyze the origin of these color deviations, we decided to measure color coordinates for all 140 color patches with a spectrophotometer ourselves. For this we used an i1Pro instrument (supplier: X-Rite), using the same measurement mode M0 (defined by ISO 13655:2009) as X-Rite used when creating the reference data [14]. In order to avoid measurement errors we took the average reading from three consecutive color measurements for each patch. We used this custom data to calculate the color difference with the supplier's reference data [14] for each patch in the chart. All calculations were done in terms of CIEDE2000, with D50 illuminant and CIE 2° standard observer.

The results are shown in Fig. 3. Colors indicate the magnitude of the color difference. It is clear from Fig. 3 that our custom data shows the largest deviations from the reference data for the black patches. This is the case for all black patches, both those dispersed along the circumference of the chart as well as patch E6. Based on many years of experience as professional photographers for the Rijksmuseum, two of the present co-authors (HB and CW) confirm that this pattern is typically found also when evaluating image quality after building camera profiles. The black patches almost always give large deviations in the value for lightness L*, producing color differences large enough to make it more difficult to satisfy the strictest criteria of FADGI and Metamorfoze guidelines [39].

Fig. 3

Color deviations CIEDE2000 on the X-Rite ColorChecker SG chart between custom data and supplier’s reference data [14]. Colors indicate the magnitude of color error, with CIEDE2000 < 1 marked green, values between 1 and 2 marked light green, values between 2 and 3 marked yellow and values above 3 marked red

In terms of CIEDE2000, the average color deviation for all 140 color patches is 0.82. For the 15 black color patches the average color deviation is CIEDE2000 = 2.74. If the black patches are ignored, the average color deviation of the remaining 125 color patches is CIEDE2000 = 0.59. For camera profiling, the strictest FADGI guidelines require the average color deviation to be smaller than CIEDE2000 ≤ 2 [22]. Since the quality achieved in camera profiling depends also on errors due to lighting and camera optics, it is concerning that using different sets of spectrophotometer data already shows errors of this magnitude.

In terms of dE(CIE 1976), the black patches show color deviations of dE(CIE 1976) = 4.23 on average (maximum: 5.23), increasing the average for all patches from dE(CIE 1976) = 0.88 to 1.24. The strictest guidelines for camera profiling using the Metamorfoze guidelines demand for each of the neutral patches that dE(CIE 1976) ≤ 2.83 [12]. This is clearly not fulfilled when comparing data from different spectrophotometers, making it impossible that camera profiles will satisfy this demand. There is no problem yet with the Metamorfoze guideline that the maximum color difference needs to be dE(CIE 1976) ≤ 10.0 and the average for all 140 patches should be dE(CIE 1976) ≤ 4.0 [12]. However, with the large color deviations reported here when comparing data from spectrophotometers there is hardly any tolerance possible for errors due to lighting and camera optics.

The large color deviations (mainly in L*) for black color patches is exemplified by black patch E6. According to X-Rite’s reference data [14], its color coordinates (L*, a*, b*) are (8.07, 0.12, -0.93), whereas our spectrophotometer data for the same patch show (12.75, 0.22, 0.07). The color difference between both sets of color coordinates is CIEDE2000 = 3.11 and dE(CIE 1976) = 4.79. Our measurement data show a much larger value especially for lightness coordinate L* = 12.75 as compared to the value L* = 8.07 provided by X-Rite. The lightness difference strongly dominates the overall color difference for this patch. Also for all other black patches our measurement data show considerably larger lightness values than the corresponding reference data. We note that in the OpenDICE software, for patch E6 a value of L* = 10.88 is used, based on measurements with a Konica Minolta FD-7 (45°/0°) spectrophotometer [40]. This confirms our conclusion that the lightness values of the black color patches may be too small in X-Rite’s publicly available reference data.

Our spectrophotometer data are based on measurements on a relatively new X-Rite ColorChecker SG chart. At the back side of the chart the production date is printed as being June 2018. To make sure that our results are not the result of a bad copy of the chart, we repeated our measurements with an older X-Rite ColorChecker SG chart, with production date July 2016. Also in this case we find a lightness value that is much larger than in X-Rite’s reference data.

We thus find that our measurement data differ from X-Rite reference data especially by yielding much larger values for L* for the dark color patches. As long as this problem is not solved, photographers are faced with the following dilemma when building camera (ICC) profiles.

If photographers would choose to use their own measurement data instead of X-Rite reference data, the darkest colors in calibrated images would not be pitch black, but they would probably correspond to values of L* ≥ 12.75. This option would effectively make the darkest spots in calibrated photographs brighter than absolute black. By using the camera profiles created in this way for the photography of artworks, many subtle nuances in dark colors on the original art work will be lost in the digital reproduction. This is unfortunate, since dark nuances form an important aspect of many art works, such as seventeenth century Dutch paintings.

Alternatively, a photographer may choose not to use own measurement data for the black patches. In that case, in order to be consistent, the photographer should then also not use own measurement data for the other color patches. Choosing this alternative option would also be unfortunate, because the photographer would have to assume that all color patches in an individual ColorChecker SG chart are well described by the reference data. In practical situations, photographers may want to use a ColorChecker SG chart that is not brand new, so its colors may have slightly changed in time due to color fading. If high quality camera profiles can only be built by using new ColorChecker SG charts, this would become a costly affair.

The role of instruments in creating the color deviations

In "The nature of the color deviations and their magnitude" section we found that our measurement data on the X-Rite ColorChecker SG chart differ from X-Rite reference data especially by producing much larger values for L* for the dark color patches. One possible explanation for this problem may be that these datasets were not produced by exactly the same instrument. Our reflectance measurement data was taken with an i1Pro instrument, whereas X-Rite's reference data was produced with an i1Pro2 instrument [14]. However, these two instruments are sufficiently similar to expect that they are well aligned with each other. Both instruments have the same measurement aperture, with an illumination spot size of 3.5 mm [41, 42]. Both instruments are based on the same 45°/0° measurement geometry (with ring illumination), which should make us expect their results to be aligned by a color difference of dE(CIE 1976) between 0.4 and 0.8 on average [35]. As reported in the last paragraphs, the color deviations for which we are trying to find an explanation here are much larger than this value.

However, when we discussed these issues with an instrument supplier in 2020, it was suggested that the problem could be solved by not using an i1Pro instrument but a much more expensive spectrophotometer. Obviously, we should choose an instrument with the same 45°/0° measurement geometry as the instrument that was used to create the reference data. For example, it is well known that using a diffuse sphere d/8° instrument instead of a 45°/0° instrument would lead to large differences in reflectance data, the magnitude of the difference depending on the glossiness level of the sample [43, 44].

We considered the hypothesis that instrumental inaccuracy may be larger in our investigations than in previous investigations, because we are focusing on very dark colors. For spectrophotometers black colors may be the most difficult to obtain a good color accuracy, since they lead to a relatively low signal-to-noise ratio. We investigated if inaccuracies of the i1Pro instrument could have caused the measured color deviations.

We decided to verify the accuracy of the i1Pro with a five times more expensive high-end spectrophotometer, the spectro2guide from supplier BYK-Gardner. Like the i1Pro, the spectro2guide is a 45°/0° instrument, and also with ring illumination. It has the additional advantages of being able to also measure the gloss-level of a paint sample at 60°, and to detect fluorescence.

It is very challenging to measure the color patches of the X-Rite ColorChecker SG chart directly with a spectro2guide, since its measuring area with a diameter of 8 mm [45] is almost exactly the size of a color patch. For this reason, we first did a series of measurements on relevant paint samples with sizes large enough to enable care-free measurements.

As mentioned in the experimental section, we collected 35 samples that offer a wide range of neutral colors. Several dark up to pitch black samples are included, with a wide range of gloss levels as well. A number of brighter neutral colors were included in this investigation as well.

The measurement data for these 35 samples are summarized in Table 1. The data shows that this set of samples provides a nice selection of dark samples, many with L* values lower than 10, while also showing a variety of gloss values (at 60°) ranging between 2.0 and 93.2 Gloss Units. Therefore, this set should enable a detailed comparison between the accuracy of the i1Pro and spectro2guide instruments especially for dark samples at various gloss levels.

Table 1 Lightness values (L*) are calculated using D50 illuminant and 2° standard observer

The data of Table 1 are plotted in Fig. 4, and they show that the lightness value L* as measured by both instruments agree very well with each other. For almost all samples, the i1Pro measures an L* value that is approximately 0.6 units lighter than the value measured with the spectro2guide. Figure 4b makes it clear that the difference in L* values as measured by the two instruments does not vary systematically with lightness L*.

Fig. 4

Difference in L* values as measured by the i1Pro and spectro2guide instruments, as function of measured gloss value at 60°

Fig. 5

a Comparison between measurements of L* by two 45°/0° instruments: i1Pro and spectro2guide. The dashed line corresponds to perfect alignment of the two instruments. b Difference in L* values as measured by the two instruments, as function of measured lightness value.

On the other hand, Fig. 4b does show that atypical values for this difference in measured L* values are found for the Car Refinishes Mix & Matte samples. The reason for this becomes more clear in Fig. 5, where we show the same difference in L* values as measured by the two instruments, in this case plotted as function of measured gloss value at 60°. This shows that the Car Refinishes Mix & Matte samples do follow the same trend as for the other samples: lightness values measured by the i1Pro are generally 0.6 units lighter than the value measured with the spectro2guide, except when the measured sample has a gloss value lower than 25 gloss units. For those very matte samples, the difference may increase up to values of 1.0 or even 1.6.

These results show that the i1Pro instrument does not cause the color deviations for black patch E6 of the X-Rite ColorChecker SG chart mentioned in "The nature of the color deviations and their magnitude" section. We noticed that the i1Pro measured L* = 12.75 where X-Rite reference data mentions L* = 8.07, i.e. a difference of dL* = 4.68. Figure 5 shows that also for extremely matte colors the i1Pro is much more accurate than this. Even if patch E6 would have lost all its glossiness in time, reaching zero gloss units, this would not explain the measured color differences.

The role of the chart in creating the color deviations

By eliminating the possibility that instrumental inaccuracies cause the color deviations reported in "The nature of the color deviations and their magnitude" section, they are most likely explained by variations in the X-Rite ColorChecker SG chart itself.

However, since we find similar color deviations for charts produced at several moments in time (July 2016, June 2018, July 2019), it is unlikely that production variation can explain the measured color deviations. In that case, we would expect more variability in the color deviations. And since we find the color deviations not only charts that are 4 years old, but also for charts that are less than 1 year old, it is also unlikely that the color deviations are due to ageing of the charts or fading of its colors.

When we discussed our results with X-Rite, they were very cooperative in our search for the origin of the color deviations. X-Rite provided one of the original X-Rite ColorChecker SG charts that they had used in 2014–2015 for creating the reference data on their website [14]. With the i1Pro instrument we measured the black patches of this old X-Rite ColorChecker SG chart from 2014. This resulted in values for lightness L* varying between 8.0 and 8.87, as shown in Table 2. Only black patch C10 seems to deviate by having L* = 9.68. For patch C10 repeating the measurement with the i1Pro and also verification with the spectro2guide instrument showed consistently larger L* values than for the other black patches (as shown in Table 3). These larger L* values for patch C10 may be caused by (invisible) scratches, dust and small paper particles clustering on glue residues near the edges of the color patch, ageing due to being exposed to light etc.

Table 2 Measurements with i1Pro for L* on the ColorChecker SG chart from 2014

Table 3 Measurements with Specro2Guide for L* on two ColorChecker SG charts, produced in 2014 and in 2019

For patch E6 we find L* = 8.38 with the i1Pro. We conclude that the reference data on X-Rite’s website is correct for the original X-Rite ColorChecker SG chart from 2014, and that this data is correctly reproduced by the i1Pro instrument.

We also conclude that the color deviations for which we are seeking a solution in this article are caused by differences in color and/or glossiness in X-Rite ColorChecker SG charts produced in July 2016, June 2018 and July 2019 as compared to the version from 2014.

We investigated this further by measuring reflectance values, lightness values L* and glossiness values at 60° geometry using the spectro2guide instrument. For all black patches and all wavelengths, the reflectance values on the X-Rite ColorChecker SG chart produced in July 2019 are larger than the corresponding values on the original chart from 2014. On the original chart, all black patches have a reflectance value of 0.9% for all wavelengths. On the chart from July 2019, the reflectance is larger by 0.6% for small wavelengths, with the difference dropping to 0.1% for large wavelengths. This makes the black patches on the new chart not only lighter but also slightly more bluish than on the chart from 2014.

The difference in reflectance values between the old and the new chart may be caused by differences in glossiness. This was confirmed by measuring the Gloss value at 60° geometry using the spectro2guide instrument. Table 4 shows that for the original chart from 2014, the black patches have a gloss value that ranges from 25.7 to 27.0 Gloss Units. For the chart from July 2019, the corresponding variation is between 17.7 and 19.2 Gloss Units. These data show that in comparison to the chart from 2014, the black patches have increased in L* values by 1.7–2.9 units, while decreasing in glossiness by 7.0–9.1 Gloss Units.

Table 4 Measurements with spectro2guide for gloss on two ColorChecker SG charts, produced in 2014 and in 2019

We performed a similar analysis on the white patches of both charts. The range of L* values changed from 94.72 to 94.93 on the chart from 2014 to 95.07—95.51 on the chart from July 2019. For the glossiness, we find values ranging from 20.7 to 21.7 Gloss Units for the original chart and from 17.3 to 19.2 Gloss Units for the chart from 2019.

We conclude that in the chart from 2019 the glossiness of the black patches aligns well with those of the white patches. In the 2014 version of the chart the black and white patches were not aligned with each other in terms of gloss values. Possibly in 2014 the black patches were created with a relatively high glossiness, in order to achieve very dark L* values. We already saw in "Color charts" section that reaching darker blacks and obtaining a wider color gamut was one of the objectives when X-Rite introduced the ColorChecker SG chart. In later versions of the chart the glossiness of the black patches may have been made to align closer to those of the white patches, resulting in slightly less dark patches.

Conclusions

We have investigated the feasibility of two sets of internationally recognized guidelines for technical photography, FADGI and Metamorfoze, as far as color accuracy is concerned. In the process of camera profiling, a color accuracy of 2.0 (average) and 6.0 (maximum) is needed when following the highest level FADGI guidelines (and equivalent tolerances for Metamorfoze) [22]. Several earlier investigations showed that even when working in laboratory conditions it is difficult to meet the best color image quality level of these guidelines [7][8]. In this article, we investigate why even professional photographers, using high-end cameras and well standardized lighting conditions, often obtain camera profiles that produce color deviations that are too large to satisfy the guidelines. Only by manual tweaking of the camera profile these color deviations can sometimes be reduced to acceptable levels, but this makes the whole process time consuming and badly reproduceable.

Our results show that these problems arise predominantly for the black color patches of the X-Rite ColorChecker SG chart, which is the most commonly used target for professional photography. When we measured all color patches this chart with a spectrophotometer, our data indicated that relatively large color deviations occur already when our spectrophotometer data are compared to the spectrophotometer reference data published by X-Rite. Indeed, these color deviations occur predominantly for the black color patches, and their magnitude is large enough to be a major contributor for camera profiles to fail meeting the strictest guidelines.

Although our measurement data were created with an i1Pro, which is not a high-end spectrophotometer, we showed that the color deviations would not be solved by using a much more expensive spectrophotometer. Instead, our results show that the color deviations are caused by the X-Rite ColorChecker SG chart itself. We investigated copies of the chart that had been produced in July 2016, June 2018 and July 2019. We showed that all of them differ significantly from the version produced in 2014–2015, which was used to produce the reference data published on X-Rite’s website. The charts from 2016 and later show considerably higher values for lightness L* and lower glossiness values than the chart from 2014–2015. This completely explains the color deviations that we found when comparing our own spectrophotometer data with the reference data.

We have shown how the current version of the ColorChecker SG chart resulted from improvements over earlier versions by trying to combine low glossiness levels with an extended color gamut. In the current version of the ColorChecker SG chart this balancing act continued especially in the black patches, for which the color and glossiness have changed over the years since 2014. This makes it more difficult to satisfy the strictest tolerances implied by guidelines such as Metamorfoze and FADGI, if the X-Rite reference data from 2014 are used as a reference. In the Metamorfoze guidelines [12], for each neutral patch (which includes the black patches investigated here) the strictest maximum tolerances are dL* ≤ 2, dC* ≤ 2 and dE(CIE 1976) ≤ 2. When using the X-Rite reference data that was created with the 2014 version of the ColorChecker SG chart, these tolerances cannot even be met when using a spectrophotometer. When using a lighting set-up and a camera system, the color deviations will become even larger.

The implications of this work for attempts to meeting the FADGI guidelines [22] in combination with using X-Rite’s reference data may be less dramatic, since FADGI does not specify tolerances for individual patches. Only an average tolerance is mentioned on the total set of 140 patches of the ColorChecker SG chart. This tolerance of CIEDE2000 ≤ 2 can be met with a camera profiling system using the reference data, although also in this case part of the tolerances have already been consumed by the deviations between ColorChecker SG charts.

Our results underline the necessity to use own spectrophotometric data for specific charts, instead of generic reference data from the supplier, when the highest color accuracy is aimed for. This is in line with the recommendations by the supplier of the ColorChecker SG chart, who states that "for the highest color accuracy" it is recommended to create individually measured custom reference data [46]. The analysis in this article not only supports that recommendation, but also quantifies it.

For example, we found that when using the publicly available X-Rite reference data in combination with an X-Rite ColorChecker SG chart produced after 2014–2015, the most strict Metamorfoze guidelines can only be met if the camera profiling process is manually tweaked. In this tweaking process typically the photographer manually changes parameters in the camera profile, after they were optimized by the software. This manual process is often very time-consuming and characterized by a succession of ad hoc decisions that makes it hardly reproduceable for colleagues. The aim of this tweaking is to get a closer alignment between the generic reference data and the specific data produced by the camera profile for the individual X-Rite ColorChecker SG chart under the given lighting conditions. But since we have seen that the public reference data does not accurately describe X-Rite ColorChecker SG charts produced after 2014–2015, this manual tweaking process will not improve the actual color accuracy if the software is based on the public reference data. It is incorrect trying to optimize camera profiles by letting them reproduce reference data that are based on different black color patches than the black color patches of X-Rite ColorChecker SG charts produced after 2014–2015.

Our data show that when using the same reference data and X-Rite ColorChecker SG chart, satisfying the most strict FADGI guidelines may still be possible, although also in that case there is hardly any room left for deviations in lighting and camera set-up. Also in this case manual tweaking are better avoided.

We have shown that if the publicly available reference data are used for camera profiling, the largest color deviations occur for the black patches of the X-Rite ColorChecker SG chart. We have shown that in the digitized images, the deepest blacks would show the largest misrepresentations even on a calibrated display. On the other hand, if custom reference data is used for X-Rite ColorChecker SG charts produced after 2014–2015, the deepest blacks cannot be obtained because they are not included in the chart. In that case, subtle nuances in dark colors on the original art object may get lost in digital reproductions. This would be unfortunate in for example the photography of seventeenth century Dutch paintings, where dark nuances form an important aspect of the art works.

Therefore it is indeed recommended to create individually measured custom reference data when using an X-Rite ColorChecker SG chart produced after 2014–2015. Our data show that for these measurements it is not necessary to use high-end spectrophotometers with prices usually exceeding €10.000. Instead, instruments such as the i1Pro at a 5 times lower price range were shown to qualify. We advise photography studios that utilize more than one X-Rite ColorChecker SG chart to create separate custom reference data files for each chart. This is also important when outsourcing parts of the digital photography to external photographers. Our recommendation implies that for building camera (ICC) profiles, it is preferred to use software such as basICColor [47] and Phocus [48] that allow the user to upload custom reference data.

Since we have shown that X-Rite ColorChecker SG charts produced after 2014–2015 no longer contain the very darkest color patches, with L* = 8–9, we will investigate best ways to develop and include additional color patches, and how to apply these insights in cross-polarization photography.

Apart from that, we will also continue this investigation by developing a best practice for cases where the glossiness of the art object differs from the glossiness of the reference color patches. The current practice of camera profiling tacitly assumes that art object and X-Rite ColorChecker SG chart have the same glossiness. The current work has clearly demonstrated that even a relatively small difference in glossiness of 7 Gloss Units, on a scale that runs from 0 (dead matte) to 100 (high gloss), leads to color deviations large enough to make the profile fail the tightest guidelines, especially for black colors.

Instead of aiming for reference color patches with the same glossiness as the art object, it is also worth exploring the advantages of introducing very tight tolerances for glossiness by using highly Lambertian samples, such as the Spectralon tiles (supplier: Labsphere [49]) that are often used in hyperspectral imaging. Another option to improve color accuracy in camera profiling would be to use High Dynamic Range techniques to extend the range of contrasts that are captured, and to use color filters for introducing spectral imaging. A system based on this approach was developed at the National Gallery of Art in Washington and the Museum of Modern Art in New York [50], but this is not commercially available at this moment.

Finally, we plan to conduct a separate investigation to quantitatively evaluate in detail the improvements in color accuracy of camera profiling that result when following the recommendations from this study. For that investigation we will test the accuracy of the resulting camera profile quantitatively and independently of the colors that were used as training set, instead of using the same set of colors for training and testing the camera profile, as is common practice. We will also assess the improvements in camera profiling by investigating in detail the quality of photographic reproductions based on different camera profiles.