Identification of pigments by multispectral imaging; a flowchart method

The literature on the application of Multispectral and Hyperspectral imaging for identification of pigments on artworks is sparse. While these methods do not provide the analytical capability that spectroscopies do offer, the use of spectral imaging has the advantage of being a rapid and relatively low-cost solution for the examination of large areas. This paper presents a flowchart for the identification of historical pigments applied with gum Arabic using multispectral imaging (wavelength ranging from 360 to 1700 nm) performed with a modified digital camera for infrared, visible and ultraviolet photography; and an InGaAs camera for infrared reflectography. The flowchart method will be most successful on paint made of one layer of pure pigment, and it can selectively discriminate only a fraction of the 56 pigments analyzed. Though, considerably limited in its analytical capabilities, the low cost and speed of the workflow make the method worthwhile, even if only to localize retouching and areas appearing the same hue but painted with different pigments. The InGaAs camera is the only expensive instrument used in this study but its cost is relatively affordable for the average painting conservation studio since only a model with a low pixel count is required (320×256 pixels) rather than a more sophisticated InGaAs scanner system.


Introduction
Multispectral imaging (MSI) [1,2] and Hyperspectral Imaging [3][4][5][6], have been suggested as methods for the non-destructive identification of pigments. Though, it is mandatory to point out that these methods are problematic and the user may be subjected to draw conclusions that remain uncertain, essentially, because pigments are often mixed and overlapped in layers to make the desired color and effect.
To identify pigments with an acceptable degree of certainty, at least one other material specific technique must be used to complement hyper or multispectral imaging diagnostics. The use of MSI to tentatively identify pigments has an important advantage justifying its application: the rapid and low-cost survey of large areas. The intention of this paper is to show that with a flowchart based methodology it is possible to tentatively identify some historical pigments by means of MSI performed with simplified equipment and without the aid of imaging analysis software. This method doesn't claim to allow the identification of all different pigments, but it will work for those which present peculiar behaviors in the range of the electromagnetic spectrum readily observable with an IR-VIS-UV modified digital camera (360-1100 nm) and an InGaAs camera (900-1700 nm). In this way, selected pigments are likely to be identified by means of MSI examination. This simplified approach, though demonstrated to be limited in its analytical diagnostic capabilities, has the benefit of being accessible and easy to implement by professionals in the art conservation and examination field.
This method is more likely to succeed when applied on artworks where pigments have been applied in one single layer and not mixed; as is the case with miniatures [6,7], drawings [8] and prints. Unlike other references, which suggest the use of software algorithms to analyze the MSI images, this paper proposes a more straightforward method simply based on visual examination and the use of a photoediting software for the characterization of features apparent in the image.
There are a number of studies on the application of each of the above mentioned imaging methods specifically for the identification of pigments: UV Fluorescence (UVF) [9][10][11][12], UV Reflected (UVR) [13], Infrared False Color (IRFC) [14,15], Infrared Fluorescence [16][17][18] and Infrared Reflectography [19]. Though, there is no comparative study carried out using all of those methods, and therefore this paper intends to fill that void.

Imaging devices
The MSI images presented in this paper were acquired with a Nikon D800 DSLR (36 MP, CMOS sensor) digital camera modified for "full spectrum", ultraviolet-visibleinfrared photography (between about 360 and 1100 nm). The CMOS sensor responds both to the near infrared and near ultraviolet ranges of the spectrum, however manufacturers install an IR cut-off filter in front of the sensor to reduce infrared transmission. There are companies that will remove this filter in commercial cameras for a small fee, and then the camera is said to be "full spectrum". The camera is tethered to a computer to allow sharp focusing in non-visible modes (IR and UV) using live view mode. The MSI images created with this camera do not precisely image the same area in each waveband because it is necessary to refocus the lens at the different wavelengths. In order to allow a comparative examination through the different spectral ranges, those images are uploaded as layers of a single document file in an image editing software, such as Adobe Photoshop or GIMP, and manually resized to overlap one another. Infrared Reflectography (IRR) was performed with an InGaAs camera (320×256 pixels) Merlin NIR by Indigo Systems. Some suggested references on art documentation and examination using each spectral band and the relative instrumentation for Infrared [1,2,[20][21][22][23] and Ultraviolet [24,25] are given.

Filters
The filters chosen for use in this paper are commercially distributed for photography and so they are easy to find. Their transmission spectra are available on the manufacturers' websites, Schneider Optics for the B + W filters, Maxmax.com for the X-Nite filter and Heliopan for the infrared filter. This is the filter set used for the MSI: a) For Ultraviolet Reflected (UVR) photography, the B + W 403 filter is used together with the X-NiteCC1. B + W 403 allows just the UV light to pass, and X-NiteCC1 is necessary to stop the IR produced from the UV lamp; b) For Visible (VIS) photography, just the X-NiteCC1 filter is sufficient; c) For UV Fluorescence (UVF) photography, the B + W 420 must be mounted to stop the reflected UV, and the X-NiteCC1 is also necessary to exclude any infrared from the UV lamp; d) For Infrared (IR), Infrared Fluorescence (IRF) and Infrared Reflectography, just the Heliopan RG1000 is used.

Lenses
A Nikon Nikkor 50 mm f/1.8D AF lens was used for all the MSI photos on the pigment swatches. Standard photography lenses are generally fine for MSI. Indeed, they are transparent from 350 nm to 1700 nm. Though, it is recommended to test the lenses for hot spots in the infrared imaging. Hot spots can be caused by a number of factors, such as coatings and/or the interaction between elements within the lens and the image sensor. Lists of other lenses tested for hot spots are available online [26]. It is also recommended to use fixed focal lenses and avoid complex lenses that are likely to give flares in the infrared and ultraviolet photography. Also it is recommended to avoid telephoto lenses over 200 mm for MSI photography, so that the lens will be fast enough to work with the low intensity emission produced in the UV Fluorescence and IR Fluorescence imaging.

Lighting
Halogen lamps are used for VIS and IR photography. For UV photography (UVF and UVR) high-Flux 365 nm LED lamps are recommended. The advantages of UV LED lighting are obvious: instant start up, no heating up, very lightweight and sturdy. These LEDs are filtered to cut off any visible light with a UV-pass glass analogous to the B + W 403. This paper considers the UV Fluorescence excited by a 254 nm UV lamp (UVF254) as an additional and separate imaging method. These lamps must be used with extreme care since this UV range (UVC) is dangerous for the eyes and skin. Infrared fluorescence (IRF) requires illumination in the visible range only, which can be implemented with a white light LED lamp filtered with the X-Nite CC1 filter to cut off emission in the near-IR.

Calibration
For the camera settings and image processing of visible images it is recommended to follow normal color management procedures [2]. For the purpose of this work the American Institute of Conservation Photo Documentation (AIC PhD) target [27] was used for calibration of visible photography. The camera has been calibrated with the X-rite ColorChecker Passport and its bundled software. Due to differences in technologies and variables in manufacturing processes every camera captures colors a bit differently. Even two identical cameras from the same company are a bit different. Even if the Raw processing software could have included a profile for the specific model camera, it is possible to get even better results by creating a profile specifically for the RAW output of the particular camera. ColorChecker Passport Classic Target is the industry standard color reference target for creating DNG profiles and for evaluating specific colors. It is important to point out that the camera calibration is another step, in addition to the common white balance performed even on compact cameras. The first one corrects for different rendering of colors by different cameras, while the second one corrects for different lighting. The images were shot RAW and they were then color corrected using the camera profile above mentioned and white balanced using the N6.5 neutral grey patch in the AIC target. The grey patches are identified by the following designations (white to black): white; N8; N6.5; N5; N3.5; and black. They were also exposure corrected: N8 patch 150 +/− 5 for VIS. The same patch is also used for correcting the other images: 100 +/− 5 for IR and IRR, and 50 for UVR. On the other hand, calibration for ultraviolet fluorescence is trickier as there is not yet an officially recognized reference standard for UVF and UVF254 photography. As an aid for color balancing and in order to guarantee reproducibility of the UVF photos, the AIC PhD target was accompanied with 3 emitters activated by the LED UV and UV254 lamps: a section of a card for forensic UV photography (orange fluorescence), a swatch of zinc white (yellow fluorescence), and a fluorescent paint (green fluorescence). These 3 UV emitters together with the red fluorescence emission of the red square of the AIC PhD target itself are used for color balancing of UVF and UVF254 photos. This is performed on the RAW files editing them with Adobe Camera RaW 7.1 and assigning Temperature 15000 Kelvin and Tint 70. The images are also exposure corrected using the red fluorescence of the red patch: Red channel 70 +/− 5, Green 0, Blue 0. In order to check the correct exposure for IRF photography a swatch of cadmium red was also added to the UV emitters, and the IRF images are exposure corrected with the red cadmium swatch at RGB 30. Figure 3 shows the 8 MSI images of the AIC target together with the UV and IR fluorescence emitters. The Infrared False Color image is made by digitally editing the VIS and IR images. A copy of the VIS image is edited to become the IRFC image. The VIS green channel substitutes the blue channel and the red channel the green channel. Then, the IR image constitutes the red channel of the edited VIS.

Pigments
A collection of swatches of 56 historical pigments (Figure 4) have been applied using gum Arabic as a binder on a cellulose and cotton watercolor paper, acids and lignin free, commercialized by "Fabriano", 270 gr/m 2 . This paper is not treated with optical brighteners, it's slightly UV Fluorescent, and it reflects IR. Two cross-hair lines, 0,2 mm (vertical) and 0.4 mm (horizontal), were printed on each swatch of paper before the application of paint, in order to have a means to evaluate the pigment transparency in the IR and IRR imaging. Gum Arabic was chosen as a binder because it is the medium used in watercolor technique where pigments are more often used pure. Therefore, these are the artworks where the proposed method is likely to deliver correct identification, rather than in other painting technique, as linseed oil, were pigments are more expected to be blended and layered. The pigments were mulled into the binder which was added as needed for each pigment and applied with brush. No other means to control and measure thickness of the paint and ratio binder-pigment was implemented since this study wants to check the capabilities of this flowchart method on actual artworks where all those parameters are not likely to be easily determined. Some pigments that are not actually used with gum Arabic were included in this paper since it is planned to use the same set of pigments for future studies with other binders. Table 1  shows the list of pigments used in this study. Among all the pigments and their varieties ever used in art these 56pigments collection is not exhaustive but it attempts, at least, to be a selection of the most used ones from antiquity to early 1950'. A swatch of just gum Arabic is added as a reference for the binder.
All the pigments are distributed by Kremer Pigments and their relative information about composition and manufacturing can be found on the Kremer website [28] searching for the specified product code. The high resolution multispectral images of the pigment swatches can be accessed online [29] and browsed on an IIPImage [30] server system  that allows the user to blend between the multispectral images for immediate comparison.

Flowchart 1st step. VIS
The logic of this flowchart is to start with the most straightforward imaging methods and move toward the less specific ones. Rectangular shapes list the pigments showing the same feature such as, for example, a red UV fluorescence. If the same group of pigments shows also other properties in common, these are listed in the bottom of the same rectangular shape. The first step is to assign the pigment to one of these color categories: white, black, blue, green, yellow, brown, red. The flowchart is likely to succeed for the identification of paints made of just one pure pigment. If the paint is a mixture or a glaze, multispectral imaging loses its identification capacity. In that case, it's recommended to first perform an examination under magnification, with at least a hand-held USB microscope, to check if the paint is a mixture. Particular attention should be directed towards greens and violets. Indeed, often they are a mix of blue and yellow for the green, and red and blue for the violet.
They could also be a layered glaze, yellow over blue for the greens and red over blue for the violets.
2nd step. UVR UV light interacts just with the very surface layer of paint, so the UVR image is specific to the topmost pigment and layers of paint underneath do not influence it. Furthermore, a uniform varnish coating doesn't interfere with the pigments' appearance in the UVR but it will absorbs some UV causing just a change in the overall brightness. Strong differences in the brightness of UV reflected images such as that of lithopone and titanium white are still observable, Figure 5. Two categories are defined: bright, and dark. A pigment is defined as UVR bright if it reflects more UV than the AIC target N8 patch, i.e. its RGB > 60.

3rd step. UVF
As described for the UVR imaging, UV fluorescence comes just from the surface of the paint layer and it isn't influenced by layers underneath. Though in this case, the varnish does play a major role since it generally exhibits strong fluorescence and could overwhelm the actual fluorescence of the pigments. Consequently, MSI Figure 5 Lithopone and titanium white swatches respectively reflect and absorb UV light. The dammar varnish coating doesn't affect the UV reflected image of the two pigments. Figure 6 Prussian blue is IR absorbent. In the infrared image phthalo blue appears bright because it is IR reflective. On the other hand, Indigo appears bright because it is IR transparent. Indeed, when it is overlapped with Prussian blue it looks dark. Consequently, the infrared false color of the indigo swatch looks not the ordinary bright red but as dark as Prussian blue.
documentation is recommended when the varnish is removed from the artwork. These decision categories are defined for the UV fluorescence: none (meaning no sizeable fluorescence is observed, i.e. RGB < 30), white, blue, red, orange, yellow.

4th step. UVF 254
A 254 nm UV lamp allows the observation of fluorescence otherwise not excited by the 365 nm UV lamp discussed in the 3rd step, such as that which particularly occurs with madder lake. The same decision steps as for UVF are defined: none (meaning no sizeable fluorescence is observed), red, orange, yellow, white. Again, as for the UVF, these color categories are to be intended liberally since the perception of the color is not well defined.

5th step. IRF
Only a few pigments exhibit Infrared Fluorescence, namely cadmium based pigments and Egyptian blue, making it a highly specific method. Just two decision steps: yes (RGB > 10), or none.

6th step. IR
We can classify the behavior of each pigment in respect to the IR region as transparent, reflective or absorbent. When observing the IR image of the pigment swatches, if the pigment is IR transparent we can distinguish the carbon ink line, if it is IR reflective, the pigment will look bright and the ink line will be not visible. The pigment could also be IR absorbent and in this case the ink could not be detected since the swatch will appear pitch black. We can determine the IR behavior of a pigment with reasonable certainty only if it is applied on an IR reflective ground. Otherwise the under-paint will play a major role in the IR appearance of the above layer of paint. Note the examples in Figure 6, Prussian blue absorbs all the infrared so it is pitch dark in IR imaging. Phthalo blue and indigo both appear bright. Phthalo blue Figure 7 Cobalt-based pigments, such as cobalt blue, show an inverted behavior in the IRR. Differently than the other pigments, they become more "dark" (absorbing) in the IRR, with respect to the IR imaging. reflects IR, while indigo is transparent to the IR and therefore the white ground appears beneath. So, while phthalo blue would look bright on any ground, indigo would be bright or dark depending on the under paint. So, since we cannot know in advance if the pigment is infrared transparent, there is no way to know if its brightness or darkness in the IR image is due to the under layer. With the infrared imaging we are indeed on more uncertain ground when it comes to the identification of layered paint. Consequently, two simple categories are defined for the IR imaging: bright, dark. In order to discriminate among more pigments it is necessary to define two specific IR bright threshold values, one for the yellows (RGB > 140) and one for all the other colors (RGB > 100).

7th step. IRR
At longer IR wavelengths, such as those captured with an InGaAs camera for infrared Reflectography (IRR) most of the pigments become a bit more transparent than in the IR imaging. Cobalt-based pigments exhibit the property most useful for pigment identification as they become less transparent in the IRR, Figure 7. Looking at the change of brightness in respect to the IR imaging, two categories are defined: darker (Delta RGB > 50), and same (which encompasses the normal added transparency in the IRR range).

Figure 9
Flowchart for the identification of blue pigments by multispectral imaging.

Figure 10
Flowchart for the identification of green pigments by multispectral imaging.

8th step. IRFC
For the last step comes Infrared false color. As we saw in Figure 6, the infrared false color appearance is strongly influenced by the under layer, and as for the IR step this method is useful only for single layer paint examination. The categories are defined for the color produced as: red, pink, green, black, white, and purple.

Whites
Four white pigments have been tested and also two white minerals, gypsum and chalk, which are not properly designated as pigments because of their low hiding power, but are known to be used in art. These last ones turn grey when mixed with gum Arabic and require many applications to acquire enough opacity. Of the white pigments, only titanium white and zinc white show UV absorbance, and the latter can be identified for its yellow UV Fluorescence, Figure 8.

Blacks
The four most used carbon-based blacks were tested and, as expected, none of the imaging methods can distinguish among them.

Blues
Eleven blue pigments, including Cobalt violet, were tested. It is notable that cobalt containing pigments reflect UV (cobalt violet, cobalt blue, smalt). Also, smalt and cobalt blue turn dark in the IRR imaging, spectra are available [31], Figure 9. Natural ultramarine exhibits a characteristic white UV254 fluorescence. Figure 11 Flowchart for the identification of yellow pigments by multispectral imaging.

Figure 12
Flowchart for the identification of red pigments by multispectral imaging.

Greens
Eight green pigments were tested. Cadmium green is easily identified for its IR Fluorescence and cobalt green due to its darker IRR image, Figure 10.
Yellow Eleven yellow pigments were tested. UV Fluorescence is an intense red for Cadmium yellow, while it is yellow for Saffron and the yellow lake made of reseda. Gamboge exhibits an intense red with UV254, Figure 11.

Reds
Nine red pigments were tested, Figure 12. Cadmium red is identified for its exceptionally bright IRF, however for mineral pigments the composition is likely to be relatively consistent, and with organic lakes there could be variation in the composition; such as the case with the nine madder lakes varieties produced by Kremer (product codes 37202, 37203, 37217, 37218, 372051, 372052, 372057, 372058, 372142). They were all tested and 372051 was chosen for this flowchart since the fluorescence exhibited under UV254 illumination was the most representative when compared with the madder lakes previously documented in literature [32][33][34][35], Figure 13. Consequently, while the presence of the UVF254 fluorescence is a positive test for madder lake, its absence doesn't rule this lake out.

Browns
Five brown pigments were tested. As expected, none of the imaging methods can distinguish among them.

A case study
The Ingels Collection, Sweden, owns a Madonna and Child (painting on panel), Figure 2, which is currently being studied and has been documented with multispectral imaging by the author. The painting seems to date back to the early 14th century Sienese school, as proposed by some scholars. The lips of the Madonna are interesting to test the flowchart, see Figure 14. The lips Figure 13 Kremer Pigments produces 9 varieties of madder lake. Only one product (372051) shows the characteristic orange UVF254 Fluorescence.

Figure 14
The flowchart method helps to correlate the 7 multispectral imaging photos of the lips of the Madonna and Child panel, Ingels Collection, Sweden in attempt to arrive at pigment identification in the case of a mixture.
are retouched with a red pigment that reflects UV, has a pink UV fluorescence, is IR and IRR transparent, and IR fluorescent. Not a single pigment from the selection of reds tested here shows all these features, in particular the UVF together with the IRF. So, it seems the lips have been retouched with a mix of cadmium red, which accounts for the IRF, and madder lake for the pinkish UVF. This hypothesis seems to be supported by the micro image of an area on the lips where different red pigments are observed. Even if the pigment identification cannot be accomplished in the majority of the cases, MSI is extremely valuable to differentiate pigments of the same hue, which have been used in subsequent retouchings of an area. The mantle of the Madonna was restored at least two times using two different pigments, Figure 15. The original blue and the two blue pigments can be localized because they behave differently in the MSI images. Retouching 2 is characterized by its intense UVF which is due to its binder, while retouching 1 is differentiated from the original blue (likely azurite) because is bright in the IR.

Conclusions
This paper represents the first attempt to correlate MSI methods to achieve a preliminary pigment identification. This study focused on pigments applied with gum Arabic, which is the medium used most in watercolor, a technique where pigments are less likely to be blended and overlapped. Further work is necessary to take into account the effects of other mediums, linseed oil, tempera and acrylic, and degree of aging. It was shown that some particularly useful features such as UV Fluorescence, IR Fluorescence and the losses of infrared reflectance in the IRR imaging allow for rather reliable identification of certain pigments. On the other hand this MSI flowchart should be seen as complementary to analytical methods, in particular, elemental spectroscopies, such as X-ray Fluorescence spectroscopy. For example, on the blue pigments flowchart it is showed that Prussian blue and azurite are just distinguishable by their IRFC, which is the less reliable discriminant. Though, XRF can easily distinguish among them through the presence or absence of significant copper content.
Unfortunately, it is not always possible to perform all the 8 imaging methods to study an artwork. This is the case of paintings with the old varnish whose UVF can overwhelm that of the pigments, and the case where it is not possible to have total darkness to correctly execute IRF and UVF. In these cases, the flowchart can be still followed bypassing the missing steps, but losing, of course, part of its identification capability.
It would be valuable to compare this flowchart approach with other spectral imaging technologies [36,37], methods which are currently investigated and which use estimation techniques for visible spectrum imaging [38], and could become more powerful with the implementation of liquidcrystal tunable filters or interference filters. It would be also worth to apply the flowchart approach to other conservation studies on which Hyperspectral Imaging has proven successful such as on inks [39,40] and parchment [41].