Tensor decomposition for painting analysis. Part 2: spatio-temporal simulation

Abstract

In a previous article, we modelled the spectral and temporal dimensions of the photodegradation behaviour of pigments in the painting “A Japanese Lantern” by Oda Krohg. In particular, we extracted the endmembers and spectral fading rate of pigments by applying tensor decomposition on a time-series of spectroscopic point measurements. Now, we capture the same painting with a hyperspectral imaging setup and propose an approach to render the fading effects as 2D images. More precisely, from the hyperspectral image, we compute the concentration maps of each previously identified endmember with a least-squares unmixing method. Subsequently, by using tensor algebra, we multiply the concentration maps with the endmembers and their corresponding fading rate and obtain a 4D tensor where each pixel in the image is described by a spectrum and a fading function. This way, we generate past and future spatio-temporal simulations of the painting’s appearance by reversing and elevating light exposure, respectively.

Introduction

A wide range of materials manifests light-induced appearance changes. To name only a few: wood [1], paper [2], dyed textiles [3], pigments [4]. For this reason, it is of general interest to create models that are able to predict future degradation as a function of light exposure. In the field of cultural heritage, for older paintings where damage has already occurred, there is also an interest to reverse these changes, either through physical restoration or through digital image processing techniques. The latter are non-invasive methods because there is no intervention on the real object, and they offer a good playground for the implementation of various scenarios of degradation. Thus, digital rejuvenation has been employed for the reconstruction of Georges Seurat’s “La Grande Jatte” [5], van Huysum’s “Flowers in an Urn” [6], Vincent van Gogh’s “Field with Irises near Arles” [7] and other drawings and paintings by van Gogh [4]. In some of the previous works, the proposed solutions are based on physical-based models such as Kubelka–Munk that considers the absorption and scattering of pigments, and the non-linear mixing of the various pictorial layers [2, 5, 7]. Other approaches are hybrid, combining physical models with data-driven methods [4]. There are also purely data-driven approaches, where analytical data and measurements of accelerated aging are combined with linear regression methods to virtually restore and/or age an artwork [7, 8].

Our novel approach for spatio-temporal simulation of paintings is also a data-driven method. In a previous article [10], we showed how from a set of microfading measurements, we created a tensor decomposition model and extracted the spectral curve of the pure pigments, together with their temporal evolution. Now, we link the loadings of the tensor decomposition model with a hyperspectral capture of the same scene. More specifically, we unmix the hyperspectral image with the endmembers previously obtained by minimizing the least square error under a linear mixing assumption. Once the concentration maps are retrieved, backward and forward spatio-temporal simulations are proposed.

Related work

Microfading analysis

Microfading refers to the accelerated aging process, where gradual light-induced changes in color and spectra are gathered. While accelerated photodegradation of pigments has been studied long before [11, 12], usually on mockups, it was not until the introduction of microfading that such experiments were performed on real objects. The main advantage of microfading is its minimal invasiveness. Light is cast over an area so small, that the fading effect is almost unperceivable by the naked eye, making this technique appropriate for the measurement of real artworks. Since the pioneering work of Whitmore et al. [13], who was the first to design a microfading setup, several other systems with increased spectral accuracy and higher portability for in-situ handling have been proposed [14,15,16]. As a consequence, microfading has enabled the color degradation analysis of a series of artworks from museum collections [17, 18] and even outdoor rock art specimens [19]. In many cases, this has determined museums to improve the light policy and better control the display time of sensitive artworks, such as Islamic Ardabil carpets [3], natural history artifacts [20], paintings [18], and heterogeneous collections including prints, watercolors, curtains [21].

Mapping of photodegradation

The point-based specificity of microfading experiments poses a challenge for the holistic visualization of the light-induced damage for an artwork’s entire surface. The lack of ground-truth regarding the aging of the artwork’s surface adds to the complexity of the task. There are several publications with a rather artistic approach to this challenge, where data from accelerated aging is used in combination with image manipulation software such as Photoshop to simulate faded variants of artworks in the color domain [22,23,24]. For instance, Morris and Whitmore [22] collect microfading data of mockups that include different painting media. They transform the spectroscopic data to color coordinates, that are later fed to Photoshop to create uniformly colored swatches where the evolution of the photo-degradation becomes easier to visualize. Moreover, by using the same data and procedure, the authors render fading effects for images with variation in the spatial content. In particular, the spatial structure is taken from scans of real paintings. To avoid damage, the paintings are not measured with the microfadeometer and so, the color behaviour is borrowed from the mockups’ measurement. In addition, the authors automate the spatial rendering process in a scientific software, where a look-up table is created from the fading measurements of the mockup, connecting a color group with its faded variants at all light dosages. Hence, when the simulation at a certain dosage is queried, the closest corresponding color in the look-up table is retrieved. However, no interpolation method is applied to fill in the gaps between unmeasured values. Hendriks et al. [23] generated forecasts of the red and yellow colors in van Gogh’s “The Bedroom”, given a light exposure of up to 30 Mlux hr. The authors incorporated microfading data from aged mockups of red lakes and chrome yellow paints and based on these, altered the image of the painting in Photoshop. In a similar way, Brokerhof et al. [24] made a prognosis of color changes for a collection of Dutch city maps from the 17th century, by running various light fading scenarios in the same image editing software. While in these works the fading experiments allowed for high dosages of light, this might not be possible when the analysis is performed on real artifacts. In these latter cases, the future change could be predicted from the set of measured data using linear regression [9] or time-series models [25].

Riutort-Mayol et al. [8] proposed an interpolation method based on a multivariate Gaussian process, that correlates the set of microfaded points with a trichromatic image of the same scene. The Gaussian process considers the covariance between the faded spots and the pixels in color image based on their similarity in the HSV (hue, saturation, value) color space and spatial proximity. This way, the color fading values are extrapolated to the entire image, facilitating spatio-temporal analysis. Moreover, the sign of the partial derivative is used to ensure that the temporal change function is monotonically increasing. This method was applied to predict color changes for Spanish Levantine rock art paintings [19]. While potentially, given a hyperspectral image, the Gaussian process interpolation could be extended to the spectral domain as well, no such feasibility prospects were discussed in the papers [8] and [19].

Thomas et al. [9] suggested an interdisciplinary approach for the spatial mapping of the color degradation in “The Scream” painting in the National Museum of Norway collection. The pigments in the artwork had been previously analysed with, among other methods, X-ray fluorescence, and test points for microfading were chosen based on previous test points. Microfading data were combined with knowledge of the paintings’ support, pigments and surface layer to create digital simulations of the painting after 10 dose values ranging from 0.5 to 25 MLux hr. Moreover, unrealistic colors were eliminated based on color rendering index values. It was pointed out that it is reasonable to extrapolate up to 3 times the applied dose. If we assume that 1.5 Mlux hr were applied, this allows for projections up to 4.5 Mlux hr. Beyond this limit, the authors claim that the model performs less well. For many artworks, simulations such as these may not be possible, as they require a combination of different methods of analysis and data processing that may not be viable to carry out for most objects.

Method

Figure 1 displays the workflow of our approach. The first and core module is represented by the tensor decomposition model (thoroughly described in Part 1 of this two-article series) [10]. This model takes as input a collection of microfaded samples, and employs parallel factor analysis (PARAFAC) to find the spectra of the unmixed pigments (endmembers), their concentration in each sample and their fading rate. In the second module, we capture a hyperspectral image of the same scene to obtain the concentration of the endmembers for the full spatial dimension, beyond the microfaded points. Then, we can reconstruct each pixel in the image as a tensor product between the concentration, endmembers and the fading rate. This way, we generate spatio-temporal simulations of the whole surface, for the time steps included in the model. In the third module, we go beyond the measured time steps, by applying linear regression on the fading rates. As a result, the fading rate of each endmember is characterized by two coefficients (slope and intercept) and a given time step. Hence, fading rates for past and future moments can be computed. In tensor algebra, these new fading rates can be multiplied with the endmember and their concentration maps to render the analyzed surface backwards and forward in time.

Fig. 1

The diagram of our method. The module (1) is the core model, where we extract the endmembers and their fading rate with three-way tensor decomposition from a collection of microfading observations. Then we perform spatio-temporal fading simulation by capturing a hyperspectral image of the same microfaded scene (2). We unmix the hyperspectral image to get the concentration map for each endmember, and then recompose it as a tensor product with the fading rate for each of the modelled time steps. Finally, we extend the time span of the spatial simulations by extrapolating the fading rate for time steps not included in the trained model (3)

Spatio-temporal modelling

For the sake of brevity, we will not insist here on the tensor decomposition model as it was described in the “Method” section of Part 1 of this article series [10]. Thus, we take for granted that matrix $C$ represents the endmembers, $A$ the concentration of each endmember $f = {\left\{1 \ldots F\right\}}$ in all the input samples and $B$ the fading rate for each endmember.

Let us consider a hyperspectral image $H^{I \cdot J \cdot U}$ of the same scene from which the microfading samples were collected. Instead of sampling the scene at only few locations as the microfadeometer does, the hyperspectral image measures the scene holistically, where for every pixel at location $i,j$ with $i = {\left\{1 \ldots I\right\}}$ and $j = {\left\{1 \ldots J\right\}}$ its reflectance spectrum is recorded with a dense bandwidth for a total of $U$ bands. Given the endmembers defined by the loadings of factor $C$ of the trained PARAFAC model, we can unmix the hyperspectral image assuming the same endmembers. The unmixing is formulated as a least-square optimization problem, where the linear combination of the $F$ endmembers that best explains the image with the minimum sum of squared residuals is chosen. The result of the unmixing is given by the abundance maps, which are essentially the concentration of each pure spectrum defined at pixel level. By flattening the abundance maps $A_{map}$ into a matrix of cardinality $IJ \cdot F$, we can then replace it in Eq. 2 of Part 1 [10] and obtain spatial simulations of fading for each $k$ time step modelled with PARAFAC:

$$\begin{aligned} H_{sim}^{IJ \cdot KN} = A_{map}^{IJ \cdot F} \times (C^{N\cdot F} \otimes B^{K\cdot F}) ^T \end{aligned}$$

(1)

Then we can reshape $H_{sim}^{IJ \cdot KN}$ to $I \cdot J \cdot K \cdot N$ to access the fading simulations of the hyperspectral image at each time step $k$.

The loadings of the 2nd mode, $B$ define the fading rate of every endmember. In Part 1 [10], Eq. 3 we showed how the fading rate can be expressed as a linear function of the light exposure. By replacing $k$ in that equation with values lower than 1 or higher than $K$ (the total numbers of steps included in the model), we can estimate new values for the fading rate $R_{f}$ backward or forward in time. These new values can then substitute $B$ in Eq. 1 to get the spatial simulations for time steps other than those measured. It is important to note that while from a mathematical standpoint the backward prediction holds, from a physical perspective, it is not entirely legitimate, as the modelling was achieved with data describing only the present and future state of the artwork. Thus, in this paper, we present a rather mathematically valid modelling of the past appearance.

Data collection and results

Case study: Oda Krohg’s painting “A Japanese Lantern (By the Christiania Fjord)”

In this article as well as its prequel, Part 1 [10], we show the performance of our method on the painting “A Japanese Lantern” created by the Norwegian painter Oda Krohg in 1886 and present in the collection of the National Museum of Norway (inventory number NG.M.00879). The painting is made on canvas, and with pastel media. The highlight of the painting, as indicated by its title, is the Japanese lantern in the top central part (see Fig. 2). The lantern is interesting from a scientific point as well, because it has an intricate color composition. For this reason, in this two-part series of articles, the lantern was chosen as the region of interest to characterize the reflectance and lightfastness of the constituent pigments, based on microfading spectrometry. In this article, we present an approach to visualize the fading effects for the entire surface of the lantern.

Fig. 2

Locations measured with MFT on the central lantern. The samples correspond to five colour groups: pink (P), red (R), green (G), orange (O), violet blue (V). Courtesy of Børre Høstland, National Museum

Microfading analysis

In Part 1 [10] of this two-series article, we explained how we collected a set of measurements (see Fig. 2) from a fragment of the painting, i.e. the lantern in the top central part, with a portable microfadeometer (MFT) [26]. The temporal evolution of the color coordinates shows an overview of the color degradation mechanisms. As it emerges from the plots in Fig. 3, the tendency is for the colors to darken (the negative change of lightness) and desaturate (negative change of chroma). The desaturation effect is evident for the pink, red and orange samples. However, in the case of the green samples, it seems that chroma remains stable. Similarly, the chroma of the violet samples has a rather small shift in comparison to the red, pink and orange. In addition, V1 saturates, while V2 desaturates. This contrasting trend is probably related to the underlying materials that albeit similar in color, have different spectral composition.

Fig. 3

Alteration of L*, C* coordinates for the microfaded samples in the central lantern. The black square marks the initial value, before fading, while the diamond marks the final value, after fading. The lightness of all samples decreases after fading. While the red, pink and orange sample desaturate, the chroma of the green samples keeps constant. Interestingly, the two violet samples have opposite behaviours, where chroma increases for V1, whereas it decreases for V2

Hyperspectral image capture

The painting was imaged with the HySpex VNIR-1800 [27]. HySpex VNIR-1800 is a pushbroom hyperspectral scanner, with a CMOS sensor that records the spectral response every 3.2 nm between 400 and 1000 nm for a line of 1800 pixels at a time. In order to acquire a full surface, either the camera or the object needs to be moved so that sequential frames are acquired. This results in a 3D data array with size equal to (number of frames) * 1800 pixels * 186 spectral bands. In this case, the camera was set on a rotational stage, parallel to the painting that was held in a vertical position with a motorized easel, as shown in Fig. 4. A lens focusing at 1 m was mounted on the camera, resulting in a pixel size equivalent to that of 0.2 mm. The distance between the camera and the painting was approximately 1 m. To maximize the signal, two halogen studio lamps were placed on the left and right side of the camera, at 45$^{\circ }$ with respect to the painting. To account for the light non-uniformity of the captured line and to obtain the reflectance factor, a standard gray target with known reflectance of approximately 60% was placed alongside the painting. Each pixel in the captured line is corrected for the dark current noise. Then, to get the reflectance factor, the pixels in the region of interest are divided by the response of the gray target and multiplied by the known reflectance of the target. Similar to the microfaded data, the hyperspectral signals were smoothed with a Savitzky–Golay filter [28] of order 2 and window size 17 to reduce the noise.

Fig. 4

In-situ setup for hyperspectral imaging

Due to various constraints related to the in-situ measurements and museum logistics, the microfading was performed before the hyperspectral imaging. Moreover, at the time of the imaging campaign, the painting was reframed, whereas it was without frame during the microfading experiment. The framing includes a thin sheet of Optium acrylic [29] overlaid on the painting. The acrylic sheet blocks 99% of the UV light, is anti-reflective and is transparent, so it should have minimal to no impact on the imaged signal of the painting’s reflectance in the visible range. It could be implied that during the gap in-between the two measurements, some intermediate fading might have occurred. Nonetheless, the painting was not on display during this period, so there should have been very little light exposure mostly due to the reframing after the microfading experiment, packing, and setting up for the hyperspectral scanning.

Exploratory analysis

In order to get a quick understanding of the hyperspectral image, principal component analysis (PCA) was computed to compress and visualize the significant variation in the spectral data for the full spectral range (400–1000 nm). Before PCA was applied, the data was normalized along the spectral dimension with the standard normal variate preprocessing technique [30], to reduce the influence of the spectral signal’s intensity in favour of its shape. The data was split into 10 principal components, where the components are sorted descendingly by the amount of data variance they explain. By nature, principal components (PC) are orthogonal, and they could suggest distinct materials in the data. However, it is difficult to give chemical meaning to the spectra of the PC especially since the mean-centering in pre-processing implies loadings with negative values. Fig. 5a, c display false color visualizations of the first 6 principal components in the central lantern, where we can see a spatial distribution of different materials. We can parallel these distributions with the true color rendering of the central lantern (Fig. 5b) to attribute more meaning to the components. For example, the red in Fig. 5a seems to correspond to the yellow areas in the true color image, while green segments the violet-blue strokes and cyan covers the pink regions. Moreover, the false green color in Fig. 5c delineates stripes that overlap with the false red color regions in Fig. 5a. Actually, considering the known fact that pastels are usually applied in layers, we could hypothesize that PCA manages to separate some of the layers in the painting, even though it is difficulty to quantify the order. Nonetheless, if we carefully inspect the true color image, we can notice areas of yellow covered with green in the top left and right sides of the lantern.

Fig. 5

True color visualization of the central lantern based on the hyperspectral image (b), together with PCA false color renderings (a, c). a R, G, B correspond to: PC1—45.23% retained variance, PC2—36.55% variance, PC3—7.3% variance. b R, G, B correspond to the most informative spectral bands: 650 nm, 551 nm and 470 nm. c R, G, B correspond to: PC5—2.47% retained variance, PC6—10.97% variance, PC4—5.53% variance

Spatio-temporal spectral fading simulation

One of the advantages given by our tensor decomposition model and facilitated by the multi-modal acquisition, is the possibility to spatially map the temporal changes triggered by fading. For generating spatio-temporal simulations, we first recover the concentration maps of the endmembers (factor $C$ loadings extracted with the PARAFAC model, displayed in Fig. 6b) in the hyperspectral image using least-squares method. We enforce the non-negativity constraint so that we get realistic, positive concentrations. Also, we interpolate the spectral sampling of the hyperspectral image to match that of the microfading data. The resulting abundance maps have the same spatial dimension as the input hyperspectral image, 736 * 790 (height * width), and are shown in Fig. 7. A clustering effect can be noticed, where the same endmember is present in contiguous and adjacent regions, which indicates that the abundance maps are realistic. The most spatially predominant pigment is endmember 1, while the least extensive is endmember 6. The uniformity of endmember 1 seems to confirm the likelihood of our previously mentioned layering theory, suggesting the presence of a common bottom red layer applied on the central lantern before the other colors.

Fig. 6

Loadings of the two factors (B, C) in our tensor decomposition model, fitted for 6 components

Fig. 7

Abundance maps, estimated with the least-square method, of endmembers 1–6 (previously extracted with the tensor decomposition model from the microfading data) in the hyperspectral image of the central lantern. Endmember 1 seems to be the pigment most uniformly distributed. This hints to its application as an underneath layer, beneath the rest of the colors

The abundance maps with size 736 * 790 * 6 are then flattened to 581,440 * 6 arrays so that they can be combined with the fading rate and endmember loadings (see Fig. 6) using the Khatri-Rao product, as formulated in Eq. 1. The result is a 2D matrix with dimensions 581,440 * 648, where the second dimension corresponds to the number of time steps (8) multiplied by the number of wavelengths (81). This is then reshaped to a 4D array with size 736 * 790 * 8 * 81, that contains the spectral simulations along the 81 wavelengths for the 8 time steps, spatially distributed for the entire surface of the central lantern. We then separate the images at each temporal slice, and transform them to CIE L*a*b* using CIE 1931 2$^{\circ }$ standard observer and D65 standard illuminant. This way, we can check whether the degradation pattern of the spatio-temporal simulations fits with the findings of the colorimetric analysis in “Microfading analysis” section.

Figure 8 shows the $\Delta E_{00}$ difference between the spatio-temporal simulations at time step 8 and time step 1. We would expect that the maximum difference is 2 $\Delta E_{00}$ units, since this is the span of the microfading data used to train the tensor decomposition model. While, indeed, the range is mostly below 2 $\Delta E_{00}$ units, the maximum difference reaches 2.5 units for few isolated groups of pixels. This can be caused by a number of factors: the imperfections of our model in finding all the endmembers; the least-square fitting error in the estimation of the abundance maps; the setup differences between the microfading and hyperspectral measurements that might generate a difference of scale when computing the abundance maps. In addition, we graphically represented the subtraction of the color coordinates L* and C* between the simulations at time step 1 and time step 8. Hence, in Fig. 9 the false colors red and green stand for positive and negative change, respectively. The previous colorimetric analysis (see  “Microfading analysis” section) revealed a negative trend for the L* and C* coordinates for almost all of the single point measurements. This trend seems to be preserved for the spatial simulations as well. Nonetheless, we can notice small red areas in Fig. 9, indicating a positive change, which is contrary to the colorimetric analysis of the individual microfaded samples. This may be explained by the fact that those areas in the hyperspectral image simulations that change positively in the L* and C* coordinates were not measured with the microfadeometer. Actually, this raises one potential limitation of our spatio-temporal simulation model: if the microfading experiment doesn’t sample the full range of the materials spatially present, then the performance and accuracy of the model might be affected.

Fig. 8

$\Delta E_{00}$ between the spatio-temporal simulations at time step 1 and time step 8, where the exposure is equivalent to 0.027 Mlux hr

Fig. 9

Comparison of the CIE L*, C* coordinates between the spatio-temporal simulations at time step 8 (equivalent to 77 s of light exposure) and time step 1. Red depicts a positive change, while green a negative change. The negative difference prevails, which is in agreement with the colorimetric analysis of the microfading data (“Exploratory analysis” section)

We know that the hyperspectral image was captured 1 year after the microfading experiment. In this period, the painting was not on display. Hence, the exposure should have been minimal, mostly occurring during the reframing process after the microfading data collection and during the hyperspectral scanning. By comparing the spatio-temporal simulations with the hyperspectral image, it is possible to estimate the amount of light exposure in-between the two measurements. Therefore, we computed the $\Delta E_{00}$ and root mean square error (RMSE) between the hyperspectral image and the spatio-temporal simulations for time steps 1–8. According to both metrics, the simulation at time step 2 is the one with the highest colorimetric and spectral similarity with the original image, as shown in Fig. 10. This implies that in the gap between the two acquisitions, the painting was exposed for approximately 0.0385 Mlux hr. However, there are a number of factors other than light exposure that may influence the precision of this result. For instance, there were a number of differences between the microfading data collection and hyperspectral imaging: the state of the painting (without and with the acrylic sheet), illumination (LED vs. halogen) and acquisition geometry. In addition, the neutral reference tiles used to calibrate the light sources for the two instruments were different. Moreover, there is the problem of unidentifiable scale of the endmembers extracted with the parallel factor analysis (see “Tensor decomposition with parallel factor analysis” section in Part 1 [10] prequel). To account for these variations in magnitude, before computing the difference between the simulations with respect to the hyperspectral image, we applied normalization in each case by division with the maximum values.

Fig. 10

The spatio-temporal simulation at time step 2 is most similar to the hyperspectral image, both colorimetrically and spectrally, with an average $\Delta E_{00}$ of 0.86 and an average RMSE of 0.13. This indicates that light-induced aging happened in-between the two measurements (microfading and hyperspectral). However, there are other factors, mainly related to the differences between the two acquisitions setups, that could influence the accuracy of this result

Rendering of past and future appearance

In a previous article we derived the linear approximations of the fading rates characteristic to each endmember and estimated the slope and intercept, (see Eq. 3 and Table 1 of Part 1) [10]. In addition, the goodness of the linear approximations was proved based on a test dataset. To visually acknowledge the effect of reversed and increased fading, we computed new fading rates for $k = {\left\{-31 \ldots 0\right\}}$ and $k = {\left\{9 \ldots 32\right\}}$, which together with the measured $k = {\left\{1 \ldots 8\right\}}$, cover a light exposure range of $\pm 341$ seconds or $\pm 1.19$ Mlux hr. The number of 32 time steps was inspired from the analysis of future modelling of the point measurements in Part 1 [10], section “Data collection and results”, where the reconstruction error was showed to increase proportionally with the extent of the temporal range. For this reason, we chose a moderate value as a way to cap the amount of error and at the same time, achieve the visible effect of aging. Then, we input the new fading rates in Eq. 1 and created a total of 64 hyperspectral images for $k = {\left\{-31 \ldots 32\right\}}$. In Fig. 11, we show sRGB renderings (based on the CIE L*, a*, b* coordinates integrated for D65 illuminant and 2$^{\circ }$ observer) of the past, present and future appearance of the Japanese lantern. The past, simulated for the smallest k can be considered a digital rejuvenation and shows a brighter and more colorful appearance than the consequent renderings. Indeed, the future appearance (Fig. 11c) looks darker and less colorful. This means that the color degradation trend, previously discussed in the case of a sparser spatial sampling (Fig. 3) or for a more confined period of time (Fig. 9) is consistent for bigger cutouts in the spatio-temporal volume. The animation in Additional file 1 portrays a more gradual visualization of the change, that takes into account the intermediate time steps between the past and future simulations in Fig. 11. We chose these boundaries for the light exposure $k = {\left\{-31 \ldots 32\right\}}$ because they are sufficient to show the aging process. Mathematically, we have no clear boundaries and potentially, we could generate simulations far beyond this range. However, from a physical point of view, simulations might get unrealistic beyond a certain threshold. Defining this threshold requires precise knowledge about long-term kinetics of the pigments and a full monitoring of the painting’s storage and display conditions, and it is out of scope for our current work.

Fig. 11

Spatio-temporal sRGB renderings of the lantern, for D65 illuminant and 2$^{\circ }$ observer. For simulating the past, we reverse the light exposure, up to − 1.19 Mlux hr (equivalent to − 341 s or k = − 31 time steps). Present refers to 0 Mlux hr (k = 1) and future to + 1.19 Mlux hr (k = 32 or + 341 s). We can see that the overall appearance turns darker due to photodegradation. Also we can notice the shift to greener hues of the orange colours in the center (corresponding to points O1, O2, O3 in the microfaded samples). For a more gradual display of the change, for intermediate time steps, please watch the animation in the Additional file 1

Every 2D rendering for a certain time step has a full spectral representation. This facilitates the visualization of the degradation for particular wavelengths. For instance, Fig. 12 shows the difference between the rejuvenation (− 1.19 Mlux hr) and aged (+ 1.19 Mlux hr) versions of the lantern, in RGB composite images. Each R, G, B channel corresponds to the most representative spectral bands in the long-wave, mid-wave and short-wave parts of the visible electromagnetic spectrum according to the correlation coefficient metric. These informative bands are 650 nm, 551 nm and 470 nm. By visualizing the difference hyperspectral image between past and future, we can see which wavelengths change most for certain areas in the lantern. For example, in the areas surrounding the O1 and O2 microfaded points, we can see the predominant color in Fig. 12 is yellow, meaning that the most significant shift happens in G and R channels, corresponding to 551 nm and 650 nm. A full animation highlighting the photodegradation as the per-wavelength difference between the rejuvenation and aged simulations is included in Additional file 2.

Fig. 12

RGB image of the difference between past and future simulation of the lantern. The R, G, B channels correspond to the most informative spectral bands in the ranges 600–700 nm, 500–600 nm, 400–500 nm, namely 650 nm, 551 nm and 470 nm. The contrast of the image is stretched for visualization purposes

Discussion

To summarize our results, we showed how the fusion of microfading spectrometry with reflectance image spectrometry can be useful to the analysis of an artwork on various levels. First, both acquisition methods enable informative preliminary analysis. Microfading gives us an overview of the color degradation of the materials, while the spatio-spectral dimension of the hyperspectral imaging enables principal component analysis that shows a distribution of probably distinct materials, even though the components have low interpretability with regard to the chemical meaningfulness. However, these techniques alone have a number of limitations. Microfading doesn’t have a spatial component making it a difficult task to extrapolate the degradation behaviour to other points on a surface. On the other hand, hyperspectral imaging, while having a good resolution in the spatial and spectral dimensions contains little information about the fading mechanisms of the pigments. Also, while pigment classification and unmixing techniques can be applied, hyperspectral imaging has a limited reach beyond the surface layer of a painting. In our case, the hyperspectral system is sensitive in the near-infrared region, and able to see through several pigments that are transparent in the near-infrared. In addition, we can argue that the microfading, based on an accelerated aging procedure, can reach as well layers underneath the superficial pictorial layer. In other words, through fading, some components disappear, uncovering pigments from underneath layers. Seeing beyond the surface turns out to be a useful property when it comes to pigment unmixing.

Given all these considerations, by fusing microfading data with hyperspectral image analysis we get a more holistic representation of an artwork and its constitutive pigments together with their fading mechanisms. In other words, we get a spatio-spectro-temporal representation.

Towards future validation of our approach, we intend to conduct psycho-physical studies where more experts could assess if the spatio-temporal simulations look realistic. Moreover, because the tensor decomposition method is sensitive to the training data, we are aware that with a limited sampling of the microfading observations, some pure pigments might be disregarded, which in turn affects the spatial mapping. This could be accounted for with a more thorough sampling during the microfading experiment. In addition, the tensor decomposition method is ultimately a linear unmixing method, which might not capture all the non-linear mixing mechanisms in the current pastel painting. Because this has an impact on the spatial representation, non-linear models such as Kubelka–Munk are under consideration for further evaluation. Finally, although we show results for a single case study, our method can be applied on any other type of colorant (ink, dyes, etc.) and artworks given a set of overlapping microfading and hyperspectral measurements.

Conclusion

In this study, we elaborated a method that combines microfading spectrometry with hyperspectral image analysis towards spatio-temporal simulations of an artwork. The current approach builds on our previous work, where we distilled a time-series of point spectroscopic measurements into the spectral signals of pure pigments and their fading functions. In this follow-up article, by coupling the basis factors recovered beforehand with a hyperspectral image of the same scene, we are able to render the appearance of the artwork by modulating the amount of light exposure. As a result, we can undo the fading and achieve a digital rejuvenation. Similarly, we can simulate the future photodegradation by virtually increasing the light exposure. Our proposed spatio-temporal simulations have a full spectral representation, meaning that they can be rendered for specific wavelengths, as well as adapted to the color response of various illuminants.