Pigment analysis
The brilliant colours on the surface of murals are often made up of different types and proportions of pigments, which were carefully mixed and blended by artists. The ways pigments were used have obvious characteristics of the times and regions. On the other hand, lines are the basic elements of murals, which play an important role in the contours, light and dark changes and spatial composition of mural patterns. The investigation of pigments and lines in murals can provide auxiliary support for research on the origin, craftsmanship, painting style and time evolution. It is of great significance for the protection, research and restoration of murals.
Most of the long-lasting pigments used in murals are composed of mineral components. Each pigment has its own unique spectral features due to the different compositions of different pigments. Hyperspectral technology can provide image and spectral information of a target at the same time with a high spectral resolution, which can provide an approximate continuous spectral curve for each pixel in the image. All these advantages can support the scientific determination of the types of pigments in murals, the quantitative inversion of the spatial distribution and content of pigments, and the extraction of lines in murals.
Pigment endmember extraction
The endmembers of the hyperspectral image of the mural were extracted by the SISAL algorithm, which unmixes the pixels by finding the minimum volume simplex that contains hyperspectral vectors. The hinge function whose strength is controlled by a regularization operation is introduced as the soft constraint, and the results are optimized by a sequence of augmented Lagrangian algorithms. It is robust to noise and anomalies and can process large-scale hyperspectral images. We selected four endmembers according to the visual colours presented on the mural, which are the red background, yellow clothes, blue body and black line. Figure 3 shows the extracted red, yellow, blue and black endmember spectral curves.
The extracted endmember spectral data were matched with the spectrum library of pure pigments constructed by our team. SAM, SFF and the binary encoding comprehensive matching algorithm were used to determine the types of pigments. Material identification was performed by comparing the angle between the data and the endmember spectrum, by comparing the positions of the spectral absorption feature, and by using a logic function to compare each encoded data point and endmember spectrum. The pure pigment spectrum library applied was painted on a white wall and included more than 30 kinds of common ancient mural painting pigments of red, green, blue, yellow, black and white, which can be used as the benchmark data of pigments in cultural relic protection and restoration. Based on final matching of endmembers, a tentative hypothesis of identification could be that the yellow of the clothes may be orpiment. Based on the final matching of endmembers and the spectral characteristics given in [31], the red endmember of the background may be cinnabar or red ochre, which is difficult to identify the pigment from the spectrum alone. The spectrum of blue endmember is quite different from that of known blue pigment, and it is difficult to identify the type of blue pigment only from the spectrum. Wang et al. [32] investigated the pigments of murals of Qutan Temple using X-ray diffraction and isotope X-ray fluorescent. Niu et al. [33] analysed the pigments of murals of Qutan Temple by means of optical microscopy, cross sectional analysis, scanning electron microscopy with energy dispersive spectrometry, and X-ray diffraction. Their results show that blue pigments were lapis and azurite. In addition, they also pointed out that the red pigments were cinnabar and lead, and the yellow pigment is orpiment, which is partly consistent with the results identified by spectral technique.
Abundance inversion
The abundance of each pigment was inversed by the FCLS algorithm to solve the abundance of the extracted endmember by using the minimum error principle [34]. For each pixel in the abundance map of an endmember, the digital number (DN) of the pixel is generally 0–1, which represents the abundance that also denotes the content of this endmember. Finally, the eighteen hyperspectral images were mosaicked to calculate the abundance maps for red, yellow and blue endmembers, as shown in Fig. 4. This map intuitively shows the spatial content distribution of each pigment on the mural. Its value range is 0–1, which represents the content of the corresponding pigment. The greater the value is, the higher the content.
The same method was used to extract black line endmembers and calculate their abundance map. The line extraction results of the eighteen images were mosaicked, as shown in Fig. 5. In the line extraction image, the background lines were clear, and the contour lines and navel of the Buddha, which were not visible in the true colour image, were clearly visible.
Therefore, endmember extraction, spectral matching and abundance inversion algorithms can be combined to analyse the pigments used in ancient murals. They can be used to identify the types of pigments and calculate their content distribution in murals, which can provide a quantitative and scientific reference for the documentation, research and restoration of cultural relics.
Information enhancement
Hidden information refers to information that is difficult to recognize by human eyes, such as signs of repair and altered information. Information enhancement and hidden information mining of ancient murals can improve the effect of artistic expression and provide new implications for the study of ancient murals. Hyperspectral images have rich wavebands, which can highlight subtle differences in objects at different wavelengths. They include the visible light to near-infrared bands, which can help us to identify information under the cover of pigment or surface material and mine the information that is difficult to detect with the human eye [35]. These unique advantages make hyperspectral imaging technology an appropriate method for mural information enhancement and hidden information mining.
Image enhancement by continuum removal and linear stretching
First, continuum removal was used to enhance the spectral features of the preprocessed mural hyperspectral image. Continuum removal is an effective method to enhance spectral features by highlighting the absorption and reflection features of spectral curves and normalizing them into a consistent spectral background. Then, three bands with wavelengths of 640.31 nm, 549.79 nm, and 460.20 nm were selected as red, green, and blue channels to synthesize false colour images, which were linearly stretched by the histogram to realize information enhancement. Among them, histogram linear stretching is a method to improve the image quality by scaling up the brightness range of the original image to saturate both ends of the transformed image histogram. Finally, the eighteen enhanced images were mosaicked to obtain the information enhancement map. As shown in Fig. 6, the colour of lines in the background of the enhanced mural image were white, the separability between the characters and the background was higher, and the lines in the extremely soot-affected areas in the true colour image were also clearly visible. The contrast of the mural image was higher, and the detail was clearer. In addition, in the background of the enhanced mural image, the white fog-like substance that is thick in the middle and light at the bottom can also reflect the extent and degree of the soot contamination of the mural to a certain extent.
Hidden information extraction by using spectral difference
In addition, it can be found that in the true colour image taken by the ordinary digital camera, the body, neck, face, eyeballs and edges of the eyes of the last character were all blue–black, as shown in Fig. 7a. However, in the enhanced mural images, the colours of the edge of the body, the neck, the eyeball and the edge of the eyes were white and were obviously different from those of the whole body and the face of the character in Fig. 7b. In the preprocessed hyperspectral mural image, the regions of interest were extracted, and the average spectral curves were calculated. As shown in Fig. 7c, the absorption features and trends of the spectral curves of the body and face were similar, and only the reflectance between 700 and 1000 nm was different. The spectral features of the four curves of the clothing edge, the neck, the eyeballs and the edge of the eyes were similar and were obviously different from those of the spectral curves of the body and face. Therefore, the pigment used on the edge of the body may be different from the pigment used on the body and neck.
In order to investigate the differences in the use of pigments, three other Buddhas in the mural were selected and compared. Figure 8 shows the four selected Buddha regions.
As can be seen from Fig. 8, in the true colour image of the Buddha, it can be seen that the bodies of the four Buddha are blue black, the faces of the second, third and fourth Buddhas are blue black, and the ribbons of the first, second and third Buddhas are brown. However, there are some differences in the colours of the face, body and ribbon in the enhanced mural images in Fig. 8. Therefore, in order to compare whether there are different pigments in the face, body and ribbon areas of different Buddha, we select the region of interest in the preprocessed hyperspectral mural image and calculate the average spectral curve, as shown in Fig. 9.
It can be seen from Fig. 9a that for the face areas of different Buddhas, the reflectance curves of the third and fourth Buddha are similar, while the second curve is slightly different from the other two. The reflectance difference is close to 3%. However, the positions of the absorption valley near 700 nm and the trend of the curves are relatively similar. The difference in reflectance may be caused by the different brightness of the light or other noise.
For the body regions of different Buddha shown in Fig. 9b, the spectral curves of the four Buddha are highly similar. The largest difference was between the second and other curves, about 1%.
It is worth noting that for the ribbon regions of different Buddha, as shown in Fig. 8e–g, in the enhanced image, the ribbon colours of the first, second and third Buddha are relative different. In Fig. 9c, the spectral curves of the three Buddhas are also different. The maximum difference in reflectance is more than 4%. Therefore, the pigments of this part may be different.
Therefore, the hyperspectral spectral feature enhancement and image enhancement methods can improve the quality of mural images, enrich the amount of information, and enhance the interpretation and recognition effect of ancient murals. It can mine illegible information and reveal altered areas. Thus, it can be used to increase the readability and artistic expression effect of ancient murals and provide new implications for the research of ancient murals.
Virtual restoration
Because of their long history, murals are deteriorated to varying degrees due to the influence of the natural environment, such as humidity and high temperatures, and human activities, such as burning incense and worshipping Buddha in temples. With the help of image restoration methods, the deterioration of murals can be virtually restored without interfering with the current situation of murals. It is a valuable complement to documentation and actual restoration for murals. Virtual restoration can provide useful information for the actual restoration and improve the efficiency of the protection and restoration of ancient murals. Hyperspectral imaging technology provides a new possibility for virtual restoration due to its wide spectral coverage and stronger penetration ability than visible light.
Soot-affected mural image synthesis
As shown in Fig. 10a, the mural is seriously contaminated by soot, and some of the patterns are covered. The entire image is blackened, and some of the lines in the background are even illegible. In the preprocessed hyperspectral images of the mural, because the regions of interest of the red background areas were less affected by soot, the soot-affected red background and soot-affected black lines of the areas with relatively serious soot damage were extracted, and the average spectral curve was calculated. As shown in Fig. 10b, the trend and spectral feature positions of the two curves of the red background and soot-affected red background were similar. The cross occurred near the wavelength of 800 nm; that is, the effect of soot on the red background may be less near this band. After 550 nm, the difference between the two spectral curves of the soot-affected red background and soot-affected black lines increased with increasing wavelength; that is, the separability of the background and lines increased.
Figure 11 shows a true colour image (Fig. 11a) of a small area of the mural and the images with wavelengths of 405.79 nm (Fig. 11b), 605.40 nm (Fig. 11c), and 805.53 nm (Fig. 11d). The black lines on the red background were clearer in the band with a wavelength of 805.53 nm, and the black marks at the edge of the white paint loss on the right side of the character disappeared in this band.
Therefore, three bands with wavelengths of 805.53 nm, 549.79 nm and 460.20 nm were selected as the red, green and blue channels to synthesize the false colour image with the preprocessed hyperspectral image. Based on the true colour image, block histogram matching was performed on the synthesized false colour image to obtain the soot-affected mural image with clearer patterns and realistic colour.
Preliminary soot removal
Different from general degradation, soot often covers large areas of mural patterns, and the spatial distribution of soot-covered images is similar to that of foggy images.
In computer vision and computer graphics, an atmospheric scattering model is usually used to describe the formation process and principle of foggy images. Although the particles of sootiness and fog are different, they will lead to the scattering of some light by particles, and the intensity of light will be weakened when the incident light contacts the particles. The model is shown in Eq. (2).
$$I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right)$$
(2)
where \(I\left(x\right)\) is the DN of the observed image, representing the soot-affected mural image, J(x) is the image after smoke removal, which is the part to be solved in Eq. (2), t(x) is the transmittance of soot medium, and A is the scattered light value caused by soot.
The dark channel prior is a statistical rule proposed by He [36]. It was pointed out that there will be some areas, and at least one colour channel, that have some pixels whose intensities are very low and close to zero in most of the fog-free images without sky areas. The parameters of A and t(x) in atmospheric scattering model (2) can be solved by using the dark channel prior.
$${P}^{dark}\left(x\right)=\underset{z\in \Omega (x)}{\mathit{min}}(\underset{c\in (r,g,b)}{\mathit{min}}{P}^{c}(y))$$
(3)
where x is a pixel; \(c\) is a color channel among \(r\), \(g\), and \(b\); \({P}^{C}\) is the gray value of a channel of \(P\); and \(\Omega (x)\) is a local patch centered at \(x\); \(\underset{c\in (r,g,b)}{min}\) is the minimum value of each pixel in the r, g, b channel; \(\underset{z\in \Omega (x)}{min}\) is a minimum filter.
The dark channel image \({P}^{dark}\left(x\right)\) was calculated by the Eq. (3), and the maximum value of dark channel was selected as the A in the Eq. (2). According to the dark channel prior rule, the dark channel is \({P}^{dark}\to 0\) for the fog-free image. The t(x) in Eq. (2) can be figure out by Eq. (4).
$$t\left(x\right)=1-\omega \underset{c}{\mathit{min}}(\underset{y\in \Omega \left(x\right)}{\mathit{min}}(\frac{{P}^{c}\left(y\right)}{{A}^{c}}))$$
(4)
where \(\omega (0 < \omega \le 1)\) is a constant parameter to retain the perspective depth of the image.
The preliminary soot removal of the synthetic soot-covered mural image was performed by combining the false colour image, atmospheric scattering model and dark channel prior. First, the dark channel image was calculated by the false colour image, and the atmospheric light value and transmission were obtained according to the dark channel image. Second, the soot-free image was obtained from the synthetic soot-covered mural image according to the atmospheric scattering model. Finally, the brightness was adjusted to realize the preliminary removal of soot. These images were then transformed to HSV space, where the V component was multiplied by the set brightness factor to form a new V component that was used to perform inversive HSV transformation to obtain the image with adjusted brightness. As shown in Fig. 12, compared with the original image, the influence of soot on the mural image after preliminary soot removal was reduced, the details were highlighted, and the black lines in the red background were clearer.
Inpainting of paint loss
There are a number of damaged areas due to paint loss in the background of the mural, which causes the exposure of the white wall at the bottom. To further improve the visual effect of the mural, an image inpainting algorithm named Criminisi algorithm was used to restore the paint loss. The Criminisi algorithm is a patch-based image inpainting algorithm that can synchronously utilize the texture and structure information in the image to better realize the filling of the target area [37]. Firstly, the pixels of the area to be inpainted were masked, and the priority of the patches at the edge of the masked area was calculated to find the image patch with the highest priority. Then this patch was replaced by an optimal target patch that was searched in the whole image under the similarity criterion. Final, the remaining masked area and the corresponding confidence priority and data priority were updated. And the next image patch with the highest priority would be filled in the same way. This process was repeated until all pixel blocks were repaired. Before using Criminisi algorithm to inpaint the image, we need to know the areas to be restored. Therefore, we proposed a method to locate the paint loss areas by using support vector machine (SVM). First, the figure region was distinguished from the background region in the mural for the paint loss regions are mainly located in the background region. Another reason is that the characters in the murals are very delicate and rich in colour, which may affect the accuracy of the extracted areas and the effect to be inpainted. The image was masked as Fig. 13a. Second, the regions of interest of each colour, mask area and paint loss area in the image are selected as the training data, as shown in Fig. 13b. The SVM classification method [38] was used to classify the masked background region to several classes including the paint loss areas, as shown in Fig. 13c. Third, in order to make the extraction area completely cover the deteriorated area, the dilation operation in morphological filtering was performed three times to expand the original extracted areas. The final extracted paint loss areas are shown in Fig. 13d.
Finally, the Criminisi algorithm was used to inpaint the paint loss areas in the image after the preliminary soot removal. As shown in Fig. 14, from the perspective of visual effects, most of the white walls in the mural appeared after the paint loss was repaired, making the whole image more coherent.
Virtual restoration of soot-covered mural images
The Retinex method considers the object brightness perceived by the human eyes as a combination of the illumination of the environment and the reflection of the object surface [39]. The illumination component can be estimated from the original image to obtain the reflection component, that is, to obtain the colour of the object itself. It was shown as Eq. (5):
$$I\left(x,y\right)=R\left(x,y\right)\cdot L\left(x,y\right)$$
(5)
where \(I\left(x,y\right)\) is the DN of the image, \(L\left(x,y\right)\) is the component of ambient light, and \(R\left(x,y\right)\) is the reflectance image of the mural, which is the result image after inpainted.
To further restore the soot-affected mural image, according to the Retinex method, two bilateral filters with different weights and parameters were set to solve the illumination and reflection of the image after the inpainting of paint loss to realize the virtual restoration of the mural image. Among them, the bilateral filter is a kind of nonlinear filter that can consider spatial information and grey similarity at the same time and can better achieve the purpose of edge preservation and denoising [40]. Finally, the eighteen restored images were mosaicked, and the virtual restoration map was obtained. From a visual point of view, as shown in Fig. 15, the restored image basically eliminated the influence of soot on the mural content and repaired the paint loss in the background. It was clear and coherent; hence, the method largely restored the original appearance of the mural.
Therefore, using the advantages of hyperspectral imaging as well as relevant defogging methods and image inpainting algorithms, it is possible to restore the mural image blurred by soot and repair the paint loss damage in the background. This method is more applicable to mural with light soot and small paint loss area. The restoration of murals suffered by serious soot (basically invisible) or large areas of paint loss would be further investigated. Nevertheless, this method can still help to promote the artistic expression of ancient murals, improve the circulation ability of online exhibitions so that they are available to more audiences. It can also provide valuable guidance for mural restoration.
