Application of hyperspectral imaging technology to digitally protect murals in the Qutan temple

Abstract

Hyperspectral imaging technology is a research hotspot in the field of cultural heritage protection. It can be used to quickly and noninvasively obtain detailed spectral information from the surfaces of cultural relics of different categories. We can intuitively analyse pigment compositions, line characteristics, painting skills and patterns using spectral information. Hyperspectral imaging has high scientific significance and application value for the protection, restoration and research of ancient murals and other cultural relics. In this study, a mural from Daheitian hall in the Qutan temple, Qinghai Province, China, was used as a sample. The hyperspectral data were acquired and analysed for several purposes. Pigment spectral matching and abundance inversion were carried out to obtain the pigment distribution. These data were enhanced by continuum removal and histogram stretching to obtain hidden information. The dark channel prior, Criminisi and Retinex methods were used to virtually restore the image of the mural. The results indicated that by using hyperspectral imaging data, the constructed pure pigment spectrum library and suitable approaches, the types and distributions of mural pigments can be quantitatively analysed, and the lines in murals can be extracted. Hyperspectral images are helpful for identifying information hidden by pigments or surface materials. Mural images can be enhanced, and hidden information can be highlighted using enhancement methods, such as continuum removal and histogram linear stretching. In addition, hyperspectral imaging data have unique advantages in the restoration of mural images, and the combination of defogging methods and image inpainting algorithms can realize the virtual restoration of mural images. In brief, hyperspectral imaging technology was found to have a highly favourable effect on pigment analysis, line extraction, information enhancement, hidden information extraction and the virtual restoration of ancient murals.

Introduction

Cultural heritage is an important component of civilization and transmits historical culture. As one of the important kinds of cultural heritage, murals contain the wisdom and craftsmanship of predecessors. They are abundant in content, bright in colour and vivid in imagery, which can fully show the ingenious thinking and outstanding skills of the creator. Murals are immeasurable cultural treasures left to later generations, which can be used to study religious art, political economy, cultural history, folk costumes and so on at that time. The history of murals can be traced back to the Stone Age and includes prehistoric rock paintings, painted pottery, palace murals, tomb murals and temple murals. These categories of murals constitute a long history of murals. They are the most precious and non-renewable wealth of human civilization, reflect the social style of their time and have far-reaching historical significance and research value. However, due to years of natural erosion and anthropogenic activities, murals inevitably suffer from varying degrees of natural ageing and deterioration.

It is of great significance for the protection, restoration and research of murals to analyse the pigment composition, line characteristics, painting skills and pattern contents of murals. In recent years, with the miniaturization and convenience of the instrument, hyperspectral technology has been widely used in the field of cultural relic protection [1,2,3,4,5,6]. Hyperspectral technology can provide a scientific basis for the identification of cultural relic materials and ages [7,8,9,10], deeply and intuitively reflect the pigment information in cultural relics [11,12,13,14,15,16,17], enhance and mine hidden information [18,19,20,21,22,23,24], and restore mural images and patterns [25,26,27]. Wu et al. [2] applied hyperspectral shortwave infrared imaging technology to the analysis of ancient paintings. They achieved good results in the extraction of linear features of ancient paintings, recognition and classification of pigment information, and extraction of hidden information. Cucci et al. [6] acquired the hyperspectral data of wall paintings and mural inscriptions at the Pompeii archaeological site, and they analysed them from the aspects of pigment identification and mapping, assessed the gypsum abundance, and enhanced the faded details and traits, which proved the feasibility of hyperspectral technology in the field of heritage research.

Wang et al. [7] extracted the spatial and spectral features of hyperspectral images of ancient Chinese paintings through principal component analysis and convolutional neural networks, which were fused with support vector machines for classification. Real and fake paintings were identified with an accuracy rate of 84.6%. Pottier et al. [13] proposed three spectral feature extraction methods, including baseline subtraction, internal normalization and second derivative operations, which were applied to three kinds of cultural heritage materials to analyse the pigments and draw macroscopic chemical maps. Viguerie et al. [16] used visible light and near-infrared hyperspectral imaging technology to analyse red, black and white pigments and binders in fifteenth century Gothic art works. They provided an overall pigment distribution map. Guo et al. [19] analysed the types of pigments on the crowns of the figures based on the hyperspectral data of ancient paintings and used principal component analysis to weaken the background information and enhance the smear information around the crowns. Peng et al. [23] automatically obtained the optimal principal component of hyperspectral images of pottery and murals with a salient object detection method, and the images were fused with true colour images to effectively mine pattern information that is not easily detected by human eyes in cultural relics. Zhou et al. [26] selected characteristic bands from hyperspectral data and used the Poisson editing method with colour constraints to inpaint stains on four pairs of ancient Chinese paintings with different colours and materials. The inpainted paintings had better visual effects and objective evaluations. Han et al. [27] proposed a digital virtual restoration method for bronze chariot patterns by using hyperspectral images and ordinary pictures in two aspects: edge restoration and colour restoration. The missing edges were recovered by enhancing and fusing the edges of patterns in different characteristic bands. Different pigments were distinguished by matching their spectra with the known spectra, and the damaged image was reconstructed by fusing the edge and colour information.

Compared with portable X-ray fluorescence and Raman spectrum analysis methods, hyperspectral technology can obtain large area images and reflection spectrum information at the same time with higher efficiency. Due to its high spectral resolution, this approach can provide an approximately continuous spectral curve for each pixel in an image and can be used to analyse the types and contents of pigments used in chromatic relics. In addition, the wavelength of hyperspectral data spans from visible light bands to near-infrared bands, is wider than the wavelength of commonly used cameras and would be helpful for mining some hidden information covered by pigments or other materials. Hyperspectral data can be used to enhance and restore mural images and patterns.

In this study, a mural created in 1390s was scanned by ground hyperspectral imaging equipment to obtain images. The detailed spectral information of the mural surface was retained. Then, endmember extraction, spectral matching and abundance inversion algorithms were used to analyse the pigments and extract the lines, which can provide scientific and quantitative information for mural research. Furthermore, spectral feature enhancement and image enhancement methods were used for information enhancement, and virtual restoration of mural images was performed by combining image defogging and inpainting algorithms, which can improve the connotation and artistic expression ability of murals. Finally, the pigment abundance map, line extraction map, information enhancement map, hidden information map and virtual restoration map of the ancient mural were obtained. The application of hyperspectral imaging technology provides a beneficial basis for the digital documentation, preservation and restoration research and circulation display of murals.

Materials and workflow

The mural painting of the Qutan temple

The mural studied was painted indoors on the north wall of Daheitian hall in the Qutan temple, which is located in Qutan town, 21 kms south of Ledu District, Haidong city, Qinghai Province, China. According to records, the temple was built in the 25th year of Hongwu in the Ming Dynasty (1392 A.D.), with a history of more than 600 years. As shown in Fig. 1, there are four Daheitian deities in the mural; these deities are important Dharma protectors in Buddhism with far-reaching historical significance and cultural value. The mural is painted on the wall about 1 m above the ground. The overall condition of the mural is relatively good because most paintings are complete. The primary colours used in this mural are mainly red and blue–black. It can be found that the entire mural is very dark, and some patterns are illegible because it suffers from degradation due to some paint loss and soot. We scanned the murals with the hyperspectral imager and carried out several algorithms on their images to provide useful thematic maps for the protection and restoration of these murals.

Fig. 1

An image of the mural on the north wall of Daheitian hall, Qutan temple, Qinghai Province, China

Data acquisition and preprocessing

In July 2018, the hyperspectral data of the mural were captured by using the VNIR400H hyperspectral imaging system of Themis Vision Systems in the United States of America. The image spatial resolution of the imaging system is 1392 × 1000 pixels, the field of view (FOV) is 30 degrees, the sampling interval is 0.6 nm, and the spectral resolution is 2.8 nm. The image was collected in 1040 bands spanning the Visible and Near infrared region, from 377.45 to 1033.10 nm. The hyperspectral camera was placed approximately 1 m in front of the mural, and its main optical axis was perpendicular to the mural. A total of 18 hyperspectral images were collected by shutting the doors and windows to prevent natural light, and two halogen lamps were used as the light source. These 18 images were mosaicked into a whole image, including the four Daheitian deities in the mural, with a width of 2.11 m and a height of 0.97 m. The mosaicking process was realized by the Registration module and Mosaicking module under the Map menu of the Environment for Visualizing Images (ENVI) software. ENVI is a software of remote sensing image processing platform of Harris Geospatial Solutions in the United States of America, which is a powerful remote sensing image processing software developed by scientists using Interactive Data Language. It is a fast, convenient and accurate software solution to extract information from images. With the image taken by the ordinary digital camera as the benchmark, the geometric rectification of the 18 hyperspectral images were performed using the tool of Warp from GCPs: Image to Image under Registration module. Then, they were stitched to a whole hyperspectral image using the tool of Pixel Based under the Mosaicking module. Affected by the wall on the right, the hyperspectral images did not completely cover the deity on the far-right mural. In order to compare the effect of the method proposed with the high-resolution digital image, we also used Nikon D810 to take the digital image of the mural, with a resolution of 7360 × 4912.

The original data obtained by the hyperspectral imaging system are radiance images, which cannot be directly used for spectral analysis. In addition, the change in environmental parameters and the interference of dark current noise lead to noise in the data. Therefore, it is necessary to preprocess the data by radiometric correction and denoising. The radiometric correction formula is as follows:

$$\mathrm{R}=\frac{{R}_{raw}-{R}_{dark}}{{R}_{white}-{R}_{dark}}\cdot \rho ,$$

(1)

where $R$ is the data after radiometric correction, ${R}_{raw}$ is the original hyperspectral data of the mural, ${R}_{white}$ is the standard reflector data, which is the hyperspectral image of the standard reflector acquired in situ under the same conditions as acquiring the mural’s hyperspectral image, $\rho$ is the reflectance of the standard reflector given by the producer, and ${R}_{dark}$ is the dark current noise data obtained after the lights were turned off and the lens was covered.

It was observed that the noise in the bands at both ends of the hyperspectral data wavelength is relatively serious. Therefore, among the 1040 bands acquired, 51–990 bands (405.79–1000.79 nm) were selected for the minimum noise fraction (MNF) [28]. By transforming the noise covariance matrix of the data and the noise whitening data, the principal component with a high signal-to-noise ratio was retained to realize dimensionality reduction and denoising of hyperspectral data. Finally, the top n components with more than 95% cumulative information content were selected for an inverse MNF transformation to restore the hyperspectral data dimension and achieve data denoising.

Overall workflow

The overall workflow of this study is shown in Fig. 2. The hyperspectral imaging data of the murals were captured and denoised by MNF and inverse MNF. Then, the hyperspectral data of murals were analysed from three aspects: (1) Pigment analysis. Simplex identification via the split augmented Lagrangian (SISAL) method was used to extract the endmembers (single pigment spectrum) from the hyperspectral images [29]. These single pigment spectra were matched with a standard spectral library to confirm their types by the combination of spectral angle mapping (SAM), spectral feature fitting (SFF) and binary encoding methods. Then, the content distribution of each single pigment in the image, also denoted as the abundance map, was calculated by fully constrained least squares (FCLS) abundance inversion algorithms; (2) Information enhancement. The hyperspectral images were enhanced by the methods of continuum removal and histogram stretching to highlight the faded or hidden information. The resulting image was called the information enhancement map; and (3) Virtual restoration. The dark channel prior, Retinex by bilateral filtering and Criminisi algorithms were combined to restore the deteriorated images to produce a virtual restoration map [30]. The step of data preprocessing is performed by the hyperspectral image acquisition software provided by the instrument producer and the ENVI. The step of pigment analysis and information enhancement are processed by the software of the ENVI and by the code in Matlab. The step of virtual restoration is mainly carried out by the code building in Matlab.

Fig. 2

The overall workflow

Results and discussion

Pigment analysis

The brilliant colours on the surface of murals are often made up of different types and proportions of pigments, which were carefully mixed and blended by artists. The ways pigments were used have obvious characteristics of the times and regions. On the other hand, lines are the basic elements of murals, which play an important role in the contours, light and dark changes and spatial composition of mural patterns. The investigation of pigments and lines in murals can provide auxiliary support for research on the origin, craftsmanship, painting style and time evolution. It is of great significance for the protection, research and restoration of murals.

Most of the long-lasting pigments used in murals are composed of mineral components. Each pigment has its own unique spectral features due to the different compositions of different pigments. Hyperspectral technology can provide image and spectral information of a target at the same time with a high spectral resolution, which can provide an approximate continuous spectral curve for each pixel in the image. All these advantages can support the scientific determination of the types of pigments in murals, the quantitative inversion of the spatial distribution and content of pigments, and the extraction of lines in murals.

Pigment endmember extraction

The endmembers of the hyperspectral image of the mural were extracted by the SISAL algorithm, which unmixes the pixels by finding the minimum volume simplex that contains hyperspectral vectors. The hinge function whose strength is controlled by a regularization operation is introduced as the soft constraint, and the results are optimized by a sequence of augmented Lagrangian algorithms. It is robust to noise and anomalies and can process large-scale hyperspectral images. We selected four endmembers according to the visual colours presented on the mural, which are the red background, yellow clothes, blue body and black line. Figure 3 shows the extracted red, yellow, blue and black endmember spectral curves.

Fig. 3

Four endmember spectral curves extracted by the SISAL algorithm

The extracted endmember spectral data were matched with the spectrum library of pure pigments constructed by our team. SAM, SFF and the binary encoding comprehensive matching algorithm were used to determine the types of pigments. Material identification was performed by comparing the angle between the data and the endmember spectrum, by comparing the positions of the spectral absorption feature, and by using a logic function to compare each encoded data point and endmember spectrum. The pure pigment spectrum library applied was painted on a white wall and included more than 30 kinds of common ancient mural painting pigments of red, green, blue, yellow, black and white, which can be used as the benchmark data of pigments in cultural relic protection and restoration. Based on final matching of endmembers, a tentative hypothesis of identification could be that the yellow of the clothes may be orpiment. Based on the final matching of endmembers and the spectral characteristics given in [31], the red endmember of the background may be cinnabar or red ochre, which is difficult to identify the pigment from the spectrum alone. The spectrum of blue endmember is quite different from that of known blue pigment, and it is difficult to identify the type of blue pigment only from the spectrum. Wang et al. [32] investigated the pigments of murals of Qutan Temple using X-ray diffraction and isotope X-ray fluorescent. Niu et al. [33] analysed the pigments of murals of Qutan Temple by means of optical microscopy, cross sectional analysis, scanning electron microscopy with energy dispersive spectrometry, and X-ray diffraction. Their results show that blue pigments were lapis and azurite. In addition, they also pointed out that the red pigments were cinnabar and lead, and the yellow pigment is orpiment, which is partly consistent with the results identified by spectral technique.

Abundance inversion

The abundance of each pigment was inversed by the FCLS algorithm to solve the abundance of the extracted endmember by using the minimum error principle [34]. For each pixel in the abundance map of an endmember, the digital number (DN) of the pixel is generally 0–1, which represents the abundance that also denotes the content of this endmember. Finally, the eighteen hyperspectral images were mosaicked to calculate the abundance maps for red, yellow and blue endmembers, as shown in Fig. 4. This map intuitively shows the spatial content distribution of each pigment on the mural. Its value range is 0–1, which represents the content of the corresponding pigment. The greater the value is, the higher the content.

Fig. 4

Pigment abundance map inversed by the FCLS algorithm: a red endmember abundance map; b yellow endmember abundance map; c blue endmember abundance map

The same method was used to extract black line endmembers and calculate their abundance map. The line extraction results of the eighteen images were mosaicked, as shown in Fig. 5. In the line extraction image, the background lines were clear, and the contour lines and navel of the Buddha, which were not visible in the true colour image, were clearly visible.

Fig. 5

Abundance map of black line endmember inversed by the FCLS algorithm

Therefore, endmember extraction, spectral matching and abundance inversion algorithms can be combined to analyse the pigments used in ancient murals. They can be used to identify the types of pigments and calculate their content distribution in murals, which can provide a quantitative and scientific reference for the documentation, research and restoration of cultural relics.

Information enhancement

Hidden information refers to information that is difficult to recognize by human eyes, such as signs of repair and altered information. Information enhancement and hidden information mining of ancient murals can improve the effect of artistic expression and provide new implications for the study of ancient murals. Hyperspectral images have rich wavebands, which can highlight subtle differences in objects at different wavelengths. They include the visible light to near-infrared bands, which can help us to identify information under the cover of pigment or surface material and mine the information that is difficult to detect with the human eye [35]. These unique advantages make hyperspectral imaging technology an appropriate method for mural information enhancement and hidden information mining.

Image enhancement by continuum removal and linear stretching

First, continuum removal was used to enhance the spectral features of the preprocessed mural hyperspectral image. Continuum removal is an effective method to enhance spectral features by highlighting the absorption and reflection features of spectral curves and normalizing them into a consistent spectral background. Then, three bands with wavelengths of 640.31 nm, 549.79 nm, and 460.20 nm were selected as red, green, and blue channels to synthesize false colour images, which were linearly stretched by the histogram to realize information enhancement. Among them, histogram linear stretching is a method to improve the image quality by scaling up the brightness range of the original image to saturate both ends of the transformed image histogram. Finally, the eighteen enhanced images were mosaicked to obtain the information enhancement map. As shown in Fig. 6, the colour of lines in the background of the enhanced mural image were white, the separability between the characters and the background was higher, and the lines in the extremely soot-affected areas in the true colour image were also clearly visible. The contrast of the mural image was higher, and the detail was clearer. In addition, in the background of the enhanced mural image, the white fog-like substance that is thick in the middle and light at the bottom can also reflect the extent and degree of the soot contamination of the mural to a certain extent.

Fig. 6

Information enhancement image by continuum removal and linear stretching method

Hidden information extraction by using spectral difference

In addition, it can be found that in the true colour image taken by the ordinary digital camera, the body, neck, face, eyeballs and edges of the eyes of the last character were all blue–black, as shown in Fig. 7a. However, in the enhanced mural images, the colours of the edge of the body, the neck, the eyeball and the edge of the eyes were white and were obviously different from those of the whole body and the face of the character in Fig. 7b. In the preprocessed hyperspectral mural image, the regions of interest were extracted, and the average spectral curves were calculated. As shown in Fig. 7c, the absorption features and trends of the spectral curves of the body and face were similar, and only the reflectance between 700 and 1000 nm was different. The spectral features of the four curves of the clothing edge, the neck, the eyeballs and the edge of the eyes were similar and were obviously different from those of the spectral curves of the body and face. Therefore, the pigment used on the edge of the body may be different from the pigment used on the body and neck.

Fig. 7

Hidden information extracted from hyperspectral images: a the right Buddha image taken by ordinary digital camera; b result image of the right Buddha after hidden information extraction; c average spectral curves of typical objects in the image

In order to investigate the differences in the use of pigments, three other Buddhas in the mural were selected and compared. Figure 8 shows the four selected Buddha regions.

Fig. 8

The local images of four Buddhas before and after information enhancement: a local image of the first Buddha taken by ordinary digital camera; b local image of the second Buddha taken by ordinary digital camera; c local image of the third Buddha taken by ordinary digital camera; d local image of the fourth Buddha taken by ordinary digital camera; e local image of the first Buddha after image enhancement; f local image of the second Buddha after image enhancement; g local image of the third Buddha after image enhancement; h local image of the fourth Buddha after image enhancement

As can be seen from Fig. 8, in the true colour image of the Buddha, it can be seen that the bodies of the four Buddha are blue black, the faces of the second, third and fourth Buddhas are blue black, and the ribbons of the first, second and third Buddhas are brown. However, there are some differences in the colours of the face, body and ribbon in the enhanced mural images in Fig. 8. Therefore, in order to compare whether there are different pigments in the face, body and ribbon areas of different Buddha, we select the region of interest in the preprocessed hyperspectral mural image and calculate the average spectral curve, as shown in Fig. 9.

Fig. 9

Spectral curves of typical objects of four Buddhas in hyperspectral images: a the spectra of face for second, third and fourth Buddha; b the spectra of body for the four Buddhas; c the spectra of ribbon for the first, third and fourth Buddha

It can be seen from Fig. 9a that for the face areas of different Buddhas, the reflectance curves of the third and fourth Buddha are similar, while the second curve is slightly different from the other two. The reflectance difference is close to 3%. However, the positions of the absorption valley near 700 nm and the trend of the curves are relatively similar. The difference in reflectance may be caused by the different brightness of the light or other noise.

For the body regions of different Buddha shown in Fig. 9b, the spectral curves of the four Buddha are highly similar. The largest difference was between the second and other curves, about 1%.

It is worth noting that for the ribbon regions of different Buddha, as shown in Fig. 8e–g, in the enhanced image, the ribbon colours of the first, second and third Buddha are relative different. In Fig. 9c, the spectral curves of the three Buddhas are also different. The maximum difference in reflectance is more than 4%. Therefore, the pigments of this part may be different.

Therefore, the hyperspectral spectral feature enhancement and image enhancement methods can improve the quality of mural images, enrich the amount of information, and enhance the interpretation and recognition effect of ancient murals. It can mine illegible information and reveal altered areas. Thus, it can be used to increase the readability and artistic expression effect of ancient murals and provide new implications for the research of ancient murals.

Virtual restoration

Because of their long history, murals are deteriorated to varying degrees due to the influence of the natural environment, such as humidity and high temperatures, and human activities, such as burning incense and worshipping Buddha in temples. With the help of image restoration methods, the deterioration of murals can be virtually restored without interfering with the current situation of murals. It is a valuable complement to documentation and actual restoration for murals. Virtual restoration can provide useful information for the actual restoration and improve the efficiency of the protection and restoration of ancient murals. Hyperspectral imaging technology provides a new possibility for virtual restoration due to its wide spectral coverage and stronger penetration ability than visible light.

Soot-affected mural image synthesis

As shown in Fig. 10a, the mural is seriously contaminated by soot, and some of the patterns are covered. The entire image is blackened, and some of the lines in the background are even illegible. In the preprocessed hyperspectral images of the mural, because the regions of interest of the red background areas were less affected by soot, the soot-affected red background and soot-affected black lines of the areas with relatively serious soot damage were extracted, and the average spectral curve was calculated. As shown in Fig. 10b, the trend and spectral feature positions of the two curves of the red background and soot-affected red background were similar. The cross occurred near the wavelength of 800 nm; that is, the effect of soot on the red background may be less near this band. After 550 nm, the difference between the two spectral curves of the soot-affected red background and soot-affected black lines increased with increasing wavelength; that is, the separability of the background and lines increased.

Fig. 10

Spectral analysis of patterns under different levels of soot: a mural image; b regions of interest and their average spectral curves

Figure 11 shows a true colour image (Fig. 11a) of a small area of the mural and the images with wavelengths of 405.79 nm (Fig. 11b), 605.40 nm (Fig. 11c), and 805.53 nm (Fig. 11d). The black lines on the red background were clearer in the band with a wavelength of 805.53 nm, and the black marks at the edge of the white paint loss on the right side of the character disappeared in this band.

Fig. 11

Hyperspectral images with different wavelengths: a true colour image; b band with 405.79 nm; c band with 605.40 nm; d band with 805.53 nm

Therefore, three bands with wavelengths of 805.53 nm, 549.79 nm and 460.20 nm were selected as the red, green and blue channels to synthesize the false colour image with the preprocessed hyperspectral image. Based on the true colour image, block histogram matching was performed on the synthesized false colour image to obtain the soot-affected mural image with clearer patterns and realistic colour.

Preliminary soot removal

Different from general degradation, soot often covers large areas of mural patterns, and the spatial distribution of soot-covered images is similar to that of foggy images.

In computer vision and computer graphics, an atmospheric scattering model is usually used to describe the formation process and principle of foggy images. Although the particles of sootiness and fog are different, they will lead to the scattering of some light by particles, and the intensity of light will be weakened when the incident light contacts the particles. The model is shown in Eq. (2).

$$I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right)$$

(2)

where $I\left(x\right)$ is the DN of the observed image, representing the soot-affected mural image, J(x) is the image after smoke removal, which is the part to be solved in Eq. (2), t(x) is the transmittance of soot medium, and A is the scattered light value caused by soot.

The dark channel prior is a statistical rule proposed by He [36]. It was pointed out that there will be some areas, and at least one colour channel, that have some pixels whose intensities are very low and close to zero in most of the fog-free images without sky areas. The parameters of A and t(x) in atmospheric scattering model (2) can be solved by using the dark channel prior.

$${P}^{dark}\left(x\right)=\underset{z\in \Omega (x)}{\mathit{min}}(\underset{c\in (r,g,b)}{\mathit{min}}{P}^{c}(y))$$

(3)

where x is a pixel; $c$ is a color channel among $r$, $g$, and $b$; ${P}^{C}$ is the gray value of a channel of $P$; and $\Omega (x)$ is a local patch centered at $x$; $\underset{c\in (r,g,b)}{min}$ is the minimum value of each pixel in the r, g, b channel; $\underset{z\in \Omega (x)}{min}$ is a minimum filter.

The dark channel image ${P}^{dark}\left(x\right)$ was calculated by the Eq. (3), and the maximum value of dark channel was selected as the A in the Eq. (2). According to the dark channel prior rule, the dark channel is ${P}^{dark}\to 0$ for the fog-free image. The t(x) in Eq. (2) can be figure out by Eq. (4).

$$t\left(x\right)=1-\omega \underset{c}{\mathit{min}}(\underset{y\in \Omega \left(x\right)}{\mathit{min}}(\frac{{P}^{c}\left(y\right)}{{A}^{c}}))$$

(4)

where $\omega (0 < \omega \le 1)$ is a constant parameter to retain the perspective depth of the image.

The preliminary soot removal of the synthetic soot-covered mural image was performed by combining the false colour image, atmospheric scattering model and dark channel prior. First, the dark channel image was calculated by the false colour image, and the atmospheric light value and transmission were obtained according to the dark channel image. Second, the soot-free image was obtained from the synthetic soot-covered mural image according to the atmospheric scattering model. Finally, the brightness was adjusted to realize the preliminary removal of soot. These images were then transformed to HSV space, where the V component was multiplied by the set brightness factor to form a new V component that was used to perform inversive HSV transformation to obtain the image with adjusted brightness. As shown in Fig. 12, compared with the original image, the influence of soot on the mural image after preliminary soot removal was reduced, the details were highlighted, and the black lines in the red background were clearer.

Fig. 12

Synthetic soot-covered mural image

Inpainting of paint loss

There are a number of damaged areas due to paint loss in the background of the mural, which causes the exposure of the white wall at the bottom. To further improve the visual effect of the mural, an image inpainting algorithm named Criminisi algorithm was used to restore the paint loss. The Criminisi algorithm is a patch-based image inpainting algorithm that can synchronously utilize the texture and structure information in the image to better realize the filling of the target area [37]. Firstly, the pixels of the area to be inpainted were masked, and the priority of the patches at the edge of the masked area was calculated to find the image patch with the highest priority. Then this patch was replaced by an optimal target patch that was searched in the whole image under the similarity criterion. Final, the remaining masked area and the corresponding confidence priority and data priority were updated. And the next image patch with the highest priority would be filled in the same way. This process was repeated until all pixel blocks were repaired. Before using Criminisi algorithm to inpaint the image, we need to know the areas to be restored. Therefore, we proposed a method to locate the paint loss areas by using support vector machine (SVM). First, the figure region was distinguished from the background region in the mural for the paint loss regions are mainly located in the background region. Another reason is that the characters in the murals are very delicate and rich in colour, which may affect the accuracy of the extracted areas and the effect to be inpainted. The image was masked as Fig. 13a. Second, the regions of interest of each colour, mask area and paint loss area in the image are selected as the training data, as shown in Fig. 13b. The SVM classification method [38] was used to classify the masked background region to several classes including the paint loss areas, as shown in Fig. 13c. Third, in order to make the extraction area completely cover the deteriorated area, the dilation operation in morphological filtering was performed three times to expand the original extracted areas. The final extracted paint loss areas are shown in Fig. 13d.

Fig. 13

Extraction of paint loss areas: a the mask distinguished between the Buddha and the background area; b training data of each colour, mask area and paint loss area; c SVM classification map; d the extracted paint loss areas

Finally, the Criminisi algorithm was used to inpaint the paint loss areas in the image after the preliminary soot removal. As shown in Fig. 14, from the perspective of visual effects, most of the white walls in the mural appeared after the paint loss was repaired, making the whole image more coherent.

Fig. 14

Inpainting of paint loss

Virtual restoration of soot-covered mural images

The Retinex method considers the object brightness perceived by the human eyes as a combination of the illumination of the environment and the reflection of the object surface [39]. The illumination component can be estimated from the original image to obtain the reflection component, that is, to obtain the colour of the object itself. It was shown as Eq. (5):

$$I\left(x,y\right)=R\left(x,y\right)\cdot L\left(x,y\right)$$

(5)

where $I\left(x,y\right)$ is the DN of the image, $L\left(x,y\right)$ is the component of ambient light, and $R\left(x,y\right)$ is the reflectance image of the mural, which is the result image after inpainted.

To further restore the soot-affected mural image, according to the Retinex method, two bilateral filters with different weights and parameters were set to solve the illumination and reflection of the image after the inpainting of paint loss to realize the virtual restoration of the mural image. Among them, the bilateral filter is a kind of nonlinear filter that can consider spatial information and grey similarity at the same time and can better achieve the purpose of edge preservation and denoising [40]. Finally, the eighteen restored images were mosaicked, and the virtual restoration map was obtained. From a visual point of view, as shown in Fig. 15, the restored image basically eliminated the influence of soot on the mural content and repaired the paint loss in the background. It was clear and coherent; hence, the method largely restored the original appearance of the mural.

Fig. 15

Virtual restoration map

Therefore, using the advantages of hyperspectral imaging as well as relevant defogging methods and image inpainting algorithms, it is possible to restore the mural image blurred by soot and repair the paint loss damage in the background. This method is more applicable to mural with light soot and small paint loss area. The restoration of murals suffered by serious soot (basically invisible) or large areas of paint loss would be further investigated. Nevertheless, this method can still help to promote the artistic expression of ancient murals, improve the circulation ability of online exhibitions so that they are available to more audiences. It can also provide valuable guidance for mural restoration.

Conclusions

In this study, the hyperspectral imaging data of an ancient mural were acquired and analysed. The outcomes of the study can be summarized as follows: (1) Based on hyperspectral imaging data and the pure pigment spectrum library, the types and contents of ancient mural pigments can be quantitatively analysed by endmember extraction, spectral matching and abundance inversion algorithms. The lines in murals can also be extracted; (2) Hyperspectral technology is helpful to identify information covered by pigments or surface materials. The use of enhancement methods, such as continuum removal and histogram linear stretching, can enhance mural images and highlight the hidden information in murals; and (3) Hyperspectral images have unique advantages in the restoration of mural images. Combining defogging methods and image inpainting algorithms can achieve the virtual restoration of murals. The present research will expand the application of hyperspectral imaging technology to the protection and research of ancient murals. It can enhance the ability to digitally document ancient murals and improve the artistic expression and circulation display ability of ancient murals. It should also be pointed out that some involved methods, such as pigment analysis, information extraction and virtual restoration, need to be further studied to achieve higher performance.