Restoration method of sootiness mural images based on dark channel prior and Retinex by bilateral filter

Abstract

Environmental changes and human activities can cause serious degradation of murals, where sootiness is one of the most common problems of ancient Chinese indoor murals. In order to improve the visual quality of the murals, a restoration method is proposed for sootiness murals based on dark channel prior and Retinex by bilateral filter using hyperspectral imaging technology. First, radiometric correction and denoising through band clipping and minimum noise fraction rotation forward and inverse transform were applied to the hyperspectral data of the sootiness mural to produce its denoised reflectance image. Second, a near-infrared band was selected from the reflectance image and combined with the green and blue visible bands to produce a pseudo color image for the subsequent sootiness removal processing. The near-infrared band is selected because it is better penetrating the sootiness layer to a certain extent comparing to other bands. Third, the sootiness covered on the pseudo color image was preliminarily removed by using the method of dark channel prior and by adjusting the brightness of the image. Finally, the Retinex by bilateral filter was performed on the image to get the final restored image, where the sootiness was removed. The results show that the images restored by the proposed method are superior in variance, average gradient, information entropy and gray scale contrast comparing to the results from the traditional methods of homomorphic filtering and Gaussian stretching. The results also show the highest score in comprehensive evaluation of edges, hue and structure; thus, the method proposed can support more potential studies or sootiness removal in real mural paintings with more detailed information. The method proposed shows strong evidence that it can effectively reduce the influence of sootiness on the moral images with more details that can reveal the original appearance of the mural and improve its visual quality.

Introduction

As one of the most important components of cultural heritage, murals have profound historical significance and research value [1]. Temple murals are part of the dominant types of Chinese murals, usually painted on the walls of Buddhist or Taoist temples. However, murals deteriorated to different degrees due to the changes of the environment and human activities. Burning incense while praying in temples were very common phenomena for a long period of time, which made temple murals vulnerable to sootiness, resulting in blurred patterns. At present, the study on the sootiness mural is mainly focused on the influence of the sootiness on the pigments in the mural [2] and the sootiness cleaning [3]. Traditional methods of removing sootiness mostly use chemical reagents, which are tedious, time-consuming and laborious, and the chemical reagents may cause a certain degree of damage to cultural relics. Therefore, it has become a research hotspot of mural digital protection to virtually repair the degradation of mural quality and restore the original appearance of mural by using digital photography technology.

In recent years, the study on virtual restoration of murals mostly focuses on the automatic filling method of damaged areas [4], mainly including image restoration methods based on diffusion, exemplar, sparse and deep learning, etc. For the diffusion-based method, Jaidilert et al. [5], used a variety of existing variational in painting methods to inpaint the scratches in Thai murals extracted by region growing and morphological methods based on seed points provided by users. Shen et al. [6], improved the morphological component analysis method based on sparse method to decompose the image into structure and texture parts, and simplified total variation algorithm and K-singular value decomposition algorithm. As a result of applying the described method, the authors were able to inpaint the cracks in the murals and improve the inpainting accuracy. However, these methods are suitable for inpainting the narrow or small texture damaged areas. When they are used to recover the texture of large areas, the results will tend to blur. In the method based on exemplar, Li et al. [7], used the existing Criminisi algorithm to inpaint the mud spots extracted by threshold segmentation based on the characteristics of brightness, colourity and texture of the mud spots on the mural of Tang Dynasty tomb, which improved the accuracy of mud spots extraction. Purkait et al. [8], used patch similarity measurement method based on spatial coherence to improve patch-based restoration algorithm to achieve automatic coherent texture synthesis, and completed the inpainting of colorful Indian temple murals. Wang et al. [9] improved the algorithm of Criminisi by using the wavelet energy factors and virtually restored the scratches and paint losses of Ming Dynasty murals in Guanyin Temple, Xinjin County, Sichuan Province, China, so that the overall texture and color of the restored murals were continuous and natural. This kind of method can be applied to inpaint large area of damaged image, but when the structure of the region to be repaired is curved or the texture is irregular, the inpainted structure is often discontinuous. For the sparse-based method, Wang [10], used the method of directions for dictionary learning, and realized the informatization and restoration of the murals in Potala Palace by sparse representation, so that the restored images can better retain the details and edge information in the murals. Wang et al. [11], based on the structure information collected from the line drawing, used the sparse model to construct the target patch, and the constraints of texture similarity and structure continuity were introduced to inpaint the damaged murals with missing structure in a large area of Mogao Grottoes, Dunhuang, China. For the inpainting method based on deep learning, Cao et al. [12], proposed a consistency enhanced generative adversarial network model, which achieved the restoration of the temple murals in Wutai Mountain, China, so that the overall consistency and structural continuity of the restored images were better. However, at present, most of the virtual restoration methods were aimed at filling methods, such as paint losses and cracks, and there are few methods to restore the sootiness murals.

Distinct from other problems, the sootiness usually covers a large area on mural. The influence of sootiness on mural image is similar to that of fog image, which attenuates the transmission of electromagnetic wave in a large space, so that the target image is blurred. The current study on image defogging algorithms mainly includes the following two types of methods [13]. The first one is based on image enhancement, such as histogram equalization method [14] or the enhancement method based on Retinex method [15,16,17]. The method based on the enhancement only enhances the contrast of image without considering the cause of image degradation. It can effectively improve the visual effect, but the resulting image will appear as information loss and oversaturation. The second one is the image defogging algorithm based on the physical imaging model [18, 19]. This method can restore the image to its original state before degradation based on physical imaging model, which makes the image more natural. However, due to the lack of known information, the image defogging is uncertain and prone to halo phenomenon.

With the development of remote sensing technology, hyperspectral imaging has been widely used in many fields. It can provide spectral information with high spectral resolution and nearly continuous spectral curve for each pixel of the image. It can usually cover visible light to near-infrared wavelength, which is helpful to identify the information covered by pigments or surface materials, and to mine the information that is difficult to be recognized by human eyes [20]. Because of these unique advantages, hyperspectral imaging technology has been introduced into the study of cultural relics protection, including virtual restoration [21, 22] and visual enhancement [23,24,25,26,27], which adds potential to the use of the hyperspectral technology to remove sootiness from murals.

The main objective of this study is to propose a method to restore the sootiness mural hyperspectral images based on dark channel prior and Retinex by bilateral filtering algorithms. First, the near-infrared band (845.77 nm) with strong penetration ability, the visible bands (549.79 nm and 460.20 nm) were selected as the RGB channel to produce a pseudo color image. It was then matched with the corresponding true color image by the block histogram matching algorithm. The result image, denoted as sootiness mural image, is more suitable for restoration with better contrast. Second, the dark channel image was calculated on the pseudo color image by dark channel prior algorithm to estimate the scattering light value and transmittance of sootiness. And based on the atmospheric scattering model, the preliminary sootiness removal image was obtained from the sootiness mural image. Third, according to the Retinex method, the illumination image and the reflection image were solved respectively by bilateral filtering algorithm from the preliminary sootiness removal image, and the sootiness mural image restoration was realized. In this way, a clearer mural image with more details will be obtained.

Materials and restoration methods

Materials

Sootiness

According to the Chinese National Standard for the Protection of Cultural Relics, Ancient wall painting deterioration and legends (GB/T 30237-2013) [28], sootiness is the mark of the mural being polluted by soot or incense, as shown as an example, in Fig. 1. Due to the contamination of sootiness, the patterns in the murals are often blurred, which seriously affects the value of appreciation of the murals.

Fig. 1

Sootiness legend of mural

Mural data

The mural selected in this study is located on the north indoor wall of Daheitian Hall on the east side of Qutan Temple, located at Ledu District, Haidong City, Qinghai Province, China. According to the site’s records in China, the temple was built in the 25th year of Minghongwu (1392 A.D.), with a history of more than six hundred years [29]. As shown in Fig. 2a, due to the activities of incense burning and worship Buddha for a long time, the mural is seriously contaminated by sootiness. Some of the patterns in the mural are covered and cannot be recognized by the naked eye. In order to reduce the influence of sootiness on the mural and restore the information covered by sootiness, the hyperspectral data of the mural were captured and analyzed.

Fig. 2

The image of the mural and the two study areas: a complete image of the mural; b image of the first study area; c image of the second study area

Data acquisition

In July 2018, the mural data of the study areas were captured by using the VNIR400H hyperspectral imaging system of Themis Vision Systems with the spatial resolution 1392 × 1000 pixels and 1040 bands covering from 377.45 nm (visible light) to 1033.10 nm (near-infrared). The spectral sampling interval was 0.6 nm, and the spectral resolution was 2.8 nm.

During the data acquisition, the distance between the hyperspectral camera and the mural was about 1 m. The sunlight was blocked by closed doors and windows, and a pair of halogen lamps whose spectral distribution is close to that of sunlight were used for illumination. A total of 24 hyperspectral images were collected, covering most of the sootiness area of the north wall of the Daheitian Hall. The images of the two study areas, shown in Fig. 2b, c, are the true color images produced by the red, green and blue bands with wavelengths of 460.20 nm, 549.79 nm and 640.31 nm after radiometric correction and data denoising.

Restoration method

Figure 3 shows the overall workflow of the proposed method for the restoration of sootiness mural images, including four main steps: (1) Data preprocessing using radiometric correction and data denoising, (2) sootiness mural image produce using block histogram matching of pseudo color image and true color image, (3) preliminary sootiness removal using dark channel prior and image brightness adjustment, and (4) restoration of sootiness mural using Retinex by bilateral filter in HSV (hue, saturation, value) color space. The details of each step are discussed in the following sections.

Fig. 3

The overall workflow of the proposed method

Data preprocessing and sootiness mural image produce

The original data captured by hyperspectral imaging system are in radiance and the data range is depending on the number of digitalization bits of the system. The radiance is the radiation energy reflected by the target and received by the sensor in a certain direction of space per unit area per unit time and per unit solid angle. Even for the same target point, the radiance will change with the change of the incident energy. However, the reflectance of a certain material is usually unique and independent of the external illumination, so it is commonly used to study the natural characteristic of the target. Therefore, we will convert the original hyperspectral data into reflectance images by using Eq. (1) [30, 31]:

$$R = \frac{{R_{raw} - R_{dark} }}{{R_{white} - R_{dark} }}$$

(1)

where $R{ }$ is the corrected reflectance image; $R_{raw}$ is the original image of the mural; $R_{white}$ is the standard reflector data obtained on site; and $R_{dark}$ is the dark current data acquired with the light source off and the lens covered. The reflectance of the standard reflector is 99%.

In addition, in the data acquisition of hyperspectral imaging system, there will be some noise bands due to the changes of environmental parameters and the interference of dark current noise. Through the inspection of the data, it was found that the bands at both ends of the sensor’s wavelength spectrum were noisy, that is, the bands with the shortest wavelength and the bands with the longest wavelength. After observing the spectral curve of random pixels on the image, the 51–990 bands (405.79–1000.79 nm) were manually selected for subsequent processing in the 1040 bands acquired.

In order to further reduce the noise, the selected data were processed by the minimum noise fraction rotation (MNF), which is a common method of dimension reduction and denoising in hyperspectral data processing [32]. It can transform the noise covariance matrix of the data and the noise whitening data, and retain the principal component with relatively large signal-to-noise ratio, so as to realize the dimensionality reduction and denoising of hyperspectral data. In this paper, the MNF transformation was performed on the hyperspectral data after manual selection to separate the noise from the information in the data. The top n components with more than 95% information content were selected for inverse MNF transformation, and the hyperspectral data dimension was restored, and the data denoising was realized.

It was observed that the near-infrared bands can reveal the information under the surface material coverage to a certain extent, which is conducive to the removal of sootiness. Therefore, the near-infrared, green and blue bands with wavelengths of 845.77 nm, 549.79 nm and 460.20 nm were selected to produce a pseudo color image (PCI) in the denoising hyperspectral image. The red, green, and blue bands with wavelengths of 640.31 nm, 549.79 nm, and 460.20 nm were selected to produce a true color image (TCI). However, by comparing the image quality of PCI and TCI, it was found that some of the patterns in the murals were clearer and more realistic in the TCI. Thus, PCI is calibrated to obtain the sootiness mural image (SMI) by block histogram matching method which adopted the TCI as the reference, as shown in Fig. 4. In the following sections, the TCI is used as a reference image in comparison of restoration effect. The PCI is used to obtain the dark channel image in the dark channel prior sootiness removal, so as to solve the scattering light value and transmittance. SMI is used as the sootiness mural image to be restored for the subsequent restoration of sootiness mural.

Fig. 4

The hyperspectral image of area 1: a true color image (TCI); b pseudo color image (PCI); c sootiness mural image (SMI)

Preliminary sootiness removal

In the proposed method, the dark channel prior was applied to solve the atmospheric scattering model for the preliminary removal of sootiness in mural images. The influence of sootiness on mural image is similar to that of fog, which usually covers the mural pattern in a large area. The more serious the fog or sootiness, the more blurred the pattern and color in the image, or even completely invisible. Although the particles of sootiness and fog are different, they will cause some light to be scattered by the particles and the light intensity will be weakened when the incident light contacts with the particles. Therefore, the atmospheric scattering model was used to describe the formation process and method of sootiness mural images, such as Eq. (2) [33]:

$$S\left( x \right) = D\left( x \right)t\left( x \right) + A\left( {1 - t\left( x \right)} \right)$$

(2)

where $S\left( x \right)$ is the sootiness mural image to be restored; $D\left( x \right)$ is the image of target scene after preliminary sootiness removal; $t\left( x \right)$ is the medium transmittance of the sootiness; and $A$ is the scattering light value caused by sootiness. The purpose of the restoration of sootiness mural is to recover $D$, $A$ and $t$ from $S$.Step 1: Estimate the scattering light $A$. The dark channel prior is a statistical rule proposed by He et al. [16]. It points out that in most fog-free images (non-sky areas), there will be some areas, at least one color channel has some pixels whose intensity are very low and close to zero. This channel is called dark channel. This is similar to the situation of mural images covered by sootiness in that in the areas with light or even no sootiness, the pattern and color of the image are visible, and at least one color channel has some pixels whose value are very low and close to zero for the red, blue, or green color areas. But in areas with heavy fog or sootiness, the content of the image will be covered and illegible. The mathematical expression is as follows:

$$P^{dark} \left( x \right) = \mathop {\min }\limits_{z \in \Omega \left( x \right)} \left( {\mathop {\min }\limits_{{c \in \left( {r,g,b} \right)}} P^{c} \left( y \right)} \right)$$

(3)

where$c$ is a color channel among $r$, $g$, and $b$; $P^{C}$ is the gray value of a channel of $P$; $\Omega \left( x \right)$ is a local patch centered at $x$. A dark channel image is the result of two minimum operators: $\mathop {min}\nolimits_{{c \in \left( {r,g,b} \right)}}$ is the minimum value of each pixel in the $r, g,{ }b$ channel; $\mathop {min}\nolimits_{z \in \Omega \left( x \right)}$ is a minimum filter.

The dark channel image was calculated from the PCI using the Eq. (3), and the maximum value of dark channel was selected as the estimated value of scattering light.

Step 2: Calculate the transmission $t$. According to the method of dark channel prior, for the fog-free image, the dark channel is $P^{dark} \to 0$. So that when the scattering light value had been obtained, the transmission $t$ can be obtained. The constant parameter $\omega (0 < \omega \le 1)$ was introduced to retain the perspective depth of the image and make it more realistic:

$$t\left( x \right) = 1 - \omega \mathop {\min }\limits_{c} \left( {\mathop {\min }\limits_{y \in \Omega \left( x \right)} \left( {\frac{{P^{c} \left( y \right)}}{{A^{c} }}} \right)} \right)$$

(4)

Step 3: Restore the image $D\left( x \right)$. After the scattering light value A and transmittance t were obtained, the SMI was taken as the $S\left( x \right)$ to be restored, and then the image $D\left( x \right)$ was solved according to the atmospheric scattering model. When the transmission approached zero, direct recovery of $D\left( x \right)$ was prone to noise. Therefore, the minimum threshold $t_{0}$ was set to control the transmission, so:

$$D\left( x \right) = \frac{S\left( x \right) - A}{{max\left( {t\left( x \right),t_{0} } \right)}} + A$$

(5)

Step 4: Brightness adjustment. The brightness of the image $D\left( x \right)$ was low after removing the sootiness via the dark channel prior, so the image was converted to the HSV color space, and the image brightness was adjusted by setting a brightness factor and multiplying it with the Value component. Thus, the mural image $B\left( x \right)$ with preliminary sootiness removal was obtained, and the details of the mural were enhanced while removing the interference of sootiness.

Fine sootiness removal

In order to further remove the sootiness from mural image, defogging method of Retinex by bilateral filter is applied to the sootiness mural image. Retinex method considers the brightness of the object perceived by the human eye as an organic combination of the illumination of the environment and the reflection of the object surface [34]. The illumination component can be estimated from the original image to obtain the reflection component, that is, the color of the object itself. The mathematical expression of Retinex method is:

$$B\left( x \right) = R\left( x \right) \cdot L\left( x \right)$$

(6)

where $B\left( x \right)$ is the image pixel value received by the human eye or camera, that is the image after preliminary sootiness removal in the proposed method; illumination image $L\left( x \right)$ is the illumination component of the ambient light; and reflection image $R\left( x \right)$ is the reflection component of the object.

Take logarithm on both sides of Eq. (6):

$$\ln \left[ {B\left( x \right)} \right] = \ln \left[ {R\left( x \right)} \right] + ln\left[ {L\left( x \right)} \right]$$

(7)

The reflection component of each color channel of the image is

$$\ln \left[ {R_{i} \left( x \right)} \right] = \ln \left[ {B_{i} \left( x \right)} \right] - ln\left[ {L_{i} \left( x \right)} \right]$$

(8)

where $B_{i} \left( x \right)$, $R_{i} \left( x \right)$, $L_{i} \left( x \right)$ are the channel image, the reflection component and the illumination component of the $i$ color channel of the image respectively. The usual method is to use Gaussian function to estimate the illumination image from the image, so as to solve the reflection image.

Bilateral filter [35] is a nonlinear filter, which combines the spatial proximity and pixel similarity of image, and can consider both spatial information and gray similarity. Compared with Gaussian filtering, bilateral filtering can effectively remove most noise while keeping image details. During the filtering, edge keeping and denoising are better realized by adjusting the filter size p which represents the diameter of each pixel neighbourhood, the weight $\sigma_{r}$ which controls the change of gray scale, and the weight $\sigma_{s}$ which controls the change of spatial distance.

In the proposed method, the defogging method of Retinex by bilateral filter [16] was used to further improve the visual effect of the sootiness mural by setting two bilateral filters with different weights and parameters. First, the image $B\left( x \right)$ after the preliminary sootiness removal was converted from RGB space to HSV color space. Second, only for Value component, the illumination image was estimated by bilateral filtering. Then, the logarithm of the original image and illumination image were taken, and the reflection image was solved by using the second bilateral filtering according to the Retinex method, so as to obtain the recovered image of the V component. Finally, the HSV space image was converted to RGB (red, green, blue) space to realize the restoration of sootiness mural.

Other method to be discussed

Homomorphic filtering is a method to compress the image brightness range and enhance the contrast in frequency domain [36]. It is based on the illumination and reflection model of the image. By adjusting the gray range of the image, it can enhance the detail information of the dark area without losing the details of the bright area. Gaussian stretching is an interactive histogram stretching method in radiation enhancement. By stretching the histogram of the output image into a Gaussian function, the detail information of the image is enhanced. Homomorphic filtering and Gaussian stretching will be used in the discussion and compared with the proposed method.

Results

Sootiness mural image produce

As shown in Fig. 5a, the mural is seriously contaminated by sootiness. The entire image is blackened, and some of the lines in the background are blurry. Figure 5b shows the near-infrared band with a wavelength of 845.77 nm in area 1. In this band, the lines on the red background were clearer, and the black marks on the edge of the white paint losses on the right side of the character disappeared. So, the three bands of near-infrared, green and blue with wavelengths of 845.77 nm, 549.79 nm and 460.20 nm were selected as red, green and blue channel to produce the PCI, as shown in Fig. 5c. The selection of near-infrared band has little effect on the visualization of sootiness mural image. However, compared with the PCI, the clothes of the lower left part of the image were clearer and the color was more realistic in the TCI. Therefore, based on the TCI, block histogram matching was performed on the PCI. For the study areas of this article, the resolution was 1392 × 1000 pixels, and the blocking standard was to divide the image length and width into 8 equal parts, that is, the image was divided into 64 blocks, the spatial resolution of each block was 174 × 125 pixels. The smaller the block is, the clearer the clothing is, and the more realistic the color is. But if the block is too small, the black marks on the edge of the paint losses will reappear. Figure 4d shows the SMI after the block histogram matching, and Fig. 5e and f are the enlarged images in the red frame of TCI and SMI, respectively. It can be seen that in SMI, the black marks on the edge of the paint losses region disappeared, and the color of the clothes was more realistic.

Fig. 5

The sootiness mural image of area 1: a true color image (TCI); b near-infrared band image; c pseudo color image (PCI); d sootiness mural image (SMI) after block histogram matching; e enlarged image in the red frame of TCI; f enlarged image in the red frame of SMI

Preliminary sootiness removal

In the preliminary sootiness removal, the dark channel image was first calculated from the PCI, and the SMI was obtained after block histogram matching. Then, the sootiness removal was performed on the sootiness mural image via the dark channel prior. For the study areas of the proposed method, the minimum filter size was 3 × 3. If the filter size is too large, the sootiness removal effect will be poor, and if it is too small, some information in the image will be blurred. The dark channel image is shown in Fig. 6b. The effect of dark channel priori sootiness removal is shown in Fig. 6c. Finally, through the image brightness adjustment, the mural image after the preliminary sootiness removal was obtained, and the brightness factor is 2.0, as shown in Fig. 6d. Compared with the TCI, the image after removing the sootiness via the dark channel prior and brightness adjustment preliminarily reduced the sootiness effect and made the black textures in the red background clearer.

Fig. 6

The preliminary sootiness removal results of area 1: a true color image; b dark channel image; c result of dark channel priori sootiness removal; d result of brightness adjustment

Fine sootiness removal

In order to further restore the sootiness mural, the preliminary sootiness removal image obtained in the previous step was converted to HSV color space. The Retinex by bilateral filter was used to solve the illumination image and reflection image respectively. Then the HSV space image was converted to RGB space to realize the restoration of sootiness mural. In the proposed method, the filter size $p$, the weight to control the change of gray scale $\sigma_{r}$ and the weight to control the change of spatial distance $\sigma_{s}$ of two bilateral filters are shown in Table 1. The larger the filter size is, the more obvious the denoising effect is, but it will slow down the calculation speed. The gray scale change weight and space distance weight have little influence on the restoration effect of sootiness mural image. From a visual point of view, as shown in Fig. 7b, the restored image basically removed the influence of sootiness on the content of the mural. The lines at the background were clear, and the black marks on the edge of the paint losses region disappeared. The mural image was clear as a whole, and the pattern on it were more attractive. Therefore, this process can restore the blurred mural caused by sootiness contamination, increase its readability, and realize the restoration of sootiness mural.

Table 1 The parameters of Retinex by bilateral filter

Fig. 7

The restoration result of area 1: a true color image; b restoration result

In addition, the processing time of the proposed method was counted as shown in Table 2, which is the processing time of the image with 1392 × 1000 pixels. As the original data is hyperspectral data, it usually takes a long time for radiometric correction and denoising preprocessing. And the subsequent processing is relatively fast. All the experiments were implemented in the environment of an AMD 7 4800H CPU 2.90 GHz PC with 16-GB RAM.

Table 2 The processing time of the proposed method

Discussion

Digital image and hyperspectral image

Hyperspectral image has a high spectral resolution, which is helpful for mining the information covered by surface materials. However, its spatial resolution is relatively low, so a high spatial resolution photo taken by a digital camera was used for comparative experiments. Figure 7a shows a high spatial resolution digital photo taken by Nikon DB810 camera in the study area 1, with a resolution of 4841 × 3688 pixels. The method proposed in this paper and the same parameters were used for processing. The effects are shown in Fig. 8.

Fig. 8

The restoration effects of digital photo and hyperspectral synthetic image of area 1: a digital photo; b restoration effect of digital photo; c hyperspectral image; d restoration effect of hyperspectral image

It can be seen from Fig. 8 that although the image taken by the digital camera has a higher spatial resolution, its restoration effect was not as good as that of the sootiness mural image produced by hyperspectral. Especially for the lines located among the red region, they were clearly visible in the restored image using the hyperspectral image, but still blurred in the restored image using the digital photo.

Image selection during the dark channel prior sootiness removal

Due to the large number of bands provided by the hyperspectral image, it is possible to combine many images with different bands to carry out the method. In order to select the optimal $S\left( x \right)$ and the most suitable image for calculating the dark channel image in the dark channel prior sootiness removal, the following experiments were performed. Take the TCI, the PCI and the SMI as the $S\left( x \right)$ or the image used to calculate the dark channel image, and perform the dark channel prior to remove sootiness. In order to facilitate the observation, the brightness adjustment and the Retinex by bilateral filter in HSV Color Space processing were also carried out for the image after the dark channel prior sootiness removal. It can be seen from Figure a-f in Table 3 that when the PCI was used to calculate the dark channel image, the clothes and details of the character is clearer. It can be seen from Figure g-l that when the PCI was used as the image to be restored, some black masks are disappeared, such as the edge of the white paint losses area on the right side of the image. Therefore, in the proposed method, the PCI was selected to calculate the dark channel image, and the SMI as the $S\left( x \right)$. The restoration effects were shown in Figure p–r.

Table 3 The comparison of sootiness removal results at different steps with different image combination

Combination of different steps

In order to further compare the effect of the proposed method, MNF transformation and dark channel prior sootiness removal steps were omitted, and other processing and parameters were consistent with the proposed method. The effects are shown in Fig. 9.

Fig. 9

The restoration effects of area 1 without different steps of the proposed method: a true color image; b restoration effect without MNF transformation; c restoration effect without prior dark channel; d restoration effect of the proposed method

It can be seen from Fig. 9 that omitting the MNF transformation resulted in more noise in the image, which affected the effect of sootiness removal. If the dark channel prior sootiness removal was omitted, the image was brighter after restoration; however, the details were not prominent, and the restoration effect of sootiness mural was not significant.

Comparison with other methods

Homomorphic filtering is another common algorithm for image defogging, and Gaussian stretching is a common method for image enhancement. In order to compare their effects with that by the proposed method, the above methods were applied to the true color image produced by hyperspectral data in area 1, as shown in Fig. 10. Moreover, variance, average gradient, information entropy and gray scale contrast were introduced as objective evaluation indexes, as shown in Table 4. The larger the values of variance, average gradient, information entropy, and gray contrast, the better is the sootiness restoration effect. In addition, according to the no-reference quality assessment method for defogged images proposed by Li et al. [37], the restoration quality of sootiness mural image was objectively evaluated from three aspects: effective edge strength, color restoration ability and structural information. The calculation results are shown in Table 5. Similarly, the larger the values of effective edge intensity, color restoration ability, structure information, and comprehensive evaluation, the better is the sootiness restoration quality.

Fig. 10

The restoration effects of area 1 with different methods: a true color image; b restoration effect of homomorphic filtering method; c restoration effect of Gaussian stretching method; d restoration effect of the proposed method

Table 4 The statistics performance of different methods

Table 5 The no-reference quality assessment of different methods

In terms of visual effects, as shown in Fig. 10, the Gaussian stretching algorithms significantly improved the brightness of the image, but the processed image was generally white, the contrast was reduced, and details were lost seriously. The brightness of the image restored by homomorphic filtering algorithm was still dim, and the visual effect was not good. The algorithm in this paper can not only improve the brightness of the sootiness mural image, but also enhance the details. In terms of objective quality evaluation, the results in Tables 3 and 4 show that the variance, average gradient, information entropy and gray contrast of the proposed method were superior to other algorithms, and the evaluation of comprehensive edges, hue and structure were better too.

Applicability of restoration method proposed

The study area 2 of the mural on the same wall of Qutan Temple was restored with the same method and parameters. The effect is shown in Fig. 11b. In addition, other six images were restored and mosaicked to one image covering the first Buddha in the left of the mural on the same wall of Qutan Temple, with a size of about 0.6 × 1 m, as shown in Fig. 11c.

Fig. 11

The restoration results of sootiness murals in other study areas of Qutan Temple: a true color image of area 2; b restoration results of area 2; c restoration and mosaic result of the left Buddha

In order to test the applicability of restoration method proposed, other mural images suffered from sootiness were restored in the same way. The same approach was performed on an area on the west wall of Guanyin Temple, Heilongmiao Village, Yanqing District, Beijing, China, which was acquired in July 2017. This temple was built in the reign of Daoguang in the Qing Dynasty (A.D. 1821–1850), with a history of nearly 200 years. As shown in Fig. 12a, the lower right corner of the mural is seriously contaminated by sootiness, the patterns are covered, and it is illegible to the naked eye. After restoration and mosaicking, the pattern under the sootiness can be more clearly identified, and the almost invisible edge in the serious sootiness area reappeared, as shown in Fig. 12b. Among them, there are three images of the lower right corner area of the mural, with the size of about 0.5 × 1.0 m. Therefore, the restoration method of sootiness mural in this paper can achieve the removal of sootiness in different murals and the restoration of large-scale sootiness murals.

Fig. 12

The restoration results of sootiness murals of the lower right corner patterns of Guanyin Temple: a digital image; b restoration and mosaic result

Inpainting of the paint losses in the background

Finally, in order to further improve the visual quality of the mural restoration, the paint losses in the background was inpainted. After the experiment, the image after the preliminary sootiness removal step was selected, the paint losses in the image was inpainted by Criminisi algorithm, and the restoration of sootiness mural by Retinex using bilateral filter was performed on the inpainted image. The final restoration effect is shown in the Fig. 13.

Fig. 13

The final restoration result of area 1: a true color image; b result of preliminary sootiness removal; c result of paint losses inpainting; d final restoration re

In terms of visual effects, after sootiness restoration and paint losses inpainting, the mural image was more enjoyable and attractive, and revealed the original appearance of mural to a large extent.

Conclusions

It is an arduous task in the protection of cultural relics to remove sootiness from murals. The objective of this study is to develop a new method to virtually restore sootiness mural images using hyperspectral imaging technology. By using the advantages of near-infrared bands, the related defogging methods, dark channel prior and Retinex by bilateral filter, were combined to reveal the mural patterns blurred by sootiness contamination. The approach was carried out on several sootiness mural images. The readability and artistic expression of the murals were increased effectively. The experimental results show that the restoration method can remove most of the sootiness, highlight the detail information, make the image brighter, the lines clearer and the pattern more attractive. Good results have also been achieved in the objective evaluation of variance, average gradient, information entropy, gray contrast and comprehensive evaluation. Although, the block histogram matching can make the clothes clearer and color more realistic, the details processing of clothes with complex patterns and dense paint losses still need to be improved. It is necessary to further study the radiometric transmission model which is more suitable for mural sootiness, so as to realize the restoration of sootiness mural more effectively.