Materials
Sootiness
According to the Chinese National Standard for the Protection of Cultural Relics, Ancient wall painting deterioration and legends (GB/T 30237-2013) [28], sootiness is the mark of the mural being polluted by soot or incense, as shown as an example, in Fig. 1. Due to the contamination of sootiness, the patterns in the murals are often blurred, which seriously affects the value of appreciation of the murals.
Mural data
The mural selected in this study is located on the north indoor wall of Daheitian Hall on the east side of Qutan Temple, located at Ledu District, Haidong City, Qinghai Province, China. According to the site’s records in China, the temple was built in the 25th year of Minghongwu (1392 A.D.), with a history of more than six hundred years [29]. As shown in Fig. 2a, due to the activities of incense burning and worship Buddha for a long time, the mural is seriously contaminated by sootiness. Some of the patterns in the mural are covered and cannot be recognized by the naked eye. In order to reduce the influence of sootiness on the mural and restore the information covered by sootiness, the hyperspectral data of the mural were captured and analyzed.
Data acquisition
In July 2018, the mural data of the study areas were captured by using the VNIR400H hyperspectral imaging system of Themis Vision Systems with the spatial resolution 1392 × 1000 pixels and 1040 bands covering from 377.45 nm (visible light) to 1033.10 nm (near-infrared). The spectral sampling interval was 0.6 nm, and the spectral resolution was 2.8 nm.
During the data acquisition, the distance between the hyperspectral camera and the mural was about 1 m. The sunlight was blocked by closed doors and windows, and a pair of halogen lamps whose spectral distribution is close to that of sunlight were used for illumination. A total of 24 hyperspectral images were collected, covering most of the sootiness area of the north wall of the Daheitian Hall. The images of the two study areas, shown in Fig. 2b, c, are the true color images produced by the red, green and blue bands with wavelengths of 460.20 nm, 549.79 nm and 640.31 nm after radiometric correction and data denoising.
Restoration method
Figure 3 shows the overall workflow of the proposed method for the restoration of sootiness mural images, including four main steps: (1) Data preprocessing using radiometric correction and data denoising, (2) sootiness mural image produce using block histogram matching of pseudo color image and true color image, (3) preliminary sootiness removal using dark channel prior and image brightness adjustment, and (4) restoration of sootiness mural using Retinex by bilateral filter in HSV (hue, saturation, value) color space. The details of each step are discussed in the following sections.
Data preprocessing and sootiness mural image produce
The original data captured by hyperspectral imaging system are in radiance and the data range is depending on the number of digitalization bits of the system. The radiance is the radiation energy reflected by the target and received by the sensor in a certain direction of space per unit area per unit time and per unit solid angle. Even for the same target point, the radiance will change with the change of the incident energy. However, the reflectance of a certain material is usually unique and independent of the external illumination, so it is commonly used to study the natural characteristic of the target. Therefore, we will convert the original hyperspectral data into reflectance images by using Eq. (1) [30, 31]:
$$R = \frac{{R_{raw} - R_{dark} }}{{R_{white} - R_{dark} }}$$
(1)
where \(R{ }\) is the corrected reflectance image; \(R_{raw}\) is the original image of the mural; \(R_{white}\) is the standard reflector data obtained on site; and \(R_{dark}\) is the dark current data acquired with the light source off and the lens covered. The reflectance of the standard reflector is 99%.
In addition, in the data acquisition of hyperspectral imaging system, there will be some noise bands due to the changes of environmental parameters and the interference of dark current noise. Through the inspection of the data, it was found that the bands at both ends of the sensor’s wavelength spectrum were noisy, that is, the bands with the shortest wavelength and the bands with the longest wavelength. After observing the spectral curve of random pixels on the image, the 51–990 bands (405.79–1000.79 nm) were manually selected for subsequent processing in the 1040 bands acquired.
In order to further reduce the noise, the selected data were processed by the minimum noise fraction rotation (MNF), which is a common method of dimension reduction and denoising in hyperspectral data processing [32]. It can transform the noise covariance matrix of the data and the noise whitening data, and retain the principal component with relatively large signal-to-noise ratio, so as to realize the dimensionality reduction and denoising of hyperspectral data. In this paper, the MNF transformation was performed on the hyperspectral data after manual selection to separate the noise from the information in the data. The top n components with more than 95% information content were selected for inverse MNF transformation, and the hyperspectral data dimension was restored, and the data denoising was realized.
It was observed that the near-infrared bands can reveal the information under the surface material coverage to a certain extent, which is conducive to the removal of sootiness. Therefore, the near-infrared, green and blue bands with wavelengths of 845.77 nm, 549.79 nm and 460.20 nm were selected to produce a pseudo color image (PCI) in the denoising hyperspectral image. The red, green, and blue bands with wavelengths of 640.31 nm, 549.79 nm, and 460.20 nm were selected to produce a true color image (TCI). However, by comparing the image quality of PCI and TCI, it was found that some of the patterns in the murals were clearer and more realistic in the TCI. Thus, PCI is calibrated to obtain the sootiness mural image (SMI) by block histogram matching method which adopted the TCI as the reference, as shown in Fig. 4. In the following sections, the TCI is used as a reference image in comparison of restoration effect. The PCI is used to obtain the dark channel image in the dark channel prior sootiness removal, so as to solve the scattering light value and transmittance. SMI is used as the sootiness mural image to be restored for the subsequent restoration of sootiness mural.
Preliminary sootiness removal
In the proposed method, the dark channel prior was applied to solve the atmospheric scattering model for the preliminary removal of sootiness in mural images. The influence of sootiness on mural image is similar to that of fog, which usually covers the mural pattern in a large area. The more serious the fog or sootiness, the more blurred the pattern and color in the image, or even completely invisible. Although the particles of sootiness and fog are different, they will cause some light to be scattered by the particles and the light intensity will be weakened when the incident light contacts with the particles. Therefore, the atmospheric scattering model was used to describe the formation process and method of sootiness mural images, such as Eq. (2) [33]:
$$S\left( x \right) = D\left( x \right)t\left( x \right) + A\left( {1 - t\left( x \right)} \right)$$
(2)
where \(S\left( x \right)\) is the sootiness mural image to be restored; \(D\left( x \right)\) is the image of target scene after preliminary sootiness removal; \(t\left( x \right)\) is the medium transmittance of the sootiness; and \(A\) is the scattering light value caused by sootiness. The purpose of the restoration of sootiness mural is to recover \(D\), \(A\) and \(t\) from \(S\).Step 1: Estimate the scattering light \(A\). The dark channel prior is a statistical rule proposed by He et al. [16]. It points out that in most fog-free images (non-sky areas), there will be some areas, at least one color channel has some pixels whose intensity are very low and close to zero. This channel is called dark channel. This is similar to the situation of mural images covered by sootiness in that in the areas with light or even no sootiness, the pattern and color of the image are visible, and at least one color channel has some pixels whose value are very low and close to zero for the red, blue, or green color areas. But in areas with heavy fog or sootiness, the content of the image will be covered and illegible. The mathematical expression is as follows:
$$P^{dark} \left( x \right) = \mathop {\min }\limits_{z \in \Omega \left( x \right)} \left( {\mathop {\min }\limits_{{c \in \left( {r,g,b} \right)}} P^{c} \left( y \right)} \right)$$
(3)
where\(c\) is a color channel among \(r\), \(g\), and \(b\); \(P^{C}\) is the gray value of a channel of \(P\); \(\Omega \left( x \right)\) is a local patch centered at \(x\). A dark channel image is the result of two minimum operators: \(\mathop {min}\nolimits_{{c \in \left( {r,g,b} \right)}}\) is the minimum value of each pixel in the \(r, g,{ }b\) channel; \(\mathop {min}\nolimits_{z \in \Omega \left( x \right)}\) is a minimum filter.
The dark channel image was calculated from the PCI using the Eq. (3), and the maximum value of dark channel was selected as the estimated value of scattering light.
Step 2: Calculate the transmission \(t\). According to the method of dark channel prior, for the fog-free image, the dark channel is \(P^{dark} \to 0\). So that when the scattering light value had been obtained, the transmission \(t\) can be obtained. The constant parameter \(\omega (0 < \omega \le 1)\) was introduced to retain the perspective depth of the image and make it more realistic:
$$t\left( x \right) = 1 - \omega \mathop {\min }\limits_{c} \left( {\mathop {\min }\limits_{y \in \Omega \left( x \right)} \left( {\frac{{P^{c} \left( y \right)}}{{A^{c} }}} \right)} \right)$$
(4)
Step 3: Restore the image \(D\left( x \right)\). After the scattering light value A and transmittance t were obtained, the SMI was taken as the \(S\left( x \right)\) to be restored, and then the image \(D\left( x \right)\) was solved according to the atmospheric scattering model. When the transmission approached zero, direct recovery of \(D\left( x \right)\) was prone to noise. Therefore, the minimum threshold \(t_{0}\) was set to control the transmission, so:
$$D\left( x \right) = \frac{S\left( x \right) - A}{{max\left( {t\left( x \right),t_{0} } \right)}} + A$$
(5)
Step 4: Brightness adjustment. The brightness of the image \(D\left( x \right)\) was low after removing the sootiness via the dark channel prior, so the image was converted to the HSV color space, and the image brightness was adjusted by setting a brightness factor and multiplying it with the Value component. Thus, the mural image \(B\left( x \right)\) with preliminary sootiness removal was obtained, and the details of the mural were enhanced while removing the interference of sootiness.
Fine sootiness removal
In order to further remove the sootiness from mural image, defogging method of Retinex by bilateral filter is applied to the sootiness mural image. Retinex method considers the brightness of the object perceived by the human eye as an organic combination of the illumination of the environment and the reflection of the object surface [34]. The illumination component can be estimated from the original image to obtain the reflection component, that is, the color of the object itself. The mathematical expression of Retinex method is:
$$B\left( x \right) = R\left( x \right) \cdot L\left( x \right)$$
(6)
where \(B\left( x \right)\) is the image pixel value received by the human eye or camera, that is the image after preliminary sootiness removal in the proposed method; illumination image \(L\left( x \right)\) is the illumination component of the ambient light; and reflection image \(R\left( x \right)\) is the reflection component of the object.
Take logarithm on both sides of Eq. (6):
$$\ln \left[ {B\left( x \right)} \right] = \ln \left[ {R\left( x \right)} \right] + ln\left[ {L\left( x \right)} \right]$$
(7)
The reflection component of each color channel of the image is
$$\ln \left[ {R_{i} \left( x \right)} \right] = \ln \left[ {B_{i} \left( x \right)} \right] - ln\left[ {L_{i} \left( x \right)} \right]$$
(8)
where \(B_{i} \left( x \right)\), \(R_{i} \left( x \right)\), \(L_{i} \left( x \right)\) are the channel image, the reflection component and the illumination component of the \(i\) color channel of the image respectively. The usual method is to use Gaussian function to estimate the illumination image from the image, so as to solve the reflection image.
Bilateral filter [35] is a nonlinear filter, which combines the spatial proximity and pixel similarity of image, and can consider both spatial information and gray similarity. Compared with Gaussian filtering, bilateral filtering can effectively remove most noise while keeping image details. During the filtering, edge keeping and denoising are better realized by adjusting the filter size p which represents the diameter of each pixel neighbourhood, the weight \(\sigma_{r}\) which controls the change of gray scale, and the weight \(\sigma_{s}\) which controls the change of spatial distance.
In the proposed method, the defogging method of Retinex by bilateral filter [16] was used to further improve the visual effect of the sootiness mural by setting two bilateral filters with different weights and parameters. First, the image \(B\left( x \right)\) after the preliminary sootiness removal was converted from RGB space to HSV color space. Second, only for Value component, the illumination image was estimated by bilateral filtering. Then, the logarithm of the original image and illumination image were taken, and the reflection image was solved by using the second bilateral filtering according to the Retinex method, so as to obtain the recovered image of the V component. Finally, the HSV space image was converted to RGB (red, green, blue) space to realize the restoration of sootiness mural.
Other method to be discussed
Homomorphic filtering is a method to compress the image brightness range and enhance the contrast in frequency domain [36]. It is based on the illumination and reflection model of the image. By adjusting the gray range of the image, it can enhance the detail information of the dark area without losing the details of the bright area. Gaussian stretching is an interactive histogram stretching method in radiation enhancement. By stretching the histogram of the output image into a Gaussian function, the detail information of the image is enhanced. Homomorphic filtering and Gaussian stretching will be used in the discussion and compared with the proposed method.