Restoration of an ancient temple mural by a local search algorithm of an adaptive sample block

Abstract

Long-term influences in the external environment, including light, temperature and humidity, have caused varying degrees of damage to ancient Chinese murals. Allowing people to appreciate the original style of the murals has become important to experts, and the development of image processing and machine learning technology has allowed intelligent restoration of ancient murals. This paper proposed the adaptive sample block and local search (ASB–LS) algorithm based on the Criminisi algorithm to address the flaking deterioration of Kaihua Temple murals from the Song Dynasty. ASB–LS achieved virtual restoration of damaged areas. First, the mural’s compositional characteristics were analyzed, the structure tensor was introduced, and the data items were redefined using eigenvalues to ensure accurate transmission of the image’s structural information. Then, the data item was used to form a new priority function to improve the image filling order. Finally, the sample block size was adaptively selected by the average correlation of the structure tensor, and a local search strategy was used to improve matching efficiency, which effectively avoided mispropagation of the restored image structure and blinded search of the matching block. Experiments were performed on the Song Dynasty murals in the Kaihua Temple for two types of deterioration: flaking deterioration and artificial destruction. Compared with the Criminisi algorithm and two improved algorithms, the proposed ASB–LS algorithm had better subjective analysis and objective evaluation. Subjective visuals significantly improved and conformed to the image’s compositional characteristics, and the inpainting time efficiency improved, establishing a good foundation for restoring ancient murals.

Introduction

Ancient murals record large amounts of important information about people’s production, life and hobbies at that time. These murals were witnesses of ancient civilizations and are treasures of historical material and cultural heritage. They provide an important reference for people to understand changes in ancient painting and artistic style. However, due to natural weathering and destruction by human factors, ancient murals show considerable signs of deterioration, such as cracks, flaking, and fading. Traditional mural restoration work is irreversible and risky. However, applying digital image restoration technology to the restoration of ancient murals can provide advanced technical support for mural restoration work. Digital murals use computers to perform virtual restoration to protect and restore murals, greatly reducing the risk of protection and restoration work. This approach can permanently preserve mural information. Therefore, to preserve these precious cultural relics in the long term, the digital protection and restoration of murals has become a popular research topic.

In recent years, research scholars have examined the issues of intelligent protection and digital restoration for ancient murals. Izzo et al. [1] focused on two cases of Mario Sironi and Edmondo Bacci in Venice and conducted in-depth research on various materials and ages of Italian mural painting techniques to understand their protection requirements and to develop a sustainable protection plan. Sakr et al. [2] studied the color effect and solution of Streptomyces on ancient Egyptian tomb murals, which laid the foundation for the digital protection of ancient murals. Abdel-Haliem et al. [3] isolated and identified Streptomyces, which mainly affect the color change in ancient Egyptian tomb murals, and provided new ideas for their removal. Li et al. [4] focused on the reinforcement and effectiveness of the Mogao Grottoes and performed an on-site inspection to determine whether a previously repaired mural had new deterioration. The digital restoration of ancient Chinese murals began with the study of the Dunhuang murals. Pan et al. [5] constructed a Dunhuang mural protection and restoration system architecture from a macro perspective using digital image restoration technology. Yang et al. [6, 7] applied the improved Criminisi algorithm, Markov sampling and image decomposition techniques to the digital restoration of the Dunhuang murals and achieved good repair results. Later scholars studied the digital restoration of other ancient murals. Shen et al. [8] used morphological component analysis (MCA) [9, 10] to decompose murals into two parts, the structure and the texture, and repaired them separately, aiming at the cracks and mud spots in the murals of the Tang Dynasty tombs and further improving the repair effect. Jiang et al. [11] proposed an automatic calibration and restoration method for Tibetan murals with mud spot disease based on sample block priority that could accurately calibrate the mud spot disease in Tibetan digital murals and perform simulated restoration. Jiao et al. [12] proposed an improved block matching Wutai Mountain mural digital repair algorithm for the irregular shape and uneven size of the damaged area of the Wutai Mountain mural that contributed to solving the problem of error filling. Although certain results have been achieved in the intelligent restoration of ancient murals, the digital restoration of murals is still in its infancy, and the restoration effects require further improvement.

Image restoration algorithms play an important role in the mural restoration process. Currently, the algorithms for image restoration are divided into two main categories. One category is a digital image restoration algorithm based on the partial differential equation (PDE) proposed by Bertalmio et al. [13]. The algorithm has a good restoration effect on small-scale damage, such as scratches, in the processed image. However, when the damaged area is large or the surrounding texture is very rich, the repair effect is poor. The other category is the exemplar-based repair algorithm proposed by Criminisi et al. [14], which achieves satisfactory results in repairing large-area damage. However, due to the seriousness of flaking deterioration in ancient temple murals and the large size of flaking areas, if the Criminisi algorithm is used for virtual repair, the repair effect heavily depends on the repair order of the boundary points. Furthermore, a global search is used in the matching block search, which requires considerable time. To address this defect, Liang et al. [15] changed the priority calculation method from a multiplication to an additive form for data items and confidence items and added a normalization factor. Liu et al. [16] modified the confidence term into an exponential form, which can reduce its tendency to rapidly fall to zero and can add a positive planning factor that the user can choose. These two improved algorithms have a better effect on the repair of damaged images with a simple structure, but for images with rich texture, the structure is discontinuous. Liu et al. [17] constructed a local structure metric function by introducing structural tensor theory, optimized the priority function and proposed a new matching criterion. The algorithm has good preservation for strong edge structure of images, but the phenomenon of structural fracture occurs when weak edge structure is repaired. Siadati et al. [18] more accurately distinguished the significant structure of an image by combining the structural tensor and gradient and defined a new priority function to repair the structural part of the image first. When the algorithm repairs complex structures, its effect is not ideal, and the running time is very long. Alilou et al. [19] reduced the calculations between matching blocks by setting the optimal distance in advance, which reduced the time complexity of the algorithm but resulted in a mismatched block.

The improved algorithm described above can still cause structural discontinuity when inpainting complex mural images. The reason for this analysis is mainly the priority function and the use of fixed sample blocks when inpainting different structural regions. To address the above problems, this paper proposed an adaptive sample block and local search algorithm (ASB–LS) based on the composition features of murals. The structure tensor was introduced in the calculation of the data item, and the adaptive sample block was used to fill the failing area in different structural areas. At the same time, a local search strategy was used to improve the search mode of the matching block, which improved the repair effect and repair efficiency of temple murals.

Methods

Basic theories

Criminisi algorithm

The Criminisi algorithm is an image sampling process driven by isophotes that is able to effectively retain the structural texture information of the image and achieve a satisfactory inpainting effect for images with wide-range missing information [14]. The core of this algorithm is that it takes the filling priority of the region to be restored into consideration. Specifically, it calculates the priority levels of all pixels of the boundary contour first. Then, it takes the pixel with the highest priority as the center to generate a to-be-inpainted block of a fixed size. Subsequently, an optimum matched block is searched in the intact region based on the matching criterion. Finally, the unknown pixels in the to-be-repaired block are filled with the known ones in the optimum matched block, and the confidence levels of the boundary and the post-filling to-be-repaired block are updated. The inpainting diagram is shown in Fig. 1, where $\varOmega$ is the flaking area, $\delta \varOmega$ is the boundary of the flaking area, $\psi$ is the intact area, and $\psi_{p}$ is a pixel block with a size of 9 × 9 at the pixel P on the boundary. The use of this algorithm to inpaint images includes the following three steps.

Fig. 1

Schematic diagram of the Criminisi algorithm

1. Priority calculation of the boundary points

$$P\left( p \right) = C\left( p \right) \times D\left( p \right),$$

(1)

where $C(p)$ represents the confidence item and $D(p)$ represents the data item. They are defined as follows:

$$C(p) = \frac{{\sum\nolimits_{{q \in \psi_{p} \cap \psi }} {C(q)} }}{{\psi_{p} }}$$

(2)

$$D(p) = \frac{{\left| {\nabla I_{p}^{ \bot } \cdot n_{p} } \right|}}{\alpha },$$

(3)

where $\left| {\psi_{p} } \right|$ indicates the total number of pixels in the to-be-repaired block, $\alpha$ is a normalization factor (for grayscale images, $\alpha$ is usually set at 255), $n_{p}$ is the unit normal vector of the area to be repaired at point P, $\nabla I_{p}^{ \bot }$ indicates the direction of the illuminance line of the P point, which can also be expressed by the gradation gradient, $\nabla I_{p}^{ \bot } = \left( { - I_{y} ,I_{x} } \right)$, and $I_{x}$ and $I_{y}$ represent gradients in the x-direction and the y-direction, respectively.

2. Select the best matching block

First, the boundary point $\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{p}$ with the highest priority is found and then the block to be repaired $\psi_{{\hat{p}}}$ centering on the point is generated. The sum of squared difference (SSD) criterion is used to find the best-matching block $\psi_{q}$ with the squared difference between the known partial pixels of $\psi_{{\hat{p}}}$ and the smallest in the intact region ($\psi$). Finally, the failing area of the corresponding position in $\psi_{q}$ is replaced with $\psi_{{\hat{p}}}$ as follows:

$$\psi_{q} = \arg \hbox{min} d(\psi_{{\hat{p}}} ,\psi_{q} )$$

(4)

$$d(\psi_{{\hat{p}}} ,\psi_{q} ) = \sum {(R(x) - R(y))^{2} + (G(x) - G(y))^{2} + (B(x) - B(y))^{2} } ,$$

(5)

where $x \in \psi_{{\hat{p}}} \cap \psi$, y is the point corresponding to the position in $\psi_{q}$, R, G, and B are the pixel values on each channel of each pixel, $\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{p}$ represents the pixel with the highest priority on the boundary, $\psi_{{\hat{p}}}$ represents the to-be-repaired 9 × 9 block with $\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{p}$ as the center, and $\psi_{q}$ represents the optimum matched block of $\psi_{{\hat{p}}}$ (i.e., with the least SSD). After the optimum matched block $\psi_{q}$ is determined, the pixel values in the unknown area are replaced with the pixel values at the corresponding position in $\psi_{q}$.

3. Update the boundary and confidence

After the failing area in $\psi_{{\hat{p}}}$ is repaired, it is necessary to update the boundary value $\partial \varOmega$ of the failing area and the confidence value of the boundary point:

$$C(p) = C(\hat{p}),$$

(6)

where $p \in \psi_{{\hat{p}}} \cap \varOmega$ is the detached pixel point in block $\psi_{{\hat{p}}}$ to be repaired and $\hat{p}$ is the pixel point with the highest priority. We repeat steps (1)–(3) until the entire flaking area $\varOmega$ is filled.

However, the Criminisi algorithm suffers from the following drawbacks. First, this algorithm employs the product of the multiplication of the data and confidence items directly in priority calculation. Therefore, if one of the items is zero, the calculated priority will be zero. In this case, the calculation will result in an unreasonable inpainting order. Second, this algorithm adopts a global search for the matched block without considering the possibility of local resemblance of the image. When the image is too large, the inpainting time will be long, which leads to low efficiency. In addition, the Criminisi algorithm does not take the composition features of murals into consideration during inpainting. Considering these drawbacks of the Criminisi algorithm, this study introduced the structural tensor of the image, redefined the data item and priority function, and selected different block sizes adaptively for different structural regions of the mural to perform local searches for the optimum matched block. The inpainting effect was efficiently improved.

Structural tensor of the image and composition characteristic of murals

1. Structural tensor of the image

The structural tensor was introduced into image processing by Förstner and Gülch in 1987 and is usually used to analyze the local geometric structure of an image, mainly distinguishing between flat, edge and corner regions of the image [20, 21]. The tensor here is a structural matrix of the image that contains the intensity information of the local region, the main directions in the gradient of the specific pixel neighborhood and the coherence degree of these directions.

Assume that the gradient vector $I$ of the image is

$$\nabla I = \left(_{{I_{y} }}^{{I_{x} }}\right).$$

(7)

In Eq. (7), $I_{x}$, $I_{\text{y}}$ is the partial derivative of the $x,y$ direction, $I_{x} = \partial I/\partial x$, and ${\text{I}}_{y} = \partial I/\partial y$. Then, the structural tensor ${\text{J}}_{p}$ of the image is defined as

$$J_{0} = \nabla I\nabla I^{\rm T} = \left[ {_{{I_{x} I_{y} }}^{{I_{x}^{2} }} \left. {{}_{{I_{y}^{2} }}^{{I_{x} I_{y} }} } \right]} \right.,$$

(8)

$$J_{p} = G_{\sigma } *J_{0} = \left[ {_{{G_{P} *I_{x} I_{y} }}^{{G_{p} *I_{x}^{2} }} \left. {_{{G_{p} *I_{y}^{2} }}^{{G_{P} *I_{x} I_{y} }} } \right]} \right..$$

(9)

In Eq. (9), ${\text{G}}_{\sigma }$ is a two-dimensional Gaussian kernel function with a mean of 0 and a variance in $\sigma ,G_{\sigma } = \frac{1}{{2\pi \sigma^{2} }}\exp \left( { - \frac{{x^{2} + y^{2} }}{{2\sigma^{2} }}} \right).$ $J_{p}$ is convoluted after the Gaussian function, which suppresses the noise level and better reflects the image edge information and direction information [22]. Since $J_{p}$ is a symmetric and semipositive two-dimensional matrix, there are two nonnegative eigenvalues. The sizes are as follows:

$$\lambda_{ 1, 2} { = }\frac{ 1}{ 2}\left( {{\text{j}}_{11} + J_{22} \pm \sqrt {(j_{11} - j_{22} )^{2} + 4j_{12}^{2} } } \right),$$

(10)

where $\lambda_{ 1} ,\lambda_{ 2}$ represent the maximum and minimum values, respectively, of the eigenvalues for the structure tensors at the pixel points reflecting the intensity on the local edges of the image. When both eigenvalues are small, the gradation value of the pixel in the domain is small and belongs to a smooth region. When both feature values are large, the gray value of the pixel in the domain changes substantially, and the pixel point is in the corner region of the image. When it is larger or smaller, the gray value has a strong change in one direction.

2. Compositional characteristics of murals

The compositional characteristics are unique features of murals, landscape paintings, and other artwork that are different from ordinary pictures and are structural configuration features of the artistic image in the works. Good composition can make the work content clear and distinct and the theme outstanding and pleasing to the eye [23]. The composition characteristics are roughly divided into horizontal, vertical, and S-shaped categories. The mural compositions of different dynasties have their own characteristics. The repair of murals not only completes the filling of the flaking area but also maintains the overall composition of the image during repair.

Adaptive sample block and local search (ASB–LS) algorithm

This paper made full use of the characteristics of structural tensors and combined the mural composition features to improve the Criminisi algorithm. An adaptive sample and local search (ASB–LS) algorithm were proposed.

Priority function

According to the structure tensor, to distinguish the characteristics of different structural regions in the image, the feature values of the structural tensor were introduced into the data item, and the new data image is defined as follows [18]:

$$D(p) \, = \, 1 - \exp \left[ { - (\lambda_{1} - \lambda_{2} )^{2} } \right].$$

(11)

As seen in Eq. (11), when pixel point P is at the edge of the structure, $\lambda_{1} - \lambda_{2} > > 0,$ then $D(p) \approx 1.$ When P is in a flat area, $\lambda_{1} - \lambda_{2} \approx 0,$ then $D(p) \approx 0.$ When $\lambda_{1} - \lambda_{2} > 0,$ then $0 < D(p) < 1,$ so the pixel point is in the texture area and the data item $D(p)$ is a more accurate representation of the structural characteristics of the image.

In addition, in the traditional Criminisi algorithm, the priority function $P(p)$ is defined in the form of a product according to formula (1). When the data item or confidence item suddenly becomes zero, the repair priority of this pixel point will also be zero, which will affect the repair order and produce incorrect repair results. To make the confidence term and the data item smooth, this paper used the redefinition priority function in which the improved data term and exponential function form of the confidence term were linearly weighted as follows:

$$P(p) = \alpha \cdot \exp (C(p)) + \beta \cdot D(p),$$

(12)

where $\alpha$, $\beta$ are the weights of the data items and confidence items, respectively, and $\alpha + \beta = 1$. To prioritize repair of the structural part of the image, the weight of the priority function is set to $\alpha < \beta$.

Adaptive sample block

In the traditional Criminisi algorithm, when searching for the optimal matching block, the sample block is usually set to a fixed size, that is, a 9 × 9 pixel block, which results in structural disorder and discontinuity after the complicated image is repaired. To address this defect, this paper distinguished the characteristics of image flatness, texture and edge according to the eigenvalues of the structure tensor and used different sample block sizes for different structural regions to search, which effectively suppressed the mispropagation of structural information. To determine the size of the adaptive sample block, the average correlation factor $Avg_{cor}$ of the pixel block [24] was introduced and defined as follows:

$$Avg_{cor} = \left( {\frac{{\bar{\lambda }_{1} - \bar{\lambda }_{2} }}{{\bar{\lambda }_{1} + \bar{\lambda }_{2} }}} \right)^{2} ,$$

(13)

where $Avg_{cor}$ represents an average correlation factor of 9 × 9 blocks to be repaired, $\psi_{p}$ is centered on one pixel point P, and $\bar{\lambda }_{ 1} ,\bar{\lambda }_{ 2}$ are the average of the known pixel point structure tensors in the block to be repaired $\psi_{p}$. When $Avg_{cor}$ is large, the pixel block is at the edge of the image or the texture structure area, so a smaller sample block should be used for matching repair. When $Avg_{cor}$ is small, the pixel block is located in the smooth region of the image, so a larger sample block is used for the matching repair. In this study, the appropriate sample block size was determined by repeating experiments. The area marked by the green frame in Fig. 2 was a pixel block of 9 × 9 in different structure areas.

Fig. 2

Pixel blocks of 9 × 9 in different structure areas

Table 1 presents the average correlation factors for different structural regions in the marked mural in Fig. 2, where 1, 2, and 3 pixel blocks are in the smooth region of the image; 4, 5, and 6 pixel blocks are in the texture region of the image; and 8 and 9 pixel blocks are in the edge area of the image.

Table 1 Average correlation factors of pixel blocks in different structural regions

As shown in Table 1, when the pixel block is located in the smooth region, the value of $Avg_{cor}$ is usually less than 0.65; when the pixel block is located in the texture region, the value of $Avg_{cor}$ fluctuates at approximately 0.8; and when the pixel block is located in the edge region, the values of $Avg_{cor}$ are all above 0.9. Therefore, to maintain the composition characteristic of the mural, the structural information of the inpainting image was accurately propagated. When the matching block is searched, the different structural regions of the image can be accurately distinguished according to the average correlation factor. In this paper, the change in $Avg_{cor}$ was used to adaptively select the size of the sample block to fill the failing area. The size of the adaptive sample block was determined as follows [18]:

$$size\,(p) = \left\{ {\begin{array}{*{20}l} {11 \times 11} , & \quad {Avg_{{cor}} \le 0.65} \\ {9 \times 9} , & \quad {0.65 < Avg_{{cor}} } \\ {5 \times 5} , & \quad {Avg_{{cor}} \ge 0.9} \\ \end{array} } \right.$$

(14)

The value of $size(p)$ was determined based on the post-repair effect as well as the time span for repair. First, three intervals were divided according to $Avg_{cor}$. Then, repeated experiments on multiple images were performed in each interval, and the sample block size $size(p)$ was determined when the best repair effect was achieved.

Matching block search strategy

The traditional Criminisi algorithm uses the global search strategy when searching for the best matching block, that is, the one that is most similar to the sample block in the unbroken area of the whole image. This method can search for the best matching block, but it takes a long time. According to the composition characteristic of the mural image, the matching blocks of many failing areas are located near the sample block. Therefore, the global search has blindness, which reduces the efficiency of the algorithm. To reduce the search space of the algorithm and ensure the effect of the repair, this paper adopted a local search strategy, namely, searching for the best-matching block in the local window space. The size $Length$ of the partial window was set as follows:

$$Length = \eta * \sqrt {size(p)} ,$$

(15)

where $\eta$ is the step size. The local search space is centered on the pixel P with the highest priority and extends $Length$ long distances in the upward, downward, left, and right directions, so the actual search space of the local search strategy is $\left( {2 * Length + 1} \right)\left( {2 * Length + 1} \right)$. Compared to the global search strategy, the size of the search window is reduced from $M \times N$ (M and N are the rows and columns of the image, respectively) to 2 (m and n are the rows and columns of the partial window, respectively). The local search shortens the search time of the matching block, thereby improving the repair efficiency of the algorithm.

ASB–LS algorithm description

The specific implementation steps of the ASB-SL algorithm proposed in this paper were as follows:

• Input: The mural image to be repaired in which automatic calibration of the flaking area has been completed [25].

• Output: The restored mural image.

• Step 1. Determine the boundary of the shedding area;

• Step 2. Calculate the data item according to Eq. (11) and calculate the priority of each pixel on the boundary according to Eq. (12);

• Step 3. Find the pixel point with the highest priority among the boundary pixel points and calculate the average correlation factor of the 9 × 9 size pixel block centered on the pixel point according to Eq. (13);

• Step 4. Adaptively determine the optimal size $size(p)$ of the block to be repaired $\psi_{{\hat{p}}}$ according to Eq. (14);

• Step 5. Find the best-matching block in the known area of the image according to the local search space determined by Eq. (15);

• Step 6. Copy the best-matching block to the failing area of the corresponding position in $\psi_{{\hat{p}}}$;

• Step 7. Update the confidence value of the failing area according to Eq. (6);

• Step 8. Repeat steps 1–7 until the pixels in the flaking area $\varOmega$ are filled.

The ASB–LS algorithm flow chart is shown in Fig. 3.

Fig. 3

ASB-SL algorithm flow chart

Results and discussion

To verify the effectiveness of the proposed ASB–LS algorithm in mural image restoration, this study used VS2015 to conduct simulation experiments on a computer with a 3.20 GHz processor and 4G memory. For artificially damaged mural restoration, the damaged region was artificially demarcated. When some pixels with a pixel value < 255 exist on the boundary of the damaged region, noticeable restoration traces will be left after restoration. Therefore, preprocessing of the damaged image is necessary. To do this, the masks of the damaged region should be obtained first, and then appropriate corrosion and expansion operation should be performed. Finally, additive operation of the masks and the original image is performed to obtain the calibrated image. For naturally damaged images, however, no such pretreatment is needed. In this study, the values of $\alpha ,\beta ,\sigma ,\eta$ were set at 0.35, 0.65, 0.4, and 10, respectively, according to the outcomes of repeated experiments. The time efficiency and repair effect based on the proposed method in this study were compared with those based on the Criminisi algorithm, the method by Liang et al. [15], and the method by Siadati et al. [18], respectively. Because the actual flaking mural image lacks an intact reference object, it is difficult to evaluate with PSNR (peak signal-to-noise ratio). This paper added a repair experiment after the artificial destruction of the intact mural image to further prove the effectiveness of the ASB–LS algorithm.

Actual flaking mural repair comparison

In this paper, the mural paintings of the Song Dynasty in the Kaihua Temple were taken as the research object, and the virtual damage was solved to address the flaking deterioration of the mural image. The repair results of each algorithm are shown in Fig. 4.

Fig. 4

Comparison of the inpainting effects for different algorithms on damaged murals. a Murals with calibrated flaking disease. b Restoration effect of the Criminisi algorithm. c Restoration effect of the algorithm in the literature [15]. d Restoration effect of the algorithm in the literature [18]. e Restoration effect of the algorithm proposed in this study

Since the image flaking area was large, the image content on the center of the flaking area was not known, so it could only be compared from the repair result at the flaking boundary. The comparison results in Fig. 4 show that the red box marking the inpainting result with the Criminisi algorithm in images 5, 6, and 7 has obvious diffusion marks at the edge of the image. The literature [15, 18] improved on the basis of the Criminisi algorithm, and the diffusion trace problem also improved. However, the resulting information of the part marked by the green frame was confusing, and the part marked by the yellow frame was visually disconnected and had a breakage problem. In this paper, the structure tensor redefined the data item and the priority function so that the image structure-rich information area was preferentially repaired. The repaired image had fewer diffuse traces at the shedding boundary and better visual connectivity. Next, we analyzed the repair effect from both subjective and objective aspects.

Subjective analysis

Because subjective evaluation results tend to vary from person to person, this paper used the scoring method. Twenty students with normal vision (10 male students and 10 female students) were randomly selected from the laboratory for appropriate training. The participants scored the effect structure continuity, and the overall mural composition after different algorithms was used for repair. In addition, to ensure higher persuasiveness of the outcomes, three experts in image restoration (a professor from the Institute of Graphics and Images of the same university of the current team and two experts from other universities) and three experts in mural restoration (one professor specializing in arts and crafts from the same university of the current team and two investigators from the Mural Research Center of the Shanxi Provincial Bureau of Cultural Relics) were invited to evaluate and score the repair effect. The score was divided into 5 levels (very satisfied, satisfied, average, dissatisfied, very unsatisfied), and the corresponding scores were 5, 4, 3, 2, and 1. Table 2 summarizes the average scores assigned by the 20 students and by the 6 experts on the repair effect on the murals shown in Fig. 4 (detailed scores for each of the mural images are provided in Additional file 1: Table S1, Additional file 2: Table S2, Additional file 3: Table S3, Additional file 4: Table S4).

Table 2 The average scoring results of the subjective evaluation of the repair effect

As shown in Table 2, although there are differences between the average scores assigned by the experts and those assigned by the students, they show a consistent tendency. The algorithm proposed in this study exhibited advantages over other methods with regard to the structural continuity and overall image composition of the repaired images.

Objective analysis

In this paper, the search mode of matching blocks was changed from a global search to a local search, and different sizes of sample blocks were searched in different ranges of the local space. The search space of the matching block was reduced from the original $M \times N$ (M, N is the row and column of the image, respectively) to $m \times n$ (m, n is the row and column of the local window, respectively), which greatly improved the efficiency of the repair while ensuring the repair effect. The virtual repair experiment was performed on the image in Fig. 4. The repair time of different algorithms is shown in Table 3.

Table 3 Objective comparison of repair time (S) for different algorithms

The results in Table 3 show that the algorithm proposed in this paper significantly improves the image restoration efficiency for different murals. The time improvement is between 106 and 151 s, and the time performance is considerable.

Artificial destruction mural repair comparison

To further verify the effectiveness of the proposed ASB–LS algorithm, this study also added and repaired artificially flaked areas to the intact mural. peak signal to noise ratio (PSNR) was used to measure the similarity between the restored image and the original intact image [26]. The larger the PSNR value, the more similar the two images are, and the better the repair effect is. PSNR is defined as follows:

$$PSNR = 10*\log \left( {\frac{{255^{2} }}{MSE}} \right)$$

$$MSE = \frac{1}{n}\sum\limits_{i = 1}^{n} {(x_{i} - u_{i} )^{2} } ,$$

where $n$ represents the number of pixels in the image and $x_{i}$ and $u_{i}$ are the original image and the repaired image, respectively.

Figure 5 shows the overall restoration effects of different algorithms, and Fig. 6 shows the local magnifications of the restoration effects of different algorithms.

Fig. 5

Comparison of the effects of different algorithms on artificial destruction repair. A Image 1 in Fig. 2. B Image 2 in Fig. 2. C Image 3 in Fig. 2. D Image 4 in Fig. 2. (a) The original mural image. (b) Artificially damaged mural image. (c) Restoration effect of the Criminisi algorithm. (d) Restoration effect of the algorithm in the literature [15]. (e) Restoration effect of the algorithm in the literature [18]. (f) Restoration effect of the algorithm proposed in this study

Fig. 6

Comparison of the local magnifications of the repair effects of different algorithms on artificial destruction. A Image 1 in Fig. 2. B Image 2 in Fig. 2. C Image 3 in Fig. 2. D Image 4 in Fig. 2. (a) Restoration effect of the Criminisi algorithm. (b) Restoration effect of the algorithm in the literature [15]. (c) Restoration effect of the algorithm in the literature [18]. (d) Restoration effect of the algorithm proposed in this study

The Criminisi algorithm suffered from disruption of the line in the lower right corner (Fig. 5A(c) and Fig. 6A(a)) and incorrect extension of the texture-rich region of the damaged dragon beard part (Fig. 5B(c) and Fig. 6B(a)), the fringe area of the hair part (Fig. 5C(c) and Fig. 6C(a)) and the upper texture part (Fig. 5D(c) and Fig. 6D(a)) after repair. The algorithms in the literature [15, 18] suffered distortion and interruption of the line in the right corner (Fig. 5A(c–d)) and disruption on the right side of the dragon body (Fig. 5B(c–d)), in the fringe region of the damaged shoulder part (Fig. 5C(c–d)) and in the green band region after repair (Fig. 5D(c–d)). Compared with these methods, the algorithm proposed in this study effectively solved the problem of post-repair inaccurate extension and exhibited better continuity of the fringe area.

A comparison of PSNR values repaired by different algorithms is shown in Table 4. The algorithm improved the PSNR by 1–3 dB. Compared with the PSNR values in the literature [15, 18], the PSNR value of the proposed algorithm also improved, which further demonstrates that the proposed ASB–LS algorithm works best for mural restoration.

Table 4 Comparison of PSNR values repaired by different algorithms

Based on the analytical results above, the algorithm proposed in this study saved 100–150 s when repairing the flaking region of the mural with improved overall visual continuity and increased the PSNR by 1–3 dB when repairing artificially damaged murals compared with the classical Criminisi algorithm and those in the literature [15, 18].

Conclusion

Based on the Criminisi algorithm, this paper proposed the ASB–LS algorithm to address the erosion problem of flaking deterioration of Kaihua Temple murals from the Song Dynasty. The structural tensor of the image was introduced into the data item, and the data item and the priority function were redefined. Then, the average correlation factor of the sample block was used to adaptively adjust the size of the sample block to different structural regions of the image, thereby effectively avoiding the error diffusion of the structural information. A local search strategy was used to improve the repair efficiency while ensuring the repair effect. Compared with the traditional Criminisi algorithm and those in the literature [15, 18], the proposed algorithm in this study improved the visual continuity, increased the PSNR by 1–3 dB, and reduced the repair time by 100–150 s when used to repair the flaking region of murals, providing a reference for the actual practice of mural restoration.

However, the algorithm proposed in this study was sensitive to the structure tensor when used to repair flaking disease of murals, which determined not only the repair order of the boundary pixels but also the size of the matched block. Furthermore, when the flaking region was large, the restoration effect on the center of the region based on the proposed algorithm was not satisfactory. In addition, the types of mural diseases repaired in this study were limited; thus, further research on the proposed algorithm is needed. In future work, the digital restoration of various types of deterioration in ancient murals will involve in-depth research and attempts to introduce image semantic analysis techniques to complete the automatic repair of mural damage from various types of deterioration. Additionally, an evaluation standard for mural image restoration will be designed to better describe the quality of the repair results.