Restoration of Tanjore paintings using segmentation and in-painting techniques

Abstract

India has its unique and rich cultural heritage. One such uniqueness in India is ancient paintings. Especially in South India, Tanjore paintings are very popular. These paintings are made during 1010 AD with vibrant colours, gold, silver and precious stones. These paintings are the memorabilia of the great Chola kingdom. These paintings can be seen in great Brahadeeshwara Temple walls till now. Damages to these paintings happen due to varying environmental conditions and rituals followed throughout the year. Hence, preserving these heritages could be an additional source in National Cultural Museum and cultural libraries. This paper focuses towards the restoration of such ancient painting images that can be digitized and archived for the future use of aesthete. The painting images are preprocessed using Weiner filter for removing the background noises since its PSNR value is higher than Gaussian and Median filters. The preprocessed image is then applied to restoration algorithm. Two types of restoration algorithm is attempted, image segmentation and in-painting algorithm. The degraded image was restored efficiently with in-painting algorithm than segmentation algorithm. Further research can be focused towards automatic adaptive selection of patch based on the nature of images. From the results it is observed that with in-painting algorithm the image restoration is better than the segmentation algorithm for degraded painting images.

Introduction

India is famous for its culture and civilization. Indian ancestors excelled in the field of Architecture, Art, Medicine, Astrology, etc., that are recognized and admired worldwide. They passed on the information about the heritage to their future generation in the form of sculptures, paintings and inscriptions. Paintings are one such art form which depicts the ancient history and culture. Chittannavasal and Tanjore paintings are very prominent in India which attracts people around the world. Tanjore painting play a significant role in Indian paintings which are made of vibrant colours, gold and precious stones. But due to climatic changes, lack of maintenance and rituals these paintings are degraded. Though many efforts are carried out to save these paintings from further damage, it is very challenging to restore the paintings from the existing degradation. However, with developing technology in the field of image processing has made this challenge achievable. These painting can be restored by formulating a degradation model and then developing an algorithm to restore the degraded portions. This paper deals with the various methodologies adopted for restoration of damaged images. This section describes the various restoration techniques adapted for paintings.

Several image processing techniques has been adopted by researcher to restore ancient painting and images. In Chinese paintings, nearest neighbor methodology [1] was adopted for the restoration of tears, flakes and cracks in the painting. Hierarchy based restoration technique [2] is also used for the restoration of Chinese paintings. In this algorithm, painting is split into foreground and background layers. Then restoration algorithm is applied to these layers and finally the results are combined to produce the synthesized image restoration.

Crack detection and elimination in digitized paintings are performed using morphological top-hat transformation [3, 4]. The authors were able to achieved true positive rate of 98.3%. Though this algorithm provides better result for digitized images, it has to be explored for real time painting images. Non-uniform illumination enhancement technique [5] was applied for illumination enhancement and color restoration. The authors could achieve local contrast improvement, detail enhancement thus preserving the originality of the image in the cultural heritage dataset.

In computer vision system for robust restoration [6] of prehistoric Tehran wall paintings is presented. The authors applied an image stitching algorithm for image restoration. An area of relevant semantics, geometry and color in a different spot of the wall paintings was selected and stitched into the damaged area. Their key focus was the identification of damaged or missing area in the painting performed using morphological algorithm in addition with edge information.

Similarly, computer-guided and virtual artwork restoration techniques [7] are implemented in paintings. These technique aids the restorer with virtual cleaning software to identify the best suitable cleaning procedure with a small patch of the paintings. Thus with initial study, it could be extended to the painting upon successful implementation with the small patch.

Two different methodologies for color image restoration [8] such as, blending of the standard deviation- weighted gray world and the Combined Gray World and Retinex (CGWR). The second technique was based on alteration of the Multi Scale Retinex (MSR) theory. In these techniques, the effect of neighboring pixels on the human eye is replicated for modifying the algorithms. In addition, the modified MSR is applied on CGWR technique to improve the performance of the basic algorithm. Their experimental results depicted the comparison between these two techniques with the basic traditional technique. A non blind deconvolution [9] is applied for restoring the degraded image with point spread function estimation. With this method, incorrect estimation may occur if the signal is narrower than the blur size. This limitation is overcome with use of gradient Reliability map (R-Map).

From the literature studies it is evident that restoration of images plays a vital role in preserving the cultural heritage. Though the various restoration techniques have been applied to Chinese paintings, it is observed from the literatures that digital restoration techniques are not explored for Tanjore paintings which were made during 1010 AD. Though many years passed, these paintings still remains astonishing to the spectators who come across it. Hence, the motivation of the work is to preserve the ancient paintings through digital technology for the future generation to cherish the culture and heritage. Since, the digital restoration process is not explored; it is a first of its kind attempt for the restoration of Tanjore paintings. Hence, this paper focuses on restoration of ancient Tanjore painting using image processing techniques.

Methodology

Figure 1 shows the various steps involved in the restoration of ancient Tanjore paintings. The major techniques involved are (1) Image preprocessing, (2) Image restoration. The ancient painting images are captured in Brahadeeshwara Temple, Tanjore using a digital camera during a bright sunny day. From the captured image, 40 images which are more degraded are considered for the study. Most of the images taken for the study are within the size of 1220 × 775 pixels. The captured image contains background noises due to the ambient conditions. Hence image pre-processing is essential to suppress the background noises in the image. “Pre-processing of paintings” section describes the study made by applying various filtering techniques for removing noises on the captured images. The pre-processed images are then subjected to image restoration techniques. The results obtained with restoration techniques are given in "Image restoration" section

Fig. 1

Work flow of the proposed system

Pre-processing of paintings

Pre-processing techniques involves the suppression of background noises from the degraded images. Filters such as Weiner, Median and Gaussian filter had produced promising results in noise removal of degraded images [10, 11]. Hence a study is attempted by applying those filtering techniques on the degraded images considered for restoration. The Peak Signal to Noise Ratio (PSNR) is estimated and best suitable filter is selected for pre-processing steps. Figure 2 shows the results obtained through the application of various filters on degraded images.

Fig. 2

Performance of various Filters on the image

Figure 3 shows the performance of various filters on the degraded images. It is observed that the PSNR value is higher for Weiner filter than Gaussian and median filter for the degraded images. Hence in the study, Weiner filter is applied for pre-processing of painting images which is effective in eliminating any noise that might have occurred during acquisition of images.Table 1 shows the performance of various filters for the degraded painting images.

Fig. 3

PSNR of various Filters for degraded painting Images

Table 1 MSE of various filters for degraded painting images

Image restoration

The next step in image restoration is recovery of degraded portions from the filtered images. Two techniques such as segmentation technique and In-painting technique were applied in this stage of study. A comparison is made on the restored images from the results obtained with these two techniques to estimate the best fit.

Segmentation techniques

In segmentation technique, the degraded images are segmented into various blocks. Initially the image was segmented in to four parts and the restored image was analyzed. The RGB image in each segmented block is converted into binary image. From the binary image the region of interest (ROI), which is the degraded part in the specific block, is identified based on region based segmentation technique by applying threshold value. Various trials were done and finally a threshold value of 0.6 is fixed which provided better result for all the degraded images. The average RGB component in each block is computed by the algorithm. The colour corresponding to the computed average RGB is filled in the degraded part. This process is performed for all the segmented blocks in the image and retrieval of the degraded parts in the image is done. With four blocks of segmentation, the ROI is not properly identified and hence filling was very poor. When the same has been analyzed with sixteen blocks, the restoration is better than 4 blocks. However, the colours are not properly averaged, and it is filled with inappropriate RGB colour. Hence based on heuristic approach, with 8 blocks segmentation best result are produced. Figure 4a–c shows the original, degraded and restored painting images respectively through this segmentation technique. Though segmentation technique restores the degraded portions of the image, the outcome is not satisfactory. Hence, to achieve better restoration of ancient images, in-painting technique is attempted.

Fig. 4

a Original images b degraded images c restored image through segmentation techniques

In-painting technique

In-painting technique is applied to restore degraded paintings, remove certain portions in an image without changing the nature of the image [12,13,14,15,16,17,18]. A patch based method is adopted for degrade identification in images using a binary mask (M) according to Eq. (1).

$$\mathrm{M}\left(\mathrm{p}\right)=\left\{\begin{array}{l}1 for p\in D \\ 0 for p \in U\end{array}\right.$$

(1)

where, p is the pixel index, D is the damage area of the painting and U is the undamaged area of the painting. The RGB image corresponding to the damaged area D from the binary mask is selected as ROI. Initially the image is decomposed to group into texture and structural features. The structural features such as lines and edges are extracted. Similarly the texture features such as pixel intensity is extracted from the image. Once the features are extracted the interpolation of images is done for both extracted images. A patch (${\varphi }_{d}$) at the edge of the damaged area and undamaged area that has the highest priority is selected. A similar patch with the same texture and structural features in the undamaged area (${\varphi }_{u}$) is searched as per Eq. (2). A patch size of 4 is fixed for this work. After searching the best patch (${\varphi }_{u}$), the same is filled in the damaged area.

$${\varphi }_{u}=\mathrm{arg}(\mathrm{min}({\varphi }_{d}, {\varphi }_{u}))$$

(2)

where, $\mathrm{min}({\varphi }_{d}, {\varphi }_{u}))$ is the sum of squared difference. The selected patch ${\varphi }_{u}$ is then copied to the damaged portion of the image patch by patch using interpolation. Thus through interpolation, the structural and texture features propagates in to the damaged portion of the image for restoration of damaged portions of the painting image. The flow of the work has been given in Fig. 5. Figure 6a–d shows the original image, degraded image, binary mask and restored images respectively through in-painting technique are given in Fig. 6.

Fig. 5

Flowchart of in-painting technique

Fig. 6

a Original image b degraded image c binary mask d restored image using in-painting technique

Results and discussion

The algorithm developed for the restoration of degraded images of Tanjore paintings are discussed in the previous sections. From the overall results obtained, the restored images are compared and analysed for its restoration quality. The parameters such as the quantity of degraded portions retrieved, matching of colours between the non-degraded and degraded portions in ROI are considered for assessment. The retrieved images are compared with the images from internet sources. However, the restored images are not cross validated with the Real-time images of historical evidence material, since there are no such non-degraded images available with authenticated sources. From the study made, Weiner filter performed well as compared to other filters used in the study for the noise removal of degraded images. Weiner filter produced a minimum MSE as low as 0.001 in this application as given in Table 1.

From Fig. 4 it is observed that the images have been filled with the average of RGB colours in the segmented blocks which restores the degraded portions. However the exact match of RGB colours as in the original historical evidence images are not obtained in filled portions of the degraded images. In addition, identification of degraded portions in the images for the selection of ROI is still remains challenging in few images. Also, the image is resized to a lesser dimension and image is blurred. With In-painting technique, the restoration is better than the segmentation technique. However few degraded portions are not filled with proper RGB value.

Conclusions

This research focuses on the restoration of Tanjore painting images based on segmentation and In-painting techniques. From the results, it is concluded that in-painting techniques performs better than segmentation process for the restoration of ancient painting images. The restoration performed using segmentation technique by averaging RGB components was not able to fill the colour properly. The ROI of the segmented block is filled with the average colour of the corresponding segment. However, the region of interest may have a dominant colour of the neighbouring block. Hence segmentation of blocks should be optimized in such a way that appropriate colour should be chosen for filling. At the same time the number of blocks could be varied dynamically based on the colours used in images. Images having less number of colours are restored efficiently than the images with a large number of colour combinations. It is evident that in Tanjore paintings vibrant and number of colours are used. Hence segmentation technique is not suitable for painting images. In-painting performs better for restoration of ancient painting images. Further research can be carried out with adaptive patch selection algorithm without compromising on the quality of restored image for improving the performance of restoration process. In addition, the restores images can be digitally archived in a cloud portal with the information related with specific Tanjore paintings. This will aid in preserving the restored Tanjore paintings digitally which will last for years in spite of the degradation of the real-time image.