Skip to main content

Restoration of Tanjore paintings using segmentation and in-painting techniques

Abstract

India has its unique and rich cultural heritage. One such uniqueness in India is ancient paintings. Especially in South India, Tanjore paintings are very popular. These paintings are made during 1010 AD with vibrant colours, gold, silver and precious stones. These paintings are the memorabilia of the great Chola kingdom. These paintings can be seen in great Brahadeeshwara Temple walls till now. Damages to these paintings happen due to varying environmental conditions and rituals followed throughout the year. Hence, preserving these heritages could be an additional source in National Cultural Museum and cultural libraries. This paper focuses towards the restoration of such ancient painting images that can be digitized and archived for the future use of aesthete. The painting images are preprocessed using Weiner filter for removing the background noises since its PSNR value is higher than Gaussian and Median filters. The preprocessed image is then applied to restoration algorithm. Two types of restoration algorithm is attempted, image segmentation and in-painting algorithm. The degraded image was restored efficiently with in-painting algorithm than segmentation algorithm. Further research can be focused towards automatic adaptive selection of patch based on the nature of images. From the results it is observed that with in-painting algorithm the image restoration is better than the segmentation algorithm for degraded painting images.

Introduction

India is famous for its culture and civilization. Indian ancestors excelled in the field of Architecture, Art, Medicine, Astrology, etc., that are recognized and admired worldwide. They passed on the information about the heritage to their future generation in the form of sculptures, paintings and inscriptions. Paintings are one such art form which depicts the ancient history and culture. Chittannavasal and Tanjore paintings are very prominent in India which attracts people around the world. Tanjore painting play a significant role in Indian paintings which are made of vibrant colours, gold and precious stones. But due to climatic changes, lack of maintenance and rituals these paintings are degraded. Though many efforts are carried out to save these paintings from further damage, it is very challenging to restore the paintings from the existing degradation. However, with developing technology in the field of image processing has made this challenge achievable. These painting can be restored by formulating a degradation model and then developing an algorithm to restore the degraded portions. This paper deals with the various methodologies adopted for restoration of damaged images. This section describes the various restoration techniques adapted for paintings.

Several image processing techniques has been adopted by researcher to restore ancient painting and images. In Chinese paintings, nearest neighbor methodology [1] was adopted for the restoration of tears, flakes and cracks in the painting. Hierarchy based restoration technique [2] is also used for the restoration of Chinese paintings. In this algorithm, painting is split into foreground and background layers. Then restoration algorithm is applied to these layers and finally the results are combined to produce the synthesized image restoration.

Crack detection and elimination in digitized paintings are performed using morphological top-hat transformation [3, 4]. The authors were able to achieved true positive rate of 98.3%. Though this algorithm provides better result for digitized images, it has to be explored for real time painting images. Non-uniform illumination enhancement technique [5] was applied for illumination enhancement and color restoration. The authors could achieve local contrast improvement, detail enhancement thus preserving the originality of the image in the cultural heritage dataset.

In computer vision system for robust restoration [6] of prehistoric Tehran wall paintings is presented. The authors applied an image stitching algorithm for image restoration. An area of relevant semantics, geometry and color in a different spot of the wall paintings was selected and stitched into the damaged area. Their key focus was the identification of damaged or missing area in the painting performed using morphological algorithm in addition with edge information.

Similarly, computer-guided and virtual artwork restoration techniques [7] are implemented in paintings. These technique aids the restorer with virtual cleaning software to identify the best suitable cleaning procedure with a small patch of the paintings. Thus with initial study, it could be extended to the painting upon successful implementation with the small patch.

Two different methodologies for color image restoration [8] such as, blending of the standard deviation- weighted gray world and the Combined Gray World and Retinex (CGWR). The second technique was based on alteration of the Multi Scale Retinex (MSR) theory. In these techniques, the effect of neighboring pixels on the human eye is replicated for modifying the algorithms. In addition, the modified MSR is applied on CGWR technique to improve the performance of the basic algorithm. Their experimental results depicted the comparison between these two techniques with the basic traditional technique. A non blind deconvolution [9] is applied for restoring the degraded image with point spread function estimation. With this method, incorrect estimation may occur if the signal is narrower than the blur size. This limitation is overcome with use of gradient Reliability map (R-Map).

From the literature studies it is evident that restoration of images plays a vital role in preserving the cultural heritage. Though the various restoration techniques have been applied to Chinese paintings, it is observed from the literatures that digital restoration techniques are not explored for Tanjore paintings which were made during 1010 AD. Though many years passed, these paintings still remains astonishing to the spectators who come across it. Hence, the motivation of the work is to preserve the ancient paintings through digital technology for the future generation to cherish the culture and heritage. Since, the digital restoration process is not explored; it is a first of its kind attempt for the restoration of Tanjore paintings. Hence, this paper focuses on restoration of ancient Tanjore painting using image processing techniques.

Methodology

Figure 1 shows the various steps involved in the restoration of ancient Tanjore paintings. The major techniques involved are (1) Image preprocessing, (2) Image restoration. The ancient painting images are captured in Brahadeeshwara Temple, Tanjore using a digital camera during a bright sunny day. From the captured image, 40 images which are more degraded are considered for the study. Most of the images taken for the study are within the size of 1220 × 775 pixels. The captured image contains background noises due to the ambient conditions. Hence image pre-processing is essential to suppress the background noises in the image. “Pre-processing of paintings” section describes the study made by applying various filtering techniques for removing noises on the captured images. The pre-processed images are then subjected to image restoration techniques. The results obtained with restoration techniques are given in "Image restoration" section

Fig. 1
figure 1

Work flow of the proposed system

Pre-processing of paintings

Pre-processing techniques involves the suppression of background noises from the degraded images. Filters such as Weiner, Median and Gaussian filter had produced promising results in noise removal of degraded images [10, 11]. Hence a study is attempted by applying those filtering techniques on the degraded images considered for restoration. The Peak Signal to Noise Ratio (PSNR) is estimated and best suitable filter is selected for pre-processing steps. Figure 2 shows the results obtained through the application of various filters on degraded images.

Fig. 2
figure 2

Performance of various Filters on the image

Figure 3 shows the performance of various filters on the degraded images. It is observed that the PSNR value is higher for Weiner filter than Gaussian and median filter for the degraded images. Hence in the study, Weiner filter is applied for pre-processing of painting images which is effective in eliminating any noise that might have occurred during acquisition of images.Table 1 shows the performance of various filters for the degraded painting images.

Fig. 3
figure 3

PSNR of various Filters for degraded painting Images

Table 1 MSE of various filters for degraded painting images

Image restoration

The next step in image restoration is recovery of degraded portions from the filtered images. Two techniques such as segmentation technique and In-painting technique were applied in this stage of study. A comparison is made on the restored images from the results obtained with these two techniques to estimate the best fit.

Segmentation techniques

In segmentation technique, the degraded images are segmented into various blocks. Initially the image was segmented in to four parts and the restored image was analyzed. The RGB image in each segmented block is converted into binary image. From the binary image the region of interest (ROI), which is the degraded part in the specific block, is identified based on region based segmentation technique by applying threshold value. Various trials were done and finally a threshold value of 0.6 is fixed which provided better result for all the degraded images. The average RGB component in each block is computed by the algorithm. The colour corresponding to the computed average RGB is filled in the degraded part. This process is performed for all the segmented blocks in the image and retrieval of the degraded parts in the image is done. With four blocks of segmentation, the ROI is not properly identified and hence filling was very poor. When the same has been analyzed with sixteen blocks, the restoration is better than 4 blocks. However, the colours are not properly averaged, and it is filled with inappropriate RGB colour. Hence based on heuristic approach, with 8 blocks segmentation best result are produced. Figure 4a–c shows the original, degraded and restored painting images respectively through this segmentation technique. Though segmentation technique restores the degraded portions of the image, the outcome is not satisfactory. Hence, to achieve better restoration of ancient images, in-painting technique is attempted.

Fig. 4
figure 4

a Original images b degraded images c restored image through segmentation techniques

In-painting technique

In-painting technique is applied to restore degraded paintings, remove certain portions in an image without changing the nature of the image [12,13,14,15,16,17,18]. A patch based method is adopted for degrade identification in images using a binary mask (M) according to Eq. (1).

$$\mathrm{M}\left(\mathrm{p}\right)=\left\{\begin{array}{l}1 for p\in D \\ 0 for p \in U\end{array}\right.$$
(1)

where, p is the pixel index, D is the damage area of the painting and U is the undamaged area of the painting. The RGB image corresponding to the damaged area D from the binary mask is selected as ROI. Initially the image is decomposed to group into texture and structural features. The structural features such as lines and edges are extracted. Similarly the texture features such as pixel intensity is extracted from the image. Once the features are extracted the interpolation of images is done for both extracted images. A patch (\({\varphi }_{d}\)) at the edge of the damaged area and undamaged area that has the highest priority is selected. A similar patch with the same texture and structural features in the undamaged area (\({\varphi }_{u}\)) is searched as per Eq. (2). A patch size of 4 is fixed for this work. After searching the best patch (\({\varphi }_{u}\)), the same is filled in the damaged area.

$${\varphi }_{u}=\mathrm{arg}(\mathrm{min}({\varphi }_{d}, {\varphi }_{u}))$$
(2)

where, \(\mathrm{min}({\varphi }_{d}, {\varphi }_{u}))\) is the sum of squared difference. The selected patch \({\varphi }_{u}\) is then copied to the damaged portion of the image patch by patch using interpolation. Thus through interpolation, the structural and texture features propagates in to the damaged portion of the image for restoration of damaged portions of the painting image. The flow of the work has been given in Fig. 5. Figure 6a–d shows the original image, degraded image, binary mask and restored images respectively through in-painting technique are given in Fig. 6.

Fig. 5
figure 5

Flowchart of in-painting technique

Fig. 6
figure 6

a Original image b degraded image c binary mask d restored image using in-painting technique

Results and discussion

The algorithm developed for the restoration of degraded images of Tanjore paintings are discussed in the previous sections. From the overall results obtained, the restored images are compared and analysed for its restoration quality. The parameters such as the quantity of degraded portions retrieved, matching of colours between the non-degraded and degraded portions in ROI are considered for assessment. The retrieved images are compared with the images from internet sources. However, the restored images are not cross validated with the Real-time images of historical evidence material, since there are no such non-degraded images available with authenticated sources. From the study made, Weiner filter performed well as compared to other filters used in the study for the noise removal of degraded images. Weiner filter produced a minimum MSE as low as 0.001 in this application as given in Table 1.

From Fig. 4 it is observed that the images have been filled with the average of RGB colours in the segmented blocks which restores the degraded portions. However the exact match of RGB colours as in the original historical evidence images are not obtained in filled portions of the degraded images. In addition, identification of degraded portions in the images for the selection of ROI is still remains challenging in few images. Also, the image is resized to a lesser dimension and image is blurred. With In-painting technique, the restoration is better than the segmentation technique. However few degraded portions are not filled with proper RGB value.

Conclusions

This research focuses on the restoration of Tanjore painting images based on segmentation and In-painting techniques. From the results, it is concluded that in-painting techniques performs better than segmentation process for the restoration of ancient painting images. The restoration performed using segmentation technique by averaging RGB components was not able to fill the colour properly. The ROI of the segmented block is filled with the average colour of the corresponding segment. However, the region of interest may have a dominant colour of the neighbouring block. Hence segmentation of blocks should be optimized in such a way that appropriate colour should be chosen for filling. At the same time the number of blocks could be varied dynamically based on the colours used in images. Images having less number of colours are restored efficiently than the images with a large number of colour combinations. It is evident that in Tanjore paintings vibrant and number of colours are used. Hence segmentation technique is not suitable for painting images. In-painting performs better for restoration of ancient painting images. Further research can be carried out with adaptive patch selection algorithm without compromising on the quality of restored image for improving the performance of restoration process. In addition, the restores images can be digitally archived in a cloud portal with the information related with specific Tanjore paintings. This will aid in preserving the restored Tanjore paintings digitally which will last for years in spite of the degradation of the real-time image.

Availability of data and materials

The data was collected through capturing the degraded images using a digital camera by the Authors of this paper at Bhahadeeshwara Temple, Tanjore, India.

References

  1. Zeng Y, Gong Y. Nearest neighbor based digital restoration of damaged ancient Chinese paintings. In: 23rd international conference on digital signal processing, Shanghai, China; 2018.

  2. Wei S. Research on hierarchical image restoration of Chinese Painting. In: The 9th international conference on computer science & education (ICCSE 2014) August 22–24, 2014. Vancouver, Canada.

  3. Giakoumis I, Nikolaidis N, Pitas I. Digital image processing techniques for the detection and removal of cracks in digitized paintings. IEEE Trans Image Process. 2006;15(1):178–88.

    Article  Google Scholar 

  4. Desai SD, Kavita V. Horadi, Navaneet P, Niriksha B, Siddeshvar V. User intervention based detection & removal of cracks from digitized paintings. In: 2014 fifth international conference on signals and image processing; 2013. https://doi.org/10.1109/ICSIP.2014.7

  5. Jmal M, Souidene W, Attia R. Efficient cultural heritage image restoration with nonuniform illumination enhancement. J Electron Imaging. 2017;26(1): 011020. https://doi.org/10.1117/1.JEI.26.1.011020.

    Article  Google Scholar 

  6. Karianakis N, Maragos P. An integrated system for digital restoration of prehistoric theran wall paintings. In: 18th international conference on digital signal processing (DSP); 2013. https://doi.org/10.1109/ICDSP.2013.6622838.

  7. Barni M, Bartolini F, Cappellini V. Image processing for virtual restoration of artworks. IEEE Multimedia. 2000;7(2):34–7.

    Article  Google Scholar 

  8. Ahmed AMT. Color restoration techniques for faded colors of old photos, printings and paintings. In: IEEE international conference on electro/information technology 2009, p. 10804195. https://doi.org/10.1109/EIT.2009.5189600

  9. Motohashi S, Nagata T, Goto T, Aoki R, Chen H. A study on blind image restoration of blurred images using R-map. In: International workshop on advanced image technology; 2018, p. 7–9. https://doi.org/10.1109/IWAIT.2018.8369650

  10. Mbarki Z, Seddik H, Braiek EB. Non blind image restoration scheme combining parametric Wiener filtering and BM3D denoising technique. In: 4th international conference on advanced technologies for signal and image processing; 2018. Sousse, Tunisia.

  11. Akbar JM. Joint method using Akamatsu and discrete wavelet transform for image restoration. Applied computing and informatics; 2020.

  12. Cao, et al. Restoration of an ancient temple mural by a local search algorithm of an adaptive sample block. Herit Sci. 2019;7:39. https://doi.org/10.1186/s40494-019-0281-y.

    Article  Google Scholar 

  13. Liang SF, Guo M, Liang XQ. Enhanced Criminisi algorithm of digital image inpainting technology. Comput Eng Des. 2016;37(1314–8):1345.

    Google Scholar 

  14. Siadati SZ, Yaghmaee F, Mahdavi P. A new exemplar-based image inpainting algorithm using image structure tensors. In: Proceeding of the 24th Iranian conference on electrical engineering; 2016, p. 995–1001.

  15. Michaeli T, Irani M. Blind deblurring using internal patch recurrence. In: European conference on conputer vision; 2014, p. 783–98.

  16. Chan TF, Shen J. Mathematical models for local nontexture inpaintings. SIAM J Appl Math 62:3(Dec., 2001–Feb., 2002): 1019–43. https://www.jstor.org/stable/3061798

  17. Papyan V, Elad M. Multi-scale patch-based image restoration. IEEE Trans Image Process. 2016;25(1):249–61. https://doi.org/10.1109/TIP.2015.2499698.

    Article  Google Scholar 

  18. Wang W, Jia Y. Damaged region filling and evaluation by symmetrical exemplar-based image inpainting for Thangka. EURASIP J Image Video Process. 2017;2017:38. https://doi.org/10.1186/s13640-017-0186-1.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank ICSSR and Sathyabama Institute of Science and Technology for extending their support.

Funding

This work is funded by Indian Council of Social Science Research (ICSSR) under IMPRESS scheme (IMPRESS/P1415 /2018–19/ICSSR).

Author information

Affiliations

Authors

Contributions

SP—Image acquisition and preprocessing, preparation of the manuscript. SB—Developed the restoration algorithm, preparation of manuscript. MS—Image acquisition and processing. TV—Supported in documentation of literature reviews. All authors read and approved the final manuscript.

Corresponding author

Correspondence to S. Barani.

Ethics declarations

Competing interests

The authors declare that they have no conflicts of interest to report regarding the present study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Poornapushpakala, S., Barani, S., Subramoniam, M. et al. Restoration of Tanjore paintings using segmentation and in-painting techniques. Herit Sci 10, 41 (2022). https://doi.org/10.1186/s40494-022-00661-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40494-022-00661-1

Keywords

  • Tanjore paintings
  • Distorted paintings
  • Restoration
  • In-painting technique
  • Culture
  • Heritage