In situ virtual restoration of artifacts by imaging technology

The restoration of incomplete artifacts is important for presenting the value of artifacts. Even though the existing methods provide rich solutions for the restoration of artifacts, they may not be the ultimate and perfect methods. The innovation of restoration methods may provide better restoration effects in some aspects or bring people a different visiting experience, which motivates people to explore new restoration methods. In this paper, based on virtual imaging technology, a virtual imaging restoration method for incomplete artifacts were proposed. In order to examine the effects and potential of the virtual imaging restoration, a bronze mirror and a mural were restored through our newly invented virtual imaging restoration device. The effects of the restorations were evaluated through the illumination test and the questionnaire surveys. The results proved that the virtual imaging restoration can visually achieve in situ restoration, and it is completely reversible. Virtual imaging restoration also performed well in terms of visiting experience, authenticity, discernibility, and economy. Limitations of this restoration method are related to the scope of application, the viewing angle, and the imaging effects. Overall, virtual imaging restoration is worth recommending and the potential of this method and its encouraging results may provide a choice or pretest tool for the restoration of some incomplete artifacts in the near future.


Introduction
Artifacts are a kind of precious human treasure. However, a considerable portion of artifacts is incomplete, which hinders the presentation of their values. One way to give people complete artifacts is restoration, which mostly consists of five conventional methods: (1) Physical restoration, restoring incomplete artifacts to complete state by applying physical materials [1][2][3][4]. (2) Digital restoration, restoring the digitalized incomplete artifacts to complete state by using multimedia technology [5][6][7][8].
(3) Digital projection restoration, restoring the original color and texture of incomplete artifacts by projecting the image onto the surface of the objects [9,10]. (4) Patterns restoration, restoring incomplete artifacts to complete state by placing the patterns of the incomplete areas behind artifacts (as shown in Fig. 1). (5) AR restoration, using Augmented Reality (AR) technology to recognize and track the incomplete artifacts, and then digitally restore them on smart devices [11,12]. The existing methods provide rich solutions for the restoration of artifacts. Due to the preciousness of artifacts, restoration is an issue that needs to be carefully considered and evaluated. When evaluating a restoration method, some fundamental principles of protection/restoration are usually referred to, which includes: • Authenticity authenticity is a core concept in contemporary conservation/restoration principles, which is rich in connotation. There are specific attributes that are often listed as references for the definition of authenticity, such as the authenticity of form and  [13,14]. • Reversibility reversibility is a widely used but relatively vague concept. In simple terms, reversibility refers to recover the restored object to the previous form for any future need [15]. Some concepts with similar meanings have also developed as supplements to the reversibility principle, such as repeatability [16] and removability [17]. • Discernibility discernibility has been paid more and more attention since the Venice Charter in 1964. Discernibility holds that "replacements of missing parts must integrate harmoniously with the whole, but at the same time must be distinguishable from the original so that restoration does not falsify the artistic or historic evidence [18] ".
Even though the existing methods provide rich solutions for the restoration of artifacts, they may not be the ultimate and perfect methods. To a certain extent, the diversity of restoration methods reflects the diverse demands for restorations. With the development of the restoration concepts, the restoration has not only been used as a means to make artifacts more stable or complete but also to enhance people's understanding of artifacts [13]. The innovation of restoration methods may provide better restoration effects in some aspects or bring people a different visiting experience, which motivates people to explore new restoration methods.
In this paper, a new restoration method based on virtual imaging technology was proposed. Virtual imaging technology, also known as phantom imaging is not a new technology and has been widely used in museums to exhibit virtual artifacts or related scenes [19][20][21].
However, the potential of virtual imaging technology in restoration has not been paid enough attention in the past, and it has not been found the application of this technology in the restoration of artifacts. To examine the effects and potential of virtual imaging technology in the restoration of artifacts. Two artifacts were restored via our newly invented virtual imaging restoration device and the effects were evaluated through the illumination test and the questionnaire surveys. The results showed that virtual imaging restoration can restore artifacts well in situ and perform well in terms of authenticity, reversibility, discernibility, and economy. At the same time, virtual imaging restoration can bring a novel and good visiting experience.

Methods
A bronze mirror and a mural were chosen as the restoring objects, both of which were partially incomplete. The restorations were accomplished by our virtual image restoration device. For the evaluation of the restoration effects, we conducted an illumination test and the questionnaire surveys.

Basic principle of virtual imaging restoration
The virtual imaging restoration is established based on the principle of a kind of visual illusion phenomenon that human common sense can sometimes cause erroneous spatial perception. This principle can further be explained by the example in Fig. 2. As shown in the figure, an incomplete bronze mirror is placed behind a transparent sheet that is inclined 45 degrees. Since the sheet is transparent, the incomplete bronze mirror behind the transparent sheet can be clearly observed. Just as sometimes people can see themselves from a piece of transparent glass, the transparent sheet also has a certain degree of reflection. A mirror-reversed digital image of the missing part of the bronze mirror is placed below the transparent sheet. As the transparent sheet is with an inclination of 45 degrees and has a certain reflectivity, the virtual image will be horizontally reflected into observers' eyes. With the common sense that "light travels in a straight line, " the brain tends to think that the imaging light comes directly from the transparent space in front of it, that is to say, the brain will believe that the image of the missing part of the bronze mirror is located in the transparent space behind the transparent sheet. Because the virtual image has a focal length, that is, the virtual image has a sense of distance in our eyes and we can perceive its spatial position. Therefore, the visual distance of the virtual image can be controlled by adjusting the distance between the projection screen and the transparent sheet. When the projection screen is in a certain spatial position, the virtual image of the missing part and the bronze mirror can be at the same visual distance and can be completely joined together in the observers' eyes, thereby achieving the virtual restoration in situ.

Virtual imaging restoration device
As shown in Fig. 3, the device for the virtual imaging restoration was established based on the above principle. The structure mainly consists of the imaging plate, the  platform, the base, the projection screen, the lifting platform and the cabinet. The imaging plate is a 45 degree inclined transparent sheet made of an electronic grade soda-lime glass with a thickness of 1 mm, 90% transparency, and good rigidity. It is important to note that the properties of the transparent sheet play a key role in the imaging effect. Firstly, it should maintain as thin as possible (≤ 1 mm), to reduce the ghosting. That is because the transparent sheet has a certain thickness, and both the inside surface and the outside surface of the sheet can reflect light. The thickness of the sheet determines the positions where the same light beam generates reflected light on the inner and outer surfaces is different, so the reflected light of the two surfaces does not entirely coincide, resulting in the ghosting. The thicker the transparent sheet, the more obvious the ghosting. Secondly, the transparent sheet should possess high transmissivity to ensure people can see a clear and undistorted image through it. At the same time, the transparent sheet should also have reasonable reflectivity to ensure sufficient light be reflected in observers' eyes so that people can see a clear virtual image. Finally, the transparent sheet needs to be rigid and unbending to reduce the interference of deformation caused by its self-weight.
The upper surface of the base is covered with a layer of black light-absorbing cloth to minimize the extra reflected light. An opening is arranged on the base as a place to hide the projection screen. The projection screen is a mobile phone with an OLED screen with 89% NTSC color gamut and 345ppi pixel density. The lifting platform can adjust the height of the projection screen to control the imaging distance of the virtual image. The remote control is connected to the projection screen through an adapter and OTG (USB On-The-Go) cable.

The restorations of artifacts
According to the above principle and device, the restorations of artifacts followed the steps listed below.

Digital reconstruction and inversion of the miss-
ing parts: the digital images of the missing parts were obtained by Photoshop software. For the bronze mirror, the digital reconstruction of the missing part was based on its geometrically symmetrical structure. For the mural, the digital reconstruction was based on the picture when the mural was complete. Then the digital images were mirror-inverted by Photoshop. This is because the image will be mirrorinverted after being reflected. Therefore, the digital image needs to be mirror-inverted in advance, so that the reflected image is original and not inverted. The pure black images (RGB (0,0,0)) was generated by PowerPoint, which will be used in the discernibility of the restoration. 2. Projection of the digital images: the digital images and the pure black images were copied to the projection. Then the digital images were projected to the imaging plate and the virtual images of the missing part appeared behind the imaging plate. 3. Put together the virtual images and the restoration objects: the restoration objects were placed on the platform behind the imaging plate. Then the distances of the projection screen and the restoration objects to the imaging plate were adjusted to ensure that the virtual missing parts and the incomplete restoration objects were completely put together. 4. Control of the projection: the brightness of the projection screen was adjusted so that the brightness of the virtual image matches the restoration object. The projection screen was connected to the remote control to freely switch the digital images and the pure black images so that people can control the projection of virtual images by themselves.

Illumination test
To examine whether the light of virtual imaging restoration will touch the artifacts. The illuminance of the reflected light on the surface of artifacts before and after virtual imaging restoration was monitored by the TASI TA8133 DIGITAL LIGHT METER (accuracy ± 2%). The distance between the illuminance sensor and the artifacts was 100 mm. The illumination data was recorded every 2 s, and 20 data were recorded in each state.

Questionnaire surveys on the evaluation of virtual imaging restoration
In order to evaluate the effects of virtual imaging restoration, the restorations were displayed to 33 visitors, including 12 restoration scholars and 21 ordinary visitors. In the meantime, questionnaires (see Additional file 1: Table S1) was carried out. Ordinary visitors were only asked to answer questions related to the experience and make some comments. Scholars were asked to answer more questions about restoration theories and to compare virtual imaging restoration with other methods.
To help the respondents understand the virtual imaging restoration, the information about the basic principles and structure of the restoration device was provided. The information on other restoration methods and restoration theories (the same as in the introduction) was provided to scholars for reference.

Restorations of the bronze mirror and the mural
The bronze mirror and the mural before and after virtual imaging restoration are shown in Fig. 6 and Fig. 7. Before restoration (the projection screen played a pure black picture), the bronze mirror and the mural presented the original incomplete state. When the screen projected the digital images, the artifacts were restored to the complete state. The pure black picture and the digital image can be switched through the remote control, so visitors are allowed to freely control the restorations. In other words, virtual imaging restoration is a discernible method. In real restorations, there are some discrepancies in people's understanding of the discernibility. Some people want it to look the same from a far distance and different from a near place, such as the very common tratteggio retouch [22]. Others want to make it indistinguishable to the naked eyes, but distinguishable from special angles or by the special treatment. Such as dyeing the outer surface of ancient bronzes to make it indistinguishable but does not dye the inner surface to distinguish the restored area [23], or adding the fluorescent agent to restoration materials to make it distinguishable under the ultraviolet [24]. However, the main group of visitors is the ordinary people rather than the professionals who have access to view artifacts through ultraviolet lights or from the inside of artifacts. The ordinary people's rights to knowing the original state of artifacts cannot be sacrificed. In contrast, the virtual imaging restoration can present both the complete and incomplete state of the artifacts, allowing the restoration areas to be easily discerned by visitors. The virtual images and the artifact were perfectly joined together, which looks like a kind of in situ restoration. Based on the rigorous simulation of the missing parts, the outlines of the virtual images were perfectly matched with the incomplete artifacts. The fine patterns of the virtual images were also accurately docked with that of incomplete artifacts.
Of course, there were still some shortcomings in our virtual image restorations. As shown in Figs. 4 and 5, the first is that the virtual images have a certain degree of ghosting, of which the reason has been explained in Sect. "Virtual imaging restoration device" . In the future, the use of thinner transparent sheets can make the ghosting be eliminated to an ideal degree. Second, the color differences between virtual images and artifacts were still relatively large. This is mainly because the imaging plate only reflects part of the light of the digital image, there are some differences between the color of the virtual image and that of the digital image. Theoretically, by adjusting the color parameters of the digital image, the color of real artifacts can be approximated. Third, as shown in Fig. 6 and 7, although virtual images can still match the artifacts when viewed obliquely, from a very high or very low viewing angle, virtual images will be misaligned with the artifacts to a certain extent. This is probably due to the thickness of the imaging plate is still somewhat large. Finally, because virtual images were two-dimensional, depth of field cannot be produced. Therefore, the current virtual imaging restoration is only suitable for artifacts with flat surfaces, such as paintings, frescos or murals, textiles, and papers.

Illumination test
Although in principle, the light of virtual images will not reach the artifacts, we still conservatively tested the illumination of the reflected light from the bronze mirror before and after virtual image restoration. The illuminance data is shown in Fig. 8. It can be seen from the figure that the reflected light from the bronze mirror were not changed before and after the restoration, which   proved that the virtual imaging restoration is a completely reversible restoration method. Table 1 shows the results of single choice questions and scoring questions of the questionnaire. In Question 1, all respondents said that this was the first time they have seen the application of virtual imaging technology in the restoration of artifacts, which means that the work is innovative. The results of Question 2 show that 83.33% of scholars and all ordinary visitors believed that virtual imaging restoration can improve their visiting experience or understanding of artifacts, which affirms the value of our work. In Question 3, restoration scholars gave 4.33 points for the visiting experience of virtual imaging restoration, while ordinary visitors gave 4.47 points, confirming the good visiting experience of virtual imaging restoration. Most restoration scholars (83.33%) and all ordinary visitors thought that this method is worth recommending (Question 4), and the recommendation scores are 4.58 and 4.62 respectively (Question 5). This indicates that most people welcomed the virtual imaging restoration and were willing to recommend it. In the scoring of the effect of virtual imaging restoration (Question 6), restoration scholars gave 4.17 points, while ordinary visitors gave a higher score of 4.57. The difference in this score possibly mean that ordinary visitors depended more on intuitive feelings, but restoration scholars considered more evaluation factors. 75% of the scholars agreed that virtual imaging restoration does well in preserving the authenticity of artifacts, while the remaining 25% were uncertain (question 7). To a certain extent, it reflects that there were some disputes on the understanding of authenticity. All scholars agreed that virtual imaging restoration performs well in terms of reversibility (question 8), which is supported by the obtained scientific data (Fig. 8). Similarly, all scholars also agreed that virtual imaging restoration performs well in terms of discernibility. Table 2 shows the comparison results of virtual imaging restoration with other methods in several aspects by restoration scholars. The visualization results of Table 2 are shown in Fig. 9, which is represented by a radar chart.

Comparison with physical restoration
As shown in Table 2 and Fig. 9(a), most scholars believed that the visiting experience of virtual imaging restoration is better than physical restoration, which may be that virtual imaging restoration can bring more freshness and interactivity. Most scholars agreed that virtual imaging restoration has more advantages in authenticity than physical restoration, which is consistent with our point of view. On the one hand, virtual imaging restoration can present the true incomplete state of artifacts to a greater extent. On the other hand, physical restoration

Table 2 Comparison results of virtual imaging restoration and other methods
The numbers represent the number of people + virtual imaging restoration is better than the compared method -virtual imaging restoration is worse than the compared method ○ the two methods are equivalent or incomparable

Restoration methods
Visiting experience Authenticity Reversibility Discernibility Economy Scope of application

Physical restoration
Digital restoration Digital projection restoration Patterns restoration + 11 may be exposed to more complicated authenticity problems. Authenticity judgments may be linked to a great variety of sources of information, including form and design, materials and substance, traditions and techniques, and other internal and external factors [14], which undoubtedly increases the difficulty of restoration.
Obviously, the more complex information the restoration materials carried, the more content deviating from the authenticity may be introduced to restoration objects to probably confuse visitors or restorers. In the case of rerestorations, the added materials may interfere with the judgment of restorers. The restorers usually focus on presenting the truth, but in the process they may lose track of it or may have to settle for a compromise [13]. Such was the dilemma in the Tirsted Church, where about 50% of the non-original pictorial contents were removed when the Gothic wall paintings were re-restored for the fourth time in 2000 [13,25]. On the contrary, for  Fig. 9 The results of restoration scholars comparing virtual imaging restoration with other methods. a Compared with physical restoration; b compared with digital restoration; c compared with digital projection restoration; d compared with patterns restoration; e compared with AR restoration. + means that virtual imaging restoration is better than the compared method, − means that virtual imaging restoration is worse than the compared method; ○ means that the two methods are equivalent or incomparable virtual imaging restoration, people only need to take into account the authenticity of the appearance, which is easier to achieve under the technical conditions nowadays. Therefore, the virtual imaging restoration can restore artifacts in a more cost-effective way with less authenticity offset, while physical restoration may introduce more inauthenticity. In terms of reversibility, most scholars believed that virtual imaging restoration is better than physical restoration. This is because absolute reversibility is almost impossible for physical restoration since the treatments implemented on the body of the objects will produce irreversible effects [15]. In contrast, virtual imaging restoration is completely reversible. A majority of scholars also agreed that virtual imaging restoration is better in terms of discernibility. This is possibly owing to the harmony and discernibility of physical restoration areas are contradictory, which makes it difficult for physical restoration to achieve significant distinguishability while maintaining the harmony of the restoration areas. Also, in terms of the economy, most restoration scholars agreed that virtual imaging restoration is better than physical restoration. This may due to because virtual imaging restoration saves materials and labor costs in comparison with physical restoration. For the scope of application, unexpectedly, up to one-third of scholars believed that the scope of application of virtual imaging restoration is wider than physical restoration. These scholars may think that quite a lot of artifacts are not suitable for physical restoration, and virtual restoration is a better alternative. However, we think they still overestimate the scope of application of virtual imaging restoration, because it is currently only suitable for artifacts with flat surfaces.

Comparison with digital restoration
The results in Table 2 and Fig. 9(b) show that virtual image restoration has obvious advantages over digital restoration in visiting experience. This possibly due to the scholars agreed that virtual imaging restoration presents the true appearance of artifacts, while digital restoration only shows digital artifacts. For authenticity, up to half of the scholars agreed that the two methods are equivalent or incomparable. This is a confusing result, in comparison with physical restoration, most scholars believed that virtual imaging restoration is better in terms of authenticity. However, when comparing with digital restoration, scholars seemed more neutral or hesitant. We think that this is a manifestation of authenticity disputes. As for reversibility and discernibility, the statistical results are more biased towards virtual imaging restoration. We believe that since digital restoration does not directly involve artifacts, the meaning of reversibility and discernibility is not very prominent for digital restoration, which may influence the judgment of scholars to some extent. In terms of economy, scholars had clear and contradictory views, that probably based on different cost estimation models. For the scope of application, the results are unexpected that most of the scholars chose " + ". This is possibly that some scholars did not fully understand the application limitations of virtual imaging restoration.

Comparison with digital projection restoration
The results in Table 2 and Fig. 9(c) show that, compared with digital projection restoration, the most prominent advantage of virtual imaging restoration is the visiting experience. This may be that virtual imaging restoration is more innovative, while digital projection restoration is to some extent just a kind of projection. For authenticity, the statistical results are again confusing, since three options are preferred equally by scholars. We believe that the two are equivalent in terms of authenticity because they both show the real state of artifacts and restore the appearance of artifacts through virtual methods. The statistical results are the same in terms of reversibility and discernibility. Half of the scholars chose " + " and the others chose "○". Since the light from the projector is in direct contact with the artifacts, which may be harmful to artifacts, especially for the artifacts with organic substances on' surface [26]. Therefore, the reversibility of virtual imaging restoration should be better than digital projection restoration, and scholars who chose "○" may have overlooked the destructiveness of the light. Digital projection restoration can also be discernible by controlling the projection, which may be the reason why half of the scholars choose "○". The other half of scholars chose " + ", perhaps because the current digital projection restoration has not fully highlighted the discernibility, although it has this potential. According to statistics, the advantages of virtual imaging restoration in terms of the economy and the scope of application are not outstanding. As virtual methods, both methods are relatively costeffective. The scope of application of the two is limited. The former is suitable for artifacts with flat surfaces, and the latter is suitable for artifacts with damaged surfaces.

Comparison with patterns restoration
The results in Table 2 and Fig. 9(d) show that virtual image restoration has huge advantages over patterns restoration in visiting experience, authenticity, reversibility, and discernibility. It is obvious that both methods are inexpensive and have a limited scope of application. Since pattern restoration is a bit crude, virtual imaging restoration may be used as a better alternative.

Comparison with AR restoration
One of the most impressive parts of AR technology is its rich visiting experience. In fact, AR restoration and virtual imaging restoration have different advantages in visiting experience. AR restoration is undoubtedly more advantageous in terms of interaction and immersion.
With the development of AR technology, AR restoration will be more playable for artifacts. However, compared with AR restoration, virtual imaging restoration has the advantage of exhibiting artifacts more directly. Visitors of virtual imaging restoration observe the artifacts with naked eyes, while AR restoration requires AR equipment. Besides, AR restoration often requires certain training to experience, and virtual imaging restoration is more simple and direct. In an AR restoration case, visitors tried to touch the electronic screen to enlarge the artifacts and commented that the augmented object hides the real world [11]. It shows that visitors who view artifacts through AR subconsciously think that artifacts in front of them are electronic. In a sense, AR equipment cuts the direct connection between visitors and artifacts. The statistical results in Table 2 and Figure (e) do not highlight the advantages of AR restoration on visiting experience. It is possible that AR restoration is not a common technology at present, and most scholars have not experienced AR restoration in person, which affected the results to a certain extent. In the comparison of authenticity and reversibility, most scholars agreed that virtual imaging restoration and AR restoration are equivalent or incomparable in these two aspects, which is consistent with our expectations. Since the two methods are based on the combination of virtual and real, there is no much difference in maintaining authenticity and achieving reversibility. AR restoration is capable of being discernible, but half of the scholars in the statistical results believed that virtual imaging restoration is better on discernibility. This may be that in the virtual imaging restoration, discernibility is currently the only interactive function, which impressed the visitors. However, discernibility in AR restoration is only one of many interactive functions, which may dilute the attention of scholars to a certain extent. As for the economy, statistical results show that virtual imaging restoration performs better on the economy. Because AR restoration requires AR equipment and specific algorithms, which is usually expensive.
As for the scope of application, most scholars agreed that virtual imaging restoration is more limited than AR restoration because AR restoration is not limited to be used on artifacts with flat surfaces.
In the free comments, the advantages of virtual imaging restoration proposed by the respondents include: "novel method and experience", "inexpensive", "good viewing", "interactive and intuitive", "convenient and easy to promote", "you can see with naked eyes", "the original information and the restoration information can be displayed well", "good discernibility and reversibility", "completely non-destructive restoration, providing a reference for the physical restoration of artifacts". Correspondingly, some disadvantages are also highlighted in respondents' comments: "the viewing angle is limited", "the types of artifacts are limited", "the color difference is a little obvious", "the virtual image has ghosting", "it cannot be displayed in all directions". There was also a comment from respondents that "it is a means of exhibition rather than restoration". However, as already mentioned, the restoration is now not only a means to make artifacts more stable and complete but also to enhance people's understanding of artifacts [13]. In fact, virtual imaging restoration as well as AR restoration and digital restoration all have the dual attributes of restoration and exhibition.
Suggestions for improvement from respondents include: "add images save function", "improve the degree of integration of the device", "add a dynamic demonstration of the restoration steps", "optimize imaging clarity and color difference", "add sound and other elements", "realize the restoration of three-dimensional artifacts if possible".
To sum up, the results of the questionnaire gave an overall positive evaluation. Virtual imaging restoration performed well in terms of visiting experience, authenticity, reversibility, discernibility, and economy, but it is only suitable for limited artifacts. Besides, virtual imaging restoration needs to address shortcomings such as ghosting, color differences, and limited viewing angles. There is room for improvement in the functions, interactivity, and integration of devices.

The future of virtual imaging restoration
As a low-tech and low-cost method, virtual imaging technology presented encouraging results in the restoration of artifacts. Although virtual imaging restoration still has some limitations, its advantages of cost-effective and easy to implement, making it an exciting and potential application in suitable artifacts, especially for small museums. The current shortcomings of virtual imaging restoration, such as the ghosting, the limited viewing angles, and color differences, are expected to be solved by replacing the thinner imaging plate and adjusting color parameters. Moreover, richer functions and interactive content can be added to further develop this method. In addition, due to it is flexible in showing the state before and after restoration, it is also very suitable for pretesting the restoration plans and providing references for actual restoration work, which could reduce the possible operating errors.