Using virtual imaging technology to restore artifacts in situ

The restoration of incomplete artifacts is important for presenting the value of artifacts. Even though the existing methods provide rich solutions for the restoration of artifacts, they may not be the ultimate and perfect methods. The innovation of restoration methods may provide better restoration effects in some aspects or bring people a different visiting experience, which motivates people to explore new restoration methods. In this paper, based on virtual imaging technology, a virtual imaging restoration method for incomplete artifacts were proposed. In order to examine the effects and potential of the virtual imaging restoration, a bronze mirror and a mural were restored through our newly invented virtual imaging restoration device. The effects of the restorations were evaluated through the illumination test and the questionnaire surveys. The results proved that the virtual imaging restoration can visually achieve in-situ restoration, and it is completely reversible. Virtual imaging restoration also performed well in terms of visiting experience, authenticity, discernibility, and economy. Limitations of this restoration method are related to the scope of application, the viewing angle, and the imaging effects. Overall, virtual imaging restoration is worth recommending and the potential of this method and its encouraging results may provide a choice or pretest tool for the restoration of some incomplete artifacts in the near future.

with similar meanings have also developed as supplements to the reversibility principle, such as repeatability [16] and removability [17].
Discernibility: discernibility has been paid more and more attention since the Venice Charter in 1964.
Discernibility holds that "replacements of missing parts must integrate harmoniously with the whole, but at the same time must be distinguishable from the original so that restoration does not falsify the artistic or historic evidence [18]".
Even though the existing methods provide rich solutions for the restoration of artifacts, they may not be the ultimate and perfect methods. To a certain extent, the diversity of restoration methods re ects the diverse demands for restorations. With the development of the restoration concepts, the restoration has not only been used as a means to make artifacts more stable or complete but also to enhance people's understanding of artifacts [13]. The innovation of restoration methods may provide better restoration effects in some aspects or bring people a different visiting experience, which motivates people to explore new restoration methods.
In this paper, a new restoration method based on virtual imaging technology was proposed. Virtual imaging technology, also known as phantom imaging is not a new technology and has been widely used in museums to exhibit virtual artifacts or related scenes [19][20][21]. However, the potential of virtual imaging technology in restoration has not been paid enough attention in the past, and it has not been found the application of this technology in the restoration of artifacts. To examine the effects and potential of virtual imaging technology in the restoration of artifacts. Two artifacts were restored via our newly invented virtual imaging restoration device and the effects were evaluated through the illumination test and the questionnaire surveys. The results showed that virtual imaging restoration can restore artifacts well in situ and perform well in terms of authenticity, reversibility, discernibility, and economy. At the same time, virtual imaging restoration can bring a novel and good visiting experience.

Methods
A bronze mirror and a mural were chosen as the restoring objects, both of which were partially incomplete. The restorations were accomplished by our virtual image restoration device. For the evaluation of the restoration effects, we conducted an illumination test and the questionnaire surveys.

Basic principle of virtual imaging restoration
The virtual imaging restoration is established based on the principle of a kind of visual illusion phenomenon that human common sense can sometimes cause erroneous spatial perception. This principle can further be explained by the example in Fig. 2. As shown in the gure, an incomplete bronze mirror is placed behind a transparent sheet that is inclined 45 degrees. Since the sheet is transparent, the incomplete bronze mirror behind the transparent sheet can be clearly observed. Just as sometimes people can see themselves from a piece of transparent glass, the transparent sheet also has a certain degree of re ection. A mirror-reversed digital image of the missing part of the bronze mirror is placed below the transparent sheet. As the transparent sheet is with an inclination of 45 degrees and has a certain re ectivity, the virtual image will be horizontally re ected into observers' eyes. With the common sense that "light travels in a straight line," the brain tends to think that the imaging light comes directly from the transparent space in front of it, that is to say, the brain will believe that the image of the missing part of the bronze mirror is located in the transparent space behind the transparent sheet.
Because the virtual image has a focal length, that is, the virtual image has a sense of distance in our eyes and we can perceive its spatial position. Therefore, the visual distance of the virtual image can be controlled by adjusting the distance between the projection screen and the transparent sheet. When the projection screen is in a certain spatial position, the virtual image of the missing part and the bronze mirror can be at the same visual distance and can be completely joined together in the observers' eyes, thereby achieving the virtual restoration in situ.

Virtual imaging restoration device
As shown in Fig. 3, the device for the virtual imaging restoration was established based on the above principle. The structure mainly consists of the imaging plate, the platform, the base, the projection screen, the lifting platform and the cabinet. The imaging plate is a 45 degree inclined transparent sheet made of an electronic grade soda-lime glass with a thickness of 1 mm, 90% transparency, and good rigidity. It is important to note that the properties of the transparent sheet play a key role in the imaging effect. Firstly, it should maintain as thin as possible (≤1mm), to reduce the ghosting (That is because the transparent sheet has a certain thickness, and both the inside surface and the outside surface of the sheet can re ect light. The thickness of the sheet determines the positions where the same light beam generates re ected light on the inner and outer surfaces is different, so the re ected light of the two surfaces does not entirely coincide, resulting in the ghosting. The thicker the transparent sheet, the more obvious the ghosting). Secondly, the transparent sheet should possess high transmissivity to ensure people can see a clear and undistorted image through it. At the same time, the transparent sheet should also have reasonable re ectivity to ensure su cient light be re ected in observers' eyes so that people can see a clear virtual image. Finally, the transparent sheet needs to be rigid and unbending to reduce the interference of deformation caused by its self-weight.
The upper surface of the base is covered with a layer of black light-absorbing cloth to minimize the extra re ected light. An opening is arranged on the base as a place to hide the projection screen. The projection screen is a mobile phone with an OLED screen with 89% NTSC color gamut and 345ppi pixel density. The lifting platform can adjust the height of the projection screen to control the imaging distance of the virtual image. The remote control is connected to the projection screen through an adapter and OTG (USB On-The-Go) cable.

The restorations of artifacts
According to the above principle and device, the restorations of artifacts followed the steps listed below. 1) Digital reconstruction and inversion of the missing parts: the digital images of the missing parts were obtained by Photoshop software. For the bronze mirror, the digital reconstruction of the missing part was based on its geometrically symmetrical structure. For the mural, the digital reconstruction was based on the picture when the mural was complete. Then the digital images were mirror-inverted by Photoshop. This is because the image will be mirror-inverted after being re ected. Therefore, the digital image needs to be mirror-inverted in advance, so that the re ected image is original and not inverted. The pure black images (RGB (0,0,0)) was generated by PowerPoint, which will be used in the discernibility of the restoration.
2) Projection of the digital images: the digital images and the pure black images were copied to the projection. Then the digital images were projected to the imaging plate and the virtual images of the missing part appeared behind the imaging plate.
3) Put together the virtual images and the restoration objects: the restoration objects were placed on the platform behind the imaging plate. Then the distances of the projection screen and the restoration objects to the imaging plate were adjusted to ensure that the virtual missing parts and the incomplete restoration objects were completely put together. 4) Control of the projection: the brightness of the projection screen was adjusted so that the brightness of the virtual image matches the restoration object. The projection screen was connected to the remote control to freely switch the digital images and the pure black images so that people can control the projection of virtual images by themselves.

Illumination test
To examine whether the light of virtual imaging restoration will touch the artifacts. The illuminance of the re ected light on the surface of artifacts before and after virtual imaging restoration was monitored by the TASI TA8133 DIGITAL LIGHT METER (accuracy ±2%). The distance between the illuminance sensor and the artifacts was 10 cm. The illumination data was recorded every 2 seconds, and 20 data were recorded in each state.

Questionnaire surveys on the evaluation of virtual imaging restoration
In order to evaluate the effects of virtual imaging restoration, the restorations were displayed to 33 visitors, including 12 restoration scholars and 21 ordinary visitors. In the meantime, questionnaires see Additional le 1 was carried out. Ordinary visitors were only asked to answer questions related to the experience and make some comments. Scholars were asked to answer more questions about restoration theories and to compare virtual imaging restoration with other methods. To help the respondents understand the virtual imaging restoration, the information about the basic principles and structure of the restoration device was provided. The information on other restoration methods and restoration theories (the same as in the introduction) was provided to scholars for reference.

Restorations of the bronze mirror and the mural
The bronze mirror and the mural before and after virtual imaging restoration are shown in Fig. 6 and Fig.   7. Before restoration (the projection screen played a pure black picture), the bronze mirror and the mural presented the original incomplete state. When the screen projected the digital images, the artifacts were restored to the complete state. The pure black picture and the digital image can be switched through the remote control, so visitors are allowed to freely control the restorations. In other words, virtual imaging restoration is a discernible method. In real restorations, there are some discrepancies in people's understanding of the discernibility. Some people want it to look the same from a far distance and different from a near place, such as the very common tratteggio retouch [22]. Others want to make it indistinguishable to the naked eyes, but distinguishable from special angles or by the special treatment.
Such as dyeing the outer surface of ancient bronzes to make it indistinguishable but doesn't dye the inner surface to distinguish the restored area [23], or adding the uorescent agent to restoration materials to make it distinguishable under the ultraviolet [24]. However, the main group of visitors is the ordinary people rather than the professionals who have access to view artifacts through ultraviolet lights or from the inside of artifacts. The ordinary peoples' rights to knowing the original state of artifacts cannot be sacri ced. In contrast, the virtual imaging restoration can present both the complete and incomplete state of the artifacts, allowing the restoration areas to be easily discerned by visitors.
What was impressive was that the virtual images and the artifact were perfectly joined together, which looks like a kind of in-situ restoration. Based on the rigorous simulation of the missing parts, the outlines of the virtual images were perfectly matched with the incomplete artifacts. The ne patterns of the virtual images were also accurately docked with that of incomplete artifacts.
Of course, there were still some shortcomings in our virtual image restorations. As shown in Figures 4 and 5, the rst is that the virtual images have a certain degree of ghosting, of which the reason has been explained in Section 2.2. In the future, the use of thinner transparent sheets can make the ghosting be eliminated to an ideal degree. Second, the color differences between virtual images and artifacts were still relatively large. This is mainly because the imaging plate only re ects part of the light of the digital image, there are some differences between the color of the virtual image and that of the digital image. Theoretically, by adjusting the color parameters of the digital image, the color of real artifacts can be approximated. Third, as shown in Fig 6 and 7, although virtual images can still match the artifacts when viewed obliquely, from a very high or very low viewing angle, virtual images will be misaligned with the artifacts to a certain extent. This is probably due to the thickness of the imaging plate is still somewhat large. Finally, because virtual images were two-dimensional, depth of eld cannot be produced. Therefore, the current virtual imaging restoration is only suitable for artifacts with at surfaces, such as paintings, frescos or murals, textiles, and papers.

Illumination test
Although in principle, the light of virtual images will not reach the artifacts, we still conservatively tested the illumination of the re ected light from the bronze mirror before and after virtual image restoration. The illuminance data is shown in Figure 8. It can be seen from the gure that the re ected light from the bronze mirror were not changed before and after the restoration, which proved that the virtual imaging restoration is a completely reversible restoration method.
3.3 Questionnaire surveys on the evaluation of virtual imaging restoration Table 1 shows the results of single choice questions and scoring questions of the questionnaire. In Question 1, all respondents said that this was the rst time they have seen the application of virtual imaging technology in the restoration of artifacts, which means that the work is innovative. The results of Question 2 show that 83.33% of scholars and all ordinary visitors believed that virtual imaging restoration can improve their visiting experience or understanding of artifacts, which a rms the value of our work. In Question 3, restoration scholars gave 4.33 points for the visiting experience of virtual imaging restoration, while ordinary visitors gave 4.47 points, con rming the good visiting experience of virtual imaging restoration. Most restoration scholars (83.33%) and all ordinary visitors thought that this method is worth recommending (Question 4), and the recommendation scores are 4.58 and 4.62 respectively (Question 5). This indicates that most people welcomed the virtual imaging restoration and were willing to recommend it. In the scoring of the effect of virtual imaging restoration (Question 6), restoration scholars gave 4.17 points, while ordinary visitors gave a higher score of 4.57. The difference in this score possibly mean that ordinary visitors depended more on intuitive feelings, but restoration scholars considered more evaluation factors. 75% of the scholars agreed that virtual imaging restoration does well in preserving the authenticity of artifacts, while the remaining 25% were uncertain (question 7). To a certain extent, it re ects that there were some disputes on the understanding of authenticity. All scholars agreed that virtual imaging restoration performs well in terms of reversibility (question 8), which is supported by the obtained scienti c data (Fig. 8). Similarly, all scholars also agreed that virtual imaging restoration performs well in terms of discernibility.  Table 2 shows the comparison results of virtual imaging restoration with other methods in several aspects by restoration scholars. The visualization results of Table 2 are shown in Figure 9, which is represented by a radar chart.
Compared with physical restoration. As shown in Table 2 and Fig. 9(a), most scholars believed that the visiting experience of virtual imaging restoration is better than physical restoration, which may be that virtual imaging restoration can bring more freshness and interactivity. Most scholars agreed that virtual imaging restoration has more advantages in authenticity than physical restoration, which is consistent with our point of view. On the one hand, virtual imaging restoration can present the true incomplete state of artifacts to a greater extent. On the other hand, physical restoration may be exposed to more complicated authenticity problems. Authenticity judgments may be linked to a great variety of sources of information, including form and design, materials and substance, traditions and techniques, and other internal and external factors [14], which undoubtedly increases the di culty of restoration. Obviously, the more complex information the restoration materials carried, the more content deviating from the authenticity may be introduced to restoration objects to probably confuse visitors or restorers. In the case of re-restorations, the added materials may interfere with the judgment of restorers. The restorers usually focus on presenting the truth, but in the process they may lose track of it or may have to settle for a compromise [13]. Such was the dilemma in the Tirsted Church, where about 50% of the non-original pictorial contents were removed when the Gothic wall paintings were re-restored for the fourth time in 2000 [13,25]. On the contrary, for virtual imaging restoration, people only need to take into account the authenticity of the appearance, which is easier to achieve under the technical conditions nowadays.
Therefore, the virtual imaging restoration can restore artifacts in a more cost-effective way with less authenticity offset, while physical restoration may introduce more inauthenticity. In terms of reversibility, most scholars believed that virtual imaging restoration is better than physical restoration. This is because absolute reversibility is almost impossible for physical restoration since the treatments implemented on the body of the objects will produce irreversible effects [15]. In contrast, virtual imaging restoration is completely reversible. A majority of scholars also agreed that virtual imaging restoration is better in terms of discernibility. This is possibly owing to the harmony and discernibility of physical restoration areas are contradictory, which makes it di cult for physical restoration to achieve signi cant distinguishability while maintaining the harmony of the restoration areas. Also, in terms of the economy, most restoration scholars agreed that virtual imaging restoration is better than physical restoration. This may due to because virtual imaging restoration saves materials and labor costs in comparison with physical restoration. For the scope of application, unexpectedly, up to one-third of scholars believed that the scope of application of virtual imaging restoration is wider than physical restoration. These scholars may think that quite a lot of artifacts are not suitable for physical restoration, and virtual restoration is a better alternative. However, we think they still overestimate the scope of application of virtual imaging restoration, because it is currently only suitable for artifacts with at surfaces.
Digital Restoration Digital Projection Restoration Patterns Restoration the numbers represent the number of people + virtual imaging restoration is better than the compared method -virtual imaging restoration is worse than the compared method ○ the two methods are equivalent or incomparable Compared with digital restoration. The results in Table 2 and Figure 9(b) show that virtual image restoration has obvious advantages over digital restoration in visiting experience. This possibly due to the scholars agreed that virtual imaging restoration presents the true appearance of artifacts, while digital restoration only shows digital artifacts. For authenticity, up to half of the scholars agreed that the two methods are equivalent or incomparable. This is a confusing result, in comparison with physical restoration, most scholars believed that virtual imaging restoration is better in terms of authenticity. However, when comparing with digital restoration, scholars seemed more neutral or hesitant. We think that this is a manifestation of authenticity disputes. As for reversibility and discernibility, the statistical results are more biased towards virtual imaging restoration. We believe that since digital restoration does not directly involve artifacts, the meaning of reversibility and discernibility is not very prominent for digital restoration, which may in uence the judgment of scholars to some extent. In terms of economy, scholars had clear and contradictory views, that probably based on different cost estimation models. For the scope of application, the results are unexpected that most of the scholars chose "+". This is possibly that some scholars did not fully understand the application limitations of virtual imaging restoration.
Compared with digital projection restoration. The results in Table 2 and Figure 9(c) show that, compared with digital projection restoration, the most prominent advantage of virtual imaging restoration is the visiting experience. This may be that virtual imaging restoration is more innovative, while digital projection restoration is to some extent just a kind of projection. For authenticity, the statistical results are again confusing, since three options are preferred equally by scholars. We believe that the two are equivalent in terms of authenticity because they both show the real state of artifacts and restore artifacts' appearance through virtual methods. The statistical results are the same in terms of reversibility and discernibility. Half of the scholars chose "+" and the others chose "○". Since the light from the projector is in direct contact with the artifacts, which may be harmful to artifacts, especially for the artifacts with organic substances on' surface [26]. Therefore, the reversibility of virtual imaging restoration should be better than digital projection restoration, and scholars who chose "○" may have overlooked the destructiveness of the light. Digital projection restoration can also be discernible by controlling the projection, which may be the reason why half of the scholars choose "○". The other half of scholars chose "+", perhaps because the current digital projection restoration has not fully highlighted the discernibility, although it has this potential. According to statistics, the advantages of virtual imaging restoration in terms of the economy and the scope of application are not outstanding. As virtual methods, both methods are relatively cost-effective. The scope of application of the two is limited. The former is suitable for artifacts with at surfaces, and the latter is suitable for artifacts with damaged surfaces.
Compared with patterns restoration. The results in Table 2 and Figure 9(d) show that virtual image restoration has huge advantages over patterns restoration in visiting experience, authenticity, reversibility, and discernibility. It is obvious that both methods are inexpensive and have a limited scope of application. Since pattern restoration is a bit crude, virtual imaging restoration may be used as a better alternative.
Compared with AR restoration. One of the most impressive parts of AR technology is its rich visiting experience. In fact, AR restoration and virtual imaging restoration have different advantages in visiting experience. AR restoration is undoubtedly more advantageous in terms of interaction and immersion.
With the development of AR technology, AR restoration will be more playable for artifacts. However, compared with AR restoration, virtual imaging restoration has the advantage of exhibiting artifacts more directly. Visitors of virtual imaging restoration observe the artifacts with naked eyes, while AR restoration requires AR equipment. Besides, AR restoration often requires certain training to experience, and virtual imaging restoration is more simple and direct. In an AR restoration case, visitors tried to touch the electronic screen to enlarge the artifacts and commented that the augmented object hides the real world [11]. It shows that visitors who view artifacts through AR subconsciously think that artifacts in front of them are electronic. In a sense, AR equipment cuts the direct connection between visitors and artifacts.
The statistical results in Table 2 and Figure (e) do not highlight the advantages of AR restoration on visiting experience. It is possible that AR restoration is not a common technology at present, and most scholars have not experienced AR restoration in person, which affected the results to a certain extent. In the comparison of authenticity and reversibility, most scholars agreed that virtual imaging restoration and AR restoration are equivalent or incomparable in these two aspects, which is consistent with our expectations. Since the two methods are based on the combination of virtual and real, there is no much difference in maintaining authenticity and achieving reversibility. AR restoration is capable of being discernible, but half of the scholars in the statistical results believed that virtual imaging restoration is better on discernibility. This may be that in the virtual imaging restoration, discernibility is currently the only interactive function, which impressed the visitors. However, discernibility in AR restoration is only one of many interactive functions, which may dilute the attention of scholars to a certain extent. As for the economy, statistical results show that virtual imaging restoration performs better on the economy.
Because AR restoration requires AR equipment and speci c algorithms, which is usually expensive. As for the scope of application, most scholars agreed that virtual imaging restoration is more limited than AR restoration because AR restoration is not limited to be used on artifacts with at surfaces.
In the free comments, the advantages of virtual imaging restoration proposed by the respondents include: "novel method and experience", "inexpensive", "good viewing", "interactive and intuitive", "convenient and easy to promote", "you can see with naked eyes", "the original information and the restoration information can be displayed well", "good discernibility and reversibility", "completely non-destructive restoration, providing a reference for the physical restoration of artifacts". Correspondingly, some disadvantages are also highlighted in respondents' comments: "the viewing angle is limited", "the types of artifacts are limited", "the color difference is a little obvious", "the virtual image has ghosting", "it cannot be displayed in all directions". There was also a comment from respondents that "it is a means of exhibition rather than restoration". However, as already mentioned, the restoration is now not only a means to make artifacts more stable and complete but also to enhance people's understanding of artifacts [13]. In fact, virtual imaging restoration as well as AR restoration and digital restoration all have the dual attributes of restoration and exhibition.
Suggestions for improvement from respondents include: "add images save function", "improve the degree of integration of the device", "add a dynamic demonstration of the restoration steps", "optimize imaging clarity and color difference", "add sound and other elements", "realize the restoration of three-dimensional artifacts if possible".
To sum up, the results of the questionnaire gave an overall positive evaluation. Virtual imaging restoration performed well in terms of visiting experience, authenticity, reversibility, discernibility, and economy, but it is only suitable for limited artifacts. Besides, virtual imaging restoration needs to address shortcomings such as ghosting, color differences, and limited viewing angles. There is room for improvement in the functions, interactivity, and integration of devices.
3.4 The future of virtual imaging restoration As a low-tech and low-cost method, virtual imaging technology presented encouraging results in the restoration of artifacts. Although virtual imaging restoration still has some limitations, its advantages of cost-effective and easy to implement, making it an exciting and potential application in suitable artifacts, especially for small museums. The current shortcomings of virtual imaging restoration, such as the ghosting, the limited viewing angles, and color differences, are expected to be solved by replacing the thinner imaging plate and adjusting color parameters. Moreover, richer functions and interactive content can be added to further develop this method. In addition, due to it is exible in showing the state before and after restoration, it is also very suitable for pretesting the restoration plans and providing references for actual restoration work, which could reduce the possible operating errors.

Conclusion
Based on the virtual imaging technology, a virtual imaging restoration method and its device for the restoration of incomplete artifacts were presented. This method was successfully applied in the restorations of a bronze mirror and a mural. Virtual imaging restoration allowed the presence of the true incomplete state and the in situ virtual restorations of the artifacts as well as allowed visitors to freely control the restoration. This restoration method has encouraging advantages in the aspects of visiting experience, authenticity, reversibility, discernibility, and economy, and was highly recommended by ordinary visitors and restoration scholars.
Limitations of this restoration method are related to the scope of application, the viewing angle, and the imaging effects. It is only suitable for the artifacts with at surfaces and can only be viewed from limited angles. The ghosting and color differences of virtual imaging restoration are still somewhat obvious.
Overall, virtual imaging restoration has shown exciting application potential in the restoration of some artifacts. With the improvement of this method, it may provide a choice or pretest tool for the restoration of some incomplete artifacts in the near future.

Declarations
Availability of data and materials.
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare that they have no competing interests.

Funding
The research is nancially supported by "the Fundamental Research Funds for the Central Universities".  The schematic diagram of the virtual imaging restoration.

Figure 3
The schematic structure of the virtual imaging restoration device. the imaging plate; the platform; the base; the opening of the base; the projection screen; the lifting platform; the cabinet; the remote control.

Figure 4
Images of the bronze mirror before (a) and after (b) virtual imaging restoration.

Figure 5
Images of the mural before (a) and after (b) virtual imaging restoration.  Images of the mural viewed from an inclined (a), very high (b), and very low (c) angle.

Figure 8
Illumination of re ected light on the surface of artifacts before and after virtual imaging restoration Figure 9 The results of restoration scholars comparing virtual imaging restoration with other methods. (a) Compared with physical restoration; (b) compared with digital restoration; (c) compared with digital projection restoration; (d) compared with patterns restoration; (e) compared with AR restoration. + means that virtual imaging restoration is better than the compared method, -means that virtual imaging restoration is worse than the compared method; ○ means that the two methods are equivalent or incomparable.