PROTEUS: an immersive tool for exploring the world of cultural heritage across space and time scales

Abstract

In the field of digital humanities, it is increasingly necessary to develop and validate virtual reality tools that are capable of combining various scientific data in a virtualized context providing also access and user friendly consultation of online repositories. This paper reports the main aspects of the implementation of a virtual reality tool integrated with an online repository for storing 3D models, metadata and chemical analyses related to different sectors of digital humanities. The virtual reality software, developed for the Oculus Quest 2 hardware, is called PROTEUS and allows for seamless transition from the macroscopic world of digital humanities to the microscopic world of molecular sciences. The paper illustrates, by means of some case studies, the performances of this innovative tool that permits the researcher to understand and manipulate objects, to test hypotheses and to seek meaningful results, visualising the metadata while changing the parameters of the simulation in a dynamic and interactive way. This represents also a significant step forward in the democratisation of science, thanks to an user-friendly and immersive access to advanced scientific algorithms, which allow the natural perception of structural and topological features of the underlying molecular and supra-molecular systems.

Graphical Abstract

Introduction

The rapid obsolescence of the hardware and software tools used for virtual reconstructions is a serious problem in the field of digital humanities. In few years, if not months, substantial updates occur both in display devices (helmets for virtual reality, screens, CAVE devices, etc.) and in software with which the applications are created (graphics and rendering engines, visualisation tools). PROTEUS was conceived to be easily upgradeable both for new hardware and for new software releases. The other key element of the PROTEUS platform is its ability of shifting in a seamless way from the macroscopic to the microscopic level, thus allowing en easy and effective interaction between the two worlds and putting them in correlation through a direct approach in the same application.

PROTEUS allows a real time interaction with the models displayed in the virtual environment providing the ability to enter metadata and create interconnections between them. Traditional visualisation software is either focused on the microscopic world of molecules or on case-by-case macroscopic scenarios. In this context there is a lack of a general integrated software for the simultaneous visualisation of both kinds of data. In fact, in most cultural heritage applications [1], sample points are selected for further chemical analysis in well defined positions of a macroscopic model (e.g., a statue, a painting, etc.). However, the visualisation of the sample data (that is the chemical data associated with the sample points) is currently separated from the visualisation of the macroscopic object. The goal of PROTEUS is to rethink the visualisation process completely, by providing an universal and easily scalable workflow so as to cover a wide range of possible scenarios.

The main features of PROTEUS are illustrated in this work by means of two case studies concerning virtual reconstructions previously developed within the SMART laboratory of the Scuola Normale Superiore, such as the virtual reconstruction of the archaeological site of Kaulonia in Punta di Stilo. The benefits deriving from a fully scalable tool guarantee a constant updating of the projects themselves over the time. They can, indeed, be integrated and updated directly in a single application. PROTEUS therefore derives from the experience of the team’s previous work and from the need to have a tool capable of managing the various reconstructions made over the time.

Literature review

One of the first approaches for the visualization of data outside the realm of hard sciences was proposed in the early sixties of the past century by Binford and Quimby [2], leading actors of Processual Archeology (also called New Archaeology). Their aim was the establishment of a strong connection between apparently distant disciplines such as digital humanities and chemistry by means of the revolutionary use of scientific and technological techniques within archaeology. The evolution of graphical engines and the ease of use of software for the creation of 3D models derived from photogrammetry and 3D modelling, have allowed a rapid proliferation of projects related to 3D reconstructions in the field of digital humanities [3]. Over the years, also thanks to the advancements of visualisation technologies, the attention has been focused mainly on the creation of virtual environments with high visual fidelity, at the cost of a less refined user interaction, that is the ability to "immerse" the user in the virtual environment in order to carry out studies and analyses directly from within the reconstructed environment. Research has to focus mainly on the creation of different layers, easy to be prompted and able to suddenly switch from the actual state of conservation to the corresponding 3D model resulting from the cultural heritage interpretation. The final aim is the achievement of a so called "simulation slice" of that very specific context/object, which is the actual aspect of that context/object in a given period of time [4]. In the context of scientific visualisation, the main challenge is the achievement of a good balance between the accurate representation of scientific data (computed or stored in databases) and the visual quality of the virtual experience itself [5,6,7,8,9]. Traditional scientific software relies on custom tools meant to be used on conventional terminals or software developed with mouse and keyboard in mind. However, the true advantage of virtual reality (VR) is the enhanced ability of finding connections between apparently heterogeneous information, discovering correlations that would otherwise be harder to detect. The simulations are now context-dependent, so that the virtual reconstruction is not just a simple visual feature, but, rather, a reference for the user to better understand, manipulate and test different hypotheses. For instance, the term "Cyber Archaeology" has been introduced to define the simulation process that derives from the interconnection and feedback between the user and the virtual simulation [10,11,12]. In this way, it has become possible to overcome the general idea of "reconstruction", normally referred in Archaeology to a static reconstruction of an object or a context in a "realistic" and "tangible" way. The term "Virtual Archaeology" was proposed in 1990 by Paul Reilly, with the aim of describing the use of the term computer graphics and computer simulations for archaeological excavations [13]. The concept of "virtual simulation" expands what is an usually closed and unalterable context, creating a virtual space capable of increasing the perception and the knowledge of a given context and potential past by emphasising affordances [14]. VR and augmented reality (AR) are changing the way users can interact with archaeological artefacts or cultural heritage sites, especially for unreachable settings, such as underwater locations that are inaccessible to the regular visitor or tourist [15]. Despite the debates and criticisms on the method and applications of visualisation and computer graphics in the field of cultural heritage discussed in several conferences during the nineties, the technological developments during the last three decades have revolutionised the visualisation experience thanks to a constant improvement in the rendering of graphics and realism, leading to a wide use in the archaeological field. This advance has meant that the technological term, introduced by Reilly, is abandoned and replaced by "reconstruction", with reference to the visualisation environment [16].

The use of virtual and augmented reality software in the context of cultural heritage involves, in addition to all the aspects mentioned above, a constant evaluation of the usability of the interface and of the interaction dynamics. [17]

Materials and methods

Case studies

Two case studies were selected, specifically created for viewing via HMD (Head Mounted Display) and which required an updated interface with the new devices available on the market.

"Sala stemmi"

The ”Sala Stemmi” is a room with an important historic value located at Palazzo della Carovana in Pisa (Italy): all the walls are decorated with precious coats of arms representing the most important aristocratic families in Pisa. In 2012 a diagnostic survey started to evaluate the state of conservation of the constituent materials and to support a restoration intervention, particularly on the coffered ceiling. At the same time, the environment has been acquired by photogrammetry procedure: this process outputs a 3D model with high resolution texture, maintaining the real dimensions for an optimal user experience in the VR application. 359 pictures were taken using a Nikon D800 camera with the af-s Nikkor 24–120mm 1:4 camera lens, and a tripod. Aperture was set to f8 and shutter speed never under 1/160, to avoid blurriness. The ISO value was never over 800, to avoid image noise. The images were used as input to Agisoft Metashape [18], a software that performs photogrammetric processing of digital images and generates point cloud data and 3D textured models. All the parameters are tweaked and adjusted to obtain a high quality mesh (about 2.5 million vertex count) and 4k texture asset. The high density 3D model generated by Metashape is optimised for the VR implementation by retopology, new UV mapping and baking texture procedures, executed in Autodesk 3ds Max, a 3D modelling and rendering software for design visualisation, games and animations. The outcome model has 200–300k vertex count. 3ds Max has been also used to traditionally model and texture the wooden beam.

"Punta Stilo"

Punta Stilo at Kaulonia (Monasterace) hosted an ancient Greek colony founded by the Achaeans people at the end of the 8th century BC in the extreme south of the Italian peninsula. Among the most notable discoveries, there is an area dedicated to the late-archaic votive offerings, characterised by the presence of several cippus fixed in the ground (the largest of them with a dedicatory inscription) and a deposition of bronze weapons, composed of two anatomical greaves and a chalcedon helmet dedicated to Zeus. The LaDiRe laboratory at the University of Pisa provided aerial photogrammetry scanning for the site, and terrestrial photogrammetry for the reconstruction of some remains, such as the greaves, the helmet and the votive cippus [19]. A dedicated application was developed to visualise a general overview of the excavation site and a few 3D models contextualised where they were originally found: it was also possible to display a brief historical description of the object. It was developed with Unity 3D being thus usable through HTC Vive, a room scale VR HMD, whereby a user can walk freely around a definedy play area rather than be constrained in a stationary position. The HMD contains two OLED display panels with a resolution of 1080 $\times$ 1200 per-eye, with a refresh rate of 90 Hz and a 110° field of view.

Software

In the following section, the main features of the different softwares used to build PROTEUS are shortly described.

Unity

Unity [20] is a cross-platform graphical engine developed by Unity Technologies that allows the development of real time applications, games and other interactive content, such as architectural visualisations or 3D animations. Unity has been chosen as a development platform for this work, since it is one of the few graphical engines of this kind that can boast support for practically all platforms on the market leaving the code unchanged, or with minor modifications related to particular devices. The content creation task is facilitated by the fact that Unity supports the vast majority of existing formats, and interfaces very easily with the most used editing programs (whether they are programs for editing 3D models, audio, textures, etc.).

Database

Regarding the transfer of data between the VR application and the web server, the Unity API was used to allow for effective integration with the XAMPP multi platform web development tool [21]. XAMPP is composed of a Relational Database(MySql), an Open Source software compliant with the ANSI SQL and ODBC SQL standard licence GNU General Public License, which supports programming languages such as Java, PHP, Python and many more. The use of MySql is matched with the XAMPP multiplatform software made of Apache HTTP Server and all programming tools in PHP and Perl. Specifically, the database was used as a joining system for the scenes built inside PROTEUS, through the storage of interchange formats (JavaScript Object Notation—JSON)

Proxima

The PROTEUS software also includes the Proxima Molecular Perception library [22] for handling chemical data. In particular, the Proxima software is a C++ library (integrated within Unity [20] using underlying C functions) that provides methods for the parsing of cartesian coordinates (XYZ file format), together with the detection and evaluation of chemical bonds and non-covalent interactions. The integration of Proxima within PROTEUS allows for the integration of further chemical tools in future releases. Another advantage of using Proxima is the correct handling of periodic structures such as the ones obtained from X-Ray Crystallography or Molecular Dynamics (MD) simulations. In these systems, a unit cell is repeated along three vectors in order to produce the entire system, possibly adding some bonds that might be missing because they involve atoms that are on two opposite sides of the unitary cell. The representation of molecules through Proxima in Unity is done with a ball-and-stick representation because of its common use and advanced possible optimizations available in the literature [23]. The use of Proxima in a VR application represents, in our opinion, a huge step forward in the democratisation of science, since more and more non-expert users can have access to advanced perception algorithms.

Meshlab

Modelling and colour projection were performed with MeshLab, which allows for the creation of 3D models with high-resolution geometry and high quality texture. Meshlab is an Open source system for processing and editing 3D triangular meshes. It is provided with a set of tools for editing, cleaning, healing, inspecting, rendering, texturing and converting meshes.

Results

The first step was the implementation of a general infrastructure including: (1) the loading of macroscopic 3D models (e.g., those obtained from photogrammetry or other approaches) with associated sample points; (2) molecular specific visualisation tools (e.g., the so-called ball-and-stick representation in VR applications); and (3) the definition of a common file format for the association of the sampling data to the sample points. At this stage, the sampling data could be either a molecule generated from a file containing atom types and coordinates (XYZ file) or one from the Protein Data Bank (PDB) [24], or it could also be some general text, for example, with a description of the sample point. The choice of the PDB file format was motivated by the large amount of available chemical data (obtained either from experimental studies and/or quantum chemical computations) that is compatible with such format. In particular, the Protein Data Bank acts as a huge repository of chemical structures obtained by different experimental techniques [(e.g., Crystallography, nuclear magnetic resonance (NMR), etc.]. While PDB and XYZ file formats are conceptually similar (both of them store types and cartesian coordinates of the atoms forming the considered molecule), the XYZ file format has the advantage of being simpler to parse, lighter and easily obtainable from several quantum chemistry softwares (e.g., Gaussian [25]). The PROTEUS development has therefore been focused on the design and implementation of different levels of visualisation, which allow for a seamless transition from the macroscopic object under analysis to its microscopic components (e.g., the chemical compounds detected through spectroscopy), in order to stimulate the search for one of the possible "simulation slices" for that context. [4] In a previous work [26], an application devoted to the Sala Stemmi was implemented with the main aim of testing a multi-scale approach, with direct integration with UnityMol [27], a separate software capable of displaying molecules from the PDB database.The purpose of PROTEUS was to reconsider the results of the first exploratory study bringing it to a new level of visualisation and interaction, by transforming a closed application, devised for a single specific case study, to an open application, dynamically integrated with new types of simulations. To this end, PROTEUS supports different types of metadata (e.g., images, text, molecules) useful for the immersive visualisation of the results of different chemical analyses performed during the diagnostic campaign, such as X-Ray (XR), ultraviolet–visible (UV–vis), and infrared (IR) spectroscopic analyses, reflectography, and UV light shooting. The first result was a dynamical integration of external data with model reconstructions, openly connected online with various heterogeneous databases and repositories, which contain extremely different data. The process of linking a web repository is facilitated by the need of inserting a link to the web server folder. After establishing the connection parameters, it is possible to connect to a web server and download models in real time within the application via an upload interface. Then, the imported models can be translated, rotated, and inserted in the simulation, creating a virtual scenario, which allows for the addition of metadata to the models, such as links to other repositories or databases. As sketched in Fig. 1, from a chemical perspective it is possible to connect to a generic online repository and download datasets from databases, with the possibility of dynamically linking molecules to 3D models.

Fig. 1

PROTEUS. Schematic diagram showing the main skills of PROTEUS software for Cultural Heritage application

As mentioned above, the PROTEUS tool derives from the prototype of a VR viewer, capable of managing macro-elements (Unity reconstruction) interfaced with a software that manages the micro-elements (UnityMol). PROTEUS overcomes the typical limitations of a closed software, restricted to the specific case study, interfaced with an external tool for chemical management. As a matter of fact, the creation of a modular software allows for ease updates and implementations on different platforms, always retaining a powerful molecular viewer. In this way, it became possible to add or modify sampling points on the 3D models in the Sala Stemmi (coffered ceiling) by downloading the 3D models of the "Sala Stemmi" from the online repository and then entering the points (see Fig. 2). The implementation of a new relational database able to combine the macroscopic level (i.e., the 3D models and the web directory) with the microscopic level (the chemical molecule repository) allows for linking the analysed points with the chemical data. For instance, in Fig. 3 an example is reported, where the chemical structure of a dimethyl ether molecule detected in the binder used to realize the decoration of the wooden beam is visualized in the 3D model of Sala Stemmi room. Proxima handles the solvation procedure by generating a sphere of water molecules around the solute in real-time locally on the HMD. In the same way, the tool allows for visualising all the available spectroscopic measurements that can be managed by the operator. The use of a JSON file permits to represent the complex hierarchy of a macro object containing different sample points with different metadata associated in a simple plain text file. Then, the implemented database allows for the creation of a junction system, which is scalable (especially for a future-proof vision) with respect to both the number of scenarios and the number of directories. It will therefore be possible to further add or modify in the future the support for new repositories of both molecules and 3D models, keeping intact the structure of the database, which manages all the links to the assets through the JSON files. This degree of interactivity and flexibility allows for accurate and up-to-date reconstructions. Furthermore, PROTEUS can be connected with a generic web server, thus enhancing the choice of conceivable applications, concerning both organisation and implementation. It is also possible to employ relational databases that are capable of communicating with PROTEUS, providing a more structured access to information (see the sketch of Fig. 4) . As already highlighted, a key feature of the tool, together with the possibility of interconnecting the macroscopic data with the microscopic counterparts, is the availability of a simulation software that is constantly updated and capable of dynamically loading different simulations. For this purpose, the necessary assets and data were recovered from previous virtual reconstructions regarding the case study of Kaulonia [28, 29]. The Punta Stilo sanctuary in Kaulonia was an ideal experimentation field thanks to the possibility of operating at different scales, with various 3D surveying and modelling procedures, which guarantee the acquisition of different pieces of information. As shown in Fig. 5, it is possible to load in real time the various models from the web repository coming from the virtual reconstructions, out of date by now, and which can be updated and recompiled for the new hardware, directly within PROTEUS, adding the metadata and various information to the finds and analysing them freely by touch. A Head Mounted Display was chosen to provide improved immersion and above all to ensure greater portability of the application. Once the method of interaction was established, a visualisation system was activated for the archaeological finds with the creation of a user-friendly interface that is able to manage different tasks customised according to the user needs and to be available in any position within the excavation. It was possible to implement the simulation in PROTEUS, by importing the various archaeological finds with relative descriptions and data, as shown in Fig. 6.

Fig. 2

The wooden beam. The data sampled on the wooden beam of sala stemmi. On the top left a calcium carbonate molecule in the Sala Stemmi

Fig. 3

Molecular metadata. The molecular metadata representing a molecule of dimethyl ether spherically solvated by Proxima is loaded directly in the Sala Stemmi virtual reconstruction

Fig. 4

The software architecture. The overall architecture of the software

Fig. 5

Sample points. The different types of metadata available

Fig. 6

Macroscopic database. The macroscopic database can be interrogated in Virtual Reality so as to retrieve 3D models of interest

Discussion

The internal architecture of PROTEUS employs a hierarchy of objects where SamplePoints are child of a MacroObject. The MacroScene script handles all the MacroObjects. It has been decided to employ the MRT mixed reality toolkit for handling natural hand interactions so as to simplify the visualisation process from the user’s perspective. The MacroObject script has also a public reference to the 3D model representing the corresponding object. The Sample Points are inserted as children of the MacroObject, with each Sample Point being a prefab with a script indicating whether the associated metadata is a text or a molecule (in the latter case, the content of an XYZ file is put in the text field of the script and Proxima handles the correct molecular representation). In order to specify the relative position of the metadata to be displayed with respect to the Sample Point position, a Sample Placement flag is specified that could be of type:

• XY;

• XZ; and

• YZ.

Together with a distance and angle value the above data permit to place the metadata in the desired location. In Fig. 7 the metadata position with respect to the Sample Point is shown. The final results, in the case of the Sala Stemmi, are the sample points shown in Fig. 2, where the metadata are the blue labels with the relative text describing the age and the type of sample collected. For the integration with the online repository, the goal was to create a system as simple, scalable and intuitive as possible. For this reason, PROTEUS can be interfaced both to simple storage systems, such as a web directory, and to structured databases. The simplest method for uploading models in the platform is from the web repository. It is in fact possible to connect, through a specific script, to a folder in a web server in php to check the content and download the 3D models. Thanks to the ease of use and the possibility of simply selecting and downloading from a web directory, it is possible to link PROTEUS with any web host that supports apache and php, simply by uploading assets into a directory, indicating their location to the the tool configuration interface. Then all the contents can be viewed and used directly from the application. The need for large repositories of chemical data is well established in molecular sciences, as witnessed by the many data banks already available, such as the PDB [https://www.rcsb.org]. Crystallographic Open Database [http://www.crystallography.net/cod/] etc. In most cases, these data banks store data so as to make them available through public URLs. For instance, our team has made available very accurate geometrical structures of a large number of prototypical organic molecules contained in an internal repository available through the team’s internal gitlab. These URLs can be used in the same way as the ones from the Protein Data Bank in the JSON file format (as explained in the next section) as a source of chemical information to be linked to a given macroscopic model. The JSON file has been implemented to automate the composition of the scenes with the metadata. The file stores the information needed by PROTEUS to set the virtual environment selected by the user. The attribute-value pairs determine the 3D objects and HDRI images and the textual or molecular information to be visualised in a specific spot in the scene. As described above, one of the focal points of the project is to import data and 3D models into the project’s data asset without rebuilding the entire application from scratch.To this end, the application is connected to a PHP script, which checks the content of the web directory and returns the content accessible directly from the application. Once the chosen content has been selected, the application generates the URL for the data asset request and the server returns the updated unity data asset (see Fig. 8). In the case of internal use in the SMART laboratory and of the case studies listed below, both the simple (in the case study of Kaulonia) and advanced (Sala Stemmi) settings were used, which consist in a connection to an internal 3D models and molecule information database. The application of PROTEUS to different case studies, where the acquisition of the models is carried out by different methods, shows not only the potential of the tool, but also demonstrates that it employs an innovative approach able to prevent the loss of virtual reconstructions developed for obsolete hardware. In fact, the implementation of different data sources leads to the availability of a system capable of keeping the reconstruction support updated for the new Virtual Reality display devices. The problem of obsolescence of immersive hardware technologies is particularly prominent, given that in few years different types of helmets can follow one another, even from the same manufacturer. The change of hardware and the continuous updates therefore entail an obsolescence of the software and the need to dedicate resources to the constant updating of projects. From a technical point of view, the added value of PROTEUS with respect to the exiting tools can be summarized as follows:

• Support of virtual reconstructions over time: it is in fact possible to use it for all those virtual reconstructions in which objects have to be placed in a context of discovery or study, adding scenarios, virtual models, descriptions and archaeometric data.

• Ease of use also by non-specialists, who do not have advanced programming background, being capable of displaying 3D models in VR, importing them directly within the application available for an immersive analysis.

• Resilience to the obsolescence of virtual reconstructions, providing a tool capable of managing the individual elements and combining them in a single application.

• Free and simple interface with several tools, equipped with different ways of uploading assets, ranging from simple offline upload, to uploading models from a web directory to integration with more structured databases.

• Straightforward addition of new models, studies and data without the need to re-implementation, re-compilation and re-distribution.

• Possibility of expanding the application for the use of other HMDs but also via desktop systems thanks to the deployment flexibility of the Unity 3D game engine.

From the point of view of use and interconnection between artistic and chemical data related to archeometry PROTEUS allows for:

• Real-time interaction with virtual reconstructions, adding elements and testing hypotheses directly within the simulation even in multi-session.

• Immersive access (through VR) to advanced scientific algorithms, which allow for the automatic perception of structural and topological features of the underlying molecular and supra-molecular systems.

• Thanks to the integration with the Proxima molecular viewer, it is possible to directly access the existing data banks such as the Protein Data bank or the open Crystallography database.

• Local access in real-time on the HMD is also possible to all of the molecular analysis tools (solvation procedures, computation of charges, etc.) implemented in Proxima.

Fig. 7

The metadata coordinates. The relative position of metadata with respect to the Sample Point position. The position is specified indicating a plane (XY, YZ, XZ), a distance and an angle. The angle determines a rotation around the third axis not included in the specified plane

Fig. 8

Database consultation. The internal architecture for database consultation

Conclusions

The PROTEUS application developed and optimised for VR allows for an immediate and intellegible interpretation of heterogeneous data; VR, indeed, offers a unique and highly immersive point of view within the reconstructed context, a factor that makes understanding closer to our perceptive sensitivity. The possibility of having an open tool, freely interfaceable with other applications and online repositories, permits great flexibility and paves the route towards new methods of visualisation and shared interaction between professionals with different skills. Thanks to the dynamic loading of 3D models and molecular data, it is possible to understand and manipulate objects, to test hypotheses and to seek meaningful results, visualising the metadata and changing the parameters of the simulation in a dynamic and interactive way. Its use promotes collaboration and knowledge sharing, favouring multidisciplinarity and effectively creating a strong point of contact between two apparently very different domains, such as chemistry and cultural heritage. The software was tested for different case studies demonstrating the capability of interaction between the macroscopic and microscopic levels and of highlighting and analysing artefacts dynamically loaded in real time from the web. Moreover, PROTEUS allowed to adapt and implement previous works, based on obsolete hardware and software, integrating and using them through a development tool capable of being constantly and individually updated. Even if the PROTEUS framework is at present available on Oculus Quest 2, the software is easily exportable on other types of HMDs and devices that can be interfaced with Unity. One of the planned developments is to move not only from a dimensional point of view, but also from a temporal point of view, giving access to multiple levels of reconstruction in in such a way as to analyse the evolution and degradation of the works and artefacts included in the simulation. The potential of virtual simulations can be expanded with the possibility of multi-user sessions and offers the opportunity for scholars to collaborate, working together in the same virtual space, even if they are physically distant.