Knowledge graph representation method for semantic 3D modeling of Chinese grottoes

Abstract

The integration of 3D geometric models with semantic information significantly improves the applicability and comprehensibility of cultural heritage. The semantic 3D modeling of Chinese grottoes poses challenges for individuals without expertise in cultural heritage due to gaps in domain knowledge and discrepancies in understanding. However, the existing domain ontology and knowledge graph provide an insufficient representation of the knowledge of Chinese grottoes. To overcome these obstacles, we propose a knowledge graph representation method to provide explicit knowledge for participants at different stages of semantic 3D modeling of Chinese grottoes, which includes schema layer construction and data layer construction. On the schema layer, we design a domain ontology named ChgOnto (Chinese Grottoes Ontology) that consists of four high-level concept classes: spatial object, informational object, digital device, and temporal object. Among the classes in the ChgOnto, the components (e.g., cliff wall, cave roof, cliff wall footing), elements (e.g., primary Buddha statue, pedestal, decoration), the properties (e.g., length, width, depth) of caves and niches in Chinese grottoes as well as the spatial relationships between them are all precisely defined. ChgOnto also reuse the classes from the renowned CIDOC CRM ontology in the cultural heritage field and GeoSPARQL in the geospatial domain, facilitating integration between the two subjects. Considering the schema layer as the conceptual data model, the data layer extracts knowledge from unstructured text through natural language processing tools to instantiate the abstract classes and fill the properties of the schema layer. Finally, the knowledge required for semantic 3D modeling of Chinese grottoes is expressed in the data layer by a knowledge graph in a fixed expression form. Dazu Rock Carvings, a World Heritage site in China, is selected as a case study to validate the practicality and effectiveness of the proposed method. The results reveal that our method offers a robust knowledge-sharing platform for the semantic 3D modeling of Chinese grottoes and demonstrates excellent scalability. The method proposed in this paper can also serve as an informative reference for other types of cultural heritage.

Introduction

It is no longer difficult to obtain high-precision 3D geometric models of cultural heritage based on reality [1,2,3]. With regard to the applications involving 3D models of cultural heritage, the relevance extends beyond geometric and graphical aspects to encompass object semantics. For example, it is crucial to add physical and mechanical parameters to the cliff in a 3D model and establish its structural relations with the mountain for analyzing the protection and reinforcement of the grottoes [4]. Assigning rockfall hazard levels to cliff-top slopes can mitigate potential safety risks for tourists [5]. Semantic annotations on 3D models of the artistic buddha statues in the grottoes facilitate cultural dissemination. However, purely geometric models are insufficient for the abovementioned applications. Semantic 3D modeling extracts semantic classes of the geometric primitives from 3D models and associates the attributes of those extracted classes with rich information [6, 7], which have emerged as research trends recently [8, 9]. Integrating external knowledge can significantly enhance the semantic information of the digital 3D model, improving its interpretability and usability [10]. Automated algorithms for processing point cloud data have been developed to efficiently and accurately extract geometric elements [11,12,13,14,15]. It requires domain knowledge to relate the semantic meaning to the geometric elements. Semantic 3D modeling is a complex process, where the lack of domain knowledge of cultural heritage among personnel involved in data collection, data processing and 3D modeling from different subject backgrounds leads to the inconsistency of the geometry and attributes in the 3D semantic model. Therefore, an effective knowledge-sharing platform is needed to support semantic 3D modeling and ensure the consistency and completeness of knowledge among different participants, particularly in the context of Chinese grottoes.

Chinese grottoes, which are carved stone caves or niches into cliffs functioning as buddhist temples, include UNESCO World Heritage sites like Mogao Caves, Yungang Grottoes, Longmen Grottoes, and Dazu Rock Carvings. Understanding the spatial structure (e.g., cliff wall, cave roof, cliff wall footing), basic elements (e.g.,buddha statue, pedestal, decoration), and attributes (e.g., length, width, depth) of Chinese grottoes poses significant challenges for people who are not experts in cultural heritage due to the lack of a guiding mechanism for the process of semantic 3D modeling. For instance, when performing the task of point cloud semantic segmentation on a grotto scene, the extraction of geometric primitives by data managers may not align with the requirements of cultural heritage. Similarly, when designing the underlying data structure of semantic 3D models, it may be difficult for information system developers who are usually specialized in computer science to correctly define the properties of geometric objects to meet the requirements of applications. Even data managers and information system developers may have different understandings of the same geometric elements and attributes in the grotto scene. As such, in the different stages of semantic 3D modeling, it is necessary to provide adequate knowledge for people who are not experts in cultural heritage and to maintain the consistency of knowledge among different participants. Urgently, we need to establish a knowledge representation method for Chinese grottoes that facilitates knowledge sharing and interdisciplinary communication, thereby promoting collaboration among different stakeholders within a unified knowledge system [16].

Knowledge graphs, which are structured forms of human knowledge, have emerged as effective carriers for representing information and knowledge in the digital realm, gaining widespread attention from academia and industry [17]. Knowledge graphs accumulate and convey real-world knowledge, with nodes representing entities of interest and edges denoting relationships between entities [18]. Due to their impressive information integration and knowledge expression capabilities, knowledge graphs can provide valuable guidance for the semantic 3D modeling of Chinese grottoes.

This paper presented a knowledge graph representation method for Chinese grottoes to address the problems of knowledge scarcity and inconsistency in semantic 3D modeling. The method consisted of two steps: schema layer construction and data layer construction. In the schema layer stage, we constructed an ontology model called ChgOnto following the ontology design processes, refining top-level concepts such as space objects, information objects, digital device objects and time objects. ChgOnto provides a comprehensive representation of the spatial structure composition, elements of caves and niches and the spatial relationships between them. Additionally, it defines important properties of caves, niches, and the main buddha statues. Finally, we utilized natural language processing technology to construct a knowledge graph of Chinese grottoes as the data layer, building upon the schema layer. By adopting the knowledge representation form of a knowledge graph, this method offers unified, comprehensive and intelligent knowledge guidance for professionals from various disciplines.

The main contributions of this paper are as follows:

• We constructed a domain ontology called ChgOnto for 3D semantic modeling of Chinese grottoes. ChgOnto addresses the existing gap in ontology design and knowledge representation specific to Chinese grottoes.

• ChgOnto reused classes from the standardized ontology CIDOC CRM (Conceptual Reference Model) in the field of cultural heritage and GeoSPARQL in geographic information. This ensures the extensibility of the ontology across these two domains and effectively combines geographic information with cultural heritage.

• The spatial relationships were used as semantic encoding to express the geometric structure between the geometric primitives of the grottoes.

• A knowledge graph representation method was proposed for Chinese grottoes, utilizing ChgOnto as a conceptual data model. This method leverages the schema layer of the knowledge structure, which acts as a constraint, facilitating knowledge aggregation between nodes in the data layer.

This study provides a knowledge graph representation method in the absence of domain knowledge with regard to the semantic 3D modeling of Chinese grottoes, providing the necessary knowledge to participants from interdisciplinary backgrounds in the modeling process. The rest of this paper is organized as follows. In Sect. "Related work", related works and a summary is presented. The methodology of the knowledge graph representation is given in Sect. "The method of knowledge graph representation". In Sect. "Experiment and result", the feasibility of this method was demonstrated through a case study. Dynamic knowledge graph, ontology extensibility, and limitations are discussed in Sect. "Discussions". This paper ends with concluding remarks in Sect. "Conclusions".

Related work

Semantic 3D modeling

3D Geographic Information Systems (3D GIS) and Historic Building Information Modeling (HBIM) technologies greatly facilitate the fusion of semantics objects, attributes, and relationships in semantic 3D modeling of cultural heritage. This has expanded the applications of 3D models in cultural heritage such as visualization, information management, scene understanding, and decision-making support [19,20,21]. Through combining laser scanning, photogrammetry, and computer vision with 3D GIS, it was critical to create thematic vector layers for archaeological sites, which linked the geometric shapes with relevant information regarding their preservation status, such as levels of decay, crack patterns, and materials characterization [22]. CityGML is the international standard of the Open Geospatial Consortium (OGC) for the semantic definitions of all objects (i.e., features) that are relevant for applications of 3D city models: buildings, components, and the relations between them [23, 24]. This principal focus is on 3D city models, but Application Domain Extension (ADE) as its built-in mechanism also adds new modules for city walls and monuments [25]. Li et al. [26] developed the ACRoofADE as an extension model of CityGML to express and ensure the consistency of the geometrical, semantic and topological relationships with regard to rebuilding the ancient Chinese-style architectural roof. Pepe et al. [27] proposed a specialized method for creating the CityGML model of a historically and architecturally important bridge. This model served as a valuable tool to support spatial planning process as well as implement measures for protection, monitoring and preservation of urban elements.

According to the history document (for HBIM) and reality-based recording data (for as-built HBIM), a library of parametric and semantic elements (such as a wall, roof, stair, door, and window) can be created and used to establish the heritage [28]. Converting point cloud to a semantically rich HBIM model involves several tasks, including: (1) semantically segmenting the point clouds into geometric components, (2) creating geometric models of the components, (3) allocating semantic information such as component category and material to the components, and (4) restoring the spatial relationships between the geometric components [29]. HBIM enables the exchange of information and semantic interoperability among components in historic buildings through the use of Industry Foundation Classes (IFC) standards [30]. Cursi et al. [31] examined BIM methods to enhance semantic enrichment and expand the knowledge representation through taking the context of HBIM process into consideration. Simeone et al. [32] developed a knowledge base prototypal platform to enhance the semantic representation capability of applying BIM for architectural heritage. The INCEPTION project proposed a workflow that placed focus on efficient 3D digitization methods, post-processing tools for semantic 3D modeling and web-based solutions in order to provide access for both experts and non-experts [33]. Through the HBIM modeling process, it identified the cultural heritage buildings semantic ontology and data structure of information catalogue, which allowed the integration of semantic attributes with hierarchically and mutually aggregated 3D digital geometric models for heritage information management [34, 35].

Semantic 3D modeling, which incorporates 3D GIS and HBIM technologies, aims to convert unstructured geometric models to semantically interpretable geometric objects in cultural heritage and to associate with additional knowledge [36]. However, due to the diversity and geometric complexity of cultural heritage, this is a complex, time-consuming, and labor-intensive task that requires significant efforts and resources [37, 38]. To improve reconstruction efficiency, many researchers have focused on developing automated feature extraction and semantic segmentation algorithms of the point cloud, aiming to extract or segment semantically meaningful geometric primitives economically and rapidly. However, another crucial issue has been overlooked, i.e., the process of semantic 3D modeling requires professionals with diverse backgrounds to collaborate, including data collection personnel, data processing personnel, point cloud segmentation personnel, 3D modeling personnel, and system development personnel. The semantic meaning requires the formalized knowledge representation through a machine-readable and controlled specific glossary, aiming to form a common basis to be recognized and shared by different experts working in the cultural heritage field. The formalized knowledge representation ensures the continuous update and refinement of semantics over time, regardless of the type of 3D model used [39]. Due to the diversity of cultural heritage, the 3D sematic modeling requires organizing the knowledge with formalized representation method [40].

Knowledge graph

In 2012, Google introduced the knowledge graph, aiming to enhance the semantic understanding of search engine and improve the quality of user searches on the web [41]. Since its inception, knowledge graphs have been extensively used across various industries and applications such as multi-source knowledge fusion [42], intelligent question answering [43], complex parts assembly processes [44], and knowledge sharing in product development [45]. Cultural heritage is one of the typical representatives of applying the knowledge graph method for the structured representation of domain knowledge [46]. According to the knowledge domains, knowledge graphs can be broadly categorized into open knowledge graphs and domain-specific knowledge graphs.

Open knowledge graph

In recent years, open knowledge graphs have been used to solve the problem of correlation between spatial entities and semantics. For example, Chen et al. [47] presented a crowdsourced geographic knowledge graph named CrowdGeoKG which extracted different types of geo-entities from OpenStreetMap and enriched them with human geography knowledge from Wikidata. Dsouza et al. [48] introduced WorldKG with the aim of addressing the issues of high heterogeneity, diversity, and incompleteness in OpenStreetMap through providing a comprehensive semantic portrayal of geographic entities such as buildings, mountains and cities. Chadzynski et al. [49] developed autonomous intelligent software agent system based on cognitive architecture, which was capable of automated instantiation, visualization and analysis of multifaceted City Information Models (CIM) in dynamic geospatial knowledge graphs. However, these knowledge graphs are applicable to macroscopic geographic entities but cannot meet the application requirements of expressing the microscopic geometric structure of spatial entities.

In the field of cultural heritage, Europeana [50], ArCo [51], and WarSampo [52] have been developed as semantic infrastructure and knowledge-sharing platforms, which advocate public participation and exploration. Europeana Data Model (EDM) used the standardized thesauri and vocabularies, providing the access to creating a semantic contextualization for objects. ESM allowed the semantic operations on the metadata and the enrichment with Linked Open Data on the web [53]. Nishanbaev et al. [54] integrated 3D models of cultural heritage with linked open data (LOD) from platforms like DBpedia and GeoNames via Web-GIS, which significantly enhanced digital cultural heritage exploration. Liu et al. [55] constructed a large-scale knowledge graph of ancient Chinese history and culture to facilitate public understanding of historical and cultural knowledge promptly and accurately. However, the data used for constructing the open knowledge graph comes from the Internet, which contains cross-domain knowledge with various data structure, emphasizing the breadth of knowledge while limiting the practicability in real-world applications [56, 57]. In specific domains and applications, more knowledge nodes do not necessarily equate to better results. Users expect the underlying structure and knowledge content of the knowledge graph to satisfy their application requirements entirely. Therefore, the construction method of open knowledge graph cannot meet the specific requirements of providing efficient knowledge for the semantic 3D modeling of grottoes, such as the basic components of the grottoes, the size of caves and niches, the name and the number of buddha statues, etc. Defining the abstract structure underlying the knowledge graph can aid in the effective conceptualization of domain knowledge.

Domain-specific knowledge graph

Many scholars used abstract knowledge representation model as the constraint to build spatial-relevant knowledge graphs to ensure that knowledge content and structure meet the application requirements. For instance, Wang et al. [58] proposed a geographical knowledge representation model for constructing the map of geographical knowledge to present the characteristics of geographical knowledge from spatial and temporal views. Zheng et al. [59] constructed a hierarchical structure knowledge representation model to express the spatiotemporal characteristics as well as the evolution process of geographical elements. Berta et al. [60] conceptually transformed the knowledge in the urban space domain into a semantic structure with concepts, elements and their interrelationships. They constructed the knowledge representation model using the ontology method to improve the quality of urban space planning. Such spatial entity knowledge representation method requires either ontology design or conceptual model to provide the basic structure for building knowledge graphs.

CIDOC CRM stands out as one of the most successful formal ontologies in the cultural heritage field. It became the ISO 21127 standard in 2006 and was developed by an interdisciplinary team of experts from the International Documentation Committee (CIDOC) of the International Committee of Museums (ICOM) [61, 62]. Currently, CRM has spawned many extended models to handle specific subfields. For instance, CRMgeo is used as a global schema for integrating spatiotemporal properties of temporal entities and persistent items [63]. Niccolucci [64] based the CIDOC CRM and its extensions (CRM_sci and CRM_dig) to record scientific experiments involved in archaeological investigations, called CRM_as. Gergatsoulis et al. [65] used CRM-based models (CRM_archaeo and CRM_sci) to represent archaeological excavation activities and observations of archaeologists in excavation site work. Messaoudi et al. [66] based CIDOC core, CRM_sci, CRM_dig, and CRM_inf to develop an ontology model for integrating semantic, spatial, and morphological dimensions. Ranjgar et al. [67] developed a POI (points of interest)—Based on data model for heritage sites in Iran, which integrates spatial semantics with cultural heritage information. Fafalios et al. [68] extended CIDOC CRM to construct the SeaLiT ontology as the architecture of the knowledge graph. Ronzion et al. [69] applied CRM_ba to Roman architectural documentation, using semantic models to encode architectural structural information. Kim et al. [70] proposed an ontology-based Korean Cultural Heritage Data Model (KCHDM), support the extraction of semantic patterns from the abundant textual data collected from the heritage database of related institutions.

In addition, there are also some novel ontology models used to express the specific knowledge structure in the field of cultural heritage. For example, Acierno et al. [71] constructed four main knowledge domains, including artifacts, life cycle, architectural heritage investigation process, and actors, for supporting the representation of information and knowledge in architectural heritage conservation and investigation activities. Cacciotti et al. [72] proposed a computer-readable ontology representation of historical building damage diagnosis containing triggering events, mechanisms, agents, and damages. Quattrini et al. [73] expressed historical buildings from the spatial scale as building, building components (e.g., nave, apse, vestibule, etc.), and building elements (e.g., doors, walls, columns, etc.), providing a data structure for the knowledge graph.

In the domain-specific knowledge graphs based on ontology or conceptual model, Bai et al. [74] constructed a cultural knowledge graph of Beijing, the ancient capital of China, based on domain ontology designed for material culture, system culture, behavioral culture, and mental culture, and developed a visual and interactive question-and-answer platform. Fan et al. [75] constructed an OpeOnto (opera ontology) containing deep semantic (theme, emotion) classes, based on which they constructed a multi-modal (semantic, image, and music) knowledge graph in the OpeOnto-driven way. Dou et al. [76] constructed the domain ontology and knowledge graph of Chinese intangible cultural heritage. Lu et al. [77] constructed the YunJin knowledge graph with video as a data source by the YunJin Video Resource Ontology model (YJVO).

Brief summary

The CityGML and IFC standards in GIS and HBIM have not yet provided usable semantic specifications and knowledge for Chinese grottoes. Existing ontologies and knowledge graphs have not found relevant studies on knowledge representation in Chinese grottoes. Therefore, the unified description of entities and attributes, as well as the knowledge representation method for 3D semantic modeling tasks of Chinese grottoes, still requires further development and research.

In the knowledge expression of Chinese grottoes, open knowledge graphs cannot reach the depth and structure of knowledge in subdivided domains. Nonetheless, ontologies can mitigate this challenge, which provide a structured and formal way to represent knowledge. Through providing a shared vocabulary and a set of concepts for a specific domain, ontology provides the basic concepts and structure for knowledge representation. Domain-specific knowledge graphs provide platforms for knowledge sharing and collaboration, allowing different individuals to access while ensuring knowledge consistency in the production process. Using ontology-based conceptual structures can enhance and update knowledge graphs without diluting knowledge due to the addition of nodes and edges.

The method of knowledge graph representation

This section provides a detailed introduction to the knowledge graph representation method for Chinese grottoes, which combines the "top-down" and "bottom-up" strategies. The method consists of two stages: schema layer construction and data layer construction. Through summarizing the core knowledge and key terms in Chinese grottoes, the schema layer adopted a "top-down" approach to defining abstract classes and attributes and designed an ontology model which is called ChgOnto. The ChgOnto contains the hierarchical, semantic, and spatial relationships between classes, which provides a logical framework and pattern constraint for knowledge representation in the data layer. The data layer used a "bottom-up" approach to extract instances of the specific knowledge required by the schema layer from highly specialized textual data such as monographs, literature resources, etc. For example, the schema layer defines "grottoes" as a conceptual term, and the data layer needs to extract concrete instances such as Longmen Grottoes and Yungang Grottoes from the data sources. Instances are expressed as nodes and those instances are connected by relationships to form a triple as < Entity, relationship, Entity > and < Entity, property, Property value > . Subsequently, based on the knowledge structure provided by the schema layer, the triples are associated to form a semantic web. The overall process of the method is shown in Fig. 1.

Fig. 1

The process of knowledge graph representation method for 3D semantic modeling

Schema layer construction

The schema layer construction used the Ontology Development 101 method [78] for ontology design, which serves as the conceptual data model and logical foundation for knowledge graph representation. The method consists of the following steps: (1) determine the domain and scope; (2) reuse existing ontologies; (3) enumerate important terms; (4) define the classes and the class hierarchy; (5) define the properties of classes and the relation between classes; (6) define the facets of the properties; (7) create instances. The steps will be discussed in the following subparts.

Ontology design

Step1: Determine the domain and scope

Defining the domain and scope of an ontology is crucial for targeting the Chinese grottoes and clarifying the purpose and knowledge content of the ontology design. In this study, the ChgOnto was developed for the knowledge representation of Chinese grottoes, thus providing corresponding knowledge for supporting the 3D semantic modeling process. Point clouds are the primary data source in the digital preservation of cultural heritage, but they pose several challenges for real-world applications, such as being massive, non-structured, and lacking neighboring and semantic information. To gain a high-level understanding of the 3D scene, it is necessary to extract meaningful geometric units with explicit semantics and functions from the point clouds and convert the unordered and discrete point clouds into a combination of geometric primitives with topological relationships.

A typical workflow from point cloud data collection, point cloud semantic segmentation and 3D reconstruction to application system development is illustrated in Fig. 2. In the semantic 3D modeling process of Chinese grottoes, participants need to clarify the following questions:

• For data collectors, what is the area of data collection and what is the point of interest to be collected?

• For data managers, what is the spatial composition of the cave and niche in the grotto scene, and which entities should be segmented?

• For 3D modelers, how should the topological relations between entities be constructed? What are the attributes of the entities, and what are their corresponding attribute values?

• For software developers, how should the data model of the database be designed?

Fig. 2

The typical workflow from point cloud data acquisition to applications

The above questions indicate that the semantic 3D modeling of Chinese grottoes involves the cooperation of participants from different disciplinary backgrounds. These participants usually do not come from the cultural heritage field. The knowledge graphs based on ChgOnto for semantic 3D modeling can provide these participants with unified knowledge and attribute information of grottoes. This can reduce the difficulty of individuals without cultural heritage expertise to complete their work independently and alleviate misunderstanding of the grottoes among people with different backgrounds.

Step2: Reuse existing ontologies

The purpose of reusing existing ontologies is to improve the reusability and maintainability of an ontology, provide opportunities for integration with other ontologies, and reduce ontology design costs. Ontology reuse can typically be achieved through reference and inheritance.

In the ChgOnto model, the classes related to time, location, and data types were reused through referencing them from the CRM standard ontology and its extension. For example, the E2 Temporal Entity, E53 Place, and E59 Primitive Value were referenced from the CRM core. Additionally, CRM_dig provided concepts and properties for describing digital data and digital devices, such as D1 Digital Object and D8 Digital Device.

GeoSPARQL is a standard in the Semantic Web of the OGC for representing and querying geospatial-related data [79]. In GeoSPARQL, the geo: SpatialObject is defined as "The class Spatial Object represents everything that can have a spatial representation". The geo: SpatialObject class in GeoSPARQL was adopted to represent all objects of Chinese grottoes and their surrounding environment. By inheriting the class geo: SpatialObject, these objects are treated as spatial objects within ChgOnto. Furthermore, the Egenhofer Topological Relations from GeoSPARQL were reused to describe the topological relationships between different objects in grottoes. Table 1 provides a list of the main classes reused in ChgOnto and their respective sources.

Table 1 The list of the main classes reused in ChgOnto and their respective sources

Step3: Enumerate important terms

The purpose of enumerating important terms is to capture the basic vocabulary used in the field and ensure the ontology's comprehensiveness within its domain coverage. By identifying the most important terms, ontology developers can establish a hierarchical structure of classes and properties that reflect the knowledge structure of the domain, enabling the construction of more complex models. Conceptual terms, which represent the highest level of abstraction in an ontology, describe the knowledge classification within the ontology. These terms have semantic relationships that form a knowledge graph structure through linkage.

In this paper, the representation of Chinese Grottoes was divided into several categories (i.e., classes defined or referenced in this paper are indicated in italics, such as the class Chinese Grottoes): Spatial Object, Digital object, Digital Device Objects, and Temporal Object. The Spatial Object includes Chinese grottoes, grotto groups, cave and niche, environments, and cave and niche Components. The Chinese grottoes refers to the collective name of multiple grottoes in a region, including multiple grotto groups. A grotto group consists of a series of cave and niche. For instance, the Longmen Grottoes, as an instance of Chinese grottoes, are composed of four grotto groups: Xishan Grottoes, Dongshan Grottoes, Xiangshan Temple, and Baiyuan. The basic units that make up a group are called “a cave” or “a niche”. The environments class is composed of natural entity and artificial entity surrounding the cave and niche, coexisting with the grottoes. The place object describes the geographical range of the grottoes, while the site object refers to the specific address of the grotto groups. The Digital object is the basic data source for 3D reconstruction and is obtained through Digital Device Objects from the digitalization of the grottoes and their surrounding environment. The Temporal Object represents time-dependent entities, such as periods (Qing Dynasty, Tang Dynasty and Ming Dynasty, etc.) and time span (e.g., 1900–1950). Figure 3 presents the conceptual framework composed of important terms, providing the knowledge structure for the knowledge graph representation. More specific key terms, such as spatial structure and elements of a cave and niche, will be discussed in detail below.

Fig. 3

Conceptual framework of ontology consisting of important terms

Step4: Define the classes and the class hierarchy

The purpose of defining classes and class hierarchies is to classify and organize concepts within the ontology. In this research, the main objects of focus are caves and niches. From a spatial composition perspective, this step involves defining the structural composition of caves and niches, as well as identifying the elements contained within each structure.

There are various types of grottoes in China. For instance, grottoes in the Central Plains region are primarily caves formed by natural cliffs or artificially cut mountains, creating roughly vertical cliffs that are then carved into caves with Buddha statues. Representative examples of such grottoes include the Yungang Grottoes in Datong, Shanxi Province, and the Longmen Grottoes in Luoyang, Henan Province. On the other hand, grottoes in southern China mainly consist of carved niches on cliff walls, where Buddha statues are placed. Notable examples of such niches include the Dazu Rock Carvings in Chongqing and the Qianfo Temple in Guangyuan, Sichuan.

From the geometric composition of the cave and niche, the cave has a large depth (like a room, usually can accommodate people), which can be divided into front cliff wall, left cliff wall, right cliff wall, and cave roof (Fig. 4a), and some caves have pillars with sculptures on them. The niche (Fig. 4b) is shallow in depth, with only one front cliff wall and no cave roof, while there is a niche eave (like the eaves of a modern building) at the junction of the cliff and the mountain. In addition, a cave or niche may contain a smaller one (Fig. 4b). Specifically, “cave and niche (窟龛 in Chinese)” is a special term in Chinese grottoes, which represents the basic units that make up a grottoes group. Although “Cave” and “Niche” may have different geometric shapes as indicated in Fig. 4, they own the same semantic properties including name, number, height, width and depth as indicated in Table 3. Therefore, we do not divide “cave and niche” into two separate classes in this paper.

Fig. 4

Schematic representation of the spatial structure of the cave niche

Grottoes' artistic creations are primarily reflected on the cliff wall. Consequently, the various types of artistic content found on the cliff wall are abstracted as elements. This abstraction allows each element to possess clear semantics and properties within grotto art. The elements within caves and niches are categorized as primary Buddha statues, other Buddha statues, decorations, pedestals, and inscriptions. The primary Buddha statue refers to the largest statue within the cave or niche or a series of statues of the same size. On the other hand, the other Buddha statues encompass disciples, guards, or small decorative Buddhas associated with the main Buddhas. Figure 5 illustrates the elements found within a specific niche. Additionally, environmental entities and digital data entities are further subdivided. The main classes and class hierarchy that have been defined are presented in Table 2.

Fig. 5

the main elements of the niche

Table 2 The list of main classes and the class hierarchy

Step5: Define the properties of classes and the relation between classes

The properties of classes play a vital role in defining the conceptual structure within a class. A property consists of a property name, a property domain, and a property range. The property domain specifies the class to which the property applies, while the property range indicates the type of value that the property can have. Table 3 presents the definitions of properties for the Cave and Niche class and the Primary Buddha Statue class. Although Cave and Niche may have different geometric shapes as explained in Step 4, they own the same semantic properties including name, number, height, width and depth.

Table 3 The properties of class Cave and Niche and class Primary Buddha statue

The relationships between classes encompass both semantic relations and spatial relations. Semantic relations involve verbs such as “located in”, “found in”, “digitized”, “composed of”, and others, which establish connections between classes based on their meanings. Spatial relations, on the other hand, reflect the spatial connections that arise from the interaction of different entities in a grotto’s scene. In the context of 3D reconstruction, understanding the spatial relations between objects is crucial for modelers to accurately restore the scene. The relationships between classes can be expressed using triples such as < Entity, semantic relation, Entity > and < Entity, spatial relation, Entity > .

The ChgOnto primarily defined two types of spatial relationships, i.e., topological relationships and relative positional relationships. The topological relationship captures the proximity and association between entities and is adapted from the Egenhofer Topological Relation in GeoSPARQL. It consists of eight distinct types of topological relations (Fig. 6). The relative positional relationship describes the spatial positioning of entities relative to each other in space, as depicted in Fig. 7. For a clearer understanding, we show the spatial relationship of partly entities in the HuaYanSanSheng niche through a schematic diagram (Fig. 8).

Fig. 6

The description of Egenhofer topological relations

Fig. 7

The description of relative position relation

Fig. 8

Schematic drawing of the topological relation and relative position relationships between entities (partly) in the HuaYanSanSheng Niche

Step6: Define the facets of the properties

Facts refer to the metric information related to the properties of a class. For instance, the measurement of the height and shoulder width of a Buddha statue is represented by a numerical value in centimeters. The properties describing the posture and clothing characteristics of a Buddha statue are represented by strings. In the context of the CRM, the E59 Primitive Value is used to represent facts, which can include E60 Number, E61 Time Primitive, and E62 String.

Step7 Create instances

The last step in ontology design is to create an instance of the class. To define an instance, first need to determine the class to which the instance belongs, and then match the actual property values for the properties of the class. Taking the scene in Fig. 6 as an example for the class “Cave and Niche”, the conceptual graphs (Fig. 9) of the scene instance were constructed. The conceptual graphs mainly show the structural composition of the HuaYanSanSheng Niche. The niche eaves include the two rays as the Decoration. The cliff wall consists of the class Primary Buddha statues as three entities (Wenshu Buddha, PuXian Buddha, and LuSheNa Buddha), 81 small round niches as the class of Other Buddha Statues, and two lotuses as the class of Dedestal. The cliff wall footing includes the Inscription.

Fig. 9

The conceptual graphs of the HuaYanSanSheng Niche as an instance

The conceptual data model of the schema layer

Based on the ontology design, a domain ontology for 3D semantic modeling of Chinese grottoes can be constructed. This ontology model serves as a data conceptual model, which is essential for constraining the knowledge graph representation within the data layer. Figure 10 provides a clear visualization of the basic structure of the schema layer, including the classes and the class hierarchies. Different colors are employed to distinguish the classes that have been reused from other ontologies, aiding in visualizing the distinct origins of the classes. The properties of the Cave and Niche Component, and the Cave and Niche Element, are also provided in this model.

Fig. 10

The conceptual data model of the schema layer

Data layer construction

The schema layer provides a standardized framework for representing knowledge graphs. The data layer instantiates the abstract classes of the schema layer through knowledge extraction and fills in the attributes of the classes. The instantiated knowledge exists as a set of nodes and relations. These nodes consist of instances of conceptual classes, while the relations connect the nodes, forming a knowledge web. The construction of the data layer involves knowledge extraction, entity alignment, and knowledge storage.

Knowledge extraction

Knowledge extraction is the process of extracting knowledge units such as entities, relations, and properties from semi-structured or unstructured text data [80]. In natural language processing, knowledge extraction is also known as named entity recognition (NER), which is used to identify named entities, such as countries, institutions, places, and people, from text data sets. The text data of this study came from professional books and academic papers related to Chinese grottoes, which are more reliable than open data on the Internet. In this research, NER is mainly utilized to extract caves and niches, buddha statues, and other entities from text. For example, “HuaYanSanSheng Niche” is recognized as an instance of Cave and Niche, while “LuSheNa Buddha” is identified as an instance of the Primary Buddha Statue.

After the NER process, a discrete set of named entities is obtained. To form a semantic web as a knowledge structure, it is necessary to extract relations between entities and link entities through relations to triples. For instance, < HuaYanSanSheng Niche, include, LuShaNa Buddha > , < HuaYanSanSheng Niche, include, WenShu Buddha > , and < HuaYanSanSheng Niche, include, PuXian Buddha > . Properties extraction involves obtaining attribute names and attribute values of an instance. For example, < HuaYanSanSheng Niche, height, 820 cm > and < HuaYanSanSheng Niche, width, 1550 cm > are examples of extracted properties.

Entity alignment

Entity alignment involves associating data after knowledge extraction through entity alignment and property fusion. The key to entity alignment lies in entity disambiguation and property disambiguation. Entity disambiguation aims to determine whether entities with different names in various data sources refer to the same real-world object, thus eliminating the phenomenon of “different words describing one object”. For example, “HuaYanSanSheng”, “HuaYanSanSheng Niche”, “HuaYanSanSheng Rock Carving”, and "HuaYanSanSheng Image" all represent the same entity. Therefore, semantic consistency is necessary, and these entities should be named as “HuaYanSanSheng Niche”. Moreover, there are cases where named entities have the same name but represent different real entities, known as the phenomenon of “one word describing different objects”. For instance, both the HuaYanSanSheng Niche and the YuanJue Cave contain the LuSheNa Buddha statue. To distinguish these entities, uniquely identified properties are added. For example, the unique identifier of the LuSheNa Buddha in the HuaYanSanSheng Niche is set as DZ-BD-04-F-MB-01, while the unique identifier of the same statue in the YuanJue Cave is set as DZ-BD-08-R-MB-03.

Similar to entity disambiguation, attribute disambiguation addresses the issue of multiple attribute names referring to the same attribute or the absence of uniform units of measure for attributes. For example, when describing the width of a niche, there are synonyms such as “niche width”, “width”, and “image width”, but they all represent the same attribute. Additionally, it is necessary to standardize the measurement of attribute values. For instance, 820 cm and 8.2 m represent the same attribute value.

Knowledge storage

Knowledge storage involves transforming massive real-world information into structured data that conforms to computer processing models. To achieve this goal, structured description languages such as XML, RDF, RDFS, and OWL are used to represent knowledge. OWL language is particularly useful for representing knowledge in complex scenarios and adding additional predefined vocabulary to describe resources [81]. Graph databases have become the mainstream method for knowledge storage. Based on graph models, data is represented in the form of nodes and edges, clearly illustrating the dependencies between data nodes. Compared to traditional relational databases, graph databases have a complete graph query language and various graph mining algorithms, providing advantages in the speed of deep association queries. In this paper, Neo4j [82], a popular graph database management system, was utilized to store and maintain knowledge triples. In this study, we first utilized CSV files to store the extracted entities and properties. The Neo4j's LOAD CSV command was then employed to transfer the data from the CSV file into the graph database. This command treats each row in the CSV file as a node and each column as a property associated with that node. We used the MERGE clause to create or match the nodes and the relationships between nodes based on the entities and relationships found in the extracted knowledge triples. Finally, all the triples were imported and mapped, and further stored in the database to ensure the persistence of the changes in the Neo4j database.

Experiment and result

Data collection

Dazu Rock Carvings are a remarkable collection of Buddhist sculptures and rock-cut caves located in Dazu County, Chongqing Municipality, China. Dating back to the 9^th to the thirteenth centuries, these carvings have been well-known for their exceptional artistry, intricate details, and diverse themes. Dazu Rock Carvings was recognized as a United Nations Educational, Scientific and Cultural Organization (UNESCO) World Cultural Heritage site in 1999, representing significant cultural and historical treasure in China. Since Dazu Rock Carvings consists of a large number of grotto groups, caves and niches, it is a typical representative among Chinese grottoes. In addition, the records and description of Dazu Rock Carvings in professional publications and books are abundant, which can be used as effective data sources for analysis. As such, we chose the Dazu Rock Carvings as the experimental area to conduct case study in this work. The Dazu rock carvings cover a vast area, which comprises five grotto groups: Beishan, Baodingshan, Shimenshan, Nanshan and Shizhuanshan (as shown in Fig. 11a). The yellow line in Fig. 11b represents the Baodingshan grotto group, which consists of numerous caves and niches. To ensure the quality of the knowledge, the experimental data was sourced from two Chinese monographs, namely “Dazu Rock Carvings (大足石刻)” (ISBN 978-7-5647-8384-6) and “Research and Appreciation of Dazu Rock Carvings (大足石刻研究与欣赏)” (ISBN 978-7-229-06584-3), as well as Chinese academic papers. Text data was extracted from these sources as experimental data.

Fig. 11

The geographical location of the experiment area

Ontology model

This research employed the OWL language and the Protégé 5.6.1 ontology editor to formalize domain knowledge. Protégé is a widely used ontology construction tool developed by Stanford University that enables users to create and edit ontologies [83]. It supports multiple ontology representation languages, including OWL, RDF, RDFS, and OWL2. Figure 12 shows a visual representation of developing the ChgOnto in the Protégé platform. The ChgOnto defined 44 classes, 48 relationships, and 16 attributes. Among them, 11 classes are reused in CRM Core, starting with CRM_E; 3 classes in CRM_dig are reference, starting with CRM_dig_D; one GeoSPARQL class and 8 types of topology relationships are inheritance. The main entities and relations in Protégé software are shown in Fig. 13.

Fig. 12

The screenshot of the ChgOnto in the Protégé environment

Fig. 13

The main entities, properties, and relations defined in the ChgOnto by Protégé

Knowledge graph construction

The NER is implemented using the BERT-BiLSTM-CRF combination model, which can handle long-term dependencies in text and capture contextual information [84]. Relations and properties extraction are achieved through the method of rule matching. The knowledge extraction process in this paper relied on two open-source natural language processing toolkits, OpenNRE [85] and CRF +  +  [86]. The knowledge triples were structured based on the schema layer and stored in the Neo4j graph database for management and storage.

The knowledge graph of Baodingshan grotto group contains 19 classes, including Conceptual node, Place, Site, Periods, Chinese grottoes, Grotto groups, Cave and niche, Cave roof, Niche eave, Cliff wall footing, Left cliff wall, Right cliff wall, Front cliff wall, Pillar, Primary buddha statue, Other buddha statue, Inscription, Pedestal and Decoration. As shown in Fig. 14a, each class is composed of a different number of entities. For example, the class Periods includes 7 entities, such as Tang Dynasty, Five Dynasties and Song Dynasty etc. The class Chinese grottoes includes 5 entities, such as Dazu rock carvings, Longmen grottoes and Anyue grottoes etc. The class Pedestal includes 32 entities, such as two lotus pedestal, cloudy pedestal and quadrate pedestal etc. The class Decoration includes 32 entities, such as Bodhi trees, auspicious clouds and lotus leaves etc.

Fig. 14

The number of entities and properties in the proposed knowledge graph

A total of 42 entities belonging to the Class Cave and niche were extracted, such as Peacock Ming King cave, Shuiyue Guanyin niche, and PiLu cave etc. Each entity was described by 5 properties of name, number, height, width, and depth. Figure 14b shows the number of extracts for each property in the class Cave and niche. Take the entity of Peacock Ming King cave for example. The property value of name in this entity is “Peacock Ming King cave”, the property value of number is “No.155”, the property value of height is 347 cm, the property value of width is 290 cm and the property value of depth is 603 cm. The number of properties extracted for height, width and depth is less than 42 due to the lack of relevant descriptions in the data source.

A total of 122 entities belonging to the Class Primary buddha statue were extracted. As illustrated in Fig. 14c, each entity was described by 11 properties, including name with 122 distinct values (e.g., WenShu buddha, PuXian buddha), statue height with 39 distinct values (e.g., 191 cm), shoulder width (e.g., 40 cm), body posture (e.g., sitting, playing and standing), hairstyle (e.g., high square corolla, crown, and nautilus and bun hair), facial expression (e.g., handsome, dignified and solemn), left hand posture (e.g., placed on the chest and lotus in hand), right hand posture (e.g., holding a bow and arrow, and holding a shield), dress (e.g., U-collar coat and monk's coat), accessories (e.g., necklace), preservation state (e.g., damaged and intact).

The proposed knowledge graph visualized using Neo4j is shown in Fig. 15, which included a total of 19 classes, 483 entities and 492 relationships. For example, the entity of Chinese grotto includes the entity of grotto groups, thus the “contains” relationship exists between Dazu Rock Carvings and Baodingshan (see Fig. 11) and the entity of grotto groups consists of the entity of cave and niche. In the case where the environmental entities and the relationships between the structural compositions and elements of the cave and niche were lacking in the textual description, we manually supplemented them with common sense or by referring to relevant images. As indicated in Fig. 8, the cliff wall is covered by several small round niches. Such commonly acknowledged information is usually abbreviatory in the textual data sources and cannot be automatically extracted by the aforementioned methods. As such, we manually added the “cover” and “covered by” relationships between cliff wall and small round niche in order to improve the integrity of the knowledge graph.

Fig. 15

Visual interface of knowledge graph in Neo4j

Application of knowledge graph

Knowledge retrieval

To ensure unified knowledge guidance in different stages of semantic modeling for the Baodingshan Grottoes group, the Cypher language is used to search the constructed knowledge graph. The retrieval conditions are as follows:

“match (n:Grotto_Group{name:'Baodingshan'}) - [r*0..]-> (result) return result”

As shown in Area 1 in Fig. 16, the searches results display the components and elements of the caves and niches contained in the Baodingshan grotto group. Detailed node properties can be queried by node names. The Area 2 and Area 3 in Fig. 16 show the property value of the entity belonging to the Class Primary Buddha Statue and Cave and Niche. These entities, relationships and property values provide meaningful information and knowledge for different participants in the process of semantic 3D modeling.

Fig. 16

Search results of the knowledge graph of the Baodingshan Grotto group (Area1 is a capture of knowledge graph showing entity names and relationships. Area2 and Area3 show examples of property values of the entities)

The Knowledge graph also enables attribute retrieval based on entity name. For instance, with regard to querying the attribute value of the Primary Buddha Statue named "WenShu Buddha", the following search condition can be used:

“match (n:Primary_Buddha_Statue{Name:'WenShu Buddha'}) return n”

The search results returned by the database in JSON format are as follows:

Topological relation retrieval

In 3D semantic modeling, restoring the correct topological relationship between different geometric units is crucial to ensure the model's quality. The topological relationships among components and elements provided in the knowledge graph can serve as modeling references for 3D modelers. Figure 17 illustrates the topological relationships among different components and elements of the HuaYanSanSheng Niche.

Fig. 17

The topological relationship among the structural components and elements of the HuaYanSanSheng Niche

In the actual application process, to retrieve entities that have a "meet" relation with "LuSheNa Buddha", the following retrieval conditions can be used:

“match (n:Primary_Buddha_Statue{Name:'LuSheNa Buddha'}) -[r:has_topological_relation{type:'meet'}]-> (result) return result”

The retrieval results are displayed in Fig. 18. A total of 4 nodes were queried, indicating that the entities that have a "meet" relation with "LuSheNa Buddha" include three components of "niche eaves", "front of the cliff", and "the cliff wall footing", and one pedestal element, namely "two lotus pedestal".

Fig. 18

The retrieval results of the entities that have a “meet” relationship with “LuSheNa Buddha”

Discussions

Dynamic knowledge graph

The dynamic knowledge graph refers to the revision of inaccurate or outdated attribute information in the knowledge graph over time [87]. The process of updating knowledge within a knowledge graph can be logically divided into two categories: schema layer updates and data layer updates. The schema layer updates involve the introduction of new concepts into the schema layer whenever fresh data is obtained. The data layer updates, on the other hand, primarily focus on the addition or revision of entities, relations and attribute values. During data layer updates, it is crucial to ensure the reliability and consistency of the data source. For instance, when analyzing the HuaYanSanSheng Niche in the Dazu Rock Carvings, the size of the Buddha statue provided by professional texts may lack precision due to the timeliness of the information and the use of different measurement methods. In such circumstances, attribute values can be refined by measuring the Euclidean distance in point cloud data (see Fig. 19). Table 4 provides a comparison of attribute measures in professional books with values measured in point cloud data.

Fig. 19

The property values in the knowledge graph are measured in the point cloud for knowledge updating

Table 4 The property values of the entities from different data sources

Extensibility

Grottoes, as immovable cultural relics, have specific geographical locations and embody both natural and cultural characteristics. The domain ontology proposed in this research has utilized many ontology classes already defined in CIDOC CRM and GeoSPARQL. The geo: SpatialObject class in GeoSPARQL, a generic spatial object description class, plays a significant role in this. Consequently, the ChgOnto proposed here has high expandability. For instance, it can incorporate domain ontologies and knowledge graphs related to rock and soil types (e.g., sandstone, limestone, and conglomerate), climate conditions (e.g., aridity, humidity, and high temperatures) and disease characteristics (e.g., weathering, cracking, and waterlogging). This would enhance its utility in natural disaster risk assessment and rock mass stability analysis of immovable cultural relics.

Analysis of limitations

The knowledge graph construction approach based on the "schema layer-data layer" model can be negatively affected by the incompleteness or inadequacy of abstraction during the ontology design process. Such issues are likely to compromise the usefulness of knowledge graphs in practical applications. For example, due to the uniqueness of cultural heritage and the limitations of cognition, the ChgOnto model designed in this study may not be directly applied for all Chinese grottoes. In this case, the ChgOnto model can be used as a basis on which further extension and adjustment can be made. Therefore, more extensive research is needed to revise and extend the ontology model. Also, gathering user feedbacks and suggestions and adjusting the ontology model to meet the requirements of practical applications, is a critical step for ensuring ontology usability.

Entity disambiguation and attribute disambiguation still relies on manual processing and verification. Although this process is labor-intensive and time-consuming, it yields high-quality and highly reliable domain knowledge graphs, which provides robust support for practical applications. As mentioned in the disambiguation issue in Sect. “Entity alignment”, there are a large number of entity names and units of attribute values need to be unified. Striking a balance between quality and efficiency remains crucial in the construction of domain knowledge graphs. Therefore, it is necessary to systematically establish a glossary or dictionary of Chinese grottoes and reduce manual processing in the process of entity disambiguation and attribute disambiguation.

Although the creation of domain-specific knowledge graphs based on ontology construction ensures that the structure and content of knowledge are more in line with users’ requirements, it falls short in terms of efficiency when compared to automatic knowledge graph construction methods. The automatic construction method of knowledge graph of Chinese grottoes based on ontology is likely to be developed through defining a set of semantic rules. In addition, as mentioned in Sect. "Knowledge Graph Construction", due to the lack of description of topological relations of various geometric elements in the textual data, those relations are constructed manually. It is promising that the topological relations of the geometric elements extracted from point cloud data can be identified by adopting advanced image processing technologies in order to alleviate labor cost.

Conclusions

This research proposes a knowledge graph representation method for the 3D semantic modeling of Chinese grottoes. It establishes a standardized knowledge-sharing mechanism and solves the problem of lacking knowledge guidance in the modeling process, which results in inconsistent understanding of grotto knowledge among personnel with multidisciplinary backgrounds. In this work, the ontology model named ChgOnto is first designed, which is used as the schema layer defining the structure of building the domain-specific knowledge graph of Dazu rock carvings. This knowledge graph contains 19 classes, 483 entities and 492 relationships. Among them, 5 properties are defined in the cave and niche class, of which 189 distinct property values are extracted. The primary buddha statues defines 11 properties and extracts 593 property values. The entity and attribute information can be requested to support the semantic 3D modeling of Chinese grottoes through conducting the knowledge query operation on the knowledge graph. It reveals that the proposed method has strong scalability and replicability, enabling provide the knowledge sharing in the process of 3D semantic modeling of Chinese grottoes. In the future, the proposed knowledge graph representation method can be reused, adjusted and extended to accommodate other types of cultural heritage.