Design and research of digital twin platform for handicraft intangible cultural heritage -Yangxin Cloth Paste

Abstract

In the context of the 5G era, the rapid development of digital technology and its integration with intangible cultural heritage (ICH) can facilitate the dynamic transmission of ICH.The research purposes to construct a virtual experience platform for handmade ICH using the handmade ICH of East Hubei Province in China—Yangxin Cloth Paste as a case study through Digital Twin technology. It explores the application of digital twin technology in the field of handmade ICH transmission and aids the dynamic transmission of handmade ICH. Firstly, the research collected tangible and procedural data of the Yangxin Cloth Paste. By using photogrammetric techniques, a model of the handicraft was built and an effective digital twin conversion procedure was designed. Next, the research set up a framework for a digital twin platform for handmade ICH, designing systems for the production, display, and transaction of ICH handicrafts. Lastly, its effectiveness was validated by user satisfaction evaluation guiding subsequent optimization direction. The platform innovatively uses digital twin technology to help users visualize handicraft ICH. Through the combination of digital twin technology and virtual reality technology, it creates a realistic virtual reality experience of ICH of handicraft, stimulates users' interest in exploring ICH of handicraft, and contributes to the process protection, dissemination and development of handicraft ICH.

Introduction

Research background

Intangible cultural heritage (ICH), made up of all immaterial manifestations of culture, represents the variety of living heritage of humanity as well as the most important vehicle of cultural diversity [1]. Among them, the craft category of ICH represents a living artisan culture, embodying the labor and wisdom accumulated by the nation over a long process of social evolution. One of the elements that acts as a carrier in the transfer of cultural heritage from generation to generation is the tradition of handicrafts [2]. It is also the most tangible expression of ICH [3]. Yangxin Cloth Paste, an existing folk craft in the Yangxin area of Huangshi, Hubei, China, covers clothing, daily necessities, and temple decorations [4]. As a national-level ICH of folk crafts in the Central Plains region of China, it plays a representative role.

Handicraft ICH is characterized by the use of materials with local characteristics, with unique process methods and procedures to create beautiful, elegant, easy to use handicrafts, in the process of historical evolution perfectly fit the needs of local people. Latitude, terrain, climate, farming conditions, water conservancy and other elements of civilization, different geographical locations have created different civilizations, and different lifestyles have also created great differences in handicrafts. The differences in the external performance of handicrafts such as material, function, pattern and shape actually represent the internal differences in the production mode, creation thinking and group cognition of handicrafts users.

Digital twins represent a virtual reenactment of an object or system throughout its entire lifecycle, updated based on real-time data and utilizing simulation, machine learning, and inference to assist in decision-making [5]. However, the definition of the heritage digital twin (HDT) and the focus of research are different.Cultural heritage entities can be made up of tangible entities and intangible entities, and pure intangible cultural heritatives have no tangible ingredients, so the digital twin of the cultural heritage is defined as:The Heritage Digital Twin (HDT) of a Heritage Asset (HA) is the digital representation of the complex of knowledge about that heritage asset, organised according to a specific ontology, called the HDT ontology [6].

The digital twin of a heritage (HDT) includes the digital representation of a Heritage Asset,such as the 3d model of the heritage, the virtual reality (VR), the augmented reality (AR) model, other types of visual digital artifacts related to the heritage and the digital twin of all the details of forming cultural heritage. HDT also includes a Heritage Story, such as a narrative, a historical source, a popular belonging, and the content of common creation.HDT also includes a Heritage Document composed of data generated by the corresponding production activities,such as the study report, the number of images, video, and the collection of artists and architects.A Heritage Asset is an external manifestation of the heritage; A Heritage Story links tangible legacy entities and intangible components to the reference community, giving regional historical and cultural values; A Heritage Document makes the cultural heritage be preserved in various forms.A Heritage Asset,a Heritage Story and a Heritage Document together constitute the complex of knowledge about that heritage asset.They convert digital technology into heritage digital twin(HDT) [6].

The use of digital twin technology to virtually reproduce the entire production process of intangible handicrafts can not only digitally protect the handicrafts, but also explore the unique regional cultural characteristics and lifestyle details in their history through the observation and analysis of materials, processes and processes. The history and culture behind it is an invisible treasure. Digital twin technology can observe and analyze handicraft ICH in the virtual world without affecting the actual production and inheritance of handicraft.

The focus of this study is to introduce digital twin technology into the craft category of ICH, taking Yangxin Cloth Paste as an example. By building a digital twin platform, the process of Yangxin Cloth Paste will be digitally and virtually presented from aspects such as raw materials, pattern drawing, cutting, piecing, sewing, etc [5]. Digital Twin is committed to the high reality reproduction of virtual experience. Users can interact with artisans who make real heart-nourishing cloth products through virtual items,and experience and learn to inherit ICH in virtual reality. Users can explore new paths for the inheritance, protection, dissemination, rejuvenation, and innovation of ICH in crafts through digital twins.

Document review and theoretical basis

Digital protection of cultural heritage

The protection of cultural heritage based on digital technology can be divided into two types according to the classification of the main body of cultural heritage: demiurgic cultural heritage and architectural cultural heritage. With the breakthrough of virtual digital technology, related research is also thriving. The main studies fall into these two categories:

First, 3D scanning and digital modeling techniques are used to simulate the restoration and preservation of art artifacts and applied to physical artifacts. For example, Tong Y. et al. [7] used 3D scanning and high-fidelity 3D modeling to digitally present the colored Bodhidharma statue in Shandong's Lingyan Temple, performing virtual restoration of color and texture on the digital model. Jo Y.H. et al. [8] used three-dimensional virtual modeling to establish a digital model of a broken stone Buddha statue, virtually restored it, then used 3D printing for physical repairs, achieving non-contact restoration of the relic.

Second, through images, film and television, text materials, physical photography and other materials, the digital reconstruction of historical buildings and cultural relics, and preview planning protection measures in advance.For example, Ferwati M.S. et al. [9] built the Zubarah model based on the Qatar Museum Authority database, completing a virtual reconstruction of Qatar's historic city of Zubarah from existing ruins, establishing a reference object for real reconstruction. Ovidia Soto-Martin et al. [10] digitally rebuilt St. Augustine Church, utilizing VR technology to document the deteriorating cultural heritage, virtually integrating the murals to create an interactive immersive virtual reality experience. Li, D. et al. [11] put forward a digitalization method for cultural heritage sites which rationally integrates and utilized multiform surveying measurements, and applied it to two projects, namely the Digital Mogao Grottos and the Chi Lin Nunnery reconstruction.

Digital inheritance and protection of ICH

To strengthen the inheritance and dissemination of ICH, it is necessary to integrate various modern technologies. At present, the research involving emerging digital technologies empowering ICH inheritance and protection mainly falls into three categories: digitization of ICH content, digitization of ICH dissemination channels, and innovative digital dynamic inheritance of ICH.

The digitization of ICH content mainly involves recording and archiving existing ICH historical artifacts, inheritors, detailed information, etc., through digital media technologies such as text, images, videos, and 3D technology. For instance, Erturk, N. [12] contributed the digitalization practices to the safeguarding of the ICH, major issues and challenges related to the digital preservation in museum storage, and taking the problems encountered by Smithsonian’s National Museum of the American Indian and Musée du quai Branly in the protection of ICH as an example, he put forward the measures of sustainable collection management of digital ICH. Zhanna, R. [13] made a visual reconstruction based on the choreographed texts of famous folk dance choreographers. That is, the digitalization of folk dance movement patterns.Wijesundara, C.et al. [14] presented a model called Cultural Heritage in Digital Environment (CHDE) for organizing various digital CHI organized as a digital archive.

The digitization of ICH dissemination channels primarily uses new media platforms, virtual museums, etc., to achieve non-physical display and interaction of ICH. For example, Hammou, I.et al. [15] verified the close relationship between social media communication and the promotion of the ICH in Morocco, and promoted the ICH of the city of Marrakech through the development of Moroccan handicrafts. By making full use of its superior technology “Street View”, it created a yearround online museum [16]. Apollonio, F. I. et al. [17] introduced an automated combination of acquisition, based on mobile equipment and visualization, based on Real-Time Rendering, and also tested in some digital heritage museum cases. Marra, A. et al. [18] reported the results of an approach to the maintenance and preservation of art objects housed in a museum complex based on a comprehensive digital path towards a Historical Digital Twin (HDT). Yang, C. et al. [19] used virtual reality technique to reconstruct and imitate cultural space and crowd cultural activity in warring States time,reconstructed characters, finery and architecture by 3d modeling technique. Selim, G. et al. [20] discussed a novel technological approach using virtual heritage technology to reflect Umm Qais heritage's intertwining and interdependent nature and developed the first model of a “virtual living museum”.

Innovative digital dynamic inheritance of ICH mainly focuses on innovation aligned with modern consumption patterns and communication between craftsmen and users. For example, Partarakis, N. et al. [21] defined cultural crafts(HCs), embrace craftsmanship as a form of ICH, and leverage the need for display to transform traditional crafts into engaging digital experiences.Zhang, Y. et al. [22] systematically introduced the digital technology in the scenic area planning, project initiation, tourism management model, and other aspects to create a culture-first, three-dimensional virtual reality scenic area of Celadon Cultural Industrial Park.

Practice of digital twinning of cultural heritage

The research and application of the existing digital twin technology in the direction of cultural heritage has produced the definition of heritage digital twin—Heritage Digital Twin (HDT). HDT is a composite of corresponding (digital) information about heritage entities, consisting of its own information (such as twin identifiers) and other related digital information [23].

Relevant research trials include:Puxiang Wang and others used Quanzhou's ships as an example, building a sensor network on the relics, using digital twin technology for vulnerability analysis and predictive reinforcement, achieving preventive protection of movable wooden artifacts. Jouan, P., & Hallot, P. [24] suggested the application of Digital Twin (DT) principles to support site managers in the preventive conservation of their assets. They used digital twin technology to digitally protect and simulate a medieval castle in the south of France, analyzing and simulating data collected by on-site sensors to predict threats to site integrity in a DT environment and corresponding prevention plans.Vuoto, A. et al. [25] proposed a prototype of the digital twin paradigm for the preservation of heritage buildings' structural integrity.

Research and exploration of digital twin technology in the field of ICH

Emerging digital technology can effectively enable the protection and dissemination of various cultural heritage through various ways. With the further development of digital technology, digital twin technology has been applied to the virtual reconstruction and protection of cultural heritage, and the practice and methods are becoming more and more progressive. However,in the field of ICH protection, digital twin technology also has great application space.The digitization of the ICH requires extension of the standard ontology for digitalization of cultural heritage and the ways of interaction with the data [26]. For instance, Wei J.T. et al. [27], based on digital twin technology, proposed a double-loop five-dimensional structure digital design transformation model suitable for ICH and built a digital display system for the “Ten Miles of Bridal Makeup” custom for experimental verification. Tang X.Y. et al. [28] used digital twins to construct a digital virtual culture IP of lion dance, exploring strategies and methods for sustainable design and dissemination of intangible heritage based on immersive sensory interactive experience design. Zhao, Z. [29] proposed a novel digital protection method of ICH, include (1) Intelligent terminal, (2) Fusion of cultural digital content and realistic scene video images, and (3) Object target tracking in realistic scene. Stewart, R. et al. [26] developed an extended model that fits the process of ICH digitalization Europe wide, which based on the DigiCult project. Jouan, P. et al. [30] advocated the application of Digital Twin's (DT) principles, using HBIM models as a digital replica to support the preventive conservation of heritage places, and proposed a data model for structuring information on preventive conservation strategies. Wang, M. & Lau, N. [31] created the NFT digital twin series of Miao silverware crafts co-produced by Miao silversmiths and 3D technicians through design experiments that combine digital technology with traditional Miao silverware making techniques.

In summary, the use of digital technology for the inheritance and dissemination of ICH has gradually become a trend. As a regional technology combining 3D technology and virtual reality technology, digital twin technology has a high similarity with the protection of cultural heritage and ICH, but it is an emerging topic and has few studies. Based on this background, this paper takes Yangxin Cloth Paste as the representative of handicraft ICH, and explores the path of living inheritance of handicraft ICH empowered by digital twin technology.

Methods and materials

Research framework Overview

A digital twin platform design is being undertaken for Yangxin Cloth Paste, a handcrafted ICH in the eastern part of Hubei Province, China. First, through field investigations of Yangxin County's Yangxin Cultural Hall, folk Yangxin Cloth Paste inheritance sites (using Xianjia Village as an example), etc., historical and physical data of Yangxin Cloth Paste ICH are collected. Additionally, surveys and observations of the production process by inheritors of the intangible heritage are conducted, summarizing the actual manufacturing techniques of Yangxin Cloth Paste, thus streamlining the process of Yangxin Cloth Paste craft. At the same time, the entire lifecycle of Yangxin Cloth Paste is segmented for categorized digital twin modeling.

Next, the transformation program for Yangxin Cloth Paste digital twins is designed, using 3D scanning equipment and 3D reconstruction technology to digitalize the entire process of Yangxin Cloth Paste production. Then, the design of an intangible heritage handcraft trading system, a digital twin collection display system, and a digital twin system for intangible handcrafted products is done to constitute the final handcrafted ICH digital twin platform. This is combined with systems and technologies such as nanite, lumen, PBR, LOD, etc., to achieve the platform's design and implementation.

Finally, the platform's user experience process is refined, taking Yangxin Cloth Paste ICH as an example to allow users to experience the platform. The effectiveness of the platform is verified through user satisfaction surveys, and suggestions for subsequent improvements are made. See Fig. 1 for the research process.

Fig. 1

Study flow chart

Yangxin Cloth Paste collection and process digital twinning classification

Investigation and process summary of craftsmen's production

Yangxin Cloth Paste is a folk fabric art popular in the Yangxin County area of Huangshi City, Hubei Province, China. It belongs to folk practical art, and its techniques and products have a wide range of applications. It is one of China's national-level ICH. In order to understand the specific craftsmanship of Yangxin Cloth Paste, a field investigation was conducted in Xianjia Village in Yangxin County. Through observation and interviews with artisans, a series of information about Yangxin Cloth Paste was gathered. According to the survey and interviews, the folk inheritance of Yangxin Cloth Paste is severely lacking, but there is strong interest among the public in learning it. More convenient and efficient ways of passing down the craft are needed to lower the barriers to inheritance. See Fig. 2 for the site of the investigation.

Fig. 2

YangXin Cloth Paste field survey map—Artisan (Photo taken by the author)

Yangxin cloth paste material data

According to different functional and decorative focuses, Yangxin Cloth Paste handcrafts are divided into two categories: cloth daily necessities and cloth flat decorative paintings.

Cloth articles of daily use more emphasis on functionality, representative common cloth articles of daily use including sachets, pillows, insoles, bibs, handbags, etc., the pattern selection is more unified, mainly divided into three categories: the first category is the local common flowers and plants (pomegranate flowers, magnolia, cloves, rose, cuckoos, etc.); the second category is the local common zodiac animals (tiger, rabbit, pig, chicken etc.); and the third category is the local traditional faith symbols (the Eight Trigrams, tai chi, prosperity brought by the dragon and the phoenix, etc.). In the past, villagers pasted on ordinary cloth products with regional characteristics and historical and cultural characteristics of cloth patterns to make them more spiritual value. These cloth daily necessities well reflect the local cultural environment and way of thinking in the past.

Cloth flat decorative painting more emphasis on its decorative and artistic, in modern times after the productivity has been rapid development, the style is more unified, used for frame painting decoration, hanging cloth screen, but its pattern choice is rich and colorful. The flat decorative paintings on cloth paste have well adopted the core image of traditional Chinese culture as the basis for creation. The existing finished products include the plum, orchid, bamboo and chrysanthemum representing noble personality, auspicious flowers, calligraphy characters of various styles, facial masks of local dramas, 12 zodiac signs, 24 solar terms, marriage folk customs, etc. The patterns are more fashionable and artistic. The content carried is also upgraded from the regional culture to the national culture level, which is more aesthetic and communicative. These decorative paintings well reflect the development of modern local spiritual culture and the inheritance of historical culture.

Digital photography was used to record the various types of Yangxin Cloth Paste crafts and categorize their styles. Representative subjects were captured with 3D photography to record digital reconstruction data. See Fig. 3 for the site of the investigation.

Fig. 3

Yangxin Cloth Paste field survey map—Yangxin Cloth Paste handicraft object (self-taken by the author)

Process processing and digital twin classification

To achieve the active inheritance of Yangxin Cloth Paste, an ICH of handcraft, it is necessary to create a digital twin that covers the entire process of producing Yangxin Cloth Paste handcrafted items. Therefore, the production process of Yangxin Cloth Paste was carefully sorted and summarized according to digital twin categories, as follows:

(1) Yangxin Cloth Paste's production requires initial drawing and coloring of the pattern. The pattern can be created using computer graphic design software, and the data can be directly imported into the digital twin system.

(2) The production materials for different Yangxin Cloth Paste crafts are basically the same, so only a few commonly used similar models need to be digitized and stored in the Yangxin Cloth Paste digital twin system database as cases.

(3) In the core production process of Yangxin Cloth Paste, the fractal cutting technique is implemented, and although the process remains consistent, the specific presentation varies completely. Therefore, each step of Yangxin Cloth Paste needs to be digitally twinned to understand in detail the process and real-time progress.

(4) After the patching process of Yangxin Cloth Paste is completed, it basically presents the final effect, with room for adjustment. The patched Yangxin Cloth Paste is digitally twinned, allowing users to propose adjustments to the digitized virtual Yangxin Cloth Paste effect, and artisans can make modifications.

(5) After the sewing process of Yangxin Cloth Paste is completed, representing the completion of the handcrafted item, the final product is digitally twinned and recorded in the Yangxin Cloth Paste digital twin exhibition system.

See Fig. 4 for the organization of the Yangxin Cloth Paste production process and digital twin classification.

Fig. 4

Yangxin Cloth Paste production process and digital twin classification diagram

Digital twin transformation program design and implementation

Digital twin transformation program design

To realize the digital twin of Yangxin Cloth Paste, a conversion program for handcrafted physical digital twin entities is designed, with low latency, low cost, high efficiency, and high feasibility.

The specific operation of the Yangxin Cloth Paste digital twin program involves a combination of hardware and software. Photogrammetry technology is used to achieve the digital modeling of the physical Yangxin Cloth Paste. By taking photos using this technology(Canon-EOS800D used in this article), calculating the camera's position, generating point clouds, and constructing meshes, a model is created [32]. Photogrammetry is a technology that compares, measures, and translates the image information in a series of photos into useful information such as models, maps, etc. Its advantages include:

(1) Low cost, avoiding the use of expensive 3D scanning equipment.

(2) Ease of use, being user-friendly even for those without much experience.

(3) Good results, as it can generate accurate visual three-dimensional digital models based on the photographs.

In order to stably collect the physical details of each step of Yangxin Cloth Paste and realize the digital twin of the whole process, it is necessary to build a flat rotating table.The camera is fixed on the axis of longitudinal movement, and the lens is always aligned with the center of the table, and then the object of different stages of Yangxin Cloth Paste is placed on the table, and the table is rotated at a constant speed for photographic record. When the Angle between the camera Angle and the horizontal plane is 75°, 60°, 45°, 30°, 15°, the table is rotated once to record the point image of the photographic object at this height, and finally shoot three high-definition images under the Angle of 90°. The above five images and three images are a group to record a step of YangXin Cloth Paste posting. According to Fig. 4, the record of a Yangxin Cloth Paste handicraft is divided into material group, cutting sample group, collage group and physical group, with a total of 20 groups of images and 12 top images. The operation mode of the photogrammetric part is shown in Fig. 5.

Fig. 5

Photogrammetric operation demonstration diagram

Next, frame reduction software (Adobe After Effects used in this article) is applied to reduce the video's frame rate, exporting hundreds of high-definition photos for aligning the point cloud.

In order to pursue photo accuracy, the video is shot at 25fps, 1080p resolution (1920 × 1080), and the rotating speed of the table is controlled to make it circle for 30 s, and each circle corresponds to a 30-s image data. The establishment of the digital model requires the observation photos from different angles, so the video needs to be converted into high-definition images. After creating the project in AE, import the captured video, adjust the frame rate to 3 frames/second in the composition Settings, and drag it into the render sequence, adjust the output module to a png sequence, render to generate about 150 images with 1080p resolution, and so on, convert each set of images into pictures and package them.

Then, the exported photos are imported into photo modeling software (Reality Capture used in this article), computing each photo's corresponding camera position and generating a point cloud model, followed by digital modeling and the attachment of color and texture.

Reality Capture is a 3D model making software, which mainly generates data by scanning images, then extracts useful data from it, and finally generates 3D models. The advantage is that it uses many of its functions directly through the command line or running scripts. Here, the scene arrangement in Yangxin Museum that recorded the handmade weaving process in the past is used as a model generation case.The specific operation process of using the reality capture software is shown in Fig. 6.

Fig. 6

Detailed flow chart of model construction by Reality Capture

The final result is shown in the Fig. 7. Based on the good reproduction performance of the color and material of complex scene-like objects, we choose this software for the digital modeling of Yangxin Cloth Paste.

Fig. 7

Reality Capture digital modeling effect of cultural travel resources

Finally, the digital model is imported into a real-time interactive virtual platform (Unreal Engine 5.1 used in this article), where lighting is adjusted and a simple interactive interface is set up to display the handcrafted physical items virtually, completing the digital twin conversion, and the specific implementation will be explained in Sect. "Design and practice of handicraft intangible digital twin platform".

In summary, the digital twin conversion program design of Yangxin Cloth Paste is shown in Fig. 8.

Fig. 8

Yangxin Cloth Paste digital twin conversion program diagram

Yangxin Cloth Paste digital twinning implementation: plum, orchid, bamboo, chrysanthemum

In the specific implementation process of the digital twin transformation of Yangxin Cloth Paste, a set of handcrafted Yangxin Cloth Paste artworks depicting plum, orchid, bamboo, and chrysanthemum (commonly referred to as “Plum, Orchid, Bamboo and Chrysanthemum”) is selected as a case study for creating a digital twin entity. Based on the captured video of Yangxin Cloth Paste, the footage is imported into Adobe After Effects for frame reduction, exported as PNG photos, and then the photo set is imported into Reality Capture for alignment, generating a point cloud and constructing preliminary models for plum, orchid, bamboo, and chrysanthemum respectively. The initial models are influenced by factors such as shooting light, angle of shooting details, photo accuracy, and the complexity of the physical object (structural obstruction), resulting in defects, rough surfaces, and uneven details between light and dark parts.

The problem of model defect is mainly due to the fact that the pictures used for alignment are not accurate to each Angle of view,because some of the pictures exported by video frame reduction will be fuzzy. These fuzzy pictures will cause the software to fail to automatically identify the camera position, and the image alignment of this Angle will be confused, so that subsequent photos cannot be aligned, resulting in the loss of model area.At this time, the solution is to select fuzzy pictures in the imported photo group and delete the camera points corresponding to the Angle, and retain the clear images before and after.

The roughness of the surface of the model is mainly due to the fact that the material of the object reflects the ambient light during shooting, and there is a slight thickness difference and material difference between the paste and the background cloth. The material of the paste is more dense and silky, and the material of the background cloth is more rough and dark, and the cloth is not flat with a slight wrinkle phenomenon.The camera preserves all of these details, and the resulting digital model also appears rough and uneven.However, in the real world, the human eye is unable to detect subtle bumps and thickness differences, and digitization only aggravates these "defects", but loses the sense of reality.At this time, the solution is to use a Tool, a Smoothing Tool in the software function module SCENE, to smooth the convex and convex point to make the model smoother to increase the sense of vision.

The problem of uneven details in the bright and dark parts of the model is mainly due to the fact that the light intensity of each Angle of the stationary object in the real world cannot be the same, so there will be light and dark differences between the various faces of the object itself, which will be more obvious after camera sampling.At this time, the solution is to use a rotating table to place the subject, and shoot by fixing the camera and rotating the table, so that the location of the camera will always shoot the bright spot, control the camera exposure and environmental light conditions remain unchanged, and steadily sample the details of the subject.

The processed digital twin entities of Yangxin Cloth Paste of “Plum, Orchid, Bamboo and Chrysanthemum” can be seen in Fig. 9.

Fig. 9

YangXin Cloth Paste Digital twinning Practice—Plum, Orchid, Bamboo and Chrysanthemum Decorative Painting (digital model)

Design and practice of handicraft intangible digital twin platform

Construction of handicraft ICH digital twin platform framework

The digital twin of cultural heritage is different from the digital twin of industrial manufacturing, and the HDT is a digital representation of the knowledge complex of this heritage asset, covering both tangible and intangible parts.

Meanwhile, the digital preservation of handcrafted ICH differs from tangible cultural heritage. For tangible cultural heritage, the focus of digital twin preservation research lies in the ultra-high precision digital modeling of the cultural heritage itself, attempting to accurately reproduce their color details and material texture in the digital twin. Many use sensors to detect the extent of damage in each site for future repair and protection.

However, ICH is intangible, and it is a collection of geographical cultivation, handicraft skills, community culture and historical beliefs behind handicrafts. With the change of times and the collision of cultures, the past handicrafts no longer adapt to the modern community culture and lose their circulation, which is the main reason why these handicrafts have gradually become heritage. As for the digital twin of ICH, it should reproduce the historical community atmosphere at the birth of ICH, rather than elaborate the digital model itself. The practice of copying the digital twin in the industrial field to the field of cultural heritage protection is debatable. The fundamental purpose of the digital twin of ICH is to find the adaptation point of the heritage in the current community culture, so that it can be transformed into skills more in line with modern culture while retaining its own characteristics, so as to rejuvenate.

In conclusion, the preservation of handcrafted ICH focuses on its living transmission.Therefore, the emphasis on digital twin preservation research for handcrafted intangible heritage is on the comprehensiveness of digital information, such as recording the entire production process of intangible handcrafted heritage.As the most tangible type of ICH, the techniques and processes of handcrafting are its essence, the handcrafted goods are its external manifestation, and the artisans are the channels that transform abstract handcrafting into concrete handcrafted products. The three elements are interdependent, and none can be missing. See Fig. 10 for details.

Fig. 10

The inheritance elements of handicraft ICH constitute

Therefore, for the digital twin platform of handcrafted ICH, it is not only necessary to digitize and preserve the physical handcrafted artifacts but also to stimulate the creative passion of paltform users and digitally twin the entire process of handcrafting itself. Based on these requirements, the platform separately constructs a digital twin collection display system, an ICH handcrafted product transaction system, and an ICH handcrafted product production digital twin system. The platform framework can be seen in Fig. 11.

Fig. 11

Yangxin Cloth Paste affixed digital twin platform frame diagram

Digital twin system for the production of ICH handicrafts

To achieve the living transmission of handcrafted ICH, it is necessary to continue its social value in the dynamic production and creation of handcraft. The digital twin system of ICH handcrafted product production covers the entire process of handcrafting, twinning all the physical objects and the state of handcrafted goods in all stages, guiding users to make their own handicrafts in detail, and record and save them in a virtual reality environment.

In the process of building a digital twin system for handcrafted ICH, using traditional digital modeling methods can lead to key problems such as slow model creation, low simulation of color and texture rendering, oversized model volume, slow data transmission, and poor user interaction experience. Ballabeni, A. et al. [33] resented an efficient pipeline based on color enhancement, image denoising, RGB to Gray conversion and image content enrichment. To solve these problems, the following practical technologies are used for alternative optimization in this study:

(1) Optimization Measures for Slow Model Creation: Utilizing Photogrammetry for 3D reconstruction to create digital twins. Compared to traditional digital modeling, the advantages of photographic 3D reconstruction include: (1) better restoration of physical color and texture effects under lighting; (2) faster model generation, enabling digital twinning of physical objects.

(2) Optimization Measures for Low Simulation of Color and Texture Rendering.Using PBR (Physically-Based Rendering) for baking and texture processing, reducing the polygon count of the model while retaining the surface details of the high-polygon model, to fit the actual effects more closely. PBR simulates properties like metallic and roughness through special lighting algorithms, reflecting the interaction between light and the surface of the material more precisely. Unlike texture rendering, the PBR rendering pipeline requires attaching the texture to the material sphere, adjusting the material effect, and then rendering it to the model entity. Taking Yangxin Cloth Paste as an example, PBR material ball is composed of basic color, metallicity, high brightness, roughness, anisotropy, self-luminous color, normal, tangent and other main adjustment parameters. The initial material ball has a default value, and the material tends to be translucent rubber texture; Adjust the metal degree to 1, roughness to 0.5 will show a silver metal texture; If the roughness is adjusted to 0, the material ball will show a mirror effect; In order to present the visual effect of cloth products, the roughness is adjusted to 0.8, the high light is adjusted to 0.2, and the metal is adjusted to 0.8. Specific visual differences are shown in Fig. 12.

Fig. 12

Material ball differential adjustment

The material texture map is generated when the model is exported from Reality Capture, so the normal and tangent values do not need to be adjusted, just import the map into the normal. The display effect of the digital model obtained before and after adjusting the value is obviously different, and the adjusted digital model is more suitable for the fabric effect,as seen in Fig. 13.

Fig. 13

Application effect of PBR pipeline rendering technology (Taking fabric effect as an example)

(3) Optimization Measures for Excessive Model Volume.Utilizing Nanite virtual geometry technology to reduce the size of the digital model without affecting the rendering effect. Models obtained through camera scanning tend to lose many details in the digital conversion process, usually composed of triangular mesh bodies. Simultaneously, since the physical details are much greater than those produced by traditional digital modeling, the number of mesh bodies can reach an enormous scale. This means that the model occupies excessive computational storage space, making it difficult to edit in virtual digital software. Nanite is a virtualized geometry system in Unreal Engine 5, using a brand-new internal mesh format and rendering technology to render pixel-level details and massive objects. It can intelligently handle only perceptible details. Furthermore, Nanite employs a highly compressed data format and supports fine-grained streaming with automatic levels of detail. This means that if the Nanite system is used, it can not only reduce the memory pressure of real-time editing in the editor but also support high-precision models composed of tens of millions of triangular mesh bodies. Taking the digital twin of Yangxin Cloth Paste of “Plum, Orchid, Bamboo, and Chrysanthemum” as an example, enabling Nanite support has reduced at least one-third of the storage pressure without losing model accuracy, as seen in Fig. 14.

Fig. 14

Nanite technology improves disk storage without changing rendering accuracy

(4) Optimization Measures for Slow Data Transmission. Utilizing 5G (Fifth Generation) for real-time data transmission. The digital twin system for ICH handcraft production requires real-time updates of the digital twins of handcrafted products. Virtual digital twins and real handcrafted products need to be consistent, enabling users to understand specific processes and production progress, and thereby propose modifications to actual production. Simultaneously, since the digital twin system uses camera modeling, large files such as HD videos, massive high-precision photos, generated massive mesh models, and texture maps must be transmitted. Therefore, data transmission in the ICH handcraft production digital twin system must meet the standards for instantaneous transmission of large files. With millimeter-level wavelengths, ultra-wide bandwidth, ultra-high speed, and ultra-low latency, 5G perfectly fits the data transmission requirements.

(5) Optimization Measures for Low User Interaction Experience.Using UMG Editor's Unreal Motion Graphics UI Designer to create user interaction interfaces. UMG is implemented based on the Slate framework, allowing direct construction of UI widgets within Unreal Engine. Interaction methods can be designed through widget blueprints, and corresponding text, animations, and events can be adjusted to create interactive effects. In the ICH handcraft production digital twin system of Yangxin Cloth Paste, several main interactions were achieved through UMG:

① Users can control a virtual person to observe and interact with background walls representing each craft in the digital twin room of the handcrafted product. When a craft is completed in actual production, its digital twin background wall will be updated within the system room.

② Each process's digital twin model has text information and video demonstrations available for interaction, helping users understand more about Yangxin Cloth Paste craft.

③ Once the final product is completed, its digital twin will be displayed on the podium, allowing users to experience the details of the handcrafted product in a virtual space, approaching realism.

After improving the above issues, the system is run on a workstation connected to Unreal Engine 5 (unreal engine 5.1). Taking the making of a Yangxin Cloth Paste bag as an example, some sample screen captures are provided, seen in Fig. 15. Figures a, b generate digital twin background walls for processes including material preparation, hard clay cloth shape cutting, floral cloth shape cutting, cloth patching layering, cloth patching piecing, and sewing techniques. Figures c, d set interactive podiums in front of the digital twin background walls for each process, allowing users to read craft text by pressing "E" and watch craft videos by pressing "R". Figures e, f set a display podium for the finished digital twin in the virtual room of the Yangxin Cloth Paste bag production system, showing whether it is completed. Figure g places the digital twin of the finished Yangxin Cloth Paste bag on the podium, allowing users to appreciate the simulation details from various angles. Figure h shows an actual photograph of the finished Yangxin Cloth Paste bag.

Fig. 15

Screenshot of system interaction: a and b Yangxin Cloth Paste technology display wall; c, d process text video interaction; e, f finished DT display platform; g Yang new package DT; h Yang new package reality entity

Digital twin collection display system

In the digital preservation aspect, the greatest advantage of handcraft-based ICH compared to other types of ICH lies in the ability to produce corresponding physical products for retention. Treating the produced ICH handcrafts as physical cultural heritage, it's possible to create Heritage Digital Twins (HDT) for these handcrafted items, thereby constructing a collection display system for handcraft-based ICH. This allows users to experience the craftsmanship charm of handcraft-based ICH within a virtual space.

Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which properties of the worlds can be modified [34]. To enable users to experience nearly lifelike handcrafted items in a digitized virtual environment, the following techniques were employed to optimize the high-detail display of the handcraft DT, based on the DT conversion program described in this article:

(1) To make the DT collection display system run more smoothly, cloud storage technology is utilized. The principle of HDT is to digitally record all the information related to the heritage as comprehensively as possible. The purpose of the ICH DTis to transform the digital model into a knowledge complex of the heritage assets, so the digital preservation and transmission of a large amount of information needs to be considered.Cloud storage is a cloud computing model that allows data and files to be stored on the Internet through cloud computing providers. Users can access these data and files through the public Internet or dedicated private network connections, offering agility, scalability, and durability, along with anytime, anywhere data access. The handcraft-based ICH's production case models and handcraft art artifact models are transmitted to the cloud. When users choose to browse the corresponding handcraft-based ICH, the relevant model package is downloaded from the cloud. In the future, with the increase of the virtual twints of different types of crafts, cloud storage technologies will be classified and sorted in the cloud so that the corresponding cultural data resources packages are loaded faster when the user selects the virtual collection to interact. The cloud storage architecture of this system is shown in Fig. 16.

Fig. 16

Handicraft ICH-DT collection display system-cloud storage architecture

(2) To ensure that users do not experience lag when browsing the exhibition environment, LOD technology is used to optimize the model, reducing the file size while preserving the model's quality, facilitating data storage. LOD (Level of Detail) translates to a level of detail technique that, without affecting the visual effect of the image, simplifies the surface details of the scene step by step to reduce the scene's geometric complexity, thereby enhancing the efficiency of the drawing algorithm [35]. This technique usually constructs several different geometric models with varying approximation accuracies for each original polyhedral model. Depending on the object's distance to the viewpoint, different levels of detail are used [27], greatly improving computational efficiency.

(3) To make the display of the collection more in line with the real texture, the Lumen global illumination technology is used, rendering light rays in real time to beautify the display effect of the collection. Lumen is Unreal Engine 5’s fully dynamic global illumination and reflection system, capable of rendering indirect diffuse reflections in grand scenes with many details, ensuring unlimited bounces and indirect high-luminosity reflection effects, thereby solving the problem of indirect diffuse illumination, making the scene more realistic.

After adopting the above technologies to improve the exhibition space, the system is connected to Unreal Engine 5.1’s workstation for operation, taking Yangxin Cloth Paste collection as an example, with some sample screenshots provided in Fig. 17. Figure a sets up the Yangxin Cloth Paste collection display space, placing the DT of Yangxin Cloth Paste handcrafts on a wooden platform. Figure b provides an interactive button at the corresponding collection spot, allowing users to jump to the DT production system’s virtual room by pressing the “E” key. Figures c and d take Yangxin Cloth Paste's green handbag as an example, and e and f take Yangxin Cloth Paste’s shoulder bag as an example. Compared to the actual pictures, they display the system's product simulation and detail restoration levels, highlighting the visual display advantage of the DT system where the virtual and real blend seamlessly. Figures g and h use the Yangxin Cloth Paste art painting “Plum, Orchid, Bamboo, and Chrysanthemum” as an example, showcasing the DT collection's visual effects within the system.

Fig. 17

Screenshots of Yangxin Cloth Paste collection display system: a, b display space and interaction; c, d green bags and exhibits; e, f handbags and exhibits; g, h plum, orchid, bamboo and chrysanthemum display effect

ICH handicraft products trading system

Handicraft artisans and craftspeople are increasingly taking advantage of digital information technologies to disseminate their skills and expertise, market their products and attract a wider range of ICH inheritors [36]. The DT platform for ICH in handicrafts establishes interactive channels between artisans and users, facilitating real-time communication between both parties through the internet. This interaction promotes the trade of ICH handicrafts, stimulates the artisans' creativity, and, guided by user demands and relying on the collaborative development advantages of the digital ecosystem platform, fully leverages the professional value of handicraft practitioners, designers, and other personnel to satisfy users’ personalized creative service needs [37]. Relying on the ICH handicraft trading system, users can upload their works to the platform to provide other users with experience by understanding and making detailed ICH crafts. While exchanging crafts through the virtual platform, users can also contact the author to purchase the ICH handicrafts they have experienced, so as to realize the purchase experience combining virtual and real ICH handicrafts, thereby attracting more users to invest in the ICH of handicrafts.

Stage summary

Due to the characteristics of immaterial HDT, the focus of research on Yangxin Cloth Paste DT is not the protection and feedback of individuals, but more comprehensive digital record of relevant information. The DT is used as a medium to integrate heritage information, so that users can interact with the DT in the virtual world and experience it personally, so as to achieve the purpose of in-depth understanding and learning of ICH. Therefore, in the practice of digital twinning of handicraft ICH, its performance is more inclined to the combination of digitalization of the whole production process, digitalization of heritage-related information and virtual space experience.

In the production process of Yangxin Cloth Paste, the real-time update is the digital record of the state of the production process, which is achieved through the transformation degree of the camera head and 2.3.1, including the digital model under different processes, the production of videos, the introduction of craftsmen, etc. In 2.3.1, it is mainly divided into four groups, which are essentially virtualized restoration of the actual production process.

Users first enter the collection display system introduced in 2.4.3, feel the visual charm of Yang new cloth pastes in the virtual World-Expo space, and then select the collection for interaction, enter the virtual room of the production DT system proposed in 2.4.2, and the DT of the entire production process are displayed on the production table and production background wall in the room. By interacting with the production table in front of the background wall, you can see the production video of the corresponding process, the image of the craftsman who made the handicraft, the text explanation of the cloth pattern and the process introduction. There is a cloud head in the virtual room, and the digital virtual body of the handicrafts that have been made will be presented on it. Users can feel the charm of more details and crafts in the virtual room of production.

Users can only interact with the DT under the updated process in the virtual space, and learn about the actual production video media, the production process, the corresponding craftsmen and other text information. For the adjustment of handicrafts themselves, it is necessary to communicate with the corresponding handicrafts through the ICH handicraft product trading system, and put forward their own improvement suggestions outside the virtual space. The same kind of feedback also makes it easier for users to connect with each other. After the craftsmen adopt the advice, the subsequent processes will be adjusted, in order to keep the camera data acquisition procedure unchanged, the existing DT production virtual body plate will be retained.

Results and discussion

User experience process

Taking Yangxin Cloth Paste as an example, ten ICH experts and one individual capable of creating Yangxin Cloth Paste crafts were invited to experience the DT platform for ICH in handicrafts and conduct a full video recording. The universally applicable user experience process of this platform was then organized, as shown in Fig. 18. While viewing past examples of handicrafts to understand the ICH craft, users can interact with the craftsmen to agree on designs and product categories. They can also monitor the real-time progress on the platform and propose further personalized requirements for the artisans to modify and adjust. The artisans will gradually upload their crafting process to the platform, generating corresponding step boards in virtual space and updating them in real time based on adjustments. The finished product will be displayed on the platform, and after confirming the DT effect through virtual scenarios, users can purchase the corresponding physical product. Completed cases will be archived into the collection display system.

Fig. 18

Yangxin Cloth Paste DT platform user experience flow chart

Customer satisfaction survey

In order to evaluate the performance and user satisfaction of the DT platform for ICH in handicrafts, 50 users were invited to participate in a survey experience of the platform using Yangxin Cloth Paste, a type of ICH, as an example. A total of 47 valid survey questionnaires were collected. The questionnaire was designed to examine various aspects such as interest, immersion, interactivity, and satisfaction, and the Likert scale was used for data collection and assessment. The metrics for interest focused on understanding desire, experience willingness, and purchase intent; the metrics for immersion emphasized realism, presence, and focus [38]; the metrics for interactivity examined ease of operation, accessibility, and freedom; the metrics for satisfaction concentrated on promotion willingness, frequency of use, and update expectations. Based on these metrics, the questionnaire survey results are shown in Table 1.

Table 1 Design user satisfaction questionnaire for handicraft ICH DT platform

The experimental data shows that the average value of all the user feedback is 3.66, indicating an overall inclination towards satisfaction. Among the four evaluation aspects, the average score for satisfaction was the highest (4.06), and the average score for interactivity was the lowest (3.15). This suggests that after experiencing the platform, users appreciated its innovation and value, but the specific virtual interaction did not meet expectations. Immersion and interest both scored above 3.5, reflecting users' general interest in the digitization of ICH in handicrafts, and that the platform provided a relatively authentic virtual experience. Among all the evaluated metrics, experience willingness, frequency of use, and update expectations all scored above 4, indicating that the DT platform for ICH in handicrafts designed in this study preliminarily met users' pre-existing expectations for digitalized ICH experiences. Ease of operation and accessibility both scored below 3, indicating that the platform's interaction needs further optimization.

Overall, the DT platform for ICH in handicrafts designed in this study has initially achieved its goals in intangible heritage inheritance and attracting users. Users have a generally positive attitude towards the platform and have high expectations for subsequent iterative updates.

Conclusion

In the digital age, to breathe new life into traditional culture and achieve sustainable development, it's crucial to integrate traditional cultural foundations with digital technology as the medium, and creativity at its core [39]. This study takes “Yangxin Cloth Paste” from the eastern part of Hubei Province in China, an ICH of handicraft, as its research subject. By introducing DT technology to the protection of handicraft-related ICH, this research aims to explore new pathways for the living inheritance of such heritages through the digitization of the entire production cycle and a combination of virtual and real user experiences.

Taking the case of Yangxin Cloth Paste, the study digitalizes the handicraft through processes such as photography, point cloud processing, model generation, and engine imports. This leads to efficient and feasible physical virtualization, offering tangible reference value.

To construct the DT platform, the study first established a DT system for the production of intangible handicraft heritage products. A digitally-oriented process classification was carried out for Yangxin Cloth Paste, with 3D reconstruction of the entire production cycle. The constructed 3D models utilized PBR pipeline rendering to enhance realism. Triangle mesh models imported into the virtual engine were optimized using nanite virtual geometry technology, reducing model size and lessening editor memory pressure. The platform uses 5G to transmit data, achieving real-time updates between virtual and real worlds. The UMG editor designed an interactive interface, allowing users to understand product and craft details. A DT collection display system was then created, showcasing existing and soon-to-be-produced handicrafts virtually. Using cloud storage for extensive model data and LOD technology to optimize model displays, combined with lumen global illumination, the platform presents near-real ICH displays. Finally, a transaction system for intangible heritage crafts was established to promote innovation and inheritance.

However, there are limitations. Firstly, Yangxin Cloth Paste’s representation of intangible cultural handicraft heritage is limited. There are other crafts like metallurgy, ceramics, food, tools, toys, weaving, and paper cutting, each with unique production techniques and DT manifestations. Future research will delve into these areas. Secondly, the difficulty in widespread adoption due to the complexity and variety of equipment, software, and techniques involved, though cost-effective, is challenging for users. This necessitates specialized platform operators for the digitization of heritage resources. Lastly, the focus of this study was on the replication of the craftsmanship process and products in virtual spaces, lacking in the design of interaction and presentation in those spaces.

In the future, the platform’s design goal is to incorporate a broader range of handicraft heritages and strengthen its experience and education by introducing thematic events and new interactive methods. DT technology will be used to guide more high-quality ancient intangible cultural relics handicrafts to log on this virtual platform, and split their processes, so that users can learn and carry out their own interest production, increase social interaction to promote exchanges between users on intangible cultural relics, and promote the inheritance and innovation of handicraft intangible cultural relics.