DIGAN: distillation model for generating 3D-aware Terracotta Warrior faces

Abstract

Utilizing Generative Adversarial Networks (GANs) to generate 3D representations of the Terracotta Warriors offers a novel approach for the preservation and restoration of cultural heritage. Through GAN technology, we can produce complete 3D models of the Terracotta Warriors’ faces, aiding in the repair of damaged or partially destroyed figures. This paper proposes a distillation model, DIGAN, for generating 3D Terracotta Warrior faces. By extracting knowledge from StyleGAN2, we train an innovative 3D generative network. G2D, the primary component of the generative network, produces detailed and realistic 2D images. The 3D generator modularly decomposes the generation process, covering texture, shape, lighting, and pose, ultimately rendering 2D images of the Terracotta Warriors’ faces. The model enhances the learning of 3D shapes through symmetry constraints and multi-view data, resulting in high-quality 2D images that closely resemble real faces. Experimental results demonstrate that our method outperforms existing GAN-based generation methods.

Introduction

Employing GAN technology to generate 3D representations of the Terracotta Warriors can significantly aid experts in the preservation and restoration of cultural heritage. Many of the warriors are damaged or partially destroyed; using GAN to create complete 3D models provides reference frameworks for restoration efforts. Digitizing the warriors’ features and creating 3D models substantially enhances methods of cultural heritage preservation and exhibition. These digital 3D models can be utilized in virtual museums, online exhibitions, and educational platforms, allowing global audiences to explore the intricate details of the Terracotta Warriors and amplifying their cultural impact. Generating 3D representations of the warriors’ faces using GAN has significant implications not only for heritage preservation and restoration, digital conservation and display, academic research, education, and the cultural creative industries but also for driving technological innovation and development. Through these applications, we can better protect and transmit invaluable cultural heritage while fostering new creativity and technological advancements.

Generative adversarial networks (GANs) [1,2,3], such as StyleGAN, can generate high-quality and diverse facial images. These networks learn features from large datasets and generate realistic faces through machine learning [4] and transfer learning [5]. These generated images are valuable across various applications, from entertainment and security to art and scientific research. However, these models have limitations. The primary issue is the randomness of the generated faces, which prevents users from directly controlling the semantic attributes of the images, such as pose and expression. This inability to specify details means users must accept the model’s randomly generated outcomes, posing a significant constraint for applications requiring precise control over image features. Current research aims to develop methods that provide control during the generation process, allowing users to adjust the semantic attributes, particularly 3D information like pose. This control is beneficial for applications such as facial recognition [6, 7] and image synthesis [8,9,10], as it enables the generation of faces with specific features, enhancing accuracy and flexibility. Most existing methods rely on external supervision [11], such as pose labels [12], landmarks, or synthetic images [13]. These external data are used to disentangle semantic factors in GAN’s latent space, enabling better control of these features during generation. Additionally, some methods explore unsupervised approaches, investigating the latent space [14,15,16] of GANs to achieve controllability of 3D features. These unsupervised methods rely on a deep understanding of data distribution and GAN’s internal mechanisms, employing clever designs to control the generated features.

Compared to solutions that only generate 2D images, constructing generative models with explicit 3D shapes offers significant advantages, including detailed adjustments to shape, pose, and texture. However, existing 3D generation solutions struggle with complex objects, particularly when generating fine-grained details and diverse facial expressions. 3D generative models [17,18,19,20] not only create realistic 3D forms but also provide stricter and more precise control over content. This control extends to detailed adjustments of object shape [21, 22], pose [23, 24], and texture [25, 26], better meeting specific application requirements. Recently, some research has attempted to train 3D generative models using 2D images, but these efforts mainly focus on relatively simple and coarse objects, like cars. For such objects, training data from 2D images can adequately support 3D shape reconstruction. However, this approach shows clear limitations when dealing with more complex objects. When applied to objects with fine-grained details, especially faces, existing 3D generation solutions encounter significant difficulties in learning reasonable 3D shapes. Faces possess rich details and varied expressions [27,28,29,30], imposing higher demands on generative models.

Despite significant advancements in GAN-based image generation techniques in recent years, particularly in generating high-quality 2D facial images, existing studies still face substantial challenges in producing 3D-aware facial models. These challenges primarily stem from the limited control over the generation of complex facial features and a lack of precise understanding of the generated images’ three-dimensional morphology. Current approaches often rely on external supervision data, which not only complicates data preparation but also restricts the broader application of these models. Additionally, these methods frequently fail to generate coherent and consistent shapes when handling large pose variations, resulting in suboptimal visual outcomes that do not meet practical application requirements.

To address these issues, we have developed a 3D generation model based on a distillation approach (DIGAN) that enables high-precision 3D facial generation of Terracotta Warriors without the need for external supervision. Our goal is to provide a more flexible and efficient technology for 3D facial generation, driving innovation in cultural heritage preservation and digital display.

In this paper, we develop a distillation model to generate 3D representations of the Terracotta Warriors’ faces. By refining and leveraging knowledge from StyleGAN2, we train a 3D generative network. Specifically, we employ an innovative approach that transfers the strengths and insights of the existing StyleGAN2 model to a novel 3D generator, enabling it to produce high-quality 3D models of the Terracotta Warriors’ faces. The G2D network, as the primary component, generates detailed and realistic 2D images through layered convolution operations and feature fusion. The main objective of the 3D generator is to modularly decompose G2D’s generation process, encompassing different 3D modules such as texture, shape, lighting, and pose. These modular 3D components, when integrated, render 2D images of the Terracotta Warriors’ faces. This decomposition and integration process results in images that are visually more realistic and three-dimensional. Through these steps, the G2D model acts as a teacher model, providing direct supervisory signals to G3D, thereby avoiding adversarial training. Additionally, the use of symmetry constraints and multi-view data effectively enhances the learning of 3D shapes. By modularizing StyleGAN2’s generation process, we construct a powerful 3D generative network capable of producing high-quality 2D images of the Terracotta Warriors. This new 3D generator not only inherits StyleGAN2’s ability to generate high-quality images but also improves image realism and three-dimensionality through modular design, resulting in visuals that closely resemble real human faces.

The main contributions of this paper can be summarized as follows:

• We developed a distillation-based framework for generating 3D perceptual models of the Terracotta Warriors’ faces, named DIGAN.

• We constructed a G3D model comprising five sub-networks to complete the transformation from style codes to final rendered images.

• Our method can produce high-quality 3D information for human faces and Terracotta Warriors’ faces, allowing precise control over pose and lighting.

Related work

Attitude disentanglement 2D GAN

Recent research on Generative Adversarial Networks (GANs) has demonstrated their capability to generate high-quality, realistic images. In recent years, significant advancements in the fineness and realism of image generation have been achieved, making GAN-generated images nearly indistinguishable from real photographs. Increasingly, studies are focused on disentangling various factors within images, such as pose, lighting, and background. These studies not only enable GANs to produce realistic images but also allow better control over different elements of the images, enhancing image editing and manipulation capabilities. The first category of methods introduces specific modules or loss functions during training to ensure the disentanglement of pose information. Specifically: Tran et al. [31] and Tian et al. [32] utilize pose labels during training to ensure the independence of pose information. Hu et al. [33] and Zhao et al. [34] employ landmarks and 3D Morphable Models (3DMM) to assist training, thereby improving pose disentanglement. Deng et al. [35] guide the learning of pose factors during GAN training using 3DMM, resulting in more natural and accurate pose generation. The CONFIG method [36] combines real and synthetic images for training to enhance GAN’s image quality and pose control capabilities. HoloGAN [37] uses a 3D feature projection module to generate rotatable face images, enabling manipulation in 3D space. The second category of methods manipulates pre-trained GAN networks to alter the output’s 3D information. Specifically: Labels [38] or 3DMM [39] are used to control 3D information in generated images by disentangling the latent space. Shen et al. [40] propose an unsupervised method to decompose the GAN’s latent space, achieving control over 3D information in generated images without additional label data. Despite the progress in generating high-quality images and disentangling pose information, these methods still have a notable drawback: the inability to explicitly output the object’s 3D shape. This limitation is particularly critical for applications requiring strict control over 3D information. Overall, while GANs have made significant strides in image quality and pose disentanglement, their limitation in explicitly outputting 3D shapes persists. This defect constrains their practicality in applications requiring precise 3D control. Future research may need to explore ways to overcome this limitation to achieve more comprehensive 3D image generation and control.

3D reconstruction and generation of 2D images

Research on reconstructing and generating 3D shapes from unlabelled 2D images aims to extract three-dimensional information from flat image data, producing realistic 3D shapes. In the domain of unsupervised reconstruction methods, Tulsiani et al. [41] proposed a method that uses multi-view consistency as a supervisory signal, leveraging consistency between images from different viewpoints to infer 3D shapes. Kanazawa et al. [42] used the consistency of similar objects to learn reconstruction models, improving accuracy by comparing and learning from multiple instances of the same class of objects. Wu et al. [43] presented a method that utilizes the symmetry of objects to learn detailed shape and reflectance information for more refined reconstruction. Handerson et al. [44,45,46] reconstructed 3D meshes using shadow information from synthetic images and extended this approach to generate models capable of creating new 3D shapes. In the realm of generative models, GAN architectures (Generative Adversarial Networks) provide signals through adversarial loss, making the generated 3D shapes more realistic. Gadelha et al. [47] applied discriminators to the contours of generated voxels, enhancing the details of the 3D shapes produced. Lunz et al. [48] used commercial renderers to guide neural renderers in outputting shadowed images, increasing the realism of the generated images. Voxel-based methods, however, have limitations in recovering fine-grained details and 3D surface colors, often resulting in 3D shapes lacking sufficient detail and color information. Among the latest methods, Szabo et al. [49] employed vertex position mapping to train GANs with textured mesh outputs directly, yet they still encountered shape distortion issues during generation. Some studies use StyleGAN to generate multi-view synthetic data. For instance, Zhang et al. [50] improved data quality through manual annotation, while Pan et al. [51] iteratively synthesized data and trained reconstruction networks to enhance generation outcomes. This paper proposes a novel 3D generative model that innovatively addresses distortion and detail recovery issues in previous methods by simultaneously learning to manipulate StyleGAN2 generation and estimate 3D shapes. This approach produces more accurate and detailed 3D shapes, offering higher quality 3D reconstruction and generation results.

Methods

This chapter provides a detailed overview of the overall network architecture, the implementation details of the five sub-networks, and the loss functions used. First, the design of the entire network architecture is described, including how the various modules collaborate. Next, the specific implementation methods and technical details of each sub-network are elaborated, covering the generation of texture and shape, as well as adjustments to lighting and pose. Finally, the chapter discusses the loss functions employed, which play a crucial role during training to ensure the quality and realism of the generated images.

Overall framework

As shown in Fig. 1, our network first implements G2D, based on StyleGAN2, to generate 2D images of Terracotta Warriors, serving as a teacher model to supervise the learning process of G3D. For each random sampled image input from StyleGAN2 into G3D, five sub-networks are utilized: DV, DL, DS, DT, and DW. These sub-networks are each responsible for different tasks, working together to complete the transformation from style code to the final rendered image. The sub-network DV, using the KAN structure, maps the input random sampled feature ${w}$ to a 6-dimensional viewpoint representation ${V}$. This viewpoint representation includes translation and rotation information of the image, determining the perspective of the final rendered image. The sub-network DL, also using the KAN structure, decodes the style code ${w}$ into a 4-dimensional output ${L}$. This output ${L}$ includes the x–y direction of the light, ambient light, and diffuse light intensity, thus determining the lighting effects of the image. The sub-network DW generates texture mappings through StyleGAN2, defining the detailed texture of the image. The sub-network DS generates depth maps, representing the depth information of different parts of the image. The sub-network DT generates transformation maps, describing the shape and positional transformations within the image. DS and DT map the disentangled style code ${w_0}$ to shape representation ${S}$ and transformation mapping ${T}$. This separate handling of shape and transformation makes the disentangling of styles clearer and more intuitive.

Fig. 1

The overall framework of our DIGAN. The framework includes a teacher model of G2D and an academic model of G3D. G3D consists of five sub-networks and a renderer R

Using G3D, the style code ${w}$ is converted into three-dimensional shapes and views, and a differentiable renderer ${R}$ generates 2D images ${I_{3D}}$. The images generated by G2D, $I_{2D}$, are used as a supervisory signal, and a loss function is defined to measure the difference between ${I_{2D}}$ and ${I_{3D}}$. By minimizing the loss function through backpropagation, the parameters of the G3D model are updated, making the generated images $I_{3D}$ as close as possible to the images ${I_{2D}}$ generated by G2D. The renderer $R$ uses shape representation ${S}$, transformation mapping ${T}$, viewpoint representation ${V}$, and lighting effects ${L}$ to output the final rendered image. This process integrates the outputs of all sub-networks, achieving the transformation from style code ${w}$ to rendered image ${I_w}$. Moreover, the depth maps and texture maps obtained from the aforementioned sub-networks are correlated so that each depth value can be mapped to the corresponding texture position, generating realistic 3D appearances of the Terracotta Warriors. By combining the depth maps and transformation maps, complex shape representations and transformations can be realized.

G3D student model

The primary goal of the G3D student generator is to modularize the G2D generation process, encompassing various 3D components such as texture, shape, lighting, and pose. Acting as the teacher model, G2D provides direct supervisory signals for G3D, circumventing the need for adversarial training. Additionally, the use of symmetry constraints and multi-view data effectively enhances the learning of 3D shapes. The G3D student generator not only inherits the advantage of high-quality image generation from StyleGAN2 but also improves the realism and three-dimensional effect of the images through its modular design, making the visual representation of the Terracotta Warriors more lifelike.

DV and DL Sub-networks

Through the KAN structure, sub-networks DV and DL process the input style code ${w}$, generating information related to the image’s viewpoint and lighting effects. Specifically, sub-network DV produces a 6-dimensional viewpoint representation ${V}$, which includes translation and rotation information, thus determining the image’s perspective. Sub-network DL generates a 4-dimensional lighting information ${L}$, encompassing the direction of light, ambient light, and diffuse light intensity, thereby determining the image’s lighting effects. This approach allows for flexible adjustments to the image’s perspective and lighting effects based on different style codes ${w}$, achieving highly controllable image rendering. The detailed implementation is as follows. The DV sub-network outputs a vector that contains viewpoint features. Similar to the lighting information, we also constrain the viewpoint information. This is achieved by applying the Tanh activation function to limit the values within the range of [− 1, 1]. Randomly sampled feature ${w}$ is used to generate the viewpoint information ${V}$, which is produced by the viewmlp module:

$${\mathbf{V}} = \tanh \left( {{\text{MLP}}_{{{\text{view}}}} \left( {\mathbf{w}} \right)} \right)$$

(1)

The DL sub-network generates a vector containing lighting features. To ensure that the lighting information remains within a reasonable range, we typically constrain it. The output is limited to the range of [− 1, 1] using the Tanh activation function.

Given the randomly sampled feature ${w}$, the lighting information ${L}$ is produced by the lightmlp module:

$$\textbf{L} = \tanh \left(\text{MLP}_\text{light}(\textbf{w})\right)$$

(2)

The MLP is responsible for generating the latent space vectors, consisting of multiple layers and including components such as KANLinear, GroupNorm, and the LeakyReLU non-linear activation function. Inspired by the literature, the specific MLP can be represented as follows:

The overall representation of the MLP is:

$$\textbf{y} = f_n \circ f_{n-1} \circ \cdots \circ f_2 \circ f_1(\textbf{x})$$

(3)

where each layer ${f_i}$ is defined as:

$$f_i(\textbf{h}_{i-1}) = \text {LeakyReLU} \left(0.2, \text {GroupNorm}\left( \frac{d}{4}, \text {KANLinear}(\textbf{h}_{i-1})\right)\right)$$

(4)

The KAN operation is defined as:

$$\text{KAN}(\textbf{Z}) = (\Phi _{K-1} \circ \Phi _{K-2} \circ \cdots \circ \Phi _{1} \circ \Phi _{0}) \textbf{Z}$$

(5)

where ${\Phi _i}$ represents the ${i}$-th layer of the entire KAN network.

For the final layer:

$$f_{n} \left( {{\mathbf{h}}_{{n - 1}} } \right) = {\text{activation}}\left( {{\mathbf{W}}_{n} {\mathbf{h}}_{{n - 1}} + {\mathbf{b}}_{n} } \right)$$

(6)

This representation ensures a structured and effective transformation of the input latent vector $\textbf{x}$ through multiple layers, incorporating normalization and activation functions to achieve the desired output $\textbf{y}$.

DS sub-network

The DS sub-network is designed to generate depth maps. The process involves an optional MLP section, followed by an upsampling section with partial information fusion, and finally a head section that produces the final output. The detailed process is as follows:

Given input $\textbf{x}$, it first passes through multiple fully connected layers with LeakyReLU activations:

$$\textbf{h}_{\text {MLP}} = \text {LeakyReLU}(0.2, \textbf{W}_i \textbf{h}_{\text {MLP}} + \textbf{b}_i)$$

(7)

In the upsampling section, initial upsampling uses transposed convolution (ConvTranspose2d) and ReLU activation:

$${\mathbf{h}}_{{{\text{upsample}}}} = {\text{ReLU}}\left( {{\text{ConvTranspose}}2{\text{d}}\left( {{\mathbf{h}}_{{{\text{MLP}}}} } \right)} \right)$$

(8)

Next, a Partial_conv layer with ReLU activation is introduced:

$${\mathbf{h}}_{{{\text{upsample}}}} = {\text{ReLU}}\left( {{\text{Partial}}\_{\text{conv}}3\left( {{\mathbf{h}}_{{{\text{upsample}}}} } \right)} \right)$$

(9)

Our network includes five upsampling blocks, each consisting of GroupNorm, transposed convolution (ConvTranspose2d), and ReLU activation, as well as GroupNorm, Partial_conv3, and ReLU activation.

In the injection section, a 1x1 convolution processes input $\textbf{x}_2$ to obtain $\textbf{h}_{\text {inject}}$:

$$\textbf{h}_{\text {inject}} = \text {Conv}2{\text{d}}_{1 \times 1}(\textbf{x}_2)$$

(10)

The upsampled result is added to the injected result to get $\textbf{h}_{\text {combined}}$:

$$\textbf{h}_{\text {combined}} = \textbf{h}_{\text {upsample}} + \textbf{h}_{\text {inject}}$$

(11)

In the head section, $\textbf{h}_{\text {combined}}$ is processed through GroupNorm and Partial_conv, with ReLU activation:

$${\mathbf{y}} = {\text{ReLU}}\left( {{\text{GroupNorm}}\left( {{\text{Partial}}\_{\text{conv}}3\left( {{\mathbf{h}}_{{{\text{combined}}}} } \right)} \right)} \right)$$

(12)

Further processing of $\textbf{y}$ is done through GroupNorm and Conv2d, with ReLU activation:

$$\textbf{y} = \text {ReLU}\left(\text {GroupNorm}\left(\text {Conv}2{\text{d}}(\textbf{y})\right)\right)$$

(13)

Finally, the output convolution layer produces the final output $\textbf{y}$:

$$\textbf{y} = \text {Conv}2{\text{d}}_{\text {out}}(\textbf{y})$$

(14)

This structure ensures that the depth map generated by the DS sub-network is accurate and detailed, by incorporating multiple layers of upsampling, partial convolution, and normalization.

DW sub-network

The DW sub-network applies the Laplacian operator during backpropagation to perform edge detection on the gradients, retaining only the gradients at edge regions while suppressing noise in non-edge regions. The process involves several steps:

Given the gradients from downstream $\textbf{g}_{out}$, we first compute their absolute values and generate a mask $\textbf{M}$ that marks all non-zero gradient positions:

$$\textbf{M} = |\textbf{g}_{out}| > 0$$

(15)

The shape of the mask needs to be expanded for subsequent convolution operations, denoted as:

$$\textbf{M}_{\text {expanded}} \in \mathbb {R}^{N \times 1 \times H \times W}$$

(16)

The Laplacian operator $\textbf{L}$ is applied to the expanded mask to detect edges. We use a discrete Laplacian operator for convolution:

$$\textbf{C} = \textbf{L}\left(\textbf{M}_{\text {expanded}}\right)$$

(17)

The result of the Laplacian convolution $\textbf{C}$ is checked to detect edges. If the absolute value is less than a small constant $\epsilon$, it is considered an edge point:

$$\textbf{E} = |\textbf{C}| < \epsilon$$

(18)

Here, $\textbf{E}$ is a binary mask, $\textbf{E} \in \mathbb {R}^{N \times H \times W}$, and it needs to be adjusted to remove extra dimensions.

Finally, using the edge mask $\textbf{E}$, we clip the gradients to retain only the gradients at edge positions:

$$\textbf{g}_{in} = \textbf{E} \odot \textbf{g}_{out}$$

(19)

where $\odot$ represents element-wise multiplication. This process helps suppress gradient noise in non-edge regions during training by detecting edges in the gradients and retaining only the gradients at edge locations.

DT sub-network

The DT sub-network is a neural network model designed for feature transformation, primarily consisting of an optional MLP part and a progressively expanding transposed convolution network. It is suitable for mapping input data to a specified output space and enhancing the model’s expressive capability using activation functions. The detailed implementation is as follows:

Given input $\textbf{x}$ passing through multiple fully connected layers with LeakyReLU activations:

$$\textbf{h}_{MLP} = \text {LeakyReLU}(0.2, \textbf{W}_i \textbf{h}_{MLP} + \textbf{b}_i)$$

(20)

In the upsampling part, the initial upsampling uses transposed convolution (ConvTranspose2d) and ReLU activation:

$${\mathbf{h}}_{{upsample}} = {\text{ReLU}}\left( {{\text{ConvTranspose}}2{\text{d}}\left( {{\mathbf{h}}_{{MLP}} } \right)} \right)$$

(21)

After the initial upsampling, there are five upsampling blocks. Each block contains transposed convolution (ConvTranspose2d), GroupNorm, and ReLU activation:

$$\textbf{h}_{upsample} = \text {ReLU}\left(\text {GroupNorm}\left(\text {ConvTranspose}2{\text{d}}(\textbf{h}_{upsample})\right)\right)$$

(22)

The final step includes upsampling, convolution (Conv2d), GroupNorm, and ReLU activation:

$$\textbf{h}_{\text {upsample}} = \text {Upsample}(\text {scale}{\_}\text{factor} = 2)$$

(23)

$$\textbf{h}_{upsample} = \text {Conv}2{\text{d}}(\textbf{h}_{upsample})$$

(24)

$$\textbf{y} = \text {activation}(\textbf{h}_{upsample})$$

(25)

By following this process, the DT sub-network effectively transforms the input features into the desired output space, enhancing the expressive power of the model through carefully structured layers and activations.

Loss function

Reconstruction loss

The reconstruction loss for each sample measures the difference between the proxy image ${I_0}$ output by StyleGAN2 and the reconstructed image ${I_w}$. Let ${w}$ represent the randomly sampled code, and the image generated by StyleGAN2’s teacher model is given by ${w} = G2D(w)$. The rendered image is ${I_w} = R(A, S, T, V, L)$, where ${R}$ is the renderer with five parameters: ${W, S, T, V, L}$. The reconstruction loss comprises the L1 loss and the perceptual loss, with the L1 loss representing pixel-level differences between the original image ${I_w}$ and the reconstructed image ${\hat{I}_w}$. The perceptual loss, ${L_{\text {perc}}}$, multiplied by the weight parameter $\lambda _{\text {perc}}$, captures higher-level feature differences.

$$\mathcal {L}_{\text{rec}} = \Vert I_w - \hat{I}_w\Vert _1 + \lambda _{\text{perc}} \mathcal {L}_{\text{perc}}\left(I_w, \hat{I}_w\right)$$

(26)

where ${N}$ is the total number of pixels, and ${I_w[i]}$ and ${\hat{I}_w[i]}$ are the values of the original and reconstructed images at pixel ${i}$, respectively. The reconstruction loss ${L_{\text {rec}}}$ combines low-level pixel differences and high-level perceptual differences to more comprehensively assess the quality of the reconstructed image. By adjusting the value of $\lambda_{\text {perc}}$, the contributions of these two losses to the total loss can be controlled.

To further constrain the model’s output, we add several regularization losses, including identity variance regularization loss and albedo map regularization loss.

Identity variance regularization loss

Assuming that identity should not change under viewpoint perturbation, we achieve this through identity variance regularization loss:

$$\mathcal {L}_{\text{idt}} = \Vert f(I_{w0}) - f(I_w')\Vert _2^2$$

(27)

where ${I_{w0}}$ is the texture image, ${I_w'}$ is the transformed image, and ${f}$ is a pre-trained face recognition network. Since the recognition network may not have pose invariance, this regularization is applied only to images within a certain rotation range.

Albedo map regularization loss

To better utilize shadow information, we regularize the albedo map, obtained by removing illumination from standard Terracotta Warrior facial images:

$$\mathcal {L}_{\text{regA}} = \Vert KA\Vert _*$$

(28)

where ${KA} \in \mathbb {R}^{B \times HW}$ is the albedo matrix, composed of filtered and vectorized albedo maps, and $\Vert \cdot \Vert_*$ denotes the nuclear norm. The Laplacian kernel is used to filter the grayscale albedo map, retaining only high-frequency information. The nuclear norm acts as a soft approximation for low-rank regularization, encouraging sample consistency while ensuring that different albedo maps within a batch have small Laplacian values.

Total loss function

Combining these losses, the total loss function for our network model training is:

$$\mathcal {L}_{\text{total}} = \mathcal {L}_{\text{rec}} + \lambda _{\text{idt}} \mathcal {L}_{\text{idt}} + \lambda _{\text{regA}} \mathcal {L}_{\text{regA}}$$

(29)

The weight parameters $\lambda _{\text{idt}}$ and $\lambda _{\text{regA}}$ control the contributions of the respective regularization terms to the total loss. This combined loss aims to improve the overall quality of the generated image while preserving identity and detailed features, leveraging shadow information to ensure more consistent and reliable albedo maps.

Experiments

Implementation details

All modules in this study are implemented using PyTorch 1.8, ensuring system stability and scalability. For rendering techniques, the mesh rasterizer from PyTorch3D is utilized for differentiable rendering. This technique allows fine-tuning of images during training, enhancing the precision and effectiveness of the generative model. The training phase employs a reimplementation of the StyleGAN2 model in PyTorch, specifically trained on the Terracotta Warriors dataset. StyleGAN2 is recognized as one of the leading models in generative adversarial networks, capable of producing high-quality and high-resolution images. To optimize the training data, the MTCNN face detector is used to crop images and resize them to 256 × 256 pixels. This process ensures the consistency and quality of the training images, improving the efficiency and effectiveness of model training. In the second phase of training, the pre-trained 2D StyleGAN2 model is used to further train the generator of the teacher-student 3D model. This approach enables the model to better learn complex 3D structures and textures, enhancing the realism and detail of the generated images. Additionally, we trained the 3D generator for 30,000 steps. For the 3D renderer configuration, the field of view was set to 10°, a parameter that facilitates the generation of more detailed and realistic images. The depth map’s output range was confined to 0.9 to 1.1 to ensure the precision of depth information and minimize potential errors. Furthermore, the rotation and translation ranges of the viewpoint were normalized to ($-60^{\circ }$, $60^{\circ }$) and (− 0.1, 0.1), respectively, to simulate perspective variations in real-world scenarios. The lighting coefficients were normalized to (0,1) to maintain consistent lighting conditions during training.

Datasets

Our method was tested on three datasets to verify its broad applicability and effectiveness: the CelebA dataset [52], the cat dataset [53], and the Terracotta Warriors dataset. These datasets represent three distinct application scenarios: human faces, animals, and historical artifacts. The CelebA dataset comprises 200,000 real human face images, each annotated with bounding boxes. These images cover a wide range of poses, expressions, and lighting conditions, making it a standard benchmark in the fields of face recognition and detection. The cat dataset includes 12,000 images of cats with bounding box annotations. Each image is cropped to ensure the bounding box accurately encloses the cat’s head, making it suitable for pet image processing and animal recognition research. The Terracotta Warriors dataset consists of 300 images of the heads of excavated Terracotta Warriors. These images capture the authentic appearance and details of the Terracotta Warriors, providing valuable data for the digital preservation and study of historical artifacts.

Qualitative experimental results

The qualitative results of our method, DIGAN, are illustrated in Fig. 2 These results display the outcomes obtained through training with StyleGAN2, and the 3D shapes generated using our method. These results clearly demonstrate the superiority of our approach in generating 3D shapes. The final row presents side views of the generated 3D shapes, showcasing the facial rotation effects from different angles, and highlighting our method’s ability to capture and reproduce facial details accurately. By leveraging self-predictive transformation mapping, our method maintains dynamic stability for foreground objects while keeping the background relatively static. This characteristic ensures the stability and consistency of the generated 3D shapes between the foreground and background. Our 3D generator not only produces realistic 3D human faces but also extends to other objects, such as cats and Terracotta Warriors. This demonstrates the generality and efficiency of our generator across different subjects, maintaining high-quality shape recovery even with extensive pose variations. In the generated 3D shapes, detailed features are well-preserved. Whether it is the wrinkles on human faces or the intricate details on the edges of Terracotta Warriors, our method captures and presents these details accurately, showing its precision and accuracy in handling fine details. Our method exhibits outstanding capability in generating high-quality 3D shapes and accurately preserving and restoring details during pose variations. By utilizing advanced self-predictive transformation mapping, our approach ensures dynamic stability between foreground and background and demonstrates broad applicability and detail-capturing ability across various objects. These features make our 3D generation method highly practical and valuable for a wide range of application scenarios. Figure 3 presents sample images generated under varying lighting conditions, visually illustrating the influence of light sources on the generated images. These samples demonstrate the flexibility and robustness of our method in handling lighting variations. By generating normal maps of the shapes, our approach can create images under any lighting conditions. The normal maps not only capture subtle shape variations on the Terracotta Warriors’ surfaces but also retain detailed information of the original surfaces, ensuring the generated images maintain a high degree of realism under different lighting conditions. The second row of Fig. 3 displays shadow maps created under lighting, which are used to re-illuminate the images in the first row. This technique allows us to simulate shadow effects in real lighting environments, making the final generated images more realistic and natural. Our method showcases the ability to generate high-quality facial images of the Terracotta Warriors under any lighting condition, even for images not present in the training dataset. This indicates that our method has high generalization and adaptability, capable of handling complex and variable lighting environments, and provides reliable technical support for various application scenarios.

Fig. 2

The results of the Terracotta Warriors’ faces generated by StyleGAN2 and the 3D shape results of the Terracotta Warriors’ faces generated by our method are shown, including the front and side views

Fig. 3

Shows examples of the Terracotta Warriors’ faces generated using different lighting. The first row shows the Terracotta Warriors’ faces under any lighting conditions, and the second row shows the shadow maps generated by the lighting

Quantitative experimental results

We evaluated the visual fidelity of the generated images using the Fréchet Inception Distance (FID). DIGAN was thoroughly compared with other mainstream Generative Adversarial Network (GAN) models, such as StyleGAN [52], APA [54], ADA [55], Few-Diffusion [56], FastGAN [57], and FreGAN [58]. Table 1 shows that DIGAN demonstrates competitive or superior FID scores on challenging datasets such as CelebA, cat, and Terracotta Warriors. When trained on different datasets with the same set of hyperparameters, DIGAN’s architecture remained stable, consistently generating images with high visual fidelity. Thus, DIGAN exhibited exceptional visual fidelity and robustness across multiple datasets, underscoring its superior performance in the field of image generation.

Table 1 Comparison of FID (lower is better) of different methods on three datasets

This study provides a detailed quantitative comparison of DIGAN with three renowned open-source methods: HoloGAN [37], DiscoFaceGAN [35], CONFIGNet [59], and LiftGAN [60] as shown in Table 2 to evaluate their performance in face generation and identity preservation. For each method, 1,000 faces were randomly generated and output images were produced in multiple specified poses. The ArcFace model was used to compute the similarity between generated frontal and non-frontal faces, assessing each method’s ability to preserve identity information after facial rotation. All methods excelled in the visual quality of the generated images, producing facial images with high realism and clarity. However, DiscoFaceGAN performed relatively poorly in preserving identity information after facial rotation, indicating the challenge of maintaining identity consistency with varying facial poses. Without any supervision, DIGAN exhibited superior identity preservation capabilities compared to most methods. Within the framework of 3D controllable generation models, DIGAN’s average identity similarity across different rotation angles was significantly better than other methods, demonstrating its superior performance in maintaining identity consistency. Overall, DIGAN showcased exceptional performance in face generation and identity preservation, making it a powerful tool for relevant applications and research.

Table 2 Quantitative comparison of the proposed DIGAN with state-of-the-art works

Ablation studies

In the ablation study, we thoroughly examined the impact of various loss functions on the generated results in our model, with a particular focus on their performance in terms of image quality and realism. Through experimental comparisons, we revealed the roles and effects of different loss functions in the generative model. To evaluate the importance of these loss functions, we conducted ablation experiments by sequentially removing the symmetric reconstruction loss, albedo consistency loss, and identity regularization loss, and retraining the generative model. This approach allowed us to observe the impact of each loss function on the final image quality and feature preservation. During the evaluation process, we employed both quantitative and qualitative methods. Quantitative evaluation included using specific metrics, such as FID scores, to measure the quality of the generated images. Qualitative evaluation involved visual inspection and comparison of the generated image details to assess their realism and consistency. The results of these evaluations are presented in Table 3 and Fig. 4 providing clear data support. To verify the generality and robustness of the experiments, we conducted multiple ablation studies using a specific Terracotta Warriors dataset. These experiments provided a more comprehensive understanding of the specific impact of different loss functions on model performance and ensured that the results have broad applicability.

Table 3 Quantitative evaluation of the ablation study

Fig. 4

Qualitative results of ablation study

We remove the symmetric reconstruction loss (i.e., wo_flip), the identity regularization loss (i.e., wo_idt), and the albedo consistency loss (i.e., wo_rega), respectively. Experiments showed that when the symmetric reconstruction loss was removed, the generative model struggled to produce reasonable facial shapes, often resulting in inconsistent and unnatural shapes. However, the model still managed to partially achieve facial rotation functionality, indicating the crucial role of symmetric reconstruction loss in shape generation. Further experiments revealed that albedo consistency loss and identity regularization loss had similar effects in regularizing the output shapes. Both loss functions helped the model generate more consistent and believable image shapes, thereby improving the quality and credibility of the generated images. By adding additional regularization losses to the output of StyleGAN2, we can significantly enhance the plausibility and reasonableness of the generated facial shapes. This approach not only improves the visual quality of the generated images but also boosts the model’s performance in complex shape generation tasks.

Conclusion

This paper explores the application of Generative Adversarial Networks (GANs) in generating 3D Terracotta Warrior faces, emphasizing their significance in cultural heritage preservation and restoration, digital archiving and exhibition, academic research, education, and the cultural and creative industries. Using GAN technology, particularly StyleGAN and its variants, allows for the generation of high-quality and diverse 3D Terracotta Warrior images. To address these challenges, this paper proposes an innovative distillation model, DIGAN, for generating 3D Terracotta Warrior faces. By distilling knowledge from StyleGAN2, we train a novel 3D generative network. The 3D generator modularly decomposes the generation process, encompassing 3D modules for texture, shape, lighting, and pose, and ultimately renders 2D Terracotta Warrior images. The model enhances 3D shape learning through symmetric constraints and multi-view data, resulting in images with a more pronounced three-dimensional appearance and realism. The proposed method can generate high-quality 3D information for faces and Terracotta Warrior images, allowing precise control over pose and lighting. Our approach has made significant progress in generating high-quality 3D Terracotta Warrior faces, providing new technological means for cultural heritage preservation and restoration, and advancing digital archiving and exhibition. However, our work has certain limitations. Since our model is a generative method, we do not have a real 3D model for its output, so it is difficult to directly evaluate the various performance of the 3D output. Therefore, we will carry out related work on the reconstruction of the 3D Terracotta Warriors in the future to obtain a real 3D model.