Deep transfer learning for visual analysis and attribution of paintings by Raphael

Abstract

Visual analysis and authentication of artworks are challenging tasks central to art history and criticism. This preliminary study presents a computational tool for scholars examining and authenticating a restricted class of paintings, with a specific focus on the paintings of Raffaello Sanzio da Urbino, more popularly known as Raphael. We applied transfer learning to the ResNet50 deep neural network for feature extraction and used a support vector machine (SVM) binary classifier in support of authentication. Edge detection and analysis algorithms, considered to be crucial for capturing the essence of Raphael’s artistic style, including the brushwork signatures, were also integrated and are used as an authentication tool. The machine learning approach we have developed demonstrates an accuracy of 98% in image-based classification tasks during validation using a test set of well known and authentic paintings by Raphael. Of course, a full authentication protocol relies on provenance, history, material studies, iconography, studies of a work’s condition, and more. Our work, then, contributes to just a portion of a full authentication protocol. Our findings suggest that machine learning methods, properly employed by experts aware of context, may enhance and expand traditional visual analysis for problems in art authentication.

Introduction

Art attribution and authentication are complex and often vexing tasks but of great consequence to academic art history, art criticism, and, of course, the commercial art market. Attribution protocols typically include a study of provenance (the documentary record of sales, ownership, and display of works), material studies (chemical and spectral analysis of pigments, supports, etc.), iconography (study of the items, costumes, material culture, and so on depicted in a work), study of the condition of the work, analysis of candidate authors’ careers and oeuvre (styles at different periods of career), derivative or ancillary works (preparatory studies, cartoons, copies, x-ray, hyperspectral images that reveal spectral information, penitent, and other physical characteristics, as well as yielding information of previous states of a work), and connoisseurship (close visual analysis of the composition, style, brush strokes, shading, etc.). In this paper, we consider just this last component—connoisseurship—and present a computational machine learning approach to the visual analysis of paintings as a tool in service of connoisseurship. We demonstrate such computational approaches to the authentication of paintings by Raphael, a seminal and influential artist of the High Renaissance whose work is frequently the subject of authentication debates in large part because he had a thriving atelier of students and assistants.

Raffaello Sanzio da Urbino, commonly known as Raphael, was an Italian painter and architect of the High Renaissance [1, 2]. Born in 1483 in Urbino, Raphael is celebrated for the perfection and grace of his art. His work is often placed alongside those of Leonardo da Vinci and Michelangelo as one of the trinity of great masters of the Renaissance. Raphael’s oeuvre spans various subjects, from religious themes to portraits and frescoes that depict classical narratives. His most renowned frescoes are in the Vatican’s Stanza della Segnatura, which houses his celebrated School of Athens [3]. This masterpiece features prominent philosophers from various periods engaged in dialogue, representative of the Renaissance’s rebirth of classical knowledge.

Given the significant consequences to art scholarship and the commercial world of art forgery, mis-attribution, and the natural ageing of artworks, establishing a painting’s authenticity is of paramount importance [4, 5]. Historically, the authentication of artworks relied heavily on manual examinations, as well as provenance research, expert opinions, and, more recently, radiographic and spectroscopic methodologies [6, 7]. Such methods have played a significant role in understanding the choice of materials, deciphering pentimenti and underdrawings, and assessing the age of paintings.

The integration of machine learning digital tools into art analysis promises to offer a range of benefits, including authentication [8]. When carefully employed, such methods may provide an analytical stance highlighting clarity and precision of analysis, one that can, in principle, incorporate more art information than available to any scholar without such computational tools. Computational approaches present a structured method to categorise extensive art collections based on intricate stylistic details, including brushstroke pattern analysis [9].

Prior work

Prior studies have also employed traditional machine learning techniques such as Support Vector Machines (SVMs) with feature sets like Dense Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), and Structural Similarity (SSIM) to achieve reasonable classification accuracies [8, 10]. Machine learning assisted computational algorithms are becoming more common for the analysis of brushwork, pigment distributions, and support structures [11]. In this context, techniques like principal component analysis, clustering, and neural networks have yielded promising results [12].

Recent advancements have also exploited deep learning, particularly the utility of transfer learning, and attained success in classifying artworks by a range of artists [13, 14]. Convolutional Neural Networks (CNNs) were developed by mimicking certain properties of biological neural networks, such as those occurring in the human visual cortex. Neurons in the visual cortex process images through a sequence of hierarchical layers, transforming images from edge boundaries to complex features and feature groupings [15, 16]. CNNs aim to mimic this architecture through convolutional layers, enabling them to classify images based on low- to high-level features ranging from colour and texture to semantic meaning and artistic style [17]. As a result, especially in the last decade, we have witnessed a shift towards machine learning, notably deep learning, as a potential tool for art analysis.

Early efforts in deep learning, exemplified by Shamir et al., [18] employed various types of image descriptors to identify the authors of artworks in a variety of styles. These studies were augmented by the use of CNNs as feature extractors, a strategy also tested by Bar et al., who used CNN models pre-trained on the ImageNet dataset along with lower-level descriptors like PiCoDes to develop a robust system for painter recognition [19]. Saleh et al., [20] extended the methodology by integrating multiple feature extractors to create models capable of identifying the artist and discerning the artwork’s style and genre. Similar work using CNNs for art analysis was conducted by Brachmann et al., [21] in which they analysed traditional Western, Islamic, and Chinese art and other types of images. Further, Karayev et al., [22] addressed a broader range of images that include photographs, aiming for a more generic style classification. Similarly, the work by Bhushan et al., [11] demonstrated the ability of deep neural networks to analyse the style and authenticity of artworks. Yang and Fan explored the use of deep facial feature analysis to analyse the ancient Thangka Buddha face [23]. Gatys et al., [24] developed an innovative technique using CNNs to translate the stylistic features of a painting into a vector format and quantified into what is known as the Gram matrix. More recently, similar to the work of Yang and Fan [23] Ugail et al., [25], carried out a face focused study that explored the use of face matching through the analysis of deep features for analysing the work of Raphael. Other deep feature assisted analyses include the utilisation of knowledge graphs [26] of more popular adversarial networks for analysis and even the generation of artwork [27]. Furthermore, there are studies focusing on painting classification using multi-task deep learning and a large-scale database for computational painting categorisation [28].

While machine learning has made initial strides in the field of art analysis, many challenges must be addressed before it will be accepted by the wider scholarly art community [29]. One of the major challenges is the scarcity of high-quality data required for training robust multi-class classification models [30]. This data scarcity also complicates distinguishing between an artist’s natural stylistic evolution and any anomalies that may appear in their work. To mitigate this, one of the primary objectives of our research is to demonstrate how smaller, artist-specific models can be constructed to analyse and authenticate individual paintings more accurately. Our study builds on this fundamental idea in that we incorporate a deep transfer learning approach to construct artist specific models, specifically here, one for Raphael. We use the ResNet50 model for in-depth feature extraction coupled with SVM for efficient classification [31, 32]. In addition, edge detection algorithms are integrated into the framework to enhance the authentication process.

Overview of work presented here

In this paper, we discuss a deep transfer learning based computational approach aimed at analysing and authenticating the works of the High Renaissance artist Raphael. Our methodology, outlined in Sect. Methodology, utilises a three-tiered approach beginning with feature extraction from a pre-defined dataset using the pre-trained ResNet50 model. These features are then used to train an SVM classifier. The effectiveness of this model is evaluated through test subsets of data, leading to final classification and feature evaluation using SVM and ResNet50. We incorporate edge detection techniques, including operators like Canny, Sobel, Laplacian, and Scharr, to help estimate the authenticity of the artwork. Section Results provides empirical evidence demonstrating the methodology’s utility in authenticating Raphael’s paintings, while Sect. Discussion delves into intriguing cases requiring further exploration. The paper describes some conclusions and future directions in Sect. Conclusion.

Methodology

Our methodology is based on a structured, three-fold procedure for the analysis and authentication of the works of Raphael. The initial phase involves a pre-trained ResNet50 architecture, with the omission of its top layers, to derive features from a predefined dataset of art images. Subsequently, this dataset is partitioned into training and test subsets, with the SVM performing authentication. Upon completion of the training, the model’s performance is assessed using the test subset. In the testing segment, features of the test images are revealed via the ResNet50 model, which then undergoes class prediction using the SVM. The concluding segment accentuates the role of edge detection in determining authenticity. Operators, including Canny, Sobel, Laplacian, and Scharr are employed to capture the edge features of the test images. These features are then normalised to produce weights. We compare these extracted features to a weighted average derived from a reference image set and integrate the results with the SVM’s outputs to yield the final estimation of the style and likely authorship. This integrating approach, which marries deep learning with edge detection, offers a comprehensive framework for artwork evaluation.

Transfer learning with ResNet50

Transfer learning is a machine learning technique where a pre-trained model, typically developed for a benchmark task, is repurposed for a new, related task. The core principle behind transfer learning is the observation that knowledge gained during training on one task can assist performance on a related task. This method is particularly valuable in scenarios where there is a paucity of data for the new task, a situation common in art analysis due to the uniqueness and rarity of certain pieces.

Hence, transfer learning leverages a model pre-trained on a source task to improve performance on a related target task [33]. Formally, given a source domain ${\mathcal {D}}_S$ and learning task ${\mathcal {T}}_S$, transfer learning guides the learning of the target predictive function $f(\cdot )$ in domain ${\mathcal {D}}_T$ for task ${\mathcal {T}}_T$ using the knowledge in ${\mathcal {D}}_S$ and ${\mathcal {T}}_S$, where ${\mathcal {D}}_S \ne {\mathcal {D}}_T$ or ${\mathcal {T}}_S \ne {\mathcal {T}}_T$.

In essence, transfer learning involves two main stages. In the first stage, the pre-trained model acts as a feature extractor. Given that the earlier layers of a deep neural network generally capture universal features (like variation in colours, hues, tonal values, edges and textures), these layers can be frozen and used to transform input data into a more informative representation. The second stage involves fine-tuning of the network weights. Depending on the similarity between the source and target tasks, certain deeper layers of the model can be fine-tuned to better capture the specifics of the new task. This training step involves unfreezing some of the model’s layers and retraining them on the new dataset.

ResNet (Residual Network) included a novel architectural concept termed as “residual blocks” to facilitate the training of much deeper neural networks while avoiding problems due to vanishing error gradients [31]. The defining feature of a residual block is its skip connection, which allows gradients to bypass layers during backpropagation. Each residual block can be represented by,

$$F(x) = h(x) + H(x),$$

(1)

where ${\mathcal {F}}(x)$ is the residual mapping to be learned, h(x) is the identity mapping, and ${\mathcal {H}}(x)$ represents the stacked non-linear layers of the block.

In ResNet50, the architecture is mathematically structured as, $\begin{gathered}{x_0}\to {\text{Initial\,Convolutional\,Layer\,with\,Max-Pooling}}, \\ {x_i}\to {\text{Residual\,Block\,of\,Type\,I}} \quad {for }\,i = 1, \ldots , 3, \\ {x_j}\to {\text{Residual\,Block\,of\,Type\,II}} \quad {for }\,j = 4, \ldots , 7, \\ {x_k}\to {\text{Residual\,Block\,of\,Type\,III}} \quad {for }\,k = 8, \ldots , 13, \\ {x_l}\to {\text{Residual\,Block\,of\,Type\,IV}} \quad {for }\,l = 14, \ldots , 16 ,\\ x_{17}\to {\text{Average\,Pooling\,Layer,}} \\ x_{18}\to {\text{Final\,Fully\,Connected\,Layer}},\end{gathered}$ where $x_i$, $x_j$, $x_k$, and $x_l$ are the outputs after the corresponding layers or blocks.

To employ ResNet50 for binary classification, the following steps can be mathematically formulated:

• 1. Model Initialisation: Load ResNet50 pre-trained on ImageNet. Let $W_{pre}$ represent the pre-trained weights of ResNet50.

• 2. Model Modification: Remove the top layer and append a new layer,

  $${\hat{y}} = \sigma (W_{new} \cdot x_{17} + b),$$

  (2)

  where ${\hat{y}}$ is the predicted output, $\sigma$ is the sigmoid activation function, $W_{new}$ are the weights of the new layer, $x_{17}$ is the output from the last average pooling layer, and b is the scalar bias.

• 3. Feature Extraction: Using $W_{pre}$, extract features F from the dataset,

  $$F = W_{pre} *P,$$

  (3)

  where $*$ represents the convolution operation, and P represents the dataset.

• 4. Training: For model training, we use the following formulation such that,

  $${\mathcal {L}}({\hat{y}}, y)= {} -y \log ({\hat{y}}) - (1 - y) \log (1 - {\hat{y}}),$$

  (4)
  $$W_{new}^{(t+1)}= {} W_{new}^{(t)} - \alpha \nabla {\mathcal {L}},$$

  (5)

  where ${\mathcal {L}}({\hat{y}}, y)$ is the binary cross-entropy loss, y is the true label, and $\alpha$ is the learning rate.

• 5. Evaluation: The resulting model is then evaluated using,

  $$\text {Accuracy} = \frac{1}{N} \sum _{i=1}^{N} {\mathbb {I}}({\hat{y}}_i = y_i),$$

  (6)

  where N is the number of samples, and ${\mathbb {I}}(\cdot )$ is the indicator function.

Thus, in essence, our approach involves transfer learning in which the ResNet50 model, which was originally trained on the ImageNet dataset for a wide array of object recognition tasks, is being repurposed for the specific task of artwork authentication. In this case, the top layers of the ResNet50 model are omitted, and the remaining layers are used to extract feature vectors from images of the images in a dataset. These feature vectors are then used to train an SVM classifier. Here, the transfer learning approach takes advantage of the rich, hierarchical features of a deep neural network learned on a large, general dataset to improve performance on a different, more specific task.

Image data, model training, and validation

Our primary objective was to devise a machine learning model to classify paintings as autograph Raphael or not. This binary classification task was established using two distinct sets of images: a collection of authenticated Raphael paintings (labelled as Class 0 or “Raphael”) and a mixed array of paintings from artists like Rembrandt, Peter Lely, van Dyck (labelled as Class 1 or “Not Raphael”). Of course, the success of our system and its value in authentication studies of a particular candidate author depend upon the artists whose works represent the Class 1 data. We did not explore a wide range of candidate data.

The initial step involved getting a collection of images, as discussed above. The essential tasks in this process are,

• Design a function to load and preprocess the image data.

• Develop another function to augment the data, providing variations of the data to increase the sample size of the training set.

• Develop another function to distil pertinent features from the images through the ResNet50 model.

• Utilise the visual feature vectors to train the SVM classifier for use in classifying, with a probability score, of an unknown painting as “Raphael” or “Not Raphael” based on image data.

We constructed our dataset using two different groups of images. As mentioned, the first consists of verified paintings by Raphael, labelled as Class 0 or “Raphael,” while the second features artworks from other renowned artists like Rembrandt, Peter Lely, and van Dyck, labelled as Class 1 or “Not Raphael.” Given the low number of authenticated Raphael paintings available, our dataset for Class 0 was necessarily limited, but we included 49 images. Similarly, for Class 1, we utilised 49 images from other well-known artists, chosen specifically because their styles bear some resemblance to Raphael’s yet can be reliably distinguished by art historians. This careful selection ensures that the deep features extracted by our model are both representative and distinct for each class.

Data augmentation plays a crucial role in enhancing the performance and generalisation capability of the model. Specifically, the following data augmentation techniques are applied to the training images.

• Rotation Range: Each image is randomly rotated within a range of $-20^\circ$ to $+20^\circ$.

• Width and Height Shift: Random width and height shifts are applied to the images, with a range of up to 0.2 times the original dimensions.

• Horizontal Flip: With a probability of 0.5, each image is flipped horizontally.

• Fill Mode: To address the empty pixels created by the above transformations, the “nearest” fill mode (using a $3 \times 3$ convolution filter) is applied to fill in missing pixel values.

By employing this data augmentation strategy, the model is trained on a more diverse set of images, which reduces errors due to overfitting and enhances the model’s ability to generalise to new data.

For the classification task, the SVM classifier was employed [34]. This method was chosen for its effectiveness in handling high-dimensional spaces. We utilised the SVM with a linear kernel, which is adept at finding an optimal hyperplane, effectively distinguishing between the two image classes. The model was also configured to calculate class probabilities, which are essential for certain decision-making scenarios. Furthermore, a fixed random seed was set to ensure consistent and reproducible results across different runs.

We evaluated the performance of the SVM’s on a validation subset. The SVM was re-trained for optimal outcomes, encompassing the full image data and combining both the training and test sets. This approach was taken to produce a more refined model, ensuring it was robust against potential real-world variability. Once training was complete, the finalised SVM model was saved for future deployments.

The devised classifier demonstrated an impressive test accuracy of $98\%$. The detailed metrics for the performance of the resulting model are given in Table 1. These results demonstrate the transfer learned SVM model possesses sufficient robustness and accuracy for it to be utilised in practical applications.

Table 1 Results from the test dataset on the classification task

Authenticity analysis using edge features

Brushstrokes and other marks are often highly informative in humanistic and authentication studies “by eye.” For this reason, computational edge detection and analysis play a key role in understanding images, especially in revealing intricate details that are integral to the evaluation of artworks. The brushstrokes, for example, which carry the essence of an artist’s technique, often manifest as edges in an image. Therefore, capturing these subtle cues through edge detection can provide insights into the artist’s unique style and potentially aid in verifying the authenticity of a painting. Below, we discuss the performance of each edge detection analysis operator and how it is tested.

• Canny Edge Detection: Proposed by John Canny in 1986 [35], this technique is renowned for its multi-stage process to detect the visual edges of a wide range of scales. A Canny edge detector starts with a Gaussian filter for image smoothing and then calculates the image gradient to highlight regions with sharp intensity variations. It utilises non-maximum suppression to thin out edge candidates and double thresholding to categorise potential edges. It effectively identifies true edges and filters out noise, making it useful for capturing subtle brushstroke style in paintings.

• Sobel Operator: This operator centres around convolving the image with a pair of 3x3 matrices [36]. It accentuates regions of rapid intensity variation and computes the gradient magnitude of image intensity for each pixel. It also provides a distinct representation that marks boundaries. It is particularly useful for delineating variations caused by sharp brushstrokes, thereby revealing the artist’s technique.

• Laplacian of Gaussian (LoG): This is a two-step process that starts with applying a Gaussian smoothing filter to the image and then determining the Laplacian [37]. It highlights regions of rapid intensity changes and is commonly used for blob detection. The technique can effectively uncover fine details in artworks, including intricate brushwork patterns.

• Scharr Operator: This operator is often considered an optimised version of the Sobel operator. It provides enhanced edge gradient detection and captures finer details compared to the Sobel operator [38]. It is especially valuable for detecting delicate brushstrokes or nuanced painting techniques.

By combining these edge detection techniques, we harness a comprehensive representation of the painting’s low-level visual features, focusing especially on the unique brushstroke patterns that often reveal the author’s identity.

After assessing each edge detection technique individually, a combined approach was implemented to harness the strengths of each operator. Given the complementary nature of these techniques in identifying different types of edges and nuances in paintings, a weighted combination can better represent the overall edge information present in an artwork.

Mathematically, the combined edge image $E_{combined}$ can be represented as,

$$E_{combined} = w_{canny}E_{canny} + w_{sobel}E_{sobel} + w_{LoG}E_{LoG} + w_{scharr}E_{scharr},$$

(7)

where, $E_{canny}$, $E_{sobel}$, $E_{LoG}$, and $E_{scharr}$ are the edge images obtained from the Canny, Sobel, Laplacian of Gaussian, and Scharr operators respectively. $w_{canny}$, $w_{sobel}$, $w_{LoG}$, and $w_{scharr}$ represent the weights assigned to each edge detection technique, based on their effectiveness in capturing distinct features of Raphael’s brushstroke patterns. The individual values for the weights are found through experimentation on the dataset used for Class 0.

Once the edge features are computed, the standard deviation is calculated for every individual edge feature obtained from an image. The standard deviation serves as an effective metric to quantify the variability and intensity of edge features in the image. To ensure that the weighted combined edge detector optimally captures the characteristic edges for Raphael paintings, the threshold and weights ($w_{canny}$, $w_{sobel}$, $w_{LoG}$, $w_{scharr}$) were determined through experimentation and calibration using the 40 authenticated images in the training set for Class 0. This process guarantees that the most representative and recurring edge patterns across the authenticated paintings are captured, setting an accurate benchmark for further authenticity assessments.

Algorithm 1

Procedure for Painting Analysis and Verification

Algorithmic implementation

Our methodology, designed to authenticate artworks with a particular emphasis on those of Raphael, harnesses the power of computational techniques, specifically deep learning and edge detection. In the first phase, features are extracted from artworks using the ResNet50 architecture. This extracted data is then divided into training and test sets, which facilitate the training and evaluation of an SVM classifier, respectively. Once the SVM model is trained, it can predict the category of a test image. We employ edge detection to further bolster the validation process, extracting and comparing edge features from test artworks to reference sets. Based on the combined evidence from the SVM predictions and edge feature comparisons, a decision is made regarding the artwork’s authenticity. Below is a representation of how this algorithm can be implemented, as shown in Algorithm 1.

Results

To accurately determine whether a given painting belongs to Raphael, we have implemented a dual-layered verification process that employs both edge feature comparison and SVM model output. We initially relied solely on the deep features extracted through transfer learning from a pre-trained model, which appeared to be effective in identifying the distinctive artistic style attributed to Raphael.

However, we encountered challenges with this approach; the SVM model sometimes failed to discern imitations from authentic works. This led us to introduce an additional layer of verification using the edge features of the image. Through extensive experimentation with authenticated images of Raphael’s works, which serve as our training set, we discovered that there exists a narrow range of values for the combined edge features. This was then used as a threshold that reliably attributes a painting to Raphael.

By combining these two methods, we not only leverage deep features’ powerful style recognition capabilities but also introduce a fine-grained layer of scrutiny through the combined edge feature comparison. This hybrid methodology enhances the model’s robustness and accuracy, effectively differentiating genuine Raphael paintings from imitations. Therefore, a painting is only classified as likely to be by Raphael if it meets both the edge feature threshold, denoted by $\delta$, and the SVM model’s probability threshold, denoted by $\gamma$. This composite approach led to greater classification accuracy.

Fig. 1

Results for a section of The Sistine Madonna - an authenticated painting by Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael creator QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raphael_-_The_Sistine_Madonna_-_Google_Art_Project.jpg), “Raphael - The Sistine Madonna - Google Art Project”)

Fig. 2

Results for the painting of Diana Kirke, Late Countess of Oxford by Peter Lely. (Image reproduced under Wikimedia Commons public domain licence - Peter Lely artist QS:P170,Q161336 (https://commons.wikimedia.org/wiki/File:Peter_Lely_-_Diana_Kirke,_later_Countess_of_Oxford_-_Google_Art_Project.jpg), “Peter Lely - Diana Kirke, later Countess of Oxford - Google Art Project”)

Fig. 3

Results for the painting initially attributed to Innocenzo Francucci da Imola but suspected (and later confirmed) to be a Raphael. (Image reproduced by kind permission of National Trust for Scotland)

Table 2 Results from the test dataset on the classification task

Fig. 4

Results for the Lo Sposalizio or The Marriage of the Virgin, another authenticated work by Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_-_Spozalizio_-_Web_Gallery_of_Art.jpg), “Raffaello - Spozalizio - Web Gallery of Art”)

Fig. 5

Results for an imitation of Raphael’s style by Seward, one of the authors of this paper. (Image reproduced by kind permission of Steven Seward Portraits)

To demonstrate the predictive power of our algorithm, we selected several paintings that were not part of any of the training classes. These examples are shown in Figs. 1 to 5, and the quantitative results are tabulated in Table 2.

Figure 1 features a section of The Sistine Madonna, also known as Madonna di San Sisto, which is an authenticated oil painting by Raphael. The masterpiece was commissioned in 1512 by Pope Julius II for the church of San Sisto in Piacenza and is thought to have been completed between 1513 and 1514. It is considered one of Raphael’s final Madonna paintings. Our algorithm’s analysis corroborates that Raphael most likely created this artwork, thus confirming historical records.

Figure 2 displays Diana Kirke, Late Countess of Oxford, a painting by Peter Lely. Given the stylistic differences and time periods between Peter Lely and Raphael, our algorithm correctly determined that this painting is unlikely to be a work by Raphael.

In Fig. 3, we explore a particularly interesting case: a painting initially attributed to Innocenzo Francucci da Imola but suspected to be a Raphael. The artwork, titled The Virgin, was discovered in Haddo House, managed by the National Trust for Scotland. Previously obscured by discoloured varnish, it was dated to between 1505 and 1510 and had a paltry valuation of just $\pounds$20 in 1899. Given the potential for the painting to be a genuine Raphael, its worth from $\pounds$20 could have skyrocketed to around $\pounds$20 million. Historian Bendor Grosvenor’s reported underdrawings and unique artistic elements that strongly suggest that the painting is an original Raphael. Our algorithm supports these claims, indicating a high likelihood that it is indeed a work by Raphael.

Figure 4 shows Lo Sposalizio or The Marriage of the Virgin, another authenticated work by Raphael. Executed in 1504 for the Franciscan church of San Francesco in Città di Castello, the painting was eventually acquired by the Pinacoteca di Brera in 1806. Our algorithm confidently verified that Raphael indeed crafted this artwork.

Finally, Fig. 5 serves as a test for the algorithm’s capability to detect imitations. The painting in question is an imitation of Raphael’s style by Seward, one of the authors of this paper. Despite the visual similarity in terms of colour palette and brushstroke patterns, the intrinsic qualities differ substantially from Raphael’s signature techniques. Our algorithm effectively detected this discrepancy, suggesting that the painting is not a signature Raphael.

These examples demonstrate the robustness and efficacy of our algorithm, proving its utility in distinguishing between genuine Raphael artworks and imitations or works by different artists.

Fig. 6

From the sketch to the final painting on how a meticulous imitation of Raphael is created. (Images reproduced by kind permission of Steven Seward Portraits)

Fig. 7

Results for the Madonna della rosa painting - currently at the Museo del Prado, Madrid, which is solely attributed to Raphael. However, the SVM model suggests that it cannot be solely attributed to Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), “Raffaello Santi - Madonna della Rosa (Prado)”)

Fig. 8

Parts of the Madonna della rosa painting identified for individual analysis. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), “Raffaello Santi - Madonna della Rosa (Prado)”)

Fig. 9

Results for image part A shown in Fig. 8. This part of the painting is likely to have had significant input from someone other than Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), “Raffaello Santi - Madonna della Rosa (Prado)”)

Fig. 10

Results for image part B shown in Fig. 8. This part of the painting is likely to have had major input from Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), “Raffaello Santi - Madonna della Rosa (Prado)”)

Discussion

The deep transfer learning technique discussed in the paper is aimed at analysing and authenticating paintings, specifically focusing on works by the Renaissance artist Raphael. It employs a hybrid approach that combines deep learning and traditional machine learning. A pre-trained ResNet50 model is used for feature extraction, while an SVM is employed for classification. The ResNet50 model, initially trained on the ImageNet dataset, is adapted to process images of Raphael’s and other artists’paintings. These images are pre-processed, augmented to enhance the dataset’s robustness, and then split into training and test sets. The SVM is trained on the feature vectors extracted by ResNet50 and evaluated for its accuracy in classifying the artworks. This methodology exemplifies transfer learning, as it re-purposes a general deep learning model for a specific task-artwork authentication.

Our work presented here demonstrates the capacity of the algorithm to identify potential imitations of Raphael, as shown in the results for Fig. 5. Figure 6 shows from the sketch to the final painting how this meticulous imitation of Raphael was created. To do this, Raphael’s painting techniques were comprehensively studied, particularly focusing on the article on this subject by Faldi and Paolini [39], which details the techniques of Raphael. The original self-portrait by Raphael was painted on a wooden panel, whilst it was opted for a PVC panel for this imitation. However, both are similar in size. While the exact shade of Raphael’s priming remains unclear, it possibly ranged from off-white to light yellow. The artist’s traditional gesso technique typically exhibited a fine craquelure, which was not observed in high-resolution images. Raphael’s practice of using animal hide glue as a covering was not replicated in this attempt due to the different nature of the ground. The replication began with a light pencil sketch to capture the main proportions. Building on the initial sketch, the details were elaborated using a very fine brush with diluted black oil paint, incorporating a cross-hatching technique. In [39], they mention that Raphael occasionally used a pen, but the lines in the original were too fine to discern the tool used clearly. Subsequently, the entire artwork was painted in transparent earth tones to set the chiaroscuro. It was observed that the original had a less polished layer, which could be attributed to age or abrasion. The final stage involved painting the true colours over the dry earth tones, maintaining the visibility of the undertones and adhering to Raphael’s vertical brush stroke pattern. Emphasis was given to specific features such as the sharp edges of attire and the colours on the face, allowing some of the original sketch to show.

Fig. 11

Results for image part C shown in Fig. 8. This part of the painting is likely to have had major input from Raphael. This part of the painting is likely to have had major input from Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), “Raffaello Santi - Madonna della Rosa (Prado)”)

Fig. 12

Results for image part D shown in Fig. 8. This part of the painting is likely to have had major input from Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), “Raffaello Santi - Madonna della Rosa (Prado)”)

Fig. 13

Results for image part E shown in Fig. 8. This part of the painting is likely to have had major input from Raphael. (Image reproduced under Wikimedia Commons public domain licence - Raphael artist QS:P170,Q5597 (https://commons.wikimedia.org/wiki/File:Raffaello_Santi_-_Madonna_della_Rosa_(Prado).jpg), "Raffaello Santi - Madonna della Rosa (Prado)")

Table 3 Results from the test dataset on the classification task

One property of the technique we have discussed is its capability to analyse not just entire paintings but also isolated components or sections of them. This feature is particularly relevant given the historical context of a Renaissance artist’s workshop, which functioned as both a studio and a training school. Apprentices often assisted masters, contributing to backgrounds, still-life details, and secondary figures, particularly in large-scale commissions [40]. Our methodology could enable one to explore these details further. To illustrate, we examined the Madonna of the Rose (Madonna della rosa), a painting from 1518–1520 now housed in the Museo del Prado, Madrid. Initially, our algorithm suggested a probability of just 0.57 (as shown in Fig. 7) that the entire painting was the work of Raphael- a comparatively low figure. Intrigued by this, we analysed individual sections of the artwork as shown in Fig. 8. The results for these individual parts are shown in Figs. 9 - 13 and Table 3. Interestingly, the results raised questions about whether Raphael indeed painted the face of Joseph in the painting.

It is useful to note that some art historians have previously questioned the full attribution of this painting to Raphael alone, suggesting that his associate, Giulio Romano, might have had a hand in it [41]. Our machine learning analysis supports this academic discourse. The algorithmic assessment indicates a probability of 0.57 that the entire painting belongs to Raphael, which gives credence to the idea that other artists may have contributed to the painting. Consequently, our machine learning findings appear to corroborate the views of scholars who argue for a shared attribution of this painting involving artists other than Raphael alone.

It is important to point out that for this work we have chosen the ResNet50 which appears to be popular for the type of image analysis we have discussed [31, 33, 42]. Other variations of Resenet such as weights from ResNet101, ResNet152 or versions of Densnet and EfficientNet may work equally well. Our preliminary work in comparing Resnet50, Resnet101 and Resnet152 for the current does not appear to give any distinct advantages for using ResNet with higher filter sizes such as Resnet101 and Resnet152. However, this would form a topic for further investigation.

Similarly, the question of how the resolution of images used for training and testing affects the accuracy of results is important. In this work, we have the resolution of our training and testing images ranged between 330 x 330 pixels to 10,000 x 10,000 pixels. Experiments for measuring the impact of image resolution on the performance of deep learning suggest that higher image resolutions lead to better performance. In the work presented in [43], although, in a different context, they tested images ranging from the performance of accuracy on images ranging from 32 $\times$ 32 pixels to 512 $\times$ 512 pixels and found that the variation in accuracy for images between 256 x 256 pixels and 512 x 512 pixels is in the order of 0.4%. However, within the context of the analysis of portraits, the quantity and quality of images to be used for training and testing would be a topic for further investigation.

Finally, the question of extending the proposed method for multi-class classification is interesting. The essence of our approach for artwork verification is to utilise high level (deep) features to first pin down the ‘style’ of the artist and then use local features (through edge feature analysis) for detailed analysis of features that are specific to the artist. This last step provides a degree of assurance on the potential authenticity of the artwork. Thus, for multi-class classification (i.e., for classification of paintings involving multiple artists) one approach to extend this method would be to use transfer learning using a pre-trained model and then use a specific feature based model either sequentially or through the use of attention transformers.

Conclusion

This work offers a considered and effective approach to the challenge of artwork verification, with a particular focus on the works of the Renaissance artist Raphael. By employing a hybrid methodology combining deep learning for feature extraction and conventional machine learning for classification, the study effectively utilises a pre-trained ResNet50 model and an SVM classifier. Our proposed approach exhibits encouraging accuracy and serves as an example of transfer learning, where a model trained for a more generalised task is adapted for a specific case.

Of course, the potential applications of this work are not confined to the works of Raphael alone. The methodology can be readily modified to construct individual models for authenticating paintings by different artists, given that an adequate number of verified example paintings are available for training. This could be a useful resource for art historians and collectors alike, supplementing existing methods such as scholarly analysis, spectroscopic imaging, and dating techniques.

As advances continue to be made in machine learning and image processing technologies, this method has the potential to become part of an array of tools for artwork analysis and verification. It can operate in conjunction with other methods currently in use, including in-depth scrutiny by art historians and various advanced imaging techniques, thus contributing to a more thorough and dependable framework for artwork authentication and analysis. Future research may explore the application of this approach to a wider variety of artists and artistic styles and its integration with other computational and non-computational methods for more robust verification and authentication.