Unveiling the invisible: mathematical methods for restoring and interpreting illuminated manuscripts

The digital processing, analysis and archiving of databases and collections in the arts and humanities is becoming increasingly important. This is because of a myriad of possibilities that digitisation opens up that go well beyond the organisation and manipulation of the actual physical objects, allowing, for instance, the creation of digital databases that are searchable with respect to several parameters (keywords), the digital processing and analysis of objects that are non-destructive to the original object, and the application of automated algorithms for sorting newly found objects into existing digital databases by classifying them into pre-defined groups in the database. These possibilities go hand-in-hand with ever-growing advances in data science that are developing mathematical methodology for analysing and processing digital data. A large component of digital data in the arts and humanities is composed of digital images. Despite many developments of mathematical image analysis methods in applications like biomedicine, the physical sciences and various forms of engineering, the arts and humanities have been mostly overlooked as an application in need of bespoke mathematical image analysis methods. Still, a few examples in this context exist and encompass works on forgery detection [1], the digital restoration of paintings with the Ghent Altarpiece [2–7] and Van Gogh’s Field with Irises [8–10] being prominent examples in these efforts, the digitally guided restoration of frescoes as done for the Mantegna frescoes [11, 12] and the Neidhart frescoes [13, 14], the algorithm-based analysis and classification of texture in paintings [15, 16], learned representations of artists’ styles and painting techniques [17, 18], and multi-modal image registration and colour analysis in paintings [19–23], just to name a few.

In this work we discuss a range of mathematical methods for correcting and enhancing images of illuminated manuscripts. In particular, we consider automated and semi-automated models for digital image restoration based on partial differential equations, exemplar-based image inpainting and osmosis filtering, and their translation to the digital interpretation of illuminated manuscripts. Here, we refer to mathematical image processing as the task of digital image restoration (or reconstruction), that is the digital processing of a given image to correct for its visual imperfections. Generally, this is done with the main intention of producing a final result where imperfections have been corrected in a visually least distracting way. This is the case for several imaging tasks such as image denoising, deblurring and also image inpainting.

Medieval and Renaissance illuminated manuscripts present a particular challenge, but also an opportunity to transform current understanding of European visual culture between the 6th and 16th century. Illuminated manuscripts are the largest and best preserved resource for the study of European painting before 1500. Nevertheless, the images in some manuscripts have been affected by wear-and-tear, degradation over time, iconoclasm, censorship or updating. Unlike the conservation of other painted artefacts, the conservation of illuminated manuscripts preserved in institutional collections is non-invasive, usually restricted to repairs of the binding and of torn parchment or paper, and rarely involves the consolidation of flaking pigments. For the study of illuminated manuscripts, physical restoration and repairs are often disregarded. This minimal approach is due largely to the fact that when compared to wall or easel paintings, the images in illuminated manuscripts are relatively small and their pigment layers are few and very delicate. It is not possible to remove over-painting without damaging or completely removing the original painting beneath. The removal of even the smallest sample or the restoration of even the smallest painted area would constitute a considerable change to the overall image. As a consequence, pigment losses are often not filled in and over-paintings added on top of the superficial layers can often not be removed to reveal the original images. Virtual restoration is thus the only way to recover damaged illuminations, whether by infilling paint losses or by removing over-painted layers or indeed both. Bringing the images as close as possible to their original form would ensure both their accurate scholarly interpretation and their full appreciation by wider audiences. Damaged or inaccurately restored illuminations can lead to the exclusion of seminal works of art from academic debates or to incomplete and misleading interpretations of the dating, origin and artists involved. Preserving the current state of the illuminations in line with conservation ethics, faithful digital restoration would serve as a reliable surrogate for multiple reconstructions, enabling research, teaching and wider appreciation for manuscripts.

The reliable processing of illuminated manuscripts requires a multi-disciplinary collaboration as the current work is based on. In what follows we discuss a range of new adaptive, semi-automated restoration methods that (a) reconstruct image-structures using partial differential equations [13, 14, 24–28], (b) mimic the human-expert behaviour, using texture- and structure patches sampled from the intact part of the illuminated manuscript at hand and integrating them in exemplar-based inpainting approaches [29, 30] in order to provide a digital restoration in agreement with the available information and pleasant to the eye (c) exploit infrared imaging data, correlating the visible image content with its traces in the hidden layers of paint [31, 32], and (d) create new 3D interpretations of illuminated manuscripts through a new 3D conversion pipeline [33]. The pre-sequel of this work is an article in the exhibition catalogue [32].

Organisation. In “Retrieving missing contents via image inpainting” section we propose a semi-supervised approach for the segmentation of damaged areas of colour accurate images (in the following referred to simply as RGB images) of illuminated manuscripts and for the retrieval of missing information via a two-step image inpainting model. In “Looking through the layers via osmosis filtering” section we consider the mathematical model of image osmosis to integrate super-painted visible image information on a manuscript with hidden infrared ones for looking through the layers of a restoration process. Finally, in “Creating a 3D virtual scene from illuminated manuscripts” section we present a mathematical pipeline to convert a 2D painting into a 3D scene by means of the construction of an appropriate depth map.

The problem of image inpainting can be described as the task of filling in damaged (or occluded) areas in an image f defined on a rectangular domain \(\Omega\) by transferring the information available in the intact areas of the image to the damaged areas in the image. Over the last 30 years a large variety of mathematical models solving the image inpainting problem have been proposed, see, e.g., [28, 34] for a review. In some of them, image information is transferred into the damaged areas (the so-called inpainting domain, denoted by D in the following) by using local information only, i.e. by means of suitable diffusion and transport processes which interpolate image structures in the immediate vicinity of the boundary of D in the occluded region. Such techniques have been shown to be effective for the transfer of geometric image structures, even in the presence of large damaged areas [28]. However, because of their local nature, such methods do not make use of the entire information contained in the intact image regions. In particular, such methods do not take into account non-local image information in terms of patterns and textures nor image contents located far away of D. For this reason, non-local mathematical models exploiting self-similarities in the whole image have been proposed [29, 30, 35, 36]. Such models operate on image patches rather than single pixels. Small patches inside D are iteratively reconstructed by comparison with patches outside D in a suitable distance. Missing patches are then reconstructed by copy and paste of a closest patch (or its centre pixel) from the intact part of the image. These models have been proven to be impressively effective in a very large variety of applications and rendered computationally feasible in recent years with the well-known PatchMatch algorithm [37].

The first step of any inpainting algorithm is the decomposition of the image domain in damaged and undamaged areas. This is an image segmentation problem, decomposing a given image into its constituting regions, cf. for instance [34]. Its solution may be rendered very hard in the presence of fuzzy and irregular region boundaries and small scale objects.

In the following we describe an algorithm which detects damaged areas in images with possibly large and non-homogeneous missing regions using few examples provided by the user. This is then used as a necessary initial step for the subsequent application of a two-stages inpainting procedure based on total variation inpainting [38] and exemplar-based image inpainting proposed in [36] for the reconstruction of image contents in the images of the illuminated manuscripts in Fig. 1. Our proposed segmentation is semi-supervised since user input is required for training, while the inpainting procedure is fully automated.

A semi-supervised algorithm for the detection of the damaged areas

For identifying the damaged areas in the image (mainly missing gold leaves) we propose in the following a two-step semi-supervised algorithm. Here, a classical binary segmentation model is used first for the extraction of a small training region as described in “Chan-Vese segmentation” section which subsequently serves as an input for a labelling algorithm which segments the whole inpainting domain based on appropriate intensity-based image features in “Image descriptors: feature extraction” and “A clustering algorithm with training” sections.

Chan-Vese segmentation

In binary image segmentation one seeks to partition an image in two disjoint regions, each characterised by distinctive features. Typically, RGB intensity values are used to describe image contents and mathematical image segmentation methods often compute the required segmented image as the minimiser of an appropriate functional.

Let f be the given image. We seek a binary image u so that

$$\begin{aligned} u(x) = {\left\{ \begin{array}{ll} c_1, & {}\quad {\text {if}} \; x \text{ is } \text{ inside } C, \\ c_2, & {} \quad {\text {if}} \; x \text{ is } \text{ outside } C, \end{array}\right. } \end{aligned}$$

(1)

where C is a closed curve. In this work, we consider the Chan-Vese segmentation functional for binary image segmentation [39], that is

$$\begin{aligned} \mathcal {F}(c_1,c_2,C):= \, & {} \mu ~ \text {Length}(C) + \nu ~\text {Area}\left( int(C)\right) \\&+ \nonumber \lambda _1\sum _{x\in int(C)} | f(x)-c_1 |^2 + \lambda _2\sum _{x\in ext(C)} |f(x)-c_2|^2. \end{aligned}$$

(2)

The functional \(\mathcal F\) is minimised for constants \(c_1\) and \(c_2\) and the contour C, i.e. the optimal u of the form (1). Here, \(\mu ,\,\nu ,\,\lambda _1,\,\lambda _2>0\) are positive parameters and \(int(C),\, ext(C)\) denote the inner and the outer part of C, respectively. In (2) the first and second term penalise the length of C and the area of the region inside C, respectively, giving control on the smoothness of C and the size of the regions. The two other terms penalise the discrepancy between the fitting of the piecewise constant u in (1) and the given image f in the interior and exterior of C, respectively. By computing a minimum of (2) one retrieves a binary approximation u of f.

Despite being very popular and widely used in applications, the Chan-Vese model and its extensions present intrinsic limitations. Firstly, the segmentation result is strongly dependent on the initialisation: in order to get a good result, the initial condition needs to be chosen within (or sufficiently close to) the domain one aims to segment. Secondly, due to the modelling assumption (1), the Chan-Vese model works well for images whose intensity is locally homogeneous. If this is not the case, the contour curve C may evolve along image information different from the one we want to detect. Images with significant presence of texture, for instance, can exhibit such problems. Furthermore, the model is very sensitive to the length and area parameters \(\mu\) and \(\nu\), which may make the segmentation of very small objects in the image difficult.

For our application, we make use of the Chan-Vese model¹ to segment a sub-region \(D_1\) of D that will serve as a training set for the classification described in the following two subsections. To do that, we ask the user (typically, an expert in the field) simply to click on a few pixels inside the inpainting domain D to identity a candidate initial condition for the segmentation model (1), which is then run to segment the subregion \(D_1\). In Fig. 2 we show the results of this approach with a superimposed mask of the computed region \(D_1\) for some details cropped from the original images.

Because of the intrinsic limitations of the Chan-Vese approach, we observe that the segmentation result is not satisfactory (see, for instance, the example in the first row of Fig. 2) since it generally detects with high precision only the largest uniform region around the user selection. To detect the whole inpainting domain D in this manner, the user should in principle give many initialisation points, which may be very demanding in the presence of several disconnected and possibly tiny inpainting regions.

For this reason, we proceed differently and make use of a feature-based approach to use the area \(D_1\) as a training region for a clustering algorithm running over the whole set of image pixels. This procedure is described in the next two sections.

Image descriptors: feature extraction

In order to describe the different regions in the image in a distinctive way, we consider intensity-type features. Namely, for every pixel x in the image we apply non-linear colour transformations to compute the HSV (Hue, Saturation, Value), the geometric mean chromaticity GMCR [40], the CIELAB and the CMYK (Cyan, Magenta, Yellow, Key) values (see [41] for more details). Once this is done, we append all these values and store them in a feature vector \(\varvec{\psi }\) of the form

$$\begin{aligned} \varvec{\psi }(x)= [ \text {HSV}(x), \text {GMCR}, \text {CIELAB}(x), \text {CMYK}(x)]. \end{aligned}$$

(3)

For our purpose the feature vector (3), essentially based on RGB intensities, rendered precise segmentations. For more general segmentation purposes, one could add texture-based features and, if available, multi-spectral measurements such as infrared IR or ultraviolet UV images.

A clustering algorithm with training

Once the feature vectors are built for every pixel in the image, we use the training region \(D_1\) detected as described in “Chan-Vese segmentation” section as a dictionary to drive the segmentation procedure extended to the whole image domain. We proceed as follows. First, we run a clustering algorithm over the whole image domain comparing the features defined in (3) in order to partition the image in a fixed number of K clusters. To do that, we use the well-known k-means algorithm.² After this preliminary step, we check which cluster has been assigned to the training region \(D_1\) and simply identify in the clustered image which pixels lie in the same cluster. By construction, this corresponds to finding the regions in the image ‘best-fitting’ the training region in terms of the features defined in “Image descriptors: feature extraction” section, which is our objective. After a refinement step based on erosion/dilation of extracted regions, so as to remove or fill-in possibly misclassified pixels, we can finally extract the whole area to inpaint D. We report the results corresponding to Fig. 2 in Fig 3a, b.

Inpainting models

Once an accurate segmentation of the damaged areas is provided, the task becomes the actual restoration of the image contents in D by means of the available information in the region \(\Omega \setminus D\). A standard mathematical approach solving such an inpainting problem consists in minimising an appropriate function \(\mathcal {E}\) defined over the image domain \(\Omega\), i.e. in

$$\begin{aligned} \text { finding }\qquad u\qquad \text {s.t.}\qquad u\in \text {argmin}_v~ \mathcal {E}(v). \end{aligned}$$

(4)

A standard choice for \(\mathcal {E}\) in the case of local inpainting models is the functional

$$\begin{aligned} \mathcal {E}(v) = R(v) + \lambda \Vert \upchi _{\Omega \setminus D}(f- v)\Vert ^2_2, \end{aligned}$$

(5)

where f denotes the given image to restore, \(\Vert \cdot \Vert _2\) is the Euclidean norm, \(\lambda\) and appropriately chosen positive parameter and \(\upchi _{\Omega \setminus D}\) denotes the characteristic function of the non-occluded image areas, so that for every pixel \(x\in \Omega\):

$$\begin{aligned} \upchi _{\Omega \setminus D}(x) = {\left\{ \begin{array}{ll} 1\quad &{}\text {if }\; x\in \Omega \setminus D\\ 0 \quad &{} \text {if }\; x\in D. \end{array}\right. } \end{aligned}$$

The second term in (5) is as a distance function between the given image f and the sought after restored image u in the intact part of the image. The multiplication of \(f-u\) by the characteristic function \(\upchi\) implies that this term is simply zero for the points in D, since there is no information available, while \(f-u\) for all the points in \(\Omega \setminus D\) has to be as small as possible. The term R typically encodes local information (such as gradient magnitude) which is the responsible of the transfer of information inside D by means of possibly non-linear models [28, 34]. The transfer process is balanced with the trust in the data by the positive parameter \(\lambda\). A classical choice of a gradient-based inpainting model consists in choosing

$$\begin{aligned} R(v) = \Vert \nabla v \Vert _1 = \sum _{x\in \Omega } | \nabla v(x) | \end{aligned}$$

(6)

i.e. the Total Variation of v [38]. As mentioned above such an image inpainting technique is not designed to transfer texture information. Furthermore, it fails in the inpainting of large missing areas. For our purposes we use (6) as an initial ‘good’ guess with which we initialise a different approach based on a non-local inpainting procedure as described in the following section.

Exemplar-based inpainting

We describe here the non-local patch-based inpainting procedure studied in [30, 36] and carefully described in [42] from an implementation point of view.³ In the following, we define for any point \(x\in \Omega\) the patch neighbourhood \(\mathcal {N}_x\) as the set of points in \(\Omega\) in a neighbourhood of x. Assuming that the patch neighbourhood has cardinality n, by patch around x we denote the 3n-dimensional vector \(P_{x} = (u(x_1), u(x_2),\ldots ,u(x_n) )\) where the points \(x_i, i=1,\ldots n\) belong to patch neighbourhood \(\mathcal {N}_x\). In order to measure ‘distance’ between patches, a suitable patch measure d can be defined, so that \(d(P_{x},P_{y})\) stands for the patch measure between the patches around the two points x and y. We define then the Nearest Neighbour (NN) of \(P_{x}\) as the patch \(P_y\) around some point y minimising d.

For an inpainting application the task consists then in finding for each point x in the inpainting domain D the best-matching patch \(P_y\) outside D. Assuming that each NN patch can be characterised in terms of a shift vector \(\phi\) defined for every point in \(\Omega\) (i.e. assuming there exists a rigid transformation \(\phi\) which shifts any patch to its NN), the problem can be formulated as the minimisation problem

$$\begin{aligned} \min ~ \mathcal {E}(u,\phi ) = \sum _{x\in D}~ d^2\left( P_{x},P_{x+\phi (x)}\right) . \end{aligned}$$

(7)

Heuristically, every patch in the solution of the problem above is constructed in such a way that in the damaged region D the patch has a correspondence (in the sense of the measure d) with its NN patch in the intact region \(\Omega \setminus D\). Following [42], we use the following distance:

$$\begin{aligned} d^2\left( P_{x},P_{x+\phi (x)}\right) = \sum _{y\in \mathcal {N}_{x}} \left( u(y)- u(y+\phi (x))\right) ^2. \end{aligned}$$

(8)

From an algorithmic point of view, solving the model involves two steps: the first consists in computing (approximately) the NN patch for each point in D, so as to provide a complete representation of the shift map \(\phi\). This can be computationally expensive for large images. In order to solve this efficiently, a PatchMatch [37] strategy can be applied. Afterwards a proper image reconstruction step is performed, where for every point in D the actual corresponding patch is computed. We refer the reader to [42] for full algorithmic details.

A crucial ingredient for a good performance of the exemplar-based inpainting algorithm [30, 36] is its initialisation. In particular, once the inpainting domain is known, a pre-processing step where a local inpainting model, such as the TV inpainting model (5) with (6), can be run to provide a rough, but reliable initialisation of the algorithm.⁴

We report the results of the combined procedure in Fig. 4 and the overall work-flow of the algorithm in the diagram in Fig. 5.

Model parameters

For the segmentation of the training region \(D_1\) within the inpainting domain D we use the activecontour MATLAB function by which the Chan-Vese algorithm can be called. For this we fixed the maximum number of iterations to maxiter\(=1000\) and use the default value as a tolerance on the relative error between iterates as a stopping criterion. We use the default values for the parameters \(\mu\) and \(\nu\) in (2). The subsequent clustering phase was performed by means of the standard MATLAB kmeans function after specifying a total of \(K=35\) labels to assign. The use of such a large value for K turned out to be crucial for an accurate discrimination. The automatic choice of the value of K for this type of applications is a matter of future research. The clustering was iteratively repeated 5 times to improve accuracy. Once the detection of the inpainting domain is completed, in order to provide a good initialisation to the exemplar-based model we use the TV inpainting model (4) with (6) with the value \(\lambda =1000\) and a maximum number of iterations equal to maxiter2\(=1000\) with a stopping criterion on the relative error between iterates depending on a default tolerance. Finally, we followed [42] for the implementation of the exemplar-based inpainting model: for this we specified 12 propagation of iterations and tested different sizes for the patches. In order to avoid memory shortage, we restricted ourselves to patches of size \(5\times 5\), \(7\times 7\) and \(9\times 9\).

The numerical tests were performed on a standard MacBook Pro (Retina, 13-inch, Early 2015), 2.9 GHz Intel Core i5, 8 GB 1867 MHz DDR3 using MATLAB 2016b.

Discussion and outlook

We proposed in this section a combined algorithm to retrieve image contents from two images of illuminated manuscripts shown in Fig. 1 where very large regions have been damaged. At first, our algorithm computes an accurate segmentation of the inpainting domain which is performed by means of a semi-supervised method exploiting distinctive features in the image. Then, taking the segmentation result as an input, the procedure is followed by an exemplar-based inpainting strategy (upon suitable initialisation) by which the damaged regions are filled.

The results reported in Figs. 4 and 6 confirm the effectiveness of the combined method proposed. In particular, when looking at the difference between standard local (TV) image inpainting methods and the exemplar-based one we immediately appreciate the higher reconstruction quality in the damaged regions, especially in terms of texture information. The method has been validated on several image details extracted from the entire images, and has been shown effective also for very large image portions with highly damaged regions.

In term of computational times, the segmentations in Fig. 3 are obtained in approximatively 15 min. The inpainting results in Fig. 4 are obtained in about 3 min for patches of size \(5 \times 5\) and about 7 min for patches of size \(7 \times 7\). Overall the whole task of segmenting and inpainting the occluded regions takes approximatively 20 min per image of size \(690 \times 690\). However, these results highly depend on the size of the image, the size of the inpainting domain and the size of the patches chosen.

Future work could address the use of different features for the segmentation of the inpainting domain with similar methodologies, such as for instance texture features [43]. Furthermore, at an inpainting level, we observe that the reconstruction of fine details in very large damaged regions (such as the strings of the harp in Fig. 6) is very challenging due to the lack of correspondence with similar training patches in the undamaged region. For solving this problem a combination of exemplar-based and local structure-preserving inpainting models could be used.

In the previous section the image content in the damaged areas of the illuminations is completely lost and it was estimated only from the information available in the rest of the picture. This, however, is not the only kind of degradation encountered in the process of restoration of illuminated manuscripts. In some cases parts of an illumination are painted over. In this section we discuss as such an example the illuminations from the primer of Claude de France which illustrate the story of Adam and Eve in the garden of Eden. The two figures were originally depicted naked, as described in the book of Genesis but a later owner wanted them clothed with additional veils, leaves or beast skin added in the illumination, cf. Fig. 7. The use of infrared imaging as shown for instance in Fig. 8 allows to look through these added layers, unveiling hidden structural information underneath the painted layer. All the input colour images and their reflectogram are freely available on the Fitzwilliam museum website⁵ along with some more information about the manuscript, in particular the pigments used.

In this section we aim to fuse the details appearing in the near infrared reflectogram (IR) with the colours of the visible colour image, in particular the skin tones, to create a digital version of the illuminations as they could have looked before overpainting. Since we only have access to one near infrared reflectogram and we cannot chose the wavelength and have no information on the pigments used, we find ourselves in one of the following three situations: (i) the added cloth is transparent in the IR; (ii) the added cloth appears in the IR but without texture; (iii) the added cloth and its texture appear in the IR. The fact that the original pigments can also be IR transparent poses an additional challenge. For these different situations we use different methods all based on the use of the linear image osmosis model studied by Weickert et al. in [31].

In the following we first present the original parabolic linear osmosis equation studied in [31] and our slightly modified local elliptic formulation of osmosis [44]. Then we recall some of its common applications in image processing and finally apply our methods to digitally unveiling Adam and Eve in Claude De France’s Primer in each of the different situations (i)–(iii) described above (cf. “IR transparent original pigments”, “Over-paint with IR transparent texture” and “Non IR transparent over-paint texture: adding an inpainting step” sections).

Applications to illuminated manuscripts

In an ideal case, the added pigments do not appear on the IR while the colours to be restored are perfectly encoded in the IR. In this case the problem is reduced to a simple seamless cloning application with Dirichlet boundary conditions. The drift-field of the colour image is replaced by the one from the infrared image on the sub-domain to be restored. However, unfortunately, such an ideal case is uncommon. For the illuminations of the primer, we encounter rather different scenarios. For instance, when the added cloth is IR transparent or has no texture in the IR, the osmosis equation is enough to get a satisfying result. When the texture of the added cloth appears in the IR, the osmosis equation is no longer enough and we have to add an inpainting step to our method. We describe this in a greater detail in the following.

IR transparent original pigments

In Fig. 8, the IR along with a careful examination of the colour image reveals the existence of an original fig leaf under the added leaves of the over-paint. Here the over-paint is IR transparent so it should be a simple seamless cloning problem with the colour image being the background and the infrared being the foreground image. Yet, the colour distinction between the original fig leaves and the skin of Adam and Eve is hard in the IR. If we simply follow the seamless cloning method, we get back not only the skin colour but also the fig leaves colour from the small parts left untouched in the colour image. However because they appear the same in the IR, some diffusion occurs across the edges between the skin and the fig leaves. To prevent this, we enforce Neumann boundary conditions along these edges to prevent any such diffusion. The results with and without the use of Neumann boundary conditions (represented as red lines in the mask) are presented in Fig. 8.

Over-paint with IR transparent texture

In Fig. 9, the added cloth on Adam is not IR transparent but it has little texture discernible on the IR and the original drawings appear clearly by transparency under it. This looks like a shadow in the IR as well as in the solution obtained with the method of the previous “IR transparent original pigments” section. Thus we mix seamless cloning with mixed boundary conditions and the shadow removal method. We replace the canonical drift-field of the colour image by the one of the IR in the region of interest. Then we put the drift-field to zero on the edge of the over-paint appearing in the IR. This method is illustrated in Fig. 9. The white lines of the mask are the areas where the drift-field is put to zero. In this figure we observe some transparent texture from the over-paint (over Adam’s hip and at the bottom of Eve’s veil). As expected, this texture appears in the final result.

Non IR transparent over-paint texture: adding an inpainting step

In the case of Fig. 10, the IR adds some useful information to the colour image, as shown by the result obtained using the method from the previous “Over-paint with IR transparent texture” section but a large amount of the added skirt texture, visible in the IR, is also present. To get rid of this unwanted texture, we put the drift field to zeros on the area corresponding to Adam’s skin and manually segment the lines we want to keep. Note that this leads to a complete loss of texture in this region. To have a more natural looking result, we want to have some texture for the skin. While we can’t recover the original texture with our inputs, the untouched part of the illumination gives us some example of texture for Adam’s skin. This information is enough to use the exemplar-based inpainting algorithm described in “Exemplar-based inpainting” section, using as initialisation our result with missing texture. The final result on Adam’s skin has probably not much in common with the original painting but it appears natural enough, so it can help to get a better idea of the illumination in its original state.