Extracting chain lines and laid lines from digital images of medieval paper using spectral total variation decomposition

Abstract

Medieval paper, a handmade product, is made with a mould which leaves an indelible imprint on the sheet of paper. This imprint includes chain lines, laid lines and watermarks which are often visible on the sheet. Extracting these features allows the identification of the paper stock and gives information about the chronology, localisation and movement of manuscripts and people. Most computational work for feature extraction of paper analysis has so far focused on radiography or transmitted light images. While these imaging methods provide clear visualisation of the features of interest, they are expensive and time consuming in their acquisition and not feasible for smaller institutions. However, reflected light images of medieval paper manuscripts are abundant and possibly cheaper in their acquisition. In this paper, we propose algorithms to detect and extract the laid and chain lines from reflected light images. We tackle the main drawback of reflected light images, that is, the low contrast attenuation of chain and laid lines and intensity jumps due to noise and degradation, by employing the spectral total variation decomposition and develop methods for subsequent chain and laid line extraction. Our results clearly demonstrate the feasibility of using reflected light images in paper analysis. This work enables feature extraction for paper manuscripts that have otherwise not been analysed due to a lack of appropriate images. We also open the door for paper stock identification at scale.

Introduction

Paper, one of the most versatile and lasting material technologies from the medieval period to the present day, has carried great importance throughout the ages. It has been used to record and preserve history, was the carrier of information and as such tells the story of the agents and sites that it encountered. While nowadays paper is machine-made, each sheet of medieval paper was handmade by taking a sieve - the mould - and dipping it into a large vat of pulp made from fibres. When lifting the mould out of the vat, excess water drips out and the newly formed paper sheet is couched on a pile of felts, pressed and hung up to dry. The imprint of the mould left on each sheet of paper at the point of its making is like a ‘fingerprint’ which can be identified, described and traced [1]. A mould consists of horizontal (laid) wires and vertical (chain) wires in a wooden frame with additional wires twisted into a shape which produces the watermark. The thickness of the metal wires and distance between them, the size of the watermark and the mould vary, and this variation constitutes the signature that the mould leaves on each sheet of paper. Such fingerprint can change overtime with the use of the mould, because the wires move, however, its identification allows individual sheets of paper to be connected and grouped. Papermakers used two moulds, with often similar features, to make sheets of paper simultaneously. It has been proposed that the paper produced by these two moulds is considered a paper stock [2]. The study of the morphological features of a paper stock has enormous potential to shed light on the chronology, localisation and movement of books and people.

With the advent of digital technologies, a large number of paper manuscripts have been digitised in the last two decades amounting to an impressive catalogue of images ranging from radiography to transmitted and reflected light images. However, each image type comes with its own set of challenges. Radiography equipment is expensive, obtaining the images can take a significant amount of time and there are considerably fewer images readily available. On the other hand, transmitted light images are attainable at a lower equipment cost but they are relatively scarce and burdened with selection bias. They also contain shadows of the ink from the verso side of the paper, that is the side that is closest to the light source. Therefore, analysing transmitted light images of paper that contain written text has to include the digital removal of any ink shining through the paper, adding an extra layer of pre-processing in the image analysis pipeline. In turn, reflected light images are abundant and freely available from many archives and libraries. The analysis of reflective light images of paper is challenging due to low contrast attenuation of the expected chain and laid lines, the jumps in the intensity values due to several factors (including image noise, degradation of the manuscripts, dirt and spots), and the strong penetration of the ink.

Context

The fingerprint of a mould is made up of at least three main paper features: the watermark, the chain lines and the laid lines. Extraction and quantification of these features has mostly been done based on radiography and transmitted light images. The work of watermark identification, measurement and extraction has been the focus of many publications. Whelan et al. [3] propose using morphological filters for watermark segmentation from transmitted light images. For laid paper, they additionally remove any visible laid or chain lines by Fourier filtering. However, their method is derived for paper that does not contain any writing. Hiary et al. [4] propose using morphological filters to separate foreground and background interference in transmitted light images, enabling the analysis of papers that have ink potentially on both sides. Afterwards, they use edge detection to extract the watermark. In contrast, Gorske et al. [5] take a different approach in that they measure the distance between the watermark and the neighbouring chain lines instead of using the segmented watermark for paper stock identification and matching. The Bernstein project [6] has built a large watermark catalogue and portal that combines the digitised watermarks from many research institutions and standardised the terminology. Their platform also provides an advanced search option to filter for chain line distances around the watermark and laid line densities [7]. It must be noted, however, that chain line distances around a watermark tend to be smaller as compared to the rest of the paper sheet. This is due to the chain lines acting as a support to the watermark wires. The Bernstein project additionally provides software for watermark and line detection and extraction for radiography and transmitted light images. Watermarks as a feature of handmade paper can be classified according to iconography and this type of clustering can show differences also within the same design. However, it is well known that not all paper contains a watermark and it is important to test the extent to which chain and laid lines can also help with the identification of the mould. Therefore, focusing on chain and laid lines has its merit and it is imperative to have accurate chain and laid line extraction methods for paper stock identification. To this end, this paper focuses on chain and laid lines extraction and we will expand our methods to watermark segmentation in future work.

State of the art for chain and laid line extraction

The overarching methods used for chain and laid line extraction are based on the Fourier transform and Radon transform or involve morphological filters. Van der Lubbe et al. [8] detect chain lines in radiography images using morphological filters and vertical projection, assuming the chain lines to be vertical and straight. Leveraging two paper features - chain and laid lines - van Staalduinen et al. [9] extract chain lines based on their shadows, a larger imprint due to the paper making process, via Fourier transform, Radon transform and Gaussian filtering. They additionally estimate the laid line density from the Fourier frequency with the highest magnitude and then define similarity measures for mould matching. Johnson et al. [10] use radiography images of Rembrandt’s prints for chain line detection. Their method uses the Radon transform to estimate the orientation of chain lines and a combination of vertical filtering and the Hough transform to extract the lines. This forms the basis of their mould matching algorithm. Shifting gears to transmitted light images for chain line extraction, Hiary et al. [11] develop a model to separate background and foreground and then use the Radon transform to detect chain lines. They use canny edge detection to additionally extract the watermark. Most recently, both Biendl et al. [12] and Sindel et al. [13] develop deep learning approaches for chain line segmentation and parametrisation from transmitted light images. While [12] follow a supervised approach to train a UNet for chain line segmentation, [13] train a conditional generative adversarial network to predict a segmentation mask. However, both approaches rely on supervised training which in turn requires manually annotated ground truth segmentation masks and is therefore costly. The work by Xi et al. [14] additionally shows that mould matching using chain line information alone is not enough to successfully identify different paper stock. Considerably less work has been published on the extraction of laid lines. Atanasiu [15] first uses emboss filtering for noise removal and subsequently applies the discrete fast Fourier transform to obtain the highest frequency in the Fourier domain representing the frequency of laid lines. More recently, Gorske et al. [5] propose a laid line density approximation based on the 1D fast Fourier transform and refinement of the frequency estimate through a phase vocoder. While they do not produce a paper average number of laid lines per centimetre, as commonly used in laid line analysis, their results show a map of local laid line densities for the full image and enable paper matching based on the similarity of density distributions throughout the image.

Considering the vast amount of reflected light images readily available from many archives and libraries, there is a need to broaden the study of the evidence provided by the mould. The use of automatic feature extraction from reflected light images has the potential to improve the speed, scale and accuracy of paper stock identification, facilitate research into heritage questions we can only answer at scale and transform this field of study. Without the need for expensive equipment or re-imaging of paper manuscripts, paper stock identification becomes accessible even for small institutions. However, the work on chain and laid line detection so far has focused on radiography and transmitted light images. Building on those results, the extraction from reflected light images requires additional layers of processing and specialised approaches to deal with challenges such as low contrast attenuation of the chain and laid lines, high influence of noise and strong penetration of the ink.

This paper establishes a proof of concept for the analysis and extraction of paper features from reflected light images by means of mathematical multi-scale image decomposition methods (cf. Figure 1). In particular, that means we determine the locations of the lines in the image and enable the measurement of the line distances as well as calculating the line density. We show that the spectral total variation (TV) decomposition [16, 17] is able to separate the features of interest (chain and laid lines) from ink and noise. In a sense, this can be seen as a simulation of transmitted light images from reflected light images. Furthermore, we develop a semi-automated approach for the extraction and measurement of chain and laid lines from the decomposed images. We identify straight lines and count their number and density. We are then able to measure the distance between them to identify these elements of the paper stock signatures. We demonstrate that this method of extraction is replicable on different types of image degradation. We consider high resolution and high quality images, which are widely available from digitised archives. The extension to lower resolution images such as those taken with a phone camera will be part of future work. To the best of our knowledge, the application of the proposed method to reflected light images has not been attempted before. In what follows, we show how we have developed mathematical imaging approaches for extracting granular paper feature information from regular reflected light images of paper manuscripts. We show that the spectral TV decomposition of the images is vital to the success of our methods. The accompanying code is available at https://github.com/TamaraGrossmann/HiddenKnowledge.

Fig. 1

Overview of the features and outcomes of interest. Original image courtesy of The Parker Library, Cambridge, Corpus Christi College, MS 81: Homer, Iliad, Odyssey. Quintus of Smyrna, Posthomerica, p. 55

Organisation

The paper is structured as follows: First, we introduce the spectral TV decomposition that forms the basis of our method in the second section. We present the proposed algorithms for chain and laid line extraction. In the third section “Results and Discussion”, we show results of the line extraction on example images. Additionally, we discuss the selection of parameters in the semi-automated algorithms. In the last section we draw conclusions and outline future work.

Methods

The analysis of medieval paper features from reflected light images is challenging for three main reasons. First, the features of interest, the chain and laid lines, have low contrast in the image and therefore can be barely visible to the human eye and hard to detect, needing specialised image analysis approaches. Additionally, many sheets contain writing with ink occluding or interrupting paper features. Finally, medieval manuscripts tend to show varying levels of paper degradation, dirt spots and image noise that interfere with the chain and laid line extraction. For these reasons, the direct chain and laid line extraction from reflected light images is not feasible. Instead, we apply spectral TV decomposition to separate the granular, low contrast features of interest from the rest of the reflected light image.

Pre-processing

In order to produce absolute measurements from the chain and laid line extraction in terms of distances in millimetre and line density per centimetre, the first step is the measurement of the pixel size. For reflected light images, we typically observe two types of data. That is, either the image contains a ruler in one of the corners or the paper size (height and width) is provided in the metadata. Using the paper measurements to extract the pixel size can introduce some inaccuracies due to paper being, for example, warped on the imaging platform. However, for the images that we consider, these inaccuracies are minor and we have found that they do not obscure the absolute measurements of line distances and densities. Both types of data require the detection of edges of either the ruler lines or the edges of the paper itself. We employ canny edge detection [18], a well-established method that is based on the image intensity gradient and involves Gaussian smoothing and thresholding. For the images of paper manuscripts that we consider, canny edge detection is sufficient as the contrast between the paper (light) and the background (dark) is sharp. However, for images of lower quality, more recent approaches such as [19, 20] may be more suitable. When a ruler is present in the image, the size of each pixel can be determined by taking the pixel distance between the ruler lines, for example, the two outer lines. When using the metadata information on the paper size, we determine the pixel size by measuring the pixel distance between the top and bottom edge of the paper. The final step in pre-processing is transforming the images into greyscale for simplicity. While colour information may provide additional information, our methods have worked successfully leveraging the greyscale information alone which also reduces computational cost in the analysis.

Spectral total variation decomposition

The decomposition of an image into its scale-dependent components is a common and successful tool in image analysis. The intuitive idea is to separate components loosely related to their size/scale. Exact representations can be modelled by mathematical approaches relating to different types of components. A prominent example is the Fourier transform that decomposes an image based on the frequency of its structures. The bases of this decomposition are trigonometric functions. Especially for data with repeating structures, such as signals made up of multiple waves with varying frequencies, the Fourier transform is a powerful tool to separate the different frequency components. By defining filters, the structures can be separated in the transform domain relating to scale/frequency. One such filter is the Gaussian filter. The standard deviation $\sigma$ of the Gaussian filter determines the frequencies that are attenuated or removed in the Fourier domain. The resulting images depict varying degrees of blurring. In a two-dimensional setting, Gaussian filtering in the Fourier domain of an image f(x) is defined as ${\mathcal {F}}[G_{\sigma }](w){\mathcal {F}}[f](w)$, where ${\mathcal {F}}$ is the Fourier transform and $G_{\sigma }(x) = \frac{1}{2 \pi \sigma ^2}\exp {(-|x |^2/2\sigma ^2)}$ the Gaussian filter kernel. This Gaussian filtering can be equivalently modelled by the solution to the heat equation $u_t(x,t) = \Delta u(x,t)$ with the initial condition $u(x,0) = f(x)$ and for $t = 2 \sigma ^2$. It is also called the Gaussian scale-space representation of f(x). The temporal component of the heat equation is related to the standard deviation of the Gaussian kernel and describes the scale in the decomposition. That is, the solution of the heat equation at larger time instances equates to Gaussian filtering in Fourier domain with higher standard deviations and therefore higher degrees of blurring. An example of Fourier decomposition with Gaussian band-pass filtering at different frequencies is shown in Fig. 2.

Fig. 2

Example of spectral TV decomposition with band-pass filtering (left column) in comparison to Fourier decomposition with Gaussian band-pass filtering (right column) for a synthetic image containing disks at different sizes and contrasts (original image on the top left). Images represent small to large scales/ high to low frequencies. For the spectral TV decomposition, the spectral response is plotted (bottom left), the red crosses mark the spectral responses of the individual disks. Note that to aid visualisation in the Fourier decomposition each image has been rescaled independently

Linear scale-space representations of an image, such as for Gaussian filtering, however, are not always the best choice for decomposition as images are inherently non-linear. Fourier decomposition can approximate edges of an image only at high frequencies, as seen in the Gaussian filtered example in Fig. 2. Edges define the boundary of objects in an image and a representation of these features only at high frequencies can therefore make the clean separation of image structures challenging. Instead, the spectral TV decomposition as introduced by [16, 17] is based on the non-linear edge-preserving total variation functional and retains sharp edges at all scales. The spectral TV decomposition is driven by the size and the contrast of the structures with small and low contrast features being represented at lower scales and large, high contrast features at higher scales. The basic atoms are induced by eigenfunctions to the subdifferential of the TV functional. These are, for example, disks and disk-like structures. The spectral TV decomposition for an image containing disks is shown in Fig. 2. In contrast to the Fourier decomposition, the spectral TV decomposition is able to clearly separate each of the disks while retaining sharp edges. The relation between structure size and contrast is particularly relevant for the analysis of paper images. In reflected light images, the chain and laid lines are small features and only present at low contrast while the noise, dirt spots, paper degradation such as water damage, and ink are larger features or are present at higher contrast. The spectral TV decomposition is therefore ideal to separate the structures of interest (chain and laid lines) from any components that might impede the feature analysis of paper.

The total variation functional is defined as $J_{TV}(u) = \int _{\Omega } |D u |$ with $\Omega$ being the image domain and Du the distributional gradient. For u smooth enough, this becomes $J_{TV}(u) = \int _{\Omega } |\nabla u |dx$ [21]. The TV scale-space representation of an image f(x) is modelled by the TV flow with Neumann boundary conditions, formally written as [22, 23]:

$$\begin{aligned} {\left\{ \begin{array}{ll} u_t(t,x) &{}= \text {div}\left( \frac{D u}{|D u |}\right) \quad t \in (0, \infty ), \,\, x \in \Omega \\ u(0,x) &{}= f(x) \quad x \in \Omega \\ \frac{\partial u}{\partial \nu }(t,x) &{}= 0, \quad t \in (0, \infty ), \,\, x \in \partial \Omega , \end{array}\right. } \end{aligned}$$

(1)

where $\Omega$ is the image domain and $\partial \Omega$ denotes the boundary of the image domain. The spectral TV decomposition is based on the solution u(t, x) to the TV flow (1) given an image f(x). The TV transform is then derived from the second temporal derivative of the TV flow solution [16, 17]:

$$\begin{aligned} \phi (t,x) = u_{tt}(t,x)t, \end{aligned}$$

where t refers to the scale parameter. The transform generates impulses at the basic atoms or structures. To give an idea, in the case of an image containing disks the TV transform will create an impulse at a scale based on the radius and contrast of the disk. Disks of different radii and contrasts will therefore result in impulses at different scales t and enable the separation of the disks. One such example is shown in Fig. 2. Notice the four peaks or impulses in the spectral response plot that are related to the scales at which the disks are separated. Given the spectral responses $\phi (t,x)$, the initial image can be recovered via the inverse TV transform $f(x) = \int _{0}^{\infty } \phi (t,x) dt + {\bar{f}}$, with ${\bar{f}}$ the mean value of f. Instead of recovering the full images, we can define filters and manipulate the image in the transform domain, similar to Gaussian filtering in the Fourier domain. In the following, we will be using band-pass filtering, i.e., separating the image based on specific scales. The TV-band-pass filtering in an interval $[t_k, t_{k+1}]$ for $t_k < t_{k+1}$ is defined as

$$\begin{aligned} b^k&= \int _{t_k}^{t_{k+1}} \phi (t,x) dt \quad \text {for} \quad k = 1,\dots , K-1 \\ b^K&= \int _{t_{K-1}}^{t_{K}} \phi (t,x) dt + f_r(t_K,x). \end{aligned}$$

The filtered images $b_k$ are referred to as the K decomposed spectral bands of the initial image f and the residual is derived as $f_r(t_K,x) = u(t_K,x)-u_t(t_K,x)t_K$. The spectral bands sum up to the initial image $f = \sum _{k = 1}^{K} b_k$. For the numerical derivation of the TV flow solution and the TV decomposition we refer to the original papers by Gilboa [16, 17]. Additionally, [24, 25] have developed deep learning approaches for fast computations. Code is openly available for both methods.¹

The classical spectral TV decomposition as introduced above is based on the isotropic total variation, i.e. the discrete TV functional is rotation invariant and in the two-dimensional setting defined as

$$\begin{aligned} J_{TV}^{iso}(u) = \sum _{i,j}\sqrt{u_{x_1}^2(i,j) + u_{x_2}^2(i,j)}, \end{aligned}$$

where $\nabla u = (u_{x_1},u_{x_2})$ denotes the derivatives in $x=(x_1,x_2)$ direction respectively and i, j the pixel locations. The eigenfunctions or basic atoms related to isotropic TV are therefore disk-like shapes as described above. However, we can also discretise the TV functional to include directionality in x- and y-direction by using the 1-norm to derive the gradient magnitude, i.e.

$$\begin{aligned} J_{TV}^{ani}(u) = \sum _{i,j}|u_{x_1}(i,j) |+ |u_{x_2}(i,j)|. \end{aligned}$$

Let us call this discretisation anisotropic TV. It is not rotation invariant and the related eigenfunctions are rectangular shapes. An example of the effect of isotropic versus anisotropic TV decomposition of an image containing rectangles is shown in Fig. 3. The corners of the rectangles are recovered as sharp edges in the anisotropic case, however, the isotropic TV decomposition shows rounded corners on the rectangles and noise at the corner locations in the other bands. While we will mostly use the classical spectral TV decomposition with isotropic TV for paper feature analysis, we will show in the results section that there are cases in which anisotropic TV will be more beneficial.

Fig. 3

Example of spectral TV decomposition with isotropic (top row) and anisotropic (bottom row) TV on a synthetic image containing rectangles of different size and contrast

We apply the spectral TV decomposition to the reflected light images of paper and use band-pass filtering to separate the chain and laid lines from the rest of the paper. In a sense, this decomposition can be seen as a simulation of transmitted light images as it creates images that show similar clarity of the chain and laid lines to transmitted light. While spectral TV decomposition does not reconstruct lines that were previously obstructed by ink or dirt spots, it does reduce the influence on the analysis and obstruction thereof caused by these high contrast features and it enhances the chain and laid lines that were visible before. In Fig. 4, an example of the spectral TV decomposition for paper is shown. To separate the relevant features, the spectral bands need to be chosen appropriately. Image noise is typically very small and will be present in the lowest scales, i.e., around $t=0$. The chain and laid lines are slightly larger structures with low contrast and are represented in low scales. In contrast, the ink has a high contrast and can therefore be found at the higher bands of the decomposition. We have found the best results for extracting the chain and laid lines by selecting the band-pass filter interval around $t \in [0.026, 1]$. We will discuss the selection of spectral bands in more detail in the results section.

Fig. 4

Example of spectral TV decomposition of a reflected light image - patch taken from Cambridge University Library MS Ii.5.41, fol. 10r. a) the original image, b) high-pass filtered image containing only image noise, c) band-pass filtered image including the chain and laid lines of interest, d) low-pass filtered image displaying larger and high contrast structures such as the ink and dirt spot. Reproduced by kind permission of the Syndics of Cambridge University Library, MS Ii.5.41, fol. 10r. Full paper image is displayed in Fig. 6a)

Chain line extraction

Chain line imprints on medieval paper are produced by the vertical chain wires in the mould. As they sit on top of the laid wires, the imprint tends to have higher contrast and casts a shadow, therefore being more visible than laid lines in reflected light images. On average, chain lines are placed at a distance of between 1.5–5 cm apart [9]. Around a watermark, the chain line distances can be reduced as the vertical wires act as supports onto which the watermark is fixed. A variation in the chain line distance across a paper can therefore give an indication as to where the watermark is placed. To this end, we are interested in quantifying the distance between each chain line across the entire page for extracting the fingerprint of the paper. While the average distance between chain lines can give relevant insights, more detailed information and precise matching can be performed when taking distance variations across a paper into account and using the entire chain line sequence. This process gives an overall idea of the size of the mould and its construction.

The first step in our chain line extraction pipeline is the selection of a patch of the image to reduce computational cost. Spectral TV decomposition can become computationally expensive especially when considering high resolution images such as those available for paper. However, in order to get the chain line distances we do not need to analyse the entire page, but rather a subsection. Two main features need to be considered in the selection of patches: Depending on the size and cut of the paper, e.g., folio or bi-folio, the orientation of the chain lines can appear rotated compared to the mould such that they are horizontal instead of vertical. Additionally, as we aim to extract the sequence of chain line distances, the patch needs to contain every line at least partially. That is, we do not need the full length of the line to extract the distances. An example for patch selection is shown in Fig. 5a. In this case, the chain lines are vertical and we select a patch in the lower quarter of the paper page. We assume the chain line orientation to be either horizontal or vertical. For images with lines at different orientation, the Radon transform can be used to determine the dominating orientation and rotate the image accordingly to have chain lines presented horizontally or vertically, compare e.g., [10]. After patch selection, the next step is computing the spectral TV decomposition of the patch as introduced above. The band-pass filtered image forms the base for the detection of chain lines. We are left with an image that now only contains the chain and laid lines and any structures that interfere with the line detection and extraction are removed (cf. Figure 5c). We assume the chain lines to be straight and parallel to each other and employ Fourier filtering to detect the lines. While we have argued above that Fourier decomposition is not ideal to separate small, low contrast structures from high contrast features in images in general, it is suited for periodic structures. Therefore, the task at hand lends itself to Fourier filtering. In other words, as the chain lines in the images are present with relative regularity, using a frequency-based method can enhance and detect the lines. We use the fast Fourier transform on the band-pass filtered image. In the centred Fourier domain, repetitive structures are presented as peaks along the axis orthogonal to their orientation in the image with the distance to the centre marking the frequency. For chain lines that lie vertically in the image, the lines will generate a peak along the horizontal line in Fourier domain. We employ rectangular filtering similar to [10] along the relevant axis to enhance the lines and remove any other structures. The filter width is 1–3 pixels and the height is determined by the image height. The resulting filtered image (cf. Figure 5d) displays only the lines in the relevant direction with the chain lines enhanced. Due to noise, however, additional lines will be present with lower intensity. The next step is therefore to project the image onto the axis perpendicular to the chain lines and employ peak detection. The peaks with the highest magnitude correspond to the chain lines, while the noisy lines will only generate low magnitude peaks. Gaussian filtering of the 1D signal removes some of the noise. We perform a simple local peak detection (cf. Figure 5e) by comparing directly neighbouring values above a specified threshold. The threshold is manually selected and chosen to filter out any remaining low peaks that are caused by noise. Once the chain line peaks are located, we can determine their pixel distance. Using the pixel measurements from the pre-processing step, the pixel distances can then be transformed into metric values. Finally, we draw the detected chain lines in the original image for visualisation as shown in Fig. 5f.

Fig. 5

Chain line extraction pipeline: a original colour image with patch marked; b grayscale patch; c spectral TV band-pass filtered image; d Fourier filtered image; e plot of d projected onto x-axis, orange crosses mark the peaks and the grey dotted line represents the threshold; f original image with detected chain lines. Original image courtesy of The Parker Library, Cambridge, Corpus Christi College, MS 81: Homer, Iliad, Odyssey. Quintus of Smyrna, Posthomerica, p. 9

Laid line extraction

The laid lines on medieval paper are imprints from the horizontal wires in the mould. They are densely spaced and appear at low intensity in reflected light images. For laid line density, we choose the practice of measuring lines per 1 cm. Other common practices include the measurement of centimetres per 20 lines as described in [26]. On average, medieval paper will have laid line densities of 5–15 lines per centimetre [9]. While the distances between laid lines can also be of interest, we will focus on the laid line density in this paper. Note that laid line distances can be computed from the extraction results. Our approach is, however, not tailored to have the extracted lines placed in the centre of the laid lines and distance measurements between lines can therefore be slightly imprecise. We extract the number of lines per centimetre from a patch of the image. In that, our method is more similar to works from Atanasiu [15] or van Staalduinen et al. [9] that extract an average line density as opposed to the work by Gorske et al. [5] that produce laid line density variations across the entire paper.

Similar to chain line extraction, the first step is the selection of a patch on which the laid line extraction is implemented. As the aim is to obtain the density of laid lines per centimetre, the patch needs to be large enough to contain 1 cm of the paper sheet. We select square patches at places of the paper that have the best visibility of the laid lines. These can often be found near the inner hinge of the book. Next, the spectral TV decomposition of the patch is computed. The resulting band-pass filtered image only contains the chain and laid lines (cf. Figure 6c). While for the chain line extraction Fourier filtering with a rectangular filter mask was sufficient to extract the lines, in reflected light images the method fails for laid lines. An example of rectangular filtering of the laid lines is shown in Fig. 7. Laid lines tend to have lower contrast in the images. Chain lines sit on top of the laid lines and therefore leave a larger imprint on the paper sheet in the paper making process. This has also been described as a shadow in [9]. Lower contrast of lines means that noise and paper degradation has a larger impact on any analysis of the images. Therefore, the band-pass filtered images remain too noisy to define an appropriate filter mask to extract the densely spaced laid lines in the Fourier domain. Instead, we propose the use of the Radon transform, similar to [11, 14] only for laid line detection. The Radon transform computes parallel line integrals through the image at different angles. That is, pixel values of the image are projected along lines rotated around the image centre. Lines in the images are then represented as peaks in the Radon domain. Where there is a line present in the image, the value of the line integral will be higher than at other angles or positions. The resulting image depicts the line integral value for each projection angle and projection position related to the image centre. An example of this for three lines at different angles of 45$^{\circ }$, 90$^{\circ }$ and 165$^{\circ }$ is shown in Fig. 8. The location of each laid line and the angle at which it is present can then be extracted. To this end, we determine the projection angle with the highest magnitude to obtain the orientation of the laid lines by calculating the maximum value of the Radon transform for every angle considered. We then plot the projection positions along that angle (cf. Figure 6e). Based on the resulting 1D signal, we perform peak detection. Note that we do not use any filtering in the transform domain or inverse transform as we did in the chain line extraction. However, there is a distinct relationship between the Radon and the Fourier transform through the Fourier slice theorem [27]. More specifically, the 2D Fourier transform of an image along a line at angle $\alpha$ is equivalent to taking the 1D Fourier transform of the Radon transform at angle $\alpha$ for the same image. Essentially, we leverage the Fourier transform for both the chain and laid line extraction. While the chain line extraction looks at angles of only 0$^{\circ }$ and 90$^{\circ }$ and uses rectangular filtering, the laid line extraction takes all angles into account and instead of filtering, the line location is directly extracted from the transform domain. Therefore, the basic idea for the line detection is similar. The 1D signal in the laid line extraction is smoothed using a Gaussian kernel to remove any remaining noise. We then perform the same local peak detection approach and thresholding as in the chain line detection. As the peaks in the 1D signal contain the location of the laid lines, we can count the lines per centimetre and draw the detected laid lines in the original image for visualisation (cf. Figure 6f).

Fig. 6

Laid line extraction pipeline: a original colour image with patch marked; b grayscale patch; c spectral TV band-pass filtered image; d Radon transform image depicting projection angles between 0$^{\circ }$ and 180$^{\circ }$ and the projection position; e plot line through image d at the 90$^{\circ }$ angle, orange crosses mark the peaks and the grey dotted line represents the threshold; f original image with detected laid lines. Reproduced by kind permission of the Syndics of Cambridge University Library, MS Ii.5.41, fol. 10r

Fig. 7

Laid line extraction based on the Radon transform (top) compared to extraction using rectangular filtering in the Fourier domain (bottom). The latter is highly influenced by noise and detects too many lines and at incorrect positions. Reproduced by kind permission of the Syndics of Cambridge University Library, MS Ii.5.41, fol. 10r

Fig. 8

Example for a a synthetic image showing three lines at different angles of 45$^{\circ }$ (top left), 90$^{\circ }$ (bottom) and 165$^{\circ }$ (top right). and b the corresponding Radon transform displaying a peak at the 45$^{\circ }$, the 90$^{\circ }$ and the 165$^{\circ }$ angle

Code for both chain and laid line extraction is available at https://github.com/TamaraGrossmann/HiddenKnowledge.

Results and discussion

In this section, we demonstrate the performance of the proposed algorithms for chain and laid line detection on a set of reflected light images of paper manuscripts from The Parker Library, Corpus Christi College, Cambridge, UK. We additionally discuss the parameter selection in the semi-automatic chain and laid line extraction algorithms.

Materials

The handmade medieval paper manuscripts that we use to evaluate the proposed algorithms are sourced from The Parker Library, Corpus Christi College, Cambridge and the University Library Cambridge. The Parker Library has 120 fully digitised medieval manuscripts containing paper with dates ranging from the end of the fourteenth century to the beginning of the sixteenth century. Of these 120 manuscripts, 78 are possibly written in England according to the provenance data that we have. The 78 manuscripts are paper or mixed media. The reflected light images are of high quality and openly available online including manuscript measurements and approximate dating of the writing. All manuscripts that we analyse were approximately written in the mid-fifteenth to sixteenth century. From the University Library Cambridge, we extract chain and laid lines from multiple folios of the manuscript MS li.5.41 that was written approximately 1400–1410. The quality of the reflected light images we consider in the following is high. That is, they all have a high image resolution (upwards of 3000x3000px), the paper was photographed in a well-lit setting and from an appropriate camera angle to cause the least amount of distortion. Most of the manuscripts from The Parker Library are available at a resolution of about 4000x6000px. The paper folios are chosen to demonstrate the ability of the algorithms to deal with paper degradation, discolouration, noise and ink to varying degrees. While our algorithms are able to successfully handle these types of distortion, it needs to be said that the patch selection should be made carefully. The more that the chain and laid lines are visible even at very low contrast, the less noise will be in the analysis.

With regards to computational resources, the most expensive part in terms of speed, the spectral TV decomposition, is run on a GPU. A more in depth analysis of the computation time can be found in [24, 25]. The remainder of the algorithms is run on a CPU.

Chain line extraction

For chain line extraction, images were processed as described in Algorithm 1 and results of the detected line locations were confirmed by a paper expert. For four example images, the results are shown in Fig. 9. We are able to successfully extract the chain lines and measure the distances between all detected lines. Two of the example images (Fig. 9a and b) show discolouration along the edges of the paper. These types of paper degradation are not uncommon in medieval paper. The spectral TV decomposition is, however, able to separate the discoloured, darker patches from the lighter and finer lines in the image enabling chain line detection without interference of noise. In Fig. 9a, a line is visible on the far left of the paper sheet that has not been detected by the algorithm. This line might or might not be a chain line, or more likely a fold of the page. In Fig. 9b, two of the chain line distances are significantly below the mean and therefore give an indication of the position of the watermark. Further inspection of the full page confirms the finding. On the other hand, for three of the example images shown in Fig. 9, the algorithm detects additional lines that are not in fact chain lines but rather the result of noise in the paper. This can be rectified in two ways. Either the algorithm is re-run with changed parameters (see below for parameter selection), or by making an informed decision on which lines are to be omitted in the distance measurement. The edge of the paper is one of the lines that the algorithm may detect additionally (cf. Figure 9a, b and d). However, those lines can easily be omitted by the user.

Fig. 9

Results for chain line extraction of two different papers, the top images show the reflected light image and the bottom images the spectral TV decomposed images with the detected lines drawn. a Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 151: Iohannes Auerbach OFM, Processus iudiciarius. François de Meyronnes OFM, De indulgentiis, etc., p. iiv. The average chain line distance is 2.63cm. Individual chain line distances in cm are shown in the image. The algorithm detected two additional lines, marked by red arrows. b Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 171A: Scotichronicon (Volume 1), p. xvir. The average chain line distance is 3.93cm. Individual chain line distances in cm are shown in the image. The algorithm detected two additional lines, marked by red arrows. c Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 81: Homer, Iliad, Odyssey. Quintus of Smyrna, Posthomerica, p. 55. The average chain line distance is 3.156cm. d Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 81: Homer, Iliad, Odyssey. Quintus of Smyrna, Posthomerica, p. 9. The average chain line distance is 3.37cm. The algorithm detected one additional line at the fold, marked by a red arrow

The four examples highlight the feasibility of the proposed algorithm for chain line detection even for paper with degraded quality and noise.

Laid line extraction

For laid line extraction, we process images as described in Algorithm 2 and results of the detected line locations were confirmed by a paper expert. Results for four example images are shown in Fig. 10. The proposed algorithm is able to successfully extract all laid lines of the paper patch and obtain the laid line density as the number of lines in centimetres. The paper shown in Fig. 10a has a laid line density of 8 lines per centimetre. Despite the two large folds across the paper patch that interrupt the laid lines, our algorithm is able to detect all laid lines. The folds have a higher contrast and the spectral TV decomposition can therefore separate those structures from the laid lines. Additionally, the folds are present at a different angle to the lines of interest and the plot of the Radon transform at the laid line angle will not pick up the fold. In Fig. 10b, the laid lines are obstructed by a large amount of ink. However, due to the clear difference in contrast and size, the spectral TV decomposition is able to separate ink and paper. This example demonstrates that our method detects the laid lines even in this highly obstructed case. The resulting density is 10 lines per centimetre. Similarly, in Fig. 10c and d, the ink and spot do not hinder the analysis. In Fig. 10b and d, ruling lines are visible in the original image, but the algorithm is able to extract the relevant lines nonetheless.

Fig. 10

Results for laid line extraction of two different papers, the left images in each box show the reflected light image and the right images the spectral TV decomposed images with the detected lines drawn. a Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 100: Transcripts (16th century). Simeon of Durham OSB. Geoffrey of Coldingham OSB. Tito Livio Frulovisi. Walter of Guisborough OSA. Asser, p. iiv. The laid line density measured in the 1 cm section marked by the red line is 8 lines per cm. b Detail from Cambridge University Library, MS Ii.5.41, fol. 336r. Reproduced by kind permission of the Syndics of Cambridge University Library. The laid line density measured in the 1 cm section marked by the red line is 10 lines per cm. c Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 81: Homer, Iliad, Odyssey. Quintus of Smyrna, Posthomerica, p. 55. The laid line density measured in the 1 cm section marked by the red line is 11 lines per cm. d Detail from Cambridge University Library, MS Ii.5.41, fol. 10r. Reproduced by kind permission of the Syndics of Cambridge University Library. The laid line density measured in the 1 cm section marked by the red line is 8 lines per cm

The extraction of laid lines tends to be more challenging than the extraction of chain lines. The imprint of the chain wires is often more visible, that is, it has a higher contrast. This is due to the chain wires sitting on top of the laid wires. Additionally, the laid lines not only have a lower contrast, but are spaced densely resulting in noise having a bigger influence on the results. For paper or images with reduced quality, this can lead to lines not being detected with the proposed algorithm that uses isotropic TV in the decomposition. An example is shown in Fig. 11b. Instead, we propose anisotropic TV for spectral TV decomposition in cases where isotropic TV decomposition fails. As introduced in the method section above and visualised in Fig. 3, the basic atoms of anisotropic TV decomposition are rectangular shapes. The evolution of the TV flow (1) for anisotropic TV will result in more block-shaped structures. We leverage this in cases of high noise levels or low image quality to be able to extract the laid lines. As shown in Fig. 11, we are then able to extract all laid lines despite lower image quality.

Fig. 11

Comparison of laid line extraction with a isotropic TV decomposition and b anisotropic TV decomposition. Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 78: Domenico di Bandino, Fons memorabilium uniuersi (book 5, De uiris claris), p. 3

Parameter selection

The proposed algorithms for chain and laid line extraction are semi-automated and some parameters need to be manually tuned and adjusted heuristically for each paper image depending on the image resolution, quality and lines present. These parameters are the interval for band-pass filtering of the spectral TV decomposition and the smoothing factor and thresholding for peak detection. For chain line extraction, the width of the filtering mask for Fourier filtering needs to be additionally selected.

The band-pass filtering interval is related to the image resolution. As the decomposition is dependent on the size of the structures and chain and laid lines in images with higher resolution will typically be represented by a larger number of pixels, the scale at which they appear in the scale-space representation will be larger. Conversely, images with a lower resolution will need band-pass filtering at lower scales. For the Corpus Christi manuscripts, we have found that $t \in [0.026,1]$ for band-pass filtering gives the best results. This selection ensures that small image noise is filtered out but all chain and laid line features are retained. However, the algorithm is robust to small changes in the interval and will detect correct lines with varied bands as long as the line information is contained in the spectral bands. In chain line extraction, the second parameter is the filtering mask for Fourier filtering. Johnson et al. [10] suggest a rectangular mask of 1/3 image height and 3px width. This parameter is also dependent on the image resolution and the number of pixels that the lines are represented by. For the example images shown, a rectangular mask with 2/3 image height and 1px width has given the best results, where the choice of width has greater importance. Finally, for both extraction algorithms the smoothing factor and threshold for peak detection is to be determined. This parameter is dependent on the image noise and degradation. Images with higher noise level require a larger smoothing factor. In the algorithms presented, the threshold is chosen manually, however, there are approaches to automate the peak detection and thresholding [28]. This will be part of future work.

Spectral total variation decomposition

Finally, we show that the spectral TV decomposition step in both algorithms is vital to the success of the methods presented here. While approaches such as Fourier filtering and using the Radon transform to extract the chain and laid lines have been proposed in various forms and combinations for transmitted light and radiography images (e.g. [9,10,11]), they do not work on reflected light images without additional decomposition of the images. In Fig. 12, we show results for the same algorithms 1 and 2 with and without the use of the spectral TV decomposition. For the case of chain line detection in Fig. 12a, the lines detected via the algorithm without the spectral TV decomposition (red) cluster around the region of the actual chain lines. However, the algorithm detects many more lines than are present, and the locations do not match up with the chain lines. Considering the laid line detection, Fig. 12b shows the difference in lines detected. Not only is the angle of the lines detected without spectral TV decomposition incorrect, many lines are also missed or incorrectly spaced. We have observed this phenomenon not only on the two examples presented here, but on many other patches taken from the manuscripts that we consider.

Fig. 12

Comparison of the performance with (blue lines) and without (red lines) spectral TV decomposition on the images. In a, the chain line detection is shown. Patch taken from The Parker Library, Cambridge, Corpus Christi College, MS 81: Homer, Iliad, Odyssey. Quintus of Smyrna, Posthomerica, p. 55. In b, the laid line detection is shown. The image on the left displays laid lines detected with the spectral TV decomposition in the algorithm, the middle image shows the lines detected without the spectral TV decomposition and the image on the right has both versions overlaid. The patch is taken from Cambridge University Library, MS Ii.5.41, fol. 336r. Reproduced by kind permission of the Syndics of Cambridge University Library

Conclusions

The extraction of the fingerprint of medieval paper has been at the core of much research and for many scholars. In this paper, we proposed algorithms to detect and extract the chain and laid lines from reflected light images. In that, we showed the feasibility of leveraging the so far untapped resource of reflected light images for paper analysis where most other work has been focused on radiography or transmitted light images.

We introduced the use of spectral TV decomposition for separating high contrast and large size structures such as ink and discolouration from the low contrast and fine structures that are the lines of interest. Subsequently, we proposed algorithms for chain and laid line extraction based on the decomposed images. The results clearly demonstrate the successful detection and extraction of all chain and laid lines even in cases of paper degradation such as discolouration or folds. We additionally showed an alternative decomposition approach for cases in which paper or image quality made the detection more challenging.

This proof of concept demonstrates that mathematical analysis can be used to extract features of the imprint that the mould left on the sheet of paper from reflected light images. Chain lines and laid lines can be detected from these type of images and its extraction can then offer future data for analysis at scale. These algorithms can be successfully applied to research in book history, bibliography and conservation. Our methods still require some manual fine tuning and parameter selection, however, future work will aim to fully automate the extraction pipeline to make the algorithms even more accessible to all scholars without any prior knowledge of coding or mathematical imaging and allow for the extension to large scale analysis. The work we present is not limited to the use on reflected light images, but can be applied to radiography and transmitted light images to enhance the line features via the spectral TV decomposition. Additional work will look into future extensions to low quality images.