Hyperspectral imaging and convolutional neural networks for augmented documentation of ancient Egyptian artefacts

Abstract

The study aims at investigating the use of reflectance Hyperspectral Imaging (HSI) in the Visible (Vis) and Near Infrared (NIR) range in combination with Deep Convolutional Neural Networks (CNN) to address the tasks related to ancient Egyptian hieroglyphs recognition. Recently, well-established CNN architectures trained to address segmentation of objects within images have been successfully tested also for trial sets of hieroglyphs. In real conditions, however, the surfaces of the artefacts can be highly degraded, featuring corrupted and scarcely readable inscriptions which highly reduce the CNNs capabilities in automated recognition of symbols. In this study, the use of HSI technique in the extended Vis-NIR range is proposed to retrieve readability of degraded symbols by exploiting spectral images. Using different algorithmic chains, HSI data are processed to obtain enhanced images to be fed to the CNN architectures. In this pilot study, an ancient Egyptian coffin (XXV Dynasty), featuring a degraded hieroglyphic inscription, was used as a benchmark to test, in real conditions, the proposed methodological approaches. A set of Vis-NIR HSI data acquired on-site, in the framework of a non-invasive diagnostic campaign, was used in combination with CNN architectures to perform hieroglyphs segmentation. The outcomes of the different methodological approaches are presented and compared to each other and to the results obtained using standard RGB images.

Introduction

Reflectance Hyperspectral imaging (HSI) is an imaging analytical spectroscopic technique originally born for Remote Sensing applications, but today widespread in several sectors, including the Cultural Heritage (CH) field. When the HSI technique is implemented in reflectance mode, a sequence of reflectographic images of the same scene is collected over an extended region of the electromagnetic spectrum, usually spanning the Visible and Near Infrared (VNIR) up to the Short-Wave InfraRed (SWIR) ranges. The HSI data-set is called a data-cube or image-cube, since it includes both spectral and spatial information, and is defined in a pseudo 3D-space with each datum being defined by two spatial (x,y) and one spectral (λ) coordinates. The data-cube is a stack containing a reflectance spectrum per each pixel of the imaged area. Being non-invasive, HSI was initially introduced in CH as particularly suited for investigations of polychrome artworks. Indeed, VNIR-SWIR reflectance spectroscopy is an ideal, well-established approach for the analysis of polychrome materials since it enables non-invasive identification of pigments and several coloured materials. However, the inherent technological complexity, cost and lack of user-friendly HSI instrumentation limited extensive use of this technology for a long period after its initial introduction in CH. For these reasons, until recently most of the HSI applications were primarily focussed on the study of very selected paintings and valuable artefacts [1,2,3]. In the last decade, significant technological advances have led to commercialisation of a new generation of HSI devices, featuring portability, lightness and operability on-site. This has opened new applicative perspectives, with a rapid extension of HSI to new contexts, including assets outdoors and several new types of artefacts [3,4,5,6,7,8,9]. Among the emerging applications of HSI in CH, those in the archaeological field appear as some of the most promising. The earlier applications of HSI to the investigation of pigments in mural paintings and to the retrieval of faded traits in archaeological sites are reported in [10]. In parallel, imaging spectroscopy (including hyperspectral and multispectral methods) has also started to expand in several domains of digital humanities, such as palaeography and epigraphy, as a tool to improve the readability of ancient manuscripts and faded texts on degraded supports [6, 11,12,13,14,15]. To the best of authors’ knowledge, so far HSI technique has never been applied to the study of ancient Egyptian hieroglyphic inscriptions, while recently it has been proposed for the analysis of pigments and manufacturing materials in polychrome Egyptian artefacts [16, 17].

The research here proposed originates from the observation of the enormous potential that HSI may have for the study of hieroglyphic texts. Specifically, a suitable use of the spectral images could be highly beneficial for digital enhancement of faded signs, retrieval of corrupted inscriptions and improvement of readability of texts. Indeed, considering that a peculiarity of Egyptian artefacts is that writing, symbols and polychrome decorations occur together in the same surface and are often interleaved, with colours playing also a semantic role, HSI can be regarded as a unique tool for addressing, at once, multiple goals—such as: colour analysis, augmented imaging, retrieval of texts—by means of a single non-invasive acquisition session.

Starting from these premises, in the present study, we are proposing the use of HSI as a further step of our on-going investigation into the application of Artificial Intelligence (AI) methods to the automated segmentation of hieroglyphic inscriptions [18,19,20,21,22]. In particular, authors have recently shown that a class of Deep Learning algorithms, namely Convolutional Neural Networks (CNNs) [23], can successfully address various tasks of interest for the overall challenge of hieroglyphic deciphering, notably symbol segmentation, recognition and transliteration [18,19,20,21,22]. In these previous works, CNNs architectures were primarily used to separate glyphs from their background, a procedure also known as instance segmentation, which is a necessary step for the successive transliteration of the ancient Egyptian hieroglyphs. This approach was considered interesting because it might be not only oriented to the mere transliteration task, but could also be a starting point for harder objectives, like the linguistic analysis of hieroglyphic texts, the recognition of corrupted, rewritten, and erased signs, and even the identification of the scribe’s “hand” or sculptor’s school. To this aim, well-established architectures, e.g. the Mask-R CNN [24], originally designed to address segmentation and classification tasks in natural scenarios, were re-trained using purpose-build data-sets of photographs of hieroglyphs, obtaining very encouraging results on trial data-sets of images of glyphs [18,19,20,21]. Even though the signs in the training sets were selected from sharp and clearly readable images, in practical cases antique objects may have degraded surfaces, so that the capability of recognition of CNNs can be compromised by the presence of corrupted, rewritten, and erased signs. A possible solution to this issue could be including damaged hieroglyphs into the training set to improve detection accuracy. A different strategy is proposed here.

In this study, we investigate the additional contribution to network segmentation performance that could result from using the rich HSI information: in fact, it is expected that some critical segmentation regions might be better revealed by using the wider spectrum of HSI data-cubes with respect to standard RGB images. We propose novel methods to apply CNN segmentation networks to HSI data rather than RGB images. Such an extension is not straightforward, since original methods are not able to deal with hundreds of bands. Two possible methods to exploit the richness of HSI data for feeding the segmentation network are proposed. All these aspects are investigated starting from a simplified, yet real, context by using HSI data acquired within an in-field measurements campaign. Specifically, the results of a pilot study—carried out on an original Egyptian coffin belonging to the collection of the Fiesole Ethnographic Museum—are presented. The sarcophagus features an extended hieroglyphic inscription, lying along the central band of the body and bearing numerous symbols that are partly degraded, but still recognisable. Such an inscription was thus used as a test set for combining HSI and CNN. Thanks to the use of a HSI portable camera, a series of data-cubes of the hieroglyphic symbols were acquired in order to obtain augmented images to feed a CNN previously developed [20] for hieroglyphic segmentation tasks. The potential advantages of using either raw or pre-processed HSI were explored in a series of demonstrative tests. Far from claiming to be conclusive, the present paper aimed at studying the feasibility of combining HSI and CNN as a novel methodological approach in the wider context of AI methods applied to the hieroglyphic language analysis.

Materials and methods

Case-study: the Egyptian coffin at the Fiesole Ethnographic Museum

The examined object is a wooden coffin belonging to the Egyptian collection of the Franciscan Ethnographic Museum in Fiesole, near Florence (Italy). This museum, founded in 1920 by the Franciscan friars, hosts a heterogeneous, yet valuable, collection of historical and archaeological objects which were gathered worldwide during the missionary activities of the friars. The Egyptian nucleus of the collection was collected from 1923 to 1929 and today includes about 240 items, some of which are of great archaeological interest. The polychrome wooden coffin here examined is dated approximately to the XXV Dynasty, 747–656 BC. The central part of the coffin features a hieroglyphic inscription, laying along the entire length of the lid. Despite some lacking parts, the text has been recognised as a ritual formula to wish post-mortem survival to the departed. The part of the inscription located on the feet, which traditionally report the name of the departed, is lost, appearing fully abraded and thus preventing further attribution. Nevertheless, a sentence referring to the “Lady of the house” indicates that the coffin was likely dedicated to a woman. The wooden sarcophagus also hosts a still intact human mummy, whose provenance and dating remain however uncertain. Indeed, previous analytical studies report that the mummified body belongs to a 25 to 40-year-old male [25], thus suggesting that the sarcophagus does not contain its original body, and that it has been somewhat altered over the centuries. Despite this evidence, today the ensemble of coffin and mummy is kept together, as they arrived at the Museum and are displayed in their historical showcase (see Fig. 1). This is a wooden structure with glass walls, and is considered by itself a valuable item of the historical collection. The surface of the sarcophagus presents several signs of degradation on the painted surface, featuring lacunae, flaking areas and several cracks in the wooden structure. Thus, it was deemed mandatory to perform HSI measurements in-situ, without any displacement of the coffin. The fragility of the showcase poses further practical constraints for measurements.

Fig. 1

The showcase with the Egyptian coffin and the human mummy on display at the Franciscan Ethnographic Museum in Fiesole

HSI experimental set-up and data-processing

Portable VNIR hyperspectral imaging

In this study, a portable push-broom HSI camera, mod. Specim IQ^®, was used. The camera is designed for in-field applications, and is very compact (207 × 91 × 74 mm) and light (1,3 kg). It operates in the VNIR region, with a nominal spectral range 400–1000 nm and 7 nm spectral resolution. The CMOS sensor is 512 × 512 pixels. The camera can be mounted on a tripod and can operate at variable distances (from 10 cm to infinity). The F/number is 1,7 and the FOV is 31 × 31 deg., with manual focusing. The optimal working distance is established on the basis of the operational needs, taking into account that when the distance increases, then also the framed area increases at the expense of a poorer spatial resolution, and vice-versa. In the case examined here, an average working distance of about 50 cm was chosen, using a bespoke mechanical structure to hang the camera above the coffin lid. HSI acquisition was performed from the open top of the showcase, thus avoiding the displacement of the object (see Fig. 2). By moving the camera to subsequent positions at fixed distance from each other, 22 juxtaposed frames were imaged tiling the entire lid surface. Three-dimensionality and non-flatness of the target entailed slight variability among the different imaged frames: considering the chosen average working distance, each frame corresponds approximately to a square area of about 20 × 20 cm on the target surface. Calibration of reflectance was performed before the measurements session (white reference “custom-acquisition” mode) by acquiring the image of a white Spectralon^® certified target, that was placed close to the area of interest on the lid surface, upon the same illumination conditions. The IQ camera is equipped with an internal viewfinder camera, which displays in real time the HSI acquisition and provides a RGB photograph of the same area. The output of each acquisition is a HSI data-cube comprising 204 spectral bands in the 400–1000 nm interval, along with an associated RGB image, as well as an identification file number. The sequence of the acquired HSI data-cubes is reported in Fig. 3. In order to test the ability of the Mask R-CNN to segment hieroglyphs [20], only the data-cubes containing glyphs were selected from the sequence, namely those identified with the numbers: 526, 527, 529, 531 and 533, which are shown in Fig. 4.

Fig. 2

The experimental set-up adopted to perform the in-situ measurements campaign on the Egyptian coffin: the HSI portable camera along with the illumination system on the bespoke mechanical structure

Fig. 3

A view of the sequence of the areas acquired with the HSI camera to cover the entire coffin surface: the white box in each frame corresponds to the imaged area of the HSI data-cube, comprising 204 bands in the VNIR range

Fig. 4

The selection of the HSI data-cubes presenting inscriptions and used in the segmentation tests. The identification number of each data-set is also reported

HSI data processing

In general, a HSI image-cube can be regarded as either a collection of spectra or a collection of spectral images of the same scene. Such a dual nature of HSI data makes this technique particularly attractive for the investigation of polychrome artworks and decorated surfaces. Indeed, the outputs of algorithms applied to HSI data-cubes can be maps, false-colour or grey-scale elaborated images, which are superimposable to the RGB image and, at the same time, highlight details that are not visible to the naked eye. Thus, by acquiring spectral imaging data in the VNIR up to the SWIR ranges, not just the artwork surface, but also inner features, which might be concealed under the pictorial layers (like underdrawings, pentimenti, signatures, retouches, restorations, etc.), are easily visualised and registered to the RGB.

Since HSI data are inherently redundant, significant information needs to be extracted with suitable algorithms, such as data-reduction techniques. Usually, the HSI data-processing chain involves use of suites of different multivariate and statistical methods. The choice of these algorithms strongly depends on the applicative contexts and on the scientific queries. In pre-processing the HSI data acquired on polychrome surfaces, one of the most well-established techniques is Principal Component Analysis (PCA) [26]. This method is used not only for data-dimensionality reduction, but also to facilitate the straightforward visualisation of salient features which might not be discernible to the naked eye. From a technical point of view, applying PCA to the HSI data-cube reduces the original stack of spectral images (typically hundreds) to a short sequence of a few (usually less than ten) images, which still show the whole scene but highlight new details, or areas deserving attention, not evident in the colour RGB [3]. This enhancement effect is because the PCA algorithm stresses the internal variability of spectral data, thus strongly differentiating the areas of the pictorial surface that differently respond to the light. This means that PCA images enhance changes of materials, retouches, different thickness, etc., which emerge in spite of their similar colours or their same appearance in the RGB image. Mathematically, the PCA algorithm operates a change (rotation) of the axes of the original representation space into a new coordinate system (PC space). This new representation enables a reorganisation of the original data by concentrating the meaningful informative content into the few first variables, while the remaining ones can be neglected. This is the key of dimensionality-reduction. When dealing with HSI data, the original representation space is a n-dimensional hyperspace with as many axes as the number of wavelengths (variables). In the data-cube, each pixel of the imaged area is associated with a reflectance spectrum and it is represented as a point in the n-dimensional hyperspace, with n equal to the number of imaged bands (wavelengths). However, the information contained in a reflectance spectrum is spread out all over the spectral interval, with some wavelengths (variables) being correlated with others. The PC space is a new set of axes, which: a) are orthogonal with each other and b) are oriented along the directions of maximum variance of the data. The PCs are linear combinations of the original variables; they are uncorrelated and, more importantly, are hierarchically ordered, meaning that: the first axis, PC1, is along the direction of maximum variance of the data-set, the second axis, PC2, is chosen as orthogonal to PC1 and along the residual maximum variance, and so on. Since the variance of data is connected to their informative content, the salient information of the original data-set is now summarised by the first few PCs variables, while the residual ones include noise and can be neglected.

In practical cases, the sequence of the first PCs images is expected to reproduce the salient contents of the original spectral sequence of the HSI data-cube. This aspect is exploitable in presence of inscriptions with corrupted symbols and degraded decorations, where the PC images can be used to retrieve the legibility of signs [10]. Besides PCA, other methods can also be adopted to pretreat HSI image-cubes with the aim of improving legibility, such as Minimum Noise Fraction (MNF), Independent Component Analysis (ICA), or simpler methods such as single spectral bands selection, or mathematical operations (e.g. bands ratio, subtraction, addition, derivative, etc.). Once the HSI processing work-flow is established, a set of output elaborated images is obtained, emphasising different salient aspects of the original scene. In this study, ENVI^® NV5 Geospatial Software was used to process the HSI-data-cubes. When applying the PCA transformation to the data-cubes, the method of Covariance Matrix was used.

In the present study we limit the treatment to the more basic approach. We assume that, instead of conventional RGB photographs, enhanced HSI images could be used as input to CNN architectures trained for symbol segmentation, so as to improve the net performances when applied to degraded symbols. As preliminary tests two alternative approaches were proposed to provide HSI-data as CNN inputs: using selected PC images, or using the entire raw sequence of single spectral bands. The details of such processing chains are reported in Sect. “Segmentation”.

Segmentation

Architectures for image segmentation

Segmentation of images is a basic task in image processing, often used as the initial step in a chain of operations to reach more complex objectives. In the field of hieroglyphs (and, in general, of any other text) transliteration, segmentation is a preliminary action needed to separate any instance of an object present in a picture, where, as in object-oriented programming, by instance we mean a specific realisation of a class of objects.

In this work, we investigate whether segmentation may benefit from the extremely rich information contained in HSI data. In the following, some commonly used CNN, previously proposed for the segmentation of grey level and colour images, are reviewed, with a specific focus on those CNNs that have been selected in this study for the purpose of hieroglyph segmentation.

Mask R-CNN architecture [24] is one of the most popular networks proposed for segmentation; it is available thanks to the Detectron2 platform [27] and has been developed by the Facebook AI Research Group. Mask R-CNN is an extension of an already existing structure, Faster R-CNN [28]. Faster R-CNN combines two tasks in a single network, both based on CNNs: detecting the regions of the image that are likely to contain an object of interest (Regions of Interest, RoI), and performing the classification of the objects within the RoI. The former task is faced by the Region Proposal Network (RPN), which basically is a Fully Connected Network (FCN) that returns the RoIs as bounding-boxes, as well as, for each of them, an “objectiveness score”, i.e., an index related to the confidence of finding an object in that window. The latter task operates on each detected RoI and it is in charge of refining the final bounding-box and assigning a classification score. It is important to underline that finding bounding-boxes and providing classification scores are the outputs of two parallel branches; even though such operations are independent, in order to speed up the training process, the RPN and the classifier share the first convolutional layers, reducing the amount of parameters that need to be learnt by the network. With respect to the architecture of Faster R-CNN, Mask R-CNN adds a third branch, represented by a simple FCN, which is responsible for creating precise segmentation masks of the object within the RoI; examples of such masks, which show the hieroglyphs as white objects over a black background, can be seen in the rightmost pictures within Figs. 5 and 6. Again, the masking branch is independent from the other ones, so that the results obtained for bounding-boxes regression, classification and segmentation are unrelated.

Fig. 5

Scheme of the processing chain of the PCA-Seg method

Fig. 6

The processing chain of the SBC-Seg method

In this work, we used Mask R-CNN, driven by a Python code developed to interact with Detectron2’s [29] API. The default backbone used by Detectron2 is the ResNet50 with a Feature Pyramid Network (FPN) [30], used to extract multiscale features. Detectron2’s API lets the user set a large variety of parameters, in order to fully customise the network and find the most suitable one for a specific application.

Algorithms for HSI hieroglyph segmentation

In this study, we focus primarily on the problem of hieroglyphs instance segmentation; the task of classification has been treated in previous works (see, e.g., [18] and references therein).

In [19, 20], Mask R-CNN has been configured for the segmentation of ancient Egyptian hieroglyphs contained in RGB or grey-level images, as those captured with standard digital photographic cameras, while the network parameters (like number of iterations in the learning process, thresholds used to discard instances, etc.) were selected on a trial-and-test basis.

In this study, we extend the use of the same architectures to the segmentation of augmented images, as those provided by HSI data in order to retrieve the readability of lost or corrupted symbols. To this aim, two different approaches were tested, considering that HSI data processing can be introduced at different stages of the algorithmic chain.

In the first proposed method, the HSI data-cube is processed at first by using PCA, and then by applying CNN for segmentation separately to each selected PC image, that is PCA1, PCA2, etc. The final result is a segmentation map per each PC image used in input. In the following, we will refer to such a method as PCA-Seg, whose scheme is sketched in Fig. 5.

In the second approach, the CNN segmentation stage is applied to each spectral image of the HSI data-cube, thus providing a sequence of masks (referred to as single bands masks). Subsequently, all the output masks (as many as the number of HSI bands) are suitably combined to achieve a unique final segmentation map. The simplest way to combine the single band masks is associating, for each pixel, the value 1 to the presence of a hieroglyph and 0 to the background, and then summing up the resulting arrays. After that, applying a threshold to the accumulated values yields the final classification of each pixel as belonging to a hieroglyph or background. This method will be referred to as the single-band and combination segmentation (SBC-Seg) approach. The SBC-Seg scheme is sketched in Fig. 6.

From a signal processing standpoint, the main difference between these two algorithmic chains is where deep learning comes into play to process the HSI data: in the SBC-Seg approach, the segmentation network operates on the original raw HSI data, whereas, in the PCA-Seg approach, CNN operates on HSI data only after a preprocessing algorithm, i.e., the PCA, which produces a reduction of dimensionality.

Evaluation metrics

The two proposed methods have been applied to the data-sets denoted as 526, 527, 529, 531 and 533 (Fig. 4) and the results have been compared.

In the results that will be discussed below, for the PCA-Seg method the first ten PC images were considered, whereas for the SBC-Seg approach all the available 204 spectral bands covering the 400–1000 nm range were used.

As to the assessment of the performance of the two methods, both qualitative and quantitative evaluation criteria have been used. The results were qualitatively evaluated by using visual inspection: indeed, the visual comparison of the output mask with respect to the original image (e.g., in RGB format) is one of most reliable ways to ascertain if a method yields satisfying results. Visual evaluation is simple and feasible when the test dataset is limited, like in our case; however, it is a subjective criterion and is not automatable.

In order to provide also a quantitative way for evaluating the performance of the two methods, an index known as Intersection over Union (IoU) was used [31]. The IoU is evaluated by comparing the prediction segmentation mask, that is the output of the CNN, and the ground truth, i.e., the “actual” segmentation mask that, in our case, has been manually extracted by a human for each symbol contained in the examined frames. In the practice, the IoU is calculated as the ratio of the area of the intersection between the prediction and the ground truth masks, over the area of their union. From its definition, it is apparent that the IoU ranges from 0 to 1, with 1 indicating a perfect superposition of the predicted mask onto the ground truth.

Results and discussion

In this section, the results of the segmentation obtained on the HSI cubes shown in Fig. 4 are discussed. As can be seen, these images contain blocks of several hieroglyphs that constitute the coffin inscription. Each image included a variable number of symbols. The results obtained with the two segmentation approaches previously described, i.e., PCA-Seg and SBC-Seg, are compared, taking into account the two evaluation criteria based on visual inspection and IoU. Performances of these methods are also compared to each other and to those obtained from standard RGB images. In all cases, the Detectron2 segmentation network is applied.

Results by using PCA-Seg

As previously mentioned, the PCA pre-processing of HSI data-cubes produces a sequence of grey level images, referred to as PCn. In the PCA-Seg method, segmentation is simply obtained by feeding the PCn image to the segmentation network. Figure 7 shows some examples of the segmentation masks produced by the network when the input was each single PCn, with n = 1,2,…,10, obtained from the data-set n. 529, which is taken here as an illustrative example. As can be seen, even though the first PC image is visually closer to the grey-scale version of the RGB image, the most salient information for segmentation of the symbols can be recovered from the successive PCs. This fact is also confirmed by the objective evaluation criterion, i.e., by the results in terms of IoU given in Table 1 for every analysed HSI cube and for the first ten PCs. As can be seen, for the data-cube n. 529, depicted in Fig. 7, the highest score is obtained for PC3, for which, indeed, symbols shapes appear as better recognisable. Looking at the results of Table 1, with reference to all HSI data-cubes, they demonstrate that the segmentation results are highly dependent on the PCs and not necessarily the most energetic ones yield the best performance.

Fig. 7

Examples of single PC segmentation for the dataset n. 529. In the upper part, the PCs, ranging from 1 (leftmost) to 10 (rightmost); in the bottom part, the relative masks obtained after applying the segmentation CNN

Table 1 IoU index for the different HSI data-cubes and for each of the first ten PC extracted from them

Results by using SBC-Seg

Unlike the PCA-Seg method, SBC-Seg works on the raw sequence of HSI data and segmentation is directly applied to each spectral band. Figure 8 shows some examples of segmentation masks achieved at given spectral bands extracted from the HSI data-cube n. 526, which are shown here for illustrative purposes. In SBC-Seg, for each pixel a number of classifications (hieroglyph/background) as large as the number of bands (in our case, 204 bands) is available.

Fig. 8

Examples of single band segmentation. The masks are obtained from spectral images extracted at the wavelengths; 400, 420, 563, 586, 631, 690, 871, 972 nm from the data-cube n. 526

A quantitative assessment of the classification relative to each single band is shown in the curves plotted in Fig. 9, where the IoU computed in each band and for all the analysed HSI data-cubes is plotted; this also allows the contribution of the single spectral bands to the final accumulated map to be estimated. As expected, the segmentation results are highly dependent on the band that is considered. Even though it is difficult to uniquely identify a band (or a spectral interval) yielding the best results, it seems that for all the cubes poor performance occurs for wavelengths approximately below 600 nm, whereas for wavelengths above 600 nm the IoU increases and reaches a plateau above 700 nm.

Fig. 9

IoU index computed in each band and for every analysed HSI data-cube

In order to achieve a final unique segmentation mask, we need to combine all the SBC-Seg results. As already mentioned, each mask resulting from a single spectral band classification is processed as a binary array, with 0 corresponding to the background and 1 to a hieroglyph: the sum of all these arrays represents the occurrence, for each pixel, of being classified as a hieroglyph. Figure 10 shows the frequencies (normalised to their maximum) with which each pixel is classified as belonging to a hieroglyph. The final segmentation can be achieved by thresholding the number of occurrences. Figure 11 shows, for every analysed HSI data-cube, the IoU curves that can be achieved for different choices of the threshold applied to the cumulated array; some numerical values are reported in Table 2 for three different values of the threshold.

Fig. 10

Pixel occurrence of being classified as a hieroglyph during single band segmentation

Fig. 11

IoU computed for every HSI cube on the segmentation masks achieved after applying a threshold on the cumulated classification arrays vs. the threshold value

Table 2 IoU index, for each HSI data-cube, for different choices of the thresholds used on the cumulated classification arrays

Discussion

The two proposed methods, PCA-Seg and SBC-Seg, are compared to each other and also to the results obtained from the network applied to the RGB images of the HSI data-cubes.

Figure 12 shows a pictorial view of the overall results of the segmentation, obtained from the RGB image, two of the most significant PCs (PC1 and PC3), and two choices of the threshold in the SBC-Seg method. The ground-truth, i.e., the manually-extracted segmentation mask, is reported as well.

Fig. 12

Examples of segmentations obtained from the RGB image, the PCA-Seg method (PC1 and PC3) and the SBC-Seg method (for two choices of the thresholds applied to the cumulated classification arrays); figures from (a) to (e) refer to the various data-cubes (indicated in the top-left corner)

For a quantitative, though preliminary, comparison, we report in Table 3, for each HSI data-cube, the values of the best IoU obtained by using each method (choosing the optimal PC from Table 1 and the best threshold from Fig. 11). Comparing the performances of the two proposed methods, PCA-Seg and SBC-Seg, on the five data-cubes we can see that there is no a clear predominance of one method over the other: in two cases performances are almost equivalent (cubes 527 and 529), in one case PCA-Seg surpasses SBC-Seg (cube 526), in two cases the viceversa (cubes 531 and 533). However, the HSI-based segmentation methods outperform the RGB in three cases. In addition, looking at the averages values of IoUs shown in Table 3, we observe that the PCA-Seg and the RGB methods yields the same performance, whereas SBC-Seg seems superior.

Table 3 Comparison of the performances, in terms of IoU, obtained by using different methods of segmentation

These findings are based on a limited dataset and cannot be considered as statistically representative for drawing general guidelines. However, it is crucial to highlight that in specific cases the automated segmentation workflow highly benefits from the use of the HSI richness. Indeed, it can be observed that HSI methods outperform the RGB one for data-cubes 529, 531, and 533 that are exactly those featuring extensive lacks in the paint layers and appear visually more degraded. In fact, in these cases, IoU values are significantly lower for RGB images than for HSI data, clearly indicating that failures of CNNs due to the presence of corrupted symbols can be remediated by resorting to HSI methods. Therefore, the obtained results indicate a promising path for overcoming the scarce readability of RGB images in the CNN automated segmentation process by means of suitably processed hyperspectral images.

Summarising, the richness of HSI data-cubes emerges as an important feature to be exploited in recognition tasks, especially in the presence of evident degradations or artefacts, even though the signal processing chains need to be tuned on the kind of imaged objects and on the specific application [32, 33].

Conclusions

In this work we explored the possibility of automatically identifying ancient hieroglyphs on the surface of an ancient Egyptian artefact by combining the use of trained CNN and hyperspectral images. As it has been proved by several applications in the cultural heritage field, HSI technology can provide a spectral representation of a scene much richer than standard RGB images. The richness of HSI-data can be highly advantageous for retrieval of faded and corrupted signs in ancient documents, and is here considered in the framework of the ancient Egyptian writing and hieroglyphic segmentation. Specifically, in this study we have investigated how HSI and deep learning techniques can be combined to perform the segmentation of the hieroglyphs even when inscriptions are partially degraded. A segmentation network, based on convolutional neural networks and operating on grey level or RGB images, was specifically trained for hieroglyph segmentation in previous works by the authors. In this paper, we have presented possible ways to exploit such a network to segment images obtained from HSI-data, which is not a straightforward task. Two strategies are here proposed: the first method uses the dimensionality reduction provided by PCA—a classical HSI pre-processing—and then applies a segmentation network to the resulting data; the second method applies the segmentation network directly to every spectral image composing the HSI cube, and then uses all the resulting masks to single out the presence of hieroglyphs. The two proposed methods were validated by means of a case-study, that is the inscription of an Egyptian coffin (XXV Dynasty) belonging to the Franciscan Ethnographic Museum in Fiesole (Italy). Using a portable HSI camera the coffin lid was non-invasively imaged in-situ and a HSI dataset comprising five HSI data-cubes, each including a set of hieroglyphic symbols in different states of degradation, was obtained. These data were used to test the above mentioned processing chains and evaluate their effectiveness. A quantitative criterion for evaluating the performances of the HSI-based methods in comparison with the standard method RGB images was also introduced. Even though a clear predominance of a single method does not emerge, the obtained results clearly indicate that the automated segmentation workflow can greatly benefits from the use of HSI-images. In particular, in the presence of data-sets including degraded symbols the quantitative evaluation criteria indicated that one of the proposed methods, the SBC-Seg one, produces noticeably better performances than the others. Even though preliminary, the obtained results are promising and clearly demonstrate the potential of the proposed methodological approach as a new tool to segment HSI. These preliminary results highlight the soundness of exploiting the spectral richness of information contained in the HSI data as input of the CNNs architectures.