Web-based diagnostic platform for microorganism-induced deterioration on paper-based cultural relics with iterative training from human feedback

Abstract

Purpose

Paper-based artifacts hold significant cultural and social values. However, paper is intrinsically fragile to microorganisms, such as mold, due to its cellulose composition, which can serve as a microorganisms’ nutrient source. Mold not only can damage papers’ structural integrity and pose significant challenges to conservation works but also may subject individuals attending the contaminated artifacts to health risks. Current approaches for strain identification usually require extensive training, prolonged time for analysis, expensive operation costs, and higher risks of secondary damage due to sampling. Thus, in current conservation practices with mold-contaminated artifacts, little pre-screening or strain identification was performed before mold removal, and the cleaning techniques are usually broad-spectrum rather than strain-specific. With deep learning showing promising applications across various domains, this study investigated the feasibility of using a convolutional neural network (CNN) for fast in-situ recognition and classification of mold on paper.

Methods

Molds were first non-invasively sampled from ancient Xuan Paper-based Chinese books from the Qing and Ming dynasties. Strains were identified using molecular biology methods and the four most prevalent strains were inoculated on Xuan paper to create mockups for image collection. Microscopic images of the molds as well as their stains situated on paper were collected using a compound microscope and commercial microscope lens for cell phone cameras, which were then used for training CNN models with a transfer learning scheme to perform the classification of mold. To enable involvement and contribution from the research community, a web interface that actuates the process while providing interactive features for users to learn about the information of the classified strain was constructed. Moreover, a feedback functionality in the web interface was embedded for catching potential classification errors, adding additional training images, or introducing new strains, all to refine the generalizability and robustness of the model.

Results & Conclusion

In the study, we have constructed a suite of high-confidence classification CNN models for the diagnostic process for mold contamination in conservation. At the same time, a web interface was constructed that allows recurrently refining the model with human feedback through engaging the research community. Overall, the proposed framework opens new avenues for effective and timely identification of mold, thus enabling proactive and targeted mold remediation strategies in conservation.

Introduction

Paper has been one of the main mediums of communication and knowledge representation throughout history and across the world. However, due to its cellulose composition, paper is susceptible to various physical, chemical, and biological agents present in the environment, particularly to microorganisms such as molds [1]. Aged papers are especially prone due to spontaneous and environment-induced acidification, which creates a hospitable environment for microorganisms to grow and reproduce [2]. The byproducts of microorganisms’ growth and metabolism contain enzymes and acidic compounds that will disintegrate the fibrous cellulose structure of the paper, resulting in both aesthetic and mechanical deterioration to the paper material itself as well as the information it carries [3]. Furthermore, given the omnipresence of microorganisms [4], there are constant threats of contamination to paper-based materials. The contamination process is usually undetectable by the naked eye, making it difficult to intervene at an early stage [5]. When the contamination is visibly apparent, a substantial colony composed of microorganisms has already been engraved in the cellulose structure of the paper, causing irreversible deterioration, making the time window for effective remedial measures highly limited. Thus, the impacts of microorganisms on books, documents, and other paper-based materials may result in inestimable cultural losses [1, 6] and may induce huge financial burdens [7, 8]. Therefore, effective detection and treatment of bio-contamination due to microorganisms’ contamination are crucial for the preservation of unique artifacts for humanity and future generations [9].

Current mold remediation practices in conservation are generally wide-spectrum. A general workflow consists of mechanical cleaning of microorganism bodies through brushing and vacuum cleaning and sometimes coupled with chemical or physical treatment to further eradicate the microorganism spores [10]. Chemical methods involve fumigation using membrane-active microbicides, such as alcohol, salicylanilides, and quaternary ammonium salts. These microbicides coat the cell wall and then cause damage to the structure of the mold bodies. Alternatively, electrophilically active microbicides such as aldehydes and organometallic compounds act through electrophilic addition or substitution to cause enzyme inactivation [1]. These wide-spectrum biocides, however, could induce more rapid mutations in the microorganism strain, and elevate environmental risks and public health concerns [11]. On the other hand, physical techniques involve dehydration, radiation, and deoxygenation that perturb the hospitable environmental conditions that are essential for molds to survive and further reproduce [1]. However, the extreme environmental conditions applied in the physical methods could subject the delicate ancient paper material to secondary damage. The development of species-targeted approaches has the potential to minimize the risk of secondary destruction and aligns with the conservation principle of “minimum intervention” [12]. For example, species-targeted biocides could be designed based on different biocidal resistances observed in different microorganism species [13]. After all, successful implementation of such treatments relies heavily on fast and accessible microorganism identification methods.

Additionally, the identification of mold species has the potential to resolve practical and safety problems in conservation work settings. First, different consortia of microorganisms display varying mechanisms of destruction [6], making it challenging for conservators to provide an accurate condition report and propose the optimal treatment options without knowing the contaminating species. However, if the strain of the mold is known when diagnosing the material, the best approach to conserve can be devised accordingly. More importantly, according to the Centers for Disease Control and Prevention (CDC), mold is associated with many health complications. People who have access to or frequently attend to the contaminated materials are subjected to potential health risks given the presence of mold [6, 14]. Molds can invade living tissues and cause diseases or allergies, such as allergic skin, respiratory symptoms, or gastrointestinal disorders [6, 15]. The pathogenicity, transmission route, and infection severity are also distinctive with various species. As a result, to ensure efficiency and safety in the treatment of microbial contamination, species of the mold should be carefully identified and evaluated [3].

The identification of mold species before conservation practices is not common in the current conservation field. This is primarily because the mainstream identification techniques, namely culture-based and molecular biological methods, are not widely accessible from the conservators’ point of view [15]. Culture-based techniques rely heavily on culturing the microorganisms in a qualified laboratory environment, which is time-consuming in terms of both making connections with specialized labs and conducting the analysis. Meanwhile, molecular biology techniques that directly analyze the sequence of the target organisms, even though has high authenticity and accuracy, take a prolonged period to generate results [8, 16]. For example, pyrosequencing, which is based on the luminometric detection of pyrophosphate can achieve an accuracy of 99% but a single run takes 24 h [17]. Illumina, a sequencing method based on a reversible dye-terminator and engineered polymerase can achieve single-shot accuracy of 99.9%, however, the operation time is 1–11 days [18]. Another significant drawback of sequencing techniques is that, to obtain sufficient amounts of sample, they require direct sampling from the object being conserved, which is considered invasive and would induce secondary damage, especially to delicate historical material [3]. In summary, fast and non-invasive tools are still absent for conservators and other practitioners in everyday work.

AI-enabled applications are gradually integrating into various aspects of our lives. Benefiting from the quick advancements in artificial neural networks called deep neural networks (DNN), more and more challenging tasks that once required profound human expertise can now be achieved by computers [16, 19]. In particular, computer vision (CV), a branch of DNN application that enables computers to identify and understand objects and people in images and videos [19], is gaining momentum in assisting with expert-level image and video-related tasks and can capture intricate features that may not be apparent to the naked eye [20]. For example, CV has shown huge potential in the field of medicine to enhance healthcare workflows [20]. A team of experts in Poland applied DNN and a bag-of-words approach to classify microscopic images of various fungal species. The approach significantly reduced the diagnostic time from 4 to 10 days down to less than 3 days. This not only allows faster decisions regarding antifungal treatments to be prescribed to patients who had a fungal infection, resulting in shorter recovery time but also reduces the cost of diagnosis due to the replacement of biochemical testing procedures [21]. There have also been attempts to integrate DL techniques into conservation practices. For example, Hatir et al. proposed to use deep neural networks to assist with the task of classifying the weathering condition for historical stone monuments which can help in conservation and restoration practices [22]. Another team in China proposed using FSNet to automatically recognize and count fungal spores in microscopic images to help monitor grain storage and detect signs of spoiling and fungi contamination [23]. Given the nearly real-time feedback and minimal invasiveness, CV is an ideal candidate for diagnosing delicate cultural heritage like paper. Thus, this study investigated the feasibility of a CV-based diagnostic tool for identifying mold strains on paper-based relics non-invasively to assist conservation practices. Convolutional Neural Network (CNN) architectures, a type of mainstream CV algorithm, were experimented with to classify the strains with microscopic images. At the same time, the study provides a framework to allow the tool to grow in tandem with the growing knowledge in the field of conservation.

Methods

Sample preparation

Microscopic images of the visually apparent stained regions on paper were collected for modeling “Modeling” section. The sample preparation process involves extracting the molds from ancient artifacts, creating contaminated mockups for the ancient artifacts, and collecting microscopic images for model training (Fig. 1).

Fig. 1

Sample acquisition pipeline

Mold acquisition

Microorganisms were sampled and revived from four ancient archives with minimal intervention, including “Sequel of Comprehensive Reflections to Aid in Governance” from the Ming Dynasty, “Supervised Copy of I Ching” from the Qing Dynasty, “True Interpretation of Journey to the West” from Qing dynasty, and “Veritable Records of the Qing Dynasty” from Qing dynasty (Fig. 2B), that were stored in the department of rare books and special collection at Liaoning University Library. Over 10 different strains of microorganism were resuscitated from the books, and four strains of molds that were most commonly found to contaminate paper-based materials and pose more serious threat were selected for this project: Aspergillus niger (A. Niger), Aspergillus oschraceus (A. Oschraceus), Cladosporium, and Paenibacillus polymyxa (P. Polymyxa) [6, 24,25,26,27]. A. Niger is one of the most commonly mentioned molds in literature. Besides producing cellulolytic enzymes that can digest cellulose, A. Niger can also secrete a wide spectrum of biological enzymes, including amylase, pectinase, and protease, whose by-products, can increase the overall acidity in the paper, and then induce severe loss of papers’ folding endurance in the long-term [25, 28]. Cladosporium can also secrete cellulolytic enzymes. Even though its activity is not as high as that of A. Niger, Cladosporium is capable of secreting pigments with high concealment over the original material [29,30,31]. P. Polymyxa is a type of bacteria that is not only potent in protein hydrolysis and cellulose degradation but also tends to secrete humus compounds consisting of polysaccharide, lipid, protein, and nucleic acid, which cause adhesion between papers [32, 33]. A. Oschraceus was selected for its potency in producing spores and causing large-scale air contamination that may induce asthma and even cancer for people who have been in contact with the contaminated materials [34]. The following step-by-step procedures outline the strain extraction process:

• 1. Visually identify regions of paper that show significant colonization of microorganism contamination from ancient books.

• 2. Gently swab the discolored region to collect microorganism strains and place the swab, together with liquid beef extract peptone medium (BPM), in a conical flask and culture the suspension in a lab shaker at 37 ℃ for 125r/min for 12–72 h.

• 3. Use dilution plating to transfer bacteria and fungus from liquid-state medium to solid-state BPM and potato dextrose agar medium (PDA) respectively and cultivate for 1–3 days at 28 ℃ for initial isolation and purification.

• 4. Select an isolated colony for streak plating and cultivate for 2–7 days at 28 ℃ for secondary isolation and purification.

• 5. Store the isolated strains using slant streaking at 4℃ for later use.

• 6. Use microorganism colony characteristics for preliminary strain classification and then use 16 s rDNA, and 18 s/ITS rDNA for strain determination for bacteria and fungus respectively.

• 7. Select the aforementioned four microorganism strains and suspend them in a liquid medium. Place in lab shaker at 28 ℃ for 125r/min to further proliferate.

• 8. Centrifuge the suspension when reaches the logarithmic growth phase, collect condensation (microorganism bodies), and re-suspend in sterile water. Save the solution containing the suspended microorganisms for creating mockup samples.

• 9. Bio-waste disposal: The residual material, including all apparatus that came in contact with the microorganism such as the contaminated paper, underwent neutralization procedures as enforced by the health administration agency.

Fig. 2

Image processing and modeling pipeline. A Microscopic images of the strains being classified. B Ancient paper artifacts were used to retrieve the mold strains, and strain growth in the culturing environment. C Image augmentation pipeline with random cropping and rotation. D Transfer learning modeling architecture with different partial training paradigms

Mockup construction

Because the ancient documents that the molds were sampled from are fragile due to natural aging and are off the limit for frequent access, mockups of mold-contaminated papers were constructed. The microorganism suspension solutions, obtained in “Mold acquisition” section, are evenly sprayed onto pre-cut 15 cm × 15 cm papers using a fog spray, with three duplicate samples for each strain. The sprayed papers are then sealed in Ziplock bags to avoid contamination and inoculated for 7–10 days in an incubator at 28 ℃. As a way to mimic the growth environment of microorganisms on paper-based relics, routinely spray sterile water onto contaminated papers every 12 h to facilitate the growth and reproduction of the microorganisms.

Image collection

Microscopic images of the mold-contaminated papers, obtained from step 2.1.2, were collected for the four strains of microorganisms with three different magnifications, including 10 × and 40 × from a standard laboratory compound microscope, as well as one from a commercially available phone-attached microscope lens that was bought online (Fig. 2A). The images obtained with the commercial microscope lens were included to enhance the accessibility and portability of the model. Costing less than $40 on Amazon, the lens would allow people to capture microscopic features similar to the ones captured by the standard laboratory microscope. Additionally, microscopic images without any microorganisms under the field of view were added to the image collection to serve as blank controls, allowing the CV algorithm to only learn about the representative features of the mold instead of the paper cellulose background.

Modeling

The modeling process contains the training phase and the testing phase. The training phase involves training a CNN model to identify mold species. Images are first labeled with the corresponding classes, in this case, the strains, and then undergo data preprocessing to obtain the training data before being fed into the training process. The model gradually improves itself over training epochs until it achieves optimal results for the set of evaluation criteria. The trained models are further validated using testing data, a completely new set of images that the model had not seen before, to evaluate the robustness of the classification model (Fig. 3).

Fig. 3

Modeling Pipeline

The process was carried out in the cloud-based Jupyter Notebook environment provided by Google Colab, using NVIDIA A100 GPU. The deep neural network architecture was constructed using the TensorFlow API version 2.11.0.

Data preprocessing

The image dataset was divided into training and testing sets before further preprocessing. This prevents the risk of testing images inadvertently being included in the training set, often referred to as data leakage, which prevents the model from memorizing specific images and causing a pseudo-confident model. Further, given the limited number of images that can be generated from ancient books with highly restricted access, data augmentation is used to augment the number of images to reduce the risk of overfitting. Overfitting refers to the situation that, when the number of training samples is limited, the models will become overly familiar with the training data that they almost remember certain features but cannot generalize well to features that it was not exposed to before. Traditional data augmentation techniques for image data include rotating the images within a range of angles, translating or flipping the images in various directions, and cropping the images [35]. In this study, data augmentation was achieved through random cropping and rotation (Fig. 2C). The crop was generated by first initializing the x and y coordinates of a random starting point (i.e. the upper-left corner coordinate) and pinpointing the other three corner coordinates to create 1000pixels × 1000pixels images. The randomized cropping procedure was repeated on each image for 50 times. Then, the cropped sections were rotated in angles of n × $\frac{\pi }{ 8}$, where 0 ≤ n < 8, to further augment the number of images. In the end, images that do not contain sufficient representative features were manually discarded from the image dataset to avoid false positives in the learning process.

Modeling scheme

In this study, a transfer learning scheme was used for training the classification model. To further prevent overfitting, transfer learning offers a solution by leveraging the knowledge, represented as weights in the model, gained from off-the-shelf pre-trained models that were trained on well-rounded datasets, such as ImageNet, a large natural image database containing over 14 million images of 20,000 types of objects. Through transfer learning, the development pipeline is simplified by avoiding training a new model all from scratch [36]. The modified model is then trained on data from the target domain to further tune the model weights, through a process called fine-tuning, to adapt to more niche tasks [35, 37,38,39,40].

The train/freeze trade-off was considered when fine-tuning the transfer learning scheme used in this study. The initial weights of the pre-trained models enable the model to perform well on the pretraining dataset (i.e. ImageNet) so that, during the later fine-tuning on the target data, knowledge learned about general graphical features can be transferred, rather than learning from scratch. In practice, the transferred knowledge was retained at various levels by freezing different numbers of layers in the model. Freezing more layers in the pre-trained model can reduce the amount of computation needed. However, when the pretraining dataset is largely different from the target dataset, it would make the model underfitting on the target features. On the other hand, unfreezing more layers in the pre-trained model can increase the fitting performance on the target set. However, unfreezing too many would be equivalent to training the model from scratch, which demands longer training time and more computational resources.

In this study, three types of finetuning schemes were experimented with, considering the train/freeze trade-off, namely full finetuning, partial finetuning, and no fine-tuning (Fig. 2D). No finetuning refers to setting the feature extraction block that was transferred from the pre-trained models to untrainable, and only the classification layer is set to be trainable so that the architecture can adapt to the new classification task. Partial finetuning refers to partially unfreezing the feature extraction block to allow some layers to be trainable with the new set of data while the rest are not trainable. In particular, shallow layers of the feature extraction layer that capture higher-level information are set to be trainable to better learn the features that are unique to the new image set, while deeper layers that capture lower-level information such as edges, lines, and colors, are set to be non-trainable because those are characteristics shared by nearly all objects. Finally, the full finetune refers to setting the overall architecture, including the transferred feature extraction layers as well as the classification layer, to be trainable by the new image data.

Model architecture

Convolutional Neural Networks (CNN) is one of the mainstream algorithms powered by DL [41], widely explored in numerous CV application scenarios, such as medicine [42], robotics [43], etc. CNN models are achieved with supervised learning, where a large number of images are presented with labels to the model, then the model, which is composed of a series of layers to detect different features of the input images, will gradually learn the representative features corresponding to each label.

Visual Geometry Group (VGG) and Residual Network (ResNet) architectures, both widely applied CNN architectures [41, 44], were investigated in this study because they have shown consistent performance in microscopic image classification through training with large amounts of data or transfer learning [16, 39].

The feature extraction layers of the two pre-trained networks, VGG16 and ResNet50, are transferred for the new classification task, as explained in “Modeling Scheme” section. A Global Average Pooling (GAP) layer, proposed as an alternative for fully connected layers in classical CNNs [45], is added after the output of the feature extraction to take the average of each of the feature maps from the last layer of the feature extraction block and flatten it into a one-dimensional vector. GAP layers can efficiently reduce the spatial dimensions of three-dimensional feature maps by downsampling the entire feature map to a single value [46]. The one-dimensional output from the GAP layer is then passed into a fully connected layer with five nodes, each representing a class label of the classification task (Fig. 2D).

Modeling experiment

A dynamic adjustment of the number of training epochs and learning rate was experimented in the study. Early stopping is implemented with pre-defined stopping criteria during the training process to avoid overfitting. In the study, parameters including categorical cross-entropy loss, accuracy, recall, precision, and F1 scores of both the training and validation sets were used as stopping criteria with a wait time of 20 epochs. So, if the parameters being monitored remain unchanged (i.e. the validation accuracy does not further increase and the validation loss does not further decrease) after 10 epochs, then the training process will terminate.

On the other hand, learning rate specifies the rate at which the weights are being adjusted. A higher learning rate adjusts the weights more at a time, while a smaller learning rate leads to smaller adjustments. Early into training, a smaller learning rate may not allow the model to achieve its best potential due to convergence to a local minimum instead of a global minimum, while a larger learning rate would allow more rapid optimization. However, as the number of training epochs gets higher, a larger learning rate may negatively affect the optimization by causing non-convergence, while a smaller learning rate can help slowly optimize the model to achieve a global minimum in the loss. Thus, in the study, an exponential decay scheme was used to decrease the learning rate with the increasing number of training epochs.

The models were all trained using ADAM optimizer, and the architectures experimented on are summarized in Table 1, including the pre-trained model architecture, hidden layer architecture, and unfrozen (i.e. trainable) layers, as well as the pre-trained weights.

Table 1 Modeling Experiment Architectures

Web-based application

To allow the conservation community access to the classification tool, a web-based interface was constructed using Streamlit. Streamlit is an open-source platform and Python library that allows machine learning and AI development teams to quickly construct interactive web-based platforms. The models fine-tuned for the classification task were saved as TensorFlow H5 file formats and were reconstructed in the Streamlit environment. The web widgets, including the image upload function, model selection button widgets, and drop-down boxes are built-in functions of the Streamlit library. The interface allows users to upload their microscopic images of mold-contaminated artifacts and classify the strain of the mold presented. A knowledge base, stored as a Python dictionary, was also constructed to store relevant information about each strain, such as the health risks associated with the strain and recommendations for treatment. Future expansion and refinement of the model can be accelerated with the involvement of the research community. Therefore, a feedback mechanism was implemented to enable researchers and experts to contribute new images to the current image repository, correct labels if the model provides an erroneous classification, and introduce new strains.

Results and discussion

Model performance

The respective performances of the models are summarized in Table 2 corresponding to the models by index number. The parameters recorded to evaluate the model performance were loss, accuracy, precision, recall, and F1 score, for both the training and validation sets. Since early stopping was implemented during the training process, the evaluation parameters are all optimal values that triggered the early termination. More specifically, the maximum values of accuracy, precision, recall, and F1 score, as well as the minimum value of loss are recorded in the table. The epoch at which the training process terminated is recorded in the last column as a reference for convergence speed.

Table 2 Model performance summary

Overall, according to the training performance summarized in Table 2 as well as Fig. 4. Both types of CNN model architectures, VGG16 and ResNet50, demonstrate sufficient convergence between the training and validation sets, indicating the capability of the traditional convolutional neural network for this classification task.

Fig. 4

Model performance summary. A Training and validation accuracy and loss for the two major CNN architectures investigated: VGG16 and ResNet50. Training and validation showing convergence tendencies, demonstrating the potential for microscopic image classification. B Confusion matrix for the classification accuracy among five categories, including blank control. C Feature map of the CNN model

The performance of the developed model was tested using a testing set containing new images of the pre-identified microorganism strains, with the confusion matrix showcased in Fig. 4B. According to the confusion matrix, for the total 752 test cases, 679 cases were correctly predicted, achieving a testing accuracy of 90.29%. In detail, A. Niger achieved 88.1% accuracy, A. Oschraceus achieved 99.3% accuracy, Cladosporium achieved 85.6% accuracy, and P. Polymyxa achieved the highest prediction accuracy among all four strains of 99.5%. There are 8 false positive predictions of A. Oschraceus that should be A. Niger, and 1 false positive prediction of A. Niger that should be A. Oschraceus. These misclassifications could be due to the structural similarity between the two species since they both belong to the same Aspergillus genus. The misclassification cases of Cladosporium concentrate at P. Polymyxa, because the two strains share high commonalities in their microscopic features. The blank control has false positive classifications from all of the four strains particularly from A. Niger. This could be due to A. Niger having the smallest bodies among the four strains, which makes the background paper cellulose structures have more confounding power during the classification.

Further, feature maps of the trained classification models were investigated. Visualizing feature maps in a deep CNN can provide insights into the learned representations, including patterns, shapes, and textures, to help us understand how the model processes and interprets the input data. Figure 4C shows the feature map vs. the original image for A. Niger. The dark dots that scatter the original image are the microorganism bodies and the gray shadows in the background are cellulose fibers. The brighter plasma colors in the feature map are regions that contain the information that the model deems important for the classification task, in other words, the representative features that are learned by the model to identify the respective microorganism. When comparing the feature map with the original image, promising alignments can be seen between the microorganism bodies’ contours and the highlighted area on the feature map as well as between the cellulose fibers and the darker shadows in the feature map, indicating the model has successfully learned about the actual morphological features of the microorganism bodies with contrast to the background cellulose fibrous structures for the classification task.

Web-application & community engagement

The purpose of this segment of the work is to promote resource sharing and establish protocols for the collective advancement of deep learning applications in cultural heritage. As [37] mentioned, the success of deep learning models depends on the power of the dataset that the model was trained on, which includes the number of data and the quality of data. Specifically, a database that incorporates metadata of nucleotide sequences, microbial strains, and potential enzymatic properties as well as their destructive mechanism in the biodeterioration of different materials should be created and curated to assist more efficient identification, removable, and prevention of contaminating fungal strains [9]. At the same time, better resource sharing can be achieved by developing online platforms. For example, through the collective force of people who share an interest in fungi, a fine-grained classification dataset was developed—Danish Fungi 2020 (DF20) [47]. A mobile application based on CV, named FungiVision, was then established to classify different types of fungi and assist Mycology.

The web-based interface (https://biodegrade-diagnostics.streamlit.app/) developed in this study allows three streams of activities that form a closed loop, which we call an “iterative learning scheme” (Fig. 5A). First, when a new image of an unknown strain is uploaded to the application, users can choose from different classification models, which were trained on existing data, to perform the classification task and produce the predicted class label of the unknown strain (Fig. 5B, C). Then, the second stream of functionality is to retrieve the knowledge stored in the knowledge base about the classified strain (Fig. 5D). Currently, the knowledge base that the users can access incorporates information including morphology, cultural and molecular analyses, potential enzymatic properties, and their destructive mechanism to assist practitioners in developing the most suitable treatment techniques. Lastly, the third stream of function allows the user, after receiving classification results from the algorithm, to decide whether to accept the provided label and information. If the human user/expert raises a concern regarding the accuracy of the label provided, they can modify the label or add a new label if the species does not exist in the existing list of strains. The activity, including both the machine output and human input, will be logged in the back-end. Then, users can initiate a pulling request of the administrator to both update the image database and re-train the classification models.

Fig. 5

Mold classification iterative learning pipeline and web-based application. A Pipeline for iterative learning of the mold classification model with human feedbacks. B Web-based application: main page. C Web-based application: classification page. D Web-based application: knowledge retrieval and representation wiki page

Outlooks & improvements

This project trained a multi-class model that can distinguish between four strains of molds by feeding in images that contain only one strain per image. In real-life situations, such as archives and museums, or even freshly excavated cultural remains, there could be multiple strains of microorganisms contaminating the same surface area of the paper. To adhere to the principle of minimal intervention and apply strain-specific treatments, a multi-label classification model can be further developed, in case multiple different strains of microorganisms are captured in the same microscopic image.

The current project is just the prototype of AI integration in conservation practice. While there are many more strains of molds presented on cultural heritage, the refinement of the model, to let the models become more accurate and more generalizable, requires effort from the research community. The web-based application, implemented with the human feedback functionality, would serve as a platform to achieve such an endeavor as more and more specialists participate.

Conclusion

Our study showed the competency of CNN models in classifying microscopic images to achieve the end of identifying the microorganism strain presented. A suite of models that are competent in the classification task were generated as deliverables for the conservation community to use and contribute. At the same time, we proposed a sustainable application framework, accessible as a web-based application, to engage the conservation community to collaborate and contribute to the further improvement of the models.