Scratch detection of archival films: modeling and trajectory calculation

Abstract

Scratches are a common continuous structural type of damage in archival films. Scratch pixel level segmentation algorithms still use general image segmentation, and with poor the detection. Based on the characteristics of scratches, a scratch model based on a Gaussian probability density function is proposed, a local adaptive one-dimensional Gaussian weighted segmentation algorithm is designed, and an invalid region filtering mechanism is described. Based on experimental results, the local one-dimensional Gaussian weighted segmentation algorithm proposed in this paper can preserve the integrity and continuity of scratches, and effectively suppresses edge defects; the invalid region filtering mechanism can effectively reduce the false detection of scratches. Adding a scratch trajectory can effectively reduce the chances of missing the detection of scratches, and the stability of the traditional algorithm using Kalman filters to predict the scratch trajectory is poor. According to the similarity of multiple scratch trajectories, a calculation method of trajectories based on U-net is described. Experimental results show that this method can calculate the scratch trajectory stably and effectively.

Introduction

Before the advent of electronic storage devices, films played a pivotal role as information carriers, preserving precious historical information. Generally, during the playback and copying of films, sharp protrusions on the surfaces of the equipment rub against the films, causing the superficial gel layer to peel off and leading to scratch damage [1]. In the standard definition television format (720 × 576), scratches appear as distinctive 3 to 10-pixel black vertical stripes of variable lengths, with a tilt angle of less than 5^°. During playback, scratches are usually darker than the surrounding visual information, obscuring part of the picture and persisting for several frames. Consequently, scratches are a prominent and undesirable part of filmstrip movies.

Traditional scratch-detection algorithms are divided into two categories based on the use of context information: intra-frame detection and detection based on the characteristics of scratches in consecutive frames. Representative scratch-detection algorithms in the first class are the Kokaram algorithm based on Bayesian theory [2] and the Buni algorithm based on Weber's law [3]. So far, both these algorithms have achieved good results. A scratch-detection algorithm based on wavelet decomposition [4] has been proposed by Chen Zhao at the Shanghai Jiaotong University. This algorithm decomposes wavelets to extract high-frequency information from scratch images, yielding good results. Additional scratch-detection techniques include median filters based on edge protection [5], genetic algorithms [6], nonlinear operators [7], and parallel algorithms [8]. The second category includes multi-frame detection algorithms such as the Joyeux algorithm [9], which employs the Kalman filter to track and assess time-domain information based on single-frame detection, considering the continuity of scratches. This approach is more complex as it requires a Kalman filter and motion estimation [10] but outperforms single-frame detection algorithms.

With the emergence of convolution neural networks (CNNs), some scholars have employed CNNs to detect and repair scratches. Unlike conventional methods that are constrained to specific gray value information, deep learning can control deep features at different scales more flexibly, addressing problems that cannot be solved by conventional methods. However, CNNs may encounter challenges in handling complex degradation, limiting the learning of features, such as the blurring of scratch edges. For example, a U-net was previously used to detect scratch damage in archival films [11, 12], but only parts with simple background information in the main scratches could be successfully detected. Iizuka et al. introduced DeepRemaster [13], which uses time convolution to restore archival movie damage and repair video artifacts but has a poor perception of scratch damage. Wan achieved visually appealing reconstruction results by learning features through a recurrent transformer network [14]. Inspired by the above studies, in this study, we combine the advantages of CNNs with projected detection of scratches.

Various single-frame scratch detection techniques are now the key research findings in scratch detection, whereas general image segmentation techniques are used in pixel-level scratch segmentation. The unique way that scratches perform is reflected in the intricacy of historical film deterioration. As seen in Fig. 1, the boundaries of scratches will be unevenly jagged, their gray values will vary, and the influence of content will cause the boundaries of scratches to break. Because of this, single-frame detection is challenging to apply, and the general picture segmentation algorithm's results are not optimal. Joyeux’s technique addresses the issue of missing detection in single-frame detection by using Kalman to trace the course of scratches in the temporal domain. Archival film finishes its deterioration process on film. The most practical way to create line scratches is to rub against spinning mechanical parts that exhibit complicated time-domain trajectories. The accuracy of the scratch trajectories cannot be ensured via Kalman prediction tracking.

Fig. 1

Scratches in archival films

For the aforementioned reasons, the calculation of the scratch trajectory and the pixel-level segmentation of scratches are the main topics of this article. In the task of scratch detection, the trajectory of the scratch is calculated using a convolution neural network. The Gaussian probability density function's mathematical model is created. This model informs the construction of a local adaptive one-dimensional Gaussian weighted detection algorithm as well as the introduction of the invalid region filtering mechanism. Compare the steps of the standard approach with the steps of this paper's algorithm. As shown in Fig. 2.

Fig. 2

Comparison before and after optimization of scratch detection process, With barchival indicating the optimization part

The related work

Scratch trajectory

In principle, scratches have continuity in successive frames according to the physical causes of scratches, but real observation reveals that this continuity is broken. The curved trajectory of the scratch shows in a continuous sequence of numerous frames, as seen in Fig. 3, instead than in a straight line. According to the investigation, line scratches are typically caused by rubbing against moving mechanical parts, which causes offset and hinders the ability of the scratch identification procedure to effectively employ context information. The scratch trajectory in the time domain is likewise compatible with the proposed scratch offset trajectory. The Joyeux technique uses the Kalman filter to track and evaluate the time domain information based on single frame detection. A schematic illustration of the Joyeux algorithm's tracking trajectory may be found in Fig. 4. The calculation of the scratch offset reveals that it is challenging to determine the scratch offset using continuous frame image content. Tracking every scrape by Kalman involves a lot of computation. As can be seen on the right side of Fig. 4, there will be errors in the Joyeux algorithm employing Kalman tracking judgment because to the trajectory's complexity.

Fig. 3

200*500 matrix composed of 200 consecutive frames of scratch horizontal coordinates and 200 consecutive frames of scratch horizontal coordinates with multiple scratch sequences, where white areas represent scratches and black areas represent background

Fig. 4

Line scratch tracking over 30 frames (left). Wrong detection behavior (right). Units along the x-axis are pixels, along the time axis projected stripes

Upon examining Fig. 3, it is evident that each frame contains many scratches with comparable trajectories. The trajectories of these multiple conspicuous scratches can be used to compute the scratch offset. When calculating scratch offset, scratches have two advantages over the film's content: first, they are physical damage with a straightforward structure that has nothing to do with the content; second, the scratch offset determined using the film's content cannot be verified because there are typically multiple scratches and multiple position information ensures the accuracy of the scratch offset calculation. Because content affects the classic single frame projection approach, which makes detection unsteady, and because context information about the scratch is lacking, it is difficult to calculate scratch offset from numerous obvious scratch tracks. In continuous multi-frame pictures, the semantic segmentation network with strong robustness can reliably recognize certain visible information in some scratches. Single-frame projection detection helps lessen the issue of missed detection through scratch trajectory if the trajectory can be computed.

Scratch model

The original mathematical model of scratch is the trigonometric function model proposed by Kokaram:

$$ l(i) = {\text{A}} \cdot \alpha^{{\left| {C - \left. i \right|} \right.}} \cos \left( {\frac{c - i}{w}} \right) + {\text{f}}_{0} $$

(1)

${\text{A}}$ represents the brightness of the scratch,$\alpha$ represents the brightness attenuation coefficient,${\text{c}}$ is the center position of the scratch,${\text{W}}$ is the width of the scratch, and ${\text{f}}_{0}$ is the offset determined by the local average gray value. Vectors $i$ represent the coordinates of pixels.

The horizontal section and projection curve of the scratch are determined to be similar to the Gaussian probability density function during the scratch projection detection method. This study proposes a scratch model whose expression is based on the Gaussian probability density function:

$$ \begin{array}{*{20}c} {l\left( {x,y} \right) = {\text{A}}\left( {1 - \frac{1}{{\sqrt {2\pi } \sigma }}e^{{ - \frac{{\left( {x - \mu } \right)^{2} }}{{2\sigma^{2} }}}} } \right) + N\left( {x,y} \right)} & {,x \in \left( {0,{\text{X}}} \right),y \in \left( {0,{\text{Y}}} \right)} \\ \end{array} $$

(2)

Among them, ${\text{A}}$ represents the brightness of the scratch,$\mu$ is the center position of the scratch, and $\sigma$ represents the scale factor. $x$ represents horizontal direction coordinates,$y$ vertical direction coordinates, ${\text{X}}$ horizontal direction length of the image, ${\text{Y}}$ vertical direction length of the image, and $N\left( {x,y} \right)$ interference items, as shown in Fig. 5.

Fig. 5

Scratch model; a Represents a three-dimensional image of a scratch model based on a Gaussian probability density; b Represents a true scratch three-dimensional image

Invalid area filtering mechanism

Film-to-magnetic technology must first preserve the film's information as a digital copy in order to preserve the original archive film documentation. In actuality, the digital copy is detected throughout the detection procedure. Additionally, analog to digital signals are transformed from the scratches on the archive film. A perfect replica of the scratch information is not guaranteed by this method. Furthermore, the scratch’s limited width makes it easy for surrounding information to influence the conversion process. as depicted in Fig. 6. There is no doubt that the background information has an impact on the scratches at the planes of strips 2 and 3 in Fig. 6.

Fig. 6

Features of different backgrounds of scratches, and the gray value line diagram of horizontal tangent plane with serial numbers

The proposed method

Scratch trajectory calculation based on U-net

U-net is a popular network with a straightforward design and great efficiency. The encoding and decoding networks that make up U-net, along with a layer-hopping connection, allow for the end-to-end production of semantic segmentation findings at the pixel level. U-net possesses the subsequent two attributes: Through layer hopping connections, U-net combines the network's shallow layer and deep layer information equally, resulting in an output that includes the distinctive characteristics of various image depths.

To get the results of semantic segmentation, it is sent to the trained network model after preprocessing. The outputs of the semantic segmentation process are projected vertically, and a projection mask comprising the horizontal spatial position information of the scratches is obtained by setting the filter's threshold value. The process is shown in Fig. 7.

Fig. 7

U-net scratch detection process

A schematic diagram of the search for a match in Step2 is shown in Fig. 8.

Fig. 8

Step 2 Search the matching process diagram. T represents the image sequence, X represents the offset pixel, and the black box represents the scratch position

Local adaptive detection

Weight function

The idea behind projection detection is that the scratch information needs to be in the center position of the projection mask, and the scratch position should be determined by extending to both sides of the local extreme value of the projection curve. The weight function of the Gaussian probability density function is intended to scale the projection area, based on the scratch model of the Gaussian probability density function.

Weight function model:

$$ \begin{array}{*{20}c} {W\left( {x,y} \right) = 1 - \frac{1}{{\sqrt {2\pi } \sigma }}e^{{ - \frac{{\left( {x - \mu } \right)^{2} }}{{2\sigma^{2} }}}} } & {,x,y \in \left( {0,{\text{X}}_{{\text{n}}} } \right)} \\ \end{array} $$

(3)

where x represents horizontal coordinates and y represents vertical coordinates. Make $\mu { = }{{{\text{X}}_{{\text{n}}} } \mathord{\left/ {\vphantom {{{\text{X}}_{{\text{n}}} } 2}} \right. \kern-0pt} 2}$, conform to scratches in the center of the projection mask. According to the gray value of the observed scratch is generally more than 20 lower than the nearby gray value, scaling the scale of the middle and both sides of the scratch area within 10% can highlight key information without affecting the real content, so that $\sigma { = }4$. Substituting $\mu { = }{{{\text{X}}_{{\text{n}}} } \mathord{\left/ {\vphantom {{{\text{X}}_{{\text{n}}} } 2}} \right. \kern-0pt} 2}$ and $\sigma { = }4$ into Eq. 3 yields Eq. 4:

$$ \begin{array}{*{20}c} {W\left( {x,y} \right) = 1 - \frac{1}{{\sqrt {2\pi } 4}}e^{{ - \frac{{\left( {x - \frac{{{\text{X}}_{{\text{n}}} }}{2}} \right)^{2} }}{32}}} } & {,x,y \in \left( {0,{\text{X}}_{{\text{n}}} } \right)} \\ \end{array} $$

(4)

The weight function is shown in Fig. 9a and the weight function conforms to the Gaussian probability density function in the horizontal direction as shown in Fig. 9b.

Fig. 9

Schematic diagram of weight function; a Weight function; b Tangent plane of weight function

Adaptive detection

The convolution neural network's detection results are projected vertically, and the test results are used to determine the threshold value. Finding the scratch region can help with later processing by removing the majority of image information that interferes with scratch detection. The little area should be set up as a square with the side length equal to the width of the scratch. Continue to divide the scratch area into blocks. The bar-shaped scratch is broken down into many square detection areas, each of which has the potential to have a scratch. An adaptive threshold is used by each square block to identify scratches in the area. as shown in Fig. 10, ${\text{X}}_{i}$ is the scratch width.

Fig. 10

Scratch block diagram

The square block expression is $\begin{array}{*{20}c} {f\left( {x,y} \right)} & {,x,y \in \left( {0,{\text{X}}_{{\text{n}}} } \right)} \\ \end{array}$, and the mask of scratch is as Eq. 5:

$$ M\left( {x,y} \right){\text{ = }}\left\{ {\begin{array}{*{20}c} {255} & {f\left( {x,y} \right) \times W\left( {x,y} \right) < {\text{T + C}}} \\ 0 & {f\left( {x,y} \right) \times W\left( {x,y} \right) \ge {\text{T + C}}} \\ \end{array} } \right.,\,\,\,x,y \in \left[ {0,{\text{X}}_{{\text{n}}} } \right] $$

(5)

Threshold ${\text{T = }}\frac{{\sum\limits_{y = 0}^{{{\text{X}}_{{\text{n}}} }} {\sum\limits_{x = 0}^{{{\text{X}}_{{\text{n}}} }} {f\left( {x,y} \right) \times W\left( {x,y} \right)} } }}{{{\text{X}}_{{\text{n}}}^{2} }}$ in Eq. 5. The threshold ${\text{C}}$ is to set an offset value adjustment amount for the threshold value, and select the appropriate value according to the prior knowledge of scratches. Compared with the threshold value less than the threshold value is regarded as a scratch pixel, and the scratch mask image is obtained by traversing the square block.

Scratching performance will be impacted by backdrop information that is either too bright or too dark. The visual performance is not evident when the difference between the background information and the scratch is smaller than 10, which is not consistent with the features of the scratch. There are two categories for these situations: the first is when the visual performance is too bright and the gray value is too great. The first is that the gray value is too small (the visual representation is too dark); if the black scratch is identified, the result will not include too bright information, thus it can be ignored. The segmentation method mentioned above will interpret the background data as scratch data in this scenario. Therefore, according to the projection mask tangent plane area, the original pixel value is small (for example, less than 35), and the mask tangent plane is regarded as an invalid area and filtered.

Simulation experiment and result

Stability analysis of scratch detection and calculation of scratch trajectory

In order to create a scratch compensation algorithm in this paper, two prerequisites must be met: first, as Fig. 3 illustrates, multiple groups of scratch positions were observed, and it was found that the distance between scratches is significantly greater than the value of the scratch offset. Second, U-net has the stability and efficacy to segment some large scratches. The purpose of the experiments is to demonstrate the stability of U-net in scratch detection.

Real scratches are currently detected using a number of primary scratch detection techniques. As seen in Fig. 11, a comparison of numerous traditional projection detection techniques is first conducted. The motion-compensated frame difference projection in Fig. 11 provides the best projection detection result because it can retain scratches and filter the majority of the frame information. The wavelet-based scratch detection technique can increase the amplitude of the horizontal change in scratches, but it is unable to completely remove the interference caused by the image content information. Figure 12 illustrates how readily the image content can alter the first-order vertical wavelet’s projection information. The difference projection between the front and back frames in the single frame detection algorithm mentioned above performs better after motion compensation. This is mostly because sequence time domain information is correlated, filtering out most information and minimizing content-induced interference.

Fig. 11

Comparison of horizontal projection detection algorithms; a Horizontal projection of the original image; b Horizontal projection of histogram equalization; c Horizontal projection of frame difference after motion compensation

Fig. 12

Scratch detection based on wavelet transform; a Original drawings; b A first level vertical detail component; c Projection of the horizontal direction of the original image; d Horizontal projection of first-order vertical wavelet

Direct scratch repair is DeepRemaster's approach of handling scratch damage. Scratches are automatically identified and fixed based on the time domain information of the sequence using a time convolution network. Figure 13 displays the test findings. Many of the scratches in Fig. 13 have not been found and fixed, and the stability of the detection findings is not good. Analysis is done on the causes. The intricate degradation of archival films necessitates a strong nonlinear expression ability from the model. The long scratch length also necessitates a big receptive field and a deep depth curl neural network, which will significantly raise the time convolution network's computation cost. For jobs involving the identification and repair of scratches, the time convolution network is therefore unsuitable. The task of scratch detection is still one that cannot be disregarded.

Fig. 13

Time convolution network detection and repair results, row 1 represents the original sequence, and row 2 represents the sequence after detection and repair

Even though there are four noticeable major scratches on the left side, the projection results are quite different, as shown in Fig. 14. The motion-compensated reference frame difference mask image can filter static information through the reference frame. However, the projection results of continuous multiple frames are unstable. Additionally, the results demonstrate that while projection detection is a useful technique for detecting scratches, its outcomes are not consistent, necessitating the addition of an auxiliary method for calculating scratch trajectories. U-net is commonly used as a semantic segmentation network for scratch detection, and as Fig. 15 illustrates, its stability for segmenting the primary scratch sections is greater than that of reference frame difference projection following motion compensation. The stability of U-net detection of major scratches can fulfill the scratch compensation calculation when combined with the preceding experimental results.

Fig. 14

Continuous 3-frame motion compensated reference frame difference projection

Fig. 15

Continuous multi-frame scratch detection, wherein a first row represents a sequence to be detected, a second row represents a U-net semantic segmentation result, a third row represents a scratch detection result based on the U-net semantic segmentation result, and a fourth row represents a scratch detection result based on motion compensated frame difference projection

Figure 15 further illustrates how U-net detection's applicability range is limited. Small groove scratches and secondary scratches on the right side cannot be used, but the detecting effect is reliable and effective for scratches with very obvious vision. Scratch tracks cannot be calculated using this method if there are no visible scratches in continuous frames. This does not imply that the approach is worthless; on the contrary, U-net is more resilient and has the ability to compute the trajectory of scratches steadily. Additionally, noticeable scratches are more likely to result in visual perception, and lowering the missed rate of scratches is more important.

U-net is used to detect a continuous 15-frame test sequence in order to produce a scratch detection result; the horizontal coordinate information of the scratch is displayed in Fig. 16a. Information about the scratch center is computed, as Fig. 16b illustrates. The complete offset data is displayed in Fig. 16c, and the computed scratch trajectory is displayed in Fig. 16d, based on the search strategy and matching criteria. The effectiveness of the scratch trajectory calculation approach suggested in this paper is demonstrated by the findings in Fig. 16. The scratch trajectory in this work is computed using several real scratch location data, and the findings have a greater degree of dependability when compared to Kalman's predicted trajectory.

Fig. 16

Calculation process of scratch track. a Scratch detection information; b Scratch center position; c Calculate scratch frame offset; d Draw scratch trajectory

Analysis of segmentation detection results

This research presents a comparison between the standard method and the proposed local adaptive detection. Direct use of the projection level will make later repairs more challenging. Typically, the threshold segments each scratch to produce a pixel-level mask based on the projection level mask. The test sequence is displayed in line 1 of Fig. 17, and the projection level mask of this sequence is displayed in line 2. The Otsu and triangle methods, which may automatically calculate thresholds according on global information, are used in classical global segmentation. For double peaks, the Otsu approach works well, and for single peaks, the triangle method works well. Lines 3 and 4 of Fig. 17 illustrate the comparison of the global segmentation outcomes. The Otsu method's global segmentation in Fig. 17 is marginally higher than the projection level. The triangle approach is poor and cannot effectively meet the requirements of scratch detection, while the gray value histogram of the image produced by the projection level mask corresponds to double peaks. Two-dimensional Gaussian weighting and the local mean are examples of traditional local adaptive segmentation techniques. This work employs a one-dimensional Gaussian function as the weighting function to emphasize the center information and guarantee the continuity of scratches from top to bottom in accordance with the Gaussian probability function model of scratches. In Fig. 13, the local block size is divided into blocks and set based on the scratch width in order to more precisely compare the efficacy of the three ways. The comparative impact of the three approaches in 5, 6, and 7 is displayed in Fig. 17.

Fig. 17

Comparison of segmentation algorithms; row 1 represents a test image sequence with scratches, row 2 represents a projection level mask, row 3 represents a Otsu pixel level mask, row 3 represents a triangle pixel level mask, row 4 represents a local mean pixel level mask, row 5 represents a local two-dimensional Gaussian weighted pixel level mask, and row 6 represents a local one-dimensional Gaussian weighted pixel level mask

The comparison and analysis primarily focus on three aspects: (1) constraining the projection area's edge information; (2) examining the continuity and integrity of scratches; and (3) evaluating the efficacy of the invalid area filtering technique. As seen in Figs. 18, 19, some scratches have been magnified due to their thin width in order to more easily compare the results in Fig. 17. In contrast to Fig. 18b, c, d efficiently reduces edge information while retaining a greater amount of information regarding scratches. The scratches in Fig. 19a are thinner than those in Fig. 18a, which helps to more clearly illustrate the algorithm's continuous and comprehensive segmentation effect on scratches. The continuity and completeness of the scratch segmentation result in Fig. 19d is superior than Figs. 19b, c, and the scratch region in Fig. 19d maintains more information on fine scratches than Fig. 19b, c. One-dimensional Gaussian weighted segmentation emphasizes the significance of the center information and can segment scratches more thoroughly and suppress the edge information of the projection region when compared to mean and two-dimensional Gaussian segmentation. In addition, an invalid region filtering technique is added to the local segmentation algorithm introduced in this research to filter the regions that are scratched due to background information. In contrast to Fig. 19b, c, the background information in Fig. 19d's middle scratch region is too dark. If the invalid area filtering technique is not used, this will result in the introduction of a sizable false detection area. The comparison's findings demonstrate that reducing false detection can be accomplished successfully by including an invalid region filtering algorithm.

Fig. 18

Comparison of three local cut algorithms 1. a Image 1; b Local mean; c Local two-dimensional Gaussian weighting; d Local one-dimensional Gaussian weighting

Fig. 19

Comparison of three local cut algorithms 2. a Image 1; b Local mean; c Local two-dimensional Gaussian weighting; d Local one-dimensional Gaussian weighting

The study’s proposed local one-dimensional Gaussian weighted scratch segmentation algorithm exhibits good detection performance based on the aforementioned experimental results. It can effectively preserve the integrity and continuity of the scratch region or suppress edge information. Additionally, the invalid region filtering mechanism effectively reduces false detections caused by background information. As a result, this paper's proposed local one-dimensional scratch detection algorithm has some real-world applicability.

Conclusion

Currently, single-frame detection and Kalman tracking prediction are used to detect scratches on a trajectory. The study's scratch segmentation algorithm is based on widely used techniques for segmenting images. In examining the features of scratch damage in archival films, this work finds that many primary scratch paths share common properties. Multiple scratches are used in conjunction with U-net semantic segmentation stability to determine scratch trajectories in order to overcome the instability of Kalman tracking prediction results. Furthermore, the Gaussian probability density function is utilized to create a scratch model, and a local adaptive one-dimensional Gaussian weighted segmentation approach is suggested. Additionally, a system for filtering invalid regions is integrated. The results of the experiments show that U-net semantic segmentation is stable enough to compute scratch trajectories with numerous scratches. In terms of maintaining the continuity and integrity of scratches, suppressing edge information, and removing invalid areas, the suggested scratch segmentation algorithm performs better than conventional techniques. In the field, it is useful in practice.