Overview

Recently, the research group of Prof. Bing Pan (Beihang University), chief scientist of Haytham Technology, published the paper “User-Independent, Accurate and Pixel-Wise DIC Measurements with a Task-Optimized Neural Network” in Experimental Mechanics. The paper introduces a new task-optimized deep learning digital image correlation method (Digital Image Correlation, DIC) —RAFT-DIC, which achieves fully automatic, pixel-wise, high-precision displacement field measurement by targeted improvements to the deep learning DIC framework and a new dataset generation method for model training. Compared with existing deep learning DIC methods, the sub-pixel displacement measurement accuracy of RAFT-DIC is about an order of magnitude higher, with stronger cross-dataset generalization and practicality.

Article: Pan B, Liu Y. User-Independent, Accurate and Pixel-Wise DIC Measurements with a Task-Optimized Neural Network[J]. Experimental Mechanics, 2024: 1-15.

Link: https://link.springer.com/article/10.1007/s11340-024-01088-4

Research Background

With advantages such as non-contact full-field measurement, ease of use and wide applicability, the digital image correlation method has become the mainstream displacement and deformation measurement technology in experimental mechanics and is widely used in different scientific and engineering fields. Although subset-based local DIC algorithms (conventional DIC) have made major progress in measurement accuracy, computational efficiency and robustness, and are widely used in commercial software, they still have the following two shortcomings:

1) Conventional DIC computation requires users to explicitly input key calculation parameters (subset size, shape function order, etc.) for the subsequent correlation calculation, and cannot achieve user-independent fully automatic measurement;

2) Conventional DIC computation is performed on uniformly distributed calculation points specified on the reference image (the distance between adjacent points is the user-specified step size), so the output is discrete results and pixel-wise dense displacement field measurement cannot be achieved.

To solve these inherent limitations of conventional DIC, deep learning is a possible solution. Since deep learning was first introduced to DIC in 2021, various deep-learning-based DIC methods have demonstrated the potential and advantages of neural networks in image deformation measurement. Compared with conventional DIC, deep learning DIC can output dense displacement fields in an end-to-end manner after network training, without users manually selecting calculation parameters. Although many scholars have tried to introduce deep learning into DIC, the sub-pixel displacement measurement accuracy of existing deep learning DIC has no obvious advantage over conventional DIC, especially for low-frequency displacement fields (translation, rotation, stretching and shear or their combinations). In addition, the generalization ability of existing deep learning DIC is insufficient, so its performance on real experimental images is worse than on test datasets. 

Technical Approach

For the network framework, this paper draws on progress in deep learning optical flow estimation and fully utilizes the prior information of DIC measurement (computing pixel-wise similarity and iteratively updating the displacement field), fundamentally optimizing the network framework. In optical flow estimation, the Recurrent All-Pairs Field Transforms (RAFT) framework achieved the highest optical flow estimation accuracy at the time it was proposed (error of about 3 pixels) and became the third-generation standard framework for deep learning optical flow estimation. Although RAFT shows excellent performance in optical flow estimation, its sub-pixel accuracy falls far short of DIC measurement needs, so the RAFT framework is not directly suitable for DIC measurement.

To this end, this paper made targeted modifications to the original RAFT framework to improve its measurement performance. The modified displacement estimation framework is called RAFT-DIC (main modules shown in Figure 1); its overall framework consists of three modules: 1) the encoder module, which extracts feature information from the reference and deformed speckle images; 2) the 4D correlation layer, which computes the similarity between the reference and deformed images by constructing a 4D correlation layer over all pairs of feature vectors; 3) the iterative update module, which updates the displacement field estimated by RAFT-DIC using a CNN-based iterative update module.

Figure 1. Main modules of the RAFT-DIC framework: 1) the feature encoder and context encoder outputting full-resolution features; 2) the multi-scale correlation pyramid built from 4D correlation layers pooled at two scales (1 and 1/2); 3) the update operator for updating and refining the displacement field.

RAFT-DIC made two targeted modifications to the original RAFT framework: first, all downsampling operations were removed from the encoding module so that the module perceives more spatial information, greatly improving displacement estimation accuracy; second, the number of pyramid layers of the 4D correlation layer was reduced to two, so the network can focus more efficiently on small-displacement accuracy.

For training datasets, this paper also developed random speckle image and displacement field dataset generation technology. Fully considering factors of real measurement scenarios (illumination changes, defocus, camera magnification, etc.), a widely distributed speckle image dataset was established (Figure 2).

Figure 2. Example images of the synthetic speckle dataset

In addition, this paper proposes a model-free displacement field dataset generation method that converts the grayscale distribution of random natural images into displacement field distributions through a series of operations — grayscale normalization, random blurring and random scaling (Figure 3). Compared with existing displacement field generation techniques using specific mathematical models, the diversity and complexity of the displacement field dataset are improved. Based on the established speckle image and displacement field dataset generation technology, a widely distributed and robust dataset was constructed, greatly improving the generalization performance of RAFT-DIC. 

Figure 3. (a) The displacement field generation process based on random natural images; (b) random natural image samples and the corresponding generated displacement fields.

Experimental Results

To verify the sub-pixel displacement measurement accuracy of RAFT-DIC relative to existing conventional DIC based on first-order and second-order shape functions (using the state-of-the-art IC-GN algorithm) and different deep learning DIC methods (StrainNet-f, DisplacementNet and original RAFT), tests were first performed on the numerical simulation images in Figure 4, and the results of different methods are shown in Figure 5. The results show that the accuracy of the proposed RAFT-DIC is about an order of magnitude higher than the other methods.

Figure 4. (a) Reference speckle image and (b) the applied sinusoidal displacement field



Figure 5. Comparison of the u displacement (left) and error maps (right) measured by first-order DIC (subset size 21×21 pixels, first-order shape function), second-order DIC (subset size: 21*21 pixels), StrainNet-f, DisplacementNet, original RAFT and RAFT-DIC.

To further verify the performance of RAFT-DIC on real experimental images, further tests were performed on the three-point bending test of a rubber specimen and the uniaxial tension experiment in the DIC challenge. The results further demonstrate that RAFT-DIC has strong cross-dataset generalization and practicality.

Figure 6. (a) Reference image and (b) deformed image in the three-point bending experiment



Figure 7. u and v displacement fields computed by standard DIC, RAFT-DIC, StrainNet-f, DisplacementNet and original RAFT





Figure 8. (a) Reference and deformed images of the uniaxial tensile test; (b) u and v displacement fields of conventional DIC (subset size: 21 × 21 pixels, step size: 3 pixels), RAFT-DIC, StrainNet-f, DisplacementNet and original RAFT

Summary and Outlook

This paper proposes a deep learning DIC method using a task-optimized neural network framework —RAFT-DIC, which can achieve precise, pixel-wise dense displacement field measurement in an end-to-end manner without user input. Simulation and real experimental results show that RAFT-DIC outperforms existing deep learning DIC methods in measurement accuracy and cross-dataset generalization. Compared with conventional DIC, RAFT-DIC can not only output pixel-wise dense displacement without manual selection of key calculation parameters, but also has significantly higher accuracy for displacement measurement with complex deformation images, and achieves accuracy comparable to conventional DIC for simple deformation images. In the future, researchers will further improve the practicality of RAFT-DIC — for example, combining it with other neural networks to expand its displacement measurement range, and achieving fully automatic, precise and dense 3D-DIC measurement based on RAFT-DIC. This advanced algorithm will be applied in the new version of Haytham Technology's software.