Automated Real‐Time Detection and Prediction of Interlayer Imperfections in Additive Manufacturing Processes Using Artificial Intelligence

Although fused deposition modeling (FDM) additive manufacturing technologies have advanced in the past decade, interlayer imperfections such as delamination and warping are still dominant when printing complex parts. Herein, a self‐monitoring system based on real‐time camera images and deep learning algorithms is developed to classify the various extents of delamination in a printed part. In addition, a novel method incorporating strain measurements is established to measure and predict the onset of warping. Results show that the machine‐learning model is capable of detecting different levels of delamination conditions, and the strain measurements setup successfully reflects and determines the extent and tendency of warping before it actually occurs in the print job. This multifunctional system can be applied to assess other manufacturing processes to realize autocalibration and prediagnosis of imperfections without human interaction.

Advances in the field of additive manufacturing (AM) have made it possible for researchers to fabricate multimaterial and multifunctional designs with complex geometric features in the past decade. [1][2][3][4][5][6][7] Among the numerous 3D printing techniques, fused deposition modeling (FDM) is one of the most widely used ones for its easy operation and low cost. A typical FDM process slices a computer-aided design (CAD) model into thin layers of 2D patterns, which are then portrayed with extruded polymer rasters. Despite the advances in FDM technologies, they are plagued with interlayer imperfections that can cause delamination and warping, which necessitates a print job to be restarted and the material used will be wasted. Delamination occurs when the adhesion between two layers is weak and warping is caused by the residual thermal strain accumulated during the printing. [8,9] It turns out that these imperfections are largely dictated by the settings of printing parameters, first-layer calibration, and model geometry. [10][11][12] With recent advances in applying artificial intelligence and machine learning to materials science and engineering problems, [13][14][15][16][17][18] researchers have started using machine-learning algorithms to classify and predict different printing defections including blob, warp, and delamination based on the settings of printing parameters. [19][20][21][22][23] Another interesting approach in the field adopts a new slicing mechanism which splits prints into spatially locked bricks to reduce warping. [24] However, previous works in literature are not capable of real-time analyzing or predicting interlayer issues before they spread into the rest of the build part. In this work, we have developed a method based on computer vision and strain measurements to detect and predict interlayer imperfections such as delamination and warping in printed parts. Our proposed approach utilizes camera-based images with deep learning algorithms to classify and detect delamination conditions, and additionally, a novel setup is established based on strain gauge measurements to measure and predict the tendency of warping.
The first interlayer imperfection considered is delamination. As mentioned in the previous paragraph, delamination is mainly caused by an improper gap between the current nozzle height and the print, which results in a weak bonding between layers. Therefore, it has been shown in the literature [25,26] that the key to solve this issue is to adjust the nozzle offset value properly in the first place, where the adjustment of this variable is known as the first-layer calibration. Inspired by the fact that this calibration mostly relies on the users' bare eye observation of the extruded polymer shape at the nozzle tip, we have developed a setup that mimics the manual calibration through an application of deep learning. The monitoring setup has a universal serial bus (USB) Logitech camera attached at the left side of printing nozzle with an in-house designed and printed camera mount ( Figure 1a). The mount is reinforced on the back of its cantilevered component to minimize vibration during printing. In addition, the plastic shell of the camera is removed to fit itself in the limited printing space and provides a near horizontal filming angle. Since the offset nozzle height of the first layer stays in the range of 0.1-0.2 mm, a near horizontal view gives a more clear view to monitor the distance between the nozzle and the current raster. The offset nozzle height is classified into four categories: "Highþ," "High," "Good," and "Low." Since the nozzle cannot be adjusted to be lower than the print bed itself, no lower category is considered in the classification. Images representing the four categories are shown in Figure 1b, with a scale bar of 5 mm. It can be seen that the "High" nozzle height will result in poor adhesion conditions between the filament and print bed. Moreover, a "Highþ" nozzle height intensifies the delamination even worse as shown in the red box compared with the orange one. On the contrary, a "Low" nozzle height causes filaments to be extruded all around due to the restricted space beneath the nozzle and hence leading to a nonuniform surface condition shown in the green box.
After defining the condition of the four cases, image data are then collected during the printing process. Customized first-layer calibration Gcode file is made which sends commands to the printer to deposit ten single rasters directly on the print bed. Each raster is parallel and separated 5 mm from each other to provide the proper angle to balance image capturing quality and image collecting efficiency. During the image data acquisition, www.advancedsciencenews.com www.advintellsyst.com within each category, nozzle height is adjusted in three levels to ensure full coverage of the category. The specific values of each nozzle height are shown in Table 1. It is of note that there is a 0.04 mm gap between each category which may be caused by the print bed not being perfectly horizontal. For each level in one category, 900 images are captured continuously during the printing of the calibration model (the 10 rasters). Among all the image data, %5% of them are not useful since they are captured during the idle transition path between parallel rasters. As there may be some slight vibration from the camera mount and variation of the external environment, each image is augmented into four images with the following details. First, a fixed 224 Â 224 yellow "extraction box" is chosen as the reference of interest. Then, a red-dotted square box in the same size is extracted randomly from a uniform distribution (normal distribution was not chosen as we have no information about the expected position and the variance) within 10 pixels at the upper-right corner of the yellow box as shown in cycle 1 ( Figure 1c). Finally, the same extraction procedure is repeated at the remaining three directions for generality: upper-left, lower-left, and lower-right. To sum up, each category has around 900 Â 4 Â 3 ¼ 10 800 images altogether, and 30% of them are randomly picked out as validation data set, whereas the rest are grouped as training data. To ensure the applicability of our classification model, testing images are collected separately on a different day. For each category, one raster is printed with the nozzle height adjusting from level 1 to level 3 uniformly according to Table 1, and a yellow box representing the focused area is extracted; these images are treated as the testing data set. After preparing all the needed image data sets, the training data set is fed into a convolutional neural network (CNN) model for training, and the accuracy is evaluated on the validation data set by the updated model after each epoch. Here, a pretrained residual network (ResNet) model is applied with its final layer removed and two layers of fully-connected layer added to scale down the output to a vector with four elements. The training procedure terminates when the accuracy of the validation data set reaches convergence after 40 epochs. The loss and accuracy against the number of epochs are shown by solid curves in Figure 1d. The orange curve (solid line) indicates a 95.5% accuracy for our classification model based on the validation data set. Suppose that the 5% of meaningless data are randomly classified into the four categories with the same probability, the highest theoretical accuracy achievable is 96.25%, where the rest of the 3.75% accuracy comes from the three-fourths of the incorrectly categorized not useful images.
With the converged trained CNN model obtained, its performance is evaluated on the testing data set. The result of the accuracy for the testing data set is: "Good" ¼ 70.0%, "High" ¼ 96.8%, "Highþ" ¼ 97.7%, and "Low" ¼ 65.5%. To interpret the low accuracy on the "Good" and "Low" category, the output vector is transferred by a Softmax function that normalizes the sum of the vector elements to 1 and represents the probability of an image belongs to which category. The mean value of the normalized vector is calculated for each category, and the results are shown by a bar chart in Figure 1e. Comparing the two data sets, both "High" and "Highþ" categories reach a high accuracy, whereas "Good" and "Low" categories have a lower confidence of accuracy in the testing data set. It can be seen that under the "Good" category, images have a higher chance to be viewed as "High" as these two categories are adjacent. Similarly, in terms of "Low" category, images have a higher tendency to be regarded as "Good." One of the reasons attributing to the low accuracy seen in the "Low" and "Good" categories is that extruded materials that fall under these two categories are usually both flat and uniform, making it difficult to distinguish between the two. The only feature to differentiate between the two categories is from the raster width and a slight tilt at the margin of the raster. Whereas when the categories of "High" and "Highþ" are compared, the lack of feature difference in "Low" and "Good" can direct the CNN model to fit the noise of the training data. To resolve the overfitting issue, we hypothesize that the main noise causing the low accuracy in these two categories can be attributed to the nozzle condition difference. It can be seen that in the extracted images (Figure 1c), the nozzle occupies almost half of the space, and the surface conditions may change differently after each printing. Filaments, dust, and oil residues may attach to the nozzle. In addition, the 5% of meaningless data may also be another source of error in the actual testing process. As a result, we looked into methods to resolve these two potential sources that may increase the accuracy of our model.
To further improve our model and structure, the 5% of data which tend to be blurry images during idle nozzle traveling are divided out as a fifth category called "Null." In addition, to eliminate the nozzle condition variation effects, images are preprocessed by using edge detection to mask the nozzle area into a black color before training and testing (Figure 1f ). The training results based on the validation data set can be seen in the dashed curves in Figure 1d. It can be seen in the figure that the accuracy curve converges faster and better than our previous model, which increased the overall accuracy from 95.5% to 97.8%. A confusion matrix that represents the result of the testing data set is shown in Figure 1g to provide an intuitive interpretation of the distribution. The overall accuracy can be calculated as the average value on its diagonal which is 91.0%, where the accuracy is increased by 8.5% compared to the previous model. If the data are treated as "Valid" and "Null" in two categories, where "Valid" is the combination of "Highþ," "High," "Good," and "Low" category, the matrix can be transformed into Table 2.  shows that the model has an accuracy of 97.5% to distinguish correctly between valid and null images and that the refined model shows considerable improvement when it comes to dealing with null data and varying nozzle conditions in the image. In addition, this extra fifth category helps us remove null data on the fly during the actual implementation of the trained algorithm in real-time printing situations. Other extrusion-based methods and 3D-printing technologies that have layer-by-layer transitions such as PolyJet and selective laser sintering (SLS) will also have the existence of null data when taking camera-based images. Therefore, we believe that our monitoring method of adding an additional fifth category of null data will be applicable to other systems and be able to discard on the fly not useful data in real time.
Here, we would like to make a note about the different factors that may influence the results of our model and our plans for future work. First, in terms of how geometry or color of the part may affect the results of our system, we are using a fixed Gcode file and unchanged white filament that both constrain and standardize the calibration process. The advantage of this standard calibration process is that it ensures easy adaptation to other systems and also reduces the effects of different shapes or colors of the printed part that is fabricated. Second, the quality of the image data is believed to be another important parameter within the model as it is believed that improved image quality will potentially improve the model accuracy by providing more pixel information within the same extraction box. To improve the image quality, enhanced image processing algorithms such as super resolution (SR) can be applied to further improve the image quality. SR techniques have been shown in the literature to construct high-resolution images from several low-resolution ones by increasing high-frequency components and removing the degradations. [27] These methods will be explored in future work. Third, the type of filament material and type of printing process, where different materials will cause disparate features in the image and the image acquiring location will vary for different types of printing processes. It is believed that our monitoring algorithm will be applicable to other types of materials (such as acrylonitrile butadiene styrene (ABS)) and printing processes as long as the materials can be extruded or jetted in a layer-by-layer process and camera-based images can be obtained as a result. An interesting extension of this work is to apply our machine learning driven monitoring system to print a variety of different types of materials (from very soft to very stiff ) with different printing processes (from extrusion-based to binder jetting).
Another interlayer issue in AM includes warping. In this article, the method to predict warping is conducted by a setup established on the print bed based on strain gauges. First, print bed tape is peeled off from its plastic protective film and stuck to the print bed to prevent any contamination. Second, a smaller size of plastic film (larger than the printing sample) from the last step is fixed to the print bed tape along its perimeter with super glue as shown in the light gray layer at the right part of Figure 2a. Third, the first strain gauge is fixed on the plastic film with the rear half of it glued exactly on the ring of superglue in the last step. In this case, any small expansion of the plastic film caused by the printing sample deformation can be detected by the strain gauge. The cross-sectional configuration is shown in the left part of Figure 2a. Finally, the third step is repeated at the other side of the plastic film and is denoted as Strain gauge 2. The detailed physical dimension of the plastic film and printing object can be seen in Figure 2b. Since the elongation of the thin film is very small, a sensitive measuring and signal amplifying system is needed. Here, a combined application of the Wheatstone bridge and LM741 amplifier circuit is established and shown in Figure 2c. During the printing, the voltage signal is collected by an Arduino microcontroller board and corresponding strain curve against time is calculated and plotted in real time. The detail of the circuits and the governing equations are discussed in the Experimental Section. Nozzle height information is recorded at the same timestamp via Pronterface, [28] which is an open-source framework that is able to communicate with the 3D printer.
With all the measurement settings prepared, two rectangular blocks with different percentages of infill are printed and two sets of data are collected. Since warping is mainly caused by the accumulated residual strain as the more layers stack up, the data in the time domain are then mapped into the layer number according to the recorded nozzle height information. Two sets of results on both Strain gauge 1 and Strain gauge 2 are shown in Figure 2d,e with a plot of strain against layer number and an envelope curve fitting of the data. Local maxima and minima are picked at a certain interval, and polynomial curve fittings are implemented on those points. Here, we define the amplitude of strain (Δε) as the difference of the upper boundary (red curve) and the lower boundary (blue curve). From the plot, it can be hypothesized that the amplitude of strain is steady for the 50% infill printing object while expanding for the 100% infill one. To further probe the variation of the strain signal, the amplitude of strain against layer number curve is plotted for both cases in Figure 2f,g. The warping conditions on strain gauge 1 side are also shown in the figure at layer numbers 17 and 21. On the one hand, for the 50% infill printing object, there is no sign of warping at both layer heights. On the other hand, in terms of the 100% infill printing sample, slight warping is shown at layer 17 and severe warping occurs at layer 21. The findings are consistent with the amplitude of the strain curve, and in other words, the whole experimental setup is able to show the condition and tendency of warping according to the amplitude of strain. Since the accumulation of internal stress during printing is irreversible, once a slight warping condition happens, continuing printing will be a waste of time and material. By setting a threshold, which can be defined as the ratio of current and initial amplitude of strain, a prediction of warping can be realized. Take strain gauge 1 in 100% infill sample as an example, at layer 17, the amplitude of strain is about 0.85 Â 10 À3 , and the average of first five layers is taken as the initial value of the amplitude of strain which is about 0.68 Â 10 À3 . Therefore, the ratio between the aforementioned two values, which equals 1.25, can be regarded as the threshold of predicting warping. Different from the fluctuating strain pattern of the 100% infill sample, this ratio varies between 0.90 and 1.20 throughout the entire printing process of the 50% infill sample. The value of the threshold is indeed dependent on the shape and dimension of the printing sample which requires further study; therefore, future work includes predicting warping based on the ratio analyzed from the input of a given CAD model.
In summary, we have developed a system which is able to detect and predict delamination and warping issue in real time www.advancedsciencenews.com www.advintellsyst.com www.advancedsciencenews.com www.advintellsyst.com based on the trained machine-learning model and strain measurement. In terms of delamination, we focus our effort on the calibration of nozzle offset height where our observation model achieves an accuracy of 97.8% on validation data set and 91.0% on testing data set. For predicting warping, the strain gauge setup effectively quantifies the warping tendency of a print. By setting a threshold of the amplitude of strain, a prediction mechanism of warping can be realized. Future work involves an autocorrection first-layer calibration system based on the classification results, and other sensors such as infrared camera may be adapted for in situ assessment. The framework developed in this article can also be potentially applied to other materials and AM systems to improve the process with less human interaction.

Experimental Section
Experimental Equipment: The 3D-printer used in the experiments was PRUSA i3 MK3 FDM printer with polylactic acid (PLA) filament and the model of camera was Logitech C270. The model of strain gauge used in the measurement was BF120-3AA, and other circuit components information are shown in Table 3.
Governing Equations in Strain Gauge Measurement: The circuit is shown in Figure 2c and related equations were described later. The voltage difference across m and n in Wheatstone bridge, denoted by V mn , can be expressed by The right part of the circuit is an ideal difference amplifier, where R 5 ¼ R 8 and R 6 ¼ R 7 . Thus, the output voltage transferred to Arduino V out is given by Plug Equation (1) to Equation (2) and take the variation of R 2 , the following is obtained Given that the gauge factor (GF) is approximately a constant equal to 2 and has the following relationship where ΔR 2 is the resistance change of the strain gauge and ε is the strain. We can finally express the strain in terms of the output voltage difference and other constant variables as follows ε ¼ 1 GF where ΔV out is the difference in the current output voltage and initial steady-state value.