A Deep Learning Quality Control Loop of the Extrusion-based Bioprinting Process
Extrusion-based bioprinting (EBB) represents one of the most used deposition technologies in the field of bioprinting, thanks to key advantages such as the easy-to-use hardware and the wide variety of materials that can be successfully printed. In recent years, research efforts have been focused on implementing a quality control loop for EBB, which can reduce the trial-and-error process necessary to optimize the printing parameters for a specific ink, standardize the results of a print across multiple laboratories, and so accelerate the translation of extrusion bioprinted products to more impactful clinical applications. Due to its capacity to acquire relevant features from a training dataset and generalize to unseen data, machine learning (ML) is currently being studied in literature as a relevant enabling technology for quality control in EBB. In this context, we propose a robust, deep learning-based control loop to automatically optimize the printing parameters and monitor the printing process online. We collected a comprehensive dataset of EBB prints by recording the process with a high-resolution webcam. To model multiple printing scenarios, each video represents a combination of multiple parameters, including printing set-up (either mechanical or pneumatic extrusion), material color, layer height, and infill density. After pre-processing, the collected dataset was used to thoroughly train and evaluate an ad hoc defined convolutional neural network by controlling over-fitting and optimizing the prediction time of the network. Finally, the ML model was used in a control loop to: (i) monitor the printing process and detect if a print with an error could be stopped before completion to save material and time and (ii) automatically optimize the printing parameters by combining the ML model with a previously published mathematical model of the EBB process. Altogether, we demonstrated for the first time how ML techniques can be used to automatize the EBB process, paving the way for a total quality control loop of the printing process.
1. Santoni S, Gugliandolo SG, Sponchioni M, et al., 2021, 3D bioprinting: Current status and trends a guide to the literature and industrial practice. Biodes Manuf, 5: 14–42. https://doi.org/10.1007/s42242-021-00165-0
2. Moroni L, Boland T, Burdick JA, et al., 2018, Biofabrication: A guide to technology and terminology. Trends Biotechnol, 36: 384–402. https://doi.org/10.1016/j.tibtech.2017.10.015
3. Bonatti AF, Fortunato GM, de Maria C, et al., 2022, Bioprinting technologies: An overview. Bioprinting, 19-49. https://doi.org/10.1016/B978-0-323-85430-6.00006-6
4. Pati F, Jang J, Lee JW, et al., 2015, Extrusion bioprinting. In: Essentials of 3D Biofabrication and Translation. Academic Press, Cambridge, Massachusetts, p123–152.
5. Choudhury D, Anand S, Naing MW, 2018, The arrival of commercial bioprinters-Towards 3D bioprinting revolution! Int J Bioprint, 4: 139. https://doi.org/10.18063/IJB.v4i2.139
6. Groll J, Burdick JA, Cho DW, et al., 2018, A definition of bioinks and their distinction from biomaterial inks. Biofabrication, 11: 013001.
7. Bonatti AF, Chiesa I, Vozzi G, et al., 2021, Open-source CAD-CAM simulator of the extrusion-based bioprinting process. Bioprinting, 24: e00172. https://doi.org/10.1016/j.bprint.2021.e00172
8. Schwab A, Levato R, D’Este M, et al., 2020, Printability and shape fidelity of bioinks in 3D bioprinting. Chem Rev, 120: 11028–11055. https://doi.org/10.1021/acs.chemrev.0c00084
9. Paxton N, Smolan W, Böck T, et al., 2017, Proposal to assess printability of bioinks for extrusion-based bioprinting and evaluation of rheological properties governing bioprintability. Biofabrication, 9: 044107. https://doi.org/10.1088/1758-5090/aa8dd8
10. Di Pietro L, Ravizza A, Vozzi G, et al., 2019, European regulatory framework for the clinical translation of bioprinted scaffolds and tissues. Biomed Sci Eng, 3: 108. https://doi.org/10.4081/bse.2019.108
11. Yu C, Jiang J, 2020, A perspective on using machine learning in 3D bioprinting. Int J Bioprint, 6: 253. https://doi.org/10.18063/ijb.v6i1.253
12. An J, Chua CK, Mironov V, 2021, Application of machine learning in 3D bioprinting: Focus on development of big data and digital twin. Int J Bioprint, 7: 342. https://doi.org/10.18063/ijb.v7i1.342
13. Freeman S, Calabro S, Williams R, et al., 2022, Bioink formulation and machine learning-empowered bioprinting optimization. Front Bioeng Biotechnol, 10: 913579. https://doi.org/10.3389/fbioe.2022.913579
14. Bone JM, Childs CM, Menon A, et al., 2020, Hierarchical machine learning for high-fidelity 3D printed biopolymers. ACS Biomater Sci Eng, 6: 7021–7031. https://doi.org/10.1021/acsbiomaterials.0c00755
15. Fu Z, Angeline V, Sun W, 2021, Evaluation of printing parameters on 3D extrusion printing of pluronic hydrogels and machine learning guided parameter recommendation. Int J Bioprint, 7: 434. https://doi.org/10.18063/ijb.v7i4.434
16. Ruberu K, Senadeera M, Rana S, et al., 2021, Coupling machine learning with 3D bioprinting to fast track optimization of extrusion printing. Appl Mater Today, 22: 100914. https://doi.org/10.1016/j.apmt.2020.100914
17. Jin Z, Zhang Z, Shao X, et al., 2021, Monitoring anomalies in 3D bioprinting with deep neural networks. ACS Biomater Sci Eng. https://doi.org/10.1021/acsbiomaterials.0c01761
18. Chandola V, Banerjee A, Kumar V, 2009, Anomaly detection: A survey. ACM Comput Surv (CSUR), 41: 1–58. https://doi.org/10.1145/1541880.1541882
19. Wang R, Nie K, Wang T, et al., 2020, Deep learning for anomaly detection. In: Proceedings of the 13th International Conference on Web Search and Data Mining. p894–896. https://doi.org/10.1145/3336191.3371876
20. Pang G, Shen C, Cao L, et al., 2021, Deep learning for anomaly detection: A review. ACM Comput Surv (CSUR), 54: 1–38. https://doi.org/10.1145/3439950
21. Goodfellow I, Bengio Y, Courville A, 2016, Deep Learning. Cambridge, Massachusetts, MIT Press.
22. Chalapathy R, Chawla S, 2019, Deep learning for anomaly detection: A survey. arXiv preprint arXiv, 1901: 03407. https://doi.org/10.48550/arXiv.1901.03407
23. LeCun Y, Bengio Y, Hinton G, 2015, Deep learning. Nature, 521: 436–444.
24. Wang C, Tan XP, Tor SB, et al., 2020, Machine learning in additive manufacturing: State-of-the-art and perspectives. Addit Manuf, 36: 101538. https://doi.org/10.1016/j.addma.2020.101538
25. Meng L, McWilliams B, Jarosinski W, et al., 2020, Machine learning in additive manufacturing: A review. JOM, 72: 2363–2377. https://doi.org/10.1007/s11837-020-04155-y
26. Wu M, Phoha VV, Moon YB, et al., 2016, Detecting malicious defects in 3D printing process using machine learning and image classification. In: ASME International Mechanical Engineering Congress and Exposition. Vol. 50688. American Society of Mechanical Engineers, New York, pV014T07A004.
27. Liu C, Roberson D, Kong Z, 2017, Textural analysis-based online closed-loop quality control for additive manufacturing processes. In: IIE Annual Conference. Proceedings. Institute of Industrial and Systems Engineers, Peachtree Corners, Georgia, p1127–1132.
28. Ogunsanya M, Isichei J, Parupelli SK, et al., 2021, In-situ droplet monitoring of inkjet 3D printing process using image analysis and machine learning models. Procedia Manuf, 53: 427–434. https://doi.org/10.1016/j.promfg.2021.06.045
29. Gobert C, Reutzel EW, Petrich J, et al., 2018, Application of supervised machine learning for defect detection during metallic powder bed fusion additive manufacturing using high resolution imaging. Addit Manuf, 21: 517–528. https://doi.org/10.1016/j.addma.2018.04.005
30. Sing SL, Kuo CN, Shih CT, et al., 2021, Perspectives of using machine learning in laser powder bed fusion for metal additive manufacturing. Virtual Phys Prototyp, 16: 372–386. https://doi.org/10.1080/17452759.2021.1944229
31. Jin Z, Zhang Z, Gu GX, 2019, Autonomous in-situ correction of fused deposition modeling printers using computer vision and deep learning. Manuf Lett, 22, 11–15. https://doi.org/10.1016/j.mfglet.2019.09.005
32. Tonnaer L, Li J, Osin V, et al., 2019, Anomaly detection for visual quality control of 3D-printed products. In: 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, Piscataway, New Jersey, p1–8.
33. Zhang J, Wang P, Gao RX, 2019, Deep learning-based tensile strength prediction in fused deposition modeling. Comput Ind, 107: 11–21. https://doi.org/10.1016/j.compind.2019.01.011
34. Zhang B, Liu S, Shin YC, 2019, In-process monitoring of porosity during laser additive manufacturing process. Addit Manuf, 28: 497–505. https://doi.org/10.1016/j.addma.2019.05.030
35. Wang T, Kwok TH, Zhou C, et al., 2018. In-situ droplet inspection and closed-loop control system using machine learning for liquid metal jet printing. J Manuf Syst, 47: 83–92. https://doi.org/10.1016/j.jmsy.2018.04.003
36. McKinney W, 2011, pandas: A foundational Python library for data analysis and statistics. Python High Perform Sci Comput, 14: 1–9.
37. Bradski G, 2000, The openCV library. Dr Dobbs J Softw Tools, 25: 120–123.
38. Abadi M, Agarwal A, Barham P, et al., 2016, Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv, 1603: 04467. https://doi.org/10.48550/arXiv.1603.04467
39. Chollet F, 2018, Keras: The python deep learning library. Astrophysics Source Code Library, ascl-1806.
40. Pedregosa F, Varoquaux G, Gramfort A, et al., 2011, Scikit learn: Machine learning in Python. J Mach Learn Res, 12: 2825–2830.
41. Bonatti AF, Fortunato GM, Lapomarda A, et al., 2021, Advanced Firmware and Hardware for Multiscale and Multimaterial Bioprinting. Proceedings of the Seventh National Congress of Bioengineering.
42. Zhao M, Bu J, Chen C, 2002, Robust background subtraction in HSV color space. In: Multimedia systems and Applications, Vol. 4861. International Society for Optics and Photonics, United States, p325–332. https://doi.org/10.1117/12.456333
43. Simonyan K, Zisserman A, 2014, Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv, 1409: 1556. https://doi.org/10.48550/arXiv.1409.1556
44. He K, Zhang X, Ren S, et al., 2016, Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. p770–778.
45. He K, Zhang X, Ren S, et al., 2015, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision. p1026–1034.
46. Kingma DP, Ba J, 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv, 1412: 6980. https://doi.org/10.48550/arXiv.1412.6980
47. Selvaraju RR, Das A, Vedantam R, et al., 2016, Grad-CAM: Why did you say that? arXiv preprint arXiv, 1611: 07450. https://doi.org/10.48550/arXiv.1611.07450