Machine Learning in the Microscopy Lab

How Machine Learning Supports Materials Microscopy

  • Credit: Aalen UniversityCredit: Aalen University
  • Credit: Aalen University
  • Fig. 1: Exemplary schema for training a supervised ML model for image data [1].
  • Fig. 2: Multiphase segmentation of a magnetic alloy for electrical machines by pixel-by-pixel classification. Left: Light microscopic brightfield image with difficult distinction between the seven components. Right: Result of an ML-based segmentation with reliable differentiation of the optically similar phases after expert training of the algorithm
  • Fig. 3: Automated layer thickness measurement over large sample areas in lithium-ion cells. Top: Image montage. Detail of a large area image of a PHEV2 cell with a typical layer structure of anodes, cathodes and isolating separators on the left side. The segmentation into the active material layers to be measured (green on the anode side, blue on the cathode side) as well as the separators and collector foils in between (red) runs to the right. The two lower graphs: Exemplary layer thickness gradients for one anode and one cathode each.
  • Fig. 3: Automated layer thickness measurement over large sample areas in lithium-ion cells. Top: Image montage. Detail of a large area image of a PHEV2 cell with a typical layer structure of anodes, cathodes and isolating separators on the left side. The segmentation into the active material layers to be measured (green on the anode side, blue on the cathode side) as well as the separators and collector foils in between (red) runs to the right. The two lower graphs: Exemplary layer thickness gradients for one anode and one cathode each.
  • Fig. 3: Automated layer thickness measurement over large sample areas in lithium-ion cells. Top: Image montage. Detail of a large area image of a PHEV2 cell with a typical layer structure of anodes, cathodes and isolating separators on the left side. The segmentation into the active material layers to be measured (green on the anode side, blue on the cathode side) as well as the separators and collector foils in between (red) runs to the right. The two lower graphs: Exemplary layer thickness gradients for one anode and one cathode each.
  • Fig. 4: Machine Learning helps to distinguish real non-metallic inclusions (NMI) in steel from preparation artefacts and “cleans” the image data for standard-compliant evaluation.
  • Fig. 5: Examples of “worst picture galleries” of abnormalities in the microstructure of lithium-ion batteries (left) and sinter magnets (right), based on CNN / Deep Learning.
  • Fig. 6: Examples of automatically detected defects in a titanium casting from the aerospace industry [9]. Class Activation Heatmaps are superimposed in color: The brighter an area, the higher the net classifies the image area as a defect.

Digitalization is increasingly penetrating materials microscopy. Microscopic image acquisition is increasingly automated, enabling large sample areas or even complex materials to be digitally imaged. The data-driven methods of machine learning can help to obtain relevant information from images and thus simplify laboratory work and make it more reproducible.

Machine Learning for the Analysis of Microscopy Data

Machine Learning (ML) tries to implicitly learn information from available data. The computer learns from example data, which describe the problem as generally as possible, in connection with an associated target quantity such as an image or material property. The aim of ML methods is to recognize patterns in data and to derive underlying principles from them. These patterns can also represent complex, non-linear relationships. Machine Learning is particularly suitable for problems in which cause-and-effect relationships are not completely known or can hardly be grasped by humans due to too many influencing variables. For ML to work, it needs a sufficient amount of high-quality data, as well as experts to label - that is, to annotate the data to train the algorithms.

Multiphase Segmentation as the Basis for Quantitative Microstructural Analysis

The segmentation of microstructural images - i.e. the partition of the microstructure in the images into the phases and components to be quantified - is a central task in quantitative microstructural analysis. The requirements for accuracy and reproducibility are particularly high here. Multiphase materials that appear similar under the microscope are challenging. Similar gray or color levels can hardly be distinguished with conventional threshold-based methods. The ML-based classification of individual pixels is much less susceptible. This makes it possible to differentiate a complex magnetic alloy for electrical machines into quality- and property-relevant components.

ML also makes it possible to optically separate the complex inner layer sequence in a lithium-ion cell consisting of cathode, anode and separator and thus to measure a large area of the layer thickness.

This makes it possible to measure deviations from a cell specification. This is relevant for the control of the production and finally the application of lithium-ion batteries for mobile or stationary energy storage [2].

Clean Up Data with ML

The purity of steels - i.e. the measurement of non-metallic inclusions (NMI) - is important for their machinability and mechanical strength. For the measurement, samples have to be polished and cleaned for microscopy - often insufficiently. Preparation artefacts such as scratches, foreign particles or dry spots appear almost identical to real inclusions. Existing approaches to image processing fail here and cannot separate real from unreal in many cases. The result is either falsified data or a high amount of time required for human post-processing. This is precisely where machine learning can help by training the computer in a two-step process to distinguish real inclusions from artefacts on the basis of texture parameters in the image. With this “cleaned” image, a standard-compliant analysis (e.g. EN 10247, ASTM E45 or ISO 4967) can be implemented. This eliminates a lot of unnecessary post-processing time for the operator.

Conspicuous Features? ML Takes Over the Search...

Materials and components are never free of defects due to production. Defects can be disadvantageous for material properties, but they can also be relevant for operational safety. It is therefore of particular interest for the engineer to know these anomalies in order to be able to evaluate their relevance in general. For this purpose, deep artificial convolutional neural networks (CNN) trained on certain structures are used. With the help of these CNNs, security-relevant foreign inclusions or deformations - i.e. conspicuous structures - can be automatically found in microscopic images of Li-ion cells and output in the form of a “worst picture gallery”. A similar approach is used for sintered magnets. Here it is defective or inhomogeneous microstructure areas that have a negative effect on magnetic properties. In this case, these were detected using an anomaly detection approach, i.e. a Generative Adversarial Network was only trained with error-free image sections [2-6].

...also Non-destructive with High-precision and Reproducibility: ML for NDT

Automated ML methods such as CNNs are also relevant for non-destructive testing (NDT), such as X-ray tomography, scanning acoustic microscopy or thermography. CNN results are 100% reproducible. With detection rates of sometimes more than 99%, they are also more reliable than purely visual image evaluation by an individual. Approaches with ML are particularly interesting in safety-critical industries, such as the X-ray inspection of components in aviation. Unfortunately, the result can vary greatly depending on the inspector. The identification and characterization of potential defects is therefore prone to errors. CNNs can make a significant contribution to reducing the error rate. However, it must also be emphasized that the ML methods have to be adequately trained by the experts [7, 8].

Machine Learning Ready for Use?

The approaches presented offer enormous potential to automate time-consuming or repetitive evaluations in microscopy. Many routine tasks can already be solved. Deep CNNs offer even more possibilities. Although these are more computationally intensive than classical ML methods, they are capable of independently extracting relevant features from the images. Thus, mainly relevant features for the problem are taken into account. In many domains CNNs have already become the frontrunners compared to classical ML methods and digital image processing. They often outperform humans. One vision in the field of material microscopy is not only to use the methods for quantification and evaluation, but also to link the information obtained with manufacturing parameters and/or application properties of the materials [10].

Authors
Andreas Jansche1, Amit Kumar Choudhary1, Olatomiwa Badmos1, Hermann Baumgartl1, Daniel Sauter1, Timo Bernthaler1, Ricardo Büttner1, Gerhard Schneider1

Affiliation
1Aalen University, Aalen, Germany

Contact
Andreas Jansche

Aalen University
Aalen, Germany
andreas.jansche@hs-aalen.de
www.hs-aalen.de/imfaa

References:

[1]     A. Jansche, A.-K. Choudhary, O. Badmos, H. Baumgartl, T. Bernthaler, R. Büttner, G. Schneider: „Künstliche Intelligenz in der Materialmikroskopie", Jahresmagazin Ingenieurwissenschaften / Werkstofftechnik 2019, S. 14-17

[2]     A. Jansche, F. Trier, A. Kopp, P. Schirle, T. Bernthaler, U. Klauck, G. Schneider: Maschinelles Lernen zur Quantifizierung und Defekterkennung an lichtmikroskopischen Bildern prismatischer Li-Ionen Zellen, Tagungsband des 22 Workshop Farbbildverarbeitung 2016 in Ilmenau, 61–74 (2016)

[3]     Y. LeCun, Y. Bengio, G. Hinton: Deep learning, Nature, Vol. 521, 7553, 436–444 (2015)

[4]     H. Baumgartl, R. Buettner, T. Bernthaler, I. J. Timm, A. Jansche, G. Schneider: Colored micrographs significantly outperform grayscale ones in modern machine learning, in MCM 2017 Proceedings, 99–101 (2017)

[5]     O. Badmos, A. Jansche, T. Bernthaler, G. Schneider: Application of deep learning in materials microscopy to evaluate Lithium-Ion batteries, in DGaO-Proceedings 2018, Aalen (2018)

[6]     A. Jansche, O. Badmos, A. K. Choudhary, T. Bernthaler, G. Schneider: Machine Learning als Tool in der Materialmikroskopie, in DGaO-Proceedings 2018, Aalen (2018)

[7]     H. Baumgartl, J. Tomas, R. Buettner, M. Merkel: A novel Deep-Learning Approach for Automated Non-Destructive Testing in Quality Assurance based on Convolutional Neural Networks, in ACEX-2019 Proceedings, Athen [2019]

[8]     M. Bertović: Human Factors in Non-Destructive Testing (NDT): Risks and Challenges of Mechanised NDT, Dissertation, Universität Berlin, Berlin [2015]

[9]     Anton du Plessis, Pierre Rossouw: X-ray computed tomography of a titanium aerospace investment casting, Case Studies in Nondestructive Testing and Evaluation, Vol. 3, 21-26 (2015)

[10]   A. Fuchs, T. Bernthaler, B. Stahl, U. Klauck, B. Reinsch, G. Schneider: Bildanalyse komplexer Werkstoffgefüge durch Texturanalyse und Korrelation mit den Eigenschaften durch neuronale Netze, Zeitschrift für Metallkunde, Vol. 92, 8, 979 -985 (2001)

More on data analysis
 

 

 

You may also be interested in

Register now!

The latest information directly via newsletter.

To prevent automated spam submissions leave this field empty.