Automatic crack detection in forged metal parts an image analysis approach Master of Science Thesis MARTIN LUNDH Department of Signals and Systems Division of Signal Processing and Biomedical Engineering CHALMERS UNIVERSITY OF TECHNOLOGY Göteborg, Sweden 2012 Report No. EX073/2012 REPORT NO. EX073/2012 Automatic crack detection in forged metal parts an image analysis approach MARTIN LUNDH Department of Signals and Systems Division of Signal Processing and Biomedical Engineering CHALMERS UNIVERSITY OF TECHNOLOGY Göteborg, Sweden 2012 Automatic crack detection in forged metal parts an image analysis approach MARTIN LUNDH c©MARTIN LUNDH, 2012. Technical Report no. EX073/2012 Department of Signals and Systems Division of Signal Processing and Biomedical Engineering Chalmers University of Technology SE–412 96 Göteborg Sweden Telephone +46 (0)31–772 1000 Cover: CAD model of connecting rod with 3D mapped indications Chalmers Bibliotek, Reproservice Göteborg, Sweden 2012 Automatic crack detection in forged metal parts an image analysis approach MARTIN LUNDH Department of Signals and Systems Division of Signal Processing and Biomedical Engineering Chalmers University of Technology Abstract Manual quality control is always associated with a certain degree of variation in both throughput and accuracy. An automated vision system can significantly improve both, and this master thesis aims to develop such a system for the specific application of crack detection in forged metal parts. The work has been carried out at Consat Engineering in Stockholm. This thesis combines advanced image analysis techniques with the established method of magnetic particle testing. The source images generated by magnetic particle testing has characteristically low signal to noise ratio, which has hindered previous attempts to create an automated system. The vision system consists of a camera equipped with a UV filter, two UV lights and a software library called HALCON. The system uses an edge detection algorithm that produces sub pixel precise results to indicate the locations of potential flaws. Review of the images showed that the properties of a real indication stays the same independent of viewing angle, but false indications change properties. 3D techniques were used to reduce the number of false positives by taking advantage of this phenomenon. A 3D calibrated area scan camera was used in combination with a 3D matching algorithm to find the 3D pose of the connecting rod in the image. The crack indications were mapped onto a 3D model of the connection rod based on this pose. The results show that the number of false indications can safely be assumed to decrease substantially while using the 3D techniques, while still maintaining detection of the real flaws. All subsystems were verified and the thesis concludes that further development is recommended. Index Terms: Image analysis, Connecting rod, Crack detection, 3D vision, Nondestructive testing iii iv Acknowledgments This thesis has been carried out at Consat Engineering in Stockholm in collaboration with the Signals and Systems department at Chalmers University of Technology. I would like to thank my supervisors, Håkan Kroksmyr at Consat Engineering and Artur Chodorowski at Chalmers University of Technology. Further I would like to thank a number of people at Consat, mainly Lars-Åke Johansson for his extensive support and advice along the whole project, and my colleagues at the Stockholm office. I would also like to thank Thomas Ringvall at Karl Deutsch Nordiska for his help in getting the parts tested with penetrant testing, as well as Jonas Höjer at Volvo Penta for his help with magnetic testing and destructive testing. Finally I would like to thank my girlfriend for her patience and support during the long hours of the project. Martin Lundh Stockholm, Sweden, 2012 v vi Contents Abstract iii Acknowledgments v Contents vii 1 Introduction 1 1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.3 Previous work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Technical background 3 2.1 Nondestructive testing methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1.1 Magnetic particle testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1.2 Dye penetrant testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Destructive testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Image analysis techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3.1 Noise reduction using spatial filtering . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3.2 Morphological image analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3.3 Canny edge detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.4 Segmentation and feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.5 Line detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.6 3D calibration of a 2D camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.7 3D pose from a 2D image using shape-based matching . . . . . . . . . . . . . . . 11 2.3.8 2D to 3D mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3 Hardware set-up 13 3.1 Image acquisition setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Destructive testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4 Evaluation of non destructive testing methods 17 4.1 Magnetic particle testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2 Dye penetrant testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5 The crack detection algorithm 19 5.1 Assumptions of flaw properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.2 Shape based crack detection algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.2.1 Pre-processing the image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.2.2 Image segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.2.3 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.2.4 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.2.5 Parameter robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.3 3D projection based crack detection algorithm . . . . . . . . . . . . . . . . . . . . . . . . 22 vii Contents 5.3.1 Overview of the 3D projection based algorithm . . . . . . . . . . . . . . . . . . . 22 5.3.2 Crack detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.3.3 3D matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.3.4 3D mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.3.5 Processing the feature vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.3.6 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.3.7 Implementation of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.3.8 Comments on image analysis approaches for crack detection . . . . . . . . . . . . 25 6 Results and Conclusions 27 6.1 Results from present work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 6.1.1 Crack detection using 3D projection based image analysis . . . . . . . . . . . . . 27 6.1.2 3D matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.1.3 3D mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.1.4 Destructive testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 References 41 viii Nomenclature Abbreviations ET: Eddy current testing KDN: Karl Deuch Nordiska MT: Magnetic particle testing NDT: Nondestructive testing PT: Dye penetrant testing Key words 3D pose: Position in 3D space, rotation and translation in relation to a reference system Crack indication: Indication of material separation created by an NDT Crack: Material separation False indication: Crack indication not corresponding to a real crack True indication: Crack indication corresponding to a real crack Undetected crack: Real crack not detected by NDT ix Contents x List of Figures 1.1 The connecting rods used in the project . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 Field lines in a ferromagnetic material with a crack. Image courtesy of Karl Deutsch Nordiska 4 2.2 Part inspected with MT under ultraviolet light. Square marking on part is only visual aid and not present on parts in production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Part inspected with PT using red dye. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.4 Example of how a crack looks like using destructive testing. . . . . . . . . . . . . . . . . 6 2.5 Camera coordinate system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.6 Calibration plate used for camera calibration in HALCON . . . . . . . . . . . . . . . . . 11 2.7 Calibration plate seen in different perspectives . . . . . . . . . . . . . . . . . . . . . . . . 11 2.8 Line of sight from camera to image and the intersection with a triangle from the model. . . 12 3.1 Magnetic particle testing equipment at the plant . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Different mean for anything but multiples of 50Hz when sine wave has that frequency . . . 14 3.3 Image acquisition setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.4 Test setup, the black plastic is put over the setup to block ambient light. . . . . . . . . . . 15 4.1 MT testing clearly shows the defect. Image courtesy of Karl Deutsch Nordiska . . . . . . 17 4.2 PT testing does not give clear indications. Image courtesy of Karl Deutsch Nordiska . . . . 18 5.1 Flowchart of the shape based crack detection algorithm. . . . . . . . . . . . . . . . . . . . 20 5.2 Flowchart of the 3D projection based crack detection algorithm. . . . . . . . . . . . . . . 23 6.1 Crack detection, image 1-3 in series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 6.2 Crack detection, image 4-6 in series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 6.3 Crack detection, image 7-9 in series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 6.4 Crack detection on image provided by KDN . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.5 3D matching in daylight, camera poorly calibrated. . . . . . . . . . . . . . . . . . . . . . 32 6.6 Crack indication and mapping first image in series . . . . . . . . . . . . . . . . . . . . . . 33 6.7 Crack indication and mapping second image in series . . . . . . . . . . . . . . . . . . . . 34 6.8 Destructive testing, sample 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.9 Destructive testing, sample 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 xi LIST OF FIGURES xii Chapter 1 Introduction The rapid development of digital computers has led to a vast expansion of applications for computer vision systems. Processes that previously had to be done manually can now be automated using computers and cameras. Computer vision and image analysis have a very high demands for processing power, something that still is a problem for real time applications. While the computers and vision algorithms are getting very advanced, to a level where they supersede human capacity for certain tasks, there are still tasks that has been deemed too difficult. The human visual cortex is a vastly advanced and highly trainable computer that has an amazing ability to analyze images, and the tasks it performs are in many cases difficult, if not impossible, to replicate digitally. It it however limited by a shorter endurance. This is true for most kinds of real-world applications; there is seldom a one best way to solve a computer vision problem, and it is therefore important to be able to quickly draft an algorithm and test its effectiveness [6]. 1.1 Purpose This project aims to develop a system that automatically detects cracks in forged metal parts. This includes, but is not limited to, testing other non destructive testing methods as well as different image analysis tools. 1.2 Background Connecting rods are used in all types of internal combustion engines, ranging from simple one cylinder moped engines to large ship engines. They connect the piston to the crank shaft, and they are subject to considerable forces. The cost of a single connection rod is relatively small, and quality control in the factory can discard a fair number of forged parts without any significant losses. If a rod breaks when it is in use in an engine, the damage will be considerably larger. The vehicle will have to be taken out of service, the engine unmounted and the connection rod replaced. A failure might also damage other parts of the engine. The quality control during manufacturing of these engine parts is hence very important and can potentially save considerably sums. The connecting rods used in this project are shown in figure 1.1. They have been forged and sand blasted, but not machined. Today, most stages during the manufacturing of connection rods are automated. However, one part that almost always needs manual labor is the visual inspection, i.e. crack detection. Research has shown that a human operator loses focus after a short period of time when performing tasks that require a high level of concentration. If the operator is working longer shifts, it was shown that products that did not meet the acceptance level were approved anyway. This thesis aims to develop method to automate this quality control of connection rods. This would increase both the throughput and accuracy in the quality control, while improving the cost efficiency. Previously, a shorter project investigated the possibility of adding a visual system to the already existing inspection station. The visual system consisted of a camera, two UV lights and an image processing tool called Manto. The results indicated the great difficulties in detecting cracks without a large amount of false 1 Chapter 1. Introduction Figure 1.1: The connecting rods used in the project indications. The report recommended that future work should investigate different detection methods and the possibility to automate them fully or partially. The innovation of this thesis lies not in the image analysis tools themselves since existing tools have in fact been used, but rather in the combination of the existing tools and the development of an algorithm that completes the task. The task of the system is highly specialized as it aims to detect very small indications. 1.3 Previous work A number of attempts had been made to automatically detect surface defects in forged metal parts, all of which have had limited success. An attempt using a neural network approach has been made by Consat, the result of which is described in the internal report [7]. They conclude that the methods used were insufficient for the demands of the task. Other studies have been done in the area of using image analysis to detect cracks and defects in mate- rials, a comparison of image analysis techniques for detecting cracks in concrete structures can be found in [14], but no study of automatic crack detection in metal using image analysis has been found. There are however methods that automatically detects cracks in metal, they are however relying on other techniques than image analysis and are not suited for the task this thesis aims to solve. 1.4 Limitations Due to time limitations no investigations will be performed as to the acceptance level of cracks. The ac- ceptance level will instead be based on the knowledge of the current operator, and be based on a larger collection of samples. • The thesis will not consider the cost of a completed system, only the performance of the visual system will be investigated. • A complete system will not be designed, only a design proposal will be presented. 2 Chapter 2 Technical background The work performed in this thesis is based on a theoretical foundation, some key parts of which will be presented in the following sections. Both nondestructive testing methods and image analysis tools will be described. 2.1 Nondestructive testing methods There are a number of different methods to test whether a material has flaws in it or not. The methods that are collectively called ”nondestructive testing” has the common feature that they do not harm the material being tested but uses material properties to create indications of flaws. Each method has its own special properties and depending on the the task, one will be more suitable than the others. Some of the methods are able to detect flaws inside the material and some only detects surface and near surface flaws. A crack is by definition a very small material separation. In [4] they state that a crack usually is more than ten times longer along the surface than it is deep into the material. The width of the crack is narrow, at least ten times smaller than the depth. In addition, the crack is usually sharp at its base. There are a number of different types of flaws within a material, cracks, impurity and so on. The different methods detect different types of flaws. Once a forged metal part is used, the stresses produced will be mainly located on the surface of the product. Cracks and other defects that lie close to the surface are therefore more dangerous than those that lie within the material. It is also true that, in general, fatigue cracks start developing on the surface of the material and, as they grow larger, goes deeper into it. The complex geometry of the connecting rod causes problems for some of the methods, like ultrasonic testing and eddy current testing. This means that these the methods are unsuited for the purposes of this the- sis. The methods that are viable for further testing are Magnetic particle Testing and dye Penetrant Testing. This section will describe these two methods, the basic principles of how they work and the requirements on the products being tested. The section will also show some example images of how an indication might look with the different methods. 2.1.1 Magnetic particle testing Magnetic particle testing (MT) uses properties of magnetic fields to indicate defects. The method is able to detect surface and near-surface flaws in ferromagnetic materials. The part to be inspected must be made out of a ferromagnetic material in order to be able to sustain magnetic flux. The basic principle of magnetic particle testing uses the fact that a ferromagnetic material can sustain more magnetic flux than air. If a crack is present in the test material, the gap will not be able to sustain the same amount of flux as the surrounding material ,and some of the flux will be forced out of the part. The part of the field lines that are forced out by the crack is called ”stray flux” [4]. This ”stray flux” will, together with the created poles in the crack, attract other magnetic material to a larger extent than the surrounding material, see image 2.1. This is used in magnetic particle testing by introducing small magnetic particles onto the tested part. The particles are colored with a dye to create a contrast to the background color of the part, it could be black (for a light colored background) or fluorescent (for any color background, except 3 Chapter 2. Technical background florescent). The result of adding these small magnetic particles is that the crack is indicated visually by a collection of particles at the location of the flaw, there is a higher density of particles at the locations of flaws. Figure 2.1: Field lines in a ferromagnetic material with a crack. Image courtesy of Karl Deutsch Nordiska The test procedure consists of a number of steps. The part is magnetized and a carrier fluid with magnetic indicators is applied. It is important to remember to magnetize in multiple directions so that all cracks are detected. The magnetic flux will be interrupted by any anomaly such as a crack or cavity since they can not sustain the same amount of flux. This will force the flux to travel a different path. The changed flux will disturb the magnetic particles applied, these disturbed regions are seen as markings and are thus called ’crack indicators’ and become the object of analysis in order to determine the quality of the part. [11]. Figure 2.2: Part inspected with MT under ultraviolet light. Square marking on part is only visual aid and not present on parts in production. 4 2.1. Nondestructive testing methods Magnetic particle testing is extensively used in the metal forming industry for purposes of quality con- trol. It can also be used as a method of detecting fatigue cracks, a test which could for example be performed during normal service of the product. An important property of the MT method is that the test piece does not have to be completely clean, some dirt and corrosion does not effect the result to any significant extent. This reduces the need to clean the part prior to the test, which saves time and reduces cost. The method only detects cracks perpendicular to the induced magnetic field, so the part has to be mag- netized twice, once in each direction, to ensure that all flaws are detected. The operator has to be trained to determine what kind of anomaly an indication is, and whether it is dangerous for the application. 2.1.2 Dye penetrant testing Dye penetrant testing (PT) was first used by railway pioneers to ensure the quality of the rails. The technique has been refined since then, but the main principle stays the same. Capillary action draws a dye into surface defects that later are visualized with a developer, also using capillary action. The work flow for PT is quite simple, First the part is cleaned thoroughly to remove any dirt from the cracks. The part to be inspected is then covered in a dye fluid, usually with a contrast agent in it. The fluid penetrates into any crack or cavity of the object. The remaining dye is cleaned and a developer is applied. The part is then visually inspected to detect any indications of a flaw. The method can only detect surface defects that can hold the fluid (penetrant). Both the dye and the developer relies on capillary action to draw the dye into/out of the cavities. The dye can be either colored with a dye directly visible to the human eye, or with fluorescent dye. The fluorescent dye has to be illuminated with a black light ( ultra violet light) and the fluorescence emitted is then visible to the human eye. This method is usually more accurate, and can reveal smaller cracks. Figure 2.3: Part inspected with PT using red dye. To ensure that the dye penetrates the defect properly the product is soaked for a period of time, the so-called ”dwelling time”. It is important that this time is long enough or the defects will not be detected. The same is true for the developer, it is important that the part is left to develop long enough or the defects might not be visible. It is however not possible to over-soak it, assuming the dye does not dry. Cleaning is an important part of PT since it first ensures that the crack can take dye, and later removes any excess dye so that the crack can be seen after the developing stage. The amount of cleaning necessary 5 Chapter 2. Technical background might vary with the type of product being tested and the size of cracks that are to be detected. The amount of cleaning after the part has been soaked with dye is crucial, as to little cleaning will give false indications and too much cleaning will remove real indications. [11] 2.2 Destructive testing In contrast to nondestructive testing, destructive testing damages the tested material. Methods of destructive testing range from collecting surface samples to sawing perpendicular to a crack indication. The destructive testing method used in this thesis to determine the actual reason of a flaw indication is the latter one, sawing and viewing the defect under a microscope. This method will accurately determine the size and type of defect present. The part is first tested using MT, and indications from that test are marked. A cut is made perpendicular to the indication and the cut surface is viewed under a microscope. Figure 2.4: Example of how a crack looks like using destructive testing. The destructive testing can accurately determine the type and size of a flaw, but not the severity of the flaw. The severity is directly connected to the manufacturing methods used when the part is produced, and the load cases it will be subject to in regular use. Shot blasting the part will induce tension in the surface of the product, and if the defect lies within the region of the tension it might not be dangerous. If, however, the flaw extends beyond that depth it might grow and finally result in a failure of the part. 2.3 Image analysis techniques The algorithm developed is based on a few basic image analysis techniques. The innovation lies instead in the combination of the existing techniques. The techniques are not new, and extensive literature exists within the area of image analysis. Some basic methods used in the project will be described here, only in detail necessary to understand the project. For a deeper explanation and description the reader is advised to refer to textbooks in the area, such as [6] and [5]. These two books has been the main source of information to this text. 6 2.3. Image analysis techniques 2.3.1 Noise reduction using spatial filtering In image analysis it is common to filter images either in the image domain or in the transform domain, for example the Fourier Domain. Spatial filtering takes place in the image domain and is hence applied to the image directly without transforming the image. A spatial filter uses a filter mask to define the properties of the filter. The response for a specific pixel position in the output space is the weighted sum of all the pixels in the neighborhood specified by the filter mask. The method used is called two dimensional convolution. The design of the filter mask can be used to produce various results ranging from edge detection and gradient estimation to smoothing and contrast enhancement, even sharpening. One filter that has been extensively used in the thesis is the Gaussian low pass filter. The filter is defined according to equation 2.1 [6] g(x, y) = 1 2πσ2 e− x2+y2 2σ2 (2.1) This equation describes a two dimensional Gaussian distribution g(x, y) where x and y are spatial image coordinates and σ being the standard deviation of the Gaussian distribution. Values from this distribution are sampled (usually at the pixel centers) and used as a filter mask. The filter is used to smoothen images prior to other operations that might be noise sensitive. It is, for example used in the Canny edge detector as a pre-processing stage to remove unwanted noise that would yield false edge locations. Other filters are derived using similar methods, sampling of a theoretical function. 2.3.2 Morphological image analysis The most fundamental morphological operators in image analysis are erosion and dilation. They both use a structure called a structuring element which controls the behavior of the operator. The structuring element is a matrix of 1s and 0s or as a set of background and foreground pixels. The structuring element has a fixed origin, and moving the origin within the structuring element usually creates a different behavior of the operator. Morphological dilation is defined by A⊕B = {z|(B̂z) ∩A 6= ∅} (2.2) And erosion is defined by A B = {z|(Bz) ∩Ac 6= ∅} (2.3) Where A is the image on which the erosion/dilation is applied and B is the structuring element. B̂ is the reflection of the set B and Bz is the translation of set B by vector z. ∅ is the empty set, and B ∩ A is the intersection of set A and B, Ac is the complement of set A. The dilation of A with B is in other words the set consisting of all the structuring element origin locations where the reflected and translated B overlaps at least one element of A. It is clear from the definition that a dilation of an image will ”thicken” its content (in the binary image). Erosion is the counterpart to dilation, the output image contains 1s in the structuring element origin locations where all elements of the structuring element overlaps positions with 1s. Opening and closing are two useful morphological operators that are commonly used to remove salt and pepper grain for example. Opening is defined as the erosion of A with B followed by the dilation of the result with B: A ◦B = (A B)⊕B (2.4) and morphological closing is defined as a dilation followed by erosion: A •B = (A⊕B) B (2.5) These and more have been used in the development of the algorithm described in detail in chapter 5. 7 Chapter 2. Technical background 2.3.3 Canny edge detector The Canny edge detector uses a combination of different techniques to give a robust and accurate localiza- tion of edges in an image. The main benefit of the Canny edge detector over the more classical detectors such as Sobel or Previtt is that it is more accurate and less sensitive to noise. It should be said that the Canny method uses one of these classical methods within, but with pre- and post-processing methods to improve quality of the detections. The method was developed by John F. Canny in 1986 [3] and follows the steps in the list below: 1. Smoothening of the image. This eliminates noise that otherwise would yield false edges. This smoothening is usually done with a spatial Gaussian low pass filter. 2. Finding the intensity gradient of the image. This can be done using one of many edge detector filters. Normally they filter the image in two directions, horizontal and vertical as they only can detect edges in one direction every pass. From the edge detectors it is possible to extract estimations of the edge gradient and edge direction. 3. Non-maximum suppression. An edge point is defined so that it is the local maximum in the direction of the gradient. These edges are marked and all pixels not belonging to the edge are set to zero. 4. Double thresholding. The pixels that remains after the non-maximum suppression still have their strength information. The double thresholding divides these into three categories, false edges, weak edges and strong edges. The pixels with intensity smaller than the low threshold are considered false, and are discarded as edge points. The pixels with a gradient intensity in between the low and the high thresholds are called weak edges and finally all pixels with a gradient intensity larger than the high threshold are called strong edges. 5. Hysteresis tracking of the image gradient. The strong edges are considered to be true edges and are all included in the output. The weak edges are only included if they are connected to a strong edge. The edge tracking can be performed in a number of different ways, one being to connect all edge pixels using 8-connectivity and saving all regions including at least one strong edge pixel. This results in a binary image. Even though the Canny edge detector is fairly advanced and can yield very good results, it still has some limitations. It is a method that is based on thresholding and therefore the normal problems of deciding the threshold values apply. It is difficult to provide thresholds that always will give good answer, and so thresholds are strongly related to the fundamental challenges of the vision system. 2.3.4 Segmentation and feature extraction The term image segmentation implies dividing the image into smaller regions containing certain parts of the image. The number and properties of these regions vary dependent on the quality of the segmentation but also with the aim of the segmentation. In feature extraction, the aim of segmentation is to isolate an object without the background and determine certain properties of that region. There are a number of different im- age analysis methods available if the goal of the analysis is to find a specific object with distinct properties. One of these methods is for example, correlation. Image segmentation is a very important task when working with machine vision and there exist many different techniques to achieve good segmentation. In order to choose between the different methods it is advisable to look at the requirements of the segmentation, the properties of the object to be segmented and lastly, to try many different approaches as there is no ”one best way” when dealing with image analysis. Point, line and edge detectors can be created using different spatial filter masks. This is in particular the way the Sobel edge detector works. It first creates a gradient map of the image and thresholds that map at a specific value for the gradient. Since the gradient is highest at the points where the change in image intensity is greatest, it is invariant to the actual intensity of the image, only relative intensity is of interest. The gradient map contains large values in regions with rapid changes and low values for regions with uniform intensity. A line for example, usually consists of one sharp edge on each side and a narrow band of constant intensity in the middle. This means that in the gradient map, a line will be represented as 8 2.3. Image analysis techniques two edges separated by a region of low values. This property is often used to distinguish between lines and other artifacts. The over all most intuitive method of segmentation is global thresholding. A intensity value is chosen as the threshold (T ) and everything above that is set to 1s and everything below is set to 0s: h(x, y) = { 1 if h(x, y) > T 0 if h(x, y) < T (2.6) Where h(x, y) is a spatial function and x and y are spatial coordinates. The threshold T can also be chosen dynamically or specific for a certain region instead of for the whole image. This is known as local thresholding and can often give better results than the global one. The drawback is computational complex- ity and more design parameters. 2.3.5 Line detection The line detector used takes advantage of partial derivatives of a Gaussian kernel to determine parameters of a polynomial describing the image for each pixel in the image. The standard deviation of the Gaussian kernel determines the amount of smoothing that will be applied to the image. A larger amount of smoothing results in fewer detected lines, but also to a worse localization of the lines. The parameters of the polynomial are used to calculate the direction of the line in the image plane. The pixels that have a local maximum in the second derivative perpendicular to the line direction are marked to be lines. The lines found in this manner are connected to contours if they fulfill the conditions of a hysteresis threshold, similar to the Canny edge detector. [1] The laplacian, or second derivative, of an image I , is given by: L(x, y) = ∂2I ∂x2 + ∂2I ∂y2 A common digital approximation of this is given by [6]: L(x, y) = [f(x+ 1, y) + f(x− 1, y) + f(x, y + 1) + f(x, y − 1)]− 4f(x, y) This kernel is an approximation to the second derivative of the image, and is therefore very sensitive to noise. To counter this the image is usually smoothened with a Gaussian kernel first. Because convolution is associative it is possible to first convolve the kernel of the second derivative with the Gaussian kernel to produce the ”Laplacian of Gaussian” that will provide cleaner results. 2.3.6 3D calibration of a 2D camera A pin hole camera projects a 3D point into the image plane using a so called perspective projection. If the 3D point is given in the camera coordinate system, where the X and Y axis are parallel to the column and rows of the image respectively, and the Z axis is perpendicular to the image (see image 2.5), then the projection is given by 2.7 [8]. The point P = (xc, yc, zc) is the 3D point given in camera coordinates, f is the focal length and u and v are image plane coordinates. [12] ( u v ) = f zc ( xc yc ) (2.7) For an ideal case, the image coordinates calculated by the perspective projection would exactly corre- spond to the pixels in the image. This is not the case since the lens distortion causes the ideal projected positions be modified. This lens distortion can be modeled in the image plane alone without the need for 3D information. [12] There are a number of different ways of modeling the lens distortion, each with its strengths and weak- nesses. The one used in this thesis is the polynomial model that uses five parameters, assuming the focal length of the lens is already known. The equations that transform the distorted image plane coordinates to undistorted image plane coordinates are described in equation 2.8 and 2.9 9 Chapter 2. Technical background O O=(0,0,0) X Y Z Figure 2.5: Camera coordinate system u = ũ+ ũ(K1r 2 +K2r 4 +K3r 6) + 2P1ũṽ + P2(r 2 + 2ũ2) (2.8) u = ṽ + ṽ(K1r 2 +K2r 4 +K3r 6) + P2(r 2 + 2ṽ2 + P2ũṽ) (2.9) Where ṽ and ṽ are the distorted image plane coordinates, and r = √ ũ+ ṽ. The equations cannot be ana- lytically inverted and numerical methods have to be used to calculate the distorted image plane coordinates from the undistorted image plane coordinates. In addition to this, the point in the image plane coordinate system has to be transformed into the image coordinate system, wich is a pixelated system. This is done by the use of the following expression: ( r c ) = ( ṽ Sy + Cy ũ Sx + Cx ) (2.10) Where Sx and Sy are scaling factors that represent horizontal and vertical distance between the pixels on the sensor. Cx and Cy represent the principal point in the image. Notation follows the style in [2]. There are hence ten (f, k1, k2, k3, P1, P2, Sx, Sy, Cx, Cy) parameters needed to describe the complete projection from 3D to 2D image coordinates. These parameters can be determined by numerically solving the previous equations using known 3D points and their corresponding 2d image points. [12] The calibration in HALCON uses a specific calibration plate with known properties, such as size, shape and color. A number of images of the calibration plate is collected, and the calibration plate is found in the image using normal image analysis techniques such as thresholding and connected regions. An image of a typical calibration plate can be seen in figure 2.6. 10 2.3. Image analysis techniques Figure 2.6: Calibration plate used for camera calibration in HALCON By collecting images of the calibration plate in many different positions (see figure 2.7) and orientations within the frame, internal functions in HALCON optimizes the camera parameters to describe the projection as well as possible. These images provide the program with a large number of points in 3D space and corresponding points in the 2D image. Figure 2.7: Calibration plate seen in different perspectives 2.3.7 3D pose from a 2D image using shape-based matching Determining the 3D pose of a known object in a 2D image can be done using one of multiple methods. Here, a method that uses a large number of virtual projections of the 3D object into the image plane is used. The method is based on the assumption that a sharp edge in the CAD model will represent a sharp edge in the image of the object. The matching performs the following steps: [1] 1. Create shape model The shape model consists of the previously mentioned projections. The projec- tion uses the parameters derived in the camera calibration to ensure a realistic projection to image coordinates. The equations involved in the projections are presented in section 2.3.6. Determining the pose range in which the object can be found will increase the time required for matching. It is also important to determine the so called ”minimum face angle”, a measurement that determines which 11 Chapter 2. Technical background lines in the model should be projected. Only those lines that appear in the real image as clear edges should be included. 2. Edge filter the image Use a spatial filter to find edges in the image. This edge image will be the base to which the projections are compared. 3. Find best match The search is performed in levels, where each level represents a lower resolution image. In each level, the projection that best matches the edge filtered image is chosen. The next level of the search fine-tunes the match by comparing projections from poses close to the one determined in the previous level. 4. Return the pose The pose with the best match is returned. This pose is then used by other parts of the program, such as the 2D to 3D mapping. 2.3.8 2D to 3D mapping Image plane coordinates can be mapped back to 3D world coordinates under the following conditions: 1. the camera is 3D calibrated 2. the pixels lie on a known object 3. there exist a 3D model of the object 4. the objects pose in 3D space is known If all of these parameters are known, the mapping can be performed by use of ray-tracing. Since the camera is already calibrated it is possible to track the line of sight from the origin of the camera coordinate system to a pixel in the rectified image. This will be a straight line and can thus be represented by two points. Considering the line as a ray originating from the center of the camera and moving forwards, one can investigate the intersection between the ray and the 3D model. If the 3D model is represented by triangles in 3D space, the intersection between one of those triangles and the line of sight represents a mapping from 2D to 3D, see image 2.8. Figure 2.8: Line of sight from camera to image and the intersection with a triangle from the model. It is important to remember that, in these calculations, the line of sight extends beyond the surface of the 3d model, and will hence intersect the model in more than one place. All intersections that occur beyond the intersection closest to the camera will therefore be false and should not be considered. 12 Chapter 3 Hardware set-up The first step of the project was to decide which non destructive testing method to use. The MT method, currently in use by the manufacturer, sometimes creates false indications and the visibility of the indications might vary dependent on the location of the flaw. With this in mind the PT method was also evaluated. Both MT and PT was performed on the same samples to ensure a reliable comparison. Both methods also requires a trained and certified operator to perform the tests. It was from the start evident that the tests had to be preformed by experts, and Karl-Deutch nordiska was contacted for the penetrant test. The MT test was performed by the operator at the current plant, and images captured by the author and by Lars-Åke at Consat. For explanation of MT and PT see chapter 2 and for a more thorough review of the techniques see [11] and [4]. The magnetic testing at the plant uses a current-flow method of inducing a magnetic field in the part. The setup can be seen in image 3.1. The connecting rod is only magnetized in one direction which means that only cracks in the direction of current can be detected. Figure 3.1: Magnetic particle testing equipment at the plant 3.1 Image acquisition setup The non destructive testing was performed by certified operators. The parts were sent to two companies that performed the tests and the test setup used is not known in detail. The images from MT were acquired by the author and Lars-Åke at Consat. The equipment used for this purpose was an area scan camera and an ultra-violet florescent light. The florescent light was converted to use DC current instead of AC in order to 13 Chapter 3. Hardware set-up get rid of the 50Hz interference pattern induced by the AC. Doing this ensures equal exposure regardless of shutter speed and timing with the AC waveform. See figure 3.2. Mean 1/50 s Mean 1/50 + X s 1/50 s 1/50 + X s (a) Mean 1/50 s Mean 1/50 + X s 1/50 s 1/50 + X s (b) Figure 3.2: Different mean for anything but multiples of 50Hz when sine wave has that frequency The positioning of the light in relation to the test part and the camera is very important in order to get an even illumination without sharp reflections potentially hiding defects. A large diffuse light source placed at a certain angle away from the camera will not produce any sharp highlights on planar surfaces. It is however considerable more difficult to achieve an even, diffuse reflection on a part with complex 3D geometry. The setup used in the acquisition of images from the MT test is shown schematically in figure 3.4 Figure 3.3: Image acquisition setup. This setup has been used for all images using the MT method in this report. It is recommended that the setup be reviewed before final implementation, where the automatic turning of the products has to be taken into account. The hardware used in the image acquisition for MT is listed below. • Camera Allied vision Marlin f-080 C • 16 mm lens • Ultra violet light source • UV filter attached to the lens • Solid colored background • Dark room 14 3.2. Destructive testing • Software for capturing images This equipment was used in combination with miscellaneous clamps and supports to collect the im- ages used. The image acquisition can be seen in figure 3.4. Some of the images were taken in a different environment, but the setup was overall the same. Figure 3.4: Test setup, the black plastic is put over the setup to block ambient light. The camera sensor is, contrary to the human eye, able to detect light in the UV spectrum. This is an un- wanted property in this application and therefore a UV filer was attached to the lens to block direct reflec- tions from the UV lights. The florescent dye on the connecting rod emits light in the visible spectrum, and the indications can therefore be masked by ambient light. Hence the ambient light had to be controlled to not interfere with the image acquisition and thus compromise the quality of the images. This was done by covering the whole setup with a dark, light blocking sheet. 3.2 Destructive testing Destructive tests were used to investigate the real nature of the MT indications. The destructive testing was performed by Jonas Höjer at Volvo Penta. The setup consisted of a MT test bench, a saw and a microscope with a built in camera. The parts were first MT tested to determine where the indications were located. They were then cut perpendicular to the indication and the cut surface inspected under microscope. 15 Chapter 3. Hardware set-up 16 Chapter 4 Evaluation of non destructive testing methods The preliminary tests performed with MT and PT provided useful insight into the strength and weaknesses of the two methods in this application. Test images and comments on the methods will be presented here. The tests themselves are difficult and require a trained operator, which is why this job had to be laid out to a third party. The analysis of the tests were in part difficult due to the lack of detail in the reports provided by the third party companies. 4.1 Magnetic particle testing Tests were performed both by the customer company as well as two third-party companies. All the tests show similar results, although the number of cracks detected varied slightly. The smallest indications that were marked as defects were not detected by any test but the one at the customer company. This might be due to the fact that they have been able to tune their system over a longer period of time, and therefore have more sensitive equipment. It could also be due to different criteria as to what constitutes a defect. The method generated a highly visible indication with relatively high signal to noise ratio as can be seen in figure 4.1 Figure 4.1: MT testing clearly shows the defect. Image courtesy of Karl Deutsch Nordiska 17 Chapter 4. Evaluation of non destructive testing methods 4.2 Dye penetrant testing The tests performed at Karl Deuch Nordiska showed that the surface of the connection rod is too rough to be viable for PT. As indicated in section 2.1.2 dye penetrant testing needs the surface to be smooth enough to allow the dye to be washed off the surface without simultaneously washing it out of the defects. The defects also need to be deep and sharp enough to keep the dye in. Figure 4.2: PT testing does not give clear indications. Image courtesy of Karl Deutsch Nordiska As seen in figure 4.2 the PT did not provide any clear indications of the cracks. Instead the dye bleed over the whole product giving false indications all over. This was also stated in the report that followed the test, [13]. The type of penetrant testing used for these tests was based on a red dye instead of a fluorescent one. The main difference between the two is the sensitivity and visibility of the indication. 4.3 Conclusion Even though the dye penetrant testing method seemed promising at first, the rough surface of the connection rod proved to be a difficult obstacle. The results of the tests clearly show that the method is unusable for the application since the dye gives indications all over. However, it must be said that other investigations might have yielded a different result; it is not known to the author exactly how the tests were performed, only the result of the penetrant test were delivered. It would have been interesting to see what would have happen if the part was cleaned in such a way that there would be no more false indications. Either the crack would have been visible or the dye in the crack would have been washed away with the excess dye. It is possible that this was a part of the test procedure, but it was not communicated through the delivered report. It should be stated that the operator performing the test were professionals, and should be trusted even though more information would have been preferable. For example, the two images supplied (one from MT and one from PT) showed different connecting rods. Ideally, they would have shown the same connecting rod with the same defect visible. Other types of non destructive testing methods have been investigated, but non of them were viable candidates and were therefore not tested further. For example, eddy current testing seemed promising with its ability to detect very fine cracks accuracy and using automated procedures. It is however very difficult to scan the whole connection rod because of the complex shape, and the rough surface could yield false results. Ultrasonic testing was also discarded because of the complex geometry of the part. A very high frequency in a special chamber with multiple access points would have to be used. The final conclusion is that the magnetic particle testing is the best candidate and will be used in the continued work. 18 Chapter 5 The crack detection algorithm The first task of the development of a complete image analysis system for detecting cracks was to develop an algorithm that detects cracks in a single image with reasonable performance. Then, an algorithm that uses 3D techniques was developed to improve the performance. 5.1 Assumptions of flaw properties In order to determine whether an indication is a crack or not it is necessary to define what constitutes a crack or flaw. Based on the knowledge of the operators and the sample collection they provided, a list of relative statements describing a flaw was determined. The statements are vague, but they still give an indications of the general properties a crack can be expected to have. The indications of flaws: • are of high contrast type. • has an elongated shape, eg. are narrow and long. • are brighter than the background. • could occur in any position on the part. • could only occur in a direction that lies within the margins of the MT In addition certain assumptions had to be made to reduce the number of false indications. The indications of flaws are not: • coinciding with an sharp contour of the connection rod. • smaller than a certain size. • covering a larger area. These assumptions lead to the conclusions that the defect is line-shaped with sharp edges and a certain length. The absolute measurements could be defined in SI units, but it would still only be valid for the particular product and would have to be adjusted for every new part number. It can also be seen from the lists, that cracks coinciding with sharp contours in the geometry of the part could be misclassified. This problem will be further discussed later. 5.2 Shape based crack detection algorithm First, a shape based crack detection algorithm was developed. It uses simple image analysis operators to determine whether there is a crack in the material. The shape based crack detection algorithm follows the conceptual flowchart in figure 5.1. A more de- tailed explanation of the algorithm will be presented in the following sections. 19 Chapter 5. The crack detection algorithm Background correction Canny edge detector Dilate and create regions of interest Thresholding within each region of intrest Feature extraction Classifier List of detected cracks Source image Figure 5.1: Flowchart of the shape based crack detection algorithm. 20 5.2. Shape based crack detection algorithm 5.2.1 Pre-processing the image First a number of pre-processing steps had to be taken to minimize the number of false indications when using the edge detection based segmentation. This included correction of the background. The background correction was performed using the gray level morphological operator open. Gray level opening differs slightly from the binary version described in the technical background. For further information regarding gray level morphology please see [5]. The background of an image can be neutralized using the following operators: C = A− (A ◦B) (5.1) Where C is the resulting image, A is the original image, B is the structuring element and ◦ is the opening operator. The gray level opening procedure removes all image components smaller than the structuring element. Smoothing and subtracting the resulting image from the original image results in an image with a uniform background. The size of the structuring element should be chosen so that the width of the widest crack can fit inside the structuring element in all rotational directions. In other words, the structuring element should be a circle with a diameter slightly larger than the width of the widest crack that should be detected. This step removes a great deal of unnecessary information in the image. 5.2.2 Image segmentation Image segmentation was performed in four steps, not including the above mentioned pre-processing. First a Canny edge filter was applied. The thresholds of the filter was experimentally determined by trial and error. This resulted in a binary image with thin lines where all strong edges in the image were located. It was not possible to distinguish between a crack and a non crack from that image, even though all cracks were marked. The binary operator dilate was then used to expand the regions where the edge filter had marked an edge. This created a number of Regions Of Interest (ROI) in which all possible cracks were located. These locations were mapped back to the original (but pre-processed) gray scale image. Thirdly, to determine if the ROI contained a crack, each of them were thresholded with a threshold that was relative to the dynamic range in that specific ROI. The relation to the dynamic range was experimentally determined. This resulted in all the ROI being binary again, and lastly a blob analysis were performed on them. The result of the blob analysis was used to extract features for each blob. 5.2.3 Feature extraction Due to the fact that there are constraints on the direction in which the cracks can occur, the direction was used as a feature, other features were chosen to be rotation- and translation-invariant to accommodate coming improvements in the non destructive testing methodology. The features do not have to be scale invariant since the small differences in distance from camera can be covered in the robustness, or simply be deemed negligible. The crack detection algorithm uses a small number of simple shape based features extracted from the binary images produced in the segmentation. The features used were: • Eccentricity • Absolute length of major moment • Area • Convexity • Angle of major axis. The main property of a crack is the eccentricity, the ratio between the length of the two main moments of the object. 21 Chapter 5. The crack detection algorithm 5.2.4 Classification The crack detection algorithm uses a simple box-classifier. Each of the features listed above was given a minimum as well as a maximum threshold. The threshold values were experimentally determined by trail and error. 5.2.5 Parameter robustness A disadvantage of the algorithm is that it is sensitive to parameter adjustment, specifically to adjustments of the threshold values for the canny edge detector. It is also very important that the structuring element in the background estimation has a wider diameter than the actual width of the cracks themselves, otherwise they will be removed along with the background. All parameters that are given a number in pixels should be adapted to be a percentage of the length of the connection rod. Doing this ensures a robustness to changes in resolution or distance to the object. There should be very little need to adjust these parameters once the system (in the future) is installed at a factory; it is however important to ensure that it is easy to adjust the parameters to fit a different product of similar type. It should for example be easy to adjust the algorithm to check for cracks in a larger connection rod. 5.3 3D projection based crack detection algorithm An significant part of the problem of detecting the indications of cracks using the shape based algorithm presented in the previous section was the large number of false positives. Once an algorithm was tuned to detect all flaws, the number of false positives increased dramatically. It was noted during the collection of sample images, that the visual indications that induced false positives changed appearance depending on which angle they were photographed from, whereas the real indications were independent of the angle. The main effort in writing the 3D projection based algorithm was thus spent on determining the 3D position of the indications within a sequence of images, each viewing the connection rod from different angles. These positions was then mapped to a CAD model of the connecting rod, and the result analyzed. The result shows which of the indications stayed visible from all directions and which did not. Each connecting rod were therefore photographed in a number of different angles, these images created a set of images. Please note that the sequence described by 5.3.1 to 5.3.4 were performed for each individual image of the set. The result of applying the sequence to all the images in the set is a feature vector that consists of 3D positions for each crack. That feature vector is then passed to the classifier described in 5.3.6 for crack indication classification. 5.3.1 Overview of the 3D projection based algorithm The crack detection was only slightly changed in comparison to the previously discussed shape based crack detection algorithm. It was, however, simplified and the classifier was removed. The algorithm is divided into three main parts: crack detection, 3D matching and 3D mapping. In ad- dition a simple classifier based on the euclidean distance was used. A conceptual flowchart of the 3D projection based crack detection algorithm can be seen in image 5.2. 5.3.2 Crack detection The 3D based algorithm uses the same type of pre-processing with background correction as the previously presented shape based crack detection algorithm. Here, however, background correction is complemented by a filter that further enhanced the contrast in the image without saturating any channel. The core function of the crack detector was however changed from an edge detector to a line detector. The function can simply be described as a canny edge detector followed by a function that selects all double edges and traces the local maxima/minima between these lines. The detection was then tuned to be slightly over-sensitive to ensure that no real cracks stayed undetected. This meant that many false indications got detected, a problem that was dealt with in the 3D matching part of the algorithm, in combination with the classifier. 22 5.3. 3D projection based crack detection algorithm Source image 3D object modelBackground correction 3D matchingShape based crack detection 3D mapping Classifier List of detected cracks Perform once for each image in a set Perform once on whole set Figure 5.2: Flowchart of the 3D projection based crack detection algorithm. 23 Chapter 5. The crack detection algorithm 5.3.3 3D matching In order to determine where on the connecting rod the indication is located, it is necessary to determine the 3D pose of the part in the image. This was done via a 3D matching method that takes a 3D CAD model and searches for it in the image. Based on the size and rotation of the part in the image it is possible to determine the pose of the part. This pose is then used for mapping the crack indications from the 2D image to the 3D CAD model. 5.3.4 3D mapping A custom algorithm was written to map 2D image coordinates to a 3D model with a pre-defined pose. It takes the vertices’s of a triangle and two points on a line as input arguments and returns the boolean ”true” if there is an intersection between the triangle and the line that is defined by the two points and ”false” otherwise. The function first checks whether the line intersects the plane that is spanned by the three points. If it does, it checks whether the intersection point lies within the triangle. To use the function within the algorithm, first a line of sight is calculated from the camera to the image plane for each pixel that represents an indication. The previously calculated 3D pose is used to determine the 3D locations of all the vertices’s of the triangles that represent the 3D object. Once the triangles position as well as the line of sight is known, the line of sight for the current pixel is checked against all triangles in the 3D object. This procedure will produce from one to many mappings for each pixel, as long as it is in fact located on the connecting rod. The mapping resulting in the position closest to the camera is the correct one and is kept. [9] This produces 3D positions for all crack indications in a single image. This 3D position of each crack is that cracks individual feature vector. 5.3.5 Processing the feature vectors The feature vectors(containing only 3D positions) produced by applying the previously described sequence to all images in a set, was processed to describe the euclidean distance between all cracks in that set. This means that the feature vector for a single crack indication after processing contains information about the euclidean distance from that specific indication to all other indications in the set of images. 5.3.6 Classification The classifier takes the processed feature set , and classifies each indication into one of two classes: true indication, meaning that it is a crack, or false indication, meaning that there is no crack. The classifier developed for the 3D projection based crack detection algorithm is, as the one for the shape based crack detection algorithm, a very simple box classifier with thresholds manually determined by trial and error. The classifier uses only two thresholds; the maximum distance between two indications to consider them to represent the same indication, and the minimum number of indications considered to represent the same indication to consider that indication to be a real crack. Example: • The set of images contains a total of 100 crack indications. This means that each crack indication has a feature vector(after processing) consisting of 99 distances. • Threshold 1 = 2mm • Threshold 2 = 5 • The feature vector of indication number 1 in image number 1 has 4 distances smaller than 2mm • Since threshold 2 = 5 > 4 determines that indication number 1 in image number 1 is classified as a false indication. 24 5.3. 3D projection based crack detection algorithm 5.3.7 Implementation of the algorithms The shape based crack detention algorithm was developed in MATLAB while the final development and im- plementation of the 3D projection based algorithm was performed in HALCON. The switch from MATLAB was motivated by the simplified conversion to c# code from HALCON, and also by the more comprehensive library of built-in algorithms. HALCON allows the user to apply a more high level approach to the problem while maintaining and improving performance. A great advantage and the first and main reason for the switch was that HALCON has built in 3D to 2D matching algorithms, something needed to exclude false indications caused by the geometry of the connection rod. HALCON can be described as a comprehensive machine vision library with a built in development tool. [10] The algorithms were run on a PC with a dual core i5 processor and 4 GB of ram. The software used was MATLAB version 7.13.0.564 (R2011b), HALCON verion 10 and Windows 7 64 bit. 5.3.8 Comments on image analysis approaches for crack detection A number of experiments were carried out before the development of the above presented algorithms. Some comments on those experiments, as well as from the above algorithms, are presented here. Methods based on absolute pixel intensities were quickly discarded as they were to sensitive too changes in image exposure. It is in an industrial application, very difficult to maintain a constant exposure. The light source changes properties over time and ambient light changes characteristic etc. all which effect the exposure. Global thresholding was also discarded based on the fact that direct reflections from the light source would create highlights in the image brighter than the indication itself. Segmentation based on edge detection showed great promise, as did methods based on the standard deviation. These methods are robust against luminosity changes as well as rotation. They both handled highlights from the light source well as the transition from dark to light areas of the image was slower at those areas than at the defects. The methods are however sensitive to scale, and it is relatively difficult to tune the parameters. A drawback of the method was also that the geometry of the connecting rod had some sharp corners that were marked as cracks. This was the reason for using 3D techniques. An important notice from the shape based crack detection algorithm, that uses edge detection and binary operators, was that all real cracks had two edges close to each other. This was used in the 3D projection based algorithm to simplify the crack detection part of the algorithm by use of a line detector instead of an edge detector. The 3D mapping part of the algorithm uses a brute force approach that require a lot of processing power. This could be done in a more optimized way, which would improve the speed performance of the whole algorithm significantly. This methods of classification presented here had relatively good performance, but could most likely be improved by instead using, for example, a neural net classifier. The classifier could also include more features than just the distance between cracks. 25 Chapter 5. The crack detection algorithm 26 Chapter 6 Results and Conclusions 6.1 Results from present work This section will present the results of the 3D projection based crack detection algorithm. The shape based crack detection algorithm has very similar results as the crack detection in the 3D based one, and are therefore not presented here. Since the algorithm is divided into three larger parts and one smaller (the classifier), that is how the re- sults will be presented. A complete evaluation of the program will not be possible since there was problems with the 3D matching under ultra violet light. All major parts of the program have, however, been tested as separate systems, and the results will be presented here. 6.1.1 Crack detection using 3D projection based image analysis The developed crack detection algorithm performed relatively well. It can, in a robust way, detect crack indications, but with a number of false responses as well. Figure 6.3 shows the crack detection program applied on a sequence of nine images taken at increasing viewing angles. It is clear that many of the false indications disappear/appear depending on the viewing angle while the true indication stays detected in many consecutive images. It can also be seen that a number of false indications stay detected in the sequence, most of which have to do with the geometry of the connection rod, and could be eliminated based on the position when mapped to the 3D model. 27 Chapter 6. Results and Conclusions (a) (b) (c) Figure 6.1: Crack detection, image 1-3 in series 28 6.1. Results from present work (a) (b) (c) Figure 6.2: Crack detection, image 4-6 in series 29 Chapter 6. Results and Conclusions (a) (b) (c) Figure 6.3: Crack detection, image 7-9 in series 30 6.1. Results from present work In addition to the sequence of images on the previous page, the algorithm was tested on a number of different images. If the images were taken in approximately the same setup, with the same camera, lens and focusing distance, the results were very similar, even for other types of crack indications. A number of parts were tested with penetrant testing at KDN, and the results of that test have already been presented. They did, however, also perform MT on the parts in addition to penetrant testing. The images from that test had significantly different properties than the ones taken by the author. It is therefore interesting to present the results of the crack detection on those images. As can be seen in image 6.4 there is a much smaller amount of false indications, and the real crack gets marked more accurately with less sensitivity to parameter changes. The images differ from the ones taken in-house in a number of ways, such as the luminance, the resolution and the noise level. This leads to that the images are more suited for the task of crack detection. (a) Original image (b) Crack detection with higher sensitivity (c) Crack detection with lower sensitivity Figure 6.4: Crack detection on image provided by KDN The same connecting rod with the same crack indication has also been tested using destructive testing. The result of that test can be seen in section 6.1.4, specifically in figure 6.9. 31 Chapter 6. Results and Conclusions 6.1.2 3D matching As described in the theory section, 2.3.7, the 3D matching is based on an edge filtered version of the image. The UV light in combination with the florescent dye introduces extra edges into the image. The crack detection algorithm relies on this to detect flaws, but it corrupts the result of the matching algorithm. However, the results from matching in daylight can be shown to work, as demonstrated in figure 6.5. Figure 6.5: 3D matching in daylight, camera poorly calibrated. The remaining issues with pose estimation is due to a poorly calibrated camera. The problems in 3D matching can be handled in a number of ways, a more in-depth discussion on this can be found in chapter 6.2. 6.1.3 3D mapping It is not possible to directly display the result of the algorithm that maps image coordinates to the 3D model in a printed report. It is, however, possible to visualize the process by transforming all mapped positions to a reference pose in 3D space. From the reference pose a projection back to the image plane is computed and the result of this is presented in figure 6.6 and 6.7. The 3D model of the connecting rod is also projected in the reference position, and it is possible to see that the cracks are mapped accurately. The small deviations are due to the fact that the 3D pose of the connecting rod was poor as a result of poor 3D matching. This is not of great importance for the purposes of this thesis since it aims to evaluate the method, not to create a working industrial application. 32 6.1. Results from present work (a) Indications found using the crack detection part of the program (b) Indications mapped to the 3D model and projected in a reference position Figure 6.6: Crack indication and mapping first image in series 33 Chapter 6. Results and Conclusions (a) Indications found using the crack detection part of the program (b) Indications mapped to the 3D model and projected in a reference position Figure 6.7: Crack indication and mapping second image in series 34 6.1. Results from present work It is clear that the mapping from 2D to 3D works well. The problems that can be seen are due to poor matching results, something that can be fixed in a final implementation of the system. It is also worth noticing that the lines in 6.6(a) and 6.7(a) have been closed (the endpoint connected by a straight line) before mapping to 3D. This is not a desired behavior, and should be fixed before a real implementation. 35 Chapter 6. Results and Conclusions 6.1.4 Destructive testing The destructive tests show the real flaws that magnetic particle inspection indicates. It is clear that there are real flaws at location of indications. The severity of the flaws are on the other had difficult to determine as it is dependent on a number of factors. (a) Magnetic particle testing (b) Cut guidelines, 1A (c) Cut surface 1A viewed under microscope, notice the scale Figure 6.8: Destructive testing, sample 1 In figure 6.8(a) the indication from magnetic particle inspection is seen. It is located to the right of the symmetry axis of the connecting rod, as indicated by the marking ”1A” in figure 6.8(b) that also indicates the cutting guide line. Figure 6.8(c) show the result of the destructive testing of that indication. It is clear that there is, in fact, a material defect at that location. The scale is very important in this figure, the defect is approximately 50µm deep and does not break the surface at the location of the cut. 36 6.1. Results from present work (a) Magnetic particle testing (b) Cut guidelines, 2A (c) Cut surface 2A viewed under microscope, notice the scale Figure 6.9: Destructive testing, sample 2 Figure 6.9 show the same type of test results, but the size of the flaw is considerably larger. It is a surface breaking defect that extends down to approximately 200µm. It should be noted that the connecting rod in 6.9 is the same one (with the same flaw visible) as the one that in figure 6.4 had been passes through the crack detection system. 37 Chapter 6. Results and Conclusions 6.2 Discussion The results show that the chosen method, magnetic particle testing in combination with an image analysis tool, shows great promise for the detection of cracks in forged metal parts. An important aspect of the de- veloped algorithm is that i can be considered modular, it consists of four independent parts; crack detection, 3D matching, 3D mapping and a classifier. This modularity means that any of the four parts could be easily changed without the need to modify any other part, with the exception of the classifier that might have to be augmented to take new inputs. This is a very important property since it means that the actual crack detection can be heavily modified or even completely changed and it would still work well with the rest of the system. The interfaces between the parts are simple, such as that the classifier takes a set of regions with corre- sponding information and determines whether the region represents a crack. There are still some problems with the program, such as that the 3D matching does not provide any robust 3D pose in images acquired under ultra violet light. This is because the uv light in combination with the MT method introduces edges in the image not present in the 3D cad model. It is this property that is sought after to detect cracks, but it throws off the 3D matching algorithm. One way of overcoming this problem is to add a new, known, element in a fix position in relation to the connecting rod. This could be done by adding a marker to the device that rotates the connecting rod between images in a sequence. It is yet to be determined if the developed crack detection algorithm is sufficiently robust for an industrial application. It is however, as stated previously, easy to change one part of the program if the need would arise. It should also be noted that the algorithm works well once tuned to the test setup. There are some key benefits of introducing an automated system instead of relying on a human operator. First, there are the purely technical advantages such as a more consistent quality control and the possibility to run the system 24h/day without breaks. Then, there are also benefits for the operator who is excused from working in the harsh environment of a dark room with strong UV light. The job also demands a high level of concentration, something that gets very tiering after a even a short period of time. The workers’ conditions are, in short, uncomfortable. It is possible that an automated system still would need a human operator to make the final decision if there are uncertainties, and a human would most definitely need to maintain the system and tune the process. This would however be possible to do behind the comfort of a computer without the need for a demanding work environment. The initial hypothesis that false indications seemed to change appearance depending on the angle of view holds up in experiments. As presented in the results, the sequence of images taken at gradually in- creasing angles show that the false indications appear and disappear seemingly randomly between images. The true indication on the other hand stays marked once it is marked the first time. While it is true that some false indications stay marked over a sequence of images, they are also the false indications that are related to the complex geometry of the part. They can therefore be discarded based on the 3D location. This validates that the approach of mapping indications from the images to a 3D model is a good method to reduce the number of false indications. The mapping of crack indications from the image plane to the 3D model is slow. A brute force solution has been applied as a proof of concept, and the computational time is directly related to the number and size of the indications detected. A more sophisticated solution could significantly reduce the time required of the algorithm. The same is true for the part that determines if the crack is in the same position during the sequence of images. This algorithm simply compares 3D positions using the euclidean distance between points on the cracks. This can be done significantly more elegantly and effectively. These optimizations have not been addressed in this thesis since it is not within the scope to write optimal code, but to provide an answer to whether it is possible to automatically detect cracks. Optimization and improvements are recommended, and necessary, in future work. It is interesting for that the images produced at Karl Deuch Nordiska proved to be much better suited for crack detection. This means that there is a lot that can be done with the image acquisition system to improve performance. A better light source and a better camera would probably produce cleaner images. Using the cleaner source images in combination with the software developed should, theoretically, be able to increase the performance and efficiency of the system significantly. The destructive testing show the real properties of the cracks. Both flaws shown in images 6.8 and 6.9 can be detected by the computer vision system developed. The indication in figure 6.9 is very clear and 38 6.3. Future work relatively easy to detect , as can be seen in figure 6.4, while the indication in figure 6.8 demands more careful tuning of the system. The destructive testing show that the flaw that created the smaller indication is very small, in the region of 50µ. If the defect would have been much smaller, the system would not be able to detect it. This is a drawback of the system, and to determine whether the defect is dangerous or not, further investigations into the production methods and load cases has to be studied. It is important to have knowledge of how sensitive the designed system is in order to determine whether it is sufficient to replace a human operator. The operator can still detect indications smaller than the system can detect without a large amount of false positives. If the sensitivity is tuned down slightly, so that the smallest indications are like the one showed in figure 6.8(a), the number of false positives decrease significantly. From destructive testing the question arises whether it is necessary to detect indications smaller than those, since they in general are very small. This question is not answered within this thesis and is left for future work. It is the authors opinion that the method shows such promise that further development is recommended. The method seems, after the investigations performed in this thesis, to be able to detect cracks with sufficient accuracy for an industrial application. 6.3 Future work There is still a lot of work needed to be done before a system based on the methods presented in this thesis can be implemented in an industrial application. The program has to be optimized for faster execution, and the image acquisition needs to be revised. The setup used by Karl Deutch Nordiska gave much clearer indications than those produced in the test setup. The problems with the 3D matching have to be addressed, possibly with one of the solutions discussed previously. Mechanical design and control logic has not been a part of the project, but needs careful attention if the system is to be developed further. There are a number of possible approaches to crack detection that, in combination with the rest of the program developed here, can provide good results. One of them is the use of a neural network as the previous project investigated at Consat. That project concluded that there would be severe over-detection if the process was tuned to detect all flaws. The 3D methods presented here could most likely improve the result significantly. This has yet to be tested in reality however. The destructive testing gave some questions that remained unanswered. What size of cracks are danger- ous, how sensitive does the system really have to be. This investigation into material science and manufac- turing techniques should be done in order to proceed. 39 Chapter 6. Results and Conclusions 40 References [1] “Halcon manual,” Website, 2012, as available 2012-05-18, http://download.mvtec.com/halcon-11.0- hdevelop-users-guide.pdf. [2] “Halcon user guide,” Website, 2012, as available 2012-05-18, http://download.mvtec.com/halcon- 11.0-solution-guide-iii-c-3d-vision.pdf. [3] J. Canny, “A computational approach to edge detection,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. PAMI-8, no. 6, pp. 679 –698, nov. 1986. [4] V. Deutsch, M. Vogt, M. Platte, V. Schuster, and W. A. K. Deutsch, The Magnetic Particle Crack Detection, 1st ed. Wuppertal: Castell Publication Inc., 2002. [5] R. Gonzalez and R. Woods, Digital Image Processing Using MATLAB, 3rd ed. New Jersey: Prentice Hall, 2008. [6] R. Gonzalez, R. Woods, and S. Eddins, Digital Image Processing Using MATLAB, 2nd ed. Knoxville, TN: Gatesmark Publishing, 2009. [7] L. Johansson, “P11-4686rp01,” Consat, Tech. Rep., 2011. [8] D. Lay, Linear Algebra and Its Applications, 4th ed. New Jersey: Pearson Education (US), 2008. [9] T. Moller and B. Trumbore, “Fast minimum storage ray/triangle intersection.” [10] MVTec, “Mvtec halcon,” Website, 2012, as available 2012-03-28, http://www.mvtec.com/halcon/. [11] N. resource center, “Ndt resource center,” Website, 2012, as available 2012-03-19, http://www.ndt- ed.org/EducationResources/educationresource.htm. [12] R. Tsai, “A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses,” Robotics and Automation, IEEE Journal of, vol. 3, no. 4, pp. 323 –344, august 1987. [13] Unknown, “Bericht 408 kdn 4 pleuel 201203sy,” Carl Deutch Nordiska, Tech. Rep., 2012. [14] P. Wang and H. Huang, “Comparison analysis on present image-based crack detection methods in concrete structures,” in Image and Signal Processing (CISP), 2010 3rd International Congress on, vol. 5, oct. 2010, pp. 2530 –2533. 41 Abstract Acknowledgments Contents Introduction Purpose Background Previous work Limitations Technical background Nondestructive testing methods Magnetic particle testing Dye penetrant testing Destructive testing Image analysis techniques Noise reduction using spatial filtering Morphological image analysis Canny edge detector Segmentation and feature extraction Line detection 3D calibration of a 2D camera 3D pose from a 2D image using shape-based matching 2D to 3D mapping Hardware set-up Image acquisition setup Destructive testing Evaluation of non destructive testing methods Magnetic particle testing Dye penetrant testing Conclusion The crack detection algorithm Assumptions of flaw properties Shape based crack detection algorithm Pre-processing the image Image segmentation Feature extraction Classification Parameter robustness 3D projection based crack detection algorithm Overview of the 3D projection based algorithm Crack detection 3D matching 3D mapping Processing the feature vectors Classification Implementation of the algorithms Comments on image analysis approaches for crack detection Results and Conclusions Results from present work Crack detection using 3D projection based image analysis 3D matching 3D mapping Destructive testing Discussion Future work References