Deep learning-based inspection

AT ITS Notre-Dame-de-Bondeville, France, pharmaceutical company Aspen weighs and mixes drug formula components and fills them into ampoules in a subsequent process. Once the ampoules have been filled and sealed, they are transported to an inspection and packaging area. The ampoules and their contents are then checked for defects. Previously, this process involved employees picking up each ampoule individually and checking it from all sides to see if the ampoule and the fill level were OK and if there were any foreign objects in the liquid.

The big challenge here is that the contents of the ampoules may contain bubbles which are very difficult to distinguish from foreign objects. The particles floating in the vials are not always easy to detect, even for inspectors. They may be located on the side, sink to the bottom, or be unclear due to the viscosity of the liquid. It is therefore understandable that manual inspection is very time-consuming and costly.

Automating the process, 12 cameras that comply with industry standard GigE Vision are used in a new system. Good lighting is also important to make the particles clearly visible. Machine Vision is performed on an industrial PC. On the software side, the machine vision solution is based on MVTec HALCON, the standard software for machine vision with over 2100 operators for almost all image processing tasks including deep learning. Due to the extremely difficult conditions in Aspen’s application, deep learning had clear advantages over classic rule-based methods. With classic machine vision methods, it was not possible to find a set of rules that was robust and flexible enough to detect the defects.

The 12 cameras are positioned so that they capture up to 14 images of each ampoule from different stations. The large number of images is helpful for deep learning-based inspection, as there are images in which the particles are not visible but can be seen when viewed from a different angle. It should also be noted that a particle is only identified as such if it is found in a certain number of images. This successfully reduces the number of false positives.

Once the images have been captured, they are transmitted to MVTec HALCON. There, various machine vision methods are used to perform the checks. Aspen uses the deep learning-based semantic segmentation included in HALCON to detect foreign matters. In addition to inspecting the liquid for particles, other inspection tasks are also performed in parallel with MVTec HALCON. These include so-called cosmetic defects.

The system checks whether the fill level is correct, whether the color is appropriate, and whether the closure complies with specifications. Classic machine vision methods such as matching and blob analysis are used for these tasks. Classic methods have the advantage that they deliver very robust results for suitable applications and enable very short processing times. At the end of the inspection, a clear decision is made as to whether the ampoule in question is OK or NOK.

“The task was very complex – certainly one of the most challenging we have ever faced. This was particularly true when it came to preparing the images for training. We at MVTec were called in to assist with conceptual preparation, process implementation, and documentation,” explains Patrick Ratzinger, project manager at MVTec.

Deep learning is only able to make robust decisions if the images are prepared appropriately. MVTec's support consisted of sorting, post-processing, and recompiling the data previously labelled by Aspen, and then training it multiple times. The same data set was trained multiple times in order to compare the different results. MVTec thus created a neural network that Aspen can use to perform inspections on its own equipment. This network is trained to segment the particles from the background and thus reliably detect them. For the training, test ampoules were manipulated to simulate possible defects that could occur.

Images of these test ampoules were captured and labeled using the Deep Learning Tool. The Deep Learning Tool is an MVTec software product that allows images to be labeled for deep learning applications. The labeled data was combined with good images, meaning ampoules without any defects, to create a dataset used for training the deep learning networks. The training runs, which are carried out multiple times, are then evaluated to see if they work effectively in real-world use.

“Through our consulting and support work, we have built up extensive knowledge about the data and its use. For example, how best to label defects, what the ideal composition of datasets looks like, and how best to interpret the results. We have passed on this knowledge to Aspen as part of our consulting services,” explains Ratzinger.

www.mvtec.com