3D vision helps precise parts palletising

The integrated 3D vision software MVTec HALCON ensures precise gripping of those parts. Reliable part recognition allows the entire workflow to be optimised and automated, which speeds up palletising, saves costs, and relieves employees.

Fully automated part picking by robots is a major technical challenge. A particular difficulty is that the components are not always delivered to the robot in an orderly fashion, but rather unsorted and even overlapping each other. This means that humans still have to intervene frequently. The Spanish company Tekniker has developed an application based on machine vision that enables fully automated bin-picking with two robots. A particular highlight is the 3D vision technology 'Surface-based Matching' which reliably recognises the delivered components, i.e., acts as the eye of the robots and enables precise gripping.

Tekniker, based in Eibar in northern Spain, specialises in the development of technology solutions for automation, industrial robotics, and sensor technology, among others. The company serves a wide range of industries such as aerospace, automotive, or machinery and plant engineering.

One of Tekniker's customers is the CIKAUTXO GROUP, a Spanish manufacturer of vibration-damping and fluid-conducting components for the automotive industry. On behalf of the company, Tekniker's team of experts has developed a machine-vision-controlled robot system for automated palletising components for industrial production. Robots remove components that are lying randomly in a box and then place them in an orderly manner into a container that is specially adapted to the geometric features of the parts. They are then ready for further processing in the production workflow.

Process automation

"Until now, the palletising of parts was carried out purely manually. Our goal was to automate this process end-to-end in order to speed it up, increase productivity, and save costs. In addition, our workers were to be relieved of monotonous, ergonomically unsound tasks so that they could devote themselves to more demanding tasks," explains Kepa Laka, automation and robotics area manager at CIKAUTXO GROUP. To implement process automation, one option was to implement a mechatronic system. However, this approach offers a low level of flexibility and reliability and, on top of that, generates a lot of noise. Therefore, Tekniker decided to use a solution based on machine vision.

The system setup consists of two industrial robots from KUKA, a 3D scanner from Photoneo, a depth camera from Intel, a PC running the machine vision software, and a SIMATIC PLC from Siemens to control the cell. In the process, one of the robots first removes the stamped parts in large quantities from the input bin using a magnetic multifunction gripper and deposits them on two conveyor belts.

The removal is stopped as soon as a certain volume of parts is reached on both belts. This is calculated using depth data provided by the depth camera as well as a customised HALCON algorithm that combines 2D and 3D vision tools.

The machine vision software on the PC also continuously processes the data from the 3D scanner and provides the robots with the position and orientation of the parts best suited for gripping on the conveyor belt.

With this data, the second robot already begins to pick up the parts from the conveyor belts and, also with the help of 3D vision, aligns them so that it can sort them into two target bins in an orderly manner. And, after the first robot stops picking from the input bin, it "helps" the second: it also picks up parts from the conveyor belt, aligns them and sorts them into the target container.

This works at a speed of 8 seconds per part. During this process, the parts must be inserted into guide bars within the target container with the front facing upwards and with a tolerance of 0.5mm. Should the gripped part face downwards, it is first placed in a rotating device to pick it up again in the correct position.

Precise localisation

The particular challenge in this setup is that the robots need to precisely recognise the position and orientation of components in 3D space in order to pick them up and put them down again safely.

"To implement this requirement, we have developed the SMARTPICKING software and integrated it into the overall solution. The software is able to accurately identify objects by analysing 3D data and using the surface model as well as the corresponding geometric information. At the heart of the SMARTPICKING solution is the machine vision software MVTec HALCON. It provides sophisticated algorithms and tools to precisely locate the parts lying chaotically on the conveyor belt and transmit the coordinates of the pick point to the robots," explains Jorge Molina, researcher at smart and autonomous systems department at Tekniker.

Classic 3D-based image processing methods are used; first and foremost, this includes the Surface-based Matching feature integrated in HALCON. The technology uses data from 3D point clouds captured by the 3D scanner. The data is further processed using different filters and preprocessing operators to improve the pose estimation of the parts. This allows for a particularly robust positioning of objects – regardless of the nature of the surface.

Now, the robots can recognise the parts and their exact position and to grip them unerringly. This works for a wide range of different parts as they occur in industrial production processes. Reliable position determination is thus ensured even for deformed or smooth surfaces without distinctive edges, which do not show any significant gray value differences in conventional images.

In addition, for a safe gripping process, the robots must also be able to determine the relative position of the 3D camera in relation to the respective object exactly in their coordinate system. For this purpose, a so-called hand-eye calibration is performed with the help of MVTec HALCON, e.g., with an associated calibration plate. This calibration is essential to establish the relationship between camera, robot, and the parts to be gripped. Only then does the robot "know" exactly where its gripper is in relation to the object to be gripped – i.e., where it must move it in order not to grasp at nothing.

Avoiding collisions

In addition to detecting and determining the position of the parts to be gripped, possible collisions between the robot arms and individual objects (components, containers, etc.) must be avoided during the entire pick-and-place workflow. This is where the 3D Object Processing technology integrated in HALCON comes in. This makes it possible to determine those parts that lie favorably on the conveyor belt and can thus initially be gripped without collision. Based on this initial selection, the software then determines the optimal candidate using various decision criteria such as the height and orientation of the parts or the position of the gripping point and returns the coordinates of its position to the robot. In this way, the robot can safely carry out the next gripping process.

The controlled sequence allows the removal of components in such a way that collisions and thus disruptions to the workflow are reliably prevented. The software also detects the remaining capacity of the target container. For this purpose, operators precisely calculate its volume. A previously defined threshold value can then be used to determine whether further parts can be added to the container.

Increased process speed

"Thanks to the robust machine vision capabilities in MVTec HALCON, we can automate and optimise the robotic palletising process end-to-end. The software offers a large number of sophisticated machine vision tools that can be used to precisely locate the parts to be picked up. This ensures safe and smooth pick-and-place processes. In addition, we can categorically exclude collisions of the gripper arms, thus ensuring the safety and continuity of automated palletising," Jorge Molina sums up.

"As a result of using the system, we are able to increase process speed by up to 100 percent, improve productivity accordingly, and, on top of that, relieve our employees of routine tasks," confirms Laka.

www.mvtec.com