The Case Study Balluff series are articles written by key partners of our company, integrators, specialists implementing dedicated solutions for manufacturing companies using Balluff tools and solutions.
It happens that a fully autonomous industrial robot does not find its place in the factory. It is hard to define in advance the paths it would take due to the fact that the organisation of the shop floor changes or there are no set corridors. Sometimes robots are delegated to previously undefined tasks (for example, delivering a missing component to a specific location). Then comes the need for the robot to follow an operator who understands the task well and is able to chart a path.
Hi, follow me
We often encounter situations in which the production hall is rearranged and the robot cannot follow the same paths permanently, because they do not exist. That is why we took it upon ourselves to create the right system – a robot that tracks the operator. For this purpose, we used a stereo camera with two lenses. Based on the image obtained from them, we can estimate the distance between the operator and the camera, which makes it easier to track the operator.
We also used deep machine learning in the system, networks such as Faster R-CNN for human detection, and tracking algorithms – classic computer vision algorithms and one-shot learning. The whole is integrated with the AWS cloud responsible for speech interpretation, controlling the robot with voice, and biometrics and recognition of the operator on the basis of their photo. This way, we make sure that no unauthorised person takes control of the robot.
In order for the robot to start following the operator, the operator must approach and introduce themselves. We use DeepLens for this, a device created by Amazon that can stream audio and video to the AWS cloud. The audio from this device is analysed in the cloud. When the phrase “Hi, follow me” is detected, the robot verifies that the operator standing in front of the camera has been entered into the database of people who have permission to control it. The use of Amazon Rekognition ensures that only authorised people will control the robot. After the first verification, using the Fast R-CNN network, we detect the operator, and then thanks to proprietary algorithms, the robot starts to follow him, acting for example as a porter. The unit turns and accelerates as the operator does, so it will not bump into any obstacle. This makes it easier to operate the production line and supply it with the necessary components that are in short supply.
Communication with the hardware
For self-driving strollers, it is important to equip them with additional elements that collect external stimuli. If a cart is tasked with moving an item from point “A” to multiple points, it must receive information about them. To do this, we used technology from PGS Software that uses ultrasonic waves. It allows you to read information from the items that are on the cart and deliver them to the right places. In practice, it looks like the operator “leads” the truck behind him completing the order, and the robot, having information about what and where it carries, continues to work independently.
In order for our system to communicate with network solutions, we use the TCP/IP communication protocol that allows us to transfer information to and from the cart. We are able to fully satisfy all application requirements using one network, one topology. Thanks to this, we can easily combine all the elements in such a way that they fulfil the assumed order.
Importantly, we can equip the cart with different components and further senses of self-awareness of the application and condition of the cart. By using sensors to monitor the operating status of the machine, we are able to inform operators of any errors that may have occurred along the way.
A standard RFID system consists of a processor, a head and a data carrier. The RFID processor is responsible for communication with the master control system and transmits control information to the head. The head (or antenna), in turn, is responsible for the physical implementation of the identification of reading or writing data. The data carrier is placed on the object being identified. It contains an ID number and may also contain a user space, a data space that can be freely written to and freely read from.
In the aforementioned project, we used the compact and universal BIS V processor. Up to four heads can be connected to it. This particular model has Ethernet TCP/IP based protocol communication and an IO-Link port which is an IO-Link master. Any IO-Link device can be connected to it.
We can connect any frequency standard to any of the four ports. Depending on the application requirements, we can use the UHF (BIS VU), HF (BIS V
M) and LF (BIS C and BIS VL) technologies. Each of these three technologies is distinguished by certain characteristics to meet different application requirements. UHF technology allows multiple media to be read at one time, from a long distance. Depending on the settings, size and type of antenna, we are talking about distances of up to six metres. Unfortunately, this technology is sensitive to environmental conditions – the presence of metal structures or water, adversely affects the operation of the entire system.
With HF technology, only one media is served at a time by a single head. The working distances are also much smaller – we are talking about a few, to several dozen centimetres. Nevertheless, this technology allows us to handle a lot of data in a short time.
We have many heads and carriers to choose from, including for special tasks and conditions. LF technology is slower than HF, and handles less data, but is much cheaper and metal-proof. If you choose to only read the ID number, and the application is done stationary, not on the move, perhaps such a system will suffice.
In smart warehouses, it is important that information reaches the systems that manage them in near real-time. The use of a forklift that follows the operator and picks up specific packages from the warehouse shelves increases worker efficiency. The cart is always at hand, there is no need to move it, and thanks to RFID technology we are able to analyse its current status in real time. This allows the warehouse management system to know what to expect in the near future, what packages will drive up to the next stage of shipment handling.
Using this solution, an employee who knows the list of products can very quickly complete the order by simply placing the items on the trolley. The robot recognises them and it traces its own path along which it should continue.
If the order is not complete, or if there is a mistake and the wrong item is added to the order, the technology will apply a self-checking mechanism and indicate where the error occurred. This relieves the burden of performing post-checks on order fulfilment.
On a modular production line, a changeover worker can load a die or tools onto a trolley. The robot is able to transport them to their destination and use the production slots where the upgrade is performed. In this case, the important thing is that an operator tracking trolley is perfect for an environment where obstacles need to be avoided and routes need to be modified.
Thanks to the companionship of the robot, the operator who manually refits the line is able to take many more components from the warehouse, saving time on walking to pick up the parts needed to refit the next slot. In this way, line changeover becomes a less time-consuming task and allows for quicker adaptation of the line to current production plans.
Adam Gurgul | Tomasz Dawid | Rafał Siwek