Advanced altitude picking in warehouses for more efficiency, less damage & lower costs

The essence

At IKEA’s distribution center, forklift picking is one of the main daily tasks. Doing this at a higher altitude is difficult because there isn’t a clear view of the exact position. This results in ergonomic discomfort, rack & product damage, and loss in efficiency. To tackle this, we attached a 3D camera to the fork of the reach truck that streams a stereoscopic 3D video to the reach truck driver’s smartglasses.

Play Video about Altitude picking support for Ikea

The challenge

Picking orders at a high altitude

IKEA has many distribution centers all over the world. Forklift picking is one of the main daily tasks with over 200 items to collect each day. Knowing that racks go up to 9 meters high, many of those items are stored at high altitude. However, forklift picking at a higher altitude is difficult as the operator doesn’t have a good view on the exact position of the fork. IKEA saw 3 opportunities to improve their current way of working.

The first one is to improve ergonomics. What if it wasn’t necessary for the order pickers to look up all the time? The second one is to create a safer environment. Sometimes it might be difficult to clearly see pallets at high altitudes, racks & products get damaged, and rack stability can’t be assured. When this happens, the entire aisle must be unloaded to do a safety check. This causes delays and unforeseen labor. And lastly, they want to boost efficiency. IKEA assumes that when orders are picked on ground level, employees can work easier and more efficiently. Therefore, they wanted to investigate whether an innovative solution allowing order pickers to see all racks as if they were on ground level, can boost confidence and efficiency.

The Solution

Because of our experience with real-time stereoscopic 3D video combined with smartglasses (solution for DEME), we knew such a solution could suit IKEA as well. By attaching a 3D camera to the fork of the reach truck, and streaming a 3D video to the smartglasses, we would be able to bring the eyes of the reach truck driver at the same height of the fork – no matter the altitude.

The HoloLens is the best fit for this solution, as it allows for pin sharp stereoscopic projection of the 3D camera image. This is very important as the operator requires an accurate sense of depth in order to control the reach truck. Besides that, it is essential for the reach truck driver to still see his real environment as he’s driving around the warehouse or simply moving forward or backward just a little bit. As the HoloLens is an augmented reality device that adds virtual information on top of the real world, the device can be safely used in a warehouse environment.

To stream the video from the 3D camera to the HoloLens, we needed two things: (1) a steady network connection and (2) a micro-computer broadcasting the video. This is not only necessary to stream the video to the HoloLens, but also to achieve low latency. This is the time between the camera that records the 2 videos (left eye and right eye), and the moment it is visualized in full HD in the HoloLens.

The idea is simple, but the implementation requires some further explanation. We installed the Jetson micro-computer into the mast of the reach truck. The Jetson is a full Linux computer and is responsible for streaming the video from the 3D camera to the HoloLens. To stream the video from the Jetson to the HoloLens, we plugged in two Wi-Fi antennas via USB and set those up to act as Wi-Fi hotspots. The HoloLens is connected to that Wi-Fi network, allowing it to communicate with the Jetson and thus receiving a video stream. The benefit of this Wi-Fi network is that it moves along with the reach truck, wherever it goes.

On the other hand, the Jetson is also connected to the Wi-Fi network of the warehouse, and in consequence also to the internet. Because of our set-up described above, this internet connection can in return be passed through to the HoloLens. Even though the HoloLens application works without internet connectivity, we added this functionality for sending important data to the cloud. For instance, employees logging into their Microsoft account. In the future, the Jetson can also be used for computer vision interpretation of the video stream.

Thanks to this approach, we were able to get a latency between 100 and 130 milliseconds – which is extremely fast! To give you an idea:

Perception of real-time: 400 milliseconds
Ultra-low latency: 200 milliseconds
IKEA solution latency: 100 – 130 milliseconds

In phase 2 of this project, we will exploit the stereoscopic vision of the 3D camera and the HoloLens. By means of AI and Computer Vision, we will support the reach truck drivers with – well… let’s call it “parking sensors”. When picking furniture or pallets out of the rack, we will provide the reach truck drivers with extra visual assistance such as a green overlay when the alignment is correct, and real-time distance measuring from the fork to the rack.

Our approach

Partnerships were made

An inspiration workshop with both C-level management and reach truck drivers was the start of this project. From day one, the needs and vision of IKEA’s reach truck drivers was of high importance. After all, they are the ones using the application in the end.

Throughout the process of analyzing and developing the solution, we could always rely on a task force created by IKEA. They gave us the input we needed to create the best possible solution for them, with a strong focus on human centric design.

This project wasn’t only a collaboration with IKEA, but it also became a partnership with Toyota. As part of the think phase, we contacted Toyota to define not only the software solution, but also the hardware solution. Parallel to our wireless solution, we also developed a wired solution. To make the solution as clean as possible, we decided to retrofit the reach truck (meaning make some small adjustments to the truck) in order to include the cables into the pulley system of the reach truck. To test the HoloLens app in a first phase, we will use our wireless solution.