Advanced altitude picking in warehouses for more efficiency, less damage & lower costs
At IKEA’s distribution center, forklift picking is one of the main daily tasks. Doing this at a higher altitude is difficult because there isn’t a clear view of the exact position. This results in ergonomic discomfort, rack & product damage, and loss in efficiency. To tackle this, we attached a 3D camera to the fork of the reach truck that streams a stereoscopic 3D video to the reach truck driver’s smartglasses.
The HoloLens is the best fit for this solution, as it allows for pin sharp stereoscopic projection of the 3D camera image. This is very important as the operator requires an accurate sense of depth in order to control the reach truck. Besides that, it is essential for the reach truck driver to still see his real environment as he’s driving around the warehouse or simply moving forward or backward just a little bit. As the HoloLens is an augmented reality device that adds virtual information on top of the real world, the device can be safely used in a warehouse environment.
To stream the video from the 3D camera to the HoloLens, we needed two things: (1) a steady network connection and (2) a micro-computer broadcasting the video. This is not only necessary to stream the video to the HoloLens, but also to achieve low latency. This is the time between the camera that records the 2 videos (left eye and right eye), and the moment it is visualized in full HD in the HoloLens.
The idea is simple, but the implementation requires some further explanation. We installed the Jetson micro-computer into the mast of the reach truck. The Jetson is a full Linux computer and is responsible for streaming the video from the 3D camera to the HoloLens. To stream the video from the Jetson to the HoloLens, we plugged in two Wi-Fi antennas via USB and set those up to act as Wi-Fi hotspots. The HoloLens is connected to that Wi-Fi network, allowing it to communicate with the Jetson and thus receiving a video stream. The benefit of this Wi-Fi network is that it moves along with the reach truck, wherever it goes.
On the other hand, the Jetson is also connected to the Wi-Fi network of the warehouse, and in consequence also to the internet. Because of our set-up described above, this internet connection can in return be passed through to the HoloLens. Even though the HoloLens application works without internet connectivity, we added this functionality for sending important data to the cloud. For instance, employees logging into their Microsoft account. In the future, the Jetson can also be used for computer vision interpretation of the video stream.
Thanks to this approach, we were able to get a latency between 100 and 130 milliseconds – which is extremely fast! To give you an idea:
- Perception of real-time: 400 milliseconds
- Ultra-low latency: 200 milliseconds
- IKEA solution latency: 100 – 130 milliseconds