Because of our experience with real-time stereoscopic 3D video combined with smartglasses (solution for DEME), we knew such a solution could suit IKEA as well. By attaching a 3D camera to the fork of the reach truck, and streaming a 3D video to the smartglasses, we would be able to bring the eyes of the reach truck driver at the same height of the fork – no matter the altitude.
The HoloLens is the best fit for this solution, as it allows for pin sharp stereoscopic projection of the 3D camera image. This is very important as the operator requires an accurate sense of depth in order to control the reach truck. Besides that, it is essential for the reach truck driver to still see his real environment as he’s driving around the warehouse or simply moving forward or backward just a little bit. As the HoloLens is an augmented reality device that adds virtual information on top of the real world, the device can be safely used in a warehouse environment.
To stream the video from the 3D camera to the HoloLens, we needed two things: (1) a steady network connection and (2) a micro-computer broadcasting the video. This is not only necessary to stream the video to the HoloLens, but also to achieve low latency. This is the time between the camera that records the 2 videos (left eye and right eye), and the moment it is visualized in full HD in the HoloLens.
The idea is simple, but the implementation requires some further explanation. We installed the Jetson micro-computer into the mast of the reach truck. The Jetson is a full Linux computer and is responsible for streaming the video from the 3D camera to the HoloLens. To stream the video from the Jetson to the HoloLens, we plugged in two Wi-Fi antennas via USB and set those up to act as Wi-Fi hotspots. The HoloLens is connected to that Wi-Fi network, allowing it to communicate with the Jetson and thus receiving a video stream. The benefit of this Wi-Fi network is that it moves along with the reach truck, wherever it goes.
On the other hand, the Jetson is also connected to the Wi-Fi network of the warehouse, and in consequence also to the internet. Because of our set-up described above, this internet connection can in return be passed through to the HoloLens. Even though the HoloLens application works without internet connectivity, we added this functionality for sending important data to the cloud. For instance, employees logging into their Microsoft account. In the future, the Jetson can also be used for computer vision interpretation of the video stream.
Thanks to this approach, we were able to get a latency between 100 and 130 milliseconds – which is extremely fast! To give you an idea:
- Perception of real-time: 400 milliseconds
- Ultra-low latency: 200 milliseconds
- IKEA solution latency: 100 – 130 milliseconds
In phase 2 of this project, we will exploit the stereoscopic vision of the 3D camera and the HoloLens. By means of AI and Computer Vision, we will support the reach truck drivers with – well… let’s call it “parking sensors”. When picking furniture or pallets out of the rack, we will provide the reach truck drivers with extra visual assistance such as a green overlay when the alignment is correct, and real-time distance measuring from the fork to the rack.