The crane was already equipped with VR technology, but since it is essential for the operator to always see his physical controller, VR headsets were not an option. To work safely, an accurate sense of depth is also a must. We therefore decided to create an augmented reality solution for the Microsoft HoloLens.

By leveraging a 3D stereoscopic camera, the app visualizes the actions of the crane and its environment in a true 3D livestream, projected on a holographic hemisphere. Being an augmented reality device, the HoloLens allows for pin-sharp stereoscopic projection of the 3D camera image while still giving the operator a view of his physical remote control and real-life surroundings.

Also, the HoloLens is a fully standalone, untethered headset. It can be used from anywhere, in every office or any location in the field, as long as good network connectivity is provided. No need for an additional laptop or desktop computer. Having this functionality running on a HoloLens proves to be a very convenient and portable solution.

What makes the project truly robust is the secondary “controller” mechanism keeping track of the latency. Since there is no room for errors, the end-to-end latency (the time between making a single video frame, streaming it, and visualizing on the HoloLens) of the stream was highly important and a determining factor to give the project a ‘No‘ or a ‘Go‘. By using a mix of the industry standard streaming protocols and a carefully chosen combination of the videocodec, container and transport parameters, we were able to achieve a latency of around 120 milliseconds! Generally, anything below 200 milliseconds is considered “real-time”. Should the stream slow down and be interrupted due to network issues for example, a warning and even an alarm appear on the screen to instruct the operator to pause his process until the latency returns to normal values.