Graphical visualization and 3D simulations in TV shows are well suited for explaining abstract concepts and progresses. Since the 1960s the blue/greenscreen-technique is used in German productions to add graphics. Within the last ten years many broadcasters used virtual studios for their production.  To escape the “green hell”, Augmented Reality (AR) has gotten more popular in the studio environment. Both techniques enable interaction between the moderator and virtual graphics and therefore support the storytelling. Besides the huge amount of equipment, that is necessary for those techniques, the moderator is also not able to see or control the virtual objects.
With the help of AR enabled mobile devices, such as AR glasses or smartphones, users can now enhance their experience by interacting with virtual objects that were added to the real environment.

First prototype: Microsoft Spectator View and HoloLens

In 2018, the IRT started to test if these new devices could be used for production in broadcasting. For the first approach we used the Microsoft Spectator View, that utilizes two HoloLenses for camera tracking and moderator vision. The prototype has been demonstrated to several editors from the German public broadcasters to collect their feedback. As a result, the handling of the HoloLens for the moderator was challenging and cumbersome, even though the produced content was satisfying.

 

Second prototype: iPad interaction with iOS Spectator View

To improve the moderator’s ability in controlling the scene, a second prototype has been developed with an iPad to interact with the virtual objects and an iPhone to record the session.  This approach focuses on the representation of drone delivery in a spectator view scenario. The goal is to integrate new technologies like AR in an everyday Moderator/spectator scenario for broadcasting manners. The use of AR should help to better understand the idea of drone delivery with new ways of illustration and visualization. Furthermore, it should be easier for the moderator to explain the scenario while interacting with an intuitive app. For the viewer it is also more tangible and easier to understand if they see the animations and the objects in three-dimensional space. Additionally, we focused on AR for mobile devices to enhance the ease of use and the accessibility to the public. Our scenario therefore consists of two mobile devices, where one device is controlled by the moderator for interaction¨ and the other device is used by the user as the spectator.

Technical Details

The application was developed in Swift for iOS 12 or higher. For AR functionality we used the framework AR Kit in connection with Scene Kit. The connectivity between devices was handled with Multipeer Connectivity, which takes care of all the network capabilities and the synchronization of events. The 3D models were designed in Blender and then imported as objects into Xcode.

Watch the demo video

Authors:

Duc Anh Le, Simon von der Au, Niloofar Samimi