There is an explosion of virtual set technologies. Once reserved for top-tier broadcasters, video magic is now becoming available to anyone

Virtual sets are now a common sight on our screens, replacing the plywood and fabric of studios past and placing presenters, pundits and news anchors in real-time rendered, shiny-floored virtual spaces, complete with augmented reality pop-ups and CG characters. 

“Without doubt we are seeing a substantial increase in virtual set adoption,” says Nic Hatch, CEO of real-time augmented reality technology developer, Ncam. “Predominantly because the technology is now easier than ever to use and delivers on the quality required. We know that viewership engagement has increased significantly for customers embracing our products.”

Stuart Russell, senior communications manager for Ross Video, adds that interest in virtual solutions has grown as the technology has improved and the graphics rendering has become more sophisticated. “I think customers see the value in being able to use a green screen studio for a variety of different show formats,” he says. “There is no doubt that it allows for greater flexibility in terms of how studio space is used. That said, I think we’re probably seeing more of an interest in augmented reality at the moment. 

“I think content creators like the versatility of having a physical set and being able to bring virtual elements into it to add some extra creativity to their programming. It’s definitely becoming more widely used in traditional programming and the sports and esports industries have been right at the front edge of the technology. Ross recently worked with esports company ESL and Spidercam to help deliver augmented reality content during the ESL One event in Cologne and I think we’re going to see this kind of content becoming more prevalent.”

“The market will definitely continue to increase exponentially,” agrees Andy Hook, technical solutions director at broadcast gear provider, White Light. “Right now, we are seeing a shift in a virtual studio being more acceptable for use in corporate, education and live streaming environments. We’re also seeing more hybrid mixed reality studio sets, combining real sets with virtual extensions to make them feel bigger or create elements that would be physically or financially difficult to construct.”

Today, a virtual studio means more than just a stage with a green screen, says Ulaş Kaçmaz, vice-president of sales and marketing for Zero Density, which specialises in real-time graphics and VFX. “Augmented reality elements, hybrid studio operations – part green screen,  part physical – portal windows, set extensions and so on also mean virtual studio production. Virtual studio use is surely expanding, although without a hint of physical sets becoming obsolete at any time in the future. There are immense possibilities in other areas for this technology to enhance visual storytelling. Creating a fantasy world from scratch and adding texture and spirit with invisible real-time VFX.”

Kaçmaz points out the potential for virtual set-based documentaries and historical remakes. “They already have such strong stories,” he says. “Imagine the captivating picture that could be curated by employing this tech. Also, you’d be in complete control of your scene during the shooting or previs stage instead of fixing all up in the post and settling for the inevitable glitches.”

Virtual sets technology works at its best when it complements the story rather than being gimmicky

“Storytelling is the key point,” agrees Ross Video’s Russell. “Virtual sets technology works at its best when it complements the story rather than being gimmicky, and I think the technology continues to develop in a way that makes the images we produce more realistic, natural and interesting to the viewer.”    

Imagination engines

An essential part of virtual set solutions is accurate camera tracking. Systems from the likes of Mo-Sys, Stype and Ncam allow live tracking and precise placement of virtual and physical elements. Virtual set software converts the data coming from the tracking system and converts it to virtual camera data for the rendering engine. 

“Ncam’s approach is to provide a complete virtual set solution, meaning that not only do we provide the most flexible and robust camera tracking technology available, but we also supply highly accurate lens distortion data, which is crucial to ensuring that the real world matches up seamlessly with the virtual world,” says Nic Hatch. “Ncam’s recently released AR Suite powered by version 4.23 of the Unreal Engine is able to take all the Ncam data and produce a final keyed and composited image. All these elements have been designed from the ground up internally at Ncam.”

Until now, it has been the privilege of a select group of broadcast organisations to use high-end virtual studio technology

This integration of the Unreal Engine (UE) from Epic Games as the 3D renderer in many proprietary virtual set solutions has pushed the wow factor for a couple of years now, but tools built around this photorealistic real-time game engine continue to impress and be refined. An example is the Reality Engine composting software from Turkey-based Zero Density.

“Reality Engine’s unique architecture allows Hollywood-level visual effects in live production and the node-based compositing and keying tools enable the most photorealistic output,” claims Kaçmaz. “Zero Density not only provides the main system, but also the necessary automation, monitoring and controlling interfaces. The system is designed to run on multiple Unreal Engine instances that are all controlled and managed as one.

“Our Reality Keyer product works with a clean plate and combines this with the system’s tracking functionality to produce a mesh representation of any studio green screen cyclorama,” continues Kaçmaz. “It’s implemented as a shader inside UE4. The system’s use of projection mapping of the clean plate assists the keying and makes the system much more advanced than just a normal chroma keyer. It is the world’s only real-time image-based keyer that works on GPU, preserving real environment details such as contact shadows, transparent objects and sub-pixel details like hair.”

Russell says: “With almost everyone in the virtual solutions market offering graphics solutions that are based on the Unreal gaming engine, it has become increasingly difficult for customers to differentiate between vendors, and increasingly hard for vendors to add value. Ross has chosen to focus on the twin issues of configuration and control. 

“Our virtual solutions control platform is called UX and we’ve deliberately made it intuitive and extremely customisable. The graphical user interface is very easy to use and comes installed on a touchscreen PC for virtual set camera calibration, scene manipulation, media replacement, event triggering, MOS newsroom integration, animation control and robotic camera movement control. The application provides seamless integration with tracking systems and real-time 3D rendering engines such as Ross Video’s XPression and Voyager.”

Virtual TV

VooSport and BeTV decided to go virtual for their L’Europe des onze programme covering European football. The show was powered by Reality from Zero Density. The Belgian pay TV channel was based in Keywall studios in Charleroi, Belgium.

“Dreamwall designed the set and powered it with seven Reality Engines on seven cameras, including one crane,” says Kaçmaz. “The weekly show has already recorded more than 320 hours live from the virtual studios. They wanted to transform the game nights with seamless commentaries with 3D graphics, interactive elements and visually stunning content.”

 VooSport and BeTV use virtual sets for L’Europe des onze, transforming the programme with 3D graphics

Aximmetry is a Hungarian company that had its start in designing video solutions for stage. Its most recent focus has been seamless integration of real and virtual environments and, to this end, the company has created an internal, software-based keyer as part of its virtual studio software. 

“The new advanced keyer provides superior results for keying transparent objects, contact shadows and fine details,” says chief operations officer Orsolya Dormon. “It can handle broadcast-quality real time in 4K with GPU-accelerated rendering at a fraction the GPU usage. We have also made a leap in generating realistic shadows. Apart from contact shadows, Aximmetry can cast virtual shadows on the talent and we can combine these seamlessly with the real-life shadows generated by the talent itself.”

A mixed reality environment is the standout feature in White Light’s virtual studio tool, SmartStage. “Our solution omits the need for green screens by instead utilising LED video walls, so the content is really there,” says White Light’s Andy Hook. “This makes the system much more user-friendly. You do not need to be trained in broadcast, you can just walk on set and the interaction, eyelines, reflections, lighting and colour is all real and more natural.”

Next big thing

“Making real-time ray tracing a reality has been the biggest advancement of this year, promising true-to-life scenes,” says Zero Density’s Kaçmaz. 

In this case, software has had to catch up with the hardware. “Nvidia announced the RTX cards dedicated to real-time ray tracing last year,” he adds. “Zero Density and Nvidia partnered with Fox Sports for daily presentations inside a real-time ray-traced virtual studio at NAB 2019, which was solid evidence it can be done. At IBC2019, NEP The Netherlands and Zero Density showcased a superior real-time ray-traced virtual studio. Creative and developer teams have not adopted this tech completely yet, so it is not mainstream, but will be starting from 2020.”

Spanish virtual sets company Brainstorm is also implementing the hardware developments found in the Nvidia RTX GPU for photorealistic content generation in its latest InfinitySet. As well as real-time ray tracing version 3.1, InfinitySet takes full advantage of Unreal Engine 4.22 compatibility, HDR I/O, Physically Based Rendering (PBR), new effects and new 360° output. 

With growing environmental awareness, broadcasters need to use more
environmentally friendly production methods. There is no need to cut down forests to build 3D virtual sets

Meanwhile, broadcast graphics mainstay Vizrt has built upon its recent acquisition of Newtek with a new IP-based virtual set and AR graphics system for live production. Viz Arc allows users to drive AR graphics and virtual sets from a single interface and allows AR elements to be set up on a location map or studio floor plan. 

Communicating with Viz Virtual Studio, Viz Arc can also operate multiple Viz Engines simultaneously and blend between AR and virtual set backgrounds. It includes controls for the new Fusion Keyer, which can be used to ‘immerse the presenter into a virtual environment’ using techniques such as light-wrapping, a de-noiser and multiple mattes. Customers also have direct access to Viz Artist templates and other third-party integrations, including Unreal Engine 4.

Virtual future

“We are currently working on increased interaction options within the SmartStage virtual studio by tracking the people, presenters and objects in the space,” says Andy Hook. “This will allow us to dynamically adjust content as people move around the space, as well as allowing them to see and interact with augmented reality content in a more believable way. Since we don’t need to light a green screen in SmartStage, we can give control of lighting the subjects properly back to the lighting director, and having the lighting automatically track the users opens up some really exciting opportunities for them.”

According to Aximmetry’s Dormon, more studio professionals will need to have IT knowledge in the broadcast industries to operate the systems necessary for quality production. She predicts: “Consequently, professionals familiar with IT technology in general will more likely embrace virtual studio technology as well.

“Until now, it has been the privilege of a select group of broadcast organisations to use high-end virtual studio technology. The price of such systems has been prohibitive to many and it has been obstructing small studios, educational institutions, experimental labs and individuals from testing and trying new ways with real-3D graphics and virtual environments. 

“Aximmetry aims to lift this barrier and democratise virtual studio technology,” she continues. “We have been offering our Aximmetry Community Edition, a fully functional, high-end virtual studio software, for free to anyone who wishes to experience the technology and all Aximmetry licences have been under €5k since 2018. As a result, we are seeing interest from new segments: radio stations who are interested in video podcasts and social media inserts, university labs, YouTube creators and streamers, experimental labs in various industries and corporate in-house studios, among others.“ 

She concludes: “With growing environmental awareness, broadcasters need to use more environmentally friendly production methods. There is no need to cut down forests to build 3D virtual sets.”