Extended Reality Special
Posted on Jun 26, 2024 by FEED Staff
Putting the believable into make-believe
Extended reality has the potential to produce eye-popping content both quicker and cheaper than conventional filming – bringing the director’s console into the viewer’s own home. We ask broadcasters and creatives how they make the technology work for them
Words by Paul Bray
We’ve always known that ‘nothing you see on television is real’. But as extended reality (XR) increases its hold on the industry, this could often be literally true.
The BBC can present its coverage of the Japanese Grand Prix from a studio in Manchester, a weatherman can report on a hurricane surge while up to his neck in flood water. In drama, the results can range from the epic – think The Mandalorian, an early pioneer of XR filming – to the frankly surreal: the Super Bowl simulcast for kids from the Spongebob Squarepants world of Bikini Bottom.
XR is something of a catch-all term. It can encompass virtual reality (VR), where everything the viewer sees is computer-generated; mixed reality (MR), which combines real-life action and computer-generated imagery (such as live actors or presenters and a pre-recorded backdrop); and even augmented reality (AR), where computer-generated imagery is overlaid onto a view of real life.
The concept of using imagery to defy the boundaries of studio or stage is nothing new, of course. When we see Humphrey Bogart driving a car, we know the moving background is just a back-projection. This works well enough for a static shot, where we are always looking over the driver’s shoulder. But if the camera moves, the perspective quickly gets out of kilter, as the camera is actually filming the rear projection screen a few feet away, instead of the real landscape disappearing to infinity.
To retain true perspective, you would need to continually adjust the back-projection to match the camera’s movements. This is exactly what modern XR technology can do, rendering the content in real time and mapping it onto the backdrop to fit the camera’s perspective. It’s often referred to as 3D rendering, to distinguish it from flat, Bogart-style 2D backgrounds (which still have their place, incidentally – such as in more static, presenter-behind-a-desk studio settings).
Instead of back-projection, today’s XR studios rely on video walls composed of LED panels – often several metres wide, and sometimes coming forward in a curve or three sides of a box, and even projecting across the floor. If a shot runs over the edge of the video wall, all is not lost. Virtual rendering technology can take over, creating the missing parts of the picture in real time and fusing them with the image the camera is seeing, giving the illusion of a full 360° set.
The potential benefits of XR are legion: set changes at the touch of a button or capture live talent and virtual background in a single shot without hours of post-production. Actors and presenters can experience, and even interact with, the virtual set. Less need for expensive and planet-trashing travel – and above all, win eyeballs with gripping but affordable content.
Seeing it to believe it
But sometimes, the technology must be seen to be believed. “It’s interesting to see people come into our studio for the first time – even though we may have been talking to them for weeks – and suddenly the penny drops; they see all the possibilities,” says Jonathan Brigden, managing director of Distortion Creative Group.
The company’s virtual studio in Bristol, under its Distortion Studios brand, is one of only a dozen or so such facilities in the UK, Brigden reckons. It features a Roe BP2V2 LED video wall, Brompton processing and Unreal integration, with workflows all controlled by Stage Precision software.
“People say, ‘Instead of filming 5% or 10% of shots in XR, I could do 40%,’” explains Brigden. “It’s not just the fantasy shots, but a detailed interior scene in a restaurant, or the explosion of a volcano. You can repeat one-off shots like that over and over again. You can do aerial shots you didn’t think you could get, or have the camera pull really far back. One production company is rethinking how it could make all the shows on its development slate.”
“Period documentaries – historical events, prehistoric wildlife, etc – are ideal candidates, he adds. “They tend to have big expectations but not big budgets, and VP can help them create content which otherwise wouldn’t be feasible.”
Karachi-based production house Zeekay Films is already convinced. It has used XR for video games, TV commercials and drama, including Unreal Engine for content creation, HTC Vive camera tracking and ARwall’s ARFX Pro Plugin for workflow.
“Our first project was our historical biography Salam – Fateh Hindustan, where we created detailed indoor palace shots,” begins CEO Muhammad Zeeshan Khan. “Set 830 years ago, this series required meticulous attention to detail – and XR provided the texture and historical authenticity we needed. The results exceeded our expectations.”
XR is a game changer for filmmakers, Khan believes. “Previously, we relied on green screens with tracking marks, and spent months in post-production to create believable environments. This was complex, time-consuming and often the results didn’t look entirely real. Additionally, actors and DOPs had to imagine the settings based on storyboards, which could limit the performance.
“XR and in-camera VFX allow the entire crew to see and interact with the environment in real-time,” he continues. “DOPs and compositors work together to light the set, making the shooting process much smoother. Once everything’s set and lit, recording becomes straightforward and the footage is ready for post-production, without the usual delays.”
Outdoor scenes can be shot in a controlled studio setting, without being hostage to the light or weather. “We can also consistently capture scenes like sunrises or sunsets throughout the day, which was previously impossible,” details Khan.
But the most significant benefit is time and cost. “By preparing everything before shooting, producers can complete high-VFX projects much faster, and you avoid travel and large crews,” he adds. “We’ve filmed a cosmetics commercial on a beach and a car commercial on a glacier, without needing to get on a plane or negotiate customs. Before XR, we once needed to erect a complete palace set to shoot a scene. We had to wait for days before recording could start. With XR, we designed two environments in our office and recorded them both in one session.”
Spin-off revenues are also a possibility. “With XR, a film can be produced using actors in a gaming environment, and later, you can release a game. This creates a whole new opportunity for producers and game development companies to monetise more strategically,” Khan surmises.
Gen Z-approved
For broadcasters who want to engage with a young audience – the Gen Zs and Millennials who crave graphics-heavy, social-first programming – XR can be the ideal solution.
Blinx, an online, Arabic-language news and current affairs broadcaster based in Dubai and produced for (and by) just this age group, has partnered with Vizrt to create a full XR set. It comes with a 12m wide, 1.5mm pixel pitch LED video wall and floor, plus the ability to extend into a full 360° VR environment.
“The virtual set becomes a complete sphere with the presenter inside it,” explains Fadi Radi, chief creative officer at Blinx. “We can connect the virtual camera with the actual camera – so for example, we can fly from the moon into the earth, the building or into the set. That gives us unlimited possibilities when it comes to moving the camera around, and it’s entirely seamless; the audience can’t see where the virtual world ends and the real world begins.”
The XR approach offers numerous advantages over green screen. “Instead of standing in a green box, the presenter can interact with everything they see around them,” says Radi. “It’s like being teleported into a place – you can have the same experience as if you were really there. It gives so much more realism.”
The presenter and physical set can be lit by the light from the LEDs. “All the reflections we get from the set are a plus for us, whereas in green screen, they’re something you need to remove,” adds Radi. “When the environment reflects on the talent, it gives more of the feeling of physically being there.”
There’s also a time-saving aspect. “Once you’ve set up the scene, that’s it, everything works. You can make changes to the set instantly.”
Keeping fans engaged
The potential of XR at the viewer’s end is also exciting, according to Daniel Aslam, senior manager and head of business development at Deutsche Telekom (DT). The promise of viable, affordable – and above all, comfortable – VR headsets from the likes of Apple, Metaquest, Samsung and Google has rekindled DT’s interest in creating XR content on its sports streaming channels, such as Magentasport.
For this year’s Euros, the company worked with Accedo to develop a prototype XR sports streaming application for Magentasport. Alongside the main 2D stream of the match, viewers can choose between two additional channels: one hardcore tactical, the other social media-based. Further options include live statistics, a video-on-demand stream for instant replays, an interactive mini-pitch with avatars showing each player’s current location and a 3D advertising window, where viewers can interact with 3D representations of products – look around a car, customise a running shoe, etc.
DT’s previous XR trials found that, while earlier-generation headsets left much to be desired, the concept itself was a hit. “Viewers were impressed,” admits Aslam. “They said: ‘I can be my own director, select between many camera feeds and feel as though I’m really there at the game’. We’ve also developed a co-watching experience for two people at different locations, using avatars.”
AR is especially interesting for Aslam. “You’re not completely isolated, but you can see through the virtual content into your real environment.”
XR is perfectly affordable for broadcasters, Aslam believes, since most of the technology (particularly myriad cameras at matches) is already in place. But more needs to happen before it becomes a commercial reality.
Firstly, there must be a critical mass of headsets (Aslam is sceptical of claims that there are already a million in use in Germany or the UK), and headset manufacturers need to promote their wares for enhanced TV viewing – not exclusively games or business meetings.
Furthermore, licensing must be sorted out. “The media industry in Europe isn’t yet ready,” says Aslam. “There are licences for broadcast TV, but no extra dedicated rights packages for XR. This needs to happen during the next round of licence options.”
Already in talks with major European sports organisers, Aslam hopes that they won’t be too ambitious. “For the first year, I believe you need to offer the proposition for free to get viewers used to it. You likely won’t get advertising until you have critical mass.”
From studio to stage
Broadcasters’ growing interest in XR is increasingly being paralleled in live theatre – which has, after all, been in the business of creating imaginary realities since the days of Sophocles.
“There’s always been an interest in adapting these new technologies for the stage,” observes David Gochfeld, performance lab fellow at the Royal Central School of Speech and Drama. “Since Covid-19, there’s been ample interest in extending theatre beyond the physical space.”
Audience members with the right technology – VR headsets and suitable home computers – may be invited to enter the 3D environment with the performers, and even join in the action. Some live theatre companies already offer audience participation in physical performances, often in large, warehouse settings.
“What’s new with VR is that audiences and performers don’t all have to be together in the same physical space,” affirms Gochfeld. “It also allows us to work with artists who are not in the same place.”
In the US, the Actors Theatre of Louisville has developed a mixed-reality version of that Yuletide staple, A Christmas Carol. It uses two live actors – one playing Dickens and Scrooge, the other all the ghosts – although everything the audience sees is virtual, including the pre-baked sets and lighting, as well as avatars of the performers, which are animated in real time using Unreal Engine.
VR has the capacity to engage audiences in ways physical theatre finds hard to match. “Alice eating the mushroom can be much more embracing because VR takes over all your visual and audio input – and can very effectively fool you into thinking you’re ten inches tall,” Gochfeld points out.
But it does not have to be photorealistic. “It’s about creating a sense of presence rather than realism of imagery,” conveys Gochfeld. He cautions against entering the uncanny valley, a decades-old term for the curious fact that almost-but-not-quite-realistic objects can seem strange and disturbing, while an obviously cartoon-like appearance can be perfectly acceptable.
Perfect for previsualisation
Back in the studio, XR has benefits behind the camera as well as in front. “The previs and virtual art direction process can pull much of the decision-making off set into pre-production, and DOPs like Greig Fraser have noted that tools like Unreal Engine perhaps could – and should – be used to plan every shot, as he did for Dune: Part 2,” says Rene Amador, co-founder and CEO of ARwall.
During shooting, AR on portable devices has the potential to liberate the director from the gallery. “We can run our system on an iPad or even in a web browser on a headset,” introduces Adam Leah, creative director at Nxtedition. “You could control kit miles away just by pushing buttons on your iPad or even using hand gestures.”
Using AR, sources, prompters, timing screens, rundowns or interactive shotboxes can be displayed on a headset screen so they overlay the director’s view of the real world. “This would be fantastic at Glastonbury,” enthuses Leah. “You could be on top of the tower, see where all the cameras were and do all the monitoring as though you were on the truck, but you’d be feeling and seeing the actual event.”
Leah cautions against over-using XR. “In sport, it’s fantastic but I’m not a big fan of VR news studios,” he admits. “I don’t like it when everything looks like a video game when a reporter is giving viewers the harsh realities of life.” He points out the BBC’s live news studio is all real, with no XR – just conventional screens and video walls.
Nonetheless, it seems XR is here to stay. “Stanley Kubrick once said: ‘If it can be written or thought, it can be filmed,’” concludes Khan. “With XR and in-camera VFX, this becomes a reality. Filmmakers have a vision to tell their story to a diverse audience, and XR makes this possible.”
This feature was first published in the Summer 2024 issue of FEED.
