Advancement over Replacement
Posted on Sep 8, 2023 by FEED Staff
FEED uncovers the creative ways vendors are taking advantage of AI, reinventing media technology as we know it
Computers have always been devices intended to take the drudgery out of everyday tasks. It hasn’t been long since artificial intelligence and machine learning started hinting at the possibility they could do a lot more, but it’s been long enough that AI-enabled features are becoming an increasingly big part of shipping products – saving time, or even opening up markets that were once a mile out of reach.
Reimagining the un-reimaginable
Researchers might suggest that AI and machine learning are terms we should separate. Even if we bundle them up into a single concept, some of the most visible, identifiable implementations are in software – like Blackmagic’s Resolve.
The beta release of version 18.5 emerged at this year’s NAB Show with significant new AI-enabled features, though as the company’s Craig Heffernan says, some of the software underpinnings have been there for a while.
“We’ve had AI in Resolve for four or five years,” Heffernan points out. “Our term for it is the neural engine. The earliest stuff was sync-up support… there’s a search engine which works for media management, object recognition and mask creation that would take too much time for a person. We’ve built in tools that use a degree of AI processing, calculation and internal research to support the creative user. Now in 18.5, the neural engine has dozens of features that operate like a toolset in a VFX package.”
Bringing VFX-style tools to grading is significant in the world of film post-production, where it has long been theoretically possible to create more or less any image someone can describe. Using VFX techniques to fix a problem shot, though, has always been a ruinously expensive last resort, so anything which makes more advanced techniques available to colourists is likely to go down well.
Lens flares might seem like old news, but Blackmagic’s new tools are an ample demonstration of how AI can bring unexpected levels of detail without demanding extensive manual intervention. Creatively, too, the feature makes a lot of sense in an industry which seems perpetually fascinated with the idiosyncrasies of lenses old and new. “A traditional lens flare you drop in as a plug-in,” Heffernan says. “And as you pan, it might have some motion. But the neural engine has a simulation of a lens flare – they’ve analysed lenses, spherical or anamorphic. It can look at the image using some of the depth map tools and simulate a real lens.”
An even more common demand is the removal of unwelcome background objects, something that might previously have required a round-trip to Resolve’s built-in Fusion VFX package. “We have an object recognition AI where you can use magic mask – it knows the difference between the horse and the rider. The classic example is a logo on a shirt; just go in with object removal and it goes away. That’s your wire removal tools, too.”
Sift and sort
With Resolve already boasting Fairlight digital audio tools, Blackmagic has put AI to work in sound post-production, too. “At NAB, we announced audio recognition that searches through a set of clips in a bin and sifts them depending on whether they’re dialogue or other content,” Heffernan explains, “which would be a pretty mundane task otherwise.”
The apparent simplicity of that kind of task belies the sheer time spent on it, given almost every project will need to do it. Meanwhile, the option to clean up less-than-ideal original recordings is a rarer ask, though something the world has long pined for. AI has made inroads there also.
“Within the audio package we did for neural engine, you can take the voice sample of someone on a noisy street and it will give you a completely isolated voice output.” Much as background noise separation has been tried before using conventional techniques, Heffernan describes how the AI approach works very differently. “Inside the voice isolation is an interesting example – it doesn’t remove the noise and give you the speaker’s voice. It removes all of the original audio and builds an artificial simulation of what’s been spoken.”
Some things Resolve’s AI will do are entirely new. The most commonly used things like the clip searching and object recognition could have been done by hand – in fact have been done by hand for years, but only with an impractical amount of effort. “The use of AI is to speed up tasks,” Heffernan finishes. “Our philosophy around AI is that the tool can’t replace the creative. It has to support the creative, to give you speed and precision.”
The automation era
Applying that sort of thinking to the other end of production is an attractive idea to anyone who has ever wanted to televise a football match but lacked the financial wherewithal to do so. Julien Bergeaud has worked in sports media for more than 20 years with companies including Eurosport and Discovery – and is currently vice president of sales in Europe for Pixellot, a company which aims to bring automation to live production.
The company’s work centres around a camera, albeit one which is about as tightly integrated with software as a device could ever be. Bergeaud describes the system as “the camera, which is hardware, that goes alongside software. That’s an algorithm built with AI which has learned a sport. It’s capable of automatically producing that sport as if there’s human intervention – but there isn’t. Thanks to the AI, the camera is capable of doing everything a conventional broadcast is capable of doing – zooming in, out and following the action.”
Making that a reality requires a carefully implemented camera head, with three lenses which create a 180° view of the sports field combined into a single 12K-wide image. Then, claims Bergeaud: “We learn the sport. We watch the sports and we teach the machine what’s happening on the pitch and why it’s important. Now we have 17 different sports. People think systems like ours are simply tracking the ball. There was a game in Scotland where a system got confused and tracked the head of a bald man. This is not us. What we do is machine learning, deep learning; we learn the sport.”
There are benefits for the teams taking part, too, as Bergeaud explains. “The second part is all the analytics. The AI is capable of recording everything happening on pitch – all the components of the game, for a single player or for a team. The machine does it automatically and it’s fantastic for elite teams. In France, we have already worked on Ligue 1 and now we are doing Ligue 2. If it’s available for elite teams, it’s also available for other segments.”
Sporting decision
Producing a sports match for television is nothing new. The option to do it at anything other than the very highest levels is unprecedented. “It unlocks fantastic opportunities for lower-level leagues, sports which weren’t able to find a model to finance traditional production,” Bergeaud enthuses. “If you include cameras, trucks and people, even if it’s only two or three cameras it adds up to a lot of cost and many leagues have no stream revenue to finance that. It’s a bit chicken and egg. Until they have media exposure, they can’t find money. We can give them media exposure, and they can find sponsors, advertisers, subscribers – many different models.”
Those models have proved popular worldwide. “In Israel, we’ve done basketball,” Bergeaud reports. “Now all the leagues, both men’s and women’s, are covered and have been able to find new sponsors – they wouldn’t have been able to do it without media exposure. The broadcaster for South Africa realised that the average age of their subscribers was becoming older and older – 60 plus. This broadcaster realised they needed to engage with a younger audience, and so launched SuperSport Schools on OTT.”
“It was so successful,” Bergeaud reports, “that they actually launched a new linear channel called SuperSport Schools. The old model of traditional TV is coming back thanks to the technology of AI and automated production. We’re creating engagement for the community – not only the player, but their family and fans. In the US, for instance, we have a very large project for high schools. When people were stuck at home due to Covid-19, they wanted to see their kids play. Now we have a partnership with PlayOn TV, and with an app that people can subscribe to and watch their kids play in the high school. We have a million people subscribing at $20 a month.”
Read the full article in the Autumn 2023 issue of FEED.
