Raising the IQ of sports production
Mobile Viewpoint’s AI-driven production solution makes broadcasting possible for any sport, anywhere.
The evergreen popularity of live sports is part symptom and part cause of the rapid adoption of video streaming services. But while the demand is huge, only half of the opportunity is being seized. Accelerating output and offering access to a wider range of sports coverage is the next big challenge and one that automated production systems can help deliver. “Just as we have seen costly satellite trucks replaced by backpack-sized live video transmission units, artificial intelligence will deliver similar cost savings for live production and streaming,” says Michel Bais, managing director at Mobile Viewpoint, developers of a system for sending video over the Internet using mobile networks. Whether by analysing audio signals, video images, identifying people and objects, or selecting and controlling cameras, AI raises the possibilities for working without expensive human crews. Algorithms can automatically generate replays and graphics, provide extensive metadata and then livestream the content. AI (which you might describe as augmented or assisted intelligence) could revolutionise the industry. Dutch developer Mobile Viewpoint is not alone in looking to extend its capabilities beyond the competitive cellular uplinks market. Canada’s Dejero has a mobile app that helps bridge the gap between traditional remote production and cloud production. While TVU Networks has identified automatic sports logging and production as a related field for its core bonded cellular units. Mobile Viewpoint, though, has arguably gone further than its competitors in co-opting AI into the workflow. The company has worked with Dutch research institute TNO to retune an off-the-peg AI brain from Google’s TensorFlow to fit the patterns, behaviours and relevant data sets of live streamed football. The package, branded IQ Sports Producer, comprises two parts and includes a 32MB dome camera, containing four lenses, the feed from which is stitched into a 180° panoramic field-of-view. De-warping and stitching technology provides a corrected image with straight line markings. This image can be cut out, panned, or zoomed as directed by the AI software, which is trained to follow the action as a human director would, all of which the system does with about a one-second latency. At its crudest, the AI can track groups of players (that being the most likely location for the key action), but more sophisticated applications can be taught to anticipate where a ball might be passed to. An Eye on The Ball The camera was originally built by China’s Hikvision for surveillance purposes, but it has proved well suited for this solution, not least because it is relatively cheap (around 5000 per module) and is designed to withstand all weather as well as vandalism. Plus, the camera’s four 4K feeds are reliably in sync, which, Bais says, is not always the case. The second part of IQ Sports Producer package is hardware containing three Nvidia GPU boards, plus the AI software and, of course, Mobile Viewpoint’s forte, the mobile connectivity. “IP links are our bread and butter,” says Bais. The AI software can combine motion-tracking with positional and other biometric data gathered from sensors (RFID, accelerometers, GPS) worn on special vests by the players themselves. The first (currently only) system with the 32MP camera in Europe is installed at Amsterdam’s AFC Ajax, the biggest football club in the Netherlands and part of the UEFA Champions League. Ajax is using it to film and stream its matches to its club web channel Ajax TV. The captured footage is also used to gather data for player performance analysis during training. The Ajax training academy can monitor why a player missed a goal, why they failed to make an assist and help improve player performance. Image detection means the AI technology can recognise different players and follow them, or, detect a ball on a pitch and follow its movements. “The potential for AI in this respect is huge, especially in the production of live sports content,” says Bais. “As algorithms develop, AI can detect faults (yellow or red cards) or injuries as it learns how to make productions more interesting and story-like. There are plenty of smaller sports that could use this technology to become content owners in their own right at a low cost and then monetise it. “Only 10% of professional sports are premier or first leagues. The majority are second or third tier sports, many of which are played at grounds or parks with limited Internet connectivity, even when they’re in middle of a city.” That’s where Mobile Viewpoint’s bonded cellular links come in. It’s 3G and 4G capable today, encoded in HEVC H.265, but primed for 5G, which telcos worldwide, including local giant KPN, will roll out in 2020. The Netherlands Pro women’s football league, for example, is keen to install IQ Sports Producer at six venues throughout the country as a budget-friendly means of launching a new online matchday streaming service.
The multi-camera capture of the action is live-stitched with coverage “directed” on the fly by the IQ Sports AI.
Algorithms Get Up to Speed
Completely automating live sports streaming at the top level without the need for a production crew and director might be a bridge too far – for now. AI used for assisted production alongside humans is the immediate goal.
One of the obstacles to fully automated streaming is that it takes time for an algorithm to learn the nuances of what is interesting or important for each sport. For example, an algorithm may think that capturing a fight breaking out during a football match is the same as capturing a punch being thrown during a boxing match – but for the viewer, these are two very different experiences – one that is normal and one that isn’t.
“Getting the algorithms up to speed requires time, so there is still very much a role for humans,” says Bais. “The AI tends to look for people walking or running rather than people on the ground, so it doesn’t necessarily zoom in on a player who is down following a tackle, which may be what the viewer would like to see.”
Higher image resolutions help too, since the more pixels the AI has to play with it, the more it can differentiate the ball from a piece of white paper on the pitch.
While most budget-conscious customers might want a single camera solution, multiple cameras around a venue are essential for capturing a variety of angles and gathering more data to achieve greater accuracy.
Planned enhancements to the software include AI-enabled ad insertion, auto playback of replays following a goal, auto highlights generation and continuing to train the system on a wider range of content, including motor sport and ice hockey. The system has even created a means of translating a game into 3D VR graphics within which a user can select the point of view of any player to analyse decision making during a game. It is currently being used by AFC Ajax as a training tool. “Our ambition is to bring this to the home,” says Bais. “With (Dutch telco) KPN we are working to bring a layer of interactivity to livestreamed games running on KPN’s set-top box.”
AI enabled production tools are going to create a sports content boom. The next step may well be AIs that help manage the rising tide of sports content innovations.
This article originally appeared in the January 2019 issue of FEED magazine.]]>
