Zelensky hologram: Taking on the empire
Posted on Sep 5, 2022 by FEED Staff
How a small team helped the president of Ukraine deliver a new message of hope to the world – the Zelensky hologram acting as a first for the history of tech
//Words by Neal Romanek//
Martin Williams launched Talesmith – after a career in top-quality, high-tech documentaries – with the mission of creating content he enjoyed making, particularly if it was challenging to visualise. In the spring, he got a call that fit the spec perfectly. Working on tech-heavy documentaries, with the likes of Atlantic Productions, Williams got a taste for media innovation. Projects like The Green Planet AR Experience, which brought Sir David Attenborough to life for audiences in full, interactive augmented reality, revealed the power of volumetric capture.
“It was great fun doing it, but still quite slow,” says Williams. “For every 30-second scan, you’d have to go have a cup of coffee and wait for ten minutes while the computers dealt with all the data.”
In May of this year, Williams got a call from Thomas Hoegh – tech entrepreneur and CEO of Garden Studios – on behalf of Founders Forum. This global community of businesses was planning its annual flagship event for 16 June, Founders Forum London, and had an idea to invite a very special guest speaker: president of Ukraine, Volodymyr Zelensky. But organisers dreamed of more than a Zoom call, they wanted him in person – via hologram.
There was a lot of protection – sandbags, checkpoints and airport security
To bring President Zelensky’s message to as many tech leaders as possible, Founders Forum partnered with Brilliant Minds, The Next Web and VivaTech to display the hologram via ARHT Media’s Capsule at these tech conferences. In addition, footage of the hologram at Founders Forum was livestreamed to Dublin Tech Summit, The AI Summit at London Tech Week and SuperReturn Berlin.
“Ukraine has got a huge tech skill set,” says Williams. “It was quite a big industry there, before the war. So tech is an important route to getting Ukraine rebuilt. The idea was that, if Zelensky can speak to the community and get some investment into Ukraine’s industry, they can kick-start the rebuilding of the country quickly – as opposed to building a steelworks again, which takes years.”
Williams got the call four weeks before the event. Hoegh called with a very specific game plan – get a team together, find a way of capturing a hologram of Zelensky, beam it live to the event, then devise a means of distributing it universally afterwards. If they got it right, it would be the first time any world leader in wartime had made an address via hologram.
Finding a hologram
Work started in earnest as soon as the phone was put down. The tech challenges to overcome were twofold. Firstly, it required a thorough and convincing volumetric capture of Zelensky speaking. Then, a suitable holographic display technology was needed. It quickly became apparent that there was no technology available for effectively presenting the Star Wars-style hologram we’re so familiar with. The closest approximation is the 200-year-old ‘Pepper’s ghost’ trick, still employed in amusement parks and stage shows, but it requires carefully controlled conditions to work properly. Experts decided that the video was to be delivered at each venue by Capsule telepresence monitors from ARHT Media. The Capsule monitor actually produces a 2D image, set slightly back inside a frame and manipulated to produce a 3D effect – so not strictly a hologram. Inside the Capsule is a large, vertically orientated 4K touchscreen, which uses ARHT’s HoloPresence technology to display people at life size. The Capsule comes in versions optimised for live delivery and touch-screen interactive playback.
“The appealing thing about the ARHT Media technology was that it allowed you to pretty much plug in and go,” notes Williams. Security was essential. The team communicated internally and with government organisations – the Ukrainian government and Ukrainian Embassy in London – solely through WhatsApp and Signal, never referring to Zelensky or any locations, and private security was hired. We called it ‘Project Tuesday’ and talked about ‘Roger’ instead of Zelensky,” recalls Williams.
In finding the right volumetric capture solution, Williams contacted top-tier content provider Dimension Studio, with whom he had worked when capturing Sir David Attenborough’s performances. But Dimension’s high-end capture required an entire truck. Driving a lorry into the middle of downtown Kyiv and pulling up in front of the presidential offices is not the smartest move when working in a war zone. However, Dimension referred Talesmith to a new volumetric capture company that created a solution able to fit into three Pelican cases.
Capturing a president
Ben Nunez, CEO of Evercoast, started the company with the goal of turning high-end volumetric capture – currently found on Hollywood stages at great expense – into a compact service anybody could access.
The Evercoast platform uses a battery of 21 consumer-grade depth cameras and off-the-shelf hardware, all packable into a few cases – perfect for the job of getting in and out of a war zone. Its systems are licensed to a wide variety of customers, including big telcos, major universities, medical institutions and entertainment companies. The newest iteration of Evercoast technology provides live streaming at low latency, suitable not only for one-to-many broadcasting, but live, two-way communication. Nunez was alerted on 1 June, while attending the Augmented World Expo (AWE) in Santa Clara, California – less
than three weeks out from the target date during London Tech Week.
“When we first got the call, we had no idea what we were getting into,” says Ben Nunez. “They were very cryptic. We were just told that it was going to be in a tricky part of the world to get into.”
We believe it's a seminal momenent in volumetric
It didn’t take long for Evercoast to say yes – and then get the full story. Nunez returned to New York to pack up the system and prepare it for travel to London. The kit was first set up at Garden Studios, capturing messages of support from celebrities and technologists, including Peter Gabriel, Nile Rodgers, Ukrainian pop star Oleksandr Balabanov, fashion
designer Ozwald Boateng, actor and TV personality Amanda Holden and French intellectual Bernard-Henri Lévy.
“Throughout the whole process, we weren’t sure if we would be able to go to Kyiv. The team was trying to arrange everything, from insurance to access inside Kyiv, to support from local resources.”
Running a production without insurance is, of course, a non-starter. And insurance is invalidated going to a war zone, including Talesmith’s company liability cover.
“One person tripping over a cable could have brought the whole company down,” observes Williams. After being turned down by multiple insurance brokers, the security company pointed the team towards an expensive, but adequate medical and life insurance package.
Into Kyiv
With the capture equipment shipped separately via van, Williams, Nunez and the team’s private security advisor – a former British commando – flew into Krakow in Poland. They then drove overnight to Lviv, just inside the Ukrainian border, and then to Kyiv. Information sharing was kept to a minimum, with the team knowing only that someone would meet them at a location in one of the presidential buildings.
“There was a lot of protection – sandbags, checkpoints and airport security. Plus, our phones were taken away,” says Williams.
Talesmith had engaged a local grip and gaffer team to help set up a green-screen cube for the shoot – not essential, but providing a boost in quality. Nunez and Williams got the Evercoast gear in place – until the 10pm curfew – and then returned the next morning to hurriedly finish the process. With an hour to go until shooting time, the team still hadn’t locked in a script for Zelensky. They had only written a loose draft for him to use as a departure point.
“We knew Zelensky was a comedian and had a good sense of humour, and wrote this thing about being the first leader since Princess Leia to do this,” says Williams.
Zelensky has a very young and media-savvy team
Not only did Zelensky incorporate these Star Wars parallels into his final speech, but had a bespoke T-shirt made which depicted a Mandalorian stamping on an Imperial
Stormtrooper’s head.
“It’s unusual for presidents or heads of government to use a hologram to address people,” Zelensky said. “But this is not the only aspect of Star Wars that we are putting into practice – we will defeat the Empire, too.”
“Zelensky has a very young and media-savvy team,” explains Nunez. “The president is 44, his press secretary Serhii Nykyforov is 36 – and the speechwriter was young. It was impressive collaborating with them… Zelensky was wonderful to work with and excited about the technology.”
Evercoast’s system has been refined to a near push-button level, allowing the Williams/Nunez two-man crew to operate the entire process.
The team captured an initial redundancy recording of Zelensky’s speech, in English and Ukrainian, in case the live transmission didn’t work. The arrival of President [Emmanuel] Macron, Chancellor [Olaf] Scholz and Prime Minister [Mario] Draghi for a special meeting with Zelensky ended up taking priority over the live transmission, so it was the recorded version that was sent.
The telepresident
Zelensky’s speech was broadcast to seven conference events, introduced by the pre-recorded celebrity tributes. Alongside it, the true AR version of the speech was available for download. Evercoast’s product is so automated that this was delivered 36 hours after
capture – and the volumetric output was available almost immediately.
He was wonderful to work with, and excited about the technology
“We filmed him, then had the speech on our phones within an hour and a half,” says Nunez. “We’ve spent so much time trying to make our product seamless, that you can just click record, upload, go to the web interface and launch.”
Nunez returned home to the US, and Williams to the UK, but the gear was left behind. An anonymous donor purchased the Evercoast volumetric capture system as a gift to the Ukrainian government.
Nunez had trained several of Zelensky’s media team on how to use the system, and has maintained contact ever since. They are looking towards using the gear not just for further presidential speeches, but also education and other media opportunities, building the nation’s reputation as a digital technology destination for the future.
“It’s a seminal moment in volumetric,” asserts Nunez. “It’s the biggest thing to have happened in volumetric by a landslide.”
This article first featured in the autumn 2022 issue of FEED magazine.
