The Future of Visual Effects with Guru Rob Legato — Q&A
IndieWire turns 25 this year. To mark the occasion, we’re running a series of essays about the future of everything we cover.
As part of IndieWire’s 25th anniversary series, who better to forecast the future of VFX than three-time Oscar-winning visual effects supervisor Rob Legato (“The Jungle Book,” “Hugo,” “Titanic”)? The 2021 recipient of the VES Award for Creative Excellence, Legato has translated virtual production into a live-action, photorealistic methodology for James Cameron, Martin Scorsese, and Jon Favreau. In fact, Legato’s VR experimentation with Favreau on “The Lion King” led directly to the director’s pioneering work with Industrial Light & Magic on “The Mandalorian” series on Disney+. They created the StageCraft platform, which eliminated the need for costly and time-consuming location shoots. Actors perform in an immersive and massive LED video wall and ceiling at Manhattan Beach Studios in L.A., where the practical set pieces are combined with digital extensions on the screens. This allows the filmmakers to generate complex and exotic digital backdrops in real-time (using Epic’s Unreal game engine).
As a result of this game-changer, LED wall stages have popped up around the world: ILM has three in L.A., one at Pinewood in London, and one in Vancouver; Weta and Unity have one in New Zealand; Warner Bros. Leavesden has one in England; DNEG and Dimension also have one in London; and Trilith Studios has one in Atlanta. This has further reduced the size and scope of location shoots, set builds, and crowd scenes, and Legato discusses the significance of the LED wall stage with IndieWire, as well as its future impact on the industry.
Meanwhile, another hot topic that Legato addresses is the future of digital performance, as animation software continues to advance and get beyond the uncanny valley of weirdness. This includes de-aging, which has finally come of age, resulting in more believable, youthful performances. Lola Visual Effects, the industry leader in de-aging and Marvel’s go-to specialist, has perfected the Photoshop-like technique of skin smoothing and shape warping through 2D compositing. However, the emergence of deepfake tech offers a new innovation, in which computer learning software analyzes and combines photographs and footage of young actors to create CG composites of their faces.
The interview was edited and condensed for clarity.
IndieWire: How would you assess where we are with virtual production and the impact of the LED wall stage after the success of “The Mandalorian”?
Rob Legato: We’ve been doing something akin to that, just not as easy, with blue-screen stuff. With the advent of the LED wall stage, now you can pre-make your set much like you would you’d build a set, light it, texture it — you’d do all the various set decorating, and then you’d shoot a scene against it. Only now, the LED wall stage makes it easier to visualize. It doesn’t even have to be a visual effect; it could be an office or a bar or a cave. That means that when it becomes less and less expensive to do, where everybody has the LED wall stage, you could have a greater imagination and not be restricted by budget.
Talk about being able to visualize everything in VR space on “The Lion King” as a photoreal experiment to emulate live-action.
On “The Lion King,” we had myself and [cinematographer] Caleb Deschanel, a dolly grip, a camera assistant, pulling focus, somebody doing the gimbal arm — it was very much like a live-action film, and all of the pluses and minuses. If you’re talking about the future, it’s taking advantage of computer advances and tracking and all the various things, but I think we still like the chemical reaction you have by actors on the set with the director. We’ll never totally get rid of any of that analog stuff, nor should we. It’s just that now you can make a “Lion King” or a “Jungle Book” look more photoreal and be able to shoot it in a much shorter period of time. Without the arduous task of shooting on location.
I always use “The Revenant” as an example of some films that we like to see, but no one wants to make them again because it was so difficult to do. And so physically exhausting for Leo [DiCaprio], when you’re in freezing cold and you’re there all day to get 20 minutes of usable material. He even said, “I’m never going to do this again.” But he still was highly revered and he won the Academy Award. You just don’t have to go through that kind of grief to create the same creativity. It could also be a digitally augmented film, just as powerful visually.
And we’re witnessing this now with the continuing evolution of this virtual production game-changer.
The wall makes it absolutely accessible, which used to be a big sticking point with all the CG stuff years ago. That isn’t the case anymore. You [can] throw up a scene on a video wall, shoot real actors in that moment, move the camera any way you like, and, by the time you’re done, you can move on to another set and not have to leave that studio space. It’s wildly efficient. Regardless of what worlds you could build that are science fiction–oriented or whatever, you could just do regular filmmaking and increase your production value.
But the LED wall is not a one-size-fits-all solution. It’s tailor-made for “The Mandalorian,” but not for “Dune.” And I’m hearing push back about the lighting not being authentic enough.
I don’t think it’s ever going to take over [completely], but it’s going to create a new avenue to shoot arduous scenes. I’m working on this movie right now. We’re shooting a swamp scene in New Orleans. We shut down three or four times a day because of lightning, and as soon as it rains in the middle of the day, all the vehicles get stuck in the mud. [LED walls are] all just to get the look of the scene and be able to shoot 10 hours a day instead of four. All things happen because of money. If I can’t see the difference, and you could shoot a scene that would take us two weeks on location and shoot it in three days, you can overcome your lighting problems and everything else with skill. You can light it so it’s not artificial.
Real-time engines such as Unreal 5 are making all sorts of lighting advancements.
Yeah, and since you’re talking about the future, the future is not immediately now. It’s just heading in that direction. I don’t totally understand it, but this technology allows you to have a zillion more things that produce the images and being able to do it in real time. You know, if I were to teach a cameraman how to walk into a VR environment and bring your gaffer with you, and you light it by saying, “I want a 20K over here and I want it 12 x 12 over here,” they would start to say, “Oh shit, I can light it the way I would have lit it on a major stage.” The result would be the same [but] there’s a push back and a fear.
There’s not that many Calebs or John Tolls or Vittorio Storaros, but [cinematographers] can be adapted to do this. It’s a little bit like when the synthesizers came about. Everybody hates the way they sound. And then all of a sudden, when one guy can overdub and be the entire orchestra — if he’s talented enough, the music that comes out of it is [surprising]. Now we could do more, because there’s no need for an 80-piece orchestra. He needs his imagination and a computer and he can now reproduce something that’s spectacular.
Again, it’s the future and adopting and people are very resistant to change. They fear it eliminates their job as opposed to enhancing their job, [but] the production designer is still building sets — whether it’s physical or it’s on the wall, it’s still the same artwork. The same thing with cameramen. They’re gonna have to light a digital set as they light a regular set. If they do a good job, they can light it almost identically by putting lamps almost in the same spot. They’re just virtual lights, not physical lights that have to be plugged into a wall.
Where do you see this headed in the next five years?
Within five years, we’ll shake out the companies that don’t do it very well.
I mean, I’m not gonna say any names, but I worked with a couple recently that were early adopters. They were able to get over the hump of the technical portion of it, just how to get it so it doesn’t flicker on the screen, what software to use, all the various technical stuff. What was lacking is the art of it. I went in and redirected the art of it and then the result was very good. What separates the men from the boys, so to speak, is the photographic and artistic and production value skills, and that will probably [happen] within five years. Almost every TV show, every major studio, or every stage that goes up will have one LED wall stage. Think about the efficiency of being able to shoot five scenes with my guest actor by switching sets by the flick of a switch. I can get Tom Hanks to be in my movie and only hire him for three days to do a part that I otherwise couldn’t afford.
Let’s talk about technology as it relates to performance: digital makeup and de-aging.
The advent of digital makeup makes it look photorealistic, which is about ray tracing and global illumination and all the various things. Those really lend themselves to blending the line between what was traditionally done with some limitations, to what could be done now with no limitations. You could have any actor play any other actor, could be any character, any size. It’ll get to the point, within the next few years, where it’ll be pretty seamless. [You’ll have] an actor playing another person without having to get into six hours of makeup to do it. The artwork of it and all that stuff will be here to stay; the ability to add it after the fact and make it photorealistic will be a common thing.
I think there’s never going to be a moment where you can replace an actor with a synthetic actor, but there could be a person who creates a character and the combination of the two becomes a movie star. It could happen where a small person could play an action star and no one would know. I can’t mention names, but there are action stars who literally can’t do much action and they’re replaced by doubles with some kind of face replacement. So I could imagine that there’ll be a combo movie star where the look of someone and their acting-ability avatar are very different.
Plus, de-aging keeps getting better. Look at what Lola did with Mark Hamill as Luke in “The Mandalorian” Season 2 finale. They combined their 2 1/2D approach with deepfake tech as part of their prep in compiling photographs and footage from “Return of the Jedi.”
[Deepfake’s] a remarkable piece of technology, from somebody who’s not totally an artist, that’s more natural looking and less expensive than “The Irishman” [technique at ILM that’s markerless and light-based]. And that’s going to take over the digital makeup portion, where, if you don’t have to sit in the chair for five hours and have things glued on your face, it looks just as good.
The use of it in the right hands is incredible and will aid in making it now very easy to do. You’ll see this deepfake stuff [use machine learning] where you’ll have Bill Hader on one of the talk shows doing his great impressions, and they very subtly change his face to look like Arnold [Schwarzenegger].
The line is blurring and blending and people who work in my field have to become more like a filmmaker. The look is the same and the art is the same. And, you know, it doesn’t replace anybody’s occupation.
