How ‘Avatar: The Way of Water’ senior VFX supervisor Joe Letteri created a new facial animation system [Exclusive Video Interview]

After changing the visual effects and performance capture game with “Avatar” (2009), James Cameron held a summit with his team 13 years ago to discuss how they could make the sequels even better.

“The biggest part of it was since we didn’t really know was the story was, was more about the process,” “Avatar: The Way of Water” senior visual effects supervisor and Weta director Joe Letteri tells Gold Derby at our Meet the Experts: Visual Effects panel (watch the exclusive video interview above). “So the early discussion was highly technical. It was about building out that pipeline. Later on, once we knew what the story was, the discussions became more about, are we gonna achieve the specific effects that we need — water obviously being a big one — but characters? A lot more characters with speaking parts in this film, a lot more varieties of characters — old, new, young, different species. Overall, we have not stopped working on it since that postmortem, but the actual final push on the film, when Jim started doing the performance capture, was about five years ago.”

Among the several technological advancements the four-time Oscar winner and his team created for the sequel is a new muscle-based facial animation software called APFSA (Anatomically Plausible Facial System) that more accurately captures the intricate movements of a human face. The first “Avatar” utilized the same system that Weta had developed for Gollum for the “Lord of the Rings” trilogy. Though groundbreaking at the time, the system’s limited tools couldn’t eliminate some of the “rubbery” look that would pop through.

SEE ‘Avatar: The Way of Water’: Watch Sigourney Weaver transform into 14-year-old Kiri

“We realized that for the fidelity we needed and for the number of characters, the way that we had been doing it wasn’t going to suffice,” Letteri explains. “We wrote an entirely new facial system based on a neural network that could understand the relationship between muscles and skin and what you’re seeing on the face. And that became the animation tool that we used for the facial system. We would capture the actors which a stereo camera rig, pull information from that, feed it to this network to understand what the muscles were doing and use that to drive the characters. And on top of that, the animators had access to the exact same set of muscles. It was a much more intuitive way of working than we had before. They would do the final polish and anything that needed to be done to put it together.”

In that same vein, the new creatures in “Avatar: The Way of Water” are also based in reality. “Any time you’re doing creature design, you want those touchstones,” Letteri says. “Does the audience recognize something similar? On the first film, we had the direhorses, which were basically horses but with six legs. You try to create some similarity while at the same time you’re bringing something new to it.” The sequel prominently features the tulkun, giant whale-like creatures, one of whom, Payakan, befriends Lo’ak (Britain Dalton). Cameron had drawn exactly what he envisioned the tulkun looking like.

“We would pick it up with our animation team because every character is defined by emotion. The look only gets you so far because it’s never static. Is it going to move believably? Is it going to propel itself believably? What is it going to look like from all different angles?” Letteri continues. “Especially with the tulkun because there are so many shots with Payakan… because he is giant. He’s much bigger than a whale, but Lo’ak is sitting next to him and they’re kind of conversing and he’s only got an eye in frame because that’s all you could fit in frame, so we had to do a lot of animation with just that eye to get the bonding, the moments where they were connecting, moments where there was fear. I think our animators enjoyed working with that because it’s very minimal, so you really have to nudge everything.”

Work on the next three sequels is underway. “Avatar 3” is scheduled to be released in December 2024, and by then, you can expect Letteri and his crew will have some new tricks up their sleeves. “Jim was saying a couple weeks ago [about ‘Avatar: The Way of Water’], ‘OK, well, I think we finally know how to make this movie,’” Letteri states. “Let’s apply that to the next one.”

Make your predictions at Gold Derby now. Download our free and easy app for Apple/iPhone devices or Android (Google Play) to compete against legions of other fans plus our experts and editors for best prediction accuracy scores. See our latest prediction champs. Can you top our esteemed leaderboards next? Always remember to keep your predictions updated because they impact our latest racetrack odds, which terrify Hollywood chiefs and stars. Don’t miss the fun. Speak up and share your huffy opinions in our famous forums where 5,000 showbiz leaders lurk every day to track latest awards buzz. Everybody wants to know: What do you think? Who do you predict and why?

More News from GoldDerby