by Michael S. Palmer
A few weeks ago, while we touring Skywalker Ranch and screening 'Raiders' on IMAX, Disney invited HDD to Industrial Light & Magic to talk about 'The Avengers' on Blu-ray. ILM, for those unaware, is visual effects company, which George Lucas started in 1975 to produce a little indie flick called 'Star Wars'. The company went on to pioneer computer-based visual effects on a small cult film you've probably never seen, 'Jurassic Park', and each year is involved with many of Hollywood's biggest blockbusters.
While there were many vendors involved with 'The Avengers', you could say ILM was the lead vfx house, playing an integral part in working with Joss Whedon and Marvel to update the fully-computer-generated Hulk character as well as the new Iron Man suit (every 'Iron Man' film needs a new version of the suite) and a fully rendered digital version of New York City. We'll get to all that in a moment, but first I wanted to talk a little bit about:
This free app actually debuted at Comic-Con this summer and has already had a few updates, but won't be fully functional until you buy the Blu-ray (or a digital copy) on September 27. The app's slick designed was created to mimic what it would be like to have access to Nick Fury's computer and the S.H.I.E.L.D. database. Once booted, you'll find a few different folders on the desktop. Item 47 was a Comic-Con scavenger hunt. The Avengers Initiative Timeline places everything from the Marvel Cinematic Universe - Phase 1, everything that lead up to this film, in chronological order. The S.H.I.E.L.D. Personnel Files are breakdowns of all the characters (most of these are not active just yet).
In the Comic Book Origins section, you'll be able to see the comic book panels that inspired moments in 'The Avengers' and even buy the full comic via the Marvel Digital Reader. And, when you buy a copy of the film, you can Sync it to the app and watch exclusive behind the scenes content. We saw a couple visual effects breakdowns where you could scrub forwards and backwards through evolving shots.
After looking at 'The Avengers' Second Screen Experience, we settled in for a series of round table interviews with three visual effects crew members. Here's an hour's worth of behind-the-scenes goodies about what went into creating the awesome world of 'The Avengers':
Marc Chu -- Animation Director, ILM
Robert Downey Jr's Iron Man suit has evolved over the last few movies, in part to make him more comfortable on set. Do you approach enhancing a partial suit any differently than character animation or a fully CGI character?
Marc Chu: You know, it's a different challenge and after the first movie, when Robert saw, oh, wow, look at that, he realized he could not wear as much as he needed to. It gave him freedom. He was like, wow, I don't need to wear half of the suit and I could get the same effect. And that allows him to be more comfortable for sure because that thing is not built to be worn any long duration. It'll pinch you, I can't even put the helmet on. I tried one time. We got the helmet up here and I was like, oh, good, I'm going to put it on and I couldn't even get it over here. They have to cast - the first person they had to cast, the stunt guy had to just be able to put the helmet on and that's Clay, Clay is the stunt guy who always plays Iron Man. But the approaches to doing, for this movie, when he takes the suit off, he didn't even wear anything for that. He just wore what he was going to be revealed in.
The t-shirt and the pants. So that's a different challenge. We have to imagine his movements exactly and then place the armor on him. And then figure out ways for us to take the armor off. And everything, I think when they try to plan these things, looks great on a drawing. When they actually build the platform and you go, the platform doesn't even look like the drawing, it becomes our responsibility to figure it out. We need to figure out where the arms go, what gets taken off, how the mechanisms work and fortunately, we have lots of people who are well versed in robots and things to create that from scratch.
You mentioned tailoring the Iron Man CG with Robert Downey's movements. How did you assist him, or did you need to assist him on how to walk a little bit like he's wearing the suit because he had nothing on at the time?
Marc Chu: Well we never interfere with what Robert will do, so he's just going to walk the way he does. And we just want him to be natural. In the end, all those shots are tight shots, right, they're all going to be like this on his body, on his shoulder or whatever. So it wasn't that important. When he first lands and he starts walking, that's completely CG. And that's something that we did on our mobile cap stage. And that's us interpreting his walk and putting it on the CG character. And then once it gets to him, it's just a matter of brute force, imagination, let's figure out how this thing works.
There's a misconception that motion capture (MOCAP) means that tradition key frame animation is no longer required.
Marc Chu: For this movie, for Iron Man, for Avengers, for the Hulk, it doesn't matter, it's a combination of - sometimes it's MOCAP, the suits that we use on set when we capture the stunt performer's actions. Sometimes it's MOCAP on our stage, sometimes it's pure key frame animation. I think each one of those is a tool. And each one of them has their place to be in the movie. And the sequence where he's running after black widow and he's running down that little grating, we're using a combination of key frame shots and motion capture shots. And we're cutting between, back and forth to those things. And you can't tell. Each one, I think has their place. I think motion capture is a valuable tool but as you said, you'll always need an - I firmly believe it's always the case. You will always need an animator to pulse it out.
Either because you need to preserve the size and the weight of the Hulk, for instance, so you might need to adjust it, different proportions. It's always going to be an animator that gives it that level of final polish. With the actions, with the face, especially, we had Mark as a reference, we his face as a baseline for performance but it's always going to be the animator to go, you know what, I could push that expression a little bit more in the Hulk brow. And sometimes they would ask for to be a little more wild or this is the specific tool that we give him. He's fully CG. We can start with Mark's performance. If that's not enough, we can push it. Or we can completely change it based on what we've captured with him before. We've captured all his expressions so we can come up with a performance that's from scratch that is going to look like Mark because it's based on his expressions.
It feels like the definition of performance is changing. There really is a combination of many people's skill sets and many people's opinions and sort of to get to that final result.
Marc Chu: It is. And Joss came up here to tell us, he gave us a low down. He said, this is, this needs to be the best Hulk ever. Everybody here, when we created this, we worked on the Hulk, we knew everybody would be judging that single character in all our work. It didn't matter how good the character looked or how good our digital doubles look or anything. It was the Hulk that was going to sell everything so we knew everybody was going to be critical about that and when he came up here, he told us, he actually read us a letter from Mark and it said, basically don't be constrained by what I do. If I do something, that's great, perfect, you can use it, but don't be afraid to push things if you need to. And that kind of gave us the freedom to go, okay, we don't need to follow his performance 100% but in the case that Joss wants something bigger, we can push those things and not be afraid that we're ruining his acting.
We don't want to insult his contribution. And so that was liberating and then it becomes a process of, does it look cool enough? We had Hulk dailies, specifically just for Hulk shots and that consisted of every discipline, so we would have models, painters, rigors, animators, all in one room, the VFX supervisors and there was no restriction. It was, "does this shot look cool? You as a painter, in charge of the skin, do you like the animation? Does that work for you? So we're all approaching it from, is this the best product that it can be? Is this the best Hulk that it can be?" So that is a lot of voices ultimately going into a character.
But it's exciting when you guys pull it off like Davy Jones.
Marc Chu: Yeah, I was the Davy Jones lead and that's still one of my proudest things I worked on. And this is sort of the evolution. It's like that started us. Okay, we have Bill Nighy great performer, and then we have Davy Jones. Kind of monster but there was a lot of interpretation there. Now it's something that's a little bit more hinged on an actor's character characteristics. And from Davy Jones, I also worked on 'Terminator Salvation' where we did digital Arnold. Each thing teaches us a lot about making it look real.
That's an interesting challenge, because we all know what he looked like in 'The Terminator'
Marc Chu: And that was a toughie, and the only footage that we had to work with was the footage from 1984. But for Mark, for the Hulk at least we got to capture him, this is the Mark that we want. So we poked and prodded him, took all sorts of photos until he was sick of us. But we got everything that we needed and I think when you hinge a character off of reality and you get as much detail as we did with Mark, I think you see that in the final product. If you can go up to the skin, you can see the little stubble and the skin going in, little ingrown hair, moles, I mean it's - every bit of Mark is there.
What's more challenging, working on organic characters or something mechanical like Iron Man?
Marc Chu: I don't know, I'm kind of a geeky guy, so I like mechanical stuff too. And for Iron Man it's always like, let's look at reference, let's look online like when he's putting this little thing on the pipe that diverts energy at the beginning, we looked at a ton of stuff that people were doing with underwater, I don't know. I don't even know what they're called but let's look at all this pipe cutting reference underwater and see what they do. Okay, let's make a conversion of that. It's - I think the more you always kind of hinge things off of reality, the more things look believable. For the Mark 7 suit, when it kind of unfolds and sticks onto him and transforms, we wanted to make sure that didn't feel too magical, that the volume of the pieces didn't feel like they were coming from nowhere. Like a magical tortoise of a suit, I don't know.
That's how I felt the suitcase suit was in 'Iron Man 2'. It felt like it was unfolding from nowhere. So we kind of paid attention and wanted to make sure this felt like it was physically possible, that this thing, X amount of size, could actually hold a suit and it didn't actually. The guys upstairs are crazy technical.
Do you think that will ever be another leep forward in visual effects, akin to what happened when ILM created the dinosaurs for 'Jurassic Park' in 1993?
Marc Chu: I don't know. I would hope there's something like that that could make me feel that way. But I was the same way. I wasn't working here when 'Jurassic Park' came out. And when I saw that I was like, oh my god, I want to be a part of this. And then I think the next big thing for me was Davy Jones. That was one where I felt like, people can't tell how we did it, and that's always the Hallmark of innovation. It creates, is it real, is it fake? Is it half real, half fake? I don't - I'd like to think there was something out there, I don't know. I feel like we're making incremental steps but, yeah, I don't know.
As each new challenge arrives, is part of the thrill knowing you're going to get put in a corner where something's never been done before?
Marc Chu: I think that's how ILM approaches the work that they pursue. We want to do something that is ground breaking, that you haven't seen before and getting to do the Hulk was one of those things. And it's, okay, let's do a photo reel, Bruce Banner, transforming into photo reel Hulk, a character that needs to act and performance next to live action counterparts. I think the next step will be, can you fully - can you really make a CG human and not make it seem CG? That's tough. It's tough, I mean you have to uncanny valley and everything to fight against.
Well, really, it's amazing stuff.
Marc Chu: Thanks. It was hugely fun, and I got to work on first two Iron Man films so it was, for me, a real treat to be part of this series and see them pull it off with 'The Avengers'. It's like, you're going to do what over how many years? And, oh, you did it.
Jeff White -- Visual Effects Supervisor, ILM
When working with Mark Ruffalo, did you have that curve with him where he took some convincing, but then after he got into the MOCAP process he started to realize how much of his performance was actually going to be in the Hulk?
Jeff White: I think so. You know the nice thing is we started with him coming up and just working on it talking with Joss, but not for the movie at all. Really all we were trying to get out of that day was a sort of training library of facial shapes so that we could accurately, you know recreate his kind of major expressions.
But for the rest of the day he got in the mocap suit and, you know we were doing kind of a live retarget onto an old Hulk model. Just so he could kind of play around and feel like what is this performance capture stuff really gonna turn into in the end. And then, you know I think it’s difficult on set, you know those first couple of days when everybody’s in cool costumes and you’re in the gray spandex.
And we actually had a tiara on his head that we used to track his head motion and then dots all over his face. So it’s hard – I would think it would be hard for any actor to kind of get beyond that. But once he got into I think it really started to work. On set we also did a couple hours with him where we were just off to the side and had him run through the shots without any tracking.
It was just, you know just kind of wild performance and got some really great reference material out of that. So at that point I think he was comfortable with it and we had the edit kind of sort of locked together by Octoberish. And then he came back up to ILM again with Joss and they did the full like mocap suit, the dual head camera, which is its own sort of uncomfortable part of it. But by that time I think he was pretty comfortable with the process.
Was Hulk was one of the last things finished for the movie?
Jeff White: It really was. I mean I think every shot was so much of a work in progress right down to the very end. And, we often get asked ask like how long did it take us to create the Hulk? And the reality is it took until we finished because even though the initial development took maybe six months to get the character together, every shot we put him in he didn’t look good, you know.
So we’d get into a shot where he’s turned and the camera’s over here. And now his shoulder is enormous. It looks like a pumpkin and his head is tiny. So we’d have to sort of re-craft him for each individual shot to make sure all the proportions look correct.
I think part of the fun -- Mark has done a lot of different kind of movies and a lot of independent movies and this is a huge big budget visual effects film -- was trying to make the process still feel somewhat natural. And he really got into it. As you said, once he got comfortable with it and could see how he could drive the performance, I think really gave us the best performance that we could hope for.
Joss worked very closely with the animators because there’s always that process of like, okay, now you’ve got Mark. But Hulk’s facial proportions are bigger. So it’s not a one-to-one translation, there’s that process of like what does that look like when it’s actually applied to the Hulk? And once you start to light the Hulk.
What were you most eager or excited to change about the Hulk for this film?
Jeff White: I think, you know we were really excited about the initial previs that we got because Joss clearly understood the kind of performance people wanted to see the Hulk do. And I think, for us, we’ve had a lot of developments on the technology side of it, both in terms of performance capture and in terms of rendering and muscle simulation and everything. So for us it’s kind of this ultimate okay, let’s throw everything at it that we’ve got.
My background is working as a character TD -- rigging and simulation and anatomy -- so you can’t think of a better problem to try and solve than working on something like the Hulk.
I was the most worried about it, but when we saw the initial designs where we’re going with the more de-saturated skin and we’re incorporating Mark Ruffalo into it, that gave us some really tangible goals in terms of how we needed to make this Hulk different than the previous ones.
Did Joss have this all lined up exactly what he wanted? Or did he just have a few ideas?
Jeff White: They had an art department down in Los Angeles . So we started with a piece of artwork, a maquette – and that was a great basis to start from. But there are a lot of questions and problems that come up when you try and translate that into 3D and put it on screen. So that’s where we were able to really get involved in the process.
It really came down to some initial really good design decisions by Joss in terms of what his body structure looks like, letting us put body hair all over him. That’s something the fans could have reacted very negatively to. A sort of hairy-chested Hulk. But when you zoom into his eye, everything’s there down to little nose hairs. And I think that lends a lot of credibility.
Plus we knew -- we’d already saw in the previs -- there was gonna be a foot close-up, face, hand. So we never said okay, this area you can go more low-res. We built him top to bottom to be able to hold up really close to camera.
Is it challenging maching a performer's body and size to a digital character that, in Hulk's case, is much bigger?
Jeff White: I think that’s where sometimes it can be kind of glossed over, the animation contribution. But that’s where that all gets resolved. Because you use the motion capture as a great basis. So it gets you a lot of the idiosyncrasies and the kind of things that you wouldn’t normally start key framing, where maybe they shifted left when you expected them to go right. It just kind of messes up things in a way that’s very organic and natural. But just applying Mark’s motion and speed to the Hulk doesn’t work.
We tried to figure it using our own body volume and scaling up to Hulk. He’s probably somewhere between 1,200 and 1,600 pounds. So you can imagine anybody that’s between 100 and 300 pounds isn’t gonna be able to accurately represent how fast he should stop and how fast he can get going. So that’s where, I think, using the performance as a basis, especially the face, translates better than its animation.
Key frame, massaging the mocap, that really gets the same performance. I mean there were a lot of shots where he’s doing things that Mark couldn’t approximate at all. Like jumping from building to building and that’s where it’s all key frame animation.
What kind of feedback do you give Mark like in order to sort of get where you need to be for those kind of things?
Jeff White: We gave him props to work with. Like where we’re working out the slamming shot, we’ll give him like a heavy sack full of bean bags or whatever and he can kind of play around with that. But in the end we find that it’s best to just let them do what they’re gonna do and in that process, if they’re really exerting themselves, you get great facial reference.
We spend a lot of time studying how, as he got angry, the flush that comes under the cheeks and the kind of whitening as the skin compresses. And for us, that stuff is just gold. We take that and say okay, how are we gonna incorporate this into the Hulk?
In the end, especially because you’re doing capture maybe in one place and then using it a different way in the shot, we just try to let him do the performance as it works best. But we did have like hulk hands so when we’re on set and Thor’s holding onto his arm, we have these giant foam hands that they can hold out.
And then on set we had a stand-in. His name is Troy and he was a stunt guy but he was huge and we’d put him on a little platform so he could move around. And that was great for eye line. We also had a Hulk backpack, which Mark could put the head up to the right height because just framing is a huge issue. The operators on tight shots, they’re like okay, Thor’s here and we want Hulk there. Getting that correct is a difficult relationship.
Now that we've seen this version of the Hulk, I don't think they can go back to a man painted green for the possible upcoming 'Hulk' TV series, but would you be able produce something of this quality for TV?
Jeff White: I think we’d have to have a lot of banter.
I think from beginning to end we got much faster at doing shots. We learned a lot about the process and about how to light him and how to make him look good. One of the things that was interesting was when we started off we kind of took our typical approach of really art directing the lighting, like rim, rim, rim, you know super stylized. And he really looked fake and popped out. And what we found is that we ended up having to kind of flatten out his lighting to get him to sit in there with the rest of the Avengers.
So I think after learning things like that, if you’re talking about a TV schedule and how fast you need to turn around production, it would be a matter of trying to build off everything we did for the movie and then get him in there, get him lit. You wouldn’t have time to do all the really detailed shape and frame corrections we do.
Wouldn’t it almost seem like you guys would have to follow more the animated model than a live action TV series?
Jeff White: No, I think that would be one of the big challenges. Like how do you turn that much animation around and still have it be believable. Because there is animation, then simulation, then hand correction after that. And I think where we found the biggest time suck ends up being is all the facial work, getting the eyes to look right and then how much that changes once you start lighting him.
So, as we were animating we would try very early on to start putting him in the lighting, like in the underbelly of the Helicarrier. You look at an atom render and it’s bright and you can see everything. And then you drop him in there and it’s like all of a sudden you can’t see anything.
And we had a huge problem lighting his eyes because he’s got that huge Hulk brow. So all the top light just gets shadowed and so we ended up having almost like they do with stars. Lghts that went right in here to his eye sockets. But where we’re spoiled is that we can exactly say where that little highlight on the eye’s gonna be. And that makes such a big difference as far as where you think his eye direction is.
To create a CGI version of New York City, you used a lot of digital still images. Were they special cameras?
Jeff White: No. They’re cameras you can buy. They’re Canon 1Ds Mark IIIs, so they’re high-end cameras, but anybody can purchase one. And then we have a special rig that’s kind of an arm out and an arm up. And there’s an automated one and kind of a hand-click one that just goes, shoots a frame, turns, shoots a frame, turns, shoots a frame.
So when we hang it off the side of a building you have a little computer controller. You hit go and it starts shooting the 360 degrees. Rooftops are the trickiest part. We always have to have grips up there that build the rigs for us to hang it out the window or off the side of the building.
But the other hard thing is lighting in time of day are difficult. So if we get street closure for a Park Avenue viaduct, we’ll do one side and then have to go someplace else and then come back at the same time the next morning and try and do the other side. And if it’s overcast then you almost can’t shoot. So it becomes a very time consuming process to shoot all that photography, but in the end it is just standard prosumer cameras that we’re using.
How much could you later carry over into something like the second Avengers movie?
Jeff White: Yeah. Yeah. Absolutely. You know if they wanted to set more scenes on the Helicarrier, we’ve got it built now. We’ve got a damaged version, so there’s definitely the possibility of doing a lot more with the Helicarrier if they wanted to in the future. And the Helicarrier was fun because the way they shot it was when it’s up in the sky, it’s usually essentially an all CG thing. But down on the ground we were at an airport in New Mexico.
So they had a strip of runway with the logo on it and then some shipping containers that represented the conning tower. And so in the end there’s probably this much of the Helicarrier that was real. But just having one piece of runway that was real kind of grounded the rest of it. And so we could keep pointing out wow, our stuff doesn’t look right yet. But yeah, that’s all definitely stuff we could carry forward.
It seems like you have changed Iron Man suit a little bit for every movie. Would the same tweaks happen for the helicarrier or the Hulk?
Jeff White: Well hopefully Mark Ruffalo is the Hulk again because otherwise we’d be changing it a lot. Even as far as we got with the Hulk there are improvements we’d always want to make. So the nice thing is now we’ve got a really great basis to start from. We can keep pushing some of the things that we wanted to get to, to push it even further.
As far as the Helicarrier, I’m sure there’s more that we could do on that. That was such a major undertaking that you end up getting it someplace and then building it out for the shots. So I have no doubt that if we needed to use that again there’s a lot more improvements we could make to it. But it is one of those things where in this movie you get up in running and then in the next one you could certainly improve it.
How would you improve Hulk?
Jeff White: A lot of the hair was difficult, especially his hair design and how different it came out in each shot. I think there’s more that we can push as far as the simulation of his skin and the muscles underneath. And it’s always a matter of the level of detail that we can incorporate into him.
And some of the shots, especially where he’s at a mid-ground, those are some of the toughest for us to solve in terms of the sweat starts to look a little plasticy and things like that. If we can start from the basis we have now – there were shots like the close-up smile where we got a long way, but we wanted to add little spit bubbles in between his gums and things that we see in photo reference that there just wasn’t time for. So I think it would be great to have another opportunity to keep pushing that forward.
What was the most challenging shot to complete in the film?
Jeff White: Definitely the "tie-in" – we called it the tie-in shot, but it’s essentially the part with all The Avengers characters come together. It was right in the original previs and we knew it was gonna be huge. And when you see it, the only plates that we started with were Black Widow kind of riding on the chariot, Hawk Eye on the rooftop, and then Thor hitting guys with the hammer at the end. Everything else we had to create, including Captain America. We had shot plates of a real Captain America, but the camera move was too slow and the momentum of the shot kind of died in there.
So a big problem of that one was keeping momentum. Like even Hawk Eye on the rooftop, as we come up there – they had built a set of the corner of the building but again, the camera kind of died there and the shot really slowed down. So we actually ended up replacing the set altogether with a CG rooftop and then had to roto him off there. And that one took from the beginning and we finished it the last day and it was right down to the wire. It was such a huge shot to pull together.
It’s a great shot.
Jeff White: Yeah. Thank you. I don’t know if you saw the breakdown reel, but there’s a neat breakdown of it where you can see how little is real and all the different moving parts. It was a great shot because we could really feature everything about New York that we’d built. All the destroyed cars and the fires and all that stuff. So it was a lot of fun.
Are these the things that really satisfy you, to be given this kind of almost holy grail of a challenge?
Jeff White: Definitely. I mean especially with the Hulk on this one. I mean the nice thing about New York is that nobody really talked about it. So that’s like okay, there’s a win.
The Hulk we knew everybody was going to be judging and so I think when the film first came out and the reaction seemed to be positive, it was a big sigh of relief. Because there’s so many different ways that that could be critiqued and that was – as you said, that’s kind of the holy grail. It’s like here’s something, it’s incredibly difficult and it’s gonna get judged pretty harshly, so good luck. And to have it come out as a positive result is a great thing.
Jason Smith (Associate Visual Effects Supervisor, ILM)
We were just talking about New York a few minutes ago, but what kind of research did you for New York, in addition to all the photographs?
Jason Smith: We were lucky to have a few New Yorkers on the crew, so they kept us honest. You know, we’d be developing stuff and they’d say “no, that’s not New York.” So we had them to help us keep things in line. And the photography was a huge thing. Even photography that we didn’t use for our assets, we looked through for what was actually out there on the streets because we had to remove everything from the street level like: the cars, the trashcans, the traffic lights.
Anything that would stick out and kind of give away the fact that we were presenting these images on simpler geometry. And then we had to replace it with CG things. So looking at those photographs, we were able to make lists of things that we would feel like would dress up the streets the best so we’d pick like our favorite hot dog stand and the six best construction barriers and three fire hydrants that we really liked.
And just came up with this big list of things. And tried to keep in mind that we never wanted to see the same thing kind of repeating or see it twice. And then, after creating those assets, then we dressed out all those streets with stuff that had that New York feel. But in the beginning, it all came from inspiration from those photographs. So I think in that way, we tried to keep it in-line with the true New York feel.
In recent movies like 'Dark of the Moon' with Chicago or this movie in New York, you’re dealing with things that we are intimately familiar with from our every day life. Is there is a higher degree of failure if you miss?
Jason Smith: Yes. I think that that is exactly true. And the fact that so much of this movie took place right in the viaduct in front of the Grand Central Station, such a recognizable spot. Another challenge along those lines was the Chrysler Building. You know everybody knows exactly what the Chrysler Building looks like. How it’s supposed to react to light and everything. So creating the digital Chrysler Building really was kind of a challenge in that way that we had to get the warble of those panels just right, we had to have just the right amount of weathering.
And they’ve done kind of patching on it over the years: there’s like these little lines of tar that you don’t know about until you really start to look for them. But if they’re missing, you can kinda tell it’s just not really the Chrysler Building. So there were a lot of things like that, as we were working on the New York stuff, that we kept looking back at the pictures of New York to make sure that we were nailing it. And then asking our New Yorkers too: “is this okay? Does that look okay?”
Another thing we did is we drove around a car with ultima arms sticking up filming footage of New York as we drove around. And we had that as good reference for what the exposure level should be and everything. And that was just priceless. Later, when we’d be doing shots, we could go back to that footage and say “okay, this building should actually be a lot dirtier. And this one over here should actually be a little bit cleaner.” And just use reality as a little bit of a guide there to try to make sure that we were doing it right. If you do everything from the hip, you can get something, but to do it really right and to really nail reality, you’ve got to be looking at the reference all the time.
If you could improve on one thing, what would it be?
Jason Smith: I have to keep those things secret.
No, I think that when you work in visual effects, every shot that you finish is kind of getting ripped out of your hands by the fact that it’s got to be done and you’ve put everything that you can into it. I think there are always things we can improve, but I’ve got to say that I was very happy with how the work came out in this, especially when you go to the movie and you see everything cut together with the story and everything. I was really, really happy with how it came out.
What is your favorite shot?
Jason Smith: For me, I have to say even from story boards my favorite shot was the shot where we fly around the Avengers and you see the Hulk roar and then you see each Avenger in turn and Iron Man is landing. To me, that’s just, that’s the comic book moment we’ve all been waiting for. So that’s probably my favorite.
It was just announced today, but how excited are you for Avengers 2 (set to release in 2015)?
Jason Smith: Very excited. I mean because after going through this process with Joss and seeing that there’s somebody out there who understands how to take these characters that we all love and make it work without getting too serious or too light-hearted about everything; of really weaving it together in a way that you care about the characters, you can take them seriously but still have fun. I think the idea of having that happen again – especially now that we’ve been introduced to them all working together. You know, what kinda stories might happen? I think it’s very exciting.
How did you co-ordinate with all the different visual effects companies?
Jason Smith: I think that came down to Janek Sirrs, who is the visual effects supervisor overall on the film. He really helped keep us all in line because he had visibility of every shot at any given time. So it was not uncommon to get a phone call that the skies in the helicarrier sequence that they’re cutting with the skies from another company right before that need to match up. So then we’d get on the phone with him and those other companies and make them match.
Another example is all the digital doubles like Captain America, Thor, Black Widow, all the digital versions of the Avengers that we built, we developed the assets and then gave them to Weta, who then used them. And in one case, at least the helicarrier, they damaged the wing and then gave that back to us. So I think the different companies on the film were very interested in making this process as smooth as possible. So I would say that it really came down to Yannick making sure that we were matching. And he would call out those areas where we weren’t and we would get in line.
Do all the companies use the same type of software?
Jason Smith: No. So for example with the digital doubles, for example let’s say Captain America, we’ve developed this thing where we feel we’ve hit it, we’ve got this photo real Captain America. We would send them our "turntables", like our result, plus our model and all the paint and textures and things and the lights that we used to get that rendered. But they might render it in a different renderer. And really, at the end of the day, all they have to do is match our output. And they know they can because they have the same model and textures.
So what happens afterwards as far as stowing the assets for either future productions or for archives?
Jason Smith: Yeah. Marvel has all those assets. You know they keep their own archives of everything. So I think they’re all ready for what happens next.
Is that getting more and more common to have manyVFX houses work on these giant films?
Jason Smith: Yeah. And I think that sometimes that’s what it comes down to is that there is a company that is really good and fast at iterating on a certain kind of work. And sometimes it comes down to capacity; that there’s just only a certain amount of time and a given house only has a certain number of artists during that time. So then you have to split up the work. So yeah, it happens for a lot of different reasons. But I definitely think it’s the way that things will continue.
What's it like, after working on a film for years, to finally release it to the general public?
Jason Smith: It really is insane because you go from everything being a complete secret that no one can see and no one can talk about to I’m walking through Best Buy and Jumbo’s blowing up a building on the big screen TV there. It’s like you have that initial reaction where it’s like “no, this shouldn’t be! It’s okay.” So again, it does kinda, you kinda feel like it’s ripped out of your hands but the fact is that it’s got to get out there and you’d love to keep massaging things forever. But it’s a very good feeling when you sit in a movie theater with the crowd and see them react to what you’ve done. It’s great.
You guys have built characters and environments. Is there something you haven't gotten to do that you're hoping to try?
Jason Smith: I really feel there’s a lot we can do in character work. So I’m hoping that we get more character work; things like the Incredible Hulk where we can show off the ability to pull off those things.
What would you do different than what you did in the Hulk?
Jason Smith: The Hulk only had one line in the film so I think that there is a challenge with dialog that would be great to explore with digital characters more. We’ve done that with Davy Jones in the past. But I think that’s, as far as looking at something that we haven’t done before, that’s kinda difficult because I think almost anything you think of: like giant robots, yes, explosions, yes, fire, yes, water, yeah. You know, almost everything that you can think of has been done. It’s just a matter of what can we do that we can take it to a new level. So it’s something we’ve already done but maybe there’s another step that we can take there.
I remember when I was a kid and I went and watched 'Jumanji'. Then I saw the making of, and they had done hair on the lion and I remember thinking to myself, all right! They’ve done hair. But that was so early. That was very, very early on and so many advances have been made in hair since then. And we’re still working on hair. We’re still working on skin. We had brand new eyes for the Incredible Hulk that were more anatomically based eyes than we’ve ever done before.
So it’s always getting better, even when we’ve done something. There’s always something new we can do. Like on 'The Incredible Hulk' for the first time, he had beard stubble. And when we looked at his face, we realized it just looked CG. It looked like the stubble just goes to the face and stops and then there’s green skin.
And then we looked at photos of people close-up and you can see, even though you think of skin as opaque, you watch that thing go into the skin and you see it underneath the skin. You see it continue. You can actually see the follicle, if you look close enough. And so with the Hulk, we had to incorporate that. We had to say all right the light bouncing around, the scattering inside his skin, darken it right at the root of the hair. So it’s a bunch of little incremental things like that where we notice some small piece of reality and say okay, let’s focus on that for a second. And then move on to the next one.
For facial hair on the Hulk, how do you decide the scale when you blow that up?
Jason Smith: Yeah. So in the Hulk’s case, we knew we had a shot that went kinda from lip to, actually lip to eyebrow early on in the movie. And then later on in the movie, we had something going from about chin to forehead. Right? And so when we started developing the asset, we put a camera – even though we didn’t have the camera’s on, we put a camera in about that position and started looking at the asset every day that close up.
And it was in those moments where we realized – and it was even less in the stubble than up on the eyebrows, we realized that you got this black hair and it just goes and stops in a way that your brain just says immediately “that’s not right.” And you would think who would obsess about something like that? It’s insane. But I think that to trick the brain, the brain is so subtle in what it notices that stuff that you don’t even know you’re noticing, but if one thing is an iota off, you get into that creepy realm or the realm where you know it’s CG. So yeah, we based it on shots and just kind of tried to match the camera early on.
Are there any shots in New York that you guys eventually had to scrap because of time?
Jason Smith: No, I think there was back and forth throughout the whole process about what action was taking place and where it would be. But yeah. I think the fact that Marvel and Yannick actually, Yannick Souris, had planned out where this action would take place and what photography we needed, with us early on and kind of really put that investment in ahead of time, I think really paid off.
Does the exponential growth in computer processing accelerate what you guys are able to create?
Jason Smith: I think that as the further that you go in making these super realistic creatures, the harder it gets. So any kind of exponential growth in the tools, I think just gets eaten by that. The thing that I’ve seen effect it more is the talented modelers, the talented animators, the talented painters. It’s really the talent of the artist that seems to be creating those leaps more than the technology.
Is that because they just see a need? Like you said you had to do eyes for Hulk. So is it just like: “I’ve just never been content with eyes; let me figure out a way to make eyes better?”
Jason Smith: Yeah. And we’ve always had the tool to make those eyes but it takes an artist saying “you know what? I’ve been reading up on eyes in my spare time. And they’re actually not like you think they are. They have this shape and it folds under like this and does this.” And then you can so “oh. Well, let’s invest that time and do that."
And again, you’ve always had the tools to do that but it takes a great artist to say “hey, I want to call out that we really need to do this, this and this.” So that’s what it comes down to. And I think that on the Hulk, we had a lot of people that had backgrounds in practical effects. You know the guy who painted the Hulk used to be a painter in the model shop down at ILM. The guy who modeled the Hulk and did a lot of the beautiful, sculptural art work on the shots at the very end that really give it that level of comic book pump? He’s a sculpture. So a lot of these people come from these art backgrounds where they haven’t just always lived in a computer and they know to depend on reference.