Seeing is not believing anymore

After devoting the whole week to reflecting about the industry of entertainment and the different methods it uses to tell us stories, I would like to contemplate today how the advance of the technology has enabled us to tell stories in ways that would have been unthinkable just a couple of decades ago. The other day I reflected on the changes in the technology of the distribution (in particular, the streaming platforms) but today I want to focus on the technology of the production, which has seen incredible development in recent times.

Narrative has traditionally had, as I pointed out yesterday, two different approaches: one is fundamentally visual, showing us the situations that are being narrated, while the other codes the situations into words for verbal recount. It is a well-known fact that a visual impression is much more effective that a verbal one (thus the adage "an image is worth a thousand words"), but it is equally true that images are much harder to produce than words (or at least they have been until ten years ago, now that everyone carries a camera in their pocket), since speaking requires literally no materials, while a graphic depiction requires, at the very least, a physical medium to support it (e.g. drawing in the dirt) and it scales up very quickly depending on how true you want the representation to be.

Photo: Y U

One implicit agreement between the artists and the public is that, for certain purposes, a reliable graphical capture of the events was not strictly need and instead an iconic representation was acceptable. When the first painters started to draw stick people on the walls everybody understood that they were not depicting real stick people, but instead each stick person represented, in an iconic way, a normal person. And this tacit agreement in place for thousands of years for a good reason: creating a drawing was unavoidably filtered by our conceptualization of the scene, so the painters had a tendency to paint archetypal representations of the objects in the scene rather than the objects themselves. It is like the drawing of a child where all the people in their family are typically presented with the feet at the same level and seen from the side, because the kid knows they are standing next to one another, so this idea overpowers the observation of the perspective, where most feet are actually seen from above. The result of this conceptualization is that it took centuries of detail study of the painting techniques to accept that part of the skin could actually look green even if no one has a green skin, and in the early 1400s the first painters started to reach what would later be known as photographic quality, which implicitly requires jumping over you understanding of the scene and reproducing instead what you see.

The introduction of photography made it much easier to capture views of the reality. Even if it took a number of steps before and after to prepare the plate and then develop them, the actual capture was just a few seconds, and the reproduction was quite reliable. Newspapers started replace drawings of the news with photographs, what made them much more trustworthy, since they were hard to forge. Photography also allowed to transform a theater play into a graphical novel, where snapshots of the actors where interspersed with text in the same way that preachers used to anchor their sermons on frescoes and tell in words the story between two consecutive images. Already at this point, the artists where face with the dilemma of what to do with dangerous or highly improbable situations, which were either extremely difficult to achieve or even put the lives of the actors at risk. The solution, of course, was introducing visual tricks that would later be known as special effects: a curtain of the same color as the background could hide the support of an object that seemed to fly, or a half-sword coming out of a soldier's chest would give the impression of going through.

When the movies went on fashion, the industry was faced with a three-way choice: some events could happen "off-screen", so that the scene could be shown before and after but the dangerous moment did not have to be displayed; some events could be simulated by smart tricks just in the same way that magicians had been doing all along; finally, there were some situations that just could not be created in the reality to be filmed and were therefore relegated to the cartoons. Every time that Wiley Coyote falls from a cliff, produces a coyote-shaped hole and then climbs out patting the dust off his shoulders we are seeing something that is just physically impossible but is necessary from the point of view of the story (it would be very sad to see him actually die in the middle of a bloody puddle).

However, the introduction of ever increasingly powerful computer and more refined techniques changed that trade-off for ever. The green screen techniques had been in use for years to present the subjects of a shot in front of a different background, but the introduction of computer-generated imagery (CGI) opened the possibility of adding and even moving objects that just were not there during the filming. Of course, there was initially a clear limitation of capacity to the complexity of the objects that could be handled: hairy creatures were extremely hard to simulate credibly, as were bodies of water or trees; shiny objects were also problematic with their reflections of the surrounding objects; but there were many situations where computer modeling was manageable from early on, like spacecraft, asteroids, even some types of insects were credible at the time.

One of the most salient goofs I can remember is the pastoral scene from the 2002 Star Wars: the Attack of the Clones, where Amidala and Anakin go out for a picnic and he tries to ride a kind of cow called "shaak" and ends up being thrown in the air and then trampled. The shot where this happens is entirely digital and, while the movement of the shaak is credible, Anakin come through too rubbery, almost like a rug doll. Obviously, the physics of human motion were not sufficiently refined at the time.

In fact, the technology now has split into two different paths: water and hair are now reliably modeled by the computers as was fire before, but human motion (both bodily and gestures) are now being recorded rather than modeled. There is a remarkable "behind-the-scene" video of Benedict Cumberbatch playing Smaug for the Lord of the Rings trilogy, where he is covered in white dots so that the computers can track his movements accurately and then transport them to the dragon. It is most remarkable how his body is covered with maybe twenty points, but his face has about twice as many, including two on the nostrils so that the computer can detect when he flares his nose. The result is astonishingly vivid, as  one would expect considering that it is following the actor's real movement. If this were to be modeled, someone would have to figure out when the dragon tilts its head, when it opens its mouth and so forth. But with the synchronized recording of voice and face that is not even a problem anymore.

In spite of its apparent complexity, this trickery has become extremely affordable too and is not just limited to the expensive movie productions. Nowadays almost anyone can make fake photographs with the help of Photoshop and other software of the sort. The result is not guaranteed to be credible, but anyone with a minimum of technique can get amazing results.

The consequence of this technological (r)evolution is that stories that used to be only cartoon material can now be filmed with the assistance of CGI. Consider for instance the contrast between most series of the Star Trek franchise, where almost all alien humanoid races happened to be the same size as humans (so that they could be played by an actor), with the latest installment of Men in Black, where there are aliens in almost any size. The trade-off (there is always a trade-off whenever change happens) is that we have lost our faith in photographs and even video recordings: just a few decades ago it was much harder to fake a recording, but these days seeing is not believing anymore. The devices which were intended to capture reality as it is have been overpowered by the computers, which can coerce them to display whatever the operator wants. But on the other hand, it is a lot of fun. Have a nice weekend.