Select Page
Film As A Simulation Of The Brain’s Mind

Film As A Simulation Of The Brain’s Mind

In my previous post, I suggested that it is more helpful to look at story as a simulation rather than in terms of plot and character when it comes to thinking about sound cinematically, at least in the early stages. So today I’d like to explore the concept of simulation more in detail.

A simulation is a model of some aspect of reality that allows us to safely carry out experiments, learn new things, and practice skills that we can then apply to real-life situations. To that end, it needs to have some kind of interface that gives us the means to interact with the virtual world it represents. A flight simulator, for example, has a real cockpit that can tilt in any direction, that makes real sounds, and that has real controls linked to a computer system that interprets the pilot’s actions and moves the cockpit accordingly as the pilot reacts to faithfully recreated settings and situations such as airport, mountains, dangerous weather conditions, and emergency landings. All these elements allow the pilot to become immersed in the situation at hand and experience it ‘for real’.

Stories, too, are powerful simulations that allow us to explore and learn about the social world we inhabit. As Jonathan Gottschall puts in his book Storytelling Animals, they are a place “where people go to practice the key skills of human social life”. And the interface that allows us to interact with this simulation is identification with fictional characters.

It makes sense. All species are hardwired to learn specifically about that which is essential to surviving their environment. To us humans, one of these essentials is being able to figure out other people’s needs and intentions so that we can adjust our actions accordingly. Not an easy task, since we have to cohabit and cooperate with large numbers of strangers we know nothing about.

If stories are simulations of the social world, then it makes sense that we interact with it by ‘stepping into the shoes’ of a fictional character. Through identification with them we get to feel their longings, frustrations, virtues and flaws. We experience first-hand their struggles, the moral dilemmas they face, and the consequences of the choices they make. We walk their walk and that’s how we learn.

But how exactly does identification work as an interface? It is easy to see how we can use a cockpit to interact with a simulation, but identification? Stepping into the character’s shoes sounds great as a metaphor but if you’re entrusted with the task of performing the feat, it can leave you feeling a bit baffled.

Luckily, there’s a way of bringing down to earth this concept of identification with fictional characters by looking at it from a more biological point of view.

I’ll start with transportation. This is another term that goes hand-in-hand with identification. Rule number one is that, in order to step into the character’s shoes, the audience need to be transported to the story world first. Another metaphor, but this time it is actually closer to the real thing since films transport us to the story world almost literally – though it is not our legs what take us there, but the physiological responses and emotions we feel in the body as a result of being exposed to the story events.

Physiological responses ‘transport’ us because, if we’re feeling anything, it means our brains think we’re there. If our brains thinks we’re there, we might as well be there. And why do our brains think we’re there?

Films work as an illusion because they exploit loopholes in the perceptual and cognitive processes that we evolved to help us navigate the environment. One of them is the communication time lapse that exists between the unconscious and the conscious brain.

In very simple terms, the brain works like this: it interprets the data that the senses have picked up in the environment, it establishes the context, it evaluates the information according to this context, it determines its importance, and it decides on the best course of action. Is it a threat? Run. An opportunity maybe to reproduce? Strike a sexy pose. A significant change? Get closer and find out more.

The first thing we feel as a result of this process is the physiological response, which tends to be quite basic and which main function is to prepare the muscles to move. Then comes a fully-fledged emotion, which contains more detailed information about the required response. A racing heart in itself does not tell us much. If it comes with a rush of fear or of lust, then we get a much more precise idea of what it’s all about.

On the whole, we’re not aware of these processes. This is because the brain operates in two basic modes, unconscious and conscious. The unconscious mode is much faster at processing things than the conscious. It can organise the neurochemistry and behaviours of our system within 80 milliseconds, whereas it takes the conscious mode about 250 milliseconds to catch up with things.

It is thanks to this gap that we experience films the way we do. When we’re watching a movie, all our brains know for the first 250 milliseconds is that the senses are sending information that is organised in patterns of light, colour, sound, movement and behaviour that feel just like the real world. So the brain does with this information what it evolved to do. It processes and evaluates it, and it prepares the body for the right action. Luckily, though, just before we actually stampede out of the theatre at the sight of a dinosaur, the conscious brain realises that it’s just a movie.

In short, by the time the conscious brain has figured out that we’re only watching a movie, the unconscious brain has already made a full cognitive evaluation of a situation it deemed to be real and it has triggered all sorts of physiological and emotional responses in our bodies. These bodily sensations are what anchor us in the story world. As far as the unconscious brain is concerned, we are there.

All this works very well for transportation into the story world, but that’s only the first stage. Other things still need to happen for identification with a character to take place.

One good way of understanding identification is by looking at what it is not – empathy. These two terms are often used interchangeably, but they are not the same. We can, for example, empathise with humans and animals alike, but we can only identify with humans. Making this distinction is crucial.

Empathy is when we recognise and share the emotions and feelings of another being. We see their situation from their perspective and as a result we get to feel what they feel. This is possible thanks to a mirroring system that we evolved to deal with our social world. It works by replicating in our biology the neural processes that happen in the brain for coordinating and carrying out actions.

Let’s say that a man decides to open the door. A set of neurons will fire away in his brain and will activate the regions responsible for coordinating the actions involved. The motor system will then receive instructions and perform the task. Thanks to this mirroring system, if I watch this man carry out this action, the same neurons will fire in my brain and activate the same regions involved in the operation, only the motor system will not perform the action.

This mechanism works just the same with emotions. By picking up very subtle sensory cues from things such as facial expressions, body language, and tone of voice in another person, the brain is able to replicate in our bodies all the neural processes involved.

It is easy to see from an adaptive point of view the benefits of having such a system. It gives us first-hand information about the intentions and motivations of others and about their mental and emotional states, and it helps us adjust our responses accordingly. If you’ve ever seen someone in distress and immediately understood that all they needed was a hug or a few friendly words, that was your mirror neurons giving you a tip. Or if you’ve ever had a feeling that someone was lying to you or was up to something, that was your mirror neurons too.

As for film, you can imagine how useful these mirror neurons are when it comes to getting the audience to connect both biologically and emotionally with the characters on the screen. Sadly, though, this is not enough. For identification to happen, it is not enough to simply get to experience the same emotions and thoughts as the character. The audience must reach those mental and emotional states through the same cognitive processes that took the character there.

With empathy, things happen there and then, mostly thanks to our mirroring system. Identification is much wider in scope. It requires the audience aligning with not only the character’s feelings and emotions but also with his or her cognitive processes – how he or she reasons, evaluates things, solves problems, sets goals, formulates plans, and so on. That is why we can empathise with an animal such as a dog but not identify with it. Dogs feel emotions similar to ours because they are mammals, but they use different cognitive processes to solve their problems and as a result we cannot ‘step’ into them.

There is one assumption in NLP (Neuro-Linguistic Programming) that makes it easier to understand this concept of identification with fictional characters: “The map is not the territory”.

The territory refers to reality, the physical world that exists independently of our experience of it. The map refers to our minds and to the model of the world we have built through our perceptions, personal experiences, culture, and what we have learnt from the significant others that have been present since early in our lives. It contains, among other things, our beliefs and values, which play a key part in determining the decisions we make and the actions we take.

We only ever get to know our own version of reality. Because it is so unique and personal, we can’t really step into someone else’s mind and experience the world through the lens of their model, not unless he or she is a fictional character, that is.

And that brings us back to story as simulation of the social world and identification as its interface.

If our mental model of the world determines the cognitive processes by which our brains perceive, interpret, and react to events in the environment…

And if identification is achieved by getting the audience to feel and think what the character does through the cognitive processes that took the character there, i.e. through the same mental model…

And if simulations allows us to create models of some aspect of the world that we can use in its stead…

… then we can achieve identification with a character by creating a customised mental model of the character’s world that contains the values, beliefs, and experiences that will lead to the emotions and behaviours we want to explore in the simulation-story. We can then ‘install’ such model in the audience’s minds. This way, their brains will be operating within that specific mental model, and as a result, they will walk the character’s walk, and arrive at the same thoughts and emotions by means of the same filters that took the character to do, think, and feel what he or she did.

Equally important is that all this process generates a lot of neural activity in the brain. The resulting synapses are what get stored in the nervous system as memories. When this happens, the film has served its evolutionary purpose, and the audience get a reward in the form of a shot of feel-good chemicals such as dopamine. That’s why films that get identification right tend to do well at the box office.

One of the main tools for achieving identification with characters is narrative structure. The set-up, for example, is all about ‘installing’ in the audience’s brains the specific parameters in the character’s model of the world that will cause him or her to take the actions and make the decisions that will lead to his or her success or demise. From that point on, even if the character is not present in the scene, the audience will be evaluating the events from within their position of identification. “Are these events good or bad?” And “Will they facilitate achievement of the goal or hamper it?”

This tool, narrative structure, is the staple of all forms of fiction in general. But there are different forms of storytelling – novels, theatre, film – and each, as anyone who’s had a go at adaptation will know, offers dramatic possibilities unique to their own format. Each allows you to get into the character’s mind in ways that the other formats can’t.

What is unique about film is that stories are told through the arbitrary combination of images and sounds which are arranged according to an established cinematic code.

It is easy to see the advantages such feature would bring to a simulation. First, the presence of the two primary senses adds realism and acts as a form of veridicality. The brain is prone to perceptual illusions due to loopholes and it knows it. Seeing is believing, so is hearing, although there’s plenty of room for mistakes. But when you hear and see, there is no doubt in the brain’s mind that it must have been an accurate perception. This makes film even more of an effective illusion.

Also, sound alone does an incredible job of transporting and immersing us in the story world. Being a mechanical wave, it literally touches us and can, through sympathetic resonance, influence our biorhythms in the mental and emotional domains.

Then there’s point-of-view visuals and point-of-hearing sound, both of which can take identification to deeper levels. Although, be warned, in and of itself this technique is not enough to get the audience to identify with a character. In horror films, for instance, a point-of-view shot of the victim does not necessarily lead to identification with the killer (identification, remember, is a process).

But where the real possibilities lie is in the actual relationship between image and sound – in how they are combined meaningfully for the purpose of creating a specific mental or emotional effect.

In reality, the brain uses emotions and perception to guide our actions. After carrying out its evaluation, it activates the right behavioural programs, i.e. emotions, and it determines which bits of information on each sensory channel are most useful to guide our actions. If most of the important information comes in the form of, say, sound waves, it will reduce the presence of other sensory data. The result will be a streamlined percept that includes only what is essential to perform the task at hand efficiently, and that will have excluded any distracting sensory stimuli.

In filmmaking, the director takes charge of this process. He or she creates a percept, or movie, by manipulating and combining images and sounds so as to fool the brain into interpreting things in a way that will lead it to trigger the physiological and emotional responses required by the narrative. Filmmaking is the art of hijacking the brain and tricking it into thinking that the film is a meaningful percept that it, the brain, created by itself.

So what we have on one side is that identification requires that the audience get to feel and think what the character does by the same perceptual and cognitive means.

On the other side, we have that filmmakers can capture image and sound separately and then recombine the two arbitrarily to manipulate the perceptual and cognitive processes that guide the audience through the narrative.

In summary, we can use the relationship between image and sound to deepen identification. Or to give it some slack if that’s what the narrative requires, like for example when the character is a dubious figure and you need to break identification in order for the audience to step back and have a chance to reflect on what has transpired and thus learn the moral lesson of the film. Identification, after all, moves along a continuum. The manipulation of audiovisual information can help things move along this continuum.

In the last few paragraphs I’ve started mentioning a few times the phrase ’the relationship between image and sound’. This is what film sound is all about. Not the sounds. Not even the act of combining the two. The relationship itself, as an entity in its own right, is the actual cinematic device, the dynamic that breathes life into film sound.

So, in my next post, I’ll be talking about the guiding principle that can make this partnership go from ‘mere combination of image and sound’ to ‘meaningful relationship between image and sound’.

The Story of Story

The Story of Story

I ended my first post concluding that until we begin to understand film sound at a deeper level, we will not get it past the creative impasse it is stuck at right now. If you ask me, the first thing we need to do is to start thinking of film sound as a subsystem within a system rather than in terms of sound effects and music, the sole purpose of which is to add realism and emotion to the images.

A good way to start this process is by defining the system sound belongs to: the film. A film is a story told in pictures and sound. So far so good. But that leaves us with another question. What is a story? Not so easy to answer. Yet, this is the first question we need to address if we are to understand this concept of film sound as a subsystem and as a cinematic tool that takes on an active role in the process of narration.

Over 2000 years ago, Aristotle set the precedent in his Poetics for how story would be defined in Western culture. Most attempts today still revolve around the same type of questions: is it action, character, plot, conflict, form, content…?

I spent some time delving into such questions, and I gained some interesting insights, but unfortunately, they proved to be almost (not entirely) futile when it came to grasping the cinematic role of film sound. I decided to look somewhere else, and I found the answer in a somewhat unexpected place: evolution.

The main force behind evolution is natural selection. This is the process by which a behavioural, physical, or physiological genetic mutation either makes it into the permanent genetic make-up of a species or dies away. The selective criteria is simple: Does the mutation give an organism a competitive advantage that makes it more capable of adapting to its environment and fitter to survive it? Yes? Pass. No? Out.

From an evolutionary point of view, stories are a behavioural adaptation. It follows then that they must have given us some sort of adaptive advantage. They have. And this advantage holds not only the key to understanding what makes a good story, and therefore a good film (I will talk about this in another post), but it also provides very important clues to the cinematic potential of film sound. So, let’s dive right into the story of story.

Long before we could tell tales, we were a chimp-like species, much like any other. There was nothing remarkable about us. Then, one day, the story goes, a group of us were outcompeted by other apes and found ourselves having to find another way of foraging. Smart things that we are, in the process of solving this crisis, we invented a new way of cooperating: collaboration, which is similar to cooperation in that it requires the members of a group to work together towards achieving a goal but with a number of important differences.

Cooperation involves working together towards the same goal, but ultimately, participants do it for their own benefit. In the case of chimps, for instance, if teaming up and cooperating with another chimp makes it more likely for them to achieve their aim, then they’ll go along with it, but essentially, they prefer to acquire and eat their food alone if the circumstances allow for it.

With collaboration, however, our thinking became geared toward figuring out ways to coordinate our actions with that of others in order to achieve a joint goal that had been pre-agreed (cooperation lacked this element of predetermination).

To us today, this may not seem like a big deal, but back then, it was a revolutionary way of doing things that required us to push our cognitive skills to the limit. We had to develop the ability to form shared goals, to assign each member of the group an individual task, and to understand how both our own task and that of the others fit within the scheme of things. Then we had to focus our joint attention on the same aim and synchronise our actions in order to achieve our goal. This behaviour was so radically different that it led to us splitting from the chimp lineage and becoming an altogether new species: the human race.

In itself, this was not enough to turn us into what we eventually became, which is the most successful species on earth. It allowed us to form slightly larger societies than other species, but that was about it. The true evolution would come about some few thousand years later, with a rare mutation in the brain that gave us the capacity for story.

Before this mutation happened, our brain was modular in nature. That is, it had separate modules to process different aspects of the environment. There was one module for inanimate objects, one for artefacts, one for animals, one for members of the same species, and so on. Our brain also had modules with dedicated types of intelligence designed to solve specific problems. For instance, it had a technical intelligence for building tools, such as a hammer or a knife, and a social intelligence for making sense of things like facial expressions.

These intelligences were pre-programmed by evolution to obey the rules of the natural world and they could not be consciously changed or controlled. A lot of brains still work like this. A bird, for example, cannot consciously decide to change the way it goes about building its nest since the process is hardwired into its brain.

Having such brain structure meant that we could only process the environment literally. Animals were animals, people were people, flowers were flowers, and ice was ice. There was a fire here and a lion there. Although we had the capacity to have imaginative thoughts within each module – we could imagine that if we hit something with a hammer it might break – we could not bring knowledge from one module to another. This was because there were no neural networks connecting them that could provide such cross-over of information.

Then, this magic mutation changed the wiring of the brain, causing the separate modules to be able to communicate with one another. Thus, the story of how metaphor, fantasy, and wild imagination were born, and how we went from, “Careful! Lion there!” to, “Once upon a time, there lived in a land far, far away a man with a lion’s head, surrounded by the flames of eternal fire. His wife, who had eyes blue like the sky, and was beautiful like a flower, but had snakes for hair, spent her days spinning the fates of the inhabitants of this land.”

To see this in action, you only have to visit the section of any Natural History Museum holing objects dated from 35,000 years ago. There, you’ll find practical utensils like spears, cups, hammers and so on. Skip forward a couple of thousand years or so and you’ll start finding more bizarre things, like the 33,000-year-old lion-headed man carving from Hohlenstein Stadel in Germany.

Anyone with a bit of common sense would have bet that such a futile and potentially dangerous behavioural feature alone would have been enough to bring our entire species to an end – daydreaming in the caveman days does not sound like a wise thing to do. But for some strange reason, that wasn’t the case. On the contrary. Not only did our fellow Neanderthals, who didn’t develop this capacity for telling stories about peoples and places that didn’t exist, start disappearing at an alarming rate from this point on, but we also started living in larger and larger settlements.

What happened was that stories allowed us to invent narratives about our past that gave us a sense of belonging together. Because our bonds were built on a mental ground, there were no limits to how many people these bonds could unite in one single stroke. Compared to our other fellow ape species, who still had to rely on one-to-one grooming to build trust among each other, this represented a big advantage. It meant we could build infinitely bigger communities, achieve more together, and protect each other more efficiently.

We could also imagine stories about the future, better worlds where the problems we faced had been eradicated. We could imagine the values, beliefs, and behaviours that made such worlds possible.

Something very interesting is happening here. On one side, we have a newly-acquired set of cognitive abilities that allow us to form shared goals, to focus our collective attention on them, and to synchronise our actions in order to achieve them. On the other, we have the newly-acquired ability to imagine other worlds and the behaviours required for them to exist.

When we attend to a story collectively, our minds unite and become tuned to the same scenario. The story triggers in us the same thoughts, emotions and learning experiences.

If we put the two skills together, shared intentionality and story, what we get is the ability to synchronise our collective beliefs, values, and behaviours so as to bring our imagined worlds into existence. And that’s just what we did. The outcome? The creation of a new human-made environment: the social domain.

One thing about this new environment is that it is a lot more unforeseeable than the physical and biological habitats we had been inhabiting so far. In the physical realm, we can be sure that the sun will rise in the morning and set in the evening. In the biological realm, we can safely bet that a hungry lion will make lunch of us or that a poisonous mushroom will make us ill at best. It is not so easy, however, to guess what folk with complex psychological computations dictating their behaviour will do next. And that’s what this new environment did to us. It turned us into highly unpredictable and often cunning creatures difficult to figure out.

No need to worry. We had stories, and they turned out to be the solution to the very problem they had created. They became invaluable tools for helping us learn to navigate this new complex social world. Through stories, we could learn the values, attitudes and behaviours that would help us function in our societies or we could explore new behaviours and their consequences. We could also use stories to sharpen our ability to make inferences from and scrutinise other minds – their inner worlds, expectations, intentions, and motivations. This, in turn, would help us make better decisions and adjust our behaviours accordingly, so as to get the best possible outcome.

That’s precisely why stories got the thumbs-up from natural selection. They increased our capability to adapt to our environment. They made us more suitable to survive this new cognitive niche we had just moved into.

One word that sums up very well the adaptive role of stories is LEARNING, but learning with a twist.

Biologically speaking, learning is the process by which information about the environment is stored in our nervous system as memories. It is an essential adaptation for any creature to be able to survive. The only problem is that for learning to take place, we must have some form of direct experience with the environment, something that comes with its risks. What if we don’t survive the experience?

That’s where the twist comes in. Stories provide us with an effective means for learning about the environment and for expanding our repertoire of beneficial behaviours without putting ourselves at risk. They allow us to learn, not through direct experience, but through SIMULATION.

It is when we look at story and film from that perspective – as a simulation of the social world rather than in terms of plot or character – that the cinematic role of sound slowly begins to reveal itself. In actual fact, the whole concept of filmmaking takes on a fresh dimension.

Simulation, therefore, will be the subject of my next blog post.