Cinematic Machines

Published: 5 May 2012

Edited on 13 May 2015: Improved clarity.

Movies are machines, created by people to accomplish certain things, simple as that. Sure, they are soft machines, and they may be at the limit of what we as humans can do. But they are machines.

Though we are relatively stupid in our designs of other soft machines (religious and political institutions come to mind), we seem to be extraordinarily skilled at making films. We don't seem to know much about why they work, even though individuals in the enterprise have been practicing pretty much the same crafts for a hundred years. You wouldn't expect a mason to understand what makes a cathedral special, but you do suppose that the designer of the cathedral knows a great deal about space — that she is going on something more than intuition, but that’s the way it is, though architecture is an expressive medium that has been around since the appearance of civilization.

We do have theories, scads of them, but they don't adequately explain how great architecture is created. It really does take someone who has intuitions about the way space works, and who can extend that intuition to create something special. Space has lots of moving parts in the way it interacts with those within it. We could think of this as a constant dialog between the environment and the occupant's mind and there are a lot of advantages to doing so. (I write a bit more on architecture elsewhere.)

So too with movies. It is easy enough to think of them as a simple experience to be consumed and largely forgotten, much like you probably think of most things in life: meals, sleep periods, random conversations. But here we explore the notion that a film is a coherent mix of many things. These are created by many people, who we can think of as agents of a coarse type. Some of those folks are the cast and crew, of course. One of them is you, because of what you bring and how you agree to receive. There are also others who have created previous movies and experiences that may or may not be directly referenced in the film, but which are part of its meaning and how it affects you.

In the AI community, we have somewhat well developed methods of modeling agents because all that involves is modeling then as if they were people. And we all have good intuitions about what people are and how they interact. But there are advantages to modeling things in terms of constituents of the thing itself without anthropomorphizing.

We wouldn't model a human body in terms of the things that created it would we? Sperm, egg, food, water, sunshine, teachers.... We could, but it would be more useful to model it in terms of systems, right? Each of these could be said to have agency, to be doing a job.

Instead of thinking about the people (and teams) as agents in creating the movie experience, we want to think of films in terms of its systems, and the equivalent of its biology. Just as with the human body, there are some things that can be hard coded, like the chemical and physical dynamics in the body. And there are more complex and sophisticated things in the body that sometimes seem to be reachable by precise ‘logical’ sequences. But there are far more that we know are there, and we model them softly.

Agents

What sorts of agents will we be looking for?

When we model, we have to take account of what is happening in the story, of course: the characters, motives, events, attitudes and so on. There are also genre models, familiar tropes and stereotypes, conventions (like plot twists) and structures (like the three act form). We additionally have some other discrete things in the cinematic expression that can be teased out discretely. We can make agents of:

Scene rhythms
Framing of objects and environments
Lights and movement of lights
The camera identity and indicative gestures of the camera
Elements of the score and sound track

These are a bit obvious, but at the same time hard to write about. The redframer platform will allow us to point directly to the actual effect in the film rather than having to create new terms.

But, but.

Once you get into this, you discover that there are a variety of narrative dynamics at work and these indicate an expanded notion of agency. After quite a search, we settled on situation theory to help with this. Our model has two kind of things; one of these are agents which cause things to happen. Simple enough. The other are situations which can be thought of a few ways.

The simplest is the context which generates the agent. For example, Red Riding Hood the character has a certain agency that is inherited from Red ‘Riding Hood’ the fairy tale. Contexts can also be associated ‘within’ an agent. For example, the Wolf gets its agency from the fairy tale, but in some versions of the fairy tale it matters what the Wolf is thinking. His internal state of mind is also a situation.

A strange and novel notion we use is that situations have agency.

So, the bottom line of all this is:

We want to build a model that tells us something about how the cinematic machine works, in effect building a machine in its own right
We want to do this in a way that can be both graphically displayed and programmed (in some way)
We use agents of a special kind
We also use narrative situations, based on our extension of situation theory, and these have causal agency
Some of the more interesting narrative dynamics among situations are what I loosely call folding

Where Folding Fits

Where folding fits into this agent/dynamic idea.

So where does folding fit in? So here is the whole set of components and names that we are working with.

We are building a system that we believe should be able to deal with the world the way we do. It can fall in love, be deceitful, have emotional pain. We call this system sHe, short for sHeherazade.

This being sometimes uses logic, and sometimes not. That is, sHe’s mind works something like ours. The extra ‘reasoning’ bit is a system that we call topoiesis. The functions (the math and code) in topoiesis aren’t of interest here. But what it uses instead of rules we are of interest here.

We call these narrative dynamics. They are, in fact, abstracted from narrative as we experience and build it. A goal of the redframer project is to collect these. Many of these dynamics will be relatively uninteresting to study apart from the machinery used on their behalf. But some will be interesting as all getout.

Among these interesting ones will be what we address here. In fact, I started working on these folding concepts decades ago when defining the problem, because they seem to capture the conceptual structures that seem the hardest for machine reasoning to absorb.

Elsewhere, we make the case that folding is impossible to even model in logic after the fact, and that is saying quite a bit. No one seems to be able to predict the behavior of the stock markets, and experts do less well than amateurs. But the day after the market closes, the web is full of reasonable explanations.

Agency

So the ideas explored here are done so for fun, and presented here so you can have some fun with them. I’ve included some notions not related to folding simply because they interest me. That said, there is a work-related agenda to try to get a handle on these in our own representation system.

Because they are the main way we play with the machinery. Most of the other stuff in the machine is passive syntax that gets information from a creative team to consumers. Folding is our insertion in the process, our agents in the machine.