The author explores the experience of watching “other people’s pixels,” from machinima to generative AI, and highlights how generative AI tools enable artists to blend different sources and create unique outputs. He also explores the democratization of high-end visual effects, revealing the accessibility and potential impact of these tools. And he seeks to define the experience of “the new pixel” and its potential cultural and industry impact, as he speaks to new media artists and VFX professionals to get their take.
When I proposed possible subjects for this article, Jon and Karen both immediately seized on machinima.
Machinima, or films created using real-time 3D computer graphics, is a genre I was once upon a time fascinated by, and which I had almost entirely forgotten about (the way we will all sometime soon feel about VR?).
I wondered what had become of it and why it had so totally dropped off my radar. And then I slapped myself on the forehead as something came lurching into my brain, not quite formed, but big enough to make me feel very dumb indeed.
Machinima was, at its heart, the layering of gameplay with a narrative imposed from outside, by the “filmmaker.” It was something more than just watching over a shoulder as someone played—it added an additional level of entertainment that was not part of the gamemaker’s design.
I do, in fact, watch variations of these things all the time.
The pleasures of watching Sharo speedrun The Outer Worlds in less than 11 minutes is entirely abstracted from the narrative of a game that takes a week to play properly. But pleasures they are, appreciated even by the developers, who took the time to record a commentary track over one amazing speedrun.
On Twitch, there is every kind of gameplay to watch. Let’s put aside the reviews, which don’t seem to me like they are quite in the same kind of category of entertainment as machinima, and we are still left with playthroughs, walkthroughs, speedruns, competitive play, and gaming as a background to chat or talk shows.
Without the agency provided by holding a pad in my hands, these are someone else’s pixels.
Playing a computer game is an immersive and interactive experience. For me, the experience is characterized by feelings of excitement, achievement, and immersion arising from exploration, and all of which add up to create escapism. Those are not the same feelings one gets watching someone else’s pixels.
When watching others play, one might admire their talent, admire the amazing game graphics, which one is freer to pay attention to, or admire the new narratives and entertainment modes. At the elite end of this experience, watching pro esports, or almost-impossible gaming exploits, there is an open-mouthed shock, like when watching daring parkour videos.
Cinema is the ideal analogy to the experience of these mediated pixels. The term “machinima” is a portmanteau word created from “machine” and “cinema.” In machinima, filmmakers use video game engines and characters to create scenes, record gameplay footage, and manipulate in-game assets to tell stories or create animations. This can include creating original narratives, reenacting scenes from existing movies or TV shows, or even producing music videos.
Sam Crane brings this connection into focus with his short work “We are such stuff as dreams are made on,” appearing at the Oberhausen International Short Film Festival alongside more traditional cinema fare.
It reminds me of William Gibson’s description of the “Garage Kubrick”—“an adolescent near-future Orson Welles, plugged into some unthinkable (but affordable) node of consumer tech in his parents’ garage. The ‘Garage Kubrick’ is single-handedly making a feature in there, some sort of apparently live-action epic that may or may not involve motion capture. That may or may not involve human actors, but which will seem to.”
The control is available. Today, reality can be modeled and rebuilt with tools like Arcturus’ HoloSuite. We can create not just entertainment, but a very usable simulacrum of reality, one good enough that companies like Mindtech are using to train AI for real-world use cases, while protecting privacy by not viewing real incidents.
In 1996, when machinima was new, Independence Day was in cinemas. Approximately 70% of the shots in that film had some type of visual effect, but few were wholly computer-generated. Avengers: Infinity War contained over 2,700 shots, just 80 of which were non-VFX shots. Meaning, just 3% of the shots in the film went untouched by the visual effects team, and many were wholly animated.
What we see watching Avengers: Infinity War is not our physical world, but a visual vehicle for narrative where the pixel is the dominant reality. This brings us to trailers for films that don’t exist, and the very newest and most divisive form of someone else’s pixels.
I suspect many of us (1.6 million on X, formerly Twitter, at of the time of this writing) recently shared the same “Wow!” moment when we saw the trailer for Genesis, a film that does not exist. It was created by Nicolas Neubert, a 29-year-old Volkswagen product designer, using Gen-2, a generative AI video creation tool from start-up RunwayML.
Genesis’ 45-second video was created by using still images that were created with the Midjourney generative AI tool and processing them with Gen-2, a multimodal AI system that can generate videos using either text, images, or video clips as a starting point.
According to Neubert, “It took seven hours from the beginning to the end” to create the trailer (based on pre-existing images).
This is the democratization of high-end VFX, and possibly the birth of a whole new kind of visual creative who uses prompt engineering rather than traditional VFX tools. The democratization is the point, according to Neubert: “One of the goals of not only this trailer, but what I do with Midjourney, is to show the accessibility of it.”
It’s not there yet. Marque Pierre Søndergaard, senior texture artist and host of The Naked Texture Artist podcast, told me: “The main problem is that although you can make fantastic images, we need far greater control than what you, right now, can do with a language model. I need to be able to, if needed, control exactly that pixel. These are tools that can do you a version 0 or 1, but still nowhere near the finished article.”
Are these even the pixels of the person who created the prompts? Or the machine’s pixels? Or pixels ultimately generated by the traditional VFX artists whose work the machine learned from?
Søndergaard thinks generative AI will become mainstream. “I would imagine that the industry standard tools are already scrambling to include it. We have seen bits and pieces of it in tools already,” he said/
“Long Live the New Flesh” is a phrase that originated from the 1983 Canadian science-fiction film Videodrome, directed by David Cronenberg. In the context of the movie, “Long Live the New Flesh” is a mantra associated with the dangerous and transformative power of media and technology. The phrase represents a merging of the physical and virtual worlds, blurring the boundaries between reality and fantasy. It signifies a transformation of the human body through exposure to media content, leading to a new form of hybrid existence.
Emanuela Quaranta (@Ela4zero on X) creates abstract art using generative AI tools that reminds me of work by Georgia O’Keefe. (Click on the link, it’s a delightful video.)
Collaborating with the Crosslucid artist collective, Quaranta takes care of the technical aspects of some projects involving training GANs to generate images and videos. This is a new way of making pixels divorced from any VFX tools I recognize.
Quaranta explained to me, “I’ve used Stability AI’s Stable Diffusion XL 1.0 to generate an initial image, which I then fed into Runway’s Gen-2 to animate it. The outputs of Gen-2 are actually limited to four seconds, so, to create this video, I rendered together two different generations, one of which is reversed, and then slowed down and upscaled the final video using Topaz videosoftware.”
The process is interesting, but less so than the intent. Quaranta said, “Since neural networks can produce a diverse range of textures, patterns, and forms, I find them ideal to explore the concept of hybridization, which my work deals with mainly from the perspective of synthetic biology and speculative evolution. I love to imagine possible combinations of materials from different sources and use AI tools to generate them. For this work, I was envisioning some sort of shape-shifting biomorphic artifact.”
Hybridization is the act of producing hybrids resulting from a cross between genetically unlike individuals. Perhaps the concept of hybridization is not limited to the realm of genetics. In the world of digital media and technology, we are witnessing the emergence of hybrid creations through generative AI tools. These tools enable artists and filmmakers to blend different sources, textures, and forms, resulting in unique and imaginative outputs.
The democratization of high-end visual effects is underway, allowing individuals to explore new possibilities and push the boundaries of creativity. As we navigate the blurred lines between reality and fantasy, the pixels we encounter are no longer confined to one creator or medium but have become a fusion of human and machine-generated artistry.
Long live the new pixel indeed.