Our 2nd annual Storyscapes program, created in collaboration with Bombay Sapphire®, celebrates new trends in digital media and the filmmakers and content creators who employ interactive, web-based or cross-platform approaches to story creation. James George and Jonathan Minard explore the meeting of algorithmic art, videogames, and film in their immersive new project, CLOUDS.

Tribeca: Tell us a little bit about CLOUDS. What was the impetus behind this project?

James George & Jonathan Minard: CLOUDS is an interactive documentary portraying a community of artists who use code for visual expression. The project traces a social network of collaborators through a series of interconnected conversations. It reflects how networks enable creativity with technology and facilitate collaboration across global boundaries. CLOUDS captures this story through interviews with 45 artists from the Processing, OpenFrameworks and Cinder communities: most of the artists in one way or another contribute to these open source software tools and share their work through the internet.

At Storyscapes, CLOUDS is presented as an interactive installation along with a version adapted for virtual reality on the Oculus Rift.

Tribeca: What are your thoughts on the potential for creative code? Is there more crossover between technology and the arts currently?

JG & JM: Creative coders explore how software and technology may be used expressively to directly generate audio and visual experiences and artwork. The work ranges from purely aesthetic and formal investigations to provocative applications that incorporate social media and stage online performances. Many creative coders have hybrid backgrounds, associating not just as computer programmers but also as artists, hackers, scientists or researchers. Code is a tool for them to rapidly iterate ideas, break down complex phenomena into simple rules, and visualize vast amounts of data.

As technology becomes more pervasive and average people spend much of their time interacting with software based systems, art that incorporates code will be increasingly relevant in helping us understand the rapidly changing digital landscape.  

In recent years, the motion graphics and visual effects industries have adopted code-based workflows for high-profile productions (such as Tron Legacy). We’re witnessing a convergence of computational art, videogames, and cinema. The techniques developed by coders influence the language of motion pictures, driving innovation in visual technologies.

CLOUDS was fortunate that the community surrounding creative coding is so interconnected and supportive. Without their trust and encouragement, we couldn’t have hoped to complete such a massive undertaking.

Tribeca: CLOUDS employs an innovative process that provides an immersive, responsive visual experience for viewers of the documentary. Describe the correlation between form and function that makes it so innovative.

JG & JM: Since it is driven entirely by code and generated in real time, no viewing of CLOUDS is ever the same. The interviews have been edited and annotated according to shared ideas and topics. These associations form a node-network in 3D space, which represents the web of ideas presented in the documentary. The viewer interacts with CLOUDS by first selecting a question. The documentary consults its story engine algorithm, which finds a path through the network leading through a series of relevant interviews and visualizations created by the artists. The visuals are interactive, creating a sense of immediacy and immersion.

The interviews in CLOUDS were all captured using the RGBDToolkit, an application we created as we produced the documentary. The software combines traditional Digital SLR video footage (that’s the RGB part) with 3D scans from a Microsoft Xbox Kinect video game controller (the Depth component), rendering the streams together as 3D forms in computational space. This allows the interviews themselves to be represented as data, suggesting the artists are coexisting with their code based creations.

Tribeca: Where do you see RGBD technology taking filmmaking in the future?

JG & JM: Since we released the RGBDToolkit two years ago, a number of fascinating filmmakers and storytellers who are interested in using the technique have come forward. In our conversations with this emerging community, we find that many people are excited about the possibilities that 3D live action footage suggests.

For instance, the potential to change the camera angle in post, or remove the background without the need for a green screen are things that depth cameras may soon make possible. While all of these possibilities are still some distance in the future because of low resolution of current sensors, the toolkit software has sparked the imagination of what this will be like.

Currently the software’s grainy 3D wireframe aesthetic has become a coveted way for filmmakers to create narratives and represent people occupying imaginary digital spaces.

Tribeca: How has online sharing changed the way artists think and collaborate?

JG & JM: The subjects in CLOUDS defy the traditional stereotype of the isolated artist protective of his or her process. In contrast, everyone in the film goes to great efforts to openly release the tools they invent while creating their work. With the aid of sites like GitHub and Twitter, the creator’s code and ideas can be discovered all over the world, making the community international.  As other artists discover and adopt these free and open tools, the community begins to develop a common language, which facilitates artistic collaborations. 

Tribeca: CLOUDS was partially funded by a successful crowdfunding campaign. Do you have any advice for creators looking to start their own Kickstarter?

JG & JM: The Kickstarter campaign was an amazing experience. We benefited immensely from going through the process of describing the project in a way that made it feel real, even when it was just an ambitious dream.

The challenge with Kickstarter is to know what tribe you are activating with your campaign. CLOUDS was fortunate that the community surrounding creative coding is so interconnected and supportive. Without their trust and encouragement, we couldn’t have hoped to complete such a massive undertaking.

Transmedia projects are finding ways to tell stories that permeate all different platforms to create surprising and multilayered narratives.

Tribeca: How do you approach creating an immersive installation? Was this approach something you always planned for this documentary?

JG & JM: CLOUDS was always intended to be an interactive experience that allowed viewers to choose their course through the database of interviews. With this as the core, we kept the presentation platform open until we were given an opportunity to present the project.

One serendipitous occurrence was the chance to adapt CLOUDS to the Oculus Rift virtual reality display. The Rift was released when we were already midstream in postproduction, but because of the platform agnostic approach, we were able to create a version of the project that worked with the device, allowing the viewer to interact simply by looking around. This provides the most personal and immersive way to experience CLOUDS.

The installation version is intended to feel like a small theater, where the viewer at the front is in control and the others become spectators in the journey.

Tribeca: What are you most looking forward to at Tribeca?

JG & JM: Connecting with other filmmakers, and seeing as many movies as possible!

Tribeca: Where do you think the future of interactive storytelling is headed?  

JG & JM: The craft of storytelling is as old as language itself. A good story captures our imaginations, taking us to new places and reflecting new insights back to our lives. With the prevalence of social media and online networks, we have come to expect  to participate in stories as they unfold. Interactive storytelling will continue to walk the line between traditional narrative and new ways to incorporate emerging culture. It’s a vibrant intersection that is here to stay.

Tribeca: What does the word “Transmedia” mean to you?

JG & JM: To us, transmedia means stories that can be told across many platforms simultaneously. We get media from so many different channels, from traditional film and broadcast, to sites like YouTube and social media. Transmedia projects are finding ways to tell stories that permeate all different platforms to create surprising and multilayered narratives.

Experience CLOUDS (and Oculus Rift!) in person at the Bombay Sapphire House of Imagination running April 23-26. Registration is now open!