HOME

ARTICLES

---

---

---

 

Dynamic screen / room:

 

Thore Soneson

 

During the last decades moving images, video and screens have expanded from on-the-wall projections to dynamic and multi-modulated images in different spatial settings – on multiple screens, in dynamic and interactive room environments and in an immersive physical context. Narratives have evolved from linear storytelling to open database structures, influenced by computer game-like scenarios and simulated worlds. The dramaturgic tools have evolved from the Aristotelian catharsis format to the open-ended closure, expanding from strict linear forms to generative codes using meta-tagging and computation. Pre-recorded video sequences and live-feed real-time video are combined in artworks and on-line worlds, commercial and open source software are integrating visual elements on all media platforms – from computer to handheld devices.

This expanding possibilities of media technologies have opened up the screen into a dynamic room, an interactive, real-time visual and cinematic stage. The screen is no longer a static element; its scaleable and flexible, its immersive qualities and interactive possibilities are challenging for visual artists to explore and use in new artistic contexts. This short exposé tries to pinpoint some contemporary tendencies and prolific artworks, thus creating a reference and artistic perspective. It was originally made as research for an interactive performance – the PRAMnet project "Journey to Abadyl".

The exhibition and catalogue FUTURE CINEMA / The Cinematic Imaginary after Film at ZKM curated and edited by Jeffrey Shaw and Peter Weibel in 2003, was the first large-scale expose over experimental film and digitally based video-art. The exhibition included historic and contemporary media works, and made the first comprehensive theoretical overview of the expanded narrative strategies of visual moving imagery.

Using a setup of keywords, the authors formulated dramatic tools for the future cinematic art and formulated a possible future development of the art of the moving image. In their curatorial statement, online at ZKM website, Shaw and Weibel wrote :

 

"The Future of Cinema can be delineated from two sources. One way is the expansion of existing cinematographic methods and codes into new areas. The other way is the convergence of cinema, TV and net. The classical cinema can be defined as collective experience of one fixed stable projector projecting a moving image on one screen in one room. Therefore each change of one of these factors, for example multiple screens, panorama screens, moving projections, different rooms, is already an expansion of the contemporary practises of cinema.

On the level of the material display of the image, many new techniques are also in development. Smart materials are developed which can be new sources of light and colour. New lenses are developed which contain information as a temporal code. In general, there is a change from refractive optics to diffractive optics.

Especially new results are to be expected from multi-local and multi-user virtual environments, from massive parallel virtual worlds, which can be developed based on the global net. These new communication channels of the net in combination with GPS-Systems, satellite transmission and WAP mobile phones are allowing new forms of interactive personal and collective cinema on a digital basis."

 

Since this was written Shaw and Wiebel in 2003, we have seen several of their predictions come alive; online worlds like Second Life, immersive web based multiplayer games like World of Warcraft, to name but one, and numerous examples of the emerging narrative worlds Shaw and Weibel pointed out as the future.

The following examples are primarily selected as references to the interactive narrative installation PRAMnet have investigated, they can be seen as examples of state-of-the-art narrative works where interactive and digital dynamics have transformed the visual screen during the last decade.


Keywords from FUTURE CINEMA / re-combinatory / immersive

/ Examples Re-combinatory /

Interactive cinema in its early versions meant using different user interface schemes to re-edit and re-sequence pre-recorded material. Lev Manovich developed "Soft cinema", a software which shifted pre-recorded and edited footage into pre-programmed sequences on a multiple screen divided into a Mondrian grid. A sort of meta-collage using algorithms for meta-tagging of a documentary content, a database cinema presented as an installation. "Soft Cinema" has no open interactive interface, you watch a stream of visuals, a narrative collage using voice-over and soundsscapes from the recorded footage.

The software behind "Soft Cinema" are designed to combine keywords and tags to create different flows, or streams, of experiences. The keywords are combined with different tempo and time-capsules to make innumerable and multiple variations.

Illustration 1: Lev Manovich_"Soft cinema

 

The "Decay of Fiction", an interactive drama documentary on DVD, are based on film sequences from interiors in a Hollywood hotel, THE AMBASSADOR. Here seamlessly narrated video and sound emerge from the corridors and rooms as you enter them on the screen with movements from your mouse. The visuals are interfoliated with still imagery, audio and footage from film sequences shot in the hotel during its Hollywood glamour years. These images are layered on top of the actual interiors, creating an almost ghostlike and virtual presence.

This piece use a labyrinthine structure, you explore a narrative world using screenbased interaction and navigation controlled by the mouse on a DVD-platform. The material in the piece are pre-edited and combined in an simple interface setup.

Viewing and experiencing the content of "Decay of Fiction" are more interactive than the first glance makes you believe. Stay long enough on a spot in a corridor or at a stage in the old hotel and randomly organized footage and clips linked to this spot are screened. The programming behind use a combination of metatagging and scipts that could be transformed into a roomlike experience exchanging the mouse with a body movement in an installation.

This multiple documentary are shot by filmmaker Pat O‘Neill and this expanded cinema setup are created in cooperation with The Labyrinth Project at the Annenberg centre, USC.

Illustration 2: Pat O'Neill_"Decay of Fiction"

 

Multiple screen /

Video artist have used multiple screen format to construct narratives – Eija-Liisa Ahtila with works like "Consolation Service" use divided screen and multiple viewpoints to create a narrative, which change viewpoint between the characters and move freely in time and setting – this creates an associative form of experience for the audience.

The three screens also create a roomlike experience, you get close to either of the characters in the fictional story.

Isac Julien, a visual artist and filmmaker, have been using multiple screen movies more than ten years, creating imagery that blend different point-of-views and narrative elements. In one of his recent works "True North", a three-channelled video screening, Julien tells the story of the first black person to visit the North Pole. The images mix landscape views with personal reflections and fictionalized documentary scenes that create a narrative flow and mix.

This model of using the screen creates a panoramic view and an combination of close-up and wide-angle shots creates a open narrative form, the viewer is almost placed in the editors chair. Moving from one screen to the other you compose an personal experience.

Illustration 3: Isaac Julien_"True North"



/ Examples Immersive / Video as texture / digital material.

Many media/digital artists have developed new means of combining live-feed with pre-recorded video. Digitally synthesized graphic content are used in semi-autonome processes, designed to interact with motion-movement setups or controlled by a VJ-system or programming feature.

The emphasis of these immersive videoworks lies more on texture and rhythm, using expressive colour and graphics to create an emotion and feel rather than on story and narrative. The moving images can be projected on a flat screen as graphic illusions or in a stage setting on objects or performers, creating texture and dynamic tension.

An early interactive example are "Text Rain", by american media artist Camille Utterback, where installation participants stand or move in front of a large projection screen. On the screen they see a mirrored video projection of themselves in black and white, combined with a color animation of falling letters that can be "caught" on the contours of your body and made into words and poems. A playful and engaging work.

Another examples of this artistic strategy include Norwegian media artist HC Gilje who worked with manipulated video on stage performances, as video-sound-pieces with live musicians as in the early piece "videonervous". "The main focus of the project was to make use of the immediacy which the digital technology offers, and develop video as an instrument, but also to create fluid spaces through projections, and the ability of video to function as both a set design element and a narrative element".

Gilje investigated these possibilities in a serie of stage performances in collaboration with dancers and musicians in norwegian Kreutzerkompani, creating interactive, live-feed videoloops as stage settings. On his blog "Conversation with spaces" HC Gilje keeps track on his ongoing work with programming visual space - a series of "relief projection installation" which he refers to as "a dynamic audiovisual landscape, a spatial light painting".

Illustration 4: HC Gilje_"Videonervous 2 01"


Digital noise and manipulation /

Another form of immersive performances are made by he Austrian artist duo GRANULAR SYNTHESIS, their aesthetic concept can be seen as an audiovisual abstraction, using digitally generated images and sounds. "How does an image fall into pixels as the resolution gets lower and what is falling into pixels, and is falling into pixels the right equivalent anyway? Any resulting algorithm or translation we see as a means to create an abstract reality with emotional substance and depth, a subjective artistic invention."

One of these works are "Model 5", an live-feed installation where five large screens display portraits of people that slowly dematerializes into digital noise and back into images again; here the audio and visuals are manipulated live as a large scale VJ session where the process are controlled by the artists. Real-time and hypnotic.

 

Illustration 5: Granular Synthesis_"Model 5"



Other more playful and consumer oriented works include IOO Interactive, an Italian company that create environments and events using small-scale dynamic media – sensitive wall / floor were imagery and surfaces change and reacts to audience physical movements. These setups are used as demonstrators at public fairs and exhibitions on trade shows.

Navigable /

Tracking devices and sensor tools can be used to create an immersive room were physical movements are interpreted and correlated with pre-recorded video / film sequences. These dynamic set-ups are in a sense similar to virtual reality techniques and game mechanics, you're movements in the space create an interactive visual experience. You put yourself in the players seat, you become the editor and director of the experience.

In "Threatbox.us" by artist Marie Sester, an art installation with web surveillance interface, a movie frame "attacks" visitors via a robotic video projector and computer vision tracking system. On the screen you directly targeted by documentary images, you are hold at gunpoint in extreme and violent situations. Installed in a gallery space, this setup creates a claustrophobic comment on how public space can be controlled and occupied, and used almost as a terror device.

The classic piece "I Cinema" are an immersive sound and video system inside a circular dome constructed by artist Jeffrey Shaw. His first installation piece using this dome was EVE (Extended Virtual Environment) in 1993. Here the projector are controlled by the audience, as you "shoot" the beam around on the walls, you see a display of images, video clips and sound emerging from the inside of the dome. The experience is like watching through a peep-hole into a secret world, you actually navigate in a constructed 3D environment with video, graphics and sound.

At Future Cinema exhibition French artist Jean Michel Bruyère used the dome for creating a narrative interactive piece based on Ovidius: Metamorphoses.

Illustration 6: Jeffrey Shaw_"EVE (Extended Virtual Environment)"


Smaller scale projects using sensor and tracking technologies are numerous, the one-to-one sized "Men that fall" by artists Geska Helena Andersson and Robert Brecevic collaborating at Performing Pictures group in Stockholm are one simple, but striking example. An interactive one-to-one video image of a person, who - if you get to close to the image - starts to fall down. Back away and the person rises again. Displayed in a public space, a shopping mall, this piece created laughs and playful interaction.

Immersive and navigable /

Large-scale public interactive works using video and projection techniques are often costly and therefore relatively rare. In 2005 Mexican-Canadian artist, Rafael Lozano-Hemmer developed the large-scale public art installation Under Scan featuring 1,000 interactive portraits. It was created specifically for the East Midlands region of the United Kingdom.

Here people walked around on a brightly lit square, turned into a gigantic dynamic screen. In their shadows on the ground surface video-portraits emerged. Their bodies moved and their heads turned to look straight at the passers by. When a shadow moved away from a portrait, the portrait likewise reacts by losing interest and looking away. This large-scale work combine surveillance cam-technique and as Lozano-Hemmer describes it "...a system that automatically points 14 robotic-controlled projectors to the locations where they will intercept the trajectory of pedestrians and cues a sequence of video for them. "

Illustration 7: Rafael Lozano-Hemmer_"Under Scan"


The sophisticated and advanced technology behind Lozano-Hemmer public installations are made in collaborations with programmers and software developers on large-scale sights around the world. His piece Vectorial Elevation, celebrating the arrival of the year 2000 in Mexico City's Zócalo Square used eighteen powerful searchlight beams, illuminating the night sky over the central square. The design of these light sculptures were controlled by an online 3D simulation program on a website open for everyone and transformed the sky into a screen were geometrical shapes formed stunning visuals.

In one of his latest large-scale public work, Solar Equation, Lozano-Hemmer use the new portable screens and interfaces on handheld devices and smartphones as a dynamic dynamic tool. On a large inflated ballon, video simulations of the suns turbulent sunspot acitvities are projected, the forms and intensity of these shapes are controlled from an app that can be downloaded from a website. This live-feed interactive animation where staged at the Light in Winter Festival in Melbourne 2010, uses actual data from NASA satellites as base for the simulations created by mathematical, fractal equations.

Dynamic screen / room in relation to "Journey to Abadyl" /

A variety of these techniques using video and on-screen narratives are possible to develop further. Different combinations of narratives based on computer game engines and scenarios can be investigated and tested. The examples mentioned in this brief expose are but a few of the the multiple area of possibilities made accessible with digital tools and combinations of software/hardware techniques.

In our research we concluded that the main limitations of using videostreams as content in interactive artworks are the linearity of the images. True you can find exemples of real-time editing and rendering of footage online, art-pieces that enables you to create your own moving image edition from a database on the fly. You can scroll back-forward on image sequences and you can digitize videoimages in real-time using pre-edited filters and effects from opensource software.

In PRAMnet we have worked with and devised different setups, where physical movements and positions in a sensor-tracked room can trigger images, sound, video and sequences of narrative elements.

Jörgen Callesen's Falling man was an early prototype using quick time loop and software, so a performer with his/her body movements physically control the actor on the screened video. Callesen's interactive video sculpture Emotional Fish, performed at Warehouse9 in Copenhagen September 2007, elaborated on this technique using computer generated images and sound set up with infrared light and sensors in a public space.


Illustration 8: Callesens interactive performance "Emotional Fish"



The prototype Wunderkammer used a seemingly easy setup – two opposite screens and a tracking device using a webcam capturing movements on the floor. Audience control video/sound in a setup with "good-bad" imagery by moving around in the room – together and away from one screen the opposite screen are active and vice versa. In the prototype model we tested, the setup was made on a table, using black and red miniature figures mounted on a stick, almost as puppets, with two 19" inch computer screens.

Illustration 9: PRAMnet prototype "Wunderkammer"


The complexity of these setups can be scaled up to four screens and database organized sequences. It can be used as a setup for live-feed immersive video and sound environment as well as more narrative based scenarios. Creating a dynamic screen and room where the laws of time and space can be altered and transformed.

In "Journey to Abadyl" context /

During our workshops and discussions we found it useful to percieve and organize the tools and technologies according to two basic forms and rules.

Narrative moving images can be organized in two major models – pre-recorded and edited and live-feed set ups.

The pre-recorded can be divided up in different narrative strategies like – background story, eyewitness tales, multiple-choice scenarios of events, arguments and discussions between characters.

Using these models we can establish a virtual setting, create back-stories of characters, which the audience encounter or/and recreate past or future events. Here a the game mechanics and narrative dramaturgy rules and scenarios developed by computer game designers form a backdrop of knowledge and experience on how to set up a database of content.

The live-feed set up use live manipulation and video feeds – as interaction it can be controlled by a performer/guide or by the participants. The viewing and experience can be orchestered by various scenarios such as simultaneous events in a labyrinthine setup, by a "VJ organ" controlled/triggered by physical movements, by real-time stage-set design using projected colour and synthesized images.

From these basic models we can choose to develop media for different narrative setups, using specific story tools for different city parts. Thus creating a variety of possible interactive setups with video narration and moving images.

( The JUKEBOX in TRAORA can be seen both as a visual metaphor and a real tool for eyewitness tales and ceremonies – music for the "masses" and entertainment machine. Newscasts with debates between opposing political leaders. Environments in the jungles of ARLEEANS. Historic re-creations of events. The responsive digital guide - controlled by audience movements. )

The short movie IN SEARCH OF THE MILITANT CODE, a collaboration between Thore Soneson and artist Michael Johansson, is a story set in the city part NEEBI. An outtake from the fictive world of Abadyl. This work explores the possibilities of combining narrative strategies and tools from traditional filmmaking and a computer generated universe. Video and digital sceneries in 3D are interfoliated in a mockumentary style.

Illustration 10: Thore Soneson_"In search of the Militant Code"

 

 

******

I wrote the essay DYNAMIC SCREEN as a contribution to PRAMnet seminar WIRELESS held in Copenhagen 2008. It's part of our ongoing research for the project "Journey to Abadyl". We published a book-let with essays and scenarios as a result of the seminar and our work - "Journey to Abadyl" – a pdf copy can be found at our webpage http://pramnet.org.

"In 2008, PRAMnet received a pre-production grant from Nordic Culture Point to investigate a scenic media production with the working title CubeX, an interactive performance-installation with audience participation.

At the seminar WIREFRAME held in 16-18 May we introduced our research and presented ideas for the script and drama development, for game patterns and mechanics for workable models on playdesign, and methods of interaction.

In this publication, we present results of this research work - "Journey to Abadyl" - an innovative performative synergy between performance art, computer games and interactive film.

We hope this material can form a basis for further development of this never-seen-before performance art project."

Thore Soneson march 2011
www.soneson.se

 


On-line references

FUTURE CINEMA / The Cinematic Imaginary after Film /
/ ZKM exhibition and catalogue edited by Jeffrey Shaw and Peter Weibel 2003
http://www.zkm.de/futurecinema/index_e.htm

Tracing the Decay of Fiction: Encounters with a Film by Pat O’Neill
The Labyrinth Project
http://cinema.usc.edu/programs/imap/projects/the-labyrinth-project.htm

SOFT CINEMA NAVIGATING THE DATABASE Lev Manovich
http://softcinema.net/?reload

Granular Synthesis Model 5
http://www.medienkunstnetz.de/works/modell-5/

Men that Fall
http://www.performingpictures.se

EVE (Extended Virtual Environment) Jeffrey Shaw
http://www.mediaartnet.org/works/eve/

Conversations with spaces HC Gilje project blog
http://hcgilje.wordpress.com/

HC Gilje projects 1998-2006
http://hcgilje.bek.no/projects.htm

THREATBOX.us, by Marie Sester
http://www.threatbox.us/concept/index.html

Under Scan, by Mexican-Canadian artist Rafael Lozano-Hemmer
http://www.lozano-hemmer.com/under_scan.php

Emotional Fish, by Jörgen Callesen
http://fishfingers.dk/callesen/?page_id=144

In search of the Militant Code by Thore Soneson and Michael Johansson
http://www.soneson.net/militantcode/index_eng.htm

(Links edited and checked / 2011 03 13)

 

Thore Soneson is a filmmaker/writer/producer in Sweden. His professional roles in moviemaking and for the stage have been both artistic and producer oriented, as well as a dramaturge and scriptwriter. Special interest in developing projects based on the expanding possibilities of storytelling with moving images in new media. Projects: JOURNEY TO ABADYL (performative project involving performance art, computer games and interactive film, 2009), IN SEARCH OF THE MILITANT CODE (short movie mockumentary, collaboration with Michael Johansson, 2008), LOOP 1.0 (interactive flash movie, 2005-2006), THE STORY OF A (interactive narrative script, 2002), SPEED (interactive CD-ROM, 2001). More info:www.soneson.se

 

 

 

HOME

ARTICLES

---

---

---

 

FYLKINGEN'S NET JOURNAL