Survey of Artists Working with Activated Objects, Mixed Reality, Physical Computing

Stephen Wilson, Conceptual Information Arts Program, SFSU

Toshio Iwai
Piano - as image media.
 In this work a semitransparent screen is attached to a computer-controlled grand piano. Control of the piano is linked to the computer-generated image projected on the screen, so that anyone can play the piano by altering the image.  See also Violin ~ image of strings

Composition on the Table
The installation offers series of artwork which represent the concept of Mixed Reality. Four white tables have various user interfaces such as switches, dials, turn-tables and sliding boards that a player can touch. Projectors suspended from the ceiling project computer generated images onto the tables and interfaces. Projected images change in real time as if they were physically attached to the interfaces when players operate them. Also sounds are produced in relation to the movement of images. Since the interfaces have close relation to the reaction of images, players can operate images and sounds in the same way when he/she operates ordinary interfaces and gradually feels these illusions as equivalent as the actual objects.

tom white and david small
stream of consciousness
an interactive poetic garden:    A six foot square garden sits in the middle of an otherwise ordinary computer lab. Water briskly flows down a series of cacsades into a glowing pool. Projected on the surface of the pool and flowing as if they were caught in the water's grasp are a tangle of words. You can reach out and touch the flow, blocking it or stirring up the words causing them to grow and divide, morphing into new words that are pulled into the drain and pumped back to the head of the stream to tumble down again.

Masaki Fujihata
Beyond Pages
In Beyond Pages, the observer enters an inviting and natural situation. While working, the dramaturgy is fixed: One sits down at a table and finds a book and a pen. This arrangement is clear and forces an evaluation of the medium. From childhood on, people in most cultures are familiar with the actions of turning pages and reading. For Fujihata, the book represents an information conveyer and storage medium, also serving as a symbol for textual culture and the practice of a linear method of reading. His interest in the testing and use of multi-media technology allows him to analyze the qualities and push against the limits of the medium.

Although Beyond Pages presents a book as interface and simulates the action of turning pages, the two-dimensional limits of the surface and the inflexibility of the symbol are elegantly exceeded. To expand the usually quiet and still form of illustrated text, Fujihata introduces moments of surprise. Three-dimensional moving elements appear on the surface of the pages, their movement coupled with acoustic signals. The observer�s continual smile speaks for the effect of these wonderful moments and underscores the tension between perception and recognition. The amazement intensifies as the scope of events is broadened by an unexpected and sudden change: The lamps light up and a very short projection of a smiling child within a door appears. In Beyond Pages, Fujihata refers to technology�s potential for shaping valuable content with fantasy, concentration, and curiosity.

Global Interior Project: Networked Multi-User Virtual Enviroment Project
Virtual space is designed in this project with a certain number of cubical rooms, and all the rooms are interconnected. The interior design of each room resembles the interior of the Cubical Terminal. Participants can move from room to room by navigating from the Cubical Terminal.
"Matrix Cubes" are made up of stacks of boxes with doors. These represent the links between virtual rooms or stacks of "Terminal Cubes" or stacks in the real world. The Matrix Cubes thus function as a miniature / metaphor / map of the virtual space. In this system, it is possible for a participant to have a threefold existence: Real Me, Virtual Me, and Virtual Me in the Actual World. For example, an action conducted with the image of Real Me through virtual space causes a reaction on the part of Virtual Me: Where are you? What is the meaning of your address? Where is your location? How is your existence supported? The whole topology of your existence will be shaken up.

Mersea Circles
Archived video images that are connected with a particular geographical area give rise to a collective memory. In pursuit of this phenomenon, a two-day event was organized in the summer of 2003 on the tiny island of Mersea in the vicinity of Colchester, Essex. Approximately 120 people hiked in groups along the coast and used their digital cameras to make a record of their activities, conversations and the landscape. The coordinates of the locations at which the images were shot were recorded by a GPS device.

The next phase of the project was post-processing in the studio, where a three-dimensional cyberspace was created. Within this virtual space, each video sequence was assigned—according to the GPS data—to the spot at which the recorded event actually took place. Now, when the viewer approaches the screen, the corresponding sequence is played back.

Perry Hoberman
In Timetable, an image is projected from above onto a large circular table. Twelve dials are positioned around the perimeter of the table. The functions each of these dials changes and mutates, depending on what is projected onto them at any given moment. Dials can become clocks, gauges, speedometers, switches, steering wheels, and so on. A real-time 3D scene, projected onto the central part of the table, is controlled and influenced by the movements of the dials.

At the outset, the space of Timetable seems to be rational and unified, but the longer the piece is used, the more complex and multi-dimensional it becomes, as perspectives and timeframes diverge and split off from each other. Timetable attempts to make certain paradoxical (even impossible) pseudo-scientific concepts into concrete experiences, such as time travel, multiple branching universes, alternate dimensions and shared hallucinations

Systems Maintenance
Systems Maintenance consists of three versions of a furnished room. An ensemble of life-sized furniture occupies a large circular platform on the floor, a virtual room is displayed on a computer monitor, and a 1/8 size physical scale model of the room is presented on a small pedestal. Each version is imaged by a camera (either video or virtual), and the three resulting images are combined into a single large-scale video projection. The camera position, height, angle and field of view are matched between the three cameras. By moving the furniture and camera viewpoints for each of the three rooms, visitors can match or mismatch the components of each of the rooms as they appear in the projected image. The three video signals are fed to a pair of video mixers which are used to perform an additive mix of the three signals, and this combined signal is sent to the video projector

Kyoko Kunoh   & Motoshi Chikamori
Kage - Tool's Life
When useful physical objects are gently touched, their shadow-like silhouettes magically begin to move and can assume a wide variety of forms. The objects themselves do not change their shape, whereas their shadows reveal their true character or their secret wishes. "Tool's Life" not only illustrates the function of these objects; it also brings out background factors that usually go unnoticed and the various significances of the objects' use.

ART+COM (in collaboration with Hürlimann + Lepp Ausstellungen) - Floating Numbers table

Project to illustrate the power of numbers and signs in Judaism.  On a 9-metre long table, numbers are flowing in a continuum. Digits appear randomly at the surface of this stream of numbers. Visitors could catch a number by tapping on it and it made the digits "explode" into an encyclopaedia entry for the number, featuring text and image or a movie.

Giorgio Olivero and Peggy Thoeny - Tableportation
A system that allows café guests from different tables to observe each other and be observed.  People can draw light points on their table using touch, water or metal objects. The translucent surface is a matrix of touch-sensitive points; creating contact between adjacent points causes the light source below to illuminate. Activating specific combinations of metal points triggers different light behaviours. Video images of guests' activity are projected on the wall so that their edges overlap. This mosaic of table tops creates a space for new forms of social interactions. People can play with each other in the virtual space on the wall in ways they would never do in face-to-face situations.

Maxime Lecours  -  Little table

Little table, will be able to communicate with users on the basis of the traces they leave.  Other people would know about your previous presence, because you have left for some time your prefered object, such as a cup of coffee, a pen or a book, on the table. Depending on the duration of that object on the table, the projection, the intensity or the general pattern of the light could be altered. Also, different forms of light projection could be use to express the choosen kind of information. For example, a red/white projection colors would mean the person is feeling good. Depending on how many times, and how fast you move the object, the light projection will be modulated. The table will be able to track and move the light projection as well, following the path of the object if you move it.

Sei Matsumura - Hop Step Junk

an installation which catches up people's footstep. According to the difference of their weight and the way they move, the sound of footsteps is turned into rhythm sequences and reflected into visualization.  People just step on one of the stages, make tap dancing-like footsteps and the graphical pattern reacts. If someone steps with enough loudness, the sound of footsteps is recorded and played in loop until another visitor comes.

Pasta and Vinegar - Space and Place - list of Interactive Tables
We Make Money - List of Physical Computing Projects

Servo  Smart Studio

The responsive field of lattice archipelogics   Lattice archipelogics is an installation designed by Servo where a partnership was formed with the smart studio to design the interactive apects of the installation - what we call the responsive field. Servo designed a hundred plastic modules with cavities and conduit capable of holding wiring, sensory, sonic and lighting equipment in space.

The responsive field - designed by the smart studio working together with servo - is a dynamic interactive audiovisual environment. Moving through the physical space will affect digital " agents" which exist in a hybrid physical/immaterial world. The movement of the agents will be rendered using lights in the modules and a 3D sound installation.

Our physical environments are increasingly turning into hybrids of build matter and digital effects, creating ephemeral organisations of mixed realities occurring at and beyond their geographic location. Most of the underlying technologies are modular and standardised (IP networks, graphic formats, mobile technologies, etc..), yet the scenarios and effects played out on those organisations can be highly differentiated.

Jamy Sheridan
Tree of Life  - The visual component is a real-time computer animation projected onto a bed of white sand on the floor in a visually and acoustically isolated space.  DNA structure affects visuals and sounds.

Kaeko  Murata , Eiji  Yamauchi
Fisherman's Café
Shadow fish swim across a cafe table as a means of silent communication. Participants "drink tea" as part of the process, and their movement affects the activities of the fish.

Jay Lee; Bill Keays
Suspended Window
An image of LA is influenced by images of those passing by. This interactive installation interlaces multiple layers of real and virtual surfaces, effectively suspending the normal function of the real window. As viewers wander through the interaction zone, they find themselves hovering between the laminations of this fictitious space.

Georg Ritter

AUTODROM ** — the man-machine interface, a two-way feedback loop interlinking mankind and the virtual world.

The human-controlled machines moving along an actual test track have their equivalent in virtual space in the form of 3-D characters. An additional computer-generated factor has been engineered into the behavior of these figures — the power failure for a specified period resulting from a collision with one of them, analogous to machines in the real world.

Research Projects in Mixed Reality  

(special focus on those that can be simulated with video tracking)

Dan Chak and Professor Hiroshi Ishii  -  MIT Tangible Media
AirportSim aids an airport manager interested in efficiency to distribute resources throughout a model airport, balancing cost with customer satisfaction. The AirportSim application is made up of a number of user-manipulable simulation objects that define the airport layout and parameters, such as check-in counters, waiting areas, and security checkpoints. Simulation controls affect the simulation globally, including a terror alert level control that allows the manager to plan for different emergency alert levels as defined by the United States Department of Homeland Security. As passengers walk through the virtual airport, the manager can identify bottlenecks and make changes in real-time to increase the efficiency of the workforce.

Yao Wang, Assaf Biderman, Ben Piper, Carlo Ratti, and Professor Hiroshi Ishii - MIT Tangible Media
SandScape is a tangible interface for designing and understanding landscapes through a variety of computational simulations using sand. Users view these simulations as they are projected on the surface of a sand model that represents the terrain. The users can choose from a variety of different simulations that highlight either the height, slope, contours, shadows, drainage or aspect of the landscape model

James Patten, Gian Antonio Pangaro, Matt Reynolds, Jason Alonso, Joseph Panganiban, Jim Hines, and Professor Hiroshi Ishii - MIT Tangible Media  Sensetable
Sensetable is a system that wirelessly tracks the positions of multiple objects on a flat display surface quickly and accurately. The tracked objects have a digital state, which can be controlled by physically modifying them using dials or tokens. We have developed several new interaction techniques and applications on top of this platform. Our current work focuses on business supply chain visualization using system dynamics simulation.

Brygg Ullmer, Anna Lee, Nancy Sun, and Professor Hiroshi Ishii - MIT Tangible Media

The Tangible Video Browser provides a tactile and efficient interface for viewing digital videos. Tokens act as both containers for a set of videos and controllers for selecting a video and navigating within the video. Placing a token on the interface enables access to the set of videos and depressing, releasing, and rotating the token controls the navigation.

Hiroshi Ishii, Rich Fletcher, Jay Lee, Joanna Maria Berzowska, Seungho Choo, Craig Wisneski, Charlie Cano, Andres Hernandez, Colin Bulthaup  -  Tangible Media

musicBottles present a tangible interface for experiencing a musical composition. The salient concept is that of bottles as graspable containers and controls for digital information. Physical manipulation of the bottles is the primary mode of interaction with this information. The installation consists of a specially-designed table upon which three corked bottles are placed. These glass represent the three performers in a classical music trio. Custom-designed electromagnetic tags embedded in the bottles enable each one to be wirelessly identified and sensed. The movement and uncorking of the bottles controls the different sound tracks and the patterns of colored light that are projected onto the tables' translucent surface.

Responsive Environment Group, MIT

Media Matrix is a system applying distributed, embedded computing techniques to the creation and maintenance of a queriable database of physical objects such as compact discs, video cassettes, books, and component bins. Scalable: there is no limit to the number items that can be added to the collection. This system would work for an entire library just as well as it would for a personal collection of mini DVs. Decentralized: there is no central database or computer acting as a bottleneck or weak link. Any portion of the database can cease functioning without hindering overall search performance. Self-organizing: the items in the collection never need to be manually sorted as long as they remain on the shelf. Furthermore, the act of physically adding/removing an item to/from the shelf automatically registers/unregistered it with the rest of the collection. Fast: all queries are carried out in parallel, making for a fast response time

James Patten, Ben Recht, and Professor Hiroshi Ishii  - MIT Tangible Media

Audiopad is a composition and performance instrument for electronic music which tracks the positions of objects on a tabletop surface and converts their motion into music. One can pull sounds from a giant set of samples, juxtapose archived recordings against warm synthetic melodies, cut between drum loops to create new beats, and apply digital processing all at the same time on the same table. Audiopad not only allows for spontaneous reinterpretation of musical compositions, but also creates a visual and tactile dialogue between itself, the performer, and the audience.

Professor Robert Jacob, Gian Antonio Pangaro, James Patten, and Professor Hiroshi Ishi  - MIT Tangible Media
Senseboard is one facet of our using tangible media for manipulating abstract information. It allows the user to arrange small magnetic pucks on a grid, where each puck represents a piece of information to be organized, such as a message, file, bookmark, citation, presentation slide, movie scene, or newspaper story. As the user manipulates the physical puck, the corresponding digital information is projected onto the board.

Ali Mazalek, Alison Wood, and Professor Hiroshi Ishii - MIT Tangible Media
Tangible Viewpoints is a system for interacting with a character driven narrative. The different segments of a multiple point-of-view story are organized according to the character viewpoint they represent, as well as their place in the overall narrative. These segments can consist of various types of media (video, audio, images, text), and can present character development, action and location with as much complexity as any scene of a film or chapter in a book. The interface uses wirelessly sensed graspable pawns to navigate through the multiple viewpoint story. When a pawn is placed on the sensing surface, the story segments associated with its character's point-of-view are projected around it. Users select segments to be displayed on a nearby monitor, causing the narrative to advance and new segments to become available. An aura is projected around each pawn to give a visual representation of the prominence of that viewpoint in the current telling of the story. Changes in the story space are reflected by dynamic changes in the projected graphics

UIAH Media Lab
Mixed Reality Pong
 a mixed reality version of the classic "Pong" game. The aim of the game is to score goals by hitting a virtual ball over the other end of the game area protected by the opponent player. The game counts the goals the players have scored, and they can agree to play either for a limited amount of time, or until either of them has scored a certain amount of goals.

Marek Walczak, Michael McAllister, Jakub Segen, Peter Kennard

Dialog Table is a shared interface where you use hand gestures to discover more about any data set. Several people can gather around and together explore the table’s movies, narratives and 3D journeys. The table provides an opportunity for people to discuss with each other their thoughts on what they have seen, whether it be an artwork or a service.

Dialog Table was commissioned by the Walker Art Center as a permanent installation in their museum. The table won an international design competition to promote social interactions among visitors, to provide access to the Walker's multidisciplinary colle

Lumisight Table - NTT & Tokyo U 
Lumisight Table is a novel interactive table display that can display information in each required direction on a shared screen and capture multiple users' gestures simultaneously.

Dipak Patel, Aoife Ní Mhóráin, Stefan Agamanolis - Media Lab Europe - Habitat

In family relationships, awareness of daily cycles and routines (or more importantly, deviation from these patterns) is particularly important. This awareness helps to convey reassurance and a sense of context for communication, and it provides a means for background synchronization of rhythms between those in the bond. Habitat explores the potential of using household furniture as a network of distributed ambient display appliances for conveying this kind of awareness between family members separated by a distance.