Summary: Potential Contributions of the Arts to Research Agendas in Ubiquitous Computing and Gesture Understanding.

Stephen Wilson, SFSU

The project proposes to study the potential contribution of the experimental arts/media to IT research and to study possible ways of organizing curricula to prepare non computer science students for research roles. One core premise of the project is that students are likely to become engaged when they have real experience with contemporary frontier research areas likely to have significant future cultural impact. This project identifies pervasive computing and motion/gesture understanding as two such areas. It is anticipated that the innovative collaborative research organized by this project will produce valuable insights in these fields in addition to its curricular value in providing IT training to students outside of IT. This section reviews the import of the areas and identifies areas of inquiry in which the arts may make a contribution.

Research Issues in Pervasive Computing

Many researchers have become convinced that the current paradigm of computing that posits a specialized device sitting on a desk or being carried as a PDA will change radically in the future. The evolution of hardware, software, and communication will enable the IT activation of everyday objects and settings. Sensing, computing, and communication functions will become invisible and integrated into the manufacture of many objects and the architectural arrangements of spaces. For example, a doorknob might be able to sense the identity of the opener (conceivably, via fingerprints or other biometric characteristics) and activate other objects and functions in a room that make sense when a person enters.

This cluster of research activities goes by a variety of names including Ubiquitous/Pervasive/Invisible/Embedded Computing, Tangible Bits, Augmented Reality, Smart Spaces, and Smart Objects. Mark Weiser, an early pioneer in ubiquitous computing at Xerox PARC described the rationale and questioned the metaphors used to envision the future of computing. He critiqued television, multimedia, GUI, intelligent agents, virtual reality, voice response, magic wish granters, and the like because they still postulated special devices as the center of attention.

Ubiquitous computing names the third wave in computing, just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives÷.Ubiquitous computing is roughly the opposite of virtual reality. Where virtual reality puts people inside a computer-generated world, ubiquitous computing forces the computer to live out here in the world with people÷.Our preliminary approach: Activate the world÷. This is different from PDA's, dynabooks, or information at your fingertips. It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere. (Weiser, 93A) Research in ubiquitous computing has developed in many directions, varying roughly along a continuum of how much integration is proposed. IBM's pervasive computing group's web presentation on their rationale emphasizes the need for seamless integration of specialized devices including home entertainment and other appliances that have been information-activated. They note the importance of this integration as the boundaries between workspace and home blur, "allowing them to easily take action anywhere, anytime" (IBM, Pervasive Computing, 1999)

Donald Norman, who is famous for his ideas on design of high technology devices, critiques the complexity of technology-based devices. He believes that ideally technology should become as "invisible" as the other devices that fill our worlds. In his book Invisible Computing he stresses three principles: Simplicity, Versatility, and Pleasureability. In the preface he stresses the social and cultural fit between technology and life as it is normally lived. (Norman, D. A., 1998)

Touch is an important feature of normal non technical life. Fitzmaurize note that the "graspable user interface" solves long-standing problems with keyboard and mouse interfaces. It uses a wider range of physical and cognitive human capabilities such as characterize everyday life. (Fitzmaurice,Ishii, & Buxton, 1995)

MIT's Media Lab also has several groups seeking to go even further. The Tangible Bits group seeks to move into activating of objects and architectural features of spaces. The long history of human experience with physical objects is a valuable resource that should be used and celebrated. They seek ways to add "intelligence" to the physical world. (MIT, Tangible Bits Group, 1999) Tangible Bits is part of a larger MIT Media Lab consortium called "Things That Think" which pursues this vision of adding intelligence to the most mundane of objects and spaces.

Technology can not truly be helpful unless it can provide the information that you need where you want it, when you want it, and without you needing to manage it.

Think about the engineered artifacts that we are surrounded by most of the time. We wear clothes, put on jewelry, sit on chairs, and walk on carpets that all share the same profound failing: they are blind, deaf, and very dumb. Cuff links don't, in fact, link with anything else. Fabrics look pretty but should have a brain, too÷.We must expect more from our environment.. (MIT Media Lab, Things That Think web site, 1999)

The attempt to activate physical objects and spaces requires major research efforts to understand and develop the electronic, sensor, computational, and communication capabilities that could enable these kind of systems. These visions pose significant research challenges stretching from the basic electronics, networking, and software infrastructure through software and hardware design up to cultural-technological issues about the way this technology can fit into human settings. See for example the list of issues in Microsoft and the National Institute of Standards and Testing(NIST) co-sponsored "Technologies of Invisible Computing" conference. (Microsoft, 1999).

The impact of psychology and social context on the design of pervasive computing systems are significant and often de-emphasized by IT researchers. In the special issue of the IBM Systems Journal focused on Pervasive Computing, (IBM, 1998b) several of the authors reported on research on these kind of questions. For example, Mark proposes that one cannot really design pervasive computing systems without understanding the ways it interacts with the ways people usually relate to physical spaces. (Mark, 1999)

Cultural Themes related to Pervasive Computing, Smart Objects

History of Religion:. Animism as feature of many cultures (Eliade, Jung). All objects alive; full of sensibility, spirit. Repression of Animism as major part of move toward patriarchal structures. Alienation from physical world.

Ethnomethodology (eg Garfinkle) and Situated Sociology (Suchman): Each culture and microculture develops its own structures about the meaning and relations with things and architecture.

Don Norman: Nature of objects and their context - physical, semantic, cultural, logical

Piaget: Child's conception of the world. Development of categories of live and not live. Developmental decline of animism. Dennett live-not live part of folk psychology.

Nass - people treat media and computers as social beings. Sherry Turkle - differnet

Michael Travers (MIT) Agency - judged by autonomy, purposefulness, reactivity. Self moving vs externally moved.

Debates about anthromorphism - Laurel vs Schneiderman

Philosophical issues about smart things. Dreyfus, Searle, Penrose. How do smart objects relate to discussions of embodiment?

Gershenfiekd, Neil When Things Start to Think.

Specific Areas of Inquiry

The range of research in pervasive computing is exhilarating, covering areas such as: Sensor electronics; Communication electronics; Protocols for handling mobility and identifying location of objects and persons such as UBIC, shared digital identity systems; and Approaches for communication, interoperability, identification, and registration of objects such as HomeRF and Bluetooth. Slightly less research activity is observed at the level where the fundamental capabilities are integrated into everyday processes ­ for example Spatial Awareness of Objects; Time Span of Awareness; Methods of assigning functions to objects; Bond between the physical and the virtual; Orchestration of collaboration, Security and Privacy; Context creation and maintenance; and The social aspects and costs of pervasive computing.

We propose that the experimental technological arts could provide valuable perspectives to studies that investigate this level of integration of pervasive computing with everyday life and settings. This shift in paradigm to ubiquitous computing will require major research beyond the electronic infrastructure. - For example, in interface, computational models, and in understanding the technological imperatives that are implied by the psychology and anthropology of these new kinds of settings. What will it mean to have smart objects that can communicate with each other? What sensing and communication capabilities will people want these objects to have? What intelligence should a space manifest? Should the new objects respond to normal actions or should new sublanguages of gesture be developed? How do historical cultural expectations about inanimate objects and architectural space influence the ways new pervasive computing capabilities are conceptualized and used? Some specific areas of possible inquiry include the following. Examples of technological artist experimentation with these issues can be found on the Wilson Artlinks Website (Wilson,2000b)

Location: The pervasive computing literature suggests that location is important in ubiquitous computing in a way that is not significant with traditional discrete computers. The arts have long sought to understand the importance of place. For example, the ancient art of theatre set design empowers fictional spaces by understanding everyday spaces and architecture devotes considerable energy to inventing and forming spaces. The arts would suggest the following kinds of questions: What are the important elements of a location or context that are salient for pervasive computing systems to know - its structure, history, and connotations? Should objects take on different capabilities depending on where they are? How important is the relative location of objects in a space - for example, something out of its normal place? How can a system of smart objects determine what is important and store that information?

Agency The last decade has seen a flowering of agent research investigating the power of persistent, intelligent software to aid work with computers and networks. Will the change to pervasive computing eliminate the interest in agents? Researchers differ in their models - some forecasting objects with very limited capabilities tightly linked to the specific functions of the objects while others envision new kinds of objects with expanded intelligence. Researchers working within the social computing tradition suggest that humans will project social expectations even onto the new smart objects regardless of their scope. (D. C. Dryer, C. Eisbach, and W. S. Ark, 1998) Anthropology and the Arts suggest that people have always imbued objects with personality. Animism and imminence have historically been part of the way humans deal with the inanimate world. Researchers designing the new smart objects will need to decide what intelligence and personality should the objects have? How should this agency be manifest in interactions? These questions may become even more important with everyday objects than they were with discrete computers.

Ownership - Registration of Objects In the physical world most objects and spaces do not change function based on who is using them. The doorknob is just a doorknob. As intelligence and communication capabilities are added, however, the identity of the persons engaged with the items can become important. How will systems know who is present and how might they adjust their function?

Sensual Modalities of Human Interface. How should these smart objects and spaces communicate? The historical reliance on screen communication will no longer make sense. New haptic and kinesthetic models must be found - for example, vibration, motion, temperature, texture, rigidity. One could think of activated architectural spaces as new media environments. Similarly, the shape, materials, connotations, and other aesthetic qualities of objects become important part of the equation. The tactile capabilities of objects will need to extend far beyond today's crude force feedback prototypes. IT researchers have initiated many careful studies of tactile communication. (for example Lederman, 99 and Howe, 1999)) The arts with their long attention to the aesthetics of touch and their experimentation with animated objects can help in developing the new languages of things.

Gateway to the Virtual World; The advent of ubiquitous computing does not mean that the abstract worlds of data and virtual spaces will disappear. Some believe that indeed activated objects provide vastly expanded ways for persons to comprehend, navigate, and manipulate abstract virtual worlds. For example, walls that become windows into data, everyday objects that become controllers for virtual worlds. But these new kind of linkages raise fascinating research questions. What qualities of the physical world are appropriate or possible for mapping to a virtual world? How do differences in the nature of the abstract information and virtual worlds represented impact on how the physical world should be mapped? How does the blurring of the boundaries between the physical and the virtual create opportunities and difficulties?

Invisibility: Many models such as Don Norman's ideas stress invisibility and simplicity, as a counteraction to function overload seen in today's technological devices. The arts would suggest a balance between invisibility and innovation. It is entirely possible that some smart objects should not resemble their inactivated relatives. The arts would seek to craft new, unprecedented kinds of artifacts that enhance understanding of their capabilities, intrigue and engage users, and ease their integration into life.

Utilitarian Focus: Almost all the literature focuses on the usefulness of smart spaces in utilitarian functions such as commerce, home management, or education. The arts suggest that the advent of smart objects and spaces will create other possibilities that should not be neglected. For example, these new kinds of objects open unprecedented opportunities for self-reflection, building solidarity, mystery, play, contemplation, spiritual inquiry, aesthetics, and entertainment. Basic research should be part of the research agenda for ubiquitous computing. As has been demonstrated in the recent history of digital innovation, these investigations can yield unanticipated benefits down the road.



Research Issues in Motion/ Gesture Understanding

Researchers have been working for many years to expand the interface capabilities of information technologies to understand the gestures and motions of the people that use them. The expansion of IT into everyday settings will need to reach beyond the keyboard, mouse, and specialized input devices that have traditionally been part of information systems. There are several research streams addressing these questions including speech, immersive virtual reality, eye tracking, person and face recognition, activated objects (described previously), speech, and gestural interfaces.

Eventually it is hoped that people will be able to interact with information systems merely by acting in free space as they do with the normal physical and social worlds. See for example Sandy Pentland "Dance of Bits and Atoms" MIT Media Lab, 1999) To accomplish this goal researchers are working to develop systems that can track and interpret the motions of humans and facial and body gestures via video analysis, ultrasonic, and other kinds of sensor technologies.

We seek to investigate the ways that the arts (including performance art, theatre, and dance) can enhance this effort to understand computational detection of body motion and gesture and to build systems that can intelligently interpret and respond. The proposal concentrates on recognition research projects focused on specific actions of hands or bodies, specifically those that use image-based analysis rather than glove or similar instrumentation. The long traditions of the arts in attending to motion and gesture can be a fresh resource in the conceptualization of research questions, the conduct of research, and the development of hardware and software systems that can be integrated into everyday life.

Researchers are pursuing several threads in the attempt to build gesture recognition systems. They are pursuing a variety of hardware approaches including instrumentation of bodies and hands including badges, datagloves, datasuits, pinch gloves, reflectors, transmitters, and striping and image based systems that attempt to interpret the video stream and require no on-body instrumentation. Researchers are working on a variety of recognition and segmentation techniques to extract meaningful units of motion .

Examples include the numerous projects to understand sign language (Waleed, 2000), MIT Media Lab projects on understanding hand and body actions and creation of smart spaces to understand gesture; Microsoft, Mitsubishi, and ToshibaŪs efforts to develop gesture and motion understanding. Several projects focus on multimodal approaches, using combinations of speech, gesture, motion, and context to interpret behavior.

Machine based gestural understanding will need to learn from the insights of several disciplines. For example, psychologists and anthropologists have long studied non-verbal communication. Researchers such as Ray Birdwhistell, Albert Mehrabian, Edward T. Hall, Paul Ekman, Ross Buck study subjects such as nonverbal sending and receiving accuracy, gender & cultural differences, body language, proximics, motion, facial expression, and social perception and empathy. Research labs such as the Center for Non Verbal Studies (Center for non-verbal communication, 1999) focus on the importance of non-speech communication in everyday life. Machine understanding of motion and gesture inevitably must use these understandings to make sense of the behaviors it extracts

The arts can be a critical resource in understanding gesture. Artists have long explored multimodal methods and non-verbal communication. They can offer alternatives ways to segment and interpret motion and gesture. See for example the the Laboratory of Non-verbal semiotics in Japan explanation of the value offered by cross-disciplinary research (Hirata, M., Y. Adachi, S. Kubo, and H. Abe. 1999) Here are some examples of research areas in gesture and motion based interfaces in which the arts could contribute. See the Wilson Artlinks Website (Wilson, 2000b) for references to specific artist research projects.

The quality of motion and touch: Theatre and Dance practitioners are experts of motion and touch. Their involvement in IT research could refine the conceptualization of qualities to be sensed, represented and responded to that might be missed by working within engineering frameworks. For example, movement from one spot to another is much more complex than just the spatial displacement. There is a classic theatre training exercise in which performers are asked to walk across the stage simulating a variety of emotions or in a variety of social roles. In part a performer's success depends on understanding these subtleties and finding ways to behaviorally represent them. This expertise in analyzing motion and making explicit what is usually implicit might be a major resource to those working on computational models and systems to deal with motion.

Proximics - spatial distribution of persons: Movement cannot be separated from the social loading of locations in space. The branch of anthropology called proximics studies the ways people perceive social space. For example, cultures differ on what is an acceptable body bubble with great variation in acceptable distances in conversational spatial distances and the ways people distribute themselves in public spaces. Similarly, proximics are affected by the context of the interactions and the nature of the relationship between people. The movement artsŪ expertise in understanding and orchestrating social architecture is an important resource in designing computational systems to read motions in public spaces.

Interaction of architecture and motion: Architecture is another discipline with a long tradition of interpreting and controlling motion. Architectural innovation often focuses on creation of new kinds of motion orchestration. These perspectives would contribute to gesture research by adding new emphases about the relationship of space to motion, the meaning that different spaces take on, and expansive thinking about new kinds of architectural arrangements to maximize the potential power of gesture detection.

Research Agendas beyond Utilitarian Applications: The advent of devices that attempt to interpret motion and gesture opens up many new questions about how humans move and interpret other's motions. For example, such systems might allow people to learn more about how they themselves move and how meaning is associated with movement. For example, the psychologist Howard Gardner's research demonstrates that humans possess a kind of kinematic intelligence. (Gardner, 1993) Just as the attempt to build computational artificial intelligence opened up new vistas for researchers attempting to understand cognitive functioning in general so might the attempt to build gesture interpretation systems enhance our understanding of non-computationally related gesture and kinematic intelligence. Similarly, gesture systems research might facilitate people's ability to experiment with new kinds of motion and gesture. The Arts provide a context for this open-ended inquiry.