The Aesthetics and Practice of Designing Interactive Computer Events

©Stephen Wilson, Professor, Conceptual Design and Information Art
San Francisco State University

Another form of this paper appeared in SIGGRAPH 93 Visual Proceedings Art Show Catalog ©ACM, 1993

Abstract: Much confusion and hyperbole surrounds discussions of the aesthetics of interactive computer events. This essay works to clarify some of this confusion by analyzing the differences between interactive and non-interactive events, reviewing the variety of forms included under the umbrella term of interactivity, and investigating the theoretical rationales offered to support claims of interactivity's superiority derived from psychological, political, art historical, and technohistory sources. Building on this analysis, the essay suggests extensions to current GUI design canons that uniquely attend to interactivity as an aesthetic issue. It also investigates the challenges of the interactivity possibilities of emerging technologies.

Table of Contents

Introduction : Design Challenges of Interactive Multimedia

Interactive multimedia offers great promise. At this young state it also generates lack of clarity. This article works to reduce confusion by examining the assumptions and theory that underlie the claims about its power, clarifying the variety of forms hidden under the umbrella term "interactivity", analyzing design issues unique to constructing interactive events, and suggesting some experimental forms to be explored.

Interactive design is not easy. Indeed, a developer faces all the traditional aesthetic and functional design challenges faced by those who work in single text, image, or sound media plus complex new ones.

These challenges include the following:

This article focuses specifically on the last set of issues of designing interactivity.

What is Interactive Multimedia? How is it Different from Non-Interactive Media? Creating versus Consuming.

Multimedia usually means events that include some combination of sound (music and/or voice), still image (scanned and/or synthesized computer graphics), motion image (computer animation, cinema, and/or video) and text. (Curiously, multimedia historically meant slide shows with sound.) Interactive means that the user/browser/audience has the ability to act to influence the flow of events or to modify their form. The term "hyper" in hypermedia usually is used to mean a particular kind of non-linear, flexible interactive structure with built-in linking capabilities in which viewers can each choose their own path through some material. This article will demonstrate that it is useful to begin to differentiate the very wide range of activities can fit under this umbrella definition of interactivity.

As an aid to thinking about interactivity it is useful to consider what is non-interactive media. Common examples might include a photograph, painting, movie, book, or symphony. Critics of interactive media love to point out, however, that these media may not be as non-interactive as they appear. Although their form is fixed, the act of engaging them can be highly interactive. Viewers take many subtle actions (some invisible) that interactively adjust the experience. For example, the reader of a novel or the viewer of a movie is constantly adjusting attention, internal references, identifications, emotional responses, and willingness to engage internal associations that come from personal experience, social/ ethnic/ gender positions, previous experience with the art form, etc. Some analysts would go so far as to claim there is no successful art or media without this level of engagement interactivity. Also, the sequence is not as fixed as it seems in these historical media: a book can be the ultimate inexpensive, interactive random access media with instant easy connection with any page; VCR's allow non linear access to movies with the use of fast forward or reverse capabilities; theater and music events often differ as a result of performer interactions with audience response.

Still interactive media do have some crucial differences:

Nonetheless, developers of interactive multimedia need to keep in mind the criticality of deep psychological interactivity of successful art and media. The structural incorporation of concrete choice making does not guarantee deep engagement. Indeed, some analysts suggest that the choice making itself can distract from this deep engagement by disrupting the possibilities of these internal processes of feeling and musing.

Another way to consider the definition of interactivity is to focus on the distinction between creating and consuming. There have always been people who were having interactive media experiences: the creators of art and media events (that is, the authors, poets, directors, editors, writers, composers, choreographers, etc.) have always had the challenge, opportunity, and responsibility to shape their creations. They had to decide on sequence, emphasis, and the like. The consumers (readers, listeners, movie goers, etc.) did not have this range of action. Some analysts suggest that it is useful to conceptualize the experience of interactive multimedia as the attempt to break down this distinction by providing authoring opportunities to the consumer.

Why Make a Work Interactive? Theoretical Bases of Interactivity

Why bother with interactivity? Why would one want to give up the joy being guided by a gifted storyteller? The craft traditions of historical art and media forms have cultivated centuries of design expertise focused explicitly on structuring elements such as sequence, pacing and emphasis for aesthetic reasons. Similarly, in non art situations, teachers have developed centuries of expertise in staging presentations of information so that readers learn what they need in memorable and effective ways. What is gained in an event being interactive versus non-interactive?

Developments in diverse fields such as art, psychology, cultural studies, and information technology have led to this junction in history that emphasizes interactive media. It is crucial that those working in interactive design understand these theoretical rationales for interactivity so that they can differentiate among assumptions, values, and research findings. Much of the language in the field seems to assume the superiority of interactivity. For example, this point of view is illustrated by the following statements by Trip Hawkins, head of Electronics Arts and 3D0, in a Focus magazine interview in an article called "Welcome to Your Future".

Hawkins believes in interactivity, the holy grail of multimedia, insisting that the merger of tactile expression and consciousness is not only superior to today's passive media but actually better for people. "People want and need to interact," he explains passionately. "Scientists have proven that interaction is the single best way for human beings to increase their intelligence," he adds. "The only comparable invention in terms of its effect on human intelligence was the printing press." (** Rosen, David. "Welcome to Your Future." San Francisco Focus. November, 1993)

Most analysts would agree that research findings are not so clear-cut. This survey briefly presents a consideration of theoretical rationales as a tool for clarifying design goals.

Psychology: During this century psychological research has focused on the question of learning and teaching, as researchers have tried to understand how humans learn, remember and use information. Although some of these traditions, such as behaviorism, have stressed traditional notions of teaching and learning such as drill and practice and prestructured presentation, other traditions suggested the value of "learner-centered" or "inquiry" approaches. Several different theoretical traditions thus offer rationales for the importance of interactive media.

Associationism is an approach that views the mind as a giant computer or switchboard. An article called "As We May Think" (Bush, Vannemar. "As We May Think." The Atlantic Monthy. July, 1945. pp101-108) Vannemar Bush is often identified as the classic text in the development of interactive multimedia. Bush noted that thinking is often associationistic rather than linear. That is, when people work on problems or encounter information, the experience stimulates them to think of associated ideas, which give rise to further associations and so on. The flexibility of the associations is key. He imagined - before there were any computers to use as models - a machine called a memex, which would allow researchers to flexibly and interactively follow mental associations that came up in the process of inquiry. Anticipating later developments, he envisioned this machine as incorporating sound and visual materials (microfilm) in addition to text. He saw interactivity as great support to the natural association-making tendencies of the mind, and thus a promoter of more effective thought, conceptualization, and learning.

Another tradition called cognitive psychology emphasized the constructive aspects of learning and cognition. Theorists such as Jean Piaget ( The Origins of Intelligence in Children. New York: International University Press, 1982) and Jerome Bruner (**Toward a Theory of Instruction. New York: W.W. Norton and Company, 1966 ) attacked the primitiveness of traditional notions of the mind as an empty container to be filled with information. Observational studies of children and other learners demonstrated constructive processes of assimilation and accommodation in which the mind used provisional conceptualizations to actively experiment and make sense of new experience. The new experience itself was iteratively used to refine the conceptualization in an on-going process. Interactive learning structures that encouraged these processes of exploration, experimentation, and self adjustment were likely to produce the most potent learning.

Another tradition called psychology of self came at learning from another angle. Theorists such as Carl Rogers (** Freedom to Learn. C. Merrill Publishers: Columbus,Ohio, 1983) noted that the only learning that really made a difference was "self- appropriated learning" That is, the learner had to link with internal needs and emotional states in order for the information to be incorporated in a living way. Any structure that allowed the learner to act on self identified urges was likely to be most successful.

In other areas of psychological research, studies of phenomena such as brainwashing, propaganda, advertising and conversion all point to the potentiating influence of action on the part of the target individuals. That is, the more "learner" actions that can be incorporated into the experience, the more likely the learning will "take". Interactive structures that call for more than quiet passivity are thus likely to be more effective.

The theoretical implications of these traditions have been extended beyond fields of education and training. Interactive multimedia are seen as allowing users to follow their own associationist paths; to experiment and build on their own cognitive structures; and to link their actions with internal emotional and identity needs. In learning and information retrieval applications, the theories suggest the material will be remembered, used, and integrated better. In entertainment or art situations, the theories are extrapolated to predict that interactive events can be more profound and moving than non-interactive experiences.

Anthropology and Political Science: Anthropologists and political scientists have studied the social and political participation ethos and actions of many cultures (**Stone, William & Schaffer, R. Psychology of Politics. Springer-Verlag: New York, 1988) . Although there are many anomalies in their findings (for example, the low rates of actual political participation in the United States), there do seem to be differences in the norms for participation and relationships to authority among different cultures. There is a continuum in which members of the Western European and American democracies have much higher expectations of participation in comparison to other cultures.

Some analysts have suggested that cultural forms that move against this ethos (for example, passive media) cause tension. Interactive multimedia attempts to embed a more appropriate level of participation in the media. If this analysis is accurate, however, there should be very different levels of acceptance of interactive media in different world cultures.

Art and Media Aesthetics: The current interest in interactive art and media is not without precursors. Throughout the twentieth century there has been a questioning of the traditional forms of artist/audience boundaries. In the 1920's, for example, the Dadaists established cabarets and street theater in which audience members were encouraged to participate as creators. The communist upheavals in Russia resulted in the agitprop movement in which workers were expected to become active as artists. Berthold Brecht street theater in the 30's linked politics, art and participation. In the 1960's and 70's the interactive art movement flourished all over the globe in art forms including visual art, theater, dance, music, poetry, and architecture. For example, happenings created free form installation/theater events in which the audience was often absorbed into participation into ongoing events. The Living Theater and other similar groups came down into the audience with the intent to incorporating them in the performances. French art historian Frank Popper's book Art Action and Participation (** New York U. Press: New York, 1975) documents the worldwide scope of this movement. In recent years interactive art has not been a major movement although the advent of contemporary interactive technology is resurrecting interest in these traditions.

Interactive art often arose out of a cultural critique. In part this critique attacked the separation of art from life and sought to integrate them better by bringing art into everyday settings and by involving non professionals. Another part of the critique focused on a distrust of authority and established institutions: art itself was seen as one of these ossified institutions in need of radical challenge. For some with particular political agendas, extension of the right to function as artist to the masses was part of a more general radical agenda to spread societal participation. Another part of the critique celebrated the individual. Every person was seen as having artistic potential; it was thought that life would be richer both for the individual and for the community if non artists incorporated art consciousness into everyday life. Finally there was the part of the critique that questioned order in general. Interactive art increased the repertoire of actions and thus increased the chance for fruitful randomness.

Many contemporary high tech artists are more focused on the design of systems for creation rather than one particular outcome. (** Wilson, S. Using Computers to Create Art. Prentice Hall: Englewood Cliffs, NJ, 1986) Breaking with old traditions of art, they are more interested in the family of possibilities they create than in one particular sensual manifestation. The experience of the interactive artists is useful to those outside of art because of their analysis of the relationship of culture and media, their sensitivity to the relationship between media and audience, and their attention to the aesthetics of interactivity .

Technology and Information Science: Some analysts suggest that the possibilities of interactive media are a result of technological developments. The media now being developed would be inconceivable without earlier developments in digital output, input, processing, storage, and communication capabilities. The technologies themselves seemed to have opened up further possibilities for interactivity.

Interactive multimedia demands that the systems be capable of portraying rich visual and sound information. Although interactive text systems are possible, they may not give as deep a feeling of interactivity as a full multimedia system. The last two decades have shown an accelerating enhancement of the computer's ability to display photo realistic visual information and cd quality sound. Currently capabilities are being extended to full motion video.

Input technology similarly flourished. Ivan Sutherland (**"The Ultimate Display." IFIP( 1965), 2, pp 506-508) demonstrated an early system in which accepted input via a light pen on a CRT. Engelbart ( ** "Conceptual Framework for the Augmentation of Man's Intellect" reprinted in Grief, Irene. ed. Computer-Supported Cooperative Work: A Book of Readings. San Mateo: Morgan Kaufman Publishers, 1988) conceptualized technology systems that would flexibly allow creative work. Later at Xerox PARC Alan Kay (** " Computer Software." Scientific American (September, 1984): 52) and others developed a new paradigm for computer human interaction that used a mouse device for pointing and selection and a visual metaphoric display on the screen to indicate computer resources and user actions. Later developed by Apple Computer, this system created an important ease of interaction that invited further extensions to interactivity.

Processing, storage and communications developments similarly accelerated. CPU's grew in their capacity to respond in real time to user actions and to manage resources that users might want access to. Storage capabilities increased to the extent that they could provide interactive access to significant visual and sound information. Developments in communication technology allowed access to remote resources beyond what a local system could provide. Visionaries such as Ted Nelson in his XANADU project (**Computer Lib/Dream Machines. Seattle, Wash.: Microsopft Press, 1987) described highly flexible hypertext and hypermedia systems that would allow users to create links among vast stores of information.

All these developments continue today and promise to increase the technological possibilities for interactivity and the associated perception of responsiveness.

Nature of Interactivity

Though interactive events differ in their goals, application areas, and the content they refer to, it makes sense to analyze the underlying structures of what interactive events across fields ask from participants. They differ in the kind of interactive action required - from selection of options to search for links to input of new information. They vary in the control of interaction timing and in the amount of access they give to the choice set available - ranging from total program control as in a branching presentation to total user control as in an indexed system in which a user can select any option at any time. They differ in the amount of psychic and time investment that they require ranging from causal choice to deep cogitation. The tables below illustrate some of these continua.

Kind of Interactive Action Required

Control of Interaction Timing and Amount of Access to Choices

As later sections will show, these are additional elements of design that are important. Perceived range of action may be more important that real action. For example, a hypermedia system without a wide range and depth of choice may be perceived as less interactive than a branching system with many choices. Also, the ease with which one can navigate the system or the clarity about choices may influence the perceptions of choice. It is also important to note that contributory and authoring forms of interactive events are relatively rare although some analysts see their popularity rising.

Patterns of Interaction

The illustration below clarifies some of the options in designing interactive systems. For comparison it includes some non- interactive systems. As a stimulus to further design experimentation, it includes designs that mix elements.

Full size image

Expectations of the User

Systems differ in what they expect of the user - both in what the user brings to the experience and in what the outcomes are expected to be. For example, a simple public information travel information kiosk might have no expectations of the user except general literacy. The user will not be expected to have much knowledge of either the subject matter or interactive computer systems. For a specialized information retrieval system organized for astronomers, however, every user might be expected to have relevant background knowledge of the content area and the interactive system itself. Similarly, in the art and media world an interactive art event designed for users who are assumed to have some background in relevant art or cultural issues would be quite different than one designed for novices.

Also, interactive events also differ in their goals. A simple information kiosk may intend to impart some non crucial information or an entertainment program may intend to amuse. An interactive art program may hope, on the other hand, to provoke or move in some deep way.

The Aesthetics of Interaction: Extending the Graphic User Interface to Interactive Multimedia

The graphic user interface (GUI) revolutionized the use of computers during the last decade. Prior to the pioneering work done at Xerox PARC and extended by Apple Computer, most interfaces required humans to remember and type text commands. The graphical user interface sought to make interactions more intuitive by introducing innovations, such as visual metaphors for computer capabilities and user actions, and use of the mouse as pointer and selector. As explained in the section on theoretical bases of interactivity, the graphical user interface is part of the technological developments that contributed to the current interest in interactivity.

Now after many years of experience with interactive programs, there is a growing body of design expertise. Guidelines are available. Usually these guidelines stress clarity for the user and some notions of visual appeal. It is important to note, however, that the guidelines emphasize functionality and arise out of a productivity context such as using programs to get work done.

In fact, these guidelines have almost become a standardized language. There is a danger that this canonization may stifle innovation. As interactive multimedia enters fields such as art, education, edutainment, infotainment, infotisement, and entertainment, artists and designers must seek to develop a unique aesthetics of interactivity in which elements such as screen design, user control processes, navigation actions, system responses and the like become themselves part of the magic of the new media. Although standards of clarity and functionality should not be violated wantonly, there may be worthwhile artistic or thematic reasons to do so.

The sections that follow present two well respected sets of guidelines for interface design as a base line of design wisdom. It them poses some interface variations that may make sense in interactive multimedia. One set of guidelines comes from Greg Kearsley, a well known educational software designer. He offers the following guidelines in his book Authoring: A Guide to the Design of Instructional Software (**Addison Welsely: Reading, Mass. 1986)

Kearsley Guidelines for Interface Design
Screen Design User Control Response Analysis & Helps
Do not crowd screens Always let the user set the pace State directions and questions so that errors are unlikely
Avoid use of scrolling and overlays Allow users to control sequencing Use pointing rather than typed input whenever possible
Use attentional devices sparingly Use menus as much as possible Always acknowledge user input
Use windows/ viewports to organize information Let the user customize the program Answer analysis should be tolerant of variations in response
Use different type sizes and styles for emphasis and variety Always provide defaults Allow users to change their answers
Use graphics wherever possible Provide multiple (redundant) control options Always provide corrective feedback for errors or wrong answers
Use titles and headings on all screensDifferent types of helps might be necessary for different users Feedback should be brief and neutral in tone
Screen resolution determines display quality Helps should always be accurate<, specific, and available and easy to access
* Table is condensed, excerpted, and sometimes paraphrased from Greg Kearsley Authoring: A Guide to the Design of Instructional Software. .

Apple Computer did much of the pioneering work on popularizing the graphic user interface that is the underlying basis of current interactive computer systems. Their human interface group is one of the world's preeminent research organizations on interface design issues. The table below is a summary of interface principles from their Human Interface Guidelines: The Apple Desktop Interface . (** Addison Wesley: Reading, Mass. 1993) )

User Interface Design Philosophy (excerpted from Apple Interface Guidelines)
Use concrete metaphors and make them plain. Use audio and visual effects that support the metaphor.Direct manipulation: users should feel they are in charge of the computer's activitiesSee and point: Users should select actions from alternatives presented on the screen rather than relying on memorization.
WYSIWYG (what you see is what you get) There should be no abstract commands that only promise future resultsUser control: the user, not the computer, initiates, and controls all actions.Feedback and dialog: Keep the user informed and provide immediate feedback.
Perceived stability: The computer environment remains understandable and familiar rather than changing randomly.Aesthetic integrity: visually confusing or unattractive displays detract for effectiveness. Different "things" look different on the screen. Users should be able to control superficial appearance of their computer workplaces.Use sound sparingly, make its use redundant with visual cues, make it natural and unobtrusive; use highly different sounds to indicate different states
Modelessness: A given action on the user's part should always have the same result, irrespective of past activities.Forgiveness: User's actions are generally reversible - let them know about any that aren'tConsistency: Applications are consistent within themselves and with one another.
Event loop: The user should be able to do anything at any time.Reversible actions: Always provide a way out.Good design must communicate, not just dazzle. It must inform, not just impress.
* Table is condensed, excerpted, and sometimes paraphrased from Apple Computer. Human Interface Guidelines: The Apple Desktop Interface .

Interactive multimedia designers face design tasks analogous to those faced by artists, movie directors, composers, and the like throughout history. When does it make sense to follow convention and when does it make sense to innovate? Just as the conventions of language make possible the magic of poetry's stepping outside those boundaries, so do these interactive multimedia conventions begin to suggest areas of further investigation. Moreover, interactive multimedia is such a young medium that experimentation in defining its canon is essential. However, careless violation of standards can result not in magic, but in confusing, hard -to-use works. Below is suggested several areas for experimentation. What is essential is that the interactive process selected should fit the aesthetic purposes of the work. For example, note that many of these variations might be inappropriate where productivity and functionality are the prime considerations.

Predictability and clarity cannot be the only criteria. Interactive multimedia invites use of other criteria in the crafting of its visual, sonic, and procedural qualities; indeed, its interactive and time qualities opens up new artistic possibilities. For example, intrigue, appeal, surprise, engagement, frustration, variation, and the like all are possibilities.

Future Developments

This era's first generation of interactive multimedia may be only faint indicators of future possibilities. Researchers are active in a variety of fields related to its future development. The annual meetings of organizations such SIGGRAPH (ACM special interest group on graphics ), SIGCHI (ACM computer human interface interest group), and Hypertext association are full of ideas that will eventually expand the possibilities of interactive media as they become available. This section briefly outlines some of these areas including alternative inputs, alternative outputs, telecommunications, hypermedia, and artificial intelligence.

Each of these innovations offers new extensions to the boundaries of interactive multimedia. Each enhancement is a doorway to new possibilities and challenges. All those working in design of interactive multimedia need to realize that past models such as video and computer games are not the only relevant models.

Summary: Is Interactive Multimedia Really the Doorway to the Millennium?

Those currently developing interactive multimedia and those thinking of entering the field need to carefully assess the current hyperbole surrounding it. On the other side of the excitement and high expectations could easily be disappointment and premature abandonment of culturally important lines of inquiry.

The mere inclusion of user choice in media does not automatically make engaging events: interactive entertainment programs are not necessarily more entertaining. In the same vein, educational programs do not necessarily teach more effectively and deeply and information retrieval and research assistance programs do not necessarily lead to more mastery of the material or generation of better ideas. Equally art programs do not inevitably result in more enlightening or provocative experiences. The creators' challenges are the same as they has always been with the additional challenge of interactivity. The same careful design and artistic inspiration will be necessary to make the processes of interactivity themselves key artistic or conceptual elements.

Similarly, the inclusion of choice structures does not automatically indicate a new respect for the user's autonomy, intelligence, or call out significant psychic participation. In fact, some analysts suggest that much interactive media is really a cynical manipulation of the user, who is seduced by a semblance of choice. The choices offered, however, are not significant choices - for example, the ability to choose one of three products available in an interactive shopping experience or the ability to decide when and how to kill the simulated enemy in a game. The missing choices might be more important than the "choices" offered.

A critical theory analysis of the consumer culture of television channel choice offers an important analogy. Some analysts suggest that even though there might be 40 channels to chose from or six different newscasts, there may be important realms of choice missing if the consumer has not developed sufficient awareness and critical consciousness to ask what information, points of view, forms of presentation, and options are missing. Contrary to the hopes of those who believe interactive media are the beginning of a revolutionary age, these new forms with their trappings of participation could lull users to complacency through a charade of choice.

From this author's experience, interactive media genuinely do seem to open some new cultural possibilities. They will not automatically reach their full realization as a new media and as a transformer of the culture's attitudes about the individual's abilities, possibilities and responsibilities for the generation and use of information. Those who design the events and the industries will need clarity about goals, critical consciousness of what a choice process might be, and commitment to forge new cultural forms. They will also need creativity and persistence to design new forms of events in which interactivity becomes a central aesthetic and conceptual focus.

Information About The Interactive Version of the Document

Author Contact Information:

Stephen Wilson, Head of Conceptual Design/ Information Arts Area

Art Dept, SFSU, SF, CA 94132 (415) 338-2291