Research and Development as a Source of Inspiration and Ideas for Artists

Stephen Wilson, 1989

(A revised version of this paper was published in the Journal Leonardo, vol 24:no 3, 1991)


Technological research and development is radically altering everything from everyday life to world views. In order to continue to function as interpreters of culture, artists will need to address these research developments as sources of ideas and inspiration. To stimulate artistic reflection, the article reviews new input and output possiblilities created by developments in display technologies and image sources, digital control of object making, and sensor/actuators related to touch, motion, biological functions, and speech.


I Introduction: The Need for Artistic Reflection on Technological Developments

In the contemporary era the focus of innovation and creative fire has moved to the research laboratories. Technological developments profoundly influence everything from world views to everyday life. Computer related research seems to be accelerating the pace even more. Gregory Kepes, a Bauhaus artist and one of the first critics to focus on art and technology issues, many years ago noted the radical transformation of the world being wrought by science and technology in his book New Landscape in Art and Science. (1)

The rapid expansion of knowledge and technical development have swept us into a world beyond our grasp; and the face of nature is alien once again. Like the forest and the mountains of medieval times, our new environment harbors strange menacing beasts; invisible viruses, atoms, mesons, protons, consmic rays, supersonic waves

Kepes noted that the culture desperately needed artists to "help convert this new environment into a human landscape." Like others, he feared the imbalance that came from the lack of significant artistic involvement in the technological foment. He pleaded against artist neglect and ignorance of the scientific world:

The images and symbols which can truly domesticate the newly revealed aspects of nature will be developed only if we use all our faculties to the full -- assimilating with the scientist's brain, the poet's heart and the painter's eyes. It is an integrated vision that we need...

Much artistic response to technology -- even among those who call themselves technological or computer artists -- seems relatively superficial when contrasted with the momentous changes being wrought by scientific and technological research. Artists borrow the tools and imagery or dabble with the ideas but fail to get involved with the vigor, intensity and depth of the researchers who are generating the products and ideas. The culture desperately needs artists to knowledgably analyze the implications of these developments, to celebrate the breakthroughs, caution about the dangers and to help shape the research agendas.

Traditionally validated art media may no longer be fit arenas for the exploration that is needed. Competent and powerful interpretation will require changes at several critical stages of the artistic enterprize -- in the forms and contexts chosen for expression and in the sources cultivated for information and inspiration. Art historical categories may need updating to include issues originating in the research world

Forms and Contexts:

Just using current technological tools to create new kinds of images, sounds, and sculptures does not successfully penetrate to the core of research developments. Artists must find ways to powerfully reflect on the new meanings and ways of life being created. This kind of comment is likely to require artists to use the new technologies self-reflexively to comment on their implications. It will require artists to move into the new social contexts where the technologies are unfolding rather than staying close to the traditional art contexts such as theatres and galleries, and to invent contexts and forms when necessary. For example, artists commenting on computer based telecommunications possibilities cannot just paint pictures about it. Instead, as many are beginning to do, they need to actually master the technology and to find ways to create art events that probe the technology while existing in the new contexts created by it. Only then can their works assume the immediacy necessary to expose and illuminate the new meanings and opportunities.

History may show that the currently most known forms of computer art, such as computer graphics, are actually conservative, rather than radical forms. They are attempts to assimilate the new technologies to established art traditions. While they serve important functions of popularizing and investigating some new sensual and aesthetic possiblities of computers, there is a much wider world of possibilities that begs for attention.

Sources Cultivated for Information and Inspiration:

In order to remain at the forefront of creative activity, artists will also need to develop new ways for patrolling the information frontiers. To gain access to the ideas and materials that empower the technological revolution, they must identify and join the networks by which scientific and technological innovation is stimulated and disseminated--for example, attendance at the trade shows and subscriptions to the trade journals which spread ideas to the researchers and designers of new products years before the public hears about them.

They can cultivate curiosity and skills of understanding and working with scienfific and technological ideas. Interpreting the meanings beneath the surface and seeing the implications of ideas beyond the technical specialties that created them become part of art making. Mastering the challenge of walking simultaneously in the art and science worlds becomes part of the definition of being an artist.

Illustrating the Value of Sensitivity to Technological Context:

This paper aims to illustrate the potential value of artistic sensitivity to technological context. It does so by surveying some of the emerging trends enabled by microcomputer and digital technology that seem to call for artisitic attention and response. It emphasizes the importance of this context as a source of ideas and inspiration. Because of limited space it does not consider the other part of the process concerning artist invention of new contexts and forms of expression except to note some suggestive examples.

As an artist working with the ideas, tools, and materials of the microcomputer age, I have, as part of my artistic life, systematically tried to monitor the emerging microcomputer technology through journals, trade shows, and professional contacts with researchers, developers, and manufacturers. I feel a sense of wonder at the brilliance of some of the work I encounter. The resultant files of ideas and my musings on them are the equivalents of a painter's sketch book. As described in my previous writings (2,3), some of these ideas have already inspired artworks. Other ideas intrigue many of us working with experimental technology but are still waiting for appropriate artiistic response

The goal of this paper is to describe some of these developments and their implications in order to provoke further artistic work and analysis. The information provided is varied -- stretching from descriptions of already available commercial products, to new technologies being investigated in research laboratories, to social and philosophical commentary on the developments. Similarly, the ideas range from application areas that are clearly related to established artistic endeavors (such as new image making possibilities) to areas that require the invention of innovative artistic forms and contexts (such as new biological or military applications of digital technology.)

The material presented is a summary from one section of my current book project Technological Challenges to the Arts. The described developments are only a small part of a much larger body of research topics relevant to the arts. Significant areas not extensively described here include military, security, and psychological applications of digital technology, telecommunications, hypermedia, simulations, artificial intelligence, and the social ramifications of developments. The sections that follow focus on new input and output possibilities created by developments in display technologies and image sources, digital control of object making, and sensors/actuators relating to touch, motion, biological functions, and speech. The text starts with those areas related to traditional artistic concerns of image and object making and then moves to areas with fewer precedents.


New Image Making Possibilities:

Image making has historically been a focus for artists. Computer graphics offered a unique set of aesthetic potentials, and thus it became one of the major involvements of artists with computers. Each new development in resolution or animation speed was exploited by artists who attempted to create new kinds of image aesthetics-- for example, exploring the pixel basis of images, the transformation possiblities of image processing, and the ability to generate synthetic images that approach photographic realism. One can expect this use of the evolving technology to continue. However, other research is creating very different possibilities for image generation. Areas include 3D images, information based graphics, and new computer programming algorithms.

3D images: Western artists have long been on a quest to simulate three dimensions in their creations -- for example, with the conventions of perspective. Using digital technology, researchers are beginning to have some success in creating images read by the eyes as optically true 3D. For example, Tektronics and other companies provide stereoscoptic systems in which goggles synchronized with computer presented images are timed to present each eye with position shifted images so that the brain reads them as three dimensional in a fashion analagous to the old stereoscopic viewers. Texas Instruments has developed a system called real time automatic multiplanar stereoptics in which image projectors are synchronized with rotating mirrors such that viewers see a virtual object that they can actually walk around. (Figure 1) Researchers for the military have created Heads Up Displays (HUD) in which fighter pilots see transparent electronic projections of meters and radar detected objects on their cockpit windows. These displays are optically adjusted to appear at various depths. Automobile companies are applying this technology to windshields -- for example a driver will see a transparent warning display about speed that seems to appear in the landscape 20 feet in front of the car. Researchers at Ames NASA labs are working on "video helmets" that project a 3D wire frame visual world that is sensitive to head movement -- for example, viewers turning heads to the right see images that they would see if they were really in a 3D environment. The availability of these optically true 3D image making possibilities may well force artists to identify specific kinds of subject matter that can aesthetically benefit from the new sense of depth that can be created.

Information based graphics: Progress in digital imaging is also creating new families of images that have few precedents. Satellite images provide images of the earth and other heavenly bodies that display visual information and information gleaned from other parts of the electromagnetic spectrum such as infrared, which are usually invisible. Electron microscopes provide views of tremendously complex worlds below the viewing threshold of even optical microscopes. Medical imaging technologies such as CAT, PET, Ultrasonic, and NMR scanning peer inside the body to represent static and dynamic images of body processes (Figure 2). Motion analysis systems analyze the actions of athletes and animals to reveal multiple cubist-like renditions of the subtleties of motion lost to the naked eye.

These images intrigue both because of their sensual interest and because of their linkage to information about the world. The views of the earth can easily be thought of as continuations of artistic concern with landscape and the medical and motion analysis images, as continuations of concern with figure. These images often do not represent normal visual realities but rather involve mappings of colors to other kinds of digital information. For example, different colors are chosen to represent different tissue densities read by a nuclear magnetic resonance scanner. Because these images represent important new ways our culture will see ourselves and our world, they call out for artistic treatment. Artists must involve themselves not only by superficially borrowing the new appearances but more profoundly in reflecting on the relationship between the information and its visual representation. For information on some aspects of artistic work with these imaging methods, see Cox's article on visualization with supercomputer images (5).

New Computer Programming Algorithms: Computers make possible the generation of images from mathematical abstractions. As with the information based graphics described above, the interest of these images lies as much in their relationship to their underlying mathematical structures as their sensually visible aspects. Developments in software algorithmic theory will continue to expand the repertoire of ideas requiring artistic exploration -- for example, fractals (Figure 3) and object oriented graphics. For more information see Herbert Franke (5) and Frank Dietrich (6) on the aesthetics of algorithmic graphics.

New Sculpture Possibilities-Linkage of Design to Fabrication and New Materials:

Research is also opening possibilities for those with traditional sculptural concerns of creating palpable objects. There are now new ways of linking design to the fabrication of objects so that the gap between idea and realization is minimized. Technology is moving toward the sculptural dream of having machines that will translate imagination into objects. Also there are new computer controllable materials which enable sculptors to venture in unprecedented directions in the design of kinetic sculpture.

Linkage of design to fabrication: Designers are able to trace over the contours of small 3D models with 3D digitizer styluses and have the computer record computerized mathematical models of the object. (Figure 4) Cyberware's 3D digitizer even eliminates the need to hand trace a stylus by using lasers. A computer reads distances by timing the reflections of scanning laser beams aimed at an object which is placed on a rotating platform. Once constructed by the computer, the mathematical models are available for the artist to experiment with changing shapes.

At the production end of the process, computerized models can be used to actually fabricate objects. 3D mathematical models can be used to control lathes and other machinery to actually produce the objects. For example, artist David Dameron pioneered the use of a computer controlled lathe to carve objects directly from screen designs. A company called 3D systems automates the process even more in a process called stereo lithography. A computer actually creates a plastic model by focusing laser beams to trace out the contours of a mathematical model in a chamber of special plastic foam. At those places where the beams focus, the photcurable polymers change into solid plastic. (Figure 5) General Motors has been spending billions on research on an advanced Computer Integrated Manufacturing (CIM) system. Ultimately this system aims to integrate all aspects of the design and maufacturing process. The work created by designers on CAD systems will be translated through a series of steps into instructions to automated assembly lines to actually produce cars representing the designs. The ancient artistic quest of translating ideas into physical realities may become slightly more easy.

New materials: Materials scientists are developing new materials that will give kinetic artists a unprecedented level of ease and control for realizing their dreams of capturing motion and light. Electroluminescent panels (EL) are flat phospor coated surfaces that glow with cool eery colors reminscent of fireflies when placed in low energy electrical fields. Liquid crystal displays (LCD's) are materials that change transparency and color when subjected to currents such as are now common on digital watches. Polaroid's flexible lcd extends the possibility of using this technology to cover non rigid surfaces. Fiber optics are specially fabricated glass or plastic fibers that allow efficient movement of light. These materials give an artist using computers an outlet to the sensual world beyond the customary video and sound. Artists will be able to precisely control the color and appearance of the surfaces of 3D objects

Biometal, piezo ceramics and Kevlar plastics provide radical new opportunities for controlling motion. Biometal is a special alloy that changes shape when a current is passed through it. Piezo ceramics and Kevlar plastic are unusual materials that translate electrical voltage into motion and vice versa. Greatly increasing the reliability and simplifying the computer control of motion, they radically alter traditional ways of thinking about controlled mechanical motion by creating devices with no traditional moving parts. They introduce the possibility of new kinds of elegant kinetic sculpture in which the material compostion of a work will also determine the ways it can move

Superconduction is the characteristic of a material to allow resistance--free flow of electricity when cooled to a certain point. New ceramic compositions have been discovered which allow superconduction at financially feasible temperatures instead of the near absolute zero temperatures originally thought necessary. Superconduction will make feasible many new possiblities such as cheap power, superfast computers, and levitation trains that hover above the ground on oppositional magnetic fields. Ultimately they will allow artists to reach an elusive goal of creating precisely controllable kinetic sculptures that float in space, seemingly defying gravity. (Figure 6)

New Input and Ouput Technologies - Touch/ Motion, Labeling/Scanning Systems, Biological Characteristics, Speech:

The dominance of keyboard and printer and video screen as methods for input and output to computer systems is to some degree an accident of history. These devices were efficient, capable of respectable flows of information, relatively inexpensive, and familiar. Economically and sociologically it made sense to adapt them for computer information exchange. Computers, of course, do not care what form the human communication takes as long as it is ultimately convertible to the electronic pulses computers do understand. Researchers are showing that there is almost no kind of action that is not convertible. Believing that the customary forms of human machine interaction are not universally the most effective, researchers have been developing a great variety of new possibilities. The popularity of the mouse pointer input device (originally developed in Xerox PARC research labs) demonstrates the appeal and power of approaches other than the keyboard. Researchers are investigating the use of touch/motion, labelling systems, speech, and other biological characteristics.

The use of these interactive technologies does not have as many artistic precedents as those described in previous sections. The use of touch, body motion, and biological functions in art are much less developed than the use of vision and hearing. Also, although there has been interest in interactive art in the last decades, this form, in which the audience acts as cocreators, is still in its formative stages. Some artists believe that the computer's flexible ability to process input will facilitate experimentation with interactive art and with other forms of human engagement beyond sight and sound.

Touch/Motion: Often interactive art's reliance on standard input devices such as switches, keyboards and mice seems awkward. The interface often interrupts rather than enhances the flow of the events. The technologies that make use of touch and motion create the possibility of a seamless, unself-conscious engagement. Inventors have developed relatively inexpensive touch sensitive surfaces using a variety of technologies including force sensing resistors (that read the pressure by the amount of area covered by two membranes being forced together), light (breaking the beams of invisible infrared light generated by an array of sensors), ultrasonics (locating human appendages by reading echo times for inaudible sounds sent out as in sonar), and body capacitance (which depends on the body's ability to store and transmit static electricity). These surfaces can be fabricated to be transparent so that they can be placed over monitor screens or anywhere else. Adding a pen or pencil to the equation, IBM has developed a flexible, paperlike surface that can read the movements of handwriting with the aid of special software.

Some input systems don't rely on explicit touching motions. U. of North Carolina researchers have a system in which a subject's walking on a treadmill controls videodisc displays so that the person seems to be walking through a building. A sports equipment company is marketing an exercycle in which bicycling prowess controls videodisc tours through various environments. Nintendo computer games now includes an optional Powerset system (plastic force sensing sheet placed on the floor) which reads the users' running and jumping motions to control computer graphic games such as a simulated Olympic race. (Figure 7)

Other systems don't require contact at all. Security electronics have developed a variety of motion detectors that can read the motion or proximity of humans or other objects using technologies such as ultrasonics, infrared, and light level detectors. More precision than simple motion detection is possible. For example, Polaroid uses ultrasonic range finders (reading time for bounced ultrasonic sounds to return to transmitter) to focus their cameras. More complex three axis arrangements of these sensors allow reading exact position in three dimensional space. MIT researchers are using these and other technologies to develop systems that read gesture For example one reads the position of a pointing hand to extrapolate what it is pointing at. The military has developed another system that reads the angle of reflection of an infrared beam from specially coated contact lenses to determine what someone is looking at. Ultimately, input to computers will be as effortless and looking and moving. Artist Myron Krueger (8) has begun to explore these ideas in the artificial reality installations he creates in which artificial computer graphic creatures respond to the actions of viewers.

The combination of body motion input systems with sophisticated feedback opens additional possibilties. Researchers at Oak Ridge Laboratories have developed a glove that allows workers to handle dangerous materials at a distance. A computer reads the hand motions of a worker and translates them into instructions to a distant robotic arm. Using force sensors, the computer similarly reads the resistance of the materials to force applied and translates that to solenoids in the gloves that give the worker the tactile feel of the actions. VPL connects the video helmet described earlier with a data glove that reads hand motion. The wearer is then able to manipulate objects in distant locations and receive realistic computer graphic feedback. This ability is seen as extremely important for use in places where actual presence is too costly or dangerous as in space. These technologies highlight the growing importance of the concept of telepresence. Through a combination of telecommunications and new forms of human machine communication, people will be able to perform actions in remote locations as if they were actually present. This virtual presence, in which we can experience events or take actions through machine proxies, is a change in human limitation requiring artistic comment. For example, artist Joel Slayton has already begun to create artificial computer graphic worlds explored by a glove sensor. Going even further, a developer is proposing a new industry of artificial travel, in which people will be able to experience distant places via robots with video and touch/feedback systems.

Scanning/Labeling Systems: New systems provide for the identification and movements of objects as well as humans. Technologies such as bar codes and store security systems have profoundly affected the patterns of everyday life. Bar codes are the standardized patterns of dark lines that now make up the unified product codes (UPC) that are on almost all grocery products in the United States. A detector and associated computer program can read at a distance the changing reflections of a laser beam scanned across these codes no matter what the orientation of the object (Figure 8). The technology has been extended to tasks such as keeping track of railroad freight cars and automobiles at highway toll booths (the Virginia highway department uses bar code permits on the windshield to automatically deduct bridge tolls for commuters passing through.). Some futurists believe that ultimately every manufactured object will carry these codes because of the ease they introduce into inventory, tracking, and merchandicing.

Other scanning/labeling systems extend the functionality of bar codes. Cauzin Softstrips are a system that uses two inch wide printed strips of digitial patterns of dots. These strips are read by passing them beneath a sensor wand. Although less tolerant of orientation and distance variations than bar codes, they allow much greater densities of information. Store security systems provide a different kind of scan based on specially constructed tags that produce characteristic radio or magnetic interference. Passing through detectors at the door sets off alarms if the labels have not been deactivated. More sophisticated radio frequency (RF) systems for factory inventory place labels which broadcast information about the object when activated instead of just indicating presence or absence. Hong Kong traffic officials use a related system to control car movement. Only individuals with appropriate radio activated tags are allowed to drive in the congested downtown area.

Presenting the ultimate in locatability, new satellite based systems allow the position of anyone with a transmitter to be located to within 20 feet anywhere on the earth. Automobile manufacturers are planning to offer systems which combine this facility with computerized map displays so that drivers can read their location at any time. In a more ominous development, Orange County, California law enforcement officials have introduced the electronically mediated, house arrrest ankle device that records the movements of "prisoners" and lets officials know when they illegally leave their houses. The ability to keep track of persons and things can be both a great boon and a disasterous invasion of privacy. Artists can help the society reflect on both these options.

Biological Characteristics: Researchers are now searching for ways to link the computer directly with biological characteristics of humans and other living organisms. They are trying to find ways for the computer to sense organic processes. Success in these investigations will open broad new fields of scientific and industrial activity.

As with some of the other research areas discussed above, there is little precedent in the art world for direct work with functioning biological systems. However, some earth artists and conceptualists have experimented with incorporating organic processes -- for example, Chris Burden designed performances in which his own bodily processes were highlighted, Newton Harrison presented work with living animal systems, Dennis Oppenheimer created growing sculptures and Mark Thompson created installations based on the functioning of bee colonies. The pace of this experimentation slowed in the last years although there are signs of a resurgence--for example, recent shows on animal art and the recent issue of LEONARDO focused on biological art. As ecological concerns increase our culture's awareness to the biological world, the artists will no doubt also respond. Progress in development of technological links between computers and biological processes will provide important access to artists who work with computers. Indeed, bridging the world of technology and biology may well be a uniquely appropriate contribution artists can make to the contemporary world because of their concerns that transcend narrow technical specialties.

Researchers have developed methods of tracking previously inaccessible aspects of plant and animal life. For example, Acoustic Emissions Company created moisture sensors to be put next to plants to make sounds related to water needs. Researchers at the University of Maine have developed "talking potatoes", sensors placed inside potatoes that transmit information about the vibration and pressure conditions the potatoes undergo during processing. A book called The Secret Life of Plants described still controversial research in which plants were electronically monitored for responses to events around them In one memorable experiment plants demonstrated a radical change when live brine shrimp were dumped into boiling water across the room. Exploring animal life, researchers have developed tiny radio transmitters to be put on bees to track the details of their patterns of comings and goings from their hives.

Some artists have begun to explore these capabilities to monitor processes of neighboring organisms. Minneapolis artist Lief Brush has developed extremely sensitive sensors which amplify natural sounds such as the flows of fluids in trees and the movements of insects. Joe Sanchez has connected plant sensors such that the changing resistance in plant leaves was translated into computer generated poetry.

Investigators are also working hard to perfect monitors of human biological processes. Medical researchers have developed whole families of remote sensors of human functions such as pulse and heart rate, electrocardigrams, and temperature so that doctors can keep track of patients at risk. Commercially, there are jogger's watches that give instant read outs of pulse rates. More esoterically there are new devices such as the sensing bra that provides information on a woman's fertility cycle, urine anlaysis devices that use stroboscopes to report on chemicals present, and perspiration processing developed by the Osaka Industrial Technology Institute for underwear manufacturers to assess sweat and body heat at critical locations. Examples of Artist response include designer Mary Ann Scherr's work on a line of biomedical jewelery which will change in response to body states and the state of the nearby environment and Richard Loweberg's thermographic images of heat variations in dancer bodies in motion.

Brain functioning especially intrigues researchers. There has been a long standing dream to learn more about the human mind by monitoring its functioning. The PET and NMR scanning devices previously discussed offer one way. Biofeedback offers easier access. Popularized in the 60's, biofeedback devices electronically read relatively accessible body characteristics such as galvanic skin response (resistance of the skin to flow of current related to perspiration), extremity temperature, and EEG brain waves (small voltage variations picked up by sensors on the scalp). Microcomputer technology has made these devices accessible by descreasing their cost and increasing their reliability.

There is still much controversy about biofeedback and brain wave research. Some studies suggested that characteristic brain wave patterns were related to particular mind states (for example, high alpha frequencies were associated with meditative states reached by yogis). Other research suggested that brain waves and other associated indicators were much too unreliable indicators and there relationship to mind states was questionable.

Commercially, however, these ideas are being developed. An established school of biofeedback psychotherapy now relies on these devices as critical elements and lie detectors are a standard fixture in industrial testing and crime enforcement. Newer applications continue to appear such as the Navy's Brain Aptitude Test, which analyzes patterns in the two hemispheres to indicate relative dominance of verbal and spacial intelligence, and Behavioral Engineering's Biofeedback Control System, which allows users, after a period of training, to control a robot's movement on two axes via computer monitoring of these body signals.

Undoubtedly, the research to more precisely understand and monitor body signals related to brain function will continue. For example, NYU and UCLA researchers are working on a new process called Magnetoencephalography, which reads very weak magnetic fields created by brain cell functioning. To accomplish this, they use a device called a Supercondcuting Quantum Interferece Device (SQUID), which achieves sensitivities 1000 times greater than EEG readers by filtering out irrelevant magnetic activity so that it can more distinctly isolate particular functions.

Some artists have already begun research based on these new capabilities to tap brain functioning. David Rosenboom (9) created a variety of related events including some in which musicians and dancers controlled electronic music generation by their brainwaves. The author collaborated with a biofeedback therapist to create an event called "Calmness" in which mind state, as read by extremity temperature, controlled the evolving computer graphics.

Frustrated by the imprecision of the brain wave approach, some scientists have been developing more direct approaches of working with nerve cells and reading body chemistry. For example, researchers at Stanford Medical Center has developed an IC chip to repair severed nerve cells by acting as a transceiver between ganglia. A growing field of biosensor research is working on developing sensors that can read the presence of specific enzymes, neurotransmitters, animo acids and proteins in humans and other life forms. The gap between the functioning of electronic and biological intelligence will be narrowed. Already chemists are working on ways to use viruses, light sensitive proteins, and the membranes between molecules as computer components. Scientists envision a day when picosensors implanted in the skull will allow direct communication between the brain and electronic devices. Art has always been one of man's ways to understand himself and the universe. These new approaches offer important avenues by which to continue this inquiry.

Speech: Many futurists believe that because speech serves such a central role in human to human communication, it will ultimately be the most important communication channel between humans and computerized devices. Science fiction visions of the future regularly feature computers that flawlessly understand the words spoken by humans and talk back. Typewriters that can translate speech into typing are avidly sought. Communication with computers and other devices ultimately becomes an extension of normal communication with other humans.

Speech is crucial for many art forms -- for example, drama, poetry, and music. Some anthropologists suspect speech in the form of storytelling may be one of the most primitive art forms. Technological speech, on the other hand, remains largely unexplored both because of technological barriers and its failure to fall into traditional art media categories. Speech is such a uniquely human characteristic, however, that the achievement of machine speech and speech understanding will innevitably draw artists to become involved as the technology progresses.

Currently, both speech synthesis and speech recognition pose difficult, yet unsolved problems. Electronic phoneme synthesizers provide flexible but difficult to understand, robotic speech lacking inflection and tone color. Digital recording provides high quality speech sounds without the flexible synthesis possibilites and enormous memory requirements. Speaker-dependent voice recognition systems can understand limited vocabularies of disconnected words after they are "trained" by the particular speaker who will be using them. Systems to understand free flowing speech from a general population, with whom there has been no training experience, are not yet realizable. AI systems to understand the meanings of words after the sounds have been decoded are still researcher's dreams. For specifics on speech technology, see my book Using Computers to Create Art (10)

In spite of these limitations, significant research and commercial progress has been made. Simple speech synthesis is now a feature available in everything from automobiles and appliances to vending machines and toys. For example, automobile voices warn passengers to attach seatbelts, Texas Instruments ever popular Speak and Spell toy speaks the words children spell out on its keyboard. Coleco's stuffed dog Wrinkles makes sounds related to children's actions -- for example, it giggles in response to tickles of its stomach and makes chewing sounds when fed a bone.

Speech recognition has similarly achieved some success. Speaker dependent limited vocabulary voice input systems are regular features of factory and military situations in which workers do not have hands free. Closer to home, World of Wonder's Julie doll can understand 12 spoken phrases and can respond with over 200 unique phrases in response. Voice mail answering machines with synthesized speech messages and digital recording are now in the reach of every home. Picture frames with built in digital speech recording are available such that a voice message of the person can be a regular part of the portrait viewing experience (Figure 9).

Many computer companies have active research departments investigating speech input and output. Futurists predict that many of the problems will be solved and that synthesized and recognized speech will be a prominant feature of the technological landscape. Artists have begun preliminary investigations of this new technology. For example, Paul DiMarinis has created installations which make use of synthesized voice Alan Rath has devised a mouthwriter in which a computer processed image of a mouth speaks letters as a viewer types them on a keyboard. The author produced Synthetic Speech Theatre in which artificial characters debated each other with synthesized voice and periodically responded to viewer comments via speech recognition. Machine speech, however, is a development with such profound implications for the ways we think about ourselves and relationships to machines that further artistic work is inevitable.


The methods for getting information into and out of digitally controlled devices is expanding greatly. These accomplishments are significant monuments to human curiosity and inventiveness. Computers can create images and objects that have a palpability far greater than that available on a video screen. They can keep track of objects and people anywhere on the earth. They can create speech sounds that mimic our intimate communication with each other. They can accept information from our words, brains, and bodies

To illustrate the confluence of these technologies, consider the experience of a contemporary fighter pilot flying his computer with wings. One of the worlds most sophisticated electronic displays generates computer graphic images of radar detected opponents in synthesized depth on transparent cockpit windows. The pilot peruses the incoming threats. The computer tracks the gaze of the pilot via contact lens reflections of an infared beam. When the pilot has identified the desired target, he merely looks at it. A spoken command is understood by the speech recognition system and a defensive missle is launched.

These developments will not restrict themselves to the military world but will diffuse into all aspects of every day life. More than just creating new products and gadgets, they have the potential for altering our basic attitudes toward ourselves, each other, and our environments.

The late night lights burn bright in the research labs. Scientists and developers are trying to unravel the mysteries of computer generation of true 3D images. They are finding ways for the computer to make sense of the images we get from the far cosmos, the atom, and our own bodies. They are developing exotic new materials that will make the computer control of movement as easy as the control of pixels. They are narrowing the gap between imagination and realization by creating new kinds of computer linkages between design and manufacture. They are inventing ways for computers to be controlled with our touch, motion, speech, brain waves, and even our body chemistry. Most importantly, driven by that essential human trait of curiosity, they are wondering about these possibilities made possible by computers. It is time for artists to start wondering also.

References and Notes

1. G. Kepes. New Landscapes in Art and Science. George Theobald. Chicago, 1956. Pages 19-20
2. S. Wilson. "Artists as Explorers on the Technological Frontier" Academic Computing Vol. 1: No. 2 (Autumn, 1987)
3. S. Wilson. "Editorial: Industrial Research Artist: A Proposal". Leonardo. Vol 17: No. 2 (Summer, 1984)
4. S. Wilson. "Environment-Sensing Artworks and Interactive Events: Exploring Implications of Microcomputer Developments". Leonardo. Vol 16: No. 4 (Autumn, 1983)
5. D. Cox. "Using the Super Computer to Visualize Higher Dimensions: An Artist's Contribution to scientific Visualization". Leonardo. Vol 21: No. 3 (Autumn, 1988)
6. F. Dietrich. "Visual Intelligence: The First Decade of Computer Art (1965-1975)". Leonardo. Vol 19: No. 4 (Winter, 1986)
7. H. Franke. Computer Graphics: Computer Art. Springer-Verlag. Berlin, 1985 (Revised Edition).
8. M. Krueger. Artificial Reality. Addison Wesley. Reading, Massachusetts, 1983.
9. D. Rosenboom (ed.) Biofeedback and the Arts: Results of Early Experiments. Aesthetic Research Centre of Canada. Vancouver, 1976. (Available from Frog Peak Music, Box 9911, Oakland, CA 94613)
10. S. Wilson. Using Computers to Create Art. Prentice Hall. Englewood Cliffs, New Jersey, 1986

List of Figures

(sorry not available in this web version)

1. Texas Instrument's Multiplanar 3-D Display creates an optically true three dimensional image that viewers can walk around. The image is created by the synchronization of a laser beam on a rotating, changing angle mirror

2. This Positive Emission Tomography (PET) brain scan was artifically colored by Gould computer equipment to show the physiology of the brain under the two conditions of eyes open and eyes closed

3. This Fractal Image of the Mandelbroit Set was generated at the IBM Research Center.

4 This 3Space Three-Dimensional Digitizer from McDonnel Douglas allows a user to input information into a computer by moving a stylus over a real object.

5 3-D Systems stereo lithography process creates a plastic model by using the computer graphic mathematical model of an object to control laser beams aimed at plastic polymers that solidfy wherever the beams focus.

6 This superconducting ceramic magnet available from Edmund Scientific allows experimentors to make objects float suspended by magnetic forces.

7 Nintendo's "Power Set" computer game allows users to control the game by running and jumping on top a plastic force sensing sheet.

8 Videx's Time Wand bar code scanning system allows the computer to keep track of documents and people.

9 Talking Pictures' photo frame allows people to record voice messages that are activated any time anyone lifts the frame.

þTechnological Research and Development as a Source of Ideas and Inspiration for Artists

- This page created by Stephen Wilson, Professor Conceptual Design, SFSU ( Art Dept, SFSU,1600 Holloway, SF, CA 94132
- More information about the Conceptual Design program is available at (

- The url of this page is (