The last decade has seen a large and growing body of research in computational systems embracing physical-world modalities of interaction. This work has led to the identification of several major research themes, including ubiquitous computing, augmented reality, mixed reality, and wearable computing.

At the same time, a number of research systems relating to the use of physical artifacts as representations and controls for digital information have not been well-characterized in terms of these earlier frameworks. Fitzmaurice, Ishii, and Buxton took a major step in this direction with their description of “graspable user interfaces.”^1,2

Building upon this foundation, we extended these ideas and introduced the term “tangible user interfaces” in our discussion of “Tangible Bits.”³ Among other historical inspirations, we suggested the abacus as a compelling prototypical example. In particular, a key point to note is that the abacus is not an input device. The abacus makes no distinction between “input” and “output.” Instead, the beads, rods, and frame of the abacus serve as manipulable physical representations of abstract numerical values and operations. Simultaneously, these component artifacts also serve as physical controls for directly manipulating their underlying associations.

This seamless integration of representation and control differs markedly from the mainstream graphical user interface (GUI) approaches of modern human-computer interaction (HCI). Graphical interfaces make a fundamental distinction between “input devices,” such as the keyboard and mouse, as controls, and graphical “output devices,” such as monitors and head-mounted displays, as portals for representations facilitating human interaction with computational systems. Tangible interfaces, in the tradition of the abacus, explore the conceptual space opened by the elimination of this distinction.

In this paper, we take steps toward a conceptual framework for tangible user interfaces. In the process, we hope to characterize not only systems explicitly conceived as “tangible interfaces,” but more broadly, numerous past and contemporary systems that may be productively considered in terms of tangible interface characteristics.

A first example

Figure 1

Under the mediating illumination of the I/O bulb, the building models of Urp cast graphical shadows onto the workbench surface, corresponding to solar shadows at a particular time of day. The position of the sun can be controlled by turning the physical hands of a clock tool. As the corresponding shadows are transformed, the building models can be moved and rotated to minimize intershadowing problems (shadows cast on adjacent buildings).

A physical “material wand” can be used to bind alternate material properties to individual buildings. For instance, when bound with a “glass” material property, buildings cast not only solar shadows, but also solar reflections. These reflections exhibit more complex (and less intuitive) behavior than shadows. Moreover, these reflections pose special problems for urban drivers (roadways are also physically instantiated and simulated by Urp.)

Finally, a computational fluid flow simulation is bound to a physical “wind” tool. By adding this object to the workbench, a wind-flow simulation is activated, with field lines graphically flowing around the buildings (which remain interactively manipulable). Changing the physical orientation of the wind tool correspondingly alters the orientation of the computationally simulated wind.

Tangible user interfaces

Clearly, traditional user interface elements such as keyboards, mice, and screens are also “physical” in form. Here, the role of physical representation provides an important distinction. For example, in the Urp tangible interface, physical models of buildings are used as physical representations of actual buildings.

The physical forms (representing specific buildings) of the Urp models, as well as their position and orientation on the workbench of the system, serve central roles in representing and controlling the state of the user interface. Even if the mediating computers, cameras, and projectors of Urp are turned off, many aspects of the state of the system are still concretely expressed by the configuration of its physical elements.

In contrast, the physical form of the mouse holds little “representational” significance. GUIs represent information almost entirely in visual form. Although the mouse mediates control over the graphical cursor of the GUI, its function can be equally served by a trackball, joystick, digitizer pen, or other types of input peripherals. This invariance differs sharply from the Urp example, where the interface is closely coupled to the identity and physical configuration of specific, physically representational artifacts.

Interaction model

In its original formulation, MVC served as a technical model for GUI software design, developed in conjunction with the Smalltalk-80** programming language.⁶ However, we believe the MVC model also provides a tool for studying the conceptual architecture of graphical interfaces and for relating this architecture to the tangible interface approach. Although alternate interaction models such as PAC⁷ may also hold relevance, we find MVC's exposure of the view/control distinction to be useful.

We illustrate the MVC model in Figure 2A. MVC highlights the separation of the GUI between the visual representation (or view) provided by the graphical display and the control capacity mediated by the mouse and keyboard of the GUI.

Figure 2

Figure 2B presents an alternate interaction model for tangible interfaces that we call MCRpd, for model-control-representation (physical and digital). We carry over the “model” and “control” elements from the MVC model, while dividing the “view” element into two subcomponents. In particular, we replace the “view” notion with physical representations (abbreviated “rep-p”) for the artifacts constituting the physically embodied elements of tangible interfaces, and with digital representations (“rep-d”) for the computationally mediated components of tangible interfaces without embodied physical form (e.g., video projection and audio).

Where the MVC model of Figure 2A illustrates the GUI's distinction between graphical representation and control, MCRpd highlights the TUI's integration of physical representation and control. This integration is present not only at a conceptual level, but also in physical point of fact—TUI artifacts (or “tangibles”^8-10) physically embody both the control pathway and a central representational (information-bearing) aspect of the interface.

Key characteristics

As illustrated in Figure 3, the MCRpd model highlights the following three key characteristics of tangible interfaces:

Figure 3

1. Physical representations (rep-p) are computationally coupled to underlying digital information (model).

The central characteristic of tangible interfaces is the coupling of physical representations to underlying digital information and computational models. As illustrated by the Urp example, a range of digital couplings are possible, such as the coupling of data to the building models, operations to the wind tool, and property modifiers to the material wand.

2. Physical representations embody mechanisms for interactive control (control).

The physical representations of TUIs serve simultaneously as interactive physical controls. Tangibles may be physically inert, moving only as directly manipulated by a user's hands. Tangibles may also be physically actuated, whether through motor-driven force feedback approaches as described by MacLean et al.¹¹ or by way of induced approaches such as the vibrating plates described by Reznick et al.¹²

Tangibles may be unconstrained and manipulated in free space with six degrees of freedom. They may also be weakly constrained through manipulation on a planar surface, or tightly constrained, as in the movement of the abacus beads with one degree of freedom.

3. Physical representations are perceptually coupled to actively mediated digital representations (rep-d).

Tangible interfaces rely on a balance between physical and digital representations. Although embodied physical elements play a central, defining role in the representation and control of TUIs, digital representations—especially, graphics and audio—often mediate much of the dynamic information provided by the underlying computational system.

“Representation” is a powerful term, taking on different meanings within different communities. We will consider digital representations to be computationally mediated displays that may be perceptually observed in the world but are not embodied in physically manipulable form.

In addition to the above three characteristics, which draw directly from our MCRpd model, a fourth TUI characteristic is also significant.

4. Physical state of tangibles embodies key aspects of the digital state of a system.

Tangible interfaces are generally built from systems of physical artifacts. Taken together as ensembles, TUI tangibles have several important properties. As physical artifacts, TUI tangibles are persistent—they cannot be spontaneously called into or banished from existence. Tangibles also carry physical state, with their physical configurations tightly coupled to the digital state of the systems they represent.

Building from these properties, tangible interfaces often combine tangibles together into several major interpretations. In spatial approaches, the spatial configurations of tangibles within some grounding reference frame serve as defining parameters for the underlying system. For instance, in the Urp example, the positions and orientations of building models, the wind tool, material wand, and other artifacts are all spatially framed and interpreted within the urban workspace.

In addition to spatial approaches, several other interpretations are possible. In relational approaches, the sequence, adjacencies, or other logical relationships between systems of multiple tangibles are mapped to computational interpretations. Alternatively, a kind of middle ground between spatial and relational approaches involves the constructive assembly of modular elements, often coupled together mechanically in fashions analogous (and sometimes quite literal) to the classic LEGO** assemblies of modular bricks.

A second example

Figure 4

The mediaBlocks are small, digitally tagged blocks, dynamically bound to lists of on-line media elements. The mediaBlocks support two major modes of use. First, they function as capture, transport, and playback mechanisms for moving on-line media between different media devices.

In this mode, conference room cameras, digital whiteboards, wall displays, printers, and other devices are outfitted with mediaBlock slots. Inserting one of the mediaBlocks into the slot of a recording device (e.g., a camera) activates the recording of media into on-line space, and the dynamic binding of the media to the physical block.

Similarly, inserting one of the bound mediaBlocks into a playback device (e.g., video display) activates playback of the associated on-line media. Inserting mediaBlocks into slots mounted on computer monitors provides an intermediate case, allowing mediaBlock contents to be exchanged bidirectionally with traditional computer applications using the GUI drag-and-drop operation.

The second functionality of mediaBlocks uses the blocks as physical controls on a media sequencing device. A mediaBlock “sequence rack” (partially modeled after the tile racks of the Scrabble** game) allows the media contents of multiple adjacent mediaBlocks to be dynamically bound to a new mediaBlock carrier. Similarly, a second “position rack” maps the physical position of a block to an indexing operation upon its contents. When mediaBlocks are positioned on the left edge of the position rack, the first media element of the block is selected. Intermediate physical positions on the rack provide access to later elements in the associated media list of the block.

Coupling artifacts with digital information

In the mediaBlocks system, the physical blocks act as containers for lists of images, video, and other digital media. Unlike the building models of Urp, mediaBlocks are not physically suggestive of their particular contents. Instead, they may be quickly bound and rebound to alternate media “contents” over the course of interaction by way of operations associated with the slots, racks, and pads of mediaBlock devices.

As these examples suggest, tangible interfaces afford a wide variety of associations between physical objects and digital information. Tangibles may be statically coupled or dynamically bound to computationally mediated associations including:

• Static digital media, such as images and three-dimensional (3-D) models
• Dynamic digital media, such as video and dynamic graphics
• Digital attributes, such as color or other material properties
• Computational operations, applications, and agents
• Remote people, places, and devices
• Simple data structures, such as lists of media objects
• Complex data structures, such as combinations of data, operations, and attributes

The artifacts embodying these associations take on a range of physical forms, from generic to highly representational. This range of physical and digital forms in some respects parallels the design space of GUI icons. For three decades, GUI icons have been used to represent files, folders, applications, attributes, devices, system services, and many other associations, using a range of abstract and representational graphical forms.

Noting these parallels, we introduced the term “phicon,”³ saying “we physically instantiate GUI ‘icons’ as TUI ‘phicons’ (physical icons) with varying levels of representational abstraction.”¹⁵ We also discussed a range of abstract to literal phicon forms, drawing from related icon discussions by Houde and Salomon.¹⁶

As originally posed, the phicon notion raised the possibility that tangible interfaces might profit from past attempts at frameworks for GUI icons, such as the Xerox Star.¹⁷ However, the term also faces several pitfalls. First, as the creators of the Xerox Star note, “the use of the term ‘icon’ has widened to refer to any nontextual symbol on the display … It would be more consistent with its normal meaning if ‘icon’ were reserved for objects having behavioral and intrinsic properties. Most graphical symbols and labels on computer screens are therefore not icons.”¹⁸

In our early discussions of abstract and literal phicon forms, we implicitly invoked the broader, somewhat imprecise sense of GUI icons. One path toward a more careful approach draws upon the large body of published work analyzing GUI icons. For instance, in an excellent 1993 paper on the subject, Familant and Detweiler discuss seven previous attempts at taxonomies for GUI icons.¹⁸

Symbolic and iconic representation. Many icon taxonomies have been grounded upon the discipline of semiotics—in particular, the Peircean notion of signs, icons, and symbols. Familant and Detweiler note that “according to Peirce, a sign ‘is something which stands to somebody for something in some respect or capacity.’ … For Peirce, an icon is a sign that shares characteristics with the objects to which it refers … A symbol stands in an essentially arbitrary relationship to the thing it signifies.”¹⁸

Alternatively expressed, the physical or graphical forms of iconic signs share representational properties in common with the objects to which they refer. In contrast, symbolic signs need not share such visual or physical references.

It is important to make clear that the “symbolic” versus “iconic” distinction is related, but not equivalent, to the issue of “abstract” versus “highly representational” forms. For example, Gorbet discusses the example of abstraction in comics, where the representation of a character may range from a photograph (uniquely representational) to a “smiley face” (minimally representational).^19,20 For Peirce, these continuums of representations are all instances of iconic reference. However, if we represent a person with the form of an apple or geometrical cube, we are using a symbolic reference.

From this vantage, the building models of Urp and the metaDESK¹⁵ are clearly “iconic.” Conversely, mediaBlocks and the marbles of Bishop's answering machine²¹ are “symbolic” in character—their physical forms do not share representational properties with their digital associations.

Functional roles. The notions of iconic and symbolic tangibles provide a starting point for considering the critical role of physical representation within tangible interfaces. However, these terms do not describe the specific functional roles served by TUI tangibles.

Toward these ends, Holmquist, Redström, and Ljungstrand suggest the terms “containers,” “tokens,” and “tools”²² and discuss a number of the physical and computational properties of these elements. They consider containers and tokens to be symbolic and iconic representations of digital information, respectively, while describing tools more broadly as representations of computational functions.

Aspects of this terminology have been discussed elsewhere. For instance, Fitzmaurice references the idea of objects as containers in his discussion of the LegoWall prototype,² and the container notion was discussed at some length in several theses.^19,23 Nonetheless, the selection of terms by Holmquist et al. provides a useful language for discussing some of the functional differences between, say, the buildings (tokens) of Urp, the wind, wand, and clock devices (tools) of Urp, and mediaBlocks (containers).

Tangible interface instances

Table 1 lists some of the systems that can be productively considered in terms of the emerging framework we have introduced. We have divided this table into four broad categories, corresponding to different manners in which tangibles are integrated into tangible interfaces. Individual systems are listed in order of publication.

The approaches of the first three columns rely upon the configuration of multiple interdependent tangibles, according to the spatial, constructive, and relational interpretations we have discussed earlier in the paper. These approaches are not mutually exclusive, and our table includes a subcategory of systems sharing both constructive and relational characteristics. In the fourth category of “associative” systems, tangibles are individually associated with digital information and do not reference other objects to derive meaning. This distinction will hopefully become clearer in the discussion ahead.

The organization of Table 1 is not intended as a taxonomy. For the present, our primary objective is to provide a starting point for considering these many systems not as isolated instances, but as related elements of a larger, fairly well-populated design space, with shared attributes that may be usefully compared among one another.

Spatial systems. In the first column of Table 1, we list tangible interfaces that interpret the spatial position and orientation of multiple physical artifacts within common frames of reference. Many of these systems involve the configuration of iconic tokens on a horizontal surface. The metaDESK,¹⁵ InterSim,²⁴ and Urp⁴ systems center around physical models of buildings. Twin Objects focuses on a factory planning context, with physical models of assembly-line equipment.²⁵ Illuminating Light presents a holographic simulator, with physical models of lasers, mirrors, lenses, etc.²⁶ Finally, the Zowie system is a commercial play set where physical models of game characters are manipulated to drive interactions in the play world.²⁷

Other systems use symbolic physical handles for manipulating graphical objects. The Bricks system introduced this idea in the 1995 paper by Fitzmaurice et al.,¹ accompanying it with a sample drawing application. Bricks also supported off-screen binding to graphical objects and properties by “dunking” bricks into receptacles within a physical “tray.”

BuildIt used brick-like physical handles in furniture layout and assembly-line design tasks.²⁸ The InfoBinder prototype used objects both as handles and containers for information on a table-projected GUI desktop.²⁹ The InfoBinder paper also described how these objects could be used to transport information between the graphical desktop and real-world devices such as a telephone.

Several spatial interfaces have been used in visualization-related capacities. In work by Hinckley et al.³⁰ a doll's-head physical “prop” was used to orient and scale a neurosurgical brain visualization, while cutting plane and trajectory props were manipulated with the second hand to operate on brain data. In the LEGO props work of Small,³¹ physical manipulation of a LEGO helicopter allowed the navigation of a complex spatial scene, as well as dynamic spatial selection and application of material properties.

Many spatial systems configure objects on a horizontal graphical front- or back-projected surface. Partially following in the tradition of Wellner's influential DigitalDesk,³² the InfoBinder,²⁹ BuildIt,²⁸ Illuminating Light,²⁶ and Urp⁴ systems use front-projected tables, whereas Bricks¹ and the metaDESK¹⁵ use back-projected workbenches. The remaining spatial systems display results on traditional computer monitors. Computer vision and magnetic tracking devices (e.g., Ascension Flock of Birds) are common sensing strategies.

Constructive systems. Some of the earliest tangible interfaces developed modular, electronically instrumented artifacts for constructing models of physical-world systems. Beginning in the late 1970s, Aish^33,34 and Frazer^35,36 implemented a “building block system” (BBS) and a series of “intelligent modeling” kits, respectively, for interactively representing both the structure and properties (e.g., thermal performance) of physical-world buildings. Several of Frazer's systems—e.g., the Universal Constructor,³⁶ a large system of modular interconnecting electronic cubes—were also used to represent more abstract systems, such as physically manipulable cellular automata.

Another early system, the “geometry-defining processors” (or GDP), functioned in the domain of fluid mechanics.³⁷ A system of 10-cm magnetically interlocking cubes, GDP was used to physically express—and in some respects, internally compute—three-dimensional fluid flow simulations.

Several other TUIs use blocks and tiles as primitive units for constructing computationally interpreted physical structures. Examples include the triangular, magnetic-hinging tiles of Triangles;³⁸ the square, LED (light-emitting diode)-faced Tiles;³⁹ the beads and “stackables” described by Resnick and others;^40,41 the LED-illuminated hemispheres of Nami;⁴² and the LEGO-like Blocks⁴³ and Programming Bricks.⁴⁴ In addition to their constructive aspects, several of these systems are also examples of relational approaches, as indicated in the table.

Relational systems. A number of relational systems have developed applications at the intersection of the education and programming domains. One of the earliest such examples is Perlman's Slot Machine, a physical interface for controlling the robotic (and screen-based) “Turtle” of LOGO, the computer-based learning environment.⁴⁵ In this interface, sequences of physical “action,” “number,” “variable,” and “conditional” cards were configured in physical slots to construct LOGO programs.

The AlgoBlock⁴⁶ and Programming Bricks⁴⁴ systems also support the physical expression of programs through the constructive assembly of physical blocks. Systems of programmable blocks, beads, balls, tiles, and “stackables” have also been implemented as instances of “digital manipulatives,” enabling children to explore concepts such as feedback and emergence.^39-41

Outside of the educational domain, one of the earliest works is Bishop's influential marble answering machine.²¹ This interface coupled voice messages with physical marbles, allowing these messages to be replayed, their callers to be redialed, and messages to be stored through manipulation of the physical marbles. In addition to the marble answering machine, Bishop developed a broader series of work exploring the manipulation of physically instantiated information.⁴⁷

We discussed the mediaBlocks system earlier in the paper. The LogJam video logging and ToonTown audio conferencing prototypes made earlier uses of tangibles manipulated on a multitier rack structure. In the LogJam system, domino-like physical blocks represented video annotations, which were added and removed to the racks to annotate video footage by a group of video loggers.⁹ In ToonTown, models of cartoon characters represented human participants in an audio conferencing system.¹⁰ Manipulation of tokens on the rack controlled audio panning, loudness, and token information display and assignment.

The LegoWall system implemented a wall-based matrix of electronically sensed LEGO bricks, which was applied to an example ship-scheduling application.² Matrix axes were mapped to time of day and different shipping ports. LEGO objects containing information about different ships could be plugged into grid locations corresponding to their scheduled arrivals or attached to cells allowing the display and printing of information about these ships.

The Paper Palette associates slides of a digital presentation with paper cards, giving an entire presentation the form of a deck of cards.⁴⁸ This interface facilitates the simple physical insertion, removal, and rearrangement of slides within a digital presentation, as well as the reuse of slides between different presentations.

Associative systems. In our fourth category of “associative” systems, we list several interfaces that associate individual physical artifacts with digital information but do not integrate the associations of multiple tangibles into larger-scale relationships. We are less confident of the utility of this category than those we have considered thus far. Nonetheless, the instances we have identified do seem to exhibit some consistency, suggesting the category may have merit.

To consider several examples, the musicBottles⁴⁹ and Voice Boxes⁵⁰ interfaces associate the capture and release of audio contents with physical bottles and boxes. With musicBottles, the different instruments or voices of a musical composition are stored in a set of physical bottles. As each bottle is opened, the corresponding musical contents are released. With Voice Boxes, each individual box records audio when tilted, and replays this audio when opened.

Because musicBottles are interdependent in terms of behavior (each bottle contains a different voice of a single, synchronous musical composition), we consider them to be an example of a relational interface. In contrast, since each Voice Box holds its own audio association, stored and replayed independently from other Voice Boxes, we consider them to be an associative interface.

As another example, the LegoHead,⁵¹ SAGE,⁵² and Rosebud⁵³ systems all use physical representations of conversational characters toward pedagogical ends. In LegoHead and SAGE, the characters have detachable body parts and clothing that act as “computational construction kit[s] to build creatures [which] behave differently depending on how these parts are attached.”⁵² In Rosebud, electronically instrumented stuffed animals are used as interactive containers for narratives by their owners.⁵³

Following the quoted description, we consider LegoHead and SAGE as examples of both constructive and relational systems. However, we consider Rosebud to be an associative system, given independence of its composing objects.

We also consider the POEMs,²³ Passage,⁵⁴ and WebStickers²² interfaces to be examples of associative systems. POEMs associates personally significant objects such as seashells and books with images, sounds, and annotations.²³ The Passage system binds digital associations to everyday objects such as watches, pens, and glasses, as a physical means for transporting digital information between different augmented devices.⁵⁴ The WebStickers system provides digitally coded stickers that may be attached to associate Web URLs (uniform resource locators) with artifacts such as conference proceedings, drinking mugs, and other physical objects.²²

Observations. It is neither reasonable nor productive to seek categories for tangible interfaces with the same rigor as, say, the ordering of the chemical elements in the periodic table. The semantics of user interfaces are governed by no such immutable physical laws. Nonetheless, we believe that Table 1 serves to highlight several interesting tendencies among tangible interface mappings.

For instance, the tangibles of spatial and associative systems are predominantly iconic in form, whereas those of constructive and relational approaches are predominantly symbolic. The container functionality is widespread across both relational and (predominantly iconic) associative systems, but relatively uncommon among other mappings. Also, support for dynamic binding seems to show trends across the interfaces, although this propensity appears somewhat more complex.

We believe these observations are useful both in illustrating common tendencies among present-day TUIs, as well as indicating less common properties that may suggest opportunities for future research.

Many of these trends are reasonably intuitive in nature. It is not surprising that symbolic tangibles are common among relational systems or that containers are often accompanied by support for dynamic binding (albeit not in associative systems). We also acknowledge that Table 1 is populated by a relatively small number of limited research prototypes and includes exceptions to the tendencies we have described.

Mature systems may often combine many strategies and mappings. For instance, while the Urp urban planning simulator makes static bindings between computer-aided drawing (CAD) geometries and building phicons, material properties are dynamically bound. In the continuing work of Urp, constructive approaches are also under development, where building elevations can be physically expressed through the stacking of modular layers.

Along similar lines, the musicBottles and Voice Boxes can be alternately argued to represent either iconic or symbolic approaches. Although the bottle and box artifacts are iconic with respect to their container status (in a similar fashion to the folder icon of GUIs), they are symbolic if considered directly as representations of their internal contents. In the case of the GUI folder, alternate graphical representations are provided for the container versus its contents. However, for musicBottles and Voice Boxes, the physical container itself is the only embodied mechanism for accessing (audible) contents.

Regarding such issues, Familant and Detweiler conclude: “ … many signals stand in complex relations to many referents … it should be recognized that any careful examination of signals will reveal that many of them cannot be labeled as being of one ‘kind,’ but are properly described as being composites of many different types.”¹⁸

Application domains

Information storage, retrieval, and manipulation—Perhaps the largest class of TUI applications is the use of tangibles as manipulable containers for digital media. Examples include mediaBlocks, musicBottles, Voice Boxes, Triangles, the marble answering machine, the Paper Palette, LegoWall, InfoBinder, LogJam, ToonTown, InteractiveDesk, Passage, POEMs, Rosebud, and WebStickers.

Information visualization—As we will elaborate in the next section on related areas, TUIs broadly relate to the intersection of computation and external cognition. As such, they share common ground with the area of information visualization. TUIs offer the potential for richer representations and input, trading off increased specialization for the cost of general-purpose flexibility.

Many tangible interfaces illustrate properties relating to information visualization (or more broadly, information representation). Particularly suggestive examples include Urp, neurosurgical props, LEGO props, Triangles, the Universal Constructor and intelligent modeling systems, GDP, Tiles, and Nami.

Simulation—Simulators represent another major class of tangible interfaces. Examples include Illuminating Light, Urp, GDP, the Universal Constructor, Tiles, Beads, Stackables, BuildIt, Twin Objects, LegoWall, and InterSim.

Modeling and construction—Several TUIs use cubes, blocks, and tiles as primitive units for constructing and modeling geometric physical structures, which in turn are associated with underlying digital models. Instances include the building blocks system (BBS), intelligent modeling systems, geometry-defining processors (GDP), Blocks, and Triangles.

Systems management, configuration, and control—Several TUIs illustrate the broad capacity for manipulating and controlling complex systems such as video networks, industrial plants, etc. Examples include mediaBlocks, Triangles, LegoWall, Twin Objects, AlgoBlocks, ToonTown, and LogJam.

Education—Another major grouping of TUIs relates to the education domain. Beyond the above simulator examples, related TUIs include the Slot Machine, AlgoBlock, Triangles, LegoHead, and the longstanding work of Resnick et al. with digital manipulatives and programmable bricks.⁵⁵

Programming systems—Several tangible interfaces have demonstrated techniques for programming algorithmic systems with physical objects. Examples include the Slot Machine, AlgoBlock, Tiles, and Programming Bricks.

Collocated collaborative work—Tangible interfaces naturally well-support collocated cooperative work, by virtue of their many loci of physical control. TUIs that have explicitly addressed this context include AlgoBlock, LogJam, Triangles, Urp, and Illuminating Light.

More broadly viewed, tangible interfaces offer the potential for supporting computationally mediated interactions in physical locales and social contexts where traditional computer use may be difficult or inappropriate. These include meeting spaces, living spaces, and other commercial, industrial, and domestic contexts.

Entertainment—As with many new technologies, tangible interfaces have potential in the entertainment domain. Examples include the (already commercialized) Zowie product,²⁷ as well as research systems such as curlybot,⁵⁶ Nami, Triangles, Blocks, and Digital Manipulatives.

Remote communication and awareness—Another application domain relates to systems that facilitate remote communication and awareness at the periphery of users' attention. Here, we relax the physical control and digital representation aspects of MCRpd and consider systems employing “ambient media.”³

Early examples included the Benches system,⁵⁷ which coupled physically remote benches through temperature and sound; and Live Wire,⁵⁸ which expressed network activity through the spinning of a long “dangling string.” Other ambient media examples include the ambient ROOM,⁵⁹ AROMA,⁶⁰ Pinwheels,⁶¹ the Water Lamp,⁶¹ digital/physical surrogates,⁶² personal ambient displays,⁶³ and the Information Percolator.⁶⁴

Another kind of interface in this broad domain is inTouch.⁶⁵ The inTouch prototype supports haptic gestural communication between physically remote parties through “synchronous distributed physical objects.”

Artistic expression—Several examples of tangible interfaces have been motivated strongly (or even predominantly) by artistic concerns. Examples include Benches, pinwheels, musicBottles, Triangles, and Live Wire.

Augmentation—A final application domain relates to the augmentation of pre-existing physical artifacts and usage contexts. Examples of systems include the DigitalDesk,³² Video Mosaic,⁶⁶ InteractiveDESK,⁶⁷ the paper-based audio notebook,⁶⁸ PingPongPlus,⁶⁹ TouchCounters,⁷⁰ electronic tags,⁷¹ and Object Aura.⁷²

Structured around the computational augmentation of paper documents, notebooks, game tables, storage containers, and so forth, many of these systems are also strong examples of augmented reality and ubiquitous computing approaches.

Related areas

Beyond these examples, traditional games of reasoning and chance present an interesting case example. In prototypical instances such as chess and cribbage, we find systems of physical objects, i.e., the playing pieces, boards, and cards, coupled with the abstract rules these artifacts symbolically represent. The broader space of board, card, and tile games, considered as systems of tokens and reference frames, provides an interesting conceptual parallel and grounding for modeling TUIs.⁷³

Map rooms, “war” rooms, and control rooms offer other examples of the symbolic and iconic uses of physical artifacts. Magnet boards and LEGO boards are sometimes used with reconfigurable tokens for groups to collaboratively track time-evolving processes (we know of such instances in dairies and graduate schools). Within domestic contexts, people use souvenirs and heirlooms as representations of personal histories.^74,75

Scientific and design contexts. The disciplines of cognitive science and psychology are concerned in part with “external representations.” These representations are defined as “knowledge and structure in the environment, as physical symbols, objects, or dimensions, and as external rules, constraints, or relations embedded in physical configurations.”⁷⁶ These and other theories and experiments, including analyses of the cognitive role of physical constraints in tasks like the Towers of Hanoi game,⁷⁷ seem closely applicable to tangible user interfaces.

Considerations of affordances by Gibson⁷⁸ and Norman⁷⁹ have long been of interest to the HCI community, and hold special relevance to tangible interface design. Studies of distributed cognition,^80,81 spatial representation,^82,83 and bimanual manipulation⁸⁴ also have special TUI relevance. The doctoral theses of Fitzmaurice² and Hinckley⁸⁵ have made excellent contributions both by offering perceptive analyses of this literature and also by contributing new studies in these areas.

The discipline of semiotics is concerned in part with the symbolic role of physical objects. The paper has discussed Peircean semiotics in the context of GUI icons and TUI phicons. We have also found the work of Krampen, Rossi-Landi, Prieto, Moles, Boudon, and Von Uexkull of possible relevance to TUI design. In particular, these semioticians consider the relation of physical tools to human language, grammars, and semantics.⁸⁶

The discipline of kinematics has a pervasive concern for physical degrees of freedom and has potential relevance for related TUI concerns. Analyses such as Gruebler's formula for physical constraints seem to have special applicability.⁸⁷ Finally, in the field of industrial design, the literature of product semantics focuses on the representation of interface semantics within designed physical forms.⁸⁸

HCI context. Shneiderman's three principles of “direct manipulation,”⁸⁹ while posed in the context of graphical interfaces, are also directly applicable to tangible interfaces. The first principle—“continuous representation of the object of interest”—knits especially well with the persistent nature of TUI tangibles.

As such, the sizable literature relating to direct manipulation, and associated analyses of topics such as perceptual distance, are broadly relevant to TUI design.⁹⁰ As with other direct manipulation interfaces, TUIs can be said to cultivate tool-like, rather than language-like, modalities of interaction.¹⁷ At the same time, tangible interfaces are also subject to some of the criticisms that have been levied at direct manipulation approaches, including those discussed by Frohlich⁹⁰ and by Gentner and Nielsen.⁹¹

The field of visual languages holds relevance for TUIs. Here, principles such as the “Deutsch Limit,” which suggests the implausibility of more than 50 visual primitives in simultaneous use on the screen,⁹² may have analogues for TUI systems of physical primitives. The area of diagrammatic representation, which has found contributions from both the cognitive science and visual languages communities, also holds special TUI relevance.^93,94

The areas of augmented reality,^95-97 mixed reality,⁹⁸ wearable computing,⁹⁹ and ubiquitous computing¹⁰⁰ hold the closest relation to tangible interfaces among existing major research streams. Although these areas hold in common a concern for physically contextualized interaction, we believe they generally inhabit a different conceptual and design space than tangible interfaces. In particular, where tangible interfaces are centrally concerned with the user interface properties of systems of representational physical artifacts, none of these alternate frameworks shares this emphasis.

Different researchers associate widely divergent interpretations of these terms. For instance, where many researchers consider augmented reality to be within a heavily HMD (head-mounted display)-oriented regime (e.g., Azuma⁹⁶), others hold a view of augmented reality much closer to our discussion of tangible interfaces (e.g., Mackay⁹⁷).

We do not believe these alternate stances are inconsistent, but instead offer different conceptual frameworks, different perspectives and insights, and different points of leverage for considering new kinds of physically embodied user interfaces.

The area of ubiquitous computing is somewhat more difficult to characterize, because from a user interface perspective, few conceptual frameworks have been proposed. Weiser's initial vision¹⁰⁰ has long been an inspiration and catalyst for the whole user interface community. However, from a strict user interface standpoint, most UbiComp work has followed traditional GUI approaches (with Cooperstock's Reactive Room¹⁰¹ standing as one notable exception).

Recent work with “embodied user interfaces” has somewhat extended this perspective, considering new approaches for integrating gestural input with handheld computers.¹⁰² More broadly, the UbiComp concern for bringing computation into niche physical contexts has strongly influenced TUI research. The more evolutionary user interface trajectory of UbiComp also gives it heightened practical relevance in the immediate term.

Fishkin et al. propose “invisible interfaces” as a term potentially relevant to both embodied and tangible interfaces.¹⁰² While we agree upon the importance of interface approaches that more seamlessly integrate with users' work and home environments, we do not see “invisibility” per se as a central theme of tangible interfaces. Nonetheless, we share our colleagues' enthusiasm for identifying new physically grounded approaches for interacting with computationally mediated information.

Conclusion

In researching this paper, we were both humbled and inspired by Halasz's landmark “Seven Issues” hypermedia paper¹⁰³ and equally impressive “‘Seven Issues’ Revised” address.¹⁰⁴ Reflecting on his paper after several years, Halasz remarked that “the Seven Issues paper, in retrospect, takes a very simple and narrow view of what the world of hypermedia encompasses, what was of interest to us as hypermedia researchers.”¹⁰⁴

Expanding on this theme, Halasz reflected on the diversity of the hypermedia community, ranging from differing notions of what constitutes a link, to the divergent interests of literary and technologist practitioners, to the contrasting metrics of success in academia and industry. Again, speaking in 1991, Halasz said “One of the main selling points of hypermedia [relates to] very large document collections [10K­100K documents] … Unfortunately, reality has yet to catch up to the vision.”¹⁰⁴

From the perspective of the year 2000, Halasz's words bring a wondrous reminder of how quickly realities can change and how profoundly long-latent visions can blossom. Although the areas of hypermedia and tangible interfaces are very different in character, Halasz's encounter with unexpected diversity provides an interesting benchmark. For tangible interfaces, who is the community of developers, and what are the dimensions of its diversity?

Our experience suggests this community must include practitioners of computer science and cognitive science, mechanical engineering and electrical engineering, art and design, academia and industry. The fusion of physical and digital worlds provides for an extraordinarily rich, and sparsely populated, design space. We look forward to joining with others in exploring the bounds of its potential.

Acknowledgments

**Trademark or registered trademark of ParcPlace Systems, Inc., the Lego Group, Hasbro, Inc., or J. W. Spear & Sons, PLC.

Cited references and note