0018-8670/00/$5.00 (C) 2000 IBM Emerging frameworks for tangible user interfaces by B. Ullmer and H. Ishii We present steps toward a conceptual framework for tangible user interfaces. We introduce the MCRpd interaction model for tangible interfaces, which relates the role of physical and digital representations, physical control, and underlying digital models. This model serves as a foundation for identifying and discussing several key characteristics of tangible user interfaces. We identify a number of systems exhibiting these characteristics, and situate these within 12 application domains. Finally, we discuss tangible interfaces in the context of related research themes, both within and outside of the human-computer interaction domain. The last decade has seen a large and growing body of research in computational systems embracing physical-world modalities of interaction. This work has led to the identification of several major research themes, including ubiquitous computing, augmented reality, mixed reality, and wearable computing. At the same time, a number of research systems relating to the use of physical artifacts as representations and controls for digital information have not been well-characterized in terms of these earlier frameworks. Fitzmaurice, Ishii, and Buxton took a major step in this direction with their description of "graspable user interfaces."[1,2] Building upon this foundation, we extended these ideas and introduced the term "tangible user interfaces" in our discussion of "Tangible Bits."[3] Among other historical inspirations, we suggested the abacus as a compelling prototypical example. In particular, a key point to note is that the abacus is not an input device. The abacus makes no distinction between "input" and "output." Instead, the beads, rods, and frame of the abacus serve as manipulable physical representations of abstract numerical values and operations. Simultaneously, these component artifacts also serve as physical controls for directly manipulating their underlying associations. This seamless integration of representation and control differs markedly from the mainstream graphical user interface (GUI) approaches of modern human-computer interaction (HCI). Graphical interfaces make a fundamental distinction between "input devices," such as the keyboard and mouse, as controls, and graphical "output devices," such as monitors and head-mounted displays, as portals for representations facilitating human interaction with computational systems. Tangible interfaces, in the tradition of the abacus, explore the conceptual space opened by the elimination of this distinction. In this paper, we take steps toward a conceptual framework for tangible user interfaces. In the process, we hope to characterize not only systems explicitly conceived as "tangible interfaces," but more broadly, numerous past and contemporary systems that may be productively considered in terms of tangible interface characteristics. A first example To better ground our discussions, we begin by introducing an example interface called "Urp," depicted in Figure 1. Urp is a tangible interface for urban planning, based on a workbench for simulating the interactions among buildings in an urban environment.[4,5] The interface combines a series of physical building models and interactive tools with an integrated projector/camera/computer node called the "I/O bulb." Under the mediating illumination of the I/O bulb, the building models of Urp cast graphical shadows onto the workbench surface, corresponding to solar shadows at a particular time of day. The position of the sun can be controlled by turning the physical hands of a clock tool. As the corresponding shadows are transformed, the building models can be moved and rotated to minimize intershadowing problems (shadows cast on adjacent buildings). A physical "material wand" can be used to bind alternate material properties to individual buildings. For instance, when bound with a "glass" material property, buildings cast not only solar shadows, but also solar reflections. These reflections exhibit more complex (and less intuitive) behavior than shadows. Moreover, these reflections pose special problems for urban drivers (roadways are also physically instantiated and simulated by Urp.) Finally, a computational fluid flow simulation is bound to a physical "wind" tool. By adding this object to the workbench, a wind-flow simulation is activated, with field lines graphically flowing around the buildings (which remain interactively manipulable). Changing the physical orientation of the wind tool correspondingly alters the orientation of the computationally simulated wind. Tangible user interfaces As illustrated by the previous example, tangible interfaces give physical form to digital information, employing physical artifacts both as representations and controls for computational media. Tangible user interfaces (TUIs) couple physical representations (e.g., spatially manipulable physical objects) with digital representations (e.g., graphics and audio), yielding user interfaces that are computationally mediated but generally not identifiable as "computers" per se. Clearly, traditional user interface elements such as keyboards, mice, and screens are also "physical" in form. Here, the role of physical representation provides an important distinction. For example, in the Urp tangible interface, physical models of buildings are used as physical representations of actual buildings. The physical forms (representing specific buildings) of the Urp models, as well as their position and orientation on the workbench of the system, serve central roles in representing and controlling the state of the user interface. Even if the mediating computers, cameras, and projectors of Urp are turned off, many aspects of the state of the system are still concretely expressed by the configuration of its physical elements. In contrast, the physical form of the mouse holds little "representational" significance. GUIs represent information almost entirely in visual form. Although the mouse mediates control over the graphical cursor of the GUI, its function can be equally served by a trackball, joystick, digitizer pen, or other types of input peripherals. This invariance differs sharply from the Urp example, where the interface is closely coupled to the identity and physical configuration of specific, physically representational artifacts. Interaction model Ideas about "representation" and "control" play central roles within tangible interfaces. In order to more carefully consider the relationship between these concepts, we have developed an interaction model drawing from the "model-view-controller" (MVC) archetype. In its original formulation, MVC served as a technical model for GUI software design, developed in conjunction with the Smalltalk-80** programming language.[6] However, we believe the MVC model also provides a tool for studying the conceptual architecture of graphical interfaces and for relating this architecture to the tangible interface approach. Although alternate interaction models such as PAC[7] may also hold relevance, we find MVC's exposure of the view/control distinction to be useful. We illustrate the MVC model in Figure 2A. MVC highlights the separation of the GUI between the visual representation (or view) provided by the graphical display and the control capacity mediated by the mouse and keyboard of the GUI. Figure 2B presents an alternate interaction model for tangible interfaces that we call MCRpd, for model-control-representation (physical and digital). We carry over the "model" and "control" elements from the MVC model, while dividing the "view" element into two subcomponents. In particular, we replace the "view" notion with physical representations (abbreviated "rep-p") for the artifacts constituting the physically embodied elements of tangible interfaces, and with digital representations ("rep-d") for the computationally mediated components of tangible interfaces without embodied physical form (e.g., video projection and audio). Where the MVC model of Figure 2A illustrates the GUI's distinction between graphical representation and control, MCRpd highlights the TUI's integration of physical representation and control. This integration is present not only at a conceptual level, but also in physical point of fact--TUI artifacts (or "tangibles"[8-10]) physically embody both the control pathway and a central representational (information-bearing) aspect of the interface. Key characteristics The MCRpd interaction model provides a tool for examining several important properties of tangible interfaces. In particular, it is useful to consider the three relationships shared by the physical representations ("rep-p") of TUIs. As illustrated in Figure 3, the MCRpd model highlights the following three key characteristics of tangible interfaces: 1. Physical representations (rep-p) are computationally coupled to underlying digital information (model). The central characteristic of tangible interfaces is the coupling of physical representations to underlying digital information and computational models. As illustrated by the Urp example, a range of digital couplings are possible, such as the coupling of data to the building models, operations to the wind tool, and property modifiers to the material wand. 2. Physical representations embody mechanisms for interactive control (control). The physical representations of TUIs serve simultaneously as interactive physical controls. Tangibles may be physically inert, moving only as directly manipulated by a user's hands. Tangibles may also be physically actuated, whether through motor-driven force feedback approaches as described by MacLean et al.[11] or by way of induced approaches such as the vibrating plates described by Reznick et al.[12] Tangibles may be unconstrained and manipulated in free space with six degrees of freedom. They may also be weakly constrained through manipulation on a planar surface, or tightly constrained, as in the movement of the abacus beads with one degree of freedom. 3. Physical representations are perceptually coupled to actively mediated digital representations (rep-d). Tangible interfaces rely on a balance between physical and digital representations. Although embodied physical elements play a central, defining role in the representation and control of TUIs, digital representations--especially, graphics and audio--often mediate much of the dynamic information provided by the underlying computational system. "Representation" is a powerful term, taking on different meanings within different communities. We will consider digital representations to be computationally mediated displays that may be perceptually observed in the world but are not embodied in physically manipulable form. In addition to the above three characteristics, which draw directly from our MCRpd model, a fourth TUI characteristic is also significant. 4. Physical state of tangibles embodies key aspects of the digital state of a system. Tangible interfaces are generally built from systems of physical artifacts. Taken together as ensembles, TUI tangibles have several important properties. As physical artifacts, TUI tangibles are persistent--they cannot be spontaneously called into or banished from existence. Tangibles also carry physical state, with their physical configurations tightly coupled to the digital state of the systems they represent. Building from these properties, tangible interfaces often combine tangibles together into several major interpretations. In spatial approaches, the spatial configurations of tangibles within some grounding reference frame serve as defining parameters for the underlying system. For instance, in the Urp example, the positions and orientations of building models, the wind tool, material wand, and other artifacts are all spatially framed and interpreted within the urban workspace. In addition to spatial approaches, several other interpretations are possible. In relational approaches, the sequence, adjacencies, or other logical relationships between systems of multiple tangibles are mapped to computational interpretations. Alternatively, a kind of middle ground between spatial and relational approaches involves the constructive assembly of modular elements, often coupled together mechanically in fashions analogous (and sometimes quite literal) to the classic LEGO** assemblies of modular bricks. A second example The mediaBlocks system (Figure 4) is a tangible interface for logically manipulating lists of on-line video, images, and other media elements.[13,14] Where the Urp simulator provides a spatial interface leveraging object arrangements consistent with real-world building configurations, the mediaBlocks system provides a relational interface for manipulating more abstract digital information. The mediaBlocks are small, digitally tagged blocks, dynamically bound to lists of on-line media elements. The mediaBlocks support two major modes of use. First, they function as capture, transport, and playback mechanisms for moving on-line media between different media devices. In this mode, conference room cameras, digital whiteboards, wall displays, printers, and other devices are outfitted with mediaBlock slots. Inserting one of the mediaBlocks into the slot of a recording device (e.g., a camera) activates the recording of media into on-line space, and the dynamic binding of the media to the physical block. Similarly, inserting one of the bound mediaBlocks into a playback device (e.g., video display) activates playback of the associated on-line media. Inserting mediaBlocks into slots mounted on computer monitors provides an intermediate case, allowing mediaBlock contents to be exchanged bidirectionally with traditional computer applications using the GUI drag-and-drop operation. The second functionality of mediaBlocks uses the blocks as physical controls on a media sequencing device. A mediaBlock "sequence rack" (partially modeled after the tile racks of the Scrabble** game) allows the media contents of multiple adjacent mediaBlocks to be dynamically bound to a new mediaBlock carrier. Similarly, a second "position rack" maps the physical position of a block to an indexing operation upon its contents. When mediaBlocks are positioned on the left edge of the position rack, the first media element of the block is selected. Intermediate physical positions on the rack provide access to later elements in the associated media list of the block. Coupling artifacts with digital information The Urp and mediaBlocks examples have illustrated several different approaches for using physical artifacts to represent underlying digital information. In Urp, physical models representing specific buildings are statically coupled to digital models of the geometries of these buildings. At the same time, material properties can be dynamically bound to buildings using the material wand, and a wind simulation can be invoked and oriented through manipulation of the wind tool. In the mediaBlocks system, the physical blocks act as containers for lists of images, video, and other digital media. Unlike the building models of Urp, mediaBlocks are not physically suggestive of their particular contents. Instead, they may be quickly bound and rebound to alternate media "contents" over the course of interaction by way of operations associated with the slots, racks, and pads of mediaBlock devices. As these examples suggest, tangible interfaces afford a wide variety of associations between physical objects and digital information. Tangibles may be statically coupled or dynamically bound to computationally mediated associations including: o Static digital media, such as images and three-dimensional (3-D) models o Dynamic digital media, such as video and dynamic graphics o Digital attributes, such as color or other material properties o Computational operations, applications, and agents o Remote people, places, and devices o Simple data structures, such as lists of media objects o Complex data structures, such as combinations of data, operations, and attributes The artifacts embodying these associations take on a range of physical forms, from generic to highly representational. This range of physical and digital forms in some respects parallels the design space of GUI icons. For three decades, GUI icons have been used to represent files, folders, applications, attributes, devices, system services, and many other associations, using a range of abstract and representational graphical forms. Noting these parallels, we introduced the term "phicon,"[3] saying "we physically instantiate GUI 'icons' as TUI 'phicons' (physical icons) with varying levels of representational abstraction."[15] We also discussed a range of abstract to literal phicon forms, drawing from related icon discussions by Houde and Salomon.[16] As originally posed, the phicon notion raised the possibility that tangible interfaces might profit from past attempts at frameworks for GUI icons, such as the Xerox Star.[17] However, the term also faces several pitfalls. First, as the creators of the Xerox Star note, "the use of the term 'icon' has widened to refer to any nontextual symbol on the display ... It would be more consistent with its normal meaning if 'icon' were reserved for objects having behavioral and intrinsic properties. Most graphical symbols and labels on computer screens are therefore not icons."[18] In our early discussions of abstract and literal phicon forms, we implicitly invoked the broader, somewhat imprecise sense of GUI icons. One path toward a more careful approach draws upon the large body of published work analyzing GUI icons. For instance, in an excellent 1993 paper on the subject, Familant and Detweiler discuss seven previous attempts at taxonomies for GUI icons.[18] Symbolic and iconic representation. Many icon taxonomies have been grounded upon the discipline of semiotics--in particular, the Peircean notion of signs, icons, and symbols. Familant and Detweiler note that "according to Peirce, a sign 'is something which stands to somebody for something in some respect or capacity.' ... For Peirce, an icon is a sign that shares characteristics with the objects to which it refers ... A symbol stands in an essentially arbitrary relationship to the thing it signifies."[18] Alternatively expressed, the physical or graphical forms of iconic signs share representational properties in common with the objects to which they refer. In contrast, symbolic signs need not share such visual or physical references. It is important to make clear that the "symbolic" versus "iconic" distinction is related, but not equivalent, to the issue of "abstract" versus "highly representational" forms. For example, Gorbet discusses the example of abstraction in comics, where the representation of a character may range from a photograph (uniquely representational) to a "smiley face" (minimally representational).[19,20] For Peirce, these continuums of representations are all instances of iconic reference. However, if we represent a person with the form of an apple or geometrical cube, we are using a symbolic reference. From this vantage, the building models of Urp and the metaDESK[15] are clearly "iconic." Conversely, mediaBlocks and the marbles of Bishop's answering machine[21] are "symbolic" in character--their physical forms do not share representational properties with their digital associations. Functional roles. The notions of iconic and symbolic tangibles provide a starting point for considering the critical role of physical representation within tangible interfaces. However, these terms do not describe the specific functional roles served by TUI tangibles. Toward these ends, Holmquist, Redstrom, and Ljungstrand suggest the terms "containers," "tokens," and "tools"[22] and discuss a number of the physical and computational properties of these elements. They consider containers and tokens to be symbolic and iconic representations of digital information, respectively, while describing tools more broadly as representations of computational functions. Aspects of this terminology have been discussed elsewhere. For instance, Fitzmaurice references the idea of objects as containers in his discussion of the LegoWall prototype,[2] and the container notion was discussed at some length in several theses.[19,23] Nonetheless, the selection of terms by Holmquist et al. provides a useful language for discussing some of the functional differences between, say, the buildings (tokens) of Urp, the wind, wand, and clock devices (tools) of Urp, and mediaBlocks (containers). Tangible interface instances In the previous sections, we have introduced several descriptions, models, and characteristics by which tangible interfaces can be understood. Next, we use this information to discuss systems that can be considered instances of TUIs. Table 1 lists some of the systems that can be productively considered in terms of the emerging framework we have introduced. We have divided this table into four broad categories, corresponding to different manners in which tangibles are integrated into tangible interfaces. Individual systems are listed in order of publication. Table 1 Tangible interface instances ------------------------------------------------------------------------------- SPATIAL |CONSTRUCTIVE|RELATIONAL |ASSOCIATIVE ------------------------------------------------------------------------------- Neurosurgical I|BBS[33,34] S|Slot << S|Voice Boxes[50] C I props[30] |IModeling S| Machine[4] |POEMS[23] C I Character dev[2] I| [35,36] |Marble Ans[21] C << S|Rosebud[53] C I Bricks[1] << S|GDP[37] S|Lego Wall[2] C S|Passage[54] C << I InfoBinder[29] C << S|Tiles[39] S|mediaBlocks C << S|WebStickers[22] C I metaDESK[15] I|Nami[42] S| [13] | buildIt[28] << S|Blocks[43] S|LogJam[9] C << S| Twin Objects[25] I| |ToonTown[10] C << S| InterSim[24] I| |Paper Palette C S| Illuminating I| | [48] | Light[26] | |musicBottles C I| LEGO props[31] << I| | [49] | Urp[4] S << I|----------------------------------| Zowie[27] I| AlgoBlocks[43] S | | Dr. LegoHead[50] I | | SAGE[52] I | Legend: | Triangles[38] << S C| Iconic: I | Stackables[41] S | Symbolic: S | Beads[40] << S | Container: C | Digital manipulatives[40] << S | Dynamic binding: << | Programming bricks[43] S | ------------------------------------------------------------------------------- The approaches of the first three columns rely upon the configuration of multiple interdependent tangibles, according to the spatial, constructive, and relational interpretations we have discussed earlier in the paper. These approaches are not mutually exclusive, and our table includes a subcategory of systems sharing both constructive and relational characteristics. In the fourth category of "associative" systems, tangibles are individually associated with digital information and do not reference other objects to derive meaning. This distinction will hopefully become clearer in the discussion ahead. The organization of Table 1 is not intended as a taxonomy. For the present, our primary objective is to provide a starting point for considering these many systems not as isolated instances, but as related elements of a larger, fairly well-populated design space, with shared attributes that may be usefully compared among one another. Spatial systems. In the first column of Table 1, we list tangible interfaces that interpret the spatial position and orientation of multiple physical artifacts within common frames of reference. Many of these systems involve the configuration of iconic tokens on a horizontal surface. The metaDESK,[15] InterSim,[24] and Urp[4] systems center around physical models of buildings. Twin Objects focuses on a factory planning context, with physical models of assembly-line equipment.[25] Illuminating Light presents a holographic simulator, with physical models of lasers, mirrors, lenses, etc.[26] Finally, the Zowie system is a commercial play set where physical models of game characters are manipulated to drive interactions in the play world.[27] Other systems use symbolic physical handles for manipulating graphical objects. The Bricks system introduced this idea in the 1995 paper by Fitzmaurice et al.,[1] accompanying it with a sample drawing application. Bricks also supported off-screen binding to graphical objects and properties by "dunking" bricks into receptacles within a physical "tray." BuildIt used brick-like physical handles in furniture layout and assembly-line design tasks.[28] The InfoBinder prototype used objects both as handles and containers for information on a table-projected GUI desktop.[29] The InfoBinder paper also described how these objects could be used to transport information between the graphical desktop and real-world devices such as a telephone. Several spatial interfaces have been used in visualization-related capacities. In work by Hinckley et al.[30] a doll's-head physical "prop" was used to orient and scale a neurosurgical brain visualization, while cutting plane and trajectory props were manipulated with the second hand to operate on brain data. In the LEGO props work of Small,[31] physical manipulation of a LEGO helicopter allowed the navigation of a complex spatial scene, as well as dynamic spatial selection and application of material properties. Many spatial systems configure objects on a horizontal graphical front- or back-projected surface. Partially following in the tradition of Wellner's influential DigitalDesk,[32] the InfoBinder,[29] BuildIt,[28] Illuminating Light,[26] and Urp[4] systems use front-projected tables, whereas Bricks[1] and the metaDESK[15] use back-projected workbenches. The remaining spatial systems display results on traditional computer monitors. Computer vision and magnetic tracking devices (e.g., Ascension Flock of Birds) are common sensing strategies. Constructive systems. Some of the earliest tangible interfaces developed modular, electronically instrumented artifacts for constructing models of physical-world systems. Beginning in the late 1970s, Aish[33,34] and Frazer[35,36] implemented a "building block system" (BBS) and a series of "intelligent modeling" kits, respectively, for interactively representing both the structure and properties (e.g., thermal performance) of physical-world buildings. Several of Frazer's systems--e.g., the Universal Constructor,[36] a large system of modular interconnecting electronic cubes--were also used to represent more abstract systems, such as physically manipulable cellular automata. Another early system, the "geometry-defining processors" (or GDP), functioned in the domain of fluid mechanics.[37] A system of 10-cm magnetically interlocking cubes, GDP was used to physically express--and in some respects, internally compute--three-dimensional fluid flow simulations. Several other TUIs use blocks and tiles as primitive units for constructing computationally interpreted physical structures. Examples include the triangular, magnetic-hinging tiles of Triangles;[38] the square, LED (light-emitting diode)-faced Tiles;[39] the beads and "stackables" described by Resnick and others;[40,41] the LED-illuminated hemispheres of Nami;[42] and the LEGO-like Blocks[43] and Programming Bricks.[44] In addition to their constructive aspects, several of these systems are also examples of relational approaches, as indicated in the table. Relational systems. A number of relational systems have developed applications at the intersection of the education and programming domains. One of the earliest such examples is Perlman's Slot Machine, a physical interface for controlling the robotic (and screen-based) "Turtle" of LOGO, the computer-based learning environment.[45] In this interface, sequences of physical "action," "number," "variable," and "conditional" cards were configured in physical slots to construct LOGO programs. The AlgoBlock[46] and Programming Bricks[44] systems also support the physical expression of programs through the constructive assembly of physical blocks. Systems of programmable blocks, beads, balls, tiles, and "stackables" have also been implemented as instances of "digital manipulatives," enabling children to explore concepts such as feedback and emergence.[39-41] Outside of the educational domain, one of the earliest works is Bishop's influential marble answering machine.[21] This interface coupled voice messages with physical marbles, allowing these messages to be replayed, their callers to be redialed, and messages to be stored through manipulation of the physical marbles. In addition to the marble answering machine, Bishop developed a broader series of work exploring the manipulation of physically instantiated information.[47] We discussed the mediaBlocks system earlier in the paper. The LogJam video logging and ToonTown audio conferencing prototypes made earlier uses of tangibles manipulated on a multitier rack structure. In the LogJam system, domino-like physical blocks represented video annotations, which were added and removed to the racks to annotate video footage by a group of video loggers.[9] In ToonTown, models of cartoon characters represented human participants in an audio conferencing system.[10] Manipulation of tokens on the rack controlled audio panning, loudness, and token information display and assignment. The LegoWall system implemented a wall-based matrix of electronically sensed LEGO bricks, which was applied to an example ship-scheduling application.[2] Matrix axes were mapped to time of day and different shipping ports. LEGO objects containing information about different ships could be plugged into grid locations corresponding to their scheduled arrivals or attached to cells allowing the display and printing of information about these ships. The Paper Palette associates slides of a digital presentation with paper cards, giving an entire presentation the form of a deck of cards.[48] This interface facilitates the simple physical insertion, removal, and rearrangement of slides within a digital presentation, as well as the reuse of slides between different presentations. Associative systems. In our fourth category of "associative" systems, we list several interfaces that associate individual physical artifacts with digital information but do not integrate the associations of multiple tangibles into larger-scale relationships. We are less confident of the utility of this category than those we have considered thus far. Nonetheless, the instances we have identified do seem to exhibit some consistency, suggesting the category may have merit. To consider several examples, the musicBottles[49] and Voice Boxes[50] interfaces associate the capture and release of audio contents with physical bottles and boxes. With musicBottles, the different instruments or voices of a musical composition are stored in a set of physical bottles. As each bottle is opened, the corresponding musical contents are released. With Voice Boxes, each individual box records audio when tilted, and replays this audio when opened. Because musicBottles are interdependent in terms of behavior (each bottle contains a different voice of a single, synchronous musical composition), we consider them to be an example of a relational interface. In contrast, since each Voice Box holds its own audio association, stored and replayed independently from other Voice Boxes, we consider them to be an associative interface. As another example, the LegoHead,[51] SAGE,[52] and Rosebud[53] systems all use physical representations of conversational characters toward pedagogical ends. In LegoHead and SAGE, the characters have detachable body parts and clothing that act as "computational construction kit[s] to build creatures[which] behave differently depending on how these parts are attached."[52] In Rosebud, electronically instrumented stuffed animals are used as interactive containers for narratives by their owners.[53] Following the quoted description, we consider LegoHead and SAGE as examples of both constructive and relational systems. However, we consider Rosebud to be an associative system, given independence of its composing objects. We also consider the POEMs,[23] Passage,[54] and WebStickers[22] interfaces to be examples of associative systems. POEMs associates personally significant objects such as seashells and books with images, sounds, and annotations.[23] The Passage system binds digital associations to everyday objects such as watches, pens, and glasses, as a physical means for transporting digital information between different augmented devices.[54] The WebStickers system provides digitally coded stickers that may be attached to associate Web URLs (uniform resource locators) with artifacts such as conference proceedings, drinking mugs, and other physical objects.[22] Observations. It is neither reasonable nor productive to seek categories for tangible interfaces with the same rigor as, say, the ordering of the chemical elements in the periodic table. The semantics of user interfaces are governed by no such immutable physical laws. Nonetheless, we believe that Table 1 serves to highlight several interesting tendencies among tangible interface mappings. For instance, the tangibles of spatial and associative systems are predominantly iconic in form, whereas those of constructive and relational approaches are predominantly symbolic. The container functionality is widespread across both relational and (predominantly iconic) associative systems, but relatively uncommon among other mappings. Also, support for dynamic binding seems to show trends across the interfaces, although this propensity appears somewhat more complex. We believe these observations are useful both in illustrating common tendencies among present-day TUIs, as well as indicating less common properties that may suggest opportunities for future research. Many of these trends are reasonably intuitive in nature. It is not surprising that symbolic tangibles are common among relational systems or that containers are often accompanied by support for dynamic binding (albeit not in associative systems). We also acknowledge that Table 1 is populated by a relatively small number of limited research prototypes and includes exceptions to the tendencies we have described. Mature systems may often combine many strategies and mappings. For instance, while the Urp urban planning simulator makes static bindings between computer-aided drawing (CAD) geometries and building phicons, material properties are dynamically bound. In the continuing work of Urp, constructive approaches are also under development, where building elevations can be physically expressed through the stacking of modular layers. Along similar lines, the musicBottles and Voice Boxes can be alternately argued to represent either iconic or symbolic approaches. Although the bottle and box artifacts are iconic with respect to their container status (in a similar fashion to the folder icon of GUIs), they are symbolic if considered directly as representations of their internal contents. In the case of the GUI folder, alternate graphical representations are provided for the container versus its contents. However, for musicBottles and Voice Boxes, the physical container itself is the only embodied mechanism for accessing (audible) contents. Regarding such issues, Familant and Detweiler conclude: " ... many signals stand in complex relations to many referents ... it should be recognized that any careful examination of signals will reveal that many of them cannot be labeled as being of one 'kind,' but are properly described as being composites of many different types."[18] Application domains It is interesting to consider the kinds of application domains illustrated by the previous instances of tangible interfaces. To combine legibility with compactness, we reference these systems by name only. Corresponding citations may be cross-referenced through Table 1 and the previous section. These domains include: Information storage, retrieval, and manipulation--Perhaps the largest class of TUI applications is the use of tangibles as manipulable containers for digital media. Examples include mediaBlocks, musicBottles, Voice Boxes, Triangles, the marble answering machine, the Paper Palette, LegoWall, InfoBinder, LogJam, ToonTown, InteractiveDesk, Passage, POEMs, Rosebud, and WebStickers. Information visualization--As we will elaborate in the next section on related areas, TUIs broadly relate to the intersection of computation and external cognition. As such, they share common ground with the area of information visualization. TUIs offer the potential for richer representations and input, trading off increased specialization for the cost of general-purpose flexibility. Many tangible interfaces illustrate properties relating to information visualization (or more broadly, information representation). Particularly suggestive examples include Urp, neurosurgical props, LEGO props, Triangles, the Universal Constructor and intelligent modeling systems, GDP, Tiles, and Nami. Simulation--Simulators represent another major class of tangible interfaces. Examples include Illuminating Light, Urp, GDP, the Universal Constructor, Tiles, Beads, Stackables, BuildIt, Twin Objects, LegoWall, and InterSim. Modeling and construction--Several TUIs use cubes, blocks, and tiles as primitive units for constructing and modeling geometric physical structures, which in turn are associated with underlying digital models. Instances include the building blocks system (BBS), intelligent modeling systems, geometry-defining processors (GDP), Blocks, and Triangles. Systems management, configuration, and control--Several TUIs illustrate the broad capacity for manipulating and controlling complex systems such as video networks, industrial plants, etc. Examples include mediaBlocks, Triangles, LegoWall, Twin Objects, AlgoBlocks, ToonTown, and LogJam. Education--Another major grouping of TUIs relates to the education domain. Beyond the above simulator examples, related TUIs include the Slot Machine, AlgoBlock, Triangles, LegoHead, and the longstanding work of Resnick et al. with digital manipulatives and programmable bricks.[55] Programming systems--Several tangible interfaces have demonstrated techniques for programming algorithmic systems with physical objects. Examples include the Slot Machine, AlgoBlock, Tiles, and Programming Bricks. Collocated collaborative work--Tangible interfaces naturally well-support collocated cooperative work, by virtue of their many loci of physical control. TUIs that have explicitly addressed this context include AlgoBlock, LogJam, Triangles, Urp, and Illuminating Light. More broadly viewed, tangible interfaces offer the potential for supporting computationally mediated interactions in physical locales and social contexts where traditional computer use may be difficult or inappropriate. These include meeting spaces, living spaces, and other commercial, industrial, and domestic contexts. Entertainment--As with many new technologies, tangible interfaces have potential in the entertainment domain. Examples include the (already commercialized) Zowie product,[27] as well as research systems such as curlybot,[56] Nami, Triangles, Blocks, and Digital Manipulatives. Remote communication and awareness--Another application domain relates to systems that facilitate remote communication and awareness at the periphery of users' attention. Here, we relax the physical control and digital representation aspects of MCRpd and consider systems employing "ambient media."[3] Early examples included the Benches system,[57] which coupled physically remote benches through temperature and sound; and Live Wire,[58] which expressed network activity through the spinning of a long "dangling string." Other ambient media examples include the ambient ROOM,[59] AROMA,[60] Pinwheels,[61] the Water Lamp,[61] digital/physical surrogates,[62] personal ambient displays,[63] and the Information Percolator.[64] Another kind of interface in this broad domain is inTouch.[65] The inTouch prototype supports haptic gestural communication between physically remote parties through "synchronous distributed physical objects." Artistic expression--Several examples of tangible interfaces have been motivated strongly (or even predominantly) by artistic concerns. Examples include Benches, pinwheels, musicBottles, Triangles, and Live Wire. Augmentation--A final application domain relates to the augmentation of pre-existing physical artifacts and usage contexts. Examples of systems include the DigitalDesk,[32] Video Mosaic,[66] InteractiveDESK,[67] the paper-based audio notebook,[68] PingPongPlus,[69] TouchCounters,[70] electronic tags,[71] and Object Aura.[72] Structured around the computational augmentation of paper documents, notebooks, game tables, storage containers, and so forth, many of these systems are also strong examples of augmented reality and ubiquitous computing approaches. Beyond these individual application domains, there seems to be a fairly strong relationship between tangible interfaces and networked computational systems. TUIs are frequently coupled to digital associations that depend on computer networks. Especially given the present level of enthusiasm for networked systems, the relationship between TUIs and internetworking may provide grounds for many new conceptual and practical opportunities. Related areas Broad context. Humans are clearly no newcomers to interaction with the physical world or to the process of associating symbolic function and relationships with physical artifacts. We have referenced the abacus example earlier in this paper, which we have discussed in the context of other historic scientific instruments in our paper on tangible bits.[3] Beyond these examples, traditional games of reasoning and chance present an interesting case example. In prototypical instances such as chess and cribbage, we find systems of physical objects, i.e., the playing pieces, boards, and cards, coupled with the abstract rules these artifacts symbolically represent. The broader space of board, card, and tile games, considered as systems of tokens and reference frames, provides an interesting conceptual parallel and grounding for modeling TUIs.[73] Map rooms, "war" rooms, and control rooms offer other examples of the symbolic and iconic uses of physical artifacts. Magnet boards and LEGO boards are sometimes used with reconfigurable tokens for groups to collaboratively track time-evolving processes (we know of such instances in dairies and graduate schools). Within domestic contexts, people use souvenirs and heirlooms as representations of personal histories.[74,75] Scientific and design contexts. The disciplines of cognitive science and psychology are concerned in part with "external representations." These representations are defined as "knowledge and structure in the environment, as physical symbols, objects, or dimensions, and as external rules, constraints, or relations embedded in physical configurations."[76] These and other theories and experiments, including analyses of the cognitive role of physical constraints in tasks like the Towers of Hanoi game,[77] seem closely applicable to tangible user interfaces. Considerations of affordances by Gibson[78] and Norman[79] have long been of interest to the HCI community, and hold special relevance to tangible interface design. Studies of distributed cognition,[80,81] spatial representation,[82,83] and bimanual manipulation[84] also have special TUI relevance. The doctoral theses of Fitzmaurice[2] and Hinckley[85] have made excellent contributions both by offering perceptive analyses of this literature and also by contributing new studies in these areas. The discipline of semiotics is concerned in part with the symbolic role of physical objects. The paper has discussed Peircean semiotics in the context of GUI icons and TUI phicons. We have also found the work of Krampen, Rossi-Landi, Prieto, Moles, Boudon, and Von Uexkull of possible relevance to TUI design. In particular, these semioticians consider the relation of physical tools to human language, grammars, and semantics.[86] The discipline of kinematics has a pervasive concern for physical degrees of freedom and has potential relevance for related TUI concerns. Analyses such as Gruebler's formula for physical constraints seem to have special applicability.[87] Finally, in the field of industrial design, the literature of product semantics focuses on the representation of interface semantics within designed physical forms.[88] HCI context. Shneiderman's three principles of "direct manipulation,"[89] while posed in the context of graphical interfaces, are also directly applicable to tangible interfaces. The first principle--"continuous representation of the object of interest"--knits especially well with the persistent nature of TUI tangibles. As such, the sizable literature relating to direct manipulation, and associated analyses of topics such as perceptual distance, are broadly relevant to TUI design.[90] As with other direct manipulation interfaces, TUIs can be said to cultivate tool-like, rather than language-like, modalities of interaction.[17] At the same time, tangible interfaces are also subject to some of the criticisms that have been levied at direct manipulation approaches, including those discussed by Frohlich[90] and by Gentner and Nielsen.[91] The field of visual languages holds relevance for TUIs. Here, principles such as the "Deutsch Limit," which suggests the implausibility of more than 50 visual primitives in simultaneous use on the screen,[92] may have analogues for TUI systems of physical primitives. The area of diagrammatic representation, which has found contributions from both the cognitive science and visual languages communities, also holds special TUI relevance.[93,94] The areas of augmented reality,[95-97] mixed reality,[98] wearable computing,[99] and ubiquitous computing[100] hold the closest relation to tangible interfaces among existing major research streams. Although these areas hold in common a concern for physically contextualized interaction, we believe they generally inhabit a different conceptual and design space than tangible interfaces. In particular, where tangible interfaces are centrally concerned with the user interface properties of systems of representational physical artifacts, none of these alternate frameworks shares this emphasis. Different researchers associate widely divergent interpretations of these terms. For instance, where many researchers consider augmented reality to be within a heavily HMD (head-mounted display)-oriented regime (e.g., Azuma[96]), others hold a view of augmented reality much closer to our discussion of tangible interfaces (e.g., Mackay[97]). We do not believe these alternate stances are inconsistent, but instead offer different conceptual frameworks, different perspectives and insights, and different points of leverage for considering new kinds of physically embodied user interfaces. The area of ubiquitous computing is somewhat more difficult to characterize, because from a user interface perspective, few conceptual frameworks have been proposed. Weiser's initial vision[100] has long been an inspiration and catalyst for the whole user interface community. However, from a strict user interface standpoint, most UbiComp work has followed traditional GUI approaches (with Cooperstock's Reactive Room[101] standing as one notable exception). Recent work with "embodied user interfaces" has somewhat extended this perspective, considering new approaches for integrating gestural input with handheld computers.[102] More broadly, the UbiComp concern for bringing computation into niche physical contexts has strongly influenced TUI research. The more evolutionary user interface trajectory of UbiComp also gives it heightened practical relevance in the immediate term. Fishkin et al. propose "invisible interfaces" as a term potentially relevant to both embodied and tangible interfaces.[102] While we agree upon the importance of interface approaches that more seamlessly integrate with users' work and home environments, we do not see "invisibility" per se as a central theme of tangible interfaces. Nonetheless, we share our colleagues' enthusiasm for identifying new physically grounded approaches for interacting with computationally mediated information. Conclusion In discussing a broad topic within limited space, we have necessarily left a great many concerns for future consideration. From an HCI standpoint, these include issues of situatedness and physical scale, cognitive engagement and distance, general versus special-purpose approaches, and many others. From an engineering perspective, issues include tagging and tracking technologies, hardware and software architectures, prototyping, toolkits, and beyond. And from a design viewpoint, among a great many particular challenges, there is also a more fundamental one: What makes for good tangible interface design? In researching this paper, we were both humbled and inspired by Halasz's landmark "Seven Issues" hypermedia paper[103] and equally impressive "'Seven Issues' Revised" address.[104] Reflecting on his paper after several years, Halasz remarked that "the Seven Issues paper, in retrospect, takes a very simple and narrow view of what the world of hypermedia encompasses, what was of interest to us as hypermedia researchers."[104] Expanding on this theme, Halasz reflected on the diversity of the hypermedia community, ranging from differing notions of what constitutes a link, to the divergent interests of literary and technologist practitioners, to the contrasting metrics of success in academia and industry. Again, speaking in 1991, Halasz said "One of the main selling points of hypermedia[relates to] very large document collections[10K-100K documents] ... Unfortunately, reality has yet to catch up to the vision."[104] From the perspective of the year 2000, Halasz's words bring a wondrous reminder of how quickly realities can change and how profoundly long-latent visions can blossom. Although the areas of hypermedia and tangible interfaces are very different in character, Halasz's encounter with unexpected diversity provides an interesting benchmark. For tangible interfaces, who is the community of developers, and what are the dimensions of its diversity? Our experience suggests this community must include practitioners of computer science and cognitive science, mechanical engineering and electrical engineering, art and design, academia and industry. The fusion of physical and digital worlds provides for an extraordinarily rich, and sparsely populated, design space. We look forward to joining with others in exploring the bounds of its potential. Acknowledgments We would like to thank Bill Verplank, John Frazer, Ali Mazalek, and the anonymous reviewers for valuable feedback on the paper draft. We also thank John Underkoffler, Paul Yarin, James Patten, Matt Gorbet, and other students of the Tangible Media Group, as well as Lars Erik Holmquist and Johan Redstrom of PLAY, for past and ongoing discussions of many of the ideas in this paper. We also thank Joe Marks for introducing us to the works of Frazer and Anagnostou. This work was supported in part by IBM, Steelcase, Intel, and other sponsors of the MIT Media Lab's Things That Think and Digital Life consortiums. **Trademark or registered trademark of ParcPlace Systems, Inc., the Lego Group, Hasbro, Inc., or J. W. Spear & Sons, PLC. Cited references and note 1. G. Fitzmaurice, H. Ishii, and W. Buxton, "Bricks: Laying the Foundations for Graspable User Interfaces," Proceedings of CHI'95 (1995), pp. 442-449. 2. G. Fitzmaurice, Graspable User Interfaces, Ph.D. thesis, University of Toronto, Toronto (1996). 3. H. Ishii and B. Ullmer, "Tangible Bits: Towards Seamless Interfaces Between People, Bits, and Atoms," Proceedings of CHI'97 (1997), pp. 234-241. 4. J. Underkoffler and H. Ishii, "Urp: A Luminous-Tangible Workbench for Urban Planning and Design," Proceedings of CHI'99 (1999), pp. 386-393. 5. J. Underkoffler, B. Ullmer, and H. Ishii, "Emancipated Pixels: Real-World Graphics in the Luminous Room," Computer Graphics Proceedings (SIGGRAPH 99) (1999), pp. 385-392. 6. S. Burbeck, Applications Programming in Smalltalk-80: How to Use Model-View-Controller (1987), see http://st-www.cs.uiuc.edu/users/smarch/st-docs/mvc.html. 7. J. Coutaz, "PAC, an Object Oriented Model for Dialog Design," Proceedings of Interact'87 (1987), pp. 431-436. 8. The "tangibles" term was used in this context ca. 1994 at Interval Research Corp., associated with the development of the LogJam video logging and ToonTown audio conferencing systems.[9,10] 9. J. Cohen, M. Withgott, and P. Piernot, "LogJam: A Tangible Multi-Person Interface for Video Logging," Proceedings of CHI'99 (1999). 10. A. Singer, D. Hindus, L. Stifelman, and S. White, "Tangible Progress: Less Is More in Somewire Audio Spaces," Proceedings of CHI'99 (1999). 11. K. MacLean, S. Snibbe, and G. Levin, "Tagged Handles: Merging Discrete and Continuous Manual Control," Proceedings of CHI'00 (2000), pp. 225-232. 12. D. Reznick, E. Moshkovich, and J. Canny, "Building a Universal Part Manipulator," Distributed Manipulation, Bohringer and Choset, Editors, Kluwer Academic Press, Dordrecht, the Netherlands (1999). 13. B. Ullmer, H. Ishii, and D. Glas, "mediaBlocks: Physical Containers, Transports, and Controls for Online Media," Computer Graphics Proceedings (SIGGRAPH 98) (1998), pp. 379-386. 14. B. Ullmer and H. Ishii, "mediaBlocks: Tangible Interfaces for Online Media," CHI'99 Extended Abstracts (video demonstration) (1999), pp. 31-32. 15. B. Ullmer and H. Ishii, "The metaDESK: Models and Prototypes for Tangible User Interfaces," Proceedings of UIST'97 (1997), pp. 223-232. 16. S. Houde and G. Salomon, "Working Towards Rich and Flexible File Representations," Proceedings of INTERCHI'93, Adjunct Proceedings (1993), pp. 9-10. 17. J. Johnson, T. Roberts, W. Verplank, et al., "The Xerox Star: A Retrospective," Computer 22, No. 9, 11-29 (September 1989). 18. M. Familant and M. Detweiler, "Iconic Reference: Evolving Perspectives and an Organising Framework," International Journal of Man-Machine Studies 39, 705-728 (1993). 19. M. Gorbet, Beyond Input Devices: A New Conceptual Framework for the Design of Physical-Digital Objects, M.S. thesis, MIT Media Laboratory, Cambridge, MA (1998). 20. S. McCloud, Understanding Comics: The Invisible Art, Rutgers University Press, London (1993). 21. G. Crampton-Smith, "The Hand That Rocks the Cradle," I.D., 60-65 (May/June 1995). 22. L. Holmquist, J. Redstrom, and P. Ljungstrand, "Token-Based Access to Digital Information," Proceedings of HUC'99 (1999), pp. 234-245. 23. B. Ullmer, Models and Mechanisms for Tangible User Interfaces, M.S. thesis, MIT Media Laboratory, Cambridge, MA (June 1997). 24. E. Arias, H. Eden, and G. Fisher, "Enhancing Communication, Facilitating Shared Understanding, and Creating Better Artifacts by Integrating Physical and Computational Media for Design," Proceedings of DIS'97 (1997), pp. 1-12. 25. K. Schafer, V. Brauer, and W. Bruns, "A New Approach to Human-Computer Interaction--Synchronous Modelling in Real and Virtual Spaces," Proceedings of DIS'97 (1997), pp. 335-344. 26. J. Underkoffler and H. Ishii, "Illuminating Light: An Optical Design Tool with a Luminous-Tangible Interface," Proceedings of CHI'98 (1998), pp. 542-549. 27. H. Shwe, Smarter Play for Smart Toys: The Benefits of Technology-Enhanced Play (1999), at http://www.zowie.com/about/smart.html. 28. M. Fjeld, M. Bichsel, and M. Rauterberg, "BUILD-IT: An Intuitive Design Tool Based on Direct Object Manipulation," Gesture and Sign Language in Human-Computer Interaction, Lecture Notes in Artificial Intelligence, Vol. 1371, Wachsmut and Frohlich, Editors, Springer-Verlag, Berlin (1998), pp. 297-308. 29. I. Siio, "InfoBinder: A Pointing Device for a Virtual Desktop System," Proceedings of the IHCI'95 (1995). 30. K. Hinckley, R. Pausch, J. Goble, and N. Kassel, "Passive Real-World Interface Props for Neurosurgical Visualization," Proceedings of CHI'94 (1994), pp. 452-458. 31. D. Small, Rethinking the Book, Ph.D. thesis, MIT Media Laboratory, Cambridge, MA (1999). 32. P. Wellner, "Interacting with Paper on the DigitalDesk," Communications of the ACM 36, No. 7, 86-96 (July 1993). 33. R. Aish, "Three-Dimensional Input for CAAD Systems," Computer-Aided Design 11, No. 2, 66-70 (March 1979). 34. R. Aish and P. Noakes, "Architecture Without Numbers--CAAD Based on a 3D Modelling System," Computer-Aided Design 16, No. 6, 321-328 (November 1984). 35. J. Frazer, "Three-Dimensional Data Input Devices," Computers/Graphics in the Building Process, Washington (1982). 36. J. Frazer, An Evolutionary Architecture, Architectural Association, London (1994). 37. G. Anagnostou, D. Dewey, and A. Patera, "Geometry-Defining Processors for Engineering Design and Analysis," The Visual Computer 5, 304-315 (1989). 38. M. Gorbet, M. Orth, and H. Ishii, "Triangles: Tangible Interface for Manipulation and Exploration of Digital Information Topography," Proceedings of CHI'98 (1998), pp. 49-56. 39. K. Kramer, Moveable Objects, Mobile Code, M.S. thesis, MIT Media Laboratory, Cambridge, MA (1998). 40. M. Resnick, F. Berg, et al., "Digital Manipulatives: New Toys to Think With," Proceedings of CHI'98 (1998), 281-287. 41. K. Kramer and N. Minar, Stackables: Manipulable Distributed Displays (1997), see http://el.www.media.mit.edu/projects/stackables/. 42. K. Heaton, R. Poor, and A. Wheeler, "Nami," Conference Abstracts and Applications of SIGGRAPH 99, Emerging Technologies (1999), p. 214. 43. D. Anderson, J. Frankel, J. Marks, et al., "Building Virtual Structures with Physical Blocks" (Demo Description), Proceedings of UIST'99 (1999), pp. 71-72. 44. T. McNerney, Tangible Programming Bricks: An Approach to Making Programming Accessible to Everyone, M.S. thesis, MIT Media Laboratory, Cambridge, MA (2000). 45. R. Perlman, "Using Computer Technology to Provide a Creative Learning Environment for Preschool Children," MIT Logo Memo #24, Cambridge, MA (1976). 46. H. Suzuki and H. Kato, "AlgoBlock: A Tangible Programming Language, a Tool for Collaborative Learning," Proceedings of the Fourth European Logo Conference, Athens, Greece (August 1993), pp. 297-303. 47. R. Abrams, Adventures in Tangible Computing: The Work of Interaction Designer Durrell Bishop, in Context, M.A. thesis, Royal College of Art, London (1999). 48. L. Nelson, S. Ichimura, E. Pedersen, and L. Adams, "Palette: A Paper Interface for Giving Presentations," Proceedings of CHI'99 (1999), pp. 354-361. 49. H. Ishii, R. Fletcher, et al., "musicBottles," Abstracts and Applications of SIGGRAPH 99, Emerging Technologies (1999), p. 174. 50. N. Jeremijenko, "Voice Boxes: Memesis," Ars Electronica'96 Festival Catalog, Springer, New York (1996), p. 403. 51. R. Borovoy, Genuine Object-Oriented Programming, M.S. thesis, MIT Media Laboratory, Cambridge, MA (1995). 52. M. Umaschi, "Soft Toys with Computer Hearts: Building Personal Storytelling Environments," CHI'97 Extended Abstracts (1997), pp. 20-21. 53. J. Glos and J. Cassell, "Rosebud: Technological Toys for Storytelling," CHI'97 Extended Abstracts (1997), pp. 359-360. 54. N. Streitz, J. Gei[beta]ler, et al., "i-LAND: An Interactive Landscape for Creativity and Innovation," Proceedings of CHI'99 (1999), pp. 120-127. 55. M. Resnick, "Behavior Construction Kits," Communications of the ACM 36, No. 7, 64-71 (July 1993). 56. P. Frei, V. Su, B. Mikhak, and H. Ishii, "curlybot: Designing a New Class of Computational Toys," Proceedings of CHI'00 (2000), pp. 129-136. 57. A. Dunne and F. Raby, Fields and Thresholds, Presentation at the Doors of Perception 2 (November 1994), see http://www.mediamatic.nl/doors/Doors2/Doors2Fry.html. 58. M. Weiser and J. S. Brown, The Coming Age of Calm Technology (October 5, 1996), see http://www.ubiq.com/hypertext/weiser/acmfuture2endnote.htm. 59. C. Wisneski, H. Ishii, A. Dahley, M. Gorbet, S. Brave, B. Ullmer, and P. Yarin, "Ambient Displays: Turning Architectual Space into an Interface Between People and Digital Information," Proceedings of CoBuild'98 (1998), pp. 22-32. 60. E. Pedersen and T. Sokoler, "AROMA: Abstract Representation of Presence Supporting Mutual Awareness," Proceedings of CHI'97 (1997), pp. 51-58. 61. A. Dahley, C. Wisneski, and H. Ishii, "Water Lamp and Pinwheels: Ambient Projection of Digital Information into Architectural Space," Proceedings of CHI'98 Companion (1998), pp. 269-270. 62. H. Kuzuoka and S. Greenberg, "Mediating Awareness and Communication Through Digital But Physical Surrogates," Proceedings of CHI'99 Companion (1999), pp. 11-12. 63. C. Wisneski, The Design of Personal Ambient Displays, M.S. thesis, MIT Media Laboratory, Cambridge, MA (1999). 64. J. Heiner, S. Hudson, and K. Tanaka, "The Information Percolator: Ambient Information Display in a Decorative Object," Proceedings of UIST'99 (1999), pp. 141-148. 65. S. Brave, H. Ishii, and A. Dahley, "Tangible Interfaces for Remote Collaboration and Communication," Proceedings of CSCW '98 (1998), pp. 169-178. 66. W. Mackay and D. Pagani, "Video Mosaic: Laying Out Time in a Physical Space," Proceedings of Multimedia'94 (1994), pp. 165-172. 67. T. Arai, K. Machii, and S. Kuzunuki, "Retrieving Electronic Documents with Real-World Objects on the InteractiveDESK," Proceedings of UIST'95 (1995), pp. 37-38. 68. L. Stifelman, "Augmenting Real-World Objects: A Paper-Based Audio Notebook," Proceedings of CHI'96 Companion (1996), pp. 199-200. 69. H. Ishii, C. Wisneski, J. Orbanes, B. Chun, and J. Paradiso, "PingPongPlus: Design of an Athletic-Tangible Interface for Computer-Supported Cooperative Play," Proceedings of CHI'99 (1999), pp. 394-401. 70. P. Yarin and H. Ishii, "TouchCounters: Designing Interactive Electronic Labels for Physical Containers," Proceedings of CHI'99 (1999), pp. 362-368. 71. R. Want, K. Fishkin, A. Gujar, and B. Harrison, "Bridging Physical and Virtual Worlds with Electronic Tags," Proceedings of CHI'99 (1999), pp. 370-377. 72. J. Rekimoto and M. Saitoh, "Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments," Proceedings of CHI'99 (1999), pp. 378-385. 73. B. Ullmer, Cognitive Roles of Physical Artifacts in Non-Athletic Games, unpublished manuscript (2000). 74. M. Csikszentmihalyi and E. Rochberg-Halton, The Meaning of Things: Domestic Symbols and the Self, Cambridge University Press, Cambridge, UK (1981). 75. J. Gonzalez, "Autotopographies," Prosthetic Territories: Politics and Hypertechnologies, Westview Press, Boulder, CO, pp. 133-150. 76. J. Zhang, "The Nature of External Representations in Problem Solving," Cognitive Science 21, No. 2, 179-217 (1997). 77. J. Zhang and D. Norman, "Representations in Distributed Cognitive Tasks," Cognitive Science 18, 87-122 (1994). 78. J. Gibson, The Ecological Approach to Visual Perception, Erlbaum Associates, New York (1979). 79. D. Norman, Things That Make Us Smart, Addison-Wesley Publishing Co., Reading, MA (1993). 80. E. Hutchins, Cognition in the Wild, MIT Press, Cambridge, MA (1995). 81. M. Scaife and Y. Rogers, "External Cognition: How Do Graphical Representations Work?" International Journal of Human-Computer Studies 45, No. 2, 185-213 (1996). 82. N. Eilan, R. McCarthy, and B. Brewer, Spatial Representation, Blackwell, Oxford, UK (1993). 83. D. Kirsh, "The Intelligent Use of Space," Artificial Intelligence (1995). 84. Y. Guiard, "Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model," The Journal of Motor Behavior 19, No. 4, 486-517 (1987). 85. K. Hinckley et al., "Two-Handed Virtual Manipulation," ACM TOCHI'98 (1998). 86. M. Krampen, "Objects," Encyclopedic Dictionary of Semiotics, Vol. 1, T. Sebeok, Editor, Mouton de Gruyter, New York (1986), pp. 635-639. 87. J. McCarthy, An Introduction to Theoretical Kinematics, MIT Press, Cambridge, MA (1990). 88. Semantic Visions in Design, S. Vihma, Editor, UIAH, Helsinki (1990). 89. B. Shneiderman, "Direct Manipulation: A Step Beyond Programming Languages," Computer 16, 57-69 (1983). 90. D. Frohlich, "Direct Manipulation and Other Lessons," Handbook of Human-Computer Interaction, 2e, Ch. 21, Elsevier, Amsterdam (1997). 91. D. Gentner and J. Nielsen, "The Anti-Mac Interface," Communications of the ACM 39, No. 8, 70-82 (August 1996). 92. R. Baeza-Yates, Visual Programming (1998), formerly at http://www-lsi.upc.es/~rbaeza/sem/vp/vp.html. (Visited August 19, 1998.) 93. J. Larkin and H. Simon, "Why a Diagram Is (Sometimes) Worth Ten Thousand Words," Cognitive Science 11, 65-99 (1987). 94. M. Petre, "Why Looking Isn't Always Seeing: Readership Skills and Graphical Programming," Communications of the ACM 38, 33-44 (June 1995). 95. S. Feiner, B. MacIntyre, and D. Seligmann, "Knowledge-Based Augmented Reality," Communications of the ACM 36, No. 7, 52-62 (July 1993). 96. R. Azuma, "A Survey of Augmented Reality," Presence 6, No. 4, 355-385 (August 1997). 97. W. Mackay, "Augmented Reality: Linking Real and Virtual Worlds," (published keynote address), Proceedings of ACM AVI'98 (1998). 98. P. Milgram and F. Kishino, "A Taxonomy of Mixed Reality Displays," IEICE Transactions on Information Systems, E77-D(12) (December 1994). 99. T. Starner, S. Mann, B. Rhodes, J. Levine, J. Healey, D. Kirsch, R. Picard, and A. Pentland, "Augmented Reality Through Wearable Computing," Presence 6, No. 4 (Winter 1997). 100. M. Weiser, "The Computer for the 21st Century," Scientific American 265, No. 3, 94-104 (1991). 101. J. R. Cooperstock, K. Tanikoshi, G. Beirne, T. Narine, and W. Buxton, "Evaluation of a Reactive Environment," Proceedings of CHI'95 (1995), pp. 170-177. 102. K. Fishkin, T. Moran, and B. Harrison, "Embodied User Interfaces: Towards Invisible User Interfaces," Proceedings of EHCI'98 (1998). 103. F. Halasz, "Reflections on Notecards: Seven Issues for the Next Generation of Hypermedia Systems," Communications of the ACM 31, No. 7, 836-852 (July 1988). 104. F. Halasz, " 'Seven Issues' Revisited," keynote address. Hypertext'91 Conference (1991), see http://www.parc.xerox.com/spl/projects/halasz-keynote/ and http://www.csdl.tamu.edu/~leggett/halasz.html (video). Accepted for publication April 26, 2000. Biographical sketches of authors Brygg Ullmer MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: ullmer@media.mit.edu). Mr. Ullmer is a Ph.D. candidate and IBM Media Lab Fellow in the Tangible Media Group of the MIT Media Laboratory. After graduating from the South Carolina Governor's School for Science and Mathematics in 1991, he received a B.S. in computer engineering from the University of Illinois, Urbana-Champaign in 1994, and a master's degree from the MIT Media Laboratory in 1997. He has held summer positions with the University of South Carolina Industrial Mathematics Initiative (1990-1992), Interval Research Corporation (1993-1995), and Sony CSL (2000). His current research interests include tangible and graphical user interfaces for distributed information, network and computing infrastructure, and biological systems, as well as rapid physical and functional prototyping. His publications and additional information are on line at http://tangible.media.mit.edu/~ullmer/. Hiroshi Ishii MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: ishii@media.mit.edu). Dr. Ishii is an associate professor at the MIT Media Laboratory, where he founded and directs the Tangible Media Group. His research focuses on the design of seamless interfaces between humans, digital information, and the physical environment. Prior to coming to MIT, he led a CSCW research group at the NTT Human Interface Laboratories from 1988-1994, where his team invented TeamWorkStation and ClearBoard. In 1993 and 1994, he was a visiting assistant professor at the University of Toronto, Canada. Dr. Ishii received a B.E. degree in electronic engineering in 1978, and the M.E. (1980) and Ph.D. (1992) degrees in computer engineering, all from Hokkaido University. He is active in the research fields of human-computer interaction (HCI) and computer-supported cooperative work (CSCW), and has served as a program committee member of many international conferences including ACM CHI, CSCW, UIST, SIGGRAPH, Multimedia, and ECSCW. Dr. Ishii is currently an associate editor of ACM TOCHI (Transactions on Computer Human Interactions). His publications and additional information are on line at http://tangible.media.mit.edu/~ishii/.