0018-8670/00/$5.00 (C) 2000 IBM Sensor systems for interactive surfaces by J. A. Paradiso, K. Hsiao, J. Strickon, J. Lifton, and A. Adler This paper describes four different systems that we have developed for capturing various manners of gesture near interactive surfaces. The first is a low-cost scanning laser rangefinder adapted to accurately track the position of bare hands in a plane just above a large projection display. The second is an acoustic system that detects the position of taps on a large, continuous surface (such as a table, wall, or window) by measuring the differential time-of-arrival of the acoustic shock impulse at several discrete locations. The third is a sensate carpet that uses a grid of piezoelectric wire to measure the dynamic location and pressure of footfalls. The fourth is a swept radio frequency (RF) tag reader that measures the height, approximate location, and other properties (orientation or a control variable like pressure) of objects containing passive, magnetically coupled resonant tags, and updates the continuous parameters of all tagged objects at 30 Hz. In addition to discussing the technologies and surveying different approaches, sample applications are given for each system. Large flat surfaces, such as walls, floors, tables, or windows, are common structures in everyday life, usually dictated by practical human necessity or driven by general architectural aesthetics. At present, these surfaces are mainly passive and, where appropriate, are used to display decorative items such as paintings, photographs, and rugs. Although different projects and products centered on the theme of "home automation"[1] have inspired various interactive displays, these are usually small or moderate-sized discrete devices, such as touch screens embedded into walls or tables. It is still unusual to see large portions of the walls, floors, or windows themselves used directly as interactive interfaces, except perhaps in niche applications such as those used for teleconferencing.[2] Other interactive "smart room" approaches look at sensing full three-dimensional spaces, for example with computer vision techniques,[3] and avoid concentrating expressly on the often more deliberate and precise interactions that can be expressed at the surface itself. New technologies, however, will enable such architectural surfaces to become sensate, following trends and concepts in "smart skins" that have redefined structural control and aerospace research over the last decade.[4] This paper is an overview of such sensor systems, emphasizing recent work by the MIT Media Laboratory's Responsive Environments Group on new user interface devices for interactive surfaces and large-scale public installations. In particular, we describe the technology and demonstration applications behind four systems that we have developed: an interactive wall, which tracks hand positions with a low-cost scanning laser rangefinder, a smart window that locates finger taps using differential acoustic time-of-arrival, a carpet that measures the position and pressure of feet with a grid of piezoelectric wire, and a tag reader that identifies and tracks the state of nearby objects embedded with magnetically coupled resonators. Approaches to "smart walls" Most of the commercial products that have been developed to track position across a large, responsive surface have been aimed at the digitizing tablet and "smart whiteboard" markets, where the handwriting from a set of coded pens and drawing objects is digitally captured. While many of these systems require contact or pressure to be applied against a sensitive surface[5] and act as a large touch screen[6] or trackpad,[7] others detect the position of objects just above the board or tablet. The bulk of these devices are made to work with opaque surfaces, because the sensing technology (usually nontransparent) is buried beneath the active area. One interesting example of a recent, noncommercial sensing surface is the pixilated capacitive matrix devised by Post and collaborators at the MIT Media Lab for their sensor table[8] developed for an installation at the Museum of Modern Art in New York. Although this technique can detect and track nearby bare hands through their capacitive loading, it does not scale easily in large areas and is generally opaque; therefore it is not suited to rear-projection applications. For smaller surfaces, transparent conductors such as ITO (indium-tin oxide) or conductive plastic can be used as in capacitive touchscreens,[6] but extending such fine sampling or pixilated concepts to very large displays becomes complicated and expensive with existing technologies. Most tracking systems for translucent or very large wallboards are the "squinting" type that look across from the edges of the display. Although inexpensive devices are coming on the market that use acoustic time-of-flight to a fixed receiver from active sonar pingers embedded in pens,[9] several employ optical sensing, which enables simple, passive reflecting targets on the drawing objects to be easily detected in a sensitive plane defined by a scanning fan-collimated light source, such as generated by a scanned diode laser. The best-known example of this is the SoftBoard** by Microfield Graphics,[10] where a pair of scanning lasers emanate from the two top corners of the board, identifying and tracking coded targets on pens and other objects approaching the whiteboard and intersecting the scanned plane. These sensors are unable to detect distance, thus planar position is determined by triangulating the two angular measurements. To avoid ambiguities in this triangulation, these systems generally allow only one object to be tracked at a time. Although the SoftBoard requires coded targets, earlier research systems[11] used a similar arrangement to track single fingers and bare hands. Light-Curtains, which use dense arrays of infrared light-emitting diodes (IR LEDs) that face corresponding receivers lining the perimeter of the screen, are commercially available[12] and can track multiple hands, but because of poor scalability, become expensive for large displays. A variant on this approach is the Intrepid touch system,[13] which uses an array of IR LEDs across the top of the display and two linear CCD arrays at the corners that look for reflections from the hands. Large screens can be expensive or suffer from illumination difficulties using this technique. Some smart wall hand-tracking systems use computer vision. The most straightforward versions of these use multiple cameras, squinting along the horizontal and vertical coordinates and triangulating.[14] Although this approach gives much information (potentially enabling hand gesture to be determined[15]), it involves a considerable amount of sometimes fragile video processing to detect the hand, reject background light and clutter, and solve the image-correspondence problem for multiple hands. Other video approaches are of the nonsquinting variety. The most common one that looks from the front of the screen is the standard "chromakey" technique,[16] in which the silhouette of the body is detected against a plain blue or bright screen, whereupon the hands are identified and tracked when not in front of the torso. This is familiar to many who watch weather broadcasts. Although the newscaster only gestures in front of a blue screen in the studio, the screen is replaced by the weather map in the broadcast video. For increased precision, lower ambiguity, higher speed, and the ability to work with variable background light or an image-bearing screen, many classic body-tracking systems[17] have exploited active targets made from modulated IR LEDs that must be placed on or carried by the user. Another system, the "Holowall,"[18] looks from the back of the screen, which is illuminated from behind by a bright IR source positioned next to an IR camera. Although considerable IR light penetrates the screen, the user in front is unable to see this. The IR camera captures reflections (propagating back through the screen) from the user's hands as they approach the surface. Image processing on the resulting frames is used to detect the hands. This system can unambiguously track all hands without correspondence or occlusion difficulties. It does, however, require real-time image processing, has difficulties with clutter and background IR illumination (e.g., from tungsten or outdoor lighting), needs an IR-translucent screen, and does not define a clean sensing plane at the screen surface, because the user's hands or body are detected at varying locations from the screen, depending on the local illuminations and their albedo. Computer vision has similarly been used to identify and track objects above interactive tables by recognizing a graphical code with which the object is labeled. These systems use a video camera that either looks down at the objects from above the tabletop[19] or looks up at the objects from underneath through a semi-transparent tabletop.[20] This technique requires the objects to be unambiguously labeled and needs clear lines of sight from camera to objects. A previous interactive surface arrangement designed and built by some of the authors used transmit-mode capacitive sensing.[21] This device, the Gesture Wall, injected a 50-100 kHz signal into the body of the user through an electrode on the floor (different shoe impedances were compensated by scaling the drive voltage with a servo loop that calibrated each new user[22]). The strengths of this signal, as capacitively received at electrodes placed in the four corners of the display, were used to track the position of a hand as it moved around the display surface. Although this system, diagrammed and depicted in Figure 1, was very sensitive to gesture, any absolute tracking capability for even a moderately large display required fairly stiff postural constraints on the part of the user (one hand forward and body back, as portrayed in Figure 1), since the entire body radiates the transmit signal, not just the hand to be tracked. An inexpensive scanning laser rangefinder Although many solutions exist for particular applications, as outlined above, there is no clean, simple, inexpensive, and general means of robustly tracking bare hands near very large display surfaces. Our solution to this problem, depicted in Figure 2, is to place a scanning laser rangefinder at one corner of the display to determine the polar (r, [phi]) coordinates of hands in a clearly defined plane above the projection surface. This would ideally be a compact device, enabling a simple retrofit to make any flat surface interactive. One unit, in a single corner of the screen, is able to scan the entire display surface; because it produces two coordinates simultaneously, there is no correspondence problem with multiple hands (there are still occultation issues, however, as discussed in the next section). Also, since the photodetector signal is synchronously received[23] relative to the outgoing laser, the system intrinsically rejects all background light (provided the detector is not saturated), seeing only the reflected illumination coming from the modulated laser source. Such laser rangefinders are commercially available devices, used for survey,[24] robotic,[25] and military[26] applications. A quick investigation of the current market, however, indicated that scanning rangefinders that fit even our modest requirements (approximately 1 centimeter [cm] resolution across about 4 meters at a 20-30 Hz update rate) are still considerably expensive. The least expensive devices cost several thousands of dollars per unit. As a result, we chose to pursue our own low-cost design, adapted to the task of tracking hands near interactive walls. Our system was introduced in Reference 27 and detailed in Reference 28. Triangulation rangefinders, with a displaced source and imaging receiver,[29] can provide very high depth resolution over a limited dynamic range, and thus are often used in three-dimensional object scanning.[30] Although our first device[31] was based on triangulation, it was unable to attain sufficient resolution across the entire surface of a large display because of the asymptotic perspective characteristic of the triangulation output[29] (these systems are generally tuned for optimal performance across a short range span). Therefore, we elected to pursue a CW (continuous-wave) phase-measuring system[32] in our design, which is shown schematically in Figure 3. The actual working scan head is diagrammed and shown in Figure 4. This unit includes all scanning and high frequency electronics. Even though the baseband and microcomputer electronics can also be accommodated at the head, our current devices locate them in a remote chassis. Although an IR laser would work very well (silicon photodiodes are, if anything, more sensitive to infrared), we opted to use a standard 5 milliwatt red laser diode as the light source here. Not only did this make the optical alignment much simpler, it also gave good intrinsic feedback to the user, who knows that the hand is being tracked when it is seen to intersect the visible red laser scan line. The direct current (DC) supply for the laser was 100 percent chopped by a single, fast transistor driven at a 25 MHz modulation frequency. This provides 6 meters of range before phase-wrap (this is half of the modulation wavelength, as the light makes a round-trip from the scanner and back). As seen in Figure 4, the laser was reflected from an adjustable mirror mounted just below the lens tube and aligned to point into the field of view of a photodiode "camera" (containing the photodiode, lens, and front-end amplifier). The laser and camera ensemble were then aimed at a rotating, 4-sided mirror (2 inches X 1 1/4 inch per side), which scanned both the laser and camera's field of view together across a 90[degree] span. In order to simplify the electronics chain, we used a 1-inch diameter, 45-millimeter (mm) focal-length lens focussed onto a 1.5 mm[sup]2[/sup] avalanche photodiode (APD)[33] to detect the laser light. The photodiode was biased by a ~~425-volt supply that provided maximal intrinsic gain. This was followed by a pair of monolithic microwave amplifiers, giving 25 decibels (dB) of broadband gain, then a wideband automatic-gain-control (AGC) amplifier[34] that compensated for varying complexion and reflection characteristics and was able to provide up to an additional 80 dB of gain. The bandwidth was subsequently narrowed by a 30 dB tuned amplifier[35] and adjusted to the modulation frequency (the output of this amplifier provided the AGC feedback). Although a standard rangefinder would provide additional gain at an intermediate frequency (IF),[36] the avalanche gain from the APD (and the relatively short range over which we used this device) allowed us to demodulate directly to baseband using a pair of wideband analog multipliers to obtain a quadrature pair. The quadrature reference signals came from a chain of flip-flops driven by a crystal oscillator. A first-order low-pass filter, with its breakpoint set to ~~4 kHz, selected the basebanded signals while rejecting out-of-band noise and leaving sufficient bandwidth to accommodate a 25-30 Hz scan (detecting hand-sized objects up to 4 meters away). These signals were additionally conditioned to span between 0 and 5 volts, then used as inputs to the analog/digital converter of the embedded microcomputer. The basebanded analog signals were also used as a reference for a laser-attenuator servo system that decreased the laser drive voltage if the analog signals approached the supply rails, preventing saturation for extremely bright reflections when the hand was very close to the scanner and allowing linear operation everywhere. The angular scan was synchronized in our original prototype by a phototransistor that was hit by the laser at the beginning of the sweep. In our subsequent devices, this was decoupled from the scanning laser (to simplify alignment and remove any dependence on extraneous light), and a separate shaft encoder was used. This synchronization signal is also used by a hardware safety failsafe, which shuts the laser power supply down if the mirror is not turning (at a 20-30 Hz scan rate, our system is safe for human eyes[37]). The basebanded signals produced by our scanner, as seen in Figure 5 for a pair of hands, are extremely simple to interpret. The current devices assume that the laser is scanned against a matte-black baffle, as diagrammed in Figure 2. Upon startup, the rangefinder samples its baseline across the entire scan and subsequently subtracts the scan amplitudes point-by-point from this reference to suppress residual reflection from the baffle. This allows the hands to be detected as amplitude peaks, using a simple static threshold on the Manhattan metric (i.e., sum of absolute values) of the in-phase (I) and quadrature (Q) signals. Although the hands could be detected by discriminating their associated peaks via changes in range instead of amplitude, removing the need for the baffle, this would involve more embedded processing and lead to potentially more complicated calibration issues, and hence was not pursued in our current device. At the conclusion of each scan, the microcomputer serially transmits the integrated I and Q amplitudes of each peak, the peak widths, and a set of peak times derived by taking the mean between the clock values at which the peak rose above and fell below the threshold (the clock is reset by the scan-start phototransistor signal). The initial prototype[28] worked with a Microchip Technology PIC**16C73 microcontroller,[38] but our subsequent designs replaced it with an Hitachi SH-1.[39] The SH-1 offers a 10-bit analog/digital (A/D) converter that runs roughly five times faster than that of the PIC. Our next model[40] will be based around the Analog Devices ADuC812 microconverter,[41] which offers a 12-bit ADC running at equivalent speed around an industry-standard 8051 core. The microcomputer produces a serial data stream after each scan (e.g., at 30 Hz) that is read by a connected host computer. Although there is no need for any subsequent signal processing in the classical sense, the rangefinder does off-load basic calibration to the personal computer, enabling us to keep the microcomputer minimal (its primary job is to concentrate the data stream into four relevant parameters per peak, as described above). The forward data calibration involves taking the arctangent of the I and Q parameters for each peak (finding the phase change of the returning laser light), then running this result together with the average peak time through a 15-coefficient, third-order polynomial. These coefficients are derived when the rangefinder is first set up, and relate the rangefinder coordinates (i.e., the arctangent and peak time) to screen graphics coordinates. They are determined by a linear least squares fit to a database derived from the rangefinder outputs produced when placing the user's hand on a projected icon that moves across a uniform 25-point grid (this simple calibration procedure is completed in a few minutes). Figure 6A shows the calibrated rangefinder tracking a hand across a 2 X 2 meter rear-projection surface; the reconstructed rangefinder coordinates (open circles) are seen to closely approximate the screen reference points (crosses) where the hand was actually placed. Figure 6B shows continuous data from the rangefinder system in operation, with the hand "drawing" in the air above the same screen. Rangefinder performance and applications. We have built several prototype rangefinders for hand-tracking applications, dating from our original unit in the fall of 1997[28] through the recent model shown in Figure 4. They have emerged from the laboratory, having been used in several Media Lab and outside public installations, as portrayed in Figure 7. The parts cost of this device is well under $500, strongly dominated by the APD. An improved electronics design may be able to replace the APD and its high-voltage power supply with an inexpensive PIN photodiode. Once the optics are properly aligned, the rangefinder's outputs are stable and after calibration are mapped to within a couple of inches of the hand, as shown in Figure 6, across a large (e.g., 6 X 8 foot) display. The present device gives a point-to-point noise resolution (standard deviation) of roughly [sigma] = 1.4 cm across the screen, dominated for the most part by wobble in our homemade rotating mirror assembly. This noise is easily attenuated by a four-tap FIR (finite impulse response) filter running on the host computer, which introduces a small lag but results in very smooth and stable coordinates, enabling the user to write or draw quite legibly with bare hands across a large screen, as shown in Figure 6B. To simplify application development, we have written a Windows** mouse driver for this device, enabling the hand to move the cursor. Although the polynomial calibration was usually accurate enough for open-loop interaction, we generally draw a cursor of some sort below the position of a detected hand to give direct user feedback. Aside from a few in-house demonstrations in which we used the prototype as a simple hand-controlled mouse running a common drawing program, all applications have been multimedia in nature, where the general public can move the projected graphics about with their hands while also exploring a connected musical mapping. These have been very successful. People seem to enjoy walking up to a large projection wall and interacting with the graphics and music. The scale of the presentation makes it a public spectacle, while the close-up nature of the interaction still feels somewhat intimate, forming an interesting fusion of two commonplace displays now gracing, for example, shopping malls--the touch-screen kiosk and the video wall. Figure 7A shows the first environment written to test our device with these ideas in action. The graphics routine plotted a red and green rotating square at the location of each detected hand, and changed the background objects as the hands were moved about. Drum loops would start when at least one hand was detected and change with their position. A bass-lead arpeggio would begin when more than one hand was introduced (with tonal range dependent on their mutual distance). Although quite simple, it was a very popular installation at several 1997 Media Lab events. Our next application, a port of Rice's mouse-driven Stretchable Music[42] program, was considerably more complex. Here, users could grab and stretch objects that sequentially appeared on the screen, each of which exhibited different musical behavior in accordance with the amount of stretch. If more than one hand was detected, the additional hands would draw musical "sparkles," i.e., twinkling points centered on the hand positions that made soft, ethereal sounds. This installation, shown in Figure 7B, was run for a week at SIGGRAPH 98,[43,44] where it was likewise very popular with conference visitors. The most recent application was a port of Donath's Visual Who graphical database exploration program,[45] shown in Figure 7C as it was being demonstrated at Opera Totale 5[46] in Mestre, Italy, during November of 1999. Here, users could insert graphical anchors into a complicated database, and move them around on the screen, seeing representations of related information attract and drift to the anchor being moved, depending on their attachment (e.g., degree of relationship). An interactive music system was also used, with different themes and effects tied to the different anchors and the manner in which they were manipulated (velocity, position, etc.). This system was also installed at SIGGRAPH 2000,[47] where users were able to explore various aspects of the conference databases. Although the rangefinder system is able to track multiple hands, it is subject to occlusion, where two hands at similar [phi] line up with the laser beam, hence one hand "shadows" the other. For many applications, this can be addressed in software (i.e., by introducing a tracking filter[15] or "inertia" to the plotted trackpoints, or forcing them to snap to either side of the shadowing hand), or rigorously by adding a second scanner (since each directly produces both [r, [phi]], there is no correspondence problem; all points are unambiguous). The examples of Figure 7 have not explored multihanded interactions deeply, since they were mainly ported as augmented mouse applications. This is a topic for additional investigation with this interface. Acoustic tap tracking The rangefinder system is very well suited to continuous tracking of hands, as explored in the application examples described above. It is less adapted, however, to notions like the "click" of a mouse, a commonplace action in today's standard graphical user interfaces (GUIs). There are several clean gestures by which the notion of a "click" can be adopted; for example, moving one's hand quickly in and out of the scan plane or pulsing the nontracked other hand rapidly in and out of the beam, etc. The most direct method of clicking, however, would be to just tap the screen. With only one hand being tracked, this is straightforward; a simple acoustic transducer affixed to the screen could detect the tap and announce a "click" event. With multiple hands, however, this is more of a problem, as the current planar rangefinder provides no simple way to know which hand made the tap. To resolve such issues and explore a different kind of smart surface interface, we have developed the system shown in Figure 8. This is a "tap" tracker that determines the position of a knock on a continuous surface from the differential time-of-arrival of acoustic energy at multiple locations, in this case, four transducers at the corners. The first version of this device came from a collaboration between the lead author and Hiroshi Ishii's Tangible Media Group for their PingPongPlus (PP+) installation[48] exhibited at SIGGRAPH 98.[49] Here, the impact position of a ping-pong ball on a ping-pong table was determined in real time through differential time-of-flight between the signals recorded by four microphones adhered to the table's underside near the corners. The speed of the impact shock through the table was sufficiently slow (about twice the speed of sound in air) for software timing to be performed in a PIC microcontroller using firmware developed by Wisneski and collaborators.[50] Likewise, the acoustic characteristics of ball-on-table impacts had a sharp leading edge and were fairly uniform, enabling sufficiently accurate (circa inch-scale resolution) performance using constant-threshold discriminators on the absolute value of the input signals to determine the first acoustic arrival at each microphone, after which the signals are contaminated through multipath and reflections. Three separated pickups are sufficient to locate the impact in a plane. The fourth pickup adds redundancy for consistency-checking and better resolution. Although the inverse mapping from differential timings (i.e., differential range, since the acoustic velocity is constant) to impact coordinates is nonlinear[51] (involving the closest intersection of three hyperbolas for the four microphones), we used the approximation of a linear least-squares fit for the PP+ installation.[50] This introduced some limited distortion at the perimeter of the table, but was entirely adequate for the intended interactive media applications and it executed very quickly. Because of the uniform and predictable characteristics of the hard impacts and the good acoustic propagation characteristics of the supported wood table, this approach is well suited to the ping-pong application. Because this is a very small niche, we have been exploring the utility of applying such a technique to more general scenarios, such as locating the positions of fingers knocking on a pane of glass. This would open a set of interesting applications such as easily retrofitting common store windows, for example, into simple interactive displays. This situation is more complex, however, as the finger tap excitation can now change considerably from one hit to the next. Variations will occur depending on how the glass is struck, the type of glass used, and how the glass is supported. To approach this problem, we used contact pickups made of polyvinylidene fluoride (PVDF) piezoelectric foil[52] instead of the electret microphones used for the table, placing them near the perimeter of a glass pane, as indicated in Figure 8. These pickups were insensitive to ambient sounds in the air, but produced excellent signals when the glass was hit. Figure 9 shows the response from a pair of these pickups, spaced approximately 3 feet apart and bonded with common adhesive to a 1-cm thick, 4 feet X 8 feet shatterproof glass window (the kind used to divide rooms, solidly supported by rubber anchors along its entire perimeter), to a series of ten knuckle taps running from right to left in uniform steps. Since the trigger reference is on the left pickup, it is always seen to occur at the same point in time, and the maximum plotted time difference is twice the propagation interval between sensors. As expected, the right pickup signal shows consistently increasing delay, indicating that the wavefront must propagate further through the glass to reach the transducer. Note that the observed propagation mode of these low-frequency, knock-instigated impulses moves quite slowly, at roughly 450 msec (roughly a factor of 10 below the speed of sound in glass), indicating that we are probably observing flexural waves.[53] Things are different for sharper impacts (e.g., when hitting the glass with metal), which are seen to propagate at around 3500 msec, indicating excitation of faster acoustic modes. Figure 10 shows the delay of the right signal relative to the left, as derived from a simple constant-threshold discriminator. The linear characteristic is clear (the straight line is a best fit to the points), and this simple fit resolves these data to better than 2 inches. Most of the inaccuracy comes from the leftmost hits, where this windowpane was not as solidly supported on its frame (the actual window and sensor layout used are shown in Figure 11). To track taps more reliably, however, we have found that using a simple static threshold is generally not adequate. Amplitude dependence is one factor, because the leading edge for a knuckle-tap is not sufficiently abrupt. To account for this, we have explored the use of constant-fraction discriminator circuits.[54] Most important, the characteristics of the first arrival can vary widely from transducer to transducer and impact to impact. A significant problem can be posed by the variable amount of low-amplitude, higher-frequency, dispersive deflection that often arrives before the main wavefront, as can be seen in Figure 9. Likewise, sharp impacts (e.g., snapping a metal ring against the glass instead of one's knuckle) excite rapidly moving modes. To adequately address these ambiguities, we now use a microcontroller card based on a Hitachi SH-1[55] to continuously digitize the analog signals from all four transducers into 10 bits at over 10 kHz. This enables a more detailed and robust embedded analysis to look at other waveform features (e.g., peak amplitudes and waveform shape) for each tap. The microcontroller continuously samples the signals from each transducer into a rotating buffer. Upon detecting a transducer signal above a noise threshold, a "knock" event is declared, and 10 millisecond (ms) worth of data are stored from all four inputs (including 3 ms of data before the trigger occurred). This buffer is then scanned for every significant peak in the absolute-value waveform produced by each transducer, and descriptive parameters (e.g., peak height, width, and mean arrival time relative to the initial trigger) are extracted for each peak (including any small peaks arriving earlier, as discussed above). These parameters are sent over a serial connection, together with a count of the number of zero-crossings across the data acquisition interval (too many zero crossings indicate a sharp hit with different timing, as mentioned previously). A connected personal computer then processes the timing determined for each first peak by a second- order polynomial that was obtained from a linear least-squares fit to a set of calibration points (as done with the original ping-pong table system) to produce an estimate of the impact location in Cartesian coordinates. In addition to increasing the reliability of the results, the use of a microcontroller readily enables more channels of gestural input (e.g., measuring the strike intensity and classifying the type of strike). We also extract an estimate of accuracy or validity by cross-checking the detected waveform characteristics from the different sensors and examining the differences between the four position estimates obtained from the four different sensor triplets (since there are four pickups, we have one redundant degree of freedom). Figure 11 shows this system being demonstrated on our test window. The PVDF strips can be seen in silhouette at the edges of the sensitive volume, together with rear-projected circles showing estimates of the impact position and accuracy for several successive knocks. The radius of the large circles represents the position resolution as inferred from the redundant sensor information. In our tests, this system resolved knocks in both axes to [sigma] = 1.2 inches across the 3 feet X 2.5 feet instrumented region of the glass. As noted in Figure 11, the sensor strips are very small and do not significantly block the window's view. Sensate floors Although the laser and tap trackers described earlier could conceivably work on a floor, they are inconvenient for several reasons. The laser system would have difficulty detecting dark-colored footwear at any appreciable distance and suffer considerably from occlusion in even a small crowd. The tap tracker would have to detect different kinds of impacts from different kinds of shoes across large distances on often nonhomogeneous and discretized pieces of flooring randomly loaded with furniture and people. In addition, many applications of sensate flooring also need pressure information, which is not easily derived from these techniques. Most sensor floor designs use large-area force-sensitive resistors[56-58] that respond to foot pressure. These can be fragile, however, and difficult to transport for mobile installations. Others use optical techniques, for instance by illuminating translucent floorboards with IR from below and inferring range from detected intensity reflecting off the foot.[59] While this can also measure the foot when it is above the floor, it requires calibration for variations in sole reflectance and floor transparency (which can change with time), and does not directly provide pressure signals. Other designs work through electric fields, either measuring the change in capacitance between two plates sandwiching an insulator that compresses with pressure,[60] directly measuring the loading of a capacitive electrode by the body when a foot is nearby,[61] or measuring the coupling of an external signal sent from the shoe into a receptor electrode on the floor.[62] Although these methods have potential, they likewise are prone to reliability, complication, and calibration considerations. The solution that we have pursued is shown in Figure 12. A grid of shielded cable, similar to standard coaxial wire but with a piezoelectric copolymer used for the inner insulation,[63] is placed on the floor. The wires are spaced at a roughly 4-inch pitch, so that at least one will always be under the shoe of an occupant. Successful operation is best ensured by placing a force-interpolating sheet of, for example, thin plastic above the wire grid. The piezoelectric material produces a voltage (in the 1-5 volt range if terminated with a high impedance) when the wire is stepped on, with amplitude proportional to the intensity of the dynamic foot pressure. The wires do not provide steady-state pressure due to the AC-coupled nature of the sensing medium. They respond to changes in foot pressure, and thus measure footstep dynamics, which are adequate for many applications. The piezoelectric wire is very rugged, requires minimal electronics, and is very easy to transport and embed into many types of flooring. Our current setup uses a grid of 16 X 32 wires at a 4-inch pitch below a 6 X 10 foot trapezoidal segment of carpet (Figure 12). Figure 13 diagrams the conditioning and scanning electronics that we have built to digitize the information from an array of up to 64 wires. The signal from each wire is compressed in a piecewise-linear front-end buffer that provides higher gain for smaller signals. This produces good response to soft footsteps while still yielding some proportional information for hard footfalls. The output of this buffer is then conditioned by a peak detector, giving a fast-attack, slow-decay signal that guarantees accurate sampling by an analog multiplexer that scans all 64 signals within 17 milliseconds. After a final stage of adjustable common gain, the signals are digitized into 8 bits by a 68HC11 microprocessor,[64] which sends the wire number and detected pressure to an attached host computer when a new amplitude peak is detected on any wire. Because each wire in the grid extends over the full length of the carpet, location ambiguities can be introduced for multiple individuals. In this case, their position can be estimated through simple clustering algorithms and filters[65] that group together consistent x and y wire hits that occur within one or two scan intervals (otherwise, a pixilated arrangement must be adopted that uses shorter sections of wire). Figure 14 shows 10 seconds of actual data taken from the carpet system responding to a person walking normally across a carpet diagonal, then heavily stomping back. In the bottom plot, circles are drawn for each piece of transmitted data, with the radius proportional to the detected strike force. The wire number (hence position) is plotted on the vertical axis. The data at left (corresponding to the "normal" walk) show the dynamics of the footfalls as the shoes move across the wires. Much higher pressures are seen on the hard returning steps at right; in addition they are more tightly clustered in time, as the "stomp" was essentially instantaneous, in contrast to the rolling nature of standard steps. The spread across nearby wires occurred here because the heavy stomps vibrated the suspended floor tiles on which the wire grid was placed, distributing the stomp energy over a larger area. The lower plot shows the same data discretized into footsteps and folded into three dimensions. The radius of the spheres represents the step intensity (integrated across all wires), the horizontal coordinates map to carpet position and the vertical coordinate is the elapsed time. We have used this system in several public installations,[22,66,67] most of which were paired with a simple Doppler radar system[22,68] to also detect upper body motion, providing a highly immersive sensor environment, where any movement produces an appropriate musical response. Figure 15 shows this installation as set up for demonstration in a Media Lab elevator lobby. Other groups at the Media Lab have used this carpet for different touring installations, for instance, the Interactive Cinema Group used the system to allow users to literally "walk" through a video sequence,[67] with the inter-frame segues and advances depending on the position and gait characteristics of the participants. There are many other possible interfaces for this system, tracking foot motion without requiring special footwear[69] in immersive virtual reality (VR) installations, identifying individuals based on the characteristics of their gait,[70] etc. Resonant tags The different systems described in this paper are able to locate and track activity at an intelligent surface. They are unable to identify a particular object, however; e.g., determine which hand in particular is sweeping against or tapping the wall, or which foot is in contact with the floor. Some of the other applications, namely the electronic whiteboards mentioned earlier, require this feature, so they tag their writing implements with bar codes,[10] active IR signatures,[9] or passive magnetically coupled resonant (e.g., LC) tags,[71] which are enabled when the pen presses against the surface of the board. In these systems, the resonance frequency determines the color assigned to the marker, and an array of closely spaced coils is used to locate the marker on the board. Recent interactive board games[72] have also embedded resonant magnetic tags in the bases of their action figures, enabling scores of them to be identified via their resonant frequency when on the game board and similarly tracked[73] in real time with approximately millimeter accuracy across the board and with lower resolution over a couple of inches in height. Although some developers exploit resonant tags for simple, single-tag identification (see Reference 8), this application is already well-served by commercial RFID (radio-frequency identification device) chip systems,[74] where many bits of digital identification can be coded into a low-power integrated circuit that is remotely energized and interrogated through a pulse of magnetic or RF energy. The simpler, chipless resonant tags have advantages that empower them for other applications, namely: (1) fast identification of a relatively small ensemble (circa 50) of tagged objects, (2) the ability to read many at once without introducing additional delay or interference, (3) the simple extraction of proximity and orientation through the coupling strength detected at the reader, (4) the ability to easily make the tag into a sensor (see References 75, [76) by making the resonance properties vary with physical parameters, and (5) their relatively low cost. To explore responsive-surface applications that leverage these capabilities, we have developed a simple, swept-frequency resonant tag reader, as diagrammed in Figure 16 and introduced in References 77 and [78. The lineage of these devices derives from early antishoplifting tag systems.[79] Our reader is a simple inductive bridge, with excitation swept between roughly 50 kHz and 300 kHz 30 times per second. When a magnetically coupled resonance (LC tag or magnetostrictor[80]) approaches the reader, it draws energy from the 1-foot diameter search coil, momentarily unbalancing the bridge and producing a blip that is enhanced through filtering and digitized by an onboard microcomputer that runs peak finding code similar to that developed for the laser rangefinder described earlier. The resonance frequency, resonance width, and integrated height of each detected peak are sent to a host computer after every sweep. The center frequency of each peak corresponds to the tag's ID. The maximum number of independent tags is dependent on the Q of the resonances and the breadth of the frequency sweep. And, the integrated amplitude is a function of its distance and orientation, which are coupled for each tag. As noted in Figure 16, the sinusoid driving the reader coils is derived from a simple analog voltage controlled oscillator (VCO) rather than a more expensive direct digital synthesizer (DDS). Because the VCO is subject to slow frequency drift from changes in temperature and environmental parameters, we periodically calibrate it against the microprocessor's clock by counting cycles across a fixed interval, in a similar fashion to the way in which the microprocessor-controlled analog music synthesizers of the past decades were automatically tuned.[81] Our application experiments with this tagging system have been inspired by two disparate areas of inquiry: Hiroshi Ishii's work with tangible bits[82] (where the dominant computer interface is realized by actual three-dimensional "tangible" objects rather than today's keyboard, mouse, and GUI) and John Zorn's musical performances of the early 1980s,[83] where he would improvise with a table strewn with different physical objects, each of which could make interesting acoustic sounds in various ways. Our work has culminated in the Musical Trinkets system,[84] where we have embedded LC and magnetostrictor tags into an ensemble of common objects (Figure 17), each of which produces different musical and graphical behavior when brought near a "smart table" containing the reader coil. Only a single LC tag or magnetostrictor was embedded in most of the objects of Figure 17, thus a resonance seen at a particular frequency indicates the presence of the corresponding object. Two of the tags, however, exploited wider variance in frequency to enable additional degrees of control. In one of these, the tag's coil can be stretched or compressed by the fingers while they are holding the tagged object, causing a change in the inductance, hence, a continuous shift in resonant frequency. This thus becomes a dual-parameter controller, with the resonance amplitude a function of coupling strength (therefore position and orientation) and the center frequency determined by the shape of the coil (in this case, amount of finger pull). Another is a cube with three tags mounted along orthogonal axes, enabling distance and orientation to be inferred when it is within the reader's range. Figure 18 shows our smallest tags, mounted on a set of plastic rings to enable tracking of the fingertips. These rings could be detected up to 8 inches from the search coil, and their net coupling strength indicated the combined proximity and orientation of each finger (the signal is maximal when the ring coil's axis is aligned with the local magnetic field, which tends to run vertically along the search coil's axis). Larger coils were used in the bigger objects, which could be detected more than a foot from the search coil, giving a wide continuous range of sensitivity. Magnetostrictors were also embedded in other objects. Using an antilog frequency sweep enabled them to be detected well, since they are very high-Q resonators at lower frequency (50-100 kHz). Figure 19 shows the tag reader's typical analog output (i.e., conditioned bridge unbalance vs the sweep frequency). The top trace, being fairly flat, shows that the bridge is well nulled in the absence of resonant tags. The middle trace shows the bridge signal when all four finger tags are introduced (these tags occupy the highest frequency slots), while the lower trace shows the sweep with a combination of different tags near the reader. Figure 20A shows the prototype of Musical Trinkets, with the reader, its coil, and various tagged objects. Figure 20B shows the final system in action, with interactive "magic mirror" graphics projected onto a rear screen mounted inside the reader coil. The musical mapping that we have developed to work with these objects, as detailed in Reference 78, runs on a personal computer and demonstrates several new concepts in musical interaction. All of the objects are "continuous" controllers, with the intensity of their sound, effect, or attack depending on their distance and/or velocity. The rings work as a virtual keyboard, playing notes with dynamics determined by the finger speed. The particular notes that the rings play are chosen to harmonize with the background chord determined by the frog and two goblins, which, when introduced into the reader, launch musical triads, with the chosen chord dependent on the combination of the objects that are present. The cube plays a brass sound, with the pitch bending in different ways with orientation. The pressure-tag brings up a male choir when introduced, progressively cross-fading into female choir sound when squeezed. The dinosaur is a continuous pitch bender for the background chords, the pig introduces vibrato into the ring voices, the candy-corn changes the timbre of the ring and background voices, and the pumpkin plays a shimmering sequence, which grows brighter as it approaches the reader. An eyeball contains a multiaxis tag that adds a strong chorusing effect when introduced, varying with inclination. All of these objects also launch and control related graphical events, as seen in Figure 20B. Another application of our tagging system has been pursued by Ishii's Tangible Media Group, who used it for their musicBottles system, as shown at SIGGRAPH 99.[85] Here, LC tags in the necks of an ensemble of bottles enabled individual bottles to be detected and identified when placed on a table under which the reader's coil was mounted. The bottles' stoppers contained embedded ferrite beads, which acted as cores when the stopper was inserted, correspondingly shifting the tag's resonant frequency. This shift, readily picked up by the tag reader, was sent to the host computer, which promptly began playing a corresponding musical track, in analogy to the classic "message in a bottle." We are currently perfecting another application of our tag reader,[86] where we drive pairs of search coils together that are aligned in a Hemholtz configuration, producing a uniform magnetic field between them. By enclosing a cubical volume within six coils wound around the cube's square sides and alternately driving the three opposing pairs of coils, we obtain six magnetic loading measurements that allow us to determine the position and absolute angle of any contained tags that couple into all axes. This system has many applications; e.g., ring-bearing fingers such as in Figure 18 can be independently tracked for virtual reality interactions without requiring a wired and bulky glove, or, in an entirely different venue, a small tag can be attached to a tumor on a patient receiving radiation therapy, allowing a tightly focused treatment beam to dynamically track the patient's involuntary movements (from breathing, etc.) and thus deliver a more effective dose. Conclusions and future developments In this paper we have described several different techniques for making surfaces interactive and opening up new modes of computer-user interaction for responsive environments. Most of these methods have little impact on the nature of the surface, and can efficiently "retrofit" existing walls, floors, windows, etc., with interactive capability. The application examples described in this paper are multimedia in nature. Readers are encouraged to visit the Responsive Environments Group's project Web site,[87] where a comprehensive set of video clips is posted. We are investigating the possibility of porting the laser scanner hardware to miniature commercial barcode scanners, creating a much smaller package that enables more applications. We are also looking at different ways to interpret the data, such as dealing effectively with multiple hands and occlusion. New digital algorithms are being developed for the tap tracker system to improve its performance and extract more features from tapping on poorly characterized surfaces. In addition to exploring new types of interactive content for this interface, we are exploiting its scalability potential, where taps across very large and dispersive glass panes can be tracked by a small number of sensors at its periphery. Likewise, we are extending our calibration procedure and detection electronics so the tracking algorithms can easily adapt to panels with differing responses and propagation characteristics. Robust clustering algorithms or an efficient pixellation strategy would make our piezoelectric floor useful for several simultaneous people or a small crowd. We are also exploring new techniques for tracking the multiaxis position of our resonant tags accurately across desktop-sized three-dimensional spaces for new user interface and medical applications. Acknowledgments This paper spans a large body of work to which many people have contributed. In particular, we would like to thank many of our Media Lab colleagues, namely Ari Benbasat for helping with many of these projects, Warit Wichakool for working on the tap-tracker electronics and embedded software, Jeff Hayashida for doing the mechanical design of the new laser system and Chris Yang for building it, Hiroshi Ishii for encouragement and his enthusiastic and exciting adoption of many of these systems, Craig Wisneski and Julian Orbanes for making the first tap tracker work with PP+, and Craig Abler for his help with the carpet system. We likewise appreciate the collaborations with Pete Rice from the Opera of the Future Group and Judith Donath's Sociable Media Group that developed captivating applications for our laser tracker system. We also thank Matt Reynolds, Rich Fletcher, Rehmi Post, Neil Gershenfeld, and our other associates from the Physics and Media Group for several useful discussions across the course of these projects. John Oliver of the Harvard University Physics Department gave us useful insight into front-end photodiode amplifiers and Jacques Govignon of the Draper Laboratory gave us some early guidance on laser rangefinding. Paul Seiler of the Paul Scherrer Institute in Switzerland is thanked for useful discussions about medical applications for our tagging technologies, and we appreciated the hints on structural wave propagation given by Shawn Burke at the Boston University Photonics Center. We acknowledge the support of the Things That Think consortium and the other sponsors of the MIT Media Laboratory. **Trademark or registered trademark of Microfield Graphics, Inc., or Microsoft Corporation. Cited references 1. J. Achenbach, "Future Perfect," The Washington Post (Friday, October 8, 1999), p. H01. 2. S. Greenberg, S. Hayne, and R. Rada, Groupware for Real-Time Drawing, McGraw-Hill Europe, Berkshire, England (1995). 3. P. Maes, T. Darrell, B. Blumberg, and A. Pentland, "The ALIVE System: Full-body Interaction with Animated Autonomous Agents," Multimedia Systems 5, No. 2, 105-112 (1997). 4. N. Lynn, "Sensitive Skins," Flight International 136, 34-36 (1989). 5. J. A. Parnell, Digitizing Cursor and Coordinate Grid System, U.S. Patent No. 4,479,032 (October 23, 1984). 6. R. A. Quinnell, "Touchscreen Technology Improves and Extends Its Options," EDN 40, No. 23, 52, 54-56, 58, 60, 62, 64 (November 9, 1995). 7. See http://www.glidepoint.com/index.htm and http://www.fingerworks.com. 8. O. Omojola, E. R. Post, M. D. Hancher, Y. Maguire, R. Pappu, B. Schoner, P. R. Russo, R. Fletcher, and N. Gershenfeld, "An Installation of Interactive Furniture," IBM Systems Journal 39, Nos. 3&4, 861-879 (2000, this issue). 9. C. Yates, "mimio: A Whiteboard Without the Board--Virtual Ink's Hardware/Software Combo Cuts the Cost of Collaboration and Makes It Truly Portable," PC Week 16, No. 26, 50 (June 28, 1999). 10. H. Eglowstein, "Almost as Good as Being There," Byte Magazine (April 1994), 173-175. 11. R. L. Garwin and J. L. Levine, Indicator to Data Processing Interface, U.S. Patent No. 4,558,313 (November 1, 1984). 12. F. A. Ebling, R. L. Johnson, and R. S. Goldhor, Infrared Light Beam X-Y Position Encoder for Display Devices, U.S. Patent No. 3,775,560 (November 27, 1973). 13. See Intrepid touch screen at http://www.hypertech.com/touchscreens. 14. R. Sharma, T. S. Huang, V. I. Pavlovic, Y. Zhao, Z. Lo, S. Chu, K. Schulten, A. Dalke, J. Phillips, M. Zeller, and W. Humphrey, "Speech/Gesture Interface to a Visual Computing Environment for Molecular Biologists," Proceedings of the 13th International Conference on Pattern Recognition, Volume 3, Vienna, Austria, IEEE Computer Society Press, Los Alamitos, CA (August 1996), pp. 964-968. 15. J. Rehg and T. Kanade, "Digiteyes: Vision-Based Hand Tracking for Human-Computer Interaction," Proceedings of the Workshop on Motion of Non-Rigid and Articulated Objects, Austin, TX, IEEE Computer Society Press, Los Alamitos, CA (November 1994), pp. 16-22. 16. A. Wojdala, "Challenges of Virtual Set Technology," IEEE Multimedia 5, No. 1, 50-57 (1998). 17. V. Macellari, T. Leo, P. Chistolini, and M. Bugarini, "Comparison Among Remote Sensing Systems for Human Movement Measurement," Proceedings of MELECON '85, Mediterranean Electrotechnical Conference (Cat. No. 85CH2185-7), Vol. 1, IEEE, New York (1985), pp. 1-5. 18. J. Rekimoto and N. Matsushita, "Perceptual Surfaces: Towards a Human and Object Sensitive Interactive Display," Workshop on Perceptual User Interfaces (PUI-97), Banff, Alberta, Canada, Microsoft (October 1997), pp. 30-32. 19. J. Underkoffler and H. Ishii, "Urp: A Luminous-Tangible Workbench for Urban Planning and Design," Proceedings of Conference on Human Factors in Computing Systems (CHI '99), Pittsburgh, PA, ACM Press, New York (1999), pp. 386-393. 20. B. Ullmer and H. Ishii, "The metaDESK: Models and Prototypes for Tangible User Interfaces," Proceedings of Symposium on User Interface Software and Technology (UIST '97), Banff, Alberta, Canada, ACM Press, New York (October 1997), pp. 223-232. 21. J. R. Smith, T. White, C. Dodge, J. Paradiso, N. Gershenfeld, and D. Allport, "Electric Field Sensing for Graphical Interfaces," IEEE Computer Graphics and Applications 18, No. 3, 54-60 (1998). 22. J. Paradiso, "The Brain Opera Technology: New Instruments and Gestural Sensors for Musical Interaction and Performance," Journal of New Music Research 28, No. 2, 130-149 (1999). 23. J. Paradiso and N. Gershenfeld, "Musical Applications of Electric Field Sensing," Computer Music Journal 21, No. 2, 69-89 (1997). 24. J. M. Rueger, Electronic Distance Measurement: An Introduction, Fourth Edition, Springer-Verlag, New York (1996). 25. H. R. Everett, Sensors for Mobile Robots: Theory and Application, A K Peters, Ltd., Wellesley, MA (1995). 26. E. C. Urban, "The Information Warrior," IEEE Spectrum 32, No. 11, 66-70 (1995). 27. J. Strickon and J. Paradiso, "Tracking Hands Above Large Interactive Surfaces with a Low-Cost Scanning Laser Rangefinder," Proceedings of CHI '98 Conference on Human Factors in Computing Systems: Summary, ACM Press, New York (1998), pp. 231-232. 28. J. A. Strickon, "Design and HCI Applications of a Low-Cost Scanning Laser Rangefinder," M.S. thesis, MIT, Electrical Engineering and Computer Science, and MIT Media Laboratory, Cambridge, MA (January 1999). 29. W. Chapelle and E. Gillis, "Sensing Automobile Occupant Position with Optical Triangulation," Sensors, 18-21 (December 1995). 30. M. Petrov, A. Talapov, T. Robertson, A. Lebedev, A. Zhilyaev, and L. Polonskiy, "Optical 3D Digitizers: Bringing Life to the Virtual World," IEEE Computer Graphics and Applications 18, No. 3, 28-37 (1998). 31. J. Paradiso and F. Sparacino, "Optical Tracking for Music and Dance Performance," Optical 3-D Measurement Techniques IV, A. Gruen and H. Kahmen, Editors, Heidelberg, Herbert Wichmann Verlag (1998), pp. 11-18. 32. K. S. Hashemi, P. T. Hurst, and J. N. Oliver, "Sources of Error in a Laser Rangefinder," Review of Scientific Instruments 65, No. 10, 3165-3171 (1994). 33. Application notes for the C30817E, EG&G/PerkinElmer Optoelectronics, see http://opto.perkinelmer.com/products/catalog/familylisting/family419.asp. 34. Analog Devices AD603 Datasheet, see http://www.analog.com/pdf/AD603_c.pdf. 35. Maxim MAX436 Datasheet, see http://209.1.238.250/arpdf/1564.pdf. 36. M. Zahid, J. S. Smith, and J. Lucas, "High-Frequency Phase Measurement for Optical Rangefinding System," IEE Proceedings, Science Measurement and Technology (UK) 144, No. 3, 141-148 (1997). 37. J. F. Ready, Industrial Applications of Lasers, Academic Press, New York (1997). 38. See http://www.microchip.com/. 39. See http://semiconductor.hitachi.com/superh/. 40. C. Yang, New Laser Range Finder Designs for Tracking Hands Atop Large Interactive Surfaces, M.S. thesis, MIT, Electrical Engineering and Computer Science, and MIT Media Laboratory, Cambridge, MA (January 2001). 41. See http://www.analog.com/microconverter/. 42. P. W. Rice, Stretchable Music: A Graphically Rich, Interactive Composition System, M.S. thesis, MIT Media Laboratory, Cambridge, MA (September 1998). 43. J. Strickon, P. Rice, and J. Paradiso, "Stretchable Music with Laser Rangefinder," SIGGRAPH 98 Conference Abstracts and Applications, ACM Press, New York (1998), p. 123. 44. D. Phillips-Mahoney, "Laser Technology Gives Video Walls a Hand," Computer Graphics World 21, No. 10, 17-18 (1998). 45. J. Donath, "Visual Who: Animating the Affinities and Activities of an Electronic Community," Proceedings of the Third ACM International Conference on Multimedia (Multimedia '95), San Francisco, CA, ACM Press, New York (November 1995), pp. 99-107. 46. See http://www.operatotale.org/OT5_E/N_OT5.htm. 47. J. Donath, J. Paradiso, D. Spiegel, and K. Hsiao, "LaserWho," SIGGRAPH 2000 Conference Abstracts and Applications, ACM Press, New York (July 2000), p. 86. 48. H. Ishii, C. Wisneski, J. Orbanes, B. Chun, and J. Paradiso, "PingPongPlus: Design of an Athletic-Tangible Interface for Computer-Supported Cooperative Play," Proceedings of Conference on Human Factors in Computing Systems (CHI '99), Pittsburgh, PA, ACM Press, New York (1999), pp. 394-401. 49. C. Wisneski, J. Orbanes, and H. Ishii, "PingPongPlus," Conference Abstracts and Applications of SIGGRAPH 98, Orlando, Florida, ACM Press, New York (1998), p. 111. 50. C. Wisneski, J. Orbanes, and H. Ishii, "PingPongPlus: Augmentation and Transformation of Athletic Interpersonal Interaction," Summary of Conference on Human Factors in Computing Systems (CHI '98), Los Angeles, ACM Press, New York (1998), pp. 327-328. 51. Y. T. Chan, "A Simple and Efficient Estimator for Hyperbolic Location," IEEE Transactions on Signal Processing 42, No. 8, 1905-1915 (1995). 52. J. A. Paradiso, "The Interactive Balloon: Sensing, Actuation, and Behavior in a Common Object," IBM Systems Journal 35, Nos. 3&4, 473-487 (1996). 53. L. Cremer, M. Heckl, and R. R. Ungar, Structure-Borne Sound, Second Edition, Springer-Verlag, New York (1990). 54. B. T. Turko and R. C. Smith, "A Precision Timing Discriminator for High Density Detector Systems," IEEE Transactions on Nuclear Science 39, No. 5, Pt. 1, 1311-15 (1992). 55. See http://web.mit.edu/bunnie/www/proj/sh1wh/index.html. 56. R. Pinkston, J. Kerkhoff, and M. McQuilken, "A Touch Sensitive Dance Floor MIDI Controller," Proceedings of the 1995 International Computer Music Conference, San Francisco, International Computer Music Association (1995), pp. 224-225. 57. Infusion Systems floor tile, see http://www.infusionsystems.com/products/taptile.html. 58. F. Shiraishi, P. Russo, M. Chen, and H. Tan, Sensing Floor Project, see http://vismod.www.media.mit.edu/people/hongtan/research/SmartChair.html. 59. N. Griffith and M. Fernstrom, "LiteFoot: A Floor Space for Recording Dance and Controlling Media," Proceedings of the 1998 International Computer Music Conference, International Computer Music Association, San Francisco, CA (October 1998), pp. 475-481. See also http://www.ul.ie/~pal/litefoot/Index.htm. 60. R. Post, personal communication regarding capacitive floor project (1999). 61. C. P. Mason, "Terpsitone: A New Electronic Novelty," Radio Craft, 335 (December 1936). 62. J. A. Paradiso, K. Hsiao, A. Y. Benbasat, and Z. Teegarden, "Design and Implementation of Expressive Footwear," IBM Systems Journal 39, Nos. 3&4, 511-529 (2000, this issue). 63. Measurement Specialties, Inc., see http://www.msiusa.com/Piezo_Cable.pdf. 64. See http://mcu.motsps.com/. 65. C. W. Therrien, Decision Estimation and Classification, John Wiley & Sons, Inc., New York (1989). 66. J. Paradiso, C. Abler, K. Hsiao, and M. Reynolds, "The Magic Carpet: Physical Sensing for Immersive Environments," Proceedings of 1997 Annual International Conference on Human Factors in Computing Systems, Atlanta, GA, ACM Press, New York (1997), pp. 277-278. 67. G. Davenport, S. Agamanolis, B. Barry, B. Bradley, and K. Brooks, "Synergistic Storyscapes and Constructionist Cinematic Sharing," IBM Systems Journal 39, Nos. 3&4, 456-469 (2000, this issue). 68. M. Spiwak, "Build a Radar Speed Gun," Popular Electronics 12, No. 6, 37-42, 90 (1995). 69. I. Choi and C. Ricci, "Foot-Mounted Gesture Detection and Its Application in Virtual Environments," 1997 IEEE International Conference on Systems, Man, and Cybernetics, Computational Cybernetics and Simulation, Vol. 5, Orlando, FL, IEEE, New York, NY (October 1997), pp. 4248-4253. 70. R. J. Orr and G. D. Abowd, "The Smart Floor: A Mechanism for Natural User Identification and Tracking," Proceedings of the CHI 2000 Conference on Human Factors in Computing Systems: Extended Abstracts, ACM Press, New York (2000), pp. 275-276. 71. A. Murakami, T. Yamanami, and T. Funahashi, Position Detector, U.S. Patent No. 5,466,896 (November 14, 1995). 72. See http://www.zowiepower.com/about/smart2.html#technology. 73. A. N. Dames, Position Encoder, U.S. Patent No. 5,815,091 (September 29, 1998). 74. K. Finkenzeller, RFID Handbook: Radio-Frequency Identification Fundamentals and Applications, John Wiley & Sons, Inc., New York (1999). 75. W. B. Spillman, Jr., S. Durkee, and W. W. Kuhns, "Remotely Interrogated Sensor Electronics (RISE) for Smart Structures Applications," Proceedings of the SPIE Second European Conference on Smart Structures and Materials, Volume 2361, Glasgow (October, 1994), pp. 282-284. 76. R. Fletcher, J. A. Levitan, J. Rosenberg, and N. Gershenfeld, "Application of Smart Materials to Wireless ID Tags and Remote Sensors," Materials for Smart Systems II, Proceedings of the 1997 MRS Symposium, Boston, MA, Materials Research Society, Pittsburgh, PA (1997), pp. 557-562. 77. J. Paradiso and K. Hsiao, "Swept-Frequency, Magnetically-Coupled Resonant Tags for Realtime, Continuous, Multiparameter Control," Human Factors in Computing Systems; CHI99 Extended Abstracts, ACM Press, New York (1999), pp. 212-213. 78. K. Hsiao and J. Paradiso, "A New Continuous Multimodal Musical Controller Using Wireless Magnetic Tags," Proceedings of the 1999 International Computer Music Conference, International Computer Music Association, San Francisco (October 1999), pp. 24-27. 79. G. J. Lichtblau, "Resonant Tag and Deactivator for Use in Electronic Security System," U.S. Patent No. 4,498,076 (February 5, 1985). 80. See http://www.sensormatic.com/eas_www/easum.htm. 81. M. H. Chamberlin, Musical Applications of Microprocessors, Second Edition, Hayden Book Company, Hasbrouck Heights, NJ (1985). 82. H. Ishii and B. Ullmer, "Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms," Proceedings of the 1997 Conference on Human Factors in Computing Systems, New York (1997), ACM Press, pp. 234-241. 83. F. Davis, " 'Zorn' for 'Anger,' " The Atlantic Monthly 267, No. 1 (January 1991), pp. 97-100. 84. J. Paradiso, K. Hsiao, and A. Benbasat, "Musical Trinkets: New Pieces to Play," SIGGRAPH 2000 Conference Abstracts and Applications, ACM Press, New York (July 2000), p. 90. 85. H. Ishii, R. Fletcher, J. Lee, S. Choo, J. Berzowska, C. Wisneski, C. Cano, A. Hernandez, and C. Bulthaup, "musicBottles," Conference Abstracts and Applications of SIGGRAPH 99, Los Angeles, CA, ACM Press, New York (1999), pp. 174. 86. K. Hsiao, "Fast Multi-Axis Tracking of Magnetically Resonant Passive Tags: Methods and Applications," M.S. thesis, MIT, Electrical Engineering and Computer Science, and MIT Media Laboratory, Cambridge, MA (September 2000). 87. See http://www.media.mit.edu/resenv/projects.html. Accepted for publication June 14, 2000. Biographical sketches of authors Joseph A. Paradiso MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: paradiso@media.mit.edu). Dr. Paradiso is a principal research scientist at the MIT Media Laboratory, where he leads the Responsive Environments Group and is the technology director for the Things That Think consortium. Prior to this, he has held positions at the Draper Laboratory in Cambridge, Massachusetts, and the Swiss Federal Institute of Technology (ETH) in Zurich designing high-energy physics detectors, spacecraft control algorithms, and sensor systems. He received a B.S. degree in electrical engineering and physics from Tufts University in 1977 and a Ph.D. degree in physics at the Massachusetts Institute of Technology in 1981 as a Compton Fellow. He also has designed several synthesizers and interfaces for electronic music and is the recipient of the 2000 Discover Award in digital entertainment. Further information about Dr. Paradiso may be found at http://www.media.mit.edu/~joep. Kai-yuh Hsiao MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: khsiao@mit.edu). Mr. Hsiao is a graduate student at MIT and a British Telecom Fellow in the Media Lab's Responsive Environments Group, working on physical sensors and new technologies for interactive music. He received his B.S. degree from MIT in electrical engineering and computer science in 1999. His interests in computers, electronics, and music started at a very early age. Joshua Strickon MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: strickon@media.mit.edu). Mr. Strickon is presently pursuing a Ph.D. in media arts and sciences at the Massachusetts Institute of Technology. His research work includes the study and development of interactive music and entertainment systems. Prior to that he has received a B.S. degree in 1998 and an M.Eng. degree in 1999 in electrical engineering and computer science, also at the Massachusetts Institute of Technology. Joshua Lifton MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: lifton@media.mit.edu). Mr. Lifton is a first-year graduate student in the Responsive Environments Group at the Massachusetts Institute of Technology. He received a B.A. degree in physics and mathematics from Swarthmore College in 1999. Ari Adler MIT Media Laboratory, 20 Ames Street, Cambridge, Massachusetts 02139-4307 (electronic mail: aadler@media.mit.edu). Mr. Adler is currently finishing his master's degree in mechanical engineering at the Massachusetts Institute of Technology where he is a research assistant in the Responsive Environments Group at the Media Lab. His thesis is on the development of a low-cost telemedicine system for use in developing countries. He has collaborated on several group projects at the Media Lab and was awarded an IBM fellowship for the year 1999-2000. Prior to coming to MIT, he worked as a mechanical engineer at Design Continuum, a product design consulting firm in the Boston area. He received his bachelor's degree in physics with a minor in mathematics from Brandeis University in 1996.