Combining Shape-changing Interfaces with Spatial Augmented Reality

We propose combining shape-changing interfaces and spatial augmented reality for extending the space of appearances and interactions of actuated interfaces. While shape-changing interfaces can dynamically alter the physical appearance of objects, the integration of spatial augmented reality additionally allows for dynamically changing objects’ optical appearance with high detail. This way, devices can render currently challenging features such as high frequency texture or fast motion. We frame this combination in the context of computer graphics with analogies to established techniques for increasing the realism of 3D objects such as bump mapping. This extensible framework helps us identify challenges of the two techniques and benefits of their combination. We utilize our prototype shape-changing device enriched with spatial augmented reality through projection mapping to demonstrate the concept. We present a novel mechanical distance-fields algorithm for real-time fitting of mechanically constrained shape-changing devices to arbitrary 3D graphics. Furthermore, we present a technique for increasing effective screen real estate for spatial augmented reality through view-dependent shape change.

Work presented at CHI 2016 (go to paper in the ACM DL).

D. Lindlbauer, J.E. Grønbæk, M. Birk, K. Halskov, M. Alexa, J. Müller, 2016. Combining Shape-Changing Interfaces and Spatial Augmented Reality Enables Extended Object Appearance. CHI ’16, San Jose, CA, USA

On this page, you can find the following open source data for

  • Software for mechanical distance field (MDF) algorithm (c++, Xcode)
  • Testmodels for MDF
  • Hardware files for shape-changing table prototype (.stl, .ai)

For more infos, contact David Lindlbauer (david.lindlbauer[_at_] or @da_lind).

Disclaimer: Software and hardware is provided as is. Feel free to use it, be we do not give any kind of warranty. Electrical schematics should be used with caution (i.e. use it at own risk). Do not try to build, wire or solder anything if you don’t feel confident in what you are doing. The software is most likely to contain bugs (known and unknown). Treat it more like a dictionary that should help you with your own implementation.

Mechanical distance field – Algorithm

In order to map the deformation of a physically constrained shape-changing device to arbitrary 3D graphics of spatial augmented reality, we developed a real-time algorithm for matching the two based on what we call mechanical distance fields (MDF). Our MDF encodes mechanical deformation in a single regular-spaced voxel grid. By matching desired virtual target shapes with MDF, we can determine the necessary deformation to represent any virtual object. This allows us to match the deformation of shape-changing interfaces in real-time to virtual 3D content, even if the device features deformation along 3 dimensions. For more details, please refer to the paper.

MDF example

The software is written in c++ (with Xcode, runs on OSX) with openFrameworks for GUI and CGAL for some algorithms (e.g., intersections). Below, you can download the software as zip, bundled with openFramework 0.8.4 and ofxCGAL (includes part of CGAL source). It should run pretty much out of the box. We might put it on GitHub in the future. This would, however, require a major cleanup and refactoring first.

In summary, you can load a shape-changing device (a series of .obj files, see below). The software then computes the mechanical distance field (i.e. voxel grid containing information on deformation). Since we use CGAL, we strongly recommend running the code in Release mode, otherwise it the computation might be slow (depends in the number of voxels in your MDF, we usually use 5K-20K). You can then export the MDF and use it in your own software. You can also load a virtual target and enable matching. The software will deform according to the position of the target. The source code can be used for implementing your own matching algorithm or porting the algorithm as-is to your software.

Download (100MB, all frameworks included, last update 2016-04-29):

Mechanical distance field – Test models

We created 3 different shape-changing devices as virtual models, a shape-changing phone with 4 actuators, a shape-changing tablet with 6 actuators and a shape-changing box with 54 actuators, respectively. Just place the folder in the “data” folder of the software and change the path in the software (config.h, needs recompilation).

The software parses the files by name. Naming scheme is {dimension}_{state}.obj.

Download (45MB, last update 2016-04-29):

Shape-changing tablet

We provide 3D print files for the tablet we built in our work. Printed on an Ultimaker 2. The top surface is printed from NinjaFlex. We additionally put a sheet of (regular) white paper on top for better quality of the projection. The download includes .stl files for printing, modeling files (Sketchup) and a closeup picture.

The device was heavily inspired by the work of Majken Rasmussen, who shared here device of a shape-changing phone. Thanks a lot and check out the work here:
Rasmussen et al., Balancing User and System Control in Shape-Changing Interfaces: a Designerly Exploration, TEI 2016

Download 3D print files (8MB, last update 2016-04-29)



Download Arduino driver and test software (java + processing, IntelliJ idea): 

Schematic to follow soon.

However, you basically do this:
– wire battery in series, connect to button, connect to voltage regular
– make a common connector (from breakout board) for vcc and ground and connect arduino / bluetooth / servors
– wire bluetooth to rx/tx of Ardunio
– wire servos to arduino ports