Click here to view the embedded video.

However modern browsers have the <canvas> element. This element allows for the manipulation of single pixels within this element. So I figured if I could encode the building-dates in a PNG and use canvas to display only those pixels which represent a building older than the given date it should be possible to time-slide through the buildings. Fortunately the fancy new mapping library Leaflet.js has a canvas tile layer build in. The BAG data is already available through EduGIS, so I only needed it to encode the data differently in the PNGs.

The encoding is very simple: per pixel there are 4 values available: red, green, blue and alpha. Since I only needed to encode 200-odd values I used just the red value. The years before 1850 are encoded as groups, since the data is so sparse, after 1850 each year is individually encoded. This means that from 1850 onwards the red value increases with 1. The client retrieves the encoded PNGs as normal tiles and look like this:

The image appears grey because I kept the green and blue values of the PNG the same as the red. This image is loaded into canvas and the imageData is retrieved using ctx.getImageData(0, 0, 256, 256) and stored as a jQuery data object on the canvas. This is important, since for visual effect we will manipulate the imageData on the canvas and we want to keep track of the original values. Once the imageData is attached to the canvas the colors are being calculated. It will take the original values, compares them to the current year and will decide whether or not to show the pixel and in which color.

Since only the grey tile is needed, the actual sliding through time is really fast because it doesn’t need to retrieve more data. With 4 bands of 255 values each you can encode an insane amount of data into a PNG, readily available through canvas for direct manipulation. Apart from time sliding, is detailed representation of DEM data an obvious use case.

We spend quite some time calibrating, recalibrating and reading documentation from the invite only Microsoft site without any effect. In the end Vista asked us to install an update and to reboot the Surface. When it was rebooted we ran the ‘Surface Shell’ application from the Desktop and suddenly it worked!!

The Surface, or at least our version, comes with a set of code examples. We tried several to see how well it works and found out that especially the Grand Piano demo is a crowd drawer (mainly due to the sound probably). The Piano is a limited set of piano keys in perspective view. ‘Pressing’ a key results in a tone, just like a real piano

Playing piano on a Surface

The sensitivity of the Surface is well balanced. The Surface doesn’t notice fingers which are hovering just above the surface and you don’t need to apply any pressure to get noticed once you touch it. We’ve built our own table using a similar FTIR technique and we had quite some problems getting that balance right. Also the Surface is using 5 camera’s where we used just one as such it’s resolution is better as well.

It has additional logic in the interpretation of the camera feeds. It has three different types of touches it can recognize: finger, blob and tag. From each type it gives a location, an orientation, an id and in case of the tags a value.

Surface information of a finger

The difference between a blob and a finger is the size and shape of the touch, for instance a small square object is seen as a blob and a larger oval object is seen as a finger. Furthermore the earlier white-only bias of the Surface we experienced is now changed to a no-black bias. This gives some interesting results; a black/orange marker on the surface is recognized as a single finger at the tip of the marker;

Fingers come in strange shapes

The Surface feels pretty responsive, but using the Data Visualizer you notice that the position of the touch as interpreted by the Surface lags behind the actual touch. In the demos it is not noticeable but we found a similar lag very annoying on the DiamondTouch. On that table we managed to get rid of the lag and hopefully we can do the same here.

There is the famous demo of the telephone on the Surface, where it gets recognised and you can drag and drop data. We tried it and noticed that it doesn’t like android phones; it just doesn’t see the phone. An (ancient) windows mobile phone however does get recognised for what it is: a blob;

Android on the Surface

Obviously the Surface doesn’t automagically recognize phones or any other object. The table comes with a set of fiducials which you can stick on objects. These fiducials get recognised as tags with a value and using that value you can do fun stuff in your software.

Update

I discovered that while it doesn’t respond to hovering fingers, it does notice hovering areas the size of a hand. Which is slightly annoying since in normal use you easily have your hand hovering just above the surface while one or more fingers touch it. It might be possible to filter this unwanted behavior in real applications, because the hand shows up as a blob instead of a finger.

We were supposed to have two normal and one developer edition but it wasn’t very clear which one was which so we picked one and carried it to our office. This turned out to quite tricky, this thing is heavy and doesn’t have any clear handles. In the end we carried it by holding the projection-surface …

Once setup we had to fiddle a bit to get the powercable connected and find the proper (two) powerswitches. Once it booted it prompted us an EULA we had to accept. However the Surface is designed to be a standalone machine completely controlled by touch but before the touch was working we had to accept the EULA, which was only accessible by touch … hm

You need to accept to start touch, and you need touch to accept

So we had to plug in a mouse, but apparently we hadn’t chosen the developer edition since no usb ports where visible. The manual explained that we need to open up the side of the table to access the ports. After two screws and a bit of force we managed to open up the side and it gave us full access to all normal PC ports.

The surface insides

It asked us to create an admin account and once it was setup it gave us the choice to login as that admin or as TableUser, since we had no clue what the password for TableUser was, we logged in as admin.  After a bit we were shown a fresh Vista desktop with a few shortcuts. The most obvious for us was the calibration tool. We calibrated our table and started a touch application. It failed to run complaining that the filtergraph was not found at the location specified. We neither had knew what a filtergraph was, nor what the location specified was.  This got us stuck for quite a while, the manuals didn’t help and the online site with information was only accessible by invitation and due to some miscommunication(?) we were not (yet) invited.

Digging into things like registries, hidden folders and binary blobs we finally discovered that there are two versions of the calibration tool. One which does a background-light calibration and one which does a per camera calibration. We had run both, but we failed to notice that the latter had to run 5 times (5 cameras in the surface). Once we had done so the different surface programs ran without errors and without effect.

Pretty annoyed we banged our fist on the table and that seemed to have effect. It turned out that putting a lot of force on the table gave results, indicating a calibration issue. Having no idea how to finetune the calibration we started to experiment with different ways of touching. The current consensus is that it is designed for ubergeeks who never see the sun and are pure white; touching the Surface with a finger doesn’t give a result, wrapping the finger in white tissue suddenly does …

The 'attract' application

Hopefully we will figure out soon how to tweak the calibration and otherwise we need to buy a set of white surgical gloves to hand out to users

Still I’m very much excited by the prospect of creating beautiful GIS applications and so far I’ve had enthousiast reactions from developers as well.

Basic animation of hiding a card from view

After watching the Secondlight demo from Microsoft, Johan came up with an idea to show ‘private’ information on a touchtable. For instance during a board game on the table a player wants to view his private stack of cards. In a traditional game he would have his cards turned upside down and would view them behind his hand. This behaviour can be mimiced on the table as well. If the user puts the side of his hand on the table the table can show the card within the confines of his hand.

Click here to view the embedded video.

The movie, voiced over by Geodans own Barry White

Animation showing the original map and the final map

So we came up with a Wii-mote based game which is both simple and fun, at least we hope. You are presented with four maps in sequence. Each map has various cables and tubes hidden under streets and parks. You point the Wii-mote at the screen and a ‘searchlight’ appears which allows you to look underground. Somewhere on the map there is a breach which you have to fix. Once you find the breach, you need to dig a hole using your Wii-mote. Once you have created a deep enough hole you need to fix it. This is done by hammering the broken tube until it is fixed, again with the Wii-mote. The time it took to find and fix all 4 breaches is your score.

We use an opensource library which allows to connect the Wii-mote to XNA and build the game with XNA. My task was obviously to help to design the game and do the in-game graphics. Since we only had a week time we opted for a simple style. To create the four background maps I selected four towns on a map and traced them in Inkscape to get a simplified and colorful version. Next I draw a few cables (yellow) and tubes (blue). Finally I selected 20 points where a breach could occur.

To dig and to fix the player has to make either a dig or a hammer motion with the Wii-mote. We use the accelerometer to determine if the player is digging/hammering hard enough. He has to do it 4 times and the progress is shown as a simple animation:

Fixing animation

I hope to have a movie of people playing the game soon.

I was curious what kind of feedback you get from the Wii-mote and where it fails (in light of the news about the Wii motion plus). There are some issues around getting your Wii-mote connected to a PC by bluetooth, not every stack is good, nor areall bluetooth chipsets, however the MacBook supports the Wii-mote out of the box. So I connected the Wii-mote to my MacBook instead and used the DarwiinRemote application to get a reading from the sensors.

DarwiinRemote sensor readings.

The wii-mote is equiped with an accelerometer, which does not record the position of the wii-mote. What is does is that it records the different forces (accelerations) that are applied to the controller. So in rest it will record a force of 1G down (z= -1). If you hold the controller upside down it will record a force of 1G up (z = 1) (seen from the point of view of the controller). If you hold it on its side the reading will be x=1 or x=-1 or y=1 or y= -1, depending on the side. The other axis will be 0 in rest. Actually since in rest the only force acting on the controller is earth gravitation, which is 1, the absolute sum of the forces on x, y and z will be 1.

Wii sensor reading in three orientations.

With this knowledge you can deduce the orientation of the Wii-mote from the readings above. In the beginning the Wii-mote is held horizontal pointing forward (blue line at -1). Then the Wii-mote is rotated along its body, the left side is pointing up (red line at 1). Finally the Wii-mote is pointing downward (green line at 1). Note that we do not know how the Wii-mote is pointing downwards (whether the buttons facing forward or sideways). When you start moving the Wii-mote the forces applied on the Wii-mote easily surpass the 1G and when that happens it is much more difficult to deduce the orientation. The new Wii-motion Plus extension is supposed to solve this problem. From what I’ve seen and read it most likely will use a gyroscope to calculate the orientation of the Wii-mote regardless of the forces applied to it.

‘Onderwater scherm’ of the PlectraeMenu

Once you managed to get the solution file to work with Blend (manually moving the correct project to the top of the list in the .sln file), it is major fun. It has a split screen which resembles  the reveal codes screen of word perfect, back in the day. The drawing tools are quite good, not as good as inkscape, but since you can edit the XML as well you can copy the exact coordinates over from the SVG. It would be nicer if you could import the SVG directly though. To be honest, Blend is not designed to be a real graphic design tool, MS Ex. Design is. I guess the proposed workflow is that you do your design in Design and then export that to Blend to integrate your design with C# code. I do my design in Inkscape and then import that to Blend so it’s not that much different, just a bit more hassle to get the design in.

Blend is the interaction tool of the Expression set and it shows. You can create animations (storyboards) and bind them to events (left top corner in the screenshot) on any object. Furthermore you can use databinding so the coders just manipulate the content inside the databinding without editing the XAML. On the other hand you can change anything in the XAML as long as the databinding stays intact. This way you can work parallel with the coders, their changes showup in you project and yours in their without (too much) interference.

This approach present some new problems though. Since you can do a lot in XAML regarding events and actions, but you can do everything from code aswell, you really need to think where the line of seperation is. You are forced to sit next to the coders and discuss what is done in code and what in XAML, where to use templates and databinding and what XAMLs to create. Since we are all new to WPF we still need to learn what can be done in XAML and what is better done in code. Looking back to the last three projects the general trend is that more and more is moved to XAML, but that is probably due to our previous non-existant knowledge of the possibilities of XAML.

Asymmetrical bimanual touch gestures.

Lynn pointed out earlier; ‘two handed interaction IS useful for many things‘. Bill Buxton has written a chapter on two handed input in his work in progress book on human input to computer systems. Here he states that the common view that human manual actions are generally unimanual is mistaken. Handwriting for instance is almost always seen as a one-handed activity, but in practice the non-dominant hand is used to hold the paper and as such forms a frame of reference. This bimanual task is asymmetric; the dominant hand is doing the precise work and the non-dominant hand frames the location. So I’ve been thinking how to implement two handed interaction.

Most examples I know which implement two handed touch interfaces use symmetrical actions. The left hand is doing the same as the right hand, or in case of the iPhone they are fingers of the same hand. I’m more interested in asymmetrical bimanual actions, preferably using physics. These mirror the natural way we use our hands in different tasks, or at least I hope so.

My initial brainstorm somewhere late at night gave me two possible implementations. Using you left hand to fix the object you are manipulating and the right hand to rotate it. This way you can rotate it around a different point than its centroid (lower action). This actions very much mirrors the two multi-touch rotation I disliked earlier . Another implementation is to use the left hand to hold the object and the right hand to scroll its content. (upper action). This action doesn’t have a very clear added value over more traditional approaches either, but it is a start. One benefit of this approach is that you can manipulate both the object and the content without the need of titlebars and scrollbars.

Two examples of asymmetrical bimanual touch gestures.