Click here to view the embedded video.

The Surface has been hanging around in our office for a while and we’ve done lots of interesting small and big projects for it. However they’ve been pretty fragmented and to show what we could do with the Surface you had to use several applications which had many similarities, which was confusing. So we decided that it was time to create an application which would show all our new ideas and developments. Looking for a nice name we came across ‘kõik‘ which both means everyone and every(thing), which is pretty much sums up what we want to achieve: an application which can be used by everyone and shows everything we do.

It was originally designed for landuse planning. Planners need to be able to move and zoom a map, but also annotate on it. The trouble with touch devices is that there is no difference between drawing and dragging, unlike keyboard/mouse (drawing is left mousebutton and dragging middle for instance). Having to press a button somewhere on the Surface to enable/disable drawing was too confusing, people tend to forget that they pressed them and in what modus they are now.

Having a physical object on the table turned out to be much easier to use and remember. If there was ‘the’ object on the table, you could draw and if you want to move the map, you just lift the object and move the map. Having established the object as the drawing-toggle, I then moved all drawing related interaction to a menu activated and related to the object. Currently you can choose the color and line thickness and whether you want to draw a line or polygon or you want to edit existing features, undo/redo actions and add new drawing layers.

We were supposed to have two normal and one developer edition but it wasn’t very clear which one was which so we picked one and carried it to our office. This turned out to quite tricky, this thing is heavy and doesn’t have any clear handles. In the end we carried it by holding the projection-surface …

Once setup we had to fiddle a bit to get the powercable connected and find the proper (two) powerswitches. Once it booted it prompted us an EULA we had to accept. However the Surface is designed to be a standalone machine completely controlled by touch but before the touch was working we had to accept the EULA, which was only accessible by touch … hm

You need to accept to start touch, and you need touch to accept

So we had to plug in a mouse, but apparently we hadn’t chosen the developer edition since no usb ports where visible. The manual explained that we need to open up the side of the table to access the ports. After two screws and a bit of force we managed to open up the side and it gave us full access to all normal PC ports.

The surface insides

It asked us to create an admin account and once it was setup it gave us the choice to login as that admin or as TableUser, since we had no clue what the password for TableUser was, we logged in as admin.  After a bit we were shown a fresh Vista desktop with a few shortcuts. The most obvious for us was the calibration tool. We calibrated our table and started a touch application. It failed to run complaining that the filtergraph was not found at the location specified. We neither had knew what a filtergraph was, nor what the location specified was.  This got us stuck for quite a while, the manuals didn’t help and the online site with information was only accessible by invitation and due to some miscommunication(?) we were not (yet) invited.

Digging into things like registries, hidden folders and binary blobs we finally discovered that there are two versions of the calibration tool. One which does a background-light calibration and one which does a per camera calibration. We had run both, but we failed to notice that the latter had to run 5 times (5 cameras in the surface). Once we had done so the different surface programs ran without errors and without effect.

Pretty annoyed we banged our fist on the table and that seemed to have effect. It turned out that putting a lot of force on the table gave results, indicating a calibration issue. Having no idea how to finetune the calibration we started to experiment with different ways of touching. The current consensus is that it is designed for ubergeeks who never see the sun and are pure white; touching the Surface with a finger doesn’t give a result, wrapping the finger in white tissue suddenly does …

The 'attract' application

Hopefully we will figure out soon how to tweak the calibration and otherwise we need to buy a set of white surgical gloves to hand out to users

Basic animation of hiding a card from view

After watching the Secondlight demo from Microsoft, Johan came up with an idea to show ‘private’ information on a touchtable. For instance during a board game on the table a player wants to view his private stack of cards. In a traditional game he would have his cards turned upside down and would view them behind his hand. This behaviour can be mimiced on the table as well. If the user puts the side of his hand on the table the table can show the card within the confines of his hand.

‘Onderwater scherm’ of the PlectraeMenu

Once you managed to get the solution file to work with Blend (manually moving the correct project to the top of the list in the .sln file), it is major fun. It has a split screen which resembles  the reveal codes screen of word perfect, back in the day. The drawing tools are quite good, not as good as inkscape, but since you can edit the XML as well you can copy the exact coordinates over from the SVG. It would be nicer if you could import the SVG directly though. To be honest, Blend is not designed to be a real graphic design tool, MS Ex. Design is. I guess the proposed workflow is that you do your design in Design and then export that to Blend to integrate your design with C# code. I do my design in Inkscape and then import that to Blend so it’s not that much different, just a bit more hassle to get the design in.

Blend is the interaction tool of the Expression set and it shows. You can create animations (storyboards) and bind them to events (left top corner in the screenshot) on any object. Furthermore you can use databinding so the coders just manipulate the content inside the databinding without editing the XAML. On the other hand you can change anything in the XAML as long as the databinding stays intact. This way you can work parallel with the coders, their changes showup in you project and yours in their without (too much) interference.

This approach present some new problems though. Since you can do a lot in XAML regarding events and actions, but you can do everything from code aswell, you really need to think where the line of seperation is. You are forced to sit next to the coders and discuss what is done in code and what in XAML, where to use templates and databinding and what XAMLs to create. Since we are all new to WPF we still need to learn what can be done in XAML and what is better done in code. Looking back to the last three projects the general trend is that more and more is moved to XAML, but that is probably due to our previous non-existant knowledge of the possibilities of XAML.

Asymmetrical bimanual touch gestures.

Lynn pointed out earlier; ‘two handed interaction IS useful for many things‘. Bill Buxton has written a chapter on two handed input in his work in progress book on human input to computer systems. Here he states that the common view that human manual actions are generally unimanual is mistaken. Handwriting for instance is almost always seen as a one-handed activity, but in practice the non-dominant hand is used to hold the paper and as such forms a frame of reference. This bimanual task is asymmetric; the dominant hand is doing the precise work and the non-dominant hand frames the location. So I’ve been thinking how to implement two handed interaction.

Most examples I know which implement two handed touch interfaces use symmetrical actions. The left hand is doing the same as the right hand, or in case of the iPhone they are fingers of the same hand. I’m more interested in asymmetrical bimanual actions, preferably using physics. These mirror the natural way we use our hands in different tasks, or at least I hope so.

My initial brainstorm somewhere late at night gave me two possible implementations. Using you left hand to fix the object you are manipulating and the right hand to rotate it. This way you can rotate it around a different point than its centroid (lower action). This actions very much mirrors the two multi-touch rotation I disliked earlier . Another implementation is to use the left hand to hold the object and the right hand to scroll its content. (upper action). This action doesn’t have a very clear added value over more traditional approaches either, but it is a start. One benefit of this approach is that you can manipulate both the object and the content without the need of titlebars and scrollbars.

Two examples of asymmetrical bimanual touch gestures.

manipulating object with and without pysics

The whole point of interface design is to create a natural/intuitive interface. However, since computers are not natural by definition it is hard to decide what is natural. In the classic WIMP interface the desktop has been used as a metaphor to design the interaction with the computer. Commonly a traditional desktop is horizontal and a computer screen is vertical. This and many other shortcomings of the metaphor creates a mental gap which has to be bridged by training. This training has altered our perception of ‘natural’ or intuitive interfaces for computers. For the trained user it makes sense that pressing the right mouse button pops up a context menu. Also it is only possible to move a window by dragging its handle bar and it is not possible to rotate a window. Obviously the desktop-metaphor can only go so far. There is no such thing as a menu on a classic desktop, not can one resize a piece of paper easily. It is not hard to point out the short comings of the WIMP interface, nor is it very difficult to create a ‘better’ interface. Unfortunately the average user is used to the Windows interface and invested a serious amount of time to be able to navigate it. So any new interface that works with similar in-/output devices as the standard PC has to unlearn the Windows way and learn the new way. Obviously this is quite a hurdle for people and as such new interfaces seldom survive let alone go main stream.

Enter tabletop computing…

Since tabletop computing hasn’t developed a standard interface, yet, we still have a chance to create new metaphors which will stick. Apple has managed to get a serious grip on multi-touch interface design. Many people believe that the multi-touch interface has empowered the designer to create such a novel interface. However it is my believe that only a few actions really need multi-touch, zooming being one. I don’t think people really want to use two hands to control a computer when they can do it with one hand. Since tabletop computing is much more akin to the classic desktop we can have another look at this metaphor. To rotate a piece of paper you don’t put your two index fingers on the sheet of paper and start rotating. If you want to rotate it 180degrees you probably don’t even try to turn it out of your wrist. Instead you grab it at a corner and give it a twist. – To be honest, since you can manipulate the piece of paper in 3D it is most likely that you grab the piece of paper from the table and turn it in the air -

However if we stick to the concept of 2D manipulation we can use it to enhance the current metaphor. Even if you don’t know Newton’s laws of motion, you still expect them to work. Turning a piece of paper by putting a finger in the corner and moving it, whilst counting on the inertia and drag of the paper to rotate instead of translate the paper doesn’t require any degree in physics. Few people would know which forces are at work on the paper, even fewer do care, only the result counts. Why does one not use a multi-touch gesture as used in the iPhone? Basically because it is more complicated. Look at the very simple animation below. The upper user is using physics to turn its object, while the lower user is using multi-touch. As you can see the upper user can keep on spinning if he likes, while the lower user runs into a wrist issue.

manipulating object with and without pysics

While in real life we don’t need to understand the Newtonian laws to be able to use them, in the software world it is not that simple. Since there is no natural gravity within the computer, the Newtonian laws don’t apply naturally either. We need to introduce physics into the GUI. Fortunately physics is very important in current games and a lot of effort has been put in creating physics engines which recreate the Newtonian world of physics in the virtual world of the computer. We are planning to use Farseer, since it is both open source and runs in C#. Having physics at our finger tips we can create much cooler interfaces. Instead of having two seperate actions to translate and rotate an object, you can do it in one fluid motion (upper user). Moving an object is also much easier, just give it a push in the right direction (lower user).

Using physics to manipulate objects – please note that the third hand of the lower user doesn’t touch the table anymore -

Using minimaps to allow for multiple users on the same computer

As I explained earlier, there are some interesting challenges when designing multi-touch multi-user applications. Take for instance the use of the table in a command and control center during a disaster. In this scenario there is a map which displays the area where the disaster has occurred. Additionally you can view the real-time location of different emergency vehicles. It could be that the chief fireman is interested in tracking a particular vehicle moving outside the current view, where as the rest of the staff around the table want to keep the current view. Right now the processes inside those command and control centers do not resemble the described scenario. For a starter because they don’t have a table.

There is an area though where you have similar scenarios: Gaming, especially RTS and certain simulation games. You need to keep track of different units and at the same time require a detailed zoom level which will exclude certain units. Although these games are rarely, if at all, multi-user in the sense of multiple users behind 1 computer at the same time, it does provide some interesting ideas. This is a screenshot of Transport Tycoon (from here):

It has the main mapview and two smaller mapviews. One with an overview of the entire area and one that tracks a train. Using mini-maps you could track particular vehicles are keep an eye on a specific location. Also you can use it to look something up on a different part of the map, while the main stays in view for the others. So the tug-of-war action, as described earlier, suddenly is possible without the tugging or warring bit.

One user is still zooming in, where as the other user has his own mini-map where he can drag the map at the same time. If anything interesting occurs in one of the mini-maps it can become the main view, where as the former main view becomes a mini-map itself.

Illustrating the problem of multiple users in the same interface

Working with multi-user tabletop computing produces some interesting problems. On our mapping application we discovered that every now and then multiple people want to control the map. Resulting in a tug-of-war for control over the map. On multi-user environments like the DiamondTouch it could be solved by a simple first come first serve principle, where all input of other users is ignored. However this cannot be used by single user environments like the FTIR-based tables and the Surface.

On a typical multi-touch application zooming is done by putting two fingers on the surface and increase/decrease the space between those fingers (pinching for the iPhone adepts). What happens is that the two fingers form the corners of a rectangle. If you move you fingers the box changes size, this can be used to calculate the zoom-factor. The result for the user will be that he zooms in or out on the image he is seeing. In theory he could select the corners of the object he wants to magnify and drag those corners to the corners of the table, enlarging the object to the size of the table.

However if there is another user who wants to drag the image a bit to see a different part this will give a conflict. The single-user environments cannot distinguish between the various users and will only see 3 pressure points on the table. The chances are that the software still assumes that a zooming operation is going on and it will incorporate the extra input in the rectangle. So if the new user starts dragging the map, he might accidentally decrease the size of the rectangle the first user is using to zoom in. So instead of zooming in or panning, the application will zoom out, confusing both users.

Obviously this is not the desired behavior. Since there is no way to distinguish the different users, the people around the table will need to take turns controlling the table. However it could be that for some reason you really want to take an eye on a specific area. So you want to keep that area visible all the time. The easiest solution would be to make sure that area never leaves the viewing window and as such limit the movement of the map image by other users. There is a much nice solution which is inspired by classic games as Transport Tycoon. More to follow …

Microsoft Research just released it’s fabulous InkSeine application. It’s a prototype application done by a research department and as such it lacks some features and streamlining but in general it’s cool. The biggest gripe I have that it’s very Microsoft centered, but coming from MS Research I can’t blame them.

In general if you have a tablet or an UMPC, which I both have at work, you can install this application to easily write notes, organize thoughts and link them with existing dat, like email, (word) docs, (windows) desktop search and (MSN) internet search (amongst others). It does remind me a bit of a combination of Beagle, TomBoy and InkScape and it does have some neat ideas how to work with Ink, Touch (although mainly pen-focussed) and mindmapping.

I really like their swipe-menu which makes it easy to select something: you hold you pen/finger on the icon and a circular menu appears, if you move your pen/finger in the direction of one of the items it is automatically selected. Once you know where the different menu items are, you just point and swipe, without waiting for the menu. This is a really nice way to work with a pen, especially for an application which you use a lot.

The IE icon in the center is the actual icon the circle-ish thingy is the menu, as you can see going up from the IE icon goes to cut. In all other swipe menus going up means cut/ delete/ remove/ destroy/ kill (nice career-statement

This swipe-menu does remind me of the Plectrae-menu I designed for our DiamondTouch table-top computer. (Plectrae is a pun on Plectrum and Latin plural rules, which obviously have been ignored.) A Plectrae-menu is triggered by any user touching the table and holding their finger still for a second or so, it provides the user with a set of options, which face the user in the correct orientation (more or less). This is important, since tabletop computing doesn’t have a bottom or top orientation so all information should be rotated towards the user.

Plectrae menu in action, shot by Martijn van Exel