BruTile @ the Surface

Once we got the basics running I tried to edit the XAML to make it look better. This turned out to be harder than expected. The surface has once again its own default designs, which in my opinion are worse than the WPF standard designs. The default look is grey and square. So I wanted to change it to something more akin to our other touchtable applications: white and round. However Paul used a ScatterView component to display the layerlist and the geodan logo. He had read that one can just drop components in a ScatterView, without the need to embed them in ScatterViewItems. This is true, but it is a nightmare for the (non-initiated) designer. What happens is that the grey-square design of the ScatterViewItem is drawn beneath any layout you come up with for your own item. This is a very puzzling behaviour, although not entirely unexpected once you know that the ScatterViewItem exists.

Once I solved this issue I could seriously try to change the layout of the ScatterViewItems. Here however real glitches appeared. We use rounded borders and dropshadows for our items. As you can see below there are (at least) two issues:

Rounded border on the Surface don't work

The dark rectangle in the top of the image is actually a shadow of  a ScatterViewItem which for some reason doesn’t start below it. However this makes it easy to see the first glitch in the rendering. As you can see there is a thin dark line near the edge, which makes a weird square corner at the corner. This shouldn’t be there and when you resize the ScatterViewItem down, the shadow becomes darker (as it should have been in the first place) and the line is not visible anymore. If you don’t use rounded corners on the shadow the line is still there, but harder to notice because it is exactly following the edge of the shadow.

The second annoying glitch I’ve found is the gap between the border and the content. If we zoom in a bit on the layercontrol you can see there is a pixel space between the border and the content:

A gap between border and content

So far I haven’t been able to eliminate the pixel wide gap. It doesn’t look bad in our application right now, but in general it is bad that there is such an uncontrollable gap. So I have the feeling that the Surface people are not a big fan of borders in general and rounded border specifically.

We spend quite some time calibrating, recalibrating and reading documentation from the invite only Microsoft site without any effect. In the end Vista asked us to install an update and to reboot the Surface. When it was rebooted we ran the ‘Surface Shell’ application from the Desktop and suddenly it worked!!

The Surface, or at least our version, comes with a set of code examples. We tried several to see how well it works and found out that especially the Grand Piano demo is a crowd drawer (mainly due to the sound probably). The Piano is a limited set of piano keys in perspective view. ‘Pressing’ a key results in a tone, just like a real piano

Playing piano on a Surface

The sensitivity of the Surface is well balanced. The Surface doesn’t notice fingers which are hovering just above the surface and you don’t need to apply any pressure to get noticed once you touch it. We’ve built our own table using a similar FTIR technique and we had quite some problems getting that balance right. Also the Surface is using 5 camera’s where we used just one as such it’s resolution is better as well.

It has additional logic in the interpretation of the camera feeds. It has three different types of touches it can recognize: finger, blob and tag. From each type it gives a location, an orientation, an id and in case of the tags a value.

Surface information of a finger

The difference between a blob and a finger is the size and shape of the touch, for instance a small square object is seen as a blob and a larger oval object is seen as a finger. Furthermore the earlier white-only bias of the Surface we experienced is now changed to a no-black bias. This gives some interesting results; a black/orange marker on the surface is recognized as a single finger at the tip of the marker;

Fingers come in strange shapes

The Surface feels pretty responsive, but using the Data Visualizer you notice that the position of the touch as interpreted by the Surface lags behind the actual touch. In the demos it is not noticeable but we found a similar lag very annoying on the DiamondTouch. On that table we managed to get rid of the lag and hopefully we can do the same here.

There is the famous demo of the telephone on the Surface, where it gets recognised and you can drag and drop data. We tried it and noticed that it doesn’t like android phones; it just doesn’t see the phone. An (ancient) windows mobile phone however does get recognised for what it is: a blob;

Android on the Surface

Obviously the Surface doesn’t automagically recognize phones or any other object. The table comes with a set of fiducials which you can stick on objects. These fiducials get recognised as tags with a value and using that value you can do fun stuff in your software.

Update

I discovered that while it doesn’t respond to hovering fingers, it does notice hovering areas the size of a hand. Which is slightly annoying since in normal use you easily have your hand hovering just above the surface while one or more fingers touch it. It might be possible to filter this unwanted behavior in real applications, because the hand shows up as a blob instead of a finger.

We were supposed to have two normal and one developer edition but it wasn’t very clear which one was which so we picked one and carried it to our office. This turned out to quite tricky, this thing is heavy and doesn’t have any clear handles. In the end we carried it by holding the projection-surface …

Once setup we had to fiddle a bit to get the powercable connected and find the proper (two) powerswitches. Once it booted it prompted us an EULA we had to accept. However the Surface is designed to be a standalone machine completely controlled by touch but before the touch was working we had to accept the EULA, which was only accessible by touch … hm

You need to accept to start touch, and you need touch to accept

So we had to plug in a mouse, but apparently we hadn’t chosen the developer edition since no usb ports where visible. The manual explained that we need to open up the side of the table to access the ports. After two screws and a bit of force we managed to open up the side and it gave us full access to all normal PC ports.

The surface insides

It asked us to create an admin account and once it was setup it gave us the choice to login as that admin or as TableUser, since we had no clue what the password for TableUser was, we logged in as admin.  After a bit we were shown a fresh Vista desktop with a few shortcuts. The most obvious for us was the calibration tool. We calibrated our table and started a touch application. It failed to run complaining that the filtergraph was not found at the location specified. We neither had knew what a filtergraph was, nor what the location specified was.  This got us stuck for quite a while, the manuals didn’t help and the online site with information was only accessible by invitation and due to some miscommunication(?) we were not (yet) invited.

Digging into things like registries, hidden folders and binary blobs we finally discovered that there are two versions of the calibration tool. One which does a background-light calibration and one which does a per camera calibration. We had run both, but we failed to notice that the latter had to run 5 times (5 cameras in the surface). Once we had done so the different surface programs ran without errors and without effect.

Pretty annoyed we banged our fist on the table and that seemed to have effect. It turned out that putting a lot of force on the table gave results, indicating a calibration issue. Having no idea how to finetune the calibration we started to experiment with different ways of touching. The current consensus is that it is designed for ubergeeks who never see the sun and are pure white; touching the Surface with a finger doesn’t give a result, wrapping the finger in white tissue suddenly does …

The 'attract' application

Hopefully we will figure out soon how to tweak the calibration and otherwise we need to buy a set of white surgical gloves to hand out to users

Basic animation of hiding a card from view

Most examples I know which implement two handed touch interfaces use symmetrical actions. The left hand is doing the same as the right hand, or in case of the iPhone they are fingers of the same hand. I’m more interested in asymmetrical bimanual actions, preferably using physics. These mirror the natural way we use our hands in different tasks, or at least I hope so.

My initial brainstorm somewhere late at night gave me two possible implementations. Using you left hand to fix the object you are manipulating and the right hand to rotate it. This way you can rotate it around a different point than its centroid (lower action). This actions very much mirrors the two multi-touch rotation I disliked earlier . Another implementation is to use the left hand to hold the object and the right hand to scroll its content. (upper action). This action doesn’t have a very clear added value over more traditional approaches either, but it is a start. One benefit of this approach is that you can manipulate both the object and the content without the need of titlebars and scrollbars.

Two examples of asymmetrical bimanual touch gestures.

more to come…

Enter tabletop computing…

Since tabletop computing hasn’t developed a standard interface, yet, we still have a chance to create new metaphors which will stick. Apple has managed to get a serious grip on multi-touch interface design. Many people believe that the multi-touch interface has empowered the designer to create such a novel interface. However it is my believe that only a few actions really need multi-touch, zooming being one. I don’t think people really want to use two hands to control a computer when they can do it with one hand. Since tabletop computing is much more akin to the classic desktop we can have another look at this metaphor. To rotate a piece of paper you don’t put your two index fingers on the sheet of paper and start rotating. If you want to rotate it 180degrees you probably don’t even try to turn it out of your wrist. Instead you grab it at a corner and give it a twist. – To be honest, since you can manipulate the piece of paper in 3D it is most likely that you grab the piece of paper from the table and turn it in the air -

However if we stick to the concept of 2D manipulation we can use it to enhance the current metaphor. Even if you don’t know Newton’s laws of motion, you still expect them to work. Turning a piece of paper by putting a finger in the corner and moving it, whilst counting on the inertia and drag of the paper to rotate instead of translate the paper doesn’t require any degree in physics. Few people would know which forces are at work on the paper, even fewer do care, only the result counts. Why does one not use a multi-touch gesture as used in the iPhone? Basically because it is more complicated. Look at the very simple animation below. The upper user is using physics to turn its object, while the lower user is using multi-touch. As you can see the upper user can keep on spinning if he likes, while the lower user runs into a wrist issue.

manipulating object with and without pysics

While in real life we don’t need to understand the Newtonian laws to be able to use them, in the software world it is not that simple. Since there is no natural gravity within the computer, the Newtonian laws don’t apply naturally either. We need to introduce physics into the GUI. Fortunately physics is very important in current games and a lot of effort has been put in creating physics engines which recreate the Newtonian world of physics in the virtual world of the computer. We are planning to use Farseer, since it is both open source and runs in C#. Having physics at our finger tips we can create much cooler interfaces. Instead of having two seperate actions to translate and rotate an object, you can do it in one fluid motion (upper user). Moving an object is also much easier, just give it a push in the right direction (lower user).

Using physics to manipulate objects – please note that the third hand of the lower user doesn’t touch the table anymore -

There is an area though where you have similar scenarios: Gaming, especially RTS and certain simulation games. You need to keep track of different units and at the same time require a detailed zoom level which will exclude certain units. Although these games are rarely, if at all, multi-user in the sense of multiple users behind 1 computer at the same time, it does provide some interesting ideas. This is a screenshot of Transport Tycoon (from here):

It has the main mapview and two smaller mapviews. One with an overview of the entire area and one that tracks a train. Using mini-maps you could track particular vehicles are keep an eye on a specific location. Also you can use it to look something up on a different part of the map, while the main stays in view for the others. So the tug-of-war action, as described earlier, suddenly is possible without the tugging or warring bit.

One user is still zooming in, where as the other user has his own mini-map where he can drag the map at the same time. If anything interesting occurs in one of the mini-maps it can become the main view, where as the former main view becomes a mini-map itself.

On a typical multi-touch application zooming is done by putting two fingers on the surface and increase/decrease the space between those fingers (pinching for the iPhone adepts). What happens is that the two fingers form the corners of a rectangle. If you move you fingers the box changes size, this can be used to calculate the zoom-factor. The result for the user will be that he zooms in or out on the image he is seeing. In theory he could select the corners of the object he wants to magnify and drag those corners to the corners of the table, enlarging the object to the size of the table.

However if there is another user who wants to drag the image a bit to see a different part this will give a conflict. The single-user environments cannot distinguish between the various users and will only see 3 pressure points on the table. The chances are that the software still assumes that a zooming operation is going on and it will incorporate the extra input in the rectangle. So if the new user starts dragging the map, he might accidentally decrease the size of the rectangle the first user is using to zoom in. So instead of zooming in or panning, the application will zoom out, confusing both users.

Obviously this is not the desired behavior. Since there is no way to distinguish the different users, the people around the table will need to take turns controlling the table. However it could be that for some reason you really want to take an eye on a specific area. So you want to keep that area visible all the time. The easiest solution would be to make sure that area never leaves the viewing window and as such limit the movement of the map image by other users. There is a much nice solution which is inspired by classic games as Transport Tycoon. More to follow …