Volvelles multi-touch hardware is now working

What you see in the video is the hardware for the Digital Volvelles project. The app that is being used is just a Flash application that is bundled with the touchlib project: 

 

Impressions ‘New’ Table

How to connect a Wii Remote to a Mac in Bootcamp

I tested the procedure to connect a Wii Remote (Wiimote) via Bluetooth with both a Mac Mini as well as a MacBook in Bootcamp with Windows XP. However, I cannot guarantee that it works with all Macs in Bootcamp. Definitely a problem is, that I’ve got only a German XP for testing purposes. Thus I can’t guarantee that I found the right translations for the names of buttons and labels in the XP Bluetooth wizard.

1. In “Systems Preferences” go to “Bluetooth devices”
2. If there already is a Wiimote listed, press “Remove” (yes, it’s tedious but I could not figure out another way.)
3. Click “Add Device”
4. Select “Device is ready to be found”
5. On your Wiimote click both 1 & 2
6. Immediately click “Search devices” in Bluetooth devices window.
7. After Bluetooth devices have been found select “Nintendo RVL…” and click Next.
8. Choose “No key necessary” and click “next”. Attention: Wiimote must still be blinking after this step. If you’ve done everything right a bubble should appear over Windows’ task bar which tells you “New HID device found”.
9. Start Wii yourself’s demo.exe. A Dos shell window opens up with the message “Searching for Wiimote”
10. Press 1 & 2 buttons of Wiimote if not blinking. Attention: demo.exe should now tell you “Connected!” behind “Searching for Wiimote”.
11. Close demo.exe.
12. Open up the application in which you want to use your Wiimote e.g. Johnny Chung Lee’s Wii Whiteboard. Attention: I was not able to use the exe that was provided on the download page of the project’s Sourceforge page. I had to open up the project in Visual Studio (I used the VS Express 2008 version that is freely available on Microsoft’s homepage) and press the play button in the menu bar (next to “debug”). Then everything worked just fine.

Update: Just figured out that I’ve forgot to upload Demo.exe. This is a Dos application, which shows the basic functionalities of the WiiLib project when a Wii remote is connected to the computer. Anyway you can download the source yourself from their page and compile it with Visual Studio (a VS project file is included) but somehow the compiled Demo.exe failed to work on some computers. So if you just take my version of Demo.exe you will be faster and the possibilty that you will be successful is higher. Demo.exe

 

I hope your virus scanner will try to kill if you’re about to open a downloaded, zipped exe from the Internet 😉

For some videos of our Flash multi-touch applications also visit:

• Digital Music Volvelles
• Mapstouch (A Google Maps Multi-touch Application)
• Flickr multi-touch photo viewing application

Multi-touch Table Progress

I’ve been able to get hold of an old frame made of perforated metal plates. I reassembled the parts in a way that I’ve got a frame for my multi-touch display. Currently it has the table form factor but it can easily be readjusted to be a console or even a panel.

The multi-touch display itself is made of an acrylic glass plate in which infrared light is being emitted from the sides by 24 infrared leds. Thus the light is being endlessly reflected internally in the plate. If a finger is touching the plate the optical effect of light frustration will take place that effects light to be emitted vertically on the frustration place (the point where your finger tip is touching the display). Onto the back of the display where a back projection foil is mounted an image is projected by a video beamer (a Panasonic PT-AX200E). The display is being scanned by a webcam with a removed infrared light filter (a Philips SPC900NC with this lens). A Mac Mini is producing the image that is projected by the beamer onto the display and is doing touch calculations with the help of Touchlib.

Here are some pictures from the current state of it:

Multi-touch Table First Pictures

Here are some pictures of my first multi-touch table attempts:

Multi-touch applications with Flash

I’ve written in a previous post that there is the idea of a standardised technique to connect multi-touch hardware to multi-touch software via TUIO. As we want to concentrate on Flash as visualisation and interaction layer for our digital volvelles I will present a Flash AS3 API for multi-touch with TUIO in this and further posts.

The touchlib project from the NUI Group is a collection of software tools for the realisation of a multi-touch table. It provides tools like digital image processing for the tracking of fingers on a multi-touch screen, a TUIO server and so on. It additionally provides a Flash AS3 library in order to process TUIO messages inside of Flash and to provide standard interface elements e.g. sliders.

All Flash applications that are being built with touchlib’s AS3 library can not only be used with a TUIO enabled multi-touch device but also with a mouse. This offers advantages in the development process as you do not (necessarily) need a multi-touch device in order to use your application. This will be very useful if more than one person are involved in the development process (as in our case a designer, a Flash developer and a hardware developer are involved) because everyone can get a feeling for the interaction with the application without the need for a colocated multi-touch device.

The touchlib project including the mentioned AS3 library can be downloaded here. The AS3 library with demo applications can be found in the following folder within the zip file: touchlib -> AS3 -> src.

For some videos of our Flash multi-touch applications also see:

• Digital Music Volvelles
• Mapstouch (A Google Maps Multi-touch Application)
• Flickr multi-touch photo viewing application

Links for more information at the NUI Group regarding the following topics:

• Getting Started with Flash Development and AS3
• Getting Started with TUIO and Flash
• AS3 Lbraries & APIs

First test with digital Wii Whiteboard

What do I do in the video?

1. Calibrate the Wii Whiteboard application
2. Use the IR Pen with MS Paint. In the second image I drew with Paint I wrote “Hello Mel” but the bad auto white balance function of my bad Nokia 6610 Navigator cam (which cam sucks extraordinarily!!!) prevented readability.
3. Use IR Pen with MS Explorer (NOT Internet Explorer)

I compiled Johnny Chung Lee’s Wii Whiteboard Application and used it with an infrared (IR) light pen that I built on my own (see figure 1).

I projected a videobeamer’s image onto white cardboard (the ones which are used for passepartouts) that stood on an easel. Before that I put grey cardboard onto the easel but the grey cardboard produced bad reflection of infrared light which caused bad tracking results by the Wiimote.

Figure 1: My IR pen (with old switch) and Wiimote with its “customised” stand

Anyway, this is just the kick-off for further actions with the Wiimote and IR pens but the setup for this was really hard to establish.
Finally: Guess to which music I was listening to!

Using TUIO to connect multi-touch hardware to multi-touch applications

To be able to develop multi-touch applications you must be in possession of multi-touch hardware. However, there hardly doesn’t exist any off-the-shelve multi-touch hardware with which you could start developing multi-touch applications right away. The only product in my mind that enables you to develop customised multi-touch applications for it would be the Lemur from Jazzman. But 1000$ for a Lemur mixer is not too cheap and development possibilities are constricted.

Figure 1: Lemur from JazzMutant

The result is that you have to develop your own multi-touch hardware. Lucky enough there are those pals from the Nuigroup that support you with manuals, with a software called Touchlib and with billboards where you can ask (stupid) questions and they stay patient with you.

A major downside of developing your own hardware is that there are hardly any standards, which would describe a way to connect your hardware to your computer (drivers or similar stuff) or which would describe how you can connect your applications to your hardware (usually called an application programming interface -> API). The task to connect a multi-touch table to a computer can be solved by using the already mentioned software Touchlib. In terms of connecting your multi-touch hardware to your software there is at least the idea of a standardised way: TUIO (A Protocol for Table-Top Tangible User Interfaces) invented by reactable software.

TUIO describes a way to convey touch information from your hardware over the network to your application. In figure 2 you can see how TUIO can be used e.g. with Flash:

• The multi-touch sensing hardware is connected to computer A. Computer A runs e.g. Touchlib and extracts touch information from the data yielded by the sensing hardware. Afterwards Touchlib wraps this information in TUIO data and sends this data via network to computer B.
• Computer B runs a Flash application. It receives the TUIO data from computer A (via AS3 sockets) and includes this data in the Flash application.
• The multi-touch screen displays the Flash application from computer A.

Figure 2: How TUIO integrates within multi-touch applications

Why would you want to use something like TUIO? Why does not use the computer to which the multi-touch hardware is connected to run the (Flash) application? Possible answers:

• You want to use your applications with different multi-touch hardwares. With TUIO you could easily exchange the underlying hardware if both hardwares sent their touch information via TUIO. In our case we could develop a volvelle flash application and we could use it with both an FTIR multi-touch table OR with a Wiimote based IR pen solution. Just connect the volvelle application to the TUIO enabled hardware of your choice and you’re done.
• Calculation of touch information can be a demanding task: Imagine two or more persons touching an FTIR table with both of their hands. The computer to which the FTIR table is physically connected has to calculate the touch information via image processing with the Touchlib (touch information gets extracted from a video camera image). Even with the newest hardware every bit of processing power could be needed for this task. On the other hand using touch information of e.g. 20 fingers in your Flash application is also a demanding task for which you also may want to use all availabe processing power of another computer.
• But you still can run both applications on one computer!
• Exchangable software: Imagine 100 development teams developing multi-touch software. Wouldn’t it be nice to exchange software with them? Or to create a repository of multi-touch enabled software? To be able to include your own application in a show-case of a major visualisation agency? At least I do think so!
• There should be tons of arguments more but I leave it with those indicated above.

How can you use TUIO in your own applications?

This one might be the core bit of information that is interesting for you. How can you implement the network protocol that TUIO describes in your own application? The answer is you don’t need to. The guys from Reactable provide all necessary core applications for TUIO on their homepage. If you want to use TUIO within your Flash application simply use this core template. To develop and test your TUIO applications without available multi-touch hardware the TUIO simulator should be a great help for you. However, the TUIO simulator demands a Java Runtime (1.6 or higher) on your computer.

Update: TUIO is based on the Open Sound Control (OSC) protocol. As explained on Seth Sandler’s blog Flash needs an additional component to talk with OSC components i.e. TUIO enabled hardware:

“You also need FLOSC (flash open sound control) which allows Flash to connect to programs that send OSC messages like the ones above. You can get FLOSC here. You will need java installed in order to run FLOSC.”

Furthermore, Seth Sandler’s downloads page gives a nice overview on how to set up TUIO with Flash.

There is an excellent Getting Started with TUIO and Flash post on the nuigroup forum.

For some videos of our Flash multi-touch applications also visit:

• Digital Music Volvelles
• Mapstouch (A Google Maps Multi-touch Application)
• Flickr multi-touch photo viewing application

An example for a Flash-based multi-touch application by Tim Roth

project page

via nui group