Using TUIO to connect multi-touch hardware to multi-touch applications

To be able to develop multi-touch applications you must be in possession of multi-touch hardware. However, there hardly doesn’t exist any off-the-shelve multi-touch hardware with which you could start developing multi-touch applications right away. The only product in my mind that enables you to develop customised multi-touch applications for it would be the Lemur from Jazzman. But 1000$ for a Lemur mixer is not too cheap and development possibilities are constricted.

Figure 1: Lemur from JazzMutant

The result is that you have to develop your own multi-touch hardware. Lucky enough there are those pals from the Nuigroup that support you with manuals, with a software called Touchlib and with billboards where you can ask (stupid) questions and they stay patient with you.

A major downside of developing your own hardware is that there are hardly any standards, which would describe a way to connect your hardware to your computer (drivers or similar stuff) or which would describe how you can connect your applications to your hardware (usually called an application programming interface -> API). The task to connect a multi-touch table to a computer can be solved by using the already mentioned software Touchlib. In terms of connecting your multi-touch hardware to your software there is at least the idea of a standardised way: TUIO (A Protocol for Table-Top Tangible User Interfaces) invented by reactable software.

TUIO describes a way to convey touch information from your hardware over the network to your application. In figure 2 you can see how TUIO can be used e.g. with Flash:

• The multi-touch sensing hardware is connected to computer A. Computer A runs e.g. Touchlib and extracts touch information from the data yielded by the sensing hardware. Afterwards Touchlib wraps this information in TUIO data and sends this data via network to computer B.
• Computer B runs a Flash application. It receives the TUIO data from computer A (via AS3 sockets) and includes this data in the Flash application.
• The multi-touch screen displays the Flash application from computer A.

Figure 2: How TUIO integrates within multi-touch applications

Why would you want to use something like TUIO? Why does not use the computer to which the multi-touch hardware is connected to run the (Flash) application? Possible answers:

• You want to use your applications with different multi-touch hardwares. With TUIO you could easily exchange the underlying hardware if both hardwares sent their touch information via TUIO. In our case we could develop a volvelle flash application and we could use it with both an FTIR multi-touch table OR with a Wiimote based IR pen solution. Just connect the volvelle application to the TUIO enabled hardware of your choice and you’re done.
• Calculation of touch information can be a demanding task: Imagine two or more persons touching an FTIR table with both of their hands. The computer to which the FTIR table is physically connected has to calculate the touch information via image processing with the Touchlib (touch information gets extracted from a video camera image). Even with the newest hardware every bit of processing power could be needed for this task. On the other hand using touch information of e.g. 20 fingers in your Flash application is also a demanding task for which you also may want to use all availabe processing power of another computer.
• But you still can run both applications on one computer!
• Exchangable software: Imagine 100 development teams developing multi-touch software. Wouldn’t it be nice to exchange software with them? Or to create a repository of multi-touch enabled software? To be able to include your own application in a show-case of a major visualisation agency? At least I do think so!
• There should be tons of arguments more but I leave it with those indicated above.

How can you use TUIO in your own applications?

This one might be the core bit of information that is interesting for you. How can you implement the network protocol that TUIO describes in your own application? The answer is you don’t need to. The guys from Reactable provide all necessary core applications for TUIO on their homepage. If you want to use TUIO within your Flash application simply use this core template. To develop and test your TUIO applications without available multi-touch hardware the TUIO simulator should be a great help for you. However, the TUIO simulator demands a Java Runtime (1.6 or higher) on your computer.

Update: TUIO is based on the Open Sound Control (OSC) protocol. As explained on Seth Sandler’s blog Flash needs an additional component to talk with OSC components i.e. TUIO enabled hardware:

“You also need FLOSC (flash open sound control) which allows Flash to connect to programs that send OSC messages like the ones above. You can get FLOSC here. You will need java installed in order to run FLOSC.”

Furthermore, Seth Sandler’s downloads page gives a nice overview on how to set up TUIO with Flash.

There is an excellent Getting Started with TUIO and Flash post on the nuigroup forum.

For some videos of our Flash multi-touch applications also visit:

• Digital Music Volvelles
• Mapstouch (A Google Maps Multi-touch Application)
• Flickr multi-touch photo viewing application