So it took approximately four weekends a bit of a brain jogging and I got some serious and stable marker tracking. The last big thing was actually to cut down processing. Thanks to some minor tweaks I squeezed out a 10fold increase.

Now the stats, measure with a 1280x1024 iSight on a 2.4GHz MacBook Pro:

• Speed: 119156.4 pixel/ms
• Normalized on 640x480 this are 2.56ms or a theoretical 390fps

The only bootleneck as of now is the video capture.

Results are available here