Knowing what is present in the real world and where, is a vital component for the effective operation of many Mixed Reality (MR) systems. Such information can be used to aid virtual object placement, occlusion culling, collision detection, or many other visual or simulated effects. However, building up and reconstructing this real world description, whilst also minimising user interaction and processing time, is a common hurdle for a Mixed or Augmented Reality application. The significant time and expertise required means that it is frequently performed prior to normal system operation in an off-line initialisation phase. As an alternative to more conventional approaches we have created the Mixed Reality Toolkit or MRT, which implements existing imaged based modelling techniques that can be rapidly applied over live video. The simple interactive primitive modelling techniques not only give an operator the opportunity to identify and reconstruct each object, component, volume or location in an imaged scene, but also perform these tasks in an almost instantaneous manner over live video images. 
Imaged scene modelled and calibrated using the MRT (left) and subsequently augmented with virtual characters (right). Many scenes contain reoccurring classes of object, such as chairs or tables. Therefore, instead of combining multiple primitive shapes to build up more complex shapes, by identifying, deforming and registering various classes of object to fit an image view the process can potentially be simplified. To demonstrate and examine how the MRT might be applied to this idea we adapted an existing Virtual Reality (VR) into an MR. 
Virtual Reality (VR) Scenario Turned into Mixed Reality Scenario: Original VR (left) turned into MR (right) with virtual audience integrated into real world scene. The original VR environment required human participants to interact with a virtual audience in a completely virtual world. Therefore, instead of the participants interacting with the virtual audience situated within the completely virtual world, we used the MRT to model the real environment to enable the integration of the virtual characters within the real world environment. [** video **] 
The MRT Used in Conjunction with Marker and Object Appearance Based Tacking: Real object modelled and clone relative to marker (top). Real object modelled relative to marker before being tracked directly with appearance based tracking and augmented with animated virtual worms (bottom). By further integrating the MRT with a hybrid of marker and object based tracking techniques we have also been able to demonstrate the power of the MRT to very rapidly and easily reconstruct and configure an MR system using a freely moving camera during the applications on-line run-time phase. [** video **] The toolkit consists of a small library of C++ classes and may be freely downloaded and used in accordance with the GNU Lesser General Public License. |
