AR Puppet - Animated Agents in Mixed Reality Environments

Body and facial gestures as well as speech are familiar and widely accepted means of human communication. Animated characters, often with autonomous and affective behavior, have proven to be useful in man-machine communication since they are able to exploit and deliver information through multimodal channels and thus engage the user in a natural conversation. Autonomous agents have been actively researched in recent years as an interface to computerized systems bridging the communication gap between man and computer, and the real and virtual world. Augmented Reality (AR) applications share the same goal through enhancement of the real environment with useful virtual information, where virtual objects appear to coexist with the real world.

   Download my PhD thesis "Ubiquitous Animated Agents for Augmented Reality" (2006) (5 Mb)  
   See my publication list  

Quick links

AR Lego Monkeybridge UbiAgent
AR tracker calibration tool AR piano tutor AR character rotoscoping
AR videoconferencing Cascading hand and eye movements Animated talking head

AR Lego

  This project is an important pilot application for the use of AR agents. It implements a machine maintenance scenario employing two AR agents to educate an untrained user to assemble, test and maintain machines composed of active (engines and sensors) and passive (cogwheels, gears, frames) parts. The two agents are a real, augmented LEGO Mindstorms robot and a virtual repairman. The animated virtual repairman serves as an autonomous telepointer and explains about the robot with expressive, natural gestures in the vicinity of significant physical locations during the construction steps. By monitoring the attributes of the LEGO robot and the current status of assembly application, character behavior is automatically generated for the repairman, which is synchronized to the physical actions triggered on the physical robot (turning engines on/off, setting direction/power, querying various sensors, etc.).

Selected publication:
I. Barakonyi, T. Psik, D. Schmalstieg, "Agents That Talk And Hit Back: Animated Agents in Augmented Reality", Proc. of IEEE and ACM International Symposium on Mixed and Augmented Reality 2004 (ISMAR'04), Arlington, VA, USA, pp. 141-150, November 2004.
 

Downloads:

  PDF file       YouTube video

Monkeybridge

  Lifelike animated characters have received significant attention in recent years. Augmented Reality as a new a media demands broader acceptance, which can be achieved by widening its application base to include entertainment and education. In this project we show how animated agents make autonomous decisions based on their observation of an AR environment in which they are embedded. Monkeybridge is a multiplayer game, where users place real and virtual objects onto a physical surface, thus influencing the behavior of the characters.The characters autonomously choose: the path the walk on; decide how to get from one platform to the other, e.g. climb or jump when there is a slight difference in height between platform edges; automatically choose the straightest path from several available tiles; and fall into the water if there is no suitable piece of landing stage to walk on. The game includes many spectacular virtual and physical visual elements such as animated 3D ocean waves, a flock of virtual seagull boids, a physical volcano with real smoke and a lighthouse with rotating lights. Sound effects enhance the game experience.

Selected publication:
I. Barakonyi, M. Weilguny, T. Psik, D. Schmalstieg, "MonkeyBridge: Autonomous Agents in Augmented Reality Games", Proc. of ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE'05), Valencia, Spain, pp. 172-175, June 2005.
 

Downloads:

  PDF file       YouTube video

Animated agents in a UbiComp environment

  Most of today's Augmented Reality (AR) systems operate as passive information browsers relying on a finite and deterministic world model and a predefined hardware and software infrastructure. We propose an AR framework that dynamically and proactively exploits hitherto unknown applications and hardware devices, and adapts the appearance of the user interface to persistently stored and accumulated user preferences. Our framework explores proactive computing, multi-user interface adaptation, and user interface migration. We employ mobile and autonomous agents embodied by real and virtual objects as an interface and interaction metaphor, where agent bodies are able to opportunistically migrate between multiple AR applications and computing platforms to best match the needs of the current application context. We present two pilot applications to illustrate design concepts.

Selected publication:
I. Barakonyi, D. Schmalstieg, "Ubiquitous Animated Agents for Augmented Reality", Proc. of IEEE and ACM International Symposium on Mixed and Augmented Reality 2006 (ISMAR'06), Santa Barbara, CA, USA, pp. 145-154, October 2006.
 

Downloads:

  PDF file       YouTube video

AR calibration assistant for an ultra-wideband tracking system

  We have developed a hand-held augmented reality platform exploiting a combination of multiple sensors built around an ultra-wideband tracking system. We illustrate how an environment exploiting this platform can be set up with a technician-support application, which provides intuitive in-situ instructions on how a wide area tracking system should be configured. The use of 3D registered graphics greatly assists in the debugging of common awkward use cases involving reflections off metal surfaces.

Selected publication:
J. Newman, G. Schall, I. Barakonyi, A. Schuerzinger, D. Schmalstieg, "Sentient Environments for Augmented Reality", In Advances in Pervasive Computing, Pervasive 2006, Dublin, Ireland, May 2006.
 

Downloads:

  PDF file       YouTube video coming soon

AR Piano Tutor

  In this project virtual objects are superimposed on top of a real MIDI keyboard. The input and output channels of the keyboard are connected to a computer via a MIDI interface. The application serves as a piano teacher that teaches about basic chords, scales, etc. The keys pressed by the "student" user, the order and the timing of the piano key presses are all captured by the computer with the help of the MIDI interface, therefore the teacher can immediately "check" whether the student has correctly followed the instructions and may give instant visual feedback over the real keyboard (e.g. marking wrongly pressed or missed keys with different colors). The application is based on a "fishtank" AR setup (PC+monitor+webcam), where the physical MIDI keyboard is tracked with the help of an optical marker.

Selected publication:
I. Barakonyi, D. Schmalstieg, "Augmented Reality Agents in the Development Pipeline of Computer Entertainment", Proc. of 4th International Conference on Entertainment Computer (ICEC'05), Sanda, Japan, pp. 345-356, September 2005.
 

Downloads:

  PDF file       YouTube video

Rotoscoping techniques exploiting AR

  Modellers and animators often rely on real-life references to build and animate 3D characters for games or film production. Observing the real world by photographing or videotaping the subject, or asking someone to pose helps create more believable, precise and expressive character animation. Since physical and virtual objects appear to coexist in an Augmented Reality (AR) environment, new possibilities in character animation open up. The animated virtual model and the real-world reference can be merged into one, interactive modeling tool. The main goal of this project is to develop such a tool using pose-tracked physical objects as input devices to animate skeleton-based virtual 3D models. AR enables not only a close interaction with the virtual model by using tangible objects but also the creation of complex motions like walking up stairs or lifting a ball, since the animator can use the actual physical models of the stairs or the ball together with the character to create the motion.

Selected publication:
I. Barakonyi, D. Schmalstieg, "Augmented Reality in the Character Animation Pipeline", ACM SIGGRAPH 2006 Sketches, Boston, MA, USA, July 2006.
 

Downloads:

  PDF file       YouTube video

AR videoconferencing

  The Augmented Reality Videoconferencing System is a novel remote collaboration tool combining a desktop-based AR system and a videoconference module. The novelty of this system is the combination of these tools with AR applications superimposed on live video background displaying the conference parties' real environment, merging the advantages of the natural face-to-face communication of videoconferencing and AR’s interaction capabilities with distributed virtual objects using tangible physical artifacts. The simplicity of the system makes it affordable for everyday use. The system is based on concurrent video streaming, optical tracking and 3D application sharing. The technology used provides yields superior quality compared to pure video streaming with successive optical tracking from the compressed streams. The system's collaborative features are demonstrated with a volume rendering application that allows users to display and examine volumetric data simultaneously and to highlight or explore slices of the volume by manipulating an optical marker as a cutting plane interaction device.

Selected publication:
I. Barakonyi, T. Fahmy, D. Schmalstieg, "Remote Collaboration Using Augmented Reality Videoconferencing", Proc. of Graphics Interface 2004 (GI'04), London, ON, Canada, pp. 89-96, May 2004.
 

Downloads:

  PDF file       YouTube video

Cascading Hand and Eye Movement for AR Videoconferencing

Research done at the National Institute of Informatics, Tokyo




  We have implemented an augmented reality videoconferencing system that inserts virtual graphics overlays into the live video stream of remote conference participants. The virtual objects are manipulated using a novel interaction technique cascading bimanual tangible interaction and eye tracking. User studies prove that our user interface enriches remote collaboration by offering hitherto unexplored ways for collaborative object manipulation such as gaze controlled raypicking of remote physical and virtual objects.

Selected publication:
I. Barakonyi, H. Prendinger, D. Schmalstieg, M. Ishizuka, "Cascading hand and eye movement for augmented reality videoconferencing", Proc. of 2nd IEEE Symposium on 3D User Interfaces 2007 (3DUI'07), Charlotte, NC, USA, pp 71-78, March 2007.
 

Downloads:

  PDF file       YouTube video

Animated talking head for multimodal presentations

Research done at the Ishizuka Lab at the University of Tokyo



  This project presents a three-dimensional agent with synthesized face, speech and behavior. The behavior synthesis is semiautonomous, which means the agent is capable of generating facial expressions, head and eye motion automatically with a little user support of defining behavioral rules and annotating text for the speech. Besides standard input and output devices we utilize speech synthesis and recognition for user interaction. The agent is implemented as a scriptable, individual object with a well-defined interface, which enables scripting from other applications. As the system uses external parameter files - locally stored or distributed over the Internet -, authors can freely modify both the visual appearance and the behavior. The described features allow for a versatile use of the agent, for instance performing multimodal presentations without a human presenter, making conversations with the user or acting as an interactive teacher in distance learning.

Selected publication:
N.P. Chandrasiri, I. Barakonyi, T. Naemura, M. Ishizuka, H. Harashima, "Internet Communication Using Real-time Facial Expression Analysis and Synthesis", IEEE Multimedia, Vol.11, No. 3, pp. 20-29, July-Sept 2004.
 

Downloads:

  PDF file       YouTube video

Publications

  1. Alessandro Mulloni, Daniel Wagner, Istvan Barakonyi, Dieter Schmalstieg, "Indoor Positioning and Navigation with Camera Phones", In IEEE Pervasive Computing, Vol. 8, No. 2, pp. 22-31, April-June 2009. Link to publisher

  2. Dieter Schmalstieg, Gerhard Schall, Daniel Wagner, Istvan Barakonyi, Gerhard Reitmayr, Joseph Newman, Florian Ledermann, "Managing Complex Augmented Reality Models", In IEEE Computer Graphics and Applications, Vol. 27, No. 4, pp. 48-57, July-August 2007. Download (347Kb)

  3. Istvan Barakonyi, Dieter Schmalstieg, Helmut Prendinger, Mitsuru Ishizuka, "Cascading Hand and Eye Movement for Augmented Reality Videoconferencing", In Proc. IEEE Symposium on 3D User Interfaces (3DUI'07), Charlotte NC, USA, March 2007. Download (3.3Mb)

  4. Istvan Barakonyi, Dieter Schmalstieg, "Ubiquitous Animated Agents for Augmented Reality", In Proc. of IEEE and ACM International Symposium on Mixed and Augmented Reality 2006 (ISMAR'06), Santa Barbara, CA, USA, October 2006. Download (0.96Mb)

  5. Istvan Barakonyi, Dieter Schmalstieg, "Augmented Reality in the Character Animation Pipeline", SIGGRAPH sketch, SIGGRAPH 2006, Boston, MA, USA, August 2006. Download (2.8Mb)

  6. Joseph Newman, Gerhard Schall, Istvan Barakonyi, Andreas Schuerzinger, Dieter Schmalstieg, "Sentient Environments for Augmented Reality", demo submission paper, In Advances in Pervasive Computing, Pervasive 2006, Dublin, Ireland, May 2006. Download (2.97Mb)

  7. Florian Ledermann, Istvan Barakonyi, Dieter Schmalstieg, "Abstraction and Implementation Strategies for Augmented Reality Authoring", book chapter in Emerging Technologies of Augmented Reality: Interfaces and Design, (M. Haller, B. Thomas, M. Billinghurst eds.), Idea Group Publishing, to be published in 2006. Download (231Kb)

  8. Istvan Barakonyi, Dieter Schmalstieg, "Augmented Reality Agents in the Development Pipeline of Computer Entertainment", Proc. of the 4th International Conference on Entertainment Computer (ICEC'05), Sanda, Japan, Sept. 19-21, 2005. Download (2.6Mb)

  9. Istvan Barakonyi, Markus Weilguny, Thomas Psik, Dieter Schmalstieg, "MonkeyBridge: Autonomous Agents in Augmented Reality Games", Proc. of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE'05), Valencia, Spain, June 15-17, 2005. Download (1.3Mb)

  10. Istvan Barakonyi, Dieter Schmalstieg, "Exploiting the Physical World as User Interface in Augmented Reality Applications", Proc. of the IEEE Virtual Reality 2005 Workshop on New Directions in 3D User Interfaces, Bonn, Germany, March 12, 2005. Download (1.6Mb)

  11. Istvan Barakonyi, Thomas Psik, Dieter Schmalstieg, "Agents That Talk And Hit Back: Animated Agents in Augmented Reality", Proc. of the IEEE and ACM International Symposium on Mixed and Augmented Reality 2004 (ISMAR'04), Arlington, VA, USA, Nov. 2-5, 2004, pp. 141-150. Download (1.4Mb)

  12. Istvan Barakonyi, Tamer Fahmy, Dieter Schmalstieg, "Remote Collaboration Using Augmented Reality Videoconferencing", Proc. of Graphics Interface 2004 (GI'04), London, ON, Canada, May 17-19, 2004, pp. 89-96. Download (701Kb)

  13. Naiwala P. Chandrasiri, Istvan Barakonyi, Takeshi Naemura, Mitsuru Ishizuka, Hiroshi Harashima, "Internet Communication Using Real-time Facial Expression Analysis and Synthesis", IEEE Multimedia, Vol.11, No. 3, July-Sept 2004, pp. 20-29. Download (1.98Mb)

  14. Istvan Barakonyi, Dieter Schmalstieg, "AR Puppet: Animated Agents in Augmented Reality ", Proc. of First Central European International Multimedia and Virtual Reality Conference, pp. 35-42, Veszprém, Hungary, May 6-8, 2004. Download (295Kb)

  15. Istvan Barakonyi, Tamer Fahmy, Dieter Schmalstieg, Karin Kosina, "Collaborative Work with Volumetric Data Using Augmented Reality Videoconferencing", Demo Paper, Proc. of the IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'03), Tokyo, Japan, October 2003. Download (212Kb)

  16. Istvan Barakonyi, Werner Frieb, Dieter Schmalstieg, "Augmented Reality Videoconferencing for Collaborative Work", Proc. of the 2nd Hungarian Conference on Computer Graphics and Geometry, Budapest, Hungary, June 30 - July 1, 2003.

  17. Naiwala P. Chandrasiri, Istvan Barakonyi, Takeshi Naemura, Mitsuru Ishizuka and Hiroshi Harashima, "Communication over the Internet using a 3D Agent with Real-time Facial Expression Analysis, Synthesis and Text to Speech Capabilities", Proc. of the IEEE International Conference on Communication Systems (ICCS 2002), Singapore, November 2002. Download (399Kb)

  18. Istvan Barakonyi, Naiwala P. Chandrasiri, Sylvain Descamps, Mitsuru Ishizuka, "Communicating Multimodal Information on the WWW Using a Lifelike, Animated 3D Agent", Proc. of the PRICAI'02 workshop, Tokyo, Japan, August 2002 and the 1st Hungarian Conference on Computer Graphics and Geometry, Budapest, Hungary, May 2002. Download (145Kb)

  19. Sylvain Descamps, Istvan Barakonyi, Mitsuru Ishizuka, "Making the Web Emotional: Authoring Multimodal Presentations Using a Synthetic 3D Agent", Proc. of the OZCHI 2001 Conference, Perth, Australia, November 2001, pp.25-30. Download (246Kb)

  20. Istvan Barakonyi, Mitsuru Ishizuka, "A 3D Agent with Synthetic Face and Semiautonomous Behavior for Multimodal Presentations", Proc. of the Multimedia Technology and Applications Conference (MTAC2001, IEEE Computer Society), Irvine, California, USA, November 2001, pp.21-25. Download (247Kb)