The Wayang project

The Wayang project (2009-2013) was the inspiration for this site. It was one of the projects that I led while also serving as a manager of the Computational Social Cognition programme at the Institute of High Performance Computing in Singapore. I’ve been interested in “The Illusion of Life” in puppetry and animation for a long time, and when I learned that the Indonesian wayang tradition of puppet theater was a thousand years old, and can be translated as “theater” or “shadow”, the word seemed like a great title for the project — it hinted at the intellectual history of intention perception stretching all the way back to Plato’s Cave, while also connecting it to South East Asia.


  • David Pautler (proposed project and coded the initial version)
  • Andrew Ortony (contributed some of the initial ideas)
  • Karl Fua (contributed some of the initial ideas)
  • Boon-Kiat Quek (contributed some of the initial ideas)
  • Bryan Koenig (designed and analyzed all the supporting surveys and experiments; contributed several ideas)
  • Edwin Wirawan (took over coding the cognitive model; contributed several ideas)
  • Kumaresh (coded the visualizer; contributed several ideas)
  • Kum Seong Wan (assisted Bryan with the empirical work)
  • Sepideh Sadeghi (assisted Edwin with some of the mathematical techniques for recognizing and extrapolating curved trajectories from just a few observed positions in successive frames)

Technical synopsis

The project began as a computational cognitive model based on logical abduction applied to each incoming frame of an animation. Abduction was implemented via a bottom-up incremental chart parser, and the “grammar” was a set of causal rules structured like a “feature grammar” so that many kinds of constraints could be propagated up the parse trees. See our overview paper in the journal Behavior Research Methods.

Source code

  • Wayang – the abductive inference engine and the causal rules it uses to infer intentions or physical causes for the trajectories of animated objects. Written in ECLiPSe/Prolog and Java
  • ATOM – an inference engine that supports forward and backward chaining (aka, deduction and querying), and which generates traces that can be used for Test-Driven Development (TDD) of inference rules. Written in ECLiPSe/Prolog, it’s the oldest of the three and provides backward-chaining for Wayang.
  • Wayang visualizer – Wayang tries to model how interpretations of events develop as the action unfolds, but this “unfolding” is not at all clear from its textual output which is quite verbose. To demonstrate (even to its authors) that the system can do something interesting, such development of interpretations needs to be clearly depicted. Our idea for doing so was to play the animation that the system was processing, and use a transparent overlay to draw annotations representing what the system had inferred to that point. There were two major technical challenges: (1) an animation of 150 frames requires two weeks of processing by Wayang on a fast PC, requiring us to do so offline before running the visualization “online”, (2) neither of the languages we were using (one was Java) had good support for playing Flash or placing a transparent layer above a Flash player. For anyone wanting to do the later in Java, following Kumaresh’s solution here may be your only option.

The three coding projects above are all hosted on GitHub. Guidance about how to run each one can be found in its ‘doc’ folder, which you can read by clicking through the links above. If you then want to run them, there is a GitHub tutorial if you’re not familiar with it.