Yoko v0.2

The thoughtful little chatbot

2D world

Buzzwords: Javascript, Canvas, 2D, collision detection, sensory input

To be realistic, a chat bot needs to have a 'life' as much as possible. We humans talk about stuff happening to ourselves, things we see, things we did today. All of this enters in our conversation, and conversely, any conversation worthy of that name contains experiences like that.

So, Yoko needs experiences. Let's put her in a world where she can do things, see things, experience things, and of course, communicate about it. With a virtual 2D world, we gain both a new form of input (i.e. what she sees and hears) besides chat, and a form of output (i.e. doing stuff)! Totally worth pursuing.

So to give this idea a starting point in practice, let's use HTML5 Canvas to build her a basic 2D world. To go to chat with the 2D world enabled, click here. You can move Yoko around with arrow keys and her approaching various objects will trigger stuff to happen.

One immediate success from building the 2D world is the realization that beyond the statement, sentiment and question phrase types that have so far been the only ones considered in Yoko's language processing, there is (at least) one more: commands, i.e. requests for the other person to perform some action. These will come with many challenges (as they can start with any verb for example) I'm sure, but the 2D world is a nice opportunity to start reasoning about these. (unconsciously I had already included this type without realizing it, more or less, by having Yoko answer to 'name' or 'list' questions: 'name some animals')

Because I'm rather having fun with this, you can press CTRL to have Yoko throw a rock, which will cause the cat to meow when it's hit. Note that we play sounds to indicate events like the cat meowing or a typing sound or even music...

The user shouldn't control Yoko

Obviously, in its current state (i.e. a Yoko character being controlled by the player), this world is not quite yet what we are after. Rather, Yoko should be able to:

  • Perform actions by herself, either in reaction to something the user asks her to do through chat, or as a reaction to some change in her environment. Note that 'change in environment' may include the time, i.e. she could go to sleep because it's late perhaps?
  • Respond to questions about her environment, like 'where is the cat now' or 'what do you see' which then depends on which direction she is facing, and which state the room is in.

So, in conclusion, if the user is to control anything, ideally, it should be Yoko's world and the objects in it, not Yoko itself. Annoyingly, this would require programming 'path finding' algoritms and stuff like that so yoko can move around from point A to point B without running into an object.

Let's first program a trivial case (TODO): telling Yoko 'sit down' changes the state of the room to have her sit down, and 'stand up' to change again. Perhaps this is another nice case to use Yoko's plugin architecture for?

Technical: building a 2D world with JS Canvas

The 2D world is built using JS Canvas, and contains only some very simple shapes and images. There is also some 'collision detection' so we are aware that Yoko is standing next to an object (and which one, and from which side). When she collides with a certain object, a callback function can be executed that makes stuff happen - Yoko starts action TYPING when approaching the desk from below for example. Or the light in the room switches on/off when Yoko hits it.

The user can use the keyboard arrows to walk Yoko around (which I want to remove in the end, Yoko has a life of her own after all - but perhaps on a later point the user can control another virtual character in that room?). A detailed JS object with the state is maintained and updated at all time, which contains the position of Yoko and all the objects, and whether or not the door is open, lights on or off, music is playing, where the cat is... the state also contains events that took place ('Yoko switched the light off'). This state is what we want to send to the server to use for reasoning and asking / answering / processing questions and commands.

Here's a screenshot how it currently looks:

Using the 2D world to make Yoko more 'real'

So, we have a 2D world, reminescent of the 'block world' of SHRDLU. Now what. A first step would be the ability to ask questions to Yoko about it:

  • 'Where is the cat?'
  • 'Do you see the couch?' (using the direction Yoko is looking at)
  • 'Is the red ball next to the couch?'

Or we can give her commands:

  • Sit in the couch
  • Open the door
  • Leave the room

... Lastly, the 2D world gives some inspiration for spatial reasoning, and some events for Yoko to talk about. (more on events)

Written by Wouter - copyright 2013. Questions and remarks welcome at wouter@yokobot.com!
A lot more chatbots over at chatbots.org!