When we looked at gestures, it was all about what we do when we're talking. But talking is only one half of a conversation, because there's always somebody listening. Body language when we're listening, is also very important. We may not be talking, but our bodies are still communicating. We give constant feedback to the person speaking, about whether we agree with them, whether we've understood what they're saying, and if we want to start talking. This communication, back from the listener to the speaker, is called the backchannel. It takes many forms, including gestures like head nodding, facial expressions, as well as making small sounds that aren't quite like words like, "Aha, hmm", we call these vocalizations. One of the main functions of the backchannel is simply to encourage the speaker to carry on. This positive feedback normally take the form of head nodding, smiling, and vocalizations, aha, hmm, maybe even the odd word like, "Okay, yeah." Most important of all is eye gaze. Looking at the speaker shows that we're paying attention to them, though we can also look away occasionally, to avoid excessive staring. Sometimes a conversation doesn't go so well. We might not understand what the speaker is saying. We could ask a question, but often a confused facial expression, is enough to show that we need to explain more. It's something I have to look out for a lot in a classroom of students. We can also show other feedback with our body language, for example, disagreements. This kind of backchannel feedback, can have a big effect in VR. A couple of years before I joined Mel Slater and Anthony Steed's VR Lab, University College London, they ran an experiment that showed this. They asked participants to prepare a short speech, and perform it in VR to an audience of virtual characters. The audiences responded in one of two ways. They either gave a lot of positive feedback: looking straight to the speaker, nodding and smiling, or they gave very negative feedback: looking away, leaning back in their chairs and muttering. The results were dramatic. Speakers in the negative condition, judged their performance much worse, and often had trouble carrying on speaking. This shows how important the backchannel is, and how much of an impact it can have on VR. How do we implement a character that can do backchannel feedback? We'll come back to how to do eye contact later. But a lot of the backchannel consists of gestures and short sounds, which can be implemented by playing back animations and audio. The hard part is knowing when to play them. Feedback items like nods, can be quite carefully timed in response to what the speaker is doing. Louis-Philippe Morency, Jonathan Gratch, and colleagues at the University of Southern California, have done some very advanced work using machine learning, to detect the right moment to give feedback. Having said that though, in the virtual audience experiment, the feedback wasn't timed very exactly, and still had a big impact. So, you can probably get away with playing feedback slightly at random while the speaker is speaking. A final use of the backchannel, is to show when we want to speak. In conversation, we take turns between speaking and listening. We take these turns very quickly, with almost no gap between one person stopping, and the other starting. We aren't waiting for the speaker to stop, and then picking up. We know exactly when they're going to stop, and pick up at that instant. This turn-taking is achieved through body language. For example, when we look at the speaker and move our hands into the gesturing position, it's a signal that we want to speak. With a virtual character, we probably can't do this as well. While there's been some advanced research in this area, the easiest option is to wait for the player to stop talking, and then make the character talk. Turn-taking is a fundamental part of conversation, and when simulating conversation, we need to keep track of who is taking their turn to speak. We need to do this because almost every aspect of body language changes depending on whether we're talking or listening. So, simply knowing this one bit of information, can massively increase the realism of our characters.
