It's hard to believe that it has been over a decade
since the two of us started working together.
One of the first major projects we worked on together
was developing an interactive agent
to use in some of our VR experiments.
We called it Cristina.
We'd like to talk a little bit about the challenges we faced
and how we solve them because a lot of it is still relevant today.
Cristina was able to carry on a conversation with a real person who was immersed in VR.
Her graphical appearance wasn't great even by the standard of that time.
But what made her interesting was that
her body language was quite realistic
and most important of all,
it responded to the participant.
Speech interaction isn't easy now but back then it was a big challenge.
Getting a character to respond realistically with dialogue was beyond what we could do.
Because Cristina was going to be used for experiments
where they'd always be an experiment to present,
we decided to use the approach called Wizard of Oz.
Essentially, we faked interactive conversation
by having in an experimenter choose
what Cristina was going to say from a number of options
in a menu just like the Wizard in the book.
On the other hand, this wouldn't work for a lot of the body language.
Her body language responds to other people very quickly.
The experimenter wouldn't be able to choose gestures and movements quickly enough.
Also, we mostly do body language subconsciously
so we aren't really aware of what we're doing.
That means it is very hard to explicitly control it for a character.
So, we decided to automate a lot of the body language
that involves sensing the behavior of the participant
and producing realistic responses.
The sensors we had available are very similar to
what we have now and head-mounted displays,
a head tracker and a microphone.
The most important information we needed to know
was whether Cristina was speaking or listening.
Speaking was easy when the experimenter pressed a button,
we just played back a particular piece of speech until we finished.
The audio we recorded an actress reading the script.
We then hand animated body and facial animations to go with the audio.
Our software played back the audio and animations
together to make Cristina speak and gesture.
Living experiments with motion captured actors
as they read out the lines and used that animation
which gave more realistic results.
Listening behavior was if anything more important.
It created the illusion that Cristina
was really paying attention to the participant
and made her feel real.
We used the microphone to detect whether the participant was speaking or not.
And when they were speaking,
our software would occasionally playback
nods and smiles to give some friendly encouragement.
Cristina's eyes were also very important.
We had a simple automatic gaze animation
where she would alternate
between looking at the participant and looking away.
Importantly, we change the proportion of time
she was looking depending on
whether she was speaking or listening.
She looked a lot more while listening to show attention.
This has proven to be quite effective.
Take a look at this video clip and please pay attention to his body language.
That shirt looks great on you.
How much was it?
I think it was about 20 pounds, it was quite cheap.
Thank you very much though.
You look very nice too.
Oh, thank you very much.
Here, the participant was clearly enjoying the interaction very much.
So much so that he moved all the way to the back wall
of the cave in order to get closer to Cristina.
What he didn't know was that
we have also programmed Cristina
to maintain a social distance from him.
So, the closer he moved,
therefore the back Christina moved.
She would move to make sure that she was always
at a comfortable distance to the participant.
Cristina would always turn to the face of the participant
as you would naturally do in a conversation.
And in one experiment,
we reduced the distance midway
through the conversation making her move forward
and increasing the sense of intimacy.
This had a strong emotional impact on our participants.
So, Cristina had quite realistic body language
and had a strong impact on participants
using only a few simple rules
that you can't reproduce with current VR technology.
The rest of this course is going to explain
how to do some of these things
and create socially skilled virtual characters.
