[MUSIC] Throughout this whole course, Marco and I have been asking you repetitively to test your VR application with users. This is extremely important because without testing, you are almost certainly making the wrong assumptions. Fred Brooks pointed out, in 1988, that in watching many other interfaces being designed, I observed that the uninformed and untested intuition of the designer is almost always wrong. Marco and I have both learned this the hard way. We've conducted hundreds of experiments in virtual reality and made tons of mistakes. The only thing about user variation I am certain of is not there are always situations I have failed to plan for. So I thought I should share some of my experience here with you, hopefully to save you some time. First, recruitment, you should try to recruit not only your friends and families but also people you have never met before. In particular those who fits the profile of your potential users. When they come in collect some personal information from them, for instance gender and age. This is because the testing result maybe very different from what you except and there might be big variations between different users. Collecting data on participants gender and age could help you later explain the variations and decide what you want to do next. It might also be a good idea to ask them questions about their experience in gaming and VR. If they played a lot of video games, then they might be more familiar with 3D metaphors from those games. And therefore have presumptions about what to expect, which can make them better or worse in using your application. Also in research, we always need to apply for ethical approval before running any experiment. Although you might not need that, it is always good to think about potential risks you might be putting your participants under. For instance, if your application is likely to cause simulation sickness, you will need to make sure you warn your participants in case they might be driving or cycling immediately after trying your game. Second, plan your user study to have something to compare to. It's easy to get a group of people to try your game and give you some compliments but it's not actually the most efficient way to get feedback. The more effective way is, for instance, compare two different navigation methods implemented and see which one works better. Or you can get users to play a similar game to yours and ask them to compare the two on different aspects. If you have two methods or two games you are comparing, you will want all the participants to try both. In order to have unbiased feedback, you can randomized the order. So the first participant try method one first and then method two. And then the second participant try method two first, followed by method one, so on and so forth. This is the best way to get unbiased and constructive feedback and results. And all scientists follow the same principle when running their experiments. Finally, I want to say a few words about data collection. You might want to collect at least two types of data. First, user performance. For instance, how long did it take for them to finish all the tasks? How many enemies did they manage to kill? And how many times do they get hurt? You can also collect their head rotation data. If they're doing psychotherapy for arachnophobia, you might want to measure how often do they look at the spiders in VR in order to assess their level of fear. Ideally, you should write code in your application to collect all performance related data automatically and save as log files for each user. The second types of data you can collect are more subjective. They could be in the format of questionnaires. You should use some existing questionnaires, for instance, questionnaires which measures presence or simulation sickness. Subjective data can also be collected in the format of interviews. Make sure you plan your interview questions and take notes. That's all I have to say about user evaluation for now. You might not be able to do all this, but it's good to have a rough idea of where you should be going towards. [MUSIC]
