Human judgement: Being aware of the errors we make

Loading...

Del curso dictado por Commonwealth Education Trust

Foundations of Teaching for Learning: Introduction to Student Assessment

81 calificaciones

Commonwealth Education Trust

Foundations of Teaching for Learning: Introduction to Student Assessment

81 calificaciones

The Foundations of Teaching for Learning programme is for anyone who is teaching, or who would like to teach, in any subject and any context - be it at school, at home or in the workplace. With dynamic lessons taught by established and respected professionals from across the Commonwealth, this eight course programme will see you develop and strengthen your skills in teaching, professionalism, assessment, and more. As you carry on through the programme, you will find yourself strengthening not only your skills, but your connection with colleagues across the globe. A professional development opportunity not to be missed. There are various techniques which help to assess student learning. This course helps teacher to acquire the skills to develop and use appropriate assessment procedures. Enhance your course by joining the Commonwealth teaching community on our website, Facebook and Twitter.

De la lección

Guidelines for developing and using human judgement scoring procedures

Last week we focused on assessments that could be scored against agreed correct answers. We will now look at judgement-based assessments. The main assessment method used in teaching has changed from objective-style to using judgement-based assessment. This change has brought forward some new issues for us to consider. We will introduce you to challenges that are inherent in designing and evaluating student performance in open-ended formats in the first lecture. In the next two lectures we will focus on two assessment tools that are often used under judgement-based conditions: the rubric and the essay. In the last lecture, we will explore the concepts of moderation and inter-rater reliability. The topics for this week require you to grapple with notions of accuracy, quality, reliability, validity, and error. By the end of the week you should be able to design a valid and reliable assessment task and corresponding marking scheme (rubric).

Human judgement: Being aware of the errors we make9:58

Guiding judgements: Analytic & Holistic Rubric design and use8:27

Essay marking: Working towards more reliable and valid scoring10:29

Moderation: A key to ensuring reliable and valid scoring11:13

Discussion 318:55

Conoce a los instructores

Dr Peter Keegan
Senior Lecturer, Te Puna Wānanga, Faculty of Education
University of Auckland, New Zealand
Associate Professor Gavin Brown
Director, Faculty of Education Quantitative Data Analysis and Research Unit
University of Auckland, New Zealand
Professor John MacBeath
Professor Emeritus
University of Cambridge, UK

[MUSIC]

Welcome to week five

of this course on assessment.

This week we're focusing on human judgment as the basis of scoring student work.

Before we look at the details of how we go about that, I would

first like us to look at the issues around making errors when we judge things.

Humans are notoriously bad judges, and that's what this week will

try to raise your awareness of, so that you can be a better judge.

In terms of our curriculum map, we're looking this week

at informal, student-centered approaches

to assessment that require human judgments.

We use these types of assessments, instead of just

objectively scored assessments because sometimes we want students to

construct, or write, or create, or perform their own answers.

This is an important curriculum objective.

For every response that a student carries out, there needs to

be rules for how should it be scored, interpreted, or evaluated.

And we call those rules, in general, "rubrics", and next

week's session is going to focus especially on rubrics.

And we need to carefully consider in the process of making decisions about

the quality of work, whether another competent judge would make a similar

evaluation to ourselves. If another teacher walks by, sees

the same work, and comes to a similar judgement, then we have reliable scoring.

And that's really important when there are consequences attached to student work.

And we have to be sure that the rules that we use to judge student work

are appropriate to the task being evaluated.

So, in terms of judging, we have to be worried about reliability and validity.

Judgement is the process in which experts, that's

you the teacher, decide "How good is this work?

Where does it lie on a progression from total beginner to advanced expert?"

And progress is always from none or very little up to very high levels.

In New Zealand, we use for our high school

qualifications, words like "not achieved", "achieved", "merit", and "excellence".

We could use norms for children of a certain age group -

"below average", "at the average", "above average".

We could describe it like the New Zealand curriculum in terms of

below, at, or above level 4 or 5, and so on.

Even university grades are a kind of progression from a F, fail

to an A+ or A* represents progression in quality.

And progress in good assessment is indexed

to curriculum objectives and their order of difficulty.

A curriculum statement tells us these are the things that

we teach early because they're necessary and perhaps easier.

And these are the things we teach later.

So, in terms of establishing our judgments, it's most powerful if those

judgments are indexed to curriculum difficulty and curriculum order.

So, we need tools that help us make those kinds of judgements.

And the reason we need tools is,

fundamentally, humans are not very accurate judges.

Even experts in a field disagree with each other.

Certainly the research evidence shows us that

scoring essays or performances by

university lecturers is notoriously inconsistent.

In fact, studies have shown that if you give a

lecturer the same set of essays, very few of them

will get the same grade or even close to each other.

And markers, that's you and me, have a huge impact on

the accuracy of scores, and therefore has serious implications for any decisions

we might make.

Take for example Olympic scoring.

There are sports like diving and figure skating in the Olympics, where panels of

judges make very rapid evaluations of the quality of a performance.

And on the screen, you can see a table of scores for two athletes and six judges.

And what you can notice is that the scores are pretty close to each other.

They range between eight point eight and nine point three.

Out of a ten point scale, they're really close.

But actually,

very few judges give the same score to the same athlete.

And in fact, you can see the two highlighted in

green are the same score but they are different athletes.

The two in red are low scores, but they're for different athletes.

So, none of the judges agreed, but they were close to each other.

And now this judgement happened by elite

markers, on elite performances, that happened very quickly.

This is hard work, so when we come to judging student learner

work, it's going to be hard to be similar with each other,

but it is an important ambition to aim towards.

Consider the kind of questions that we might ask.

A classic question we can ask is like the one on screen: "The President of the

U.S. during the missile crisis was:", and the student is expected to fill in the blank.

These run-on sentence completions are easy to create,

but they leave some ambiguity as to what is the correct answer.

You and I as teachers know, that the answer is supposed

to be President Kennedy during the Cuban Missile Crisis in 1962,

But, if the student writes "The President of the U.S. was worried."

Is that answer wrong?

If they say "busy playing golf," how is this answer wrong?

We need rules to say what is a valid and appropriate answer that should

be awarded and marked, and what answer should be ignored or treated as wrong?

And those rules are a rubric.

And so, in a simple, open-ended question you need a set of rules.

And on screen, are some rules that I've made

up for how this question could be scored.

Unfortunately, I think it would be better if

this were a question, not a run-on sentence.

So, there's some rules,

and we have a basis to avoid being arbitrary or capricious, which

of course, is a serious concern from a student's point of view.

"My teacher doesn't like me, therefore they marked me wrong."

So, how can we reduce error in our judgement about student work?

To create high quality judgements, we need rules.

And that's what the next session will be about, how to create rules.

We need, generally, two or more markers who work independently,

but who compare and discuss any differences in their scoring,

and we'll talk about moderation more in this week.

We have to be carrying out our marking

and our judgement, in an environment that's suitable.

If we're angry, if we're hungry, if we're tired or we're distracted, we shouldn't

be marking student work because it will come out in the scores we give.

We should deal with all performances on the same topic in one setting.

We should only compare apples with apples.

We should try to finish the scoring of all of those apples in one sitting,

and if we can't, because there's too many, or it's too late in the day, then we

should mark some without any marks on the paper,

put them aside, and come back to them tomorrow.

If we can't mark them the same way we did

yesterday, then we know there's a problem in our scoring.

We have to check our own consistency.

Human judgment is necessary in being a teacher.

We have to do it.

It's expected.

Not everything that we value can be

assessed objectively with multiple choice or sequence

and sorting and so on.

We have to use our expertise to judge the quality of student work.

But we have to be humble about this.

We have to know that we're going to make mistakes.

We're going to disagree with other judges.

And that we have to accept that our goal is not to be error free,

but to admit our error and talk about it with other judges,

and overcome and reduce that error before we give any

marks back to our students, or before we use those marks in reporting.

Later this week, we're going to get more advice on how to guide judgments.

Our next session is going to be on rubrics.

[MUSIC]

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2018 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play