Assessment Calibration

Let's go!

Running your own team Assessment Calibration

Assessment calibration

Go to Course Navigation and Content

Calibration is an activity for a group of assessors (typically for a module) whereby they collectively reflect on the grading of and feedback to student work, and consider the implications from that activity for their assessment and feedback practices.This course provides guidance for organising and taking part in an Assessment Calibration event.

Introduction

Return to start

• Getting the most from your discussion

• On the day

• Setting up

2. How to run assessment calibration:

1. Aims, purpose and objectives of Assessment Calibration

Course navigation

Return to start

Click on the index items above to take yourself through the course, and the different elements to best suits your needs.

3. Course completion/ further help from Queen Mary Academy

As you go through the course:

Clicking on the 'house' icon will return you here to Course Content

Clicking on the 'location' icon will return you to the main content slide

Where several slides relate to the same topic, 'arrow' buttons indicate this and you can use them to move up and down as required

Course content

• To improve fairness and consistency in assessment practices;
• To engage in professional development to deepen and solidify understanding of expectations of student work and standards on which judgements are based.

Objectives

A professional dialogue to calibrate the grading of and feedback to student work, and consider implications arising through that discussion for assessment task and assessment criteria.Assessment Calibration can be a professional development activity at any point in a year; scheduled to focus on a sample of scripts before the rest of the cohort to act as an 'anchor' for subsequent marking; or as effective activity for carrying out moderation.

Purpose

To develop ‘calibrated academics’ (Royce Sadler, 2013:17), professionals who are effective “custodians of standards knowledge” and can use this to inform and guide them in their independent and collective decision-making about student work.

Aim

Enabling a supportive environmentFaciliative questions you can ask.

Getting the most from your discussion

How an Assessment Calibration event works and guidance for a successful event.

On the day

Who is involved.What assessed work do we focus on.How do we prepare.

Setting up

How to run Assessment Calibration

How do we prepare?

What assessd work do we focus on?

Who is involved?

Setting up

• All colleagues engaged in the assessment of an identified module.

• An assessment calibration event also needs a facilitator, someone not involved in marking student work on that module. For example, this might be the Programme Lead, Head of Department, or a member of QMA. The facilitator: sets the date for receiving intial grades, feedback and rationales and the calibration event (that works for all members); facilitates the event; and keeps notes on all discussion themes, agreements, and decisions for action.

Who is involved?

The facilitator signposts all assessors to selected submitted student work. The instances of student work can be written coursework, videos of performance, artefacts, exam answers, etc, ie. as relevant to the module.Reflecting the purpose of your assessment calibration event (see 'Purpose), the student work could be:

• randomly selected;
• selected after first marking as a sample of firsts, seconds, thirds, fails;
• selected to include work close to grade boundary break points;
• selected because there has been discussion /confusion/ discrepancy about the work/ the task;
• selected for focus to address External Examiner feedback or past/current student feedback.

What assessed work do we focus on

All members independently mark the identified work, and record their provisional judgement (grade), rationale (such as, noting argument, words/concepts, phrases etc. from student work related to the assessment criteria), and write the feedback as they would ordinarily do so to the student-author of the work. The grade and feedback is sent to the facilitator at the agreed date in advance of the Calibration event. Members will take their grade, rationale, and feedback for each script to the meeting.

How do we prepare?

+ info

Reach consensus for the particular piece of work by agreeing on or resolving issues centred on the meaning of the assessment criteria, the merit and validity in the student work, and individual and collective emphasis and understanding or the task and assessment criteria.

Conclusions

Notice and discuss similarities and differences. Similarities can be in grade but rationales can articulate different areas of emphasis; similarities can be in rationales but differences in assigned grade; grades can be widely divergent; and, there can be significant differences in amount, quality, tone and approach of feedback. See 'Info' below for more details, and 'Getting The Most From Your Discussion'.

Professional dialogue

The facilitator identifies the piece of work for focus. One at a time, each person present shares the grade they have given it, and their rationale for doing so.

Student scripts

All members come together as a group, each bringing their documented grade, reasons, and feedback, to the Calibration meeting. The facilitator brings a copy of the programme specification and module descriptor to which the work relates; and a collation of the received reported grades and feedback that were sent to them in advance.

Come together

On the day

+ info

The success of the event is from being a collaborative, reflective, supportive environment and process. Individuals may feel vulnerable, exposed, a sense of ‘imposter syndrome’. To allay concerns and set the developmental and supportive context for Assessment, reminding members it is a process of discovery - and that research shows consensus-based collaboration builds confident, consistent assessment practices, which, in turn, positively impact student confidence and trust.You can find read more about Assessment Calibration using the resources more via the 'Info' link below.

Getting the most from your discussion: enabling a supportive enviroment

Watch this video of an Assessment Calibration event for a snapshot of focus and approach.

"Unless students see themselves as agents of their own change, and develop an identity as a productive learner who can drive their own learning, they may be neither receptive to useful information about their work, nor be able to use it." (Boud & Molloy, 2013:705).

• How will you invite students to become insiders of your assessment process, to be able to make their own consistently sound judgments and know why those judgments are justifiable?

• Are there revisions that should be made to the task and assessment briefing?
• Are there revisions that should be made to the Learning Outcomes, or the assessment criteria?
• Are there implications for our broader curriculum and instructional/signposting practices?

• What do you notice about grading the work and using the criteria?
• What do you notice about feeding back and forward to the student about this work and how we are building their understanding and performance?
• What are the implications for our individual assessment practices/what is your/our take-aways from this activity today?

Getting the most from your discussion: facilitative questions

Example questions when each person has shared their initial grade and rationale:

Example questions to connect discussion to general principles and practices of assessment and feedback:

Extending reflection to include your students

Course completed!

If you have any questions, want to discuss your Assessment Calibration event, or would like Queen Mary Academy to faciliate Assessment Calibration for your team, contact qmacademy@qmul.ac.uk

Individual markers will explain their grades and rationales. If this become defensive, personalised, or stuck, keep the process moving by reminding that:

• judgements are not a science. Collective agreement will facilitate consistent assessment and feedback practices.
• therefore judgement must be based on the assessment criteria/rubric, so differences are due to interpretation and emphasis, not personal opinion. What enhancements to the criteria might support consistent assessment practices?
• and, judgements reflect whether or not the Learning Outcomes have been met. Are they clear? Does the task align to the LOs? Is there consensus between you about what the LOs mean?

This paper is viewed as a pivotal anchor contribution to the development of Assessment Calibration

• Royce Sadler D. (2013) Assuring academic achievement standards: from moderation to calibration. Assessment in Education: Principles, Policy & Practice. 20:1, pp.5-19

Here you can read more about curriculum design that create opportunities for students to develop capabilities to judge their own learning.

• Boud D. & Molloy E. (2013) Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education 38 (6), pp.698–712