DESIGNING RUBRICS Rubric Rating Patterns and Rater Effects

Although rubrics reduce and manage the overall level of subjectivity in grading both written and performance assessments, they cannot eliminate subjectivity completely. Fatigue, distractions, implicit biases, etc. can affect the grading process and reduce the ability to differentiate between learners who are mastering the content and those who are not. The effects that typically have the greatest impact are:

Hawk vs. Dove Effects

These are also known as severity/leniency errors in which the grader systematically rates learner performance/work too harshly or too kindly.

- A Hawk is a strict grader who consistently gives lower ratings than learners should actually receive based on performance.
- A Dove is a lenient grader who consistently gives higher ratings than learners should actually receive based on performance.

If only one individual is responsible for grading rubrics in a course, all rubric assessments are likely to be graded equally severe or equally lenient. However, when there are multiple graders for a single assignment (in a Capstone course, for example), some learners may end up with higher scores simply because they were assigned to the more lenient rater.

To avoid this situation, ensure that all rubric graders are in agreement about how to interpret the rubric prior to grading (practice grading and comparing responses among raters).

Inconsistency

Inconsistency occurs when an individual does not follow the grading criteria specified on the rubric. This can happen when the rubric does not match the assignment instructions, if it is not aligned with the SLOs, or if the rubric criteria descriptions are too vague, poorly written, or overlap.

Example: An instructor uses the following rubric to assess clarity and mechanics in a final paper for a class of 25. (This is only an excerpt from the full rubric).

Excerpt from a rubric showing the criterion Clarity and Mechanics. It is a single point value rubric ranging from 4-0 points with full descriptions at each point level. The criterion descriptors for 4 points and 3 points are nearly identical with the only difference being 'few errors' versus 'some errors'.

During the grading process, the instructor has difficulty making a distinction between ‘few errors’ and ‘some errors’. Once all the papers have been graded, the instructor realizes some learners have 7 or more errors but were awarded 4 points, whereas other learners have only 3 errors and were awarded 3 points.

The criteria descriptions for 4 points and 3 points are nearly identical, so it isn’t surprising that inconsistent grading occurred. To avoid this situation, the instructor could modify the rubric so this criterion would total 3 points.

Halo Effect

The halo effect occurs when:

an instructor’s general impression of the learner influences grading,
performance on one criterion influences performance on other criteria within the same rubric, and
rubric criteria overlap too much.

General impressions that may affect grading include perceptions of which learners are ‘high performing’ or ‘low performing’ based on work in a prior course or if they are re-taking the course. This also includes awarding points/credit based on the perceived amount of effort learners contributed rather than assessing the actual product/result. Recognizing and acknowledging these implicit biases can help offset this tendency. You can also hide learner names when grading within Canvas to minimize biases. If you do this, make sure your learners do not include their names in the filename or in a header/title page.

Example: An instructor uses the following rubric to assess class presentations.

During the grading process, the instructor realizes that what is actually being assessed in Organization overlaps significantly with what is being assessed in Clarity/Mechanics/ Presentation Skills. Learners awarded 3 points for Organization are always awarded 3 points for Clarity/Mechanics/ Presentation Skills whereas learners awarded 2 points for Organization are always awarded 2 points for Clarity/Mechanics/ Presentation Skills.

To avoid this situation, this rubric can be modified so the two criteria assess different components of the presentation.

Restriction of Range

Restriction of range describes a pattern in which ratings are clustered around a specific portion of a scale. It can be one extreme (all high scores) or the other (all low scores) or exclusively in the middle (all average scores). For grading assignments, this predominantly occurs when all learners are awarded full credit regardless of actual performance. This is most likely to occur for class participation rubrics and discussion forum rubrics.

If all learners are awarded full credit, is the rubric really assessing what you intend to assess? In some cases it might be and your learners all excelled past your expectations. But in many cases, there is likely a mismatch between the intention of the assignment and how the rubric is written.

Please select the next tab above to move to the next content tab or the next button below to move to the next topic.