Calibration Report

Related Products or Features: Business Intelligence (BI) Reports, Quality Management

Required permissions: Report Viewer, Calibration

Required Employee Settings: Can View BI Reports

The Calibration report helps you to calibrate evaluation forms. During calibration, the form is sent to a number of users to evaluate. The Calibration report automatically tracks variance between evaluators and individual evaluator performance. You can run the report on different versions of the form and see the change in variance between the versions.

The report has two main views:

  • Form Calibration—helps you to quickly see the forms and questions that have a very high variance between evaluator answers. High variance indicates there may be a problem with the way the question or answers have been formulated.

  • Evaluator Calibration—helps you to identify evaluators who are consistently over- or under-evaluating forms and questions. High variance indicates an evaluator may need additional coaching to bring their scores in line with quality standards.

Example of Use

As a quality manager, Tabitha wants to improve evaluator performance. She can use the Calibration report to quickly see the variance for the forms she is responsible for, and to check the evaluator performance for each evaluator in her team.

Form Calibration View

The Form Calibration page appears when you first run the report. If you want to see the Evaluator Calibration page, select Evaluator in the Report By list.

The page displays the following summary information for each form that you selected:

  • The number of calibrations requested and completed. For example, 2/10 means ten calibrations were requested for the form, and two have been completed.

  • The calibration score:

    • If calibration is based off an evaluation, the evaluation score is used.

    • If calibration is based off an interaction, the average calibration score across the evaluators is used.

  • The average variance across all the evaluators. A high variance indicates the form is being scored inconsistently across the evaluators.

  • The number of days left for calibration. 0 indicates the calibration deadline has been reached.

Click a form name to drill down. For each question, the page shows these details:

  • The calibration score:

    • If calibration is based on an evaluation, the evaluation score is used.

    • If calibration is based on an interaction, the average calibration score across the evaluators is used.

  • The average variance across all the evaluators. A high variance indicates the question is being scored inconsistently across the evaluators.

Evaluator Calibration View

The Evaluator Calibration page appears when you select Evaluator at Report By on the Form Calibration page. If you want to see the Form Calibration page, select Form at Report By.

The page displays summary information for each evaluator who has been assigned calibrations.

The evaluators and forms displayed depend on the teams and forms you selected. You can filter the evaluators using the Evaluator drop-down.

For each evaluator, the page shows these details:

  • The number of calibrations assigned and completed. For example, 2/10 means ten calibrations were assigned to the evaluator, and two have been completed.

  • The calibration score.

  • The average variance in calibration score across all the forms for this evaluator. A high variance indicates the evaluator is scoring this form significantly higher or lower compared to other evaluators.

Click an evaluator name to see these details for each question in each form calibrated by the evaluator:

  • The calibration score.

  • The variance of the calibration score compared with the average and maximum score of other evaluators. A high variance indicates the evaluator is scoring this question significantly higher or lower compared to other evaluators.