Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Standard setting is the methodology used to determine the proficiency of candidates and the related cut scores for each exam.

Supported Standard Setting Methods 

Angoff 
Each item in the item-bank can be assigned an Angoff percentage, representing the percentage of minimally competent candidates* (borderline) that would be expected to know the correct answer to the item. 

Ebel
Practique has full support for Ebel standard setting, allowing each exam paper to be standard set by a group of invited judges. The Ebel matrix can be customised, and difficulty and relevance can be input by each judge, and the Ebel score automatically calculated for each item in the exam. Ebel score histories are kept for each item in the item bank, allowing Ebel scoring trends over different exams and exam papers to be tracked separately in the item bank. 

Borderline Group
The borderline group method only applies to examiner marked exams. For each item the examiner can indicate a global mark for each item answered by the candidate, which can have multiple levels, but minimally pass/borderline/fail. The pass mark is determined by analysing the borderline group of candidates. 

Borderline Regression
Similar to borderline group, the examiner specifies a global mark for each item answered by the candidate, which minimally has pass/borderline/fail levels. The pass mark is determined by constructing the regression line through all of the groups. 

McManus Borderline Regression
Similar to borderline regression, the examiner specifies a global mark for each item answered by the candidate, which minimally has pass/borderline/fail levels. The pass mark is determined by constructing the regression line through all of the groups and combining with a negative confidence interval.


Related terms

Cronbach’s Alpha
For each of the standard setting methods the Cronbach’s Alpha reliability metric is also calculated for the exam. This is given for the whole exam as well as what it would be if each item in turn were omitted from the analysis. This allows items that are lowering the reliability of the exam to be excluded from the results. 

Standard Error of Measurement
The SEM is calculated from the Alpha and is multiplied by a user-input number (the SEM-multiplier) allowing a single-sided confidence interval to be added to the pass mark. For example an SEM-multiplier of 1.64 is equivalent to a 90% confidence that people given a pass should have passed (based on the reliability of the exam), and 1.96 corresponds to 95% confidence.


Score Normalisation

It is possible to normalise the marks for each item in an exam to a consistent number. You may want to do this so that each item in an exam is equally weighted, despite having an inconsistent number of marks for each item. Note that the normalisation is on an item basis, and not on an exam basis. It is possible to revert to non-normalised scores.

Notes

*It is important that the assessment function has a clear definition of what a borderline or minimally competent candidate is, and whether these are equivalent

  • No labels