8.6 Exam Analysis Report

Number of candidates

Number of candidates that sat the exam.

Candidates that are excluded from exam are not included in the calculations.

Number of items

Number of items in the exam. 

Items that are excluded are not included in the calculations.

Minimum score

Smallest score achieved on exam.

Maximum score

Largest score achieved on exam.

Median

The median value is the score value in the middle of the sorted score array.

https://docs.scipy.org/doc/numpy/reference/generated/numpy.median.html

Mode

mode = scored.mode() --> scipy.mode():

Mode or Modal value is returning the most common score value in the list of scores. If there are more then oen value the smallest is returned. If there a no most common values it returns the smallest score in the exam.

https://docs.scipy.org/doc/scipy-0.19.1/reference/generated/scipy.stats.mode.html

Mean

The sum of all scores over the number of scores.

https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html

Standard error of mean

Standard deviation

First calculating the mean score of the exam. Then we calculate (x - mean)^2 for each score. Then summary of each squared differences is divided by number of scores - 1. -1 is used as standard statistical practice for better estimation. Squared root is take from last result.

https://docs.scipy.org/doc/numpy/reference/generated/numpy.std.htm

Skew

Checking if data is noramlly distributed. If > 0 it is more squeezed to left if < 0 it is more squeezed to right.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skew.html

Kurtosis

It defines sharpness of the distributed data at the peak of the curve. We are using Pearson definition.

• https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kurtosis.html ; https://en.wikipedia.org/wiki/Kurtosis

• Are the Skewness and Kurtosis Useful Statistics?

Cut Score

scored.exam_cut_score() --> sum(self.get_cut_scores().values() --> get_scored_cases() --> returns instances of Scored cases (set by standard method) : Sum of cut score of all stations divided by number of stations/questions.

Cronbach

Cronbach’s Alpha
For each of the standard setting methods the Cronbach’s Alpha reliability metric is also calculated for the exam. This is given for the whole exam as well as what it would be if each item in turn were omitted from the analysis. This allows items that are lowering the reliability of the exam to be excluded from the results. 

Standard Setting Terminology

SE of measurement

The Standard Error of Measurement (not to be confused with the Standard Error of the Mean) gives an indication of the spread of the measurement errors, when estimating candidates' true scores from the observed scores. It is calculated from the reliability coefficient (Practique uses Chronbach's alpha). It is assumed that the sampling errors are normally distributed.

The SEM is calculated as

SEM = S(1 – r_xx)^0.5

where S is the standard deviation of the exam, and r_xx is the reliability coefficient (Chronbach's alpha).

The key application of SEM in Practique is to apply a confidence interval to the cut score. For example, if you would like to be 68% sure of the pass/fail decision, the SEM indicates that the candidates within 1 SEM of the cut score may fluctuate to the other side of the cut score should they take the exam again. For example, if you wanted to be 95% sure of your decision on outcomes, an SEM multiplier of 1.96 can be applied. These figures are based on the Normal Distribution. Practique applies this on the positive side for most Standard Setting methods, as we are dealing with competency exams. In practice, what this means is that you are 95% certain that the passing candidates scores represent their true scores.

Standard Setting Terminology

SEm mulitplier

See above

Standard Setting Terminology

Error (SEm * multiplier)

Pass Score rounded

Pass Rate