Statistics used in Practique

Cronbach’s alpha is a measure used to assess the reliability, or internal consistency, of a set of scale or test items. In other words, the reliability of any given measurement refers to the extent to which it is a consistent measure of a concept, and Cronbach’s alpha is one way of measuring the strength of that consistency.

Cronbach’s alpha is computed by correlating the score for each scale item with the total score for each observation (usually individual survey respondents or test takers), and then comparing that to the variance for all individual item scores:

The resulting α coefficient of reliability ranges from 0 to 1 in providing this overall assessment of a measure’s reliability. If all of the scale items are entirely independent from one another (i.e., are not correlated or share no covariance), then α = 0; and, if all of the items have high covariances, then α will approach 1 as the number of items in the scale approaches infinity. In other words, the higher the α coefficient, the more the items have shared covariance and probably measure the same underlying concept.

Item discrimination is the degree to which students with high overall exam scores also got a particular item correct.

The Station Statistic analysis uses 33% cohort to calculate the discrimination by:

• getting all correct answer and sorting it in order,
• selecting the top third correct answers and the bottom third correct answers,
• subtracting bottom from the top

For SBA type items it works like this: all candidates sorted by score (from the highest to the lowest) are split to 5 groups and then the graph shows % of candidates who got the question correctly in each group. The graph should usually shows "steps down" because most of top scored candidates should get the question right.

For CPQ item type it shows ... something different

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

https://en.wikipedia.org/wiki/Item_response_theory

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

https://en.wikipedia.org/wiki/Item_response_theory

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

https://en.wikipedia.org/wiki/Chi-squared_test

https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html

https://en.wikipedia.org/wiki/Point-biserial_correlation_coefficient

The Standard Error of Measurement (not to be confused with the Standard Error of the Mean) gives an indication of the spread of the measurement errors, when estimating candidates' true scores from the observed scores. It is calculated from the reliability coefficient (Practique uses Chronbach's alpha). It is assumed that the sampling errors are normally distributed.

The SEM is calculated as

SEM = S(1 – r_xx)^0.5

where S is the standard deviation of the exam, and r_xx is the reliability coefficient (Chronbach's alpha).

The key application of SEM in Practique is to apply a confidence interval to the cut score. For example, if you would like to be 68% sure of the pass/fail decision, the SEM indicates that the candidates within 1 SEM of the cut score may fluctuate to the other side of the cut score should they take the exam again. For example, if you wanted to be 95% sure of your decision on outcomes, an SEM multiplier of 1.96 can be applied. These figures are based on the Normal Distribution. Practique applies this on the positive side for most Standard Setting methods, as we are dealing with competency exams. In practice, what this means is that you are 95% certain that the passing candidates scores represent their true scores.