Practique uses a range of standard statistics for reporting and standard setting. Below are some descriptions and some useful links.
...
| View file | ||||
|---|---|---|---|---|
|
| View file | ||||
|---|---|---|---|---|
|
| Item | Description | Useful links |
|---|---|---|
| Overall score | Overall score in percentage for the whole exam / candidate | |
| Cohort average | Student performance against cohort group (average of students) | |
| Pass/Fail | Student passed of failed |
...
| Item | Description | Useful links |
|---|---|---|
| 33% Discrimination | Item discrimination is the degree to which students with high overall exam scores also got a particular item correct. The Station Statistic analysis uses 33% cohort to calculate the discrimination by:
| |
| Discrimination (point-biserial) | The item discrimination index is a point biserial correlation coefficient. Its possible range is -1.00 to 1.00. A positive result indicates that there is a high correlation between higher performing candidates giving a correct response to the item. | https://en.wikipedia.org/wiki/Point-biserial_correlation_coefficient |
| Facility (difficulty) of correct answer | Facility is a measure of how easy or difficult is a question for candidates. It is calculated as: FI = (Xaverage) / Xmax where Xaverage is the mean score obtained by all users attempting the item, and Xmax is the maximum score achievable for that item. | |
| Frequency | Frequency of answers | |
| Quintile Graph | For SBA type items it works like this: all candidates sorted by score (from the highest to the lowest) are split to 5 groups and then the graph shows % of candidates who got the question correctly in each group. The graph should usually shows "steps down" because most of top scored candidates should get the question right. For CPQ item type it shows ... something different |
...
| Item | Description | Useful links |
|---|---|---|
| Facility | facility = mean_score of the station / max_score of the station | |
| Discrimination (point-biserial) | The item discrimination index is a point biserial correlation coefficient. Its possible range is -1.00 to 1.00. A positive result indicates that there is a high correlation between higher performing candidates giving a correct response to the item. | https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html https://en.wikipedia.org/wiki/Point-biserial_correlation_coefficient |
| Frequency | In SBA item type frequency of answers is calculated. If candidate have not responded it is included in calculation. Facility and Frequency of most chosen answer should be the same. From Practique 5.4.0 > , beside answer letters columns for Frequency there is No Response column as well to show the whole picture. |
...
| Item | Description | Useful links |
|---|---|---|
| Cut Score | scored.exam_cut_score() --> sum(self.get_cut_scores().values() --> get_scored_cases() --> returns instances of Scored cases (set by standard method) : Sum of cut score of all stations divided by number of stations/questions. | |
| Cronbach | Cronbach’s Alpha For each of the standard setting methods the Cronbach’s Alpha reliability metric is also calculated for the exam. This is given for the whole exam as well as what it would be if each item in turn were omitted from the analysis. This allows items that are lowering the reliability of the exam to be excluded from the results. | Standard Setting Terminology |
| SE of measurement | The Standard Error of Measurement (not to be confused with the Standard Error of the Mean) gives an indication of the spread of the measurement errors, when estimating candidates' true scores from the observed scores. It is calculated from the reliability coefficient (Practique uses Chronbach's alpha). It is assumed that the sampling errors are normally distributed. The SEM is calculated as SEM = S(1 – rxx)0.5 where S is the standard deviation of the exam, and rxx is the reliability coefficient (Chronbach's alpha). The key application of SEM in Practique is to apply a confidence interval to the cut score. For example, if you would like to be 68% sure of the pass/fail decision, the SEM indicates that the candidates within 1 SEM of the cut score may fluctuate to the other side of the cut score should they take the exam again. For example, if you wanted to be 95% sure of your decision on outcomes, an SEM multiplier of 1.96 can be applied. These figures are based on the Normal Distribution. Practique applies this on the positive side for most Standard Setting methods, as we are dealing with competency exams. In practice, what this means is that you are 95% certain that the passing candidates scores represent their true scores. | Standard Setting Terminology |
| SEm mulitplier | See above | Standard Setting Terminology |
| Error (SEm * multiplier) | ||
| Pass Score rounded | ||
| Pass Rate |
...