risr/assess enables online marking. Online marking is available for individual SAQs (Single Answer Questions) and SAQs that form part of MWQs (Multiple Written Questions). This guide outlines the key features and includes step by step instructions to help you use them effectively.Introduction
Markers can be created and assigned in 2 ways:
...
Markers can also be grouped, groups are used to divide marking between different markers; groups also facilitate double marking.
...
How to create and assign markers manually.
If you have a small number of markers or need to add an additional marker for example, this method may work best for you.
Go to settings
Select Manage Examiners
Enter examiner credentials and select Role as Marker or Senior Marker
...
To create a new examiner, set up a new user and assign them to the appropriate user group in risr/assess. Instructions on how to do this, can be found here: How to create and manage risr/assess users and here: How to create and manage risr/assess user groups
Manual assignment of markers to questions/items
Markers can be assigned to individual questions before or after an exam has been completed. Select a marker from the list and select the relevant question/s for this marker.
Eg. Select Marker101 to mark SAQ#4
...
Use of Groups
1. Multiple markers sharing marking of one item
Groups can be used to divide marking between different markers.
The candidates are divided equally between the markers and allocated randomly and anonymously.
2. Double marking
If double marking is required, markers should be assigned to different groups.
This allows each marker to assign a mark for each candidate. A senior marker is then required to adjudicate any discrepancies.
The allowed marking discrepancy is assigned in the exam details screen in the exam setup.
Click on Exams - Assign markers - click on Add Group to create your group(s). The groups will then appear in the dropdown to enable markers to be added to a specific marking group(s).
...
In this example the marking of SAQ #4 is being shared by 2 markers.
Marker101 will mark 5 candidates and Marker102 will mark 5 candidates.
SAQ#5 is being double marked.
Marker101 will mark all 10 candidates and Marker102 will mark all candidates.
NB: The allocation of one marker to a group means that all candidates are marked by that one marker.
Markers can only be removed If they haven’t completed any marking.
As soon as any marking is carried out there is no longer an option to remove the marker.
NB: Resetting an exam does not remove the allocation of markers.
Import marker details
This method is recommended if you have a number of markers to add and assign.
You can add the markers details, assign them to items and also assign to groups; these can be edited manually afterwards if required.
Go to Exams and then View Markers to access the marking assignment screen
Select Upload marker assignment and import marker details from a spreadsheet (see format required below)
...
The Marking Process - Markers
When a Marker logs in their access is restricted to see the exams and items they have been assigned to. They are shown the number of candidates they need to mark (Expected), the number of candidates they have already marked and those that are in progress.
...
As markers work through the candidates they are given the option to save or confirm marks.
NB: A standard increment for marks can be set at a server level but this can be overridden at any time when creating an item.
Candidate responses are presented anonymously and in a randomised order.
The marking criteria (allocated in the item creation) are displayed along with each candidate response.
If ‘save’ is selected the marker can come back to this candidate and change the marks.
Once ‘confirm’ is selected the marker is unable to change their mark.
Marking is not shown as complete until all candidate marks have been confirmed.
NB: It is possible to go straight to ‘confirm’ without saving first.
...
Once candidate marks have been saved they are counted as ‘In progress’.
If they are confirmed they are counted as ‘Already marked’.
Markers are automatically moved the next candidate unless they un-click ‘mark next candidate’.
...
The Marking Process - Administrators
The Administrator Role
As an administrator, the progress of markers can be monitored.
...
This shows Marker101 has confirmed marks for 5 candidates
0 are in progress and 5 were to be completed for SAQ#4.
Marker102 has not yet commenced marking.
No marking has been completed for SAQ#5.
Double Marking and the Senior Marker Role
Allowed Discrepancy
When double marking is used, if the marks awarded by the assigned markers are different the average is taken as long as the discrepancy is within the limits defined.
The allowed marking discrepancy is defined in the exam details screen - accessed from Exams
...
A Senior Marker is required to resolve any discrepancies beyond the defined limit.
A Senior Marker can be added before or after an exam has been run and if marking has commenced.
NB: The allowed discrepancy is only relevant to exams that are double marked
Adding a Senior Marker
The Senior Marker is responsible quality assuring marking and will be notified of discrepancies when double marking is enabled.
...
If the difference between the 2 markers is outside the limits of the discrepancy a message is displayed highlighting items that need attention.
...
After selecting the Candidates icon, the candidates requiring attention are highlighted: (This is in the admin view)
...
When a Senior Marker logs in their dashboard displays the number of items and candidates that they need to review.
If the Senior Marker selects ‘Mark’ they are then taken to the relevant page to review results
The Senior Marker can view the marks allocated and then enter the final mark; it is possible to save marks and then review before confirming.
Once the Senior Marker has reviewed all results the Admin view shows the results are complete and there are no candidates that require attention.
The Senior Marker reviews the mark needing attention and confirms the final mark awarded
...
Setting the Marking Type
When creating a new Written Exam in the Edit exam details screen:
Click the dropdown menu for Written marking type
Select the required marking type
...
When the Exam has been created and ready to run, or has already run:
For the required Exam click View Sessions
Click View Markers
Using the checkboxes provided click and select the required Items that require marking
Using the dropdown menu select the required marking type for the selected Items
Click Apply
| Note |
|---|
The system will provide warning messages as guidance stating the number of each role (if applicable) are required for each Item based on the marking type selected. |
...
Assigning Individual Markers
| Info |
|---|
This option should be used when specific markers need to be assigned to specific Items. This option is available for all marking types. |
How to assign individual markers (when the exam has been created and ready to run, or has already run):
For the required Exam click View Sessions
Click View Markers
Using the checkboxes provided click and select the required Items that require marking
Items can be filtered using the filtering options available
Click Assign individual markers
| Info |
|---|
Users can be filtered for selection. Only Users who satisfy the filter criteria and have the permission Exams - Mark Exams are able to be selected. If no filtering is used, all users with the permission Exams - Mark Exams are available for selection. |
Select Markers as required
Single marked (one marker is required)
Double marked (two markers are required, one for each double marking group)
Bilingual exams must have a marker for each language variant. The same marker can be assigned to both variants.
Splitting Candidates into Cohorts (SAQ/MWQ Only)
| Info |
|---|
When assigning individual markers, candidates can be split into a specified number of cohorts. This can be done to ensure an even split of candidates per marker. Select as many markers the number of cohorts. |
When selecting markers:
Click the checkbox for Split candidates into cohorts
Identify the number of cohorts required
Select the required markers appropriate to the number of cohorts identified
| Info |
|---|
Examples:
|
Assigning Reviewers
| Info |
|---|
This option is only available when the marking type is Single with reviewer of Double with reviewer. |
| Info |
|---|
A reviewer is any user with the permission Exams - Mark Exams, but when assigned as a reviewer the user will have a read-only view of the candidate’s mark. |
How to assign reviewers (when the exam has been created and ready to run, or has already run):
For the required Exam click View Sessions
Click View Markers
Using the checkboxes provided click and select the required Items that require marking
Items can be filtered using the filtering options available
Click Assign reviewer
| Info |
|---|
Users can be filtered for selection. Only Users who satisfy the filter criteria and have the permission Exams - Mark Exams are able to be selected. If no filtering is used, all users with the permission Exams - Mark Exams are available for selection. |
Select one reviewer per variant
| Note |
|---|
The reviewer cannot be the same as the marker |
Assigning Senior Markers
| Info |
|---|
This option is only applicable when the marking type is Double with senior marker |
| Info |
|---|
A senior marker can be any user that has the permission Exams - Mark Exams |
How to assign senior markers (when the exam has been created and ready to run, or has already run):
For the required Exam click View Sessions
Click View Markers
Using the checkboxes provided click and select the required Items that require marking
Items can be filtered using the filtering options available
Click Assign senior marker
| Info |
|---|
Users can be filtered for selection. Only Users who satisfy the filter criteria and have the permission Exams - Mark Exams are able to be selected. If no filtering is used, all users with the permission Exams - Mark Exams are available for selection. |
Select one senior marker per variant
| Note |
|---|
The senior marker cannot be the same as the marker |
Assigning Marker Pools
| Info |
|---|
Marker pools enables any available marker to mark as many candidates as they want within the pool. This is different from splitting the candidates into cohorts as a marker is not limited to the number of candidates they can mark. |
How to assign senior markers (when the exam has been created and ready to run, or has already run):
For the required Exam click View Sessions
Click View Markers
Using the checkboxes provided click and select the required Items that require marking
Items can be filtered using the filtering options available
Click Assign marker pool
Either
Select an unlimited number of markers per marker pool
Single marked (one marker pool)
Double marked (two marker pools)
Single marked bilingual exams (two marker pools, one for each language variant)
Double marked bilingual exams (four marker pools, two pools for each language variant)
OR
Click Add all markers to add all eligible markers
| Note |
|---|
Adding all markers is not recommended for double marking or bilingual exams |
| Note |
|---|
A marker cannot be in both marker pools when using double marking |
Click Save the marker assignment
| Info |
|---|
The number of markers assigned to each marker pool is shown in brackets. The list of assigned markers can be accessed by clicking the arrow |
Importing Marker Assignment
| Note |
|---|
The marking type must be set before the assignment spreadsheet is uploaded. Each marking type uses a unique spreadsheet format |
| Note |
|---|
All markers must have a risr/assess user account created and be assigned the permission Exams - Mark Exams before importing the marker assignment |
How to upload a marker assignment import spreadsheet (when the exam has been created and ready to run, or has already run):
For the required Exam click View Sessions
Click View Markers
Click Upload marker assignment
Identify your upload file and click Upload file
Ensure the data is mapped correctly to the spreadsheet column headers
Click Confirm
| Note |
|---|
If the spreadsheet contains a user that does not exist, the import will fail with a warning message |
Marking SAQ/MWQ items
When a marker logs in to risr/assess, their dashboard will display all marking assignments separated by exam. The following information is shown for each:
Exam / Block / Item - Exam information
Variant - Language variant used (where applicable)
Not started - the number of candidates for whom marking has not started
In progress - the number of candidates who are currently being marked
Completed - the number of candidates who have been marked
...
For the exam the marker wishes to mark:
Click Mark
The first (random) candidate's marking sheet will display
| Info |
|---|
What the marker sees as a candidate identifier is a configuration setting per instance:
|
Enter marks using the dropdown menu provided for each Marking area
The marker can then do any of the following:
Save marks (marker can access this candidate again)
Confirm marks (marker will not be able to modify marks)
Reassign to the other language marker (for bilingual exams only)
Enter notes that can be included on the candidate feedback report
Enter comments that candidates will never see. Only exam administrators, reviewers or senior markers (where applicable) will see these comments.
...
From the dashboard, the marker can click Overview to see a summary of all candidates and their status.
...
Marking VSAQ items
| Info |
|---|
For markers, VSAQ marking differs from SAQ/MWQ marking as all responses from all candidates are marked on a single screen |
When a marker logs in to risr/assess, their dashboard will display all marking assignments separated by exam. The following information is shown for each:
Exam / Block / Item - Exam information
Variant - Language variant used (where applicable)
Not started - the number of responses expected
In progress - the number of responses still being marked
Completed - the number of responses marked
...
For the exam the marker wishes to mark:
Click Mark
All responses will be displayed
...
| Info |
|---|
All known responses will automatically be marked and confirmed. No action is needed. These are shown highlighted green and with Confirmed appearing next to the score. All unknown responses will be categorised according to the Levenshtein distance set at the exam level. VSAQ items can be set to case sensitive or not case sensitive. If set to no, in the example both responses red and Red will be on the same line. The responses can be sorted by Response, Number of responses or Score. The marker can filter the responses by typing in the free text box provided and searching. |
Enter marks using the dropdown menu provided for each Marking area
The marker can then do any of the following:
Save (marks will remain editable)
Confirm (marker will not be able to modify marks)
Add (this will create a new version of the item and add the response and mark provided. The item will remain in draft state until approved)
Reassign to (for bilingual exams, the marker can reassign that response to the other language marker)
Click Save and Close (If the marker did not Save each individual response, this action will save them all before being brought back to the dashboard).
Reviewing items
| Info |
|---|
The reviewer cannot edit any marks and has more of an observational role in marking. It is important to note that if a reviewer does not finish reviewing, it will not affect scoring. Scoring and modification of marks by an administrator is still possible. |
When a reviewer logs in, the dashboard will display the following for all their assigned items:
Exam / Block / Item - Exam Information
Variant - Item variant (language) where applicable
In Progress - For SAQ/MWQ this shows the number of candidates being marked
Completed - For SAQ/MWQ this shows the number of candidates who are marked and confirmed
In Progress - For VSAQ this shows the number of responses being marked
Completed - For VSAQ this shows the number of marked and confirmed responses
| Info |
|---|
If the marker has not confirmed any marks, the Review marks and Overview buttons will be not be available and appear grey. Once the marker has confirmed marks, the Review marks and Overview buttons will become available. |
...
Once the marker has confirmed marks, the reviewer can begin:
Click Review marks
For SAQ or MWQ, the reviewer can either:
Click Needs further review which will keep this candidate in progress.
Click Review complete which will mark this candidate as complete.
...
The reviewer will be brought to the next random in progress candidate. The dashboard will show the reviewer’s progress across all assignments.
...
The reviewer can click Overview to see their progress for that specific assignment.
For VSAQ:
The reviewer can only click Review Complete beside each confirmed response
The status will indicate Confirmed Reviewed.
| Info |
|---|
The reviewer does not need to review known responses that were auto confirmed by the system. |
...
Marking as a Senior Marker
| Info |
|---|
Senior markers enter the final mark for double marked items when the marks given are outside the allowable marking discrepancy. Marks within the allowable discrepancy are automatically averaged by the system and will appear in the Completed column on the Dashboard and Overview. |
When a senior marker logs in, the dashboard will display the following information for all assigned items:
Exam / Block / Item - Exam information
Variant - Language variant for the item, where applicable
In Progress - For SAQ/MWQ this shows the number of candidates being marked
Completed - For SAQ/MWQ this shows the number of candidates marked
| Info |
|---|
Only when both markers have confirmed candidates, will they be available to the senior marker. |
| Info |
|---|
For MWQs, the discrepancy will be calculated for each part. Therefore, a senior marker may be required to enter a mark for only one part of an item. |