Responding to incidents

This page provides an overview of issues and offers suggestion on how to respond to them.

Incidents before the exam has started

A deployment fails

A deployment can fail for several reasons:

• The app registration secret is expired.
• A configuration that is incorrect (e.g., quota in the dashboard does not match actual quotas in Azure).
• An unforeseen change in your organization’s Azure tenant.
• A rare random Azure error.

If a deployment fails:

1. The Exam Orchestration job dashboard displays the most relevant Azure logs. You can find the dashboard by going to Schoolyear Dashboard -> Admin console -> Exam Orchestration Jobs -> [Exam name]. Scroll down this log and check for any errors. In most cases, the error will explicitly indicate what went wrong. If you know how to resolve the error, resolve the error. After resolving the error, you can use the retry button in the Exam Orchestration job dashboard to resume the deployment. If you don’t, continue with the next step of this list.

2. If you don’t immediately know from the dashboard what went wrong, there is a possibility it was a random Azure error (e.g. InternalServerError). Retrying the deployment from the dashboard might solve the issue. While waiting on the deployment to either succeed or run in to the same error, continue to the next step of this list.

3. You can further investigate the issue from the Azure portal. The starting point when checking the logs is the deployment of the resource group of this exam. The resource group name is logged in the Schoolyear Exam Orchestration dashboard: creating exam resource group: syexam-xxxxxxx. If you click on the resource group in the Azure Portal, the deployments can be found under Settings.

4. The error in the dashboard usually indicates which deployment went wrong. Opening this specific deployment in the Azure Portal can sometimes give you a more extensive log.

5. After resolving the error, you can retry the deployment from the Exam Orchestration Jobs page. It will try to continue where it failed. In some cases a full restart might be required because of left-over resources, in that case you can first delete the resource group of the exam in the Azure Portal and then restart. Please note: in this case, the deployment starts from scratch, which requires the regular deployment duration.

6. If you are not successful with your troubleshooting, you can contact support, more information can be found here. (Emergency support is offered. Note: this is a paid service)

Tip

These errors are common, they might give you a hint as to what went wrong:

• Authentication failures often signal an expired app registration secret or a change to the app registration role in the subscription.

• ‘CustomScriptExtensionName’ failures usually indicate session host scripts from a custom layer are failing because of mistakes in the scripts. Link to a troubleshooting guide.

• Unexpected quota errors are usually the result of changing VM sizes without changing the quota or overlapping exam times.

Incidents during the exam

A student’s experience is slow

If only one student reports slowness, the issue is usually with the student’s own device. Switch them to a spare device, their progress in the VM will remain. Most testing systems autosave; if not, ask the student to save their work first.

In case this does not resolve the issue or more students experience issues, check whether the VMs are powerful enough:

1. Go to the Azure Portal.

2. Navigate to Azure Virtual Desktop using the main search.

3. Go to Users, search for the student, and click on the name.

4. Go to sessions -> Active VM, and click on the VM name.

5. Go to Monitoring -> Metrics and check the VM’s resource consumption (CPU percentage etc.).

If the VM’s are not powerful enough, the only option (besides continuing as is) is to change the VM size and schedule a new exam. (e.g. from Standard_D2s_v5 to Standard_D4s_v5)

If the VM’s are powerful enough, the most likely cause is a slow network connection.

A virtual machine crashes or freezes

This should be rare and is typically caused by user error. If it does happen, troubleshoot as follows:

1. Go to the Azure Portal.

2. Navigate to Azure Virtual Desktop using the main search.

3. Go to Users, search for the student, and click on the name.

4. Go to sessions -> Active VM, and click on the VM name.

5. Restart the session host (virtual machine).

6. If this does not resolve the issue, consider unassigning the student and draining the session host. Please note: the student will lose all progress in the Virtual Machine.

Danger

Make sure you do not click and restart the host pool, as this restarts all virtual machines and disconnects all students.

Incidents after the exam

A deployment deletion job fails

After an exam, a deployment is scheduled for deletion. If this fails, you will get an e-mail. It is not an urgent problem, but failing to act can unnecessarily increase your Azure costs. If a deletion fails:

1. Navigate to Schoolyear Dashboard -> Admin console -> Exam Orchestration Jobs -> [Exam name]. A deletion job has the trash can icon in the list.

2. Check the logs for the error and try to resolve it.

3. Manually delete the resource group of the exam in the Azure Portal.