Guide to Judging: Award Deliberations

Overview

Award deliberation is the last vital step in the Judging Process. In this step Judges will work with the Judge Advisor and one another to select candidates for each award, and create a plan of action for gathering any follow-up information for final decisions.

Award Deliberations involve comparing teams to one another. The integrity of the Judging Process depends on all Judges being able to speak candidly during this process. What transpires during deliberations is particularly sensitive information. Therefore, all judging deliberation notes and conversations need to be kept confidential during and after the event.

Step 1: Award Nominations From Each Judge Group

After Judge groups have interviewed their subset of teams, they should decide which one or two teams from their subset of interviews are candidates for each award. Judges do not need to nominate a team for every award. They should return to the Judges’ Room and share their nominations with the rest of the Judge volunteers and Judge Advisor. Often this takes the form of Judges affixing sticky notes with team numbers written on them, under a printout of each award name, in full view of other Judge groups who are also doing the same.

Color coding can help keep the nominations from each Judge group organized (see picture below).

Deliberation_post-its.png

The end result will be a shortlist of nominations for each award from all Judge groups. When there are many award nominations for each award, the Judge Advisor may ask Judge groups to withdraw weaker candidates from consideration, based on brief arguments for and against each nomination. For example, if a team was nominated for the Think Award, but did not score highly in autonomous programming, they may not be a strong candidate. Or a Judge group, upon considering the merits of other candidates, might withdraw their nomination for their initial candidate.

Award Description sheets can be found in the Judging Resources section and can be printed out and used to help visually organize judge input/candidate teams during deliberations. For smaller events, the Final Award Nominee Ranking Sheet with Team Interview Notes may suffice.

Step 2: Cross-Checking Award Nominees

The Judge Advisor will then organize Judge groups to go out and gather further information to validate the short list of award nominees. This may take the form of observing skills or qualifying matches and observing behavior in the pits, as well as potentially conducting follow-up interviews with award nominees. The goal is to come up with a final ranking of nominees for each award being presented.

For follow-up interviews, it is recommended that the nominees are interviewed by Judges that have not interviewed them previously. If possible, put Judges together who share an area of expertise to evaluate particular awards. For example, Judges who have a background in programming/computer science would likely be best qualified to evaluate the finalist candidates for the Think Award.

Step 3: Final Ranking and Nominations

The next step is the final deliberation for each award at the event.

If follow-up interviews were conducted, the Judges who conducted the follow-up interviews should be the ones to deliberate and create a ranking among those teams. It is a best practice to have first-choice award nominees, plus three or more additional alternate candidates.

If information comes to light that a team may have violated the Code of Conduct or Student Centered Policy, either by judge observations or from Volunteer Field Notes to Judges, that team’s consideration for the judged award should be scrutinized by the Judge Advisor. If there is found to be merit in that information, the award is given to the next alternate team in the award nomination ranking.

In the case of the Excellence Award, that winner should come from the list of Design Award finalists. Moving a team from being a Design Award finalist to Excellence Award winner may result in a reshuffling of winners for other awards such that no team earns more than a single judged award at the event. The Judge Advisor should work from left to right when reconciling award winners to ensure that each award winner is earning the highest award that they are eligible for, in order of precedence. Having three or more ranked candidates for each award is very helpful in this situation and eliminates the need for additional deliberations.

An example of this is below: Team A has been selected to win the Excellence Award. Team A was also the top candidate for the Design Category, so Team B will now win the Design Award. Since Design is of higher precedence on the qualifying criteria, Team D is now the Judges Award winner. Team C, formally third place for the Innovate Award, is now the Innovate Award winner. In the case of the Think award, that award winner is unchanged.

Sample_use_of_the_Final_Award_Nominee_Ranking_Sheet.png

Step 4: Entering of Award Winners into Tournament Manager

After award nominations have been chosen, the Judge Advisor should inform the Event Partner and Tournament Manager (TM) operator, who can then put those Team numbers into Tournament Manager under the “Awards” tab. It is recommended that the TM operator print the Award Summary Sheet or Award Script Reports, so the Judge Advisor can double check that all award winners have been entered correctly.

Step 5: Collection and Treatment of Judging Materials

Prior to the award ceremony, the judge room should be secured, including the collection of all notes, rubrics, ranking sheets, and erasing any whiteboard notes. Judges should not retain copies of any notes referencing individual teams, including rubrics or award ranking sheets. If pictures of teams or robots were taken, Judges should delete them.

After the event is over, the Judge Advisor should destroy all collected judging materials off-site. These items are not to be given to the Event Partner for destruction.

Continue to the next section, Guide to Judging: Remote Judging