Setting up a Knowledge Quiz

Knowledge Quiz Help

The Knowledge Quiz section on your Qualify tab allows you to create a short quiz to ensure that candidates have the required job knowledge to be successful on the job. This quiz will have clearly correct or incorrect answers to each question (open-ended questions or questions with nuanced or conditional answers are best left to the Homework or Interview sections). This document provides you with information to help construct better questions.

General Tips

Your goal in writing questions for the Knowledge Quiz is to ensure that candidates have the required knowledge to be successful on the job. Here are some general tips for writing questions that apply across all types of questions:

Write in Simple Language

Your goal with the Knowledge Quiz is to assess required job knowledge – NOT reading comprehension, vocabulary, etc. So focus your questions and response options on the knowledge elements required upon entry to the position for which you are hiring. Avoid complicated language, sentence structure, vocabulary, and syntax. Complex language has the potential to introduce bias against protected groups, especially those candidates for whom English is not a native language.

Of course, if the language and vocabulary is job-related, then it is perfectly appropriate to use them. For example, the terms “concatenate” or “delimiter” are perfectly appropriate to use in a quiz for computer programmers, as these terms are necessary components of job-related knowledge required at entry; however, these same terms are likely inappropriate to use when hiring a sales associate, unless the sales associate is required to have knowledge relating to computer programming. Similarly, complicated sentence structure in a question for the Knowledge Quiz might be appropriate when hiring for positions with strong writing and/or editing requirements (e.g., editor, journalist), but not for positions without these requirements.

The key takeaway is that the language in your Knowledge Quiz question writing should be as simple as possible, while remaining job-related.

Be Brief

When writing job knowledge questions, it is important to be as brief as possible while still covering all the required information necessary for a candidate to successfully answer the question. Including extraneous and/or unnecessary language increases the reading requirement and time necessary to answer the question. This can introduce bias against protected groups, including candidates with increased test anxiety and those for whom English is not a native language.

Consider the following question for a data analyst position: “Could you please compute the answer to the following question: What is the mean (or average) of the following five numbers: 5, 8, 13, 21, and 34?” While this question is perfectly clear as written, it contains too many words, which increases the reading burden and time to complete the Knowledge Quiz. The first clause prior to the colon is unnecessary – though polite, it introduces more words than are necessary to understand the question. The phrase in parentheses, “(or average)”, is job-related terminology that a data analyst would be required to know upon entry; it is not necessary to define what a “mean” is to the candidate. Finally, the word “five” is unnecessary and redundant, as exactly five numbers are listed. A better form of the question would be: “What is the mean of the following numbers: 5, 8, 13, 21, 34?”

The key takeaway is that you should be as brief as possible when writing Knowledge Quiz questions, while still conveying all relevant information.

Be Able to Defend Your Questions and Answers

It is important to remember the stakes for the candidates when writing job knowledge questions – these questions you write help determine whether they are able to start a new career with your company. As such, it is extremely important to ensure accuracy in the questions in the Knowledge Quiz. For every single question, you should be able to justify why the correct answer(s) are correct, and why the incorrect answers are wrong.

When choosing the correct option, it should be correct 100% of the time. Two plus two is four. The US Declaration of Independence was signed in 1776. These are defensibly correct answers with clear sources indicating their accuracy. However, a question like “What is the best word processing program?” will have no defensibly correct answers, as any response to that is a matter of opinion. Consider the following question that might be used in the IT industry:

What operating system lacks a graphical user interface (GUI)?

1. Windows
2. iOS
3. Linux
4. Android

At first glance, it would seem as though the correct answer would be c) Linux, as Windows, iOS, and Android all clearly have GUIs, and Linux often does not. But Android is a version of Linux with a GUI, and many Linux distributions come with a GUI. So this would be a poor question to include – there is no justifiably correct answer that is always correct.

Equally as important is being able to defend that the incorrect answers are, in fact, incorrect. Each incorrect response option should never be correct under any circumstances. If a response option marked as incorrect might be correct, even under very limited circumstances, it should not be included. Consider the following question about computer usage:

Which is the best procedure to take a screenshot?

1. Press the “print screen” button on your keyboard.
2. Take a picture of your computer screen with your phone.
3. Press the buttons CTRL-ALT-DELETE at the same time.
4. Navigate to Display Settings and choose “Identify.”

Under most normal circumstances, a) is the correct answer. Options c) and d) are defensibly incorrect, as using these procedures will never result in taking a screenshot. But option b) can be correct – not only will this procedure capture what appears on the screen, but it might be the only option available in the case of a massive failure on the computer. So option b) would be a poor choice of a response option, and should be replaced with something that is very clearly wrong.

The key takeaway is that you should be able to justify why every correct response option is always correct, and why every incorrect response option is always incorrect.

Use Correct Grammar and Spelling

There are multiple reasons to use correct grammar and spelling when constructing questions for the Knowledge Quiz. First, using correct grammar and spelling makes it easier for candidates to focus on the content of the question and display their job-related knowledge. If candidates have to spend time deciphering what is intended by the question, it can distract and inhibit them from answering correctly, and well-qualified candidates may be screened out. Additionally, the questions appearing in the Knowledge Quiz help form the candidates’ first impressions about your company. If candidates are turned off by the grammar and spelling they see, then they may withdraw their applications and you may lose out on great candidates.

The key takeaway is that you should use correct grammar and spelling in your questions and response options.

Avoid Tipping the Correct Answer to the Candidate

A common mistake made during the construction of the Knowledge Quiz is for items or response options to provide unintended clues for the candidate to figure out the correct answer. In these situations, the candidates are not demonstrating their job knowledge; instead, candidates are displaying how “test savvy” they are. This inhibits proper measurement of job knowledge, leading you to pass candidates who may not have the job knowledge necessary to successfully perform on the job.

Here is a list of some common mistakes to avoid:

• Overusing “All of the above” or “None of the above” response options.
  • Though the use of these response options can be perfectly valid, it is common for test writers to overuse these as the correct response option. When this happens, the test savvy candidate would know to simply guess the “All/None of the Above” option. The best option is to simply avoid writing these types of questions. If you do decide to use these, be sure that their use as the correct answer is proportional to the number of items and number of response options. For example, a quiz with 20 multiple choice questions where each question has 4 response options and all items have an All/None of the Above response option should have approximately five of those options being marked as correct (i.e., 20 questions x 1/4 chance of guessing).
• Having the correct option(s) be significantly longer/shorter.
  • It is common for test writers to unintentionally have the correct answer(s) be noticeably longer or shorter than other response options. Avoid this mistake by having multiple response options be of similar length and detail.
• Having one question provide answers to another question.
  • Sometimes, the question text or one of the response options in one question will provide clues to the correct answer of another question in the quiz. This is problematic because it means the quiz is assessing reading comprehension, not the underlying job knowledge required to be successful on the job.

Make the Quiz Structure Intuitive

When setting up and designing quizzes, it is extremely important to keep the focus of the quiz on the job knowledge you are trying to assess, and not on the quiz itself. Using an intuitive quiz structure helps accomplish this by reducing test anxiety that candidates might feel and by helping to eliminate the effects that candidates’ test familiarity or test savviness may have. Here are some tips to help make the quiz structure intuitive:

• Order the response options (if any) in a logical way. This might include ordering numeric response options in ascending/descending order or ordering them based on the length of the response option. This helps make the quiz and options flow well together for the candidates, so they can focus on demonstrating their knowledge.
• If you are using multiple question types in your quiz, group questions of similar type together – have all the multiple-choice questions together, all the checkbox questions together, and so on. Try to avoid mixing different question types together, as that increases cognitive demand on the quiz and puts the attention back on the quiz structure and not focused on the required job knowledge.

Check the Setup for Your Quiz Once Complete

Once you have completed the setup of your Knowledge Quiz, take a look back at. Pay attention to the overall structure of it and how well it flows. Your goal should be to accurately assess candidates’ job-relevant knowledge, not how good they are at taking quizzes and tests. This might be a good time to actually take the quiz as an candidate would – this will help you spot any challenges that might arise before candidates are affected.

Tips for Multiple Choice Questions

Multiple-choice questions are the most commonly used question type and are excellent choices when you want the candidate to determine the only correct option among several incorrect answers. Here are some tips to construct better multiple-choice questions:

• Use at least four response options. Remember that it is possible for candidates to guess the correct answer even if they do not know it. The fewer response options there are, the more likely it is the candidate will guess correctly. A question with two response options (i.e., one right, one wrong) will have a 50% chance of being guessed correctly. Using four response options reduces that chance of guessing to 25%.
• Pay attention to the distribution of correct answers. You want to make sure the correct response options are about equally distributed to avoid providing any clues to candidates about the correct answer. For example, if half of the correct answers in a multiple-choice quiz appear in option B, then it might signal to the candidates that if they are in doubt, they should choose option B.

Tips for Checkboxes Questions

Checkboxes are excellent options when questions can have one or more clear answers. They are difficult for candidates to guess correctly, as the candidate has to go through each option individually and evaluate whether or not it is correct. Here are some tips maximize the usefulness of checkboxes questions:

• Avoid combining answers already included. For example, consider the question “Which of the following are word processing programs on Windows?” with response options that include “Microsoft Word”, “Notepad”, and “Both Microsoft Word and Notepad”. The third option there is not only unnecessary (candidates already have to evaluate whether Microsoft Word and Notepad are correct), but also misleading – it is unclear whether the candidate is required to select all three of those options, or only the third.
• Avoid “all of the above” responses. Similarly, because candidates are already evaluating whether each response option is correct, “all of the above” is redundant and misleading. It is best to avoid using this option.
• Remember that no partial credit is given. If you have a question with ten response options and eight of them are correct, the candidate who chooses seven of the eight will be marked wrong, as will the candidate who chose zero of those eight. If it is important to distinguish between those two candidates, consider revising the question or choosing a different question type.

Tips for Numeric Questions

Numeric questions are excellent options when there is a clear, single correct numeric answer. They are extremely difficult for candidates to guess the right answer – rather, the candidate must calculate and supply the correct answer. Be careful to clearly set expectations for the correct answer within the question text:

• Specify the number of decimal points. If the correct answer has decimal values, make sure the candidate knows how many digits after the decimal to enter. For example, consider the question “What is the value of pi?” If an candidate enters “3.14159”, it is clear this candidate knows the correct answer. However, if you entered the correct answer as “3.14”, the candidate may be marked as incorrect, simply for supplying too much information. A better question would be “What is the value of pi to two decimal places?” as that clearly specifies how much information the candidate should provide.
• Avoid units in the correct answer. Many numeric questions might require units (e.g., meters, gallons) in the correct answer. However, it is advisable not to require candidates to supply the units in their answer, as not all possible correct answers can be accounted for. For example, if you specify the correct answer as “12 kilometers”, the candidate might be marked as incorrect for answering “12 km” or even “12”, despite clearly knowing and supplying the correct answer. A better option would be to only require the number in the answer field, and provide the correct unit in the question itself.