Writing good quiz questions

Why it matters

A quiz question has two jobs: to help learning (retrieval practice) and to tell you what learners actually understand. A badly written multiple-choice question fails at both. If the wrong answers are obviously silly, learners can pass by elimination. If the question is a trick, good learners get it wrong. Well-written questions, especially well-written distractors, let you diagnose specific misconceptions.

Anatomy of a multiple-choice item

• Stem: the question or problem.
• Key: the correct answer.
• Distractors: the incorrect options.

Thomas Haladyna, Steven Downing and Michael Rodriguez (2002) reviewed item-writing advice and research and published a widely used set of guidelines. The core of it:

Stem guidelines

• Put the main idea in the stem; a learner should be able to answer it without looking at the options.
• Ask about one important point, tied to an objective.
• Use a realistic scenario when the objective is Apply or above.
• Word it positively. If you must use NOT or EXCEPT, make it bold.

Option guidelines

• Make every distractor plausible to someone who has not mastered the objective.
• Keep options similar in length and grammar (the longest, most detailed option is often the key, and learners notice).
• Avoid "all of the above" and "none of the above".
• Vary the position of the key.
• Do not overlap options or make two of them correct.

How many options?

Rodriguez's 2005 meta-analysis found that three options (one key, two good distractors) is usually enough: adding a weak fourth option barely improves the item and takes longer to write and to read. Two strong distractors beat three weak ones.

Distractors that diagnose

The best distractors come from real mistakes. Collect them from practice sessions, support tickets and open-ended answers. Each one should map to a specific misconception, so that a wrong answer tells you what to reteach.

Example from the Git workshop:

You changed app.js, ran git commit -m "Update", and Git said "nothing added to commit". Why?

• A. You need to push before you can commit. (misconception: confuses local commit with remote push)
• B. You did not stage the file with git add first. (key)
• C. Your commit message was too short. (misconception: thinks Git validates messages)

If many learners pick A, they have not separated local and remote repositories, so you know which part to revisit. That is far more useful than knowing that 60% got it right.

Compare a weak version:

What does git add do?

• A. Stages changes for the next commit
• B. Makes coffee
• C. Deletes the repository
• D. All of the above

It tests only recall, two distractors are absurd, and "all of the above" adds nothing.

Writing for higher levels of Bloom's taxonomy

Multiple-choice items can test more than recall when the stem presents a situation:

• Apply: "Which command creates and switches to a branch called docs?"
• Analyze: show a git log --oneline --graph output and ask which branch a commit is on, or why a merge created a merge commit.
• Evaluate: "Your team forbids rewriting shared history. Which of these is the best way to undo a pushed commit?"

For skills, combine multiple-choice checks with practical tasks (see Backward design).

Other question types

• Short answer reveals misconceptions you did not anticipate. Use it in pilots to harvest distractors.
• Two-tier questions ask for an answer and then the reason, separating lucky guesses from understanding.
• Confidence ratings ("How sure are you?") help learners calibrate.

Feedback on quiz items

In practice quizzes, give feedback that explains why the key is right and why the chosen distractor is wrong. Feedback that addresses the specific misconception is more useful than "Incorrect, the answer is B".

Review your items with data

After a run, look at two simple statistics:

• Difficulty (p-value): the proportion who answered correctly. Very high (above about 0.9) may be too easy; very low may be confusing or untaught.
• Discrimination: do high scorers on the whole quiz get this item right more often than low scorers? If not, the item may be flawed.
• Distractor analysis: a distractor nobody picks is doing no work; replace it.

Common mistakes

• Testing trivia because it is easy to write.
• Implausible distractors.
• Clues in grammar or length.
• Trick questions and double negatives.
• No link to an objective.

Quick recap

• Clear stem, one key, plausible distractors, no "all of the above".
• Three good options are usually enough.
• Base distractors on real misconceptions so wrong answers diagnose.
• Use scenarios for higher-level thinking; review items with data.

Try it

On the Do tab, write diagnostic questions for a topic you teach.