Use the science of learning to design lessons, objectives, materials and assessments that actually change what people can do.

Instructional Design & How People Learn

Pick one objective at Apply level or higher.

List three real mistakes or misconceptions learners have about it (from experience, forums, support tickets or your own early mistakes).

Write a scenario stem that can be answered without looking at the options.

Write the key and two distractors, each based on one of the misconceptions. Note beside each distractor which misconception it reveals.

Check the options: similar length and grammar, no 'all of the above', no overlap, only one defensible answer.

Write feedback for each option that explains why it is right or wrong.

Repeat for two more questions, placing the key in a different position each time.

Give the three questions to someone who knows the topic and someone who does not. If the novice can pass by elimination, rewrite the weak distractors.

## Why it matters

A quiz question has two jobs: to help learning (retrieval practice) and to tell you what learners actually understand. A badly written multiple-choice question fails at both. If the wrong answers are obviously silly, learners can pass by elimination. If the question is a trick, good learners get it wrong. Well-written questions, especially well-written **distractors**, let you diagnose specific misconceptions.

## Anatomy of a multiple-choice item

- **Stem:** the question or problem.
- **Key:** the correct answer.
- **Distractors:** the incorrect options.

Thomas Haladyna, Steven Downing and Michael Rodriguez (2002) reviewed item-writing advice and research and published a widely used set of guidelines. The core of it:

### Stem guidelines

- Put the main idea in the stem; a learner should be able to answer it without looking at the options.
- Ask about one important point, tied to an objective.
- Use a realistic scenario when the objective is Apply or above.
- Word it positively. If you must use NOT or EXCEPT, make it bold.

### Option guidelines

- Make every distractor **plausible** to someone who has not mastered the objective.
- Keep options similar in length and grammar (the longest, most detailed option is often the key, and learners notice).
- Avoid "all of the above" and "none of the above".
- Vary the position of the key.
- Do not overlap options or make two of them correct.

### How many options?

Rodriguez's 2005 meta-analysis found that **three options** (one key, two good distractors) is usually enough: adding a weak fourth option barely improves the item and takes longer to write and to read. Two strong distractors beat three weak ones.

## Distractors that diagnose

The best distractors come from **real mistakes**. Collect them from practice sessions, support tickets and open-ended answers. Each one should map to a specific misconception, so that a wrong answer tells you what to reteach.

Example from the Git workshop:

> You changed `app.js`, ran `git commit -m "Update"`, and Git said "nothing added to commit". Why?
> - A. You need to push before you can commit. *(misconception: confuses local commit with remote push)*
> - B. **You did not stage the file with `git add` first.** *(key)*
> - C. Your commit message was too short. *(misconception: thinks Git validates messages)*

If many learners pick A, they have not separated local and remote repositories, so you know which part to revisit. That is far more useful than knowing that 60% got it right.

Compare a weak version:

> What does `git add` do?
> - A. Stages changes for the next commit
> - B. Makes coffee
> - C. Deletes the repository
> - D. All of the above

It tests only recall, two distractors are absurd, and "all of the above" adds nothing.

## Writing for higher levels of Bloom's taxonomy

Multiple-choice items can test more than recall when the stem presents a situation:

- **Apply:** "Which command creates and switches to a branch called `docs`?"
- **Analyze:** show a `git log --oneline --graph` output and ask which branch a commit is on, or why a merge created a merge commit.
- **Evaluate:** "Your team forbids rewriting shared history. Which of these is the best way to undo a pushed commit?"

For skills, combine multiple-choice checks with practical tasks (see Backward design).

## Other question types

- **Short answer** reveals misconceptions you did not anticipate. Use it in pilots to harvest distractors.
- **Two-tier questions** ask for an answer and then the reason, separating lucky guesses from understanding.
- **Confidence ratings** ("How sure are you?") help learners calibrate.

## Feedback on quiz items

In practice quizzes, give feedback that explains why the key is right *and* why the chosen distractor is wrong. Feedback that addresses the specific misconception is more useful than "Incorrect, the answer is B".

## Review your items with data

After a run, look at two simple statistics:

- **Difficulty (p-value):** the proportion who answered correctly. Very high (above about 0.9) may be too easy; very low may be confusing or untaught.
- **Discrimination:** do high scorers on the whole quiz get this item right more often than low scorers? If not, the item may be flawed.
- **Distractor analysis:** a distractor nobody picks is doing no work; replace it.

## Common mistakes

- Testing trivia because it is easy to write.
- Implausible distractors.
- Clues in grammar or length.
- Trick questions and double negatives.
- No link to an objective.

## Quick recap

- Clear stem, one key, plausible distractors, no "all of the above".
- Three good options are usually enough.
- Base distractors on real misconceptions so wrong answers diagnose.
- Use scenarios for higher-level thinking; review items with data.

## Try it

On the Do tab, write diagnostic questions for a topic you teach.


Write multiple-choice items with clear stems and distractors drawn from real misconceptions.

Writing good quiz questions

How to write MCQ distractors that work: use your students' real mistakes

Writing distractors for multiple-choice questions

Describe the task in two sentences and list the objectives it assesses.

Turn the objectives into three to five criteria. Avoid vague criteria such as 'effort' or 'professionalism' unless you can describe them.

For each criterion, write the 'proficient' description first, in observable terms.

Write the lower levels by what is missing or wrong, and the top level by what goes further.

Read each row and replace any adjective-only differences ('good', 'very good') with observable ones.

Find or create two samples of work and score them with the rubric. If possible, have a colleague score them too and compare.

Rewrite any description you and your colleague scored differently.

For one sample, write feedback with at most three points that answers: where am I going, how am I going, and where to next.

## Why it matters

Practical tasks, projects and presentations cannot be marked by an answer key. Without clear criteria, two assessors give different scores, learners do not know what "good" looks like, and feedback becomes vague ("nice work, tidy up the code"). A **rubric** makes quality visible, and good **feedback** turns an assessment into the next step of learning.

## What a rubric is

A rubric lists the **criteria** you will judge, the **performance levels** for each, and a **description** of what each level looks like. Susan Brookhart's *How to Create and Use Rubrics for Formative Assessment and Grading* (2013) is a practical reference.

Two main types:

- **Analytic rubric:** scores each criterion separately. Better for feedback, because learners see which part needs work.
- **Holistic rubric:** one overall score from a description of the whole. Faster, useful for quick judgments, but less diagnostic.

## Example: analytic rubric for a Git team project

Task: in pairs, add a feature to a small app using a branch and a pull request.

| Criterion | 1 Beginning | 2 Developing | 3 Proficient | 4 Exemplary |
|---|---|---|---|---|
| Branching | Work committed directly to `main` | Feature branch used, but unrelated changes mixed in | One feature branch with only related changes | As 3, and branch name and scope follow the team convention |
| Commits | One large commit, unclear message | Several commits, messages vague ("fix") | Small commits, each message says what and why | As 3, and history tells a clear story a reviewer can follow |
| Pull request | No PR, or no description | PR with a title only | PR explains the change and how to test it | As 3, and responds to review comments with follow-up commits |
| Conflict handling | Conflict left unresolved or work lost | Resolved, but tests fail | Resolved, tests pass | As 3, and explains the resolution in the PR |

Notice what makes it work:

- **Criteria match the objectives**, not generic qualities like "effort".
- **Descriptions are observable**, so two assessors would agree.
- **Levels describe quality**, not just quantity ("several commits" vs "small commits, each with what and why").
- Four levels avoid a lazy middle choice; some teams prefer three.

## Writing a rubric step by step

1. List the objectives the task assesses.
2. Turn each into a criterion (three to six is plenty).
3. Describe **proficient** first: what meets the standard?
4. Describe the levels below by what is missing or wrong, and the level above by what goes further.
5. Test it: score two or three real samples with a colleague and compare. Rewrite any description you disagreed on.
6. Share the rubric with learners **before** the task, ideally with an example of proficient work.

## Feedback that moves learning forward

John Hattie and Helen Timperley (2007) framed effective feedback as answering three questions:

- **Where am I going?** (the goal, which the rubric shows)
- **How am I going?** (current performance against it)
- **Where to next?** (the specific next step)

Feedback can also do harm. Avraham Kluger and Angelo DeNisi's 1996 meta-analysis found that in roughly a third of cases, feedback interventions actually *lowered* performance, especially when feedback pointed attention at the person ("you're a natural") rather than the task.

Practical rules, drawing on Paul Black and Dylan Wiliam's work on formative assessment ("Inside the Black Box", 1998):

- **Be specific and about the task:** "Your commit messages say what changed but not why; add the reason in one clause" beats "improve your commits".
- **Make it actionable now:** give feedback while learners can still use it, before the final submission.
- **Limit it:** two or three points learners can act on, not twenty.
- **Make learners do something with it:** for example, ask them to revise one commit message and resubmit. Wiliam puts it this way: feedback should be more work for the recipient than the donor.
- **Comments and grades:** Ruth Butler (1988) found that students given comments only improved more than those given grades, or grades plus comments, which suggests a grade can draw attention away from the comment. When you can, give comments first and the grade later.

## Peer and self-assessment

A clear rubric lets learners assess their own and each other's work. Have pairs swap pull requests and score them with the rubric before submitting. They learn the standard by applying it, and you get fewer weak submissions.

## Common mistakes

- Criteria like "creativity" or "professionalism" with no description.
- Levels that differ only by adjectives ("good", "very good", "excellent").
- Rubrics shared only after marking.
- Long feedback lists with no clear next step.
- Praise aimed at the person rather than the work.

## Quick recap

- A rubric = criteria + levels + observable descriptions.
- Analytic rubrics give better feedback; holistic rubrics are faster.
- Write proficient first; test with real samples and a colleague.
- Feedback answers where am I going, how am I going, where to next, and asks learners to act on it.

## Try it

On the Do tab, write an analytic rubric for a project in your subject.


Write an analytic rubric for a project and give feedback that learners can act on.

Rubrics and feedback

How to Create Rubrics for Assignments

Providing feedback that moves learning forward

## Why it matters

In any group you teach, some people have disabilities (visible or not), some are working in a second language, some join on a phone in a noisy space, and some are tired or anxious. If accessibility is an afterthought, these learners hit barriers: an untagged PDF, an uncaptioned video, a color-only diagram, a timed quiz. Designing for this variety from the start is cheaper than retrofitting, often helps everyone, and in many workplaces and schools it is also a legal requirement.

## Universal Design for Learning

**Universal Design for Learning (UDL)** is a framework developed by CAST, a US nonprofit, beginning in the 1980s and 1990s (David Rose and Anne Meyer were among its founders). It takes its name from universal design in architecture, where curb cuts built for wheelchair users turned out to help people with strollers, carts and luggage too.

UDL asks designers to provide options along three principles (CAST released version 3.0 of the UDL Guidelines in 2024):

| Principle | The "why", "what" or "how" of learning | Examples in a Git workshop |
|---|---|---|
| Multiple means of **engagement** | Why: interest, effort, self-regulation | Choice of practice scenario; clear goals; a safe practice repo where mistakes cost nothing |
| Multiple means of **representation** | What: how information is presented | Diagram plus spoken explanation plus written handout; captions; commands available as copyable text; key terms defined |
| Multiple means of **action and expression** | How: how learners show what they know | Use the terminal or a graphical client; explain a resolution verbally or in writing; keyboard-accessible tools |

UDL is not about lowering standards. The objective stays "resolve a merge conflict so tests pass"; UDL removes barriers that are not part of that objective. If the objective is about Git, then reading a low-contrast screenshot is not a fair part of the test.

A note on evidence: UDL is a design framework with broad adoption and a growing research base, but reviews describe the evidence for its overall effect on learning outcomes as still developing. Its concrete accessibility practices (captions, alt text, structure) rest on firmer ground and on legal standards.

## Accessibility standards you will meet

- **WCAG** (Web Content Accessibility Guidelines) from the W3C, currently version 2.2, organized around content being **perceivable, operable, understandable and robust**. Most organizations aim for level AA.
- In the US, **Section 508** (federal agencies and their vendors) and the **ADA**; in the EU, the **European Accessibility Act**. Requirements and deadlines differ by country and change over time, so check with your organization's accessibility or legal team.

## A practical checklist for training materials

**Documents and slides**
- Use real headings, lists and built-in layouts so screen readers can navigate.
- Add meaningful alt text to informative images; mark decorative ones as decorative.
- Contrast at least 4.5:1 for normal text; never rely on color alone (add labels or patterns).
- Use the accessibility checker built into Word, PowerPoint or Google tools.

**Video and audio**
- Accurate captions (correct technical terms); transcripts for audio.
- Describe important visual actions in the narration.

**E-learning and quizzes**
- Everything works with a keyboard; focus is visible.
- Avoid time limits unless speed is the skill; if used, allow extensions.
- Clear instructions and error messages.

**Live and virtual sessions**
- Share materials in advance so people can preview or enlarge them.
- Turn on live captions in the meeting tool; speak one at a time.
- Offer several ways to participate: speak, chat, poll, or follow-up message.
- Ask, privately and early, whether anyone needs adjustments.

## Applying UDL to a lesson: example

Original plan: 40-minute lecture with dense slides, a live demo in small font, and a timed 10-question quiz.

UDL revision:

1. Share slides and the worked example sheet the day before (representation, engagement).
2. State the goal and show what success looks like (engagement).
3. Demo in large font with narration, captions on, commands pasted into chat (representation).
4. Practice in pairs or solo, terminal or graphical client (action and expression).
5. Untimed practical task instead of a timed quiz; learners can explain their resolution in writing or out loud (action and expression).
6. Recording with corrected captions posted afterwards.

## Common mistakes

- Treating accessibility as a final check instead of a design input.
- Equating UDL with "learning styles": UDL is about removing barriers and offering options, not diagnosing learner types.
- Fixing only what an automated checker flags; also test with a keyboard and a screen reader.
- Assuming no one in the group needs accommodations because no one said so.

## Quick recap

- UDL: multiple means of engagement, representation, and action and expression.
- Keep the objective fixed; remove barriers that are not part of it.
- WCAG 2.2 AA is the common target; legal requirements vary, so check locally.
- Captions, alt text, structure, contrast and keyboard access are the basics.

## Try it

Answer the quiz on the Do tab. Then pick one of your own materials and run your tool's accessibility checker on it.


Apply Universal Design for Learning and basic accessibility practice so materials work for every learner.

Accessible learning

UDL At A Glance

Introduction to UDL

Write questions that diagnose misconceptions, rubrics that make quality visible, and materials that work for every learner.