How to Write eLearning Assessments That Measure Real Skill

The final quiz has ten questions, and most learners pass on the first try. The course reports look great. Then a supervisor mentions that half the new team still can't do the task without help. Somewhere between the quiz and the job, the measurement broke. It's rarely anyone's fault: most quizzes are written to check memory because memory is easy to check. So how do you write eLearning assessments that measure real skill?

To write an eLearning assessment that measures real skill, start from the job task, not the course content, and ask learners to make the same decisions or perform the same steps they'll face at work. That means scenario-based questions with realistic details, answer choices built from the mistakes people actually make, and performance tasks like simulations or work samples for the skills that matter most. Every question should trace back to a specific objective. Then check your items with real data after launch and set your pass score based on what competent performance looks like, not on a habit like 80%.

Why Do So Many eLearning Assessments Miss the Point?

Many assessments miss the point because they're written from the course content after the course is built. The writer flips through the screens and turns facts into questions. The result tests whether someone remembers screen 12, not whether they can do the job.

A second cause is misaligned objectives. If the objective says "understand the returns policy," almost any question fits, so the easiest ones get written. If the objective says "decide whether a return qualifies for a refund, an exchange or neither," the assessment practically writes itself.

Carnegie Mellon's Eberly Center has a clear guide to writing measurable learning objectives, and it's worth a read before you write a single question. Measurable verbs make measurable assessments.

How Do You Align Questions With Real Job Tasks?

You align questions with real job tasks by building a simple map before you write anything: task, objective, practice and assessment, all in one row. If a row has a gap, you've found a problem early.

Notice what's missing from the right column: definitions, dates and "Which of the following is true?" Those items have a place in quick knowledge checks during the course. They rarely belong on the assessment that decides whether someone is ready.

How Do You Write Scenario-Based Questions?

You write scenario-based questions by giving learners a short, realistic situation, asking what they would do, and offering choices that all sound reasonable to someone who doesn't fully understand the skill.

A Before-and-After Example

Here's a typical recall question:

How many days does a customer have to return an item with a receipt? (a) 14 (b) 30 (c) 60 (d) 90

And here's the same objective as a scenario:

Store policy: refunds within 30 days with a receipt for unused items; exchanges only for used items in working order. A customer brings back a blender bought 25 days ago. She has the receipt, but the box is missing and the blender has been used once. She wants her money back. What should you do? (a) Refund in full (b) Offer an exchange only (c) Decline the return (d) Call a manager

The second question tests the policy in context: timing, receipt, condition and the customer's request together. The best answer is (b), and only someone who applies every part of the policy will land on it. Each wrong answer represents a real mistake an employee might make. That's what makes it diagnostic.

Rules for Strong Answer Choices

• Build distractors from real errors. Ask supervisors what new employees get wrong.
• Keep all choices similar in length and detail, so the right one doesn't stand out.
• Avoid "all of the above" and "none of the above."
• Make sure exactly one answer is best, and have a subject matter expert (SME) confirm it.
• Avoid trick wording and double negatives. You're testing the skill, not reading stamina.

When Should You Use Performance-Based Assessments?

Use performance-based assessments when the skill involves steps, tools or judgment under realistic conditions, and when getting it wrong has real consequences. Multiple-choice can test decisions. It can't test whether someone can actually do the procedure.

Practical formats include:

• Software simulations where learners complete a task in a copy of the system.
• Branching scenarios where each choice leads to realistic consequences.
• Sequencing and hotspot activities for procedures and equipment.
• Uploaded work samples, like a completed form or a recorded customer greeting.
• Observed checklists, where a supervisor confirms the learner performed the task on the job.

Blended assessment is often the best answer. The eLearning course handles knowledge and decisions, then a short observed checklist confirms the hands-on part. Each piece does what it does best.

How Many Questions Does an Assessment Need?

An assessment needs enough questions to cover every key objective at least twice, and no more. Two items per objective is a practical minimum, because a single question can be missed for reasons that have nothing to do with skill, like a misread word.

Say a course has five objectives. Ten to fifteen well-written scenario items usually give you a reliable picture. Doubling that to thirty mostly adds fatigue. If you want to reduce answer sharing between learners, build a larger pool and draw a random set from it for each attempt, making sure every objective is still covered each time.

What Kind of Feedback Should Learners Get?

Learners should get feedback that explains why an answer is right or wrong, tied to the reasoning the job requires. "Incorrect" teaches nothing. "Not quite. Because the blender has been used, the policy allows an exchange but not a refund" teaches the rule in context.

For practice questions inside the course, give that feedback right away. For a final assessment, many teams show results by objective at the end instead of revealing every answer, which keeps the item pool useful while still telling learners what to review.

How Should You Set a Passing Score?

Set a passing score based on what a minimally competent performer would get, not on a default number. Many organizations use 80% out of habit, but that number means very different things depending on how hard the questions are.

One established approach is a simplified version of the Angoff method. Several experts look at each question and estimate the probability that a just-competent employee would answer it correctly. Average those estimates across questions and experts, and you have a defensible cut score.

A quick example, with invented ratings: three supervisors rate a 10-question assessment. Their average estimates across all items come to 0.72. So a just-competent employee would be expected to score about 72%, and that becomes your passing score. For safety-critical items, you might also require that certain questions be answered correctly no matter what the overall score is.

How Do You Check Whether Your Questions Work?

Check whether your questions work by looking at item-level data after a few dozen learners have taken the assessment. Two simple numbers tell you a lot.

1. Difficulty: the share of learners who answer correctly. If nearly everyone gets it right, the question may be too easy to tell you anything. If very few do, either the question or the teaching needs work.
2. Discrimination: whether people who do well overall tend to get this question right. If strong performers miss a question as often as weak ones, the question is probably confusing or flawed.

Also look at which wrong answers are popular. If most people who miss a question pick the same distractor, that distractor is describing a common misunderstanding, and the course should address it directly.

How Do Assessments Fit Into Measuring Training Effectiveness?

Assessments fit into measuring training effectiveness as the bridge between learning and performance. A good assessment tells you whether people left the course able to do the task. It can't tell you whether they actually do it on the job, or whether business results improved.

That's why assessment scores should sit alongside other signals: manager observations, quality data and business metrics. If you want a fuller picture, our guide on how to measure training effectiveness covers the signals that matter beyond the final quiz.

Measure the Job, Not the Slides

Writing an eLearning assessment that measures real skill means starting with the job task and working backward. Map each task to an objective and an item, write scenarios with realistic details and distractors built from real mistakes, use performance tasks where the stakes are high, and set a pass score that reflects real competence.

Then let the data keep you honest. Review item results after launch, fix the questions that don't work and adjust the course where everyone stumbles. Do that, and your assessment scores will finally mean what your stakeholders assume they mean.

← All articles