Testing some multiple choice questions, I came across an issue that seems curiously little discussed in education circles.

The case is the multi-select, multiple choice question, where a learner may select x number of correct answers from y number of options.

The client particularly did not want to divulge how many correct answers there were, so out of 8 options, the learner could choose anything up to 8 answers. The client’s point of view was that telling the learner how many answers were correct was too much of a clue, and that part of what they were seeking to test was the learners’ ability to confidently distinguish the true statements from the false, rather than just guess the most plausible.

However, this can have large implications.

If we say “Choose the correct 2 options from the 8 options listed”, the learner must always choose 2 options. This means the total possible number of responses are:

8!/2!⋅(8−2)!​=28

So there are 28 ways to answer this question.

But if we don’t tell the learner there are 2 correct options they could choose 1, 2, or anything up to all 8. The total number possible responses is the total of the possible combinations. So…

8+28+56+70+56+28+8+1=255

Depending on how the marking rubric is stored, this could make quite a difference. A flat table of possible responses would require 255 rows.

Where it gets even more complicated is when we want to give partial scores for partially correct answers. Lets say the learner gets 2 marks for each correct selection. 4 marks in total are possible. But what if they choose the 2 correct options and a third incorrect one? What if they choose six? It is not obvious how such responses would be scored.

I posed this question to Co-Pilot and it simply recommended specifying the correct number of options to minimize these difficulties. So it seems there are no easy answers here!