Designing and Managing MCQs:

Chapter 2. What are Multiple Choice Questions?

• 2.1 Definition
• 2.2 Advantages of Multiple Choice Questions.
• 2.3 Problems associated with Multiple Choice Questions
• 2.4 The multiple uses of Multiple Choice Question tests
• 2.5 Managing the use of MCQs at departmental and course team level.

2.1 Definition

A "multiple choice question" (MCQ) is a question in which students are asked to select one alternative from a given list of alternatives in response to a "question stem".

For example,

Question stem:

"Which one of the following is the currency unit used in Albania?"

Alternatives:

1. Dinar
2. Rouble
3. Kwacha
4. Lek
5. Rupee

The student answers the question by selecting a number 1-5, and entering this either on a form provided for the purpose ("off-line", in computer jargon), or at a computer keyboard ("on-line"). In this example, the correct alternative (the "answer") is 4. The other incorrect alternatives are sometimes called "distractors".

One feature of this type of question is that there is a finite possibility of selecting the correct alternative by purely guessing. In fact, in this example you have 1 chance in 5 of answering the question correctly if you know absolutely nothing about the currencies in use in various countries. This problem is not insurmountable, and will be discussed further in Chapter 4 .

The so-called "TRUE/FALSE" question, in which the student chooses between two alternatives "True" or "False", is a special case of an MCQ. "True/False" questions are used extensively in the UCT Medical School.

Certain MCQs have more than one correct response in the set of alternatives. Such questions should not be used, not only because they tend to confuse students, but also because not all computer programmes available at UCT cater for such a question type. For example, the programme "MCQ", which marks the responses of students who have written the test "off- line", does not admit this type of question. On the other hand, the programme "Question Mark", which is designed principally for "on-line" testing, does allow a much broader range of question types. These questions, however, are not, strictly speaking, MCQ's. As a result, they fall outside the scope of this document.

Return to Contents of this Chapter

2.2 Advantages of Multiple Choice Questions.

• MCQs can be easily marked/scored, and this scoring can be both accurate and objective. In particular, MCQs can be MACHINE SCORED (This means that a computer will do all the marking for you!)
• MCQs can be set at different cognitive levels. For example, a question may simply challenge a student's ability to recall facts, while another may test a student's ability to apply factual knowledge to given situations; or, at a higher level, a question may test a student's ability to evaluate given information. For a discussion of this, please turn to Appendix C .
• MCQs can be designed with a diagnostic end in mind, in order to find out whether specific areas of a given subject are adequately known, or in order to detect misconceptions. This can provide feedback on the effectiveness of instruction.
• Students with poor reading skills and second-language learners need not be disadvantaged, provided the tests are designed appropriately Misreading of a question stem or an alternative may result in the loss of a few marks only; conversely, the misreading of an essay-type question, followed by the poor writing skills normally associated with a second language, can cause the student to lose an inordinate number of marks.
• Statistical information on performance can be readily obtained. One can not only find out how the class performed on a particular question, but determine whether the question was suitable in the context in which it was presented. The statistics that are gathered may also be used to rank questions with respect to their difficulty and their ability to discriminate between students of different competences.
• Tests made up of MCQs can be administered frequently, and thus provide regular information on student performance, not only to the instructor but also to the students.
• Tests made up of MCQs can provide a better coverage of content than essay-type questions, thus broadening the scope of the test. Such tests can be structured to include questions of defined grades of difficulty and discriminating power (See Chapter 4 ).
• In MCQ testing, the instructor "sets the agenda", and there are no opportunities for the candidate to avoid complexities and concentrate on the superficial aspects of the topic, as is often encountered in essay-type questions.

Return to Contents of this Chapter

2.3 Problems associated with Multiple Choice Questions.

• The construction of good test items (questions) requires special care and is therefore time-consuming.
• Instructors tend to favour "recall" type questions, as they are the easiest to design. This manual will provide you with examples which should convince you that it is possible to go far beyond this type of question.
• Experience has shown that in some environments, it may be necessary to win the acceptance of students to this type of testing. Proper communication with the students is therefore essential in order to ensure that they feel comfortable with this type of testing. Feedback from students can be very important and can only lead to an overall improvement of the existing question, for possible inclusion into a question bank.
• Creativity cannot easily be tested (the SYNTHESIS level of Bloom's taxonomy is very difficult to test by using MCQs). Creativity is best tested by discursive questions such as the "ESSAY-type" question. Bear in mind however that essay questions tend to have a poor sampling of content, have a poor scoring reliability (particularly if several persons are marking large numbers of scripts), and are notoriously time consuming to mark. (Bloom's taxonomy is discussed further in Appendix C ).
• Candidates cannot justify choices that have been deemed "incorrect" by the examiner. Two ways (apart from the necessary clarity of language in the stem and alternatives) of removing this problem are:

  [a] building the argument into the alternative. Examples of this are given in later sections of this document.

  [b] allowing students to discuss the questions with the lecturers and tutors after the test. In addition to allowing issues to be clarified, the questions might possibly need to be ignored in the calculation of test scores, and corrected for or eliminated from future tests.

Return to Contents of this Chapter

2.4 The multiple uses of Multiple Choice Question tests

The most obvious use of MCQs is in formal class tests and examinations, and this use is covered elesewhere in this handbook. In this section we want to draw your attention to a number of other uses, such as:

• Designing a tutorial around a few MCQs. They can serve to guide the tutor; stimulate discussion (perhaps in groups) and test understanding. Students who are normally reticent to speak may do so in defence of their choice of answer. The tutor prepare explanations prepared for both the right answer and distractors. Again, thereaction of students should be used to improve the question data bank where necessary.
• MCQs can be used in a variety of ways in lectures. For example, MCQs can be put on overhead (OHP) transparencies and students individually or in pairs can come up with answers. This can help to facilitate interactive lecturing and usefully 'break' up lectures. If the distractors are carefully selected to 'catch' misunderstandings, then the response of the class can be used as a diagnostic indicator. MCQs could also be used to test prior understanding and to break the ice when starting a new topic.
• Short tests - e.g., of 5 MCQs - can be administered and marked by computer to provide a quick diagnostic feedback on the progress of a class, or as a revision exercise to prepare for exams, or as a bridge into "essay" tasks. They need not carry marks or have the students' names on them: the lecturer could follow up with a general explanation of why one answer wascorrect and the others were not.

The above suggestions represent just a few of the possible uses of MCQs. Their use in wider contexts than just formal testing is important because it serves to familiarise the students with a range of answering techniques and the MCQ process in general. The methods needed to answer different kinds of MCQs should be made explicit to students and discussed with them. For a variety of reasons there may be considerable resistance from students to MCQs and it is essential that the benefits of MCQs are clearly demonstrated to them.

Return to Contents of this Chapter

2.5 Managing the use of MCQs at departmental and course team level

Limiting the use of MCQs to one or two individuals in a department is problematical for three basic reasons:

• the use of MCQs should be taken into account when designing courses, which implies the course team should be involved;
• it is unlikely that an individual lecturer can effectively develop a bank of properly evaluated questions which are used in conjunction with a range of assessment methods;
• winning acceptance of MCQs among the students is crucial and this entails action at the level of thedepartment. Each department will have its own problems and possibilities with regard to the setting up of MCQ banks and integrating MCQs into the design of courses. The section "Where to get help" indicates who on the TAT programme is available to work with departments.

Return to Contents of this Chapter