This manual is aimed at giving you an overview of the general principles used in the construction of MCQs. The "nuts and bolts" are to be found in Appendix A. In this chapter, we aim to give you an insight into the process whereby lecturers have attempted to apply these general principles in the context of three different disciplines.
3.1.1 Example A. Context: First year course test.
If we understand learning a school subject on the model of learning a language we can explain
- a) why teachers do not articulate clearly.
- b) why it is not possible to explain the point of the subject to someone who does not understand the subject.
- c) why truth is important.
- d) why subjective experiences are important.
- e) why school subjects are so important.
- f) why school subjects are taught verbally.
Lecturer's approach: Bak gives this example in her paper "How to test insight and understanding of philosophical issues by means of multiple-choicequestions". She explains that in order to answer this kind of question, the students must first understand not only the concepts but "their relation to one another as well". In the course, students were introduced to the "model of learning a language" (see question stem) in order to illustrate the educational implications of the philosophical stance that "we come to understanding only through a process which is not given, but through which we must work". The lecturer's intention was that in arriving at the correct answer (b), the students must have compared each of the statements with their essential understanding of the part of the course dealing with a model of "learning a language"; and they need to be able to understand and relate:
- the concept of a "model of learning a language";
- the general concept of understanding being a process, which in the course was linked to a particular philosophical stance;
- the argument presented in option (b) "why it is not possible to explain the point of the subject to someone who does not understand the subject".
Comments: The lecturer who set this question knew what concepts they gave prominence in her course and what relationships between the concepts she emphasised in lectures, tasks, etc. Although the lecturer's strategy appears sound, there are technical proplems with the question which might hinder some students from dealing with the question in the manner which was intended. Firstly, the correct option (b) is much longer than the distractors. For any student with experience in answering MCQs, this is a powerful clue. Secondly, the use of the double negative (not...not") in option (b) is confusing, even to first language speakers. Thirdly, one should not repeat the word "why" in front of each response, but include it as part of the stem. Such problems should be picked up when the questions are reviewed by the lecturing team and the question edited as required, or when the student answers are analysed. (See Section 4)
Return to Contents of this Chapter
3.2.1 Example B. Context: This is in a first year block dealing with organisations, bureaucracy and democracy, using trade unions as an example. This question was answered through group discussion in a supplementary tutorial designed for examination revision.
The following statements are all answers to the questionWhy have workers formed trade unions in South Africa?
Which is the best answer?
- a) They needed to resist the powerful organisations the employers could call on.
- b) They needed organisations which could advance their struggle for political rights.
- c) They needed to be come together to discuss what todo about their common problems.
- d) They needed to be organised in order to improve their working and living conditions.
- e) They needed to be organised because it was the only way they could be united.
Lecturer's approach: The lecturer who designed the question and oversees the supplementary tutorial programme worked closely with the lecturer who taught the block. In the course, students were shown how and why workers formed the first trade unions, with biographies of trade unionists used to illustrate a historical process. All the statements used as responses reflect some stages or aspects of this process, but response (d) is the "best" answer because it
- is the most fundamental (i.e. it has a higher order of generality so that it includes all the others);
- logically preceeds the others in causal terms;
- follows the structure of the argument and emphases that the lecturer presented in his block.
The intention of the person who set the question was that the students
- evaluate each statement;
- order them according to one or more or the above criteria.
The question was designed to promote maximum discussion around the responses in order to facilitate the revision process. Nine out of the twelve groups of (mainly second language) students considered to be "at risk" arrived at the correct answer within 5-8 minutes of group discussions.
Comments: This question must be seen in the context of the course, and must be considered to be on the difficult side, requiring the highest cognitive skills (see Appendix C). As such it is ideal to use in a tutorial situation as it will elicit much discussion. The repetition of the phrase "They needed..." in front of each alternative is normally undesirable, but can be defended here on the grounds that one is asked to select the BEST answer, from a list of plausible complete statements
3.2.2 Example C. Context: The question could be used in a first-year sociology tutorial or test.
Read the following statement and select the answer which you think best sums up the argument being made."Traditionally we divide ourselves into races by the twin criteria of geographic location and visible physical characteristics. But we could make an equally reasonable and arbitrary division by the presence or absence of a gene, such as the sickle-cell gene, that confers resistance to malaria. By this reckoning, we'd place Yemenites, Greeks, New Guineans, Thai and Dinkas in one "race", Norwegians and several black African peoples in another."
- a) Norwegians and some black African peoples belong to the same race.
- b) Race is determined by the presence or absence of the sickle-cell gene. Thus there are two races in the world. Those with the sickle-cell gene and those without it.
- c) We all belong to a race. Which race we belong todepends on where we come from and our skin colour.
- d) "Race" as we commonly understand it is an arbitrary means of classifying human beings. We could choose many other random criteria and get all sorts of different "races".
Lecturer's approach: The lecturer has a double agenda. Firstly, she wants to test and develop the students' ability to extract the central argument from a text. Secondly, she wants to test their grasp of a specific concept: that there is no credible biological basis on which to divide humans into "races". (the correct answer is "d").
Comments:The distractors need to be made more equal in terms of length. This can usually be done fairly easily by "padding" the shorter distractors. There are only four distractors - a fifth could make the question more challenging. Students for whom English is a second language could find the phrasing in the text difficult to penetrate. Students who have some knowledge about genes and sickle-cell anaemia might be at an advantage.
Return to Contents of this Chapter
3.3.1 Example D. Context: The question was used in a Economics first year quarterly test.
If planned aggregate expenditure is greater than National Income and the short run aggregate supply curve is horizontal, the economy:
- a) cannot be on the aggregate demand curve.
- b) cannot be on the short run aggregate supply curve.
- c) cannot be on the long run aggregate supply curve.
- d) will experience rises in both GDP and prices.
- e) both (b) and (d) are correct.
Lecturer's approach: The correct answer is (a). Although the answer will not be apparent to non-economists, the question does show how MCQs can be used to test students' ability to apply technical terms and concepts. The test consisted of about one third recall questions, one third of questions aimed at the "average" student and one third questions - like the one above - designed to discriminate between the more advanced students and the rest. To answer the question the students would have to:
- understand a number of assumptions (eg., that prices are fixed);
- understand a number of definitions (eg., of a horizontal supply curve);
- analyse a particular case and apply their understanding to it.
Students were told that they could arrive at right answers by eliminating the wrong choices - for example, if prices are fixed then option (d) must be wrong - but they could not do this reliably without some understanding of basic terms and assumptions. They were also told that in cases where a problem had to be visualised then they should use scrap paper to draw the diagram.
Comments: This is a demanding question, not only on account of the subject knowledge required, but because of the complex structure of the stem and alternatives. The conditional ("if...") combined with the negative in "cannot" (alternatives a, b, and c) contribute in making the syntax challenging, especially to students for whom English is not the mother-tongue. Students are expected to go through a number of fairly involved logical reasoning steps and they should be warned of this in advance of the test. Bearing the above points in mind, sufficient time should be allocated to allow a reasonable range of students to tackle such questions. Alternative "e" should be revised, as (i) it wrongly suggests that more than one alternative could be correct (ii) can mislead students into thinking that "b" or "d" may be correct and (iii) adds to the overall complexity of the question.
Return to Contents of this Chapter