|CAA Centre >> Resources >> FAQs >>Higher Order Skills|
Assessment of Higher-Order Skills
According to Bloom's taxonomy (Bloom and Krathwohl, 1956), the most widely used classification of educational objectives, lower order skills would include knowledge, comprehension and application, whilst analysis, synthesis and evaluation comprise the higher levels.
A common, although erroneous assumption is that CAA is only suitable for testing lower order skills, at the first three or four levels of Bloom's taxonomy. That CAA can be used for objective and non-objective testing was covered in the CAA Centre's Blueprint for Computer-Assisted Assessment.
However, it is the objective testing of higher order skills with CAA which would seem to present a problem among current or potential users. A further common, but erroneous assumption is that objective testing is limited to MCQs. It is important to keep these points in mind when considering the use of CAA for testing higher order skills.
Whilst objective testing, by definition, necessitates predetermined answers, this does not mean it is limited to the lower levels of knowledge, comprehension, and application. A predetermined answer can range from a simple, single response to far more complex arrays or combinations of responses that comprise the only appropriate answer to a given question. However, this depends on the quality of the questions and the degree of creativity used in their design, relative to maintenance of the status quo of traditional assessment methods. Although objective tests can assess a wide range of skills, knowledge and abilities, designing test questions to assess higher order skills can be time consuming and requires skill and creativity.
Using the examples in the GLOW system an attempt is made to illustrate a logical level-by-level progression from the assessment of lower order skills through to higher order skills. These examples, with the notable exception of synthesis, make use of MCQs in order to keep the explanation as simple as possible and to show how higher order learning can be assessed in this way. At the analysis level, an example of an assertion-reason question type is also given. A brief description of skills to be assessed and question words often used to test those skills is also given.
KNOWLEDGE: recall of information
COMPREHENSION: grasping meaning and interpreting in
APPLICATION: application of methods, theories, and concepts
to new situations
ANALYSIS: identification of patterns, recognition of
components and their relationships
Read carefully through the paragraph below, and decide which of the options 1-5 is correct.
"The basic premise of pragmatism is that questions posed by speculative metaphysical propositions can often be answered by determining what the practical consequences of the acceptance of a particular metaphysical proposition are in this life. Practical consequences are taken as the criterion for assessing the relevance of all statements or ideas about truth, norm and hope."
1. The word "acceptance" should be replaced by "rejection".
This question requires prior knowledge of and understanding about the concept of pragmatism. The paragraph, seen in this light, contains one word that vitiates its validity, and the student is tested on his/her ability to analyse it to see whether it fits with the accepted definition of pragmatism. With this in mind, 2. is correct. Option 1. would degrade the paragraph further, while 3. and 4. would simply result in changing to acceptable synonyms. Note that this question does not address Level 6 (Evaluation), as the student is not asked to pass a value judgement on the text.
(This is considered a difficult question, and will obviously require a high level of reading. Bear in mind that there will be a significant time factor involved.)
EVALUATION: make judgements, assess/compare value of
ideas, theories, evaluate data
A student was asked the following question: "Briefly list and explain the various stages of the creative process". As an answer, this student wrote the following:
"The creative process is believed to take place in five stages,
in the following order: ORIENTATION, when the problem must be identified
and defined, PREPARATION, when all the possible information about the
problem is collected, INCUBATION, when there is a period where no solution
seems in sight and the person is often busy with
How would you judge this student' s answer?
1. EXCELLENT (all stages correct in the right order with clear and correct
In the above question, one is expected to make value judgement on the content of the given text (knowledge of the subject is required), the meaning of the terminology used (comprehension), and structure (analysis) of the answer for the right order of events. The correct answer here is 1, but appropriate modification could provide a small bank of questions with other correct answers.
(click on 'Try It Out')
Both of these offer a wide range of tests, some of which you may find useful in the development of your own questions for assessing higher order skills. For example GRE's site has links to analytical and logical reasoning questions and a pool of argument topics which, whilst not directly suitable to CAA, may provide some inspiration on how to reformulate your assessments. For example, a complex argument could be presented on which objective questions, as deemed appropriate from the numerous types available in CAA, may be based. ETS's site provides links to their Major Field Tests and Tasks in Critical Thinking, which may be used in a similar way to that suggested above.
The extent to which questions test different levels of learning will depend on the subject, context and level of the course or module. For example, a question that seeks to test the analytical skills of first year undergraduates may test only knowledge and comprehension in third year students. An online journal article that addresses some of these issues and offers some suggestions that may be useful in overcoming some of these problems may be found at http://ifets.ieee.org/periodical/vol_2_99/jayne_brahler.html. There are also different interpretations of the terms analysis, synthesis and evaluation. Bloom's taxonomy can serve as a useful framework but may need to be refined to meet the needs of a specific assessment or course.
Other articles that are also worth reviewing:
and, if you subscribe:
Further references can be found in the Blueprint. Of particular interest would be Haladyna's 'Writing Test Items to Evaluate Higher Order Thinking' (ISBN: 0-205-17875-8) and Hakel's 'Beyond Multiple Choice' (ISBN: 0-8058-2053-1) which offer advice and examples on item construction according to skill level and alternative assessment methods respectively.
|© CAA Centre 2002|