Assessment of Higher-Order Skills

According to Bloom's taxonomy (Bloom and Krathwohl, 1956), the most widely used classification of educational objectives, lower order skills would include knowledge, comprehension and application, whilst analysis, synthesis and evaluation comprise the higher levels.

A common, although erroneous assumption is that CAA is only suitable for testing lower order skills, at the first three or four levels of Bloom's taxonomy. That CAA can be used for objective and non-objective testing was covered in the CAA Centre's Blueprint for Computer-Assisted Assessment.

However, it is the objective testing of higher order skills with CAA which would seem to present a problem among current or potential users. A further common, but erroneous assumption is that objective testing is limited to MCQs. It is important to keep these points in mind when considering the use of CAA for testing higher order skills.

Whilst objective testing, by definition, necessitates predetermined answers, this does not mean it is limited to the lower levels of knowledge, comprehension, and application. A predetermined answer can range from a simple, single response to far more complex arrays or combinations of responses that comprise the only appropriate answer to a given question. However, this depends on the quality of the questions and the degree of creativity used in their design, relative to maintenance of the status quo of traditional assessment methods. Although objective tests can assess a wide range of skills, knowledge and abilities, designing test questions to assess higher order skills can be time consuming and requires skill and creativity.

Using the examples in the GLOW system an attempt is made to illustrate a logical level-by-level progression from the assessment of lower order skills through to higher order skills. These examples, with the notable exception of synthesis, make use of MCQs in order to keep the explanation as simple as possible and to show how higher order learning can be assessed in this way. At the analysis level, an example of an assertion-reason question type is also given. A brief description of skills to be assessed and question words often used to test those skills is also given.

KNOWLEDGE: recall of information
Question words: list, define, label, describe, name

COMPREHENSION: grasping meaning and interpreting in own words
Question Words: interpret, discuss, predict, classify, summarise

APPLICATION: application of methods, theories, and concepts to new situations
Question Words: apply demonstrate, show, relate

ANALYSIS: identification of patterns, recognition of components and their relationships
Question Words: analyse, arrange, order, explain, connect, infer, compare, categorise

Example:

Read carefully through the paragraph below, and decide which of the options 1-5 is correct.

"The basic premise of pragmatism is that questions posed by speculative metaphysical propositions can often be answered by determining what the practical consequences of the acceptance of a particular metaphysical proposition are in this life. Practical consequences are taken as the criterion for assessing the relevance of all statements or ideas about truth, norm and hope."

1. The word "acceptance" should be replaced by "rejection".
2. The word "often" should be replaced by "only".
3. The word "speculative" should be replaced by "hypothetical".
4. The word "criterion" should be replaced by "measure".

This question requires prior knowledge of and understanding about the concept of pragmatism. The paragraph, seen in this light, contains one word that vitiates its validity, and the student is tested on his/her ability to analyse it to see whether it fits with the accepted definition of pragmatism. With this in mind, 2. is correct. Option 1. would degrade the paragraph further, while 3. and 4. would simply result in changing to acceptable synonyms. Note that this question does not address Level 6 (Evaluation), as the student is not asked to pass a value judgement on the text.

(This is considered a difficult question, and will obviously require a high level of reading. Bear in mind that there will be a significant time factor involved.)

SYNTHESIS: generalise, relate knowledge from several areas, create new ideas, integrate, predict
Question Words: integrate, modify, invent, design, compose, plan, formulate, arrange

EVALUATION: make judgements, assess/compare value of ideas, theories, evaluate data
Question Words: appraise, judge, evaluate, defend, rank, conclude, discriminate, recommend

Example:

A student was asked the following question: "Briefly list and explain the various stages of the creative process". As an answer, this student wrote the following:

"The creative process is believed to take place in five stages, in the following order: ORIENTATION, when the problem must be identified and defined, PREPARATION, when all the possible information about the problem is collected, INCUBATION, when there is a period where no solution seems in sight and the person is often busy with
other tasks, ILLUMINATION, when the person experiences a general idea of how to arrive at a solution to the problem, and finally VERIFICATION, when the person determines whether the solution is the right one for the problem."

How would you judge this student' s answer?

1. EXCELLENT (all stages correct in the right order with clear and correct explanations)
2. GOOD (all stages correct in the right order, but the explanations are not as clear as they should be).
3. MEDIOCRE (one or two stages are missing OR the stages are in the wrong order, OR the explanations are not clear OR the explanations are irrelevant)
4. UNACCEPTABLE (more than two stages are missing AND the order is incorrect AND the explanations are not clear AND/OR they are irrelevant)

In the above question, one is expected to make value judgement on the content of the given text (knowledge of the subject is required), the meaning of the terminology used (comprehension), and structure (analysis) of the answer for the right order of events. The correct answer here is 1, but appropriate modification could provide a small bank of questions with other correct answers.

It is interesting to note that three of the examples (one above, two in GLOW) provided are all based on the creative process. At the knowledge level, the student is asked to recall the five phases and to identify one. At the analysis level, the student is not only required to know and understand these five phases, but also to apply them to a factual example and to analyse the contents of this example to explain the appropriate phase. At the evaluation level, the student is required to know and understand all of the phases, apply these to a given text, analyse the content and synthesise in order to judge or evaluate which of the answers to the question are appropriate. All three questions have been based on different approaches to the same five phases of the creative process.

Further examples of questions at all levels of Bloom's taxonomy may be found at the following sites:

http://www.questionmark.com (click on 'Try It Out')
http://www.uct.ac.za/projects/cbe/mcqman/mcqappc.html#C2

Examples of more complex types of CAA questions, extending beyond the use of MCQs may be found at the following sites:

http://www.derby.ac.uk/assess/newdemo/newdemo.html
http://medweb/bham.ac.uk/caa
http://www.brookes.ac.uk/geology/8307/a40frame.html

Other sites worth trying include:

http://www.gre.org and http://www.ets.org/hea/index.html

Both of these offer a wide range of tests, some of which you may find useful in the development of your own questions for assessing higher order skills. For example GRE's site has links to analytical and logical reasoning questions and a pool of argument topics which, whilst not directly suitable to CAA, may provide some inspiration on how to reformulate your assessments. For example, a complex argument could be presented on which objective questions, as deemed appropriate from the numerous types available in CAA, may be based. ETS's site provides links to their Major Field Tests and Tasks in Critical Thinking, which may be used in a similar way to that suggested above.

The extent to which questions test different levels of learning will depend on the subject, context and level of the course or module. For example, a question that seeks to test the analytical skills of first year undergraduates may test only knowledge and comprehension in third year students. An online journal article that addresses some of these issues and offers some suggestions that may be useful in overcoming some of these problems may be found at http://ifets.ieee.org/periodical/vol_2_99/jayne_brahler.html. There are also different interpretations of the terms analysis, synthesis and evaluation. Bloom's taxonomy can serve as a useful framework but may need to be refined to meet the needs of a specific assessment or course.

Other articles that are also worth reviewing:

http://imej.wfu.edu/articles/a999/2/07/index.asp
comparing higher and lower level learning in a multimedia microworld.

and, if you subscribe:

http://ericae.net/ericdb/EJ446213.htm
assessing higher-order skills through interpretation of graphs in high school students

http://ericae.net/ericdb/ED362165.htm
Civil War interactive project assessing higher-order skills in social studies.

Further references can be found in the Blueprint. Of particular interest would be Haladyna's 'Writing Test Items to Evaluate Higher Order Thinking' (ISBN: 0-205-17875-8) and Hakel's 'Beyond Multiple Choice' (ISBN: 0-8058-2053-1) which offer advice and examples on item construction according to skill level and alternative assessment methods respectively.