This paper, co-authored by Richard Paul and Gerald Nosich, was commissioned by the United States Department of Education, Office of Educational Research and Improvement of the National Center for Education Statistics. It provides exactly what its title implies: a model for the national assessment of higher order thinking. The paper consists of a preface and five main sections.
The preface delineates the problem of lower order learning, summarizes the state of research into critical thinking and educational reform, and explains the five-part structure of the paper. The first main section of the paper states and explicates 21 criteria for higher order thinking assessment. The second section makes the case for how a “rich, substantive concept of critical thinking” meets those criteria. In making this case, Paul and Nosich spell out the dangers of a non-substantive concept of critical thinking. The third section of the paper spells out four domains of critical thinking: elements of thought, abilities, affective dimensions, and intellectual standards. The fourth section of the paper makes substantive recommendations regarding how to assess the various domains of critical thinking, the test strategies that may be used, the value of the proposed strategy for the reform of education, and the suggested implementation of the proposal.
What Should be the Main Objectives
of a Process to Assess Higher Order Thinking?
What Is Included in a Rich,
Substantive Concept of Critical Thinking?
Most of the language we shall use is drawn from draft statements of the National Council for Excellence in Critical Thinking Instruction. The National Council has been established precisely to articulate standards in critical thinking by 50 key leaders in critical thinking research and 105 leading educators. It is in the process of establishing regional offices and setting up 75 research-based committees to articulate the state of research in the field.
THE NATIONAL COUNCIL DEFINITION
Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.
This is the working definition of the National Council for Excellence in Critical Thinking Instruction. Though the definition as well as the other draft statements of the Council are subject to modification and refinement, the basic idea is one that is common to practitioners and researchers in critical thinking.
GLOSS ON THE DEFINITION
“In its exemplary form, [critical thinking] is based on universal intellectual values that transcend subject-matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.” (National Council Draft Statement)
a) “It entails the examination of those structures or elements of thought implicit in all reasoning: purpose; problem, or question-at-issue; assumptions; concepts; empirical grounding; inferences; implications and consequences; objections from alternative viewpoints, and frame of reference.” (National Council Draft Statement)
b) It entails larger-scale abilities of integrating elementary skills in such a way as to be able to apply, synthesize, analyze, and evaluate complicated and multidimensional issues. These include such abilities as clarifying issues, transferring insights into new contexts, analyzing arguments, questioning deeply, developing criteria for evaluation, assessing solutions, refining generalizations, and evaluating the credibility of sources of information. Among the abilities are included also the central forms of communication: critical reading, writing, speaking, and listening. Each of them is a large-scaled mode of thinking which is successful to the extent that it is informed, disciplined, and guided by critical thought and reflection. (Paraphrased from National Council Draft Statement.)
In our view, a rich, substantive concept of critical thinking, and it alone, provides an intelligible and workable means of meeting all 21 criteria. In this section we will briefly consider each objective in turn, not as a definitive response to the criteria, but merely to suggest the fuller response in Section Three below.
CRITERION # 1
Can it be used to test information processing skills? Critical thinking includes at its core “a set of information and belief generating and processing skills and abilities.”
CRITERION # 2
Can it be used to test flexible skills and abilities that can be used in a wide variety of subjects, situations, contexts, and educational levels? Since the art of critical thinking “entails proficiency in the examination of those structures or elements of thought implicit in all reasoning — purpose, problem or question-at-issue, assumptions, concepts, empirical grounding, reasoning leading to conclusions, implications and consequences, objections from alternative viewpoints and frames of reference” — it provides for maximum flexibility of use. It can be used in any subject, with respect to any situation to be figured out, in any context in which reasoning is germane, and, if adapted to the proficiency of students, at any educational level.
CRITERION # 3
Can it account for important differences among the subject areas? Subjects differ not because some make assumptions and others do not, not because some pose questions or problems and others do not, not because some have purposes and others do not, but rather because each has somewhat different purposes, and hence asks somewhat different questions, poses somewhat different problems, gathers somewhat different evidence, uses somewhat different concepts, etc. Critical thinking highlights these differences while underlining common structural features.
CRITERION # 4
Can it be used to focus on fundamental abilities fitted to the accelerating pace of change and embedded in intellectual history? Basic critical thinking skills and abilities are readily shown to be implicit in the rational development and critique of ideas at the core of intellectual history. They explain, for example, how new disciplines emerge from established ones: that is, by asking new questions, pursuing new purposes, framing new concepts, gathering new data, making new assumptions, reasoning in new directions, etc. They also explain how it is that a new field of study can ground itself, even at the outset, on definite intellectual standards that transcend any particular academic field: clarity, precision, accuracy, relevance, consistency, evidentiary force, valid reasoning, consistency . . . (standards implicit in the history of critical thinking and rational discourse in every domain).
CRITERION # 5
Can it be used to improve instruction? Critical thinking is not an isolated good, unrelated to other important goals in education. Rather it is a seminal goal which, done well, simultaneously facilitates a rainbow of other ends. It is best conceived, therefore, as the hub around which all other educational ends cluster. For example, as students learn to think more critically, they become more effective readers, writers, speakers, and listeners, because each ability requires well-reasoned thought.
They increase their mastery of content because all content is embedded in a system of understandings which, to be grasped, must be reasoned through. They become more proficient in — because they must be practiced within — a variety of modes of thinking: for example, historical, scientific, and mathematical thinking. Self-confidence increases with the intellectual empowerment critical thinking engenders. Finally, they develop skills, abilities, and traits of mind (intellectual discipline, intellectual perseverance, intellectual humility, intellectual empathy, intellectual integrity, …) crucial to success in the educational, professional, and everyday world.
CRITERION # 6
Can it make clear the inter-connectedness of our knowledge and abilities, and why expertise in one area cannot be divorced either from findings in other areas or from a sensitivity to the need for interdisciplinary integration? In learning to think critically, one learns to transfer what one has learned about the logic of questions in one field to logically similar questions in other fields. Typically this begins with a recognition of the need to ask questions based on logical parallels between all fields of study, for example, skilled practice in questioning concepts and theories, in questioning data, in questioning the source or interpretation of data, in questioning the nature or organization of data, in questioning inferences, in questioning assumptions, in questioning implications and consequences, in questioning points of view and frames of reference, etc.
CRITERION # 7
Can it be used to assess those versatile and fundamental skills essential to being a responsible, decision-making member of the work place? Critical thinking skills and abilities are highly transferable to the work place. Since in learning to think critically we learn to take increasing charge of our minds as an instrument of learning — for example, reading, writing, speaking, and listening with greater discipline and skill — we are well situated to engage in collective problem solving and goal attainment, wherever they occur. The kind of “work” increasingly required in industry and business is “intellectual”, that is, it requires workers to define goals and purposes clearly, seek out and organize relevant data, conceptualize those data, consider alternative perspectives, adjust thinking to context, question assumptions, modify thinking in the light of the continual flood of new information, and reason to legitimate conclusions. Furthermore, the intellectual work required must increasingly be coordinated with, and must profit from the critique of, fellow workers. There is no avoiding the need, therefore, to express ideas well, accurately represent and consider fairly the ideas of others, write clear and precise memos and documents, and coordinate and sequence all of these so that well-reasoned policies and decisions can be accurately understood and effectively implemented.
CRITERION # 8
Can it generate clear concepts and well-thought-out, rationally articulated goals, criteria, and standards? Since critical thinking is based on the art of monitoring one’s thinking with standards implicit in the universal structure of thought, and since the use of these standards is implicit in intellectual history from Socrates through Einstein, there is no problem using critical thinking to generate clear concepts for testing, as well as rationally articulated goals, criteria, and standards.
CRITERION # 9
Can it account for the integration of adult-level communication skills, problem solving, and critical thinking, and legitimately assess all of them without compromising essential features of any of them? Shallow concepts of critical thinking often distinguish critical thinking from problem solving and decision making as well as from reading, writing, and speaking skills.
Once one considers a rich, substantive concept of critical thinking, however, it is clear that each of the basic skills of critical thinking are presupposed by each of the other skills, just as each of them is deeply interrelated to critical thinking as a whole. Consider, does it make sense to analyze potential solutions to problems or the implications of choosing an alternative in making a decision without using critical thinking? Clearly not. Every problem to be solved (or question to be settled) requires a critical analysis of the conditions under which it can be solved or settled. We, as problem-solvers, need to look critically at the purpose for solving the problem, we need to critically examine contextual factors, our assumptions, our concepts, what we are using as data, our organization of the data, the source of the data, our reasoning, the implications of our reasoning, our point of view, objections from other points of view. All of these are essential to higher order problem solving and decision-making.
Furthermore, all of these intellectual abilities are crucial to higher order reading, writing, speaking, and listening. To read must we analyze the text and re-create its logic in our own minds. To write we must construct a logic our readers can translate into the logic of their thought. To speak we must articulate our thoughts in such a way that our audience can translate our thoughts into their experiences. To listen we must analyze the logic of the thinking of the speaker. Intellectually disciplined reading, writing, speaking, and listening require, in other words, that we work explicitly with the logic we are constructing or re-constructing, using our grasp of the standards of critical thinking to communicate accurately and precisely, effectively solve problems, and rationally make decisions.
CRITERION # 10
Does it respect cultural diversity by focusing on the common-core skills, abilities and traits useful in all cultures? As the criterion presupposes, we can respect cultural diversity best by constructing tests in higher order thinking that focus on skills and abilities necessary in all modern cultures. In this way we can legitimately justify assessing it in all cultural groups. Basic critical thinking skills and abilities — because they are based on fundamental elements implicit in the structure of all reasoned thought per se, and because their mastery is essential to higher order thinking in all academic, professional, personal, and public life — are an appropriate foundation for assessment.
CRITERION # 11
Does it test for thinking that promotes (to quote the September, 1991 Kappan) “the active engagement of students in constructing their own knowledge and understanding?” Narrow concepts of critical thinking sometimes characterize it in negative terms, as a set of tools for detecting mistakes in thinking. A rich, substantive concept of critical thinking, however, highlights its central role in all rationally defensible thinking, whether that thinking is focused on assessing thought or products already produced, or actively engaged in the construction of new knowledge or understandings. Well-reasoned thinking, whatever its end, is a form of creation and construction. It devises and articulates purposes and goals, translates them into problems or questions, seeks data that bear upon problems or questions, interprets those data on the basis of concepts and assumptions, and reasons to conclusions within some point of view. All of these are necessary acts of the reasoning mind and must be done “critically” to be done well. Hence all require critical thinking.
CRITERION # 12
Does it concentrate on assessing the fundamental cognitive structures of communication? Each of the dimensions identified in the objective is either straightforwardly a critical thinking ability or depends on a critical thinking ability. The writer’s or speaker’s purpose, implications, assumptions, point of view, etc., are all elements of thought, and the ability to identify and assess those as one reads or listens — the ability to construct in one’s mind an accurate and fertile interpretation — is simply thinking by listening, thinking by reading.
A similar reliance on elements of thought is central to writing or speaking effectively at any educational level. The knowledge of how to gather and present evidence, to make clear one’s own assumptions, to see the implications of a position: these are critical thinking abilities.
All forms of communication, moreover, rely on critical thinking standards. Essays and interpretations of essays, utterances and interpretations of utterances, need to be relevant, logical, consistently worked out; evidence needs to be recorded and reported accurately; points need to be made clearly and with as much precision as the subject permits; topics need to be covered in depth and presented fairly.
CRITERION # 13
Can it be used to assess the central features of making rational decisions as a citizen, a consumer, and a part of a world economy? Both public and private life increasingly requires mastery of the basic skills and abilities of critical thinking. When this mastery is absent the public degenerates into a mass society susceptible to manipulation by public relations specialists who can engineer political victories by an adroit use of mud slinging, scare tactics, shallow nationalism, fear, envy, stereotypes, greed, false idealism, and maudlin sentimentality.
Modern citizenship requires basic critical thinking skills and abilities throughout. The modern citizen should be able to assess the arguments presented for his or her assent, must rationally adjudicate between conflicting points of view, must attempt to understand a culturally complex world, must assess the credibility of diverse sources of information, must translate between conflicting points of view and diverse appeals, must rationally decide priorities, must seek to understand complex issues that involve multiple domains (for example, the environmental, moral, economic, political, scientific, social, and historical domains). Without a solid grounding in critical thinking, citizens are intellectually disarmed, incapable of discharging their civic responsibilities or rationally exercising their rights.
CRITERION # 14
Can it avoid reducing a complex whole to oversimplified parts? Testing for a rich, substantive concept of critical thinking is testing for skills of reasoning in terms of elements of thought, for the ability to orchestrate those elementary skills, for the affective dimensions that make critical thinking actualizable in practice, and for universal intellectual standards, in short for a rich and complex whole rather than for fragmented parts.
CRITERION # 15
Can it articulate what is central to basic skills for the future? Basic skills are constituted by the structures explicated in a rich, substantive concept of critical thinking. To teach reading is to teach the ability not merely to repeat content, but to reconceptualize that content, to see applications of the main ideas, to generalize from them, critique them, see them in context, to enter with empathy into another’s point of view. To teach writing as a basic skill is to teach not merely grammar and punctuation, but the ability to arrange one’s ideas logically and consistently, to anticipate reasonable objections, to transfer ideas to the page in a way that makes them decipherable in all their complexity by a reader. To teach math as a basic skill is not primarily to teach how to solve pre-selected, individual, isolated problems out of context, but to teach the ability to begin to make sense of the world mathematically, to think quantitatively, to be able to see mathematical patterns, to set up the construction of problems and then creatively go about solving them. Critical thinking abilities like these do not exist somehow in addition to the basic skills of life; they constitute the basic skills of life.
Can it provide the kind of skills that are seen as valuable outside the school as well as inside it? Critical thinking provides skills that are seen as valuable by practitioners of the academic disciplines, by responsible leaders of government, of the professions, of business, by citizens interested in their environmental, physical, and economic welfare. In all such areas what is needed are ways to adapt to rapidly changing knowledge, to recognize problems and see their implications before they become acute, to formulate approaches to their solution that recognize legitimately different points of view, to draw reasonable conclusions about what to do.
Increasingly, one is hearing statements such as the one made by David Kennedy, the president of Stanford University, to 3,000 college and university presidents:
"It simply will not do for our schools to produce a small elite to power our scientific establishment and a larger cadre of workers with basic skills to do routine work. Millions of people around the world now have these same basic skills and are willing to work twice as long for as little as one-tenth our basic wages. To maintain and enhance our quality of life, we must develop a leading-edge economy based on workers who can think for a living. If skills are equal, in the long run wages will be too. This means we have to educate a vast mass of people capable of thinking critically, creatively, and imaginatively."
CRITERIA #17 AND #18
Can critical thinking be assessed in a way that requires evaluation of authentic problems in realistic contexts, where the abilities assessed include those of formulating the problem and initial screening of plausible solutions? Yes. Testing of authentic skills, abilities and dispositions in authentic contexts can be accomplished by using a combination of a) standard multiple-choice items, b) machine-gradable multiple-rating items and c) short essay items.
|a) The standard multiple-choice part of the assessment would be an expanded version of established critical thinking tests, such as the Watson-Glaser or Cornell tests. It is suitable for assessing micro-dimensional critical thinking skills, like identifying the most plausible assumption, recognizing an author’s purpose, selecting the most defensible inferences, and such like.
b) The multiple-rating part of the assessment would test more open-ended and larger-domained abilities, like thinking within opposing points of view, the willingness to suspend judgment, the ability to synthesize disparate data into a logical scheme, to take established findings and generalize them into new contexts, etc.
The multiple-rating portion of the assessment, to be reliable, must:
|c) The essay part of the assessment would be designed to address critical thinking abilities and traits that involve creating a logic to capture a situation rather than selecting from among possibilities suggested by the test. Examples include the ability to construct an interpretation, to make a logical outline of a text, to figure out ways to gather information, to take an unclear and complex real issue and reformulate it so as to make it more amenable to solution.
Validity on the essay part of the assessment requires that the test be:
Can critical thinking be assessed nationally in a way that is financially affordable? To make it affordable, the constructed response segment of the assessment should be administered not to the population of students as a whole, but rather to a representative sample of the student population of a school system. The assessment should be a) paid for by school systems that contract to have their students tested, and b) constructed, monitored, administered, and graded by a private agency with critical thinking credentials, or at least under the direction of scholars with a solid grounding in the research into critical thinking.
CRITERIA #20 AND #21
Can critical thinking be assessed so as to gauge the improvement of students over the course of their education and to measure the achievement of students against national standards? To evaluate students in both these dimensions requires:
a) an assessment administered as a pre-test at the 6th grade and then as a follow up at the 9th and 12th grades (to provide for value-added judgments)
b) a criterion-referenced assessment that is built on clear, consistently applied quality-norms that are derived from a rich and substantive concept of critical thinking (to provide for the measuring of national progress)
It is important to be alert to the dangers posed by a non-substantive concept of critical thinking. Such a concept exists when, separate from a consideration of the research in the field, a person or institution presupposes a) that the meaning or terminology of critical thinking is intuitively obvious (hence not in need of scholarly analysis), or b) that each concept underlying critical thinking (such as assumption, inference, implication, reasoning . . . ) can be analyzed separately from a theory that accounts for the interrelation of these concepts, or c) that the skills of critical thinking can be adequately cultivated without reference to the values, traits of mind, and dispositions that underlie those skills.
There Are At Least Three Serious Problems That May Result
From the Use of a Theoretically Superficial Concept of Critical Thinking
a) important critical thinking concepts, which must be clearly defined to be used effectively in assessment, may be used vaguely, inconsistently, incorrectly, or misleadingly
b) a false, misleading, or simplistic over-arching concept of critical thinking may be fostered
c) an unrealistic strategy for the assessment and cultivation of critical thinking may be incorporated into testing and teaching
Many examples of the unwitting use of a non-substantive concept of critical thinking could be cited — such as “thinking skills” programs devoid of intellectual standards (which, for example, systematically confuse “inferences” with “valid inferences” and “analogies” with “sound analogies”), or testing personnel who lack adequate grounding in critical thinking theory (and so, for example, frequently confuse assumptions with inferences or inferences with implications). The most far-reaching danger occurs when influential educational systems or institutions, like state departments of education, inadvertently incorporate a non-substantive concept of critical thinking into statewide curriculum standards or into statewide testing programs. This can result in significant, unintended negative consequences, for example: thousands of teachers encouraged to follow a misconceived model for the assessment of reasoning, leading to mis-instruction on a grand scale.
ILLUSTRATION: THE CALIFORNIA DIRECT WRITING ASSESSMENT
We shall look at one important case. Unfortunately, given the brevity of this paper, one case must stand for all. The case we have chosen concerns the Integrated Language Arts Assessment of the California Assessment Program, a massive statewide program that has impact not only on every student in the public schools of California, but also, because of the leadership role of California in assessment, on national teaching and testing practices as well. It appears that three fundamental mistakes occurred in the design of the direct writing assessment:
a) Though one of the goals of the program was to place an emphasis on the quality of reasoning and critical thinking in writing, it appears that no one with a research background in critical thinking reviewed the articulation or implementation of the assessment prompts. (We infer this from the fact that fundamental conceptual errors occur both in the prompts themselves and in the application of criteria to student constructed responses.)
b) It was assumed, inappropriately, that classroom teachers without extended training in critical thinking are able to effectively assess student essays that call for evaluative reasoning. We infer this from statements descriptive of the assessment design like:
|Teachers on the CAP writing Development Team develop all the testing and instructional materials for assessment. For every type of writing assessed, the team develops a special set of prompts … and a scoring guide that identifies the thinking and writing requirements for that type of writing . . . Essays are scored in four to six days by several hundred teachers at four regional scoring centers. A special handbook for each grade level provides teachers with practical instructional materials for each type of writing, including sample prompts, illustrative essays, and related readings.|
c) The resulting assessment was not monitored by anyone with a research background in critical thinking. (We infer this from the fact that model “strong” answers purporting to illustrate critical reasoning are showcased that are in fact patently very weak answers, containing virtually no reasoning at all.)
Consider Figure 1 and Figure 2 used as illustrations of the nature and quality of the writing assessment program in an article authorized and developed by the staff of the California Assessment Program. It is entitled “California: The State of Assessment” and was written for an important national anthology, Developing Minds (more than 150,000 copies disseminated by ASCD). The show-piece article, in which these figures occur, argues that the examples illustrate a “state-of-the-art teacher-developed writing assessment” that is sophisticated in “its testing, scoring, and reporting systems” and designed to “include only those tasks that will stimulate high-quality instruction.”
There are a number of problems illustrated in these figures that a substantive understanding of critical thinking would have avoided:
Fundamental misconceptions of the nature of critical thinking and reasoned discourse, such as those documented above, must not be replicated in a national assessment program. Steps should be taken to insure that a substantive concept of critical thinking and a well-supervised implementation of that concept form the basis of the finished assessment program.
What are the four component domains of critical thinking and their implications for the assessment of higher order thinking?
Elements of Thought
As soon as we move from thought which is purely associational and undisciplined, to thinking which is conceptual and inferential, thinking which attempts in some intelligible way to figure something out, to use the power of reason, then it is helpful to think about what can be called “the elements of thought.” The elements of thought are the basic building blocks of thinking, essential dimensions of reasoning whenever and wherever it occurs. Working together, they shape reasoning and provide a general logic to reason. We can articulate these elements by paying close attention to what is implicit in the attempt on the part of the mind to figure anything out whatsoever. Once we make them clear, it will be obvious that each of them can serve as an important touchstone or point of assessment in critical analysis and in the assessment of thinking.
For each of the elements of thought there is a cluster of attendant basic thinking skills. Because they involve fundamental structures of thought, these skills can be characterized as micro-skills, those skills out of which larger-domained critical thinking abilities are built. Being able to think critically about a particular issue, then, will include the ability to identify, clarify, and argue for and against alternative formulations of the elements of thought.
The basic conditions implicit whenever we gather, conceptualize, apply, analyze, synthesize, or evaluate information the elements of thought are as follows:
Assessment would test, in a way geared to their educational level, students’ ability to recognize faulty and justified inferences in a passage, to rank inferences with respect to both their plausibility and their relevance, to make good inferences in their own reasoning, to discriminate among various formulations of an author’s inferences with respect to which is most accurate, to take something they do not believe but to entertain it for the sake of argument and draw reasonable inferences from it.
Assessment of Elements of Thought
Any program for the assessment of critical thinking skills must itself be assessed in terms of its validity and reliability in testing for the ability to think about, and in terms of, the elements of thought. These abilities can be successfully assessed in three related ways: by a restricted use of standard multiple-choice items, by multiple-rating items, and by short essay items. Both multiple-choice and multiple-rating items are machine-gradable, while essay items are not.
Although our recommendations about the content of the assessment will be spelled out in detail in Section Four, some of these can be anticipated here with respect to the assessment of reasoning abilities centering on the elements of thought.
Multiple-choice testing (as in the existing Watson-Glaser Critical Thinking Appraisal or the Cornell Critical Thinking Tests) is an important part of an assessment of critical thinking, but its legitimate use is restricted to testing only the most basic skills of identifying and recognizing elements of thought, and then only as they occur in relatively short and unambiguous excerpts.
Within this domain, multiple-choice questions will require students:
The elements of thought do not exist in isolation from one another, nor — more importantly for the concept of an assessment procedure — do they exist outside a particular context of application. In the practice of good critical thinking, skills more closely associated with elements of thought are orchestrated into larger-domained abilities which are applied to thinking about complex and sometimes ambiguous issues, problems, decisions, theories, states of affairs, social institutions, and human artifacts.
These critical thinking abilities include being skillful at:
Abilities like these play a central role in a rich and substantive concept of critical thinking. They are essential to approaching actual issues, problems, and situations rationally. Understanding the rights and duties of citizenship, for example, requires that one at least have the ability to compare perspectives and interpretations, to read and listen critically, to analyze and evaluate policies. In fact, there is no macro-ability on the list that would not be relevant or even crucial to thinking deeply about the rights and duties of citizenship.
Similarly, the capacity to make sound decisions, to participate knowledgeably in the work-place, to function as part of a global economy, to master the content in anything as complex as the academic disciplines, to apply those subject area insights to real-life situations, to make insightful cross-disciplinary connections, to communicate effectively — each of these relies in a fundamental way on having a significant number of the abilities listed. Take, for example, the capacity to make sound decisions: such decision-making is hardly possible without an attendant ability to (going down the list of abilities in order) refine generalizations, compare analogous situations, develop one’s perspective, clarify issues, and so forth.
The last four abilities listed — the ability to read, write, listen, and speak, each in a critical, informed, constructive way — are best considered not as in the usual model, not as manifestations of thinking already accomplished, but as being themselves actual modes of constructive thinking. As such, they are structured amalgams of elementary skills together with any number of other abilities.
Assessment of Abilities
The assessment of abilities, too often neglected, is essential to assessment of critical thinking. Since these are the abilities implicit in the realistic use of thinking, no assessment tool that fails to assess a significant number of these abilities could justifiably be called an assessment of higher order thinking. The assessment, moreover, needs to address such abilities directly (rather than through secondary indicators), systematically (rather than haphazardly as a result of an attempt to assess other variables like academic achievement), and in settings as authentic as possible given the requirement of uniform, relevant grading.
Assessment of abilities that meets these four criteria cannot be accomplished within the confines of a standard multiple-choice-type test. It can be accomplished, however, for all of the abilities (except those having to do with oral communication), by means of a combination of machine-gradable multiple-rating items and essay items.
For any macro-ability, there will be dimensions of the ability that are generative and other dimensions of it that are selective. In trying to solve a real problem, for example, much of one’s thinking is devoted to generating a formulation of the problem that will make it more susceptible to solution. Another, and quite different, aspect of problem solving, is the ability to select, from among a large variety of possibilities, that avenue of thought which will most likely result in a solution. Students who are trained using a rich, substantive concept of critical thinking tend to improve in both dimensions of this ability, and both are genuine dimensions of real problem-solving.
The selective dimensions of an ability can be assessed accurately, even in complex, ambiguous, and subtle cases, using multiple-rating items. The generative dimension, on the other hand, cannot. Since it requires students to come up with their own critical thinking approaches within that macro-ability, this dimension can be assessed adequately only by carefully constructed and carefully graded essay tests. Details of the assessment and samples of assessment items will be presented in Section Four.
Higher order thinking requires more than higher order thinking skills. Critical thinking, in any substantive sense, includes more than abilities. The concept also includes, in a crucial way, certain attitudes, dispositions, passions, traits of mind. These affective dimensions are not merely important to critical thinking; they are essential to the effective use of higher order thinking in real settings.
These affective dimensions include:
Without intellectual perseverance, one could not solve the complicated, multi-faceted problems one confronts in industry. Without intellectual courage, one could not maintain a defense of citizenship rights in the face of scare tactics. Without fair-mindedness, one could not enter into another’s point of view and thus would lack that empathetic understanding necessary for a reasonable approach to living in a pluralistic society. Without developing insight into egocentricity and socio-centricity one could employ one’s reasoning skills in a merely self-serving and prejudiced way. Without confidence in reason one could not adequately address those complex and frequently ambiguous real-life problems that require reasonable decisions in the face of crucial uncertainties.
Assessment of Affective Dimensions
The assessment of affective dimensions of critical thinking is an important part of an assessment of higher order thinking. An initial problem is that from the fact that all these dimensions are essential, it does not follow that all are directly testable, nor does it follow that any of them is easily testable. For some of these affective dimensions (intellectual perseverance, for example), any testing would have to take place over an appropriately long period of time and thus could not be legitimately assessed at all during a time frame suitable for a national test.
Nevertheless, a number of affective dimensions can be assessed in a relatively straightforward way using essay items and, especially, machine-gradable multiple-rating items.
“Reasoning Within Conflicting Points of View,” a central aspect of the disposition of fair-mindedness, is already being assessed on a machine scorable test developed by Dr. Richard Paul. This Appraisal asks students to select the strongest (that is, the most defensible) argument in favor of each side of a pair of conflicting and sometimes emotionally charged points of view. Proficiency on these items indicates a fair-minded willingness to distinguish the concept of reasonable defensibility from that of personal belief.
Multiple-rating items are currently being prepared that address aspects of intellectual courage, other aspects of fair-mindedness, aspects of intellectual humility, and aspects of the development of insight into one’s own egocentricity and socio-centricity.
In any domain where assessment is taking place, there are standards implicit in that assessment. Higher order thinking is thinking that meets universal intellectual standards. Thus, when assessing a student’s ability to compare and evaluate perspectives (a macro-ability) and to do so with fair-mindedness (a trait of mind), we would judge whether she had made such evaluations in a relevant and consistent way, with attention to accuracy, fairness, and completeness in describing each perspective, and with a sensitivity to the degree of precision appropriate to the topic.
We would assess critical thinking about and in terms of the elements of thought in very much the same