Critical Thinking: A Model for the National Assessment of Higher Order Thinking

Critical Thinking: A Model for the National Assessment of Higher Order Thinking



This paper, co-authored by Richard Paul and Gerald Nosich, was commissioned by the United States Department of Education, Office of Educational Research and Improvement of the National Center for Education Statistics. It provides exactly what its title implies: a model for the national assessment of higher order thinking. The paper consists of a preface and five main sections.

The preface delineates the problem of lower order learning, summarizes the state of research into critical thinking and educational reform, and explains the five-part structure of the paper. The first main section of the paper states and explicates 21 criteria for higher order thinking assessment. The second section makes the case for how a “rich, substantive concept of critical thinking” meets those criteria. In making this case, Paul and Nosich spell out the dangers of a non-substantive concept of critical thinking. The third section of the paper spells out four domains of critical thinking: elements of thought, abilities, affective dimensions, and intellectual standards. The fourth section of the paper makes substantive recommendations regarding how to assess the various domains of critical thinking, the test strategies that may be used, the value of the proposed strategy for the reform of education, and the suggested implementation of the proposal.


Section One: Objectives

What Should be the Main Objectives 
of a Process to Assess Higher Order Thinking?

  1. It should assess students’ skills and abilities in analyzing, synthesizing, applying, and evaluating information.

  2. It should concentrate on thinking skills that can be employed with maximum flexibility, in a wide variety of subjects, situations, contexts, and educational levels. 

  3. It should account for both the important differences among subjects and the skills, processes, and affective dispositions that are crucial to all the subjects. 

  4. It should focus on fundamental, enduring forms of intellectual ability that are both fitted to the accelerating pace of change and deeply embedded in the history of the advancement of the disciplines. 

  5. It should readily lead to the improvement of instruction. 

  6. It should make clear the inter-connectedness of our knowledge and abilities, and why expertise in one area cannot be divorced either from findings in other areas or from a sensitivity to the need for interdisciplinary integration. 

  7. It should assess those versatile and fundamental skills that are essential to being a responsible, decision-making member of the work-place. 

  8. It should be based on clear concepts and have well-thought-out, rationally articulated goals, criteria, and standards. 

  9. It should account for the integration of communication skills, problem-solving, and critical thinking, and it should assess all of them without compromising essential features of any of them. 

  10. It should respect cultural diversity by focusing on the common-core skills, abilities, and traits useful in all cultures. 

  11. It should test for thinking that is empowering and that, when incorporated into instruction, promotes (to quote the September, 1991 Kappan) “the active engagement of students in constructing their own knowledge and understanding.” 

  12. It should concentrate on assessing the fundamental cognitive structures of communication, for example: 

    • with reading and listening, the ability to
    • create an accurate interpretation,
    • assess the author’s or speaker’s purpose,
    • accurately identify the question-at-issue or problem being discussed,
    • accurately identify basic concepts at the heart of what is said or written,
    • see significant implications of the advocated position,
    • identify, understand, and evaluate the assumptions underlying someone’s position,
    • recognize evidence, argument, inference (or their lack) in oral and written presentations,
    • reasonably assess the credibility of an author or speaker,
    • accurately grasp the point of view of the author or speaker,
    • empathetically reason within the point of view of the author or speaker.
      with writing and speaking, the ability to
    • identify and explicate one’s own point of view and its implications,
    • be clear about and communicate clearly, in either spoken or written form, the problem one is addressing,
    • be clear about what one is assuming, presupposing, or taking for granted,
    • present one’s position precisely, accurately, completely, and give relevant, logical, and fair arguments for it,
    • cite relevant evidence and experiences to support one’s position,
    • see, formulate, and take account of alternative positions and opposing points of view, recognizing and evaluating evidence and key assumptions on both sides,
    • illustrate one’s central concepts with significant examples and show how they apply in real situations,
    • empathetically entertain strong objections from points of view other than one’s own.


  13. It should assess the skills, abilities, and attitudes that are central to making sound decisions and acting on them in the context of learning to understand our rights and responsibilities as citizens, as well-informed and thinking consumers, and as participants in a symbiotic world economy.

  14. It should avoid any reductionism that allows a multi-faceted, theoretically complex, and authentically usable body of abilities and dispositions to be assessed by means of oversimplified parts that do not adequately reflect the whole.

  15. It should enable educators to see what kinds of skills are basic for the future.

  16. It should be of a kind that will assess valuable skills applied to genuine problems as seen by a large body of the populace, both inside and outside of the educational community.

  17. It should include items that assess both the skills of thoughtfully choosing the most reasonable answer to a problem from among a pre-selected set and the skills of formulating the problem itself and of making the initial selection of relevant alternatives.

  18. It should contain items that, as much as possible, are examples of the real-life problems and issues that people will have to think out and act upon.

  19. It should be affordable.

  20. It should enable school districts and educators to assess the gains they are making in teaching higher order thinking.

  21. It should provide for a measure of achievement against national standards.



Section Two: Critical Thinking and Criteria for Assessment

What Is Included in a Rich, 
Substantive Concept of Critical Thinking?

Most of the language we shall use is drawn from draft statements of the National Council for Excellence in Critical Thinking Instruction. The National Council has been established precisely to articulate standards in critical thinking by 50 key leaders in critical thinking research and 105 leading educators. It is in the process of establishing regional offices and setting up 75 research-based committees to articulate the state of research in the field.


Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.
This is the working definition of the National Council for Excellence in Critical Thinking Instruction. Though the definition as well as the other draft statements of the Council are subject to modification and refinement, the basic idea is one that is common to practitioners and researchers in critical thinking.


“In its exemplary form, [critical thinking] is based on universal intellectual values that transcend subject-matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.” (National Council Draft Statement)


a) “It entails the examination of those structures or elements of thought implicit in all reasoning: purpose; problem, or question-at-issue; assumptions; concepts; empirical grounding; inferences; implications and consequences; objections from alternative viewpoints, and frame of reference.” (National Council Draft Statement)

b) It entails larger-scale abilities of integrating elementary skills in such a way as to be able to apply, synthesize, analyze, and evaluate complicated and multidimensional issues. These include such abilities as clarifying issues, transferring insights into new contexts, analyzing arguments, questioning deeply, developing criteria for evaluation, assessing solutions, refining generalizations, and evaluating the credibility of sources of information. Among the abilities are included also the central forms of communication: critical reading, writing, speaking, and listening. Each of them is a large-scaled mode of thinking which is successful to the extent that it is informed, disciplined, and guided by critical thought and reflection. (Paraphrased from National Council Draft Statement.)

c) Critical thinking entails the possession and active use of a set of traits of mind and affective dimensions: independence of thought, fair-mindedness, intellectual humility, intellectual courage, intellectual perseverance, intellectual integrity, curiosity, confidence in reason, and the willingness to see objections, to enter sympathetically into another’s point of view, and to recognize one’s own egocentricity or ethnocentricity. (Paraphrased from National Council Draft Statement.)
Critical thinking — in being responsive to variable subject areas, issues, and purposes — is incorporated in a family of interrelated modes of thinking, among them: scientific thinking, mathematical thinking, historical thinking, anthropological thinking, economic thinking, moral thinking, and philosophical thinking (National Council Draft Statement).


How Does a Rich, Substantive Concept 
of Critical Thinking Meet the 21 Criteria?

In our view, a rich, substantive concept of critical thinking, and it alone, provides an intelligible and workable means of meeting all 21 criteria. In this section we will briefly consider each objective in turn, not as a definitive response to the criteria, but merely to suggest the fuller response in Section Three below.


Can it be used to test information processing skills? Critical thinking includes at its core “a set of information and belief generating and processing skills and abilities.”


Can it be used to test flexible skills and abilities that can be used in a wide variety of subjects, situations, contexts, and educational levels? Since the art of critical thinking “entails proficiency in the examination of those structures or elements of thought implicit in all reasoning — purpose, problem or question-at-issue, assumptions, concepts, empirical grounding, reasoning leading to conclusions, implications and consequences, objections from alternative viewpoints and frames of reference” — it provides for maximum flexibility of use. It can be used in any subject, with respect to any situation to be figured out, in any context in which reasoning is germane, and, if adapted to the proficiency of students, at any educational level.


Can it account for important differences among the subject areas? Subjects differ not because some make assumptions and others do not, not because some pose questions or problems and others do not, not because some have purposes and others do not, but rather because each has somewhat different purposes, and hence asks somewhat different questions, poses somewhat different problems, gathers somewhat different evidence, uses somewhat different concepts, etc. Critical thinking highlights these differences while underlining common structural features.


Can it be used to focus on fundamental abilities fitted to the accelerating pace of change and embedded in intellectual history? Basic critical thinking skills and abilities are readily shown to be implicit in the rational development and critique of ideas at the core of intellectual history. They explain, for example, how new disciplines emerge from established ones: that is, by asking new questions, pursuing new purposes, framing new concepts, gathering new data, making new assumptions, reasoning in new directions, etc. They also explain how it is that a new field of study can ground itself, even at the outset, on definite intellectual standards that transcend any particular academic field: clarity, precision, accuracy, relevance, consistency, evidentiary force, valid reasoning, consistency . . . (standards implicit in the history of critical thinking and rational discourse in every domain).


Can it be used to improve instruction? Critical thinking is not an isolated good, unrelated to other important goals in education. Rather it is a seminal goal which, done well, simultaneously facilitates a rainbow of other ends. It is best conceived, therefore, as the hub around which all other educational ends cluster. For example, as students learn to think more critically, they become more effective readers, writers, speakers, and listeners, because each ability requires well-reasoned thought.

They increase their mastery of content because all content is embedded in a system of understandings which, to be grasped, must be reasoned through. They become more proficient in — because they must be practiced within — a variety of modes of thinking: for example, historical, scientific, and mathematical thinking. Self-confidence increases with the intellectual empowerment critical thinking engenders. Finally, they develop skills, abilities, and traits of mind (intellectual discipline, intellectual perseverance, intellectual humility, intellectual empathy, intellectual integrity, …) crucial to success in the educational, professional, and everyday world.


Can it make clear the inter-connectedness of our knowledge and abilities, and why expertise in one area cannot be divorced either from findings in other areas or from a sensitivity to the need for interdisciplinary integration? In learning to think critically, one learns to transfer what one has learned about the logic of questions in one field to logically similar questions in other fields. Typically this begins with a recognition of the need to ask questions based on logical parallels between all fields of study, for example, skilled practice in questioning concepts and theories, in questioning data, in questioning the source or interpretation of data, in questioning the nature or organization of data, in questioning inferences, in questioning assumptions, in questioning implications and consequences, in questioning points of view and frames of reference, etc.


Can it be used to assess those versatile and fundamental skills essential to being a responsible, decision-making member of the work place? Critical thinking skills and abilities are highly transferable to the work place. Since in learning to think critically we learn to take increasing charge of our minds as an instrument of learning — for example, reading, writing, speaking, and listening with greater discipline and skill — we are well situated to engage in collective problem solving and goal attainment, wherever they occur. The kind of “work” increasingly required in industry and business is “intellectual”, that is, it requires workers to define goals and purposes clearly, seek out and organize relevant data, conceptualize those data, consider alternative perspectives, adjust thinking to context, question assumptions, modify thinking in the light of the continual flood of new information, and reason to legitimate conclusions. Furthermore, the intellectual work required must increasingly be coordinated with, and must profit from the critique of, fellow workers. There is no avoiding the need, therefore, to express ideas well, accurately represent and consider fairly the ideas of others, write clear and precise memos and documents, and coordinate and sequence all of these so that well-reasoned policies and decisions can be accurately understood and effectively implemented.


Can it generate clear concepts and well-thought-out, rationally articulated goals, criteria, and standards? Since critical thinking is based on the art of monitoring one’s thinking with standards implicit in the universal structure of thought, and since the use of these standards is implicit in intellectual history from Socrates through Einstein, there is no problem using critical thinking to generate clear concepts for testing, as well as rationally articulated goals, criteria, and standards.


Can it account for the integration of adult-level communication skills, problem solving, and critical thinking, and legitimately assess all of them without compromising essential features of any of them? Shallow concepts of critical thinking often distinguish critical thinking from problem solving and decision making as well as from reading, writing, and speaking skills.

Once one considers a rich, substantive concept of critical thinking, however, it is clear that each of the basic skills of critical thinking are presupposed by each of the other skills, just as each of them is deeply interrelated to critical thinking as a whole. Consider, does it make sense to analyze potential solutions to problems or the implications of choosing an alternative in making a decision without using critical thinking? Clearly not. Every problem to be solved (or question to be settled) requires a critical analysis of the conditions under which it can be solved or settled. We, as problem-solvers, need to look critically at the purpose for solving the problem, we need to critically examine contextual factors, our assumptions, our concepts, what we are using as data, our organization of the data, the source of the data, our reasoning, the implications of our reasoning, our point of view, objections from other points of view. All of these are essential to higher order problem solving and decision-making.

Furthermore, all of these intellectual abilities are crucial to higher order reading, writing, speaking, and listening. To read must we analyze the text and re-create its logic in our own minds. To write we must construct a logic our readers can translate into the logic of their thought. To speak we must articulate our thoughts in such a way that our audience can translate our thoughts into their experiences. To listen we must analyze the logic of the thinking of the speaker. Intellectually disciplined reading, writing, speaking, and listening require, in other words, that we work explicitly with the logic we are constructing or re-constructing, using our grasp of the standards of critical thinking to communicate accurately and precisely, effectively solve problems, and rationally make decisions.


Does it respect cultural diversity by focusing on the common-core skills, abilities and traits useful in all cultures? As the criterion presupposes, we can respect cultural diversity best by constructing tests in higher order thinking that focus on skills and abilities necessary in all modern cultures. In this way we can legitimately justify assessing it in all cultural groups. Basic critical thinking skills and abilities — because they are based on fundamental elements implicit in the structure of all reasoned thought per se, and because their mastery is essential to higher order thinking in all academic, professional, personal, and public life — are an appropriate foundation for assessment.


Does it test for thinking that promotes (to quote the September, 1991 Kappan) “the active engagement of students in constructing their own knowledge and understanding?” Narrow concepts of critical thinking sometimes characterize it in negative terms, as a set of tools for detecting mistakes in thinking. A rich, substantive concept of critical thinking, however, highlights its central role in all rationally defensible thinking, whether that thinking is focused on assessing thought or products already produced, or actively engaged in the construction of new knowledge or understandings. Well-reasoned thinking, whatever its end, is a form of creation and construction. It devises and articulates purposes and goals, translates them into problems or questions, seeks data that bear upon problems or questions, interprets those data on the basis of concepts and assumptions, and reasons to conclusions within some point of view. All of these are necessary acts of the reasoning mind and must be done “critically” to be done well. Hence all require critical thinking.


Does it concentrate on assessing the fundamental cognitive structures of communication? Each of the dimensions identified in the objective is either straightforwardly a critical thinking ability or depends on a critical thinking ability. The writer’s or speaker’s purpose, implications, assumptions, point of view, etc., are all elements of thought, and the ability to identify and assess those as one reads or listens — the ability to construct in one’s mind an accurate and fertile interpretation — is simply thinking by listening, thinking by reading.
A similar reliance on elements of thought is central to writing or speaking effectively at any educational level. The knowledge of how to gather and present evidence, to make clear one’s own assumptions, to see the implications of a position: these are critical thinking abilities.

All forms of communication, moreover, rely on critical thinking standards. Essays and interpretations of essays, utterances and interpretations of utterances, need to be relevant, logical, consistently worked out; evidence needs to be recorded and reported accurately; points need to be made clearly and with as much precision as the subject permits; topics need to be covered in depth and presented fairly.


Can it be used to assess the central features of making rational decisions as a citizen, a consumer, and a part of a world economy? Both public and private life increasingly requires mastery of the basic skills and abilities of critical thinking. When this mastery is absent the public degenerates into a mass society susceptible to manipulation by public relations specialists who can engineer political victories by an adroit use of mud slinging, scare tactics, shallow nationalism, fear, envy, stereotypes, greed, false idealism, and maudlin sentimentality.

Modern citizenship requires basic critical thinking skills and abilities throughout. The modern citizen should be able to assess the arguments presented for his or her assent, must rationally adjudicate between conflicting points of view, must attempt to understand a culturally complex world, must assess the credibility of diverse sources of information, must translate between conflicting points of view and diverse appeals, must rationally decide priorities, must seek to understand complex issues that involve multiple domains (for example, the environmental, moral, economic, political, scientific, social, and historical domains). Without a solid grounding in critical thinking, citizens are intellectually disarmed, incapable of discharging their civic responsibilities or rationally exercising their rights.


Can it avoid reducing a complex whole to oversimplified parts? Testing for a rich, substantive concept of critical thinking is testing for skills of reasoning in terms of elements of thought, for the ability to orchestrate those elementary skills, for the affective dimensions that make critical thinking actualizable in practice, and for universal intellectual standards, in short for a rich and complex whole rather than for fragmented parts.


Can it articulate what is central to basic skills for the future? Basic skills are constituted by the structures explicated in a rich, substantive concept of critical thinking. To teach reading is to teach the ability not merely to repeat content, but to reconceptualize that content, to see applications of the main ideas, to generalize from them, critique them, see them in context, to enter with empathy into another’s point of view. To teach writing as a basic skill is to teach not merely grammar and punctuation, but the ability to arrange one’s ideas logically and consistently, to anticipate reasonable objections, to transfer ideas to the page in a way that makes them decipherable in all their complexity by a reader. To teach math as a basic skill is not primarily to teach how to solve pre-selected, individual, isolated problems out of context, but to teach the ability to begin to make sense of the world mathematically, to think quantitatively, to be able to see mathematical patterns, to set up the construction of problems and then creatively go about solving them. Critical thinking abilities like these do not exist somehow in addition to the basic skills of life; they constitute the basic skills of life.


Can it provide the kind of skills that are seen as valuable outside the school as well as inside it? Critical thinking provides skills that are seen as valuable by practitioners of the academic disciplines, by responsible leaders of government, of the professions, of business, by citizens interested in their environmental, physical, and economic welfare. In all such areas what is needed are ways to adapt to rapidly changing knowledge, to recognize problems and see their implications before they become acute, to formulate approaches to their solution that recognize legitimately different points of view, to draw reasonable conclusions about what to do.

Increasingly, one is hearing statements such as the one made by David Kennedy, the president of Stanford University, to 3,000 college and university presidents: 

"It simply will not do for our schools to produce a small elite to power our scientific establishment and a larger cadre of workers with basic skills to do routine work. Millions of people around the world now have these same basic skills and are willing to work twice as long for as little as one-tenth our basic wages. To maintain and enhance our quality of life, we must develop a leading-edge economy based on workers who can think for a living. If skills are equal, in the long run wages will be too. This means we have to educate a vast mass of people capable of thinking critically, creatively, and imaginatively."


Can critical thinking be assessed in a way that requires evaluation of authentic problems in realistic contexts, where the abilities assessed include those of formulating the problem and initial screening of plausible solutions? Yes. Testing of authentic skills, abilities and dispositions in authentic contexts can be accomplished by using a combination of a) standard multiple-choice items, b) machine-gradable multiple-rating items and c) short essay items.

     a) The standard multiple-choice part of the assessment would be an expanded version of established critical thinking tests, such as the Watson-Glaser or Cornell tests. It is suitable for assessing micro-dimensional critical thinking skills, like identifying the most plausible assumption, recognizing an author’s purpose, selecting the most defensible inferences, and such like.

b) The multiple-rating part of the assessment would test more open-ended and larger-domained abilities, like thinking within opposing points of view, the willingness to suspend judgment, the ability to synthesize disparate data into a logical scheme, to take established findings and generalize them into new contexts, etc.

The multiple-rating portion of the assessment, to be reliable, must:


i) embody a rich and substantive idea of critical thinking

ii) be constructed and monitored by critical thinking experts who have such a concept

iii) be changed often (5% annually) to assess critical thinking with respect to authentic contemporary issues

  c) The essay part of the assessment would be designed to address critical thinking abilities and traits that involve creating a logic to capture a situation rather than selecting from among possibilities suggested by the test. Examples include the ability to construct an interpretation, to make a logical outline of a text, to figure out ways to gather information, to take an unclear and complex real issue and reformulate it so as to make it more amenable to solution. 

Validity on the essay part of the assessment requires that the test be:
i) constructed by experts in critical thinking,
ii) assembled from a large and rotating bank of short essay questions to allow for items that show no significant differences,
iii) centrally graded by teams well-trained in a full concept of critical thinking in order to assure quality control.


Can critical thinking be assessed nationally in a way that is financially affordable? To make it affordable, the constructed response segment of the assessment should be administered not to the population of students as a whole, but rather to a representative sample of the student population of a school system. The assessment should be a) paid for by school systems that contract to have their students tested, and b) constructed, monitored, administered, and graded by a private agency with critical thinking credentials, or at least under the direction of scholars with a solid grounding in the research into critical thinking.


Can critical thinking be assessed so as to gauge the improvement of students over the course of their education and to measure the achievement of students against national standards? To evaluate students in both these dimensions requires:

     a) an assessment administered as a pre-test at the 6th grade and then as a follow up at the 9th and 12th grades (to provide for value-added judgments)

     b) a criterion-referenced assessment that is built on clear, consistently applied quality-norms that are derived from a rich and substantive concept of critical thinking (to provide for the measuring of national progress)

What, Specifically, Are the Dangers of a 
Non-Substantive Concept of Critical Thinking?

It is important to be alert to the dangers posed by a non-substantive concept of critical thinking. Such a concept exists when, separate from a consideration of the research in the field, a person or institution presupposes a) that the meaning or terminology of critical thinking is intuitively obvious (hence not in need of scholarly analysis), or b) that each concept underlying critical thinking (such as assumption, inference, implication, reasoning . . . ) can be analyzed separately from a theory that accounts for the interrelation of these concepts, or c) that the skills of critical thinking can be adequately cultivated without reference to the values, traits of mind, and dispositions that underlie those skills.

There Are At Least Three Serious Problems That May Result 
From the Use of a Theoretically Superficial Concept of Critical Thinking

    a) important critical thinking concepts, which must be clearly defined to be used effectively in assessment, may be used vaguely,  inconsistently, incorrectly, or misleadingly
    b) a false, misleading, or simplistic over-arching concept of critical thinking may be fostered

    c) an unrealistic strategy for the assessment and cultivation of critical thinking may be incorporated into testing and teaching

Many examples of the unwitting use of a non-substantive concept of critical thinking could be cited — such as “thinking skills” programs devoid of intellectual standards (which, for example, systematically confuse “inferences” with “valid inferences” and “analogies” with “sound analogies”), or testing personnel who lack adequate grounding in critical thinking theory (and so, for example, frequently confuse assumptions with inferences or inferences with implications). The most far-reaching danger occurs when influential educational systems or institutions, like state departments of education, inadvertently incorporate a non-substantive concept of critical thinking into statewide curriculum standards or into statewide testing programs. This can result in significant, unintended negative consequences, for example: thousands of teachers encouraged to follow a misconceived model for the assessment of reasoning, leading to mis-instruction on a grand scale.


We shall look at one important case. Unfortunately, given the brevity of this paper, one case must stand for all. The case we have chosen concerns the Integrated Language Arts Assessment of the California Assessment Program, a massive statewide program that has impact not only on every student in the public schools of California, but also, because of the leadership role of California in assessment, on national teaching and testing practices as well. It appears that three fundamental mistakes occurred in the design of the direct writing assessment:


a) Though one of the goals of the program was to place an emphasis on the     quality of reasoning and critical thinking in writing, it appears that no one     with a research background in critical thinking reviewed the articulation or     implementation of the assessment prompts. (We infer this from the fact     that fundamental conceptual errors occur both in the prompts themselves     and in the application of criteria to student constructed responses.)

b) It was assumed, inappropriately, that classroom teachers without     extended training in critical thinking are able to effectively assess student     essays that call for evaluative reasoning. We infer this from statements     descriptive of the assessment design like:

Teachers on the CAP writing Development Team develop all the testing and instructional materials for assessment. For every type of writing assessed, the team develops a special set of prompts … and a scoring guide that identifies the thinking and writing requirements for that type of writing . . .  Essays are scored in four to six days by several hundred teachers at four regional scoring centers. A special handbook for each grade level provides teachers with practical instructional materials for each type of writing, including sample prompts, illustrative essays, and related readings.

c) The resulting assessment was not monitored by anyone with a research background in critical thinking. (We infer this from the fact that model “strong” answers purporting to illustrate critical reasoning are showcased that are in fact patently very weak answers, containing virtually no reasoning at all.)

Consider Figure 1 and Figure 2 used as illustrations of the nature and quality of the writing assessment program in an article authorized and developed by the staff of the California Assessment Program. It is entitled “California: The State of Assessment” and was written for an important national anthology, Developing Minds (more than 150,000 copies disseminated by ASCD). The show-piece article, in which these figures occur, argues that the examples illustrate a “state-of-the-art teacher-developed writing assessment” that is sophisticated in “its testing, scoring, and reporting systems” and designed to “include only those tasks that will stimulate high-quality instruction.”

There are a number of problems illustrated in these figures that a substantive understanding of critical thinking would have avoided:

  1. A description of subjective reactions was systematically confused with sound evaluative reasoning. It is important to distinguish questions like, “Is rock music good music?” or “Does rock music excel as a form of music?” (which call for objective evaluation) from questions like, “Do you enjoy rock music?” or “Does rock music stir powerful emotions in you?” (which call, not for reasoning, but for the description of subjective reactions). Apparently the test developers were unclear about this distinction.
  2. The assessing teachers did not notice that the student failed to respond to the directions. The student did not develop evaluative reasoning, did not support his judgment with reasons and evidence, did not consider possible criteria on which to base his judgment, did not analyze the subject in the light of the criteria, and did not select evidence that clearly supported his judgment. Instead the student described an emotional exchange, asserted — without evidence — some questionable claims, and expressed a variety of subjective preferences (a fuller critique of the student essay is available on page 170 of, “Why Students — and Teachers — Don’t Reason Well”). The assessing teachers were apparently too confused about the nature of evaluative reasoning or the basic notions of criteria, evidence, reasons, and well-supported judgment to notice the discrepancy.
  3. The California State Department of Education assessment staff did not notice these errors once they were made. Instead of catching the errors once made, the California Department of Education chose to use the mis-graded student essay as a showcase model to disseminate nationally as illustrating “exceptional achievement” in reasoned evaluation, and as a model of their assessment of reasoned writing. We conclude that the California Assessment Program does not use scholars with a background in critical thinking research, any of whom would surely have recognized the problem.

Fundamental misconceptions of the nature of critical thinking and reasoned discourse, such as those documented above, must not be replicated in a national assessment program. Steps should be taken to insure that a substantive concept of critical thinking and a well-supervised implementation of that concept form the basis of the finished assessment program.


Section Three: The Four Domains of Critical Thinking

What are the four component domains of critical thinking and their implications for the assessment of higher order thinking?

Elements of Thought

As soon as we move from thought which is purely associational and undisciplined, to thinking which is conceptual and inferential, thinking which attempts in some intelligible way to figure something out, to use the power of reason, then it is helpful to think about what can be called “the elements of thought.” The elements of thought are the basic building blocks of thinking, essential dimensions of reasoning whenever and wherever it occurs. Working together, they shape reasoning and provide a general logic to reason. We can articulate these elements by paying close attention to what is implicit in the attempt on the part of the mind to figure anything out whatsoever. Once we make them clear, it will be obvious that each of them can serve as an important touchstone or point of assessment in critical analysis and in the assessment of thinking.

For each of the elements of thought there is a cluster of attendant basic thinking skills. Because they involve fundamental structures of thought, these skills can be characterized as micro-skills, those skills out of which larger-domained critical thinking abilities are built. Being able to think critically about a particular issue, then, will include the ability to identify, clarify, and argue for and against alternative formulations of the elements of thought.

The basic conditions implicit whenever we gather, conceptualize, apply, analyze, synthesize, or evaluate information the elements of thought are as follows:

  1. Purpose, Goal, or End in View. Whenever we reason, we reason to some end, to achieve some objective, to satisfy some desire or fulfill some need. One source of problems in reasoning is traceable to defects at the level of goal, purpose, or end. If the goal is unrealistic, for example, or contradictory to other goals we have, confused or muddled in some way, then the reasoning used to achieve it is problematic. An assessment of critical thinking, then, would test, at the appropriate educational level, skills of being able to state an author’s purpose, to identify a plausible statement of an author’s goals from a list provided, to rank formulations of an author’s objectives according to which are more or less reasonable in light of a particular passage, to distinguish clearly between purposes, consequences, assumptions, and other elements of thought.
  2. Question at Issue, or Problem to be Solved. Whenever we attempt to reason something out, there is at least one question at issue, at least one problem to be solved. One area of concern for reasoners, therefore, will be the formulation of the question to be answered or problem to be solved, whether with respect to their own reasoning or to that of others.
    Assessing skills of mastery of this element of thought would test students’ ability to formulate a problem in a clear and relevant way, to choose from among alternative formulations, to discuss the merits of different versions of the question at issue, to recognize key common elements in statements of different problems, to structure the articulation of problems so as to make possible lines of solution more apparent.
  3. Point of View, or Frame of Reference. Whenever we reason, we must reason within some point of view or frame of reference. Any “defect” in that point of view or frame of reference is a possible source of problems in the reasoning. A point of view may be too narrow, too parochial, may be based on false or misleading analogies or metaphors, may contain contradictions, and so forth. 
    Levels of skill here would be tested with reference to being able to enunciate an author’s point of view in a passage, to adjudicate between different statements of that point of view, to recognize bias, narrowness, and contradictions when they occur in the point of view, to recognize relations between the frame of reference being used and its implications, assumptions, and main concepts. 
  4. The Empirical Dimension of Reasoning. Whenever we reason, there is some “stuff”, some phenomena about which we are reasoning. Any “defect”, then, in the experiences, data, evidence, or raw material upon which a person’s reasoning is based is a possible source of problems. Students would be tested, again, based on their level, on their ability to distinguish evidence from conclusions based on that evidence, to give evidence themselves, to identify from a pre-selected list data that would support an author’s positions, data that would oppose it, data that would be neutral, to notice the presence or lack of relevant evidence, to recognize, to be intellectually courageous in recognizing (and labeling as such) mere speculation that goes beyond the evidence. 
  5. The Conceptual Dimension of Reasoning. All reasoning uses some ideas or concepts and not others. These concepts can include the theories, principles, axioms and rules implicit in our reasoning. Any “defect” in the concepts or ideas of the reasoning is a possible source of problems. The assessment of the relevant higher order thinking would test the ability to identify main concepts of a passage, to choose among different versions of those concepts (some perhaps equally good), to see relations among concepts, to reason about the similarity of points of view on the basis of similarity of fundamental concepts, to distinguish central from peripheral concepts, derived concepts from basic concepts, to see the implications of using one concept rather than another.
  6. Assumptions. All reasoning must begin somewhere, must take some things for granted. Any “defect” in the assumptions or presuppositions with which the reasoning begins is a possible source of problems. Assessing skills of reasoning about assumptions would test the ability to identify assumptions underlying given inferences, points of view, and goals, to evaluate the accuracy of different formulations of the assumptions, to distinguish between assumptions and inferences, to rank assumptions with respect to their plausibility, to be intellectually fair-minded by choosing the most plausible version of assumptions underlying points of view with which they disagree.
  7. Implications and Consequences. No matter where we stop our reasoning, it will always have further implications and consequences. As reasoning develops, statements will logically be entailed by it. Any “defect” in the implications or consequences of our reasoning is a possible source of problems. Skills to be assessed would include the ability to identify important implications, to do so by selecting from a list of possible implications, to make fine discriminations among necessary, probable, and improbable consequences, to distinguish between implications and assumptions, to recognize the weakness of an author’s position as shown by the implausibility of its implications, to exercise intellectual fair-mindedness in discriminating between the likelihood of dire and mild consequences of an action to which one is opposed.
  8. Inferences. Reasoning proceeds by steps in which we reason as follows: “Because this is so, that also is so (or probably so),” or “Since this, therefore that.” Any “defect” in such inferences is a possible problem in our reasoning.

Assessment would test, in a way geared to their educational level, students’ ability to recognize faulty and justified inferences in a passage, to rank inferences with respect to both their plausibility and their relevance, to make good inferences in their own reasoning, to discriminate among various formulations of an author’s inferences with respect to which is most accurate, to take something they do not believe but to entertain it for the sake of argument and draw reasonable inferences from it.

Assessment of Elements of Thought

Any program for the assessment of critical thinking skills must itself be assessed in terms of its validity and reliability in testing for the ability to think about, and in terms of, the elements of thought. These abilities can be successfully assessed in three related ways: by a restricted use of standard multiple-choice items, by multiple-rating items, and by short essay items. Both multiple-choice and multiple-rating items are machine-gradable, while essay items are not.

Although our recommendations about the content of the assessment will be spelled out in detail in Section Four, some of these can be anticipated here with respect to the assessment of reasoning abilities centering on the elements of thought. 
Multiple-choice testing (as in the existing Watson-Glaser Critical Thinking Appraisal or the Cornell Critical Thinking Tests) is an important part of an assessment of critical thinking, but its legitimate use is restricted to testing only the most basic skills of identifying and recognizing elements of thought, and then only as they occur in relatively short and unambiguous excerpts.

Within this domain, multiple-choice questions will require students:

  • to identify an author’s purpose in a passage;
  • to rate selected inferences as justified, probably true, insufficiently evidenced, probably false, unjustified;
  • to select among formulations of the problem at issue in a passage those that are clearly reasonable, probably reasonable, probably unreasonable, clearly unreasonable;
  • to recognize unstated assumptions;
  • to distinguish evidence from hypotheses and conclusions;
  • to rate described evidence as reliable, probably reliable, probably not reliable, unreliable.


The elements of thought do not exist in isolation from one another, nor — more importantly for the concept of an assessment procedure — do they exist outside a particular context of application. In the practice of good critical thinking, skills more closely associated with elements of thought are orchestrated into larger-domained abilities which are applied to thinking about complex and sometimes ambiguous issues, problems, decisions, theories, states of affairs, social institutions, and human artifacts.

These critical thinking abilities include being skillful at:

  1. refining generalizations and avoiding over-simplifications,
  2. comparing analogous situations: transferring insights into new contexts,
  3. developing one’s perspective: creating or exploring the implications of beliefs, arguments, or theories,
  4. clarifying issues, conclusions, or beliefs,
  5. clarifying and analyzing the meanings of words and phrases,
  6. developing criteria for evaluation: clarifying values and standards,
  7. evaluating the credibility of sources of information,
  8. questioning deeply: raising and pursuing root or significant questions,
  9. analyzing or evaluating arguments, interpretations, beliefs, or theories,
  10. generating or assessing solutions,
  11. analyzing or evaluating actions or policies,
  12. reasoning dialogically: comparing perspectives, interpretations, or theories,
  13. reasoning dialectically: evaluating perspectives, interpretations, or theories,
  14. reading critically: constructing an accurate interpretation of, understanding the elements of thought in, and evaluating, the reasoning of a text,
  15. listening critically: constructing an accurate interpretation of, understanding the elements of thought in, and evaluating, the reasoning of an oral communication,
  16. writing critically: creating, developing, clarifying, and conveying, in written form, the logic of one’s thinking,
  17. speaking critically: creating, developing, clarifying, and conveying, in spoken form, the logic of one’s thinking.

Abilities like these play a central role in a rich and substantive concept of critical thinking. They are essential to approaching actual issues, problems, and situations rationally. Understanding the rights and duties of citizenship, for example, requires that one at least have the ability to compare perspectives and interpretations, to read and listen critically, to analyze and evaluate policies. In fact, there is no macro-ability on the list that would not be relevant or even crucial to thinking deeply about the rights and duties of citizenship.

Similarly, the capacity to make sound decisions, to participate knowledgeably in the work-place, to function as part of a global economy, to master the content in anything as complex as the academic disciplines, to apply those subject area insights to real-life situations, to make insightful cross-disciplinary connections, to communicate effectively — each of these relies in a fundamental way on having a significant number of the abilities listed. Take, for example, the capacity to make sound decisions: such decision-making is hardly possible without an attendant ability to (going down the list of abilities in order) refine generalizations, compare analogous situations, develop one’s perspective, clarify issues, and so forth.

The last four abilities listed — the ability to read, write, listen, and speak, each in a critical, informed, constructive way — are best considered not as in the usual model, not as manifestations of thinking already accomplished, but as being themselves actual modes of constructive thinking. As such, they are structured amalgams of elementary skills together with any number of other abilities.

Assessment of Abilities

The assessment of abilities, too often neglected, is essential to assessment of critical thinking. Since these are the abilities implicit in the realistic use of thinking, no assessment tool that fails to assess a significant number of these abilities could justifiably be called an assessment of higher order thinking. The assessment, moreover, needs to address such abilities directly (rather than through secondary indicators), systematically (rather than haphazardly as a result of an attempt to assess other variables like academic achievement), and in settings as authentic as possible given the requirement of uniform, relevant grading.
Assessment of abilities that meets these four criteria cannot be accomplished within the confines of a standard multiple-choice-type test. It can be accomplished, however, for all of the abilities (except those having to do with oral communication), by means of a combination of machine-gradable multiple-rating items and essay items.

For any macro-ability, there will be dimensions of the ability that are generative and other dimensions of it that are selective. In trying to solve a real problem, for example, much of one’s thinking is devoted to generating a formulation of the problem that will make it more susceptible to solution. Another, and quite different, aspect of problem solving, is the ability to select, from among a large variety of possibilities, that avenue of thought which will most likely result in a solution. Students who are trained using a rich, substantive concept of critical thinking tend to improve in both dimensions of this ability, and both are genuine dimensions of real problem-solving.

The selective dimensions of an ability can be assessed accurately, even in complex, ambiguous, and subtle cases, using multiple-rating items. The generative dimension, on the other hand, cannot. Since it requires students to come up with their own critical thinking approaches within that macro-ability, this dimension can be assessed adequately only by carefully constructed and carefully graded essay tests. Details of the assessment and samples of assessment items will be presented in Section Four.

Affective Dimensions

Higher order thinking requires more than higher order thinking skills. Critical thinking, in any substantive sense, includes more than abilities. The concept also includes, in a crucial way, certain attitudes, dispositions, passions, traits of mind. These affective dimensions are not merely important to critical thinking; they are essential to the effective use of higher order thinking in real settings. 
These affective dimensions include:

  1. thinking independently,
  2. exercising fair-mindedness,
  3. developing insight into egocentricity and socio-centricity,
  4. developing intellectual humility and suspending judgment,
  5. developing intellectual courage,
  6. developing intellectual good faith and integrity,
  7. developing intellectual perseverance,
  8. developing confidence in reason,
  9. exploring thoughts underlying feelings and feelings underlying thoughts,
  10. developing intellectual curiosity.

Without intellectual perseverance, one could not solve the complicated, multi-faceted problems one confronts in industry. Without intellectual courage, one could not maintain a defense of citizenship rights in the face of scare tactics. Without fair-mindedness, one could not enter into another’s point of view and thus would lack that empathetic understanding necessary for a reasonable approach to living in a pluralistic society. Without developing insight into egocentricity and socio-centricity one could employ one’s reasoning skills in a merely self-serving and prejudiced way. Without confidence in reason one could not adequately address those complex and frequently ambiguous real-life problems that require reasonable decisions in the face of crucial uncertainties.

Assessment of Affective Dimensions

The assessment of affective dimensions of critical thinking is an important part of an assessment of higher order thinking. An initial problem is that from the fact that all these dimensions are essential, it does not follow that all are directly testable, nor does it follow that any of them is easily testable. For some of these affective dimensions (intellectual perseverance, for example), any testing would have to take place over an appropriately long period of time and thus could not be legitimately assessed at all during a time frame suitable for a national test.
Nevertheless, a number of affective dimensions can be assessed in a relatively straightforward way using essay items and, especially, machine-gradable multiple-rating items.

“Reasoning Within Conflicting Points of View,” a central aspect of the disposition of fair-mindedness, is already being assessed on a machine scorable test developed by Dr. Richard Paul. This Appraisal asks students to select the strongest (that is, the most defensible) argument in favor of each side of a pair of conflicting and sometimes emotionally charged points of view. Proficiency on these items indicates a fair-minded willingness to distinguish the concept of reasonable defensibility from that of personal belief.

Multiple-rating items are currently being prepared that address aspects of intellectual courage, other aspects of fair-mindedness, aspects of intellectual humility, and aspects of the development of insight into one’s own egocentricity and socio-centricity.

Intellectual Standards

In any domain where assessment is taking place, there are standards implicit in that assessment. Higher order thinking is thinking that meets universal intellectual standards. Thus, when assessing a student’s ability to compare and evaluate perspectives (a macro-ability) and to do so with fair-mindedness (a trait of mind), we would judge whether she had made such evaluations in a relevant and consistent way, with attention to accuracy, fairness, and completeness in describing each perspective, and with a sensitivity to the degree of precision appropriate to the topic.

We would assess critical thinking about and in terms of the elements of thought in very much the same

    Enrique Rubio

    Enrique Rubio

    Reflexiones sobre aprendizaje, tecnología y sostenibilidad