Useful Higher Ed Assessment Website

The National Institute for Learning Outcomes Assessment (NILOA), based at the University of Illinois, Urbana-Champaign, publishes briefs and articles about the challenges of learning assessment in higher education. Peter Ewell reports on progress with the Degree Qualifications Profile, an effort supported by the Lumina Foundation.

Technologies for Assessing Deeper Learning in Higher Education

More and more college faculty are grappling with the challenges of crossing disciplinary boundaries to ensure their students are learning critical thinking skills. Both accreditation agencies and employers increasingly seek evidence that such skills are learned in college. Faculty commonly focus on teaching the rules of reasoning in their own disciplines, but branching out to teach reasoning across the disciplines is something new. I gave a presentation today at the faculty professional development day at Notre Dame de Namur University, a small, Catholic 4-year college in Belmont, CA. The talk presents an overview of new educational technologies in higher education and some links to online tools that faculty can use. In addition, I presented a perspective from educational and cognitive psychology on the concept of "critical thinking." The practical implication of this perspective is that unless faculty explicitly help students make connections across disciplines through assignments and assessments--most students will not make such connections. Why? Cognitive psychology tells us based on empirical evidence that people learn to reason within domains. As a result, what constitutes good reasoning in one domain does not precisely constitute good reasoning in another domain. The underlying logic does not easily "transfer." The sidebar conversations with faculty generated some interesting ideas. Maybe simply preparing students to know that whenever they enter a new domain, they need to appreciate that the "rules of the reasoning game" change constitutes a kind of critical thinking skill. If faculty in different disciplines can work together to articulate the "rules of the game" within their respective disciplines, and then discuss how to share the essentials across disciplines, they may move into a more coherent and effective approach to demonstrating results in improved critical thinking. One professor discussed how he teaches modus ponens to his students, and then has them apply this logical rule to different contexts. While such a logical reasoning framework has broad application and relevance, it does not necessarily help students apply, say, the rules of foreign language conjugation, the rules of biological metabolic processes, or the rules of supply and demand in economics. These latter reasoning systems are forms of "model-based reasoning" that have a non-intuitive quality. They constitute a domain-specific form of reasoning that is essential to be a critical thinker in those domains and essential for quality reasoning for a wide range of problems in the world. I look forward to hearing more about how the NDN faculty address their challenges around critical thinking. They may soon be exploring the idea of identifying the various essential forms of "model-based reasoning" per domain and using these as a basis for providing evidence of critical thinking across the disciplines.

Two interesting perspectives on the way higher ed assessment is turning. First, a communications professor from Florida State University, is using scores of social networking popularity called Klout in course grading. Second, a professor for a massive open online course (MOOC), which includes "tens of thousands" of students, is using "peer grading" to grade papers. An article from Inside Higher Ed captures the lack of consistency and rigor in peer grading. It is a course on the history of cyberstructure and is offered by Coursera through the University of Michigan. Two points: Asking peers to apply rubrics might be a first step, although I've noted in my own research that no two instructors apply the same rubrics in the same ways. Obtaining reliable grading takes time and practice. Perhaps such training and calibration processes need to be set up online? Second, I don't see anything wrong with incorporating a real world metric to students get a feel for it, but expecting them to develop a following in the 12 to 16 weeks of a course seems a bit unrealistic.

Accountability Analysis

States vary widely in how they track college student achievement, according to a new report cited in Inside Higher Ed. Responses to the report are mixed. While the call to account is generally lauded, but attempts to characterize progress as a horse race strike some as gimmicky. In our project, Domain-Specific Assessment, which is sponsored by the U.S. Department of Education's Institute of Education Sciences, we are finding what cognitive scientists such as John Anderson and Richard Clark have found all along: Getting experts, such as postsecondary educators, to articulate the learning outcomes that matter is not so easy. Part of the problem is psychological: When someone becomes an expert in any field, lots of the cognitive work becomes automatic, chunked, and unconscious. Multiple steps become routines that are no longer carefully parsed. This can lead to a gap in how we articulate what it is that students need to learn in higher education. To see full report, check column to left.

Higher Education Accountability

In my morning reading of Inside Higher Ed, I noted that accountability for student learning outcomes was on the docket at a meeting this week among representatives of the U.S. Department of Education, various higher education associations, and accreditation agencies. The upshot of the negotiations, which have been tense in the past, was that institutions of higher learning will continue to set their own agenda for the outcomes they seek to achieve. Efforts to deputize accreditation agencies to define and collect data on other outcomes appeared to be overruled. This action may be framed as a step forward for continued higher education academic freedom or a loss for those concerned about assured consistent quality of learning across various higher education institutions.

21st Century Skills

If you want students to learn 21st-century skills, one approach involves giving them problems in school that resemble those they may tackle in their everyday lives. The thinking is that by solving such problems, students will learn the skills of teamwork, communication, project management, and complex problem solving. Yet, it might be difficult to convince teachers to use such time-consuming activities in their classrooms, and the research on the effectiveness of such activities is mixed at best.

Of course, the ho-hum effectiveness results might have more than a little to do with the limitations of the tests of effectiveness than the actual classroom activities. And so begins the assessment debate. Currently, most American teachers at the K-12 level and beyond, are required to teach to tests that check student performance on more academically-defined standards. These standards generally focus on knowledge that is easy to test: core facts, concepts, and procedures--knowledge that cognitive psychologists characterize as "declarative" and "procedural."

You don't have to be a psychologist to know that life does not present itself as a multiple choice test, but more as an unstructured series of problems that require a bit more than reciting vocabulary and demonstrating you can transform an equation. Proponents of teaching 21st-century skills fear that students focused strictly on proving they have learned the more academic standards will fail to learn the kinds of higher-level skills that will keep them competitive in the global economy. We all can remember those first years out of college when we realized how little we learned in school seemed to matter...

Incidentally, despite the press of standardized testing, teachers have long been encouraged to teach students higher-level skills through instructional categorization systems that hierarchically rate classroom activities according to their cognitive qualities. Bloom's Taxonomy is a well-known hierarchy. It involves rating activities that require students to apply, evaluate, argue, or synthesize knowledge as the higher-order kinds of activities. Check the tasks required on your child's state tests. You won't see too many questions that require those kinds of skills.

I have provided a link to the discussion around the problem of how to assess 21st-century skills on the blog. The blogosphere and journalism world are already batting about these ideas, noting that the so-called 21st-century skills depend on knowing the NCLB basics. There's also some concern about how expensive it would be to score tests that measure such skills. While these policy debates heat up, various efforts unfold in the educational community that may point the way toward a rapprochement between the advocates of standards-based education and 21st-century skills-based education. The Partnership for 21st Century Skills has been providing skills standards for K12 subjects.

In my own work, we have taken the approach of helping community college instructors who want to engage their students in problem-based learning activities--the kind designed to teach 21st-century skills--to define the learning outcomes they seek to achieve and develop effective tools to assess student performance. Our team from the Foothill-DeAnza Community College District in Los Altos, CA, and SRI International is starting to share these materials, which were developed under a grant from the National Science Foundation. To review them, go to this link: http://elc.fhda.edu/