Competency vs. Credit Hours

Like MOOCs, the call for "competency-based" higher education is an effort to provide more flexibility around seat-time, which has been a mainstay of the nation's systems for adult learning. The notion is, if you know your stuff and you can pass the test quickly, why should you be required to put in a certain amount of hours? The idea is receiving increasing scrutiny and experimentation from both the U.S. Department of Education and foundations. Pioneers in this concept include non-traditional institutions such as Western Governor's University, Excelsior, and Capella, and Southern Hampshire University has become the first traditional higher education institution to decouple seat time ("the credit hour") from degree attainment. Assessment becomes the central indicator of competency, but not all assessments are created equal. Some tests focus on measuring how much terminology and facts someone knows. Some focus on the testing the skill of communicating and explaining connections among ideas. Others focus on applying concepts while solving problems. Some focus on hands-on skill and situational decision making. Many of the more complex skills are not easily or accurately captured by test formats that are easiest to administer and auto score; they require complex judgments based on multiple factors. Few people like to think about tests, but getting the metrics right offers a way out of proxies such as "seat time." The only way to do this well is to think about what few like to think about: assessment.

MOOCs: Less image management, more design principles, please

I begin with the news and then offer a perspective of an educational researcher. First, the news--The MOOC experiment at San Jose State University continues, but one of the pioneers of the experiment is expressing reservations and its would-be faculty guinea pigs are using the machinery of campus politics to slow down the effort.

The SJSU faculty have requested a review of the campus chancellor's leadership after last spring and sunmer's online education experiment showed weak results: http://www.insidehighered.com/news/2013/11/19/facultyi -members-san-jose-state-u-urge-outside-review-institutions-governance

In the meantime, the leader of the SJSU experiment, Sebastian Thrun of the MOOC provider Udacity, recently told a writer for the magazine Fast Company that he is downshifting his MOOC ambitions from the loftier climes of higher education learning to the supposedly much more straightforward and achievable workd of workforce training (????), specifically for computer scientists: http://www.fastcompany.com/3021473/udacity-sebastian-thrun-uphill-climb.

Although Udacity continues its work with SJSU after a very poor success rate (25% passed) compared to face-to-face classes (53% passed) last spring, the (entirely predictable!) lack of evidence of actual course completion and learning in many MOOCs has been (finally!) grabbing media  attention lately:

"And yet, all of these efforts have been hampered by the same basic problem: Very few people seem to complete courses when they're not sitting in a lecture hall. Udacity employs state-of-the-art technology and sophisticated pedagogical strategies to keep their users engaged, peppering students with quizzes and gamifying their education with progress meters and badges. But a recent study found that only 7% of students in this type of class actually make it to the end. (This is even worse than for-profit colleges such as the University of Phoenix, which graduates 17% of its full-time online students, according to the Department of Education.)"

"Among those pupils who took remedial math during the pilot program, just 25% passed. And when the online class was compared with the in-person variety, the numbers were even more discouraging. A student taking college algebra in person was 52% more likely to pass than one taking a Udacity class, making the $150 price tag--roughly one-third the normal in-state tuition--seem like something less than a bargain. The one bright spot: Completion rates shot through the roof; 86% of students made it all the way through the classes, better than eight times Udacity's old rate. (The program is supposed to resume this January; for more on the pilot, see "Mission Impossible.")

I have been reporting these media reports rather dutifully for several months now, and at last, I have an actual opinion and perspective to express. Let's start here: Note that the experience with the SJSU developmental math MOOC was somewhat different from other MOOCs.  The low completion rates that have plagued most online learning for the past 15 years were not replicated. In this case, students finished; they just didn't pass. This is an interesting finding that no one seems to be discussing much. 

One wonders why these students didn't do what most savvy students do when they are gaming the traditional higher Ed system: Drop the class you're clearly not passing to "maintain" your GPA--and the illusion that you are a traditional "good student."  Perhaps they're not savvy students. There is good circumstantial evidence to support this claim. For example, perhaps not being savvy students is why they are in dev Ed. Somewhere along the line, they didn't get the memo about playing the school game. They didn't cram before the placement test to avoid the dev Ed class when they came to the admissions office. But make no mistake--the savvy students do that. Further, we are at SJSU, a tier 2 college. These students likely didn't take lots of SAT prep classes and GPA-goosing AP classes in high school either. After all, that is what the savvy students do. So these students were able "only" to get into SJSU rather than a supposedly "better" institution of higher Ed.

And think about that last statement. What is it with the brand names in higher Ed? I have worked with some very terrific SJSU grads and some pretty lame Stanford grads. Enough on the brand name nonsense! Is anyone else noticing how perverse things have become? What does any of the business of being a savvy student and embroidering your resume with the right schools have to do with actual learning? It really boils down to a ridiculous level of image management. Do we really want, as a society, our youngest, most idealistic citizens focused on image management, which is really what being a savvy student has become? Where does that lead us, as a society? 

Now, that's my opinion. Here is the substantive perspective part as an educational researcher: What is the difference between finishing and passing, and what does this tell us about learning? Could it be that the 75% of folks who did not pass may tell us something about the failures of thinking you can simply put typical course material online (video lectures, slides, readings) and then make magic happen? What needs to change in the design of learning content to permit the other 75% to learn online? As it happens, there is plenty of research into online learning that has clarified that point, and yet no one discusses that research in these various reports.

Back to opinion: Instead, the reporters seek the answer from entrepreneurs with pedigrees from brand name institutions who have not spent not more than a moment studying pedagogy. Hey, why would they? Studying pedagogy--let's face it, studying "education"--isn't good for their "image." Education is not a prestige field. Everybody knows that, right? Why do it then?

Back to perspective: But, for the record, here are some design principles we know... You need social support to learn, mainly to provide what is called learner "self-regulation," which means keeping the learner focused consistently to finish. (In the old days when mostly women taught in schools, we called it "helping." But I digress...) Anyway, you also need multiple representations of knowledge, frequent feedback, and -- get this -- multiple trials to reach mastery. Yes, it turns out that the notion that you learn "fast and efficiently" is simply goofy. You learn slow and painstakingly, through lots of hard work and effort. Yes, the sad truth is that we must be "grinds" to learn. But again, what would saying that do for our "image?" Oh by the way, that reminds me, we in the Ed field are calling hard work "grit" now. 

Could it also be that there is an implicit identification of the central problem of the design of traditional higher Ed learning in these data? Perhaps those 75% non-completers represent the kinds of students who do not learn very well under the old model of "time pressured, image-managed, competitive grade learning" the first time around, but ultimately could learn if permitted to study with some form of support on their own time and less pressure around image management? The MOOC experiment does not answer these questions, but these results raise such questions. 

Instead, in the MOOC coverage, we see the predictable narrow focus on one data point or two (pass rates! completion rates!), and on to the next personality piece about the next entrepreneur with something to sell us. There is very little thought about what the evidence of pass and completion rates tells us about the design of education. Instead, we see the predictable political positioning to protect the status quo without any overt evidence of a good faith effort to try to improve the design of the status quo (e.g., SJSU and UCSC faculty "protests"). We see the sort of half-baked questioning about the "social justice" of such proposed innovations without really looking at the details of what is actually happening in the cases of the oft-quoted stereotypes of the "underrepresented, needy student." So, we use these stereotypes as a prop to block change and we learn nothing about how the design of learning could be improved to actually help them (and, hey, if we are honest, everyone else too)-- (for the PC perspective, see this: http://www.hackeducation.com/2013/11/14/thrun-as-saint/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+HackEducation+%28Hack+Education%29 ). 

None of this is constructive or focused on solving the education problem in this country. That takes real work and a laser focus on the problem and designing a solution -- and not a focus on the personalities, political strategy, or political correctness. 

Be advised, that kind of focus on the design of learning IS happening, but coverage of that work is in short supply in the media of all forms and all channels. 

Why not? Because it isn't a sexy story, and that isn't good for the image of the reporter or the media outlet publishing it. 

Study Shows Effort Not MOOC Tech Spells Success

Student effort, not whiz bang technology, made the difference for passing mathematics courses designed by online provider Udacity through San Jose State University last spring, according to a report released by the university. The research, funded by the National Science Foundation and conducted by a team from the state's community college system and San Jose State, described problems collecting data from any but the top students and from Udacity, which countered these claims. See the full story here and the original report appears here.

Pushing college math remediation into high school

An experiment to use online self-directed learning software in high school has shown impressive results in increasing the numbers of students ready for college in Tennessee. The program that brings community college instructors into high school computer labs to assist students working on Pearson's MyMathLab program has yielded pass rates as high as 83% among students who are at high risk of failing college math placement tests. In addition, 25% of the students have gone on to complete college credit mathematics courses while still in high school. As many as 60% of entering community college students test into remedial math, and only about half of those ultimately pass those non credit-bearing courses. The high cost of remedial math and low success rate at the college level has the states of Florida and Connecticut passing laws to remove college requirements forcing students to take such non credit remedial courses.

Community Colleges offer own measures of success

In response to federal survey metrics that they say favor the mission of 4-year universities, an association of U.S. community colleges has offered its own approach to reporting how part-time and transfer students do on their campuses. Development began two years ago and 40 colleges served in the pilot group. See a sample here.

Blog bill pullback raises questions about on-time graduation options for CA students

A California bill to permit MOOC companies to provide public postsecondary students an alternate way to get required course credits for on-time graduation has been postponed for a year. Faced with opposition from faculty and the desire of university systems to create their own online offerings, Senate Pro Tem Darrell Steinberg has delayed his push for SB520. In his May budget, Gov. Jerry Brown proposed $37 million  for higher ed technology to support the university-driven approach. Several efforts will be developed, each crafted within the three distinct tiers of the state college system, but it is unclear how these are being studied or evaluated, how transfer articulation will be addressed with these online courses from three different systems, how that information will be shared with the public, or how these efforts will ensure more timely educational offerings for California public university students and their families. A report by the 20 Million Minds Foundation has found that California's 145 public colleges and universities have failed to offer timely courses to their students and have offered only a patchwork set of "homemade" online courses at different campuses that fail to transfer across campuses. 

Udacity effort at San Jose State University on "pause"

San Jose State University's has announced its plan to place on hold its collaboration with Udacity, an online MOOC company, after spring semester testing showed students using the online program performing more poorly than peers in traditional classes. The experiment, announced in January by Gov. Jerry Brown and encouraged by the Bill & Melinda Gates Foundation, showed that only 51% students in Udacity's developmental mathematics courses passed compared to 74% of those in the regular courses. These results were announced by the university's vice provost at a meeting of fellow California State University provosts last month and shared with the Inside Higher Education news publication by the California Faculty Association, whose members have been critiquing the university's plans to adopt MOOCs as a way to cut costs rather than improve educational quality. By contrast, the university's EdX MOOC experiment, designed as a course supplement rather than a course replacement, is going relatively better. University leaders are attributing some of the Udacity course problems to its rushed creation last winter and selection of particularly high risk students. They plan to absorb the lessons learned and continue design work with Udacity in the spring of 2014.

Old fashioned shoe leather and MOOCs

Ry Rivard, a reporter for Inside Higher Ed, filed a solid piece of old fashioned journalism today on how MOOCs are securing service agreements in 21 universities in 16 states without going through the usual bidding process intended to keep costs down to taxpayers. The novelty and uncertainty of how to make money from these MOOC services appear to have contributed to the trend. Universities typically engage multiple bidders for similar services, such as learning management systems. In the wake of the MOOC hype over the past year, traditional LMS businesses have begun offering their own MOOC support services too. The universities' diversion from standard procurement practices reflects the similar diversion from standard internal review and privacy waivers associated with the MOOC trend. Universities have permitted MOOCs to have access to student learning data without securing waivers to FERPA regulations or going through an internal review board. They do this by designating MOOCs as "service providers." By contrast, traditional researchers measuring the effectiveness of educational interventions have to devote substantial time and research resources to securing such permissions, and often must grapple with highly incomplete data as a result.

Calling Bruce Bochy and Jim Harbaugh: City College needs you

To anyone who has spent even a few moments at a board meeting of the City College of San Francisco, the news of its possible shutdown comes as no surprise. The dysfunction is painfully apparent. If CCSF were a play, it would feel as though all the players, from faculty to college president to board members, were elbowing each other like a bunch of third-rate drama queens on a stage, each one striving to play Camille in her famously wordy death scene.

It is perhaps in disgust with this spectacle that last week the college's accreditor gave the motley troupe of CCSF a year to shut down its 11 campuses or somehow come together to file a cogent appeal. The letter cited the college's failure to address 11 out of 14 accreditation requirements. These include relatively mundane bureaucratic requests, such running a healthy budget and showing how well students are learning. Reading between the lines, one can almost hear the accreditor saying: "Enough Camille! More Baron de Varville!" (The baron was the social climber's dull, cuckholded husband who underwrote her colorful Parisian lifestyle.)

For models of what could be done, we turn from the theater to sports, which is one institution that actually works in San Francisco. Perhaps the CCSF group should borrow some tips from the head coach of the 49ers or the talented manager of the currently slumping, but ever classy and team-focused, Giants. Instead, reports indicate that the CCSF troupe plans another--get ready of it-- "mass campus protest." They cannot get enough drama, apparently. As Camille said: "I always look well when I'm near death."

Labor Department's Top Workforce Training Exec Heads to University of Phoenix

After running the federal government's largest community college workforce training program during the Obama administration, Jane Oates will now be doing similar outreach to industry for the nation's largest for-profit university. The University of Phoenix announced Oates' appointment this week. This switch says a lot about how fluid the workforce training field is today, when a top political official crosses the pitched battle lines from public workforce training programs to for-profit. A key priority in Oates' work will be to continue to identify employers' needs and ensure Phoenix can demonstrate its graduates meet job requirements. Assessment is a key lynchpin to this kind of effort--and finding ways to make the work products of school more transparent and aligned with those of the workplace.

For those not up to speed on beltway politics, Oates resigned earlier this spring from the Department of Labor's Education and Training Administration, accepting blame for budgetary overruns in the Job Corps program that runs 125 training centers nationwide. ETA has played a key role in disbursing $1 billion in funds to community colleges to accelerate retraining for displaced workers through the Trade Adjustment Assistance Community College and Career Training (TAACCCT). The budgetary problems led the department to turn away trainees beginning in late January of this year. Oates estimates roughly 10,000 trainees were turned away and about 700 people lost their jobs at the training centers. The training program was to resume in April.

Assessment for the humanities, MOOCs, and students

With all the attention that the sciences have received in recent years, now humanities educators say they need to defend the relevance of their field. A new report outlines their argument, which underscores the importance of preparing for a "career," not just a job, and being a good communicator across multiple cultures and fields. Accurate measurement of learning and clear, actionable feedback have ever been a problem in these fields, as anyone who has fled them can attest. Perhaps the reformers could start there. How many students have felt their humanities courses required the skill of "reading the professor's mind"? This perception stems from the lack of transparent or clearly defined learning objectives and grading standards, and these are practices every educator should cultivate.

In the meantime, more colleges are claiming they can "roll their own" MOOCs, from the Midwest to Australia. The Australians seem to be capturing the promise of offering low-cost alternatives--courses with assessments. The Americans continue to talk a lot about protecting faculty control and IP, the ever vexing problem of assessment, and student data privacy. What is interesting to me, as a researcher, is how tough it is for educational researchers to obtain access to student data, particularly learning data. We routinely must jump through IRB and FERPA hoops to show evidence of effectiveness, often ending with scant and flawed information. But in the MOOC era, the same universities that run external researchers in circles seem to be granting MOOC partners free access to student data by dubbing them "institutional partners," and the like. Student data is the core of the assessment problem, so the conversation needs to advance there--and students, not just administrators and faculty, should be involved.

MOOCs and faculty IP entitlements

College faculty should seek to preserve their intellectual property claims for courses they create as MOOCs, according to former leader of a national college faculty association, who plans a book on the topic later this year. The issue arose some months ago when the academic senate at the University of California, Santa Cruz, questioned a MOOC agreement that awarded the university rights to online courses its professors created for provider, Coursera. Lots of righteous anger here, but what is really going on? And who might get stuck with the bill?

Apparently, some universities' faculty unions had won collective bargaining agreements in the past to give faculty, not their university employer, IP rights to the courses they created. But recently the game has changed as MOOC providers like Coursera set up course distribution agreements with those same universities. Coursera provides broad access to the professors' courses only when they voluntarily sign away those rights, a move unions perceive as undermining the agreements they have with their university bosses. For those scratching their heads, these agreements appear to have grown out of the scientific places on the campuses, where inventions created during research can lead to real money.

We'll see how the course story unfolds, but from what I can tell, the IP story appears a little different for all the academics who write courses. A course isn't so much an invention as a kind of written product, and like most written products, it depends on a distribution channel to give it life. Here's the rub: The owner of the distribution channel has rights too, and this arrangement has long been understood in the mass media realm. For example, journalists routinely sign away their creative rights to have their writing distributed on mass media venues. Researchers do the same when their work is published in peer-reviewed journals. Further, when that research or journalistic report is completed on company time, the researcher or journalist forfeits additional proceeds back to the employer because the employer already paid for its creation. Based on these analogous situations, under what terms should professors expect any proceeds from the courses they create on university time? How is having your course distributed on a MOOC mass media channel any different from having your work distributed via a newspaper or journal or the television airwaves? 

Read more: http://www.insidehighered.com/news/2013/03/19/u-california-faculty-union-says-moocs-undermine-professors-intellectual-property#ixzz2W6ZULlUt 

Inside Higher Ed 

California Dreams for MOOC Credits Getting Trimmed

In response to faculty protests during a hearing this spring, California legislators are trimming the ambitions of MOOC providers to offer academic credit at all three levels of the state's higher education system. The latest amendments to Senate President Pro Tem Darrell Steinberg's SB 520 transform the originally envisioned universal online platform to an "incentive grant program." The change shifts downward the number of eligible courses for MOOC credits beginning fall 2014 from the "50 most impacted" to a set of "20 high-demand" lower division courses. The change also moves away from framing the MOOC system as a "one stop" platform for California students to a series of grant-supported efforts led by faculty and leaders within each of the three educational segments. The legislation now calls for the grants to foster collaboration among the three higher education systems--the University of California, the California State University, and the California Community Colleges--and to offer the courses to high school students. In an April hearing, faculty leaders expressed concern about "unproven" private companies being put in charge of students' education.

More studies questioning the U.S. "STEM crisis"

A new survey of U.S. workers shows that fewer than a quarter use advanced mathematics in their work, and most of those using such math hold blue-collar technician jobs--not white collar positions. As reported in The Atlantic, the study surveyed 2,300 workers in a range of jobs over two phases from 2004 through 2009. The findings suggest that high-tech blue collar context could be a useful one for high school mathematics teachers. A couple other items of interest: A study around early childhood that indicates gesticulation drives home abstract concepts in mathematics instruction. And the Community College Research Center at Columbia University concludes from an interview study of 46 Virginia community college students that they prefer to take only "easy" courses online rather than "hard" courses.

Skepticism about supposed shortage of STEM workers

A new study finds that even if U.S. students on average are not as stellar in mathematics and science as their peers in some other developed nations, the nation still produces enough top-tier performers to fill a third of the so-called STEM (science, technology, engineering, and mathematics) workforce jobs available in the world today. This finding, from a report by a liberal-learning economic policy institution, counters the view that the nation is potentially facing a shortage of STEM workers because of educational system problems and needs to hire better educated temporary workers from the outside. Other findings that the legion of educational researchers who have been studying science and mathematics learning for the past 20 years may find surprising: The nation's post-secondary system is actually overproducing STEM majors relative to the jobs available. Apparently majoring in science or mathematics is not the guarantee to job security that we've been told. The report indicates that only half the U.S. STEM graduates actually secure STEM jobs, largely because there is not sufficient demand for STEM workers, not because these graduates lack knowledge or skills. Further, a surprising proportion of those working in STEM fields have landed those jobs without a college degree.

Useful Higher Ed Assessment Website

The National Institute for Learning Outcomes Assessment (NILOA), based at the University of Illinois, Urbana-Champaign, publishes briefs and articles about the challenges of learning assessment in higher education. Peter Ewell reports on progress with the Degree Qualifications Profile, an effort supported by the Lumina Foundation.

Technologies for Assessing Deeper Learning in Higher Education

More and more college faculty are grappling with the challenges of crossing disciplinary boundaries to ensure their students are learning critical thinking skills. Both accreditation agencies and employers increasingly seek evidence that such skills are learned in college. Faculty commonly focus on teaching the rules of reasoning in their own disciplines, but branching out to teach reasoning across the disciplines is something new. I gave a presentation today at the faculty professional development day at Notre Dame de Namur University, a small, Catholic 4-year college in Belmont, CA. The talk presents an overview of new educational technologies in higher education and some links to online tools that faculty can use. In addition, I presented a perspective from educational and cognitive psychology on the concept of "critical thinking." The practical implication of this perspective is that unless faculty explicitly help students make connections across disciplines through assignments and assessments--most students will not make such connections. Why? Cognitive psychology tells us based on empirical evidence that people learn to reason within domains. As a result, what constitutes good reasoning in one domain does not precisely constitute good reasoning in another domain. The underlying logic does not easily "transfer." The sidebar conversations with faculty generated some interesting ideas. Maybe simply preparing students to know that whenever they enter a new domain, they need to appreciate that the "rules of the reasoning game" change constitutes a kind of critical thinking skill. If faculty in different disciplines can work together to articulate the "rules of the game" within their respective disciplines, and then discuss how to share the essentials across disciplines, they may move into a more coherent and effective approach to demonstrating results in improved critical thinking. One professor discussed how he teaches modus ponens to his students, and then has them apply this logical rule to different contexts. While such a logical reasoning framework has broad application and relevance, it does not necessarily help students apply, say, the rules of foreign language conjugation, the rules of biological metabolic processes, or the rules of supply and demand in economics. These latter reasoning systems are forms of "model-based reasoning" that have a non-intuitive quality. They constitute a domain-specific form of reasoning that is essential to be a critical thinker in those domains and essential for quality reasoning for a wide range of problems in the world. I look forward to hearing more about how the NDN faculty address their challenges around critical thinking. They may soon be exploring the idea of identifying the various essential forms of "model-based reasoning" per domain and using these as a basis for providing evidence of critical thinking across the disciplines.

Higher ed in-depth interviews

Radio Higher Ed  is a kind of CSPAN for higher ed geeks, particularly those interested in accreditation, assessment, and student learning. Good source of background about this layer of the higher education system, and an opportunity to gain insights from those directly working at this level.