Dear Commons Community,

The Chronicle of Higher Education is reporting on an evaluation of seventeen MOOCs developed by edX and offered at Harvard and MIT.  The results (see Executive Summary below) are similar to other preliminary reports on MOOCs.   The Chronicle summarized the results as follows:

“The papers released on Tuesday draw on data from 17 MOOCs offered by Harvard and MIT in 2012 and 2013. A number of academics have begun studying aspects of the MOOC phenomenon, but few academic papers have been published so far.

The first of the working papers, which was written jointly by researchers at both universities, provides an overview of the data from those 17 MOOCs. Some findings:

• 841,687 people registered for the 17 MOOCs from Harvard and MIT.
• 5 percent of all registrants earned a certificate of completion.
• 35 percent never viewed any of the course materials.
• 54 percent of those who “explored” at least half of the course content earned a certificate of completion.
• 66 percent of all registrants already held a bachelor’s degree or higher.
• 74 percent of those who earned a certificate of completion held a bachelor’s degree or higher.
• 29 percent of all registrants were female.
• 3 percent of all registrants were from underdeveloped countries.

Some of these findings reinforce what others have already observed about MOOCs: Few of those who sign up for a course end up completing it. Most MOOC students already hold traditional degrees. Students who sign up for MOOCs are overwhelmingly male.”

In sum, these courses seem more attuned for adult and continuing education and are appealing to students who are looking to develop skills rather than enroll in a traditional college education.  I am sure we will see more MOOC evaluations in the coming year.

Tony

Executive Summary

In the year from the fall of 2012 to the summer of 2013, the first 17 HarvardX and MITx courses launched on the edX platform. In that year, 43,196 registrants earned certificates of completion. Another 35,937 registrants explored half or more of course content without certification. An additional 469,702 registrants viewed less than half of the content. And 292,852 registrants never engaged with the online content. In total, there were 841,687 registrations from 597,692 unique users across the first year of HarvardX and MITx courses. (See Table 2.)

The most typical course registrant is a male with a bachelor’s degree who is 26 or older; however, this profile describes fewer than one in three registrants (222,847, 31%). A total of 213,672 (29%) registrants report their gender as female; 234,463 (33%) report a high school education or lower; 45,884 (6.3%) report that they are 50 or older; and 20,745 (2.7%) have IP or mailing addresses from countries on the United Nations list of Least Developed Countries. Small percentages are not small numbers. The diversity of registrants resists singular profiles; registrants are notable for their differences. (See Table 3 and Table 4.)

Course certification rates are misleading and counterproductive indicators of the impact and potential of open online courses. (See Figure 1.)

o Large numbers of non-certified registrants access substantial amounts of course content.

o Open online registration is not equivalent to enrollment in conventional courses, where traditional enrollment generally entails monetary costs, opportunity costs, and accountability.

o Certification rates can be useful indicators when enrollments are limited. For a fixed number of registrants, higher certification rates accurately reflect larger numbers of certified registrants. For open online courses that support large-scale enrollment, there is no forced tradeoff between numbers of certified and non-certified registrants—both numbers can increase freely by design. In these circumstances, focusing on certification rates alone penalizes desirable activities like browsing and exploring courses, which open online courses are generally designed to support.

o Pressure to increase certification rates may decrease the impact of open online courses, by encouraging instructors and administrators to suppress or restrict registration, lower certification standards, deemphasize recruitment of target subpopulations, or disregard interventions that may disproportionately increase numbers of non-certified registrants over certified registrants.

There are considerable differences in average demographics across courses, in terms of gender (13%-49% female), college degree attainment (54%-85%), median age (23-30), and percentage from the US (16%-36%). These differences are best appreciated in the context of the diversity of course offerings, the intentions of the instructor teams, and the outreach and dissemination efforts of course teams. In spite of average differences, all large-scale courses had hundreds of registrants with only high school degrees or who are under 15, and also had hundreds of registrants with postdoctoral degrees or who are over 50. (See Figures 3-8, and Tables 2-6.)

Unlike conventional courses, open online enrollment occurs continuously throughout courses, with enrollment rates rising as course launch dates approach and then declining more quickly after launch dates pass. Exploration and certification is more likely among registrants who enroll near the launch dates, but viewing likelihood is stable through the run of the courses. Course exploration and certification may benefit from synchronous course schedules and the cohorts that they build. Managing asynchronicity to maintain registrant involvement regardless of enrollment date is an ongoing challenge for instructors and a fertile area for future research. (See Table 5, Figure 8, and Figure 9.)

New metrics, far beyond grades and course certification, are necessary to capture the diverse usage patterns in the data. A simple comparison of grades and viewed content shows thousands of users who fit a range of profiles. Of particular interest may be those students who accessed substantial course content but did not participate in assessments. Metrics include course chapters accessed, forum usage, total numbers of “clicks,” and numbers of active days in the course. (See Table 6 and Figure 13.)

The average percentage of registrants who cease activity in these open online courses is highest in the first week at around 50%. The average percentage of registrants who cease activity in the second week declines sharply to 16% for registrants who persist to that point, and these percentages continue to decline over subsequent weeks. This indicates that registrants who are active after the first week have a relatively high chance of visiting again in subsequent weeks. (See Figure 12.)

Over four thousand registrants earned more than one certificate across HarvardX and MITx, including 1,912 who earned at least one certificate from both institutions. A total of 76 registrants earned 5 or more certificates from the first 17 courses.