In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs)

“, instructors teach differently online (Palloff & Pratt, 1999; Salmon, 2000; Wiley, 2002), and this line of inquiry typically places face-to-face instruction inappropriately as the gold standard (Duffy & Kirkley, 2004; McDonald, 2002).”

“, instructors teach differently online (Palloff & Pratt, 1999; Salmon, 2000; Wiley, 2002), and this line of inquiry typically places face-to-face instruction inappropriately as the gold standard (Duffy & Kirkley, 2004; McDonald, 2002).”

“The alluring promise of MOOCs and what keeps people interested in them is their ability to offer free or low-cost education to anyone, anytime, anywhere, and on a massive scale (Delbanco, 2013; Jordan, 2014; Yang et al., 2013).”

“It is important to note that our findings suggest something different than the work of Margaryan et al. (2015) Margaryan et al.”

“Among other things, they question whether meaningful and effective educational experiences are even possible in large enrollment courses (see Glass & Smith, 1979; Lederman & Barwick, 2007; Mulryan-Kyne, 2010), let alone completely online courses with massive enrollments (see Leddy, 2013; Mazoue,…”

“Among other things, they question whether meaningful and effective educational experiences are even possible in large enrollment courses (see Glass & Smith, 1979; Lederman & Barwick, 2007; Mulryan­Kyne, 2010), let alone completely online courses with massive enrollments (see Leddy, 2013; Mazoue, 2013; Rees, 2013).”

“But for every study that suggested no significant difference or that students possibly might learn better online (e.g., Means, Toyama, Murphy, Bakia, & Jones, 2010), another study suggested the opposite (see Jaggars, 2011; Jaggars & Bailey, 2010; Jaggars & Xu, 2010; Xu & Jaggars, 2011).”

“These issues, as they relate to MOOCs, have been fully discussed and studied by numerous researchers (Lowenthal & Hodges, 2015; Rodriguez, 2012; Margaryan, Bianco, & Littlejohn, 2015; Jordan, 2015).”

“The QM Rubric is frequently linked to quality in higher education online course literature (e.g., Legon 2015; Lowenthal and Hodges 2015; Shattuck 2010).”

“Mejia-Madrid et al. (2020) explored how to evaluate quality in online environment and emphasized the significance of teaching method and technology.”

“Lowenthal & Hodges (2015) reviewed six MOOCs applying the Quality Matters rubric intended for traditional for-credit online courses.”

“While the joint bodies recommended' by section 13 would probably have little effective power to act independently, their scope of authority as outlined in section 13 would be similar to that of existing international institutions, notably the International Joint Com mission between the United…”

“In fact, the availability of course information is one of the standards employed by the Quality Matters (QM) consortium to indicate the quality of an online course (Lowenthal and Hodges 2015).”

“Finally, questions of instructional quality have been raised in the literature (Lowenthal & Hodges, 2015; Margaryan, Bianco, & Littlejohn, 2015).”

“A second differential element in open compared to conventional courses is the problem of determining and measuring quality (Hayes 2015; Kocdar and Aydin 2015; Lowenthal and Hodges 2015; Margaryan et al. 2015).”

“riculum to great extent as online courses tend to follow their prescribed structure (Lowenthal and Hodges 2015).”

“Unfortunately, active learning as a quality standard of online course design (Lowenthal and Hodges, 2015) remains largely undefined.”

“However, despite its growing popularity, many researchers have raised concerns regarding the overall quality of self-directed online instruction (Lowenthal and Hodges 2015; Mishra and Kunwar 2015; Wiley and Gurrell 2009), arguing that it is often poorly designed and fails to draw on empirically…”

“Consequently, several scholars have expressed their desire for quality benchmarks in MOOCs (e.g. Lowenthal & Hodges, 2015; Morrison, 2016).”

“The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 26 learners interact in online systems toward learning goals (Lowenthal & Hodges, 2015; Means, Toyama, Murphy, Bakia, & Jones, 2010).”

“For example, a dominant segment of research on instructional quality in MOOCs focuses on the presence or absence of specific design elements or categorizes quality according to existing scales (Bali, 2014; Kasch, van Rosmalen, & Kalz, 2017; Lowenthal & Hodges, 2015).”

“This approach is not as expensive as the previous approach, still, it suffers from the same drawback concerning the users’ perspectives (Lowenthal & Hodges, 2015). iii.”

“The standardized approach utilizes well-established rubrics, such as Quality Matters (QM) or Quality Assurance, as the evaluation criteria (Lowenthal & Hodges, 2015; Parscal & Riemer, 2010); however, the approach often overlooks and dismisses learners ’ experiences.”

“Lowenthal and Hodges (2015) used all eight standards to evaluate quality of massive, open, online courses (MOOCS) from Coursera, EdX and Udacity (three of the main leading massive, open, online course providers).”

“Tools such as navigational maps or frames with indices are elements that can ensure quality and help students understand their progress, where they are, and what tasks remain to be completed [3,5,25].”

“The challenge for IHEs is to design and deliver relevant, engaging educational experiences that meet learner needs given that student attrition in remote learning is higher than traditional settings (Croxton, 2014; Lowenthal & Hodges, 2015).”

“Third, we relied on the Quality Matters (QM) framework (Lowenthal & Hodges, 2015), which focuses on eight standards to evaluate courses.”

“Courses designed on objectives can promote the use of limited designs with video recording and multiple-choice tests Online Learning Journal – Volume 22 Issue 2 – June 2018 173 7 (Lowenthal & Hodges, 2015).”

“On the other hand, there were studies reporting the design aspects of MOOCs based on the evaluation of their instructional and design quality using existing frameworks (e.g., Lowenthal & Hodges, 2015; Margaryan et al., 2015; Oh et al., 2020).”

“Lowenthal and Hodges [12] investigate the quality of MOOCs by using the quality matters quality control framework.”

“Furthermore, some contend that instructional and learning quality are poor in most MOOCs (Margaryan, Bianco, & Littlejohn, 2015), and yet others claim it is feasible that MOOCs meet the standards of quality set for other online courses (Lowenthal & Hodges, 2015).”

“…and those whose main objective is to assess the quality of courses through the use of various tools and rating scales (Aguaded & MedinaSalguero, 2015; Alemán, Sancho-Vinuesa, & Gómez, 2015; Baldomero, Salmerón, & López, 2015; Chapman, Goodman, Jawitz, & Deacon, 2016; Lowenthal & Hodges, 2015).”

“For example, Lowenthal and Hodges (2015) applied Quality Matters (QM), a common quality-assurance framework from the United States. The QM peerreview process is centered around eight general standards, with at least five among them directly linked to instructional quality (e.g., Standard 3: Assessment and measurement,…”

“The research on the various dimensions that potentially influence course quality (for example, see Lowenthal & Hodges, 2015; MacDonald & Thompson, 2005) looks at the quality of a course as a whole but not at quality dimensions for the individual tasks of course production, which was required for…”

“In tandem with its growing importance, standards have been created to keep online course quality high, and models have been developed for how best to motivate online students and support their learning [24, 28, 49, 71].”

“The researcher listed two requirements needed to support student engagement and guide the student to become an active learner, namely: (1) course tools and (2) media [44].”

“There is widespread skepticism of the quality of MOOCs and the learning methodologies used and there is evidence that supports this skeptical view (Margaryan, Bianco, & Littlejohn, 2015; Lowenthal & Hodges, 2015).”

“2013); Very little research has been carried out to study the quality matters of Mooc (Lowenthal and Hodges 2015).”

“MOOCs have also impacted global online learning in the last decade be cause they “offer free or low -cost education to anyone, anytime, anywhere, and on a massive scale” (Lowenthal & Hodges, 2015, p. 84).”

“It is supported by Lowenthal and Hodges (2015), who also indicated the positive role of QM for MOOC evaluation.”

“Rather than post hoc removal of standards which are not relevant to MOOCs (Lowenthal & Hodges, 2015), QM might consider the development of a fit-for-purpose rubric specific for MOOCs which incorporates contemporary research regarding the hallmark features of MOOCs.”

“However, such MOOC initiatives also raise concerns as the quality of learning experience in MOOCs still remains debatable (Lowenthal & Hodges, 2015; Margaryan et al., 2015; Toven-Lindsey et al., 2015).”

“There is also a dearth of research on instructional innovations in designing MOOCs and other forms of open education (Bonk et al., 2015; Brouns et al., 2014; Lowenthal & Hodges, 2015; Margaryan, Bianco, & Littlejohn, 2015).”

“Such concern today is central to academic discussions of online learning effectiveness (Lowenthal & Hodges, 2015).”

“MOOQ was started due to the huge demand for improving the quality of MOOCs from research [7 – 10, 16, 17] and from practice [4, 6, 10, 11].”

“However, since most MOOCs do not meet those points [14] and with additional points from iNACOL standards, the paper does not give CEH v10 an unfair evaluation.”

“Scholars like Lowenthal and Hodges (2015) and Xiao, Qiu and Cheng (2019) examined quality assurance challenges specific to MOOCs, such as scalability, learner engagement, and credential recognition.”

“Much has been discussed about the potential of online computer science (CS) courses to revolutionize education by expanding access to a broad range of students and drawing in a large number of individuals eager to learn [2, 42, 51].”

“Massive Open Online Courses (MOOC) is a new vision of education that aims to increase the quality of learning and the level of knowledge since the mass of knowledge already doubles every two years [1].”

“However, some researchers pointed out that this approach might address too much on the fundamental areas and is inadequate on instructional approaches for active engagement, communication, and collaboration [10].”

“The Quality Matters Program was originally created for online courses, but was used often also for MOOC (Lowenthal & Hodges, 2015).”

“In this context, it could be of interest to implement some system to verify if online courses comply with an established quality rubric, such as those of the Quality Matters framework used in other institutions (Gregory et al., 2020; Lowenthal & Hodges, 2015).”

“Second, pedagogical experts often use structured tools (Zimmerman et al., 2020) such as the Quality Matters Rubric (Gregory et al., 2020; Lowenthal & Hodges, 2015; Woods, 2014), the Rubric for Online Instruction (Debattista, 2018; Lee et al., 2020; Xu et al., 2020) and checklists (Baldwin & Ching,…”

“It combines an intellectually formative experience, responsibility and autonomy in the choice of subjects, as well as individual interests and social competences (Eynon et.al., 2021; Lee et. al., 2016; Lowenthal et. al., 2015).”

“Q uality matters rubric sudah digunakan oleh beberapa institusi untuk memastikan kualitas rancangan MOOC dalam meningkatkan keterlibatan, kemandirian dan penyelesaian MOOC (Gaston & Lynch, 2018; Gregory et al., 2020; Lowenthal & Hodges, 2015; Murillo & Jones, 2020).”

“Cluster #4 (MOOC) focuses on massive open online courses (Onan, 2021), user experience on online course platforms (S. Wang et al., 2021), quality matters about online courses (Lowenthal et al., 2015), quality assurance management, and automated assessment, among others.”

“…have often used existing scales such as Quality Matters and Course Scan and deductively analyzed instructional design features from the perspectives of researchers and instructional designers (Lowenthal & Hodges, 2015; Margaryan et al., 2015; Oh et al., 2020; Wang, 2023; Wang et al., 2021).”

“In a study over six different MOOCs, The authors in [12] noticed that all of them failed to meet the standards concerning the definition of course or units learning objectives.”

“Although it is intuitive to apply standardized quality control models, the diversity of MOOCs and their purposes makes it challenging to establish criteria that apply to all situations [36].”

“Such a broad approach is expected to meet some quality standards (Lowenthal & Hodges, 2015).”

“Some courses lack rigorous instructional design and academic review, resulting in insufficient accuracy and depth of knowledge, which adversely affects learners' learning outcomes [26].”

“However, the quality of MOOCs has been criticized from the start and many recent works focused on the evaluation of the quality of MOOCs (for example [19-21]).”

“MOOCs are not a solution to every educational need and Lowenthal & Hodges (2015) argues that it is an opportunity to rethink how to design and teach online courses rather than either a complete threat or gimmick.”

“Given the role of QM as the dominant course design certification tool in the industry (Lowenthal & Hodges, 2015), and the influence that course design has on learner motivation and outcomes, it is important to understand how QM functions to shape the online learner experience by observing the…”