PORTAAL: A Classroom Observation Tool Assessing Evidence-Based Teaching Practices for Active Learning in Large Science, Technology, Engineering, and Mathematics Classes

“For example, when asked to define active learning, faculty members in one study offered a range of answers, including simply “using clickers” or “group work” to “emphasizing higher-order skills” (Freeman et al., 2014).”

“In addition to the amount of practice, the quality and distribution of the practice is also important (Ericsson et al., 1993).”

“The resulting ICCs were all > 0.8, and thus all were in the “excellent” range (Supplemental Table 2; Cicchetti 1994).”

“The resulting ICCs were all > 0.8, and thus all were in the “excellent” range (Supplemental Table 2; Cicchetti 1994).”

“One of the most popular definitions of active learning comes from Hake (1998): “those [practices] designed at least in part to promote conceptual understanding through interactive engagement of students in heads-on (always) and hands-on (usually) activities which yield immediate feedback through…”

“To provide students with opportunities to practice their logic development, it is necessary for instructors to formulate questions that require a higher level of thinking (PORTAAL element logic development 1, L1; Morris et al., 1977; Ericsson et al.,1993; Jensen et al., 2014).”

“At the college level, an intervention that had students reflect on a situation in their own life in which hard work helped them improve also increased their achievement in their classes (Aronson et al., 2002).”

“Praise that focuses on effort encourages a mastery goal and can increase student willingness to participate in challenging tasks (R6; Mueller and Dweck, 1998).”

“Furthermore, for some college students, the perception that their instructor holds mastery goals for the class leads to improved performance relative to students who believe the instructor has a performance orientation (Good et al., 2012).”

“…teaching as they do their research, may not recognize that the teaching strategies they use are not as effective as other strategies, or may not recognize that comfort and familiarity often dictate their choices of teaching method (Cole, 1982; Bouwma-Gearhart, 2012; Brownell and Tanner, 2012).”

“In the Bell and Kozlowski (2008) study, students in the treatment group were simply told: “errors are a positive part of the training process” and “you can learn from your mistakes and develop a better understanding of the [topic/activity].””

“Furthermore, the education research literature frequently does not provide sufficient detail for the proper implementation of the educational innovations presented (Borrego et al., 2013).”

“14, Summer 2015 14:ar23, 5 into determining the correct answer and therefore reduces learning outcomes (Kulhavy, 1977).”

“The blue instructor decreased failure rates (instances of a “C−” or lower) on exams by 41% (Eddy and Hogan, 2014) when the instructor changed from traditional lecturing to a more active-learning environment that incorporated greater use of group work in class and weekly reading quizzes.”

“, 2010) and increase overall class attendance and performance (Freeman et al., 2007).”

“Second, a deeper understanding of the material is often gained when students share and explain their answers to other students (i.e., work in small groups; L4; Wood et al., 1995, 1999; Renkl, 2002; Sampson and Clark, 2009; Menekse et al., 2013).”

“…and reward in a large class (students are fairly anonymous and will not get personally called out for not participating, and, even if they do participate, the chance they will be rewarded by being called on is low), which decreases motivation to participate (Kidwell and Bennett, 1993).”

“guarantee improvement in student learning, but it does increase the odds of success over less scientifically based approaches (Fraser et al., 2014).”

“Students also self-report that having time to think independently before discussions allows time to think through the question and come up with their own ideas (Nicol and Boyle, 2003).”

“Some students consider this explanations for their answers (Okada and Simon, 1997).”

“To provide students with opportunities to practice their logic development, it is necessary for instructors to formulate questions that require a higher level of thinking (PORTAAL element logic development 1, L1; Morris et al., 1977; Ericsson et al.,1993; Jensen et al., 2014).”

“…psychology studies in classrooms have demonstrated that the smaller the group a student is in (e.g., pairs vs. the entire class) the more likely he or she is to participate in group work and the higher the quality of individual answers (Chidambaram and Tung, 2005; Aggarwal and O’Brien, 2008).”

“Immediate feedback and correction also improves student performance (Trowbridge and Carson, 1932; Ericsson et al., 1993; Epstein et al., 2002; Renkl, 2002).”

“With these changes, the failure rate decreased dramatically (a 65% reduction; Freeman et al., 2011).”

“PART 1: PORTAAL DEVELOPMENT”

“PART 1: PORTAAL DEVELOPMENT”

“Second, a deeper understanding of the material is often gained when students share and explain their answers to other students (i.e., work in small groups; L4; Wood et al., 1995, 1999; Renkl, 2002; Sampson and Clark, 2009; Menekse et al., 2013).”

“This group work increases performance on isomorphic clicker questions and posttests compared with performance of students who only listened to an instructor explanation (Schworm and Renkl, 2006; Smith et al., 2011).”

“Second, a deeper understanding of the material is often gained when students share and explain their answers to other students (i.e., work in small groups; L4; Wood et al., 1995, 1999; Renkl, 2002; Sampson and Clark, 2009; Menekse et al., 2013).”

“Immediate feedback and correction also improves student performance (Trowbridge and Carson, 1932; Ericsson et al., 1993; Epstein et al., 2002; Renkl, 2002).”

“There are many articles validating the importance of opportunities to practice: student learning is positively correlated with the number of in-class clicker questions asked (Preszler et al., 2007); students who created their own explanations performed better on related exam questions than students who read expert exp…”

“These behaviors can also influence students’ willingness to participate in class (Fritschner, 2000; Goodboy and Myers, 2008).”

“This result is not exclusive to biology, as it has also been seen in engineering (Borrego et al., 2013), physics (Henderson and Dancy, 2009), and extensively at the K–12 level (O’Donnell, 2008).”

“PART 1: PORTAAL DEVELOPMENT”

“For a more nuanced picture of instructor confirmation, instructors could use the survey developed by Ellis (2000).”

“First, there is a lack of clarity as to what active learning is, and this lack of clarity can lead to lack of fidelity of implementation of the teaching technique (O’Donnell, 2008; Turpen and Finkelstein, 2009; Borrego et al., 2013).”

“These results, paired with the other studies, demonstrate that both conditions lead to equal Vol. 14, Summer 2015 14:ar23, 7 (how much students like a subject or domain) and the more typical cognitive learning (Ellis, 2000, 2004; Goodboy and Myers, 2008).”

“This result is not exclusive to biology, as it has also been seen in engineering (Borrego et al., 2013), physics (Henderson and Dancy, 2009), and extensively at the K–12 level (O’Donnell, 2008).”

“PART 1: PORTAAL DEVELOPMENT”

“Instead, we focused on teacher-provided incentives that create attention around the activity (Table 3; deWinstanley and Bjork, 2002).”

“…students with immediate feedback on their logic from peers or the instructor, which helps them correct or develop their ideas (deWinstanley and Bjork, 2002) and sets the tone that the instructor values student logic and critical thinking, not just the right answer (Turpen and Finkelstein, 2010).”

“14:ar23, 4 CBE—Life Sciences Education One of the simplest ways to improve student conversations and increase student focus on sense-making and logic when introducing activities is for instructors to remind students to provide the rationale for their answers (L2; Turpen and Finkelstein, 2010; Knight et al., 2013).”

“PART 1: PORTAAL DEVELOPMENT”

“, 2007), but it has also been shown to reduce the quality of the small-group discussions, as 1) these interactions were dominated by the student in the group with the greatest knowledge, 2) there were fewer exchanges between students, and 3) fewer students participated in the discussions overall (James, 2006; James an…”

“PART 1: PORTAAL DEVELOPMENT”

“, 2007), but it has also been shown to reduce the quality of the small-group discussions, as 1) these interactions were dominated by the student in the group with the greatest knowledge, 2) there were fewer exchanges between students, and 3) fewer students participated in the discussions overall (James, 2006; James an…”

“Awarding course points has been shown to increase student participation (especially the participation of low-performing students; Perez et al., 2010) and increase overall class attendance and performance (Freeman et al.”

“, 2013) and student metacognition (Thomas and McDaniel, 2007).”

“…particularly the attitudes of low-performing students, toward the discussion topic (Len, 2006–2007); 2) improve social cohesion and, through that, feelings of accountability (Hoekstra and Mollborn, 2011); and 3) develop students’ argumentation skills (Kuhn et al., 1997; Eichinger et al., 1991).”

“In an experiment explicitly designed to test how grading scheme impacts learning, Willoughby and Gustafson (2009) found that although students answered more clicker questions correctly when points were earned based on correctness, this grading scheme did not lead to an increase in achievement on a…”

“Group work is particularly important for tasks that require students to transfer knowledge from one context to another (Gadgil and Nokes-Malach, 2012).”

“Additional studies have supported this finding that students learn what they are tested on (Morgan et al., 2007; Wormald et al., 2009).”

“Second, a deeper understanding of the material is often gained when students share and explain their answers to other students (i.e., work in small groups; L4; Wood et al., 1995, 1999; Renkl, 2002; Sampson and Clark, 2009; Menekse et al., 2013).”

“At the institutional level, these barriers include a reward system that can lead faculty members to devote less time and effort to teaching (Lee, 2000) and limited institutional effort to train graduate students or faculty members on teaching methods (Cole, 1982; Weimer, 1990).”

“Second, a deeper understanding of the material is often gained when students share and explain their answers to other students (i.e., work in small groups; L4; Wood et al., 1995, 1999; Renkl, 2002; Sampson and Clark, 2009; Menekse et al., 2013).”

“This complexity, in and of itself, is a second barrier: changes in teaching practice can feel overwhelming because of the number of aspects that must be considered, and this can lead to paralysis and inaction (Kreber and Cranton, 2000).”

“Examples include the Reformed Teaching Observation Protocol (RTOP; Sawada et al., 2002), the Teaching Dimensions Observation Protocol (Hora et al., 2013), the Classroom Observation Protocol for Undergraduate STEM (COPUS; Smith et al., 2013), the Practical Observation Rubric to Assess Active Learning (Eddy et al., 2015…”

“There are many different ways to enact active learning, but typically active learning includes students working with other students during class and more frequent assessment of student learning in the classroom (Eddy et al. 2015b; Freeman et al. 2014).”

“…can reduce bias with regard to who is called on in college science courses; in biology courses in which random call is not practiced and students volunteer to answer questions, only 40% of female voices are heard, even when females make up 60% of the class (Eddy et al., 2014, 2015; Tanner, 2013).”

“Second, collaborations between researchers and instructors could help increase the implementation of evidence-based teaching practices by instructors/faculty who are not education researchers and would benefit from support while shifting their instructional approaches (Eddy et al., 2015).”

“…a growing interest in understanding how undergraduate anxiety may impact motivation, performance, and persistence, particularly for demographic subsets of students (Bledsoe and Baskin, 2014; Eddy et al., 2014, 2015a,b; Broeckelman-Post et al., 2016; Cooper and Brownell, 2016; Cooper et al., 2018).”

“…engagement between two instructors; paired with additional classroom observation data about instructor habits (for example from a tool like PORTAAL [Eddy et al., 2015] or COPUS [Smith et al., 2013]), instructors may be able to enhance their own soft skills to increase engagement in their own…”

“…definition for the BER community; the fact that active learning is a complex process encompassing both teaching methods and student learning (Eddy et al., 2015); the dichotomous nature of the term as both a theory and as a set of pedagogical strategies (Drew and Mackie, 2011); the…”

“…Protocol (TDOP), Reformed Teaching Observation Protocol (RTOP), Classroom Observation Protocol for Undergraduate STEM (COPUS), Practical Observation Rubric To Assess Active Learning (PORTAAL)] (11 – 14) require trained human observers and are not feasible for addressing questions at this scale.”

“A literature review revealed that about half of the instructional practices demonstrated to improve student achievement in active-learning instruction in large undergraduate STEM courses focused on student motivation (Eddy et al. 2015).”

“These studies should include inspection of finer-grain analyses of student intersectionality/identity, as well as a better understanding of the diverse relationships within working groups (Eddy et al., 2015).”

“There are also recent observation protocols that would indicate whether the GTA PD is being taught using current best practices in science teaching (e.g., Smith et al., 2013; Eddy et al., 2015).”

“A study that recently investigated the teaching strategies of the classrooms used in our study documented that instructional practices varied between instructors, ranging from almost entirely traditional lecture to more student-centered interactive methods (Eddy et al., 2015).”

“…to relevant literature to inform our thinking about conceptual relationships among codes, including literature on active-learning instruction (e.g., Eddy et al., 2015), student motivation (e.g., Glynn et al., 2009), cognitive science (e.g., Vermunt, 1996), and course climate (e.g., Ambrose et al.,…”

“In particular, such Instructor Talk sampling would complement information being collected with currently available classroom observation tools (e.g., Reformed Teaching Observation Protocol, Teaching Dimensions Observation Protocol, Classroom Observation Protocol for Undergraduate STEM, Practical Observation Rubric To…”

“…the focus of their pedagogies from preparing students to pass exams to preparing them to be life-long learners who are able to think critically, collaborate, solve problems, and communicate effectively (Conference Board of Mathematical Sciences, 2016; Eddy et al., 2015; Maron, 2016; Pienta, 2015).”

“These results corroborate the improvement in science education through active methodologies already described by many authors in the last few decades [41,42].”

“…using additional approaches, such as analyzing the instructor talk exhibited (Seidel et al., 2015), the 21 specific practices documented in the PORTAAL (Eddy et al., 2015), or the participation patterns in classroom discourse provided by Equity Quantified in Participation (Reinholz and Shah, 2018).”

“In the past decade, the biology education research community has identified a number of evidence-based instructional practices (Freeman et al., 2007, 2011; Handelsman et al., 2007; Bradforth et al., 2015; Eddy et al., 2015; James and Singer, 2016; Jordt et al., 2017).”

“The Practical Observation Rubric to Assess Active Learning (PORTAAL) is used by observers to document the frequency and duration of class activities that employ specific active-learning techniques documented to improve student learning (Eddy et al., 2015).”

“Although there are many different aspects of active learning, important components with demonstrated positive effects on student learning have recently been reviewed (Eddy, Converse, and Wenderoth 2015).”

“The Ignored event could have been mitigated if the instructor had been cautious with praise and not judged student responses (Aronson et al., 2002; Tanner, 2013; Eddy et al., 2015).”

“We adapted our definitions from the biology education literature (e.g., Freeman et al., 2014; Eddy, Brownell, et al., 2015; Eddy et al., 2015).”

“These efforts have led to the development of multiple classroom observation instruments (for a review, see Hora and Ferrare, 2012), such as the Reformed Teaching Observation Protocol (i.e., RTOP; Sawada et al., 2002), Practical Observation Rubric to Assess Active Learning (Eddy et al., 2015), Teaching Dimensions Obser…”

“, 2015) and the Faculty Practice/Faculty Support rubric (D’Avanzo, 2013; Smith et al., 2013; Wieman and Gilbert, 2014; Eddy et al., 2015).”

“First, it is possible that additional or other approaches to active learning would have been more effective than those used in this study; future research should be directed at determining what types of activities are most effective in eliciting active learning (Andrews et al., 2011; Freeman et al., 2014; Eddy et al.,…”

“Several classroom observation tools have been designed to measure student (and instructor) participation and behavior in the classroom (Sawada et al., 2002; Hora and Ferrare, 2010; Smith et al., 2013; Eddy et al., 2015; Lane and Harris, 2015).”

“The changes in course design that occurred over the three experiments (Table 1) are backed by data from a classroom observation tool [11] (S3 Text).”

“, teaching practices inventories) could be used to give faculty the tools to support student buy-in of evidence-based teaching approaches and to be successful in implementing these teaching models (Wieman and Gilbert, 2014; Eddy et al., 2015).”

“random to participate in a multiple-choice test on the course topic provided on the online platform Moodle, to incorporate a typical active learning assessment-component into the online teaching concept (Freeman et al., 2014; Eddy et al., 2015).”

“Other tools could also be useful for investigating the implementation and fidelity of evidence-based practices as a result of co-teaching experiences (e.g., Crowe et al., 2008; Eddy et al., 2015; Tekkumru‐Kisa et al., 2015; Arneson and Offerdahl, 2018; Kranzfelder et al., 2019).”

“…method [26], the Teaching Dimensions Ob-servation Protocol (TDOP) [12], the Practical Observation Rubric To Assess Active Learning (PORTAAL) [7], the Classroom Assessment Scoring System (CLASS) [29], and the Classroom Observation Protocol for Undergraduate STEM (COPUS) [34]; where observers…”

“Eddy et al. (2015) found that students explaining their answers to peers, hearing other students describe their logic, and participating in activities that require group work are all elements of best practices in AL.”

“The possibility of biased PRT has been stressed and can be countered by conducting several independent class observations or by involving multiple reviewers (Eddy et al., 2015).”

“In either case, the large difference in the fractions of more engaged students in these two classes calls for the use of more observation codes when active learning occurs such as the 21 elements used in the PORTAAL classroom observation tool (Eddy et al. 2015).”

“Other disadvantages include lack of collaboration and teamwork (key International Engineering Alliance and Engineering Council of South Africa qualification standard graduate attributes [9,14]) as well as the risk of plagiarism during the assessment, which may misrepresent the benefits of virtual experimentation [8].”

“The approach was purely directed to student self-directed learning (SDL) with an all-inclusive scheme to direct the online activities to student academic improvement (Darmawan & Hidayah, 2017; Eddy et al., 2015).”

“Many factors contribute to students’ success in their coursework, including but not limited to scaffolding of course material, homework assignments, students’ motivation, and family responsibilities (Eddy et al., 2015a).”

“Obviously, the study conducted by Eddy, Converse, and Wenderoth (2015) emphasized that learners’ participation can be taken to a higher level by feeling the sense of responsibility during grasping of their own knowledge and skills.”

“…necessary to combine these resources with those of the students: previous school experience and their own personality (dotted arrows) (modi fi ed from Eddy et al. 2015). for molecular biology students (Laneuville & Sikora 2015), suggesting that this kind of learning activity is not…”

“Eddy, Converse, and Wenderoth (2015) created an observational tool for capturing the presence, absence, and frequency of best practices for active learning, although it is specific to large ALCs (i.e., 50–700 students) in STEM domains.”

“Error framing is proposed in the literature as a pedagogical tool to reduce student apprehension (Eddy et al., 2015) and anxiety (Cooper et al.”

“Although other instruments exist, such as the Classroom Observation Protocol for Undergraduate STEM (COPUS) [43] and the Practice Observation Rubric to Assess Active Learning (PORTAAL), [44] the TPI provides a relatively simple, frequency-based system for characterizing research-based teaching practices with little to…”

“The training guide provided by COPUS and PORTAAL (Smith et al., 2013; Eddy et al., 2015) was also reviewed.”

“While activity within lectures is varied by lecturer, the majority of lecture time is predominantly the lecturer talking (Smith et al., 2013b; Eddy et al., 2015; Kinnear et al., 2021).”

“…the volume and variance of classroom audio recordings to estimate the percentage of time spent in active learning (56); and the Practical Observation Rubric To Assess Active Learning, which measures the minutes students could spend in thinking or discussion as a component of its score (57).”

“This increase in interaction, based on the evidence-based active learning strategy [44,45], was accompanied by the requirement to turn on the students’ cameras to enable a more personal learning atmosphere.”

“For example, Eddy et al. (2015) proposed that active learning typically involves students collaborating with peers in the classroom and engaging in mutual knowledge evaluation.”

“…tools are the Teaching Dimen - sions Observation Protocol (TDOP) [ 6 ], Practical Observa - tion Rubric To Assess Active Learning (PORTAAL) [7], the Classroom Assessment Scoring System (CLASS) [8], and the Classroom Observation Protocol for Undergradu - ate STEM (COPUS) [9] where observers…”

“Other tools, such as the Practical Observation Rubric to Assess Active Learning (PORTAAL) (Eddy et al., 2015) and the Teaching Dimensions Observation Protocol (TDOP) (Hora, 2015; Hora & Ferrare, 2014) were designed to measure the types and prevalence of active learning instructional strategies in…”

“Traditionally, knowing how class time is spent requires bringing in a trained observer to hand-annotate during lecture, for example, using one of the popular classroom annotation schemes (Smith et al. 2013; Sawada et al. 2002; Velasco et al. 2016; Eddy, Converse, and Wenderoth 2015).”

“…(e.g., Dasgupta et al., 2014), surveys (e.g., Glynn et al., 2011; Hanauer and Dolan, 2014), concept inventories (e.g., Anderson et al., 2002; Price et al., 2014; Couch et al., 2015), or observation protocols (e.g., Smith et al., 2013; Eddy et al., 2015), for use by the biology education community.”

“…in specific disciplines for traditional face to face classroom, for example, the Practical Observation Rubric to Assess Active Learning (PORTAAL; Eddy et al., 2015), The Classroom Observation Protocol for Undergraduate STEM (COPUS; Smith et al., 2013), Peer Review of Teaching (PRoT) program for…”

“In recent years, “Active Learning (AL),” the teaching method which puts more emphasis on students’ active participation in class and their abilities to discover problems and solve them, has been coming under the spotlight worldwide [1][2][3][4][5][6].”

“It is probable that students would actively participate in their studies, contribute to debates, and work collaboratively with peers when they feel respected, encouraged, and included (Eddy et al., 2020).”