Questions for Assessing Higher-Order Cognitive Skills: It's Not Just Bloom’s
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
We present an exploratory study of biologists’ ideas about higher-order cognition questions. We documented the conversations of biologists who were writing and reviewing a set of higher-order cognition questions. Using a qualitative approach, we identified the themes of these conversations. Biologists in our study used Bloom's Taxonomy to logically analyze questions. However, biologists were also concerned with question difficulty, the length of time required for students to address questions, and students’ experience with questions. Finally, some biologists demonstrated an assumption that questions should have one correct answer, not multiple reasonable solutions; this assumption undermined their comfort with some higher-order cognition questions. We generated a framework for further research that provides an interpretation of participants’ ideas about higher-order questions and a model of the relationships among these ideas. Two hypotheses emerge from this framework. First, we propose that biologists look for ways to measure difficulty when writing higher-order questions. Second, we propose that biologists’ assumptions about the role of questions in student learning strongly influence the types of higher-order questions they write.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Why Some Material Is Difficult to Learn
Sweller, J., & Chandler, P. (1994) · Cognition and Instruction · doi:10.1207/s1532690xci1203_1 · Cited by 1,455
Qualitative Analysis on Stage: Making the Research Process More Public
Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002) · Educational Researcher · doi:10.3102/0013189x031007028 · Cited by 1,362 · cited 1×
“Rigor in qualitative research has been defined as the attempt to make “data and explanatory schemes as public and replicable as possible” (Denzen, 1978, p. 7, quoted in Anfara et al., 2002).”
Henderson, C., Beach, A., & Finkelstein, N. (2011) · Journal of Research in Science Teaching · doi:10.1002/tea.20439 · Cited by 1,055 · cited 2×
“Indeed, recent research on change in STEM undergraduate education shows that changing the practices of STEM educators requires not only dissemination of curricula and techniques (like Bloom’s) but also attention to educators’ beliefs about teaching and learning (Henderson, 2005; Henderson et al., 2011).”
BackgroundHandelsman, J., Ebert-May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., Gentile, J., Lauffer, S., Stewart, J., Tilghman, S. M., et al. (2004) · Science · doi:10.1126/science.1096022 · Cited by 928
Why are some problems hard? Evidence from Tower of Hanoi
Kotovsky, K., Hayes, J. R., & Simon, H. A. (1985) · Cognitive Psychology · doi:10.1016/0010-0285(85)90009-x · Cited by 722 · cited 1×
“In educational psychology, a standard measure for the difficulty of problems is the amount of time needed to complete the problem (e.g., Kotovsky et al., 1985; Sweller and Chandler, 1994).”
BackgroundBiology in Bloom: Implementing Bloom's Taxonomy to Enhance Student Learning in Biology
Crowe, A., Dirks, C., & Wenderoth, M. P. (2008) · CBE—Life Sciences Education · doi:10.1187/cbe.08-05-0024 · Cited by 629 · cited 6×
“These are similar to the evaluation questions described in the BBT (Crowe et al., 2008).”
BackgroundMethodResultInfluentialPrescribed Active Learning Increases Performance in Introductory Biology
Freeman, S., O'Connor, E., Parks, J. W., Cunningham, M., Hurley, D., Haak, D., Dirks, C., & Wenderoth, M. P. (2007) · CBE—Life Sciences Education · doi:10.1187/cbe.06-09-0194 · Cited by 455
What We Say Is Not What We Do: Effective Evaluation of Faculty Professional Development Programs
Ebert-May, D., Derting, T. L., Hodder, J., Momsen, J. L., Long, T. M., & Jardeleza, S. E. (2011) · BioScience · doi:10.1525/bio.2011.61.7.9 · Cited by 420 · cited 1×
“Similarly, science, technology, engineering, and mathematics (STEM) courses and assignments often fail to promote the development of HOCS, as several studies have shown (Reynolds and Moskovitz, 2008; Momsen et al., 2010; Ebert-May et al., 2011).”
BackgroundIncreased Course Structure Improves Performance in Introductory Biology
Freeman, S., Haak, D., & Wenderoth, M. P. (2011) · CBE—Life Sciences Education · doi:10.1187/cbe.10-08-0105 · Cited by 306 · cited 2×
“…students practice science is the idea behind many national calls for reform in undergraduate science classrooms (AAAS, 1989, 2011; NRC, 2003; Handelsman et al., 2004), and many published reports show how this can be done (e.g., Bogucka and Wood, 2009; Freeman et al., 2011; Hoskins et al., 2011).”
BackgroundMultiple-Choice Exams: An Obstacle for Higher-Level Thinking in Introductory Science Classes
Stanger-Hall, K. F. (2012) · CBE—Life Sciences Education · doi:10.1187/cbe.11-11-0100 · Cited by 243
Hoskins, S. G., Lopatto, D., & Stevens, L. M. (2011) · CBE—Life Sciences Education · doi:10.1187/cbe.11-03-0027 · Cited by 207
Freeman, S., & Parks, J. W. (2010) · CBE—Life Sciences Education · doi:10.1187/cbe.10-03-0017 · Cited by 44 · cited 1×
“Some research has shown that students do not perform as well on exams with higher Bloom’s levels (Knecht, 2001; Freeman and Parks, 2010).”
BackgroundBush, S. D., Pelaez, N. J., Rudd, J. A., Stevens, M. T., Tanner, K. D., & Williams, K. S. (2011) · CBE—Life Sciences Education · doi:10.1187/cbe.10-08-0106 · Cited by 42 · cited 1×
“Recently, educational researchers in biology have used grounded theory analysis to study science faculty with education specialties (Bush et al., 2011) and biology undergraduates’ misconceptions about genetic drift (Andrews et al., 2012).”
Background
Cited by
Smith, M. K., Jones, F. H. M., Gilbert, S. L., & Wieman, C. E. (2013) · CBE—Life Sciences Education · doi:10.1187/cbe.13-08-0154 · Cited by 571
Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol
Laverty, J. T., Underwood, S. M., Matz, R. L., Posey, L. A., Carmel, J. H., Caballero, M. D., Fata-Hartley, C. L., Ebert-May, D., Jardeleza, S. E., & Cooper, M. M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0162333 · Cited by 262
Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187
Ghosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67
Probing Internal Assumptions of the Revised Bloom’s Taxonomy
Larsen, T. M., Endo, B. H., Yee, A. T., Do, T., & Lo, S. M. (2022) · CBE—Life Sciences Education · doi:10.1187/cbe.20-08-0170 · Cited by 54
Metadata
- Authors
- Lemons, P. P., & Lemons, J. D.
- Venue
- CBE—Life Sciences Education
- Published
- 2013-03
- Volume
- 12
- Issue
- 1
- Pages
- 47-58
- ISSN
- 1931-7913 (ISSN-L)
- OpenAlex
- W2138662581
- Cited by
- 95
- References
- 14
- FWCI
- 27.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.