MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback

May combine two claims.

Generating questions on specific concepts requires precise prompts, and students may not know how to formulate them.

“Secondly, generating questions on specific concepts requires precise prompts (Limit2), which students might not know how to formulate (Sahoo et al., 2024).”

• cites Sahoo et al., 2024 sourcePartly supported

  The Few-Shot Prompting section (page unavailable) says careful prompt engineering is important for optimal performance, and §2.5 describes crafting effective prompts as laborious even for expert annotators. These passages support the general difficulty of formulating effective prompts, but they do not establish that generating questions on specific concepts requires precise prompts or that students may not know how to formulate them. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

GPT-4 and similar LLMs have trouble generating professional multiple-choice questions because of outdated training data, hallucination, and prompt sensitivity, which leads to poor quality and difficulty.

“However, current large language models (LLMs) like GPT-4 struggle with professional MCQG due to outdated knowledge, hallucination issues, and prompt sensitivity, resulting in unsatisfactory quality and difficulty.”

• Also stated in: § Introduction · plain

  “Firstly, their performance is constrained by their training data, leading to two major issues: outdated knowledge (Mousavi et al., 2024) and hallucination (Zhang et al., 2023) (Limit1).”

• cites Mousavi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Sahoo et al., 2024 sourcePartly supported

  The Retrieval Augmented Generation (RAG) section says limited, static training data can hinder accurate LLM responses, and the Chain-of-Knowledge (CoK) Prompting section describes factual hallucinations and unreliable outputs. These support general concerns about LLMs, but the passages do not establish that GPT-4 struggles specifically with professional multiple-choice question generation, that prompt sensitivity causes this problem, or that the resulting questions have unsatisfactory quality and difficulty. No page numbers are available in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Natural language or correction feedback carries more information than preference or fine-grained feedback, making it especially useful for self-correcting language models.

“Unlike preference and fine-grained feedback, which provide scalar values as training signals, natural language or correction feedback offers richer information (Scheurer et al., 2022; Ma et al., 2023; Yao et al., 2023b; Mishra et al., 2024), making it particularly effective for self-correcting language models (Welleck et al., 2022; Pan et al., 2023).”

• cites Scheurer et al., 2022 sourcePartly supported

  The Abstract and Introduction (p. 1) say natural-language feedback conveys more information per evaluation than pairwise preference comparisons. They do not make the same comparison with fine-grained feedback or characterize it as a scalar training signal. The self-correction claim is attached to different citations in the citing sentence. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Welleck et al., 2022 sourceSupported
• cites Pan et al., 2023 sourceSupported

May combine two claims.

Although LLMs can rapidly produce MCQs, the resulting questions often fall short in quality and difficulty.

“Overall, while LLMs can quickly generate MCQs, their quality and difficulty often do not meet expectations.”

• cites Cheung et al. (2023) sourcePartly supported

  The Question construction section reports that ChatGPT produced 50 MCQs in 20 minutes 25 seconds, versus 211 minutes 33 seconds for human examiners, supporting the speed claim. But the Assessment of question quality section calls the scores satisfactory and finds no significant AI–human difference in overall quality or the exam-suitability domain, which assessed level of challenge. The claim that quality and difficulty often fall short is not supported by these results. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites Klang et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Agarwal et al. (2023) sourcePartly supported

  The Discussion reports that ChatGPT produced the least difficult MCQs, none of the three models produced many questions requiring high-level reasoning, and Bard and Bing produced questions with quality shortcomings. The study examined MCQs in medical physiology; it did not measure how quickly the models generated them, so the speed claim is unsupported. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Feedback-based learning helps LLMs align with intended outcomes and improves their instruction-following across various feedback types.

“Learning from feedback helps align LLMs with desired outcomes, enhancing their ability to follow instructions through different forms of feedback, such as human preference feedback (Ouyang et al., 2022), AI-generated preference feedback (Lee et al., 2023; Dubois et al., 2024), or fine-grained feedback (Wu et al., 2024; Lightman et al., 2023).”

• cites Ouyang et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lee et al., 2023 sourceSupported
• cites Dubois et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In medicine, LLM judges have shown some promise for diagnostic support, documentation, clinical conversation, medical question answering, medical reasoning, and patient education.

“In the medical field (Li et al., 2024c; Raju et al., 2024), LLMs-as-judges have demonstrated certain potential in areas such as diagnostic support (Yao et al., 2024), medical documentation (Brake and Schaaf, 2024,?), clinical conversation (Wang et al., 2023a; Li et al., 2024a), medical question answering (Wang et al., 2024b; Krolik et al., 2024), medical reasoning (Jeong et al., 2024), and patient”

• cites Li et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Raju et al., 2024 sourceSupported
• cites Yao et al., 2024 sourceCould not verify

  The Method section uses GPT-3.5-turbo as an examiner of medical definitions for patient education, not as a judge for diagnostic support. A References entry names a separate study on imaging diagnostic support, but the searched passages do not report the claimed use of LLM judges in diagnosis. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Self-correction typically entails producing a critique of the initial response's weaknesses and then revising it accordingly; this iterative approach has shown potential to improve LLM output quality.

“This self-correction process generally involves generating a critique that identifies shortcomings in the initial response, followed by revising it based on the self-critique—an iterative process that has shown promise in enhancing LLM output quality (Pan et al., 2023).”

• cites Pan et al., 2023 sourceSupported

May combine two claims.

The proposed framework markedly improves the quality and difficulty of generated questions and shows close agreement with expert assessments.

“The framework significantly enhances question quality and difficulty while aligning closely with expert evaluations.”

• Also stated in: § Potential for Over-Self-Critique: · plain

  “Our iterative self-refinement method (MCQG-SRefine) improves question quality and difficulty based on critique and correction stages.”

May combine two claims.

When MCQG-SRefine removed too much information to raise quality and difficulty, inferring the correct answer sometimes became harder, though this was rare (7.5%) in human evaluation.

“However, when MCQG-SRefine reduced the amount of useless information provided in the questions to increase quality and difficulty, omitting too much information sometimes made inferring the correct answer more challenging (Case Study 2), although this was a rare occurrence in our human evaluation (7.5%).”

May combine two claims.

GPT-4's judgments are inconsistent in about 35% of instances, varying with the order in which options are presented (A-B versus B-A).

“it still exhibits inconsistencies in approximately 35% of cases. These inconsistencies, where GPT-4’s judgments vary based on the order of presented options (A-B vs. B-A), highlight the challenges of relying on LLMs for evaluation.”

May combine two claims.

With expert-identified topics and test points, MCQG-SRefine reduced easy questions by 80%, increased medium questions 2.25-fold, and increased hard questions 4-fold.

“Specifically, when provided with expert-identified topics and test points, there is an 80% reduction in easy questions, a 2.25-fold increase in medium questions, and a 4-fold increase in hard questions.”

• Also stated in: § Main results · hedged

  “In Figure 4, we also observe a decrease of about 80% in easy questions, a 2.25 times increase in medium questions, and a 4 times increase in hard questions with the input of these medical cases with expert-provided topics and key points.”

• AssumptionRests on an assumption: The AMBOSS difficulty labels provide a valid reference for USMLE question difficulty.

May combine two claims.

For human-generated topics and key points, 30% of best scores came from the first round, while the remaining 70% were evenly spread across rounds 2, 3, and 4.

“For human-generated topics and key points, 30% of the best scores came from the first round of output, while the remaining 70% were evenly distributed across rounds 2, 3, and 4.”

May combine two claims.

With machine-provided topics and key points, easy questions fell by 33.3%, medium questions doubled, and hard questions did not increase.

“For machine-provided topics and key point cases, the proportion of easy questions decreased by 33.3%, a 2 times increase in medium questions, but there was no increase in the proportion of hard questions (RQ2).”

• AssumptionRests on an assumption: The AMBOSS difficulty labels provide a valid reference for USMLE question difficulty.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Grounding the key elements of generated questions in the original medical cases helps LLMs reduce hallucinations and improves reliability.

“Additionally, grounding questions’ key elements in the original medical cases can help LLMs reduce hallucinations, enhancing the reliability of the generated content.”

May combine two claims.

Human-provided topics and key points were clearer, making high-quality first-round generation more likely, while machine-provided ones needed more improvement.

“This is consistent with the main results, where topics and key points provided by humans were clearer, making it more likely for the LLM to generate high-quality questions in the first round, while those provided by machines required more improvement.”

Probably not a substantial claim: may not pass S4 (based).

If LLMs are not given targeted guidance like chosen topics and test points, their questions frequently miss relevance, quality, and suitable difficulty.

“As discussed in Limit2, generating questions using LLMs without specific guidance, such as defined topics t and test points k, often results in questions that lack relevance, quality, and appropriate difficulty.”

Classified as probably not a substantial claim. May combine two claims.

Human evaluators did not know which system generated each question, and question order was randomized per data point.

“The evaluators were blinded to the source of the questions, and the order of the questions was randomized for each data point.”

May combine two claims.

Generative AI models may create content well without deep understanding, reducing their dependability as evaluators.

“Recent studies (West et al., 2023) also reveal that generative AI models can excel at creating content while lacking deep understanding, which limits their reliability as evaluators.”

• cites West et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

GPT-4 and similar LLMs perform very well on USMLE QA tasks, reaching human-expert accuracy and reasoning.

“Our motivation stems from the fact that LLMs like GPT-4 have proven to perform exceptionally well on USMLE QA tasks, achieving human-expert levels in both accuracy and reasoning processes.”

• cites Achiam et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Using recent medical cases as input lets LLMs generate more up-to-date questions and reduces the outdated-knowledge limitation.

“Using the latest medical cases as input, LLMs can generate up-to-date questions, thereby minimizing the limitations of outdated knowledge.”

May combine two claims.

An LLM-as-Judge automatic metric can replace costly expert evaluation and provide reliable, expert-aligned assessments.

“Furthermore, we introduce an LLM-as-Judge-based automatic metric to replace the complex and costly expert evaluation process, ensuring reliable and expert-aligned assessments.”

• AssumptionRests on an assumption: The 30 expert-designed criteria cover the relevant aspects of USMLE question quality.

May combine two claims. The quote only partly states this claim.

GPT-4 used a conservative strategy of copying and pasting from raw medical cases to avoid hallucination, which sacrificed simplicity and logical coherence.

“Our experiments show that GPT-4 adopted a very conservative strategy when generating context due to our emphasis on hallucination issues in the prompts. This strategy involves copying and pasting much information from raw medical case inputs to avoid generating potentially incorrect new facts.”

May combine two claims.

The two human evaluators' preference agreement was 87.5% with Cohen's kappa 0.66722, indicating substantial reliability.

“We calculated the Percentage Agreement (87.5%) and Cohen’s kappa (0.66722) for the two evaluators’ preferences, indicating substantial reliability of our human evaluation settings.”

May combine two claims.

GPT-4 plus MCQG-SRefine can turn answer-revealing information into hints through critique and correction, which experts judged as better question construction.

“So the questions directly generated by GPT-4 often make the answers obvious, but GPT-4 + MCQG-SRefine can modify this information into hints for the correct answer through its critical and corrective steps, which experts consider a better way to construct questions for candidates (Case Study 4).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Top LLMs such as GPT-4 have achieved over 90% accuracy on sample USMLE questions.

“Top LLMs like GPT-4 have shown over 90% accuracy on sample USMLE questions (Achiam et al., 2023).”

• cites Achiam et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Expert-provided topics and key points are higher quality than machine-provided ones, allowing the LLM to generate harder questions through deeper critique and correction.

“This also indicates that the quality of topics and key points provided by experts is higher, so LLM can generate more difficult questions by thinking more deeply during the critique and correction steps.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This work is the first to apply iterative self-refinement to the generation of USMLE-style MCQs.

“Inspired by the success of these iterative self-refinement methods, we are the first to explore using this approach for generating USMLE-style MCQs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Strong LLM reviewers like GPT-4 reach more than 80% agreement with human preferences in multi-turn dialogue, in both controlled and crowd-sourced settings, matching human evaluator agreement levels.

“Similarly, research by Zheng et al. (2024) and Zhang et al. (2024a) indicates that powerful LLM reviewers, such as GPT-4, achieve over 80% consistency with human preferences in multi-turn dialogue scenarios, both in controlled and crowd-sourced settings, reaching agreement levels similar to human evaluators.”

• cites Zheng et al. (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhang et al. (2024a)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In tasks like summarization and story generation, LLM evaluations correlate strongly with human judgments, scoring or comparing texts along specific criteria.

“LLMs have shown a high correlation with human evaluations in tasks such as summarization and story generation, effectively scoring candidate texts or comparing them based on specific evaluation aspects (Chen et al., 2023a; Li et al., 2024c,b; Gu et al., 2024).”

• cites Chen et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Li et al., 2024c,bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

The authors' work tackles these challenges to produce dependable, domain-specific evaluations suited to medical education.

“Our work addresses these challenges to create reliable, domain-specific evaluations that meet the unique requirements of medical education.”

Classified as probably not a substantial claim.

LLMs are able to revise their own outputs to satisfy diverse user requirements, including reducing harmful content, adding keywords, generating diverse responses, or debugging code.

“Recent research has demonstrated that LLMs (Achiam et al., 2023; Heng et al., 2024; Wang et al., 2024a) can self-correct their responses to meet various user requirements, such as reducing harmful content, incorporating specific keywords, diversity requirement generation, or debugging code (Madaan et al., 2024; Chen et al., 2023b).”

• cites Achiam et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Madaan et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chen et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.