Dermacen Analytica: A novel methodology integrating multi-modal large language models with machine learning in tele-dermatology

“ViTs are, in fact, a variation of the Transformer model (Vaswani et al., 2017; Devlin et al., 2018), which was originally developed for natural language processing (NLP) tasks.”

“ViTs are, in fact, a variation of the Transformer model (Vaswani et al., 2017; Devlin et al., 2018), which was originally developed for natural language processing (NLP) tasks.”

“We use RoBERTa, a pre-trained model on English language text which uses masked language modeling (MLM) That is, it applies masking on a portion of the input tokens in a sentence and asks the model to predict the masked tokens (Liu et al., 2019).”

“• Advanced Textual Similarity Models : Utilizing pre-trained models like BERT (Bidirectional Encoder Representations from Transformers) (Reimers & Gurevych, 2019), which can understand the context of words in text, one can compare the embeddings of the provided answer and the correct response.”

“Vision Transformers (ViTs) (Zhai et al., 2022) constitute a type of AI model that has revolutionized the field of computer vision, i.e. the AI discipline focused on enabling machines to interpret and understand visual information from the world (Khan et al., 2022).”

“The roberta-large-mnli is a variant of the RoBERTa (Robustly Optimized BERT Pretraining Approach) model that has been specifically fine-tuned on the Multi-Genre Natural Language Inference (MNLI) dataset (Williams et al., 2018).”

“The primary goal of NER is to identify and categorize key information in text into predefined categories or “entities” such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. NER systems are designed to recognize entities within a body of text.”

“During training (Atito et al., 2021), ViTs learn to identify features and patterns within the patches, leveraging the self-attention mechanism to understand the context and relationships between different parts of an image (Yuan et al., 2021; Zhai et al., 2022).”

“Classifiers play a critical role in categorizing data, such as distinguishing between different types of tumors or identifying patients at high risk for certain conditions (Johnson et al., 2021).”

“Dermatological conditions (Kerr et al., 2010; Hahnel et al., 2017; Hay et al., 2014) can be broadly categorized into several types, including, but not limited to: 1.”

“Segmentation algorithms are crucial in imaging analysis, helping to isolate and analyze specific areas within medical images for more accurate diagnoses (Wang et al., 2022).”

“Machine learning algorithms, for instance, are instrumental in identifying patterns and insights from large datasets, enabling the prediction of disease progression and response to treatments (Ahsan et al., 2022).”

“The evaluation of medical students is a rigorous process and other studies (Labrak et al., 2024) have also followed a simulated approach of the OSCE to evaluate LLMs in diagnostic tasks.”

“Machine learning (ML), particularly its sub-field of deep learning (DL), has become increasingly popular in dermatology due to its ability to learn from large datasets of skin images (Chan et al., 2020; Du et al., 2020; Hogarty et al., 2020).”

“In skin lesion segmentation, U-Net, a CNN variant, has shown particular efficiency in handling medical images with fewer data samples (Iranpoor et al., 2020; Anand et al., 2023).”

“These algorithms, especially convolutional neural networks (CNNs), have shown proficiency in identifying patterns and features in images that may be subtle or invisible to the human eye (Liopyris et al., 2022).”

“Machine learning (ML), particularly its sub-field of deep learning (DL), has become increasingly popular in dermatology due to its ability to learn from large datasets of skin images (Chan et al., 2020; Du et al., 2020; Hogarty et al., 2020).”

“The process of diagnosing a skin lesion or condition by dermatologists involves several key steps and requires a thorough consideration and evaluation approach (Marwaha et al., 2019; Gachon et al., 2005).”

“Dermatological conditions (Kerr et al., 2010; Hahnel et al., 2017; Hay et al., 2014) can be broadly categorized into several types, including, but not limited to: 1.”

“Techniques such as Otsu’s method or K -means clustering have been widely used for segmenting skin lesions from healthy skin based on color and texture differences (Fan et al., 2017).”

“However, in order to navigate through the complexity of the medical field effectively, a multifaceted approach is essential that integrates several AI technologies including explainability (Panagou-lias et al., 2024b), trustworthiness and holistic evaluation properties (Virvou, 2023).”

“…medical education, a diverse range of assessment methods supplements Objective Structured Clinical Examinations (OSCEs) and multiple-choice quizzes (MCQs) to comprehensively evaluate medical students (Majumder et al., 2019; Vanderbilt et al., 2013; Norcini & McKinley, 2007; Ferris & O’Flynn, 2015).”

“The most common rule used for evaluating a skin lesion, is the following ABCDE rule (Ring et al., 2021; Duarte et al., 2021), which, however, is considered as a guideline and not as a diagnosis tool.”

“Dermatological conditions (Kerr et al., 2010; Hahnel et al., 2017; Hay et al., 2014) can be broadly categorized into several types, including, but not limited to: 1.”

“…medical education, a diverse range of assessment methods supplements Objective Structured Clinical Examinations (OSCEs) and multiple-choice quizzes (MCQs) to comprehensively evaluate medical students (Majumder et al., 2019; Vanderbilt et al., 2013; Norcini & McKinley, 2007; Ferris & O’Flynn, 2015).”

“This suggests the combination of Otsu’s thresholding method with the GrabCut algorithm (Setiawan, 2020) to accurately delineate the le-sion’s boundaries.”

“In the premise of evaluating multi-modal LLMs in the medical domain, we have also constructed an evaluation pipeline based on images and/or text (Panagoulias et al., 2024a, 2023b).”

“During training (Atito et al., 2021), ViTs learn to identify features and patterns within the patches, leveraging the self-attention mechanism to understand the context and relationships between different parts of an image (Yuan et al., 2021; Zhai et al., 2022).”

“…medical education, a diverse range of assessment methods supplements Objective Structured Clinical Examinations (OSCEs) and multiple-choice quizzes (MCQs) to comprehensively evaluate medical students (Majumder et al., 2019; Vanderbilt et al., 2013; Norcini & McKinley, 2007; Ferris & O’Flynn, 2015).”

“The process of diagnosing a skin lesion or condition by dermatologists involves several key steps and requires a thorough consideration and evaluation approach (Marwaha et al., 2019; Gachon et al., 2005).”

“The most common rule used for evaluating a skin lesion, is the following ABCDE rule (Ring et al., 2021; Duarte et al., 2021), which, however, is considered as a guideline and not as a diagnosis tool.”

“Machine learning (ML), particularly its sub-field of deep learning (DL), has become increasingly popular in dermatology due to its ability to learn from large datasets of skin images (Chan et al., 2020; Du et al., 2020; Hogarty et al., 2020).”

“In the premise of evaluating multi-modal LLMs in the medical domain, we have also constructed an evaluation pipeline based on images and/or text (Panagoulias et al., 2024a, 2023b).”

“In skin lesion segmentation, U-Net, a CNN variant, has shown particular efficiency in handling medical images with fewer data samples (Iranpoor et al., 2020; Anand et al., 2023).”

“In the premise of evaluating multi-modal LLMs in the medical domain, we have also constructed an evaluation pipeline based on images and/or text (Panagoulias et al., 2024a, 2023b).”

“Named Entity Recognition (NER) (Nadeau & Sekine, 2007; Hong et al., 2020) is a sub-field of NLP and information retrieval.”

“The segmentation process involves the use of the “apply otsu and grabcut refined” function (Zhang et al., 2018).”

“In the premise of evaluating multi-modal LLMs in the medical domain, we have also constructed an evaluation pipeline based on images and/or text (Panagoulias et al., 2024a, 2023b).”

“Panagoulias et al. (32) created Dermacen Analytica, an improved AI model based on machine learning that can help teledermatology doctors do better diagnoses.”