CoChat
Sign Up
PreprintCited by 3Topic ModelingAuthorship Attribution and ProfilingAdversarial Robustness in Machine Learning

Idiosyncrasies in Large Language Models

Open access
Sun, Mingjie, Yin, Yida, Xu, Zhiqiu, Kolter, J. Zico, Liu, Zhuang·2025·arXiv
DOI 10.48550/arxiv.2502.12150arXiv 2502.12150
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work, we unveil and study idiosyncrasies in Large Language Models (LLMs) -- unique patterns in their outputs that can be used to distinguish the models. To do so, we consider a simple classification task: given a particular text output, the objective is to predict the source LLM that generates the text. We evaluate this synthetic task across various groups of LLMs and find that simply fine-tuning text embedding models on LLM-generated texts yields excellent classification accuracy. Notably, we achieve 97.1% accuracy on held-out validation data in the five-way classification problem involving ChatGPT, Claude, Grok, Gemini, and DeepSeek. Our further investigation reveals that these idiosyncrasies are rooted in word-level distributions. These patterns persist even when the texts are rewritten, translated, or summarized by an external LLM, suggesting that they are also encoded in the semantic content. Additionally, we leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies. Finally, we discuss the broader implications of our findings, including training on synthetic data, inferring model similarity, and robust evaluation of LLMs. Code is available at https://github.com/locuslab/llm-idiosyncrasies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.12150

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI-generated text systematically overuses em-dashes, words like 'delve' and 'tapestry,' and other surface-level patterns that even simple classifiers detect reliably.

    “AI-generated text systematically overuses em-dashes, words like 'delve' and 'tapestry,' and other surfacelevel patterns that even simple classifiers detect reliably (Sun et al., 2025; Shaib et al., 2026).”

    From StoryScope: Investigating idiosyncrasies in AI fiction (2026) · cites this paper as Sun et al., 2025

    Section 3.1 (p. 6) finds model-specific word choices and characteristic phrases that a simple TF-IDF logistic-regression classifier identifies with high accuracy. But the paper does not establish overuse of em-dashes, “delve,” or “tapestry”; its classifiers distinguish among LLMs, rather than reliably detecting AI-generated text against human writing (Introduction, p. 1; Related Work, p. 9). (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Similar classifiers have been used to attribute text to its source LLM (Sun et al., 2025).

    “Similar classifiers have been used to attribute text to its source LLM (Sun et al., 2025).”

    From Stealing Reasoning Traces from Proprietary LLM APIs (2026) · cites this paper as Sun et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Cai, Q., Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., and et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024a. · arXiv
  2. Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang, Y. Phi-4 technical report. arXiv preprint arXiv:2412.08905 , 2024b. · arXiv
  3. Antoun, W., Sagot, B., and Seddah, D. From text to source: Results in detecting large language model-generated content. arXiv preprint arXiv:2309.13322 , 2023. · arXiv
  4. Bao, G., Zhao, Y., Teng, Z., Yang, L., and Zhang, Y. Fast-detectgpt: Efficient zero-shot detection of machinegenerated text via conditional probability curvature. In ICLR , 2024.
  5. Cai, W., Shi, T., Zhao, X., and Song, D. Are you getting what you pay for? auditing model substitution in llm apis. arXiv preprint arXiv:2504.04715 , 2025. · arXiv
  6. Chakrabarty, T., Laban, P., and Wu, C.-S. Can ai writing be salvaged? mitigating idiosyncrasies and improving human-ai alignment in the writing process through edits. arXiv preprint arXiv:2409.14509 , 2024. · arXiv
  7. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M. I., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating llms by human preference. In ICML , 2024.
  8. DeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024. · arXiv
  9. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2018.
  10. Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP , 2023.
  11. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., and et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. · arXiv
  12. Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., and Yeung-Levy, S. Describing differences in image sets with natural language. In CVPR , 2024.
  13. Dunlap, L., Mandal, K., Darrell, T., Steinhardt, J., and Gonzalez, J. E. Vibecheck: Discover and quantify qualitative differences in large language models. In ICLR , 2025.
  14. Gao, I., Liang, P., and Guestrin, C. Model equality testing: Which model is this api serving? In ICLR , 2025.
  15. Gehrmann, S., Strobel, H., and Rush, A. M. Gltr: Statistical detection and visualization of generated text. In ACL , 2019.
  16. Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024. · arXiv
  17. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. · arXiv
  18. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR , 2022.
  19. Huang, B., Chen, C., and Shu, K. Authorship attribution in the era of llms: Problems, methodologies, and challenges. arXiv preprint arXiv:2408.08946 , 2024. · arXiv
  20. Huang, Y., Nasr, M., Angelopoulos, A., Carlini, N., Chiang, W.-L., Choquette-Choo, C. A., Ippolito, D., Jagielski, M., Lee, K., Liu, K. Z., Stoica, I., Tram` er, F., and Zhang, C. Exploring and mitigating adversarial manipulation of voting-based leaderboards. arXiv preprint arXiv:2501.07493 , 2025. · arXiv
  21. Ippolito, D., Duckworth, D., Callison-Burch, C., and Eck, D. Automatic detection of generated text is easiest when humans are fooled. In ACL , 2020.
  22. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023. · arXiv
  23. Lavergne, T., Urvoy, T., and Yvon, F. Detecting fake content with relative entropy scoring. In International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse , 2008.
  24. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In ACL , 2004.
  25. Liu, Z. and He, K. A decade's battle on dataset bias: Are we there yet? In ICLR , 2025.
  26. Mansour, Y. and Heckel, R. Measuring bias of web-filtered text datasets and bias propagation through training. arXiv preprint arXiv:2412.02857 , 2024. · arXiv
  27. McGovern, H., Stureborg, R., Suhara, Y., and Alikaniotis, D. Your large language models are leaving fingerprints. In ICCL Workshops , 2024.
  28. Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., and Finn, C. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In ICML , 2023.
  29. Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992 , 2025. · arXiv
  30. Penedo, G., Kydl´ ıˇ cek, H., Allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17575 , 2024. · arXiv
  31. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In NAACL , 2018.
  32. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2024. · arXiv
  33. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR , 2020.
  34. Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussonot, L., Mesnard, T., Shahriari, B., and et al. Gemma2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 , 2024. · arXiv
  35. Russell, J., Karpinska, M., and Iyer, M. People who frequently use chatgpt for writing tasks are accurate and robust detectors of ai-generated text. In ACL , 2025.
  36. Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., and Feizi, S. Can ai-generated text be reliably detected? TMLR , 2025.
  37. Singh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., ¨ Ust¨ un, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., Ermis, B., Fadaee, M., and Hooker, S. The leaderboard illusion. arXiv preprint arXiv:2504.20879 , 2025. · arXiv
  38. Sun, C., Qiu, X., Xu, Y., and Huang, X. How to fine-tune bert for text classification? arXiv preprint arXiv:1905.05583 , 2019. · arXiv
  39. Torralba, A. and Efros, A. A. Unbiased look at dataset bias. In CVPR , 2011.
  40. Uchendu, A., Le, T., Shu, K., and Lee, D. Authorship attribution for neural text generation. In EMNLP , 2020.
  41. Uchendu, A., Ma, Z., Le, T., Zhang, R., and Lee, D. Turingbench: A benchmark environment for turing test in the age of neural text generation. In EMNLP , 2021.
  42. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurIPS , 2017.
  43. Wu, J., Yang, S., Zhai, R., Yuan, Y., Wong, D. F., and Chao, L. S. A survey on llm-generated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724 , 2023. xAI. Grok-2 beta release. x . ai/blog/grok-2 , 2024. · arXiv
  44. You, Z., Zhang, X., Guo, H., Wang, J., and Li, C. Are images indistinguishable to humans also indistinguishable to classifiers? arXiv preprint arXiv:2405.18029 , 2024. · arXiv
  45. Zaitsu, W. and Jin, M. Distinguishing chatgpt(-3.5, -4)generated and human-written papers through japanese stylometric analysis. arXiv preprint arXiv:2304.05534 , 2023. · arXiv
  46. Zeng, B., Yin, Y., and Liu, Z. Understanding bias in largescale visual datasets. In NeurIPS , 2024.
  47. Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild. In ICLR , 2024.
  48. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In ICLR , 2024.
  49. Zhong, R., Wang, H., Klein, D., and Steinhardt, J. Explaining datasets in words: Statistical models with natural language parameters. In NeurIPS , 2024.

Cited by

Sorted by citations

Authors
Sun, M., Yin, Y., Xu, Z., Kolter, J. Z., & Liu, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.12150
arXiv
2502.12150
OpenAlex
W4407759132
Cited by
3
References
1
Citations by year
2026: 1 · 2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, idiosyncrasy, semantic content, LLM-as-a-judge, model similarity, synthetic data training, LLM evaluation, ChatGPT, Claude, Grok, Gemini, DeepSeek

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review