REVIEW 2 minor 45 references
Binary detectors reach perfect F1 scores separating human from AI text while model attribution tops out at 0.9531 F1.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-30 17:30 UTC pith:DRTIUUA6
load-bearing objection This is a shared-task findings report summarizing competition results on AI-text detection, with no new methods or claims.
Findings of the Counter Turing Test: AI-Generated Text Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The Counter Turing Test shared-task results establish that current detection pipelines can achieve perfect separation between human-written and AI-generated text in binary classification, while the same pipelines reach a lower ceiling when required to attribute a text sample to one of several specific large language models.
What carries the argument
The two shared tasks (binary classification and model attribution) evaluated on held-out test data using F1, with top entries built from fine-tuned DeBERTa and BART transformers plus ensembles.
Load-bearing premise
The shared-task test data is assumed to be representative of real-world usage and not specially constructed to favor or defeat particular detection methods.
What would settle it
A new test collection drawn from everyday mixed human and AI sources on which the reported top binary system scores below 0.95 F1 would falsify the claim of reliable binary detection.
If this is right
- Binary detectors can be applied with high reliability in domains that match the shared-task distribution.
- Model attribution requires additional work on features that survive cross-model similarity.
- Ensemble and hybrid transformer approaches deliver the strongest observed results for both tasks.
- Further gains in attribution will likely depend on adversarial robustness and cross-domain generalization.
Where Pith is reading between the lines
- If real-world text mixes differ from the task data, binary performance could drop below the reported perfect score.
- High binary accuracy suggests institutions could use such detectors to triage content for further human review.
- Improved attribution might eventually allow tracing generated text back to specific model providers or fine-tunes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports findings from the Counter Turing Test (CT2) shared tasks on detecting AI-generated text. Task A requires binary classification of human-written vs. AI-generated text and reports a top F1 of 1.0000; Task B requires attribution to the specific generating model and reports a top score of 0.9531. The paper describes the top-performing approaches (primarily fine-tuned DeBERTa- and BART-based transformers and ensembles) and notes that attribution remains substantially harder than binary detection.
Significance. If the reported competition outcomes hold, the work supplies a concrete benchmark of current detector performance against recent LLMs (GPT-4, Claude 3.5, Llama). The near-perfect binary result contrasted with the 0.9531 attribution ceiling usefully quantifies the added difficulty of model identification and points to concrete research directions (adversarial robustness, feature extraction, cross-domain generalization). The empirical, multi-system nature of the report is a strength.
minor comments (2)
- [Abstract] Abstract: the models are listed as “Claude 3.5, and Llama”; supplying the precise variants (e.g., Claude-3.5-Sonnet, Llama-3-70B) would improve reproducibility and context.
- [Abstract] Abstract and results sections: the reported F1 scores are given without accompanying information on test-set size, domain distribution, or generation prompts; a brief summary table or paragraph would help readers assess the scope of the claimed performance.
Simulated Author's Rebuttal
We thank the referee for their positive assessment of the CT2 shared-task findings and for recommending minor revision. No major comments were provided in the report.
Circularity Check
No significant circularity
full rationale
The document is a shared-task findings report presenting empirical competition outcomes (top binary F1 of 1.0000, top attribution score of 0.9531) achieved by submitted systems on the organizers' test sets. No derivations, equations, fitted parameters, or predictive claims are present that could reduce to inputs by construction. All stated results are direct measurements from the competition evaluation, with no self-citation chains or ansatzes invoked to support a methodological derivation.
Axiom & Free-Parameter Ledger
read the original abstract
The growing capability of large language models to produce fluent, contextually coherent text has created mounting pressure on the systems and institutions responsible for ensuring the authenticity of digital content. Advanced generative models such as GPT-4, Claude 3.5, and Llama can produce highly coherent and human-like text, making it increasingly difficult to differentiate between human-written and AI-generated content. While these models have transformative applications, their misuse has raised concerns about misinformation, biased narratives, and security threats. This paper provides a comprehensive analysis of state-of-the-art AI-generated text detection techniques and evaluates their effectiveness through the Counter Turing Test (CT2) shared tasks. Task A (Binary Classification) required participants to distinguish between human-written and AI-generated text, while Task B (Model Attribution) focused on identifying the specific language model responsible for generating a given text. The results demonstrated high performance in binary classification, with the top system achieving an F1 score of 1.0000, but significantly lower scores in model attribution, where the best system achieved 0.9531, highlighting the increased complexity of this task. The top-performing teams leveraged fine-tuned transformer models, ensemble learning, and hybrid detection approaches, with DeBERTa-based and BART-based methods demonstrating strong results. However, the lower scores in Task B underscore the challenges of distinguishing outputs from different LLMs, necessitating further research into adversarial robustness, feature extraction, and cross-domain generalization.
Figures
Reference graph
Works this paper leans on
-
[1]
Gpt-4 technical report.OpenAI Technical Report, 2023
OpenAI. Gpt-4 technical report.OpenAI Technical Report, 2023. URL https://arxiv.org/abs/2303. 08774
2023
-
[2]
Claude ai: Conversational ai assistant
Anthropic. Claude ai: Conversational ai assistant. https://www.anthropic.com/claude, 2024. Accessed: 2025-01-25
2024
-
[3]
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. URL https://arxiv.org/abs/2302.13971
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[4]
Release strategies and the social impacts of language models
Irene Solaiman and Miles Brundage. Release strategies and the social impacts of language models. OpenAI Technical Report, 2019
2019
-
[5]
Deception in ai-generated text: Adversarial evaluation.ACL Workshop on Fact-Checking, 2023
Prakhar Krishna et al. Deception in ai-generated text: Adversarial evaluation.ACL Workshop on Fact-Checking, 2023
2023
-
[6]
Voter suppression AI robocall investigation update
New Hampshire Department of Justice, Office of the Attorney General. Voter suppression AI robocall investigation update. Press release, February 2024. URL https://www.doj.nh.gov/ news-and-media/voter-suppression-ai-robocall-investigation-update
2024
-
[7]
In the matter of Steve Kramer: Notice of apparent liability for forfeiture
Federal Communications Commission. In the matter of Steve Kramer: Notice of apparent liability for forfeiture. FCC 24-59, File No. EB-TCD-24-00036094, May 2024. URL https://docs.fcc.gov/ public/attachments/FCC-24-59A1.pdf
2024
-
[8]
Disinformation affecting the EU: Tackled but not tamed
European Court of Auditors. Disinformation affecting the EU: Tackled but not tamed. Special Report No. 09/2021, Publications Office of the European Union, June 2021. URL https://op.europa. eu/webpub/eca/special-reports/disinformation-9-2021/en/
2021
-
[9]
Overview of text counter turing test: Ai generated text detection
Rajarshi Roy, Gurpreet Singh, Ashhar Aziz, Shashwat Bajpai, Nasrin Imanpour, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Amitava Das, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, and Aman Chadha. Overview of text counter turing test: Ai generated tex...
2025
-
[10]
DetectGPT: Zero-Shot Machine-Generated Text Detection Using Probability Curvature.ICML, 2023
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature, 2023. URL https://arxiv.org/abs/2301.11305
-
[11]
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text, 2019. URL https://arxiv.org/abs/1906.04043
work page internal anchor Pith review Pith/arXiv arXiv 2019
-
[12]
Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna- gpt: Divergent n-gram analysis for training-free detection of gpt-generated text, 2023. URL https://arxiv.org/abs/2305.17359
-
[13]
Spotting LLM s with binoculars: Zero-shot detection of machine-generated text
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024. URL https://arxiv.org/abs/2401.12070
-
[14]
Defending against neural fake news, 2020
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news, 2020. URL https://arxiv.org/abs/1905.12616
-
[15]
Automatic detection of generated text is easiest when humans are fooled,
Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled, 2020. URL https://arxiv.org/abs/1911.00650
-
[16]
Ghostbuster: Detecting text ghostwrit- ten by large language models, 2024
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwrit- ten by large language models, 2024. URL https://arxiv.org/abs/2305.15047
-
[17]
Authorship attribution for neural text generation
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384–8395, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/...
-
[18]
Raidar: generative AI detection via rewriting.arXiv preprint arXiv:2401.12970, 2024
Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting, 2024. URL https://arxiv.org/abs/2401.12970
-
[19]
Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023. URL https: //arxiv.org/abs/2303.13408
-
[20]
A Watermark for Large Language Models.ICML, 2023
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models, 2024. URL https://arxiv.org/abs/2301.10226
-
[21]
Modeling the attack: Detecting ai-generated text by quantifying adversarial perturbations,
Lekkala Sai Teja, Annepaka Yadagiri, Sangam Sai Anish, Siva Gopala Krishna Nuthakki, and Partha Pakray. Modeling the attack: Detecting ai-generated text by quantifying adversarial perturbations,
- [22]
-
[23]
A Comprehensive Dataset for Human vs. AI Generated Text Detection
Rajarshi Roy, Gurpreet Singh, Ashhar Aziz, Shashwat Bajpai, Nasrin Imanpour, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Gaytri Jena, Amitava Das, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, and Aman Chadha. A comprehensive dataset for human vs. ai ...
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[24]
Gemma 2: Improving Open Language Models at a Practical Size
Gemma Team. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024. URL https://arxiv.org/abs/2408.00118
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[25]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Re- nard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B.arXiv preprint arXiv:...
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.06825 2023
-
[26]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. doi: 10.48550/arXiv.2407.10671. URL https://arxiv.org/abs/2407.10671
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.10671 2024
-
[27]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. doi: 10.48550/arXiv.2407.21783. URL https://arxiv.org/abs/2407.21783
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024
-
[28]
Yi: Open Foundation Models by 01.AI
01.AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zon...
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[29]
Hello GPT-4o
OpenAI. Hello GPT-4o. OpenAI Blog, May 2024. URL https://openai.com/index/hello-gpt-4o/
2024
-
[30]
Defactify-text: A comprehensive dataset for human vs
Rajarshi Roy, Gurpreet Singh, Ashhar Aziz, Shashwat Bajpai, Nasrin Imanpour, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Amitava Das, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, and Aman Chadha. Defactify-text: A comprehensive dataset for human vs. ...
2025
-
[31]
Avinash Trivedi and Sangeetha Sivanesan. Sarang at defactify 4.0: Detecting ai-generated text using noised data and an ensemble of deberta models.arXiv preprint arXiv:2502.16857, 2025
-
[32]
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021. URL https://arxiv.org/abs/2006.03654
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[33]
Scalable framework for classifying ai-generated content across modalities, 2025
Anh-Kiet Duong and Petra Gomez-Krämer. Scalable framework for classifying ai-generated content across modalities, 2025. URL https://arxiv.org/abs/2502.00375
-
[34]
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019. URL https://arxiv.org/abs/ 1910.13461
work page internal anchor Pith review Pith/arXiv arXiv 2019
-
[35]
Identifying machine generated text with stylometric features
Vijayasaradhi Indurthi and Vasudeva Varma. Identifying machine generated text with stylometric features. proceedings of DeFactify 4: Fourth workshop on Multimodal Fact-Checking and Hate Speech Detection
-
[36]
A scalable tree boosting system
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794. ACM, August 2016. doi: 10.1145/2939672.2939785. URL http://dx.doi.org/10.1145/ 2939672.2939785
-
[37]
Shrikant Malviya, Pablo Arnau-González, Miguel Arevalillo-Herráez, and Stamos Katsigiannis. Skdu at de-factify 4.0: Natural language features for ai-generated text-detection.arXiv preprint arXiv:2503.22338, 2025
-
[38]
Harika Abburi, Sanmitra Bhattacharya, Edward Bowen, and Nirmala Pudota. Ai-generated text detection: A multifaceted approach to binary and multiclass classification.arXiv preprint arXiv:2505.11550, 2025
-
[39]
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024. URL https://arxiv.org/abs/2212.03533
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[40]
InProceedings of the 16th ACM workshop on artificial intelligence and security, pages 79–90
Chinnappa Guggilla, Budhaditya Roy, Trupti Ramdas Chavan, Abdul Rahman, and Edward Bowen. Ai generated text detection using instruction fine-tuned large language and transformer-based models.arXiv preprint arXiv:2507.05157, 2025
-
[41]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[42]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805
work page internal anchor Pith review Pith/arXiv arXiv 2019
-
[43]
Shifali Agrahari and Sanasam Ranbir Singh. Tracing thought: Using chain-of-thought reasoning to identify the llm behind ai-generated text.arXiv preprint arXiv:2504.16913, 2025
-
[44]
Chain-of-thought prompting elicits reasoning in large language models,
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models,
-
[45]
URL https://arxiv.org/abs/2201.11903
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.