Pith. sign in

REVIEW 2 minor 45 references

Binary detectors reach perfect F1 scores separating human from AI text while model attribution tops out at 0.9531 F1.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.3

2026-06-30 17:30 UTC pith:DRTIUUA6

load-bearing objection This is a shared-task findings report summarizing competition results on AI-text detection, with no new methods or claims.

arxiv 2605.20761 v2 pith:DRTIUUA6 submitted 2026-05-20 cs.CL

Findings of the Counter Turing Test: AI-Generated Text Detection

classification cs.CL
keywords AI-generated text detectionshared taskbinary classificationmodel attributiontransformer modelslarge language modelsCounter Turing TestF1 evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper reports findings from the Counter Turing Test shared task on detecting AI-generated text. In Task A, systems had to classify text as human-written or machine-generated, and the leading entry scored an F1 of 1.0000. In Task B, systems had to name the specific language model that produced the text, with the best entry reaching only 0.9531 F1. Top solutions relied on fine-tuned transformer models such as DeBERTa and BART, often combined in ensembles. The results indicate that binary separation is tractable under the task conditions, yet distinguishing among different generative models remains measurably harder.

Core claim

The Counter Turing Test shared-task results establish that current detection pipelines can achieve perfect separation between human-written and AI-generated text in binary classification, while the same pipelines reach a lower ceiling when required to attribute a text sample to one of several specific large language models.

What carries the argument

The two shared tasks (binary classification and model attribution) evaluated on held-out test data using F1, with top entries built from fine-tuned DeBERTa and BART transformers plus ensembles.

Load-bearing premise

The shared-task test data is assumed to be representative of real-world usage and not specially constructed to favor or defeat particular detection methods.

What would settle it

A new test collection drawn from everyday mixed human and AI sources on which the reported top binary system scores below 0.95 F1 would falsify the claim of reliable binary detection.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Binary detectors can be applied with high reliability in domains that match the shared-task distribution.
  • Model attribution requires additional work on features that survive cross-model similarity.
  • Ensemble and hybrid transformer approaches deliver the strongest observed results for both tasks.
  • Further gains in attribution will likely depend on adversarial robustness and cross-domain generalization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If real-world text mixes differ from the task data, binary performance could drop below the reported perfect score.
  • High binary accuracy suggests institutions could use such detectors to triage content for further human review.
  • Improved attribution might eventually allow tracing generated text back to specific model providers or fine-tunes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript reports findings from the Counter Turing Test (CT2) shared tasks on detecting AI-generated text. Task A requires binary classification of human-written vs. AI-generated text and reports a top F1 of 1.0000; Task B requires attribution to the specific generating model and reports a top score of 0.9531. The paper describes the top-performing approaches (primarily fine-tuned DeBERTa- and BART-based transformers and ensembles) and notes that attribution remains substantially harder than binary detection.

Significance. If the reported competition outcomes hold, the work supplies a concrete benchmark of current detector performance against recent LLMs (GPT-4, Claude 3.5, Llama). The near-perfect binary result contrasted with the 0.9531 attribution ceiling usefully quantifies the added difficulty of model identification and points to concrete research directions (adversarial robustness, feature extraction, cross-domain generalization). The empirical, multi-system nature of the report is a strength.

minor comments (2)
  1. [Abstract] Abstract: the models are listed as “Claude 3.5, and Llama”; supplying the precise variants (e.g., Claude-3.5-Sonnet, Llama-3-70B) would improve reproducibility and context.
  2. [Abstract] Abstract and results sections: the reported F1 scores are given without accompanying information on test-set size, domain distribution, or generation prompts; a brief summary table or paragraph would help readers assess the scope of the claimed performance.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of the CT2 shared-task findings and for recommending minor revision. No major comments were provided in the report.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The document is a shared-task findings report presenting empirical competition outcomes (top binary F1 of 1.0000, top attribution score of 0.9531) achieved by submitted systems on the organizers' test sets. No derivations, equations, fitted parameters, or predictive claims are present that could reduce to inputs by construction. All stated results are direct measurements from the competition evaluation, with no self-citation chains or ansatzes invoked to support a methodological derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical report on shared-task results with no mathematical derivations, free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5878 in / 1003 out tokens · 33384 ms · 2026-06-30T17:30:28.281422+00:00 · methodology

0 comments
read the original abstract

The growing capability of large language models to produce fluent, contextually coherent text has created mounting pressure on the systems and institutions responsible for ensuring the authenticity of digital content. Advanced generative models such as GPT-4, Claude 3.5, and Llama can produce highly coherent and human-like text, making it increasingly difficult to differentiate between human-written and AI-generated content. While these models have transformative applications, their misuse has raised concerns about misinformation, biased narratives, and security threats. This paper provides a comprehensive analysis of state-of-the-art AI-generated text detection techniques and evaluates their effectiveness through the Counter Turing Test (CT2) shared tasks. Task A (Binary Classification) required participants to distinguish between human-written and AI-generated text, while Task B (Model Attribution) focused on identifying the specific language model responsible for generating a given text. The results demonstrated high performance in binary classification, with the top system achieving an F1 score of 1.0000, but significantly lower scores in model attribution, where the best system achieved 0.9531, highlighting the increased complexity of this task. The top-performing teams leveraged fine-tuned transformer models, ensemble learning, and hybrid detection approaches, with DeBERTa-based and BART-based methods demonstrating strong results. However, the lower scores in Task B underscore the challenges of distinguishing outputs from different LLMs, necessitating further research into adversarial robustness, feature extraction, and cross-domain generalization.

Figures

Figures reproduced from arXiv: 2605.20761 by Aishwarya Naresh Reganti, Aman Chadha, Amitava Das, Amit Sheth, Ashhar Aziz, Gurpreet Singh, Kapil Wanaskar, Nasrin Imanpour, Nilesh Ranjan Pal, Parth Patwa, Rajarshi Roy, Ritvik Garimella, Shashwat Bajpai, Shreyas Dixit, Shwetangshu Biswas, Subhankar Ghosh, Vasu Sharma, Vinija Jain, Vipula Rawte.

Figure 1
Figure 1. Figure 1: Illustration of Raidar concept. Given a News data text and an LLM-generated text, the same LLM is asked to rewrite the inputs while preserving meaning. The rewriting of a human-written text undergoes more character-level edits (highlighted in red/yellow), while the rewriting of an LLM-generated text remains largely unchanged. 4. Participating Systems With over 52 registrations on the competition web page, … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 32 canonical work pages · 14 internal anchors

  1. [1]

    Gpt-4 technical report.OpenAI Technical Report, 2023

    OpenAI. Gpt-4 technical report.OpenAI Technical Report, 2023. URL https://arxiv.org/abs/2303. 08774

  2. [2]

    Claude ai: Conversational ai assistant

    Anthropic. Claude ai: Conversational ai assistant. https://www.anthropic.com/claude, 2024. Accessed: 2025-01-25

  3. [3]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. URL https://arxiv.org/abs/2302.13971

  4. [4]

    Release strategies and the social impacts of language models

    Irene Solaiman and Miles Brundage. Release strategies and the social impacts of language models. OpenAI Technical Report, 2019

  5. [5]

    Deception in ai-generated text: Adversarial evaluation.ACL Workshop on Fact-Checking, 2023

    Prakhar Krishna et al. Deception in ai-generated text: Adversarial evaluation.ACL Workshop on Fact-Checking, 2023

  6. [6]

    Voter suppression AI robocall investigation update

    New Hampshire Department of Justice, Office of the Attorney General. Voter suppression AI robocall investigation update. Press release, February 2024. URL https://www.doj.nh.gov/ news-and-media/voter-suppression-ai-robocall-investigation-update

  7. [7]

    In the matter of Steve Kramer: Notice of apparent liability for forfeiture

    Federal Communications Commission. In the matter of Steve Kramer: Notice of apparent liability for forfeiture. FCC 24-59, File No. EB-TCD-24-00036094, May 2024. URL https://docs.fcc.gov/ public/attachments/FCC-24-59A1.pdf

  8. [8]

    Disinformation affecting the EU: Tackled but not tamed

    European Court of Auditors. Disinformation affecting the EU: Tackled but not tamed. Special Report No. 09/2021, Publications Office of the European Union, June 2021. URL https://op.europa. eu/webpub/eca/special-reports/disinformation-9-2021/en/

  9. [9]

    Overview of text counter turing test: Ai generated text detection

    Rajarshi Roy, Gurpreet Singh, Ashhar Aziz, Shashwat Bajpai, Nasrin Imanpour, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Amitava Das, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, and Aman Chadha. Overview of text counter turing test: Ai generated tex...

  10. [10]

    DetectGPT: Zero-Shot Machine-Generated Text Detection Using Probability Curvature.ICML, 2023

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature, 2023. URL https://arxiv.org/abs/2301.11305

  11. [11]

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text, 2019. URL https://arxiv.org/abs/1906.04043

  12. [12]

    Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text.arXiv preprint arXiv:2305.17359, 2023

    Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. Dna- gpt: Divergent n-gram analysis for training-free detection of gpt-generated text, 2023. URL https://arxiv.org/abs/2305.17359

  13. [13]

    Spotting LLM s with binoculars: Zero-shot detection of machine-generated text

    Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024. URL https://arxiv.org/abs/2401.12070

  14. [14]

    Defending against neural fake news, 2020

    Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news, 2020. URL https://arxiv.org/abs/1905.12616

  15. [15]

    Automatic detection of generated text is easiest when humans are fooled,

    Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled, 2020. URL https://arxiv.org/abs/1911.00650

  16. [16]

    Ghostbuster: Detecting text ghostwrit- ten by large language models, 2024

    Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwrit- ten by large language models, 2024. URL https://arxiv.org/abs/2305.15047

  17. [17]

    Authorship attribution for neural text generation

    Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384–8395, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/...

  18. [18]

    Raidar: generative AI detection via rewriting.arXiv preprint arXiv:2401.12970, 2024

    Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting, 2024. URL https://arxiv.org/abs/2401.12970

  19. [19]

    Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023

    Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023. URL https: //arxiv.org/abs/2303.13408

  20. [20]

    A Watermark for Large Language Models.ICML, 2023

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models, 2024. URL https://arxiv.org/abs/2301.10226

  21. [21]

    Modeling the attack: Detecting ai-generated text by quantifying adversarial perturbations,

    Lekkala Sai Teja, Annepaka Yadagiri, Sangam Sai Anish, Siva Gopala Krishna Nuthakki, and Partha Pakray. Modeling the attack: Detecting ai-generated text by quantifying adversarial perturbations,

  22. [22]

    URL https://arxiv.org/abs/2510.02319

  23. [23]

    A Comprehensive Dataset for Human vs. AI Generated Text Detection

    Rajarshi Roy, Gurpreet Singh, Ashhar Aziz, Shashwat Bajpai, Nasrin Imanpour, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Gaytri Jena, Amitava Das, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, and Aman Chadha. A comprehensive dataset for human vs. ai ...

  24. [24]

    Gemma 2: Improving Open Language Models at a Practical Size

    Gemma Team. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024. URL https://arxiv.org/abs/2408.00118

  25. [25]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Re- nard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B.arXiv preprint arXiv:...

  26. [26]

    Qwen2 Technical Report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. doi: 10.48550/arXiv.2407.10671. URL https://arxiv.org/abs/2407.10671

  27. [27]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. doi: 10.48550/arXiv.2407.21783. URL https://arxiv.org/abs/2407.21783

  28. [28]

    Yi: Open Foundation Models by 01.AI

    01.AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zon...

  29. [29]

    Hello GPT-4o

    OpenAI. Hello GPT-4o. OpenAI Blog, May 2024. URL https://openai.com/index/hello-gpt-4o/

  30. [30]

    Defactify-text: A comprehensive dataset for human vs

    Rajarshi Roy, Gurpreet Singh, Ashhar Aziz, Shashwat Bajpai, Nasrin Imanpour, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Amitava Das, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, and Aman Chadha. Defactify-text: A comprehensive dataset for human vs. ...

  31. [31]

    Sarang at defactify 4.0: Detecting ai-generated text using noised data and an ensemble of deberta models.arXiv preprint arXiv:2502.16857, 2025

    Avinash Trivedi and Sangeetha Sivanesan. Sarang at defactify 4.0: Detecting ai-generated text using noised data and an ensemble of deberta models.arXiv preprint arXiv:2502.16857, 2025

  32. [32]

    DeBERTa: Decoding-enhanced BERT with Disentangled Attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021. URL https://arxiv.org/abs/2006.03654

  33. [33]

    Scalable framework for classifying ai-generated content across modalities, 2025

    Anh-Kiet Duong and Petra Gomez-Krämer. Scalable framework for classifying ai-generated content across modalities, 2025. URL https://arxiv.org/abs/2502.00375

  34. [34]

    BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019. URL https://arxiv.org/abs/ 1910.13461

  35. [35]

    Identifying machine generated text with stylometric features

    Vijayasaradhi Indurthi and Vasudeva Varma. Identifying machine generated text with stylometric features. proceedings of DeFactify 4: Fourth workshop on Multimodal Fact-Checking and Hate Speech Detection

  36. [36]

    A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794. ACM, August 2016. doi: 10.1145/2939672.2939785. URL http://dx.doi.org/10.1145/ 2939672.2939785

  37. [37]

    Skdu at de-factify 4.0: Natural language features for ai-generated text-detection.arXiv preprint arXiv:2503.22338, 2025

    Shrikant Malviya, Pablo Arnau-González, Miguel Arevalillo-Herráez, and Stamos Katsigiannis. Skdu at de-factify 4.0: Natural language features for ai-generated text-detection.arXiv preprint arXiv:2503.22338, 2025

  38. [38]

    Ai-generated text detection: A multifaceted approach to binary and multiclass classification.arXiv preprint arXiv:2505.11550, 2025

    Harika Abburi, Sanmitra Bhattacharya, Edward Bowen, and Nirmala Pudota. Ai-generated text detection: A multifaceted approach to binary and multiclass classification.arXiv preprint arXiv:2505.11550, 2025

  39. [39]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024. URL https://arxiv.org/abs/2212.03533

  40. [40]

    InProceedings of the 16th ACM workshop on artificial intelligence and security, pages 79–90

    Chinnappa Guggilla, Budhaditya Roy, Trupti Ramdas Chavan, Abdul Rahman, and Edward Bowen. Ai generated text detection using instruction fine-tuned large language and transformer-based models.arXiv preprint arXiv:2507.05157, 2025

  41. [41]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

  42. [42]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

  43. [43]

    Tracing thought: Using chain-of-thought reasoning to identify the llm behind ai-generated text.arXiv preprint arXiv:2504.16913, 2025

    Shifali Agrahari and Sanasam Ranbir Singh. Tracing thought: Using chain-of-thought reasoning to identify the llm behind ai-generated text.arXiv preprint arXiv:2504.16913, 2025

  44. [44]

    Chain-of-thought prompting elicits reasoning in large language models,

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models,

  45. [45]

    URL https://arxiv.org/abs/2201.11903