Pith. sign in

REVIEW 2 major objections 14 references

Training user simulators to produce responses indistinguishable from real users by an LLM judge outperforms training them to match exact ground-truth replies.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 20:50 UTC pith:EXLHBMIH

load-bearing objection Turing-RL gets better indistinguishability scores than matching baselines on two domains, but the paper shows no evidence these gains improve the downstream uses it claims to target. the 2 major comments →

arxiv 2606.19336 v2 pith:EXLHBMIH submitted 2026-06-17 cs.CL

Learning User Simulators with Turing Rewards

Yingshan Susan Wang , Cedegao E. Zhang , Linlu Qiu , Zexue He , Pengyuan Li , Alex Pentland , Roger P. Levy , Yoon Kim This is my paper
classification cs.CL
keywords user simulatorsTuring rewardsreinforcement learningLLM judgesindistinguishabilityconversational agentsReddit discussionssocial simulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes training user simulator models through reinforcement learning where an LLM judge scores responses for how well they could have come from the real user given the conversation history. This Turing-test style reward replaces the usual goal of matching a single correct reply. The method is evaluated in conversational chat and Reddit forum domains, where it beats standard approaches on both automated and human metrics. A sympathetic reader would care because improved simulators could make agent training, system evaluation, and social research more realistic without needing perfect response copies for every turn.

Core claim

Turing-RL trains an LLM user simulator via reinforcement learning with a discriminative Turing reward that an LLM judge assigns based on how indistinguishable the generated response is from what the real user might have said. Rather than maximizing log probability or similarity to a ground-truth reply, the simulator optimizes for passing the judge's human-likeness test. Across conversational chat and Reddit discussion domains the resulting models score higher on both LLM and human evaluation metrics than baselines that rely on response matching.

What carries the argument

The discriminative Turing reward, an LLM-judge score measuring how well a generated response blends with the user's history as if produced by the real user.

Load-bearing premise

An LLM judge supplies a reliable signal of human-likeness that corresponds to the simulator's actual usefulness in downstream tasks rather than just stylistic similarity.

What would settle it

A downstream experiment in which agents trained on Turing-RL simulators show no measurable advantage over agents trained on matching-based simulators when both interact with real human users.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • User simulators no longer require exact ground-truth responses at every turn to be effective.
  • The indistinguishability objective improves performance consistently in both open chat and structured forum settings.
  • LLM-based and human judgments both favor simulators trained with the Turing reward over matching baselines.
  • Better simulators can directly support agent assistant training and personalization system evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the LLM judge signal holds, the approach could lower the cost of collecting large human response datasets for simulator training.
  • The same reward structure might apply to simulating other interactive behaviors where exact matching is difficult or unnecessary.
  • Substituting human judges for the LLM judge could be tested in high-stakes evaluation settings to check transfer of the signal.
  • Measuring real-human performance of agents trained on these simulators would directly test whether the claimed utility gains materialize.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper proposes Turing-RL, a reinforcement learning approach to train LLM-based user simulators. Instead of maximizing likelihood or similarity to ground-truth responses, it uses a discriminative Turing reward from a separate LLM judge that scores how indistinguishable a generated response is from a real user's response given conversation history. Experiments in conversational chat and Reddit discussion domains report that Turing-RL outperforms response-matching baselines on both LLM-judge and human indistinguishability metrics, leading to the claim that optimizing for indistinguishability is effective for learning user simulators applicable to agent training, personalization evaluation, and social science research.

Significance. If the indistinguishability improvements transfer to downstream utility, the method offers a principled alternative to direct response matching for user simulation. The Turing-reward formulation is a clear conceptual contribution and the consistent outperformance on proxy metrics across two domains is a positive signal. However, the significance is limited by the absence of any experiments linking the reported metric gains to the applications listed in the abstract.

major comments (2)
  1. [Abstract] Abstract: The central claim that Turing-RL is effective for the stated applications (training agent assistants, personalization evaluation, etc.) rests on an untested correlation between higher LLM-judge/human indistinguishability scores and functional utility. No experiments measure downstream performance such as agent success rates, personalization accuracy, or other utility proxies when the learned simulators are placed in the loop.
  2. [Abstract] Abstract: No information is supplied on the judge LLM (model identity, prompting procedure, training data, or fine-tuning), statistical significance of the reported gains, or controls for potential confounds such as judge-simulator data overlap. These omissions make it impossible to assess whether the Turing reward is reliable or whether the outperformance is robust.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments point by point below and outline planned revisions to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that Turing-RL is effective for the stated applications (training agent assistants, personalization evaluation, etc.) rests on an untested correlation between higher LLM-judge/human indistinguishability scores and functional utility. No experiments measure downstream performance such as agent success rates, personalization accuracy, or other utility proxies when the learned simulators are placed in the loop.

    Authors: We agree that the manuscript contains no downstream experiments linking indistinguishability gains to the motivating applications. The abstract and introduction present those applications as potential use cases for improved user simulators rather than as claims of demonstrated utility. The core contribution is the demonstration that optimizing for indistinguishability yields better simulators on the reported metrics. We will revise the abstract and add a dedicated limitations paragraph to clarify the scope and note the absence of downstream validation. revision: partial

  2. Referee: [Abstract] Abstract: No information is supplied on the judge LLM (model identity, prompting procedure, training data, or fine-tuning), statistical significance of the reported gains, or controls for potential confounds such as judge-simulator data overlap. These omissions make it impossible to assess whether the Turing reward is reliable or whether the outperformance is robust.

    Authors: We acknowledge these omissions in the current draft. In the revised version we will add an appendix section detailing the judge LLM (including model name and version), the exact prompting procedure, any fine-tuning or training data used for the judge, statistical significance tests (e.g., paired t-tests or bootstrap confidence intervals) for all reported gains, and an explicit discussion of data-overlap controls between the judge and the simulator training sets. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical outperformance shown against external benchmarks

full rationale

The paper presents an empirical RL method (Turing-RL) that trains a simulator LLM using rewards from a separate LLM judge to maximize response indistinguishability, then reports higher scores than response-matching baselines on both LLM and human evaluation metrics. No equations, self-definitional reductions, or fitted parameters renamed as predictions appear in the abstract or described chain. Human evaluations serve as an independent external check, and the judge is described as external to the simulator. No load-bearing self-citations or uniqueness claims are present. The central effectiveness claim rests on comparative empirical results rather than reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities beyond the standard assumption that LLMs can serve as reliable judges; full text would be needed to audit any hidden fitting or modeling choices.

pith-pipeline@v0.9.1-grok · 5713 in / 1051 out tokens · 19638 ms · 2026-06-26T20:50:42.657230+00:00 · methodology

0 comments
read the original abstract

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

Figures

Figures reproduced from arXiv: 2606.19336 by Alex Pentland, Cedegao E. Zhang, Linlu Qiu, Pengyuan Li, Roger P. Levy, Yingshan Susan Wang, Yoon Kim, Zexue He.

Figure 1
Figure 1. Figure 1: Overview of Turing-RL. Given a user’s history, induced persona, and current [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: LLMs trained with Turing rewards (Turing-RL) outperforms other training signals on human-likeness in both domains. Turing judge scores (1–7 Likert; higher = more human-like) from Sonnet 4.6 on Chat and Reddit, with 95% CIs. Hollow markers indicate untrained baselines, while solid markers denote trained Qwen3-8B variants. The dashed line with shaded band marks the Qwen3-8B base model performance with 95% CI… view at source ↗
Figure 3
Figure 3. Figure 3: Turing-RL matches Sim-RL even though Sim-RL is explicitly trained to maxi￾mize similarity, showing that optimizing for indistinguishability does not sacrifice content alignment. Response similarity to ground truth (Sim, %; higher = more similar to what the user actually said) from Sonnet 4.6 on Chat and Reddit, with 95% CIs. Hollow markers indicate untrained baselines, while solid markers denote trained Qw… view at source ↗
Figure 4
Figure 4. Figure 4: Turing-RL is among the strongest on Chat, while Turing-RL and Sim-RL are strongest among the trained models on Reddit, outperforming SFT-Init and Logprob￾RL. Response specificity ([0, 1]; higher = more grounded in the interaction context and compatible with the target user) from Sonnet 4.6, with 95% CIs. Hollow markers indicate untrained baselines, while solid markers denote trained Qwen3-8B variants. GT i… view at source ↗
Figure 5
Figure 5. Figure 5: Comparing human and LLM judge accuracy at identifying the real user’s response. For each of 50 target users per domain, we take the majority vote from ∼6 human annotators (solid) and from the Sonnet 4.6 Turing judge. GT Accuracy is the fraction of targets correctly identified. Both evaluators agree that Turing-RL is the hardest model to distinguish from real users. Sonnet 4.6 matches or exceeds human accur… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison of the ground truth, GPT-5, Qwen3.5-397B, SFT-Init, Sim [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Persona induction prompt (part 1 of 2). The inducer receives the user’s reserved [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Persona induction prompt (part 2 of 2). Continuation of rules and field descriptions, [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: System message shared across all prompt configurations. The [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: User message layout for PRISM (multi-turn dialogue). History conversations [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt layout for ConvoKit (Reddit forum). History threads are wrapped in [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Thinking trace elicitation prompt. A teacher model receives the user’s history [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: User interface used for human annotation in the Reddit domain. [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: User interface used for human annotation in the Chat domain. [PITH_FULL_IMAGE:figures/full_fig_p028_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: User interface consent and instructions. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Exact Turing distinguishability judge prompt used in the experiments, split across [PITH_FULL_IMAGE:figures/full_fig_p030_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Exact HumanLM response-only similarity judge prompt used in sim evaluation, [PITH_FULL_IMAGE:figures/full_fig_p040_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Exact batched response specificity judge prompt used in the experiments, split [PITH_FULL_IMAGE:figures/full_fig_p042_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Raw per-step GRPO training scores for Reddit and Chat. The first row shows [PITH_FULL_IMAGE:figures/full_fig_p047_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Qualitative examples from Chat and Reddit. Each column shows the conversation [PITH_FULL_IMAGE:figures/full_fig_p048_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages

  1. [1]

    values":

    arXiv:2404.16019. David Lazer, Alex Pentland, Lada Adamic, Sinan Aral, Albert-László Barabási, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, et al. Computational social science.Science, 323(5915):721–723, 2009. Chance Jiajie Li, Jiayi Wu, Zhenze Mo, Ao Qu, Yuhan Tang, Kaiya Zhao, Yulu Gan, Jie Fan, Jiangbo Yu, Jinhua Z...

    work page doi:10.18653/v1/2026.eacl-long.200 2009

  2. [2]

    Past responses from [HUMAN]

  3. [3]

    The current conversation context

  4. [4]

    Two candidate responses, Response A and Response B

  5. [5]

    the real issue is

    An advisory watchlist for source-copy checks One candidate is the real [HUMAN] response. The other candidate is AI-generated. ## User History <|User History|> {user_history} <|End User History|> ## Context <|Context|> {context} <|End Context|> ## Candidate Responses ### Response A <|Response A|> {response_a} <|End Response A|> ### Response B <|Response B|...

  6. [6]

    stance", pick key points related to the stance like

    Extract 1-3 key points: - Extract K key points from the ground truth response along the response dimension (e.g., if evaluating a "stance", pick key points related to the stance like "clearly disagrees with X", if evaluating a "response", pick key points about the response like "offers a solution to Y"). - Because you are evaluating the full response, con...

  7. [7]

    - [0.7, 0.9]: Mostly reflected with small imperfections

    Score how well the generated response matches each key point: - For each key point i, compare it with the generated response and assign a match value m_i in range [0, 1]: - 1.0: The key point is precisely and perfectly reflected. - [0.7, 0.9]: Mostly reflected with small imperfections. - [0.4, 0.6]: Partially reflected or vague, but still leaning in the c...

  8. [8]

    + m_K) / K, which measures how comprehensive the generated response reflects the ground truth response

    Compute coverage C = (m_1 + m_2 + ... + m_K) / K, which measures how comprehensive the generated response reflects the ground truth response

  9. [9]

    you're absolutely right

    Compute penalty P for extra or conflicting content: - Examine additional content in the generated response beyond those key points: - Does it introduce unsupported evidence and assumptions? - Is it irrelevant to what ground truth response expresses? - Is it only using generic commentary or high-level framing that misses the ground truth's goals, values, c...

  10. [10]

    Response-specific checks: - The generated response may or may not reuse phrases from the context; however, if the generated response just directly copies previous context, without quoting it , treat that as off-task behavior and give a score of 0. - Wrong-perspective hard zero: if the generated response treats another user's perspective, identity, or expe...

  11. [11]

    key_points

    Compute the final score = max(0, min(1, C - P)) Additional considerations: - Follow the instruction carefully. - Be strict and reserve scores above 0.8 for clearly outstanding matches. - Do not reward verbosity or generic topical plausibility; reward user-specific evidence. Output format (JSON): { "key_points": "<analysis of key points from ground truth a...

  12. [12]

    Target user evidence

  13. [13]

    The current interaction context

  14. [14]

    label_1": {

    {num_candidates} candidate responses <|Target User Evidence|> {user_history} <|End Target User Evidence|> <|Interaction Context|> {context} <|End Interaction Context|> <|Candidate Responses|> {candidates} <|End Candidate Responses|> Figure 18: Exact batched response specificity judge prompt used in the experiments, split across five parts for typesetting....