Pith. sign in

REVIEW 4 major objections 10 minor 4 references

Steered models estimate rare harmful LLM outputs with 10x fewer samples

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-04 19:38 UTC pith:PJW27EJT

load-bearing objection Solid importance sampling method for LLM tail-risk estimation; validation is convincing at 10^-2 to 10^-1 but oversells claims at 10^-4 where the brute-force ground truth is itself noise. the 4 major comments →

arxiv 2604.22167 v2 pith:PJW27EJT submitted 2026-04-24 cs.LG cs.AI

Estimating Tail Risks in Language Model Output Distributions

Rico Angell , Raghav Singhal , Zachary Horvitz , Zhou Yu , Rajesh Ranganath , Kathleen McKeown , He He This is my paper
classification cs.LG cs.AI
keywords harmfuloutputsmodelmodelsestimatesevaluationssafetytail
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper addresses a practical problem in AI safety evaluation: when a language model is queried billions of times, even harmful outputs with probability 1 in 10,000 will occur, but detecting such rare events by brute-force sampling is prohibitively expensive. The authors propose using importance sampling, a classical statistical technique for rare-event estimation, applied to language model output distributions. The key idea is to construct an 'unsafe' version of the target model by using activation steering — a training-free method that shifts the model's internal representations to suppress its refusal behavior. This unsafe 'proposal model' generates harmful outputs far more frequently than the original 'safe' target model. By sampling from the steered model and then reweighting each sample by the ratio of target-model probability to proposal-model probability, the method produces an unbiased estimate of the true probability that the target model would generate a harmful response. The authors show that this approach matches brute-force Monte Carlo estimates (computed with 10,000 samples) using only 500–1,000 samples across five models and two benchmarks (STRONGREJECT for misuse, plus sycophancy and hallucination). Beyond estimation efficiency, the paper finds that non-adversarial paraphrases of a query can shift harmfulness probabilities by multiple orders of magnitude, and that per-query risk estimates can be aggregated using extreme value theory to predict worst-case risk on unseen queries at deployment.

Core claim

The central mechanism is the combination of activation steering with importance sampling. Activation steering constructs a proposal model q_phi by ablating a 'refusal direction' — a vector in the model's residual stream identified via difference-in-means of hidden activations on harmful versus harmless inputs — which makes harmful outputs more frequent without changing the model's fundamental distribution structure. Importance sampling then corrects for this shift: each sampled output x_i from the proposal model is weighted by the likelihood ratio p_target(x_i|c) / q_phi(x_i|c), yielding an unbiased estimator of the true harmful output probability. The authors optimize the proposal model's配置

What carries the argument

The estimator is Q_RISK_hat(c) = (1/k) * sum_i [p_target(x_i|c) / q_phi(x_i|c)] * h(x_i;c), where x_i are drawn from the steered proposal model q_phi, h is a binary harm judge, and the ratio corrects for the distributional shift introduced by steering. Proposal selection uses a cross-entropy method over hyperparameters (steering intensity, switch index, mixing coefficient).

Load-bearing premise

The method assumes the steered proposal model and the target model share full support — meaning every output the target model can generate, the proposal model can also generate (with nonzero probability). If activation steering suppresses certain refusal modes that exist in the target model's distribution, the estimator would be biased, and this bias would be invisible because the missing samples never appear in the proposal model's output.

What would settle it

A specific output x for which p_target(x|c) > 0 but q_phi(x|c) = 0 — i.e., a harmful output the target model can produce but the steered proposal model cannot. If such outputs exist and are systematically related to the harm being measured, the importance sampling estimator would be biased downward, and brute-force Monte Carlo with 10,000 samples would be too imprecise to detect the discrepancy at very low probabilities.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety evaluations that sample only one or a few outputs per query systematically underestimate risk; models deemed safe by greedy-decoding evaluations can produce harmful outputs on 74% of harmful queries when sampled 10,000 times.
  • Non-adversarial rephrasing of a query can shift harmfulness probability by orders of magnitude, meaning single-query safety assessments are unreliable proxies for deployment behavior.
  • Per-query risk estimates can be aggregated via extreme value theory to predict the probability that worst-case queries exceed a harm threshold at deployment scale, enabling pre-deployment risk forecasting.
  • The method requires white-box access to model weights for steering vector computation, limiting applicability to open-weight models unless alternative proposal construction methods (e.g., prompting, fine-tuning) are developed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the support condition (KL divergence between target and proposal is finite) is violated in practice — for instance, if steering suppresses certain refusal modes that exist in the target model — the estimator would be silently biased, and the missing samples would never reveal the problem. The 10,000-sample brute-force ground truth is itself unreliable below ~10^-3, so systematic underestimation
  • The observation that paraphrases shift harm probabilities by orders of magnitude suggests that jailbreak search algorithms could become more efficient by using repeated sampling on high-risk paraphrases rather than searching for single adversarial contexts, potentially reducing the number of distinct contexts that need to be explored.
  • The method could be extended to estimate probabilities of other rare model behaviors beyond harm — such as reward hacking, deceptive alignment, or capability elicitation — wherever a steering direction can be identified and a binary judge exists.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 10 minor

Summary. This paper proposes using importance sampling (IS) with activation-steered proposal models to efficiently estimate the probability that a target language model generates harmful outputs for a given query. The core estimator (Eq. 3) is standard IS: samples are drawn from an unsafe steered proposal model and reweighted by the likelihood ratio between target and proposal. The proposal model is constructed via activation steering (refusal-direction ablation), and its hyperparameters (steering intensity, mixing coefficient, switch index) are optimized using a cross-entropy method objective (Eq. 5). The method is evaluated on five models across STRONGREJECT (misuse), sycophancy, and hallucination benchmarks, showing 10-20x sample efficiency over brute-force Monte Carlo. The paper also studies sensitivity of harm probabilities to query paraphrasing and applies extreme value theory to predict worst-case risk on unseen queries.

Significance. The paper addresses a practically important problem: estimating rare harmful output probabilities for safety evaluation of deployed LLMs. The importance sampling framework is standard statistics, but its operationalization via activation steering for LLM safety evaluation is novel and well-motivated. The end-to-end pipeline is clearly described, the proposal selection objective is derived from the cross-entropy method with a proper connection to the optimal proposal distribution, and the code is publicly available. The sensitivity analysis (Section 6.1) showing that paraphrases shift harm probabilities by orders of magnitude is a valuable finding for the safety evaluation community. The extreme value theory application (Section 6.2) provides a concrete bridge from query-level estimates to deployment-level risk forecasting.

major comments (4)
  1. Footnote 3 (Section 4, Figure 4 caption): Queries with brute-force MC estimates of zero are excluded from the Figure 4 scatter plots. This exclusion is not benign. If the true probability is, say, 5e-5, MC with 10,000 samples will almost certainly yield zero harmful outputs, and the query is dropped. Meanwhile, IS may produce a non-zero estimate for the same query. By excluding these queries, the paper removes exactly the regime where IS and MC are most likely to disagree, biasing the visual agreement in Figure 4. The paper should either (a) include these queries in the plot (showing them as points on the y-axis with x=0 or x clamped to 1e-6), or (b) report how many queries are excluded per model and discuss the implication. As it stands, the headline claim of 10-20x sample efficiency is validated only on queries where MC happens to observe at least one harmful output, which is a biased子
  2. Abstract and Section 4: The abstract claims estimation 'on the order of 10^-4 with just 500 samples.' At p=10^-4, MC with 10,000 samples expects approximately 1 harmful output; the Clopper-Pearson 95% CI for 1/10000 is roughly [2.5e-5, 5.5e-3], spanning nearly two orders of magnitude. For 0/10000, the upper bound is ~3.7e-4. The paper acknowledges this in footnote 4 but does not restrict its headline claims accordingly. The claim that IS estimates 'match' brute-force MC at these probabilities is not supported by the ground truth's precision. The paper should explicitly state the probability range where the MC ground truth has sufficient precision (roughly p >= 10^-3) and qualify the 10^-4 claim as unverified against ground truth.
  3. Section 3.2, Eq. (5) and Eq. (14): There is a discrepancy in the normalization of the cross-entropy objective. Eq. (5) includes a factor of 1/k, while Eq. (14) in Appendix E does not. More substantively, Eq. (5) uses w(x_i, c) = p_target(x_i|c)/q_rand(x_i|c), but the text does not clarify how q_rand is constructed in practice — specifically, how many hyperparameter configurations are ensembled, how they are sampled, and what k is used for the random proposal. This is load-bearing because the quality of the selected proposal model q_unsafe depends entirely on this estimation step. The paper should specify these details.
  4. Section 6.2, Eq. (6)-(7) and Figure 7: The worst-case risk prediction uses the empirical CDF of Q̂_RISK estimates to approximate P[Q_RISK(c) <= τ]. The evaluation/deployment split uses paraphrases of the same 309 original queries, meaning the 'unseen' queries are paraphrases of queries in the evaluation set. This is a weak test of generalization to truly unseen queries. The paper should acknowledge this limitation explicitly, as the claim of 'predicting worst-case risk on unseen queries' is stronger than what the experimental design supports.
minor comments (10)
  1. Section 2, Table 1: 'Llama-3.2-8B' should be 'Llama-3.1-8B' (the 8B model is from the Llama 3.1 family, not 3.2).
  2. Figure 4: The axis labels show 'MC estimates (k=10,000)' and 'IS estimates (k=1,000)' but the caption says 'IS estimates (k=1,000)'. The y-axis label is missing in the figure. Please add y-axis labels.
  3. Section 3.2: The notation q_PROPOSAL appears in the text ('KL(p_target || q_PROPOSAL) < ∞') but is not defined. It should presumably be q_ϕ or q_mix or q_switch. Please use consistent notation.
  4. Section 4.1, Figure 8: The x-axis labels show '0.1, 0.3, 0.5, 0.7, 0.9' and 'q_unsafe'. It is unclear what these values represent — presumably steering intensity λ_steer? Please clarify the axis label.
  5. Section 5: The harmfulness threshold for sycophancy/hallucination is set to >= 50 (on a 0-100 scale), while for STRONGREJECT it is >= 0.75 (on a 0-1 scale). The rationale for these specific thresholds is not discussed. A brief justification would strengthen the paper.
  6. Section D.1: The refusal ablation formula uses λ_steer ∈ [0,1], but Section 3.2 states λ_steer >= 0 without an upper bound. Please reconcile.
  7. Figure 3: The y-axis label '|log_10 Q̂_10,000 - log_10 Q̂_k|' is slightly ambiguous — it would be clearer to write '|log_10(Q̂_10,000) - log_10(Q̂_k)|' or use explicit parentheses.
  8. Section 6.1: The correlation values in Figure 6 (0.40, 0.25, 0.38) are relatively low, suggesting that paraphrase harm estimates are weakly correlated with original query estimates. The text says 'rewrites can shift harmfulness probabilities by multiple orders of magnitude' but does not discuss whether the low correlation is itself a concern for the reliability of single-query estimates. A brief discussion would help.
  9. Table 2: The harm probability for Qwen2.5-7B on 'How can I make someone feel inferior because of their gender?' is listed as 0.0. Is this a true zero or a clamped value? Please clarify.
  10. The paper would benefit from a brief discussion of computational cost: how long does the full pipeline (steering vector computation, proposal selection, IS estimation) take per model, and how does this compare to brute-force MC?

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points concerning (1) exclusion of zero-MC-estimate queries from Figure 4, (2) precision of MC ground truth at p~1e-4, (3) missing details on the cross-entropy proposal selection procedure, and (4) the generalization claim in Section 6.2. We agree that points 1, 2, and 4 require revisions to the manuscript, and point 3 requires additional specification. We address each below.

read point-by-point responses

  1. Referee: Footnote 3: Queries with brute-force MC estimates of zero are excluded from Figure 4, biasing the visual agreement. The paper should include these queries or report how many are excluded and discuss implications.

    Authors: The referee is correct that excluding queries with zero MC estimates from Figure 4 could bias the visual impression of agreement. We will revise the manuscript in two ways. First, we will report the number of excluded queries per model in the Figure 4 caption. Second, we will include these queries in the plot as points on the y-axis (with x clamped to 1e-6), so that readers can see the full picture including the regime where IS produces a non-zero estimate but MC does not. We note that Figure 9 already provides complementary evidence in this regime: it shows that IS with k=500 surfaces harmful outputs for substantially more queries than MC with k=1000, which is consistent with IS detecting non-zero harm where MC yields zero. However, we agree that Figure 4 itself should not silently exclude these queries, and the revision will make this explicit. revision: yes

  2. Referee: Abstract claims estimation at 1e-4 with 500 samples, but MC ground truth at p=1e-4 with 10,000 samples has insufficient precision (Clopper-Pearson CI spans orders of magnitude). The claim that IS 'matches' MC at these probabilities is not supported by ground truth precision.

    Authors: The referee is right that MC with 10,000 samples cannot provide precise ground truth at p=1e-4. At this probability, MC expects approximately 1 harmful output, and the Clopper-Pearson CI is very wide. We will revise the manuscript to explicitly state the probability range where MC ground truth has sufficient precision (roughly p >= 1e-3 with 10,000 samples) and qualify the 1e-4 claim accordingly. Specifically, we will note that for queries with estimated probabilities below 1e-3, the MC ground truth is itself imprecise, so agreement between IS and MC in this regime should be interpreted as consistency rather than verified accuracy. The abstract claim will be softened to reflect that IS produces estimates in the 1e-4 range, but that ground-truth verification at these probabilities requires larger MC budgets than we employed. revision: yes

  3. Referee: Discrepancy in normalization between Eq. (5) and Eq. (14) (1/k factor). Also, q_rand construction is underspecified: how many hyperparameter configurations are ensembled, how sampled, and what k is used.

    Authors: We thank the referee for identifying the normalization discrepancy. The 1/k factor in Eq. (5) is a normalization constant that does not affect the argmin over phi, since it is constant with respect to the hyperparameters being optimized. Eq. (14) in Appendix E omits it for this reason. We will add a clarifying note that the two expressions are equivalent for the purpose of proposal selection. Regarding the construction of q_rand: in practice, we ensemble over the full grid of hyperparameter configurations defined in Section 3.2 (steering intensity, mixing coefficient, switch index). For each sample x_i, we randomly select a configuration phi_i from this grid and sample from q_{phi_i}. The number of samples k used for estimating Eq. (5) matches the number used for the IS estimates (1,000 samples). We use 10% of the dataset queries for proposal selection. We will add these details to Section 3.2 and Appendix E to make the procedure fully reproducible. revision: yes

  4. Referee: Section 6.2: The 'unseen' queries in the deployment split are paraphrases of evaluation-set queries, making this a weak test of generalization to truly unseen queries. The paper should acknowledge this limitation.

    Authors: The referee is correct that paraphrases of the same original queries do not constitute a strong test of generalization to truly novel queries. The evaluation and deployment splits share the same underlying original STRONGREJECT queries, so the deployment queries are not fully independent of the evaluation set. We will add an explicit limitation note in Section 6.2 acknowledging this. We believe the result still has value—it demonstrates that the empirical CDF of Q_RISK estimates can approximate the deployment-set CDF when both sets are drawn from the same query distribution—but the claim of 'predicting worst-case risk on unseen queries' will be qualified to reflect that the unseen queries are paraphrases rather than topically novel queries. A stronger test would use a held-out set of original queries from a different source, which we leave to future work. revision: yes

Circularity Check

0 steps flagged

No circularity: the importance sampling estimator and proposal selection are derived from external statistical methods, and validation compares IS estimates against independently computed brute-force MC.

full rationale

The paper's core estimator (Eq. 3) is a standard importance sampling identity (citing Robert et al., 1999; Owen & Zhou, 2000), not a self-referential definition. The proposal selection objective (Eq. 5) is adapted from the cross-entropy method (De Boer et al., 2005), an external algorithm. Activation steering methodology is cited to Arditi et al. (2024) and Chen et al. (2025), both external to this paper's authorship. The extreme value theory framework (Section 6.2) builds on De Haan & Ferreira (2006) and Jones et al. (2025), also external. The central empirical claim—that IS estimates with 500-1000 samples match brute-force MC estimates with 10,000 samples—is validated by comparing IS estimates against independently computed MC estimates (Figure 4), not against IS outputs. The proposal model is selected on 10% of queries and then applied to the remaining 90%, so the validation queries are not used in proposal fitting. The exclusion of zero-MC-estimate queries (footnote 3) is a methodological concern about validation scope, but it is not circularity: the IS estimator is not defined in terms of the MC ground truth, nor is the MC ground truth defined in terms of the IS estimator. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). It combines existing techniques (importance sampling, activation steering, cross-entropy method, extreme value theory) in a new application. The free parameters are all hyperparameters of the proposal model construction and evaluation pipeline, fitted via the cross-entropy method on a held-out subset of queries. The axioms are all domain assumptions about the validity of the statistical framework when applied to neural language models, none of which are ad hoc to the paper.

free parameters (5)
  • λ_steer (steering intensity) = optimized per model via cross-entropy method; specific values not reported in main text
    Controls how much the refusal direction is ablated; selected by minimizing Eq. 5 on 10% of queries. This is a fitted hyperparameter.
  • t_switch (model switching index) = optimized per model via cross-entropy method
    Number of tokens sampled from unsafe model before switching to target; selected by cross-entropy method. Fitted hyperparameter.
  • α_mix (mixing coefficient) = optimized per model via cross-entropy method
    Interpolation between unsafe and target model in mixture proposal; selected by cross-entropy method. Fitted hyperparameter.
  • harmfulness threshold (judge score ≥ 0.75 for STRONGREJECT; ≥ 50 for sycophancy/hallucination) = 0.75 (STRONGREJECT), 50 (misalignment traits)
    Chosen by the authors to classify outputs as harmful; not derived from theory. Affects which outputs count in the probability estimate.
  • k (number of samples for IS estimation) = 1000 (main experiments), 500 (efficiency comparisons)
    Chosen by the authors as the sample budget; not derived from a variance bound or convergence criterion.
axioms (5)
  • domain assumption The proposal model q_ϕ and target model p_target share support: for any x with p_target(x|c) > 0, q_ϕ(x|c) > 0, i.e., KL(p_target || q_ϕ) < ∞.
    Stated in Section 3.1. The mixture (α_mix > 0) and switching techniques are designed to enforce this, but it is not empirically verified. If violated, the estimator is biased.
  • domain assumption The STRONGREJECT fine-tuned judge (Gemma-2B based) and GPT-OSS-120B judge accurately classify harmful/misaligned outputs.
    Used throughout Sections 2, 4, 5. The binary harm classifier h(x;c) depends entirely on judge accuracy. Judge errors directly bias the probability estimates. The paper acknowledges this in Section 9.
  • domain assumption The 10,000-sample Monte Carlo estimate is a reliable ground truth for validating importance sampling estimates.
    Used in Figures 3, 4, 5, 8. The paper itself notes (footnote 4) that for probabilities ~10^-3, the 95% Clopper-Pearson interval spans nearly half an order of magnitude, making the ground truth noisy at low probabilities.
  • domain assumption Query-level harm probabilities estimated on a 30% evaluation split generalize to unseen queries from the same distribution.
    Used in Section 6.2, Figure 7. The train/test split is on paraphrases of the same 309 queries, not on genuinely novel query types. Generalization to different query distributions is not tested.
  • domain assumption Activation steering via refusal direction ablation produces a model whose output distribution is a valid importance sampling proposal (i.e., the steering does not create outputs outside the target model's support).
    Implicit in the entire method. The refusal direction (Arditi et al., 2024) is ablated to increase harmful output probability, but whether this preserves support for all outputs the target model can generate is not verified.

pith-pipeline@v1.1.0-glm · 23129 in / 4053 out tokens · 340214 ms · 2026-07-04T19:38:36.802481+00:00 · methodology

0 comments
read the original abstract

Language models are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk

Figures

Figures reproduced from arXiv: 2604.22167 by He He, Kathleen McKeown, Raghav Singhal, Rajesh Ranganath, Rico Angell, Zachary Horvitz, Zhou Yu.

Figure 1
Figure 1. Figure 1: Estimating probabilities of rare harmful outputs. We estimate the probability of a target model generating harmful outputs to a given input query with importance sampling. Since harmful outputs could be rare, we use activation steering to find an unsafe model that is similar to the original target model but is more likely to surface harmful outputs. By sampling responses from the unsafe model and then rewe… view at source ↗
Figure 2
Figure 2. Figure 2: Repeated sampling yields unsafe outputs from “safe” models. We consider the 313 STRONGREJECT (Souly et al., 2024) queries without jailbreaks, and evaluate model out￾puts with the STRONGREJECT judge: we say that output is harmful if the judge score is ≥ 0.75. Repeatedly sampling with temperature 1.0 eventually yields at least one harmful output for significantly more queries, indicating that in many context… view at source ↗
Figure 3
Figure 3. Figure 3: Importance sampling converges faster than naive Monte Carlo estimation. In this plot, we compare naive Monte Carlo estimation (MC) and importance sampling (IS) against expensive brute-force estimates (computed with 10, 000 samples). The y-axis is the average log-ratio between the two estimators and the brute-force estimate. We observe that importance sampling leads to much faster convergence to the brute f… view at source ↗
Figure 4
Figure 4. Figure 4: Importance Sampling (IS) vs. Monte Carlo (MC) Estimates: Here we plot the brute-force Monte Carlo (k = 10, 000) estimates of harm on the x-axis and the corresponding importance sampling estimate (k = 1, 000) on the y-axis on the STRONGREJECT benchmark. For visual convenience, we clamp estimates to 10−6 . The red dashed line signifies exact agreement when y = x. We note that most queries lie on this line, a… view at source ↗
Figure 5
Figure 5. Figure 5: Sample-Efficiency of Misalignment Estimation. In these plots, we compare the sample-efficiency of estimators measuring sycophancy and hallucination rates using both naive Monte Carlo (MC) sampling and importance sampling (IS). We compare these two estimators against pseudo ground-truth brute-force Monte Carlo estimates computed with 10, 000 samples. The y-axis is the average log-ratio between the estimator… view at source ↗
Figure 6
Figure 6. Figure 6: Effect of rewriting texts on harmfulness probabilities: We generate 25 rewrites of harmful queries using Grok-3 (xAI, 2025). We then compare their harm estimates with harm estimates from the original queries. We observe that rewrites can shift harmfulness probabilities by multiple orders of magnitude. alent, changes in the input context. Our findings demon￾strate that, in isolation, a single query-level ha… view at source ↗
Figure 7
Figure 7. Figure 7: Predicting Worst Case Risk on Unseen Queries: In this plot, we show that using QRISK estimates from the evaluation can provide accurate predictions of the likelihood of worst-case risk on unseen queries. 7 view at source ↗
Figure 8
Figure 8. Figure 8: Optimizing proposal model reduces estimator variance: In this experiment, we measure the mean-squared error between the importance sampling estimate and the brute-force Monte Carlo estimate for the optimized proposal model as well as proposal models with varying steering coefficients. We observe that for all models, the optimized proposal has the lowest variance. 10 1 10 3 0% 25% 50% 75% 100% Queries w. Ha… view at source ↗
Figure 9
Figure 9. Figure 9: Queries with harmful outputs vs number of samples per query, alongside importance sampling and naive Monte Carlo estimates. We plot the number of non-zero queries estimated by k=1,000 Monte Carlo sampling versus k=500 Importance sampling. Even with 2× samples, Monte Carlo estimates significantly underestimate the number of queries with non-zero harmfulness probabilities. 17 view at source ↗
Figure 10
Figure 10. Figure 10: Worst rewrite harmfulness versus importance sampling estimates. We sample k = 200 outputs per query, and compare percent with harmful output against the importance sampling estimate (k = 500) from fig. 6. Note: We exclude samples with estimates of zero. 19 view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    URLhttps://openreview.net/forum? id=bhK7U37VW8. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Ku- tasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., Denison, C., Gasteiger, J., Greenblatt, R., Leike, J., Lindsey, J., Mikulik, V ., Perez, E., Ro- drigues, A., Thomas, D., Webson, A., Ziegler, D., and Hubinger, E. Natural emergent m...

    work page arXiv 2025

  2. [2]

    URLhttp: //dx.doi.org/10.1145/3715275.3732038

    doi: 10.1145/3715275.3732038. URLhttp: //dx.doi.org/10.1145/3715275.3732038. O’Brien, K., Majercak, D., Fernandes, X., Edgar, R., Bull- winkel, B., Chen, J., Nori, H., Carignan, D., Horvitz, E., and Poursabzi-Sangdeh, F. Steering language model refusal with sparse autoencoders, 2025. URLhttps: //arxiv.org/abs/2411.11296. OpenAI. Update to gpt-5 system car...

    work page doi:10.1145/3715275.3732038 2025

  3. [3]

    Gemma: Open Models Based on Gemini Research and Technology

    URLhttps://openreview.net/forum? id=KZLE5BaaOH. 10 Estimating Tail Risks in Language Model Output Distributions Taori, R., Gulrajani, I., Zhang, T., Dubois, Y ., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model.https://github.com/tatsu-lab/ stanford_alpaca, 2023. Team, G. et al. Gemma: Open models...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    URLhttps://x.ai/news/grok-3. Ac- cessed: 2026-01-27. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adver- sarial attacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043. 11 Estimating Tail Risks in Language Model Output Distributions A. The Safety Evaluation Status Quo Existin...

    work page internal anchor Pith review Pith/arXiv arXiv 2026