pith. sign in

arxiv: 2303.08112 · v6 · submitted 2023-03-14 · 💻 cs.LG

Eliciting Latent Predictions from Transformers with the Tuned Lens

Nora Belrose , Igor Ostrovsky , Lev McKinney , Zach Furman , Logan Smith , Danny Halawi , Stella Biderman , Jacob Steinhardt This is my paper

Pith reviewed 2026-05-12 16:49 UTC · model grok-4.3

classification 💻 cs.LG
keywords tuned lenslogit lenstransformer interpretabilitylatent predictionsaffine probeslayer-wise decodingmalicious input detection
0
0 comments X

The pith

Affine probes trained per layer decode transformer hidden states into reliable vocabulary predictions, outperforming the logit lens.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper trains a small affine probe on the hidden state of each block in a frozen pretrained transformer, allowing every layer to be decoded into a distribution over tokens. This tuned lens refines the logit lens by learning a per-layer adjustment rather than using a fixed projection. Experiments across models up to 20B parameters show the resulting latent predictions are more accurate, stable, and less biased. Causal interventions confirm the probes rely on features the model itself uses. The sequence of these predictions across layers also serves as an effective signal for identifying malicious inputs.

Core claim

We train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier logit lens technique. We show it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. The trajectory of latent predictions can be used to detect malicious inputs with high accuracy.

What carries the argument

The tuned lens: a learned affine probe fitted independently to each layer's hidden state that maps it to logits over the vocabulary.

If this is right

  • The method provides a stable view of how token predictions are refined layer by layer during inference.
  • Prediction trajectories across layers can flag anomalous or adversarial inputs without additional model training.
  • The probes generalize across autoregressive language models of varying sizes up to 20 billion parameters.
  • Causal evidence indicates the decoded distributions reflect features the model actually computes rather than spurious correlations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The technique could support layer-specific model editing by identifying which layers hold the decisive prediction shifts.
  • Similar per-layer probes might apply to non-language transformers for inspecting feature refinement in other domains.
  • Tracking divergence between tuned-lens predictions and final output could serve as an online monitor for unexpected model behavior.

Load-bearing premise

The learned affine probes recover the model's actual internal computation rather than merely fitting a convenient readout that happens to correlate with the final output.

What would settle it

A causal intervention that changes the model's final output but leaves the tuned lens predictions at intermediate layers unchanged, or a direct comparison showing the tuned lens no more predictive than the logit lens on held-out data.

read the original abstract

We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier "logit lens" technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at https://github.com/AlignmentResearch/tuned-lens.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper introduces the tuned lens: for each layer of a frozen pretrained autoregressive transformer, an affine probe is trained to map the hidden state h_l to a distribution over the vocabulary. This refines the logit lens by learning per-layer readouts. Experiments on models up to 20B parameters show the tuned lens yields more predictive, reliable, and unbiased per-layer predictions than the logit lens. Causal interventions (feature ablations and edits) indicate that the probes rely on features similar to those used by the model itself. The sequence of latent predictions across layers is also shown to detect malicious inputs with high accuracy. Full reproduction code is released.

Significance. If the central claims hold, the tuned lens supplies a practical, scalable tool for inspecting how transformers iteratively refine next-token predictions layer by layer, strengthening mechanistic interpretability research. The multi-scale experiments, causal interventions, and released code are concrete strengths that increase the work's utility. The malicious-input detection result points to a potential safety application, though it is secondary to the interpretability contribution.

major comments (2)
  1. [§4.3] §4.3 (Causal Experiments): The feature interventions and ablations are performed on activations after the affine probes have already been fit to those same activations (or to final logits). This design tests consistency between the learned readout and the intervention effect but does not directly demonstrate that the probes recover the model's native layer-wise computation rather than a convenient linear approximation of downstream layers. A control that compares probe predictions to the model's own un-probed intermediate computations (e.g., via direct logit extraction without regression) would be needed to support the claim that the tuned lens 'uses similar features to the model.'
  2. [§3.2] §3.2 (Definition of 'unbiased'): The claim that the tuned lens is 'unbiased' relative to the logit lens is central to the comparison, yet the precise metric (e.g., whether it refers to calibration error, KL divergence to final logits, or something else) is not formalized before the experiments. Without an explicit definition or derivation showing that the affine map removes a specific bias term, the superiority claim rests on empirical tables whose interpretation depends on this choice.
minor comments (3)
  1. [Table 1, Figure 2] Table 1 and Figure 2: axis labels and legends use inconsistent notation for 'logit lens' vs. 'tuned lens' across panels; standardize to improve readability.
  2. [§4.4] §4.4 (Malicious input detection): the reported accuracy is given without a baseline that uses only final-layer logits or a simple perplexity threshold; adding this control would clarify the incremental value of the trajectory.
  3. [Abstract / §4.1] The abstract states results 'across multiple model sizes up to 20B' but does not list the exact models or token counts used for probe training; this detail belongs in §4.1.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment point by point below, indicating where we agree and will revise the manuscript for clarity and rigor.

read point-by-point responses

  1. Referee: [§4.3] §4.3 (Causal Experiments): The feature interventions and ablations are performed on activations after the affine probes have already been fit to those same activations (or to final logits). This design tests consistency between the learned readout and the intervention effect but does not directly demonstrate that the probes recover the model's native layer-wise computation rather than a convenient linear approximation of downstream layers. A control that compares probe predictions to the model's own un-probed intermediate computations (e.g., via direct logit extraction without regression) would be needed to support the claim that the tuned lens 'uses similar features to the model.'

    Authors: We agree that the interventions are applied after probe fitting, so they primarily confirm consistency between the readout and the causal effects on activations. Because the tuned lens is trained to recover the model's final output distribution from each hidden state, the fact that interventions on the same features affect both the probe and the model's downstream computation in aligned ways provides evidence that the probes rely on features the model itself uses to refine predictions. The logit lens (direct logit extraction without learned regression) is already used as the suggested control throughout our comparisons. In the revision we will explicitly reframe §4.3 to present the logit-lens results as this control and discuss how the tuned lens improves upon it under interventions. revision: partial

  2. Referee: [§3.2] §3.2 (Definition of 'unbiased'): The claim that the tuned lens is 'unbiased' relative to the logit lens is central to the comparison, yet the precise metric (e.g., whether it refers to calibration error, KL divergence to final logits, or something else) is not formalized before the experiments. Without an explicit definition or derivation showing that the affine map removes a specific bias term, the superiority claim rests on empirical tables whose interpretation depends on this choice.

    Authors: We acknowledge that 'unbiased' was introduced without a formal definition in §3.2. In the revised manuscript we will add an explicit definition: the tuned lens is unbiased relative to the logit lens when its per-layer predictions exhibit lower average KL divergence to the final-layer output distribution (measured across models and datasets). We will also briefly explain that the learned affine map corrects for the distributional shift between intermediate hidden states and the final unembedding, while noting that this is an empirical correction rather than a full theoretical derivation of bias removal. revision: yes

Circularity Check

0 steps flagged

No significant circularity in claimed derivation chain

full rationale

The paper trains affine probes on held-out data to map intermediate hidden states to vocabulary distributions and evaluates the resulting tuned lens on separate malicious-input detection benchmarks plus causal interventions. These steps do not reduce the core claims (greater predictiveness than logit lens, similar features to the model, detection utility) to fitted quantities by construction; the validation data and experiments are independent of the probe-fitting objective.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The method rests on the assumption that an affine map per layer is sufficient to decode hidden states and that the model's own next-token predictions are the correct supervision signal. No new physical entities or unstated mathematical axioms are introduced.

free parameters (1)
  • per-layer affine probe weights and biases
    Trained on the model's own outputs for each block; these are the central fitted objects.
axioms (1)
  • domain assumption Hidden states at each layer contain information that can be linearly mapped to the final vocabulary distribution
    Invoked when the authors train the probes and interpret their outputs as latent predictions.

pith-pipeline@v0.9.0 · 5470 in / 1180 out tokens · 36860 ms · 2026-05-12T16:49:22.662071+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models

    cs.LG 2026-05 unverdicted novelty 8.0

    Adaptive scheduling of interventions in discrete diffusion language models, timed to attribute-specific commitment schedules discovered with sparse autoencoders, delivers precise multi-attribute steering up to 93% str...

  2. Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens

    cs.LG 2026-04 accept novelty 8.0

    Function vectors steer LLMs successfully where the logit lens fails to decode the target answer, showing the two properties come apart.

  3. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    cs.CL 2023-04 accept novelty 8.0

    Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.

  4. Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

    cs.CL 2026-07 unverdicted novelty 7.0

    LOCOS scores attention heads via OV-circuit output projection onto answer-token unembedding directions and identifies non-literal retrieval heads whose ablation collapses performance on non-literal benchmarks more tha...

  5. SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    SemRF supplies fixed semantic anchors and pseudo-inverse tying to produce stable coordinates for residual dynamics, Voronoi traces, and minimum-action canonical paths that link to parameter efficiency under controlled...

  6. IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients

    cs.LG 2026-06 unverdicted novelty 7.0

    IG-Lens computes exact layer-wise attributions of target token probability changes in transformers via telescoping integrated gradients, ensuring the attributions sum precisely to the total probability delta.

  7. IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients

    cs.LG 2026-06 unverdicted novelty 7.0

    IG-Lens introduces telescoping integrated gradients that attribute exact changes in target probability to individual transformer layers, with the attributions summing precisely to the observed probability shift.

  8. Plans Don't Persist: Why Context Management Is Load Bearing for LLM Agents

    cs.AI 2026-06 unverdicted novelty 7.0

    Replay pairing shows LLM agents do not persist plans in hidden states but rely on plans remaining in context, with rapid signal decay and task performance drops when plans are evicted.

  9. Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    SWITCH uses explicit <swi> and </swi> boundary tokens to make latent chain-of-thought compatible with on-policy RL (GRPO) and open to causal mechanistic probing, outperforming prior hidden-state recurrence methods.

  10. Bag of Dims: Training-Free Mechanistic Interpretability via Dimension-Level Sign Patterns

    cs.LG 2026-06 unverdicted novelty 7.0

    Sign patterns in the unrotated standard basis of transformer activations form independent binary feature registers that support training-free detection, prediction, and causal intervention across language, vision, and...

  11. Forecasting Future Behavior as a Learning Task

    cs.AI 2026-06 unverdicted novelty 7.0

    Behavior Forecasters trained on LRM trajectories outperform larger models in predicting repeatability and input sensitivity at low cost.

  12. Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering

    cs.CL 2026-06 unverdicted novelty 7.0

    FD-SLMs exhibit state inertia during abrupt interruptions that a training-free perception-vector steering intervention mitigates, lifting correctness from 28% to 45% and IWOR from 40% to 72% on the Zero-Buffer Benchmark.

  13. Bernstein-Schur Kernels: Random Features by Sketched Modulation and Radial Randomization

    cs.LG 2026-06 unverdicted novelty 7.0

    A doubly-randomized feature map for Bernstein-Schur kernels achieves unbiased kernel approximation with variance and operator-norm bounds controlled by effective dimension rather than crude maxima.

  14. PRISM: Recovering Instruction Sets from Language Model Activations

    cs.AI 2026-06 unverdicted novelty 7.0

    PRISM is a new activation-conditioned model that recovers full sets of simultaneous instructions from LLM hidden states via judge-guided GRPO training and outperforms prior activation-to-language methods on security-r...

  15. Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery

    cs.CL 2026-06 unverdicted novelty 7.0

    Structurally distinct circuits for literal sequence copying across token frequency bands implement the same computation, shown by broad transfer of band-specific edges, a shared core recovering 99% performance, and in...

  16. Toward Calibrated, Fair, and accurate Deepfake Detection

    cs.LG 2026-06 unverdicted novelty 7.0

    Face-Feature Tuning is a label-free logit remapping method that reduces FPR/TPR gaps across groups in deepfake detection while preserving overall accuracy.

  17. When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    Mechanistic analysis of GLMs shows graph sink tokens have high activation but low importance for predictions, indicating decoupling between saliency and graph-semantic utility.

  18. How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings

    cs.CL 2026-05 unverdicted novelty 7.0

    Auto-interpretation labels for SAE features generalize poorly across languages and scripts, missing the same semantic content up to 4x more often in Serbian than English and more in Cyrillic than Latin despite determi...

  19. Training-Free Looped Transformers

    cs.LG 2026-05 unverdicted novelty 7.0

    Training-free looped transformers retrofit recurrence to frozen models via damped ODE sub-steps on mid-stack blocks, yielding gains such as +2.64 pp on MMLU-Pro for Qwen3-4B.

  20. The physics of AI weather models

    physics.ao-ph 2026-05 unverdicted novelty 7.0

    AI weather models may simulate the atmosphere via particle positions in latent space whose updates follow gradient flow on a learned free energy functional rather than conventional physical equations.

  21. Scale-Dependent Collective Adaptation in Self-Amending LLM Societies: A Cross-Family Study of Emergent Governance

    nlin.AO 2026-05 unverdicted novelty 7.0

    LLM societies in Nomic show non-monotonic collective adaptation peaking at mid-scales, with smaller models rule-inert and larger ones restrictive.

  22. Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space

    cs.LG 2026-05 unverdicted novelty 7.0

    Pretraining and alignment induce asymmetric geometric traces in transformer weights because alignment updates concentrate in read pathways due to activation covariance while write pathways inherit less structure from ...

  23. From Noise to Diversity: Random Embedding Injection in LLM Reasoning

    cs.AI 2026-05 conditional novelty 7.0

    Random Soft Prompts (RSPs) sampled from the embedding distribution improve Pass@N on reasoning benchmarks by increasing early-stage token diversity without any training.

  24. Deep Minds and Shallow Probes

    cs.LG 2026-05 unverdicted novelty 7.0

    Symmetry under affine reparameterizations of hidden coordinates selects a unique hierarchy of shallow coordinate-stable probes and a probe-visible quotient for cross-model transfer.

  25. In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification

    cs.LG 2026-05 conditional novelty 7.0

    In-context learning binds model outputs to the demonstrated label tokens as an exhaustive vocabulary, overriding semantic plausibility and causing fixation even with homogeneous or nonsense labels.

  26. The Convergence Gap: Instruction-Tuned Language Models Stabilize Later in the Forward Pass

    cs.LG 2026-05 unverdicted novelty 7.0

    Instruction-tuned language models stabilize their next-token predictions later in the forward pass than pretrained models, with late MLP layers providing the strongest tested control point under matched histories.

  27. Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions

    cs.CL 2026-05 unverdicted novelty 7.0

    Performance collapse in layer-pruned LLMs stems from disrupting the Silent Phase of decision-making, which blocks the transition to correct predictions, while the later Decisive Phase is robust to pruning.

  28. Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Tabular foundation models show substantial depthwise redundancy, so a looped single-layer version achieves comparable results with 20% of the original parameters.

  29. Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training

    cs.CL 2026-05 unverdicted novelty 7.0

    Transformer circuits show free evolution during SFT, rendering static mechanistic localization inadequate for future parameter updates due to inherent temporal latency.

  30. The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

    cs.LG 2026-05 unverdicted novelty 7.0

    Transformers encode counts correctly internally but fail to read them out due to misalignment with digit output directions, fixable by updating 37k output parameters or small LoRA on attention.

  31. The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

    cs.LG 2026-05 accept novelty 7.0

    Transformers store count information internally but cannot read it out as digits due to near-orthogonal alignment with output-head rows; updating digit rows or applying LoRA to attention layers improves constrained an...

  32. Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2

    cs.CL 2026-04 unverdicted novelty 7.0

    DF-SSM distills Mamba-2 to 1-bit scaffold plus int8 low-rank correction for 9.7x compression and 21.4x faster inference, plus analysis showing three distinct processing phases across layers.

  33. PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training

    cs.LG 2026-04 unverdicted novelty 7.0

    Stealth Pretraining Seeding plants persistent unsafe behaviors in LLMs via diffuse poisoned web content that activates on precise triggers and evades standard evaluation.

  34. Graph-Native Reinforcement Learning Enables Traceable Scientific Hypothesis Generation through Conceptual Recombination

    cs.AI 2026-07 unverdicted novelty 6.0

    Graph-PRefLexOR fine-tunes graph-native models with GRPO to organize reasoning into phases, yielding 40-65% gains in traceable hypothesis generation and 2-3x semantic diversity on 100 materials science questions.

  35. Prototype Language Models

    cs.LG 2026-07 unverdicted novelty 6.0

    PRISM forms predictions as sparse mixtures of learned prototypes trained with clustering objectives, matching dense model accuracy while enabling ~500x faster data attribution and behavior editing without finetuning.

  36. FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

    cs.AI 2026-06 unverdicted novelty 6.0

    FADE attenuates FFN outputs in LVLMs based on layer-wise information flow analysis to mitigate hallucinations, shown effective on POPE, CHAIR, and MME benchmarks.

  37. Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

    cs.CL 2026-06 unverdicted novelty 6.0

    Generation accuracy exceeds self-evaluation on three of four in-context QA benchmarks, with attention analysis showing evaluators attend 3-5x less to context and barely read the candidate answer.

  38. Understanding How MLLMs Describe Artworks Using Token Activation Maps

    cs.CV 2026-06 unverdicted novelty 6.0

    Token Activation Maps applied to MLLM art descriptions reveal that visual grounding strength varies by token category, with better artist identification than title prediction.

  39. Skin-Deep: A Geometric Diagnostic for Alignment Fragility in Large Language Model Representations

    cs.AI 2026-06 unverdicted novelty 6.0

    Skin-Deep extracts a Geometric Fragility Score from LLM activations that identifies which initially safe models retain the most refusal after small LoRA fine-tuning.

  40. Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

    cs.CL 2026-06 unverdicted novelty 6.0

    Confident Decoding selects reliable intermediate layers via entropy-guided backward search to bypass final-layer alignment perturbations in LLMs, improving reasoning performance.

  41. A Validation-Gated Mechanistic Account of Suicidality Detection in LLMs

    cs.CL 2026-06 unverdicted novelty 6.0

    A validation-gated framework rules out analysis for implicit suicidal intent separation but identifies a recurring low-rank semantic mid-network feature causally implicated in binary suicide detection across models an...

  42. BrainSurgery: Reproducible and Reliable Declarative Weight Manipulations for Model Editing and Upcycling

    cs.LG 2026-06 unverdicted novelty 6.0

    Introduces BrainSurgery, a declarative YAML-based tool for reproducible tensor surgery on deep learning checkpoints with built-in validation assertions, shown via examples and case studies in model upcycling and LoRA ...

  43. Inside the LLM Word Factory

    cs.CL 2026-06 unverdicted novelty 6.0

    Activation patching localizes English detokenization in Llama2-7B to a two-stage attention-then-MLP process at layer 1 that generalizes to 12 models from 8 families, with depth varying by positional encoding, plus an ...

  44. Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings

    cs.CL 2026-06 unverdicted novelty 6.0

    EmbedFilter applies a linear filter derived from the LLM unembedding matrix to suppress high-frequency token influences in text embeddings, yielding improved zero-shot performance and inherent dimensionality reduction.

  45. Diagnosing Visual Ignorance in Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    VLMs show language-prior reliance via multi-stage bottlenecks in visual retrieval and suppression, with many benchmark examples remaining answerable under severe visual obfuscation.

  46. Child-directed speech facilitates production, not comprehension, in BabyLMs

    cs.CL 2026-05 unverdicted novelty 6.0

    CDS-trained BabyLMs show earlier and more appropriate production in a new frame-completion task while FineWeb-edu models lead on comprehension benchmarks, indicating current tests underestimate CDS benefits.

  47. TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection

    cs.AI 2026-05 unverdicted novelty 6.0

    TriLens detects hallucinations via per-layer entropy trajectories of logit-lens readouts from three internal modules across LLMs and QA benchmarks.

  48. Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations

    cs.CL 2026-05 unverdicted novelty 6.0

    Deception probes in LLMs collapse under stylistic shifts but recover with style-augmented training, rejecting single-direction and entropy hypotheses in favor of distributed multi-dimensional signals.

  49. Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation

    cs.CL 2026-05 unverdicted novelty 6.0

    LLMs represent semantic operations in NLI via decodable, partially distinct activation subspaces whose steering causally affects predictions, though with model-dependent steerability and cross-operation interference.

  50. Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography

    cs.CL 2026-05 unverdicted novelty 6.0

    Sparse autoencoders applied to GPT-2 and Llama models recover semantic features accounting for 94% of peak brain encoding performance and map onto distinct cortical semantic regions across three languages.

  51. How Many Different Outputs Can a Transformer Generate?

    cs.LG 2026-05 unverdicted novelty 6.0

    Transformers are limited to a linearly growing number of accessible output sequences with prompt length, with exponential decay in accessible proportion beyond a critical point, even under unbounded context.

  52. Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)

    cs.LG 2026-05 unverdicted novelty 6.0

    SVD on the lm_head weight matrix of transformers reveals interpretable vocabulary clusters that indicate training data composition, model differences, and ethical concerns in models like GPT-OSS, Gemma, and Qwen.

  53. Geometry-Lite: Interpretable Safety Probing via Layer-Wise Margin Geometry

    cs.LG 2026-05 unverdicted novelty 6.0

    Geometry-Lite decomposes LLM safety detection into layer-wise margin geometries and finds that persistent boundary positions, not layer-to-layer drift, drive most detection performance across nine models and seven benchmarks.

  54. Scale Determines Whether Language Models Organize Representation Geometry for Prediction

    cs.LG 2026-05 unverdicted novelty 6.0

    Representation geometry in language models aligns with the unembedding readout subspace in a scale-dependent manner, preserved throughout training in large models but progressively lost in late layers of small models ...

  55. Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

    cs.LG 2026-05 unverdicted novelty 6.0

    Distinguishable Deletion unifies knowledge erasure and refusal for LLM unlearning via an energy index that enforces boundaries during training and enables refusal at inference.

  56. Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

    cs.LG 2026-05 unverdicted novelty 6.0

    Training installs a depth-dependent spectral gradient and low-rank bottleneck in LLM residual streams whose amplification or suppression of graph communities is predicted by local operator type.

  57. N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation

    cs.LG 2026-05 unverdicted novelty 6.0

    N-vium achieves 57.9% wall-clock speedup over matched standard transformers at no perplexity cost by mixing exact predictions from multiple model depths.

  58. Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy

    cs.LG 2026-05 unverdicted novelty 6.0

    Base LLMs show multi-agent yield to peer pressure at rates equal to or higher than aligned models, localized by activation patching to mid-layers where attention dominates, with one dissenter cutting yield by 54-73 po...

  59. Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy

    cs.LG 2026-05 unverdicted novelty 6.0

    Pretrained base models exhibit higher yield to peer disagreement than RLHF instruct variants, with the effect localized to mid-layer attention and mitigated by structured dissent rather than prompt defenses.

  60. Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces

    cs.LG 2026-05 unverdicted novelty 6.0

    A latent mediation framework with sparse autoencoders enables non-additive token-level influence attribution in LLMs by learning orthogonal features and back-propagating attributions.

Reference graph

Works this paper leans on

105 extracted references · 105 canonical work pages · cited by 103 Pith papers · 16 internal anchors

  1. [1]

    Journal of the Royal Statistical Society: Series B (Methodological) , volume=

    The statistical analysis of compositional data , author=. Journal of the Royal Statistical Society: Series B (Methodological) , volume=. 1982 , publisher=

    work page 1982

  2. [2]

    Alabdulmohsin, B

    Revisiting neural scaling laws in language and vision , author=. arXiv preprint arXiv:2209.06640 , year=

    work page arXiv

  3. [3]

    Understanding intermediate layers using linear classifier probes

    Understanding intermediate layers using linear classifier probes , author=. arXiv preprint arXiv:1610.01644 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Submitted to The Eleventh International Conference on Learning Representations , year=

    Overthinking the Truth: Understanding how Language Models process False Demonstrations , author=. Submitted to The Eleventh International Conference on Learning Representations , year=

    work page

  5. [5]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Training a helpful and harmless assistant with reinforcement learning from human feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Deep learning through the lens of example difficulty , author=. Advances in Neural Information Processing Systems , volume=

    work page

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Revisiting model stitching to compare neural representations , author=. Advances in Neural Information Processing Systems , volume=

    work page

  8. [8]

    Computational Linguistics , volume=

    Probing classifiers: Promises, shortcomings, and advances , author=. Computational Linguistics , volume=. 2022 , publisher=

    work page 2022

  9. [9]

    If you use this software, please cite it using these metadata , volume=

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , author=. If you use this software, please cite it using these metadata , volume=

    work page

  10. [10]

    Breakthroughs in statistics , volume=

    Foresight: Its logical laws, its subjective sources , author=. Breakthroughs in statistics , volume=

    work page

  11. [11]

    arXiv preprint arXiv:1609.03543 , year=

    Logical induction , author=. arXiv preprint arXiv:1609.03543 , year=

    work page arXiv

  12. [12]

    Philosophy of Science , volume=

    Slightly more realistic personal probability , author=. Philosophy of Science , volume=. 1967 , publisher=

    work page 1967

  13. [13]

    , author=

    Truth and probability. , author=. Studies in subjective probability , pages=. 1926 , publisher=

    work page 1926

  14. [14]

    2007 , howpublished =

    Yudkowsky, Eliezer , title =. 2007 , howpublished =

    work page 2007

  15. [15]

    doi:10.5281/zenodo.5879544 , url =

    Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel , url =. 2021 , version =. doi:10.5281/zenodo.5879544 , month =

    work page doi:10.5281/zenodo.5879544 2021

  16. [16]

    GPT-NeoX-20B: An Open-Source Autoregressive Language Model

    Gpt-neox-20b: An open-source autoregressive language model , author=. arXiv preprint arXiv:2204.06745 , year=

    work page Pith review arXiv

  17. [18]

    Datasheet for the pile.arXiv preprint arXiv:2201.07311, 2022

    Pythia: a Scaling Suite for Language Model Interpretability Research , author=. Computing Research Repository , eprint=. doi:10.48550/arXiv.2201.07311 , url=

    work page doi:10.48550/arxiv.2201.07311

  18. [19]

    2006 , publisher=

    Pattern recognition and machine learning , author=. 2006 , publisher=

    work page 2006

  19. [20]

    Enriching Word Vectors with Subword Information

    Enriching Word Vectors with Subword Information , author=. arXiv preprint arXiv:1607.04606 , year=

    work page Pith review arXiv

  20. [21]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

    work page

  21. [22]

    Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages=

    LOF: identifying density-based local outliers , author=. Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages=

    work page 2000

  22. [23]

    Alignment Forum , year=

    Causal Scrubbing: a method for rigorously testing interpretability hypotheses , author=. Alignment Forum , year=

    work page

  23. [24]

    Advances in Neural Information Processing Systems , volume=

    Similarity and Matching of Neural Network Representations , author=. Advances in Neural Information Processing Systems , volume=

    work page

  24. [25]

    Knowledge Neurons in Pretrained Transformers

    Dai, Damai and Dong, Li and Hao, Yaru and Sui, Zhifang and Chang, Baobao and Wei, Furu. Knowledge Neurons in Pretrained Transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. doi:10.18653/v1/2022.acl-long.581

    work page doi:10.18653/v1/2022.acl-long.581 2022

  25. [26]

    arXiv preprint arXiv:2209.02535 , year=

    Analyzing transformers in embedding space , author=. arXiv preprint arXiv:2209.02535 , year=

    work page arXiv

  26. [27]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    An image is worth 16x16 words: Transformers for image recognition at scale , author=. arXiv preprint arXiv:2010.11929 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  27. [28]

    Austrian Journal of Statistics , volume=

    Changing the reference measure in the simplex and its weighting effects , author=. Austrian Journal of Statistics , volume=

    work page

  28. [29]

    Transactions of the Association for Computational Linguistics , volume=

    Amnesic probing: Behavioral explanation with amnesic counterfactuals , author=. Transactions of the Association for Computational Linguistics , volume=. 2021 , publisher=

    work page 2021

  29. [30]

    2021 , journal=

    A Mathematical Framework for Transformer Circuits , author=. 2021 , journal=

    work page 2021

  30. [31]

    Toy Models of Superposition

    Toy Models of Superposition , author=. arXiv preprint arXiv:2209.10652 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [32]

    2021 , publisher =

    Gao, Leo and Tow, Jonathan and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and McDonell, Kyle and Muennighoff, Niklas and Phang, Jason and Reynolds, Laria and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy , title =. doi:10.5281/zenodo.5371628 , url =

    work page doi:10.5281/zenodo.5371628

  32. [33]

    Reducing transformer depth on demand with structured dropout.arXiv preprint arXiv:1909.11556,

    Reducing transformer depth on demand with structured dropout , author=. arXiv preprint arXiv:1909.11556 , year=

    work page arXiv 1909

  33. [34]

    arXiv preprint arXiv:2106.03004 , year=

    Exploring the Limits of Out-of-Distribution Detection , author=. arXiv preprint arXiv:2106.03004 , year=

    work page arXiv

  34. [35]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor , year=. Computing Research Repository , eprint=. doi:10.48550/arXiv.2101.00027 , url=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2101.00027

  35. [36]

    Transformer Feed-Forward Layers Are Key-Value Memories

    Transformer feed-forward layers are key-value memories , author=. arXiv preprint arXiv:2012.14913 , year=

    work page internal anchor Pith review arXiv 2012

  36. [37]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space, 2022

    Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space , author=. arXiv preprint arXiv:2203.14680 , year=

    work page arXiv

  37. [38]

    Highway and Residual Networks learn Unrolled Iterative Estimation

    Highway and residual networks learn unrolled iterative estimation , author=. arXiv preprint arXiv:1612.07771 , year=

    work page Pith review arXiv

  38. [39]

    Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

    Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models , author=. arXiv preprint arXiv:2301.04213 , year=

    work page arXiv

  39. [40]

    R eal T oxicity P rompts: Evaluating Neural Toxic Degeneration in Language Models

    Gehman, Samuel and Gururangan, Suchin and Sap, Maarten and Choi, Yejin and Smith, Noah A. R eal T oxicity P rompts: Evaluating Neural Toxic Degeneration in Language Models. Findings of the Association for Computational Linguistics: EMNLP 2020. 2020. doi:10.18653/v1/2020.findings-emnlp.301

    work page doi:10.18653/v1/2020.findings-emnlp.301 2020

  40. [41]

    Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , year =

    Benjamin Heinzerling and Michael Strube , title = ". Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , year =

    work page 2018

  41. [42]

    Proceedings of the 2019 Con , year=

    Designing and Interpreting Probes with Control Tasks , author=. Proceedings of the 2019 Con , year=

    work page 2019

  42. [43]

    A structural probe for finding syntax in word representations , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

    work page 2019

  43. [44]

    European conference on computer vision , pages=

    Deep networks with stochastic depth , author=. European conference on computer vision , pages=. 2016 , organization=

    work page 2016

  44. [45]

    arXiv preprint arXiv:1710.04773 , year=

    Residual connections encourage iterative inference , author=. arXiv preprint arXiv:1710.04773 , year=

    work page internal anchor Pith review arXiv

  45. [46]

    Scaling Laws for Neural Language Models

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  46. [47]

    International conference on machine learning , pages=

    Shallow-deep networks: Understanding and mitigating network overthinking , author=. International conference on machine learning , pages=. 2019 , organization=

    work page 2019

  47. [48]

    International Conference on Machine Learning , pages=

    Similarity of neural network representations revisited , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    work page 2019

  48. [49]

    arXiv:2105.06990

    BERT busters: Outlier dimensions that disrupt transformers , author=. arXiv preprint arXiv:2105.06990 , year=

    work page arXiv

  49. [50]

    Probing Classifiers are Unreliable for Concept Removal and Detection , publisher =

    Probing classifiers are unreliable for concept removal and detection , author=. arXiv preprint arXiv:2207.04153 , year=

    work page arXiv

  50. [51]

    The BigScience

    Hugo Lauren. The BigScience. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    work page

  51. [52]

    Advances in neural information processing systems , volume=

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks , author=. Advances in neural information processing systems , volume=

    work page

  52. [53]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Understanding image representations by measuring their equivariance and equivalence , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    work page

  53. [54]

    arXiv preprint arXiv:2110.04844 , year=

    Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits , author=. arXiv preprint arXiv:2110.04844 , year=

    work page arXiv

  54. [55]

    7 LaGO: Latent Action Guidance for Online Reinforcement Learning Lin, J., Du, Y ., Watkins, O., Hafner, D., Abbeel, P., Klein, D., and Dragan, A

    Emergent world representations: Exploring a sequence model trained on a synthetic task , author=. arXiv preprint arXiv:2210.13382 , year=

    work page arXiv

  55. [56]

    2008 eighth ieee international conference on data mining , pages=

    Isolation forest , author=. 2008 eighth ieee international conference on data mining , pages=. 2008 , organization=

    work page 2008

  56. [57]

    Journal Soc

    On the generalized distances in statistics: Mahalanobis distance , author=. Journal Soc. Bengal , volume=

    work page

  57. [58]

    A SICK cure for the evaluation of compositional distributional semantic models

    Marelli, Marco and Menini, Stefano and Baroni, Marco and Bentivogli, Luisa and Bernardi, Raffaella and Zamparelli, Roberto. A SICK cure for the evaluation of compositional distributional semantic models. Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC '14). 2014

    work page 2014

  58. [59]

    Nature , volume=

    Highly accurate protein structure prediction with AlphaFold , author=. Nature , volume=. 2021 , publisher=

    work page 2021

  59. [60]

    bioRxiv , pages=

    OpenFold: Retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization , author=. bioRxiv , pages=. 2022 , publisher=

    work page 2022

  60. [61]

    Jump to conclusions: Short-cutting transformers with linear transformations.arXiv preprint arXiv:2303.09435,

    Jump to Conclusions: Short-Cutting Transformers With Linear Transformations , author=. arXiv preprint arXiv:2303.09435 , year=

    work page arXiv

  61. [62]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Roberta: A robustly optimized bert pretraining approach , author=. arXiv preprint arXiv:1907.11692 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1907

  62. [63]

    Mass-Editing Memory in a Transformer

    Mass-editing memory in a transformer , author=. arXiv preprint arXiv:2210.07229 , year=

    work page internal anchor Pith review arXiv

  63. [64]

    LessWrong , year=

    The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable , author=. LessWrong , year=

    work page

  64. [65]

    TransformerLens , author =

    work page

  65. [66]

    LessWrong , year=

    interpreting GPT: the logit lens , author=. LessWrong , year=

    work page

  66. [67]

    2021 , url=

    logit lens on non-gpt2 models + extensions , author=. 2021 , url=

    work page 2021

  67. [68]

    Distill , volume=

    Zoom in: An introduction to circuits , author=. Distill , volume=

    work page

  68. [69]

    Ignore Previous Prompt: Attack Techniques For Language Models

    Ignore Previous Prompt: Attack Techniques For Language Models , author=. arXiv preprint arXiv:2211.09527 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  69. [70]

    arXiv preprint arXiv:2211.06420 , year=

    The Architectural Bottleneck Principle , author=. arXiv preprint arXiv:2211.06420 , year=

    work page arXiv

  70. [71]

    Machine Learning for Health , pages=

    Early exit ensembles for uncertainty quantification , author=. Machine Learning for Health , pages=. 2021 , organization=

    work page 2021

  71. [72]

    OpenAI Blog , year=

    Language Models are Unsupervised Multitask Learners , author=. OpenAI Blog , year=

    work page

  72. [73]

    International Conference on Machine Learning , pages=

    Linear adversarial concept erasure , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    work page 2022

  73. [74]

    11 Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto

    Null it out: Guarding protected attributes by iterative nullspace projection , author=. arXiv preprint arXiv:2004.07667 , year=

    work page arXiv 2004

  74. [75]

    arXiv preprint arXiv:2005.00719 , author =

    Probing the probing paradigm: Does probing accuracy entail task relevance? , author=. arXiv preprint arXiv:2005.00719 , year=

    work page arXiv 2005

  75. [76]

    2007 15th European signal processing conference , pages=

    The effective rank: A measure of effective dimensionality , author=. 2007 15th European signal processing conference , pages=. 2007 , organization=

    work page 2007

  76. [77]

    Computer Speech & Language , volume=

    On the effect of dropping layers of pre-trained transformer models , author=. Computer Speech & Language , volume=. 2023 , publisher=

    work page 2023

  77. [78]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , author=. arXiv preprint arXiv:1910.01108 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  78. [79]

    Scao, Teven Le and Fan, Angela and Akiki, Christopher and Pavlick, Ellie and Ilić, Suzana and Hesslow, Daniel and Castagné, Roman and Luccioni, Alexandra Sasha and Yvon, François and Gallé, Matthias and Tow, Jonathan and Rush, Alexander M. and Biderman, Stella and Webson, Albert and Ammanamanchi, Pawan Sasanka and Wang, Thomas and Sagot, Benoît and Muenni...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2211.05100

  79. [80]

    Tran, Yi Tay, and Donald Metzler

    Confident adaptive language modeling , author=. arXiv preprint arXiv:2207.07061 , year=

    work page arXiv

  80. [81]

    Neural computation , volume=

    Estimating the support of a high-dimensional distribution , author=. Neural computation , volume=. 2001 , publisher=

    work page 2001

Showing first 80 references.