On the Role of Directionality in Structural Generalization

Zichao Wei

arxiv: 2607.02307 · v1 · pith:J4NGOK5Jnew · submitted 2026-07-02 · 💻 cs.CL · cs.LG

On the Role of Directionality in Structural Generalization

Zichao Wei This is my paper

Pith reviewed 2026-07-03 14:21 UTC · model grok-4.3

classification 💻 cs.CL cs.LG

keywords CCGdirectionalitystructural generalizationsemantic parsingSLOG benchmarkAM algebraposition shiftsrecursive depth

0 comments

The pith

Directed types in CCG let a parser generalize better to position shifts than an undirected algebra does.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether adding explicit directionality to the symbolic layer improves structural generalization on the SLOG benchmark. With the same BERT-base encoder, a CCG-based system using directed types reaches 75.9 percent exact match and beats the prior AM-Parser system by five points overall. The advantage appears almost entirely on the five position-shift categories, where the CCG system gains nearly thirty points, while the undirected system stays stronger on recursive-depth categories. Replacing the encoder with DeBERTa-v3-large raises accuracy to 90.7 percent and closes the recursion gap, showing that directionality and encoder capacity address different parts of the generalization problem.

Core claim

Redesigning the symbolic backend around CCG directed types (via deterministic CKY plus a single linear decoder) encodes directional distinctions that the AM algebra lacks. Under identical BERT-base encoding this yields higher exact-match accuracy on SLOG, with the improvement concentrated on modifier-position and argument-extraction shifts. The same change leaves recursive-depth performance lower than the AM baseline. Larger encoders then improve the recursive categories more than the directional ones, indicating that directionality moves the remaining bottleneck from the symbolic layer to the neural encoder.

What carries the argument

CCG directed types inside a deterministic CKY parser paired with a single linear decoder that supplies the directional information missing from the AM algebra.

If this is right

Position-shift categories show a 29.9-point gain when directionality is added to the symbolic layer.
Recursive-depth categories remain stronger under the undirected AM algebra than under CCG.
Encoder upgrades produce their largest gains on recursive-depth categories, complementary to the directional gains.
The symbolic layer's 0 percent category ceiling disappears once direction is represented, moving the limit to the neural encoder.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A hybrid symbolic layer that combines directed types with recursive-friendly operations might remove both bottlenecks at once.
The same directional redesign could be tested on other semantic parsing benchmarks that contain argument-order or modifier-position variation.
If directionality is the key missing piece, then any undirected algebra will hit a hard ceiling on position-shift generalization regardless of encoder size.

Load-bearing premise

The SLOG category groupings separate directional effects from recursion depth and lexical choice, and any performance difference comes from the directed types rather than other differences between the two parsers.

What would settle it

Re-running the comparison after swapping the directionality component alone while keeping every other implementation detail identical, or after re-partitioning SLOG categories to mix position-shift and recursion items, would show whether the reported gap survives.

Figures

Figures reproduced from arXiv: 2607.02307 by Zichao Wei.

**Figure 2.** Figure 2: Accuracy comparison for PP and RC modifier position pairs (subject side vs. indirect-object side). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

read the original abstract

Several SLOG test categories explicitly involve directional distinctions (modifier position shifts, argument extraction positions), yet AM-Parser, the previous SOTA, uses an AM algebra whose operations do not encode direction. We redesign the symbolic backend around CCG directed types (deterministic CKY + single linear decoder, 30K learnable parameters). Under the same BERT-base encoder, the system achieves 75.9$\pm$6.4% LF exact match, surpassing AM-Parser (70.8$\pm$4.3%). Per SLOG's own category groupings, gains are highly directional: the CCG system outperforms AM-Parser on all 5 position-shift categories (+29.9pp), while AM-Parser outperforms on all 6 recursive-depth categories. Replacing the encoder with DeBERTa-v3-large yields 90.7$\pm$4.9%, with the largest encoder gains in recursive-depth categories, complementary to directionality's gains. Directional representations shift the bottleneck from the symbolic layer (AM-Parser's 0% category ceiling) to the neural layer, which improves with encoder upgrades.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CCG directional types improve SLOG position-shift categories by 30 points over AM-Parser under the same encoder, but the comparison does not isolate directionality from the switch to CKY plus linear decoder.

read the letter

The main thing to know is that the paper swaps the symbolic backend for CCG directed types and sees clear gains on the five position-shift categories in SLOG, while AM-Parser stays ahead on the recursive-depth ones. The two improvements look complementary when the encoder is upgraded to DeBERTa.

The work does a clean job of reporting the category split using SLOG's own groupings and showing that the directional lift is large and consistent across those five categories. Keeping the BERT-base encoder fixed for the head-to-head makes the comparison straightforward, and the standard deviations are reported. The observation that encoder upgrades help the recursive categories more is useful and points to different bottlenecks.

The soft spot is the missing ablation. The CCG system uses a deterministic CKY parser and a 30k-parameter linear decoder; AM-Parser uses an undirected algebra. Without an experiment that holds the parser, decoder, and training fixed while toggling only the directed types, the +29.9pp cannot be cleanly attributed to directionality rather than the parser architecture. The paper also takes the SLOG category groupings as given without showing they are orthogonal to recursion depth or lexical factors.

This paper is for people working on semantic parsing and structural generalization. Readers who care about how symbolic direction interacts with neural encoders will find the per-category numbers worth their time. It has enough concrete results on a public benchmark to deserve a serious referee, even though the causal claim about directionality needs tighter controls.

I would send it to review.

Referee Report

3 major / 2 minor

Summary. The paper claims that directionality in symbolic representations, via CCG directed types, is key to structural generalization on position-shift tasks. It redesigns the backend as a deterministic CKY parser plus single linear decoder (30K parameters) and reports 75.9±6.4% LF exact match under BERT-base (vs. AM-Parser's 70.8±4.3%), with +29.9pp gains on all 5 position-shift SLOG categories and complementary losses on recursive-depth categories. Upgrading the encoder to DeBERTa-v3-large reaches 90.7±4.9%, shifting the bottleneck from the symbolic layer to the neural encoder.

Significance. If the attribution of gains specifically to directed types holds after controlling for parser and decoder differences, the result would be significant for compositional generalization work. It supplies a concrete, low-parameter demonstration that directed types address position-shift failures that undirected AM algebras cannot, while showing that encoder upgrades and directionality provide complementary improvements. The category-specific breakdown and explicit parameter count are strengths that make the empirical comparison falsifiable.

major comments (3)

[Abstract] Abstract: the central claim attributes the +29.9pp gain on the five position-shift categories to the addition of directed types, yet the comparison pits a deterministic CKY parser + linear decoder against an undirected AM algebra. No ablation is described that holds the parser architecture, decoder, and training fixed while toggling only the presence of directed types.
[Abstract] Abstract: the claim that 'gains are highly directional' rests on SLOG's category groupings cleanly isolating directional effects, but the manuscript provides no verification (e.g., per-category recursion-depth or lexical statistics) that the groupings are orthogonal to recursion depth or other confounders.
[Abstract] Abstract: concrete numbers with standard deviations and category-specific gains are reported, yet no information is given on how the SLOG categories were constructed, whether any data filtering occurred, or the precise implementation differences between the two symbolic backends beyond the high-level description.

minor comments (2)

The manuscript would benefit from a table or appendix listing the exact SLOG category definitions and any preprocessing steps applied to the benchmark.
Clarify the training details (optimizer, learning rate schedule, number of runs) for the reported means and standard deviations to support reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract and empirical claims. We address each point below and will revise the manuscript to incorporate additional details, verifications, and discussion where feasible.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim attributes the +29.9pp gain on the five position-shift categories to the addition of directed types, yet the comparison pits a deterministic CKY parser + linear decoder against an undirected AM algebra. No ablation is described that holds the parser architecture, decoder, and training fixed while toggling only the presence of directed types.

Authors: We agree that a controlled ablation isolating only the presence/absence of directed types (while fixing parser architecture, decoder, and training) is not present and would provide stronger causal evidence. The current comparison contrasts two complete symbolic backends whose core difference is the use of directed CCG types versus an undirected AM algebra; the CKY parser and linear decoder are tailored to the directed representation. An ablation that retrofits directionality into the AM algebra would require substantial new engineering outside the scope of this work. In revision we will add an explicit limitations paragraph acknowledging this gap and noting that the category-specific pattern (gains confined to the five directional phenomena) remains consistent with the role of directionality. revision: partial
Referee: [Abstract] Abstract: the claim that 'gains are highly directional' rests on SLOG's category groupings cleanly isolating directional effects, but the manuscript provides no verification (e.g., per-category recursion-depth or lexical statistics) that the groupings are orthogonal to recursion depth or other confounders.

Authors: The referee is correct that the manuscript does not supply per-category statistics to verify orthogonality. SLOG categories are defined in the original SLOG benchmark paper by the targeted linguistic phenomenon (position shifts for directional effects versus nesting depth for recursive effects). In the revised version we will add a short appendix table reporting, for each SLOG category, mean recursion depth and basic lexical statistics (type-token ratio, average sentence length) to allow readers to assess potential confounding. revision: yes
Referee: [Abstract] Abstract: concrete numbers with standard deviations and category-specific gains are reported, yet no information is given on how the SLOG categories were constructed, whether any data filtering occurred, or the precise implementation differences between the two symbolic backends beyond the high-level description.

Authors: We will expand the experimental section and add an appendix with: (i) a concise summary of SLOG category construction drawn from the SLOG reference, (ii) explicit confirmation that the standard SLOG splits were used with no additional filtering, and (iii) precise implementation details for both backends, including the CKY chart rules for directed CCG types, the single linear decoder architecture and training objective, and exact parameter counts (30K for the CCG decoder versus the AM-Parser configuration). revision: yes

Circularity Check

0 steps flagged

No circularity: empirical head-to-head on fixed benchmark

full rationale

The paper reports an empirical comparison of a CCG-directed system (deterministic CKY + linear decoder) against AM-Parser on SLOG categories, with exact-match scores and per-category breakdowns. No derivation chain, equations, or first-principles predictions appear; performance numbers are direct outputs of the implemented systems rather than quantities defined from fitted parameters inside the paper. No self-citation load-bearing steps, uniqueness theorems, or ansatzes are invoked to justify the central attribution. The comparison is externally falsifiable on the public benchmark.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based on the abstract alone, the paper relies on standard parsing algorithms and the existing CCG formalism. No free parameters, invented entities, or ad-hoc axioms are mentioned.

axioms (1)

standard math CKY parsing is deterministic and sound for CCG categories
Invoked for the symbolic backend implementation.

pith-pipeline@v0.9.1-grok · 5716 in / 1239 out tokens · 28959 ms · 2026-07-03T14:21:09.005340+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

[1]

Cambridge: Cambridge University Press. John A. Hawkins. 2004. Efficiency and Complexity in Grammars. Oxford, New York: Oxford University Press. Pengcheng He, Jianfeng Gao, and Weizhu Chen

work page 2004
[2]

LLaMA: Open and Efficient Foundation Language Models

DeBERTaV3: Improving DeBERTa Using ELECTRA-style Pre-Training with Gradient-Disen- tangled Embedding Sharing. In The Eleventh Inter national Conference on Learning Representations. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categor - ical Reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Represen tations, ICLR 2017....

work page internal anchor Pith review Pith/arXiv arXiv 2017

[1] [1]

Cambridge: Cambridge University Press. John A. Hawkins. 2004. Efficiency and Complexity in Grammars. Oxford, New York: Oxford University Press. Pengcheng He, Jianfeng Gao, and Weizhu Chen

work page 2004

[2] [2]

LLaMA: Open and Efficient Foundation Language Models

DeBERTaV3: Improving DeBERTa Using ELECTRA-style Pre-Training with Gradient-Disen- tangled Embedding Sharing. In The Eleventh Inter national Conference on Learning Representations. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categor - ical Reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Represen tations, ICLR 2017....

work page internal anchor Pith review Pith/arXiv arXiv 2017