pith. sign in

arxiv: 2605.25527 · v1 · pith:2KIQ3J55new · submitted 2026-05-25 · 💻 cs.LG · cs.CE

DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading

Sayak Charabarty , Souradip Pal This is my paper

Pith reviewed 2026-06-29 22:32 UTC · model grok-4.3

classification 💻 cs.LG cs.CE
keywords reinforcement learninghigh-frequency tradinglimit order bookpolicy optimizationorder flowPPOgroup normalization
0
0 comments X

The pith

Group-aware policy optimization outperforms Q-learning in order-book trading backtests.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines reinforcement learning for high-frequency trading on limit order books by pairing an order-flow state model with policy-gradient methods. It deploys vanilla PPO along with group-normalized variants GRPO and GSPO that incorporate downside-aware shaping, rather than relying on tabular Q-learning. In backtests on AMZN, AAPL, and GOOG under a spread-scaled reward setup, the new policies produce higher net average PnL, improved profitability, and reduced drawdowns relative to the Q-learning baseline. The work concludes that order-flow signals provide an adequate state representation and that group-aware PPO surrogates are preferable to value-based methods.

Core claim

Pairing an order-flow-based state model with policy-gradient methods that use group-normalized updates and downside-aware shaping yields higher net average PnL, profitability, and lower drawdowns than tabular Q-learning on backtests of AMZN, AAPL, and GOOG under simplified spread-scaled rewards.

What carries the argument

Group-aware PPO surrogates (GRPO and GSPO) that apply group-normalized updates and downside-aware shaping to an order-flow state model for directional trading decisions.

If this is right

  • Order-flow signals serve as a sufficient state representation for effective policy reinforcement learning in limit order book environments.
  • Group-aware policy methods deliver measurable gains in net PnL and drawdown control over value-based baselines under the tested reward structure.
  • Policy-gradient approaches with group normalization become viable alternatives to Q-learning for high-frequency directional trading tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The group-normalization step may stabilize learning in other financial time-series control problems beyond single-asset directional trades.
  • Testing the same policies on additional assets or longer horizons would clarify whether the observed improvements generalize past the three stocks examined.
  • Replacing the simplified reward with a full transaction-cost model could expose whether the group-aware advantage survives more realistic execution conditions.

Load-bearing premise

The simplified backtesting setup based on spread-scaled rewards accurately captures real-world trading frictions, execution quality, and market impact for the tested assets.

What would settle it

A follow-up experiment that replaces the spread-scaled reward with a simulator including realistic slippage, latency, and market impact and finds that the reported PnL and drawdown gains disappear or reverse.

Figures

Figures reproduced from arXiv: 2605.25527 by Sayak Charabarty, Souradip Pal.

Figure 1
Figure 1. Figure 1: Held-out evaluation plots for GOOG showing the trade-PnL histogram, equity curve (scaled by [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Held-out evaluation plots for AMZN showing the trade-PnL histogram, equity curve (scaled by [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Held-out evaluation plots for AAPL showing the trade-PnL histogram, equity curve (scaled by [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
read the original abstract

This paper studies reinforcement learning for high-frequency trading on limit order books by pairing an Order-Flow-based state model with policy-gradient methods. Instead of value-based RL techniques like tabular Q-learning, our approach deploys policy-based methods like vanilla PPO and DeepSeekMath-inspired variants like GRPO and GSPO, that use group-normalized updates and downside-aware shaping. On backtests with financial assets AMZN, AAPL, and GOOG under a simplified backtesting setup based on spread-scaled rewards, these new policies improve net average PnL, profitability, and drawdown over the Q-Learning baseline. Our results show that (1) Order-Flow signals are an adequate state for policy RL and (2) group-aware PPO surrogates are preferable over value-based baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper studies reinforcement learning for high-frequency directional trading on limit order books, pairing an order-flow-based state representation with policy-gradient methods. It deploys vanilla PPO along with DeepSeekMath-inspired group-aware variants (GRPO and GSPO) that incorporate group-normalized updates and downside-aware reward shaping. On backtests for AMZN, AAPL, and GOOG under a simplified spread-scaled reward setup, the new policies are reported to improve net average PnL, profitability, and drawdown relative to a tabular Q-Learning baseline. The authors conclude that order-flow signals suffice as states for policy RL and that group-aware PPO surrogates are preferable to value-based methods.

Significance. If the reported ranking of policies survives more realistic execution modeling, the work would provide evidence that group-normalized policy optimization can outperform standard value-based RL in HFT settings. The adaptation of group-relative techniques from mathematical reasoning models to trading constitutes a modest methodological contribution.

major comments (1)
  1. [Backtesting / Evaluation] The central empirical claim—that GRPO/GSPO variants improve net PnL, profitability, and drawdown over Q-Learning—rests entirely on results from the 'simplified backtesting setup based on spread-scaled rewards.' This modeling choice omits partial fills, latency, adverse selection, temporary/permanent market impact, and realistic order execution. Because the performance ordering versus the baseline can reverse once these frictions are included, the claim that group-aware PPO surrogates are preferable is not yet supported by the evaluation.
minor comments (2)
  1. [Abstract] The abstract asserts quantitative improvements without supplying any numerical values, confidence intervals, or statistical tests; this should be remedied for clarity.
  2. [Methods] Provide explicit equations or pseudocode for the group-normalization step and downside-aware shaping used in GRPO and GSPO so that the variants can be reproduced.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed feedback on our work. The primary concern centers on the simplified nature of our backtesting evaluation, which we address directly below.

read point-by-point responses

  1. Referee: [Backtesting / Evaluation] The central empirical claim—that GRPO/GSPO variants improve net PnL, profitability, and drawdown over Q-Learning—rests entirely on results from the 'simplified backtesting setup based on spread-scaled rewards.' This modeling choice omits partial fills, latency, adverse selection, temporary/permanent market impact, and realistic order execution. Because the performance ordering versus the baseline can reverse once these frictions are included, the claim that group-aware PPO surrogates are preferable is not yet supported by the evaluation.

    Authors: We agree that the evaluation relies on a simplified backtesting setup that abstracts away important execution frictions including partial fills, latency, adverse selection, and market impact. This design choice was made to focus on the relative performance of the order-flow state representation and the group-aware policy optimization methods in a controlled environment. We recognize that the observed ranking of GRPO/GSPO over Q-Learning could change under more realistic execution modeling. Accordingly, we will revise the manuscript to (i) more explicitly qualify all performance claims as holding under the spread-scaled reward backtest, (ii) strengthen the limitations section with a dedicated discussion of omitted frictions, and (iii) outline concrete next steps for incorporating realistic order execution. These changes will prevent overgeneralization while preserving the contribution of the controlled comparison. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical application of existing RL methods

full rationale

The paper applies policy-gradient methods (vanilla PPO, GRPO, GSPO) drawn from external sources like DeepSeekMath to an order-book state model and reports comparative backtest metrics on AMZN/AAPL/GOOG. No derivation chain, equations, or self-referential definitions appear; group normalization and reward shaping are described as adopted techniques rather than fitted quantities renamed as predictions. Results rest on experimental evaluation against a Q-learning baseline under an explicitly simplified setup, which constitutes an external benchmark rather than a closed loop. No self-citation load-bearing steps or uniqueness theorems are invoked. This is a standard empirical RL paper whose central claims are falsifiable via replication of the backtests.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract alone supplies no concrete free parameters, axioms, or invented entities; full methods section would be required to populate this ledger.

pith-pipeline@v0.9.1-grok · 5663 in / 1062 out tokens · 25322 ms · 2026-06-29T22:32:52.923712+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 4 canonical work pages · 3 internal anchors

  1. [1]

    Limit order books.Quantitative Finance, 13(11):1709–1742, 2013

    Martin D Gould, Mason A Porter, Stacy Williams, Mark McDonald, Daniel J Fenn, and Sam D Howison. Limit order books.Quantitative Finance, 13(11):1709–1742, 2013

    2013

  2. [2]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. 8 APREPRINT- MAY26, 2026

    2024

  3. [3]

    Jaddu and Paul A

    Koti S. Jaddu and Paul A. Bilokon. Combining deep learning on order books with reinforcement learning for profitable trading, 2023

    2023

  4. [4]

    The cost of transacting.The quarterly journal of economics, 82(1):33–53, 1968

    Harold Demsetz. The cost of transacting.The quarterly journal of economics, 82(1):33–53, 1968

    1968

  5. [5]

    Market microstructure.Journal of financial Economics, 3(3):257–275, 1976

    Mark B Garman. Market microstructure.Journal of financial Economics, 3(3):257–275, 1976

    1976

  6. [6]

    The pricing of security dealer services: An empirical study of nasdaq stocks.The journal of finance, 33(4):1153–1172, 1978

    Hans R Stoll. The pricing of security dealer services: An empirical study of nasdaq stocks.The journal of finance, 33(4):1153–1172, 1978

    1978

  7. [7]

    High-frequency trading in a limit order book.Quantitative Finance, 8(3):217–224, 2008

    Marco Avellaneda and Sasha Stoikov. High-frequency trading in a limit order book.Quantitative Finance, 8(3):217–224, 2008

    2008

  8. [8]

    On dealer markets under competition.The Journal of Finance, 35(2):259–267, 1980

    Thomas Ho and Hans R Stoll. On dealer markets under competition.The Journal of Finance, 35(2):259–267, 1980

    1980

  9. [9]

    Optimal high-frequency trading with limit and market orders.Quantitative Finance, 13(1):79–94, 2013

    Fabien Guilbaud and Huyen Pham. Optimal high-frequency trading with limit and market orders.Quantitative Finance, 13(1):79–94, 2013

    2013

  10. [10]

    An electronic market-maker

    Nicholas Tung Chan and Christian Shelton. An electronic market-maker. 2001

    2001

  11. [11]

    John Wiley & Sons, 1998

    Maureen O’hara.Market microstructure theory. John Wiley & Sons, 1998

    1998

  12. [12]

    A new dynamically changing attack on review fraud systems and a dynamically changing ensemble defense

    Youzhi Zhang, Sayak Chakrabarty, Rui Liu, Andrea Pugliese, and VS Subrahmanian. A new dynamically changing attack on review fraud systems and a dynamically changing ensemble defense. in 2022 ieee intl conf on dependable, autonomic and secure computing, intl conf on pervasive intelligence and computing, intl conf on cloud and big data computing, intl conf ...

    2022

  13. [13]

    Time-Constrained Recommendations: Reinforcement Learning Strategies for E-Commerce

    Sayak Chakrabarty and Souradip Pal. Time-constrained recommendations: Reinforcement learning strategies for e-commerce.arXiv preprint arXiv:2512.13726, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity

    Riddhiman Bhattacharyya, Sayak Chakrabarty, and Imon Banerjee. Adaptive estimation and optimal control in offline contextual mdps without stationarity.arXiv preprint arXiv:2605.03393, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  15. [15]

    Small resamples, sharp guarantees: Convergence rates for resampled studentized quantile estimators.Advances in Neural Information Processing Systems, 38:62434–62482, 2026

    Imon Banerjee and Sayak Chakrabarty. Small resamples, sharp guarantees: Convergence rates for resampled studentized quantile estimators.Advances in Neural Information Processing Systems, 38:62434–62482, 2026

    2026

  16. [16]

    Deep direct reinforcement learning for financial signal representation and trading.IEEE transactions on neural networks and learning systems, 28(3):653–664, 2016

    Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. Deep direct reinforcement learning for financial signal representation and trading.IEEE transactions on neural networks and learning systems, 28(3):653–664, 2016

    2016

  17. [17]

    Model-based reinforcement learning for predictions and control for limit order books,

    Haoran Wei, Yuanbo Wang, Lidia Mangu, and Keith Decker. Model-based reinforcement learning for predictions and control for limit order books.arXiv preprint arXiv:1910.03743, 2019

    work page arXiv 1910

  18. [18]

    Reinforcement learning for optimized trade execution

    Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimized trade execution. In Proceedings of the 23rd international conference on Machine learning, pages 673–680, 2006

    2006

  19. [19]

    Deeptrader: A deep reinforcement learning approach for risk-return balanced portfolio management with market conditions embedding

    Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. Deeptrader: A deep reinforcement learning approach for risk-return balanced portfolio management with market conditions embedding. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 643–650, 2021

    2021

  20. [20]

    Portfolio management using reinforcement learning.Stanford University, 2016

    Olivier Jin and Hamza El-Saawy. Portfolio management using reinforcement learning.Stanford University, 2016

    2016

  21. [21]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    LOBSTER: Limit order book reconstruction system.SSRN Electron

    Ruihong Huang and Tomas Polak. LOBSTER: Limit order book reconstruction system.SSRN Electron. J., 2011. 9

    2011