REVIEW 3 major objections 7 minor 39 references
Complex SGD converges without analyticity, recovers superoscillations
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · glm-5.2
2026-07-04 16:14 UTC pith:UDGTCHTA
load-bearing objection Complex SGD convergence proofs are sound but incremental; the real issue is a missing 1/σ_k factor in the directional bias bound the 3 major comments →
Complex Stochastic Gradient Descent and Directional Bias in Reproducing Kernel Hilbert Spaces
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central object is the Wirtinger gradient, defined as twice the Wirtinger derivative, which serves as a replacement for the classical gradient when the objective function accepts complex inputs but need not be analytic. The authors establish that, under assumptions nearly identical to the real-valued SGD setting plus one additional assumption (stationarity of minima), this gradient yields the same convergence guarantees as classical SGD. The mechanism is straightforward: by using the real inner product (the real part of the standard complex inner product) in place of the complex inner product throughout the standard SGD convergence proofs, and by explicitly assuming that the Wirtinger梯度 (
What carries the argument
The Wirtinger gradient (2∇_z f for non-analytic f, 2∇_{z̄} f for analytic f), the real inner product ⟨z,w⟩_R := Re(⟨z,w⟩), Assumption 4 (stationarity of minima), and the connection between SGD and the Random Kaczmarz method for directional bias analysis.
Load-bearing premise
The Wirtinger gradient is not a true gradient in the classical sense: it does not automatically vanish at minima of real-valued functions over complex domains. The paper imposes this vanishing as an explicit assumption (Assumption 4) rather than deriving it, and verifies it only for the least-squares objective. The generality of the convergence results depends on whether this assumption holds broadly for other objectives, particularly non-convex ones.
What would settle it
Find a real-valued, convex objective function over a complex domain where the Wirtinger gradient does not vanish at a minimum, violating Assumption 4 and breaking the convergence guarantees.
If this is right
- Complex-valued neural networks can use SGD with the same convergence guarantees as real-valued networks, without splitting complex parameters into real and imaginary parts or requiring analyticity of the loss.
- Directional bias in the small-step-size regime, previously known only for real-valued SGD, persists in the complex setting: iterates converge preferentially along the smallest singular-value direction of the data matrix.
- Optimal functions in complex RKHS (Fock space, Hardy space, RBF spaces) that were previously obtainable only through closed-form analytic construction can be numerically recovered via stochastic iteration to machine precision.
- The same proof strategy—replacing the inner product and imposing stationarity of minima—may extend other real-valued optimization results to the complex setting, including more refined SGD variants.
Where Pith is reading between the lines
- If Assumption 4 fails for some non-convex objective encountered in complex-valued neural networks, the convergence guarantees would not hold, potentially limiting applicability to well-behaved losses such as least-squares-type objectives.
- The directional bias result suggests that complex-valued models trained with small-step SGD may inherit an implicit bias toward low-variance directions, which could affect generalization behavior in ways analogous to the real-valued case.
- The Mittag-Leffler–Fock space, mentioned by the authors as a natural next setting, would provide a test of whether the method extends to RKHS with more intricate kernel structure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a complex variant of Stochastic Gradient Descent (SGD) based on the Wirtinger gradient, providing convergence guarantees (Theorems 4–6) under assumptions paralleling the real-valued setting, without requiring analyticity of the objective. The proofs are direct adaptations of classical SGD proofs from [21], with the real inner product ⟨·,·⟩_R replacing the standard inner product. The paper also extends directional bias results for SGD in kernel regression (Corollary 1) from the real to the complex setting, borrowing analysis from the Random Kaczmarz method [28]. Finally, the paper provides numerical experiments demonstrating recovery of superoscillation functions in the Fock space and Blaschke products in the Hardy space, using analytically known optima from [13] as targets.
Significance. The paper provides a clean and self-contained treatment of complex SGD with convergence guarantees that avoid analyticity assumptions, which is a useful contribution for practitioners working with complex-valued neural networks and complex RKHS problems. The verification that the least-squares objective satisfies all assumptions (Section D.2) is thorough. The numerical experiments are well-designed, using independently established analytical targets (Theorems 8 and 9 from [13]) as ground truth, and the recovery to near machine precision in both Fock and Hardy space settings is convincing. The connection to the Random Kaczmarz method for directional bias analysis is a natural and interesting bridge. However, the convergence proofs are straightforward adaptations of existing results, and the directional bias result contains a quantitative error (see Major Comment 1) that weakens one of the paper's claimed contributions.
major comments (3)
- Corollary 1 (and its proof in Appendix C): The error bound for the inconsistent-system case is incorrect. The recurrence yields E[δ_{t+1}] = α·δ_t + β, where α = (1 − ηnσ_k²/m) and β = (ηnσ_k/m)⟨ε, u_k⟩. Iterating gives δ_{t+1} = α^{t+1}δ_0 + β(1−α^{t+1})/(1−α). Since |β| ≤ (ηnσ_k/m)||ε|| and |1−α| = ηnσ_k²/m, the steady-state error term is bounded by ||ε||/σ_k, not ||ε|| as stated in Corollary 1. This factor of 1/σ_k is significant: the directional bias result is most interesting for the smallest singular value σ_min, where 1/σ_min could be large, potentially inflating the error floor well beyond what the paper claims. The consistent-system case (where ε = 0) is unaffected and remains correct. The bound in Corollary 1 should be corrected to ||ε||/σ_k (times (1−|α|^{t+1})), and the qualitative discussion should be updated accordingly.
- Section 5.1, Assumptions 4–7: The paper presents Assumption 4 (stationarity of minima) as a load-bearing assumption needed because 'the gradient is not truly a gradient in the usual sense.' However, for any real-valued function F: C^n → R that is differentiable in the real sense, the real gradient on R^{2n} vanishes at local minima, which implies both Wirtinger derivatives ∂F/∂z and ∂F/∂z̄ vanish. Since the paper defines ∇F := 2∇_{z̄}F for non-analytic F, Assumption 4 holds automatically—it is a theorem, not an assumption. The same reasoning applies to Assumptions 5–7: for real-valued objectives, the Wirtinger gradient is a representation of the real gradient on R^{2n}, so the descent lemma, convexity, and strong convexity all follow from their real-valued counterparts. The paper should acknowledge this explicitly, as it affects how the contribution is framed (the results are more direct
- Section 5.1, Assumptions 4–7 (continued): The paper should acknowledge that these assumptions are automatically satisfied for real-valued differentiable objectives, rather than presenting them as additional conditions that must be imposed. This does not invalidate the proofs, but it strengthens the paper's positioning and clarifies that the convergence results are essentially immediate consequences of the real-valued theory via the Wirtinger calculus correspondence. The verification in Section D.2 for least-squares is still valuable as a concrete example, but the general principle should be stated.
minor comments (7)
- Section 2: The definition of the Wirtinger derivative for the analytic case appears to have a sign error. The paper writes ∂_z f = (1/2)(∂f/∂x − i∂f/∂y) when f is analytic, but the standard convention for ∂f/∂z̄ (which is what should vanish for analytic functions) is (1/2)(∂f/∂x + i∂f/∂y). Please verify the conventions are consistent throughout.
- Section 6, Corollary 1 statement: The bound uses ||ε|| without defining the norm explicitly (presumably the Euclidean norm on C^m). This should be stated.
- Appendix C, proof of Corollary 1: The line '⟨a_i^*, v_k⟩ is the conjugate of the i-th entry of Av_k' should be clarified—Av_k = σ_k u_k, so the i-th entry of Av_k is σ_k times the i-th entry of u_k, and ⟨a_i^*, v_k⟩ = conj(a_i · v_k) = conj((Av_k)_i).
- Section 7.1.1, Equation (13): The normalized update rule is introduced without full justification of why it preserves convergence guarantees. The paper states it is 'purely to improve numerical stability' and that results are 'virtually identical,' but a brief remark on why the normalization does not affect the theoretical analysis would be helpful.
- Figure 1a caption: The text mentions α^{(100000)} but the figure label says 'Iteration' up to 10^5. Please ensure consistency in iteration counts between text and figures.
- Section D.2: The statement 'Assumption 6 follows similarly, except for our choice of f_j, we have σ_min = 0' is somewhat terse. A brief expansion would help readers verify that convexity (not strong convexity) of each f_j is indeed satisfied.
- References: [13] is cited as a 2026 preprint. Please ensure the citation is complete and accessible to readers.
Simulated Author's Rebuttal
We thank the referee for a careful reading and for identifying a genuine quantitative error in Corollary 1, as well as a framing issue regarding Assumptions 4–7. Both points are well-taken and will be addressed in the revision.
read point-by-point responses
-
Referee: Major Comment 1: The error bound in Corollary 1 for the inconsistent-system case is incorrect. The steady-state error should be ||ε||/σ_k, not ||ε||.
Authors: The referee is correct. Tracing through the proof in Appendix C: the recurrence is E[δ_{t+1}] = α·δ_t + β with α = (1 − ηnσ_k²/m) and β = (ηnσ_k/m)⟨ε, u_k⟩. Iterating gives δ_{t+1} = α^{t+1}δ_0 + β(1−α^{t+1})/(1−α). The steady-state term satisfies |β/(1−α)| = |⟨ε, u_k⟩|/σ_k ≤ ||ε||/σ_k. In the proof, we incorrectly bounded the geometric sum Σ_{i=0}^{t} (ηnσ_k/m)|α|^{t-i}·||ε|| by ||ε||, when in fact it equals (||ε||/σ_k)(1−|α|^{t+1}). The consistent-system case (ε = 0) is unaffected. We will correct Corollary 1 to state the bound as (||ε||/σ_k)(1−|α|^{t+1}) and update the qualitative discussion to note that the error floor depends on 1/σ_k, which is most significant for the smallest singular value. We thank the referee for catching this. revision: yes
-
Referee: Major Comment 2: Assumptions 4–7 are automatically satisfied for real-valued differentiable objectives, since the Wirtinger gradient is a representation of the real gradient on R^{2n}. The paper should acknowledge this rather than presenting them as additional conditions.
Authors: The referee is right that for a real-valued differentiable function F: C^n → R, the Wirtinger gradient ∇F := 2∇_{z̄}F is a faithful representation of the real gradient on R^{2n} under the identification C^n ≅ R^{2n}. Consequently, Assumption 4 (stationarity of minima) is indeed a theorem rather than an additional hypothesis: if F has a local minimum at z*, the real gradient vanishes, which implies both Wirtinger derivatives vanish. The same logic applies to Assumptions 5–7: the descent lemma, convexity, and strong convexity in the Wirtinger formulation follow directly from their real-valued counterparts because ⟨∇F(z), w−z⟩_R coincides with the real inner product of the corresponding real gradient with w−z in R^{2n}. We will revise Section 5.1 to state this principle explicitly and reframe Assumptions 4–7 as properties that hold automatically for real-valued differentiable objectives, rather than as additional conditions. The verification in Section D.2 for the least-squares objective remains valuable as a concrete instantiation. This reframing does not affect any of the proofs but does clarify that the convergence results are immediate consequences of the real-valued theory via the Wirtinger calculus correspondence, which is a point we should have made from the outset. revision: yes
Circularity Check
No significant circularity; derivation chain rests on external proofs and parameter-free analytical targets
full rationale
The paper's three main components are each grounded in external sources rather than self-referential loops. (1) Theorems 4–6 are explicit adaptations of classical SGD convergence proofs from [21] (Garrigos and Gower, external authors), with the only modification being the use of ⟨·,·⟩_R and Assumptions 4–7. The proofs in Appendix B follow the standard structure and do not reduce to their own inputs. (2) Corollary 1 (directional bias) is attributed to Steinerberger [28] (external) with minor modifications for the complex setting. The skeptic correctly identifies a potential error in the bound (should be ||ε||/σ_k rather than ||ε||), but this is a correctness issue, not circularity. (3) The numerical experiments use analytical targets from [13] (self-citation by first author Alpay), but Theorems 8 and 9 of [13] provide explicit, parameter-free formulas for the optimal coefficient vectors (e.g., α* = (C_0(n,a),...,C_n(n,a))^T with C_j given by closed-form binomial expressions). The experiments verify that complex SGD recovers these known analytical solutions to machine precision—this is standard validation against independently derived targets, not a fitted input renamed as a prediction. The self-citation [13] is load-bearing for the experiments but provides parameter-free analytical results, not fitted values, so it does not raise the circularity score. Score 1 reflects the presence of a load-bearing self-citation that is nonetheless independent of the present paper's algorithmic claims.
Axiom & Free-Parameter Ledger
free parameters (5)
- Step size η_t
- L (Lipschitz constant)
- μ (strong convexity constant)
- λ (regularization parameter) =
1
- n (problem dimension) =
40 (Fock), 50 (Hardy)
axioms (5)
- ad hoc to paper Assumption 4: Minima of F are stationary points of the complex gradient
- domain assumption Assumption 5: L-Lipschitz alternative for complex gradient
- domain assumption Assumption 6: Convexity alternative for complex gradient
- domain assumption Assumption 7: μ-strong convexity of F
- domain assumption Complex representer theorem (Theorem 10)
read the original abstract
Stochastic Gradient Descent (SGD) is a known stochastic iterative method popular for large-scale convex optimization problems due to its simple implementation and scalability. Some objectives, such as those found in complex-valued neural networks, benefit from updates like in SGD and Gradient Descent (GD) with a newly defined ``gradient'' that allows for complex parameters. This complex variant of the SGD/GD methods has already been proposed, but convergence guarantees without analyticity constraints have not yet been provided. We propose a variant of SGD (complex SGD) that allows for complex parameters, and we provide convergence guarantees under assumptions that parallel those from the real setting. Notably, these results extend to GD as well, and with the same set of assumptions, we confirm that some directional bias results extend from the real to the complex setting for kernel regression problems. We provide empirical results demonstrating the efficacy of the complex SGD in kernel regression problems utilizing complex reproducing kernel Hilbert spaces. In particular, we demonstrate we may recover superoscillation functions and Blaschke products from the Fock Space and Hardy Space, respectively, as the optimal functions for a particular choice of a loss function.
Reference graph
Works this paper leans on
-
[1]
M. F. Amin, M. I. Amin, A. Y. H. Al-Nuaimi, and K. Murase, “Wirtinger calculus based gradient descent and levenberg-marquardt learning algorithms in complex-valued neural networks,” in Neural Information Processing, B.-L. Lu, L. Zhang, and J. Kwok, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 550–559
work page 2011
-
[2]
A complex gradient operator and its application in adaptive array theory ,
D. H. Brandwood, “A complex gradient operator and its application in adaptive array theory ,”IEE Proceedings F: Communications, Radar and Signal Processing, vol. 130, no. 1, pp. 11–16, 1983
work page 1983
-
[3]
The Complex Gradient Operator and the CR-Calculus
K. Kreutz-Delgado, “The complex gradient operator and the CR-calculus,” arXiv preprint arXiv:0906.4835, 2009
work page internal anchor Pith review Pith/arXiv arXiv 2009
-
[4]
B. Widrow, J. McCool, and M. Ball, “The complex LMS algorithm,”Proceedings of the IEEE, vol. 63, no. 4, pp. 719–720, 1975
work page 1975
-
[5]
A. van den Bos, “Complex gradient and hessian,” IEE Proceedings – Vision, Image and Signal Processing, vol. 141, no. 6, pp. 380–382, 1994
work page 1994
-
[6]
Complex-valued matrix differentiation: Techniques and key results,
A. Hjørungnes and D. Gesbert, “Complex-valued matrix differentiation: Techniques and key results,” IEEE Transactions on Signal Processing, vol. 55, no. 6, pp. 2740–2746, 2007
work page 2007
-
[7]
A short tutorial on wirtinger calculus with applications in quantum information,
K. Koor, Y. Qiu, L. C. Kwek, and P. Rebentrost, “A short tutorial on wirtinger calculus with applications in quantum information,” arXiv preprint arXiv:2312.04858, 2023
-
[8]
Convergence analysis of an augmented algorithm for fully complex-valued neural networks,
D. Xu, H. Zhang, and D. P. Mandic, “Convergence analysis of an augmented algorithm for fully complex-valued neural networks,” Neural Networks, vol. 69, pp. 44–50, 2015
work page 2015
-
[9]
Convergence analysis of fully complex backpropagation algorithm based on wirtinger calculus,
H. Zhang, X. Liu, D. Xu, and Y. Zhang, “Convergence analysis of fully complex backpropagation algorithm based on wirtinger calculus,”Cognitive Neurodynamics, vol. 8, no. 3, pp. 261–266, 2014
work page 2014
-
[10]
Direction matters: On the implicit bias of stochastic gradient descent with moderate learning rate,
J. Wu, D. Zou, V. Braverman, and Q. Gu, “Direction matters: On the implicit bias of stochastic gradient descent with moderate learning rate,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=3X64RLgzY6O
work page 2021
-
[11]
The directional bias helps stochastic gradient descent to generalize in kernel regression models,
Y. Luo, X. Huo, and Y. Mei, “The directional bias helps stochastic gradient descent to generalize in kernel regression models,” in 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022, pp. 678–683
work page 2022
-
[12]
Some results on Tchebycheffian spline functions,
G. S. Kimeldorf and G. Wahba, “Some results on Tchebycheffian spline functions,” Journal of Mathematical Analysis and Applications, vol. 33, no. 1, pp. 82–95, 1971
work page 1971
-
[13]
N. Alpay , A. De Martino, and K. Diki, “Representer theorem in complex reproducing kernel hilbert spaces with applications to fock and hardy spaces and superoscillations,” 2026
work page 2026
-
[14]
Superoscillations and physical applications,
A. N. Jordan, J. C. Howell, N. Vamivakas, and E. Karimi, “Superoscillations and physical applications,” in Operator Theory, D. Alpay , I. Sabadini, and F. Colombo, Eds. Basel: Springer, 2025
work page 2025
-
[15]
Superoscillation: from physics to optical applications,
G. Chen, Z.-Q. Wen, and C.-W. Qiu, “Superoscillation: from physics to optical applications,” Light: Science & Applications, vol. 8, no. 1, p. 56, 2019
work page 2019
-
[16]
Multivariable functional interpolation and adaptive networks,
D. S. Broomhead and D. Lowe, “Multivariable functional interpolation and adaptive networks,” Complex Systems, vol. 2, pp. 321–355, 1988
work page 1988
-
[18]
Analogues of finite blaschke products as inner functions,
C. Felder and T. Le, “Analogues of finite blaschke products as inner functions,” Bull. Lond. Math. Soc., vol. 54, no. 4, pp. 1197– 1219, 2022
work page 2022
-
[19]
S. R. Garcia, J. Mashreghi, and W. T. Ross, Finite Blaschke Products and Their Connections. Cham: Springer, 2018, xix+328 pp. ISBN: 978-3-319-78246-1; 978-3-319-78247-8
work page 2018
-
[20]
Mashreghi, Blaschke products and their applications
J. Mashreghi, Blaschke products and their applications. Springer, 2013
work page 2013
-
[21]
Handbook of convergence theorems for (stochastic) gradient methods,
G. Garrigos and R. M. Gower, “Handbook of convergence theorems for (stochastic) gradient methods,” 2024. [Online]. Available: https://arxiv.org/abs/2301.11235
-
[22]
Non-asymptotic analysis of stochastic approximation algorithms for machine learning,
E. Moulines and F. Bach, “Non-asymptotic analysis of stochastic approximation algorithms for machine learning,” Advances in neural information processing systems, vol. 24, 2011
work page 2011
-
[23]
Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm,
D. Needell, R. Ward, and N. Srebro, “Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm,” Advances in neural information processing systems, vol. 27, 2014
work page 2014
-
[24]
An extension of the complex–real (c–r) calculus to the bicomplex setting, with applications,
D. Alpay , K. Diki, and M. Vajiac, “An extension of the complex–real (c–r) calculus to the bicomplex setting, with applications,” Mathematische Nachrichten, vol. 297, no. 2, pp. 454–481, 2024
work page 2024
-
[25]
Phase retrieval via wirtinger flow: Theory and algorithms,
E. J. Candès, X. Li, and M. Soltanolkotabi, “Phase retrieval via wirtinger flow: Theory and algorithms,” IEEE Transactions on Information Theory, vol. 61, no. 4, pp. 1985–2007, 2015
work page 1985
-
[26]
A Randomized Kaczmarz Algorithm with Exponential Convergence,
T. Strohmer and R. Vershynin, “A Randomized Kaczmarz Algorithm with Exponential Convergence,” Journal of Fourier Analysis and Applications, vol. 15, 03 2007
work page 2007
-
[27]
Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm,
D. Needell, N. Srebro, and R. Ward, “Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm,” Mathematical Programming, vol. 155, no. 1, pp. 549–573, Jan 2016. [Online]. Available: https://doi.org/10.1007/s10107-015-0864-7
-
[28]
Randomized kaczmarz converges along small singular vectors,
S. Steinerberger, “Randomized kaczmarz converges along small singular vectors,” SIAM Journal on Matrix Analysis and Applications, vol. 42, pp. 608–615, 04 2021
work page 2021
-
[29]
Y. Aharonov, F. Colombo, I. Sabadini, D. Struppa, and J. Tollaksen,The mathematics of superoscillations. American Mathematical Society , 2017, vol. 247, no. 1174
work page 2017
-
[30]
Fractional supershifts and their associated cauchy evolution problems,
N. Alpay , “Fractional supershifts and their associated cauchy evolution problems,”arXiv preprint arXiv:2601.11829, 2026
-
[31]
Superoscillations and fock spaces,
D. Alpay , F. Colombo, K. Diki, I. Sabadini, and D. C. Struppa, “Superoscillations and fock spaces,”Journal of Mathematical Physics, vol. 64, no. 9, 2023
work page 2023
-
[32]
B. Schölkopf and A. J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA: MIT Press, 2002. 26
work page 2002
-
[33]
Kernel methods in machine learning,
T. Hofmann, B. Schölkopf, and A. J. Smola, “Kernel methods in machine learning,” The Annals of Statistics, vol. 36, no. 3, pp. 1171–1220, 2008
work page 2008
-
[34]
C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press, 2006
work page 2006
-
[35]
Theory of hp spaces academic press,
P. L. Duren, “Theory of hp spaces academic press,”New York, 1970
work page 1970
-
[36]
New York: Academic Press, 1970
——, Theory of H p spaces. New York: Academic Press, 1970
work page 1970
-
[37]
Rudin, Real and complex analysis, 3rd ed
W. Rudin, Real and complex analysis, 3rd ed. New York: McGraw-Hill Book Co., 1987
work page 1987
-
[38]
A new characterization of the hardy space and of other hilbert spaces of analytic functions,
N. Alpay , “A new characterization of the hardy space and of other hilbert spaces of analytic functions,” Istanbul Journal of Mathematics, vol. 1, no. 1, pp. 1–11, 2023
work page 2023
-
[39]
Bedrosian identity in blaschke product case,
P. Cerejeiras, C. Qiuhui, and U. Kaehler, “Bedrosian identity in blaschke product case,”Complex Anal. Oper. Theory, vol. 6, no. 1, pp. 275–300, 2012
work page 2012
-
[40]
Saitoh, Theory of Reproducing Kernels and Its Applications, ser
S. Saitoh, Theory of Reproducing Kernels and Its Applications, ser. Pitman Research Notes in Mathematics Series. Harlow: Longman Scientific & Technical, 1988, vol. 189, co-published with John Wiley & Sons, New York. (NA) DEPARTMENT OFMATHEMATICS, UNIVERSITY OFCALIFORNIA, IRVINE, IRVINE, CA 92697, USA Email address:nalpay@uci.edu (EB) DEPARTMENT OFMATHEMAT...
work page 1988
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.