New constraints on cosmological modified gravity theories from anisotropic three-point correlation functions of BOSS DR12 galaxies

We report a new test of modified gravity theories using the large-scale structure of the Universe. This paper is the first attempt to (1) apply a joint analysis of the anisotropic components of galaxy two- and three-point correlation functions (2 and 3PCFs) to actual galaxy data and (2) constrain the nonlinear effects of degenerate higher-order scalar-tensor (DHOST) theories on cosmological scales. Applying this analysis to the Baryon Oscillation Spectroscopic Survey (BOSS) data release 12, we obtain the lower bounds of $-1.655 < \xi_{\rm t}$ and $-0.504 < \xi_{\rm s}$ at the $95\%$ confidence level on the parameters characterising the time evolution of the tidal and shift terms of the second-order velocity field. These constraints are consistent with GR predictions of $\xi_{\rm t}=15/1144$ and $\xi_{\rm s}=0$. Moreover, they represent a $35$-fold and $20$-fold improvement, respectively, over the joint analysis with only the isotropic 3PCF. We ensure the validity of our results by investigating various quantities, including theoretical models of the 3PCF, window function corrections, cumulative ${\rm S/N}$, Fisher matrices, and statistical scattering effects of mock simulation data. We also find statistically significant discrepancies between the BOSS data and the Patchy mocks for the 3PCF measurement. Finally, we package all of our 3PCF analysis codes under the name \textsc{HITOMI} and make them publicly available so that readers can reproduce all the results of this paper and easily apply them to ongoing future galaxy surveys.