Neutron scattering investigation of proposed Kosterlitz-Thouless transitions in the triangular-lattice Ising antiferromagnet TmMgGaO4

The transverse-field Ising model on the triangular lattice is expected to host an intermediate finite-temperature Kosterlitz-Thouless (KT) phase through a mapping of the spins on each triangular unit to a complex order parameter. TmMgGaO$_4$ is a candidate material to realize such physics due to the non-Kramers nature of Tm$^{3+}$ ion and the resulting two-singlet single-ion ground state. Using inelastic neutron scattering, we confirm this picture by determining the leading parameters of the low-energy effective Hamiltonian of TmMgGaO$_4$. Subsequently, we track the predicted KT phase and related transitions by inspecting the field and temperature dependence of the ac susceptibility. We further probe the spin correlations in both reciprocal space and real space via single crystal neutron diffraction and magnetic total scattering techniques, respectively. Magnetic pair distribution function analysis provides evidence for the formation of vortex-antivortex pairs that characterize the proposed KT phase around 5~K. Although structural disorder influences the field-induced behavior of TmMgGaO$_4$, the magnetism in zero field appears relatively free from these effects. These results position TmMgGaO$_4$ as a strong candidate for a solid-state realization of KT physics in a dense spin system.