Self-calibrating optical galaxy cluster selection bias using cluster, galaxy, and shear cross-correlations

The clustering signals of galaxy clusters are known to be powerful tools for self-calibrating the mass-observable relation and are complementary to cluster abundance and lensing. In this work, we explore the possibility of combining three correlation functions -- cluster lensing, the cluster-galaxy cross-correlation function, and the galaxy auto-correlation function -- to self-calibrate optical cluster selection bias, the boosted clustering and lensing signals in a richness-selected sample mainly caused by projection effects. We develop mock catalogues of redMaGiC-like galaxies and redMaPPer-like clusters by applying Halo Occupation Distribution (HOD) models to N-body simulations and using counts-in-cylinders around massive haloes as a richness proxy. In addition to the previously known small-scale boost in projected correlation functions, we find that the projection effects also significantly boost 3D correlation functions out to scales 100 $h^{-1} \mathrm{Mpc}$. We perform a likelihood analysis assuming survey conditions similar to that of the Dark Energy Survey (DES) and show that the selection bias can be self-consistently constrained at the 10% level. We discuss strategies for applying this approach to real data. We expect that expanding the analysis to smaller scales and using deeper lensing data would further improve the constraints on cluster selection bias.