Enhancement of three-mode optomechanical interaction by feedback-controlled light

We realise a feedback-controlled optical Fabry-Perot cavity in which the transmitted cavity output is used to modulate the input amplitude fluctuations. The resulting phase-dependent fluctuations of the in-loop optical field, which may be either sub-shot- or super-shot-noise, can be engineered to favorably affect the optomechanical interaction with a nanomechanical membrane placed within the cavity. Here we show that in the super-shot-noise regime ("anti-squashed light") the in-loop field has a strongly reduced effective cavity linewidth, corresponding to an increased optomechanical cooperativity. In this regime feedback improves the simultaneous resolved sideband cooling of two nearly degenerate membrane mechanical modes by one order of magnitude.