Observation of the Three-Mode Parametric Instability

Three-mode parametric interactions occur in triply-resonant optomechanical systems: photons from an optical pump mode are coherently scattered to a high-order mode by mechanical motion of the cavity mirrors, and these modes resonantly interact via radiation pressure force when certain conditions are met. Such effects are predicted to occur in long baseline advanced gravitational-wave detectors. They can pump energy into acoustic modes, leading to parametric instability, but they can also extract acoustic energy, leading to optomechanical cooling. We develop a large amplitude model of three-mode interactions that explains the ring-up amplitude saturation after instability occurs. We also demonstrate both radiation-pressure cooling and mechanical amplification in two different three-mode optomechanical systems, including the first observation of the three-mode parametric instability in a free-space Fabry-Perot cavity. The experimental data agrees well with the theoretical model. Contrary to expectations, parametric instability does not lead to loss of cavity lock, a fact which may make it easier to implement control techniques to overcome instability.