An insight in the surroundings of HR4796

HR4796 is a young, early A-type star harbouring a well structured debris disk, shaped as a ring with sharp inner edges. It forms with the M-type star HR4796B a binary system, with a proj. sep. ~560 AU. Our aim is to explore the surroundings of HR4796A and B, both in terms of extended or point-like structures. Adaptive optics images at L'-band were obtained with NaCo in Angular Differential Mode and with Sparse Aperture Masking (SAM). We analyse the data as well as the artefacts that can be produced by ADI reduction on an extended structure with a shape similar to that of HR4796A dust ring. We determine constraints on the presence of companions using SAM and ADI on HR4796A, and ADI on HR4796B. We also performed dynamical simulations of a disk of planetesimals and dust produced by collisions, perturbed by a planet located close to the disk outer edge. The disk ring around HR4796A is well resolved. We highlight the potential effects of ADI reduction of the observed disk shape and surface brightness distribution, and side-to-side asymmetries. No planet is detected around the star, with masses as low as 3.5 M_Jup at 0.5" (58 AU) and less than 3 M_Jup in the 0.8-1" range along the semi-major axis. We exclude massive brown dwarfs at separations as close as 60 mas (4.5 AU) from the star thanks to SAM data. The detection limits obtained allow us to exclude a possible close companion to HR4796A as the origin of the offset of the ring center with respect to the star; they also allow to put interesting constraints on the (mass, separation) of any planet possibly responsible for the inner disk steep edge. Using detailed dynamical simulations, we show that a giant planet orbiting outside the ring could sharpen the disk outer edge and reproduce the STIS images published by Schneider et al. (2009).