Gamma-ray constraints on the infrared background excess