A large population of intermediate-mass black holes (IMBHs) might be produced at early cosmic times as a leftover of the evolution of the very massive first stars. Accretion on to IMBHs provides a source of (re)ionizing radiation. We show that the baryon mass fraction locked into IMBHs and their growth is strongly constrained by the observed residual soft X-ray background (SXRB) intensity. Thus, unless they are extremely X-ray quiet, miniquasars must be quite rare and/or have a short shining phase. As a byproduct, we find that miniquasars cannot be the only source of reionization and that their alleged contribution to the near-infrared bands is completely negligible.