The problematic growth of dust in high-redshift galaxies