Study of high-multiplicity three-prong and five-prong tau decays at BABAR