Localized Majorana-Like Modes in a Number-Conserving Setting: An Exactly Solvable Model