That’s the way I do it too Bryan, if the impeller housing isn’t grooved or looks worn all a guy needs is the impeller, make sure the key is in the right place before and after you put the impeller on the shaft and maybe the lower seal on the bottom of the water pump housing.
While your in the lower unit check the oil to see if there’s water in the gear housing, clutch dog area. If there is, its time for a new rear prop shaft seal.