I agree 100% that you were seeing little perch after the bigger one hit. Once an area gets bombarded by little perch, the bigger ones stay below them and often get tight lipped.
Two things you can do that will make a difference.
1: Move. It doesn’t take a big move. There are bigger fish there. Just hop over 50′ and try again.
2: Try a slow falling minnow. I don’t know why it works, but it does. The little ones will watch it fall and it triggers the bigger fish. Put a crappie minnow on a light jig, or even a bare hook (without any additional weight) and let it slowly free fall on it’s own with an open bail. It’s painfully slow in deep water, but try it.
Watch your line as it’s falling. You might have 20 marks on the graph (mostly dinks), but if your line stops sinking before it hits the bottom, set the hook. Chances are it’s a bigger fish. If nothing hits, reel back up above the little fish until they stop chasing and drop it again.
I’m guessing you were on a lake that starts with a “B?”
Just letting a minnow sit under a float or jigging doesn’t work nearly as well. It has to be falling through the school. 60% of the time, it works every time.